You are viewing the RapidMiner Hub documentation for version 10.0 - Check here for latest version
Project Versioning
AI Hub data storage is backed by integrated Git and Large File System (LFS) servers for keeping all data of your Projects.
The integrated git server is used to keep track of smaller files and the integrated LFS server is responsible for
handling larger files. By default, and if LFS is enabled for a Project, files
with the extensions .ioo, .rmhdf5table, .collection and .conninfo are stored in the LFS server.
It’s recommended to always store binary data like Excel sheets or pictures inside LFS and have LFS enabled for all Projects by default!
You can define additional file extensions which will be tracked by adapting the .gitattributes file
in a Project.
Storage backend
The integrated git and LFS server store data inside the RapidMiner AI Hub home directory and
all files reside in $persistent-rapidminer-home/data/repositories/git_server and persistent-rapidminer-home/data/repositories/git_lfs_server
respectively. In git terminology, the git data is stored inside bare git repositories. In the integrated LFS server,
file names always match their respective SHA-256 checksum.
Advanced configuration for upload, disk space availability and consistency checks
The integrated git and LFS server which store their data inside the RapidMiner AI Hub home directory depend on enough disk space being available.
In order to avoid corrupted files after upload, they require a certain amount of disk space being available regardless of the size of the uploaded files. In addition to that, when large files are uploaded to a Project, their expected size and SHA-256 checksum is verified by the integrated LFS server.
The following table outlines important properties which can be changed for disk space and consistency checks with environment variables.
| Property | Description | Availability |
|---|---|---|
REPOSITORIES_MAX_UPLOAD_SIZE |
By default, the LFS server only allows to upload files smaller than 5 Gigabytes. Identifiers like Gb or Mb for gigabytes and megabytes are supported. |
Any version supporting Projects |
REPOSITORIES_GIT_ENABLE_DISKSPACE_CHECK_HOOK |
Verifies that at least REPOSITORIES_GIT_DISKSPACE_CHECK_THRESHOLD is available inside the RapidMiner AI Hub home directory. |
>= 9.10.4 |
REPOSITORIES_GIT_DISKSPACE_CHECK_THRESHOLD |
Defaults to 5120M. Identifiers like G or M for gigabytes and megabytes are supported. |
>= 9.10.4 |
REPOSITORIES_LFS_ENABLE_DISKSPACE_CHECK |
Verifies that at least REPOSITORIES_MIN_LFS_DISKSPACE_CHECK_THRESHOLD is available inside the RapidMiner AI Hub home directory. |
>= 9.10.4 |
REPOSITORIES_MIN_LFS_DISKSPACE_CHECK_THRESHOLD |
Defaults to 5120M and is doubled when REPOSITORIES_LFS_REMOVE_UNSUCCESSFUL_UPLOADS is enabled. Identifiers like G or M for gigabytes and megabytes are supported. |
>= 9.10.4 |
REPOSITORIES_LFS_REMOVE_UNSUCCESSFUL_UPLOADS |
Defaults to true. When consistency checks fails during upload (checksum or size), those files will be directly removed afterwards to avoid keeping failed uploads. |
>= 9.10.4 |
REPOSITORIES_LFS_ENBLE_UPLOAD_SIZE_CHECK |
Defaults to true. Enables check of LFS files being uploaded. |
>= 9.10.4 |
REPOSITORIES_LFS_ENBLE_UPLOAD_CHECKSUM_CHECK |
Defaults to true. Enables checksum verification of LFS files being uploaded. |
>= 9.10.4 |