You are viewing the RapidMiner Hub documentation for version 10.0 - Check here for latest version

Table of system settings for RapidMiner Job Agent

This table gives an overview over all properties which can be set via environment variables. This application is a Spring Boot application and therefore those property values can be also applied, e.g. SPRING_APPLICATION_NAME=Test would change the Spring's application name to Test.

Other properties not listed here should not be changed and have reasonable defaults.

Property	Default	Description
`JOBAGENT_APP_ID`		A unique ID generated on first start and persisted in `$job-agent-home/config/.id.properties`. This ID file must be located on a persisted storage.
`JOBAGENT_BASE_DIR`	`/jobagent` (docker image)	The base directory of the Job Agent, other directory paths are derived from this base directory.
`JOBAGENT_HOME_DIR`	`$baseDir/home`	The Job Agent's home directory
`JOBAGENT_LOG_DIR`	`$baseDir/log`	The Job Agent's log directory
`JOBAGENT_DATA_DIR`	`$baseDir/data`	The Job Agent's data directory
`JOBAGENT_JOBS_DIR`	`$dataDir/jobs`	The Job Agent's jobs directory which stores information related to each job the Job Agent picks up
`JOBAGENT_CONTAINERS_DIR`	`$dataDir/containers`	The Job Agent's containers which stores information related to each spawned Job Container
`JOBAGENT_QUEUE_ACTIVEMQ_USERNAME`		Define broker user
`JOBAGENT_QUEUE_ACTIVEMQ_PASSWORD`		Define broker password
`JOBAGENT_AUTH_REALM`	`AIHub`	Realm to use for Keycloak
`JOBAGENT_AUTH_AUTH_SERVER_URL`		URL to Keycloak
`JOBAGENT_AUTH_SERVICE_CLIENT_SECRET`		The secret for `aihub-jobagent` Keycloak client
`JOBAGENT_SYNC_ENABLED`	`true`	If synchronization behavior (retrieving EXTENSIONS, JDBC, LIBS, LICENSES and the EXECUTION_CONTEXT from main application) is enabled.
`JOBAGENT_SYNC_EXTENSIONS_ENABLED`	`true`	If synchronization behavior is enabled for EXTENSIONS.
`JOBAGENT_SYNC_EXTENSIONS_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for EXTENSIONS before fetching them from remote.
`JOBAGENT_SYNC_JDBC_ENABLED`	`true`	If synchronization behavior is enabled for JDBC.
`JOBAGENT_SYNC_JDBC_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for JDBC before fetching them from remote.
`JOBAGENT_SYNC_LIBS_ENABLED`	`true`	If synchronization behavior is enabled for LIBS.
`JOBAGENT_SYNC_LIBS_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for LIBS before fetching them from remote.
`JOBAGENT_SYNC_LICENSES_ENABLED`	`true`	If synchronization behavior is enabled for LICENSES.
`JOBAGENT_SYNC_LICENSES_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for LICENSES before fetching them from remote.
`JOBAGENT_SYNC_EXECUTION_CONTEXT_ENABLED`	`true`	If synchronization behavior is enabled for EXECUTION_CONTEXT.
`JOBAGENT_SYNC_EXECUTION_CONTEXT_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for EXECUTION_CONTEXT before fetching them from remote.
`JOBAGENT_SYNC_BASE_URL`	`sync/`	The base URL for synchronization behavior.
`JOBAGENT_SYNC_HEALTH_CHECK_URL`	`$baseUrl/healthcheck`	The health check URL for synchronization behavior, derived from base URL.
`JOBAGENT_SHUTDOWN_CHECK_DELAY`	`1000`	Fixed delay in milliseconds for shutdown event checks
`JOBAGENT_RESTART_POLICY`	`NONE`	Defines when and how frequently underlying Job Containers, possible values are `NONE`, `CRON` and `JOBCOUNT`
`JOBAGENT_RESTART_MAX_JOBS`		Amount of executed jobs before Job Container is restarted, related to `JOBCOUNT`
`JOBAGENT_RESTART_CRON_EXPRESSION`		Cron expression to determine when Job Container is restarted in a scheduled manner, related to `CRON`
`JOBAGENT_RESTART_TIMEOUT`		Timeout for restarting a container. Job container is killed when the value is reached.
`JOBAGENT_CONTAINER_PROCESS_LOGGING_OVERRIDE_ENABLED`	`false`	If overriding process log level during execution is enabled or not
`JOBAGENT_CONTAINER_PROCESS_LOGGING_OVERRIDE_LEVEL`	`ALL`	If process log level overriding is enabled, this defines the verbosity, e.g. `ALL`
`JOBAGENT_CONTAINER_CONFIG_DIR`	`$job-agent-home-dir/config`	Config location for the Job Container
`JOBAGENT_CONTAINER_LIBS_DIR`	`$job-agent-base-dir/lib/container`	Library location for the Job Container used to determine what is loaded into the classpath of spawned Job Containers
`JOBAGENT_CONTAINER_JAR_DIR`	`$job-agent-base-dir/lib/container/rapidminer-jobcontainer.jar`	JAR file location to the Job Container
`JOBAGENT_CONTAINER_REPOSITORY_TYPE`	`remote`	Type of the internally loaded repository (remote) for projects. Should not be changed.
`JOBAGENT_CONTAINER_REPOSITORY_NAME`	`_LOCAL`	Name of the internally registered repository. Should not be changed.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_STRATEGY`	`ROTATE`	Defines the caching strategy of already seen Projects inside the Job Container. Possible values are `NONE` (no caching), `ROTATE` (evict latest used), `CRON` (depending on a fixed schedule), `IDLE` (depending on an idle time).
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_ROTATE_COUNT`	`2`	Used for `ROTATE` caching strategy and defines how many Projects are kept before the last used is evicted.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_CRON_EXPRESSION`		Used for `CRON` caching strategy and defines when the cache is evicted.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_IDLE_OFFSET`	`10000`	Used for `IDLE` caching strategy and defines the offset how long a repository has not been used in milliseconds.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_IDLE_INTERVAL`	`60`	Used for `IDLE` caching strategy and defines how frequently the Projects are checked in minutes.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_LOCK_BACKOFF`	`[1000, 2000, 3000, 5000, 8000, 16000]`	List of retry attempts for initially cloning the Project in milliseconds.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_USE_LFS_LIGHTWEIGHT`	`true`	If files of a Project should only be requested when they're actually required during process execution.
`JOBAGENT_CONTAINER_PLUGINS_DIR`	`$job-agent-base-dir/lib/container/plugins`	The Job Container's plugins directory used for internal RapidMiner bundled extensions
`JOBAGENT_CONTAINER_JDBC_DIR`	`$job-agent-home-dir/resources/jdbc`	The Job Container's JDBC directory
`JOBAGENT_CONTAINER_EXTENSIONS_DIR`	`$job-agent-home-dir/resources/extensions`	The Job Container's extensions directory
`JOBAGENT_CONTAINER_CUSTOM_LIBS_DIR`	`$job-agent-home-dir/resources/libs`	The Job Container's custom library directory
`JOBAGENT_CONTAINER_INIT_WITH_PROPERTIES_ENABLED`	`false`	Enable to make the Job Container read a dedicated file for initialization.
`JOBAGENT_CONTAINER_INIT_WITH_PROPERTIES_LOCATION`	`$job-agent-home-dir/config/rapidminer/rapidminer-init.properties`	Properties file used for Job Container initialization.
`JOBAGENT_CONTAINER_COUNT`	`1`	The Job Containers amount the Job Agent will spawn in parallel
`JOBAGENT_CONTAINER_MAX_ERROR_AMOUNT_BEFORE_SPAWN`	`5`	Number to determine how often a Job Container can not respond before the Job Agent will forcefully restart it.
`JOBAGENT_CONTAINER_MAX_TIME_BETWEEN_ERRORS`	`5000`	Maximum time (in milliseconds) between errors during the health check
`JOBAGENT_CONTAINER_LISTEN_SCHEME`	`http`	Listen scheme of the Job Containers spawned by the Job Agent
`JOBAGENT_CONTAINER_LISTEN_ADDRESS`	`localhost`	Listen address of the Job Containers spawned by the Job Agent
`JOBAGENT_CONTAINER_LISTEN_PORT_RANGE_START`	`10000`	Listen port range of the Job Containers spawned by the Job Agent, are incremented by one for each additional Job Container, see `COUNT` property
`JOBAGENT_CONTAINER_STARTUP_TIMEOUT`	`180000`	Maximum time to wait (in milliseconds) before a Job Container is marked as not started successfully
`JOBAGENT_CONTAINER_MEMORY_LIMIT`	`2048`	Memory in MB limit for each Job Container
`JOBAGENT_CONTAINER_USE_SYSTEM_TEMP_DIR`	`false`	By default, the Job Container's temporary directory is located inside it's working directory and does not use the system's temporary folder
`JOBAGENT_CONTAINER_JVM_PROPERTIES_DEFAULTS`	`Xms521m,XX:+UseG1GC,XX:G1HeapRegionSize=32m,XX:-HeapDumpOnOutOfMemoryError,Djava.awt.headless=true,Dlog4j2.formatMsgNoLookups=true`	List of default JVM arguments passed to each Job Container. For additions, use `JVM_CUSTOM_OPTIONS`.
`JOBAGENT_CONTAINER_JVM_CUSTOM_OPTIONS`		A simple line to add new JVM arguments passed to each Job Container, e.g. `-Dmyproperty=value`. It's recommended to use this property to transpose additional arguments to the Job Container.
`JOBAGENT_CONTAINER_JVM_CUSTOM_PROPERTIES`		List of custom JVM arguments passed to each Job Container. It's recommended to use `JVM_CUSTOM_OPTIONS` instead.
`JOBAGENT_CONTAINER_JVM_PROPERTIES`		Overwrite the default JVM properties
`JOBAGENT_CONTAINER_KILL_FORCIBLY_TIMEOUT`	`5000`	Timeout after a process is killed forcibly if it is not dead yet after receiving the destroy command
`JOBAGENT_JOB_STATE_EVENTS_CLEANUP`	`false`	If old events which have already been marked as sent to the main application, should be cleaned up.
`JOBAGENT_JOB_STATE_EVENTS_INTERVAL`	`false`	Interval in which candidates of already sent events are checked for deletion.
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_STARTING`	`1000`	Time in milliseconds between invoking check for starting Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_RUNNING`	`1000`	Time in milliseconds between invoking check for running Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_EXECUTING`	`1000`	Time in milliseconds between invoking check for executing Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_CAPACITY`	`1000`	Time in milliseconds between invoking capacity check (available Job Containers)
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_UNREACHABLE`	`1000`	Time in milliseconds between invoking check for unreachable Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_SPAWN`	`1000`	Time in milliseconds between invoking check for spawning new Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_RESTART`	`1000`	Time in milliseconds between invoking check for restarting Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_TOKEN`	`5000`	Time in milliseconds between invoking check token validity of Job Containers
`JOBAGENT_SCHEDULED_JOB_STATUS_UPDATE`	`2000`	Time in milliseconds between invoking REST request to Job Containers to fetch the progress of execution
`JOBAGENT_SCHEDULED_AGENT_STATE_UPDATE`	`1000`	Time in milliseconds between invoking check of unsent events for jobs and to send them back to the main application
`JOBAGENT_SCHEDULED_AGENT_STATUS_UPDATE`	`10000`	Time in milliseconds between invoking check of Job Agent's state and sending it back to the main application (heartbeat)
`JOBAGENT_SCHEDULED_AGENT_SYNC_REFETCH`	`5000`	Time in milliseconds between invoking check of a newly arrived sync fetch message
`JOBAGENT_SCHEDULED_JOB_TOKEN_UPDATE`	`240000`	Time in milliseconds between invoking renewal of a token
`JOBAGENT_SCHEDULED_JOB_STATE_UPDATE`	`2000`	Time in milliseconds between invoking check of unsent events for jobs and to send them back to the main application
`JOBAGENT_QUEUE_AGENT_COMMAND`	`__agentCommand`	ActiveMQ queue for agent commands
`JOBAGENT_QUEUE_AGENT_HEARTBEAT`	`__agentHeartbeat`	ActiveMQ queue for agent heartbeats
`JOBAGENT_QUEUE_JOB_REQUEST`	`DEFAULT`	ActiveMQ queue for job requests
`JOBAGENT_QUEUE_JOB_LOG`	`__jobLog`	ActiveMQ queue for log requests
`JOBAGENT_QUEUE_JOB_STATUS`	`__jobUpdate?consumer.exclusive=true`	ActiveMQ queue for job status updated
`JOBAGENT_QUEUE_SYNC_COMMAND`	`__agentCommand`	ActiveMQ queue for sync commands
`JOBAGENT_QUEUE_RECEIVE_TIMEOUT`	`5000`	ActiveMQ queue receive timeout for JMS listeners
`JOBAGENT_QUEUE_CACHE_LEVEL`	`4`	ActiveMQ queue cache level for JMS listeners
`JOBAGENT_QUEUE_TASK_EXECUTOR_COUNT`	`3`	ActiveMQ queue JMS listener threads
`JOBAGENT_QUEUE_ACTIVEMQ_MAXIMUM_REDELIVERIES`	`2`	ActiveMQ queue JMS listener maximum redelivery attempts
`JOBAGENT_QUEUE_ACTIVEMQ_MAXIMUM_JOB_REDELIVERIES`	`2`	ActiveMQ queue JMS listener per job maximum redelivery attempts
`JOBAGENT_AUDIT_ENABLED`	`false`	If audit is enabled to see additional information about user changes
`LICENSE_LICENSE`		Directly set a RapidMiner license string
`LICENSE_LOCATION`		By default, points to the parent folder where licenses reside in, e.g. `$appHome/resources/licenses` inside the Docker container
`LOGGING_...`		Set logging, e.g. `LOGGING_LEVEL_COM_RAPIDMINER=DEBUG` for debug logs of all RapidMiner classes
`BROKER_ACTIVEMQ_HOST`	`localhost`	Define broker host
`BROKER_ACTIVEMQ_PORT`	`61616`	Define broker port
`BROKER_ACTIVEMQ_USERNAME`		Define broker user
`BROKER_ACTIVEMQ_PASSWORD`		Define broker password
`BROKER_ACTIVEMQ_URI`		Define broker URI
`AIHUB_CONNECTION_PROTOCOL`	`http`	Define AI Hub protocol for internal or external connectivity
`AIHUB_CONNECTION_HOST`	`localhost`	Define AI Hub host for internal or external connectivity
`AIHUB_CONNECTION_PORT`	`8080`	Define AI Hub port for internal or external connectivity
`AIHUB_CONNECTION_CONTEXT_PATH`	`api/v1/`	Define AI Hub context path for internal or external connectivity

Attributes available during execution in a Job Agent / Job Container

rapidminer.aihub.connection.protocol=
rapidminer.aihub.connection.host=
rapidminer.aihub.connection.port=
rapidminer.aihub.connection.context-path=
rapidminer.aihub.process.location=
com.rapidminer.auth.jwt.token=
com.rapidminer.aihub.user_id=

Job Container

The Job Container properties are handled in a special way, because they are passed to the Job Container by the Job Agent during start.

All properties prefixed with JOBAGENT_CONTAINER_ are automatically transposed to the Job Container, some are transposed automatically. If you need to pass advanced properties not transposed by the Job Agent itself, please use JOBAGENT_CONTAINER_JVM_CUSTOM_OPTIONS with one of the following properties:

Property	Default	Description
`jobcontainer.process.logging.override.enabled`	`false`	Allows to override process logging
`jobcontainer.process.logging.override.level`	`ALL`	Set the override log level of the process which ignored the `log_vebosity` parameter defined in the process
`jobcontainer.system-out-log.enabled`	`false`	Allow to log `System.out` calls
`jobcontainer.register-default-symmetric-encryption-context.enabled`	`true`	Enabled registration of Project encryption context
`jobcontainer.register-default-symmetric-encryption-context.key-location`	`${container-dir}/.RapidMiner/encryption/symmetric/default-local-context.rmek`	Default location of context's key
`jobcontainer.jobs-cache-size`	`10`	This amount of jobs will be kept in cache also after they finished
`jobcontainer.progress-update`	`1000`	Update process trace every second
`jobcontainer.init-with-properties.enabled`	`false`	If RapidMiner should be initialized with additional init properties on Job Container start
`jobcontainer.init-with-properties.location`		Used properties file for init, when enabled
`jobcontainer.remote.versioned.backoff`	`[1000, 2000, 3000, 5000, 8000, 16000]`	Values in milliseconds how long to wait before each try to commit after a job finished
`jobcontainer.remote.versioned.auto-commit-strategy`	`OURS`	Possible values: `OURS` (keep changes done in the Job Container and merge them`,`NONE` to skip auto-commit and evict any change done locally (for testing)
`jobcontainer.task-scheduler.pool-size`	`10`	Job Container's scheduler used for periodic checks pool size
`jobcontainer.task-scheduler.thread-priority`	`8`	Job Container's scheduler used for periodic checks pool size
`jobcontainer.memory-check-delay`	`5000`	Time in milliseconds how frequently to check for out of memory of the Job Container
`jobcontainer.memory-consumption-threshold`	`0.95`	Threshold when to consider Job Container out of memory in decimal percent
`jobcontainer.repository.type`	`remote`	See Job Agent's explanation.
`jobcontainer.repository.caching.strategy`	`ROTATE`	See Job Agent's explanation.
`jobcontainer.repository.caching.rotate-count`	`2`	See Job Agent's explanation.
`jobcontainer.repository.caching.lock-backoff`	`[1000, 2000, 3000, 5000, 8000, 16000]`	See Job Agent's explanation.
`audit.enabled`	`false`	If audit is enabled. See Job Agent's explanation.
`logging.*`		Set logging, e.g. `logging.level.com.rapidminer.execution.jobcontainer=DEBUG` for debug logs of the Job Container classes itself

Categories

Versions

Table of system settings for RapidMiner Job Agent

Job Container