You are viewing the RapidMiner Hub documentation for version 2024.0 - Check here for latest version

Table of system settings for Job Agents

See also:

This table gives an overview over all Job Agent properties that can be set via environment variables. This application is a Spring Boot application and therefore those property values can be also applied, e.g. SPRING_APPLICATION_NAME=Test would change the Spring's application name to Test.

Other properties not listed here should not be changed and have reasonable defaults.

Property	Default	Description
`JOBAGENT_APP_ID`		A unique ID generated on first start and persisted in `$job-agent-home/config/.id.properties`. This ID file must be located on a persisted storage.
`JOBAGENT_BASE_DIR`	`/jobagent` (docker image)	The base directory of the Job Agent, other directory paths are derived from this base directory.
`JOBAGENT_HOME_DIR`	`$baseDir/home`	The Job Agent's home directory
`JOBAGENT_LOG_DIR`	`$baseDir/log`	The Job Agent's log directory
`JOBAGENT_DATA_DIR`	`$baseDir/data`	The Job Agent's data directory
`JOBAGENT_JOBS_DIR`	`$dataDir/jobs`	The Job Agent's jobs directory which stores information related to each job the Job Agent picks up
`JOBAGENT_CONTAINERS_DIR`	`$dataDir/containers`	The Job Agent's containers which stores information related to each spawned Job Container
`JOBAGENT_QUEUE_ACTIVEMQ_USERNAME`		Define broker user
`JOBAGENT_QUEUE_ACTIVEMQ_PASSWORD`		Define broker password
`JOBAGENT_AUTH_REALM`	`AIHub`	Realm to use for Keycloak
`JOBAGENT_AUTH_AUTH_SERVER_URL`		URL to Keycloak
`JOBAGENT_AUTH_SERVICE_CLIENT_ID`	`aihub-jobagent`	The configured Keycloak client
`JOBAGENT_AUTH_SERVICE_CLIENT_SECRET`		The secret for configured Keycloak client
`JOBAGENT_SYNC_ENABLED`	`true`	If synchronization behavior (retrieving EXTENSIONS, JDBC, LIBS, LICENSES and the EXECUTION_CONTEXT from main application) is enabled.
`JOBAGENT_SYNC_EXTENSIONS_ENABLED`	`true`	If synchronization behavior is enabled for EXTENSIONS.
`JOBAGENT_SYNC_EXTENSIONS_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for EXTENSIONS before fetching them from remote.
`JOBAGENT_SYNC_JDBC_ENABLED`	`true`	If synchronization behavior is enabled for JDBC.
`JOBAGENT_SYNC_JDBC_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for JDBC before fetching them from remote.
`JOBAGENT_SYNC_LIBS_ENABLED`	`true`	If synchronization behavior is enabled for LIBS.
`JOBAGENT_SYNC_LIBS_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for LIBS before fetching them from remote.
`JOBAGENT_SYNC_LICENSES_ENABLED`	`true`	If synchronization behavior is enabled for LICENSES.
`JOBAGENT_SYNC_LICENSES_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for LICENSES before fetching them from remote.
`JOBAGENT_SYNC_EXECUTION_CONTEXT_ENABLED`	`true`	If synchronization behavior is enabled for EXECUTION_CONTEXT.
`JOBAGENT_SYNC_EXECUTION_CONTEXT_CLEAN_BEFORE`	`true`	If synchronization behavior should clean local files for EXECUTION_CONTEXT before fetching them from remote.
`JOBAGENT_SYNC_BASE_URL`	`sync/`	The base URL for synchronization behavior.
`JOBAGENT_SYNC_HEALTH_CHECK_URL`	`$baseUrl/healthcheck`	The health check URL for synchronization behavior, derived from base URL.
`JOBAGENT_SHUTDOWN_CHECK_DELAY`	`1000`	Fixed delay in milliseconds for shutdown event checks
`JOBAGENT_RESTART_POLICY`	`NONE`	Defines when and how frequently underlying Job Containers, possible values are `NONE`, `CRON` and `JOBCOUNT`
`JOBAGENT_RESTART_MAX_JOBS`		Amount of executed jobs before Job Container is restarted, related to `JOBCOUNT`
`JOBAGENT_RESTART_CRON_EXPRESSION`		Cron expression to determine when Job Container is restarted in a scheduled manner, related to `CRON`
`JOBAGENT_RESTART_TIMEOUT`		Timeout for restarting a container. Job container is killed when the value is reached.
`JOBAGENT_CONTAINER_PROCESS_LOGGING_OVERRIDE_ENABLED`	`false`	If overriding process log level during execution is enabled or not
`JOBAGENT_CONTAINER_PROCESS_LOGGING_OVERRIDE_LEVEL`	`ALL`	If process log level overriding is enabled, this defines the verbosity, e.g. `ALL`
`JOBAGENT_CONTAINER_CONFIG_DIR`	`$job-agent-home-dir/config`	Config location for the Job Container
`JOBAGENT_CONTAINER_LIBS_DIR`	`$job-agent-base-dir/lib/container`	Library location for the Job Container used to determine what is loaded into the classpath of spawned Job Containers
`JOBAGENT_CONTAINER_JAR_DIR`	`$job-agent-base-dir/lib/container/rapidminer-jobcontainer.jar`	JAR file location to the Job Container
`JOBAGENT_CONTAINER_REPOSITORY_TYPE`	`remote`	Type of the internally loaded repository (remote) for projects. Should not be changed.
`JOBAGENT_CONTAINER_REPOSITORY_NAME`	`_LOCAL`	Name of the internally registered repository. Should not be changed.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_STRATEGY`	`ROTATE`	Defines the caching strategy of already seen Projects inside the Job Container. Possible values are `NONE` (no caching), `ROTATE` (evict latest used), `CRON` (depending on a fixed schedule), `IDLE` (depending on an idle time).
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_ROTATE_COUNT`	`2`	Used for `ROTATE` caching strategy and defines how many Projects are kept before the last used is evicted.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_CRON_EXPRESSION`		Used for `CRON` caching strategy and defines when the cache is evicted.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_IDLE_OFFSET`	`10000`	Used for `IDLE` caching strategy and defines the offset how long a repository has not been used in milliseconds.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_IDLE_INTERVAL`	`60`	Used for `IDLE` caching strategy and defines how frequently the Projects are checked in minutes.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_LOCK_BACKOFF`	`[1000, 2000, 3000, 5000, 8000, 16000]`	List of retry attempts for initially cloning the Project in milliseconds.
`JOBAGENT_CONTAINER_REPOSITORY_CACHING_USE_LFS_LIGHTWEIGHT`	`true`	If files of a Project should only be requested when they're actually required during process execution.
`JOBAGENT_CONTAINER_PLUGINS_DIR`	`$job-agent-base-dir/lib/container/plugins`	The Job Container's plugins directory used for bundled extensions
`JOBAGENT_CONTAINER_JDBC_DIR`	`$job-agent-home-dir/resources/jdbc`	The Job Container's JDBC directory
`JOBAGENT_CONTAINER_EXTENSIONS_DIR`	`$job-agent-home-dir/resources/extensions`	The Job Container's extensions directory
`JOBAGENT_CONTAINER_CUSTOM_LIBS_DIR`	`$job-agent-home-dir/resources/libs`	The Job Container's custom library directory
`JOBAGENT_CONTAINER_INIT_WITH_PROPERTIES_ENABLED`	`false`	Enable to make the Job Container read a dedicated file for initialization.
`JOBAGENT_CONTAINER_INIT_WITH_PROPERTIES_LOCATION`	`$job-agent-home-dir/config/rapidminer/rapidminer-init.properties`	Properties file used for Job Container initialization.
`JOBAGENT_CONTAINER_COUNT`	`1`	The Job Containers amount the Job Agent will spawn in parallel
`JOBAGENT_CONTAINER_MAX_ERROR_AMOUNT_BEFORE_SPAWN`	`5`	Number to determine how often a Job Container can not respond before the Job Agent will forcefully restart it.
`JOBAGENT_CONTAINER_MAX_TIME_BETWEEN_ERRORS`	`5000`	Maximum time (in milliseconds) between errors during the health check
`JOBAGENT_CONTAINER_LISTEN_SCHEME`	`http`	Listen scheme of the Job Containers spawned by the Job Agent
`JOBAGENT_CONTAINER_LISTEN_ADDRESS`	`localhost`	Listen address of the Job Containers spawned by the Job Agent
`JOBAGENT_CONTAINER_LISTEN_PORT_RANGE_START`	`10000`	Listen port range of the Job Containers spawned by the Job Agent, are incremented by one for each additional Job Container, see `COUNT` property
`JOBAGENT_CONTAINER_STARTUP_TIMEOUT`	`180000`	Maximum time to wait (in milliseconds) before a Job Container is marked as not started successfully
`JOBAGENT_CONTAINER_MEMORY_LIMIT`	`2048`	Memory in MB limit for each Job Container
`JOBAGENT_CONTAINER_USE_SYSTEM_TEMP_DIR`	`false`	By default, the Job Container's temporary directory is located inside it's working directory and does not use the system's temporary folder
`JOBAGENT_CONTAINER_JVM_PROPERTIES_DEFAULTS`	`Xms521m,XX:+UseG1GC,XX:G1HeapRegionSize=32m,XX:-HeapDumpOnOutOfMemoryError,Djava.awt.headless=true,Dlog4j2.formatMsgNoLookups=true`	List of default JVM arguments passed to each Job Container. For additions, use `JVM_CUSTOM_OPTIONS`.
`JOBAGENT_CONTAINER_JVM_CUSTOM_OPTIONS`		A simple line to add new JVM arguments passed to each Job Container, e.g. `-Dmyproperty=value`. It's recommended to use this property to transpose additional arguments to the Job Container.
`JOBAGENT_CONTAINER_JVM_CUSTOM_PROPERTIES`		List of custom JVM arguments passed to each Job Container. It's recommended to use `JVM_CUSTOM_OPTIONS` instead.
`JOBAGENT_CONTAINER_JVM_PROPERTIES`		Overwrite the default JVM properties
`JOBAGENT_CONTAINER_REST_TEMPLATE_BACK_OFF`	`[5000, 10000, 20000]`	The backoff used for communication with the job container
`JOBAGENT_CONTAINER_KILL_FORCIBLY_TIMEOUT`	`5000`	Timeout after a process is killed forcibly if it is not dead yet after receiving the destroy command
`JOBAGENT_JOB_STATE_EVENTS_CLEANUP`	`true`	If old events which have already been marked as sent to the main application, should be cleaned up.
`JOBAGENT_JOB_STATE_EVENTS_INTERVAL`	`60000`	Interval in which candidates of already sent events are checked for deletion.
`JOBAGENT_PLATFORM_STD_REDIRECT_ERROR_ENABLED`	`false`	If logging of the stderr for the Job Container is enabled or not.
`JOBAGENT_PLATFORM_STD_REDIRECT_ERROR_PARENT_DIR`	`$job-agent-base-dir/home/log`	The Job Container's stderr directory
`JOBAGENT_PLATFORM_STD_REDIRECT_OUT_ENABLED`	`false`	If logging of the stdout for the Job Container is enabled or not.
`JOBAGENT_PLATFORM_STD_REDIRECT_OUT_PARENT_DIR`	`$job-agent-base-dir/home/log`	The Job Container's stdout directory
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_STARTING`	`1000`	Time in milliseconds between invoking check for starting Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_RUNNING`	`1000`	Time in milliseconds between invoking check for running Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_EXECUTING`	`500`	Time in milliseconds between invoking check for executing Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_CAPACITY`	`1000`	Time in milliseconds between invoking capacity check (available Job Containers)
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_UNREACHABLE`	`1000`	Time in milliseconds between invoking check for unreachable Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_SPAWN`	`1000`	Time in milliseconds between invoking check for spawning new Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_RESTART`	`1000`	Time in milliseconds between invoking check for restarting Job Containers
`JOBAGENT_SCHEDULED_JOB_CONTAINER_CHECK_TOKEN`	`5000`	Time in milliseconds between invoking check token validity of Job Containers
`JOBAGENT_SCHEDULED_JOB_STATUS_UPDATE`	`2000`	Time in milliseconds between invoking REST request to Job Containers to fetch the progress of execution
`JOBAGENT_SCHEDULED_AGENT_STATE_UPDATE`	`1000`	Time in milliseconds between invoking check of unsent events for jobs and to send them back to the main application
`JOBAGENT_SCHEDULED_AGENT_STATUS_UPDATE`	`10000`	Time in milliseconds between invoking check of Job Agent's state and sending it back to the main application (heartbeat)
`JOBAGENT_SCHEDULED_AGENT_SYNC_REFETCH`	`5000`	Time in milliseconds between invoking check of a newly arrived sync fetch message
`JOBAGENT_SCHEDULED_JOB_TOKEN_UPDATE`	`240000`	Time in milliseconds between invoking renewal of a token
`JOBAGENT_SCHEDULED_JOB_STATE_UPDATE`	`2000`	Time in milliseconds between invoking check of unsent events for jobs and to send them back to the main application
`JOBAGENT_QUEUE_AGENT_COMMAND`	`__agentCommand`	ActiveMQ queue for agent commands
`JOBAGENT_QUEUE_AGENT_HEARTBEAT`	`__agentHeartbeat`	ActiveMQ queue for agent heartbeats
`JOBAGENT_QUEUE_JOB_REQUEST`	`DEFAULT`	ActiveMQ queue for job requests
`JOBAGENT_QUEUE_JOB_LOG`	`__jobLog`	ActiveMQ queue for log requests
`JOBAGENT_QUEUE_JOB_STATUS`	`__jobBatchUpdate`	ActiveMQ queue for job status updated
`JOBAGENT_QUEUE_SYNC_COMMAND`	`__agentCommand`	ActiveMQ queue for sync commands
`JOBAGENT_QUEUE_JOB_BLOCK`	`__repositoryBlock`	ActiveMQ persistence topic for blocking job execution.
`JOBAGENT_QUEUE_RECEIVE_TIMEOUT`	`5000`	ActiveMQ queue receive timeout for JMS listeners
`JOBAGENT_QUEUE_CACHE_LEVEL`	`4`	ActiveMQ queue cache level for JMS listeners
`JOBAGENT_QUEUE_TASK_EXECUTOR_COUNT`	`3`	ActiveMQ queue JMS listener threads
`JOBAGENT_QUEUE_ACTIVEMQ_MAXIMUM_JOB_REDELIVERIES`	`2`	ActiveMQ queue JMS listener per job maximum redelivery attempts
`JOBAGENT_AUDIT_ENABLED`	`false`	If audit is enabled to see additional information about user changes
`JOBAGENT_LICENSE_AGENT_PROXY_URL`		The URL the License Agent will use to report to
`JOBAGENT_LICENSE_AGENT_PROXY_CONNECTION_TIMEOUT`	`120`	The connection timeout to the proxy used by the License Agent
`JOBAGENT_LICENSE_AGENT_PROXY_RESPONSE_TIMEOUT`	`120`	The response timeout to the proxy used by the License Agent
`JOBAGENT_LICENSE_AGENT_MACHINE_ID`		The applications machine id, must be unique across different host machines, but can be identical for applications running on the same host
`JOBAGENT_LICENSE_AGENT_APPLICATION_NAME`	`Job Agent`	The application name the License Agent will report
`JOBAGENT_LICENSE_AGENT_POOL_SIZE`	`5`	The executor threads used by the License Agent to invoke heartbeats
`LICENSE_MODE`	`ALTAIR_UNIT`	Enabled by default to use Altair Unit licensing. Requires `_LICENSE_AGENT_`and `_AUTH_` settings. Setting `RAPIDMINER` falls back to RapidMiner licensing
`LICENSE_LICENSE`		Directly set a RapidMiner license string
`LICENSE_LOCATION`		By default, points to the parent folder where licenses reside in, e.g. `$appHome/resources/licenses` inside the Docker container
`LOGGING_...`		Set logging, e.g. `LOGGING_LEVEL_COM_RAPIDMINER=DEBUG` for debug logs of all Altair RapidMiner classes
`BROKER_ACTIVEMQ_HOST`	`localhost`	Define broker host
`BROKER_ACTIVEMQ_PORT`	`61616`	Define broker port
`BROKER_ACTIVEMQ_USERNAME`		Define broker user
`BROKER_ACTIVEMQ_PASSWORD`		Define broker password
`BROKER_ACTIVEMQ_URI`		Define broker URI
`AIHUB_CONNECTION_PROTOCOL`	`http`	Define AI Hub protocol for internal or external connectivity
`AIHUB_CONNECTION_HOST`	`localhost`	Define AI Hub host for internal or external connectivity
`AIHUB_CONNECTION_PORT`	`8080`	Define AI Hub port for internal or external connectivity
`AIHUB_CONNECTION_CONTEXT_PATH`	`api/v1/`	Define AI Hub context path for internal or external connectivity

Attributes available during execution in a Job Agent / Job Container

rapidminer.aihub.connection.protocol=
rapidminer.aihub.connection.host=
rapidminer.aihub.connection.port=
rapidminer.aihub.connection.context-path=
rapidminer.aihub.process.location=
com.rapidminer.auth.jwt.token=
com.rapidminer.aihub.user_id=

Job Container

The Job Container properties are handled in a special way, because they are passed to the Job Container by the Job Agent during start.

All properties prefixed with JOBAGENT_CONTAINER_ are automatically transposed to the Job Container, some are transposed automatically. If you need to pass advanced properties not transposed by the Job Agent itself, please use JOBAGENT_CONTAINER_JVM_CUSTOM_OPTIONS with one of the following properties:

Property	Default	Description
`jobcontainer.process.logging.override.enabled`	`false`	Allows to override process logging
`jobcontainer.process.logging.override.level`	`ALL`	Set the override log level of the process which ignored the `log_vebosity` parameter defined in the process
`jobcontainer.system-out-log.enabled`	`true`	Allow to log `System.out` calls
`jobcontainer.system-out-log.out-level`	`info`	The level of out messages
`jobcontainer.system-out-log.err-level`	`error`	The level of error messages
`jobcontainer.register-default-symmetric-encryption-context.enabled`	`true`	Enabled registration of Project encryption context
`jobcontainer.register-default-symmetric-encryption-context.key-location`	`${container-dir}/.RapidMiner/encryption/symmetric/default-local-context.rmek`	Default location of context's key
`jobcontainer.jobs-cache-size`	`10`	This amount of jobs will be kept in cache also after they finished
`jobcontainer.progress-update`	`1000`	Update process trace every second
`jobcontainer.init-with-properties.enabled`	`false`	If Altair RapidMiner should be initialized with additional init properties on Job Container start
`jobcontainer.init-with-properties.location`		Used properties file for init, when enabled
`jobcontainer.remote.versioned.backoff`	`[1000, 2000, 3000, 5000, 8000, 16000]`	Values in milliseconds how long to wait before each try to commit after a job finished
`jobcontainer.remote.versioned.auto-commit-strategy`	`OURS`	Possible values: `OURS` (keep changes done in the Job Container and merge them`,`NONE` to skip auto-commit and evict any change done locally (for testing)
`jobcontainer.task-scheduler.pool-size`	`10`	Job Container's scheduler used for periodic checks pool size
`jobcontainer.task-scheduler.thread-priority`	`8`	Job Container's scheduler used for periodic checks pool size
`jobcontainer.memory-check-delay`	`5000`	Time in milliseconds how frequently to check for out of memory of the Job Container
`jobcontainer.memory-consumption-threshold`	`0.95`	Threshold when to consider Job Container out of memory in decimal percent
`jobcontainer.repository.type`	`remote`	See Job Agent's explanation.
`jobcontainer.repository.caching.strategy`	`ROTATE`	See Job Agent's explanation.
`jobcontainer.repository.caching.rotate-count`	`2`	See Job Agent's explanation.
`jobcontainer.repository.caching.lock-backoff`	`[1000, 2000, 3000, 5000, 8000, 16000]`	See Job Agent's explanation.
`jobcontainer.load-user-certificates`	`false`	If user certificates should be loaded
`audit.enabled`	`false`	If audit is enabled. See Job Agent's explanation.
`logging.*`		Set logging, e.g. `logging.level.com.rapidminer.execution.jobcontainer=DEBUG` for debug logs of the Job Container classes itself

Categories

Versions

Table of system settings for Job Agents

Job Container