Scalable architecture

To build a RapidMiner Server environment for your data science team, two components have to be installed:

plus the following helper applications:

  • RapidMiner Studio - to design the processes you will run in the RapidMiner Server environment
  • A database - to store configuration files, cron job details, user report requests, and other support data

The following optional component can be installed, and it runs independently:

A simple schematic is shown below.

The design

The design of the RapidMiner Server environment reflects a typical data science workflow, where there are two kinds of activities:

  1. Model building, involving long-running processes that can be placed on a queue and run asynchronously

    RapidMiner Server offers a queue system for long-running jobs, which are executed externally via Job Agents. You increase processing power by adding Job Agents.

  2. Prediction, or any other application of the models, where the need for a real-time response is paramount

    There is one engine for generating predictions:

    You increase processing power by adding Real-Time Scoring Agents.

RapidMiner Server

RapidMiner Server is the central component in the architecture. You interact with it via a web interface or via RapidMiner Studio. Its main responsibilities are:

Read more: Install RapidMiner Server

In the diagram below, each blue box represents a separate machine. RapidMiner Server is installed on the big blue box on the left, while the blue boxes on the right host remote Job Agents.

Job Agent

The design with Job Agents running remotely on dedicated machines is aimed at scalability.

Each Job Agent is configured to point to one of the queues on RapidMiner Server. Its only responsibility is to pick up jobs from the queue and run them. When started, the Job Agent spawns a configurable amount of Job Containers as separate system processes. Jobs are then redirected to those Job Containers and RapidMiner processes are being executed within the spawned Job Containers. When the Job Agent is shut down, all Job Containers of this Job Agent are also shut down. For each Job Agent, the number of Job Containers that can be spawned and the available memory is configurable.

Multiple Job Agents can point to the same queue. You can manage the queues, and therefore the allocation of resources, by assigning permissions.

Read more: configure Job Agents, configure Job Containers

Job Container

The Job Container spawned by the Job Agent runs a RapidMiner Studio instance that is capable of executing a RapidMiner process. It's bound to a system port in order to accept jobs from the Job Agent via REST API.

By default, the Job Container will not terminate after each process, therefore providing a nearly instant execution of processes.

It might be desirable to run each job in its own sandbox so that the system is more robust and jobs won't have any effect on previously run jobs. This behavior can also be configured, e.g. to restart the Job Container after a process finished. Please refer to the Job Agent's configuration page for further details.

The price of this safety is latency -- the latency in restarting the Job Container is measured in seconds. Each job will then have this additional latency.

If a real-time response is paramount, we recommend using a web service or a Real-Time Scoring Agent. For example, you might build a model in a Job Container and generate predictions for that model via a Real-Time Scoring Agent.

Real-Time Scoring Agent

As mentioned previously, there is one engine for generating predictions:

When generating predictions via the Real-Time Scoring Agent, you need RapidMiner Server to create the deployment, but once it's installed, it runs independently of RapidMiner Server.

As the table below makes clear, the Real-Time Scoring Agent is the scalable, low-latency counterpart to the Job Agent / Job Container. In short, it's just what you need for real-time predictions.

Component scalability low-latency response instant execution
Job Agent / Job Container Check Icon Check Icon Check Icon
Real-Time Scoring Agent Check Icon Check Icon Check Icon

Read more: Real-Time Scoring