Scalable architecture

To build a RapidMiner Server environment for your data science team, two components have to be installed:

plus the following helper applications:

  • RapidMiner Studio - to design the processes you will run in the RapidMiner Server environment
  • A database - to store configuration files, cron job details, user report requests, and other support data

The following optional component can be installed, and it runs independently:

A simple schematic is shown below.

The design

The design of the RapidMiner Server environment reflects a typical data science workflow, where there are two kinds of activities:

  1. Model building, involving long-running processes that can be placed on a queue and run asynchronously

    RapidMiner Server offers a queue system for long-running jobs, which are executed externally via Job Agents. You increase processing power by adding Job Agents.

  2. Prediction, or any other application of the models, where the need for a real-time response is paramount

    There are two engines for generating predictions:

    Only the latter are scalable. You increase processing power by adding Real-Time Scoring Agents.

RapidMiner Server

RapidMiner Server is the central component in the architecture. You interact with it via a web interface or via RapidMiner Studio. Its main responsibilities are:

  • User, queue, and permissions management
  • Scheduling of user jobs (processes)
  • Execution of processes called via web services / web apps
  • Execution of processes running on the local Job Agent, if it exists
  • Repository management (storage of models, processes, etc. and permissions for them)
  • Connection management (DB, Hadoop/Radoop, etc.)

Read more: Install RapidMiner Server

In the diagram below, each blue box represents a separate machine. RapidMiner Server is installed on the big blue box on the left, while the blue boxes on the right host remote Job Agents.

Job Agent

The design with Job Agents running remotely on dedicated machines is aimed at scalability. However, one or more Job Agents can be installed locally, on the same machine as RapidMiner Server.

Each Job Agent is configured to point to one of the queues on RapidMiner Server. Its only responsibility is to pick up jobs from the queue and run them, by spawning a Job Container. For each Job Agent, the number of Job Containers that can be spawned and the available memory is configurable.

Multiple Job Agents can point to the same queue. You can manage the queues, and therefore the allocation of resources, by assigning permissions.

Read more: Install a Job Agent

Job Container

The Job Container spawned by the Job Agent runs a RapidMiner Studio instance that executes a process. Once that process is finished, the Job Container terminates. Because each job runs in its own sandbox, the system is highly robust; problems with one job have no effect on any other job.

The price of safety is latency -- the latency in spawning a Job Container is measured in seconds. If a real-time response is not paramount, this latency will not be important, but if it is, we recommend using a web service or a Real-Time Scoring Agent. For example, you might build a model in a Job Container and generate predictions for that model via a Real-Time Scoring Agent.

Real-Time Scoring Agent

As mentioned previously, there are two engines for generating predictions:

When generating predictions via the Real-Time Scoring Agent, you need RapidMiner Server to create the deployment, but once it's installed, it runs independently of RapidMiner Server.

As the table below makes clear, the Real-Time Scoring Agent is the scalable, low-latency counterpart to the Job Agent / Job Container. In short, it's just what you need for real-time predictions.

Component scalable low-latency
Job Agent / Job Container Check Icon Check Icon
Real-Time Scoring Agent Check Icon Check Icon
Web service Check Icon Check Icon

Read more: Web services

Read more: Real-Time Scoring