You are viewing the RapidMiner Deployment documentation for version 9.6 - Check here for latest version
Deployment Templates for Production, with High Availability
These templates cover configurations where an always-on operation of the RapidMiner platform is crucial. To achieve this, we provide an active-active redundancy of the RapidMiner Server component, and a standalone message queue to handle communications between the Server instances and the Job Agents.
This deployment consists of the following components:
- 2 RapidMiner Server instances
- 3 RapidMiner Job Agents
- Postgres database
- standalone ActiveMQ container
- load balancer container
- Python Environment Manager
To reduce the risk of outage, we always recommend doing deployments with High Availability configurations on multiple machines (see below).
To deploy the RapidMiner platform on multiple (physical or virtual) machines, we recommend a Kubernetes based approach. We support the popular public cloud vendors' Kubernetes services, as well as your own Kubernetes cluster.
The provided Docker Images are ready to deploy to any Kubernetes Cluster.
Please review the configuration below according to your environment and requirements.
The following guide requires a running Kubernetes cluster.
Rapidminer Platform is supported on the following Kubernetes services:
- Amazon Managed Kubernetes Service (Amazon EKS)
- Azure Kubernetes Service (AKS)
- MiniKube (Please read the Notices about minikube)
- MicroK8S
Volumes
Volumes provides the Elastic Block Storage for the RapidMiner Platform components to store the data permanently during container life-cycle.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rm-postgresql-pvc
labels:
app: rm-postgresql-svc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rm-server-home-pvc
labels:
app: rm-server-svc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pem-uploaded-pvc
labels:
app: pem-uploaded-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100M
Services
Services are the essential parts of the RapidMiner Platform. The services are used by containers/pods for reaching each other.
kind: Service
apiVersion: v1
metadata:
name: rapidminer-server-amq-svc
labels:
app: rapidminer-server-amq-svc
role: server
spec:
ports:
- port: 5672
targetPort: amq
selector:
app: rm-server-svc
role: server
---
kind: Service
apiVersion: v1
metadata:
name: rm-proxy-svc
labels:
app: rm-proxy-svc
role: proxy
spec:
ports:
- name: proxyhttp
protocol: TCP
port: 80
targetPort: proxyhttp
- name: proxyhttps
protocol: TCP
port: 443
targetPort: proxyhttps
selector:
app: rm-proxy-svc
role: proxy
type: LoadBalancer
---
kind: Service
apiVersion: v1
metadata:
name: postgres-svc
labels:
app: rm-postgresql-svc
spec:
ports:
- port: 5432
targetPort: postgresport
selector:
app: rm-postgresql-svc
---
kind: Service
apiVersion: v1
metadata:
name: rm-server-svc
labels:
app: rm-server-svc
role: server
spec:
ports:
- port: 8080
targetPort: rmswebui
selector:
app: rm-server-svc
role: server
---
kind: Service
apiVersion: v1
metadata:
name: pem-webui-svc
labels:
app: pem-webui-cron
role: pem
spec:
ports:
- name: pem-webuiport
port: 82
protocol: TCP
targetPort: pem-webuiport
selector:
app: rm-proxy-svc
role: proxy
Database
Database is used by RapidMiner Server.
kind: Pod
apiVersion: v1
metadata:
name: rm-postgresql-svc
labels:
app: rm-postgresql-svc
spec:
containers:
- name: rm-postgresql-svc
image: postgres:9.6
ports:
- name: postgresport
containerPort: 5432
env:
- name: POSTGRES_DB
value: rmsdb
- name: POSTGRES_USER
value: rmsdbuser
- name: POSTGRES_PASSWORD
value: rmsdbpassword
volumeMounts:
- name: pgvolume
mountPath: /var/lib/postgresql/data
subPath: postgres
volumes:
- name: pgvolume
persistentVolumeClaim:
claimName: rm-postgresql-pvc
RapidMiner Server
The main component of the RapidMiner Platform.
kind: Pod
apiVersion: v1
metadata:
name: rm-server-svc
labels:
app: rm-server-svc
role: server
spec:
hostname: rm-server-svc
containers:
- name: rapidminer-server
image: rapidminer/rapidminer-server:9.6.0
ports:
- name: rmswebui
containerPort: 8080
- name: amq
containerPort: 5672
env:
- name: JOBSERVICE_QUEUE_ACTIVEMQ_USERNAME
value: amq-user
- name: JOBSERVICE_QUEUE_ACTIVEMQ_PASSWORD
value: amq-pass
- name: JOBSERVICE_AUTH_SECRET
value: c29tZS1hdXRoLXNlY3JldAo=
- name: DBHOST
value: postgres-svc
- name: DBSCHEMA
value: rmsdb
- name: DBUSER
value: rmsdbuser
- name: DBPASS
value: rmsdbpassword
volumeMounts:
- name: rm-server-home-pvc
mountPath: /persistent-rapidminer-home
subPath: rapidminer-home
volumes:
- name: rm-server-home-pvc
persistentVolumeClaim:
claimName: rm-server-home-pvc
Job-Agent
The worker which perform the computation tasks.
kind: Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: rm-server-job-agent-svc
labels:
app: rm-server-job-agent-svc
role: execution
spec:
replicas: 3
selector:
matchLabels:
app: rm-server-job-agent-svc
template:
metadata:
labels:
app: rm-server-job-agent-svc
role: execution
spec:
containers:
- name: rm-server-job-agent-svc
image: rapidminer/rapidminer-execution-jobagent:9.6.0
env:
- name: RAPIDMINER_SERVER_HOST
value: rapidminer-server-svc
- name: RAPIDMINER_SERVER_PORT
value: '8080'
- name: JOBAGENT_QUEUE_ACTIVEMQ_URI
value: failover:(tcp://rapidminer-server-amq-svc:5672)
- name: JOBAGENT_QUEUE_ACTIVEMQ_USERNAME
value: amq-user
- name: JOBAGENT_QUEUE_ACTIVEMQ_PASSWORD
value: amq-pass
- name: JOBAGENT_AUTH_SECRET
value: c29tZS1hdXRoLXNlY3JldAo=
- name: RAPIDMINER_JOBAGENT_OPTS
value: "-Djobagent.python.registryBaseUrl=http://pem-webui-svc:82/"
RapidMiner Proxy & Python Environment Manager
The proxy component handles the incoming HTTP(S) traffic into the entire platform. Python Environment manager component (PEM) controls the python packages for job-agents. Real-Time Scoring (RTS) was designed for fast scoring use cases via web services. Those three platform pieces are MUST in one POD in kubernetes beaucuse proxy must read the certificates which are genereated by pem-cron and rts-cron containers.
apiVersion: apps/v1
kind: Deployment
metadata:
name: rm-proxy-svc
labels:
app: rm-proxy-svc
role: proxy
spec:
replicas: 1
selector:
matchLabels:
app: rm-proxy-svc
template:
metadata:
labels:
app: rm-proxy-svc
role: proxy
spec:
containers:
- name: rm-proxy-svc
image: rapidminer/rapidminer-proxy:9.6.0
imagePullPolicy: Always
env:
- name: RMSERVER_BACKEND
value: "http://rm-server-svc:8080"
- name: PEM_BACKEND
value: "http://pem-webui-svc:82/"
- name: PEM_URL_SUFFIX
value: "/pem"
- name: HTTPS_CRT_PATH
value: "/rapidminer/uploaded/certs/validated_cert.crt"
- name: HTTPS_KEY_PATH
value: "/rapidminer/uploaded/certs/validated_cert.key"
- name: HTTPS_DH_PATH
value: "/rapidminer/uploaded/certs/dhparam.pem"
- name: DEBUG_CONF_INIT
value: "true"
ports:
- name: proxyhttp
containerPort: 80
- name: proxyhttps
containerPort: 443
volumeMounts:
- name: pem-uploaded-pvc
mountPath: /rapidminer/pem/uploaded/
- name: pem-webui
image: rapidminer/python-environment-manager-webui:9.6.0
imagePullPolicy: Always
ports:
- name: pem-webuiport
containerPort: 82
volumeMounts:
- name: pem-uploaded-pvc
mountPath: /var/www/html/uploaded
- name: pem-cron
image: rapidminer/python-environment-manager-cron:9.6.0
imagePullPolicy: Always
volumeMounts:
- name: pem-uploaded-pvc
mountPath: /rapidminer/uploaded
volumes:
- name: pem-uploaded-pvc
persistentVolumeClaim:
claimName: pem-uploaded-pvc