You are viewing the RapidMiner Deployment documentation for version 9.9 - Check here for latest version

Kubernetes deployment with Helm

It has been nearly a year since our Kubernetes documentation was updated, and in the meantime large changes have occurred. It’s past time for an update.

Following some internal experiments at RapidMiner, we have attempted to reduce the complexity of Kubernetes configuration by introducing Helm Charts, but the results are still not what you would call “plug and play”.

Nevertheless, in the interest of progress, knowing that some of our users are already experienced with Kubernetes, we have decided to release some skeletal documentation – “skeletal” in the sense that it is not complete, but perhaps adequate for experienced users who know how to fill in the gaps.

In the weeks ahead, we plan to incrementally improve this documentation, based on feedback. If you made some effort to install and run the software, please let us know what we can do to improve the documentation.

For a simpler, single-host deployment of RapidMiner AI Hub, see Docker-compose deployment or the cloud images.

Our Docker Images have been improved to meet the latest security guidelines, including those of OpenShift and Rootless Docker. We tested our example configuration with the following Kubernetes services:

To deploy RapidMiner AI Hub with Kubernetes / Helm:

[Download] the Helm Chart, and follow instructions.

Note that this deployment uses RapidMiner Gen2 images, which have a more modern and secure structure than current images. The deployment is a preview, meant for testing rather than production, since there is not yet a migration mechanism from existing environments. Full support for migration will come with the next release.

Before you begin
Introduction to Helm
Instructions
The HELM configuration file (values.yaml)

Before you begin

To deploy the Helm chart, you need basic Kubernetes infrastructure. This documentation will not explain Kubernetes infrastructure setup. The links below are intended as hints for getting started.

Create Kubernetes infrastructure.
- (Amazon) Getting started with Amazon EKS
- (OpenShift) Selecting a cluster installation method and preparing it for users
As part of your Kubernetes setup, create NFS storage with a root folder.
- (Amazon) Amazon Elastic File System
- (Amazon) Creating an NFS file share
Create a namespace.
- (Kubernetes) Namespaces
- (Amazon) Multi-tenant design considerations for Amazon EKS clusters
- (Amazon) Tenant Isolation
Have your server certificate ready. Alternatively, use the built-in Let's Encrypt.

Introduction to Helm

Helm is a package manager for Kubernetes. A Helm Chart bundles the Kubernetes YAML files as templates, which you then configure via the file values.yaml. The details of this configuration depend on the details of your Kubernetes deployment. You and I may share the same templates, but our configurations (values.yaml) will differ. A typical Chart is a folder resembling the following:

mychart/
  Chart.yaml
  values.yaml
  charts/
  templates/

Chart.yaml: The Chart.yaml file contains a description of the chart. You can access it from within a template.
values.yaml: The file that defines your configuration, it contains the default values for a chart. These values may be overridden during helm install or helm upgrade.
charts/: The charts/ directory may contain other charts, called subcharts.
templates/: This folder contains the Kubernetes YAML files, as templates. When Helm evaluates a chart, it will send all of the files in the templates/ directory through the template rendering engine. It then collects the results of those templates and sends them on to Kubernetes. The placeholders in the YAML files are defined by values.yaml.

Helm documentation

Introductory videos:

What is Helm in Kubernetes? Helm and Helm Charts explained (TechWorld with Nana)
What is Helm? (IBM Cloud)
Introduction to Helm | Kubernetes Tutorial | Beginners Guide (That DevOps Guy)

Instructions

To simplify the configuration of the Kubernetes YAML files, we use Helm, the package manager for Kubernetes.

Make sure that your Kubernetes infrastructure is in place, including Helm.
Download the Helm archive, and extract values.yaml, renaming it to custom-values.yaml:
```
 helm show values ./rapidminer-aihub-9.9.2.tgz > custom-values.yaml
```
Edit custom-values.yaml and define your configuration by setting the appropriate values.
Then apply the following command to the Kubernetes cluster:

helm upgrade -n <NAMESPACE-PLACEHOLDER> --install rapidminer-aihub --values custom-values.yaml ./rapidminer-aihub-9.9.2.tgz

Note that the value <NAMESPACE-PLACEHOLDER> is the same as the one you gave in custom-values.yaml for the key productNS.

EBS volumes are sensitive to multi-attach errors during rolling updates. It is best to scale down all the deployments before the update.

The HELM configuration file (values.yaml)

common:
  domain: "<FQDN-PLACEHOLDER>"
# If you need to obtain LetsEncrypt certificate first, please temporarly change the protocol to http://
  public_url: "https://<FQDN-PLACEHOLDER>"
  sso_public_url: "https://<FQDN-PLACEHOLDER>"
# The namespace of the deployment
  productNS: "<NAMESPACE-PLACEHOLDER>"
# The docker image tag
  mainVersion: "9.9.2-gen2"
# The docker image tag for Coding Environment Storage
  cesVersion: "9.9.2-gen2"
# Docker registry prefix
  dockerURL: "rapidminerstaging/"
# The timezone of the deployment
  timeZone: "<TIMEZONE-PLACEHOLDER>"
# platform related Values, please choose one from:
# "OpenShift" : OpenShift related security and other infrastructure settings
# "EKS" : Amazon Elastic Kubernetes related security and other infrastructure settings
  platfrom: "EKS"
# If nodeSelector value is not set, then the nodeselector feature will not inserted to deployments
# any label of the worker nodes can be used
#  nodeSelector:
#    <NODE-LABEL-1-NAME-PLACEHOLDER>: "<NODE-LABEL-1-VALUE-PLACEHOLDER>"
#    <NODE-LABEL-2-NAME-PLACEHOLDER>: "<NODE-LABEL-2-VALUE-PLACEHOLDER>"
  nodeSelector: {}
# If the name of the image pull secret is empty, this part of deployment will not inserted
  imagePullSecret: []
# The name of the kubernetes secret, which contains the RapidMiner License
# If empty, the default admin user can provide it on the webui
  licenseSecret: ""
# the key of the license in the Kubernetes secret, default value is "SERVER_LICENSE"
  licenSecretKey: "SERVER_LICENSE"
# This will be the initial user, which will have admin permission in the deployment.
  initialUser: "admin"
# Initial password for the initial user
  initialPass: "<ADMIN-PASS-PLACEHOLDER>"
# The built in OIDC server realm, this realm will be used by the components in the SSO communication (KeyCloak)
  defaultSSORealm: "master"
# Default SSL requirement to access KeyCloak SSO
  ssoSSL: "external"
# Default storageclass, for example ebs-us-west-2a, see templates/002-storageclasses.yaml
  defaultstorageClass: "<STORAGECLASS-PLACEHOLDER>"
# IP / FQDN of the NFS Server
  nfsServer: "<NFS-SERVER-PLACEHOLDER>"
# Path on the NFS Server, this folder should exists before deploying
# If you have multiple deployment on the same cluster, it is recommended to include the namespace in that folder name
  nfsPath: "/<NAMESPACE-PLACEHOLDER>"
  nfsMountOpts:
    - vers=4
    - minorversion=1
    - sec=sys
# If you have multiple deployment on the same cluster, it is recommended to include the namespace in that name
  nfsStorageName: nfs-<NAMESPACE-PLACEHOLDER>
  createNamespace: "false"

AWS:
# Only the value true is supported yet
  allowEBS: "true"
  # initEBSStorageClasses should be true if the storageclasses were created by helm
  initEBSStorageClasses: "true"
# The region of the ebs volumes, for example us-west-2, this value should align to defaultStorageClass property
  EBSRegion: "<EBS-REGION-PLACEHOLDER>"
OKD: {}

proxy:
  serviceName: "proxy-svc-pub"
  imageName: "rapidminer-proxy"
  version: "9.9.2-gen2"
  configName: "proxy-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  debug: "false"

letsEncrypt:
  imageName: "rm-letsencrypt-client"
  version: "9.9.2-gen2"
  configName: "letsencrypt-client-config"
  allowLetsEncrypt: "true"
  certsHome: "/certificates/"
  webMasterEmail: "<WEBMASTER-EMAIL-PLACEHOLDER>"

landingPage:
  serviceName: "landing-page-svc-priv"
  imageName: "rapidminer-deployment-landing-page"
  version: "9.9.2-gen2"
  configName: "landing-page-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  ssoClientId: "urn:rapidminer:landing-page"
  ssoClientSecret: "<LANDING-PAGE-OIDC-CLIENT-SECRET-PLACEHOLDER>"
  debug: "false"

serverDB:
  serviceName: "rapidminer-server-postgres-svc-priv"
  imageName: "postgresql-10-centos7"
  version: "latest"
  configName: "rapidminer-server-postgres-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  dockerURL: "centos/"
  dbName: "<SERVER-DB-NAME-PLACEHOLDER>"
  dbUser: "<SERVER-DB-USER-PLACEHOLDER>"
  dbPass: "<SERVER-DB-PASS-PLACEHOLDER>"


server:
  serviceName: "rapidminer-server-svc-priv"
  imageName: "rapidminer-server"
  version: "9.9.2-gen2"
  configName: "rapidminer-server-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  activeMQServiceName: "rapidminer-server-amq-svc-priv"
  activeMQUser: "<SERVER-AMQ-USER-NAME-PLACEHOLDER>"
  activeMQPass: "<SERVER-AMQ-PASS-PLACEHOLDER>"
  ssoClientId: "urn:rapidminer:server"
  ssoClientSecret: "<SERVER-OIDC-CLIENT-SECRET-PLACEHOLDER>"
  memLimit: "2048M"
  platformAdminSyncDebug: "False"
  legacyRESTBasicAuth: "false"

jobagent:
  imageName: "rapidminer-execution-jobagent"
  version: "9.9.2-gen2"
  configName: "job-agents-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  authSecret: "<SERVER-AMQ-SECRET-PLACEHOLDER>"
  jobQueue: "DEFAULT"
  containerCount: "1"
  containerMemLimit: "2048"
  containerJavaOpts: ""
  javaOpts: "-Djobagent.container.jvmCustomProperties=Dlogging.level.com.rapidminer=INFO"

keycloak:
  serviceName: "keycloak-svc-priv"
  imageName: "rapidminer-keycloak"
  version: "9.9.2-gen2"
  configName: "keycloak-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  proxyAddrForward: "true"

keycloakDB:
  serviceName: "keycloak-postgres-svc-priv"
  imageName: "postgresql-10-centos7"
  version: "latest"
  configName: "keycloak-postgres-config"
  dockerURL: "centos/"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  vendor: "POSTGRES"
  dbName: "<KEYCLOAK-DB-NAME-PLACEHOLDER>"
  dbUser: "<KEYCLOAK-DB-USER-PLACEHOLDER>"
  dbPass: "<KEYCLOAK-DB-PASS-PLACEHOLDER>"
  dbSchema: "public"
  useSSL: "false"

rmInit:
  imageName: "rapidminer-deployment-init"
  version: "9.9.2-gen2"
  configName: "rm-deployment-init-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  debug: "false"

platformAdmin:
  serviceName: "platform-admin-webui-svc-priv"
  imageName: "rapidminer-platform-admin-webui"
  version: "9.9.2-gen2"
  configName: "platform-admin-webui-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  proxyURLSuffix: "/platform-admin"
  proxyRTSWebUISuffix: "/rts-admin"
  ssoClientId: "urn:rapidminer:platform-admin"
  ssoClientSecret: "<PLATFORM-ADMIN-OIDC-CLIENT-SECRET-PLACEHOLDER>"
  disablePython: "false"
  disableRTS: "false"
  debug: "false"

ces:
  imageName: "rapidminer-coding-environment-storage"
  version: "9.9.2-gen2"
  configName: "rapidminer-coding-environment-storage-config"
  pythonPackageLink: "git+https://github.com/rapidminer/python-rapidminer.git@9.9.0.0"
  ubuntuUid: "9999"
  ubuntuGid: "9999"
  rapidMinerUser: "rapidminer"

rts:
  serviceName: "rts-agent-svc-priv"
  imageName: "rapidminer-execution-scoring"
  version: "9.9.2-gen2"
  configName: "rts-agent-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  proxyURLSuffix: "/rts"
  initSharedCondaSettings: "true"
  waitForLicenses: "1"
  basicAuth: "true"

jupyterDB:
  serviceName: "rm-jupyterhub-db-svc"
  imageName: "rapidminer-jupyterhub-postgres"
  version: "9.9.2-gen2"
  configName: "jupyterhub-postgres-config"
  dockerURL: "rapidminerstaging/"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  vendor: "POSTGRES"
  dbName: "<JUPYTERHUB-DB-NAME-PLACEHOLDER>"
  dbUser: "<JUPYTERHUB-DB-USER-PLACEHOLDER>"
  dbPass: "<JUPYTERHUB-DB-PASS-PLACEHOLDER>"

jupyterHub:
  proxyServiceName: "jupyterhub-proxy-svc-priv"
  proxyAPIServiceName: "jupyterhub-proxy-api-svc-priv"
  serviceName: "jupyterhub-hub-svc-priv"
  imageName: "rapidminer-jupyterhub-jupyterhub"
  version: "9.9.2-gen2"
  configName: "jupyterhub-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  proxyURLSuffix: "/jupyter"
  cryptKey: "51fd01d4bdf18df039744272fe400dc06cb2b00a41add94b3f3fd90ede301df2"
  debug: "False"
  tokenDebug: "False"
  proxyDebug: "False"
  dbDebug: "False"
  spawnerDebug: "False"
  stackName: "default"
  ssoClientId: "urn:rapidminer:jupyterhub"
  ssoClientSecret: "<JUPYTERHUB-OIDC-CLIENT-SECRET-PLACEHOLDER>"
  ssoUserNameKey: "preferred_username"
  ssoResourceAccKey: "resource_access"
  spawner: "kubespawner"
  apiProtocol: "http"
  k8sCMD: "/entrypoint.sh"
  k8sArgs: "[]"
  proxyPort: "8000"
  apiPort: "8001"
  appPort: "8081"
  envVolumeName: "rm-coding-shared-vol"

jupyterNoteBook:
  imageName: "rapidminer-jupyter_notebook"
  version: "9.9.2-gen2"
  memLimit: "2G"
  cpuLimit: "100"
  ssoUidKey: "X_NB_UID"
  ssoGidKey: "X_NB_GID"
  ssoCustomBindMountsKey: "X_NB_CUSTOM_BIND_MOUNTS"
  customBindMounts: ""
  storageAccessMode: "ReadWriteOnce"
  storageSize: "5Gi"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  # Should be the name of the CES pvc (-nfs-shared-pvc)
  pvcName: "<NAMESPACE-PLACEHOLDER>-nfs-shared-pvc"
  pvcSubPath: "coding-shared"
  imagePullAtStartup: "False"
#  nodeSelector:
#    key: "<NODE-LABEL-1-NAME-PLACEHOLDER>"
#    value: "<NODE-LABEL-1-VALUE-PLACEHOLDER>"
  nodeSelector: {}

grafanaProxy:
  serviceName: "grafana-proxy-svc-priv"
  imageName: "rapidminer-grafana-proxy"
  version: "9.9.2-gen2"
  configName: "grafana-proxy-config"

grafanaAnonProxy:
  serviceName: "grafana-anonymous-proxy-svc-priv"
  imageName: "rapidminer-grafana-proxy"
  version: "9.9.2-gen2"
  configName: "grafana-anonymous-proxy-config"

grafana:
  serviceName: "grafana-svc-priv"
  imageName: "rapidminer-grafana"
  version: "9.9.2-gen2"
  configName: "grafana-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  proxyURLSuffix: "/grafana"
  ssoClientId: "urn:rapidminer:grafana"
  ssoClientSecret: "<GRAFANA-OIDC-CLIENT-SECRET-PLACEHOLDER>"
  disableSanitizeHTML: "true"

tokenTool:
  serviceName: "token-tool-svc-priv"
  imageName: "rapidminer-deployment-landing-page"
  version: "9.9.2-gen2"
  configName: "token-tool-config"
  storageClass: "<STORAGECLASS-PLACEHOLDER>"
  proxyURLSuffix: "/get-token"
  ssoClientId: "urn:rapidminer:token-tool"
  ssoClientSecret: "<TOKEN-TOOL-OIDC-CLIENT-SECRET-PLACEHOLDER>"
  ssoCustomScope: "openid info offline_access"
  customContent: "get-token"
  debug: "false"

Categories

Versions