Categories

Versions

  1. Documentation
  3. Hub
  4. Install
  5. Docker compose
  6. External Agents

You are viewing the RapidMiner Hub documentation for version 10.3 - Check here for latest version

External Web API agent

Table of contents

Set up the Web API agent

Please note that in the sample template we used

example-company.example.com for the main deployment, and

one.example.com for the external Web API agent deployment

The external Web API agent deployment is licensed independently from the main deployment

External agents have to be able to reach the aihub-backend and keycloak in the main deployment through the public url.

In the external Web API agent deployments configuration, the following environment variable must be set to the public endpoints of the main deployment:

- PUBLIC_FQDN="example-company.example.com"
- PUBLIC_PORT=80
- PUBLIC_PROTOCOL=http

Aihub-backend in the main deployment has to be able to reach the external agent deployment through the public url of the external agent deployment.

EUREKA_INSTANCE_HOSTNAME must match with the FQDN of the external Web API agent deployment.

If you are using HTTPS, the CN field of the certificate shall match with the EUREKA_INSTANCE_HOSTNAME.

The DNS name which will be called by AiHub deployment to push deployments:

EUREKA_INSTANCE_HOSTNAME1=one.example.com

Keycloak client secret shall be set according to the same value as it is in the main deployment.

- WEBAPI_AGENT_SSO_CLIENT_SECRET

The very same basic auth credentials shall be set for the external Web API agent deployment as the main deployment.

- WEBAPI_REGISTRY_USERNAME
- WEBAPI_REGISTRY_PASSWORD

Set up python environments for the Web API agent

In the external Web API agent deployment a coding-environment-manager container is fetching the list of the python environments from the platform-admin component in the main deployment and installs them on a volume, that is share with the Web API agent container.

docker-compose.yml

To be able to do that it has to be able to access the platform-admin endpoints in the main deployment.

First we have to provide a password protected access in the main deployment using the following env vars:

version: "3.9"
services:
  proxy:
    image: "${REGISTRY}rapidminer-proxy:${PROXY_VERSION}"
    hostname: proxy
    restart: always
    environment:
----> - PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_USER=${PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_USER}
----> - PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_PASSWORD=${PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_PASSWORD}
      - UNPRIVILEGED_PORTS=${UNPRIVILEGED_PORTS}
      - PROXY_DATA_UPLOAD_LIMIT=${PROXY_DATA_UPLOAD_LIMIT}
      ....
      .....

.env

In the external Web API deployment the value of the variable PLATFORM_ADMIN_BACKEND is generated, but the very same credentials shall be provided in the .env file as you set for the main deployment above:

PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_USER=<MUST MATCH WITH MASTER DEPLOYMENT ONE>
PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_PASSWORD=<MUST MATCH WITH MASTER DEPLOYMENT ONE>

Join the external Web API agent to a specific group

The Web API agent will join to a Web API group defined by the WEBAPI_GROUP_NAME variable in the .env file of the external Web API deployment.

The default value is EXTERNAL and it must created in the main deployment’s webui before the external Web API agent deployment.

WEBAPI_GROUP_NAME=EXTERNAL

HTTPS deployment

You will need a certificate for the external Web API deployment with it’s FQDN in the CN field.

The value of the EUREKA_HOSTNAME variable shall match with the sqdn and the cert as well.

To set up the deployment configuration to HTTPS you should change the following variables in the external Web API deployment's .env file:

- PUBLIC_PORT=443
- PUBLIC_PROTOCOL=https

The ssl subfolder in your deployment’s folder on the host is mapped to /etc/nginx/ssl inside the container.

When providing the private key and certificate paths, please use /etc/nginx/ssl as a directory path.

To set up the nginx reverse proxy to do the ssl offloading:

  • place your tls private key and the fullchain cert into the ssl folder
  • set the ssl_certificate and ssl_certificate_key directives in the server block to match with your filenames in the nginx-https.conf file
  • Change the mounted nginx config file from nginx-http.conf to nginx-https.conf in the docker-compose.yml
  • Change the exposed proxy port in the docker-compose.yml from 0.0.0.0:8090:80 to 0.0.0.0:8090:443
  • restart the reverse proxy
services:
  proxy:
    image: nginx:1.25.1
    hostname: webapi-proxy
    container_name: webapi-proxy
    restart: always
    volumes:
      #- ./nginx-http.conf:/etc/nginx/nginx.conf
      - ./nginx-https.conf:/etc/nginx/nginx.conf
      - ../ssl:/etc/nginx/ssl
    ports:
      - 0.0.0.0:8090:443
      #- 0.0.0.0:8090:80

Set up a custom Certificate Authority

If you already have your own Certificate Authority and already generated the certificate please skip this block.

The most seamless solution is to generate a wildcard certificate, that can be used on both the main and the external Web API agent deployment.

You can generate the necessary files on the main deployment's host with the following commands:

(please replace the hostname before running the commands below)

  cd ca-dir/
  CN='one.example.com'
  SAN="DNS:*.example.com"
  ./easyrsa --batch --subject-alt-name="${SAN}" --req-cn="${CN}" gen-req "${CN}" nopass
  ./easyrsa sign-req server "${CN}"
  cp pki/private/one.example.com.key ../ssl/
  cp pki/issued/one.example.com.crt ../ssl/
  cat pki/ca.crt >> ../ssl/one.example.com.crt

You can replace the pathname in the nginx-https.conf file with the following command:

sed -i -e 's/one.example.com.key/one.example.com.key/' -e 's/one.example.com.cert/one.example.com.crt/' external-agents/nginx-https.conf

Set up the external Web API agent deployment with custom Certificate Authority

Preparation for custom CA setup starts on the main deployment's host, where you can run the following script to transform the templates:

Please note, that the script was implemented and tested with the original templates, please run it before doing customization on the template files.

bash ./prepare-cust-ca.sh external-agent "external-agent"

Then you can finish the main deployment configuration and start the main deployment with:

docker-compose up -d deployment-init

The deployment-init service in the main deployment will create the necessary files (JKS, deb and RH certificate files) into the ssl folder.

Please check the container logs to verify that the process succeeded.

Then you can copy the files to the external Web API agent's host including the ssl folder.

Connect to custom port deployment

In this example the main deployment was deployed on port 85.

To connect external Web API agent to a custom port deployment the following settings have to be changed in the external Web API agent deployment's configuration files:

.env

- PUBLIC_PORT=85

docker-compose.yml

The port may appear only if it is a non-standard port (not 80 for HTTP and not 443 for HTTPS)

Please note that the :${PUBLIC_PORT} was added to the URLs

...
  licenseproxy:
      # Custom port
      - KEYCLOAK_AUTH_SERVER_URL=${WEBAPI_REGISTRY_PROTOCOL}://${WEBAPI_REGISTRY_HOST}:${PUBLIC_PORT}/auth/
...
...
  ext-webapi-agent-1:
      # Custom port
      - SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=${WEBAPI_REGISTRY_PROTOCOL}://${PUBLIC_FQDN}:${PUBLIC_PORT}/auth/realms/${SSO_IDP_REALM}
      - SCORING_AGENT_AUTH_AUTH_SERVER_URL=${WEBAPI_REGISTRY_PROTOCOL}://${PUBLIC_FQDN}:${PUBLIC_PORT}/auth
...
...
  coding-environment-storage:
      # Custom port
      - PLATFORM_ADMIN_BACKEND=${WEBAPI_REGISTRY_PROTOCOL}://${PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_USER}:${PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_PASSWORD}@${PUBLIC_PROTOCOL}${PUBLIC_FQDN}:${PUBLIC_PORT}/platform-admin/

HTTP deployment - Unsecure

Secure deployment is strongly recommended, use HTTPS

.env

If you would like to run the external Web API agent deployment on plain HTTP the following environment variables shall be set in the external Web API agent's .env file:

- PUBLIC_PORT=80
- PUBLIC_PROTOCOL=http

docker-compose.yml

The docker-compose.yml shall use the HTTP settings as well:

  proxy:
    image: nginx:1.25.1
    hostname: webapi-proxy
    container_name: webapi-proxy
    restart: always
    volumes:
      # Pay attantion the HTTP nginx config is used
      #
      - ./nginx-http.conf:/etc/nginx/nginx.conf
      #- ./nginx-https.conf:/etc/nginx/nginx.conf
      #- ../ssl:/etc/nginx/ssl
    ports:
      #- 0.0.0.0:8090:443
      - 0.0.0.0:8090:80
Privacy Policy © 2024 Altair Engineering Inc. All Rights Reserved. We are currently listed on Nasdaq as ALTR.