External Web API agent
Table of contents
- HTTPS deployment
- Set up a custom Certificate Authority
- Set up external Web API agent with deployment
- Set up external Web API agent to connect to deployment on custom port
- Unsecure deployment
Set up the Web API agent
Please note that in the sample template we used
example-company.example.com for the main deployment, and
one.example.com for the external Web API agent deployment
The external Web API agent deployment is licensed independently from the main deployment
External agents have to be able to reach the aihub-backend and keycloak in the main deployment through the public url.
In the external Web API agent deployments configuration, the following environment variable must be set to the public endpoints of the main deployment:
- PUBLIC_FQDN="example-company.example.com"
- PUBLIC_PORT=80
- PUBLIC_PROTOCOL=http
Aihub-backend in the main deployment has to be able to reach the external agent deployment through the public url of the external agent deployment.
EUREKA_INSTANCE_HOSTNAME
must match with the FQDN of the external Web API agent deployment.
If you are using HTTPS, the CN field of the certificate shall match with the EUREKA_INSTANCE_HOSTNAME
.
The DNS name which will be called by AiHub deployment to push deployments:
EUREKA_INSTANCE_HOSTNAME1=one.example.com
Keycloak client secret shall be set according to the same value as it is in the main deployment.
- WEBAPI_AGENT_SSO_CLIENT_SECRET
The very same basic auth credentials shall be set for the external Web API agent deployment as the main deployment.
- WEBAPI_REGISTRY_USERNAME
- WEBAPI_REGISTRY_PASSWORD
Set up python environments for the Web API agent
In the external Web API agent deployment a coding-environment-manager container is fetching the list of the python environments from the platform-admin component in the main deployment and installs them on a volume, that is share with the Web API agent container.
docker-compose.yml
To be able to do that it has to be able to access the platform-admin endpoints in the main deployment.
First we have to provide a password protected access in the main deployment using the following env vars:
version: "3.9"
services:
proxy:
image: "${REGISTRY}rapidminer-proxy:${PROXY_VERSION}"
hostname: proxy
restart: always
environment:
----> - PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_USER=${PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_USER}
----> - PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_PASSWORD=${PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_PASSWORD}
- UNPRIVILEGED_PORTS=${UNPRIVILEGED_PORTS}
- PROXY_DATA_UPLOAD_LIMIT=${PROXY_DATA_UPLOAD_LIMIT}
....
.....
.env
In the external Web API deployment the value of the variable PLATFORM_ADMIN_BACKEND
is generated, but the very same credentials shall be provided in the .env
file as you set for the main deployment above:
PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_USER=<MUST MATCH WITH MASTER DEPLOYMENT ONE>
PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_PASSWORD=<MUST MATCH WITH MASTER DEPLOYMENT ONE>
Join the external Web API agent to a specific group
The Web API agent will join to a Web API group defined by the WEBAPI_GROUP_NAME
variable in the .env file of the external Web API deployment.
The default value is EXTERNAL
and it must created in the main deployment’s webui before the external Web API agent deployment.
WEBAPI_GROUP_NAME=EXTERNAL
HTTPS deployment
You will need a certificate for the external Web API deployment with it’s FQDN in the CN field.
The value of the EUREKA_HOSTNAME
variable shall match with the sqdn and the cert as well.
To set up the deployment configuration to HTTPS you should change the following variables in the external Web API deployment's .env
file:
- PUBLIC_PORT=443
- PUBLIC_PROTOCOL=https
The ssl subfolder in your deployment’s folder on the host is mapped to /etc/nginx/ssl inside the container.
When providing the private key and certificate paths, please use /etc/nginx/ssl
as a directory path.
To set up the nginx reverse proxy to do the ssl offloading:
- place your tls private key and the fullchain cert into the
ssl
folder - set the
ssl_certificate
andssl_certificate_key
directives in the server block to match with your filenames in thenginx-https.conf
file - Change the mounted nginx config file from
nginx-http.conf
tonginx-https.conf
in thedocker-compose.yml
- Change the exposed proxy port in the
docker-compose.yml
from0.0.0.0:8090:80
to0.0.0.0:8090:443
- restart the reverse proxy
services:
proxy:
image: nginx:1.25.1
hostname: webapi-proxy
container_name: webapi-proxy
restart: always
volumes:
#- ./nginx-http.conf:/etc/nginx/nginx.conf
- ./nginx-https.conf:/etc/nginx/nginx.conf
- ../ssl:/etc/nginx/ssl
ports:
- 0.0.0.0:8090:443
#- 0.0.0.0:8090:80
Set up a custom Certificate Authority
If you already have your own Certificate Authority and already generated the certificate please skip this block.
The most seamless solution is to generate a wildcard certificate, that can be used on both the main and the external Web API agent deployment.
You can generate the necessary files on the main deployment's host with the following commands:
(please replace the hostname before running the commands below)
cd ca-dir/
CN='one.example.com'
SAN="DNS:*.example.com"
./easyrsa --batch --subject-alt-name="${SAN}" --req-cn="${CN}" gen-req "${CN}" nopass
./easyrsa sign-req server "${CN}"
cp pki/private/one.example.com.key ../ssl/
cp pki/issued/one.example.com.crt ../ssl/
cat pki/ca.crt >> ../ssl/one.example.com.crt
You can replace the pathname in the nginx-https.conf
file with the following command:
sed -i -e 's/one.example.com.key/one.example.com.key/' -e 's/one.example.com.cert/one.example.com.crt/' external-agents/nginx-https.conf
Set up the external Web API agent deployment with custom Certificate Authority
Preparation for custom CA setup starts on the main deployment's host, where you can run the following script to transform the templates:
Please note, that the script was implemented and tested with the original templates, please run it before doing customization on the template files.
bash ./prepare-cust-ca.sh external-agent "external-agent"
Then you can finish the main deployment configuration and start the main deployment with:
docker-compose up -d deployment-init
The deployment-init service in the main deployment will create the necessary files (JKS, deb and RH certificate files) into the ssl folder.
Please check the container logs to verify that the process succeeded.
Then you can copy the files to the external Web API agent's host including the ssl
folder.
Connect to custom port deployment
In this example the main deployment was deployed on port 85.
To connect external Web API agent to a custom port deployment the following settings have to be changed in the external Web API agent deployment's configuration files:
.env
- PUBLIC_PORT=85
docker-compose.yml
The port may appear only if it is a non-standard port (not 80 for HTTP and not 443 for HTTPS)
Please note that the :${PUBLIC_PORT} was added to the URLs
...
licenseproxy:
# Custom port
- KEYCLOAK_AUTH_SERVER_URL=${WEBAPI_REGISTRY_PROTOCOL}://${WEBAPI_REGISTRY_HOST}:${PUBLIC_PORT}/auth/
...
...
ext-webapi-agent-1:
# Custom port
- SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI=${WEBAPI_REGISTRY_PROTOCOL}://${PUBLIC_FQDN}:${PUBLIC_PORT}/auth/realms/${SSO_IDP_REALM}
- SCORING_AGENT_AUTH_AUTH_SERVER_URL=${WEBAPI_REGISTRY_PROTOCOL}://${PUBLIC_FQDN}:${PUBLIC_PORT}/auth
...
...
coding-environment-storage:
# Custom port
- PLATFORM_ADMIN_BACKEND=${WEBAPI_REGISTRY_PROTOCOL}://${PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_USER}:${PLATFORM_ADMIN_ENVIRONMENT_EXPORT_AUTH_BASIC_PASSWORD}@${PUBLIC_PROTOCOL}${PUBLIC_FQDN}:${PUBLIC_PORT}/platform-admin/
HTTP deployment - Unsecure
Secure deployment is strongly recommended, use HTTPS
.env
If you would like to run the external Web API agent deployment on plain HTTP the following environment variables shall be set in the external Web API agent's .env
file:
- PUBLIC_PORT=80
- PUBLIC_PROTOCOL=http
docker-compose.yml
The docker-compose.yml shall use the HTTP settings as well:
proxy:
image: nginx:1.25.1
hostname: webapi-proxy
container_name: webapi-proxy
restart: always
volumes:
# Pay attantion the HTTP nginx config is used
#
- ./nginx-http.conf:/etc/nginx/nginx.conf
#- ./nginx-https.conf:/etc/nginx/nginx.conf
#- ../ssl:/etc/nginx/ssl
ports:
#- 0.0.0.0:8090:443
- 0.0.0.0:8090:80