1. Documentation
2. 2024.1 (Latest) 2024.0 10.3 10.2 10.1 10.0 9.10 9.9 9.8 9.7 9.6 9.5 9.4 9.3 9.2 9.1 9.0 8.2 8.1 8.0 7.6
3. Radoop
4. Installation
5. Configuring Radoop Connections

Configuring RapidMiner Radoop Connections in RapidMiner Studio

You can configure connections between RapidMiner Radoop in RapidMiner Studio and one or more Hadoop clusters from the Manage Radoop Connections and Connection Settings dialogs. You can access these dialogs from the Connections menu, the Hadoop Data view, or the Design view. After configuring and saving the connection entries, you can test them before deployment. The test validates the connection to the cluster and verifies that the connection settings comply with the RapidMiner Radoop requirements described in the section on prerequisites.

1. If you have access to cluster manager software (Apache Ambari or Cloudera Manager), we strongly recommend using the Import from Cluster Manager option. This method is the simplest one.

2. If you do not use or do not have access to a cluster manager, but can ask for the client configuration files, then use the Import Hadoop Configuration Files option.

3. Otherwise, you always have the option to Add Connection Manually. This last option also allows you to import a Radoop connection that someone shared with you by clicking on the Edit XML... button, once the Connection Settings dialog appears.

Note: When configuring RapidMiner Radoop, you must provide the internal domain name or IP address of the master node (that is, the domain name or IP address that the master node knows about itself). See the networking setup overview for details on how to ensure that your data is safe from unauthorized access.

Basic RapidMiner Radoop connection configuration

Once RapidMiner Radoop is installed, you can create a connection.

1. Restart RapidMiner Studio so that it recognizes the RapidMiner Radoop extension. Once it restarts, you will see a new Manage Radoop Connections option in the Connections menu:

   

2. Select the Manage Radoop Connections menu item and the Manage Radoop Connections window opens:

   

   For more details on this dialog, see Manage Radoop Connections section below.

3. Click on New Connection button and choose Add Connection Manually:

   

   If you wish to create the connection by importing client configuration files or by using a cluster management service, read the Importing Hadoop configuration section.

4. You can edit the connection properties in the Connection Settings dialog.

   

   You can provide the name of the connection at the top of the dialog. Additional settings can be configured by selecting the appropriate tab on the left side. Complete the required connection fields listed below. Note that DNS and reverse DNS should work for all specified addresses, so the client machine must have access to the network name resolution system of the cluster or be able to resolve the addresses locally.

   Tab	Field	Description
   Global	Hadoop Version	The distribution that defines the Hadoop version type for this connection.
   Hadoop	NameNode Address	Address (usually hostname) of the node running the NameNode service.
   Hadoop	Resource Manager Address	Address (usually hostname) of the node running the Resource Manager service.
   Hive	Hive Server Address	Address (usually hostname) of the node running the Hive Server or the Impala Server.
   Spark	Spark Version	Spark version available on the cluster.
   Spark	Assembly Jar Location / Spark Archive (or libs) path	The HDFS location or local path (on all cluster nodes) of the Spark Assembly Jar file / Spark Jar files.

   For further details, see Advanced connection settings section below.

   Click OK to create the connection entry.

5. Click Save to add the entry to the available connections.

6. Test the connection between RapidMiner Radoop and the Hadoop cluster. If necessary, and with the assistance of your Hadoop administrator, set the advanced settings based on the distribution specific notes.

Your connection settings are saved in a file called radoop_connections.xml in your .RapidMiner directory.

Importing a connection

Configuring a connection manually can be cumbersome for a more complicated cluster. In this case using one of the connection import features is recommended. There are two options: you can create a connection using the cluster's client configuration files, or by providing the URL and credentials for the cluster's management service (Cloudera Manager or Ambari).

Manage Radoop Connections window

The Manage Radoop Connections window shows your already configured connections and allows you to edit them, or create and test new connections:

This window consists of 3 panels. The upper left panel lists all known connection entries. For each entry, one or more icons may be present showing some additional information, namely:

• Spark is configured for this connection
• The connection uses Impala as query engine
• Connection to a secure cluster

The basic properties of the currently selected connection are gathered on the right-hand side panel. There are also buttons executing several actions available on the selected connection:

• Configure...: Opens the Connection Settings dialog where you can configure the connection properties. Check the Advanced connection settings section for more details.
• Save: Saves the currently displayed connection.
• Save As...: Saves a copy of the currently displayed connection. Useful for saving a slightly modified connection while keeping the original entry.
• Quick Test: Runs a Quick Test on the currently displayed connection.
• Full Test...: Runs a Full Integration Test on this connection. More information on the connection tests can be found in the Testing RapidMiner Radoop connections section.
• Rename Action: Renames the current connection. Please note that all connection names should be unique.

The lower panel shows logs of the running tests. Several actions can be performed on this panel too:

• Extract logs...: This action creates a bundled zip file containing all relevant logs of your recent Radoop-related activities. See related section for more details.
• Clear logs: Clears the connection log field.
• Stop Test: The Stop Test action will halt the currently running test execution (see the Testing RapidMiner Radoop connections section).

Testing RapidMiner Radoop cluster connections

RapidMiner Radoop's built-in test functions help with troubleshooting before trouble begins.

Basic connection test

Click the Quick Test button in the Manage Radoop Connections window to test the connection to the cluster. Through a series of simple tests to different components (APIs) on the cluster, the test verifies that the cluster is running and that the RapidMiner Radoop client can access it. You can stop the test anytime by clicking the Stop Test button.

Full connection tests

Once your test succeeds, run a complete test (which may take several minutes) by clicking the Full Test... button. It's possible to customize a full connection test by clicking the Customize... button. In this panel you can enable or disable tests, change the timeout and enable or disable the cleaning after the tests. These values are reset to the defaults after closing the Manage Radoop Connections window. Click Run to start the test.

The full test initiates several jobs and applications on the cluster and then checks the results. By successfully and extensively exercising RapidMiner Radoop interactions with your cluster, you can feel confident in your RapidMiner Radoop process design and execution.

In addition to testing connections when you first create a RapidMiner Radoop configuration, you can use the Full Test if you have an error in process execution or a change in the cluster. The output of the full test results can help identify the root cause of the problem for easier troubleshooting. You can stop the Full Test anytime by clicking the Stop Test button. Stopping the current test process may take some time.

Note: The cluster connection initial test also starts automatically in the background when you open a process containing a RapidMiner Radoop Nest operator (indicated by the status bar in the bottom right corner of the RapidMiner Studio screen).

Advanced connection settings

You can use the Connection Settings dialog to edit the connection parameters. For example, you can change port numbers or define arbitrary parameters for Hadoop and Hive using key-value pairs. Do not modify the connection settings without first consulting with your organization's IT administrator. To open the Connection Settings dialog, click the Configure... button from the Manage Radoop Connections window.

Note: The fields displayed depend on the selections (for example, the selected Hadoop version). Also, some fields prepopulate based on the Hadoop version selection from the basic settings. If a field is bold in the window, it is required.

The Connection Settings dialog has multiple tabs. The following tables describe the fields in each tab. For advanced configuration details related to your environment, see the distribution specific notes.

Global

Field	Description
Hadoop Version	The distribution that defines the Hadoop version type for this connection.
Additional Libraries Directory	Any additional libraries (JAR files) on the client needed to connect to the cluster (optional, for expert users only).
Enable Kerberos	Check this box to connect to a Hadoop cluster secured by Kerberos.
Client Principal	Only with Kerberos security enabled and Server impersonation disabled. Principal of the user accessing Hadoop. The format is primary[/<instance>]@<REALM>, where primary is usually the user name, instance is optional, and REALM is the Kerberos realm. Example: user/client.rapidminer.com@RAPIDMINER.COM.
Use password instead of keytab file	Only with Kerberos security enabled. Check this box to authenticate with a password instead of a keytab file.
KeyTab File	Path of the user keytab file on the client machine. Enter or browse to the file location.
Password	Only with Kerberos security enabled and "Use password instead of keytab file" option checked. The Kerberos password that can be used to connect to the secure cluster. RapidMiner Radoop uses the cipher.key file to encrypt the password in radoop_connections.xml.
KDC Address	Only with Kerberos security enabled. Address of the Kerberos Key Distribution Center. Example: kdc.rapidminer.com.
REALM	Only with Kerberos security enabled. The Kerberos realm. It is usually the domain name in upper-case letters. Example: RAPIDMINER.COM.
Kerberos Config File	Only with Kerberos security enabled. To avoid configuration differences between the machine running RapidMiner and the Hadoop cluster, it is good practice to provide the Kerberos configuration file (usually krb5.conf or krb5.ini). Obtain this file from your security administrator. Enter or browse to the file location.
Enable MapR security	Only with some MapR Hadoop version selected. Check this box to connect to a Hadoop cluster secured by MapR Security.
MapR cluster	Only with some MapR Hadoop version selected. MapR cluster to connect to. All MapR connections must be configured in the MapR client that MapR Home points to.
Hadoop Username	The name of the Hadoop user. In most cases, the user must have appropriate permissions on the cluster. For a new connection, the default is the OS user.

Hadoop

Field	Description
NameNode Address	Address (usually hostname) of the node running the NameNode service. (Requires a working network name resolution system.)
NameNode Port	Port of the NameNode service.
Resource Manager Address	Address (usually hostname) of the node running Resource Manager service.
Resource Manager Port	Port of the Resource Manager service.
JobHistory Server Address	Address (usually hostname) of the node running the Job History Server service.
JobHistory Server Port	Port of the Job History Server service.
Retrieve Service Principals from Hive	Only with Kerberos security enabled. If checked, RapidMiner Radoop automatically retrieves all other service principals from Hive for easier configuration. Disable this setting only if there is a problem accessing other services.
NameNode Principal	Only with Kerberos security enabled and Hive principal retrieval disabled. Principal of the NameNode service. You can use the _HOST keyword as the instance. Example: nn/_HOST@RAPIDMINER.COM
Resource Manager Principal	Only with Kerberos security enabled and Hive principal retrieval disabled. Principal of the ResourceManager service. You can use the _HOST keyword as the instance. Example: rm/_HOST@RAPIDMINER.COM
JobHistory Server Principal	Only with Kerberos security enabled and Hive principal retrieval disabled. Principal of the JobHistoryServer service. You can use the _HOST keyword as the instance. Example: jhs/_HOST@RAPIDMINER.COM
Advanced Hadoop Parameters	Key-value properties to customize the Hadoop connection and Radoop's Yarn/MapReduce jobs. Some connections require certain advanced parameters. For detailed information, see the distribution specific notes.

Spark

Field	Description
Spark Version	Spark Version available on the cluster. For more information on using Spark operators, see the Configuring Spark section.
Assembly Jar Location / Spark Archive (or libs) path	The HDFS location or local path (on all cluster nodes) of the Spark Assembly Jar file / Spark Jar files.
Spark Resource Allocation Policy	The resource allocation policy for Spark jobs. The default - Dynamic Resource Allocation starting from 8.1.1 and Static, Heuristic Configuration in 8.1.0 - is typically applicable. See more Spark policy information.
Resource Allocation %	Percentage of cluster resources allocated for a Spark job. This field is only enabled when Static, Heuristic Configuration is the Spark resource allocation policy.
Use custom PySpark archive	Check this box if you want to provide your own PySpark archives.
Custom PySpark archive paths	Only when Use custom PySpark archive option is enabled. Set of archives used as PySpark libraries for PySpark job submissions. See the instructions for configuring custom PySpark/SparkR archives.
Use custom SparkR archive	Check this box if you want to provide your own SparkR archive.
Custom SparkR archive path	Only when Use custom SparkR archive option is enabled. Archive used as the SparkR library for SparkR job submissions. Set of archives used as PySpark libraries for PySpark job submissions. See the instructions for configuring custom PySpark/SparkR archives.
Advanced Spark Parameters	Key-value properties that customize RapidMiner Radoop's Spark jobs. See the instructions for configuring Spark.

Hive

Field	Description
Hive Version	Select the appropriate Data Warehouse System — HiveServer2 (Hive 0.13 or newer) or Impala. Alternatively, you can select Custom HiveServer2 and provide your own Hive jars.
Custom Hive Lib Directory	Only with Custom Hiveserver2 selected. Select a directory that contains the libraries (JAR files) needed to connect to the cluster.
Hive High Availability	Check this box if Hive High Availability is activated for this cluster (provided that HiveServer access is coordinated by ZooKeeper).
Hive Server Address/Impala Server Address	Address (usually hostname) of the node running the Hive Server or the Impala Server.
Hive Port/Impala Port	Port of the Hive Server or Impala Server.
Database name	Name of the database to connect to.
JDBC URL Postfix	Optional postfix for the JDBC URL. The default is "auth=noSasl" for Impala connections.
Username	Username for connecting to the specified database. The default is "hive" for all HiveServer2 version connections. This user should have access to the HDFS directory that Radoop uses for storing files temporarily. If this directory is located in an encryption zone, the user should also have permissions to access the encryption zone key.
Password	Password for connecting to the specified database. RapidMiner Radoop uses the cipher.key file to encrypt the password in radoop_connections.xml.
UDFs are installed manually	Check this box if the Radoop UDFs are installed on the cluster manually. More information on the manual UDF installation can be found on the Operation and Maintenance page.
Use custom database for UDFs	Check this box if a custom database should be used for storing and accessing Radoop UDFs. This is useful when more users (having different project databases and granted privileges) wish to use Radoop. This common database should be accessible by all of them. The UDFs can still be automatically or manually created.
Custom database for UDFs	Only when "Use custom database for UDFs" is checked. Define the database dedicated for storing Radoop UDFs (see above). The database must exist.
Hive on Spark / Tez container reuse	Check this box if you would like to benefit from Hive on Spark / Hive on Tez container reuse.
Hive Principal	Only with Kerberos security enabled. Principal of the Hive service. The format is primary[/<instance>]@<REALM>, where primary is usually the user name, instance is optional, and REALM is the Kerberos realm. Do not use the _HOST keyword as the instance. If Hive is not configured for Kerberos but uses another authentication mechanism (e.g., LDAP), leave this field empty. Example: hive/node02.rapidminer.com@RAPIDMINER.COM.
SASL QoP Level	Level of SASL Quality of Protection. This setting must be the same as the cluster setting. (To find the cluster setting, find the value of hive.server2.thrift.sasl.qop in hive-site.xml; the default is “auth”.)
Advanced Hive Parameters	Key-value properties to customize the behavior of Hive.

RapidMiner Server

This tab contains some multi-user configuration settings that affect the execution on RapidMiner Server. For more information and best practice solutions, see the relevant sections of Installing Radoop on Server page.

Field	Description
Enable impersonation on Server	Check this box if you want to use an impersonated (proxy) Hadoop user on RapidMiner Server.
Server Principal	Only with Kerberos security and Server impersonation enabled. Principal used by RapidMiner Server to access the cluster. The format is primary[/<instance>]@<REALM>, where primary is usually the user name, instance is optional, and REALM is the Kerberos realm. Example: user/server.rapidminer.com@RAPIDMINER.COM. Please note that this setting only affects the execution on Server.
Server Keytab File	Only with Kerberos security and Server impersonation enabled. Path of the server keytab file on the server machine.
Impersonated user for local testing	Only when impersonation on Server is enabled. Server user to impersonate for testing Server connections locally from Studio.
Access Whitelist	Regex for Server users who has access to this connection. Leave it empty or use '*' to enable access to all users.

Radoop Proxy

Field	Description
Use Radoop Proxy	Check this box if you want to access the Hadoop Cluster through a Radoop Proxy.
Radoop Proxy Connection	Only when Radoop Proxy is enabled. This field consists of two dropdown selectors, which together define the Radoop Proxy used for accessing the cluster. The first one defines the location of the Radoop Proxy. Can be local or one of the configured RapidMiner Server repositories. The second one is the identifier of the Radoop Proxy.

XML connection editor

The Radoop connection XML can be edited manually by clicking the Edit XML... button on the Connection Settings dialog. Please note that this feature should be used carefully as it's easy to make mistakes in a connection entry through the XML editor. The main purpose of the editor is to make connection sharing and copy-pasting some parts of it (e.g. Advanced Hadoop Parameters) much easier. When you close the window with the OK button, your changes appear in the fields of the Connection Settings dialog.

Note: Adding a separate key attribute to <radoop-connection-entry> tag in the XML editor will have no effect. It can only be added in radoop_connections.xml, manually.

Configuring non-default properties

If your Hadoop cluster uses non-default properties, additional key-value pairs may be required. Cluster management tools like Cloudera Manager and Ambari allow you to download the client configuration files. You may have to add cluster connection-related properties from these files to the Advanced Hadoop Parameters section of the Hadoop tab. See below for single properties that occur to be (re)set frequently, and more complex examples describing the properties required to connect to a cluster with High Availability (HA) enabled. The following tables list the keys of the potentially required client-side settings. The values should be set to the appropriate property values from the client configuration files. Note that not all keys related to these features may be required, the required set of key-value pairs depend on your cluster settings.

Configuring Spark for a RapidMiner Radoop connection

By configuring Spark for a RapidMiner Radoop connection, you enable the Spark operators. See the exact Spark version requirements for each operator on the Installing Radoop on Studio page.

To enable Spark, select a valid Spark version from the dropdown list in the Connection Settings dialog.

You must provide the following mandatory inputs on the Spark tab of the Connection Settings dialog:

Field	Description
Spark Version	Dropdown list to select the version of Spark your cluster supports. If Spark is not on the cluster or unneeded, select None. Spark 1.2.x, 1.3.x, or 1.4.x, select Spark 1.4 or below. Spark 1.5.x, 1.6.x, 2.0.x, 2.1.x, or 2.2.x, select the corresponding value. Spark 2.3.x users will need to select based on the Spark patch version number. Spark 2.3.0 will be Spark 2.3.0, for all other 2.3.x versions select Spark 2.3.1+.
Assembly Jar location / Spark Archive (or libs) path	The HDFS or local path of the distribution-specific Spark assembly JAR file / Spark JAR files. If you provide a local path then it must be the same on every node in your cluster. Specifying the local path is recommended if Spark is automatically installed (e.g. with Cloudera Manager or Ambari) on the cluster. For some Hadoop versions, the pre-built Spark assembly JAR can be downloaded from the Apache Spark download page. Some vendors (like Cloudera) provide a distribution-specific Spark assembly JAR. For the HDFS path of the JAR, contact your Hadoop administrator. For example, to install Spark 1.5 manually, refer to the Spark requirements section. If you followed the instructions there, your assembly jar is at the following location on the HDFS: hdfs:///tmp/spark/spark-assembly-1.5.2-hadoop2.6.0.jar
Spark resource allocation policy	Spark needs specification of the cluster resources it is allowed use. See the Spark resource allocation policy descriptions
Advanced Spark Parameters	Key-value pairs that can be applied to a Spark-on-YARN job. If the change has no effect on your Spark job, most likely it is ignored by YARN itself. To check the properties in the application log, set spark.logConf to true.

Configuring custom PySpark/SparkR archives for Spark

Radoop is shipped with PySpark and SparkR archives for each minor (x.y) Spark version to support Spark Scripting operator. Using these archives for all sub-versions (x.y.z) is sufficient in most cases. However certain Spark minor versions (e.g. 2.2 and 2.3) shipped by Hadoop distributions have multiple incompatible patched versions which behave differently from the aspect of Python/R process <-> JVM communication. These minor versions cannot be tackled by shipping a single set of archives. Hence custom PySpark and SparkR archive options were introduced on the connection editor. When these options are enabled, Radoop uses the user-provided archives to execute Spark Script operator, instead of the ones bundled with Radoop. These archives are usually shipped with the Hadoop distribution together with Spark so typically located close to Spark installation folder. This functionality is handled by the following extra settings:

Field	Description
Use custom PySpark archive	Check this box if you want to provide your own PySpark archives.
Custom PySpark archive paths	Only when "Use custom PySpark archive" option is enabled. Set of archives used as PySpark libraries for PySpark job submissions. You will typically need to provide two archives here, pyspark.zip and py4j-*.zip. The exact name and access path of these files depend on the Hadoop and Spark version of your cluster. Since you will need to provide at least two items, this parameter accepts multiple values. Each entry can be provided either as an HDFS location (hdfs:// protocol), as a file which is available on all cluster nodes at the same location (local:// protocol), or as a file on the client machine (file:// protocol). In a sample HDP 3 environment the necessary entries using local paths are local:///usr/hdp/current/spark2-client/python/lib/pyspark.zip and local:///usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip.
Use custom SparkR archive	Check this box if you want to provide your own SparkR archive.
Custom SparkR archive path	Only when "Use custom SparkR archive" option is enabled. Archive used as the SparkR library for SparkR job submissions. This path can be provided either as an HDFS location (hdfs:// protocol) or as a file on the client machine (file:// protocol). WARNING! Specifying archives available on the cluster nodes (local:// protocol) is not supported for this parameter. Therefore if your archive is accessible on a cluster node, you will first need to upload it to HDFS, and use the HDFS location for this parameter. In a sample HDP 3 environment this file is located at /usr/hdp/current/spark2-client/R/lib/sparkr.zip. In the example shown below, this file was uploaded to hdfs:///tmp/sparkr.zip HDFS location, which is then referenced by this parameter.

Spark resource allocation policies

RapidMiner Radoop supports the following resource allocation policies:

Dynamic Resource Allocation

• Default option starting from 8.1.1. While this policy requires configuration on the server, many server already have this installed.

• With this policy you may need to configure an external shuffle service on the cluster. For more information about the required cluster configuration steps see the Spark Dynamic Allocation documentation.

• The following properties may be defined under Advanced Spark Parameters on the Spark tab of Connection Settings dialog - they are only mandatory when Spark 1.4 or below is selected for Spark Version:

   - `spark.dynamicAllocation.minExecutors`
   - `spark.dynamicAllocation.maxExecutors`
  

• Cluster specific info

  • Cloudera - Dynamic Allocation
  • HortonWorks - How to enable Dynamic Resource Allocation in Spark
  • MapR - Enabling Dynamic Allocation in Apache Spark

Static, Heuristic Configuration

• This is the default policy in 8.1.0 and previous versions. If you use this option, you do not need to set any advanced resource allocation settings. The Resource Allocation % field sets the percentage of cluster resources (cluster memory, number of cores) to be used for a Spark job. Note that if you set this value too high, other jobs on the cluster might suffer. The default value is 70%.

Static, Default Configuration

• A policy that uses Spark's default settings for resource allocation. This value is very low and may not support a real cluster, but it may be a viable option for VMs/sandboxes.

Static, Manual Configuration

• This policy requires that you set the following properties under Advanced Spark Parameters on the Spark tab of the Connection Settings dialog. The Spark documentation describes each property. (The corresponding Spark on YARN command line arguments are shown in parentheses.)
  • spark.executor.cores (--executor-cores)
  • spark.executor.instances (--num-executors)
  • spark.executor.memory (--executor-memory)
  • (optional) spark.driver.memory (--driver-memory)
    

Note: Because of SPARK-6962, RapidMiner Radoop changes the default value of spark.shuffle.blockTransferService to nio instead of netty. To override this setting, in the Advanced Spark Parameters field configure the key spark.shuffle.blockTransferService to the value netty. Starting from 1.6.0, this setting is ignored by Spark, the BlockTransferService is always netty.

Hive on Spark & Hive on Tez container reuse

Reusing the containers of the Hive execution engine can dramatically speed up Radoop processes, especially if there are lots of Hive-only tasks. It is achieved by keeping a number of Spark / Tez containers (applications) in running state for executing Hive queries. Keep in mind that these containers will use cluster resources even if there are no running processes. Radoop tries to estimate the optimal number of containers by default, but it also can be changed to a fix number in the settings (see below). Idle containers are automatically closed after a timeout.

To use this feature, your cluster must support Hive on Spark or Hive on Tez, and in your connection have to set hive.execution.engine to spark or tez in Advanced Hive Parameters and check Hive on Spark / Tez container reuse checkbox (this is the default value):

A number of global Radoop settings can be used to control the container reuse behaviour. You may want to test different settings to use your cluster optimally, see Radoop Settings for details.

Impala connections

If your are configuring an Impala connection some Advanced Hadoop Parameters need to be added manually. If you forget to add any of those, a warning message will warn you about the missing ones. The Add Required Entries button adds the keys of these properties to the list, but their values must be set manually according to the cluster configuration.