Multi Label Performance
(Time Series)
Synopsis
This operator evaluates the performance of a Multi Label Model.Description
A Multi Label Model is a meta model able to predict multiple label attributes at once. For each label attribute, the Multi Label Model contains a Prediction Model. It can be trained by the Multi Label Modeling operator.
The Multi Label Performance operator can be used to evaluate the performance of such a Multi Label Model after it was applied on a test ExampleSet by using the Apply Model operator. The result of the application of the Multi Label Model on a test ExampleSet has to be provided to the labelled set input port.
The operator can automatically detect the label and prediction attributes for which the performances shall be evaluated. All special attributes with a role prediction_<name of label attribute> are considered to be prediction attributes, while the name of the label attribute is extracted from the role. The label attributes can also be selected by an attribute filter if the parameter auto detect label and prediction attributes is deselected.
For each of the selected label attributes, the ExampleSet at the labelled set input port has to contain the label attribute and the corresponding prediction attribute. If the label attribute is nominal, the ExampleSet has to contain also the corresponding confidence attributes.
The inner subprocess of the operator is executed for each selected label attribute. The roles of the current label, current prediction and (if existing) the current confidence attributes are set to the corresponding roles and the input data is provided at the inner labelled set port. It can be used to evaluate the performance for the corresponding label attribute. A Performance Vector has to be provided to the performance port of the inner subprocess. Additional objects can be passed in and out of the subprocess through the input and output port extender.
All Performance Vectors are collected and provided as a collection at the collection of performances output port. If possible the average of the Performance Vectors is calculated (only possible if the types and number of Performance Criteria in each Performance Vector are the same). The averaged Performance Vector for all label attributes is provided at the performance output port.
To ensure a defined behavior the input ExampleSet is not allowed to have a normal 'Label' or 'Prediction' attribute. If the input data has such attributes, their roles have to be set to different roles with the Set Role operator. Note that the role of all other attributes are not changed. The name and type of the current label attribute in the subprocess can be added as macros if the parameter add macros is selected.
Input
labelled set (Data table)The input ExampleSet for which the multi label prediction performances are evaluated. The input ExampleSet needs to have the label, the prediction attribute and (if the label is nominal) confidence attributes for each selected label attribute.
- input (IOObject)
This port is a port extender, which means if a port is connected a new input port is created. Any IOObject can be connected to the port and is passed to the corresponding inner input port for each iteration.
Output
collection of performances (Collection)A collection of Performance Vectors for each of the selected label and prediction attribute.
performance (Performance Vector)An averaged Performance Vector of the performances of all selected label and prediction attributes. The calculation of the averaged Performance Vector is only possible if the types and number of Performance Criteria in each Performance Vector are the same.
- example set (Data table)
The ExampleSet that was given as input is passed through without changes.
- output (IOObject)
This port is a port extender, which means if a port is connected a new output port is created. The port collects every result that is provided by the inner process and returns a collections of all iterations.
Parameters
- auto_detect_label_and_prediction_attributes
If selected the label and prediction attributes are automatically detected. All special attributes with a role prediction_<name of label attribute> are considered to be prediction attributes, while the name of the label attribute is extracted from the role.
Range: - attribute_filter_type
This parameter allows you to select the filter for the label attributes selection; the method you want to select the attributes for which performances are evaluated. Note that the filter is applied on all attributes indepent of their special role (the parameter include special attributes is set to true for this attribute selection and cannot be changed). The different filter types are:
- all: This option selects all attributes of the ExampleSet to be label attributes. This is the default option.
- single: This option allows the selection of a single label attribute. The required attribute is selected by the attribute parameter.
- subset: This option allows the selection of multiple label attributes through a list (see parameter attributes). If the meta data of the ExampleSet is known all attributes are present in the list and the required ones can easily be selected.
- regular_expression: This option allows you to specify a regular expression for the label attribute selection. The regular expression filter is configured by the parameters regular expression, use except expression and except expression.
- value_type: This option allows selection of all the attributes of a particular type to be label attributes. It should be noted that types are hierarchical. For example real and integer types both belong to the numeric type. The value type filter is configured by the parameters value type, use value type exception, except value type.
- block_type: This option allows the selection of all the attributes of a particular block type to be label attributes. It should be noted that block types may be hierarchical. For example value_series_start and value_series_end block types both belong to the value_series block type. The block type filter is configured by the parameters block type, use block type exception, except block type.
- no_missing_values: This option selects all attributes of the ExampleSet as label attributes which do not contain a missing value in any example. Attributes that have even a single missing value are not selected.
- numeric_value_filter: All numeric attributes whose examples all match a given numeric condition are selected as label attributes. The condition is specified by the numeric condition parameter.
- attribute
The required attribute can be selected from this option. The attribute name can be selected from the drop down box of the parameter if the meta data is known.
Range: - attributes
The required attributes can be selected from this option. This opens a new window with two lists. All attributes are present in the left list. They can be shifted to the right list, which is the list of selected label attributes.
Range: - regular_expression
Attributes whose names match this expression will be selected. The expression can be specified through the edit and preview regular expression menu. This menu gives a good idea of regular expressions and it also allows you to try different expressions and preview the results simultaneously.
Range: - use_except_expression
If enabled, an exception to the first regular expression can be specified. This exception is specified by the except regular expression parameter.
Range: - except_regular_expression
This option allows you to specify a regular expression. Attributes matching this expression will not be selected even if they match the first expression (expression that was specified in regular expression parameter).
Range: - value_type
This option allows to select a type of attribute.
Range: - use_value_type_exception
If enabled, an exception to the selected type can be specified. This exception is specified by the except value type parameter.
Range: - except_value_type
The attributes matching this type will not be selected even if they matched the before selected type, specified by the value type parameter.
Range: - block_type
This option allows to select a block type of attribute.
Range: - use_block_type_exception
If enabled, an exception to the selected block type can be specified. This exception is specified by the except block type parameter.
Range: - except_block_type
The attributes matching this block type will not be selected even if they matched the before selected type by the block type parameter.
Range: - numeric_condition
The numeric condition used by the numeric condition filter type. A numeric attribute is selected if all examples match the specified condition for this attribute. For example the numeric condition '> 6' will keep all numeric attributes having a value of greater than 6 in every example. A combination of conditions is possible: '> 6 && < 11' or '<= 5 || < 0'. But && and || cannot be used together in one numeric condition. Conditions like '(> 0 && < 2) || (>10 && < 12)' are not allowed because they use both && and ||.
Range: - invert_selection
If this parameter is set to true the selection is reversed. In that case all attributes not matching the specified condition are selected as label attributes.
Range: - add_macros
If selected macros containing the name and the value type of the current label attribute are added in each iteration of the subprocess.
Range: - current_label_name_macro
If add macros is true, this parameter defines the name of the macro holding the current label attribute name.
Range: - current_label_type_macro
If add macros is true, this parameter defines the name of the macro holding the current label attribute type.
Range: - enable_parallel_execution
This parameter enables the parallel execution of the inner processes. Please disable the parallel execution if you run into memory problems.
Range:
Tutorial Processes
Multi Label Performance on Titanic data set
This tutorial process shows the basic usage of the Multi Label Performance operator by training a Multi Label Model for the attributes 'Survived', 'Port of Embarkation' and 'Age' of the Titanic samples data set and then evaluating the prediction performance using the Multi Label Performance operator.
