Multi Horizon Performance (Time Series)
SynopsisThis operator evaluates the performance of a Multi Horizon Forecast Model.
A Multi Horizon ForecastModel is a meta model able to predict multiple horizon attributes at once. For each horizon attribute, the Multi Horizon Forecast Model contains a Prediction Model. It can be trained by the Multi Horizon Forecast operator.
The Multi Horizon Performance operator can be used to evaluate the performance of such a Multi Horizon Forecast Model after it was applied on a test ExampleSet by using the Apply Model operator. The result of the application of the Multi Horizon Forecast Model on a test ExampleSet has to be provided to the labelled set input port.
The operator can automatically detect the horizon and prediction attributes for which the performances shall be evaluated. All special attributes with a role prediction_<name of label attribute> are considered to be prediction attributes, while the name of the horizon attribute is extracted from the role. The horizon attributes can also be selected using a regular expression on the attribute roles if the parameter auto detect horizon and prediction attributes is deselected. The default regular expression (if parameter use default regular expression is selected) is: "horizon \+ (\d+)" which matches the attribute roles created by the Windowing or Process Windows operators.
For each of the selected horizon attributes, the ExampleSet at the labelled set input port has to contain the horizon attribute and the corresponding prediction attribute. If the horizon attribute is nominal, the ExampleSet has to contain also the corresponding confidence attributes.
The inner subprocess of the operator is executed for each selected horizon attribute. The roles of the current horizon, current prediction and (if existing) the current confidence attributes are set to the corresponding roles and the input data is provided at the inner labelled set port. It can be used to evaluate the performance for the corresponding horizon attribute. A Performance Vector has to be provided to the performance port of the inner subprocess. Additional objects can be passed in and out of the subprocess through the input and output port extender.
All Performance Vectors are collected and provided as a collection at the collection of performances output port. If possible the average of the Performance Vectors is calculated (only possible if the types and number of Performance Criteria in each Performance Vector are the same). The averaged Performance Vector for all horizon attributes is provided at the performance output port.
To ensure a defined behavior the input ExampleSet is not allowed to have a normal 'Label' or 'Prediction' attribute. If the input data has such attributes, their roles have to be set to different roles with the Set Role operator. Note that the role of all other attributes are not changed. The name and type of the current horizon attribute in the subprocess can be added as macros if the parameter add macros is selected.
- labelled set (Data Table)
The input ExampleSet for which the multi horizon prediction performances are evaluated. The input ExampleSet needs to have the horizon, the prediction attribute and (if the horizon is nominal) confidence attributes for each selected horizon attribute.
- input (IOObject)
This port is a port extender, which means if a port is connected a new input port is created. Any IOObject can be connected to the port and is passed to the corresponding inner input port for each iteration.
- collection of performances (Collection)
A collection of Performance Vectors for each of the selected horizon and prediction attribute.
- performance (Performance Vector)
An averaged Performance Vector of the performances of all selected horizon and prediction attributes. The calculation of the averaged Performance Vector is only possible if the types and number of Performance Criteria in each Performance Vector are the same.
- example set (Data Table)
The ExampleSet that was given as input is passed through without changes.
- output (IOObject)
This port is a port extender, which means if a port is connected a new output port is created. The port collects every result that is provided by the inner process and returns a collections of all iterations.
If selected the horizon and prediction attributes are automatically detected. All special attributes with a role prediction_<name of horizon attribute> are considered to be prediction attributes, while the name of the horizon attribute is extracted from the role.Range:
If this parameter is selected a default regular expression is used, which matches the attribute roles created by the Windowing or the Process Windows operators. The default regular expression is: "horizon \+ (\d+)"Range:
Specify the regular expression used to select the horizon attributes. The regular expression is applied on all special roles in the input ExampleSet. The expression can be specified through the edit and preview regular expression menu. This menu gives a good idea of regular expressions and it also allows you to try different expressions and preview the results simultaneously.Range:
If selected macros containing the name and the value type of the current horizon attribute are added in each iteration of the subprocess.Range:
If add macros is true, this parameter defines the name of the macro holding the current horizon attribute name.Range:
If add macros is true, this parameter defines the name of the macro holding the current horizon attribute type.Range:
This parameter enables the parallel execution of the inner processes. Please disable the parallel execution if you run into memory problems.Range:
Multi Horizon Performance on windowed Lake Huron Data
This tutorial process shows the basic usage of the Multi Horizon Performance operator by training a Multi Horizon Forecast Model on a windowed Lake Huron data set and then evaluating the horizon forecast performance by using the Multi Horizon Performance operator within a Sliding Window Validation.