Detect Outlier (Densities) (RapidMiner Studio Core)

Synopsis

This operator identifies outliers in the given ExampleSet based on the data density. All objects that have at least p proportion of all objects farther away than distance D are considered outliers.

Description

The Detect Outlier (Densities) operator is an outlier detection algorithm that calculates the DB(p,D)-outliers for the given ExampleSet. A DB(p,D)-outlier is an object which is at least D distance away from at least p proportion of all objects. The two real-valued parameters p and D can be specified through the proportion and distance parameters respectively. The DB(p,D)-outliers are distance-based outliers according to Knorr and Ng. This operator implements a global homogenous outlier search.

This operator adds a new boolean attribute named 'outlier' to the given ExampleSet. If the value of this attribute is true, that example is an outlier and vice versa. Different distance functions are supported by this operator. The desired distance function can be selected by the distance function parameter.

An outlier is an example that is numerically distant from the rest of the examples of the ExampleSet. An outlying example is one that appears to deviate markedly from other examples of the ExampleSet. Outliers are often (not always) indicative of measurement error. In this case such examples should be discarded.

Input

example set input (Data Table)
This input port expects an ExampleSet. It is the output of the Generate Data operator in the attached Example Process. The output of other operators can also be used as input.

Output

example set output (Data Table)
A new boolean attribute 'outlier' is added to the given ExampleSet and the ExampleSet is delivered through this output port.
original (Data Table)
The ExampleSet that was given as input is passed without changing to the output through this port. This is usually used to reuse the same ExampleSet in further operators or to view the ExampleSet in the Results Workspace.

Parameters

distanceThis parameter specifies the distance D parameter for calculation of the DB(p,D)-outliers. Range: real
proportionThis parameter specifies the proportion p parameter for calculation of the DB(p,D)-outliers. Range: real
distance_functionThis parameter specifies the distance function that will be used for calculating the distance between two examples. Range: selection

Tutorial Processes

Detecting outliers from an ExampleSet

The Generate Data operator is used for generating an ExampleSet. The target function parameter is set to 'gaussian mixture clusters'. The number examples and number of attributes parameters are set to 200 and 2 respectively. A breakpoint is inserted here so that you can view the ExampleSet in the Results Workspace. A good plot of the ExampleSet can be seen by switching to the 'Plot View' tab. Set Plotter to 'Scatter', x-Axis to 'att1' and y-Axis to 'att2' to view the scatter plot of the ExampleSet.

The Detect Outlier (Densities) operator is applied on the ExampleSet. The distance and proportion parameters are set to 4.0 and 0.8 respectively. The resultant ExampleSet can be viewed in the Results Workspace. For better understanding switch to the 'Plot View' tab. Set Plotter to 'Scatter', x-Axis to 'att1', y-Axis to 'att2' and Color Column to 'outlier' to view the scatter plot of the ExampleSet (the outliers are marked red). The number of outliers may differ depending on the randomization, if the random seed parameter of the process is set to 1997, you will see 5 outliers.