ROBUST OUTLIER DETECTION TECHNIQUES FOR SKEWED DISTRIBUTIONS AND APPLICATIONS TO REAL DATA

Abstract

Most of the data sets belonging to the real world contain observations at the extremes that might not be in conformity with the remaining data set. These extreme observations known to be outliers might have positive or negative effect on the data analysis like regression estimates, forecasting and ANOVA etc. Outliers are powerful tools to identify the most interesting events of the world in cross sectional data and historically important events can be picked by detecting outliers in time series data sets. Numerous outlier detection techniques have been proposed in the literature. This study provides a survey of these techniques and their properties. Most of these techniques work well under the assumption that data come from a symmetric distribution and these techniques fail to work in skewed distributions. Because of this limitation, Hubert and Vandervieren (2008) proposed a technique for outlier‟s detection in skewed data sets. Our thesis presents a new technique to measure robust skewness (SSS) and a new outlier detection technique (SSSBB) for skewed data distributions. The study shows that the proposed technique measures skewness more accurately than existing techniques and the proposed technique for outlier‟s detections works better than Hubert‟s technique on a class of theoretically skewed and symmetric distributions. The study also compares the technique with other established outlier detection techniques in the literature. This study uses simulation technique for computer generated distributions and some real data sets for comparison purposes. The study also analyzes real life data sets and compares the baby birth weight data and stock returns, both of which are known to be skewed. These results will help us in making a choice of appropriate outlier detection technique for skewed data sets for different sample sizes which might be helpful in identifying underweight babies.