Extracting Hidden Anomalies using Sketch and Non Gaussian Multiresolution Statistical Detection Procedures

Abstract : A new profile-based anomaly detection and characterization procedure is proposed. It aims at performing prompt and accurate detection of both short-lived and long-lasting low-intensity anomalies, without the recourse of any prior knowledge of the targetted traffic. Key features of the algorithm lie in the joint use of random projection techniques (sketches) and of a multiresolution non Gaussian marginal distribution modeling. The former enables both a reduction in the dimensionality of the data and the measurement of the reference (i.e., normal) traffic behavior, while the latter extracts anomalies at different aggregation levels. This procedure is used to blindly analyze a large-scale packet trace database collected on a trans-Pacific transit link from 2001 to 2006. It can detect and identify a large number of known and unknown anomalies and attacks, whose intensities are low (down to below one percent). Using sketches also makes possible a real-time identification of the source or destination IP addresses associated to the detected anomaly and hence their mitigation.