Large Scale Experimentation on Anomaly Detection Scalability and Performance

Yaya Sylla (1) , Pierre Morizet-Mahoudeaux (1) , Stephen Brobst

1 Heudiasyc - Heuristique et Diagnostic des Systèmes Complexes [Compiègne]

This paper presents a study of various standard anomalies detection techniques in internet networks, using machine learning algorithms, classification algorithms, and graph techniques working on SQL/Mapreduce and SQL-Graph implementation platforms. This approach shows its efficiency for the study of large datasets. Firstly, several algorithmic approaches and possible implementations have been studied and tested by experiment to see how the SQL-MapReduce and SQL-Graph implementation can process large-scale data. Secondly the performance and scalability of the algorithms for large volumes of data have been compared to choose the most appropriate for typical anomaly detection. The application scope is very broad, such as spam detection, crime detection, mafia or terrorism community detection, network intrusion detection, malignant tumors detection in healthcare, fraud detection on banking transactions, identity theft detection etc. The main problem we will address on this paper is the processing performance on large scale data using an implementation of Massively Parallel Processing (MPP), SQL-MapReduce, and SQl-Graph.

Mots clés: component; Anomaly; Graph; Big Data; Classification; Machine Learning; Modularity; MapReduce

Domaines: Informatique [cs]; Calcul parallèle, distribué et partagé [cs.DC]; Apprentissage [cs.LG]; Réseaux sociaux et d'information [cs.SI]; Algorithme et structure de données [cs.DS]

Yaya Sylla, Pierre Morizet-Mahoudeaux, Stephen Brobst. Large Scale Experimentation on Anomaly Detection Scalability and Performance. 20th International Conference on Artificial Intelligence (ICAI 2018), Jul 2018, Las Vegas, United States. pp.36-43. ⟨hal-02022807⟩