A Proposal of Scalable and Performing Implementation of Algorithms for Anomaly and Community Detection

Yaya Sylla (1) , Pierre Morizet-Mahoudeaux (1)

1 Heudiasyc - Heuristique et Diagnostic des Systèmes Complexes [Compiègne]

This paper presents a study of various standard anomalies detection techniques in internet networks, using machine learning algorithms, classification algorithms, and graph techniques working on SQL/Mapreduce and SQL-Graph implementation platforms. This approach shows its efficiency for the study of large datasets. Firstly, several algorithmic approaches and possible implementations have been studied and experimentally tested to see how the SQL-MapReduce and SQL-Graph implementation can process large-scale data. Secondly the performance and scalability of the algorithms for large volumes of data have been compared to choose the most appropriate for typical anomaly detection. The application scope is very broad, such as spam detection, crime detection, mafia or terrorism community detection, network intrusion detection, malignant tumors detection in healthcare, fraud detection on banking transactions, identity theft detection etc. The main problem we will address on this paper is the processing performance on large scale data using an implementation of Massively Parallel Processing (MPP), SQL-MapReduce, and SQl-Graph.

Mots clés: Anomaly; Graph; Big Data; Classification; Machine Learning; Modularity; MapReduce

Domaines: Calcul parallèle, distribué et partagé [cs.DC]; Algorithme et structure de données [cs.DS]; Apprentissage [cs.LG]; Réseaux sociaux et d'information [cs.SI]

Yaya Sylla, Pierre Morizet-Mahoudeaux. A Proposal of Scalable and Performing Implementation of Algorithms for Anomaly and Community Detection. IEEE International Conference on Big Data (BigData 2018), Dec 2018, Seatle, WA, United States. pp.3613-3621, ⟨10.1109/BigData.2018.8622510⟩. ⟨hal-02022781⟩