Collecting and Characterizing Distributed Machine Learning Workloads

Yasmine Djebrouni (1, 2) , Sara Bouchenak (1) , Khalid Benabdeslem (1)

1 INSA Lyon - Institut National des Sciences Appliquées de Lyon
2 ESI - École Nationale Supérieure d'Informatique [Alger]

Machine learning is a key for transforming data into actionable knowledge. The rapid increase in the amount of analyzed data forced the switch to distributed ML platforms. However, the complexity of such platforms is overwhelming for uninitiated users, who may not understand the trade-offs and the challenges of parameterizing such systems to achieve good performance. In order to better analyze and understand ML workloads running on ML distributed platforms, we conducted extensive experiments with various ML methods and real-world datasets, and collected the execution traces of these distributed ML workloads, that represent a total of 12 GB of traces and tens of millions of data records. We then provide a statistical analysis of the collected traces, and illustrate through a use case how different ML workloads' are characterized and their needs identified.

Mots clés: Distributed Machine Learning; ML Workload Characterization; Trace Collection; Distributed Machine Learning

Domaines: Intelligence artificielle [cs.AI]; Calcul parallèle, distribué et partagé [cs.DC]

Yasmine Djebrouni, Sara Bouchenak, Khalid Benabdeslem. Collecting and Characterizing Distributed Machine Learning Workloads. 2021. ⟨hal-03343275⟩