SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Abstract : Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG's design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.
Type de document :
Communication dans un congrès
ICDCS 2018 - IEEE 38th International Conference on Distributed Computing Systems, Jul 2018, Vienna, Austria. IEEE, pp.1-6, 〈10.1109/ICDCS.2018.00156〉
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01892685
Contributeur : Pierre Matri <>
Soumis le : mercredi 10 octobre 2018 - 18:05:33
Dernière modification le : vendredi 16 novembre 2018 - 01:28:11

Fichier

HPC_Logging___ICDCS_18_Short_P...
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Pierre Matri, Philip Carns, Robert Ross, Alexandru Costan, María Pérez, et al.. SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence. ICDCS 2018 - IEEE 38th International Conference on Distributed Computing Systems, Jul 2018, Vienna, Austria. IEEE, pp.1-6, 〈10.1109/ICDCS.2018.00156〉. 〈hal-01892685〉

Partager

Métriques

Consultations de la notice

794

Téléchargements de fichiers

55