Skip to Main content Skip to Navigation
Conference papers

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Abstract : Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG's design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.
Complete list of metadatas

Cited literature [35 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01892685
Contributor : Pierre Matri <>
Submitted on : Wednesday, October 10, 2018 - 6:05:33 PM
Last modification on : Tuesday, February 25, 2020 - 8:08:10 AM

File

HPC_Logging___ICDCS_18_Short_P...
Files produced by the author(s)

Identifiers

Citation

Pierre Matri, Philip Carns, Robert Ross, Alexandru Costan, María Pérez, et al.. SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence. ICDCS 2018 - IEEE 38th International Conference on Distributed Computing Systems, Jul 2018, Vienna, Austria. pp.1-6, ⟨10.1109/ICDCS.2018.00156⟩. ⟨hal-01892685⟩

Share

Metrics

Record views

958

Files downloads

262