Skip to Main content Skip to Navigation
Journal articles

Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer

Abstract : HPC and Big Data stacks are completely separated today. The storage layer offers opportunities for convergence, as the challenges associated with HPC and Big Data storage are similar: trading versatility for performance. This motivates a global move towards dropping file-based, POSIX-IO compliance systems. However, on HPC platforms this is made difficult by the centralized storage architecture using file-based storage. In this paper we advocate that the growing trend of equipping HPC compute nodes with local storage redistributes the cards by enabling object storage to be deployed alongside the application on the compute nodes. Such integration of application and storage not only allows fine-grained configuration of the storage system, but also improves application portability across platforms. In addition, the single-user nature of such application-specific storage obviates the need for resource-consuming storage features like permissions or file hierarchies offered by traditional file systems. In this article we propose and evaluate Blobs (Binary Large Objects) as an alternative to distributed file systems. We factually demonstrate that it offers drop-in compatibility with a variety of existing applications while improving storage throughput by up to 28%.
Complete list of metadatas

Cited literature [47 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01892682
Contributor : Gabriel Antoniu <>
Submitted on : Wednesday, October 10, 2018 - 6:01:36 PM
Last modification on : Wednesday, June 24, 2020 - 4:19:43 PM

File

HPC_BD_Convergence__Elsevier_-...
Files produced by the author(s)

Identifiers

Citation

Pierre Matri, Yevhen Alforov, Alvaro Brandon, María Pérez, Alexandru Costan, et al.. Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer. Future Generation Computer Systems, Elsevier, 2018, pp.1-10. ⟨10.1016/j.future.2018.07.035⟩. ⟨hal-01892682⟩

Share

Metrics

Record views

923

Files downloads

286