Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer

Abstract : HPC and Big Data stacks are completely separated today. The storage layer offers opportunities for convergence, as the challenges associated with HPC and Big Data storage are similar: trading versatility for performance. This motivates a global move towards dropping file-based, POSIX-IO compliance systems. However, on HPC platforms this is made difficult by the centralized storage architecture using file-based storage. In this paper we advocate that the growing trend of equipping HPC compute nodes with local storage redistributes the cards by enabling object storage to be deployed alongside the application on the compute nodes. Such integration of application and storage not only allows fine-grained configuration of the storage system, but also improves application portability across platforms. In addition, the single-user nature of such application-specific storage obviates the need for resource-consuming storage features like permissions or file hierarchies offered by traditional file systems. In this article we propose and evaluate Blobs (Binary Large Objects) as an alternative to distributed file systems. We factually demonstrate that it offers drop-in compatibility with a variety of existing applications while improving storage throughput by up to 28%.
Type de document :
Article dans une revue
Future Generation Computer Systems, Elsevier, 2018, pp.1-10. 〈10.1016/j.future.2018.07.035〉
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01892682
Contributeur : Gabriel Antoniu <>
Soumis le : mercredi 10 octobre 2018 - 18:01:36
Dernière modification le : jeudi 15 novembre 2018 - 11:58:57

Fichier

HPC_BD_Convergence__Elsevier_-...
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Pierre Matri, Yevhen Alforov, Alvaro Brandon, María Pérez, Alexandru Costan, et al.. Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer. Future Generation Computer Systems, Elsevier, 2018, pp.1-10. 〈10.1016/j.future.2018.07.035〉. 〈hal-01892682〉

Partager

Métriques

Consultations de la notice

770

Téléchargements de fichiers

87