Skip to Main content Skip to Navigation

Contribution to High Performance Computing and Big Data Infrastructure Convergence

Michael Mercier 1, 2
2 DATAMOVE - Data Aware Large Scale Computing
Inria Grenoble - Rhône-Alpes, LIG - Laboratoire d'Informatique de Grenoble
Abstract : The amount of data produced, either in the scientific community and the commercial world, is constantly growing. The field of Big Data has emerged to handle a large amount of data on distributed computing infrastructures. High-Performance Computer (HPC) infrastructures are made for intensive parallel computations. The HPC community is also facing more and more data because of new high definition sensors and large physics apparatus. The convergence of the two fields is currently happening. In fact, the HPC community is already using Big Data tools, but they are not integrated correctly, especially at the level of the file system and the Resources and Job Management System (RJMS).In order to understand how we can leverage HPC clusters for Big Data usage, and what are the challenges for the HPC infrastructures, we have studied multiple aspects of the convergence: we have made a survey on the software provisioning methods, with a focus on data-intensive applications. We also propose a new RJMS collaboration technique called BeBiDa which is based on 50 lines of code whereas similar solutions use at least 1000x more. We evaluate this mechanismon real conditions and in a simulation with our simulator Batsim.
Document type :
Complete list of metadata

Cited literature [235 references]  Display  Hide  Download
Contributor : Abes Star :  Contact
Submitted on : Thursday, October 31, 2019 - 11:15:18 AM
Last modification on : Tuesday, March 2, 2021 - 9:42:30 AM


Version validated by the jury (STAR)


  • HAL Id : tel-02284996, version 2


Michael Mercier. Contribution to High Performance Computing and Big Data Infrastructure Convergence. Symbolic Computation [cs.SC]. Université Grenoble Alpes, 2019. English. ⟨NNT : 2019GREAM031⟩. ⟨tel-02284996v2⟩



Record views


Files downloads