From Big Data to Fast Data: Efficient Stream Data Management

Alexandru Costan 1, 2
2 KerData - Scalable Storage for Clouds and Beyond
Inria Rennes – Bretagne Atlantique , IRISA_D1 - SYSTÈMES LARGE ÉCHELLE
Abstract : This manuscript provides a synthetic overview of my research journey since my PhD defense. The document does not claim to present my work in its entirety, but focuses on the contributions to data management in support of stream processing. These results address all stages of the stream processing pipeline: data collection and in-transit processing at the edge, transfer towards the cloud processing sites, ingestion and persistent storage. I start by presenting the general context of stream data management in light of the recent transition from Big to Fast Data. After highlighting the challenges at the data level associated with batch and real-time analytics, I introduce a subjective overview of my proposals to address them. They bring solutions to the problems of in-transit stream storage and processing, fast data transfers, distributed metadata management, dynamic ingestion and transactional storage. The integration of these solutions into functional prototypes and the results of the large-scale experimental evaluations on clusters, clouds and supercomputers demonstrate their effectiveness for several real-life applications ranging from neuro-science to LHC nuclear physics. Finally, these contributions are put into the perspective of the High Performance Computing - Big Data convergence.
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/tel-02059437
Contributeur : Alexandru Costan <>
Soumis le : jeudi 14 mars 2019 - 19:11:49
Dernière modification le : mercredi 20 mars 2019 - 10:57:02

Identifiants

  • HAL Id : tel-02059437, version 2

Citation

Alexandru Costan. From Big Data to Fast Data: Efficient Stream Data Management. Distributed, Parallel, and Cluster Computing [cs.DC]. ENS Rennes, 2019. 〈tel-02059437v2〉

Partager

Métriques

Consultations de la notice

101

Téléchargements de fichiers

38