A Lightweight Continuous Jobs Mechanism for MapReduce Frameworks - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2013

A Lightweight Continuous Jobs Mechanism for MapReduce Frameworks

Résumé

MapReduce is a programming model which allows the processing of vast amounts of data in parallel, on a large number of machines. It is particularly well suited to static or slow changing set of data since the execution time of a job is usually high. However, in practice data-centers collect data at fast rates which makes it very difficult to maintain up-to-date results. To address this challenge, we propose in this paper a generic mechanism for dealing with dynamic data in MapReduce frameworks. Long-standing MapReduce jobs, called continuous Jobs, are automatically re-executed to process new incoming data at a minimum cost. We present a simple and clean API which integrates nicely with the standard MapReduce model. Furthermore, we describe cHadoop, an implementation of our approach based on Hadoop which does not require modifications to the source code of the original framework. Thus, cHadoop can quickly be ported to any new version of Hadoop. We evaluate our proposal with two standard MapReduce applications (WordCount and WordCount-N-Count), and one real world application (RDF Query) on real datasets. Our evaluations on clusters ranging from 5 to 40 nodes demonstrate the benefit of our approach in terms of execution time and ease of use.
Fichier principal
Vignette du fichier
main.pdf (387.25 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-00916103 , version 1 (12-12-2013)

Identifiants

  • HAL Id : hal-00916103 , version 1

Citer

Trong-Tuan Vu, Fabrice Huet. A Lightweight Continuous Jobs Mechanism for MapReduce Frameworks. 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2013, Jun 2013, Netherlands. pp.269-276. ⟨hal-00916103⟩
324 Consultations
428 Téléchargements

Partager

Gmail Facebook X LinkedIn More