A Hadoop MapReduce Performance Prediction Method

Ge Song 1 Zide Meng 2 Fabrice Huet 1 Frederic Magoules 3 Lei Yu 4 Xuelian Lin 5
1 OASIS - Active objects, semantics, Internet and security
CRISAM - Inria Sophia Antipolis - Méditerranée , COMRED - COMmunications, Réseaux, systèmes Embarqués et Distribués
2 WIMMICS - Web-Instrumented Man-Machine Interactions, Communities and Semantics
CRISAM - Inria Sophia Antipolis - Méditerranée , SPARKS - Scalable and Pervasive softwARe and Knowledge Systems
Abstract : More and more Internet companies rely on large scale data analysis as part of their core services for tasks such as log analysis, feature extraction or data filtering. Map-Reduce, through its Hadoop implementation, has proved to be an efficient model for dealing with such data. One important challenge when performing such analysis is to predict the performance of individual jobs. In this paper, we propose a simple framework to predict the performance of Hadoop jobs. It is composed of a dynamic light-weight Hadoop job analyzer, and a prediction module using locally weighted regression methods. Our framework makes some theoretical cost models more practical, and also well fits for the diversification of the jobs and clusters. It can also help those users who want to predict the cost when applying for an on- demand cloud service. At the end, we do some experiments to verify our framework.
Type de document :
Communication dans un congrès
HPCC 2013, Nov 2013, Zhangjiajie, China. pp.820-825, 2013, <10.1109/HPCC.and.EUC.2013.118>
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-00918329
Contributeur : Ge Song <>
Soumis le : mardi 21 janvier 2014 - 15:15:13
Dernière modification le : mardi 19 janvier 2016 - 09:22:51
Document(s) archivé(s) le : mardi 22 avril 2014 - 11:37:44

Fichier

HadoopPrediction.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Collections

Citation

Ge Song, Zide Meng, Fabrice Huet, Frederic Magoules, Lei Yu, et al.. A Hadoop MapReduce Performance Prediction Method. HPCC 2013, Nov 2013, Zhangjiajie, China. pp.820-825, 2013, <10.1109/HPCC.and.EUC.2013.118>. <hal-00918329>

Partager

Métriques

Consultations de
la notice

499

Téléchargements du document

1192