Diarchy: An Optimized Management Approach for MapReduce Masters

Abstract : The MapReduce community is progressively replacing the classic Hadoop with Yarn, the second-generation Hadoop (MapReduce 2.0). This transition is being made due to many reasons, but primarily because of some scalability drawbacks of the classic Hadoop. The new framework has appropriately addressed this issue and is being praised for its multi-functionality. In this paper we carry out a probabilistic analysis that emphasizes some reliability concerns of Yarn at the job master level. This is a critical point, since the failures of a job master involves the failure of all the workers managed by such a master. In this paper, we propose Diarchy, a novel system for the management of job masters. Its aim is to increase the reliability of Yarn, based on the sharing and backup of responsibilities between two masters working as peers. The evaluation results show that Diarchy outperforms the reliability performance of Yarn in different setups, regardless of cluster size, type of job, or average failure rate and suggest a positive impact of this approach compared to the traditional, single-master Hadoop architecture.
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01249151
Contributor : Gabriel Antoniu <>
Submitted on : Wednesday, December 30, 2015 - 1:00:29 PM
Last modification on : Wednesday, December 18, 2019 - 4:57:03 PM

Links full text

Identifiers

Citation

Bunjamin Memishi, María S. Pérez-Hernández, Gabriel Antoniu. Diarchy: An Optimized Management Approach for MapReduce Masters. ICCS 2015: Proceedings of the International Conference on Computational Science, Computational Science at the Gates of Nature, Jun 2015, Reykjavík, Iceland. pp.9--18, ⟨10.1016/j.procs.2015.05.179⟩. ⟨hal-01249151⟩

Share

Metrics

Record views

314