Dynamic Scheduling of MapReduce Shuffle under Bandwidth Constraints

Sylvain Gault 1 Christian Pérez 1
1 AVALON - Algorithms and Software Architectures for Distributed and HPC Platforms
Inria Grenoble - Rhône-Alpes, LIP - Laboratoire de l'Informatique du Parallélisme
Abstract : Whether it is for e-science or business, the amount of data produced every year is growing at a high rate. Managing and processing those data raises new challenges. MapReduce is one answer to the need for scalable tools able to handle the amount of data. It imposes a general structure of computation and let the implementation perform its optimizations. During the computation, there is a phase called shuffle where every node sends a possibly large amount of data to every other node. This paper proposes and evaluates two algorithms to improve data transfers during the shuffle phase under bandwidth constraints.
Complete list of metadatas

Contributor : Christian Perez <>
Submitted on : Monday, September 7, 2015 - 10:16:10 AM
Last modification on : Friday, June 7, 2019 - 5:19:41 PM




Sylvain Gault, Christian Pérez. Dynamic Scheduling of MapReduce Shuffle under Bandwidth Constraints. Euro-Par 2014: Parallel Processing Workshops, Aug 2014, Porto, Portugal. pp.117-128, ⟨10.1007/978-3-319-14325-5_11⟩. ⟨hal-01194493⟩



Record views