Multi-Layer-Mesh: A Novel Topology and SDN-Based Path Switching for Big Data Cluster Networks

Leandro Batista de Almeida (1) , Damien Magoni (2) , Philip Perry (3) , Eduardo Cunha de Almeida (4) , John Murphy (3) , Anthony Ventresque (3)

1 UTFPR - Universidade Tecnológica Federal do Paraná [Curitiba, Brasil] = Federal Technological University of Paraná [Curitiba, Brazil] = Université technologique fédérale du Paraná [Curitiba, Brésil]
2 LaBRI - Laboratoire Bordelais de Recherche en Informatique
3 UCD - University College Dublin [Dublin]
4 UFPR - Universidade Federal do Paraná [Curitiba, Brasil] = Federal University of Paraná [Curitiba, Brazil] = Université fédérale du Paraná [Curitiba, Brésil]

Big Data technologies and tools have being used for the past decade to solve several scientific and industry problems, with Hadoop/YARN becoming the "de facto" standard for these applications, although other technologies run on top of it. As any other distributed application, those big data technologies rely heavily on the network infrastructure to read and move data from hundreds or thousands of cluster nodes. Although these technologies are based on reliable and efficient distributed algorithms, there are scenarios and conditions that can generate bottlenecks and inefficiencies, i.e., when a high number of concurrent users creates data access contention. In this paper, we propose a novel network topology called Multi-Layer-Mesh and a path switching algorithm based on SDN, that can increase the performance of a big data cluster while reducing the amount of utilized resources (network equipment), in turn reducing the energy and cooling consumption. A thorough simulation-based evaluation of our algorithms shows an average improvement in performance of 31.77% and an average decrease in resource utilization of 36.03% compared to a traditional Spine-Leaf topology, in the selected test scenarios.

Mots clés: big data; Hadoop; network topology; SDN; path switching

Domaines: Réseaux et télécommunications [cs.NI]

Leandro Batista de Almeida, Damien Magoni, Philip Perry, Eduardo Cunha de Almeida, John Murphy, et al.. Multi-Layer-Mesh: A Novel Topology and SDN-Based Path Switching for Big Data Cluster Networks. IEEE International Conference on Communications, May 2019, Shanghai, China. pp.1-7, ⟨10.1109/ICC.2019.8761785⟩. ⟨hal-02493442⟩