Handling Pregel's limits in big graphs processing in the presence of high degree vertices

Abstract : Even if specialized distributed graph processing systems such as Pregel scale better than pure MapReduce programs, in graph processing, by reducing disk I/O for iterative algorithms while offering an easy programming model using "think like vertex" paradigm, large scale graphs processing is still challenging in the presence of high degree vertices: Communication and load imbalance among processing nodes can have disastrous effects on performance. In this article, we introduce a scalable MapReduce graph partitioning approach for high degree vertices using master/slave partitioning. This partitioning makes Pregel-like systems, in graph processing, scalable and insensitive to the effects of high degree vertices while guaranteeing perfect balancing properties of communication and computation during all the stages of big graphs processing. A cost model and performance analysis are given to show the effectiveness and the scalability of our graph partitioning approach in large scale systems.
Document type :
Book sections
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01802435
Contributor : Mostafa Bamha <>
Submitted on : Tuesday, May 29, 2018 - 11:49:10 AM
Last modification on : Thursday, January 17, 2019 - 3:10:02 PM

Identifiers

Collections

Citation

Mohamad Al Hajj Hassan, Mostafa Bamha. Handling Pregel's limits in big graphs processing in the presence of high degree vertices. Applications of Big Data Analytics: Trends, Issues, and Challenges, 2018, 9783319764719. ⟨10.1007/978-3-319-76472-6⟩. ⟨hal-01802435⟩

Share

Metrics

Record views

48