A Preventive Auto-Parallelization Approach for Elastic Stream Processing

Roland Kotto-Kombi 1 Nicolas Lumineau 1 Philippe Lamarre 1
1 BD - Base de Données
LIRIS - Laboratoire d'InfoRmatique en Image et Systèmes d'information
Abstract : Nowadays, more and more sources (connected devices, social networks, etc.) emit real-time data with fluctuating rates over time. Existing distributed stream processing engines (SPE) have to resolve a difficult problem: deliver results satisfying end-users in terms of quality and latency without over-consuming resources. This paper focuses on parallelization of operators to adapt their throughput to their input rate. We suggest an approach which prevents operator congestion in order to limit degradation of results quality. This approach relies on an automatic and dynamic adaptation of resource consumption for each continuous query. This solution takes advantage of i) a metric estimating the activity level of operators in the near future ii) the AUTOSCALE approach which evaluates the need to modify parallelism degrees at local and global scope iii) an integration into the Apache Storm solution. We show performance tests comparing our approach to the native solution of this SPE.
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01585096
Contributor : Roland Kotto Kombi <>
Submitted on : Monday, September 11, 2017 - 10:52:56 AM
Last modification on : Wednesday, November 20, 2019 - 3:03:00 AM

Identifiers

Citation

Roland Kotto-Kombi, Nicolas Lumineau, Philippe Lamarre. A Preventive Auto-Parallelization Approach for Elastic Stream Processing. 37th (IEEE) International Conference on Distributed Computing Systems, Jun 2017, Atlanta, United States. pp.1532-1542, ⟨10.1109/ICDCS.2017.253⟩. ⟨hal-01585096⟩

Share

Metrics

Record views

217