PRESPS: a PREdictive model to determine the number of replicas of the operators in Stream Processing Systems

Daniel Wladdimiro (1) , Luciana Arantes (1) , Nicolas Hidalgo (2) , Pierre Sens (1)

1 DELYS - DistributEd aLgorithms and sYStems
2 UDP - Universidad Diego Portales [Santiago - Chili]

Stream Processing Systems (SPS) are used to process large amounts of data in real time, which are designed as directed acyclic graph (DAG). The vertices correspond to operators and the edges to data stream. Each component is deployed distributed in an infrastructure, which is parallelized. In this paper we propose a predictive model to dynamically and predictively determine the number of replicas required by each SPS operator, based on the input data, per operator event queue and execution time. We have performed preliminary experiments of our solution with another existing solution using a Twitter Stream, deployed on Google Cloud Platform (GCP).

Domaines: Informatique [cs]; Calcul parallèle, distribué et partagé [cs.DC]

Daniel Wladdimiro, Luciana Arantes, Nicolas Hidalgo, Pierre Sens. PRESPS: a PREdictive model to determine the number of replicas of the operators in Stream Processing Systems. Compas 2023 - Conférence francophone d'informatique en Parallélisme, Architecture et Système, Jul 2023, Annecy, France. ⟨hal-04153951⟩