A Flexible and Distributed Runtime System for High-Throughput Constrained Data Streams Generation

Abstract : Major research topics on parallel and distributed frameworks focus on reliability, performance and programmability of large scale systems for, e.g., HPC or Big Data. The solutions proposed are often directly impacted by the large scale nature of the problems. Differently, high-throughput data stream generation is an important challenge for many scientific and industrial applications which is typically well suited for small to medium scale systems, and which has to respect specific constraints about, e.g., speed, throughput or output location. In this paper we present a framework dedicated to this class of problems. We propose a performance-oriented runtime system architecture able to generate constrained data streams issued from jobs dynamically submitted by the user. Our architecture is designed to scale from a single host to a medium-sized cluster with large topology flexibility to achieve high throughput capabilities while being widely adaptive to a variety of problems. We provide experimental evidence of the ability of our framework to meet high-throughput constraints on an industrial use-case, i.e., professional digital printing, that may require tens of Gbit/s sustained output rates. We show in our measurements that our system scales and reaches data rates close to the maximum throughput of our experimental cluster.
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-02381750
Contributor : Vincent Loechner <>
Submitted on : Tuesday, November 26, 2019 - 5:50:02 PM
Last modification on : Wednesday, January 29, 2020 - 11:57:54 AM

Identifiers

Citation

Paul Godard, Vincent Loechner, Cédric Bastoul, Frederic Soulier, Guillaume Muller. A Flexible and Distributed Runtime System for High-Throughput Constrained Data Streams Generation. IPDPSW 2019 - IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum, May 2019, Rio de Janeiro, Brazil. pp.718-728, ⟨10.1109/IPDPSW.2019.00120⟩. ⟨hal-02381750⟩

Share

Metrics

Record views

23