A Performance Evaluation of Apache Kafka in Support of Big Data Streaming Applications

Evaluation de performances d'Apache Kafka en support aux applications Big Data de traitement de flux

Paul Le Noac'H (1) , Alexandru Costan (1) , Luc Bougé (1)

1 KerData - Scalable Storage for Clouds and Beyond

Producer performances when modifying batch size for several number of nodes and a message size of 50B 7. Take-aways • The variation of the batch size shows that there is a range of batches with a better performance. • When varying the number of nodes in some scenarios: a sudden performance drop (probably due to the internal Kafka synchronizations as well as the underlying network). • Future work : evaluating reference processing frameworks (Apache Spark and Flink) Parameters : • Message size • Batch size • Acquirement strategy • Network and disk I/O threads • Message replication • Hardware 2. Contribution • Isolate the performance of each Kafka component

Mots clés: Stream computing; Apache Kafka; Big Data

Domaines: Calcul parallèle, distribué et partagé [cs.DC]

Paul Le Noac'H, Alexandru Costan, Luc Bougé. A Performance Evaluation of Apache Kafka in Support of Big Data Streaming Applications. IEEE Big Data 2017, Dec 2017, Boston, United States. 2017. ⟨hal-01647229⟩