Governing Energy Consumption in Hadoop through CPU Frequency Scaling: an Analysis

Abstract : With increasingly inexpensive storage and growing processing power, the cloud has rapidly become the environment of choice to store and analyze data for a variety of applications. Most large-scale data computations in the cloud heavily rely on the MapReduce paradigm and on its Hadoop implementation. Nevertheless, this exponential growth in popularity has significantly impacted power consumption in cloud infrastructures. In this paper, we focus on MapReduce processing and we investigate the impact of dynamically scaling the frequency of compute nodes on the performance and energy consumption of a Hadoop cluster. To this end, a series of experiments are conducted to explore the implications of Dynamic Voltage and Frequency Scaling (DVFS) settings on power consumption in Hadoop clusters. By enabling various existing DVFS governors (i.e., performance, powersave, ondemand, conservative and userspace) in a Hadoop cluster, we observe significant variation in performance and power consumption across different applications: the different DVFS settings are only sub-optimal for several representative MapReduce applications. Furthermore, our results reveal that the current CPU governors do not exactly reflect their design goal and may even become ineffective to manage the power consumption in Hadoop clusters. This study aims at providing a clearer understanding of the interplay between performance and power management in Hadoop clusters and therefore offers useful insight into designing power-aware techniques for Hadoop systems.
Complete list of metadatas

Cited literature [46 references]  Display  Hide  Download

https://hal.inria.fr/hal-01166252
Contributor : Shadi Ibrahim <>
Submitted on : Friday, June 26, 2015 - 11:07:45 AM
Last modification on : Friday, October 25, 2019 - 1:21:05 AM
Long-term archiving on: Wednesday, September 16, 2015 - 1:02:07 AM

File

fgcs-ibrahim-HAL.pdf
Files produced by the author(s)

Identifiers

Citation

Shadi Ibrahim, Tien-Dat Phan, Alexandra Carpen-Amarie, Houssem-Eddine Chihoub, Diana Moise, et al.. Governing Energy Consumption in Hadoop through CPU Frequency Scaling: an Analysis. Future Generation Computer Systems, Elsevier, 2016, pp.14. ⟨10.1016/j.future.2015.01.005⟩. ⟨hal-01166252⟩

Share

Metrics

Record views

1010

Files downloads

1130