COMPUTING DATA QUALITY INDICATORS ON BIG DATA STREAMS USING A CEP

Wenlu Yang (1) , Alzennyr Gomes da Silva (2) , Marie-Luce Picard (2)

1 LFI - Learning, Fuzzy and Intelligent systems
2 EDF - EDF

Big Data is often referred to as the 3Vs: Volume, Velocity and Variety. A 4th V (validity) was introduced to address the quality dimension. Poor data quality can be costly, lead to breaks in processes and invalidate the company's efforts on regulatory compliance. In order to process data streams in real time, a new technology called CEP (complex event processing) was developed. In France, the current deployment of smart meters will generate massive electricity consumption data. In this work, we developed a diagnostic approach to compute generic quality indicators of smart meter data streams on the fly. This solution is based on Tibco StreamBase CEP. Visu-alization tools were also developed in order to give a better understanding of the interrelation between quality issues and geographical/temporal dimensions. According to the application purpose, two visualization methods can be loaded: (1) StreamBase LiveView is used to visualize quality indicators in real time; and (2) a Web application provides a posteri-ori and geographical analysis of the quality indicators which are plotted on a map within a color scale (lighter colors indicate good quality and darker colors indicate poor quality). In future works, new quality indicators could be added to the solution which can be applied in an operational context in order to monitor data quality from smart meters.

Mots clés: Data quality; Big Data; data stream; CEP; smart grids

Domaines: Informatique [cs]; Ingénierie assistée par ordinateur

Wenlu Yang, Alzennyr Gomes da Silva, Marie-Luce Picard. COMPUTING DATA QUALITY INDICATORS ON BIG DATA STREAMS USING A CEP. Computational Intelligence for Multimedia Understanding (IWCIM), 2015 International Workshop on, Oct 2015, Prague, Czech Republic. ⟨10.1109/IWCIM.2015.7347061⟩. ⟨hal-01367862⟩