Skip to Main content Skip to Navigation
Conference papers

COMPUTING DATA QUALITY INDICATORS ON BIG DATA STREAMS USING A CEP

Abstract : Big Data is often referred to as the 3Vs: Volume, Velocity and Variety. A 4th V (validity) was introduced to address the quality dimension. Poor data quality can be costly, lead to breaks in processes and invalidate the company's efforts on regulatory compliance. In order to process data streams in real time, a new technology called CEP (complex event processing) was developed. In France, the current deployment of smart meters will generate massive electricity consumption data. In this work, we developed a diagnostic approach to compute generic quality indicators of smart meter data streams on the fly. This solution is based on Tibco StreamBase CEP. Visu-alization tools were also developed in order to give a better understanding of the interrelation between quality issues and geographical/temporal dimensions. According to the application purpose, two visualization methods can be loaded: (1) StreamBase LiveView is used to visualize quality indicators in real time; and (2) a Web application provides a posteri-ori and geographical analysis of the quality indicators which are plotted on a map within a color scale (lighter colors indicate good quality and darker colors indicate poor quality). In future works, new quality indicators could be added to the solution which can be applied in an operational context in order to monitor data quality from smart meters.
Complete list of metadatas

Cited literature [8 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01367862
Contributor : Wenlu Yang <>
Submitted on : Monday, September 19, 2016 - 1:57:39 PM
Last modification on : Thursday, March 21, 2019 - 1:04:20 PM

File

final version.pdf
Files produced by the author(s)

Identifiers

Citation

Wenlu Yang, Alzennyr Gomes da Silva, Marie-Luce Picard. COMPUTING DATA QUALITY INDICATORS ON BIG DATA STREAMS USING A CEP. Computational Intelligence for Multimedia Understanding (IWCIM), 2015 International Workshop on, Oct 2015, Prague, Czech Republic. ⟨10.1109/IWCIM.2015.7347061⟩. ⟨hal-01367862⟩

Share

Metrics

Record views

250

Files downloads

341