Parallel Processing of Public Open Data with the MapReduce Paradigm : A Case Study
Résumé
Nowadays, many governments and states are involved in an opening strategy of their public data. However, the volume of these opened data is constantly increasing, and will reach in the near future limitations of current treatment and storage capacity. On the other hand, the MapReduce paradigm is one of the most used parallel programming models. With a master-slave architecture, it allows parallel processing of very large data sets. In this paper, we propose a parallel approach based on Mapreduce to process public open data. Applied, as a case study, to the official data sets from the French Ministry of Communication. We implement a parallel algorithm as a solution to define a ranking of national museums and galleries according to the accessibility degrees for people with disabilities. We studied the feasibility of our approach in two main parts: The performance in terms of execution time, and, the visualization of the obtained results in order to integrate them into solutions such as geographic BI. This work can be applied to other cases with very large data sets.