Skip to Main content Skip to Navigation
Journal articles

Raw data queries during data-intensive parallel workflow execution

Abstract : Computer simulations consume and produce huge amounts of raw data files presented in different formats, e.g., HDF5 in computational fluid dynamics simulations. Users often need to analyze domain-specific data based on related data elements from multiple files during the execution of computer simulations. In a raw data analysis, one should identify regions of interest in the data space and retrieve the content of specific related raw data files. Existing solutions, such as FastBit and RAW, are limited to a single raw data file analysis and can only be used after the execution of computer simulations. Scientific Workflow Management Systems (SWMS) can manage the dataflow of computer simulations and register related raw data files at a provenance database. This paper aims to combine the advantages of a dataflow-aware SWMS and the raw data file analysis techniques to allow for queries on raw data file elements that are related, but reside in separate files. We propose a component-based architecture, named as ARMFUL (Analysis of Raw data from Multiple Files) with raw data extraction and indexing techniques, which allows for a direct access to specific elements or regions of raw data space. ARMFUL innovates by using a SWMS provenance database to add a dataflow access path to raw data files. ARMFUL facilitates the invocation of ad-hoc programs and third party tools (e.g., FastBit tool) for raw data analyses. In our experiments, a real parallel computational fluid dynamics is executed, exploring different alternatives of raw data extraction, indexing and analysis.
Document type :
Journal articles
Complete list of metadata

Cited literature [40 references]  Display  Hide  Download
Contributor : Patrick Valduriez Connect in order to contact the contributor
Submitted on : Tuesday, January 24, 2017 - 4:24:04 PM
Last modification on : Thursday, November 25, 2021 - 11:28:10 AM
Long-term archiving on: : Tuesday, April 25, 2017 - 5:46:16 PM


paper-FGCS-2016-author version...
Files produced by the author(s)




Vítor Silva, José Leite, José Camata, Daniel de Oliveira, Alvaro Coutinho, et al.. Raw data queries during data-intensive parallel workflow execution. Future Generation Computer Systems, Elsevier, 2017, 75, pp.402-422. ⟨10.1016/j.future.2017.01.016⟩. ⟨lirmm-01445219⟩



Record views


Files downloads