Bredala: Semantic Data Redistribution for In Situ Applications

Matthieu Dreher (1) , Tom Peterka (1)

1 ANL - Argonne National Laboratory [Lemont]

In situ processing is a promising solution to the problem of imbalance between computational capabilities and I/O bandwidth in current and future supercomputers. Initially designed for staging I/O, in situ middleware now can support a wide range of domains such as visualization, machine learning, filtering, and feature tracking. Doing so requires in situ middleware to manage complex heterogeneous codes using different data structures. Data need to be transformed and reorganized along the data path to fit the analysis needs. However, redistributing complex data structures is difficult. In many cases, arbitrarily splitting the arrays of a data structure destroys the semantic integrity of the data. We present Bredala, a lightweight library to annotate a data model with enough information to preserve the semantic integrity of the data during a redistribution. Bredala allows developers to describe how to split and merge a data model safely, operations usually done by in situ middleware. We evaluate the cost and performance of our library in a molecular dynamics application. We show that our data model can simplify the workflow graph of large-scale applications, improve the reusability of tasks, and offer an efficient alternative to redistribute the data.

Domaines: Calcul parallèle, distribué et partagé [cs.DC]

Matthieu Dreher, Tom Peterka. Bredala: Semantic Data Redistribution for In Situ Applications. IEEE Cluster 2016, IEEE, Sep 2016, Taipei, Taiwan. ⟨hal-01358482⟩