ATLAS EventIndex general dataflow and monitoring infrastructure

Abstract : The ATLAS EventIndex has been running in production since mid-2015, reliably collecting information worldwide about all produced events and storing them in a central Hadoop infrastructure at CERN. A subset of this information is copied to an Oracle relational database for fast dataset discovery, event-picking, crosschecks with other ATLAS systems and checks for event duplication. The system design and its optimization is serving event picking from requests of a few events up to scales of tens of thousand of events, and in addition, data consistency checks are performed for large production campaigns. Detecting duplicate events with a scope of physics collections has recently arisen as an important use case. This paper describes the general architecture of the project and the data flow and operation issues, which are addressed by recent developments to improve the throughput of the overall system. In this direction, the data collection system is reducing the usage of the messaging infrastructure to overcome the performance shortcomings detected during production peaks; an object storage approach is instead used to convey the event index information, and messages to signal their location and status. Recent changes in the Producer/Consumer architecture are also presented in detail, as well as the monitoring infrastructure.
Type de document :
Communication dans un congrès
22nd International Conference on Computing in High Energy and Nuclear Physics, Oct 2016, San Francisco, United States. J.Phys.Conf.Ser., 898 (6), pp.062010, 2017, 〈10.1088/1742-6596/898/6/062010〉
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01669627
Contributeur : Inspire Hep <>
Soumis le : mercredi 20 décembre 2017 - 23:47:28
Dernière modification le : mardi 25 septembre 2018 - 22:08:15

Lien texte intégral

Identifiants

Collections

Citation

Á Fernández Casaní, D. Barberis, A. Favareto, C. García Montoro, S. González de La Hoz, et al.. ATLAS EventIndex general dataflow and monitoring infrastructure. 22nd International Conference on Computing in High Energy and Nuclear Physics, Oct 2016, San Francisco, United States. J.Phys.Conf.Ser., 898 (6), pp.062010, 2017, 〈10.1088/1742-6596/898/6/062010〉. 〈hal-01669627〉

Partager

Métriques

Consultations de la notice

28