Skip to Main content Skip to Navigation
Journal articles

Critical Assessment of Metagenome Interpretation – a benchmark of computational metagenomics software

Alexander Sczyrba 1, 2, * Peter Hofmann 3, 4, 5 Peter Belmann 1, 4, 5 David Koslicki 6 Stefan Janssen 4, 5, 7, 8 Johannes Dröge 3, 4, 5 Ivan Gregor 3, 9 Stephan Majda 3, 10 Jessika Fiedler 3, 4, 5 Eik Dahms 3, 4, 5 Andreas Bremges 1, 2, 11, 4, 5 Adrian Fritz 4, 5 Ruben Garrido-Oter 3, 12, 13 Tue Sparholt Jørgensen 14, 15, 16 Nicole Shapiro 17 Philip D Blood 18 Alexey Gurevich 19 Yang Bai 12, 20 Dmitrij Turaev 21 Matthew Z Demaere 22 Rayan Chikhi 23, 24 Niranjan Nagarajan 25 Christopher Quince 26 Lars Hestbjerg Hansen 14 Søren J Sørensen 15 Burton K H Chia 25 Bertrand Denis 25 Jeff L Froula 17 Zhong Wang 17 Robert Egan 17 Dongwan Don Kang 17 Jeffrey J Cook 27 Charles Deltel 28 Michael Beckstette 29 Claire Lemaitre 28 Peter Peterlongo 28 Guillaume Rizk 28 Dominique Lavenier 28 Yu-Wei Wu 30, 31 Steven W Singer 30, 32 Chirag Jain 33 Marc Strous 34 Heiner Klingenberg 35 Peter Meinicke 35 Michael D Barton 17 Thomas Lingner 36 Hsin-Hung Lin 37 Yu-Chieh Liao 37 Genivaldo Gueiros Z Silva 38 Daniel A Cuevas 38 Robert A Edwards 38 Surya Saha 39 Vitor C Piro 40, 41 Bernhard Y Renard 40 Mihai Pop 42 Hans-Peter Klenk 43 Markus Göker 44 Nikos C Kyrpides 17, 45 Tanja Woyke 17 Julia A Vorholt 46, 47 Paul Schulze-Lefert 12, 13 Edward M Rubin 17 Aaron E Darling 22 Thomas Rattei 21 Alice C Mchardy 3, 4, 5, 13
* Corresponding author
15 Section of Microbiology [Copenhagen]
Department of Biology [Copenhagen]
23 BONSAI - Bioinformatics and Sequence Analysis
CNRS - Centre National de la Recherche Scientifique, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189, Inria Lille - Nord Europe, Université de Lille, Sciences et Technologies
28 GenScale - Scalable, Optimized and Parallel Algorithms for Genomics
Inria Rennes – Bretagne Atlantique , IRISA-D7 - GESTION DES DONNÉES ET DE LA CONNAISSANCE
Abstract : In metagenome analysis, computational methods for assembly, taxonomic profilingand binning are key components facilitating downstream biological datainterpretation. However, a lack of consensus about benchmarking datasets andevaluation metrics complicates proper performance assessment. The CriticalAssessment of Metagenome Interpretation (CAMI) challenge has engaged the globaldeveloper community to benchmark their programs on datasets of unprecedentedcomplexity and realism. Benchmark metagenomes were generated from newlysequenced ~700 microorganisms and ~600 novel viruses and plasmids, includinggenomes with varying degrees of relatedness to each other and to publicly availableones and representing common experimental setups. Across all datasets, assemblyand genome binning programs performed well for species represented by individualgenomes, while performance was substantially affected by the presence of relatedstrains. Taxonomic profiling and binning programs were proficient at high taxonomicranks, with a notable performance decrease below the family level. Parametersettings substantially impacted performances, underscoring the importance ofprogram reproducibility. While highlighting current challenges in computationalmetagenomics, the CAMI results provide a roadmap for software selection to answerspecific research questions.
Document type :
Journal articles
Complete list of metadata

Cited literature [49 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01633525
Contributor : Dominique Lavenier <>
Submitted on : Wednesday, November 15, 2017 - 9:16:22 AM
Last modification on : Thursday, March 18, 2021 - 10:24:01 AM
Long-term archiving on: : Friday, February 16, 2018 - 12:55:49 PM

File

099127.full.pdf
Files produced by the author(s)

Identifiers

Citation

Alexander Sczyrba, Peter Hofmann, Peter Belmann, David Koslicki, Stefan Janssen, et al.. Critical Assessment of Metagenome Interpretation – a benchmark of computational metagenomics software. Nature Methods, Nature Publishing Group, 2017, 14 (11), pp.1063 - 1071. ⟨10.1038/nmeth.4458⟩. ⟨hal-01633525⟩

Share

Metrics

Record views

1856

Files downloads

734