Critical Assessment of Metagenome Interpretation – a benchmark of computational metagenomics software

Alexander Sczyrba 1, 2 Peter Hofmann 3, 4, 5 Peter Belmann 1, 4, 5 David Koslicki 6 Stefan Janssen 4, 5, 7, 8 Johannes Dröge 3, 4, 5 Ivan Gregor 3, 9 Stephan Majda 3, 10 Jessika Fiedler 3, 4, 5 Eik Dahms 3, 4, 5 Andreas Bremges 1, 2, 11, 4, 5 Adrian Fritz 4, 5 Ruben Garrido-Oter 3, 12, 13 Tue Sparholt Jørgensen 14, 15, 16 Nicole Shapiro 17 Philip D Blood 18 Alexey Gurevich 19 Yang Bai 12, 20 Dmitrij Turaev 21 Matthew Z Demaere 22 Rayan Chikhi 23, 24 Niranjan Nagarajan 25 Christopher Quince 26 Lars Hestbjerg Hansen 14 Søren J Sørensen 15 Burton K H Chia 25 Bertrand Denis 25 Jeff L Froula 17 Zhong Wang 17 Robert Egan 17 Dongwan Don Kang 17 Jeffrey J Cook 27 Charles Deltel 28 Michael Beckstette 29 Claire Lemaitre 28 Peter Peterlongo 28 Guillaume Rizk 28 Dominique Lavenier 28 Yu-Wei Wu 30, 31 Steven W Singer 30, 32 Chirag Jain 33 Marc Strous 34 Heiner Klingenberg 35 Peter Meinicke 35 Michael D Barton 17 Thomas Lingner 36 Hsin-Hung Lin 37 Yu-Chieh Liao 37 Genivaldo Gueiros Z Silva 38 Daniel A Cuevas 38 Robert A Edwards 38 Surya Saha 39 Vitor C Piro 40, 41 Bernhard Y Renard 40 Mihai Pop 42 Hans-Peter Klenk 43 Markus Göker 44 Nikos C Kyrpides 17, 45 Tanja Woyke 17 Julia A Vorholt 46, 47 Paul Schulze-Lefert 12, 13 Edward M Rubin 17 Aaron E Darling 22 Thomas Rattei 21 Alice C Mchardy 3, 4, 5, 13
15 Section of Microbiology [Copenhagen]
Department of Biology [Copenhagen]
23 BONSAI - Bioinformatics and Sequence Analysis
Université de Lille, Sciences et Technologies, Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189, CNRS - Centre National de la Recherche Scientifique
28 GenScale - Scalable, Optimized and Parallel Algorithms for Genomics
Inria Rennes – Bretagne Atlantique , IRISA_D7 - GESTION DES DONNÉES ET DE LA CONNAISSANCE
Abstract : In metagenome analysis, computational methods for assembly, taxonomic profiling and binning are key components facilitating downstream biological data interpretation. However, a lack of consensus about benchmarking datasets and evaluation metrics complicates proper performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on datasets of unprecedented complexity and realism. Benchmark metagenomes were generated from newly sequenced ~700 microorganisms and ~600 novel viruses and plasmids, including genomes with varying degrees of relatedness to each other and to publicly available ones and representing common experimental setups. Across all datasets, assembly and genome binning programs performed well for species represented by individual genomes, while performance was substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below the family level. Parameter settings substantially impacted performances, underscoring the importance of program reproducibility. While highlighting current challenges in computational metagenomics, the CAMI results provide a roadmap for software selection to answer specific research questions.
Document type :
Journal articles
Complete list of metadatas

Cited literature [49 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01633525
Contributor : Dominique Lavenier <>
Submitted on : Wednesday, November 15, 2017 - 9:16:22 AM
Last modification on : Wednesday, August 7, 2019 - 12:18:05 PM
Long-term archiving on : Friday, February 16, 2018 - 12:55:49 PM

File

099127.full.pdf
Files produced by the author(s)

Identifiers

Citation

Alexander Sczyrba, Peter Hofmann, Peter Belmann, David Koslicki, Stefan Janssen, et al.. Critical Assessment of Metagenome Interpretation – a benchmark of computational metagenomics software. Nature Methods, Nature Publishing Group, 2017, 14 (11), pp.1063 - 1071. ⟨10.1038/nmeth.4458⟩. ⟨hal-01633525⟩

Share

Metrics

Record views

842

Files downloads

449