Why rankings of biomedical image analysis competitions should be interpreted with care

Lena Maier-Hein (1) , Matthias Eisenmann (1) , Annika Reinke (1) , Sinan Onogur (1) , Marko Stankovic (1) , Patrick Scholz (1) , Tal Arbel (2) , Hrvoje Bogunovic (3) , Andrew P. Bradley (4) , Aaron Carass (5) , Carolin Feldmann (1) , Alejandro F. Frangi (6) , Peter M. Full (1) , Bram Van Ginneken (7) , Allan Hanbury (8) , Katrin Honauer (9) , Michal Kozubek (10) , Bennett A. Landman (11) , Keno März (1) , Oskar Maier (12) , Klaus Maier-Hein (1) , Bjoern H. Menze (13) , Henning Müller (14) , Peter F. Neher (1) , Wiro Niessen (15) , Nasir Rajpoot (16) , Gregory C. Sharp (17) , Korsuk Sirinukunwattana (18) , Stefanie Speidel (19) , Christian Stock (1) , Danail Stoyanov (20) , Abdel Aziz Taha (21) , Fons van Der Sommen (22) , Ching-Wei Wang (23) , Marc-André Weber (24) , Guoyan Zheng (25) , Pierre Jannin (26) , Annette Kopp-Schneider (1)

1 DKFZ - German Cancer Research Center - Deutsches Krebsforschungszentrum [Heidelberg]
2 McGill University = Université McGill [Montréal, Canada]
3 Medizinische Universität Wien = Medical University of Vienna
4 QUT - Queensland University of Technology [Brisbane]
5 Department of Computer Science [Baltimore]
6 University of Leeds
7 Radboud University Medical Center [Nijmegen]
8 TU Wien - Vienna University of Technology = Technische Universität Wien
9 Universität Heidelberg [Heidelberg] = Heidelberg University
10 MUNI - Masaryk University [Brno]
11 Vanderbilt University [Nashville]
12 Institute of Medical Informatics [Lübeck]
13 TUM - Technische Universität Munchen - Technical University Munich - Université Technique de Munich
14 HES-SO - Haute Ecole Spécialisée de Suisse Occidentale
15 Departments of Radiology, Nuclear Medicine and Medical Informatics
16 University of Warwick [Coventry]
17 Massachusetts General Hospital [Boston]
18 University of Oxford
19 NCT - National Center for Tumor Diseases [Dresden]
20 UCL - University College of London [London]
21 Research Studios Austria
22 TU/e - Eindhoven University of Technology [Eindhoven]
23 National Taiwan University of Science and Technology
24 University Medical Center Rostock
25 ISTB - Institute for Surgical Technology and Biomechanics [Bern]
26 LTSI - Laboratoire Traitement du Signal et de l'Image

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Domaines: Imagerie médicale; Traitement du signal et de l'image [eess.SP]; Traitement des images [eess.IV]; Imagerie

Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, et al.. Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications, 2018, 9 (1), pp.5217. ⟨10.1038/s41467-018-07619-7⟩. ⟨hal-01958848⟩