Skip to Main content Skip to Navigation
Journal articles

Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge.

Esther E Bron 1 Marion Smits 2, 3 Wiesje M van der Flier 4 Hugo Vrenken 4 Frederik Barkhof 4 Philip Scheltens 4 Janne M Papma 2 Rebecca M E Steketee 2 Carolina Méndez Orellana 2 Rozanna Meijboom 2 Madalena Pinto 5 Joana R Meireles 6 Carolina Garrett 7 António J Bastos-Leite 7 Ahmed Abdulkadir 8, 9 Olaf Ronneberger 10, 9 Nicola Amoroso 11, 12 Roberto Bellotti 11, 12 David Cárdenas-Peña 13 Andrés M Álvarez-Meza 13 Chester V Dolph 14 Khan M Iftekharuddin 14 Simon F Eskildsen 15 Pierrick Coupé 16 Vladimir S Fonov 17 Katja Franke 18 Christian Gaser 18 Christian Ledig 19 Ricardo Guerrero 19 Tong Tong 19 Katherine R Gray 19 Elaheh Moradi 20 Jussi Tohka 20 Alexandre Routier 21 Stanley Durrleman 21 Alessia Sarica 22 Giuseppe Di Fatta 23 Francesco Sensi 24 Andrea Chincarini 24 Garry M Smith 25, 23 Zhivko V Stoyanov 23, 25 Lauge Sørensen 26 Mads Nielsen 26 Sabina Tangaro 11 Paolo Inglese 11 Christian Wachinger 27 Martin Reuter 28, 29 John C van Swieten 2 Wiro J Niessen 30, 2 Stefan Klein 1
1 LERMA - Laboratoire d'Etude du Rayonnement et de la Matière en Astrophysique
2 Erasmus MC - Erasmus University Medical Center [Rotterdam]
3 WUR - Wageningen University and Research Centre [Wageningen]
4 VU University Medical Center [Amsterdam]
5 IFSTTAR/COSYS/LEPSIS - Laboratoire Exploitation, Perception, Simulateurs et Simulations
6 Hospital de São João [Porto]
7 FMUP - Faculdade de Medicina da Universidade do Porto [Porto]
8 University of Freiburg [Freiburg]
9 Department of Computer Science [Freiburg]
10 BIOSS - Centre for Biological Signaling Studies [Freiburg]
11 INFN, sezione di Bari - Istituto Nazionale di Fisica Nucleare, sezione di Bari
12 Università degli studi di Bari
13 UNAL - Universidad Nacional de Colombia [Bogotà]
14 ODU - Old Dominion University [Norfolk]
15 Aarhus University [Aarhus]
16 LaBRI - Laboratoire Bordelais de Recherche en Informatique
17 MNI - McConnell Brain Imaging Centre
18 Jena University Hospital [Jena]
19 BioMedIA - Biomedical Image Analysis Group [London]
20 Department of Signal Processing [Tampere]
21 ARAMIS - Algorithms, models and methods for images and signals of the human brain
Inria Paris-Rocquencourt, UPMC - Université Pierre et Marie Curie - Paris 6, ICM - Institut du Cerveau et de la Moëlle Epinière = Brain and Spine Institute
22 UMG - Università degli Studi "Magna Graecia" di Catanzaro [Catanzaro, Italie]
23 School of Systems Engineering [Reading]
24 Universita degli studi di Genova
25 CINN - Centre for Integrative Neuroscience and Neurodynamics [Reading]
26 DIKU - Department of Computer Science [Copenhagen]
27 CAMPAR - Computer Aided Medical Procedures & Augmented Reality
28 Massachusetts General Hospital [Boston]
29 CSAIL - Computer Science and Artificial Intelligence Laboratory [Cambridge]
30 TU Delft - Delft University of Technology
Abstract : Algorithms for computer-aided diagnosis of dementia based on structural MRI have demonstrated high performance in the literature, but are difficult to compare as different data sets and methodology were used for evaluation. In addition, it is unclear how the algorithms would perform on previously unseen data, and thus, how they would perform in clinical practice when there is no real opportunity to adapt the algorithm to the data at hand. To address these comparability, generalizability and clinical applicability issues, we organized a grand challenge that aimed to objectively compare algorithms based on a clinically representative multi-center data set. Using clinical practice as the starting point, the goal was to reproduce the clinical diagnosis. Therefore, we evaluated algorithms for multi-class classification of three diagnostic groups: patients with probable Alzheimer's disease, patients with mild cognitive impairment and healthy controls. The diagnosis based on clinical criteria was used as reference standard, as it was the best available reference despite its known limitations. For evaluation, a previously unseen test set was used consisting of 354 T1-weighted MRI scans with the diagnoses blinded. Fifteen research teams participated with a total of 29 algorithms. The algorithms were trained on a small training set (n=30) and optionally on data from other sources (e.g., the Alzheimer's Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle flagship study of aging). The best performing algorithm yielded an accuracy of 63.0% and an area under the receiver-operating-characteristic curve (AUC) of 78.8%. In general, the best performances were achieved using feature extraction based on voxel-based morphometry or a combination of features that included volume, cortical thickness, shape and intensity. The challenge is open for new submissions via the web-based framework: http://caddementia.grand-challenge.org.
Keywords : Mild cognitive impairment Computer-aided diagnosis Classification Alzheimer's disease Challenge Structural MRI
Complete list of metadatas
https://hal.archives-ouvertes.fr/hal-01220123
Contributor : Pierrick Coupé <>
Submitted on : Saturday, October 24, 2015 - 4:06:58 PM
Last modification on : Thursday, April 2, 2020 - 1:28:08 PM

Links full text

openaccess

Identifiers

Collections

CNRS | INRIA | UNIV-CERGY | OBSPM | LERMA | UPMC | ENS-PARIS | ICM | INSU | LABRI | IFSTTAR | ARAMISLAB | ARAMILAB | PSL | SORBONNE-UNIVERSITE | SU-MEDECINE | UNIV-BORDEAUX | SU-SCIENCES

Citation

Esther E Bron, Marion Smits, Wiesje M van der Flier, Hugo Vrenken, Frederik Barkhof, et al.. Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge.. NeuroImage, Elsevier, 2015, 111, pp.562-79. ⟨10.1016/j.neuroimage.2015.01.048⟩. ⟨hal-01220123⟩

Export

BibTeX TEI DC DCterms EndNote

Share

Metrics

Record views

924

Contact

Informations

CCSD