Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students

Anne Grote (1) , Nadine S Schaadt (2) , Germain Forestier (3) , Friedrich Feuerhake (2) , Cédric Wemmert (4)

1 Institute for Pathology, Hannover Medical School
2 MHH - Medizinische Hochschule Hannover = Hannover Medical School
3 IRIMAS - Institut de Recherche en Informatique Mathématiques Automatique Signal
4 LSIIT - Laboratoire des Sciences de l'Image, de l'Informatique et de la Télédétection

Crowdsourcing in pathology has been performed on tasks that are assumed to be manageable by nonexperts. Demand remains high for annotations of more complex elements in digital microscopic images, such as anatomical structures. Therefore, this work investigates conditions to enable crowdsourced annotations of high-level image objects, a complex task considered to require expert knowledge. 76 medical students without specific domain knowledge who voluntarily participated in three experiments solved two relevant annotation tasks on histopathological images: (1) Labeling of images showing tissue regions, and (2) delineation of morphologically defined image objects. We focus on methods to ensure sufficient annotation quality including several tests on the required number of participants and on the correlation of participants' performance between tasks. In a set up simulating annotation of images with limited ground truth, we validated the feasibility of a confidence score using full ground truth. For this, we computed a majority vote using weighting factors based on individual assessment of contributors against scattered gold standard annotated by pathologists. In conclusion, we provide guidance for task design and quality control to enable a crowdsourced approach to obtain accurate annotations required in the era of digital pathology.

Mots clés: Crowdsourcing; human decision making; image classification; image delineation; digital pathology; annotation; confidence score; majority vote

Domaines: Anatomie, Histologie, Anatomopathologie [q-bio.TO]; Intelligence artificielle [cs.AI]; Machine Learning [stat.ML]; Apprentissage [cs.LG]

Anne Grote, Nadine S Schaadt, Germain Forestier, Friedrich Feuerhake, Cédric Wemmert. Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students. IEEE Transactions on Medical Imaging, 2019, 38 (5), pp.1284-1294. ⟨10.1109/TMI.2018.2883237⟩. ⟨hal-02299285⟩