Reproducibility in Biomedical Natural Language Processing

Kevin Cohen (1, 2) , Aurélie Névéol (1) , Jingbo Xia (3) , Negacy Hailu , Lawrence Hunter (2) , Pierre Zweigenbaum (1)

1 ILES - Information, Langue Ecrite et Signée
2 Computational Bioscience Program. University of Colorado Denver
3 HZAU - Huazhong Agricultural University [Wuhan]

To assess the state of reproducibility in biomedical Natural Language Processing, we analyze the proceedings of the 2016 BioNLP workshop to check for code and data availability. We find that only 21% of papers report experiments where both data and code are available, with moderate to substantial inter-annotator agreement. This suggests that reproducibility in biomedical NLP would benefit from increased attention and understanding.

Domaines: Informatique [cs]; Informatique et langage [cs.CL]

Kevin Cohen, Aurélie Névéol, Jingbo Xia, Negacy Hailu, Lawrence Hunter, et al.. Reproducibility in Biomedical Natural Language Processing. AMIA Annual Symposium, Nov 2017, Washington, DC, United States. ⟨hal-01847326⟩