Skip to Main content Skip to Navigation
Conference papers

Experiments in Operations Research are Hardly Reproducible: A Bike-Sharing Case-Study.

Thomas Barzola 1 Van-Dat Cung 1 Nicolas Gast 2 Vincent Jost 1 
1 G-SCOP_ROSP - Recherche Opérationnelle pour les Systèmes de Production
G-SCOP - Laboratoire des sciences pour la conception, l'optimisation et la production
Abstract : Many researchers do believe that providing a detailed description of the experiments and the algorithms used is sufficient to guarantee reproducibility. In this paper, we argue that this is largely false. To assert this, we tried to reproduce the experiments of another work. The authors of this paper were aware of the need for their work to be reproducible: they made their data available in one of the authors' website, and they provide a detailed description of their methodology (only the code is missing). Nevertheless, despite all the care they took, we were not able to reproduce their work and our numerical findings are significantly different from theirs. Without their code, we cannot be sure if there is a bug (in their implementation or in ours) or a difference in the interpretation of the model. This raises a number of ethical questions for the community: what is the validity of science if numerical results cannot be trusted? Instead of developing new methodology, should we not spend more time reimplementing existing methods, making them available to all? This may lead to a less productive, yet more trustworthy and reliable, science.
Complete list of metadata
Contributor : CCSD Connect in order to contact the contributor
Submitted on : Thursday, March 3, 2022 - 10:48:48 AM
Last modification on : Wednesday, July 6, 2022 - 4:16:01 AM
Long-term archiving on: : Saturday, June 4, 2022 - 6:36:21 PM


Files produced by the author(s)


  • HAL Id : hal-03595289, version 1


Thomas Barzola, Van-Dat Cung, Nicolas Gast, Vincent Jost. Experiments in Operations Research are Hardly Reproducible: A Bike-Sharing Case-Study.. 23ème congrès annuel de la Société Française de Recherche Opérationnelle et d'Aide à la Décision, INSA Lyon, Feb 2022, Villeurbanne - Lyon, France. ⟨hal-03595289⟩



Record views


Files downloads