Code review poses extra challenges
Résumé
As computational scientists who are unsatisfied with the current level of quality control concerning computational work, we applaud Nature's recently announced initiative to encourage the submission of code accompanying an article, and have it checked by reviewers. We hope that more journals will follow Nature's lead. As editors-in-chief of ReScience (http://rescience.github.io/), a journal dedicated to computational replications, we regularly supervise code reviews. Our experience is that they require significant changes to the traditional reviewing process. An effective code review is possible only if reviewers can both run and inspect the software. We therefore consider the submission of source code mandatory. Moreover, inspection of the source code requires that reviewers have experience with software development in addition to scientific domain expertise. ReScience selects reviewers both according to their scientific discipline and according to their familiarity with programming languages. The first step of a code review consists of reproducing the figures and/or tables from the submitted article using the suppled software. This requires that the authors provide all input files and datasets, and clear instructions. Nevertheless, in our experience, technical problems in installing and running scientific software are inevitable. They are resolved most efficiently if authors and reviewers can communicate rapidly and interactively, without the intervention of the editorial staff. ReScience uses an open reviewing approach conducted on the GitHub platform, which has proven to be very effective. A more detailed account of our experience with publishing computational replications in ReScience can be found in PeerJ Computer Science. 2017 Dec 18;3:e142 (doi:10.7717/peerjcs.142). The currently ongoing reviews at ReScience can be viewed at https://github.com/ReScience/ReScience-submission/pulls.