There is a lot of talk about peer review and how it can be made better, but unfortunately, a lot of this happens at a level of abstraction that makes it easy to miss more modest changes that can go a long way.

For example, a common way of proceeding in certain sciences is the pre-publication review, according to which manuscripts are uploaded online for open discussion before official peer review and journal acceptance, giving the community at large an opportunity to review results and methods. The advantage of such a process is that it makes the peer selection process far more transparent, but on the downside does not allow for anonymity for either author or reviewer. The downside might seem like it clearly isn’t worth it, since anonymity is accepted as an obvious virtue. But a real life case-study indicates why it might be worth the price.

A real-life example from last year of how a larger pool of peers might be more effective than two anonymous peer reviews can be found in a recent incident surrounding an arXiv submission. arXiv.org is a site for the submission of preprints of papers in Science and Math. In 2018, two researchers from the prestigious Indian Institute of Science, Dev Kumar Thapa and Anshu Pandey, posted a paper at arXiv, where they claimed to have discovered an instance of superconductivity at room temperature in “a nanostructured material that is composed of silver particles embedded into a gold matrix”. If true, this could have been a game-changer for material science and really, all of society since we could theoretically transfer electricity without any loss.

This preprint caught the eye of a Postdoc at MIT, Brian Skinner, who probed into the data a little more and found some odd features:

Skinner wrote up his observations and posted them on arXiv himself. The story was quickly picked up on various sites, including Nature, Scientific American, and Wired. The authors, for their part, seem to have dug in their heels and have not admitted to any wrong-doing.

Most relevant for the broader point about opening up peer review is that Skinner is not an expert in the field of superconductivity, so he probably wouldn’t have been a potential reviewer for the paper in question at all. And his decision to “zoom in closely” on the data isn’t a standard method for vetting papers, so if the preprint hadn’t been posted somewhere relatively public, this discrepancy would have gone unnoticed, and the paper would have been published. The best case scenario then would be retraction.

Of course, there is the lingering question of whether such a model could be extended outside certain sciences. For example, it has been pointed out that medical journals might resist this because making results public prematurely might impede the ability to get proper press attention after full publication. And there are questions about whether the lack of anonymity at the preprint stage would effectively do away with anonymity since the authors will already be known from the preprint. So this is far from a knockdown argument. But I suspect one reason preprints aren’t more popular is simply that many people outside the sciences haven’t heard of them, but that at least can be addressed easily enough.