Reviewing in support of open science

/

Summary: I've recently signed the Peer Reviewers' Openness Initiative (PRO). Here are some thoughts on why, and more generally on how we as reviewers can have a positive impact on science.

[Note: This post has been published on The Winnower.]

I've recently signed on to the Peer Reviewers' Openness Initiative (PRO). You can read the full initiative but the basics are that as reviewers we require data and stimuli be made publicly available; and if not, that authors provide a reason as to why in the manuscript (that now becomes part of the scientific record).

Why should stimuli and data be made publicly available? Others have written in more detail on this:

Tal Yarkoni put it succinctly with his tweet: "[B]ecause I’m lazy and I’d rather point emailers to GitHub than deal with their stupid valid requests for my data and code".

My own take is that sharing materials and data promotes more accurate science because (a) it facilitates checking of one's data and analysis by others, and (perhaps equally importantly) (b) our internal checks on data organization and accuracy are typically better if we know the materials will be publicly available. So, even if no one ever looks at them, the mere fact that we are sharing them will improve the accuracy of our research. I know this is certainly true in my own work. It may slow down the process, but I think this simply reflects a speed/accuracy tradeoff shifting towards more accuracy. In the long run this seems like only a good thing for scientific discovery.

Although he had to squeeze it into 140 characters, Tal's point is also a good one. In theory, most of us support making data available "upon request", and have also probably committed to do so in agreements with publishers and funding bodies. However, I shudder to think of the difficulty of tracking down data from an old study, after the details have faded from my memory, I've changed computers half a dozen times, and everyone else involved in the study has moved on from the lab. Making data publicly available forces an up-front investment in organization that can be done while the details are fresh and personnel still around, and sharing via a public repository helps guard against loss due to moving personal computers or webservers. It makes future re-use of the data, by the original authors or others, much more realistic. So, even if the arguments about improving Science don't sway you, there's always laziness and saving your future self countless hours of frustration in trying to find and explain old datasets.

Of course, there are times when sharing data or materials is simply not possible, or not ethical. For example, if ethics approval prohibits the sharing of data, or the data being shared could compromise participants' privacy. There are frequently solutions to these problems, including sharing only anonymized and deidentified data, but if there is no solution that's fine—the PRO simply says that these reasons should be shared in the article so they are public. Speaking for myself, the reasons I have not shared data or materials in the past have nothing to do with concerns about privacy (due to the nature of the data we collect), but have to do with the additional time and effort that it costs to share materials. So, for my lab, this is not much of a practical concern (although obviously in some cases it might be).

Which brings me to an important point. Some observant readers will notice that I myself have not followed these guidelines in the past. That's true, but also changing. Obviously it doesn't make sense to have standards for the papers I review that I don't uphold myself. I'll write more on this in the future, but the short version is that whenever possible my lab is committed to making our materials and data openly available along with the publication of each article. I would welcome a PRO-signing reviewer on any of our upcoming lab papers.

It is worth emphasizing that although we have a goal of sharing everything, we may not be able to get there right away. Being OK with this fact has gone a long way towards helping me be more optimistic about the process: we are sharing more now than we were a year ago, and learning lessons along the way.

It's my hope that more reviewers will be inspired to take a more active role in shaping the future of science, making it more open and of higher quality. The PRO is a good step in this direction. I'd encourage you to read about the initiative and consider adding your signature. More important than your signature, we can all consider whether a culture of openness in our research practices—starting with making stimuli, data, and analyses publicly available—will benefit all of us by improving the quality of our science. If so, what are you waiting for?