Replicate me (or not)

/

Summary: There is a lot of interest in replication in Psychology. Regardless of whether this is the best metric to focus on, there is no question that having a study of yours selected for a replication trial can be stressful and intimidating (so I imagine). Here I volunteer to be replicated because I think it's good for science, even though it might be stressful.

Recent days and months and years have seen increased discussion about the "replication" "crisis" in Psychology, and probably for good reason. If we repeat an experiment many times, we would intuitively expect a consistent result, and not obtaining one might give us pause as to how we interpret the findings. The problems of the "file drawer" are well-known and not going away any time soon.

Of course, I can sympathize with researchers being in the position of having their work called into question. I am not in that position, so I don't get it, and I understand that I just don't get it. I am not clever enough (yet) to have a Peelle Effect that I have staked my reputation and career on. If I do someday, I will of course be stressed out if people call it into question.

But here's the thing.

We are all interested in figuring out the way things really work, right? The way that people, or the brain, or the laws of physics, behave in predictable (or predictably unpredictable) ways. So, if two scientists obtain different results, this seems important. And, yes, we have to think about effect sizes and sample sizes, and consider the possibility that running 20 undergraduates may not result in a rock-solid estimate of human behavior. (This seems to be, surprisingly, a point of contention.)

The vast majority of the dialogue I see around Psychological Science (capitals intended) and replication is honest and well-intentioned. Maybe there are pockets of mean-spirited scientists who are out to destroy careers, but I haven't met them yet. What I have seen is a group of scientists concerned about finding out what true effects are. Now, it may be that they do not have the perfect approach. But, I have yet to see a better one offered (apart from "nothing to see here, stop making trouble").

I'm also not aware of a single case of a well-intentioned junior researcher whose career was squashed due to failure to replicate in the absence of any other concerns. (If you know of one, please leave it in the comments, or email me if confidential—I'll tally but not reveal names.) The "failures to replicate" that I've seen are typically established researchers who are not in danger of losing their jobs or livelihood over the increased scrutiny. This is not to minimize the stress that failures to replicate can cause: the loss of sleep and confidence are surely formidable. And one frequent comment/criticism is that the "replicators" choose their "targets" without consultation or agreement, such that researchers whose studies are selected for possible replication feel out of control and sucked in as unwilling participants. On the one hand, this seems fair game in science. On the other hand, of course, it does not seem pleasant.

So, let me volunteer. Here I am—come replicate my studies. Sound files are available that we have used in the following behavioral and MRI studies:

  • Lee YS, Min NE, Wingfield A, Grossman M, Peelle JE (2016) Acoustic richness modulates the neural networks supporting intelligible speech processing. Hearing Research 333:108–117. doi:10.1016/j.heares.2015.12.008

  • Hassanpour MS, Eggebrecht AT, Culver JP, Peelle JE (2015) Mapping cortical responses to speech using high-density diffuse optical tomography. NeuroImage 117:319–326. doi:10.1016/j.neuroimage.2015.05.058

  • Peelle JE, Troiani V, Grossman M, Wingfield A (2011) Hearing loss in older adults affects neural systems supporting speech comprehension. Journal of Neuroscience 31:12638–12643. doi:10.1523/jneurosci.2559-11.2011

  • Peelle JE, Troiani V, Wingfield A, Grossman M (2010) Neural processing during older adults’ comprehension of spoken sentences: Age differences in resource allocation and connectivity. Cerebral Cortex 20:773–782. doi:10.1093/cercor/bhp142

  • Peelle JE, McMillan C, Moore P, Grossman M, Wingfield A (2004) Dissociable patterns of brain activity during comprehension of rapid and syntactically complex speech: Evidence from fMRI. Brain and Language 91:315–325. doi:10.1037/0096-1523.31.6.1315

  • Wingfield A, Peelle JE, Grossman M (2003) Speech rate and syntactic complexity as multiplicative factors in speech comprehension by young and older adults. Aging, Neuropsychology, and Cognition 10:310–322. doi:10.1076/anec.10.4.310.28974

I would welcome other people using these files and this paradigm, based on the descriptions in our papers which I'm happy to further clarify. I would like to have an idea of the true effect, and how the brain works—if you guys want to help me, so much the better. If you don't replicate the effect, let's figure it out, or I will stop citing the effect and we can all move on with our lives.

I'm happy to read your replications whether they appear as traditional peer-reviewed manuscripts, blog posts, preprints, Twitter comments, or typed missives delivered by courier pigeon. (I will not, however, read faxes, just because I literally don't even know where I can receive a fax. Please print it out and mail it instead.)

My goal for my own lab is to make all our materials and methods publicly available when we publish our papers, and to make it as easy as possible for others to inspect our work, and replicate it if they wish. We are not there yet, but we are moving in this direction. It is certainly scary to be open with materials, methods, and data, but I have yet to find an argument as to why this is bad for science. Rather than run from replication, we would be better off if we joined together as a community to understand and improve our approach to sampling, statistics, and inference.