the InMind blog

Can you replicate that?

• written by Aaron Moss

The two previous “Solid Science” posts for this blog have covered important changes taking place in experimental psychology. If you have not read them, I recommend you do (post1, post2). In this post I report on another, larger change occurring in the field: the replication movement. 

Every undergraduate psychology student who takes a course in research methods learns replication is critical to science. The results of one experiment prove nothing.¹ Before the conclusions of a study or a larger theory are integrated into scientific knowledge, the results need to be repeated, again, and again. This is a cornerstone of science.

But most science does not work this way. Many undergraduates who get involved in research find that replication of previous experiments almost never occurs. Some students probably have a moment similar to one I experienced as an undergraduate. After completing the first study I ever worked on, I asked my advisors if the next step was to attempt to find the same thing with another sample of participants. Although they politely responded “no,” I imagine it took nearly all of their self-control to not laugh.

My question was naïve because in practice there is little incentive for researchers to attempt replication of their own work or others’. Direct replications are hard to publish because journals prefer using their limited space to publish findings that investigate novel relationships and expand current knowledge. In fact, replications are so rare a study published in 2012 found only 1% of papers published in psychology’s top 100 journals since 1900 were direct attempts to replicate previous research (Makel, Plucker, & Hegarty, 2012). If replications take place so rarely, you may ask, how can we have confidence in the results of research?

Some researchers have argued we can’t (Ioannidis, 2005; Simmons, Nelson, & Simonsohn, 2011). The failure to replicate research findings across several disciplines in science has led to a lot of media attention (Begley, 2012) and feisty arguments over the value of replication. Recently, however, some psychologists have taken steps to change the way the science is conducted.

A stronger science

In an effort to build a more reliable body of research, some psychologists have established a number of replication projects. Earlier this year, for example, an entire issue of the journal Social Psychology was devoted to publishing studies that attempted to replicate important findings in the field. The special issue was important for its findings—at least 10 of the 27 studies were not replicated at all²—but more so for its willingness to depart from the traditional incentives that motivate research and publication (read more about what successful and unsuccessful replications mean here).

Other projects are attempting to replicate studies published in prominent journals over the past few years (the Reproducibility Project) or coordinating the efforts of many researchers to investigate the reliability of important historical and contemporary findings (The Many Labs Project). The most immediate effect of these initiatives has been the pre-registration of studies and a significant change in the publishing and reporting standards at some of psychology’s top journals (both issues covered in previous Solid Science blog posts).      

Three reasons you should care

1. The changes psychologists experiment with can be adopted by other sciences.

Psychologists have taken a lead role in addressing the replication crisis. As they learn what changes to research and publication incentives are effective other fields may be able to adopt them. In addition, the collaborative efforts’ of many psychologists to replicate important findings in the field may provide a template for researchers in other disciplines interested in working across many labs. In fact, making collaboration among researchers easier is a goal of the Open Science Framework.

2. A major source of research funding is tax dollars.

Funding from the government makes the majority of research conducted at universities possible. Research has traditionally been, and still is, a good investment. However, resources are limited and it is important the research that is conducted is reliable. Reliable research makes science more efficient because less time is spent investigating effects that do not exist.      

3. Individually and collectively, we can have more confidence applying research findings.    

There is great value in psychological research and the research conducted in other disciplines. Often, the findings can be applied to social issues. As an example consider this problem. In recent years hotels have sought to save water (and money) by asking guests to reuse towels. Before investing money to print the “Please Reuse” cards placed in bathrooms, a wise hotel should ask, what message is most effective? While many hotels opt for a message emphasizing benefits to the environment, it turns out a more effective message is emphasizing that other people, who are similar to ourselves, normally reuse towels. Many findings like this can be applied in people’s daily lives and this makes it necessary for us to increase confidence in research findings.

Notes:

1. See Roger Giner-Sorolla’s In-Mind post titled “The Year the Journals Changed” for a nice review of significance testing in research. Significance testing explains why each study is subject to chance, and thus not enough for us to conclude that an effect is “true” or reliable.
2. It is important to note, the failure of a study to replicate previous findings does not mean the previous findings are false, or that the “effect” does not exist. There are many reasons a study may fail. But when a replication study is properly executed, the results offer some information about the reliability of the original findings.

References:

Begley, S. (2012). In cancer science, many “discoveries” don’t hold up. Reuters. Retrieved from: http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12...

Ioannidis, J. P. A. (2005.) Why most published research findings are false. PLoS Med 2 (8) DOI: 10.1371/journal.pmed.0020124

Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they take place? Perspectives on Psychological Science, 7 (6): 537-542. DOI: 10.1177/1745691612460688

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False- positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22 (11): 1359 – 1366.