magazine issue 3 2015 / Issue 25

A Perfect Storm: The Record of a Revolution

• written by Eric-Jan Wagenmakers
• edited by Sam Portnow

At some point in their past, almost every country has witnessed a political revolution, a change of government following a dramatic and sometimes violent expression of discontent. As a result, emperors have been beheaded, kings dethroned, and presidents exiled. Revolutions are often caused by a slowly growing dissatisfaction in the general population, for instance due to lost wars, lack of food, or high taxes. In other words, the general population feels a strong desire for change. At some point, resentment reaches a boiling point and a single event –another tax hike, another arrest by the secret police– is enough to trigger a cascade of violent protests that can culminate in the guillotine.

The dynamics of political revolutions are in some ways similar to the academic revolution that has recently gripped the field of psychology¹. Over the last two decades, increasing levels of competition for scarce research funding have created a working environment that rewards productivity over reproducibility; this perverse incentive structure has caused some of the findings in the psychological literature to be spectacular and counter-intuitive, but likely false (Frankfurt, 2005; Ioannidis, 2005). In addition, researchers are free to analyze their data without any plan and without any restrictions, leaving the door wide open for what Theodore Barber termed Investigator Data Analysis Effects: "When not planned beforehand, data analysis can approximate a projective technique, such as the Rorschach, because the investigator can project on the data his own expectancies, desires, or biases and can pull out of the data almost any 'finding' he may desire." (Barber, 1976, p. 20). 

The general dissatisfaction with the state of the field was expressed in print only occasionally, until in 2011 two major events ignited the scientific revolution that is still in full force today. The first event was the massive fraud from well-known social psychologist Diederik Stapel, who fabricated data in at least 55 publications. Crucially, Stapel's deceit was not revealed through the scientific process of replication – instead, Stapel was caught because his graduate students became suspicious and started their own investigation. The second event was the fact that social psychologist Daryl Bem managed to publish nine experiments showing that people can look into the future. More importantly, he succeeded to publish this result in a flagship outlet, Journal of Personality and Social Psychology (Bem, 2011). To some researchers, these two events made it clear that when it comes to the publication of academic findings in premier outlets, one can apparently get away with just about anything.

Of course, these landmark events were quickly followed by others. For instance, Simmons et al., (2011) used concrete examples of Barber's Investigator Data Analysis Effects, demonstrating that “undisclosed flexibility in data collection and analysis allows presenting anything as significant”. Moreover, John et al. (2012) reported that many researchers self-report the use of “questionable research practices” such as cherry-picking, post-hoc theorizing, and optional stopping (i.e., testing more participants until the desired level of significance is obtained). In addition, more cases of academic misconduct were detected – among those accused are Lawrence Sanna, Dirk Smeesters, and, most recently, Jens Foerster. Moreover, researchers whose work failed to replicate responded in anger, accusing those who conducted the replications of incompetence or worse (e.g., Yong, 2012). Nobel-prize laureate Daniel Kahneman wrote an open letter expressing his concern and urging the field to conduct more replication studies. However, the majority of replication studies that have been published since Kahneman's open letter failed to replicate the original findings, a general trend corroborated by the contributions for the special issue on replications in the journal Social Psychology (Nosek & Lakens, 2014). As far as replicability is concerned, the proof of the pudding is in the eating, and so far the results have been flat-out disappointing.

These adverse events, as well as many others, created a perfect storm of skepticism that prompted several psychologists to take up the gauntlet and develop a series of concrete initiatives to address the endemic problems that plague not just psychology but beset empirical disciplines throughout academia. Before discussing these initiatives I will briefly present what I personally believe to be the two most important challenges that confront psychological science today.