magazine issue 3 2015 / Issue 25

The perverse incentives that stand as a roadblock to scientific reform

• written by Brent W. Roberts
• edited by Sam Portnow

And, it is difficult to see how it can be any other way.  The existing system has maximized our success.  It provides a weird p-value driven utopia.  With the infinite flexibility of our current incentive system we can, as Simmons et al (2011) showed, provide empirical evidence for any idea, no matter how absurd it may be. All it takes is a lot of data and analyses. I fear these incentives have led to a system in which many, many more people have succeeded for ideas that will not last the test of time or replication, whichever comes first.  In other words, we have an excess of success.

The problem with the current push for methodological reform is that, it is hard, unrewarding, and a will result in a science that is a lot uglier than our current system. The truth is less elegant than what we produce in our scientific journals. Adopting a sounder approach to our scientific methods, which is critical for our long-term viability as a science, will inevitably curtail our excess success. There will be fewer famous psychologists, fewer book contracts, and fewer Ted Talks.

This is one reason why people fight so strongly against the benign reforms being proposed.  We’ve had a good gig for a long time and the future will be less bright if we do things with transparency and reproducibility in mind.  Of course, it is quite possible that our future will be less bright either way.

Relatedly, my pessimism is deepened by watching a critical mass of the senior leadership in psychology protest the proposed changes of the reform movement.  On one hand, I can understand their reticence to change.  The current system has obviously served them well.  After all, they are some of the most eminent, successful researchers in our guild.

On the other hand, I simply cannot fathom leading scientists arguing against things like making their research more transparent, increasing the power of their studies, and making sure their effects are replicable.  Keeping the status quo is like keeping the leftover holiday roast to rot in the back of the refrigerator.  It festers, grows mold, and stinks up the entire refrigerator, sullying all of the other items contained therein. Why not just throw it out? Holding on to the old ways makes everyone’s work reek and does unknown harm to the entire field.  The reform movement may result in a refrigerator that is less full, but at least the food that remains will be edible.

My fear at this juncture is that the punitive nature of the reform movement, combined with the lukewarm reception by senior leaders in psychology will combine to maroon an entire cohort of young scholars.  We’ve given young scholars and impossible choice.  Do things according to the existing reward structure and produce, in the ideal, an exciting and provocative, if ephemeral set of findings.  Do things according to the reform movement and produce something sober, maybe a little messy, and real.  And, of course this vision ignores that psychology is part of a larger network of scientists, funders, and benefactors, who are also free to look into the refrigerator and find things too smelly for their liking. Without a quick, decisive change in our approach to conducting science from top to bottom, I fear we will cause our field irreparable harm.

Footnotes

This essay is based, in part, on a previous blog post (http://wp.me/p1b8ZP-3x).  I would also like to express my gratitude to Chris Fraley and the graduate students who reviewed earlier versions of this essay and provided invaluable comments intended to improve the document. 

Small grammatical changes were made to the article by the author after the article had been posted. The current version reflects those changes.

References

Cohen, J. (1990). Things I have learned (so far). American psychologist, 45(12), 1304.

Fanelli, D. (2012).  Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904.

Ross, L., Lepper, M., & Ward, A. (2010). History of social psychology: Insights, challenges, and contributions to theory and application. Handbook of social psychology.

Simmons, J.P., Nelson, L.D., & Simonshohn, U.  (2012).  False- positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.  Psychological Science.

Simonsohn, U. (2014).  Small telescopes: detectability and the evaluation of replication results.  Available at SSRN: http://ssrn.com/abstract=2259879 or http://dx.doi.org/10.2139/ssrn.2259879

Vazire, S. (2014). Why I’m optimistic. http://sometimesimwrong.typepad.com/wrong/2014/12/why-i-am-optimistic.html