Replication, Replication

“These doubts emerged and grew as a series of unhappy events unfolded in 2011: the Diederik Stapel fraud case, the publication in a major social psychology journal of an article purporting to show evidence of extrasensory perception followed by widespread public mockery, reports by Wicherts and colleagues that psychologists are often unwilling or unable to share their published data for reanalysis, and the publication of an important article in Psychological Science showing how easily researchers can, in the absence of any real effects, nonetheless obtain statistically significant differences through various questionable research practices (QRPs) such as exploring multiple dependent variables or covariates and only reporting these when they yield significant results.”3 The list continues. #

In what makes a cure even more evasive, psychology has its very own problem with replication: it’s becoming ever more rare.4 Even for graduate students, merely replicating existing findings is an ungainly endeavour, scarcely rewarded and discouraged by what Christopher J. Ferguson and Moritz Heene call an “aversion to the null” in psychology.5 If it ain’t significant, it ain’t gonna be published. The problem goes so far that it appears appealing to let undergraduates do the work.6 But when discussing my capstone project, even I was discouraged from merely replicating known findings. #

  1. Which of course is not exactly true in my case. []
  2. This, by the way, was also what deterred me most strongly from the study of economics: there appears to be little space in undergraduate curricula for research methodology, not to speak of critical questioning. It is not just that assumptions are made quite implicitly; I have hardly found them defended argumentatively upon asking. []
  3. Pashler, H. & Wagenmakers, E.-J. (2012). Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science, 7(6), 528-530. doi:10.1177/1745691612465253 []
  4. Ferguson, C. J. & Heene, M. (2012). A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null. Perspectives on Psychological Science, 7(6), 555-561. doi:10.1177/1745691612459059 []
  5. ibid. []
  6. Grahe, J.E., Reifman, A., Hermann, A. D., Walker, M., Oleson, K. C., Nario-Redmond, M., & Wiebe, R. P. (2012). Harnessing the Undiscovered Resource of Student Research Projects. Perspectives on Psychological Science, 7(6), 605-607. doi:10.1177/1745691612459057 []
  7. Fuchs, H. M., Jenny, M., & Fiedler, S. (2012). Psychologists Are Open to Change, yet Wary of Rules. Perspectives on Psychological Science, 7(6), 639-642. doi:10.1177/1745691612459521 []
  8. Another one that comes up time and again is statistical significance. How often have I read sentences à la “X was faster/higher/stronger than Y, but the difference was not significant”. If you are writing this, you’ve either slept through Stats 101, or you’re trying to imply something your data simply does not support. []
  9. Then again, it might spare us certain neuropsychology studies and the associated yellow press headlines… []
  10. Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124 []

Comments are closed.