Failed replication studies are "empty"? Not for new graduate students.

As a new graduate student, I have zero publications. That's pretty normal. So to build up my basis of knowledge in the realm of neuroeconomics, I decided it would be a smart path to replicate an already well-documented decision making task, with future studies aimed at elaborating on what I find interesting within this task and hopefully carving out an area of expertise in choice strategy and uncertainty.

And if all goes well, then my replications of this task will be successful. But what happens if they aren't? Well, Dr. Jason Mitchell thinks they are worthless. And that's insane.

In a study I'll be presenting at Society for Neuroscience, I fail to replicate Prisoner's Dilemma results (you can read a little more about The Prisoner's Dilemma in this post). In general, people in a Prisoner's Dilemma situation are more likely to defect than to cooperate (~70% defection rate for most studies). I didn't find this. But I have theories as to why this behavior occurred based around the nature of cooperation and what the instructions were in my experimental design.

Is this study worthless because I couldn't replicate a very well researched experiment (~106k links through Google Scholar for "The Prisoner's Dilemma")? No. But if I were a graduate student under Dr. Mitchell, it would have been.

I don't know how the politics in Dr. Mitchell's lab are, but say I gave him these null results I have found. A year's worth of research I've been documenting on this site. What happens? Do I get scrapped? Do I get my proposal privileges taken away? Does he make my quals so ridiculously impossible that I terminate out? What are my consequences as his student?

I suppose he would have shot down my proposal to include The Prisoner's Dilemma in my study, since it hints towards replication and he's not down for that at all, apparently. But picture yourself as one of his graduate students in this position: sitting at my desk, stats processed and I have null findings. I'm at Harvard. Ph.D student in psychology -- same department B.F. Skinner, William James, & Donald Hebb earned their Ph.D's. Dr. Mitchell says these results will be worthless if I bring them to him as null. That's some immense pressure, and completely unwelcomed -- all because Dr. Mitchell either couldn't replicate a study on his own or someone was asking him why they couldn't replicate one of his studies he's published. Dr. Mitchell mentions replication studies are doing more harm than good, but does engendering an anti-null, anti-replication culture in your lab sound like a healthy place to be as a new graduate student?

Advisors who take on graduate students shouldn't discourage replication studies. Discouraging replication is not properly training your graduate students, as there is a wealth of knowledge to be learned from doing experiments for yourself -- regardless of how impactful or groundbreaking the research is. And discouraging null results is discouraging falsification -- and falsification is a tenant of science. Having a "null results are worthless" manifesto on your lab's page seems like a bad way to recruit prospective graduate students who would be interested in elaborating on your previous works. Unless "all replication is bad unless you're replicating my studies under my supervision" is what you really mean.

5 thoughts on “Failed replication studies are "empty"? Not for new graduate students.”

  1. Bob says:

    "all replication is bad unless you're replicating my studies under my supervision"

    Bingo

  2. Mike Taylor says:

    "Art is like science in that it attemps to make sense out of an apparently meaningless universe. However, the artist, unlike the scientist, doesn't want anyone else to be able to replicate his experiment" -- Claire West.

    In the absence of replications (both successful and unsuccessful) -- in the absence of even the possibility of falsification -- what Jason Mitchell's doing isn't science. It's art.

  3. broderolle says:

    From the vantage point of mathematical statistics, Mitchell's position looks just silly, as I elaborate here: http://haggstrom.blogspot.se/2014/07/on-value-of-replications-jason-mitchell.html

  4. Mike Taylor says:

    Also, if you're looking for a perfect model of what arrogance is, you'd find it hard to do better in just fifteen words than Mitchell's statement "the likeliest explanation for any failed replication will always be that the replicator bungled something".

    Which, mean translated, means: "Everything we publish is correct. You should not attempt to verify this, but if you do then if follows that any answer you get different from ours must be wrong."

  5. sbk says:

    Simply pathetic that a professor of psychological "science" publishes such uninformed BS, and particularly sad he does not recognize it as such.

    Clearly we are credentialing folk was beyond their competencies.

Leave a Reply