Replication of Experiments

Often there is a question whether one should run lab or field experiments. While there are different merits of both methods, there is one merit of lab experiments that does not seem to get sufficient attention: They are very easy to replicate!

Replication seems to me an important part of research. In economics, I feel replication is maybe a bit more common than in psychology: Economists can have a great career even if they never come up with their own bias or their own game. While we do not have many papers that are pure replications of previous results, many papers will have a control treatment that essentially amounts to a replication of a previous result.

As such I am also not completely in favor of unifying the software, exact procedures we use in a specific experiment. Often we have to make many small choices, and the results should not depend on those choices. But if we all use the same software, we'll not easily find out...

Psychology seems to have a maybe slightly bigger problem with replication, at least they are a lot in the news lately. I'd welcome suggestions and comments as to the hypothesis that economists may have a better time with replication because different research often get to replicate the main treatment of a new result as they use it as a control in their own paper. This I think is not the standard in psychology.

Here is a link to Ed Yong's feature for Nature on the problems of replication in psychology "Replication studies: Bad copy -- In the wake of high-profile controversies, psychologists are facing up to problems with replication."

Here's a recent Chronicle article By Tom Bartlett "Power of Suggestion: The amazing influence of unconscious cues is among the most fascinating discoveries of our time­—that is, if it's true"

"Psychology may be simultaneously at the highest and lowest point in its history. Right now its niftiest findings are routinely simplified and repackaged for a mass audience; if you wish to publish a best seller sans bloodsucking or light bondage, you would be well advised to match a few dozen psychological papers with relatable anecdotes and a grabby, one-word title. That isn't true across the board. Researchers engaged in more technical work on, say, the role of grapheme units in word recognition must comfort themselves with the knowledge that science is, by its nature, incremental. But a social psychologist with a sexy theory has star potential. In the last decade or so, researchers have made astonishing discoveries about the role of consciousness, the reasons for human behavior, the motivations for why we do what we do. This stuff is anything but incremental."

and

"Fairly or not, social psychologists are perceived to be less rigorous in their methods, generally not replicating their own or one another's work, instead pressing on toward the next headline-making outcome."

That's why, when we teach Experimental Economics, we often emphasize series of experiments.

And that's also why I am very grateful for everyone who ran gender competition experiments which provided a wealth of replication of our, perhaps surprising, findings on gender differences in competitive attitudes. Thank you all!