False-Positives

Uri Simonsohn has several papers on scientific conduct, that should be a rule not only in psychology.

The first paper, "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant" is by Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn in Psychological Science, V22(11), pp.1359-1366


Their abstract reads:
"In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (less or equal to .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process."

A striking Figure from the paper is the following:

"[..] Figure 1 shows the false-positive rates from additional simulations for a researcher who has already collected either 10 or 20 observations within each of two conditions, and then tests for significance every 1, 5, 10, or 20 per-condition observations after that. The researcher stops collecting data either once statistical significance is obtained or when the number of observations in each condition reaches 50. Figure 1 shows that a researcher who starts with 10 observations per condition and then tests for significance after every
new per-condition observation finds a significant effect 22% of the time."

Stay tuned for more...