How serious is the replication crisis in psychology?

The problem is real and measurable, but its size is contested. The best-known test, a 2015 project that repeated 100 studies, found that most replications gave weaker evidence than the originals. Researchers disagree about whether this means many original findings were false, or whether the replication method and statistics exaggerate the failure. Large-scale estimates also suggest that replicability differs a great deal across subfields and research methods. [1] [2]

The Open Science Collaboration repeated 100 experimental and correlational studies from three psychology journals, using high-powered designs and the original materials where available. Ninety-seven percent of the original studies had significant results, against 36 percent of the replications. The average effect size in the replications was about half that of the originals. Other measures gave different figures: 47 percent of original effect sizes fell within the replication's confidence interval, and 39 percent of effects were subjectively rated as replicated. The authors stressed that no single indicator captures replication success. Original studies with stronger initial evidence were more likely to replicate. [1]

The team did not use the word crisis and said the results were an estimate of reproducibility, not proof that any original finding was wrong. Project leader Brian Nosek said that no study, original or replication, is definitive. [2]

One view holds that the failures are substantial and reflect weak original evidence. A reanalysis that modelled publication bias concluded that more than 90 percent of tests in eligible experiments examined negligible effects, and that p-value-based publication bias caused the observed non-reproducibility. [3]

A second view, from Daniel Gilbert, Timothy Wilson and colleagues, criticises the 2015 project itself. They argue that the studies were not randomly chosen, since some were excluded as too hard to replicate, and that many replications differed from the originals in population, country or method. They note that replications approved by the original authors succeeded more often, and they argue that the number of failures is what chance alone would produce given the replications' power and design flaws. [2]

A third view is that single replications are the wrong yardstick. One paper argues that many so-called failures may stem from low statistical power in single replication studies and that multiple replications, Bayesian methods and meta-analysis are needed. A survey of 200 meta-analyses found median statistical power of about 36 percent, with only about 8 percent of studies adequately powered, and high heterogeneity of effects, which together it says fully explain recent replication difficulties. It found only a small average amount of residual reporting bias. The results do not show which of these accounts is correct, and the sources here do not report a consensus among specialists. [4] [5]

A 2023 PNAS study applied a validated machine learning model to 14,126 papers from six top psychology journals over 20 years. Its estimates were broadly in line with manual replications, which suggest roughly a 43 percent success rate. It found that a single replication rate hides large differences between subfields, and that experiments replicate at a significantly lower rate than non-experimental studies. Authors with more publications and higher citation impact were more likely to have replicable work, while university prestige and a paper's own citation count were unrelated. Media attention was associated with a higher likelihood of replication failure. These are model-based predictions, not actual replication attempts. [6]