Replication is how science distinguishes a genuine discovery from a fluke. When an independent team repeats a study and gets the same result, that result gains credibility; when they don’t, it’s a signal something may have gone wrong the first time. This self-checking mechanism sounds straightforward, but large-scale attempts to replicate published findings have revealed that a troubling share of them don’t hold up, with success rates in some fields hovering below 50%. Understanding why replication matters means understanding both what it protects against and what happens when it’s missing.
What Replication Actually Means
The word “replication” covers more than one thing. A direct replication follows the original study’s procedures as closely as possible, using different participants or samples but the same methods. The goal is to see whether the same recipe produces the same outcome. A conceptual replication, by contrast, tests the same underlying idea but uses a different approach entirely, perhaps a different experimental technique or a different population. If both methods point to the same answer, that’s stronger evidence than either method alone, because it rules out the possibility that the finding depends on one narrow set of conditions.1eLife. Making sense of replications – Section: What does it mean to repeat the methodology?
In practice, most published studies never get replicated at all. Replication work is time-consuming, expensive, and historically undervalued by journals and hiring committees. The studies that do get replicated tend to be high-profile claims with broad implications, which means the overall picture of how well everyday science holds up is still incomplete. But the replication attempts that have been done tell a sobering story.
How Often Published Findings Fail to Replicate
The most influential test of replication came from a massive project in psychology. Researchers repeated 100 studies that had been published in top psychology journals, using rigorous methods and, where possible, the original materials. The results were stark: while 97% of the original studies had reported statistically significant findings, only 36% of the replications did. The average effect size in the replications was about half what the originals had claimed.2PubMed. Estimating the reproducibility of psychological science
Cancer biology told a similar story. The Reproducibility Project: Cancer Biology repeated 50 experiments from 23 high-impact papers, looking at 158 individual effects. For positive findings where the original study claimed to have found something, the median effect size in the replications was 85% smaller than in the originals, and 92% of replication effect sizes were smaller. Using multiple methods to judge success, about 46% of effects replicated when positive and null results were combined.3PubMed Central. Investigating the replicability of preclinical cancer biology
These numbers don’t mean that half of science is wrong. They mean that a large fraction of published findings, particularly those with dramatic effect sizes, are inflated or fragile. And they raise an uncomfortable question: if the studies we’ve tested are the high-profile ones from top journals, what does that imply about the thousands of lower-profile papers that nobody has tried to replicate?
What Makes So Many Findings Fragile
Several forces conspire to fill the literature with results that look impressive but don’t hold up. None of them require deliberate fraud, though fraud exists too. The more common problems are structural.
Publication bias is among the most corrosive. Journals overwhelmingly publish studies that find something rather than studies that find nothing. Estimates suggest that roughly half of clinical trial results go unpublished, and in social science, about two-thirds of experiments with null results end up in the “file drawer,” never seeing print, while nearly all experiments with significant results get published.4PubMed Central. fiddle: a tool to combat publication bias by getting research out of the file drawer and into the scientific community The consequence is that anyone reading the published literature gets a distorted view: the evidence looks consistently positive because the negative evidence was never made visible.
Low statistical power is another culprit. When a study is too small to reliably detect the effect it’s looking for, any significant result it does produce is likely to be a wild overestimate. Research on this problem found that when studies had power below 30%, fewer than 5% of the significant results they turned up were even roughly correct. At very low power levels, significant findings were almost systematically inflated, sometimes by a factor of two or more.5Global Epidemiology. Statistically significant results from low-power analyses: A comedy of errors Many fields routinely run underpowered studies, which means the literature is peppered with inflated findings that will shrink or vanish when someone runs a properly sized replication.
Then there are questionable research practices, sometimes called QRPs. These include things like testing many possible analyses and reporting only the one that produced significance (p-hacking), presenting a hypothesis that was actually formed after seeing the data as though it had been planned in advance (HARKing), and cherry-picking which results to highlight while burying unfavorable ones.6PubMed. HARKing, Cherry-Picking, P-Hacking, Fishing Expeditions, and Data Dredging and Mining as Questionable Research Practices Survey data suggest that HARKing in particular is widely practiced and widely seen as inappropriate, yet it persists because it helps researchers tell a cleaner story.7PubMed. HARKing: hypothesizing after the results are known
Compounding all of this, many published studies fail to report basic methodological details that would allow someone else to replicate the work. A study of event-related brain potential research, for instance, found that failing to report key parameters was “ubiquitous” and that studies were primarily powered to detect only large effects, creating substantial barriers to replication.8PubMed. Methodological reporting behavior, sample sizes, and statistical power in studies of event-related potentials: Barriers to reproducibility and replicability You can’t replicate what you can’t fully understand.
What Irreproducible Research Actually Costs
Replication isn’t just an academic exercise. When preclinical research can’t be reproduced, it has direct consequences for medicine. Drug development depends on building upon earlier published findings; if those foundations are shaky, entire drug programs can collapse years and billions of dollars down the line. Irreproducibility in preclinical work has been traced to errors and flaws at multiple stages, from the reference materials used to the study design to how data are collected and analyzed.9PubMed. The impact of preclinical irreproducibility on drug development
An economic analysis estimated that irreproducible preclinical research costs approximately $28 billion per year in the United States alone, based on the finding that the overall prevalence of irreproducible results exceeds 50%.10PubMed Central. The Economics of Reproducibility in Preclinical Research That figure captures money spent on research paths that ultimately lead nowhere because they were built on results that don’t hold up. It doesn’t capture the harder-to-quantify cost of delayed treatments for patients, or of clinical trials launched on the basis of preclinical findings that were never solid to begin with.
When Replication Is Harder Than It Sounds
Not every failure to replicate means the original finding was wrong. Some research topics are genuinely sensitive to context: the population studied, the cultural setting, the time of year, even subtle features of how participants experience the experiment. Analysis of the large-scale psychology replication project found that contextual sensitivity of the research topic predicted replication success even after accounting for methodological factors like statistical power and effect size.11PubMed Central. Contextual sensitivity in scientific reproducibility A finding about how people respond to social pressure, for example, might legitimately differ across cultures or decades in ways that a finding about basic visual perception would not.
Field sciences face an even steeper challenge. In ecology, experiments take place under the highly variable conditions of natural environments, which reduces the capacity for clean replication compared to lab-based disciplines. Researchers in field ecology have noted that concepts like reproducibility and replicability are not always clearly distinguished in their field, and that the high variability of natural conditions creates fundamental obstacles to assessing whether results can be replicated.12Wiley Online Library. Replication in field ecology: Identifying challenges and proposing solutions You can’t rerun a wildfire or a drought. This doesn’t mean replication is irrelevant to these fields, but it means the standard model of “repeat the experiment and see if you get the same answer” needs to be adapted.
How Academic Incentives Work Against Replication
One reason replication efforts remain rare is that the incentive structure of academic science actively discourages them. Career advancement, tenure, and grants are awarded largely on the basis of publishing novel findings in prestigious journals. Replication studies, by definition, are not novel. They repeat someone else’s work. This means a researcher who spends a year running a careful replication gets far less professional credit than one who publishes a new and surprising result, even if the replication is more valuable to the field.
This creates a perverse dynamic. Modeling work has shown that when scientists are rewarded chiefly on publication volume and novelty, it creates conditions where careless conduct can thrive, compounded by the tendency of top journals to favor novel, positive results over null findings or confirmatory work.13PubMed Central. Modelling science trustworthiness under publish or perish pressure Reforming the evaluation system to value replication is widely discussed, but change has been slow, in part because prestigious publications and citation-based metrics remain central to how academic credit is assigned.14PubMed Central. The misalignment of incentives in academic publishing and implications for journal reform
Fixes That Are Gaining Ground
Despite these structural headwinds, several reforms are gaining traction. Open data practices, where researchers publicly share their datasets and analysis code, make it possible for others to independently verify published results.15Royal Society Open Science. An empirical assessment of data sharing and computational reproducibility in sociology When surgical journals adopted data availability policies, reanalysis of shared data confirmed the primary outcomes of the studies that complied, though it also turned up minor discrepancies in patient counts, handling of missing values, and timing of outcomes, the kind of errors that wouldn’t necessarily change the headline result but that would remain invisible without access to the raw data.16PubMed Central. Data Sharing and Reanalyses Among Randomized Clinical Trials Published in Surgical Journals Before and After Adoption of A Data Availability and Reproducibility Policy Papers with open data also tend to accumulate more citations than those without, which creates at least a modest incentive for researchers to participate.4PubMed Central. fiddle: a tool to combat publication bias by getting research out of the file drawer and into the scientific community
Preregistration is another reform with momentum. By publicly committing to a study’s hypotheses, methods, and analysis plan before collecting data, researchers make it much harder to engage in p-hacking or HARKing after the fact. Registered Reports, a journal format where peer review happens before results are known, go even further by guaranteeing publication regardless of whether the findings are positive or null, which directly combats publication bias.
Multi-lab collaborations and adversarial collaborations represent a more ambitious approach. In adversarial collaborations, researchers who disagree about a scientific question design and conduct a study together, reducing the impact of biases and hidden agendas on both sides. Combining this with multi-lab replication, where several labs run the same protocol independently, generates stronger evidence than any single lab could produce.17PubMed Central. Next steps after airing disagreement on a scientific issue with policy implications: a meta-analysis, multi-lab replication and adversarial collaboration
How the Public Reads Replication Failures
The replication crisis has not stayed within the walls of academia. Public awareness of it has grown, and how people interpret it matters for trust in science more broadly. Experimental research found that when people learn a study has been successfully replicated, they rate the evidence as stronger and the researchers as more trustworthy. When they learn a study failed to replicate, both ratings drop.18PubMed. Replication crisis = trust crisis? The effect of successful vs failed replications on laypeople’s trust in researchers and research That’s a rational response, up to a point. The risk is that people hear “replication crisis” and conclude that science in general can’t be trusted.
Survey data from Germany found that most people interpret replication efforts as evidence of scientific quality control and science’s self-correcting nature, which is encouraging. However, supporters of certain populist political movements were more likely to interpret the crisis as proof that science can’t be trusted at all, using it as ammunition to discredit scientific expertise.19PubMed. The “replication crisis” in the public eye: Germans’ awareness and perceptions of the (ir)reproducibility of scientific research This gap highlights a communication challenge: replication failures are actually evidence that the system is working when they happen, because they catch errors. The problem was never that studies failed to replicate; it was that nobody was checking.
Replication Problems in Machine Learning
The reproducibility challenge extends well beyond traditional lab science. As machine learning has been adopted across scientific fields, it has brought its own version of the problem. A survey of the literature identified 17 fields where data leakage, a technical error where information from the test set bleeds into the training process, had been found, collectively affecting 294 papers and in some cases leading to wildly overoptimistic conclusions. When the errors were corrected, the sophisticated machine-learning models didn’t perform meaningfully better than much simpler, decades-old approaches.20PubMed Central. Leakage and the reproducibility crisis in machine-learning-based science This is a different flavor of irreproducibility than a psychology experiment that doesn’t replicate, but the root issue is the same: results that look impressive in the original paper but don’t survive independent scrutiny.
The machine-learning case also illustrates why computational reproducibility, the ability to re-run someone’s code and get the same numbers, matters alongside traditional replication. A result that depends on a specific coding error or an unreported preprocessing step isn’t really a scientific finding. It’s an artifact.
Measuring Replication Success Is Itself Complicated
Even the seemingly simple question of “did this study replicate?” turns out to be harder than it looks. The cancer biology project used seven different methods to assess replication, and they didn’t always agree. One common approach, using meta-analysis to combine the original and replication results and check whether the combined evidence is significant, has been found to perform poorly as a replication metric. It has a relatively high probability of concluding that replication succeeded even when the true effect is zero, largely because the significance of the original study biases the combined result.21PubMed Central. Evaluating meta-analysis as a replication success measure This means that some of the replication “success rates” reported in the literature may themselves be overly optimistic, depending on which metric was used.
The messiness of this measurement problem doesn’t undermine the importance of replication. It underscores it. If we can’t even agree on what counts as a successful replication, the need for multiple independent attempts at a finding becomes even more pressing.
Replication and the Legal System
Outside the laboratory, replication plays a role in settings where the stakes are immediate and personal. In courtrooms, scientific evidence is regularly used to inform decisions about guilt, innocence, and liability. The credibility of that evidence depends on whether it was produced by valid methods and whether the results are reproducible, but the adversarial structure of legal proceedings creates conditions that are, as one analysis put it, “scarcely recognizable by normal standards of scientific investigation.”22PubMed Central. A scientist’s take on scientific evidence in the courtroom.
Forensic science in particular is undergoing a shift. For decades, many forensic disciplines relied on the judgment of individual examiners, with limited empirical testing of whether their methods produced reliable, reproducible results. That is changing: the field is moving from a “trust the examiner” framework to one built around trusting the scientific method itself, which means demanding that forensic techniques demonstrate reproducibility before being treated as authoritative evidence.23PubMed Central. The scientific reinvention of forensic science This evolution is still partial and contested, but it reflects a broader principle: wherever scientific claims carry consequences, asking “can someone else get the same result?” is not optional.