How Do You Judge a Statistical Result You Cannot Reproduce?

What Can You Conclude When You Cannot Reproduce the Result?

You have examined a paper carefully, perhaps even tried to reconstruct one of its analyses, but you cannot obtain the reported result. Maybe the raw data are unavailable. Perhaps the authors did not share their code, or the statistical model is described without enough detail to rebuild it. In other cases, you have the necessary materials but your calculation simply produces a different answer.

That creates an uncomfortable appraisal problem. Should you distrust the result, accept it provisionally, or conclude that the analysis is wrong?

The answer depends heavily on why reproduction failed. Failure caused by unavailable data is not equivalent to failure caused by executable code that produces different results. Critical appraisal therefore requires you to identify what is missing or inconsistent before deciding how much the problem should affect your confidence in the finding.

Non-Reproduction Is a Reason for Scrutiny, Not an Automatic Refutation

Diagnose the Reproducibility Problem Before Judging the Finding

Start by defining what you tried to reproduce

Reproducibility is often discussed as though it were a single yes-or-no property. In practice, your appraisal should be more precise.

You might be trying to verify a percentage reported in a table, reconstruct an effect estimate from summary data, rerun a regression using shared data and code, or reproduce every table and figure in the article. Those are very different tasks.

For computational results, reproduction generally means obtaining the same or sufficiently similar results by applying the reported analysis to the same underlying data. This is distinct from replication, which asks whether a scientific claim can be supported with new data. The distinction matters because failure at one does not necessarily establish failure at the other.

There are several fundamentally different reasons reproduction can fail

Before interpreting the failure, locate the obstacle.

These scenarios should not receive the same appraisal judgment. A paper that cannot be reproduced because confidential patient-level data cannot legally be released presents a different problem from a paper whose publicly archived code and data generate estimates inconsistent with its published tables.

Ask whether the result is reproducible in principle

Some barriers to reproduction are legitimate. Individual-level health, educational, administrative, commercial, or otherwise sensitive data may be subject to ethical, legal, contractual, or privacy restrictions. In such circumstances, unrestricted public release may be inappropriate.

The relevant appraisal question becomes whether the authors provide enough transparency to understand what was done and whether appropriate mechanisms exist for qualified access or verification where feasible.

Conversely, when an analysis depends on custom computational procedures but neither code nor sufficient methodological detail is available, uncertainty about the analytical workflow increases. Code can contain consequential decisions that are difficult to communicate completely in prose. Making data, code, and outputs explicitly connected can make computational verification substantially easier.

Examine what you can verify without reproducing the computation

A failed reproduction attempt does not eliminate ordinary critical appraisal. You can still inspect whether the design addresses the research question, whether the statistical method appears appropriate, whether the variables and outcomes are clearly defined, whether the sample and exclusions are reported, and whether the analysis appears consistent with the stated design.

Then inspect the numerical evidence. Are denominators consistent? Do descriptive statistics agree across tables? Does the direction of the reported effect match the raw summaries? Are estimates accompanied by appropriate measures of uncertainty? Do the conclusions accurately represent the results?

This is one reason you generally do not need to recalculate every statistic during critical appraisal. Statistical reproduction can strengthen verification, but substantial appraisal remains possible without it.

Judge the effect estimate and its uncertainty, not just whether the P-value can be recreated

A reported result should not be reduced to whether its P-value falls above or below 0.05. Cochrane guidance emphasizes interpreting the point estimate together with its confidence interval because the interval conveys information about statistical uncertainty and precision.

That remains useful even when you cannot reproduce the underlying calculation. Ask whether the reported estimate represents a trivial, moderate, or potentially important effect in the substantive context. Examine how wide the confidence interval is and whether it includes materially different interpretations.

The relationship among P-values, effect sizes, and confidence intervals can also expose internal inconsistencies. For corresponding conventional tests and intervals, certain combinations should agree logically. Apparent contradictions deserve investigation, although differences in statistical procedures can sometimes explain them.

Look for robustness rather than one privileged analysis

A central estimate becomes more persuasive when reasonable analytical alternatives produce substantively similar conclusions. Sensitivity analyses may vary assumptions, definitions, exclusions, missing-data procedures, model specifications, or other defensible analytical decisions.

This does not mean that every possible analysis should produce the same number. Statistical analyses often contain legitimate researcher choices, and different reasonable specifications may yield different estimates. What matters is whether the substantive conclusion is unusually dependent on one narrow set of decisions.

If a claim appears only under a particular specification while plausible alternatives produce substantially different conclusions, that fragility is relevant even if the preferred model was computed correctly.

Consider whether the paper gives you enough information to understand the analysis

Statistical reporting guidelines such as SAMPL emphasize reporting statistical methods and analyses sufficiently clearly. Transparency is not merely an editorial nicety. Without adequate information about how variables were treated, models were specified, missing observations were handled, or adjustments were made, readers may be unable to determine what generated the published result.

When the analysis is sophisticated, the absence of sufficient information becomes particularly consequential. If you cannot determine whether the method was applied appropriately, it may be better to recognize the limits of your own evaluation rather than infer validity from technical complexity.

The importance of non-reproduction depends on how much rests on the result

Not every discrepancy deserves the same response. A rounding difference in a secondary descriptive table is unlikely to overturn an otherwise coherent study. Failure to reproduce the primary analysis on which the paper's main conclusion depends is much more consequential.

Ask a counterfactual question: if this result were substantially different, would your interpretation of the study change?

If yes, unresolved reproducibility becomes central to the appraisal. If no, document the problem but keep its importance proportional to its role in the study.

Reproducibility itself is not the same as scientific validity

A result can be perfectly reproducible and still arise from a biased design, inappropriate model, poorly measured outcome, unjustified adjustment strategy, or selective analysis. Reproduction asks whether the computation can be obtained again. Critical appraisal asks the broader question of whether the evidence justifies the inference.

The reverse also deserves caution. A result that cannot currently be reproduced may ultimately prove correct once missing code, data transformations, software dependencies, or analytical details are supplied.

What If Shared Data Produce a Different Result?

If the difference altered the direction of the association, moved the estimate from substantively important to trivial, or materially changed the uncertainty around it, the unresolved discrepancy would deserve considerably more weight in your appraisal.

Common Mistakes When a Result Cannot Be Reproduced

Match Your Confidence to What Can Actually Be Verified

When reproduction fails, resist the temptation to make the problem binary. Instead, identify where the evidential chain becomes uncertain and ask how important that uncertainty is to your intended use of the paper.

The central principle is proportionality. A result that cannot be reproduced deserves scrutiny, but the strength of your concern should reflect why reproduction failed, how large the discrepancy is, and how much of the paper's conclusion depends on that result.

When You Cannot Reproduce a Statistical Result

Questions About Results You Cannot Reproduce

Unexplained Non-Reproduction Should Change Your Certainty, Not End the Appraisal

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.