How Do You Know Whether Missing Studies Could Plausibly Change the Conclusion?

Missing Evidence Exists, but Would It Actually Change Your Conclusion?

You suspect that some relevant studies were never published. Perhaps registry records have no corresponding journal articles. Perhaps several studies mention outcomes that never appear in their results. Perhaps the meta-analysis shows a pattern consistent with missing small studies.

That creates a legitimate concern, but it does not answer the most important question.

Would the missing evidence merely add a little uncertainty, or could it realistically change the estimated effect, its direction, its statistical or practical interpretation, or your confidence in the conclusion?

This is much harder than simply asking whether publication bias exists.

Ask Whether a Plausible Missing-Evidence Scenario Changes the Inference

Move From Asking Whether Evidence Is Missing to Asking Whether It Matters

Missing evidence and consequential missing evidence are different questions

Almost any substantial evidence base may contain some unavailable research. Studies remain unpublished, outcomes go unreported, reports become difficult to retrieve, and data needed for synthesis may never appear.

The existence of missing evidence is therefore not enough to determine whether a conclusion is seriously biased.

Imagine a meta-analysis containing 40 large, reasonably consistent studies. Discovering one small missing study with a similar result may have negligible influence. Now imagine a meta-analysis containing four small studies with borderline effects and evidence that several completed studies remain unpublished. The same general statement, "some studies are missing," has very different implications.

Start with known missing evidence before imagining unknown studies

The strongest clues often come from studies you know exist. Registries, protocols, conference abstracts, regulatory records, dissertations, study reports, and publications themselves may reveal completed studies or measured outcomes whose results are unavailable.

Cochrane's current guidance recommends first identifying studies for which results are missing from the synthesis and considering whether that non-reporting may depend on the findings. Its ROB-ME tool separates concerns about missing results within known studies from concerns about entire studies that may be missing from the synthesis.

This distinction matters because known missing results provide more concrete evidence than hypothetical unseen studies. If a protocol shows that a relevant outcome was measured but the publication omits it, you know that a result is absent even if you do not know exactly what that result was.

Ask why the evidence is missing

Missingness becomes more concerning when it plausibly depends on the result.

A study lost because an old journal issue was not digitized presents a different bias mechanism from a study withheld because its result was disappointing. Likewise, an outcome omitted because an instrument malfunctioned differs from an outcome omitted because it failed to reach statistical significance.

ROB-ME focuses specifically on missing evidence that may arise because of the p-value, magnitude, or direction of study results. That result-dependent missingness is what can systematically move a synthesis away from the evidence that would have been observed if all eligible results were available.

The size and structure of the observed evidence matter

The potential influence of missing studies depends partly on how robust the observed evidence already is.

These are considerations rather than mechanical rules. A large number of studies does not guarantee immunity from publication bias, and a small evidence base is not automatically biased.

Think about how many studies could realistically be missing

The plausibility of an unseen evidence base depends partly on the type of research involved. Cochrane notes that some studies are harder to conceal than others. Large, expensive, publicly visible studies involving extensive infrastructure may leave more traces than small studies that can be conducted quickly and inexpensively.

Registries and other prospective records can also constrain what is plausible. If a field requires comprehensive registration and you can account for nearly every eligible registered study, a vast hidden literature becomes less plausible than in a field where studies are rarely registered and unpublished work leaves little public trace.

This does not prove completeness. It helps bound the scenarios you consider credible.

Then ask what the missing studies would need to look like

The question is not simply, "Could there be five missing studies?" You also need to consider their plausible sample sizes and results.

Five tiny studies with estimates similar to the observed literature may change little. Five large studies showing essentially no effect could matter considerably. A smaller number of studies pointing in the opposite direction might matter even more if the observed evidence base is itself small.

Thinking this way turns an abstract concern about publication bias into a robustness question: what combination of missing evidence would be necessary to change the inference, and is that combination credible given what you know about the field?

Define what you mean by "change the conclusion"

A conclusion can change in more than one way. Researchers sometimes focus exclusively on whether a p-value crosses 0.05, but that is only one possible consequence.

Missing evidence might:

• move the pooled effect toward or away from the null;
• reverse the estimated direction of an effect;
• substantially widen uncertainty;
• move an estimate across a clinically or practically meaningful threshold;
• change conclusions about harms or benefits;
• reduce confidence that the observed estimate represents the complete evidence base.

A synthesis can therefore remain conventionally statistically significant while becoming much less persuasive in practical terms. Conversely, crossing a significance threshold does not necessarily imply that the substantive conclusion has changed dramatically.

Funnel plots can provide clues, not verdicts

Funnel plots are often used to examine whether effect estimates from smaller studies are distributed differently from estimates from larger studies. Certain patterns of asymmetry can be compatible with selective non-publication.

But funnel-plot asymmetry has other possible causes, including genuine differences between smaller and larger studies, methodological differences, and sampling variation. An apparently symmetrical plot also cannot prove that no studies are missing.

Cochrane therefore recommends interpreting funnel-plot asymmetry alongside qualitative evidence about missing results and other sensitivity analyses rather than treating it as a diagnostic test that reveals publication bias by itself.

Statistical adjustments are assumption-dependent

Several statistical methods attempt to explore how missing evidence could affect meta-analysis. These approaches can be useful for sensitivity analysis, but none observes the missing studies directly.

The result therefore depends on assumptions about how studies became missing, how missing effects relate to observed effects, and other features of the evidence-generating process.

This is why the next question is not whether a statistical method can perfectly recover the hidden evidence. It is whether you can completely correct for an unknown body of unpublished research, which is a substantially stronger claim than sensitivity analysis can generally support.

ROB-ME provides a structured way to assess the problem

Cochrane incorporated the ROB-ME framework into the current Handbook for assessing risk of bias due to missing evidence in a meta-analysis. The tool uses signalling questions addressing both non-reporting within identified studies and the possibility that additional studies are missing from the synthesis.

The goal is not to calculate the number of invisible studies. It is to make a structured judgment about whether the synthesized result is at risk of bias because available evidence may differ systematically from missing evidence.

This is a useful conceptual shift. Publication bias is not merely a property of an entire field. The relevant risk can differ among outcomes and syntheses because the pattern of missing evidence may differ.

The history of the literature can strengthen or weaken the concern

Other reporting biases provide additional clues. If favorable studies are known to have appeared sooner through time-lag bias, if outcomes have been selectively reported, or if registered studies repeatedly lack published results, the case for concern becomes stronger.

Conversely, extensive prospective registration, accessible results reporting, successful retrieval of unpublished evidence, and consistency between protocols and publications can reduce some uncertainty about what is missing, although they cannot establish perfect completeness.

The judgment should therefore integrate what you know about the research system, not just the geometry of a statistical plot.

Would Plausible Missing Studies Actually Overturn the Result?

Common Mistakes When Judging the Importance of Missing Studies

Judge Robustness Under Plausible Missing-Evidence Scenarios

Do not stop after writing "publication bias may be present" in the limitations section. Ask what that possibility actually means for the conclusion you intend to draw.

Where appropriate, use structured tools such as ROB-ME and sensitivity analyses rather than relying on intuition alone. The purpose is not to manufacture certainty about evidence you cannot observe. It is to determine whether your inference survives reasonable challenges to the completeness of the evidence base.

A robust conclusion should not depend on pretending that missing evidence does not exist. It should remain defensible across the range of missing-evidence scenarios that the research context makes credible.

How to Judge Whether Missing Studies Could Matter

Frequently Asked Questions About Missing Studies

The Important Question Is Not Merely Whether Evidence Is Missing, but Whether Your Conclusion Depends on It

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.