01 · The Question
What Happens When the Evidence Entering a Meta-Analysis Is Selectively Visible?
A meta-analysis can be statistically impeccable and still face a problem that no pooling formula can solve: what if the studies or results available for analysis are not representative of all the evidence that actually exists?
Studies with favorable or statistically significant findings may sometimes be more likely to become available, while unfavorable, null, or otherwise less attractive results remain unpublished or incompletely reported. If that happens systematically, the meta-analysis may summarize a distorted evidence base with considerable numerical precision.
03 · What You Need to Know
Why Missing Evidence Can Change a Meta-Analysis
Publication Bias Is Part of a Broader Missing-Evidence Problem
"Publication bias" is often used as shorthand for a wider family of problems. Cochrane uses the broader concept of non-reporting bias for situations in which decisions about how, when, or where study results are reported are influenced by their P value, magnitude, or direction.
An entire study may remain unpublished. A published study may omit a particular outcome. Researchers may report one analysis but not another, or disseminate favorable findings more rapidly or prominently than unfavorable ones.
These mechanisms differ, but they share the same concern for meta-analysis: the evidence available for synthesis may systematically differ from the evidence that is missing.
Study-level non-reporting
An entire eligible study or report is unavailable or difficult to identify in ways related to its findings.
Result-level non-reporting
A study is available, but particular outcomes, time points, analyses, or results are selectively unavailable.
The Problem Is Not Simply That Some Studies Are Missing
Missing evidence becomes especially concerning when missingness is related to the findings. If studies were absent completely at random, losing some would mainly reduce the amount of information and precision.
Publication bias is more dangerous because the missing evidence may point systematically in a different direction from the visible evidence. Cochrane notes convincing evidence for several forms of non-reporting bias and emphasizes that available evidence can differ systematically from unavailable evidence.
A Meta-Analysis Cannot Pool Results It Never Sees
Meta-analysis works on the effect estimates supplied to it. Statistical sophistication cannot reconstruct an unknown collection of studies and outcomes with certainty.
If studies reporting large favorable effects are disproportionately visible while studies showing little effect remain unavailable, the pooled estimate may exaggerate benefit. Similar problems can occur for harms if unfavorable safety results are selectively absent.
This illustrates why a precise pooled estimate does not necessarily constitute strong evidence . Precision describes uncertainty around the evidence being analyzed. It does not guarantee that the evidence base itself is complete or unbiased.
A Comprehensive Search Helps, but Cannot Guarantee That Missing Results Exist to Be Found
Reviewers can reduce the problem by searching broadly. Depending on the question, this may involve multiple bibliographic databases, study or trial registers, regulatory sources, conference material, study authors, sponsors, or other sources.
Cochrane specifically emphasizes searching all plausible locations where study reports and results may be found because non-reporting biases can compromise the goal of identifying all eligible research.
However, even a well-designed comprehensive search cannot retrieve a result that has never been made accessible anywhere. Searching and assessing missing evidence are therefore related but distinct tasks.
Protocols and Registrations Can Reveal Missing Outcomes
When study protocols, registrations, or prespecified analysis plans are available, reviewers can compare what investigators planned to measure and analyze with what was ultimately reported.
If a registered study lists several outcomes but the publication reports only those with favorable findings, the concern becomes more concrete than a general suspicion of publication bias. Result-level missingness can sometimes be investigated more directly because reviewers know that the study exists and may know which results should have been generated. Cochrane notes that the impact of selective non-reporting within identified studies may therefore be easier to quantify than the impact of an unknown number of completely unpublished studies.
Funnel Plots Can Raise Suspicion, Not Diagnose Publication Bias
A funnel plot displays study effect estimates against a measure of study size or precision. In a simple well-behaved setting, smaller studies scatter more widely while larger studies cluster more tightly, producing an approximate inverted funnel.
If smaller studies with unfavorable or statistically non-significant findings are missing, the plot may become asymmetric. This can raise concern about missing evidence.
But funnel-plot asymmetry has several possible causes. Smaller studies may genuinely involve different populations or interventions. They may have different methodological biases. Heterogeneity or chance can also produce asymmetry. Cochrane therefore explicitly warns that funnel-plot asymmetry is not diagnostic of non-reporting bias.
Watch Out
An asymmetric funnel plot does not prove publication bias, and a symmetric funnel plot does not prove its absence. Treat the plot as one piece of evidence about possible small-study effects, not as a publication-bias detector.
Tests for Funnel-Plot Asymmetry Have Limited Power
Statistical tests such as tests for funnel-plot asymmetry can supplement visual inspection, but they have important limitations. In particular, they tend to have low power when few studies are available.
Cochrane reports a rule of thumb that such tests should generally be used only when at least ten studies contribute to the meta-analysis. Even then, evidence of asymmetry requires investigation because non-reporting bias is only one possible explanation.
This creates an uncomfortable reality: publication bias may be most consequential in small evidence bases precisely when common statistical methods for detecting it are least informative.
Small-Study Effects Are Not Synonymous With Publication Bias
If smaller studies systematically report larger effects than larger studies, researchers refer to a pattern of small-study effects. Selective publication is one possible explanation, but it is not the only one.
Smaller studies may use different populations, interventions, methods, or levels of methodological rigor. Cochrane therefore recommends investigating alternative explanations when funnel-plot asymmetry or other small-study effects appear.
This matters because incorrectly diagnosing publication bias can be almost as unhelpful as ignoring it.
Random Effects Can Sometimes Amplify the Influence of Small-Study Effects
When heterogeneity is present, random-effects meta-analysis gives relatively more weight to smaller studies than a fixed-effect analysis does. If smaller studies systematically report larger effects because of selective dissemination or within-study bias, the random-effects pooled estimate can shift toward those larger small-study effects.
That does not mean random-effects models are inherently problematic. It means that model choice does not rescue an evidence base affected by selective availability.
Publication Bias Can Reduce Certainty of Evidence
GRADE explicitly includes publication bias among the domains that can reduce certainty in a body of evidence. Cochrane notes that certainty may be downgraded when studies are not reported because of their results or when outcomes are selectively unavailable.
The appropriate response is therefore not merely to mention publication bias in the limitations section. When the concern is credible and consequential, it should change how confidently the pooled estimate is interpreted.
06 · What This Means for You
Look for Evidence About What Might Be Missing
When reading a meta-analysis, do not ask only whether publication bias was "tested." Examine the broader evidence about selective availability: search methods, registrations, protocols, unpublished sources, missing outcomes, study-size patterns, funnel plots where appropriate, and sensitivity analyses.
A simple appraisal framework
If the search relies almost entirely on published journal articles
Ask whether registers, regulatory sources, grey literature, investigators, or other sources could contain relevant missing evidence.
If protocols or registrations identify outcomes that disappear from published reports
Treat selective result non-reporting as a direct concern for syntheses of those outcomes.
If a funnel plot is asymmetric
Consider non-reporting bias among several possible explanations rather than treating the pattern as diagnostic.
If there are fewer than about ten studies
Be particularly cautious about relying on formal funnel-plot asymmetry tests because their power is generally low.
If smaller studies consistently show larger effects
Investigate publication bias, methodological differences, heterogeneity, and other possible sources of small-study effects.
If credible missing evidence could materially change the pooled result
Reduce confidence in the conclusion rather than treating the observed meta-analysis as a complete representation of the evidence.
If concerns become serious enough that the synthesis may no longer represent the underlying studies fairly, it may be necessary to return to registrations, reports, and primary studies rather than relying only on the review's pooled result .
07 · A Quick Checklist
How to Check a Meta-Analysis for Possible Publication Bias
Before assuming the available evidence is complete, check:
Did the review search beyond the most obvious published journal literature where appropriate?
Were study registers, protocols, regulatory sources, conference records, or other relevant sources considered?
Do identified studies have outcomes or analyses that were planned but not reported?
Are favorable or statistically significant findings disproportionately represented in ways that raise concern about selective availability?
If a funnel plot was used, were alternative explanations for asymmetry considered?
If an asymmetry test was used, were enough studies available for the test to be informative?
Do smaller studies systematically report different effects from larger studies?
Were sensitivity analyses used appropriately to examine how missing evidence or small-study effects might affect conclusions?
Does the review's certainty assessment reflect credible concerns about publication or other non-reporting biases?
11 · Cite this Guide
How to Cite This Guide
This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.
Recommended (Field Guide)
APA
MLA
Chicago
Copy Citation
