How Can Publication Bias Change What the Literature Appears to Say?

Can the Published Literature Give You the Wrong Impression?

You search the literature carefully. You find twelve studies addressing your question. Ten report evidence consistent with an effect, while two do not. At first glance, the literature seems fairly convincing.

But there is a problem you cannot see in your search results. What if another eight studies were conducted and most of them found little evidence of an effect, but those studies were never published?

Your search might be excellent and your reading accurate, yet your conclusion could still be distorted because the literature available for you to find is not a neutral sample of all the research that was actually conducted.

This is the central problem of publication bias.

Publication Bias Changes the Evidence You Get to See

How Publication Bias Changes the Apparent Evidence Base

Publication bias is about which studies become visible

Publication bias occurs when the likelihood that research is published is influenced by its results. The World Health Organization describes publication bias in clinical trials as a situation in which the likelihood of publication is influenced by the direction or strength of the trial results.

The basic problem extends beyond any particular discipline. Research may be conducted, analyzed, and completed without becoming part of the literature that researchers routinely search. If the probability of publication is unrelated to the findings, missing studies may reduce the amount of evidence available without necessarily pushing the literature systematically in one direction. The greater concern arises when missingness is related to what the studies found.

Why positive and statistically significant findings matter

Publication is not simply the final mechanical step after a study is completed. Investigators decide whether and when to prepare manuscripts. Sponsors may influence dissemination. Editors and reviewers make publication decisions. Studies may also appear in repositories, registries, reports, dissertations, conference proceedings, or other forms rather than conventional journal articles.

Across this process, results can influence what eventually becomes visible. Cochrane notes that statistically significant results suggesting that an intervention works may be more likely to become available, become available rapidly, appear in high-impact journals, and be cited. Consequently, a conventional database search may encounter a systematically filtered version of the underlying research.

This is why a literature review is not necessarily a census of all research that has been done. It is a review of the evidence you were able to identify under a particular search and retrieval strategy.

The literature can look more positive than the underlying research

Suppose researchers conduct many independent studies of an intervention whose true effect is small or nonexistent. Sampling variation alone means that the estimated effects will differ from study to study. Some may produce conventionally statistically significant results even when others do not.

If statistically significant positive findings are disproportionately published, the visible literature will contain more of those studies. A reader encountering only the published subset could therefore infer a stronger or more consistent effect than would be apparent if all completed studies were available.

A well-known empirical illustration comes from Turner and colleagues' comparison of published antidepressant trials with U.S. Food and Drug Administration data. Among 74 FDA-registered studies involving 12,564 participants, 31% were not published. The journal literature made 94% of the published trials appear positive, whereas the FDA's assessments classified 51% of all registered trials as positive. The authors also found that the effect size based on the published literature was larger than that based on the FDA data. This is a particularly clear demonstration of how the visible literature and a more complete evidence base can tell noticeably different stories.

Publication bias can affect more than the estimated effect size

The distortion is sometimes described simply as an overestimation of effects, but that description is too narrow. Publication bias can change several features of what the literature appears to show.

Publication bias is not the same as selective reporting within a published study

It helps to separate several related mechanisms. Publication bias concerns whether an entire study becomes publicly available in a form that can be found. A study can also be published while only some of its measured outcomes are reported. That problem is selective outcome reporting.

Researchers may also try multiple statistical specifications or analytical approaches and report only a subset. That creates a related but distinct problem involving selective analysis reporting.

These mechanisms can coexist. A study might be more likely to be published because of an appealing result, and its published paper might simultaneously emphasize selected outcomes or analyses. Thinking of them as separate layers helps you diagnose where distortion may have entered the evidence base.

Publication is not simply published versus unpublished

Visibility has degrees. A completed study might appear as a journal article, registry result, dissertation, institutional report, conference abstract, preprint, regulatory document, or other research output. Some forms are easier to discover, evaluate, and cite than others.

Other biases can therefore interact with publication bias. Positive evidence may become available sooner through time-lag bias. Evidence published in languages excluded by a review can contribute to language bias. Studies may also be harder to discover because of database indexing bias.

The practical issue is therefore not merely whether a paper exists somewhere. It is whether the relevant research enters the discoverable evidence base in a form that your review process can realistically identify.

Why systematic reviews and meta-analyses are vulnerable

Systematic searching reduces many forms of selection, but it cannot automatically recover studies whose existence is unknown. Cochrane therefore treats bias due to missing evidence as a distinct concern in evidence synthesis.

A meta-analysis is especially sensitive because its pooled estimate is calculated from the studies that contribute data. If studies are missing for reasons related to their findings, the observed set is not simply smaller. It may be systematically different from the full set that would ideally have been analyzed.

Publication bias is often difficult to prove from published papers alone

The defining evidence is partly absent by definition. You usually cannot inspect an unpublished study that you do not know exists.

Researchers therefore look for additional evidence: prospective study registrations, protocols, regulatory records, dissertations, conference records, funder databases, study reports, and other sources that can reveal research not represented by journal publications. WHO emphasizes prospective registration and public results disclosure partly because these practices make completed and ongoing trials more visible and help reduce reporting biases.

Statistical methods can also investigate patterns that may be compatible with missing results, particularly in meta-analysis. However, such methods generally require assumptions and should not be treated as machines that reveal exactly how many unpublished studies exist or what they found. The harder question is whether missing studies could plausibly change the conclusion, not merely whether a diagnostic produces a particular threshold or visual pattern.

How the Same Research Question Can Produce Two Different Stories

This distinction matters. Publication bias is fundamentally a selection problem at the level of the evidence base. You can accurately summarize every paper you find and still mischaracterize the underlying research if the probability of finding a study depends on what it found.

Common Mistakes When Thinking About Publication Bias

How Publication Bias Should Change the Way You Read a Literature

The practical lesson is not to distrust every published finding. It is to avoid treating the set of papers you can see as automatically equivalent to the complete set of evidence that exists or once existed.

When the distinction could materially affect your conclusion, ask how likely it is that relevant research would be visible under your search strategy. This question is particularly important when publication incentives strongly favor novel, statistically significant, favorable, or otherwise striking results.

For your own research, prospective registration where appropriate and timely public reporting of results can help reduce the same problem for future reviewers. In clinical research, WHO explicitly supports prospective trial registration and public disclosure of results so that healthcare decisions can be informed by a more complete evidence base.

There is also a limit to what retrospective methods can accomplish. Once an unknown body of research has disappeared from the observable record, no statistical procedure can recreate its contents without assumptions. That is why attempts to correct for unpublished research should be interpreted as sensitivity analyses or model-based adjustments rather than perfect reconstruction of the missing evidence.

What to Check When Publication Bias Could Matter

Frequently Asked Questions About Publication Bias

The Literature You Can Find Is Not Necessarily All the Evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.