How Can Selective Outcome Reporting Distort the Literature?

What If a Study Measured More Than the Paper Reports?

A published study may look complete. It has methods, results, tables, figures, and a clear conclusion. Yet the researchers may have measured outcomes that never appear in the results, or that appear only as a vague statement such as "no significant difference was found."

That matters because a paper is not necessarily a complete record of everything measured in the study. If outcomes are more likely to be fully reported when their results are statistically significant, favorable, or otherwise interesting, readers encounter a selected version of the findings.

This is selective outcome reporting, and it can distort both the interpretation of an individual paper and the larger literature built from such papers.

Selective Outcome Reporting Can Hide Part of a Study's Findings

How Outcome-Level Selection Changes the Evidence You See

The study can be published while some of its results remain missing

Publication bias and selective outcome reporting operate at different levels. With publication bias, an entire study or report may be unavailable because of its findings. With selective outcome reporting, the study itself is available, but results for some measured outcomes are absent or insufficiently reported.

This distinction is important because finding the paper does not guarantee that you have found all of its relevant results.

An outcome can disappear without disappearing completely

Selective outcome reporting is not limited to completely omitted outcomes. Cochrane distinguishes selective non-reporting, in which a result is absent, from selective under-reporting, in which some information appears but not enough to use the result properly in a synthesis.

For example, a paper might state that an outcome showed "no significant difference" without providing group estimates, an effect estimate, or sufficient information about precision. The reader knows the outcome existed, but the result may be unusable in a meta-analysis.

The bias comes from why outcomes are selected

Not every omitted outcome constitutes selective outcome reporting bias. There can be legitimate reasons why a planned outcome is unavailable, unusable, or omitted from a particular report. An instrument may fail, data may be corrupted, an outcome may prove impossible to measure, or a justified protocol amendment may change the study before results are known.

The concern is stronger when reporting depends on the result itself. If researchers are more likely to report an outcome because its effect is larger, statistically significant, favorable to the hypothesis, or otherwise attractive, the available results become a selected subset of those generated.

This distinction also means that discovering a discrepancy between a protocol and a paper is evidence that deserves investigation, not automatic proof of misconduct. Timing and explanation matter.

There is empirical evidence that outcome reporting can depend on statistical significance

A frequently cited study by Chan and colleagues compared protocols with publications from 102 randomized trials containing 3,736 outcomes. The researchers found that statistically significant outcomes had higher odds of being fully reported than nonsignificant outcomes. The pooled odds ratio was 2.4 for efficacy outcomes and 4.7 for harm outcomes.

The same study found that 50% of efficacy outcomes and 65% of harm outcomes per trial were incompletely reported. Among the trials with specified primary outcomes, discrepancies were also common: 62% had at least one primary outcome that had been changed, introduced, or omitted when protocols were compared with publications.

Those figures describe one historical cohort of clinical trials, not a universal rate that should be applied to every field. Their importance lies in demonstrating empirically that what gets reported can be associated with what the outcome showed.

Selective reporting can change the apparent balance of benefits and harms

Suppose a study measures academic performance, engagement, attendance, anxiety, workload, and dropout. If the intervention improves engagement but shows little difference on the remaining outcomes, reporting engagement prominently while omitting or inadequately reporting the others creates a different impression from presenting all six outcomes transparently.

The reported engagement effect might be entirely genuine. The distortion arises because readers lack the wider outcome context needed to understand what the intervention did and did not appear to change.

This can be particularly consequential for harms. If benefits are reported comprehensively while unfavorable outcomes are incompletely reported, the apparent benefit-harm profile may differ from the complete findings.

Outcome selection can occur through measurements and time points

An outcome domain can often be measured in several ways. Depression might be assessed with multiple scales. Achievement might be measured immediately after an intervention and again months later. Quality of life may contain several subscales.

If researchers select the measurement, scale, definition, or time point that produces the most favorable result after seeing the data, the problem overlaps with bias in selection of the reported result. Cochrane's current risk-of-bias framework explicitly considers whether a reported result may have been selected from multiple eligible outcome measurements on the basis of the findings.

This is different from choosing among several statistical procedures applied to the same outcome. That neighboring problem is selective analysis reporting.

Missing outcomes can propagate into systematic reviews

A systematic reviewer can only synthesize results that can be identified and extracted. If unfavorable or nonsignificant outcomes are systematically unavailable, excluding them does not merely reduce the sample size. It may leave the synthesis disproportionately populated by a different kind of result.

Cochrane therefore treats selective non-reporting and under-reporting as potential sources of bias due to missing results. The issue is closely related to publication bias: in one case an entire study may be missing, while in the other a result within an available study is missing.

Outcome switching and outcome non-reporting overlap but are not identical

A related issue arises when outcomes designated in advance as primary are later omitted, replaced, demoted, or supplemented by outcomes that receive primary emphasis. This can contribute to selective reporting, particularly when the change follows inspection of the results.

However, not every outcome change is necessarily biased. Understanding how outcome switching affects interpretation requires asking when the change occurred, why it occurred, and whether it was transparently documented.

How One Study Can Produce a Selectively Positive Story

Common Misunderstandings About Selective Outcome Reporting

Look Beyond the Results Section When Outcome Selection Matters

When you critically appraise a study, do not ask only whether the reported outcomes were analyzed correctly. Ask whether the reported outcomes plausibly represent the outcomes that were planned and measured.

For your own studies, the preventive principle is straightforward: specify important outcomes clearly before examining the results, preserve dated documentation of changes, report prespecified outcomes regardless of whether they produce an exciting result, and explain legitimate deviations transparently.

How to Check for Selective Outcome Reporting

Frequently Asked Questions About Selective Outcome Reporting

A Published Study May Contain Only Part of the Outcome Story

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.