How Do You Recognize Selective Reporting of Significant Analyses?

How Can You Tell Whether You Are Seeing Only the Favorable Analyses?

A paper presents a clean statistical story: several analyses, a handful of significant findings, and a Discussion built around those results. But what happened to the other outcomes, time points, subgroups, models, and analyses that researchers could have examined?

You usually cannot diagnose selective reporting simply by looking at one suspicious P-value. The problem concerns the relationship between the analyses that could or should have been reported and those that ultimately became visible.

Recognizing it therefore requires a form of statistical detective work. Compare the published paper with records created before or independently of the final results, and look for unexplained changes, missing analyses, incomplete numerical reporting, or a pattern in which favorable findings receive fuller treatment than unfavorable ones.

Look for Missing or Changed Analyses, Not Merely Suspicious P-Values

Selective Reporting Changes Which Evidence the Reader Gets to See

What selective reporting actually means

Selective reporting occurs when decisions about whether, how, or how completely to report a study result are influenced by the result itself, including its P-value, magnitude, or direction.

Cochrane describes this within the broader problem of non-reporting bias. A result may be completely missing, or it may be only partially reported. For example, authors might state that an analysis was "not significant" without providing the effect estimate or uncertainty needed for independent interpretation.

The important feature is selection. A paper does not merely contain fewer results than could theoretically have been calculated. The concern is that the visibility or completeness of results depends on what those analyses found.

Selective reporting is broader than selective outcome reporting

An outcome can be selected, but so can many other analytical choices.

This matters because readers may see a seemingly straightforward final analysis even though many analytical paths existed behind it.

The best evidence often comes from comparing documents

You cannot reliably identify selective reporting from the publication alone because the missing evidence is, rather inconveniently, missing. The strongest appraisal therefore compares multiple sources.

Cochrane recommends assembling available study materials such as registry records, protocols, statistical analysis plans, journal reports, regulatory reports, and information obtained from investigators or sponsors when assessing selective non-reporting.

CONSORT 2025 similarly expects primary and secondary outcomes in the publication to be consistent with those prespecified in the protocol and registry, and recommends reporting changes with reasons.

The key question is simple: what did the researchers say they would analyze before the results were known, and what did they eventually report?

Look for outcomes that disappear

Suppose a registered trial lists five outcomes. The publication reports four, and the missing outcome is not mentioned. That discrepancy deserves investigation.

There may be an innocent explanation. Measurement could have failed, a validated instrument might have become unavailable, or the outcome may have been dropped for a documented methodological reason. What matters is whether the change is disclosed and justified.

Concern rises when planned outcomes repeatedly disappear without explanation and available evidence suggests that unfavorable or nonsignificant results are less likely to be fully reported.

Look for primary and secondary outcomes that switch places

Outcome switching occurs when the status of outcomes changes between planning and publication. A registered primary outcome might become secondary, while another outcome becomes the headline result.

Such a change is not automatically improper. Scientific or practical circumstances can require protocol amendments. But timing and transparency matter. Was the change made before researchers had access to relevant outcome data? Was it documented? Was a reason provided?

An unexplained switch that elevates a favorable outcome after results are known deserves substantially more concern than a prospectively documented amendment.

Time points can be selected too

An intervention may be assessed immediately after treatment, three months later, six months later, and at one year. If only one time point produces a favorable result, emphasizing that occasion without showing the others can distort the apparent evidence.

Check whether the time point highlighted in the article matches the prespecified primary time point. CONSORT's outcome definition explicitly includes the measurement time point because changing when an outcome is analyzed can change the result being tested.

Watch for incomplete reporting of nonsignificant analyses

Selective reporting does not require an outcome to vanish completely. Sometimes it becomes statistically invisible.

You may encounter statements such as "there were no differences in the remaining outcomes" or "all other comparisons were nonsignificant" without estimates, confidence intervals, exact P-values, or even enough summary data to understand the findings.

Cochrane treats this kind of under-reporting as potentially important because readers may lack the information necessary to estimate an effect or incorporate it into evidence synthesis.

This is particularly problematic because a nonsignificant result may still contain substantial information. Hiding its magnitude and uncertainty behind the phrase "not significant" prevents readers from seeing that information.

Look for unexplained changes in the statistical model

Selective reporting can operate within an outcome. Researchers may have several plausible ways to define variables, handle missing data, select covariates, transform measurements, identify outliers, or specify models.

If the statistical analysis plan prespecifies one approach but the publication uses another, ask whether the change is acknowledged and justified. CONSORT 2025 recommends identifying deviations from the statistical analysis plan and distinguishing prespecified from post hoc analyses.

This does not mean that every deviation is evidence of bias. Sometimes the planned model proves inappropriate after legitimate diagnostic work. Transparency allows readers to judge that explanation.

Subgroup findings deserve particular scrutiny

Subgroups offer many analytical possibilities. Researchers can divide participants by age, sex, baseline risk, severity, institution, treatment adherence, biomarker level, or numerous other characteristics.

CONSORT notes empirical evidence of selective reporting and inadequate statistical support for subgroup claims and recommends identifying which subgroup analyses were prespecified.

If an unexpected subgroup becomes central to the paper while several planned subgroups disappear, investigate further. This is especially important when the highlighted subgroup was not prespecified.

Multiple testing and selective reporting reinforce each other

Multiplicity creates opportunities to obtain unusual findings. Selective reporting determines which of those findings the reader sees.

Suppose researchers perform 50 analyses but report all 50 transparently. The paper has a multiplicity problem that can be evaluated directly. If they instead report only the three significant analyses, the reader may not even realize that 47 alternatives existed.

Therefore, when a study involves many statistical tests, reporting transparency becomes especially important.

A cluster of P-values just below 0.05 is not proof of selective reporting

Patterns in P-values can sometimes motivate further scrutiny, but they rarely establish what happened in an individual study. A P-value of 0.047 is not evidence by itself that researchers manipulated or selectively reported an analysis.

Stronger evidence comes from documented discrepancies: a prespecified outcome that disappeared, an unacknowledged model change, missing time points, incomplete reporting, or differences between the analysis plan and publication.

Critical appraisal should distinguish suspicious patterns from demonstrated reporting discrepancies.

Not every unreported analysis should have been reported

A dataset permits an effectively enormous number of possible analyses. Researchers are not obligated to publish every conceivable model they could have run.

The more relevant questions concern analyses promised in advance, analyses needed to interpret the study fairly, and analyses whose omission could change the impression created by the reported evidence.

Prespecification is useful precisely because it provides a benchmark. Without it, distinguishing legitimate analytical development from outcome-driven selection becomes harder.

How Selective Reporting Can Change the Story of a Study

Common Mistakes When Looking for Selective Reporting

Audit the Study's Reporting Trail, Not Just Its Results Table

If a finding matters to your review, decision, or argument, do not assume the journal article contains the complete analytical history. Look for records that establish what researchers intended to measure and analyze.

Your conclusion should match the evidence you actually have. "The prespecified primary outcome is missing from the publication" is a documented observation. "The authors hid it because it was nonsignificant" is a causal claim that requires additional evidence.

How to Check a Paper for Selective Reporting

Questions About Selective Reporting of Statistical Results

To See Selective Reporting, Look for What Is Missing or Changed

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.