When Does “No Significant Difference” Fail to Show That There Is No Difference?

What Has a Study Actually Shown When the Difference Is Not Significant?

You compare two groups and obtain p =.18. The result is not statistically significant. Can you now write that there was “no difference” between the groups?

Usually, not on that basis alone. A nonsignificant result may occur because the true difference is negligible. It can also occur because the data are too imprecise to distinguish a meaningful difference from random variation.

The distinction matters because “we did not establish a difference” and “we established that the groups do not meaningfully differ” are different claims. Conventional significance testing answers the first question much more directly than the second.

A Nonsignificant Test Does Not Establish Equality

Why Statistical Nonsignificance and No Difference Are Not Equivalent

Conventional significance testing is asymmetric

In a conventional null-hypothesis significance test, researchers typically begin with a null hypothesis such as a population difference of zero. They then ask whether the observed data would be sufficiently unusual under that hypothesis to reject it at a chosen significance level.

If p falls below the threshold, researchers reject the null hypothesis according to that testing procedure. If p does not, they fail to reject it.

Those outcomes are deliberately asymmetric. Failing to reject a hypothesis is not the same inferential act as demonstrating that the hypothesis is correct. As Altman and Bland pointed out in their classic discussion of nonsignificant findings, a study that does not find a statistically significant difference has often shown only an absence of evidence for a difference rather than evidence that the difference is absent.

A p-value does not tell you how large the difference might still be

Suppose two studies both report p =.12. The identical significance classification does not mean the studies contain the same information.

One might estimate a mean difference of 4 points with a 95% confidence interval from -2 to 10. Another might estimate a difference of 0.3 points with an interval from -0.5 to 1.1. Both results can be nonsignificant, yet the first remains compatible with a fairly substantial positive difference while the second is much more tightly concentrated around small differences.

The statement “not significant” suppresses this distinction. It tells you about a threshold decision under a particular hypothesis test, not the complete range of effect sizes still compatible with the evidence.

Wide uncertainty can make “no difference” indefensible

Confidence intervals help reveal what a binary significance label hides. If an interval includes zero but also contains effects that would matter scientifically or practically, the data have not resolved whether an important difference exists.

This is particularly common in small or noisy studies. A study can fail to detect a real difference because its estimate is imprecise.

In that situation, the appropriate interpretation is uncertainty. The study has not established the difference, but neither has it provided strong evidence that the difference is negligible.

Statistical power affects how surprising a null result should be

A study with little ability to detect effects of the size researchers care about can readily produce a nonsignificant result even when such an effect exists. Consequently, a null finding from an underpowered study may carry little information about whether the underlying effect is negligible.

This is why simply accumulating many underpowered studies with nonsignificant results does not automatically establish that nothing happens.

Power calculations are particularly useful during study planning because they force researchers to consider what effect sizes a design is intended to detect. After data have been observed, interpretation should not stop at a retrospective declaration that a study was “powered” or “underpowered.” The observed effect estimate, its uncertainty, and the design assumptions provide more direct information about what the study has and has not ruled out.

Similarity requires defining what counts as similar

Two populations will rarely be exactly identical on a continuously measured outcome. With sufficiently large samples, even tiny differences may become statistically detectable. Conversely, a small study may fail to detect a substantial difference.

The scientifically useful question is therefore often not “Is the true difference exactly zero?” but “Is the difference small enough to be considered negligible for this purpose?”

Answering that requires a substantive threshold. In equivalence testing, researchers specify lower and upper equivalence bounds representing differences considered too small to matter. The two one-sided tests procedure, commonly abbreviated TOST, then tests whether effects at or beyond those bounds can be rejected.

This directly addresses a different question from a conventional significance test and can provide meaningful evidence that an effect is absent or sufficiently small.

Four outcomes are possible when significance and equivalence are considered together

Once researchers distinguish testing for a difference from testing whether meaningful differences can be excluded, a useful feature becomes visible: “significant” and “equivalent” are not simple opposites.

The last situation is especially important. Researchers sometimes interpret it as evidence for “no difference,” when the study may simply be unable to discriminate between a negligible effect and an important one.

Two Studies Can Both Be Nonsignificant but Support Different Conclusions

Reporting both studies simply as “there was no significant difference” makes them sound almost identical. They are not. The first leaves considerable uncertainty about educationally important differences. The second provides substantially tighter information about their magnitude.

Common Mistakes When Interpreting Nonsignificant Differences

Match Your Conclusion to the Question Your Analysis Actually Answered

If your ordinary significance test is nonsignificant, resist replacing the statistical statement with the stronger substantive conclusion “there is no difference.” First determine how much uncertainty remains.

Ask what differences would matter in the context of your research. Then compare that range with the effects still compatible with your data. This helps distinguish a result suggesting little meaningful effect from one that remains too uncertain to know.

The language should reflect the evidence. “We found no statistically significant difference” may be accurate when that is literally the result. “The groups did not differ” is a stronger claim. “The results were compatible only with differences smaller than our prespecified threshold of practical importance” is stronger still, but it can be justified when the analysis genuinely supports it.

Before Writing “There Was No Difference”

Questions About “No Significant Difference”

Nonsignificance Does Not Establish No Difference

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.