01 · The Question
How Can Research Support “Little or No Meaningful Effect” Rather Than Merely Fail to Find One?
Researchers are often comfortable testing whether an effect exists. The harder question comes when the expected effect does not appear: when can the study actually support the conclusion that the effect is absent or too small to matter?
A large p -value is not enough. Neither is a point estimate close to zero. Both can arise from data that remain compatible with effects of considerable magnitude.
Meaningful evidence for absence requires a study capable of discriminating between effects that matter and effects that do not. That makes “How small is small enough?” part of the scientific question, not merely a statistical detail.
02 · The Short Answer
Evidence for Absence Requires Ruling Out Effects That Matter
In Brief
A study can provide meaningful evidence that an effect is absent or very small when its design, data, and analysis are sufficiently informative to rule out effect sizes that would be substantively important under a justified criterion.
In a frequentist framework, equivalence testing is one formal way to make this inference. More generally, the conclusion depends on defining what magnitude matters, obtaining sufficiently precise and credible evidence, and showing that effects beyond that range are not well supported by the data.
03 · What You Need to Know
What Turns a Null Finding Into Evidence About Effect Size?
Start by defining what “very small” means
Claims such as “no meaningful effect” are incomplete until meaningful has been defined. An effect that is negligible in one context may matter considerably in another.
Suppose a new educational intervention improves test scores by an average of 0.2 percentage points. That difference may be inconsequential if the intervention is expensive and difficult to implement. In a different setting involving a very large population, low implementation costs, or a cumulative outcome, even a small average effect might warrant attention.
The threshold therefore requires substantive justification. Depending on the field and research question, researchers may draw on theory, prior empirical evidence, measurement properties, clinical or educational decision thresholds, cost-benefit considerations, or other defensible criteria.
The smallest effect size of interest gives the claim a target
A useful concept is the smallest effect size of interest, often abbreviated SESOI. It represents a boundary between effects considered meaningful for the research question and effects considered too small to matter for the stated purpose.
For a two-sided question, researchers may specify lower and upper bounds around zero. For example, effects between -0.20 and 0.20 on a standardized scale might be treated as practically negligible if those limits have a defensible substantive basis.
The numerical values themselves are not universal. Choosing ±0.20 merely because of a generic effect-size convention can be difficult to defend when the scientific context suggests a different threshold. The bounds should correspond to the claim you actually want to make.
A conventional nonsignificant result does not test that claim
If a standard significance test produces p >.05, researchers have not thereby shown that the effect falls inside their negligible range. They have simply failed to reject the null hypothesis under that test.
This distinction explains why “no significant difference” does not establish that there is no meaningful difference . A study may be nonsignificant while remaining compatible with effects far beyond any reasonable threshold of practical importance.
Equivalence testing reverses the relevant inferential question
Equivalence testing offers a frequentist method specifically designed to ask whether effects large enough to matter can be rejected. In the commonly used two one-sided tests procedure, researchers define lower and upper equivalence bounds and conduct tests against those boundaries.
If both relevant null hypotheses at the equivalence boundaries can be rejected at the specified significance level, the result supports the conclusion that the effect lies within the equivalence region under the assumptions of the procedure.
The conclusion is therefore not “we proved that the effect equals zero.” It is closer to “the data provide evidence against effects at least as extreme as the prespecified bounds.”
Precision determines what you can rule out
Even an appropriately formulated question cannot be answered by highly imprecise data. Suppose the estimated standardized effect is 0.03 but its confidence interval stretches from -0.45 to 0.51. The point estimate looks tiny, yet many potentially important effects remain compatible with the data.
Now suppose the estimate is 0.02 and the relevant interval is tightly concentrated near zero. If that interval and an appropriate equivalence analysis exclude the prespecified meaningful bounds, the study provides much stronger evidence against effects of those magnitudes.
This is the practical difference between evidence suggesting a negligible effect and simply having too much uncertainty to know .
Study design must make a null result diagnostic
Precision is necessary but not sufficient. A study can estimate something very precisely and still answer the wrong question.
Measurement validity matters. So do treatment fidelity, missing data, selection processes, confounding in observational studies, model assumptions, outcome timing, and other threats relevant to the design. If an intervention was barely implemented, for example, a precisely estimated near-zero contrast may provide stronger evidence about that implementation than about the intervention as theoretically intended.
Evidence for absence therefore depends on the entire inferential chain, not merely on a small coefficient and narrow standard error.
Exactly zero is usually the wrong target
Empirical researchers often speak casually of proving “no effect,” but an exact population effect of zero is generally a much stronger target than the scientific decision requires.
In many settings, the relevant question is whether the effect is smaller than some threshold of theoretical or practical importance. Equivalence testing is designed around this idea. Other inferential frameworks can address related questions differently, but all require researchers to specify what evidence would count against the effect under consideration.
Exact absence
A claim that the true effect is exactly zero, which is generally not what an equivalence test establishes.
Practical or substantive absence
A claim that effects large enough to matter under a justified criterion have been ruled out or substantially disfavored.
One study does not make every version of the effect disappear
A precise null finding for one population, outcome, intervention dose, operationalization, and follow-up period does not automatically establish absence across all contexts.
The scope of the conclusion should match the scope of the evidence. A study may provide strong evidence that a particular intervention does not produce an educationally meaningful improvement on a specified outcome over one semester. That is narrower, and more defensible, than saying the intervention “does not work.”
04 · A Practical Example
Testing Whether a Teaching Intervention Has Any Meaningful Effect
Hypothetical Example
An intervention expected to improve achievement
Researchers compare a new teaching intervention with standard instruction. Before examining the results, they justify standardized effects between -0.20 and 0.20 as too small to alter the educational decision for which the study was designed.
Define the meaningful range
The equivalence bounds are set at -0.20 and 0.20. Effects beyond either boundary would be considered large enough to matter for the stated decision.
Estimate the effect
The study estimates a standardized mean difference of 0.03, with sufficiently narrow uncertainty to test the prespecified bounds.
Test the relevant question
Rather than interpreting a conventional nonsignificant test as evidence of no effect, the researchers perform an appropriate equivalence test against the -0.20 and 0.20 boundaries.
Interpret the result
If both equivalence tests reject effects at the relevant boundaries, the researchers can conclude that the data provide evidence against effects as large as the prespecified meaningful threshold, subject to the assumptions and validity of the study.
State the claim precisely
The conclusion should concern the absence of an effect large enough to meet the specified educational threshold, not proof that the population effect equals exactly zero.
The example also shows why the threshold must be justified. If an effect of 0.10 would actually matter for the educational decision, bounds of ±0.20 would be too permissive. The statistical procedure cannot rescue a poorly chosen scientific criterion.
06 · What This Means for You
Design the Study to Learn From a Small or Null Result
If absence or practical equivalence would be scientifically informative, plan for that possibility before collecting the data. Decide what magnitude would still matter, justify that threshold, and design the study to estimate effects around that boundary with useful precision.
This changes the logic of study planning. You are no longer designing only for the possibility of detecting an effect. You are also asking whether the study can teach you something if the observed effect is small.
A simple decision framework
If your scientific question concerns whether any nonzero effect exists
Be explicit about the inferential framework and recognize that practical importance may not coincide with exact zero.
If you want to show that meaningful effects are absent
Define and justify the smallest effect size of interest and use an analysis capable of evaluating effects relative to that boundary.
If the resulting estimate remains too imprecise to exclude important effects
Treat the study as inconclusive rather than evidence of absence.
If the data precisely exclude the justified meaningful range and the design is credible
A conclusion that effects of that magnitude are absent or unlikely may be warranted within the scope of the study.
The distinction also affects how you interpret prior research. A literature containing many nonsignificant results may look consistently null while remaining remarkably uninformative if the underlying studies are imprecise. The right synthesis therefore asks what effect sizes the evidence collectively constrains, rather than simply counting how many studies were statistically significant .
Watch Out
Do not define “small enough not to matter” only after seeing the observed effect. A threshold chosen to accommodate the result can make the reasoning circular: the data look negligible because the criterion was constructed around the data.
07 · A Quick Checklist
Before Claiming That an Effect Is Absent or Very Small
Check whether the evidence supports the claim:
Define what magnitude of effect would be meaningful for the actual research question.
Justify the smallest effect size of interest using substantive evidence or reasoning rather than an arbitrary convenient threshold.
When feasible, specify the meaningful-effect threshold before examining the results.
Design the study to provide useful precision around the effect sizes you need to distinguish.
Report the effect estimate and its uncertainty rather than relying only on statistical significance.
Use equivalence testing or another inferential approach appropriate to the claim of interest.
Check whether important effects remain compatible with the evidence.
Assess whether measurement, bias, implementation, missing data, and model assumptions threaten the inference.
Limit the conclusion to the population, intervention, outcome, timing, and conditions actually studied.
09 · The Bottom Line
Evidence for Absence Is Evidence Against Effects That Matter
The Bottom Line
A study provides meaningful evidence that an effect is absent or very small when it can credibly rule out effect sizes that would matter under a justified criterion, not merely when a conventional significance test fails to reject zero.
Define what “meaningful” means before making the claim, obtain enough precision to discriminate negligible from important effects, and use an inferential method suited to that question. Even then, describe what magnitude has been ruled out and under what conditions rather than converting evidence for practical absence into a universal claim that absolutely nothing happens.
11 · Cite this Guide
How to Cite This Guide
This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.
Recommended (Field Guide)
APA
MLA
Chicago
Copy Citation
