03 · What You Need to Know
How to Choose an Effect Size for Replication Sample-Size Planning
Why the Original Effect Size Is an Attractive Starting Point
A replication study investigates a previously reported finding, so the original effect estimate is naturally relevant to planning. It provides information about the observed magnitude of the relationship or intervention effect under the original study conditions.
For example, a randomized experiment may report a standardized mean difference of 0.50 between an intervention and comparison group. A researcher can use this estimate to calculate the sample size required to detect an effect of that magnitude under a specified statistical test.
The difficulty is that an observed effect size is an estimate, not a known population parameter. It contains sampling uncertainty and may be affected by bias, measurement limitations, and analytical choices.
Replication planning should therefore distinguish the published estimate from the effect magnitude that is reasonable to anticipate in a new study.
Why Published Effect Sizes May Be Too Large
Small studies can produce highly variable effect estimates. When attention or publication depends partly on whether results achieve statistical significance, unusually large estimates may be more likely to appear among the findings selected for further investigation.
This phenomenon is often described as the winner's curse. Conditional on selection for a favorable or statistically significant result, the reported effect may overestimate the underlying effect.
Ioannidis (2008) discusses why effect sizes from initially reported findings can be exaggerated, particularly under conditions involving limited power and selective reporting.
Not every original effect is inflated. A well-designed, adequately powered study may provide a reasonably precise estimate. Nevertheless, treating the published point estimate as certain ignores the possibility that the replication effect will be smaller.
This matters because conventional power calculations are highly sensitive to the assumed effect magnitude.
How Does Effect Size Influence Required Sample Size?
For a fixed statistical design, significance threshold, and target power, smaller assumed effects generally require larger samples.
For a simple comparison of two independent group means with equal allocation and equal variances, an approximate normal-theory sample-size formula illustrates the relationship.
Halving the assumed standardized effect from 0.50 to 0.25 increases the approximate required sample size fourfold under this model.
These figures illustrate the sensitivity of sample-size planning to effect assumptions. Exact calculations may differ when using the noncentral t distribution, unequal allocation, clustering, repeated measures, covariate adjustment, or other design features.
The formula does not establish that either effect size is plausible or scientifically important. Those judgments require substantive evidence.
What Is the Difference Between an Observed Effect and a Planning Effect?
The observed effect is the estimate obtained from an existing dataset. The planning effect is the effect magnitude used to determine the resources needed for a proposed study.
They may be the same, but there is no methodological requirement that they must be.
Original observed effect
The effect estimate reported by the previous study, subject to sampling uncertainty and possible bias.
Planning effect
The effect magnitude assumed when calculating sample size or evaluating the precision of the proposed replication.
A planning effect should be justified according to the replication objective. It might be informed by the original estimate, an evidence synthesis, a theoretically meaningful threshold, or a range of plausible effects.
Choosing a smaller planning effect is not automatically more rigorous. The chosen value must remain relevant to the scientific question.
Should Researchers Use the Lower Confidence Limit of the Original Effect?
Using a lower confidence limit has been proposed as a conservative alternative to using the original point estimate.
Anderson, Kelley, and Maxwell (2017) discuss sample-size planning for replication using a lower confidence limit on the original effect as a way to address uncertainty and the possibility of overestimation.
For a positive original effect, planning around a lower plausible magnitude can require a larger sample than planning around the point estimate.
However, this approach has limitations. If the original confidence interval includes zero or negative effects, a lower confidence limit may be unsuitable as a positive effect assumption. The resulting sample size could also become impractically large.
Furthermore, a confidence limit is not automatically the smallest scientifically important effect.
Researchers should understand the rationale and limitations of the method rather than treat a particular confidence bound as a universally correct planning value.
What Is the Smallest Effect Size of Interest?
The smallest effect size of interest, sometimes called the SESOI, is the smallest effect magnitude that researchers consider scientifically or practically meaningful for the question being investigated.
For example, an educational intervention may produce a statistically detectable improvement that is too small to justify its cost or implementation burden. A meaningful-effect threshold can help distinguish effects worth detecting from effects of limited practical consequence.
The threshold should be justified using substantive considerations, such as educational consequences, clinical relevance, policy implications, or a defensible theoretical rationale.
It should not be selected merely to obtain a convenient sample size.
Lakens (2022) discusses the importance of sample-size justification and the use of meaningful effect thresholds when planning studies.
The SESOI can inform power calculations, equivalence testing, and precision targets, but its interpretation depends on the intended inference.
Can a Meta-Analysis Provide a Better Planning Estimate?
When several relevant studies are available, a systematic evidence synthesis may provide a more informative basis for planning than a single published result.
A meta-analysis can combine effect estimates and characterize uncertainty, although its usefulness depends on the quality, comparability, and completeness of the included evidence.
Researchers should examine differences in populations, interventions, outcomes, and study designs before adopting a pooled effect as the planning value.
Publication bias, selective reporting, and between-study heterogeneity may also affect the estimate.
A pooled effect should not automatically be treated as the expected effect in every new context. If meaningful effect heterogeneity exists, the replication's particular conditions may warrant a different assumption.
What If the Original Study Is the Only Available Evidence?
When only one study exists, uncertainty about the plausible effect may be substantial.
Researchers can examine the original estimate and confidence interval, evaluate the study's methodological quality, and consider what effect magnitudes are substantively meaningful.
A sensitivity analysis can show how required sample size changes across several plausible assumptions.
For example, a researcher might calculate requirements for standardized effects of 0.20, 0.30, and 0.50, explaining why each value is relevant.
These calculations do not establish which effect is true. They make the consequences of uncertainty visible.
Where resources cannot support adequate precision for the scientifically important range, researchers should reconsider the study objective rather than disguise the limitation by choosing an optimistic effect assumption.
Should Replication Sample Size Be Based on Statistical Power or Precision?
Power-based and precision-based planning address related but different objectives.
Power-based planning asks how likely a specified statistical procedure is to reject a null hypothesis under an assumed effect.
Precision-based planning asks how accurately the study is expected to estimate a quantity, often through the anticipated width of a confidence interval.
For replication research, precision can be especially valuable because the goal may be to determine whether the new estimate is compatible with the original finding or with a range of scientifically meaningful effects.
| Planning Approach |
Primary Question |
Important Consideration |
| Conventional power analysis |
What sample is needed to detect a specified effect with a chosen probability? |
Depends strongly on the assumed effect and test. |
| Precision-based planning |
What sample is needed to estimate the effect with a desired uncertainty interval? |
Requires assumptions about variability and the desired interval width. |
| Equivalence-oriented planning |
What sample is needed to evaluate whether effects fall within prespecified meaningful bounds? |
Requires justified equivalence margins and an appropriate testing framework. |
| Assurance or predictive planning |
What is the expected probability of meeting the study objective across uncertainty about the true effect? |
Requires an explicit distribution representing uncertainty in planning parameters. |
No single approach is universally preferable. The method should follow the scientific question rather than the convenience of a particular calculator.
Why Simply Repeating the Original Sample Size Is Usually Insufficient
Using the same number of participants as the original study may appear to preserve methodological similarity, but sample-size equality does not establish adequate power or precision.
If the original study was underpowered, repeating its sample size may reproduce the same limitation.
Conversely, a replication using a larger sample can preserve the original intervention, measurement, and analytical estimand while obtaining more precise evidence.
Increasing sample size is generally compatible with improving the execution of a direct replication, provided recruitment changes do not introduce consequential differences in the population.
How Do Attrition and Noncompliance Affect Sample-Size Planning?
The number of participants recruited may differ from the number contributing usable outcome data.
Researchers should anticipate plausible attrition, missing outcomes, noncompliance, and other implementation problems relevant to the design.
For example, if 250 completed observations are required and the anticipated proportion with usable outcomes is 80%, a simple recruitment calculation is 250 / 0.80 = 312.5, suggesting at least 313 recruits.
This adjustment is only an approximation. It does not correct bias caused by informative missingness, and it may be inadequate when attrition differs between groups or affects the intended estimand.
For clustered or longitudinal studies, additional design-specific considerations may be necessary.
Should Replication Planning Account for Differences in Population and Measurement?
Yes. The original effect estimate may not transport directly to a replication involving different participants, instruments, or research conditions.
For example, an educational intervention may have a smaller effect among experienced learners than among beginners. A change in the population can therefore affect the plausibility of the original effect as a planning assumption.
Similarly, replacing the outcome instrument may change score variability, reliability, or sensitivity.
Researchers should examine population comparability and measurement correspondence before adopting the original estimate.
A sample-size calculation is only as relevant as the design assumptions on which it is based.
How Should Researchers Justify the Final Sample Size?
A defensible justification identifies the primary outcome and estimand, the statistical or estimation objective, the planning assumptions, and the resulting sample-size requirement.
For a power analysis, report the assumed effect, significance level, target power, statistical procedure, allocation ratio, and relevant design parameters.
For precision-based planning, explain the desired confidence-interval width and assumptions about variability.
Researchers should also explain how the original evidence informed the planning values and why those values are plausible.
Where uncertainty is substantial, provide sensitivity calculations rather than presenting a single number as though it were inevitable.
Watch Out
Do not choose an optimistic effect size simply because it produces an affordable sample. A mathematically correct power calculation can still be scientifically misleading when its effect assumption is poorly justified.
Sample-size decisions should be coordinated with the replication's intended statistical analysis, since different estimands and models may require different planning assumptions.