Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

Should Replication Study Sample Size Be Based on the Original Effect Size Estimate?

The original effect size is a useful starting point for replication planning, but treating it as the true effect can produce an underpowered study. Learn how to account for estimation uncertainty, inflated published effects, practically meaningful effects, and precision when determining replication sample size.

1338
Using Original Effect Sizes for Replication Sample Size Guide 1338 of 1361
01 · The Question

Should You Calculate Replication Sample Size Using the Published Effect?

You are planning to replicate a published study that reports a statistically significant effect. The original researchers provide an effect size, and you need to determine how many participants your replication should recruit.

It may seem reasonable to enter the published effect into a conventional power-analysis calculator. After all, the original study provides the most directly relevant empirical estimate. But what if that estimate is unusually large because the original sample was small, the result was selected for publication, or the estimate is simply imprecise?

The sample-size decision matters because an overly optimistic effect assumption can produce a replication with inadequate power. Conversely, planning around an extremely small effect may require resources disproportionate to the scientific importance of the question.

02 · The Short Answer

Use the Original Effect as Evidence, Not as an Unquestioned Planning Value

In Brief

A replication study should not automatically base its sample size on the original published effect size. The original estimate may be imprecise or exaggerated, particularly when it comes from a small study selected for statistical significance. A defensible sample-size calculation should consider uncertainty in the original estimate, independent evidence, the smallest effect of scientific interest, and the replication's intended inferential objective.

The original effect can inform planning, especially when it is estimated precisely and supported by other studies. However, sensitivity analyses, conservative effect assumptions, and precision-based approaches may provide a more credible basis for determining the required sample size.

03 · What You Need to Know

How to Choose an Effect Size for Replication Sample-Size Planning

Why the Original Effect Size Is an Attractive Starting Point

A replication study investigates a previously reported finding, so the original effect estimate is naturally relevant to planning. It provides information about the observed magnitude of the relationship or intervention effect under the original study conditions.

For example, a randomized experiment may report a standardized mean difference of 0.50 between an intervention and comparison group. A researcher can use this estimate to calculate the sample size required to detect an effect of that magnitude under a specified statistical test.

The difficulty is that an observed effect size is an estimate, not a known population parameter. It contains sampling uncertainty and may be affected by bias, measurement limitations, and analytical choices.

Replication planning should therefore distinguish the published estimate from the effect magnitude that is reasonable to anticipate in a new study.

Why Published Effect Sizes May Be Too Large

Small studies can produce highly variable effect estimates. When attention or publication depends partly on whether results achieve statistical significance, unusually large estimates may be more likely to appear among the findings selected for further investigation.

This phenomenon is often described as the winner's curse. Conditional on selection for a favorable or statistically significant result, the reported effect may overestimate the underlying effect.

Ioannidis (2008) discusses why effect sizes from initially reported findings can be exaggerated, particularly under conditions involving limited power and selective reporting.

Not every original effect is inflated. A well-designed, adequately powered study may provide a reasonably precise estimate. Nevertheless, treating the published point estimate as certain ignores the possibility that the replication effect will be smaller.

This matters because conventional power calculations are highly sensitive to the assumed effect magnitude.

How Does Effect Size Influence Required Sample Size?

For a fixed statistical design, significance threshold, and target power, smaller assumed effects generally require larger samples.

For a simple comparison of two independent group means with equal allocation and equal variances, an approximate normal-theory sample-size formula illustrates the relationship.

Approximate Sample Size for Two Independent Groups
n per group ≈ 2(Z1−α/2 + Z1−β)² / d²
n is the approximate number of participants per group; α is the two-sided significance level; 1−β is the target statistical power; d is the assumed standardized mean difference; and Z represents the corresponding standard-normal quantile. The approximation assumes independent observations, equal allocation, and a common outcome standard deviation.
For a two-sided α of 0.05 and 80% power, Z0.975 ≈ 1.96 and Z0.80 ≈ 0.84. If d = 0.50, n ≈ 2(1.96 + 0.84)² / 0.50² = 62.72, or approximately 63 participants per group. If d = 0.25, n ≈ 250.88, or approximately 251 per group.

Halving the assumed standardized effect from 0.50 to 0.25 increases the approximate required sample size fourfold under this model.

These figures illustrate the sensitivity of sample-size planning to effect assumptions. Exact calculations may differ when using the noncentral t distribution, unequal allocation, clustering, repeated measures, covariate adjustment, or other design features.

The formula does not establish that either effect size is plausible or scientifically important. Those judgments require substantive evidence.

What Is the Difference Between an Observed Effect and a Planning Effect?

The observed effect is the estimate obtained from an existing dataset. The planning effect is the effect magnitude used to determine the resources needed for a proposed study.

They may be the same, but there is no methodological requirement that they must be.

Original observed effect The effect estimate reported by the previous study, subject to sampling uncertainty and possible bias.
Planning effect The effect magnitude assumed when calculating sample size or evaluating the precision of the proposed replication.

A planning effect should be justified according to the replication objective. It might be informed by the original estimate, an evidence synthesis, a theoretically meaningful threshold, or a range of plausible effects.

Choosing a smaller planning effect is not automatically more rigorous. The chosen value must remain relevant to the scientific question.

Should Researchers Use the Lower Confidence Limit of the Original Effect?

Using a lower confidence limit has been proposed as a conservative alternative to using the original point estimate.

Anderson, Kelley, and Maxwell (2017) discuss sample-size planning for replication using a lower confidence limit on the original effect as a way to address uncertainty and the possibility of overestimation.

For a positive original effect, planning around a lower plausible magnitude can require a larger sample than planning around the point estimate.

However, this approach has limitations. If the original confidence interval includes zero or negative effects, a lower confidence limit may be unsuitable as a positive effect assumption. The resulting sample size could also become impractically large.

Furthermore, a confidence limit is not automatically the smallest scientifically important effect.

Researchers should understand the rationale and limitations of the method rather than treat a particular confidence bound as a universally correct planning value.

What Is the Smallest Effect Size of Interest?

The smallest effect size of interest, sometimes called the SESOI, is the smallest effect magnitude that researchers consider scientifically or practically meaningful for the question being investigated.

For example, an educational intervention may produce a statistically detectable improvement that is too small to justify its cost or implementation burden. A meaningful-effect threshold can help distinguish effects worth detecting from effects of limited practical consequence.

The threshold should be justified using substantive considerations, such as educational consequences, clinical relevance, policy implications, or a defensible theoretical rationale.

It should not be selected merely to obtain a convenient sample size.

Lakens (2022) discusses the importance of sample-size justification and the use of meaningful effect thresholds when planning studies.

The SESOI can inform power calculations, equivalence testing, and precision targets, but its interpretation depends on the intended inference.

Can a Meta-Analysis Provide a Better Planning Estimate?

When several relevant studies are available, a systematic evidence synthesis may provide a more informative basis for planning than a single published result.

A meta-analysis can combine effect estimates and characterize uncertainty, although its usefulness depends on the quality, comparability, and completeness of the included evidence.

Researchers should examine differences in populations, interventions, outcomes, and study designs before adopting a pooled effect as the planning value.

Publication bias, selective reporting, and between-study heterogeneity may also affect the estimate.

A pooled effect should not automatically be treated as the expected effect in every new context. If meaningful effect heterogeneity exists, the replication's particular conditions may warrant a different assumption.

What If the Original Study Is the Only Available Evidence?

When only one study exists, uncertainty about the plausible effect may be substantial.

Researchers can examine the original estimate and confidence interval, evaluate the study's methodological quality, and consider what effect magnitudes are substantively meaningful.

A sensitivity analysis can show how required sample size changes across several plausible assumptions.

For example, a researcher might calculate requirements for standardized effects of 0.20, 0.30, and 0.50, explaining why each value is relevant.

These calculations do not establish which effect is true. They make the consequences of uncertainty visible.

Where resources cannot support adequate precision for the scientifically important range, researchers should reconsider the study objective rather than disguise the limitation by choosing an optimistic effect assumption.

Should Replication Sample Size Be Based on Statistical Power or Precision?

Power-based and precision-based planning address related but different objectives.

Power-based planning asks how likely a specified statistical procedure is to reject a null hypothesis under an assumed effect.

Precision-based planning asks how accurately the study is expected to estimate a quantity, often through the anticipated width of a confidence interval.

For replication research, precision can be especially valuable because the goal may be to determine whether the new estimate is compatible with the original finding or with a range of scientifically meaningful effects.

Planning Approach Primary Question Important Consideration
Conventional power analysis What sample is needed to detect a specified effect with a chosen probability? Depends strongly on the assumed effect and test.
Precision-based planning What sample is needed to estimate the effect with a desired uncertainty interval? Requires assumptions about variability and the desired interval width.
Equivalence-oriented planning What sample is needed to evaluate whether effects fall within prespecified meaningful bounds? Requires justified equivalence margins and an appropriate testing framework.
Assurance or predictive planning What is the expected probability of meeting the study objective across uncertainty about the true effect? Requires an explicit distribution representing uncertainty in planning parameters.

No single approach is universally preferable. The method should follow the scientific question rather than the convenience of a particular calculator.

Why Simply Repeating the Original Sample Size Is Usually Insufficient

Using the same number of participants as the original study may appear to preserve methodological similarity, but sample-size equality does not establish adequate power or precision.

If the original study was underpowered, repeating its sample size may reproduce the same limitation.

Conversely, a replication using a larger sample can preserve the original intervention, measurement, and analytical estimand while obtaining more precise evidence.

Increasing sample size is generally compatible with improving the execution of a direct replication, provided recruitment changes do not introduce consequential differences in the population.

How Do Attrition and Noncompliance Affect Sample-Size Planning?

The number of participants recruited may differ from the number contributing usable outcome data.

Researchers should anticipate plausible attrition, missing outcomes, noncompliance, and other implementation problems relevant to the design.

For example, if 250 completed observations are required and the anticipated proportion with usable outcomes is 80%, a simple recruitment calculation is 250 / 0.80 = 312.5, suggesting at least 313 recruits.

This adjustment is only an approximation. It does not correct bias caused by informative missingness, and it may be inadequate when attrition differs between groups or affects the intended estimand.

For clustered or longitudinal studies, additional design-specific considerations may be necessary.

Should Replication Planning Account for Differences in Population and Measurement?

Yes. The original effect estimate may not transport directly to a replication involving different participants, instruments, or research conditions.

For example, an educational intervention may have a smaller effect among experienced learners than among beginners. A change in the population can therefore affect the plausibility of the original effect as a planning assumption.

Similarly, replacing the outcome instrument may change score variability, reliability, or sensitivity.

Researchers should examine population comparability and measurement correspondence before adopting the original estimate.

A sample-size calculation is only as relevant as the design assumptions on which it is based.

How Should Researchers Justify the Final Sample Size?

A defensible justification identifies the primary outcome and estimand, the statistical or estimation objective, the planning assumptions, and the resulting sample-size requirement.

For a power analysis, report the assumed effect, significance level, target power, statistical procedure, allocation ratio, and relevant design parameters.

For precision-based planning, explain the desired confidence-interval width and assumptions about variability.

Researchers should also explain how the original evidence informed the planning values and why those values are plausible.

Where uncertainty is substantial, provide sensitivity calculations rather than presenting a single number as though it were inevitable.

Watch Out

Do not choose an optimistic effect size simply because it produces an affordable sample. A mathematically correct power calculation can still be scientifically misleading when its effect assumption is poorly justified.

Sample-size decisions should be coordinated with the replication's intended statistical analysis, since different estimands and models may require different planning assumptions.

04 · A Practical Example

Planning a Replication When the Original Effect May Be Inflated

Hypothetical Example

Replicating an Automated Feedback Experiment

An original randomized experiment reports a standardized mean difference of d = 0.50 in programming performance between students receiving automated feedback and those receiving conventional feedback.

The original study has a relatively small sample, and its effect estimate is imprecise. A replication team wants 80% power using a two-sided significance level of 0.05 for a two-group comparison.

Assumed Standardized Effect Approximate Participants per Group Approximate Total
0.50 63 126
0.40 99 198
0.30 175 350
0.25 251 502
0.20 393 786

These illustrative values use the approximate normal-theory formula for independent groups with equal allocation and a common standard deviation. Exact calculations and recruitment requirements may differ.

Evaluate the original estimate.

The team considers the original effect of 0.50 alongside its uncertainty, study design, and possible selection or reporting biases.

Define a meaningful planning range.

The researchers identify effects between 0.25 and 0.50 as plausible planning scenarios, while separately considering what magnitude would justify implementing the intervention.

Examine resource implications.

The team observes that planning for an effect of 0.25 requires approximately four times the sample needed for an effect of 0.50 under the stated assumptions.

Select and justify the design.

Rather than choosing the original effect automatically, the researchers select a planning assumption based on the available evidence, scientific importance, and feasible recruitment. They document the sensitivity of their design to smaller effects.

Interpret the eventual estimate.

The replication reports its effect estimate and uncertainty interval, regardless of whether the result crosses the conventional significance threshold. The interpretation considers the original finding and the range of effects compatible with the new data.

The example illustrates why the original effect is useful information but not necessarily the best single planning value. A replication designed only around an inflated estimate may have limited ability to detect or estimate a smaller, scientifically important effect.

05 · What Researchers Often Get Wrong

Common Mistakes in Replication Sample-Size Planning

Misconception

Is the Original Published Effect the True Effect Size?

No. It is an estimate subject to sampling uncertainty and potential bias. Its precision and credibility should be evaluated before using it for planning.

Misconception

Does a Statistically Significant Original Result Provide a Reliable Planning Effect?

Not necessarily. Selection for statistical significance can favor unusually large estimates, particularly when the original study has low power.

Misconception

Must a Direct Replication Use the Original Sample Size?

No. Sample size should reflect the new study's inferential objective and design. Repeating an inadequate original sample size can reproduce low precision.

Misconception

Is Planning for the Smallest Possible Effect Always Best?

No. Extremely small effects may be scientifically unimportant or require impractical resources. The planning effect should be justified through substantive relevance and available evidence.

Misconception

Does 80% Power Mean the Study Has an 80% Chance of Producing the Correct Conclusion?

No. Conventional power is the probability of rejecting a specified null hypothesis under a particular assumed alternative and model. It is not a general probability that the research conclusion is correct.

Misconception

Can a Larger Sample Correct Bias in the Original Research Design?

No. Larger samples may reduce random error but do not automatically eliminate confounding, invalid measurement, selection bias, or other systematic weaknesses.

06 · What This Means for You

Choose a Sample Size That Can Answer the Replication Question

Begin by deciding what the replication must establish. A study intended to detect a meaningful effect may require a different sample-size strategy from one intended to estimate the effect precisely or evaluate equivalence within prespecified bounds.

A simple decision framework

If the original estimate is precise and supported by independent evidence
Consider it as a planning input, while examining sensitivity to smaller effects.
If the original study is small or its estimate is imprecise
Avoid relying solely on the point estimate. Consider conservative assumptions, confidence-limit approaches, or a plausible effect range.
If the smallest meaningful effect can be justified
Use it to inform the relevant power, precision, or equivalence objective.
If several comparable studies are available
Evaluate the broader evidence, including heterogeneity and possible reporting bias, before selecting the planning effect.
If resources cannot support the required sample
Reconsider the inferential objective, design efficiency, collaboration, or feasibility rather than silently adopting an implausibly large effect.

Report the assumptions behind the calculation and the sample sizes required under plausible alternatives. This makes the design easier to evaluate and prevents an apparently precise sample-size calculation from concealing uncertainty about the expected effect.

The final decision should reflect the scientific value of the information the replication is expected to produce.

07 · A Quick Checklist

Before Calculating Replication Sample Size

Before finalizing the sample-size justification, check:
Identify the primary outcome, target estimand, and intended statistical analysis.
Retrieve the original effect estimate, uncertainty interval, sample size, and relevant methodological details.
Evaluate whether the original estimate may be imprecise or affected by selection and reporting biases.
Review other relevant studies or evidence syntheses when available.
Define the smallest scientifically meaningful effect when the research objective permits it.
Choose whether the design is primarily power-based, precision-based, or directed toward another inferential objective.
Calculate sample-size requirements across a defensible range of effect assumptions.
Account for allocation, clustering, repeated measures, attrition, and other relevant design features.
Document the final assumptions, calculation method, and implications of limited resources.
08 · Frequently Asked Questions

Frequently Asked Questions About Replication Sample Size

Can I use the original effect size in a replication power analysis?

Yes, but it should be treated as an uncertain estimate rather than the known true effect. Examine its precision, methodological context, and sensitivity to smaller plausible effects.

Why might the original effect size be inflated?

Sampling variability, low statistical power, selective reporting, and selection for significant findings can contribute to exaggerated published estimates. Inflation is possible, not inevitable.

Should I use half the original effect size?

Halving the effect is sometimes used as a sensitivity scenario, but it is not a universal rule. A planning value should be justified through evidence, uncertainty, and scientific importance.

Can I use the lower confidence limit of the original effect?

Yes, in appropriate circumstances. This may provide a more conservative planning assumption, but it can be unsuitable when the confidence interval includes zero or effects in the opposite direction.

Must a replication have more participants than the original?

No fixed multiplier applies. The required sample depends on the effect of interest, statistical model, precision, power, and design. Many replications benefit from larger samples, but the justification must be specific.

Is 80% statistical power always sufficient?

No. The appropriate power depends on the scientific stakes, resources, consequences of errors, and inferential objective. Some studies may justify higher power or a precision-based design.

Can I plan sample size without assuming an effect size?

Some estimation objectives can be planned using desired precision and assumptions about outcome variability rather than a specific nonzero treatment effect. Other designs still require assumptions about nuisance parameters or plausible outcomes.

What if the required replication sample is too large?

Consider collaborative recruitment, more efficient designs, justified alternative objectives, or transparent feasibility limitations. Avoid selecting an unsupported effect size merely to make the calculation affordable.

09 · The Bottom Line

Plan for a Plausible Effect, Not Merely the Published Estimate

The Bottom Line

Replication sample size should not automatically be calculated using the original published effect size. That estimate is uncertain and may be exaggerated, so planning should consider independent evidence, meaningful effect thresholds, plausible effect ranges, and the replication's intended power or precision.

A transparent sensitivity analysis often provides more useful information than a single calculation based on an optimistic estimate. The sample-size justification should explain what the replication can reasonably detect or estimate and what uncertainty will remain.

10 · Sources and Further Reading

Authoritative Sources on Replication Sample-Size Planning

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes