03 · What You Need to Know
How Statistical Analysis Choices Affect Replication Findings
Why Statistical Comparability Matters in Replication
A replication study provides new evidence about a previously investigated claim. The statistical analysis connects the collected data to that claim by specifying which quantities are estimated and how uncertainty is evaluated.
If the original and replication studies use different analytical approaches, their reported results may differ even when the underlying data-generating processes are similar.
For example, a comparison of unadjusted post-intervention means and a regression analysis adjusting for baseline performance may produce different effect estimates and standard errors. Both analyses can be appropriate in particular circumstances, but they are not necessarily interchangeable.
Nosek and Errington (2020) emphasize the importance of evaluating whether replication evidence is diagnostic of the original claim. Analytical comparability contributes to that evaluation because it helps establish whether the studies address the same empirical proposition.
However, reproducing an original analysis is not sufficient to establish a convincing replication. The analysis must also be appropriate for the design, data, and intended inference.
Distinguish Statistical Tests, Models, and Estimands
Researchers sometimes describe analytical equivalence by asking whether both studies used the same statistical test. That question is useful but incomplete.
A statistical test evaluates a hypothesis under specified assumptions. A statistical model represents relationships among variables and provides a basis for estimation or inference. An estimand is the precisely defined quantity that the research seeks to estimate.
Statistical test
A procedure for evaluating a hypothesis, such as a t-test, likelihood-ratio test, or permutation test.
Statistical model
A mathematical representation of relationships or distributions used for estimation and inference.
Estimand
The target quantity of scientific interest, such as a mean difference, risk ratio, or average treatment effect in a specified population.
Two different statistical procedures may estimate the same quantity under appropriate assumptions. Conversely, two analyses using similar terminology may target different quantities.
For example, a marginal risk difference and a conditional odds ratio describe different aspects of an intervention-outcome relationship. Comparing them as though they represented the same effect would be misleading.
The estimand should therefore be identified before deciding whether the original statistical procedure must be reproduced.
What Should Be Preserved in a Direct Replication Analysis?
A direct replication ordinarily benefits from preserving the original analytical comparison and outcome definition, provided they are methodologically defensible.
Important elements include the primary outcome, treatment or exposure contrast, target population, analysis time point, effect measure, handling of repeated observations, and relevant covariate adjustments.
Researchers should also examine whether the original analysis used particular exclusion criteria, transformations, missing-data procedures, or subgroup definitions.
These decisions may influence which observations contribute to the estimate and what the resulting quantity means.
Maintaining analytical correspondence is part of preserving the original study procedures that define the scientific test.
Nevertheless, analytical fidelity should not be interpreted as an obligation to reproduce known statistical errors.
When Is Reproducing the Original Statistical Test Appropriate?
Using the original test may be appropriate when the replication has a comparable design, outcome scale, data structure, and inferential objective, and when the original procedure remains statistically defensible.
For example, if the original randomized experiment compared two independent groups on a continuous outcome using a suitable t-test, the same approach may provide a straightforward replication comparison.
However, researchers should examine the assumptions and implementation details rather than reproduce the test name mechanically.
Important considerations include independence of observations, distributional characteristics, unequal variances, clustering, repeated measurements, and the consequences of missing data.
A test appropriate for independent observations may not be suitable when the replication recruits participants from intact classrooms or other clusters.
In that situation, preserving the original test without accounting for clustering could produce misleading uncertainty estimates.
Can a Different Statistical Test Still Support Direct Replication?
Yes, when the alternative procedure provides an appropriate estimate or test of the same substantive quantity.
For example, a Welch t-test may be preferable to a pooled-variance t-test when group variances differ and equal-variance assumptions are not justified. Both can address a difference in population means, although their standard-error calculations and inferential assumptions differ.
Similarly, robust standard errors may improve uncertainty estimation under certain conditions without changing the fitted regression coefficients.
These changes do not necessarily convert a direct replication into a conceptual replication because the substantive outcome and comparison may remain unchanged.
Nevertheless, analytical changes should be justified according to their effects on the estimand, assumptions, and comparability.
Researchers considering methodological improvements within direct replication should distinguish improvements in statistical implementation from changes in the research question.
Should the Same Covariates Be Included?
Covariate adjustment can influence both precision and interpretation. Whether the original covariates should be retained depends on the research design and the role those variables play.
In randomized experiments, adjustment for prespecified baseline covariates can improve precision under suitable conditions. In observational research, covariate selection is often central to addressing confounding.
However, adjustment for inappropriate variables can introduce bias. Conditioning on post-treatment variables, colliders, or mediators may change the estimand or distort a causal comparison.
Researchers should not assume that including more covariates necessarily produces a better analysis.
If the original analysis used a particular adjustment set, the replication team should identify the rationale and evaluate whether the same assumptions remain defensible.
When a revised adjustment strategy is necessary, explain how it affects the comparison with the original result.
How Should Missing Data Be Handled?
Missing data can affect replication findings through reduced precision, altered sample composition, and potential bias.
Suppose the original researchers analyzed only participants with complete outcome data. Reproducing this approach may support procedural comparability, but complete-case analysis can be biased when missingness is related to variables relevant to the outcome.
Alternative approaches, including multiple imputation or likelihood-based methods, may be appropriate under specified assumptions.
Little and colleagues' work on preventing and treating missing data emphasizes the importance of design-stage prevention and careful consideration of missingness mechanisms.
No statistical method can recover missing information without assumptions. Researchers should therefore report the extent and patterns of missingness, justify the primary approach, and consider sensitivity analyses where appropriate.
A replication may report an original-method analysis and an additional missing-data analysis, but their purposes and assumptions should be distinguished.
Should the Same Effect Size Measure Be Reported?
Using a comparable effect measure can make replication results easier to interpret.
For continuous outcomes, researchers might report mean differences or standardized mean differences. For binary outcomes, risk differences, risk ratios, or odds ratios may be relevant.
The appropriate measure depends on the outcome, design, and scientific question.
Effect measures are not automatically interchangeable. An odds ratio is not the same as a risk ratio, and standardized mean differences can be affected by differences in score variability between samples.
Where possible, report the effect on a substantively meaningful scale and provide uncertainty intervals.
If the original article reports only a p-value, researchers should examine whether an effect estimate and its uncertainty can be reconstructed from available information without introducing unsupported assumptions.
Why Statistical Significance Alone Is Insufficient
A common approach to replication interpretation is to ask whether the original study and replication both produce statistically significant results.
This approach can be misleading because statistical significance depends on the estimated effect, its uncertainty, sample size, and the selected threshold.
Two studies can estimate nearly identical effects while producing different significance decisions. Conversely, two studies can both produce significant results while estimating meaningfully different effect magnitudes.
Gelman and Stern (2006) explain why the difference between statistically significant and nonsignificant results is not itself necessarily statistically significant.
Replication evaluation should therefore consider effect estimates, uncertainty intervals, compatibility of results, and the substantive magnitude of differences.
Watch Out
Do not classify a replication as successful merely because both studies report p-values below 0.05, or as unsuccessful merely because one study is statistically significant and the other is not. These decisions discard information about effect magnitude and uncertainty.
How Can Effect Estimates Be Compared?
A direct comparison of effect estimates is often more informative than comparing significance labels.
For example, suppose the original study estimates a treatment effect of five points and the replication estimates an effect of four points. The one-point difference may be small relative to the uncertainty in both estimates.
Researchers can examine confidence intervals, estimate the difference between effects, or use appropriate evidence-synthesis methods.
When studies are independent and their effect estimates are approximately normally distributed, a simple comparison can use the difference between estimates and its standard error.
In this hypothetical calculation, the data are compatible with a range of differences, including no difference. The interval does not establish equivalence because it also includes potentially meaningful discrepancies.
The calculation assumes independent estimates and comparable effect definitions. It should not be applied mechanically to correlated estimates, incompatible estimands, or effect measures requiring different uncertainty calculations.
What Is the Role of Equivalence Testing?
Sometimes the replication question is not merely whether an effect differs from zero, but whether the evidence is sufficiently precise to rule out effects considered substantively important.
Equivalence testing can address a prespecified range of effects regarded as practically negligible. For example, researchers may define an equivalence margin based on educational or clinical relevance.
However, an equivalence margin should be justified before examining the results. Selecting a margin after observing the estimate risks producing a favorable conclusion by construction.
Equivalence testing and conventional null-hypothesis significance testing answer different questions.
A nonsignificant test against zero does not demonstrate equivalence. Likewise, evidence that an effect is statistically different from zero does not necessarily establish that it is practically important.
Lakens (2017) provides an accessible methodological discussion of equivalence tests and their interpretation.
Should Researchers Reproduce the Original Analysis and Add an Improved Analysis?
When feasible and scientifically justified, reporting both analyses can be useful.
The original-method analysis may help establish how the replication compares under similar analytical rules. A revised analysis may address an identifiable weakness or evaluate robustness under alternative assumptions.
However, the analyses should not be presented as interchangeable if they estimate different quantities.
Researchers should identify a primary inferential strategy before examining outcomes and distinguish it from supplementary analyses.
They should also avoid selecting whichever analysis produces the most favorable result as the definitive finding.
Where the original analysis is seriously invalid, it may be inappropriate to designate that procedure as the primary analysis merely for comparability. A corrected primary analysis can be justified, with the original approach discussed or reproduced only when informative and ethically appropriate.
What If the Original Study Did Not Report Its Analysis Clearly?
Incomplete reporting can make analytical replication difficult.
The article may identify a statistical test without specifying the covariates, exclusion rules, model specification, handling of missing observations, or precise outcome definition.
Researchers should examine supplementary materials, registered protocols, analysis scripts, repositories, and related publications. Contacting the original authors may help clarify unresolved details.
Where essential information remains unavailable, the replication team should document its assumptions and avoid presenting a reconstructed analysis as an exact reproduction.
The distinction between reproducing an analysis and independently selecting a defensible analysis should remain explicit.
How Should Analytical Decisions Be Preregistered?
A preregistered analysis plan can specify the primary outcome, estimand, statistical model, covariates, exclusion criteria, missing-data strategy, hypothesis tests, and planned sensitivity analyses.
It should also explain how the replication result will be compared with the original finding.
Preregistration helps distinguish planned analyses from decisions made after observing results, although it does not guarantee that the planned analysis is appropriate.
If changes become necessary, researchers should document when they occurred, why they were made, and how they affect interpretation.
Analytical transparency is especially important when correcting methodological flaws in the original research.