03 · What You Need to Know
How Research Design Similarity Affects Replication
Replication Is About Testing a Claim, Not Merely Repeating Activities
Replication is commonly understood as repeating an earlier study to determine whether its findings recur. This is useful as a starting point, but it leaves an important question unanswered: which parts of the original study must be repeated?
Nosek and Errington (2020) propose understanding replication in terms of whether a new study's possible outcomes would provide diagnostic evidence about a previous claim. Under this perspective, procedural similarity is important because it helps establish what the new evidence can reasonably tell us, not because repetition is inherently valuable.
Suppose an experiment reports that retrieval practice improves students' subsequent test performance compared with rereading. A replication should identify the precise claim being tested, including the intervention, comparison condition, outcome, and relevant population. Reproducing the classroom furniture would ordinarily be less important than preserving the instructional contrast and the meaning of the outcome measure.
However, seemingly minor environmental details may become important when theory or prior evidence suggests that they influence the effect. Decisions about similarity therefore require substantive justification rather than a mechanical checklist.
Direct and Conceptual Replication Require Different Degrees of Similarity
| Dimension |
Direct Replication |
Conceptual Replication |
| Primary purpose |
Test whether the original finding is supported under closely corresponding conditions. |
Test whether the underlying claim is supported using meaningfully different operationalizations or procedures. |
| Research design |
Preserves the defining design features as closely as feasible. |
May use a different design when it can appropriately test the same theoretical claim. |
| Intervention or exposure |
Maintains the original operational contrast where possible. |
May operationalize the same construct differently. |
| Measurement |
Generally preserves the original measures and scoring rules. |
May use alternative valid measures of the intended construct. |
| Interpretation |
Provides a relatively close test of the original result. |
Examines the claim across different operational conditions, with additional interpretive assumptions. |
These categories are useful but not perfectly discrete. Replication studies can differ in several dimensions, and disciplinary conventions vary. Researchers should describe the actual similarities and differences rather than relying exclusively on a label.
Identify the Original Study's Defining Design Features
Before deciding what to preserve, reconstruct the original study's methodological logic. Identify the research question, design, target population, sampling approach, intervention or exposure, comparison condition, measurement strategy, timing, and analysis.
For an experimental study, random assignment and the structure of the treatment contrast may be fundamental to the original causal inference. For an observational study, the operational definition of exposure, outcome measurement, and confounding-control strategy may be especially consequential. In qualitative research, replication terminology and expectations differ, so the relevant methodological commitments and intended form of corroboration must be specified.
Not every original design decision is equally important. The essential question is whether changing a feature alters the phenomenon being investigated, the estimand being targeted, or the conditions under which the original claim was expected to hold.
Which Features Usually Deserve the Closest Correspondence?
The following dimensions ordinarily require explicit evaluation:
-
Research question and hypothesis: The new study should test an identifiable claim from the original research rather than a loosely related question.
-
Design logic: Preserve features necessary for the intended descriptive, associational, or causal inference.
-
Intervention and comparison: Maintain the contrast that generated the original finding when attempting a direct replication.
-
Population and eligibility: Examine whether participant characteristics are plausible moderators of the effect.
-
Measures and timing: Ensure that constructs, assessment conditions, and observation periods remain meaningfully comparable.
-
Analysis and estimand: Identify whether both studies estimate the same quantity and whether different analytic choices change its interpretation.
The detailed question of which original procedures must be preserved depends on how each procedure contributes to the original claim.
Can Some Design Features Be Changed Without Compromising Replication?
Yes. Some changes may improve implementation without changing the substantive test. Examples include strengthening data security, correcting transcription errors in materials, improving documentation, or implementing a more reliable method of administering an otherwise equivalent procedure.
Other changes are more consequential. Replacing an intervention, changing the comparison group, adopting a different outcome construct, or substantially altering the study population may change what is being tested.
The distinction is not simply between small and large changes. A brief modification to an instruction can be consequential if it alters the intended psychological manipulation. Conversely, replacing outdated equipment may have little substantive effect when equivalence is established.
Researchers considering procedural improvements within a direct replication should distinguish implementation quality from changes to the underlying research operation.
Should the Replication Use the Same Population and Setting?
Close correspondence in population and setting can reduce uncertainty about whether contextual differences explain divergent findings. Nevertheless, recruiting exactly the same individuals is neither expected nor desirable when the purpose is to obtain independent evidence.
What matters is whether the new participants satisfy comparable eligibility criteria and whether differences in characteristics could modify the effect.
A replication conducted with university students in another country may be informative, but it introduces possible differences in language, educational systems, prior knowledge, and cultural context. Those differences do not automatically invalidate the study. They change the interpretation of what a consistent or inconsistent result might mean.
Decisions about preserving the original population should therefore be linked to the scope of the original claim.
How Closely Must Measurement and Statistical Analysis Match?
Measurement correspondence is important because a replication can appear to test the same hypothesis while actually assessing a different construct. Changes in instrument wording, scoring, administration mode, or assessment timing may affect comparability.
Similarly, different statistical models may target different estimands, apply different assumptions, or handle missing observations differently. An analysis that is statistically defensible in isolation is not necessarily equivalent to the original analysis.
Researchers should distinguish the primary comparison with the original finding from supplementary analyses intended to improve robustness. Questions about using the original measurement instruments and reproducing the original statistical analysis require separate methodological judgments.
Why Perfect Methodological Similarity Is Neither Possible Nor Sufficient
Even carefully controlled replications occur at another time, with new participants and some unavoidable differences in implementation. Exact identity across studies is therefore unrealistic.
More importantly, reproducing an original methodological weakness does not make the resulting evidence automatically stronger. An underpowered design, invalid measurement instrument, or inadequate control condition may compromise both studies.
Researchers must balance methodological fidelity against validity. Where the original methods contain serious problems, it may be preferable to conduct a corrected replication, a complementary robustness study, or both, while describing the distinction transparently.
Watch Out
Do not describe a substantially redesigned study as an exact replication simply because it investigates the same topic. Conversely, do not assume that following every original procedure guarantees a valid or informative replication.
How Should Design Differences Be Evaluated?
A useful approach is to assess each proposed change against four questions: Does it alter the target claim? Does it change the operational meaning of a key variable? Could it plausibly modify the expected effect? Would the interpretation of a different result become ambiguous?
These judgments should be made before examining the replication outcomes whenever feasible. Preregistration can document the intended design, planned deviations, hypotheses, and analyses, reducing the risk that methodological differences are rationalized after results are known.
Brandt and colleagues' Replication Recipe emphasizes careful definition of the target effect, close methodological correspondence, adequate statistical power, transparent reporting, and critical comparison of results.
06 · What This Means for You
Choose the Degree of Similarity That Serves Your Research Question
Begin by writing a precise statement of the original claim you intend to evaluate. Then identify which design features make that claim testable. This provides a more defensible basis for replication planning than attempting to copy every detail indiscriminately.
A simple decision framework
If your purpose is to test the original result under closely comparable conditions
Prioritize direct replication, preserve the defining operations, and minimize consequential changes.
If your purpose is to test whether the theoretical claim survives different operationalizations
Consider conceptual replication and explain how the alternative procedures test the same underlying claim.
If the original design contains a serious methodological flaw
Evaluate whether a corrected design, an additional comparison, or separate studies can provide more informative evidence.
If important contextual differences are unavoidable
Document them, evaluate plausible effect modification, and qualify claims about comparability and generalizability.
Prepare a design-comparison table before data collection, identifying each original feature, the proposed replication feature, the reason for any difference, and its likely implications. This record can support preregistration and transparent reporting.
Above all, distinguish evidence about the original finding from evidence about the broader theory. They are related, but they are not interchangeable.
07 · A Quick Checklist
Before Finalizing Your Replication Design
Before collecting replication data, check:
Identify the precise original claim, primary outcome, comparison, and target effect.
Obtain and examine the original article, supplementary materials, protocol, and available instruments.
Identify which original design features are essential to the intended inference.
Compare recruitment, eligibility, intervention, measurement, timing, and analysis across studies.
Document every consequential departure from the original methods and justify its purpose.
Assess statistical precision and sample-size requirements independently of the original sample size.
Specify the primary analysis and criteria for interpreting agreement or disagreement before examining outcomes.
Consider preregistration and provide sufficient methodological detail for independent evaluation.