Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

How Closely Should a Replication Study Match the Original Research Design?

A replication study does not necessarily need to reproduce every detail of the original research design. Learn which similarities are essential, when changes are defensible, and how design differences affect the interpretation of replication findings.

1330
Matching the Original Research Design Guide 1330 of 1361
01 · The Question

How Similar Must a Replication Study Be to the Original?

You have identified a published finding worth testing again, but the original research design may not be entirely suitable for your circumstances. Perhaps the participants are difficult to recruit, the original instruments are unavailable, or certain procedures could be implemented more rigorously. How much can you change before your study stops being a meaningful replication?

The difficulty is that similarity alone does not determine whether a replication is informative. What matters is whether the new study provides a credible test of the original scientific claim, and whether differences between the studies permit a defensible interpretation of their results.

02 · The Short Answer

Match the Features That Matter to the Original Claim

In Brief

A replication study should preserve the features of the original research design necessary to test the same scientific claim, but it does not need to reproduce every procedural detail. A direct replication generally requires close methodological correspondence, whereas a conceptual replication deliberately changes some operational features while testing the underlying claim.

The appropriate degree of similarity depends on the research question, the theoretical assumptions behind the original finding, and the purpose of the replication. Any consequential differences should be justified, documented, and considered when interpreting the results.

03 · What You Need to Know

How Research Design Similarity Affects Replication

Replication Is About Testing a Claim, Not Merely Repeating Activities

Replication is commonly understood as repeating an earlier study to determine whether its findings recur. This is useful as a starting point, but it leaves an important question unanswered: which parts of the original study must be repeated?

Nosek and Errington (2020) propose understanding replication in terms of whether a new study's possible outcomes would provide diagnostic evidence about a previous claim. Under this perspective, procedural similarity is important because it helps establish what the new evidence can reasonably tell us, not because repetition is inherently valuable.

Suppose an experiment reports that retrieval practice improves students' subsequent test performance compared with rereading. A replication should identify the precise claim being tested, including the intervention, comparison condition, outcome, and relevant population. Reproducing the classroom furniture would ordinarily be less important than preserving the instructional contrast and the meaning of the outcome measure.

However, seemingly minor environmental details may become important when theory or prior evidence suggests that they influence the effect. Decisions about similarity therefore require substantive justification rather than a mechanical checklist.

Direct and Conceptual Replication Require Different Degrees of Similarity

Dimension Direct Replication Conceptual Replication
Primary purpose Test whether the original finding is supported under closely corresponding conditions. Test whether the underlying claim is supported using meaningfully different operationalizations or procedures.
Research design Preserves the defining design features as closely as feasible. May use a different design when it can appropriately test the same theoretical claim.
Intervention or exposure Maintains the original operational contrast where possible. May operationalize the same construct differently.
Measurement Generally preserves the original measures and scoring rules. May use alternative valid measures of the intended construct.
Interpretation Provides a relatively close test of the original result. Examines the claim across different operational conditions, with additional interpretive assumptions.

These categories are useful but not perfectly discrete. Replication studies can differ in several dimensions, and disciplinary conventions vary. Researchers should describe the actual similarities and differences rather than relying exclusively on a label.

Identify the Original Study's Defining Design Features

Before deciding what to preserve, reconstruct the original study's methodological logic. Identify the research question, design, target population, sampling approach, intervention or exposure, comparison condition, measurement strategy, timing, and analysis.

For an experimental study, random assignment and the structure of the treatment contrast may be fundamental to the original causal inference. For an observational study, the operational definition of exposure, outcome measurement, and confounding-control strategy may be especially consequential. In qualitative research, replication terminology and expectations differ, so the relevant methodological commitments and intended form of corroboration must be specified.

Not every original design decision is equally important. The essential question is whether changing a feature alters the phenomenon being investigated, the estimand being targeted, or the conditions under which the original claim was expected to hold.

Which Features Usually Deserve the Closest Correspondence?

The following dimensions ordinarily require explicit evaluation:

  • Research question and hypothesis: The new study should test an identifiable claim from the original research rather than a loosely related question.
  • Design logic: Preserve features necessary for the intended descriptive, associational, or causal inference.
  • Intervention and comparison: Maintain the contrast that generated the original finding when attempting a direct replication.
  • Population and eligibility: Examine whether participant characteristics are plausible moderators of the effect.
  • Measures and timing: Ensure that constructs, assessment conditions, and observation periods remain meaningfully comparable.
  • Analysis and estimand: Identify whether both studies estimate the same quantity and whether different analytic choices change its interpretation.

The detailed question of which original procedures must be preserved depends on how each procedure contributes to the original claim.

Can Some Design Features Be Changed Without Compromising Replication?

Yes. Some changes may improve implementation without changing the substantive test. Examples include strengthening data security, correcting transcription errors in materials, improving documentation, or implementing a more reliable method of administering an otherwise equivalent procedure.

Other changes are more consequential. Replacing an intervention, changing the comparison group, adopting a different outcome construct, or substantially altering the study population may change what is being tested.

The distinction is not simply between small and large changes. A brief modification to an instruction can be consequential if it alters the intended psychological manipulation. Conversely, replacing outdated equipment may have little substantive effect when equivalence is established.

Researchers considering procedural improvements within a direct replication should distinguish implementation quality from changes to the underlying research operation.

Should the Replication Use the Same Population and Setting?

Close correspondence in population and setting can reduce uncertainty about whether contextual differences explain divergent findings. Nevertheless, recruiting exactly the same individuals is neither expected nor desirable when the purpose is to obtain independent evidence.

What matters is whether the new participants satisfy comparable eligibility criteria and whether differences in characteristics could modify the effect.

A replication conducted with university students in another country may be informative, but it introduces possible differences in language, educational systems, prior knowledge, and cultural context. Those differences do not automatically invalidate the study. They change the interpretation of what a consistent or inconsistent result might mean.

Decisions about preserving the original population should therefore be linked to the scope of the original claim.

How Closely Must Measurement and Statistical Analysis Match?

Measurement correspondence is important because a replication can appear to test the same hypothesis while actually assessing a different construct. Changes in instrument wording, scoring, administration mode, or assessment timing may affect comparability.

Similarly, different statistical models may target different estimands, apply different assumptions, or handle missing observations differently. An analysis that is statistically defensible in isolation is not necessarily equivalent to the original analysis.

Researchers should distinguish the primary comparison with the original finding from supplementary analyses intended to improve robustness. Questions about using the original measurement instruments and reproducing the original statistical analysis require separate methodological judgments.

Why Perfect Methodological Similarity Is Neither Possible Nor Sufficient

Even carefully controlled replications occur at another time, with new participants and some unavoidable differences in implementation. Exact identity across studies is therefore unrealistic.

More importantly, reproducing an original methodological weakness does not make the resulting evidence automatically stronger. An underpowered design, invalid measurement instrument, or inadequate control condition may compromise both studies.

Researchers must balance methodological fidelity against validity. Where the original methods contain serious problems, it may be preferable to conduct a corrected replication, a complementary robustness study, or both, while describing the distinction transparently.

Watch Out

Do not describe a substantially redesigned study as an exact replication simply because it investigates the same topic. Conversely, do not assume that following every original procedure guarantees a valid or informative replication.

How Should Design Differences Be Evaluated?

A useful approach is to assess each proposed change against four questions: Does it alter the target claim? Does it change the operational meaning of a key variable? Could it plausibly modify the expected effect? Would the interpretation of a different result become ambiguous?

These judgments should be made before examining the replication outcomes whenever feasible. Preregistration can document the intended design, planned deviations, hypotheses, and analyses, reducing the risk that methodological differences are rationalized after results are known.

Brandt and colleagues' Replication Recipe emphasizes careful definition of the target effect, close methodological correspondence, adequate statistical power, transparent reporting, and critical comparison of results.

04 · A Practical Example

Replicating a Retrieval-Practice Experiment

Hypothetical Example

Testing Whether Practice Quizzes Improve Learning

An original randomized experiment assigned undergraduate students to either complete short retrieval-practice quizzes or reread equivalent learning materials. Students completed a delayed knowledge test seven days later. The original researchers reported higher average scores in the retrieval-practice group.

A new research team wants to replicate the finding at another university.

Identify the target claim.

The primary question is whether the specified retrieval-practice intervention improves delayed test performance relative to rereading under comparable instructional conditions.

Preserve the central design.

The team retains random assignment, comparable learning materials, the two instructional conditions, the intervention schedule, and the seven-day outcome assessment.

Evaluate unavoidable differences.

Participants attend a different university, and the instructional materials require minor language adaptations. The researchers assess whether these changes preserve the intended difficulty and meaning.

Document additional improvements.

The team preregisters its protocol, establishes a justified sample size, and implements stronger allocation concealment. These improvements are distinguished from changes to the substantive intervention.

Interpret the findings cautiously.

If the effect is similar, the findings provide additional support under the new conditions. If it differs, the team examines uncertainty, implementation fidelity, measurement equivalence, and contextual differences before attributing the discrepancy to any single cause.

Now consider a different proposal: replacing the retrieval-practice quizzes with collaborative discussions and measuring students' satisfaction immediately afterward. Although the study still concerns learning activities, it no longer tests the same intervention-outcome relationship. It would not constitute a close replication of the original finding.

05 · What Researchers Often Get Wrong

Common Misunderstandings About Replication Design

Misconception

Must Every Original Procedure Be Repeated Exactly?

No. The relevant requirement is justified correspondence in features that affect the claim. Exact reproduction of incidental details may add little value, while preserving an apparently minor but theoretically important procedure may be essential.

Misconception

Does Any Methodological Change Make a Study Conceptual Replication?

Not necessarily. Minor administrative or technical adaptations may preserve the same substantive operation. Classification depends on the nature and implications of the changes, not merely their existence.

Misconception

Is a Direct Replication Automatically Stronger Than a Conceptual Replication?

Neither approach is universally superior. A close replication may provide a clearer test of a particular result, while a conceptual replication may provide valuable evidence about a broader theoretical claim. Their usefulness depends on the question and execution.

Misconception

Does a Nonsignificant Result Prove That the Original Finding Was False?

No. A nonsignificant result alone does not establish the absence of an effect. Effect estimates, uncertainty intervals, statistical power, design comparability, and the magnitude of effects compatible with the data all matter.

Misconception

Should Researchers Preserve Known Design Flaws for the Sake of Fidelity?

Not automatically. Preserving a serious flaw may reproduce the original bias rather than provide an informative test. A corrected design may be preferable, but its differences and implications for comparison must be made explicit.

06 · What This Means for You

Choose the Degree of Similarity That Serves Your Research Question

Begin by writing a precise statement of the original claim you intend to evaluate. Then identify which design features make that claim testable. This provides a more defensible basis for replication planning than attempting to copy every detail indiscriminately.

A simple decision framework

If your purpose is to test the original result under closely comparable conditions
Prioritize direct replication, preserve the defining operations, and minimize consequential changes.
If your purpose is to test whether the theoretical claim survives different operationalizations
Consider conceptual replication and explain how the alternative procedures test the same underlying claim.
If the original design contains a serious methodological flaw
Evaluate whether a corrected design, an additional comparison, or separate studies can provide more informative evidence.
If important contextual differences are unavoidable
Document them, evaluate plausible effect modification, and qualify claims about comparability and generalizability.

Prepare a design-comparison table before data collection, identifying each original feature, the proposed replication feature, the reason for any difference, and its likely implications. This record can support preregistration and transparent reporting.

Above all, distinguish evidence about the original finding from evidence about the broader theory. They are related, but they are not interchangeable.

07 · A Quick Checklist

Before Finalizing Your Replication Design

Before collecting replication data, check:
Identify the precise original claim, primary outcome, comparison, and target effect.
Obtain and examine the original article, supplementary materials, protocol, and available instruments.
Identify which original design features are essential to the intended inference.
Compare recruitment, eligibility, intervention, measurement, timing, and analysis across studies.
Document every consequential departure from the original methods and justify its purpose.
Assess statistical precision and sample-size requirements independently of the original sample size.
Specify the primary analysis and criteria for interpreting agreement or disagreement before examining outcomes.
Consider preregistration and provide sufficient methodological detail for independent evaluation.
08 · Frequently Asked Questions

Frequently Asked Questions About Replication Design Similarity

Can a replication study use a different research design?

Yes, if the alternative design can provide diagnostic evidence about the original claim. However, changing the design may shift the study toward conceptual replication and can complicate comparisons, particularly when causal identification or the target estimand changes.

Can a replication use a larger sample than the original?

Yes. A larger sample may improve precision and statistical power without changing the substantive research question. Sample size should be justified using the intended analysis, plausible effect sizes, and desired precision rather than copied automatically.

Can a replication be conducted in another country?

Yes. Such a study can provide valuable evidence, although cultural, institutional, linguistic, and population differences may influence the result. Researchers should evaluate these differences rather than assume they are irrelevant.

Can a replication improve the original methodology?

Yes, but improvements should be classified according to whether they preserve or alter the substantive test. Correcting a serious design flaw may improve validity while reducing direct comparability.

Is using the same questionnaire enough to call a study a direct replication?

No. Measurement correspondence is only one dimension. The sampling, intervention or exposure, comparison, timing, design logic, and analysis may also be consequential.

What if the original study does not report enough methodological detail?

Consult supplementary materials, protocols, repositories, and original authors where appropriate. Document unresolved uncertainties and any assumptions used to reconstruct the procedure. Avoid claiming exact methodological correspondence when essential details remain unknown.

How should researchers compare replication results with the original findings?

Compare effect estimates and their uncertainty, consider the compatibility of results, and assess methodological differences. Statistical significance alone is insufficient for determining whether findings agree.

09 · The Bottom Line

Preserve the Scientific Test, Not Every Incidental Detail

The Bottom Line

A replication study should match the original research design closely enough that its findings provide meaningful evidence about the original claim. Direct replications generally require greater methodological correspondence, while conceptual replications intentionally test the claim under different operational conditions.

The strongest design is not necessarily the one with the fewest differences. It is the one whose similarities and differences are scientifically justified, transparently documented, and appropriately reflected in the interpretation of the evidence.

10 · Sources and Further Reading

Authoritative Sources on Replication Research Design

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes