Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

Which Original Study Procedures Must Be Preserved in a Direct Replication?

Direct replication requires more than repeating the original research question. Learn which procedures ordinarily need to remain consistent, how to assess methodological equivalence, and how to document unavoidable changes.

1331
Preserving Procedures in Direct Replication Guide 1331 of 1361
01 · The Question

Which Parts of the Original Procedure Must You Reproduce?

You want to conduct a direct replication of a published study. The original article describes participant recruitment, instructions, interventions, measurements, and statistical analyses, but it may be unclear which details are indispensable and which can be adapted.

Should you reproduce the original instructions word for word? Must the same equipment, questionnaire, recruitment method, and timing be used? What happens when the original procedures are incomplete, outdated, or impractical?

These decisions matter because an apparently small procedural difference can alter the conditions under which an effect occurs. At the same time, copying incidental details does not necessarily make a replication more scientifically informative.

02 · The Short Answer

Preserve Procedures That Define the Original Test

In Brief

A direct replication should preserve the original procedures that define the research conditions, operationalize the key variables, establish the comparison, and support the intended inference. These commonly include eligibility criteria, experimental manipulation or exposure, comparison conditions, participant instructions, outcome measures, assessment timing, and the primary analytical approach.

Not every administrative or technical detail must be identical. Changes may be acceptable when they preserve the substantive operation, but their equivalence should be justified rather than assumed. Consequential departures must be reported transparently.

03 · What You Need to Know

Identifying the Procedures That Matter in Direct Replication

What Does Procedural Fidelity Mean in a Direct Replication?

Procedural fidelity refers to the extent to which the replication implements the methods and conditions specified in the original study. It involves more than using the same research design label.

Two studies may both be randomized experiments yet differ substantially in participant instructions, intervention intensity, control conditions, measurement timing, or outcome scoring. Such differences may change the effect being estimated.

Brandt and colleagues (2014) identify close correspondence in recruitment, instructions, stimuli, measures, procedures, and analyses as an important feature of convincing close replications. Their framework also emphasizes adequate statistical power, transparency, and critical evaluation of the resulting evidence.

Nevertheless, fidelity should be assessed relative to the original scientific claim. Nosek and Errington (2020) argue that the central consideration is whether the replication provides diagnostic evidence about that claim. Procedural similarity helps support this interpretation but is not an end in itself.

The broader question of how closely the research designs should correspond establishes the context for deciding which individual procedures require preservation.

Preserve Participant Eligibility and Relevant Recruitment Conditions

Eligibility criteria determine who can participate and may influence the interpretation of the original finding. Age ranges, educational background, prior experience, clinical characteristics, and other relevant attributes can affect responses to an intervention or exposure.

In a direct replication, the original inclusion and exclusion criteria should generally be preserved when they define the population to which the original claim applies.

Recruitment procedures may also matter. Participants recruited through voluntary advertisements may differ from those recruited through compulsory course participation or clinical referral. Differences in incentives, self-selection, or screening can change the composition of the sample.

However, identical recruitment channels are not always necessary. A different administrative recruitment process may be acceptable if it produces a substantively comparable eligible population and does not introduce a consequential selection mechanism.

The distinction between recruiting the same type of participants and recruiting from the same location is important when deciding whether the original study population must be preserved.

Preserve the Experimental Manipulation or Exposure Definition

For experimental research, the intervention or manipulation is often among the most important procedures to reproduce. Its content, intensity, duration, sequence, delivery method, and relevant contextual conditions may all contribute to the observed effect.

For example, a study evaluating a 20-minute retrieval-practice activity cannot automatically be replicated by substituting a 60-minute collaborative discussion. The new activity may address the same educational objective, but it represents a different intervention.

In observational research, the corresponding concern is how the exposure was defined and measured. Changing an exposure threshold, reference period, or classification rule can alter the comparison being investigated.

When the original intervention includes multiple components, researchers should identify which components are essential and whether their interactions matter. Preserving only the intervention's general name is insufficient.

Preserve the Comparison or Control Condition

An intervention effect is defined partly by what the intervention is compared against. Changing the comparator may therefore change the estimated effect even when the intervention itself remains identical.

Suppose an original experiment compares an AI-assisted writing activity with independent writing without AI assistance. A replication that compares the same AI activity with individualized human tutoring tests a different contrast.

Researchers should preserve relevant characteristics of the original comparator, including the activities performed, duration, materials, access to resources, and exposure to other treatments.

In randomized studies, the method of assignment and any important allocation restrictions also deserve attention. A replication should preserve the original randomization logic where it is central to the design, while implementing appropriate safeguards against selection bias.

Preserve Instructions, Stimuli, and Task Sequence When They Influence Responses

Participant instructions can affect expectations, motivation, comprehension, and behavior. Their wording may be especially important in psychological experiments, behavioral interventions, surveys, and educational assessments.

Where instructions operationalize the manipulation, reproducing the original wording may be appropriate. If translation or accessibility adaptation is required, researchers should assess whether the revised instructions preserve the intended meaning.

Stimuli may include images, text passages, experimental scenarios, learning materials, audiovisual content, or other research inputs. Their difficulty, emotional content, familiarity, and presentation characteristics can influence outcomes.

Task order is another potential source of variation. Changing the sequence of exposure, practice, feedback, and assessment may introduce learning, fatigue, or carryover effects.

However, the original font, file format, or software interface may be incidental when these features do not influence the intended operation. Their relevance should be evaluated rather than presumed.

Preserve Outcome Measures and Scoring Procedures

Direct replication ordinarily requires close correspondence in the operational definition of the primary outcome. Otherwise, an apparent difference between studies may reflect measurement rather than the phenomenon under investigation.

Important measurement procedures include instrument content, administration mode, response options, scoring rules, scale transformations, assessor instructions, and outcome classification thresholds.

For example, replacing an objective knowledge test with a self-reported confidence questionnaire changes the construct being measured. Even when two instruments address related constructs, their scores cannot automatically be treated as interchangeable.

Where an original instrument is unavailable, an alternative measure may be defensible, but the researcher should evaluate construct validity, reliability, and measurement comparability. The result may no longer qualify as a close replication of the original operational outcome.

Whether researchers should use exactly the same measurement instruments depends partly on the role of those instruments in the original claim.

Preserve Assessment Timing and Follow-Up Intervals

The timing of measurement can be integral to the effect being tested. Immediate post-intervention outcomes and outcomes measured several weeks later may represent different aspects of learning, behavior, recovery, or retention.

A direct replication should therefore preserve theoretically important intervals between baseline assessment, intervention delivery, follow-up, and outcome measurement.

Timing may also matter within a procedure. Exposure duration, breaks between tasks, time limits, and intervention frequency can influence performance.

Small scheduling differences may be inconsequential in some studies but important in others. Researchers should evaluate timing relative to the mechanism and outcome rather than apply an arbitrary tolerance.

Preserve Procedures That Protect the Intended Inference

Randomization, allocation concealment, blinding, standardized administration, and control of relevant confounding variables may be essential to the credibility of a study.

These procedures are not interchangeable. Randomization concerns assignment, allocation concealment prevents foreknowledge of upcoming assignments, and blinding concerns awareness of assigned conditions or other information that could influence conduct or assessment.

If the original study used appropriate safeguards, the replication should generally maintain them. If the original study omitted a necessary safeguard, reproducing that weakness is not automatically desirable.

For randomized trials, the CONSORT 2025 reporting guideline identifies important information about allocation, blinding, intervention delivery, outcomes, participant flow, and analysis. Although CONSORT is a reporting guideline rather than a design-quality instrument, its items can help researchers identify procedural information requiring careful documentation.

Preserve the Primary Analytical Comparison

Procedural correspondence extends to the analytical stage. Researchers should identify the original primary outcome, target estimand, group contrast, analysis population, statistical model, covariate adjustments, exclusion rules, and missing-data treatment.

A replication that collects equivalent data but uses an analysis estimating a different quantity may not provide a directly comparable result.

For example, comparing unadjusted post-test scores and estimating a baseline-adjusted treatment effect can yield different results, particularly when baseline imbalance, missingness, or model assumptions are consequential.

This does not mean that every original statistical choice must be copied. An original analysis may contain errors or weaknesses. Researchers can distinguish a prespecified replication analysis from additional corrected or sensitivity analyses, explaining how each informs the original claim.

Deciding which statistical procedures should remain consistent requires attention to both comparability and statistical validity.

Which Procedures May Be Adapted?

Procedure Typical Priority Reason
Eligibility criteria Preserve closely Defines the relevant participant population.
Intervention content and dosage Preserve closely Defines the treatment being evaluated.
Comparison condition Preserve closely Defines the contrast underlying the effect.
Primary outcome and scoring Preserve closely Maintains the operational meaning of the result.
Assessment timing Preserve when substantively relevant May determine which stage of an effect is observed.
Instructions and stimuli Evaluate for equivalence May influence behavior or the intended manipulation.
Recruitment administration Adapt cautiously Different methods may produce different selection patterns.
Software or equipment Adapt if equivalence is supported Technical substitutions may preserve the substantive operation.
Documentation and data security May improve Can strengthen transparency and protection without altering the tested claim.

These priorities are conditional, not universal. In a reaction-time experiment, hardware latency may be critical. In an educational intervention, the learning platform's brand may be incidental while its feedback functions are essential.

How Should Researchers Handle Missing or Poorly Reported Procedures?

Published articles often omit details necessary to reproduce a study faithfully. Researchers should examine supplementary files, registered protocols, repositories, instrument documentation, and related publications. Contacting the original authors may resolve important uncertainties.

When details remain unavailable, distinguish documented procedures from reconstructed assumptions. A plausible reconstruction is not equivalent to confirmation of what originally occurred.

Uncertainty about essential procedures should be considered when describing the replication and interpreting its findings. The appropriate response may be to narrow the replication claim or design a complementary study rather than assert methodological equivalence without evidence.

How Should Procedural Changes Be Documented?

Prepare a procedure-by-procedure comparison before data collection. For each feature, record the original method, the replication method, the evidence supporting equivalence, and the reason for any difference.

Classify changes according to whether they are administrative, implementation-related, measurement-related, or substantive alterations to the research design. This classification is an organizational aid, not a universal methodological standard.

Where possible, preregister the intended procedures and analysis plan. Record subsequent deviations with their timing and rationale, distinguishing planned adaptations from changes introduced after observing data.

Researchers considering improvements to original procedures should explain how those improvements preserve or alter the intended scientific test.

04 · A Practical Example

Preserving Procedures in an Educational Intervention Replication

Hypothetical Example

Replicating an Experiment on Automated Feedback

An original randomized study examines whether automated formative feedback improves undergraduate programming performance. Students complete a four-week sequence of programming tasks. The intervention group receives automated feedback after each submission, while the comparison group receives correctness indicators without explanatory feedback. Both groups complete the same programming assessment one week after the intervention.

A second university wants to conduct a direct replication using a newer learning management system.

Preserve the eligibility and allocation procedures.

The replication recruits students with comparable programming prerequisites and uses randomized assignment. The team documents differences in recruitment and checks whether eligibility remains consistent with the original population.

Preserve the intervention and comparator.

The same task sequence, feedback content, delivery timing, and instructional duration are retained. The comparison group continues to receive correctness indicators rather than explanatory feedback.

Preserve the primary measurement.

The researchers use the same assessment tasks, scoring rubric, and one-week follow-up interval. They standardize grading procedures and establish how disagreements between assessors will be handled.

Evaluate the platform substitution.

The new learning management system is tested to confirm that it displays the intended feedback, supports equivalent submission opportunities, and records the necessary data. Differences in interface behavior are documented.

Prespecify analysis and deviations.

The team identifies the original primary contrast and plans a comparable analysis. Any additional robustness analyses are distinguished from the main replication analysis.

In this scenario, changing the learning management system may be acceptable if its relevant functions are demonstrably equivalent. Replacing automated explanatory feedback with instructor-led tutoring, however, would substantially change the intervention and undermine a claim of close procedural replication.

05 · What Researchers Often Get Wrong

Procedural Mistakes That Weaken Direct Replications

Misconception

Is Using the Same Research Design Label Enough?

No. Two randomized experiments can differ substantially in intervention content, participant eligibility, assessment timing, and analysis. Procedural fidelity must be established through the actual methods, not the design label.

Misconception

Are Participant Instructions Too Minor to Matter?

Not necessarily. Instructions may influence expectations, task interpretation, or the experimental manipulation itself. Their importance depends on how they contribute to the hypothesized effect.

Misconception

Can the Comparison Group Be Changed Without Affecting the Study?

Changing the comparator may alter the estimated intervention effect. Even when the intervention remains unchanged, the research question may no longer be operationally equivalent.

Misconception

Does Using New Technology Automatically Improve the Replication?

No. Newer technology may improve administration, but it can also change task difficulty, feedback, timing, or participant interaction. Functional equivalence should be assessed rather than inferred from technical superiority.

Misconception

Should Known Procedural Errors Be Repeated Exactly?

Not automatically. Repeating a serious flaw can reproduce bias. Researchers should distinguish a faithful reconstruction of the original method from a corrected test of the underlying claim and report the implications of any correction.

Misconception

Can Researchers Decide Which Differences Matter After Seeing the Results?

Post hoc interpretation may be informative, but selectively emphasizing differences after observing the outcome risks confirmation bias. Anticipated moderators and consequential procedural differences should be documented before data analysis where feasible.

06 · What This Means for You

Build a Procedure-Preservation Plan Before Collecting Data

The most practical approach is to reconstruct the original procedure in chronological order, from recruitment through analysis. For each stage, determine what the procedure accomplishes and what would happen if it were changed.

Rather than asking whether a detail looks important, ask whether its alteration could change participant selection, treatment exposure, measurement, bias, or the interpretation of the target effect.

A simple decision framework

If a procedure defines the intervention, comparison, or primary outcome
Preserve it closely unless a compelling methodological reason requires a change, and explicitly assess the consequences.
If a procedure influences participant selection or exposure
Evaluate whether the alternative creates a substantively different sample or treatment condition.
If a change is primarily administrative or technical
Document the substitution and establish functional equivalence where relevant.
If the original procedure is invalid, unsafe, or ethically unacceptable
Do not preserve it merely for fidelity. Redesign appropriately and explain the resulting limits on direct comparability.
If an essential original procedure cannot be reconstructed
State the uncertainty, investigate available evidence, and qualify the replication claim.

Keep a record of actual implementation, not merely intended implementation. Intervention logs, administration records, deviations, and measurement-quality checks may reveal differences that a written protocol alone cannot establish.

When the original methods contain serious weaknesses, the question of correcting methodological flaws during replication should be considered explicitly rather than resolved through undocumented adjustments.

07 · A Quick Checklist

Check These Procedures Before Starting a Direct Replication

Before implementing the replication protocol, check:
Retrieve the original methods, supplementary materials, instruments, and available protocol documentation.
Confirm the original inclusion criteria, exclusion criteria, and relevant recruitment conditions.
Reconstruct the intervention or exposure procedures, including content, intensity, sequence, and duration.
Verify that the comparison or control condition remains substantively equivalent.
Review participant instructions, stimuli, task order, and relevant administration conditions.
Confirm the primary outcome instrument, scoring rules, assessment timing, and follow-up interval.
Preserve or appropriately strengthen procedures addressing allocation, blinding, standardization, and bias.
Specify the primary analytical comparison, analysis population, exclusion rules, and missing-data approach.
Record and justify deviations, unresolved methodological uncertainties, and evidence supporting procedural equivalence.
Consider preregistration and establish a plan for monitoring and reporting actual implementation fidelity.
08 · Frequently Asked Questions

Frequently Asked Questions About Direct Replication Procedures

Must participant instructions be identical in a direct replication?

Not always, but instructions that influence the manipulation, task interpretation, or participant behavior should be preserved closely. Translations and accessibility adaptations require attention to meaning and functional equivalence.

Can a direct replication use different research equipment or software?

Yes, when the replacement preserves the relevant research operation. Researchers should verify characteristics such as timing accuracy, stimulus presentation, measurement performance, and feedback behavior when these affect the outcome.

Must a direct replication recruit participants from the same institution?

No. New participants can be recruited elsewhere, but their relevant characteristics and selection conditions should be evaluated. Institutional or contextual differences may affect the interpretation of results.

Can the intervention duration be shortened?

Shortening the intervention may change treatment exposure and the expected effect. Unless duration is demonstrably incidental, such a change reduces procedural correspondence and requires substantive justification.

Can researchers use a revised version of the original questionnaire?

Possibly, but revisions may change the measured construct or score distribution. The researcher should examine the changes, relevant validation evidence, and implications for comparison before treating the measures as equivalent.

What if the original authors cannot provide the experimental materials?

Search supplementary publications and repositories, then document the reconstruction process and unresolved uncertainty. If essential materials cannot be recovered, the study may need a more qualified description than a close direct replication.

Should the original statistical analysis be reproduced even if it was flawed?

A comparable analysis may help evaluate the original result, but a flawed analysis should not be treated as scientifically preferable. A corrected prespecified analysis, with transparent comparison to the original approach where appropriate, may provide stronger evidence.

How can researchers demonstrate that they followed the original procedures?

Provide a detailed protocol, a comparison of original and replication procedures, relevant materials, implementation records, and a transparent account of deviations. Preregistration and accessible documentation can strengthen independent evaluation.

09 · The Bottom Line

Preserve the Procedures That Make the Original Claim Testable

The Bottom Line

A direct replication should preserve the original procedures that determine participant eligibility, intervention or exposure, comparison conditions, measurement, timing, and the primary analytical test. These procedures require the closest correspondence because changing them may alter the scientific claim being evaluated.

Administrative and technical adaptations may be defensible when they preserve the substantive operation. The central responsibility is to establish, document, and critically assess procedural equivalence rather than assume that either exact copying or modernization automatically produces a stronger replication.

10 · Sources and Further Reading

Authoritative Sources on Direct Replication Procedures

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes