03 · What You Need to Know
Identifying the Procedures That Matter in Direct Replication
What Does Procedural Fidelity Mean in a Direct Replication?
Procedural fidelity refers to the extent to which the replication implements the methods and conditions specified in the original study. It involves more than using the same research design label.
Two studies may both be randomized experiments yet differ substantially in participant instructions, intervention intensity, control conditions, measurement timing, or outcome scoring. Such differences may change the effect being estimated.
Brandt and colleagues (2014) identify close correspondence in recruitment, instructions, stimuli, measures, procedures, and analyses as an important feature of convincing close replications. Their framework also emphasizes adequate statistical power, transparency, and critical evaluation of the resulting evidence.
Nevertheless, fidelity should be assessed relative to the original scientific claim. Nosek and Errington (2020) argue that the central consideration is whether the replication provides diagnostic evidence about that claim. Procedural similarity helps support this interpretation but is not an end in itself.
The broader question of how closely the research designs should correspond establishes the context for deciding which individual procedures require preservation.
Preserve Participant Eligibility and Relevant Recruitment Conditions
Eligibility criteria determine who can participate and may influence the interpretation of the original finding. Age ranges, educational background, prior experience, clinical characteristics, and other relevant attributes can affect responses to an intervention or exposure.
In a direct replication, the original inclusion and exclusion criteria should generally be preserved when they define the population to which the original claim applies.
Recruitment procedures may also matter. Participants recruited through voluntary advertisements may differ from those recruited through compulsory course participation or clinical referral. Differences in incentives, self-selection, or screening can change the composition of the sample.
However, identical recruitment channels are not always necessary. A different administrative recruitment process may be acceptable if it produces a substantively comparable eligible population and does not introduce a consequential selection mechanism.
The distinction between recruiting the same type of participants and recruiting from the same location is important when deciding whether the original study population must be preserved.
Preserve the Experimental Manipulation or Exposure Definition
For experimental research, the intervention or manipulation is often among the most important procedures to reproduce. Its content, intensity, duration, sequence, delivery method, and relevant contextual conditions may all contribute to the observed effect.
For example, a study evaluating a 20-minute retrieval-practice activity cannot automatically be replicated by substituting a 60-minute collaborative discussion. The new activity may address the same educational objective, but it represents a different intervention.
In observational research, the corresponding concern is how the exposure was defined and measured. Changing an exposure threshold, reference period, or classification rule can alter the comparison being investigated.
When the original intervention includes multiple components, researchers should identify which components are essential and whether their interactions matter. Preserving only the intervention's general name is insufficient.
Preserve the Comparison or Control Condition
An intervention effect is defined partly by what the intervention is compared against. Changing the comparator may therefore change the estimated effect even when the intervention itself remains identical.
Suppose an original experiment compares an AI-assisted writing activity with independent writing without AI assistance. A replication that compares the same AI activity with individualized human tutoring tests a different contrast.
Researchers should preserve relevant characteristics of the original comparator, including the activities performed, duration, materials, access to resources, and exposure to other treatments.
In randomized studies, the method of assignment and any important allocation restrictions also deserve attention. A replication should preserve the original randomization logic where it is central to the design, while implementing appropriate safeguards against selection bias.
Preserve Instructions, Stimuli, and Task Sequence When They Influence Responses
Participant instructions can affect expectations, motivation, comprehension, and behavior. Their wording may be especially important in psychological experiments, behavioral interventions, surveys, and educational assessments.
Where instructions operationalize the manipulation, reproducing the original wording may be appropriate. If translation or accessibility adaptation is required, researchers should assess whether the revised instructions preserve the intended meaning.
Stimuli may include images, text passages, experimental scenarios, learning materials, audiovisual content, or other research inputs. Their difficulty, emotional content, familiarity, and presentation characteristics can influence outcomes.
Task order is another potential source of variation. Changing the sequence of exposure, practice, feedback, and assessment may introduce learning, fatigue, or carryover effects.
However, the original font, file format, or software interface may be incidental when these features do not influence the intended operation. Their relevance should be evaluated rather than presumed.
Preserve Outcome Measures and Scoring Procedures
Direct replication ordinarily requires close correspondence in the operational definition of the primary outcome. Otherwise, an apparent difference between studies may reflect measurement rather than the phenomenon under investigation.
Important measurement procedures include instrument content, administration mode, response options, scoring rules, scale transformations, assessor instructions, and outcome classification thresholds.
For example, replacing an objective knowledge test with a self-reported confidence questionnaire changes the construct being measured. Even when two instruments address related constructs, their scores cannot automatically be treated as interchangeable.
Where an original instrument is unavailable, an alternative measure may be defensible, but the researcher should evaluate construct validity, reliability, and measurement comparability. The result may no longer qualify as a close replication of the original operational outcome.
Whether researchers should use exactly the same measurement instruments depends partly on the role of those instruments in the original claim.
Preserve Assessment Timing and Follow-Up Intervals
The timing of measurement can be integral to the effect being tested. Immediate post-intervention outcomes and outcomes measured several weeks later may represent different aspects of learning, behavior, recovery, or retention.
A direct replication should therefore preserve theoretically important intervals between baseline assessment, intervention delivery, follow-up, and outcome measurement.
Timing may also matter within a procedure. Exposure duration, breaks between tasks, time limits, and intervention frequency can influence performance.
Small scheduling differences may be inconsequential in some studies but important in others. Researchers should evaluate timing relative to the mechanism and outcome rather than apply an arbitrary tolerance.
Preserve Procedures That Protect the Intended Inference
Randomization, allocation concealment, blinding, standardized administration, and control of relevant confounding variables may be essential to the credibility of a study.
These procedures are not interchangeable. Randomization concerns assignment, allocation concealment prevents foreknowledge of upcoming assignments, and blinding concerns awareness of assigned conditions or other information that could influence conduct or assessment.
If the original study used appropriate safeguards, the replication should generally maintain them. If the original study omitted a necessary safeguard, reproducing that weakness is not automatically desirable.
For randomized trials, the CONSORT 2025 reporting guideline identifies important information about allocation, blinding, intervention delivery, outcomes, participant flow, and analysis. Although CONSORT is a reporting guideline rather than a design-quality instrument, its items can help researchers identify procedural information requiring careful documentation.
Preserve the Primary Analytical Comparison
Procedural correspondence extends to the analytical stage. Researchers should identify the original primary outcome, target estimand, group contrast, analysis population, statistical model, covariate adjustments, exclusion rules, and missing-data treatment.
A replication that collects equivalent data but uses an analysis estimating a different quantity may not provide a directly comparable result.
For example, comparing unadjusted post-test scores and estimating a baseline-adjusted treatment effect can yield different results, particularly when baseline imbalance, missingness, or model assumptions are consequential.
This does not mean that every original statistical choice must be copied. An original analysis may contain errors or weaknesses. Researchers can distinguish a prespecified replication analysis from additional corrected or sensitivity analyses, explaining how each informs the original claim.
Deciding which statistical procedures should remain consistent requires attention to both comparability and statistical validity.
Which Procedures May Be Adapted?
| Procedure |
Typical Priority |
Reason |
| Eligibility criteria |
Preserve closely |
Defines the relevant participant population. |
| Intervention content and dosage |
Preserve closely |
Defines the treatment being evaluated. |
| Comparison condition |
Preserve closely |
Defines the contrast underlying the effect. |
| Primary outcome and scoring |
Preserve closely |
Maintains the operational meaning of the result. |
| Assessment timing |
Preserve when substantively relevant |
May determine which stage of an effect is observed. |
| Instructions and stimuli |
Evaluate for equivalence |
May influence behavior or the intended manipulation. |
| Recruitment administration |
Adapt cautiously |
Different methods may produce different selection patterns. |
| Software or equipment |
Adapt if equivalence is supported |
Technical substitutions may preserve the substantive operation. |
| Documentation and data security |
May improve |
Can strengthen transparency and protection without altering the tested claim. |
These priorities are conditional, not universal. In a reaction-time experiment, hardware latency may be critical. In an educational intervention, the learning platform's brand may be incidental while its feedback functions are essential.
How Should Researchers Handle Missing or Poorly Reported Procedures?
Published articles often omit details necessary to reproduce a study faithfully. Researchers should examine supplementary files, registered protocols, repositories, instrument documentation, and related publications. Contacting the original authors may resolve important uncertainties.
When details remain unavailable, distinguish documented procedures from reconstructed assumptions. A plausible reconstruction is not equivalent to confirmation of what originally occurred.
Uncertainty about essential procedures should be considered when describing the replication and interpreting its findings. The appropriate response may be to narrow the replication claim or design a complementary study rather than assert methodological equivalence without evidence.
How Should Procedural Changes Be Documented?
Prepare a procedure-by-procedure comparison before data collection. For each feature, record the original method, the replication method, the evidence supporting equivalence, and the reason for any difference.
Classify changes according to whether they are administrative, implementation-related, measurement-related, or substantive alterations to the research design. This classification is an organizational aid, not a universal methodological standard.
Where possible, preregister the intended procedures and analysis plan. Record subsequent deviations with their timing and rationale, distinguishing planned adaptations from changes introduced after observing data.
Researchers considering improvements to original procedures should explain how those improvements preserve or alter the intended scientific test.