Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

Which Research Procedures Can Be Improved Without Turning a Direct Replication Into a Conceptual Replication?

A direct replication can sometimes improve the original study's procedures without becoming a conceptual replication. Learn how to distinguish technical improvements from substantive methodological changes, evaluate procedural equivalence, and preserve the interpretability of replication findings.

1332
Improving Procedures in Direct Replication Guide 1332 of 1361
01 · The Question

Can You Improve the Original Methods and Still Conduct a Direct Replication?

You are preparing to replicate a published study, but some of its procedures could be improved. The original researchers may have used outdated software, relied on manual data entry, provided limited documentation, or implemented procedures that could now be administered more consistently.

Should you reproduce those methods exactly, including their limitations, or can you introduce improvements while still describing your study as a direct replication?

The difficulty is that not every improvement is methodologically neutral. A change that reduces recording errors may strengthen the study without altering what is tested. Another change, presented as an improvement, might modify the intervention, measurement process, or comparison condition enough to test a different claim.

02 · The Short Answer

Improve Implementation Without Changing the Substantive Test

In Brief

A direct replication can incorporate methodological improvements when those changes preserve the original study's defining operations, target estimand, and scientific claim. Improvements to documentation, data management, implementation consistency, and appropriate bias-control procedures may be compatible with direct replication, provided they do not substantively change the intervention, comparison, measurement, or research conditions.

Whether a modification is acceptable depends on its consequences, not simply its stated purpose. Researchers should assess procedural equivalence before implementation, document consequential differences, and avoid claiming direct comparability when an improvement changes what the study actually tests.

03 · What You Need to Know

Distinguishing Methodological Improvements From Changes to the Research Test

Why Direct Replication Does Not Require Reproducing Every Original Weakness

Direct replication is intended to provide additional evidence about a previous finding under closely corresponding conditions. It is not necessarily an exercise in reproducing every administrative decision, technical limitation, or avoidable error made by the original researchers.

Nosek and Errington (2020) characterize replication in terms of whether a subsequent study can provide diagnostic evidence about an earlier scientific claim. This perspective helps explain why methodological fidelity matters: it supports an interpretable comparison between the original evidence and the new evidence.

Nevertheless, procedural similarity and methodological validity are distinct considerations. A study may closely reproduce the original methods while inheriting serious limitations. Conversely, a carefully justified improvement may strengthen implementation without materially changing the claim being examined.

The challenge is determining which modifications preserve the substantive test and which introduce new operational conditions.

Before considering improvements, researchers should identify the original procedures that define the direct replication. These provide the reference against which proposed changes can be evaluated.

What Makes a Procedural Improvement Compatible With Direct Replication?

A procedural improvement is most compatible with direct replication when it changes how reliably or transparently a procedure is implemented without changing the research operation itself.

For example, replacing manual transcription of questionnaire responses with an electronic system may reduce recording errors while preserving the questionnaire, response options, administration conditions, and scoring rules.

However, administering the same questionnaire through a different medium is not automatically equivalent. The change may affect privacy perceptions, response behavior, accessibility, or completion patterns. The researcher must consider whether these effects are plausible and consequential.

Three distinctions are especially useful:

Implementation improvement A modification intended to execute the same research procedure more consistently, accurately, or transparently.
Substantive methodological change A modification that changes the operationalization, research contrast, measurement, target population, or conditions relevant to the original claim.

These are analytical categories rather than formally standardized replication classifications. Some changes occupy an uncertain middle ground and require empirical or theoretical justification.

Improving Research Documentation and Protocol Transparency

Documentation is among the clearest opportunities for improvement. Researchers may prepare a detailed protocol, publish study materials, preregister hypotheses and analyses, maintain a deviation log, or provide a reproducible account of data processing.

These activities generally improve transparency without changing the intervention or measurement itself.

For example, a replication team could preregister a protocol even if the original study was not preregistered. The new registration would make the team's planned decisions easier to distinguish from subsequent exploratory choices.

Preregistration does not guarantee methodological validity, and it does not retrospectively establish that the original study followed the same decision process. Its contribution is to clarify the replication team's intentions and support more transparent interpretation.

Researchers should also document implementation details that the original publication may have omitted. Where those details are reconstructed rather than confirmed, the uncertainty should be acknowledged.

Improving Data Collection Accuracy and Data Management

Improvements that reduce avoidable recording and processing errors may be compatible with direct replication.

Examples include automated data validation, secure electronic storage, standardized variable naming, documented data-cleaning rules, and independently checked data-entry procedures.

These changes should preserve the original variable definitions and measurement process. An automated system that records the same response more accurately is different from a system that changes the questions, response scale, or timing of measurement.

Researchers should verify that software substitutions do not introduce systematic differences. For reaction-time experiments, for instance, changes in hardware or software latency may affect the outcome even when the displayed task appears identical.

Data-management improvements should also be distinguished from analytical changes. Correcting a data-entry error is not equivalent to introducing a new exclusion rule after examining the results.

Improving Standardization and Implementation Consistency

Researchers may improve staff training, administration manuals, implementation monitoring, and adherence checks to ensure that the intended procedures are delivered consistently.

Suppose an original experiment requires participants to receive identical instructions. A replication could use a standardized script and train research assistants to follow it consistently. Such improvements may reduce unintended variation without changing the instructions themselves.

However, greater standardization is not always neutral. If the original study deliberately allowed instructors to adapt an intervention to individual needs, replacing that flexibility with rigid delivery could alter the intervention.

The relevant distinction is between reducing unintended implementation variability and changing variability that was part of the original research condition.

Can Randomization, Allocation Concealment, and Blinding Be Improved?

Procedures intended to reduce bias deserve particular attention. Researchers may consider more reliable random-sequence generation, stronger allocation concealment, independent outcome assessment, or clearer separation of intervention delivery from outcome evaluation.

These improvements can strengthen internal validity, but their compatibility with direct replication is not automatic.

For example, implementing secure allocation concealment may preserve the intended randomized comparison while reducing the possibility that recruitment decisions are influenced by upcoming assignments. In contrast, introducing participant blinding through a new placebo procedure may change participants' experiences and expectations relative to the original study.

When bias-control improvements change the study conditions, researchers should explain whether the replication remains a close test of the original result or becomes a corrected methodological test with different assumptions.

The original study's flaws should not be treated as requirements merely because they occurred previously. The more consequential the flaw, however, the more important it becomes to distinguish fidelity from correction.

Can the Sample Size Be Increased?

Increasing sample size is often compatible with direct replication because it need not change the intervention, comparison, measurement, or target estimand.

A larger sample can improve precision and, under appropriate assumptions, increase the probability of detecting an effect of a specified magnitude. It may also produce a more stable estimate than a small original study.

Nevertheless, increasing sample size does not automatically improve validity. If the larger sample is recruited from a different population or through a substantially different selection mechanism, comparability may change.

Sample-size planning should be based on a defensible effect-size assumption, the intended analysis, desired precision, and the inferential objective. Simply reproducing the original sample size is not a methodological requirement.

Researchers should separately evaluate whether the original effect estimate is a suitable basis for sample-size planning, particularly when the initial study was small or its estimate was imprecise.

Can Measurement Administration Be Improved Without Changing the Measure?

Some improvements to measurement administration preserve the intended construct and scoring procedure. Examples might include clearer assessor training, more consistent administration conditions, or independent verification of scoring accuracy.

Other changes may affect measurement equivalence. Rewording items, changing response categories, replacing an instrument, modifying the scoring rubric, or altering the follow-up interval can change the meaning or distribution of the outcome.

Even seemingly beneficial modifications require scrutiny. Making an assessment easier to understand may reduce construct-irrelevant confusion, but it may also change item difficulty or the cognitive demands being measured.

Researchers should distinguish improving the reliability of an existing measurement procedure from redefining the outcome.

Decisions about preserving the original measures should be informed by construct validity, scoring comparability, and the role of measurement in the original claim.

Can the Statistical Analysis Be Improved?

Analytical improvements require particularly careful interpretation because different analyses may answer different questions even when they use the same dataset.

Researchers may improve reproducibility by publishing analysis code, independently verifying calculations, specifying missing-data procedures, or conducting prespecified sensitivity analyses.

However, replacing the original primary analysis with another model may change the estimand, assumptions, treatment of missing observations, or interpretation of the result.

A useful approach is to distinguish a replication analysis intended to reproduce the original comparison from a corrected or supplementary analysis intended to address identifiable limitations.

This does not require presenting an invalid original analysis as authoritative. Rather, researchers should explain what each analysis estimates and why differences between the results may arise.

The choice of whether to retain the original statistical analysis should be made with attention to both inferential validity and comparability.

Which Improvements Are Most Likely to Change the Replication's Meaning?

Proposed Change Potential Benefit Implication for Direct Replication
Preregistering the protocol Clarifies planned decisions and analyses. Generally compatible when the substantive procedures remain unchanged.
Automating data-entry checks Reduces transcription and validation errors. Generally compatible if recorded variables and measurement conditions remain equivalent.
Strengthening administrator training Improves consistency. Usually compatible if the original intended procedure is preserved.
Increasing sample size May improve precision. Usually compatible if population and sampling implications are addressed.
Replacing the original questionnaire May improve psychometric properties. Potentially changes outcome operationalization and comparability.
Changing intervention duration May improve feasibility or effectiveness. May change treatment exposure and the effect being tested.
Replacing the comparison condition May create a more demanding comparison. Changes the research contrast and may prevent close replication.
Changing the primary statistical model May address analytical weaknesses. Requires evaluation of estimand and inferential differences.

The table provides conditional judgments, not universal rules. A technical substitution may be consequential in one discipline and negligible in another.

When Does an Improvement Move a Study Toward Conceptual Replication?

A study moves toward conceptual replication when it deliberately changes how the underlying construct, intervention, or theoretical relationship is operationalized while continuing to investigate the broader claim.

For example, replacing retrieval-practice quizzes with a different instructional strategy intended to promote retrieval may test a related theoretical proposition, but it no longer preserves the original intervention exactly.

Similarly, replacing a behavioral outcome with a self-report measure may shift the operational meaning of the dependent variable.

There is no universally accepted numerical threshold for how many modifications convert a direct replication into a conceptual replication. One consequential change may matter more than several administrative changes.

Rather than treating the classification as a binary judgment based on the number of differences, describe the specific dimensions of correspondence and the claims that remain testable.

What If the Original Study Contains a Serious Methodological Flaw?

Some weaknesses cannot be corrected without changing the substantive research conditions. An inadequate comparison group, invalid outcome measure, or confounded intervention may prevent a close replication from providing strong evidence about the intended causal claim.

In such cases, researchers should consider whether a corrected replication, a supplementary robustness study, or a program containing both close and improved tests would be more informative.

Correcting a flaw may improve internal validity while reducing direct comparability. This trade-off should be acknowledged rather than obscured by describing every change as a minor improvement.

Watch Out

Do not assume that an improvement preserves direct replication merely because it increases methodological rigor. A more rigorous intervention, measure, or comparison condition may also constitute a different scientific test.

The consequences of design improvements for replication comparability should be considered before finalizing the protocol.

04 · A Practical Example

Improving Data Collection Without Changing an Educational Experiment

Hypothetical Example

Replicating an Online Feedback Intervention

An original randomized experiment compares automated formative feedback with basic correctness feedback during a four-week programming course. Students complete the same programming assessment seven days after the intervention.

The original researchers collected scores manually, provided limited implementation documentation, and used a relatively small sample. A replication team wants to improve the study without changing its central experimental comparison.

Preserve the intervention and comparison.

The team retains the four-week duration, learning tasks, feedback content, timing, and comparison condition. Both groups receive the same instructional exposure as specified in the original protocol.

Improve data recording.

Assessment scores are entered into a validated electronic system with automated range checks. The assessment items and scoring rubric remain unchanged.

Improve implementation monitoring.

Researchers document whether the intended feedback was delivered, record deviations, and train administrators to follow the original instructions consistently.

Strengthen planning and transparency.

The team determines an appropriate sample size, preregisters the primary comparison, and publishes a detailed protocol describing its methodological decisions.

Evaluate a proposed substantive change.

A team member suggests extending the intervention to eight weeks. Because duration may influence the effect, the team rejects this change for the primary direct replication and considers evaluating it in a separate study.

The first improvements primarily concern execution, documentation, and precision. Extending the intervention, however, would alter treatment exposure and complicate comparison with the original finding.

If the replication produces a different result, the researchers should examine effect estimates, uncertainty, implementation fidelity, and remaining methodological differences. They should not automatically attribute the discrepancy to the improvements.

05 · What Researchers Often Get Wrong

Misconceptions About Improving Direct Replication Procedures

Misconception

Does Any Improvement Automatically Make the Study Conceptual Replication?

No. Improvements to documentation, data verification, or administrative consistency may preserve the original substantive test. Classification depends on what the change does, not whether something changed.

Misconception

Is a More Rigorous Procedure Always Better for Direct Comparability?

Not necessarily. Greater rigor may reduce bias, but a change in measurement, intervention delivery, or comparison conditions can also alter the estimated effect. Validity and comparability must be evaluated separately.

Misconception

Can Researchers Modernize Every Outdated Procedure?

Modernization is not inherently neutral. New software, equipment, or delivery systems may change participant experiences, measurement accuracy, or timing. Equivalence should be established for relevant functions.

Misconception

Does Increasing Sample Size Guarantee a Successful Replication?

No. Larger samples may improve precision but cannot eliminate systematic bias, invalid measurement, or meaningful differences in research conditions. A replication result must be interpreted through its effect estimate and uncertainty.

Misconception

Can Improved Statistical Analysis Replace the Original Analysis Without Explanation?

No. Analytical changes may alter the estimand or assumptions. Researchers should document why the analysis changed and distinguish comparable analyses from corrected or supplementary ones.

Misconception

Can Procedural Equivalence Be Established Simply by Researcher Judgment?

Expert judgment may be necessary, but consequential equivalence claims should be supported by theoretical reasoning, technical checks, pilot evidence, validation studies, or other relevant evidence where feasible.

06 · What This Means for You

Evaluate Improvements According to Their Effects on the Scientific Claim

Before adopting an improvement, specify which original procedure it replaces, what problem it addresses, and whether it could alter participant behavior, treatment exposure, outcome measurement, or the target estimand.

Use the original protocol as the reference point, but do not treat it as an unquestionable standard of validity.

A simple decision framework

If the improvement only strengthens documentation or error prevention
Implement it when appropriate, preserving the original variable definitions and substantive procedures.
If the improvement changes the delivery technology
Evaluate functional equivalence, including relevant timing, presentation, and interaction characteristics.
If the improvement changes an intervention, comparator, or outcome
Treat it as potentially substantive and reconsider whether the study remains a close direct replication.
If the original procedure contains a serious flaw
Prioritize valid and ethical research, document the correction, and qualify direct-comparability claims.

Maintain a change log specifying the original procedure, revised procedure, reason for revision, evidence of equivalence, and implications for interpretation. Where practical, preregister the changes and distinguish them from deviations introduced after data collection begins.

The goal is not to preserve every historical detail. It is to produce an informative test whose relationship to the original evidence can be independently evaluated.

07 · A Quick Checklist

Before Improving Procedures in a Direct Replication

Before adopting a methodological improvement, check:
Identify the original procedure and its role in the research design.
Specify the problem the proposed improvement is intended to address.
Determine whether the change affects the intervention, comparison, population, outcome, or estimand.
Assess whether technical substitutions preserve relevant functional characteristics.
Evaluate whether the change introduces or removes a plausible source of bias.
Support consequential equivalence claims with appropriate evidence where feasible.
Distinguish implementation improvements from substantive design modifications.
Preregister planned changes and primary analyses when appropriate.
Record implementation deviations and explain their implications for comparability.
08 · Frequently Asked Questions

Frequently Asked Questions About Improving Direct Replications

Can a direct replication use newer research software?

Yes, when the software preserves the relevant research operations. Researchers should evaluate differences in stimulus presentation, timing, data capture, scoring, and participant interaction rather than assume equivalence.

Can researchers add preregistration when the original study was not preregistered?

Yes. Preregistration can clarify planned hypotheses, methods, and analyses without changing the substantive research conditions. It should not be interpreted as evidence that the original study followed the same planning process.

Can a direct replication use a larger sample?

Yes. A larger sample is generally compatible with direct replication when relevant population characteristics and procedures remain comparable. Sample size should be justified through the replication's inferential objectives.

Can researchers improve the original intervention instructions?

Possibly, but changes in wording may influence comprehension, expectations, or behavior. Corrections to obvious administrative errors may be relatively minor, while substantive changes to task instructions require careful evaluation.

Can researchers introduce blinding that the original study did not use?

Sometimes. Blinded outcome assessment may reduce detection bias without changing the intervention, but participant blinding or placebo procedures can alter expectations and treatment conditions. The implications depend on the design.

Can an improved questionnaire still support direct replication?

It depends on whether the revised instrument preserves the original construct, operational definition, and score interpretation. Major changes may weaken direct comparability even when the new instrument has stronger psychometric evidence.

Should methodological improvements be reported even when they seem minor?

Yes, particularly when they may influence implementation or interpretation. A transparent comparison allows readers to assess which changes are consequential rather than relying entirely on the researchers' classification.

09 · The Bottom Line

Improve Research Execution While Preserving the Original Test

The Bottom Line

Direct replication can accommodate methodological improvements when they strengthen implementation without substantively changing the original research claim, intervention, comparison, measurement, or target estimand. Improvements to documentation, data accuracy, and procedural consistency are often compatible with this objective.

More consequential changes require explicit justification and may reduce direct comparability, even when they improve validity. Evaluate each modification according to its methodological consequences, document the evidence supporting equivalence, and describe the resulting replication accurately.

10 · Sources and Further Reading

Authoritative Sources on Replication and Methodological Improvement

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes