Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

How Do You Design a Replication Study When the Original Research Methods Were Poorly Reported?

Poorly reported methods can make a replication difficult without necessarily making it impossible. Learn how to recover missing procedures, distinguish documented methods from assumptions, prioritize consequential uncertainties, and design a transparent replication when the original research cannot be reconstructed completely.

1339
Replicating Poorly Reported Research Guide 1339 of 1361
01 · The Question

How Can You Replicate a Study When Important Methods Are Missing?

You have identified a published finding worth replicating, but the methods section leaves important questions unanswered. The authors may describe an intervention without providing its materials, report using a questionnaire without identifying its version, or mention random assignment without explaining how it was implemented.

You understand the original research question, yet you cannot confidently reconstruct what the researchers actually did. Should you make reasonable assumptions, contact the authors, use standard procedures from the literature, or abandon the replication?

The problem is consequential because missing methodological details can affect the intervention, measurement, analysis, and interpretation of the original finding. A replication based on unverified assumptions may test a different procedure while appearing to reproduce the original study.

02 · The Short Answer

Reconstruct What You Can, Document What You Cannot Verify

In Brief

When the original research methods are poorly reported, begin by reconstructing the study from all available documentation, including supplementary materials, protocols, repositories, instrument manuals, and clarification from the original authors. Identify which missing details are essential to the scientific claim, distinguish verified procedures from assumptions, and develop a replication protocol that explicitly documents unresolved uncertainty.

A replication may remain informative despite incomplete reporting, but the strength of its direct-replication claim depends on how confidently the defining procedures can be reconstructed. When essential features cannot be established, a qualified replication or alternative test of the underlying claim may be more defensible than claiming exact methodological correspondence.

03 · What You Need to Know

Reconstructing a Research Design From Incomplete Methodological Information

Why Poor Reporting Creates a Replication Problem

A published methods section should provide sufficient information for readers to understand how evidence was generated. In practice, however, articles may omit details needed to reproduce the research procedures.

Reporting omissions are not necessarily evidence that the original researchers failed to implement the relevant procedures. An article that does not describe allocation concealment, for example, does not establish whether allocation concealment occurred.

The distinction matters because a replication team must avoid converting missing documentation into unsupported claims about the original study's quality.

The National Academies of Sciences, Engineering, and Medicine (2019) distinguish reproducibility and replicability while emphasizing the importance of transparent methods and research practices. Incomplete reporting can obstruct both the computational reproduction of published results and the design of an independent study.

The immediate task is therefore to determine what is known about the original methods, what can reasonably be recovered, and which uncertainties threaten the intended replication.

First Identify the Scientific Claim You Want to Replicate

Before reconstructing every procedural detail, specify the original finding you intend to test.

Identify the research question, primary hypothesis, population, intervention or exposure, comparison condition, outcome, assessment timing, and target estimand.

For example, an educational study may claim that automated formative feedback improves programming performance relative to basic correctness feedback after a four-week intervention.

The relevant replication question is not simply whether automated feedback is beneficial. It concerns the particular intervention, comparator, outcome, and conditions supporting the original claim.

Defining the target finding helps prioritize missing information. An omitted detail about the feedback algorithm may be more consequential than an omitted detail about the room in which researchers held an administrative meeting.

The appropriate level of reconstruction depends on how closely the replication must correspond to the original research design.

Where Should You Look for Missing Methodological Details?

The published article is only one possible source of methodological information.

Researchers should examine the complete publication record before concluding that a procedure cannot be reconstructed.

Potential Source Information It May Provide Important Limitation
Supplementary materials Detailed procedures, instruments, intervention materials, additional analyses. Files may be incomplete or unavailable.
Registered protocols Planned design, eligibility criteria, outcomes, analysis decisions. Planned procedures may differ from actual implementation.
Research repositories Materials, questionnaires, code, data dictionaries, analysis scripts. Deposited files may not fully document how procedures were administered.
Instrument manuals Administration instructions, scoring rules, interpretation guidance. The original study may have used adaptations not described in the manual.
Related publications Earlier descriptions of interventions, measures, or research procedures. Procedures may have changed between studies.
Original authors Clarification of implementation details and access to materials. Memories may be incomplete, and records may no longer exist.

Each source should be evaluated according to what it actually establishes. A protocol can document the intended design, but it does not necessarily prove that every planned procedure was followed.

How Should You Contact the Original Authors?

Contacting the original research team can be useful when essential details remain unclear after examining available documentation.

Requests should be specific and manageable. Rather than asking the authors to explain the entire methodology again, identify the exact missing information needed for replication.

For example, a researcher might ask for the version of an instrument, the intervention instructions, the number and duration of learning activities, or the procedure used to generate random assignments.

Where appropriate, request original materials or documentation rather than relying exclusively on retrospective recollection.

Researchers should recognize that authors may have changed institutions, retired, lost access to historical files, or be unable to share materials because of licensing, confidentiality, or other restrictions.

A lack of response does not establish that the original research was conducted improperly.

Record the date and substance of clarification requests, responses received, and any remaining uncertainties. Respect confidentiality and applicable permissions when incorporating correspondence into the replication documentation.

Which Missing Procedures Matter Most?

Not every omitted detail presents the same threat to replication fidelity.

The priority is to identify whether the missing information affects the original scientific claim or the interpretation of the resulting evidence.

Important categories commonly include participant eligibility, recruitment, intervention content, comparison conditions, measurement, timing, randomization, blinding, exclusions, and statistical analysis.

For an experiment, the exact content of a manipulation may be essential. For an observational study, the definition of exposure and confounding-control strategy may be more consequential.

The following questions can help prioritize missing information:

  • Could different plausible versions of the missing procedure produce different outcomes?
  • Does the procedure define the intervention, exposure, comparator, or measured construct?
  • Could the omission conceal a source of bias or change the target estimand?
  • Would reasonable researchers reconstruct the procedure differently?
  • Can the uncertainty be resolved using independent documentation or empirical checks?

These questions should guide the assessment of which original procedures must be preserved.

Use Reporting Guidelines to Identify Missing Information

Reporting guidelines can help researchers systematically examine the completeness of the original methodological description.

For intervention research, the Template for Intervention Description and Replication, commonly called TIDieR, is particularly relevant.

Hoffmann and colleagues (2014) developed TIDieR to improve the reporting of interventions so that readers can understand, implement, and replicate them. Its 12 items address intervention rationale, materials, procedures, providers, delivery, location, dosage, tailoring, modifications, and fidelity.

The checklist can help identify whether essential intervention characteristics have been reported.

Other reporting guidelines may be relevant depending on the design. CONSORT addresses randomized trials, STROBE addresses observational studies, and PRISMA addresses systematic reviews.

However, reporting guidelines should not be treated as substitutes for a complete research protocol. They identify information that should be reported, but they do not independently establish that a study was methodologically valid.

Researchers should select guidance appropriate to the original design and use it to organize questions about missing information.

How Can TIDieR Help Reconstruct an Intervention?

Intervention descriptions often contain enough information to identify a treatment generally but not enough to reproduce its delivery.

For example, an article may state that students received AI-assisted formative feedback without describing the feedback content, frequency, platform configuration, or degree of instructor involvement.

TIDieR provides a structured way to identify these omissions.

TIDieR Dimension Reconstruction Question
Rationale What was the intended purpose or theoretical basis of the intervention?
Materials What learning resources, instructions, or other materials were used?
Procedures What activities did participants and providers perform?
Provider Who delivered or supported the intervention, and with what preparation?
Delivery Was the intervention individual, group-based, online, or face-to-face?
Location Where did intervention activities occur?
Dosage How often and for how long was the intervention delivered?
Tailoring Were procedures adapted to individual participants?
Modifications Were changes introduced during implementation?
Fidelity How was planned and actual intervention delivery assessed?

This table consolidates related elements for practical reconstruction; it does not replace the complete 12-item TIDieR checklist.

Its usefulness extends beyond identifying missing information. It can help researchers prepare a more complete description of the replication intervention itself.

How Should You Distinguish Verified Procedures From Assumptions?

A reconstruction should not treat every methodological decision as equally certain.

Some details may be directly documented in the original article or protocol. Others may be confirmed through original materials or author correspondence. Still others may be inferred from contextual evidence or selected by the replication team.

A practical reconstruction record can distinguish these sources of information.

Documented procedure A method explicitly described in an identifiable original source.
Externally clarified procedure A method supported by additional documentation or clarification from the original research team.
Reconstructed assumption A procedure selected or inferred by the replication team because the original implementation cannot be verified.

These categories are a practical documentation framework rather than a formal standard for grading replication fidelity.

For each uncertain procedure, record the available evidence, the selected reconstruction, plausible alternatives, and the consequences if the assumption is incorrect.

This allows readers to distinguish what is known about the original study from what the replication team has introduced.

Can You Use Standard Research Procedures to Fill Missing Details?

Sometimes, but standard practice should not be mistaken for evidence of what the original researchers actually did.

Suppose an article reports random assignment but does not explain the randomization procedure. A replication team could implement computer-generated random allocation with appropriate safeguards.

This may be a defensible method for the new study. However, it does not establish that the original researchers used the same procedure.

Similarly, a standard questionnaire administration protocol may provide a reasonable approach when original instructions are unavailable, but it may not reproduce undocumented adaptations.

Researchers should explain that the selected procedure is a defensible reconstruction or methodological choice, not a verified reproduction.

When the choice substantially changes the original research operation, the implications for direct replication must be acknowledged.

Should You Conduct a Pilot Study?

A pilot study can help evaluate whether the reconstructed procedures are feasible, understandable, and technically functional.

For example, pilot participants may reveal that reconstructed instructions are ambiguous, the intervention duration is unrealistic, or the assessment platform does not deliver feedback as intended.

However, a successful pilot does not establish that the reconstructed procedures match the original implementation.

Pilot evidence supports the functioning of the proposed replication protocol. It cannot independently recover historical methodological details that were never documented.

Researchers should also avoid using pilot outcomes to select procedures solely because they reproduce the original effect. Such selection risks building the desired result into the replication design.

Where pilot findings require protocol changes, those changes should be documented before the main study.

What If the Original Measurement Instrument Is Missing?

Missing instruments create a particularly important problem because measurement defines the observable outcome.

Researchers should search supplementary files, instrument repositories, manuals, related publications, and appropriate institutional records. They may also contact the instrument developer or original research team.

If the exact instrument cannot be recovered, an alternative may be necessary.

However, using another instrument with a similar construct label does not establish measurement equivalence. The alternative may differ in content, response processes, scoring, reliability, or sensitivity.

Researchers should assess the available validity evidence and explain whether the alternative supports a close replication of the original operational finding or a broader conceptual test.

The implications of using alternative measures in replication should be considered before data collection.

What If the Original Statistical Analysis Is Incompletely Described?

An article may report the name of a statistical test without providing the full model, covariate set, exclusion rules, missing-data approach, or analysis population.

These omissions can prevent researchers from reconstructing the original estimand and analytical procedure.

Where available, inspect analysis code, supplementary tables, registered plans, and data dictionaries. Published results may sometimes help identify plausible model specifications, but they rarely establish every analytical decision.

If the original approach cannot be reconstructed, the replication team should prespecify a statistically defensible analysis aligned with the original scientific claim.

The report should distinguish confirmed original procedures from analytical decisions made by the replication team.

Where different plausible analyses could materially change the interpretation, sensitivity analyses may be useful.

The decision about whether to reproduce the original statistical analysis should not be resolved by guessing undocumented model details.

Can a Study Still Be Called a Direct Replication When Methods Are Missing?

Possibly, but the description should reflect the degree of correspondence that can be established.

Missing minor administrative details may not prevent a defensible direct replication when the defining procedures are well documented.

By contrast, uncertainty about the intervention content, comparator, primary outcome, or target population may substantially weaken claims of close methodological correspondence.

There is no universally accepted numerical threshold for how much missing information disqualifies a study from being described as a direct replication.

The more informative approach is to identify which features are verified, which are reconstructed, and how the uncertainty affects the original claim.

Nosek and Errington (2020) emphasize the importance of whether replication evidence is diagnostic of the original claim. A study may still provide useful evidence when its procedures cannot be reconstructed exactly, but the scope of that evidence should be stated carefully.

How Should Researchers Handle Several Plausible Reconstructions?

Sometimes the original report permits more than one reasonable interpretation.

For example, an intervention described as providing feedback after each learning activity might have delivered feedback immediately, after a fixed delay, or after instructor review.

If feedback timing could affect learning, these possibilities may lead to different intervention effects.

Researchers should avoid selecting one interpretation without acknowledging alternatives.

Where feasible, the replication design may evaluate more than one plausible implementation. A multi-condition design can help determine whether the finding depends on the uncertain feature.

However, additional conditions increase complexity and may require larger samples. The decision should reflect the scientific importance of the uncertainty and available resources.

When only one reconstruction can be implemented, justify the choice and explain its implications for interpreting the result.

How Does Poor Reporting Affect Sample-Size Planning?

Missing methodological information may make the original effect estimate difficult to interpret.

For example, uncertainty about outcome scoring, clustering, exclusion rules, or the analysis population may affect the estimated effect and its precision.

Researchers should therefore avoid assuming that the original reported effect provides a reliable planning value.

A defensible approach may involve sensitivity calculations using a range of plausible effects and design parameters.

Where important details remain unknown, the replication team should document the assumptions used for sample-size planning and explain how those assumptions affect the study's expected precision or power.

The problem is related to using the original effect estimate to determine replication sample size, particularly when the original estimate is imprecise or its analytical basis is uncertain.

How Should the Replication Protocol Report Uncertainty?

The protocol should contain a clear account of the reconstruction process.

For each consequential procedure, identify the original evidence, unresolved uncertainty, selected replication method, and rationale for the decision.

It is useful to distinguish uncertainty about what originally happened from uncertainty about whether the selected procedure is methodologically valid.

For example, a replication team may be confident that its computer-generated randomization procedure is appropriate while remaining uncertain about the randomization method used in the original study.

These are different questions and should be reported separately.

Preregistration can document reconstruction decisions before outcomes are observed, reducing the risk that ambiguous original methods are interpreted selectively after results become available.

How Should Results Be Interpreted When the Original Methods Remain Uncertain?

When the replication produces a similar effect, the findings may provide additional support for the underlying claim. However, agreement does not establish that the original and replication procedures were identical.

When the replication produces a different effect, the discrepancy may reflect genuine effect variation, sampling uncertainty, methodological differences, or errors in the reconstruction.

The interpretation should therefore consider which uncertain procedures could plausibly influence the outcome.

Researchers should avoid describing the replication as a definitive failure of the original finding when essential conditions cannot be established.

Likewise, numerical agreement should not be treated as proof that every reconstructed assumption was correct.

Watch Out

Do not invent missing methodological details or present reasonable assumptions as verified historical facts. A transparent reconstruction with acknowledged uncertainty is more scientifically defensible than an apparently complete protocol built on undocumented guesses.

If reconstructing the original methods requires substantial redesign, the researcher should consider whether methodological changes reduce the comparability of the findings.

04 · A Practical Example

Reconstructing an Incompletely Reported AI-Assisted Learning Experiment

Hypothetical Example

Replicating an Automated Feedback Intervention With Missing Procedural Details

A published randomized experiment reports that AI-generated formative feedback improved undergraduate programming performance compared with conventional feedback. The article states that the intervention lasted four weeks, but it does not provide the feedback prompts, describe the feedback frequency, identify the precise assessment instrument, or explain how random assignment was implemented.

A replication team wants to test the original finding at another university.

Identify the defining procedures.

The team determines that feedback content, delivery timing, comparison conditions, outcome measurement, and assignment procedures are potentially consequential to the original claim.

Search the available research record.

The researchers examine supplementary files, a linked repository, related publications, and available instrument documentation. They recover the original assessment rubric but not the feedback prompts.

Request clarification.

The team contacts the original authors with specific questions. The authors provide a description of feedback frequency but cannot locate the original prompt files.

Document the reconstruction.

The researchers distinguish the verified assessment and intervention duration from the reconstructed feedback prompts. They explain how the new prompts were developed and why they are considered a plausible operationalization.

Pilot and preregister the protocol.

A pilot assesses whether the reconstructed feedback system functions consistently. The team then preregisters the primary comparison, measurement procedures, and remaining uncertainties before conducting the main study.

Qualify the replication claim.

The researchers describe the study as a replication of the original intervention claim using partially reconstructed procedures. They do not claim that the feedback prompts reproduce the original implementation exactly.

Suppose the replication produces a smaller estimated effect than the original study. The team examines uncertainty and implementation differences, including the reconstructed prompts, before drawing conclusions.

The result may still provide useful evidence about AI-assisted feedback, but it cannot establish how the original effect would have behaved under an exact reproduction of undocumented prompt content.

05 · What Researchers Often Get Wrong

Common Mistakes When Replicating Poorly Reported Research

Misconception

Does Missing Reporting Mean the Original Researchers Did Not Perform the Procedure?

No. A procedure may have been implemented but omitted from the publication. Researchers should distinguish incomplete reporting from evidence of methodological failure.

Misconception

Can Researchers Fill Missing Details With Whatever Seems Standard?

No. Standard procedures may provide defensible choices for a new study, but they do not establish what the original researchers did. Such choices should be identified as reconstructions or adaptations.

Misconception

Does Author Confirmation Guarantee an Exact Reconstruction?

Not necessarily. Clarification may be based on incomplete records or retrospective recollection. Original materials and contemporaneous documentation can provide stronger evidence when available.

Misconception

Must a Replication Be Abandoned When Some Methods Are Missing?

No. The importance of missing information depends on its role in the scientific claim. A qualified replication may remain informative, although essential unresolved details can limit direct-comparability claims.

Misconception

Does a Successful Pilot Prove That the Original Procedure Was Reconstructed Correctly?

No. A pilot can establish feasibility and identify implementation problems, but it cannot independently verify undocumented historical procedures.

Misconception

Does a Similar Replication Result Confirm That All Reconstruction Assumptions Were Correct?

No. Different procedures may produce similar results. Numerical agreement does not establish procedural identity or eliminate uncertainty about the original implementation.

Misconception

Can Researchers Hide Reconstruction Uncertainty to Make the Replication Appear Stronger?

No. Concealing uncertainty undermines the credibility of the replication. Transparent reporting allows readers to evaluate the evidence according to what is actually known.

06 · What This Means for You

Build a Transparent Reconstruction Plan Before Data Collection

Begin with the original scientific claim and identify the procedures required to test it. Search for missing documentation systematically, then classify unresolved details according to their methodological importance.

The central decision is whether the available information supports a close reconstruction or whether the replication must rely on substantive assumptions.

A simple decision framework

If the missing detail is documented elsewhere
Retrieve the source, verify its relationship to the original study, and incorporate the procedure into the replication protocol.
If the original authors can clarify the procedure
Record the clarification, identify its evidentiary basis, and preserve relevant documentation where permitted.
If a minor administrative detail remains unknown
Select a defensible procedure and document the assumption without overstating its significance.
If an essential intervention or measurement detail is missing
Evaluate plausible reconstructions, consider pilot work or alternative conditions, and qualify claims of direct replication.
If the original scientific test cannot be reconstructed adequately
Consider a conceptual replication or complementary study that tests the underlying claim using clearly specified methods.

A useful reconstruction record should identify the original procedure, available evidence, confidence in the reconstruction, selected replication method, plausible alternatives, and implications for interpretation.

Confidence judgments should be explained rather than presented as precise numerical probabilities without a defensible basis.

Where possible, preregister the completed reconstruction plan and maintain a record of subsequent deviations.

The objective is not to create an illusion of methodological certainty. It is to produce a study whose procedures and limitations can be independently evaluated.

07 · A Quick Checklist

Before Replicating a Study With Incomplete Methods

Before finalizing the reconstructed protocol, check:
Identify the precise original finding, outcome, comparison, population, and target estimand.
Review the full article, supplementary files, registered protocols, and available repositories.
Use an appropriate reporting guideline, such as TIDieR for interventions, to identify missing information.
Contact the original authors with specific questions when important details remain unresolved.
Distinguish documented procedures, externally clarified procedures, and reconstructed assumptions.
Prioritize missing details according to their implications for the scientific claim and risk of bias.
Evaluate plausible alternative reconstructions for consequential uncertainties.
Pilot reconstructed procedures when feasibility or implementation quality is uncertain.
Preregister the replication protocol and document remaining uncertainty where appropriate.
Describe the replication according to the degree of methodological correspondence actually supported by evidence.
08 · Frequently Asked Questions

Frequently Asked Questions About Replicating Poorly Reported Studies

Can I replicate a study if the methods section is incomplete?

Yes, depending on which details are missing. Search additional sources, identify consequential uncertainties, and document any reconstructed procedures. Essential missing information may limit claims of direct replication.

Should I contact the original authors before replicating their study?

Contacting the authors is often useful when important details cannot be recovered from publications, protocols, or repositories. Ask specific questions and document the clarification received.

What if the original authors do not respond?

Continue examining available evidence, document the unanswered questions, and evaluate whether a defensible reconstruction is possible. A lack of response should not be interpreted as evidence of misconduct.

Can I use a standard procedure when the original method is unknown?

Yes, if it is appropriate for the new study, but identify it as a methodological choice rather than a verified original procedure. Explain whether it affects comparability.

Can a poorly reported study still be directly replicated?

Possibly. A close replication may be defensible when the defining operations can be reconstructed from other evidence. When essential procedures remain unknown, the replication claim should be qualified.

Which reporting guideline should I use to reconstruct an intervention?

TIDieR is particularly useful for identifying missing intervention details. Other guidelines, including CONSORT and STROBE, may help evaluate reporting completeness for specific study designs.

Can a pilot study establish that my reconstruction matches the original?

No. A pilot can evaluate feasibility, implementation, and measurement procedures, but it cannot independently confirm what occurred in the original study.

What if different plausible reconstructions produce different results?

That possibility may indicate that the missing procedure is scientifically consequential. Where feasible, evaluate alternative implementations or conduct sensitivity analyses, and explain the resulting uncertainty.

09 · The Bottom Line

Transparent Reconstruction Is More Valuable Than Unsupported Precision

The Bottom Line

A replication study can remain informative when the original methods are poorly reported, but researchers must reconstruct procedures systematically, distinguish verified information from assumptions, and acknowledge uncertainty about consequential methodological details. The strength of a direct-replication claim depends on the correspondence that can actually be established.

Use available documentation, reporting guidelines, author clarification, and appropriate pilot work to develop a defensible protocol. When essential details cannot be recovered, qualify the replication's scope rather than presenting an uncertain reconstruction as an exact reproduction of the original study.

10 · Sources and Further Reading

Authoritative Sources on Reporting and Reconstructing Research Methods

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes