03 · What You Need to Know
Reconstructing a Research Design From Incomplete Methodological Information
Why Poor Reporting Creates a Replication Problem
A published methods section should provide sufficient information for readers to understand how evidence was generated. In practice, however, articles may omit details needed to reproduce the research procedures.
Reporting omissions are not necessarily evidence that the original researchers failed to implement the relevant procedures. An article that does not describe allocation concealment, for example, does not establish whether allocation concealment occurred.
The distinction matters because a replication team must avoid converting missing documentation into unsupported claims about the original study's quality.
The National Academies of Sciences, Engineering, and Medicine (2019) distinguish reproducibility and replicability while emphasizing the importance of transparent methods and research practices. Incomplete reporting can obstruct both the computational reproduction of published results and the design of an independent study.
The immediate task is therefore to determine what is known about the original methods, what can reasonably be recovered, and which uncertainties threaten the intended replication.
First Identify the Scientific Claim You Want to Replicate
Before reconstructing every procedural detail, specify the original finding you intend to test.
Identify the research question, primary hypothesis, population, intervention or exposure, comparison condition, outcome, assessment timing, and target estimand.
For example, an educational study may claim that automated formative feedback improves programming performance relative to basic correctness feedback after a four-week intervention.
The relevant replication question is not simply whether automated feedback is beneficial. It concerns the particular intervention, comparator, outcome, and conditions supporting the original claim.
Defining the target finding helps prioritize missing information. An omitted detail about the feedback algorithm may be more consequential than an omitted detail about the room in which researchers held an administrative meeting.
The appropriate level of reconstruction depends on how closely the replication must correspond to the original research design.
Where Should You Look for Missing Methodological Details?
The published article is only one possible source of methodological information.
Researchers should examine the complete publication record before concluding that a procedure cannot be reconstructed.
| Potential Source |
Information It May Provide |
Important Limitation |
| Supplementary materials |
Detailed procedures, instruments, intervention materials, additional analyses. |
Files may be incomplete or unavailable. |
| Registered protocols |
Planned design, eligibility criteria, outcomes, analysis decisions. |
Planned procedures may differ from actual implementation. |
| Research repositories |
Materials, questionnaires, code, data dictionaries, analysis scripts. |
Deposited files may not fully document how procedures were administered. |
| Instrument manuals |
Administration instructions, scoring rules, interpretation guidance. |
The original study may have used adaptations not described in the manual. |
| Related publications |
Earlier descriptions of interventions, measures, or research procedures. |
Procedures may have changed between studies. |
| Original authors |
Clarification of implementation details and access to materials. |
Memories may be incomplete, and records may no longer exist. |
Each source should be evaluated according to what it actually establishes. A protocol can document the intended design, but it does not necessarily prove that every planned procedure was followed.
How Should You Contact the Original Authors?
Contacting the original research team can be useful when essential details remain unclear after examining available documentation.
Requests should be specific and manageable. Rather than asking the authors to explain the entire methodology again, identify the exact missing information needed for replication.
For example, a researcher might ask for the version of an instrument, the intervention instructions, the number and duration of learning activities, or the procedure used to generate random assignments.
Where appropriate, request original materials or documentation rather than relying exclusively on retrospective recollection.
Researchers should recognize that authors may have changed institutions, retired, lost access to historical files, or be unable to share materials because of licensing, confidentiality, or other restrictions.
A lack of response does not establish that the original research was conducted improperly.
Record the date and substance of clarification requests, responses received, and any remaining uncertainties. Respect confidentiality and applicable permissions when incorporating correspondence into the replication documentation.
Which Missing Procedures Matter Most?
Not every omitted detail presents the same threat to replication fidelity.
The priority is to identify whether the missing information affects the original scientific claim or the interpretation of the resulting evidence.
Important categories commonly include participant eligibility, recruitment, intervention content, comparison conditions, measurement, timing, randomization, blinding, exclusions, and statistical analysis.
For an experiment, the exact content of a manipulation may be essential. For an observational study, the definition of exposure and confounding-control strategy may be more consequential.
The following questions can help prioritize missing information:
- Could different plausible versions of the missing procedure produce different outcomes?
- Does the procedure define the intervention, exposure, comparator, or measured construct?
- Could the omission conceal a source of bias or change the target estimand?
- Would reasonable researchers reconstruct the procedure differently?
- Can the uncertainty be resolved using independent documentation or empirical checks?
These questions should guide the assessment of which original procedures must be preserved.
Use Reporting Guidelines to Identify Missing Information
Reporting guidelines can help researchers systematically examine the completeness of the original methodological description.
For intervention research, the Template for Intervention Description and Replication, commonly called TIDieR, is particularly relevant.
Hoffmann and colleagues (2014) developed TIDieR to improve the reporting of interventions so that readers can understand, implement, and replicate them. Its 12 items address intervention rationale, materials, procedures, providers, delivery, location, dosage, tailoring, modifications, and fidelity.
The checklist can help identify whether essential intervention characteristics have been reported.
Other reporting guidelines may be relevant depending on the design. CONSORT addresses randomized trials, STROBE addresses observational studies, and PRISMA addresses systematic reviews.
However, reporting guidelines should not be treated as substitutes for a complete research protocol. They identify information that should be reported, but they do not independently establish that a study was methodologically valid.
Researchers should select guidance appropriate to the original design and use it to organize questions about missing information.
How Can TIDieR Help Reconstruct an Intervention?
Intervention descriptions often contain enough information to identify a treatment generally but not enough to reproduce its delivery.
For example, an article may state that students received AI-assisted formative feedback without describing the feedback content, frequency, platform configuration, or degree of instructor involvement.
TIDieR provides a structured way to identify these omissions.
| TIDieR Dimension |
Reconstruction Question |
| Rationale |
What was the intended purpose or theoretical basis of the intervention? |
| Materials |
What learning resources, instructions, or other materials were used? |
| Procedures |
What activities did participants and providers perform? |
| Provider |
Who delivered or supported the intervention, and with what preparation? |
| Delivery |
Was the intervention individual, group-based, online, or face-to-face? |
| Location |
Where did intervention activities occur? |
| Dosage |
How often and for how long was the intervention delivered? |
| Tailoring |
Were procedures adapted to individual participants? |
| Modifications |
Were changes introduced during implementation? |
| Fidelity |
How was planned and actual intervention delivery assessed? |
This table consolidates related elements for practical reconstruction; it does not replace the complete 12-item TIDieR checklist.
Its usefulness extends beyond identifying missing information. It can help researchers prepare a more complete description of the replication intervention itself.
How Should You Distinguish Verified Procedures From Assumptions?
A reconstruction should not treat every methodological decision as equally certain.
Some details may be directly documented in the original article or protocol. Others may be confirmed through original materials or author correspondence. Still others may be inferred from contextual evidence or selected by the replication team.
A practical reconstruction record can distinguish these sources of information.
Documented procedure
A method explicitly described in an identifiable original source.
Externally clarified procedure
A method supported by additional documentation or clarification from the original research team.
Reconstructed assumption
A procedure selected or inferred by the replication team because the original implementation cannot be verified.
These categories are a practical documentation framework rather than a formal standard for grading replication fidelity.
For each uncertain procedure, record the available evidence, the selected reconstruction, plausible alternatives, and the consequences if the assumption is incorrect.
This allows readers to distinguish what is known about the original study from what the replication team has introduced.
Can You Use Standard Research Procedures to Fill Missing Details?
Sometimes, but standard practice should not be mistaken for evidence of what the original researchers actually did.
Suppose an article reports random assignment but does not explain the randomization procedure. A replication team could implement computer-generated random allocation with appropriate safeguards.
This may be a defensible method for the new study. However, it does not establish that the original researchers used the same procedure.
Similarly, a standard questionnaire administration protocol may provide a reasonable approach when original instructions are unavailable, but it may not reproduce undocumented adaptations.
Researchers should explain that the selected procedure is a defensible reconstruction or methodological choice, not a verified reproduction.
When the choice substantially changes the original research operation, the implications for direct replication must be acknowledged.
Should You Conduct a Pilot Study?
A pilot study can help evaluate whether the reconstructed procedures are feasible, understandable, and technically functional.
For example, pilot participants may reveal that reconstructed instructions are ambiguous, the intervention duration is unrealistic, or the assessment platform does not deliver feedback as intended.
However, a successful pilot does not establish that the reconstructed procedures match the original implementation.
Pilot evidence supports the functioning of the proposed replication protocol. It cannot independently recover historical methodological details that were never documented.
Researchers should also avoid using pilot outcomes to select procedures solely because they reproduce the original effect. Such selection risks building the desired result into the replication design.
Where pilot findings require protocol changes, those changes should be documented before the main study.
What If the Original Measurement Instrument Is Missing?
Missing instruments create a particularly important problem because measurement defines the observable outcome.
Researchers should search supplementary files, instrument repositories, manuals, related publications, and appropriate institutional records. They may also contact the instrument developer or original research team.
If the exact instrument cannot be recovered, an alternative may be necessary.
However, using another instrument with a similar construct label does not establish measurement equivalence. The alternative may differ in content, response processes, scoring, reliability, or sensitivity.
Researchers should assess the available validity evidence and explain whether the alternative supports a close replication of the original operational finding or a broader conceptual test.
The implications of using alternative measures in replication should be considered before data collection.
What If the Original Statistical Analysis Is Incompletely Described?
An article may report the name of a statistical test without providing the full model, covariate set, exclusion rules, missing-data approach, or analysis population.
These omissions can prevent researchers from reconstructing the original estimand and analytical procedure.
Where available, inspect analysis code, supplementary tables, registered plans, and data dictionaries. Published results may sometimes help identify plausible model specifications, but they rarely establish every analytical decision.
If the original approach cannot be reconstructed, the replication team should prespecify a statistically defensible analysis aligned with the original scientific claim.
The report should distinguish confirmed original procedures from analytical decisions made by the replication team.
Where different plausible analyses could materially change the interpretation, sensitivity analyses may be useful.
The decision about whether to reproduce the original statistical analysis should not be resolved by guessing undocumented model details.
Can a Study Still Be Called a Direct Replication When Methods Are Missing?
Possibly, but the description should reflect the degree of correspondence that can be established.
Missing minor administrative details may not prevent a defensible direct replication when the defining procedures are well documented.
By contrast, uncertainty about the intervention content, comparator, primary outcome, or target population may substantially weaken claims of close methodological correspondence.
There is no universally accepted numerical threshold for how much missing information disqualifies a study from being described as a direct replication.
The more informative approach is to identify which features are verified, which are reconstructed, and how the uncertainty affects the original claim.
Nosek and Errington (2020) emphasize the importance of whether replication evidence is diagnostic of the original claim. A study may still provide useful evidence when its procedures cannot be reconstructed exactly, but the scope of that evidence should be stated carefully.
How Should Researchers Handle Several Plausible Reconstructions?
Sometimes the original report permits more than one reasonable interpretation.
For example, an intervention described as providing feedback after each learning activity might have delivered feedback immediately, after a fixed delay, or after instructor review.
If feedback timing could affect learning, these possibilities may lead to different intervention effects.
Researchers should avoid selecting one interpretation without acknowledging alternatives.
Where feasible, the replication design may evaluate more than one plausible implementation. A multi-condition design can help determine whether the finding depends on the uncertain feature.
However, additional conditions increase complexity and may require larger samples. The decision should reflect the scientific importance of the uncertainty and available resources.
When only one reconstruction can be implemented, justify the choice and explain its implications for interpreting the result.
How Does Poor Reporting Affect Sample-Size Planning?
Missing methodological information may make the original effect estimate difficult to interpret.
For example, uncertainty about outcome scoring, clustering, exclusion rules, or the analysis population may affect the estimated effect and its precision.
Researchers should therefore avoid assuming that the original reported effect provides a reliable planning value.
A defensible approach may involve sensitivity calculations using a range of plausible effects and design parameters.
Where important details remain unknown, the replication team should document the assumptions used for sample-size planning and explain how those assumptions affect the study's expected precision or power.
The problem is related to using the original effect estimate to determine replication sample size, particularly when the original estimate is imprecise or its analytical basis is uncertain.
How Should the Replication Protocol Report Uncertainty?
The protocol should contain a clear account of the reconstruction process.
For each consequential procedure, identify the original evidence, unresolved uncertainty, selected replication method, and rationale for the decision.
It is useful to distinguish uncertainty about what originally happened from uncertainty about whether the selected procedure is methodologically valid.
For example, a replication team may be confident that its computer-generated randomization procedure is appropriate while remaining uncertain about the randomization method used in the original study.
These are different questions and should be reported separately.
Preregistration can document reconstruction decisions before outcomes are observed, reducing the risk that ambiguous original methods are interpreted selectively after results become available.
How Should Results Be Interpreted When the Original Methods Remain Uncertain?
When the replication produces a similar effect, the findings may provide additional support for the underlying claim. However, agreement does not establish that the original and replication procedures were identical.
When the replication produces a different effect, the discrepancy may reflect genuine effect variation, sampling uncertainty, methodological differences, or errors in the reconstruction.
The interpretation should therefore consider which uncertain procedures could plausibly influence the outcome.
Researchers should avoid describing the replication as a definitive failure of the original finding when essential conditions cannot be established.
Likewise, numerical agreement should not be treated as proof that every reconstructed assumption was correct.
Watch Out
Do not invent missing methodological details or present reasonable assumptions as verified historical facts. A transparent reconstruction with acknowledged uncertainty is more scientifically defensible than an apparently complete protocol built on undocumented guesses.
If reconstructing the original methods requires substantial redesign, the researcher should consider whether methodological changes reduce the comparability of the findings.