01 · The Question
What should you conclude when an important part of the study cannot be checked?
You have read the paper carefully. Most of it looks reasonable, but a methodological question remains unanswered. Perhaps participant selection is unclear. Maybe the authors do not adequately explain missing data, measurement procedures, exclusions, allocation, or a crucial analytical decision. You check the supplementary materials and any available documentation, but the answer still is not there.
At this point, the problem is no longer simply poor reporting. You have an evidential decision to make: how much confidence should you place in findings when something important about how those findings were produced cannot be appraised?
02 · The Short Answer
Trust should decrease when consequential uncertainty cannot be resolved
In Brief
You should not give a study full methodological credit for procedures you cannot verify, but missing information does not automatically prove that the study was conducted badly. Your confidence should depend on what is unknown, how consequential it is for the finding you want to use, and what other evidence is available.
Sometimes the appropriate judgment is not “trust” or “do not trust,” but “this aspect remains unclear.” The more directly that uncertainty threatens a central inference, the less weight you should place on that inference.
03 · What You Need to Know
Trust in research should be calibrated rather than all-or-nothing
Start by replacing “Do I trust this paper?” with a more precise question
Trust is often too broad a category for critical appraisal. A single paper can contain claims with different levels of support. Its descriptive findings may be relatively secure while its causal interpretation is uncertain. Its primary outcome may be adequately documented while a secondary analysis depends on poorly reported decisions.
A better question is: How much confidence should I place in this particular finding or inference, given what I can and cannot evaluate?
This shifts appraisal from a verdict on the entire paper to an evaluation of specific claims. It also prevents a common mistake: treating one unresolved methodological issue as either irrelevant or sufficient to invalidate everything.
Missing information is itself a source of uncertainty
Suppose an important procedure is not adequately described. You may not know whether it was conducted rigorously, conducted poorly, or conducted appropriately but simply omitted from the report. Unless additional documentation resolves the issue, that uncertainty is real from the reader's perspective.
Formal approaches to risk-of-bias assessment recognize this problem. Guidance from the Agency for Healthcare Research and Quality notes that evaluating risk of bias depends on having a clear and complete account of study design and conduct, while also cautioning that poor reporting does not necessarily indicate poor conduct. Current CONSORT guidance similarly emphasizes that transparent reporting is necessary for readers to assess the reliability and validity of randomized-trial findings.
Evidence of a flaw
Available information indicates that a method, procedure, or decision creates a methodological problem.
Evidence is insufficient to judge
Available information does not establish whether the relevant method or procedure was adequate.
These situations should not receive identical descriptions. Yet both can affect your confidence because in the second situation you cannot rule out a consequential problem.
Ask whether the missing information could change your interpretation
Not every omission deserves the same response. The key question is what becomes uncertain because of it.
Missing information
Question it may leave unresolved
Potential consequence
Recruitment or selection procedures
How did cases enter the study?
Uncertainty about selection bias or applicability
Measurement procedures
Were the variables assessed in a way that supports the intended interpretation?
Uncertainty about measurement validity or bias
Attrition or missing-data handling
Could missing observations systematically affect the result?
Uncertainty about the direction or magnitude of findings
Allocation procedures in an experiment
Could group assignment have been anticipated or influenced?
Uncertainty about baseline comparability and bias
Analytical decisions
How exactly was the reported estimate produced?
Uncertainty about analytical validity or robustness
Minor descriptive detail
Does this detail materially affect the inference?
Possibly little or no meaningful change in confidence
The important distinction is between missing information that merely frustrates completeness and missing information that blocks evaluation of a plausible threat to the conclusion.
The importance of the unknown depends on the claim
Imagine that the method used to assign participants to groups is unclear. That uncertainty may be highly consequential if the authors make a causal claim based on group comparisons. It may be irrelevant to a completely separate descriptive observation reported in the same paper.
Likewise, uncertainty about the representativeness of a sample primarily affects claims extending findings beyond that sample. It does not necessarily erase every relationship observed within the collected data.
This claim-specific reasoning is central to critical evaluation of a research paper . You are asking not merely whether a weakness exists, but which inference depends on the uncertain methodological feature.
Look beyond the main article before accepting that the information is unavailable
A journal article may be only one component of the research record. Supplementary materials may provide methodological details omitted because of article length or presentation choices. Depending on the study type, protocols, registrations, analysis plans, repositories, or companion publications may provide further information.
Checking these sources is especially worthwhile when the missing detail is central to your appraisal. It also helps distinguish genuinely unavailable information from information that simply is not contained in the main manuscript.
As discussed when considering whether poor reporting prevents adequate appraisal , you should not infer poor conduct solely from incomplete reporting. Additional documentation can sometimes change the judgment substantially.
Uncertainty should not be converted into certainty in either direction
There are two symmetrical errors to avoid.
The first is charitable invention: “They probably followed the standard procedure.” The second is adverse invention: “If they did not report it, they must have done it badly.” Neither conclusion follows from missing information alone.
Watch Out
Do not solve uncertainty by guessing. If an important methodological feature remains unverified after reasonable checking, preserve that uncertainty explicitly in your appraisal and in any conclusions that depend on it.
One unclear feature does not automatically determine the evidential value of the entire study
A paper can have several strengths and still contain an important uncertainty. Conversely, a paper with no obvious catastrophic defect can still provide limited evidence because several smaller weaknesses and uncertainties accumulate.
This matters because critical appraisal should not become a hunt for a single disqualifying defect. A study may have no obvious fatal flaw and still provide weak evidence overall . What matters is the combined effect of the design's strengths, limitations, uncertainties, precision, consistency, and fit between evidence and claim.
04 · A Practical Example
How one missing methodological detail can affect different claims differently
Hypothetical Example
An intervention study with unexplained participant loss
Suppose 300 participants begin a study comparing two educational interventions. The results section analyzes 218 participants, but the paper provides little information about why 82 participants are missing from the final analysis or how attrition differed between groups.
What you know
Among the analyzed participants, one group obtained higher average scores.
What you do not know
You cannot determine whether participants who disappeared from the analysis differed systematically between groups or in ways related to the outcome.
Why it matters
If attrition was related to treatment assignment and likely outcomes, the observed group difference could provide a distorted picture of the effect.
Appropriate appraisal
You do not need to claim that attrition definitely biased the result. You can instead conclude that the missing information prevents adequate evaluation of an important threat to the estimated intervention effect.
If the authors provide supplementary analyses showing the pattern of attrition and demonstrating that the conclusion is reasonably robust to plausible missing-data assumptions, your confidence may increase. The appraisal changes because the evidential uncertainty has changed.
06 · What This Means for You
Let the importance of the missing information determine how cautious you should be
When you cannot fully appraise a study, document the unknown before deciding what it means. Then connect that unknown to the specific conclusion you want to use.
A simple decision framework
If the missing information is peripheral to the finding you need
Note the reporting limitation without automatically discounting an otherwise well-supported finding.
If the missing information concerns a plausible but uncertain source of bias
Reduce confidence appropriately and state what remains unclear rather than claiming that bias definitely occurred.
If the missing information is essential for evaluating the central inference
Avoid treating the study as strong support for that inference until the uncertainty can be resolved.
If additional documentation resolves the uncertainty
Revise the appraisal according to the new evidence rather than retaining the original judgment mechanically.
If the study is only one part of a larger evidence base
Consider its uncertainty alongside other studies rather than asking the single paper to carry more evidential weight than it can support.
This approach also helps when a study has known limitations rather than merely unknown ones. Even seriously limited research can sometimes contribute useful evidence if the claims are appropriately bounded and the limitations do not destroy the particular information being used.
07 · A Quick Checklist
How to appraise a study when key information remains missing
Before relying on the finding, check:
Identify the exact finding or conclusion you want to use rather than judging the paper as a single unit.
Specify exactly what methodological information is unavailable.
Check supplementary files, protocols, registrations, analysis plans, repositories, or related reports where appropriate.
Ask which potential source of bias or uncertainty the missing information prevents you from evaluating.
Determine whether that uncertainty directly affects the specific inference you intend to make.
Do not assume that an unreported procedure was either properly or improperly conducted without evidence.
Reduce confidence when consequential uncertainty remains unresolved rather than converting uncertainty into a categorical verdict.
Consider how the study fits with the wider body of evidence before relying heavily on an uncertain finding.
09 · The Bottom Line
Uncertainty should reduce confidence when it matters, not force an invented verdict
The Bottom Line
If key information is missing, do not give the study full credit for methodological features you cannot evaluate, but do not treat missing information as automatic proof of poor conduct. Calibrate your confidence according to what remains unknown and how strongly that uncertainty affects the specific claim you want to use.
When the uncertainty concerns a central source of bias or a necessary link in the study's inference, substantial caution is warranted. When it concerns a peripheral detail, its effect may be modest. Good appraisal preserves that distinction rather than forcing every paper into “trust” or “do not trust.”
11 · Cite this Guide
How to Cite This Guide
This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.
Recommended (Field Guide)
APA
MLA
Chicago
Copy Citation
