What Information Is Missing From the Paper That You Need to Judge the Study?

What do you still need to know before you can judge the study?

Critical appraisal often focuses on information that is present: the sample, intervention, outcome, effect estimate, confidence interval, or statistical model. Yet sometimes the most consequential observation is simpler: the paper has not told you enough.

Perhaps you cannot determine how participants were selected, what the control group actually received, how an outcome was defined, why the analytic sample became smaller, whether an analysis was prespecified, or how missing data were handled. The appropriate response is not to fill the gap with whatever seems methodologically plausible. First identify exactly what information is missing and what judgment that missing information prevents you from making.

Identify the missing detail and the appraisal question it blocks

How to recognize missing information that actually matters

Incomplete reporting and poor methodology are not the same thing

A paper may omit the method used to generate a random allocation sequence. That omission prevents you from judging the procedure confidently, but it does not establish that the sequence was generated improperly.

Conversely, a paper can report a flawed procedure perfectly clearly. Excellent reporting makes a methodological problem visible; it does not make the method good.

Maintaining this distinction prevents two opposite errors: giving researchers the benefit of procedures they never reported and accusing them of methodological failures that the available evidence does not establish.

Start by asking what judgment you are trying to make

Not every omitted detail deserves equal attention. The missing information matters because of the decision it prevents.

If you are judging whether a randomized comparison was protected from predictable allocation, you need information about sequence generation and allocation concealment. If you are judging applicability, you need information about participants, settings, eligibility, and intervention context. If you are judging a regression result, you may need the analysis population, variable definitions, adjustment strategy, and missing-data handling.

This keeps the exercise focused. Otherwise, critical appraisal can become an archaeological expedition for every sentence the authors might conceivably have written.

Can you reconstruct the research question?

You should be able to determine what the study was actually trying to answer. If the objective is vague and the methods support several different plausible questions, the study's purpose may remain insufficiently specified.

Missing information might include which objective was primary, whether hypotheses were directional, which outcome corresponded to the principal question, or whether the study was confirmatory or exploratory.

If you cannot state the empirical question without inventing substantial details, record that uncertainty before moving further into the appraisal.

Can you determine who was actually studied?

Knowing the sample size is not enough. To understand what population was actually studied, you may need eligibility criteria, recruitment locations, sampling frames, dates, response rates, and participant-flow information.

STROBE asks observational-study reports to describe the setting, locations, relevant dates, eligibility criteria, sources and methods of participant selection, and numbers at relevant stages. These details allow readers to reconstruct where the evidence came from rather than treating a broad population label as self-explanatory.

Can you determine how participants were selected?

A statement such as “500 adults participated” leaves unanswered how those 500 emerged from the wider population.

If the paper does not explain how participants were actually selected, you may be unable to judge sampling coverage, self-selection, recruitment mechanisms, nonresponse, or other processes relevant to the inference.

Look for the recruitment source, sampling procedure, eligibility screening, invitation process, and numbers moving through each stage.

Can you determine what was actually examined?

Broad labels can conceal crucial methodological details. “Exercise intervention,” “AI-assisted learning,” “social-media exposure,” or “usual care” may be insufficient descriptions.

For exposures, you may need the operational definition, measurement method, exposure window, categories, and timing. For interventions, you may need content, dose, provider, delivery mode, modifications, adherence, and fidelity. For qualitative research, you may need a clearer description of the phenomenon, context, sampling, data generation, and analytic process.

CONSORT 2025 requires sufficient description of interventions and comparators to allow replication and directs readers to additional intervention materials when relevant.

Can you identify the actual comparison?

A “control group” is not sufficiently informative if you do not know what control participants experienced. Likewise, “unexposed” may represent a particular reference category rather than literal absence of exposure.

If you cannot determine what was actually compared, you cannot interpret the effect estimate precisely.

This can be especially consequential when usual care varies substantially across settings or when observational categories are created using thresholds that the paper does not report.

Can you completely define the outcome?

An outcome requires more than a broad label. You may need to know the measurement instrument, event definition, analysis metric, aggregation method, threshold, assessor, and when the outcome was measured.

CONSORT-Outcomes guidance emphasizes specifying the measurement variable, participant-level analysis metric, method of aggregation, and time point, along with details such as cutoff values and composite components when applicable.

If the paper merely says that “learning improved” without explaining how learning was operationalized, you cannot determine exactly what empirical outcome supports that statement.

Can you reconstruct the study design?

Sometimes the authors give you a label but not enough information to verify it.

A study described as randomized should report enough about allocation for you to understand how randomization occurred. A longitudinal study should make the timing and repeated observations clear. A case-control study should allow you to see how cases and controls were selected.

If the defining architecture remains uncertain, describe what you can establish and identify what prevents you from determining the study design that was actually used.

Can you determine what data produced the result?

The paper may report 1,000 recruited participants while a regression table quietly uses 713 observations. If the reason is unexplained, that is consequential missing information.

For each major result, you should ideally be able to identify the analysis-specific sample, variables, transformations, exclusions, time points, and handling of missing data. CONSORT 2025 specifically requires trial reports to define who is included in each analysis and how missing data were handled.

If you cannot reconstruct what data were actually analyzed, your interpretation of the estimate remains incomplete.

Can you identify the primary analysis?

A paper containing many models should make clear which analysis carries the principal inferential role. If you cannot tell whether the main result comes from the prespecified primary model, an adjusted alternative, a subgroup, or a later exploratory analysis, the analytical hierarchy is unclear.

CONSORT 2025 asks authors to report statistical methods for primary and secondary outcomes and methods for additional analyses while distinguishing prespecified from post hoc analyses.

This matters because otherwise a striking result can become prominent without readers knowing where it stood in the original analytical plan.

Can you determine what happened to missing participants and observations?

Missingness itself is not unusual. What matters is whether you know enough to evaluate it.

You may need the number missing, reasons for missingness, whether losses differed between groups, which variables were affected, what assumptions the analytical method requires, and whether sensitivity analyses were performed.

If these details are absent, avoid declaring either that missing data caused bias or that they were harmless. The defensible conclusion may simply be that the available report does not permit a confident judgment.

Can you tell what was prespecified?

The final publication may not reveal when an outcome, subgroup, exclusion rule, or model was chosen. When this chronology matters, the paper alone may be insufficient.

CONSORT 2025 requires information on where the protocol and statistical analysis plan can be accessed and asks trial reports to disclose important changes, including outcomes or analyses that were not prespecified.

If you cannot verify what was prespecified and what was decided later, label the chronology as uncertain rather than guessing.

Can you reproduce the denominator behind each result?

Effect estimates, percentages, adverse-event rates, response rates, and subgroup findings should have identifiable denominators. If 42% improved, 42% of whom?

A denominator may change because of missing outcomes, different follow-up times, subgroup restrictions, exclusions, or analysis-specific requirements. CONSORT 2025 explicitly requires participant numbers for the primary analysis and losses and exclusions after randomization.

Missing information has different levels of consequence

Some omissions are inconvenient but do not materially affect the judgment you are making. Others make a central inference impossible to evaluate.

A reporting guideline can function as a diagnostic aid

Reporting guidelines such as CONSORT and STROBE can help identify information that should ordinarily be available for particular designs. They are reporting frameworks, not automatic study-quality scoring systems.

If an item is missing, ask why you need it and what appraisal question it affects. This avoids the rather tempting academic pastime of turning a checklist into a numerical quality score merely because Excel permits it.

Turning “the methods are unclear” into specific appraisal questions

Common mistakes when dealing with incomplete reporting

Translate every important gap into a precise uncertainty

Instead of writing “insufficient information” and moving on, complete the sentence: There is insufficient information to determine whether...

That formulation forces you to identify why the omission matters.

This produces a more disciplined appraisal. You separate what the evidence shows, what the report fails to establish, and what additional information could resolve the uncertainty.

Before finishing your appraisal, identify what you still cannot determine

Questions about missing information in research papers

Sometimes the correct appraisal is that the paper does not let you judge

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.