01 · The Question
Could the Studies Disagree Because They Treated Missing Observations Differently?
Participants miss follow-up visits. Survey respondents skip questions. Students leave a study before the final assessment. Medical records contain gaps. Missing data are common, but researchers still have to decide what to do when the values needed for an analysis are unavailable.
Those decisions can matter. One study may analyze only participants with observed outcomes. Another may replace missing values using an imputation procedure. A third may use a statistical model that incorporates assumptions about the missing data. Each approach relies on assumptions, whether those assumptions are obvious or not.
When studies produce apparently conflicting results, therefore, compare not only the analyses they performed but also what happened to observations that were missing.
02 · The Short Answer
Yes, Missing-Data Decisions Can Change the Result
In Brief
Yes. Different approaches to missing data can produce different effect estimates, confidence intervals, and sometimes different substantive conclusions because each approach makes assumptions about the unobserved values.
The crucial issue is not simply the percentage of data missing. You need to consider why the values are missing, whether missingness is related to the unobserved outcome, how much information is missing in each group, and whether the conclusions remain stable under plausible alternative assumptions.
03 · What You Need to Know
Why Missing Data Can Change What a Study Appears to Show
Missing data remove information you would have preferred to observe
Suppose a randomized trial begins with 500 participants but has outcome measurements for only 420 at the final assessment. The remaining 80 participants do not simply disappear from the scientific question. Their outcomes still exist conceptually, but the researchers do not observe them.
Cochrane identifies several reasons for missing outcome data, including withdrawal, loss to follow-up, missed study visits, failure to provide particular measurements, unavailable records, and circumstances in which participants can no longer experience the outcome. Missing outcome data can introduce bias when the observed participants differ systematically from those whose outcomes are unavailable.
Why the data are missing can matter more than how many are missing
A small amount of missing data is not automatically harmless, and a larger amount does not automatically invalidate a study. The consequences depend partly on the mechanism producing the missingness.
Imagine a treatment study in which participants experiencing severe adverse effects are especially likely to withdraw. Their missing outcomes are not merely empty cells. Missingness may contain information about what was happening to those participants. An analysis restricted to people who remained could therefore present an overly favorable picture.
Cochrane's risk-of-bias guidance emphasizes this relationship between missingness and the true unobserved outcome. Bias becomes particularly concerning when missingness depends on the outcome itself or when reasons for missing data differ between intervention groups.
Amount of missing data
How many observations or outcome measurements are unavailable.
Missingness mechanism
The process that caused values to be missing and its relationship with observed information and the unobserved values themselves.
Complete-case analysis uses only observations with the required data
One straightforward strategy is to analyze only participants for whom the variables required for an analysis are observed. This is often called complete-case or available-case analysis, depending on the precise analysis.
Its apparent simplicity can be attractive. The difficulty is that deleting incomplete observations does not make the missing-data problem disappear. If the retained participants systematically differ from those excluded in ways related to the outcome, the estimate can be biased. The analysis also discards information and may reduce precision.
Single imputation fills the gaps but can understate uncertainty
Researchers sometimes substitute one value for each missing observation. Examples include carrying forward a previous measurement, inserting a group mean, predicting a value from a regression model, or assuming a particular outcome.
The problem is not simply that the substituted value might be wrong. Treating an imputed value as though it were actually observed can fail to represent uncertainty about what the missing value could have been. Cochrane notes that simple imputation approaches that do not account for imputation uncertainty will typically produce confidence intervals that are too narrow.
Multiple imputation acknowledges uncertainty in the missing values
Multiple imputation creates multiple plausible versions of the missing data using an imputation model, analyzes each completed dataset, and combines the resulting estimates while accounting for variation between imputations.
This is more sophisticated than filling every gap with one fixed value, but it is not magic. Its validity depends on the imputation model and assumptions about the missing-data process. Variables useful for predicting missingness and the missing values may need to be incorporated appropriately.
A technically impressive imputation procedure cannot recover information that the data and assumptions do not support. Statistics occasionally demands humility before algebra, which is inconvenient but healthy.
Different assumptions can legitimately produce different estimates
Suppose participants with missing outcomes might plausibly have done somewhat worse than observed participants. One analysis assumes their outcomes are similar to those observed. Another explicitly allows the missing participants to have poorer outcomes. The resulting effect estimates may differ because the analyses answer the question under different assumptions about information that was never observed.
This is why simply asking which missing-data method is “best” can be misleading. A method needs assumptions that are defensible for the particular missingness process.
Missing data can affect uncertainty as well as the point estimate
Researchers sometimes focus only on whether the estimated effect becomes larger or smaller. Missing-data handling can also change standard errors and confidence intervals.
Two analyses could produce similar point estimates while differing substantially in precision because one method appropriately incorporates uncertainty about missing values and another effectively treats imputed observations as known. Conclusions based solely on whether a P value crosses a threshold can then appear to conflict even though the underlying estimates remain similar.
Sensitivity analysis asks whether the conclusion depends on the missing-data assumptions
Because the missing values are unobserved, their true values usually cannot be recovered with certainty. A useful question is therefore whether reasonable alternative assumptions would materially change the conclusion.
Cochrane recommends sensitivity analyses to examine the robustness of findings to assumptions about missing outcome data. The logic is straightforward: repeat the analysis under defensible alternatives and see whether the substantive interpretation changes.
Watch Out
No statistical procedure can prove what unobserved outcomes would have been. Missing-data methods formalize assumptions about unavailable information. When conclusions change under plausible alternatives, that uncertainty belongs in the interpretation rather than being hidden behind one preferred analysis.
Different missing-data handling does not automatically explain the conflict
Finding that two studies used different methods is not enough. If missingness was minimal, the alternative methods may produce virtually identical estimates. Conversely, substantial or outcome-related missingness may make analytical choices consequential.
To judge whether missing-data handling plausibly explains disagreement, look at the amount and pattern of missingness, reasons for missing observations, differences between groups, assumptions underlying each method, and results of sensitivity analyses.
04 · A Practical Example
How the Same Missing Participants Can Change the Conclusion
Hypothetical Example
An academic-support program with incomplete final assessments
Imagine a hypothetical randomized study evaluating an academic-support program. Some students do not complete the final assessment, and missingness is more common among students who had been struggling academically.
Analysis A
Includes only students with observed final scores. The program group has a higher mean score than the comparison group.
Analysis B
Uses a principled imputation model incorporating earlier academic performance and other relevant observed information. The estimated difference becomes smaller and more uncertain.
Sensitivity analysis
When missing students are assumed to have somewhat poorer outcomes than predicted under the main model, the estimated advantage becomes smaller still.
Interpretation
The substantive conclusion depends partly on assumptions about students whose final outcomes were not observed.
What to report
Rather than selecting whichever analysis produces the preferred answer, report how robust the conclusion is across plausible assumptions.
The example does not imply that imputation necessarily reduces an effect. Depending on who is missing and why, alternative missing-data approaches could increase, decrease, or leave the estimate largely unchanged.
06 · What This Means for You
How to Decide Whether Missing-Data Handling Explains the Conflict
Start before the statistical method. Determine how much data are missing, where they are missing, why they may be missing, and whether missingness differs across study groups. Only then evaluate the method used to address the problem.
A simple decision framework
If very little outcome information is missing
Different missing-data methods may have little practical effect, although the pattern and reasons for missingness still deserve checking.
If substantial data are missing but results remain similar across plausible analyses
The conclusion is more robust to the missing-data assumptions examined.
If conclusions change materially under plausible assumptions
Treat the result as sensitive to missing-data handling and preserve that uncertainty in the synthesis.
If missingness is plausibly related to the unobserved outcome
Be particularly cautious with analyses that simply discard missing cases or rely on assumptions that do not address this relationship.
If the studies use different methods but both have minimal missingness
Look more closely at other explanations for the conflicting findings.
Also separate missing-data handling from the broader question of study quality. Missing outcome data can create risk of bias, but the implications depend on the circumstances. A study should not automatically be dismissed because some observations are missing, nor should sophisticated statistical terminology substitute for examining the missingness process.
When synthesizing conflicting studies, describe the consequential difference precisely. For example, you might report that the positive finding came from a complete-case analysis with substantial attrition whereas another analysis accounting for missing observations produced a smaller and less precise estimate. That is considerably more informative than saying only that “the studies used different statistics.”
This kind of scrutiny can help determine which evidence deserves more weight without mechanically preferring one statistical method by name.
07 · A Quick Checklist
What to Check When Studies Handle Missing Data Differently
Before attributing disagreement to missing-data handling, check:
Determine how much outcome data are missing overall and within each study group.
Examine reported reasons for dropout, nonresponse, loss to follow-up, or other missing observations.
Check whether missingness could plausibly depend on the unobserved outcome.
Identify whether the analysis uses complete cases, single imputation, multiple imputation, model-based methods, or another strategy.
Identify the assumptions underlying the missing-data method rather than judging it by its label alone.
Look for sensitivity analyses using plausible alternative assumptions.
Compare both point estimates and uncertainty across the alternative analyses.
Treat conclusions that depend heavily on unverifiable assumptions as correspondingly uncertain.
09 · The Bottom Line
Missing Values Are Also Missing Information
The Bottom Line
Different missing-data handling can explain conflicting results when the studies make consequentially different assumptions about observations that were never measured or recorded.
Do not judge the issue by the amount of missing data or the sophistication of the method alone. Examine why data are missing, how each analysis treats them, and whether reasonable alternative assumptions materially change the conclusion.
11 · Cite this Guide
How to Cite This Guide
This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.
Recommended (Field Guide)
APA
MLA
Chicago
Copy Citation
