Were Important Groups Excluded From the Study?

Who Is Missing From the Evidence?

A study may describe its participants carefully while giving much less attention to people who never had a realistic opportunity to participate. Some are excluded explicitly through eligibility criteria. Others disappear earlier because recruitment occurs only in certain institutions, languages, locations, technologies, or settings.

Those absences do not automatically invalidate the research. A study often needs boundaries, and some exclusions are scientifically, ethically, or practically justified. The problem arises when a group that matters to the research question or conclusion is absent, yet the resulting evidence is interpreted as though that group had been adequately studied.

Critical appraisal therefore requires looking beyond who appears in the participant table and asking: who is missing, why are they missing, and does their absence matter for the claim being made?

Exclusion Matters When the Missing Group Matters to the Inference

Important Exclusion Is More Than a Missing Demographic Category

Start with the target population rather than a universal diversity checklist

A group can only be meaningfully described as missing relative to the population the study intends to understand. If a study explicitly concerns first-year nursing students, the absence of engineering students is not an exclusion problem. They were never part of the target population.

If the paper instead draws conclusions about university students generally, the composition of the sample deserves a different appraisal. The same sample may be appropriate for the narrower claim and poorly suited to the broader one.

Begin by identifying the population behind the authors' claims. Only then can you decide which missing groups are relevant.

Groups can disappear long before formal eligibility criteria are applied

Explicit exclusion criteria are only one mechanism. A study can effectively exclude people through its sampling frame, recruitment channels, setting, language, technology, scheduling, consent procedures, accessibility requirements, or participation burden.

For this reason, “we did not explicitly exclude them” is not enough. Ask whether members of the group actually had a plausible route into the study.

The most important missing groups are those that could change the answer

Perfect demographic balance is rarely the appropriate standard. Instead, consider whether the missing characteristic could plausibly be associated with the outcome, exposure, measurement process, intervention response, or phenomenon under investigation.

Suppose an educational intervention requires students to use a mobile application. Excluding students without compatible devices could be highly consequential if the intervention is later proposed for all students. Device access may be intertwined with socioeconomic circumstances and may also determine whether the intervention is feasible outside the study.

Likewise, evidence about a medical treatment obtained almost entirely from younger adults may leave an important uncertainty if the treatment is commonly intended for older adults and age could modify benefits, harms, metabolism, or treatment adherence.

The question is not whether every subgroup could theoretically differ. It is whether there is a credible scientific reason that the omitted group's inclusion could affect the interpretation or application of the findings.

Exclusion may prevent you from detecting heterogeneity

An average effect can conceal meaningful differences among groups. If a relevant subgroup is absent or represented by too few participants, researchers may be unable to determine whether the intervention, exposure, measurement instrument, or relationship behaves differently for that group.

This limitation matters even when the overall estimate for the enrolled participants is internally valid. The study may answer “what happened on average among the people studied?” while remaining unable to answer “does the same thing happen for the people who were not studied?”

In clinical research, this concern is reflected in inclusion policies. Current NIH policy requires women and members of racial and ethnic minority groups in NIH-funded clinical research unless a clear and compelling justification supports exclusion, and its inclusion-across-the-lifespan policy similarly expects inclusion across age groups unless scientific or ethical reasons justify otherwise. These are specific NIH requirements rather than universal rules for every study, but they illustrate the scientific concern that evidence should be applicable to populations affected by the condition being studied.

Some exclusions are scientifically or ethically necessary

Researchers may exclude participants because an intervention could be unsafe for them, because the study question specifically concerns a defined population, because valid measurement is not possible under particular conditions, or because including a group would answer a materially different question.

Such restrictions should not be treated as methodological failures merely because they narrow the sample. The appropriate appraisal asks whether the rationale is defensible and whether the conclusions respect the resulting boundary.

This distinction is important when deciding whether eligibility criteria have made the sample too narrow for the conclusions. Narrow eligibility can be entirely legitimate while broad conclusions from that narrow sample are not.

Underrepresentation and complete exclusion are different problems

A group may technically appear in a study but still contribute too little information to answer questions about that group. Five older adults in a sample of 2,000 participants are not equivalent to having no older adults, but their presence alone does not establish adequate evidence about older adults.

The relevant number depends on the study objective. If researchers only need an overall estimate and there is good reason to expect stability across age, proportional representation may not be essential. If they intend to estimate age-specific effects, however, sufficient information within age groups becomes much more important.

Presence should therefore not be confused with meaningful representation for the analysis being claimed.

Exclusion can affect measurement as well as generalization

Sometimes missing groups reveal a problem with the research instrument itself. A survey available only in one language may not merely restrict generalizability; it may indicate that the instrument has not been validated for other language groups. A digital task inaccessible to participants with visual impairments may similarly reveal that the operationalization of the construct depends on abilities unrelated to the construct itself.

In such cases, the sampling issue points to a deeper design question: is the method capable of measuring the intended phenomenon across the population the authors want to discuss?

Exclusion can intersect with selection after eligibility

Even when important groups are formally eligible, they may participate at different rates. Recruitment burden, trust, transportation, compensation, institutional access, or study procedures can make enrollment easier for some groups than others.

This means you should distinguish explicit exclusion from differential participation. The latter may require examining whether selection processes distort the study's findings, not merely checking the eligibility criteria.

The conclusion should not quietly restore people the methods removed

One of the clearest warning signs appears when the methods define a restricted sample but the discussion returns to unrestricted language. A study of English-speaking adults aged 18–40 becomes evidence about “adults.” A study of urban schools becomes evidence about “schools.” A trial excluding patients with major comorbidities becomes evidence about “patients with the disease.”

The broader wording may seem minor, but it changes the population represented by the claim. If excluded groups have not been studied and no adequate external evidence supports extension to them, the conclusion should preserve that uncertainty.

A Digital Intervention That Leaves Some Students Outside the Study

The exclusion is especially important because access is not incidental to the intervention. It determines whether some members of the intended population can use the intervention at all. That makes the missing group relevant not only to statistical generalization but also to practical implementation.

Common Mistakes When Evaluating Excluded Groups

Ask Whether the Missing Group Could Change What the Study Means

When reviewing a study, create a mental comparison between the target population and the people who could realistically enter the study. Do not stop with the demographic table. Look at the recruitment setting, eligibility criteria, language, technology, accessibility, scheduling, and other participation requirements.

The strongest criticism is specific. Instead of writing “the sample lacks diversity,” identify the group that is missing, explain why that group belongs to the relevant population, and show how its absence limits a particular conclusion. That is a methodological argument rather than a demographic headcount.

How to Check Whether Important Groups Were Excluded

Questions About Excluded and Underrepresented Groups

Missing Groups Matter When Their Absence Creates Missing Evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.