When Does Broadening a Sparse Evidence Base Make the Evidence Too Indirect?

How far can you broaden before the evidence stops answering your question?

When direct evidence is scarce, broadening can be sensible. You might include a wider population, accept related outcomes, consider a similar intervention, or draw on studies from another context.

Each decision may be individually defensible. The difficulty appears when those decisions accumulate.

A study might involve a somewhat different population, use a modified intervention, compare it against another standard of care, and measure a related outcome at a shorter follow-up. It is clearly relevant. But does it still provide evidence for your original question, or have you crossed the point where the connection depends on too many uncertain assumptions?

There is no universal distance at which evidence suddenly becomes “too indirect.” The judgment depends on what differs, whether those differences could meaningfully change the result, and what claim you intend the evidence to support.

Indirectness becomes consequential when the inferential gap could change the answer

Indirectness is a relationship between evidence and a question

A study is not inherently direct or indirect. It is direct or indirect relative to the question you want it to answer.

A randomized trial conducted in adults is direct evidence about the adults actually studied. The same trial may provide indirect evidence for children. A study of a six-month intervention is direct evidence for that intervention as delivered, but potentially indirect evidence for a two-week adaptation. A study measuring a biomarker directly answers a question about the biomarker while providing potentially indirect evidence about a clinical outcome that the biomarker is expected to predict.

This is why the target question must remain visible throughout the assessment.

Compare the target question with the evidence actually available

GRADE defines indirectness in terms of mismatches between the target PICO and the PICO represented by the available evidence. Relevant differences may involve the population, intervention, comparator, or outcome. Setting and timing can also matter through those components.

The existence of a mismatch does not automatically make the evidence unusable. GRADE guidance emphasizes whether there are compelling reasons to believe the differences could produce meaningful and systematic differences in relative or absolute effects.

Some differences matter much more than others

Suppose the target population is adults aged 18–65, while a study enrolled adults aged 20–64. Technically, the populations do not match perfectly. The discrepancy is unlikely to matter for most questions.

Now suppose the target population is children with a condition whose treatment response changes substantially with development, while all available studies involve older adults. The population difference could be central.

Indirectness therefore cannot be assessed mechanically. The same apparent difference may be negligible for one question and decisive for another.

The question is always substantive: could this difference change the conclusion?

The claim determines how much indirectness is tolerable

Evidence may be too indirect for one claim while remaining useful for another.

A study of a related intervention might be inadequate for claiming that the target intervention reduces mortality, yet useful for identifying potential adverse effects associated with a shared pharmacological mechanism. Research from another educational context might not establish effectiveness in your institution, but it could inform implementation barriers worth investigating locally.

This means “include or exclude?” is sometimes the wrong question. A better question is “What can this evidence legitimately tell me?”

Evidence can be relevant without being sufficiently direct for the primary conclusion.

Indirectness can accumulate across multiple dimensions

This is particularly important when researchers broaden sparse evidence sequentially.

You might first broaden the population. Then you may decide that a related outcome can also contribute. Finally, you might include related interventions or contexts.

None of these decisions is necessarily wrong. The problem is that each can add another inferential step between the observed evidence and the target question.

By the end, the evidence may still be relevant, but the conclusion depends on several transfer assumptions simultaneously. This cumulative inferential distance should be evaluated explicitly rather than treating each expansion as an isolated decision.

Ask what would have to be true for the transfer to work

A practical way to expose indirectness is to complete this sentence:

“For these studies to answer my target question, I must assume that...”

You may need to assume that age does not modify the effect, that a shorter intervention preserves the active mechanism, that a surrogate accurately reflects the final outcome, and that a different setting does not alter implementation.

Writing those assumptions down can reveal how much inferential work the evidence is being asked to perform.

Then ask how well each assumption is supported. Some may have strong empirical or theoretical support. Others may be plausible but uncertain. Others may amount to little more than hope wearing a methodological name tag.

Look for evidence that direct and indirect studies behave differently

When both more-direct and less-direct studies are available, their results can help evaluate whether the differences matter. Current GRADE guidance recommends examining whether effect estimates from evidence with different degrees of indirectness are concordant.

If more-direct and less-direct evidence consistently point to effects within the same decision-relevant range, concern about indirectness may be reduced. If results differ materially, that discrepancy strengthens concern that the characteristic separating the bodies of evidence matters.

This comparison should still be interpreted carefully. Differences may arise from other study characteristics, risk of bias, random error, or confounding. Concordance is informative, not magical proof of transferability.

Indirectness can affect relative and absolute effects differently

Sometimes a relative effect may transfer reasonably well while the absolute effect does not.

Suppose an intervention has a similar relative effect across populations, but the baseline risk of the outcome differs greatly. The number of people who benefit or experience harm can then differ substantially even if the relative effect remains stable.

GRADE identifies uncertainty about baseline risk as an important source of population indirectness because it affects confidence in absolute effect estimates. In some circumstances, more representative observational data can help estimate baseline risk while trial evidence informs the relative effect.

This is a useful reminder that indirectness is not always solved by accepting or rejecting an entire study. Different pieces of evidence may legitimately contribute different parts of the inference.

Indirectness is not the same as risk of bias

A beautifully conducted study can be highly indirect for your question. Conversely, a perfectly matched population and intervention can be studied with methods that create serious risk of bias.

These are separate concerns.

Risk of bias asks whether the study's estimate is credible for the question it actually studied. Indirectness asks how confidently that evidence can be transferred to the question you want answered.

Do not describe indirect evidence as “low quality” simply because it addresses a neighboring question. Its methods may be excellent. The limitation lies in the inference you want to make from it.

There is no universal cutoff for “too indirect”

You cannot calculate indirectness by assigning one point for a different population, another for a different outcome, and declaring the evidence unusable at three points.

GRADE permits certainty to be rated down according to the severity of indirectness, but that judgment depends on the likelihood that mismatches would result in meaningfully different effects and on how consequential those differences are for the decision or conclusion.

For researchers outside a formal GRADE assessment, the same logic remains useful: the more the conclusion depends on uncertain transfer assumptions, the more carefully the evidence should be separated, qualified, or restricted to a narrower claim.

When several reasonable expansions add up to an unreasonable inference

The evidence has not become worthless. The defensible conclusion has become narrower.

This is often the most useful response to substantial indirectness: preserve what the evidence genuinely contributes while refusing to make it answer a question it did not study.

Indirectness is easy to underestimate when evidence is scarce

Find the strongest conclusion the evidence can support without hiding the inferential gap

When broadening has introduced several differences, do not begin by asking whether the studies must be discarded. Begin by asking what conclusion remains defensible.

This approach matters particularly when direct research is genuinely scarce. Sparse evidence creates pressure to make every available study carry more inferential weight. Your conclusion should move in the opposite direction: as the evidence becomes less direct, the language used to describe what is known should become more precise and restrained.

That is also why avoiding overstatement when only a few studies exist is not merely a writing preference. It is part of representing the structure and uncertainty of the evidence accurately.

Before deciding that broadened evidence is direct enough

Questions about deciding when evidence is too indirect

Evidence becomes too indirect for a claim before it becomes useless

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.