03 · What You Need to Know
Heterogeneity can be evidence about conditions, not merely variation around an average
Meta-analysis often focuses attention on the average effect. Yet Cochrane distinguishes the average from the variation around it. When intervention effects differ across studies, researchers should consider whether differences in participants, interventions, outcomes, methods, or other study characteristics could explain that heterogeneity.
Sometimes no convincing explanation emerges. Sometimes the variation points toward an important effect modifier: a characteristic that changes how strongly, or occasionally in what direction, an intervention affects an outcome.
If that characteristic belongs to the surrounding environment, it may help identify a contextual mechanism.
A contextual mechanism is more specific than saying “context matters”
Context is an enormous category. Country, institution, health system, school type, resource environment, policy regime, organizational culture, infrastructure, and implementation capacity can all be described as context.
A useful mechanistic explanation goes further. It identifies a specific characteristic and proposes how that characteristic changes the pathway connecting an intervention or exposure to an outcome.
Contextual difference
Settings differ in a characteristic such as staffing, infrastructure, regulation, resources, institutional support, or social conditions.
Contextual mechanism
The characteristic plausibly changes a process required for the intervention, exposure, or phenomenon to produce its observed effect.
“The intervention works differently by country” therefore remains mostly descriptive. “The intervention appears more effective where trained personnel provide continuing support, which may improve adherence” proposes a mechanism that can potentially be examined in other settings.
Begin with effect modification
Statistically, the relevant idea is interaction or effect modification. Cochrane defines this as variation in intervention effects according to characteristics of populations or interventions and identifies subgroup analysis and meta-regression as common methods for investigating such patterns.
The contextual characteristic could be categorical, such as whether specialist support is available, or continuous, such as intervention intensity or resource availability.
But finding an association between a study characteristic and effect size does not automatically establish that the characteristic caused the difference.
First establish that there is a meaningful pattern to explain
Two studies producing different numerical estimates do not necessarily indicate effect modification. Sampling variation produces differences even when underlying effects are similar.
Researchers should examine effect estimates and their uncertainty, the magnitude and direction of heterogeneity, and whether any proposed subgroup difference is supported by a direct statistical comparison. Cochrane specifically warns that statistical significance in one subgroup but not another does not demonstrate a difference between those subgroups.
The substantive importance of the difference matters too. A tiny interaction may be statistically detectable in a large evidence base without meaningfully changing interpretation or decisions.
A mechanism needs a plausible pathway
Suppose digital interventions show larger effects in settings with reliable internet access. That pattern alone identifies an association. A plausible pathway might be that reliable connectivity increases exposure to the intervention, which improves adherence, allowing participants to receive the intended intervention dose.
The proposed chain is now more informative:
Context
Reliable connectivity is available.
Process
Participants can access the intervention consistently.
Intermediate consequence
Intervention exposure and adherence increase.
Outcome
The intervention has greater opportunity to produce its intended effect.
Evidence supporting several links in this chain is more compelling than merely observing that studies from well-connected settings report larger effects.
Prespecified explanations are generally more credible
Heterogeneous evidence can generate dozens of possible explanations. Once researchers know which studies have large and small effects, finding some characteristic that separates them is often surprisingly easy.
Cochrane therefore recommends prespecifying a small number of scientifically justified potential effect modifiers whenever possible. Post hoc exploration can still be valuable, but it should usually be presented as hypothesis-generating rather than as a definitive explanation.
Watch Out
If you examine enough contextual characteristics after seeing the results, one of them may appear to explain the heterogeneity by chance. A compelling retrospective pattern is a reason to investigate a mechanism, not automatically evidence that you have discovered one.
Between-study comparisons are observational
This limitation is fundamental.
Suppose studies conducted in high-resource settings show larger effects than studies in low-resource settings. The studies may also differ in participant characteristics, intervention intensity, publication year, study quality, comparator conditions, follow-up, or outcome measurement.
Participants were not randomized to those studies or settings. Consequently, subgroup comparisons and meta-regressions based on study-level characteristics are observational and vulnerable to confounding. Cochrane explicitly cautions that even a genuine difference between study subgroups is not necessarily caused by the characteristic used to define those subgroups.
This is why a contextual association should not casually become a causal mechanism in the discussion section.
Confounding can produce a convincing but incorrect mechanism
Imagine that every study providing intensive implementation support was also conducted in highly resourced institutions. Effects are larger in those studies.
What explains the difference? Institutional resources? Implementation support? Participant characteristics associated with those institutions? Something else entirely?
If those characteristics move together across the evidence base, aggregate data may not permit them to be disentangled. Meta-regression can investigate multiple characteristics in principle, but systematic reviews often contain too few studies to do this reliably. Cochrane generally advises against meta-regression with fewer than ten studies and notes that even larger numbers may be inadequate when characteristics are unevenly distributed.
Within-study evidence can strengthen the interpretation
A contextual mechanism becomes more convincing when relationships are observed within studies rather than only between different sets of studies.
Suppose a multicenter trial implements the same intervention across facilities with different staffing capacity, and the proposed context-effect relationship appears within that common design. If similar within-study patterns recur across independent studies, confidence in the effect-modification hypothesis can increase.
Cochrane specifically notes that replicated within-study relationships are more reliable than analyses based only on subsets of studies when evaluating participant and intervention characteristics.
They still require careful causal interpretation, but they remove some of the confounding created when entirely different studies are compared.
Country differences should be translated into specific mechanisms
A country can rarely serve as a satisfying mechanism by itself.
If effects differ between countries, ask what differs that could plausibly alter the pathway: staffing, regulation, cultural expectations, baseline services, resources, implementation, language, infrastructure, or another characteristic.
This builds directly on the principle that country differences can become part of the substantive finding when they correspond to meaningful contextual variation.
The more transferable explanation is usually the underlying contextual characteristic rather than the national label.
Resource differences can be mechanistically important
Resource availability provides a particularly clear example. Some interventions require trained personnel, technology, transportation, diagnostic capacity, or continuing support. If these resources determine whether a crucial implementation step occurs, they can become part of the causal pathway.
That does not mean a simple high-resource versus low-resource classification is sufficient. The synthesis should identify the particular resource that matters and examine how evidence behaves across different resource environments.
Different effects do not always require different directions
Effect modification may be quantitative or qualitative. In quantitative interaction, the intervention remains beneficial across settings but the magnitude changes. In qualitative interaction, the direction itself changes, such as benefit in one subgroup and harm in another. Cochrane notes that qualitative interactions are comparatively rare.
A mechanism can therefore matter even when every study points broadly in the same direction. A consistently larger benefit under one contextual condition may still be scientifically and practically important.
Sometimes the mechanism is implementation rather than the intervention itself
An intervention may possess the same theoretical active ingredient across settings while contexts determine whether participants actually receive it.
For example, a professional-development program might depend on repeated coaching. Institutions that cannot provide coaching may technically implement the same named program but deliver much less of its active component.
The contextual mechanism then operates through implementation. This distinction is valuable because it changes the practical conclusion. Instead of saying that the intervention is unsuitable for certain settings, the evidence may suggest that a particular implementation condition must be addressed.
Mechanistic hypotheses can improve future research
One benefit of investigating contextual heterogeneity is that it can transform a vague observation into a testable proposition.
Instead of asking whether an intervention “works differently in different places,” future studies can test whether the proposed contextual characteristic actually modifies the effect. Researchers may deliberately sample settings that vary on that characteristic, measure the hypothesized intermediate process, or manipulate implementation conditions where ethically and practically possible.
In this sense, heterogeneity can generate theory rather than merely complicate estimation.
