01 · The Question
Can excluding studies by language actually change what a review appears to show?
Suppose two researchers investigate the same question. One includes eligible studies regardless of publication language. The other includes only studies published in English. Could they reach different conclusions?
They could. Restricting by language changes the pool of evidence that is allowed to enter the review. If the excluded studies differ in their findings, size, precision, populations, settings, interventions, or other relevant characteristics, the resulting synthesis can change.
Yet the effect is not inevitable. Empirical research in conventional medicine has often found relatively small effects from excluding non-English publications. Other evidence shows that the effect can differ across topics and cannot be predicted safely for an individual review. The methodological problem is therefore more nuanced than either “English-only reviews are always biased” or “language never matters.”
03 · What You Need to Know
Language restrictions change the evidence set before they change the conclusion
The mechanism is straightforward: excluded studies cannot influence the synthesis
A literature review's conclusion is based on the evidence that enters it. If a language criterion removes otherwise eligible studies, those studies can no longer contribute their participants, observations, estimates, contexts, experiences, or findings.
Whether that changes the conclusion depends on what was removed.
Language restriction The review includes only publications in one or several specified languages.
Evidence loss Eligible studies published outside those languages are excluded.
Evidence composition changes The remaining studies may differ in number, sample size, findings, geography, setting, intervention, population, or methodology.
Synthesis may change Effect estimates, precision, themes, contextual interpretation, certainty, or conclusions may remain similar or shift.
The language criterion itself does not mathematically create a particular direction of bias. Its consequence depends on the characteristics of the evidence that disappears.
Language bias is not simply the presence of several languages
The concern becomes more consequential when publication language is associated with study findings or other characteristics relevant to the synthesis. Researchers have long investigated whether statistically significant or otherwise favorable findings are more likely to appear in internationally visible, English-language journals while other findings remain in local-language publications.
Empirical patterns are not uniform enough to justify assuming that every non-English evidence base behaves this way. Language can interact with publication practices, discipline, country, journal visibility, intervention type, and research tradition.
Language restriction
A methodological decision to include or search only particular publication languages.
Language bias
A systematic distortion that can arise when language-related selection produces an evidence set that differs meaningfully from the relevant evidence that was excluded.
A review can impose a language restriction without materially changing its conclusion. Conversely, a language restriction can sometimes remove evidence that matters.
Empirical studies in conventional medicine often find modest effects
The evidence here deserves careful interpretation rather than folklore. Morrison and colleagues systematically reviewed empirical studies examining the effect of excluding randomized trials published in languages other than English from systematic review-based meta-analyses. None of the included methods studies found major differences between summary treatment effects from English-restricted and language-inclusive meta-analyses of conventional medical interventions. Including non-English trials did, however, improve the precision of pooled estimates in the evidence they examined.
An earlier empirical study by Jüni and colleagues examined 50 meta-analyses containing 485 English-language and 115 non-English-language trials. In 29 of the 50 meta-analyses, excluding the non-English trials changed effect estimates by less than 5%. Other meta-analyses changed more, and the authors concluded that the importance of non-English trials was difficult to predict for an individual review.
More recently, Nussbaumer-Streit and colleagues examined 59 Cochrane Reviews of clinical interventions. Removing non-English publications excluded 31 studies from 16 reviews. For 38 of 40 affected outcomes, the size or direction of effect estimates and statistical significance were not markedly altered. Two outcomes changed statistical significance, but the original review authors indicated that their overall conclusions would still have remained the same, albeit with less certainty.
These findings provide an important corrective to the claim that excluding any non-English paper necessarily reverses a meta-analysis. They do not establish that language restrictions are harmless in every evidence base.
Topic can matter
Earlier methodological work found that the effect of language restriction differed in some areas. A study examining reports of randomized trials found substantial effects from language exclusion in reviews of complementary and alternative medicine, while similar effects were not apparent for conventional interventions.
This illustrates the central difficulty: the consequence of a language restriction depends on where relevant evidence is produced and published. You usually do not know with certainty what the excluded evidence would have contributed until you investigate it.
A conclusion can change without reversing direction
Researchers sometimes imagine only the most dramatic scenario: a review concludes that an intervention works, non-English studies are added, and the conclusion reverses completely.
Language restrictions can matter in subtler ways.
Possible consequence
What may change
Effect size
The estimated magnitude of an association or intervention effect may shift.
Precision
Adding eligible studies can narrow uncertainty around an estimate.
Statistical significance
A confidence interval or test may cross a conventional significance threshold.
Certainty
The amount or consistency of evidence available for a conclusion may change.
Contextual coverage
Populations, countries, institutions, or settings may appear less represented.
Qualitative synthesis
Themes, experiences, explanations, or counterexamples may be missing.
A review can therefore retain the same broad headline while becoming less precise, less geographically representative, or less confident after evidence is excluded.
Evidence from clinical meta-analysis does not automatically answer the question for qualitative or contextual reviews
Much of the empirical research on language restrictions has examined quantitative syntheses of medical interventions. That evidence is useful within its domain, but it should not be treated as a universal estimate of the effect of language restriction.
A qualitative evidence synthesis, historical review, educational review, policy review, or context-specific synthesis may depend on evidence in a different way. A locally published study might contribute a distinctive experience or institutional context even if it would not noticeably change a pooled numerical effect.
For geographically focused questions, language restrictions can also interact with the underrepresentation of research from particular countries . Removing local-language research may narrow both linguistic and geographic coverage at the same time.
Searching only English-language databases is not equivalent to applying an English-language filter
Language-related evidence loss can occur at more than one point. You might explicitly exclude non-English publications during screening. Alternatively, your search sources and terminology may fail to retrieve them in the first place.
Cochrane currently recommends planning systematic-review searches without language restrictions and, when appropriate, considering databases and trial registers published in languages other than English. It also recommends seeking translation assistance when needed.
This means that simply removing an English-language filter does not guarantee multilingual coverage. You may still need strategies for finding non-English research when searching primarily in English .
Language restrictions may sometimes be pragmatic rather than ideal
Translation requires time, expertise, and sometimes money. Rapid reviews and student projects may operate under genuine constraints. Methodological decisions therefore occur in real research environments rather than in a magical land of unlimited librarians and translation budgets.
The important distinction is between making a pragmatic restriction transparently and pretending that it has no implications. If you restrict languages, state which languages were eligible, why the restriction was necessary, and how it might limit the evidence base.
Watch Out
Do not claim that an English-only restriction definitely biased your conclusion, but do not claim that it could not have mattered merely because previous reviews in another field found little effect. The relevant question is what evidence your restriction could have excluded in your particular topic and context.
04 · A Practical Example
How the same language restriction can matter in different ways
Hypothetical Example
A review of digital-learning interventions across several countries
A researcher identifies 18 eligible English-language studies and four potentially eligible studies published in other languages. The four studies come from settings that are otherwise scarcely represented in the English-language evidence.
English-only synthesis The review is based on 18 studies and suggests generally positive effects across the settings represented.
Additional assessment The four non-English studies are translated sufficiently for eligibility assessment and all qualify for inclusion.
Possible numerical outcome The additional studies produce broadly similar results, so the overall direction of the quantitative finding changes little.
Possible contextual outcome The four studies nevertheless add evidence from education systems almost absent from the original evidence set and identify implementation constraints not reported in the English-language studies.
Revised interpretation The headline effect may remain similar, while the review's account of where, how, and under what conditions the intervention works becomes materially different.
Calling the language restriction “irrelevant” merely because the pooled direction did not reverse would miss an important part of what changed.
07 · A Quick Checklist
Before imposing or interpreting a language restriction
Check whether you have:
Identified which languages are plausibly important to the evidence base rather than assuming English contains nearly everything relevant.
Distinguished an explicit language eligibility restriction from incomplete retrieval of non-English research.
Considered whether language overlaps with geography, publication venue, intervention type, or other characteristics relevant to your question.
Planned how potentially relevant non-English records will be screened and translated.
Applied the same substantive eligibility and appraisal criteria regardless of publication language.
Reported any language restrictions explicitly and explained why they were necessary.
Avoided claiming that the restriction either did or did not change the conclusion unless your evidence supports that claim.
11 · Cite this Guide
How to Cite This Guide
This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.
Recommended (Field Guide)
APA
MLA
Chicago
Copy Citation
