Did You Overrepresent Highly Cited Studies in Your Literature Review?

Are You Reviewing the Evidence or the Most Visible Part of It?

Some papers seem to be everywhere. They appear near the top of search results, recur in review articles, accumulate hundreds or thousands of citations, and lead you through citation chains to still more papers connected to them.

Repeated visibility can create a subtle impression: surely a study cited this often must be especially important, reliable, or representative of what the literature shows.

Sometimes highly cited studies genuinely are influential because they introduced important methods or findings. But citation count is produced by many processes other than methodological strength. If highly cited papers receive disproportionate attention simply because they are easier to encounter, a literature review can begin reproducing the citation structure of a field rather than independently evaluating its evidence.

Visibility Is Not the Same as Evidential Importance

Why Citation Networks Can Magnify Some Evidence More Than Others

Citations measure scholarly attention, not research quality directly

A citation tells you that one document referenced another. It does not tell you why. A paper may be cited because its findings are influential, because it introduced a method, because authors disagree with it, because it is a convenient canonical reference, or simply because later authors repeatedly inherited the same citation.

Citation count therefore describes scholarly visibility and use. It should not be treated as a direct measurement of the methodological quality of the underlying study.

Citation bias can make some findings more visible than others

Citation is not necessarily neutral with respect to study findings. A systematic review and meta-analysis by Duyx and colleagues found evidence that articles reporting statistically significant results were cited more frequently than articles reporting non-significant results, with variation across scientific disciplines. The authors described this selective citation according to results as citation bias.

This does not mean every highly cited study reports a positive result or that citation counts are useless. It means that following the citation trail can reproduce existing selection processes rather than providing a neutral map of all relevant evidence.

Highly cited papers can become intellectual hubs

Once a paper becomes influential, subsequent researchers may discover the field through it, cite it as a standard reference, and use its reference list to locate earlier work. Later reviews may then cite the same study, making it even more visible to the next generation of researchers.

This cumulative process can be useful. Seminal studies genuinely deserve recognition. The danger appears when visibility begins substituting for independent retrieval and appraisal.

Citation searching is useful, but it should supplement rather than define the evidence base

Checking the reference lists of relevant studies can identify evidence missed by database searches. A Cochrane methodological review found that reference checking sometimes identified additional eligible studies, although the yield varied substantially and the supporting methodological evidence had limitations.

Backward citation searching examines references cited by a relevant paper. Forward citation searching identifies later publications that cite it. Both can be valuable supplementary methods, especially when terminology or database indexing makes evidence difficult to locate.

The problem is not citation searching itself. The problem is allowing a small set of prominent papers to determine the entire search trajectory.

Search independently of citation popularity

A structured database search gives studies an opportunity to be retrieved because they match your concepts and eligibility criteria rather than because previous authors happened to cite them. This is one reason adequate database coverage and sufficiently varied search terminology matter.

Citation searching can then supplement those methods. If your entire evidence base grows outward from two famous papers, the resulting network may inherit whatever literature those papers and their descendants happened to cite.

Do not use citation counts as an inclusion criterion

For most evidence reviews, "cited at least 50 times" is not a defensible substitute for substantive eligibility criteria. Such a threshold systematically disadvantages newer research, work from smaller fields, studies published in less visible journals, and findings that have attracted less scholarly attention for reasons unrelated to methodological quality.

If your research question is specifically bibliometric, citation count may of course be the phenomenon under investigation. That is a different purpose from using citations to decide which empirical evidence deserves inclusion.

Do not use citation counts as an informal quality score

After inclusion, another temptation appears: discussing famous studies at length while relegating less cited studies to a sentence or omitting them from the narrative synthesis.

If the review is meant to synthesize evidence, prominence should follow evidential relevance rather than reputation. A less cited study may have a stronger design, more direct population, better measurement, or more precise estimate than a famous predecessor.

The appropriate question is therefore whether you gave stronger evidence more weight than weaker evidence, not whether you gave the most cited evidence more space.

Citation bias can reinforce confirmation bias

If positive or striking studies become more frequently cited, researchers who rely heavily on citation networks may encounter those findings repeatedly. Repetition can make a conclusion appear more settled than the complete evidence base warrants.

This becomes especially consequential when evidence challenging the prevailing conclusion is less visible. A review should therefore give contradictory evidence the same opportunity for inclusion and appraisal rather than assuming that the dominant citation narrative represents the full literature.

Track how studies were found

For systematic reviews, documenting whether records were identified through databases, citation searching, websites, registers, or other methods can help reveal whether the search depends heavily on a particular discovery pathway. PRISMA 2020 and PRISMA-S support transparent reporting of the different information sources and supplementary search methods used.

This documentation does not eliminate citation bias, but it makes the search architecture visible enough to evaluate.

When One Famous Study Starts to Define an Entire Literature

The lesson is not to avoid highly cited studies. It is to prevent their visibility from deciding what else you see and how strongly you believe it.

Common Ways Citation Visibility Distorts a Literature Review

Use Citations to Find Evidence, Not to Rank Its Truth

Citation networks are powerful discovery tools. Treat them as maps of scholarly communication rather than maps of methodological quality.

Before Highly Cited Papers Take Over Your Review

Questions About Highly Cited Studies and Citation Bias

Do Not Confuse the Loudest Evidence With the Strongest Evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.