01 · The Question
What if some of the relevant evidence never became a conventional journal article?
Most literature searches naturally gravitate toward peer-reviewed journal articles. They are comparatively easy to discover, cite, and evaluate using familiar bibliographic databases. But the research that reaches journals is not necessarily a neutral sample of all the research that was conducted.
Studies can remain unpublished, appear only as conference abstracts, be reported in dissertations or theses, exist as government or organizational reports, or be registered without a corresponding full publication. This matters particularly when the probability of publication is associated with the direction or statistical significance of the findings.
The question is not whether every literature review must collect everything outside journals. It is whether excluding grey or unpublished evidence could materially distort what the available evidence appears to show.
03 · What You Need to Know
What counts as grey or unpublished evidence, and why might it matter?
Grey literature is broader than unpublished research
The terms grey literature and unpublished research overlap, but they are not interchangeable. Cochrane describes grey literature as reports published outside traditional commercial publishing and specifically identifies sources such as dissertations and conference abstracts as examples worth considering in evidence searches.
Some grey literature is very much public. Government reports, technical reports, working papers, dissertations, policy documents, and organizational research may be available online and formally archived. Other evidence may be genuinely unpublished or only partially disseminated.
Grey literature
Research or research-related material distributed outside conventional commercial or scholarly journal publishing channels, such as reports, theses, dissertations, and some conference materials.
Unpublished evidence
Research findings or study information that have not been formally published, which may include completed studies known from registries, investigator contact, or other records.
The categories are not perfectly standardized across disciplines, so the useful methodological question is less about assigning a flawless label and more about identifying which relevant evidence sources fall outside your conventional journal search.
Publication can be related to the results of the study
The strongest reason to care about unpublished evidence is not simply that more evidence exists. It is that the evidence reaching conventional publication may differ systematically from the evidence that does not.
Cochrane summarizes empirical evidence showing that studies with statistically significant or favorable findings can be more likely to reach full publication. It also notes evidence that published trials may report larger effect estimates than trials disseminated through sources such as conference abstracts, theses, books, or government reports.
If the probability of entering the visible journal literature depends partly on the result, a review restricted to published journal articles may produce a distorted picture of the underlying evidence.
Watch Out
Peer-reviewed publication is a dissemination status, not proof that the published evidence is an unbiased sample of all studies that were conducted. If unfavorable or inconclusive findings are less likely to become conventional articles, searching only journals can make an intervention or relationship look more consistently favorable than the complete evidence warrants.
Grey literature can reveal evidence that journals do not capture well
Some research questions naturally generate evidence outside academic journals. Government agencies may commission evaluations. Universities archive dissertations. Professional organizations produce technical reports. Conferences may contain early findings. Clinical and other prospective study registries can document research before results are published.
The value of these sources depends heavily on the question. If you are examining implementation of a public program, government and organizational reports might be central evidence rather than peripheral material. If you are reviewing intervention effects, study registries may help identify completed studies whose results never reached conventional publication.
For an emerging topic, non-journal sources may also provide early evidence before a substantial peer-reviewed literature has accumulated. That can be particularly relevant when trying to identify important recent research.
Conference abstracts can be useful, but they can also be frustratingly incomplete
A conference abstract can alert you that a study exists and may provide information unavailable elsewhere. But abstracts often contain substantially less methodological and results detail than full articles, which can make eligibility assessment, risk-of-bias assessment, and interpretation difficult.
Cochrane nevertheless recommends searching relevant grey-literature sources, including conference abstracts, when extensive study identification is needed to reduce publication bias.
If an abstract appears consequential, investigate whether a full publication subsequently appeared. Matching conference reports to later articles also helps avoid accidentally treating different reports from the same study as independent evidence.
Dissertations and theses may contain substantial research detail
Doctoral dissertations and master's theses can contain extensive methodological information, analyses, and findings that never appear in journal form or appear only partially in later publications.
Their usefulness varies across fields and institutions, and academic examination should not be treated as equivalent to journal peer review. Yet excluding dissertations merely because they are dissertations can remove eligible research for reasons unrelated to the actual quality of the study.
The same principle applies in the opposite direction: inclusion in an institutional repository does not establish methodological quality. Appraise the research itself.
Study registries can reveal the research that publication leaves invisible
Registries can be particularly valuable when research is prospectively registered. A registry entry may show that a study was planned or completed even when no journal article can be found. It may also preserve information about prespecified outcomes, recruitment status, or study design that can be compared with later publications.
For some evidence syntheses, searching relevant registries is therefore part of a broader attempt to identify both published and unpublished studies. Cochrane's search guidance includes sources beyond bibliographic databases when extensive study identification is required.
A registry record is not automatically a completed study, and a completed study is not automatically usable evidence. Check its status and determine what results, if any, are actually available.
Preprints require their own judgment
Preprints complicate the simple published-versus-unpublished distinction. They are publicly disseminated manuscripts but generally have not undergone the journal peer-review process at the point they are posted.
They can be particularly useful for detecting very recent work in fast-moving fields. At the same time, their status should be clearly recognized, and researchers should check whether a later peer-reviewed version, revision, withdrawal, or other update exists before relying on the version initially found.
The practical principle is consistent: dissemination route should inform your appraisal, but it should not replace appraisal.
Searching grey literature is often harder to reproduce
Bibliographic databases usually provide structured search interfaces, stable records, and exportable metadata. Websites, repositories, search engines, and organizational archives may offer much less consistency.
Search interfaces can change. Results may be ranked opaquely. Documents can move or disappear. Some sources provide limited search functionality. Consequently, grey-literature searching may require especially careful documentation of where, when, and how you searched.
This is also why the choice of sources should follow the question rather than a generic checklist. Searching fifty irrelevant websites does not make a review more comprehensive. It makes the search log longer, which is a different achievement.
Grey literature does not get a methodological free pass
Researchers sometimes respond to publication bias by swinging too far in the other direction, treating unpublished evidence as somehow more authentic because it escaped journal publication. That is equally problematic.
Grey and unpublished sources may have incomplete methods, missing results, uncertain review processes, limited metadata, or inaccessible supporting materials. Some may be excellent. Others may be impossible to evaluate adequately.
The relevant distinction is not published equals good and grey equals bad, or vice versa. The task is to critically evaluate the evidence using criteria appropriate to its design and available information.
04 · A Practical Example
How unpublished findings can change the apparent pattern of evidence
Hypothetical Example
The intervention that looked consistently successful
Suppose a researcher reviews an educational intervention and identifies six peer-reviewed journal articles. Five report favorable effects, while one reports little difference. The published literature therefore appears strongly positive.
A search of dissertation repositories, conference proceedings, and relevant study records identifies three additional eligible studies. Two report little or no evidence of benefit, while the third reports mixed findings. Their methods vary in quality, and one conference report lacks enough detail for some aspects of appraisal.
The researcher does not simply combine all nine sources as though they were equivalent. Instead, each study is appraised according to its design and available information. The researcher also checks whether any conference papers, dissertations, and journal articles represent different reports from the same underlying studies.
The expanded evidence base no longer looks uniformly favorable. That does not prove publication bias caused the difference, but it demonstrates why publication status should not silently determine which evidence becomes visible.
Published search
Six journal articles produce an apparently consistent pattern.
Broader search
Relevant dissertations, conference materials, and study records reveal additional research.
Study matching
Reports are checked to determine whether they represent unique studies or multiple outputs from the same research.
Appraisal
Each source is evaluated according to its methods and available information rather than publication status alone.
Interpretation
The researcher presents a more qualified account of the evidence and explains the limitations of incompletely reported sources.
This study-matching step is important because multiple outputs should not automatically be treated as multiple independent investigations. The distinction is examined separately when considering whether you have distinguished multiple papers from multiple independent studies.
07 · A Quick Checklist
Could relevant evidence exist outside the journal literature?
Before excluding grey or unpublished evidence, check:
I have considered whether selective publication could materially affect the apparent pattern of evidence for my question.
I know which types of non-journal evidence are plausibly relevant to this topic, such as dissertations, reports, conference materials, registries, or preprints.
I have chosen grey-literature sources because they are relevant to the question rather than simply to make the search appear broader.
I have checked whether preliminary or grey reports subsequently became full publications.
I have checked whether multiple reports represent the same underlying study.
I appraise grey and unpublished evidence rather than assuming its publication status determines its quality.
I clearly distinguish peer-reviewed, preprint, grey, and unpublished evidence where those differences matter to interpretation.
I document non-database searches sufficiently to explain where and how relevant evidence was sought.
