01 · The Question
When have you searched hard enough for unpublished evidence?
Searching for unpublished evidence creates an awkward methodological problem. You may know that completed studies, null findings, conference work, registry results, dissertations, technical reports, or data held by researchers could exist outside the conventional journal literature. Yet every additional search route costs time, and there is no database containing everything that was never formally published.
At some point, you have to stop.
The difficult question is where that point should be. Stopping after a quick database search may leave important evidence undiscovered. Continuing indefinitely through registries, repositories, institutional websites, conference records, grant databases, regulatory documents, citation trails, and unanswered emails is not realistic either. A defensible search therefore needs something more useful than either “search everything” or “search until you run out of time.”
03 · What You Need to Know
Think in terms of marginal value, not unlimited comprehensiveness
Searching more can reduce bias, but searching more always has a cost
Unpublished and difficult-to-find evidence matters because the evidence that becomes readily available may not be a random sample of all research that was conducted. Cochrane identifies non-reporting bias as a threat to evidence synthesis when decisions about whether, when, or where results are reported depend on their magnitude, direction, or statistical significance. Evidence reviewed by Cochrane indicates that studies with statistically significant results have been more likely to be published than studies with non-significant results.
This is why systematic-review guidance encourages searching beyond conventional bibliographic databases where appropriate. Cochrane's methodological standards, for example, identify searching relevant trial registers as mandatory for intervention reviews and searching relevant grey-literature sources and contacting relevant individuals or organizations as highly desirable. These recommendations are intended to reduce publication bias and improve identification of relevant evidence.
That does not imply an unlimited search. Cochrane's technical guidance also explicitly recognizes feasibility. It notes that the impact of excluding unpublished data is uncertain across reviews and advises reviewers to consider the time and effort devoted to the grey-literature component of a search.
Comprehensive search
A systematic attempt to identify the relevant evidence using sources and methods appropriate to the review question.
Exhaustive search
An attempt to pursue every conceivable source or lead regardless of its likely contribution, cost, or feasibility.
A rigorous review generally aims for the first. It cannot guarantee the second.
The consequences of missing evidence should determine how hard you search
The same search effort is not appropriate for every review. Consider what would happen if a handful of unpublished studies remained undiscovered.
If dozens of large, consistent studies already support a conclusion and plausible missing studies are unlikely to alter its interpretation, another week spent pursuing obscure leads may have little marginal value. If only three small studies exist and all report strikingly positive results, one unpublished null study could matter considerably more.
This is partly why the value of searching for unpublished research depends on the review context. Search intensity should respond to the potential consequences of missing evidence, not simply to a desire to make the methods section look impressively comprehensive.
Risk of selective non-reporting should increase search effort
Some questions provide stronger reasons than others to suspect that readily published evidence may present an incomplete picture. Intervention research is an important example because study results can influence whether and how findings become available.
A Cochrane methodological review found that published trials tended to show larger intervention effects than trials identified in grey literature. The evidence came from healthcare trials and should not be generalized mechanically to every research field, but it illustrates why publication bias can distort the apparent evidence base.
You should therefore consider greater search effort when the topic has plausible incentives for selective dissemination, when null or unfavorable findings may be less visible, or when the evidence base contains signs that completed research may be missing.
Sparse evidence raises the value of each additional study
When a review contains very few eligible studies, another study may have substantial informational value. It could affect an effect estimate, reveal an unrepresented population, provide a missing outcome, alter confidence in the evidence, or simply show that the research base is not as small as the published literature suggests.
Cochrane's current Handbook acknowledges this context dependence. Evidence discussed there suggests that unpublished studies do not necessarily change the conclusions of every review, while also recognizing that they may matter particularly when few relevant studies exist or when the published literature may be affected by vested interests.
The practical implication is important: the fact that unpublished evidence sometimes makes little difference is not a justification for ignoring it. Nor does evidence that it sometimes matters justify chasing every possible lead. You need to judge the expected value in the specific review.
Some sources are much more promising than others
Searching effort should not be distributed equally across every conceivable source. Prioritize sources based on the type of study you are seeking and how research in that field is registered, funded, disseminated, or archived.
For reviews of registered clinical trials, trial registries can be unusually valuable because they may reveal initiated, completed, terminated, or ongoing studies regardless of whether a conventional journal article was published. Cochrane reports empirical examples in which registry searches identified additional eligible trials and, in some reviews, incorporating available registry results changed meta-analytic estimates.
In other fields, dissertations, institutional repositories, conference proceedings, government databases, working-paper repositories, organizational websites, grant records, or direct researcher contact may be more productive. The first question should therefore be “Where would missing studies from this field plausibly leave traces?” rather than “What other website can I search?”
Finding a study and obtaining usable evidence are different problems
A registry entry, conference abstract, grant record, or mention in another publication may establish that a study exists without providing enough information to include it in a synthesis. You may then face a second decision: how much effort should be spent obtaining results, clarifying methods, or locating the investigators?
This distinction prevents a common source of wasted effort. Discovery may be worthwhile even when full recovery is impossible because the existence of missing studies can inform your assessment of the evidence base. But repeated attempts to retrieve unavailable numerical results may eventually produce diminishing returns.
When a potentially important study has been identified but the necessary information is unavailable, contacting the researchers may be justified. Cochrane notes that author contact has produced useful unpublished information in some systematic reviews, although it has not been uniformly successful.
Use diminishing returns as evidence, not as a magic stopping rule
Imagine that your first targeted unpublished-evidence search identifies twelve potentially relevant studies, your second source identifies four unique studies, your third identifies one, and several additional carefully chosen routes produce no unique eligible evidence. That pattern tells you something useful: the marginal yield of additional searching appears to be declining.
It does not mathematically prove that nothing else exists. An obscure source could still contain a critical study. Rather, diminishing yield becomes one piece of evidence supporting a stopping decision when combined with the importance of the question, the suspected risk of non-reporting bias, the sources already covered, and the feasibility of further searching.
Watch Out
Do not define “enough” retrospectively as the moment your deadline arrives. For a systematic review, major search sources and supplementary methods should be planned prospectively where possible. Otherwise, convenience can quietly become a methodological criterion.
Document what you did not search as well as what you did
A bounded search can still be rigorous if its boundaries are explicit and defensible. Reporting guidance emphasizes describing the sources searched, dates, search strategies, limits, and other methods used to identify studies. Cochrane likewise recommends reporting contacts with individuals and organizations and restrictions such as language or publication format.
This transparency matters because readers do not need you to claim omniscience. They need enough information to judge how likely your search was to miss relevant evidence and whether those omissions could affect the conclusions.
04 · A Practical Example
How a researcher can decide when another search is worth doing
Hypothetical Example
A review with suspiciously little null evidence
Suppose a researcher is reviewing an intervention and identifies seven published studies. Six report favorable results, most are small, and several papers mention conference presentations or registered studies that cannot immediately be matched to full publications. The researcher has already searched the main bibliographic databases but is deciding how much further to go.
Assess the consequence With only seven published studies, additional eligible studies could plausibly affect the synthesis. The unusual concentration of favorable findings also gives the researcher a reason to take missing evidence seriously.
Prioritize high-yield sources The researcher searches relevant study registries, follows references to the unmatched conference work, and checks repositories appropriate to the field rather than beginning with an unrestricted web search.
Pursue strong leads Two completed studies are identified without published results. Because both appear eligible and potentially consequential, the researcher searches for associated reports and contacts the investigators using a predefined contact procedure.
Evaluate marginal yield Further targeted searches produce no additional eligible studies, and the remaining possible sources are poorly matched to the research design and topic. Responses to the researcher contacts have also been exhausted according to the planned procedure.
Stop and report The researcher records the sources searched, the unpublished studies identified, the unsuccessful retrieval attempts, and the remaining uncertainty rather than claiming that every unpublished study has been found.
The stopping decision is defensible because it follows the expected value of further searching. If the first registry search had instead uncovered twenty unmatched completed studies, stopping at that point would be much harder to justify.
06 · What This Means for You
Set the search effort according to what is at stake
A useful way to allocate effort is to ask two questions together: how plausible is it that important evidence remains unpublished, and how consequential would that evidence be if it existed?
A simple decision framework
If the evidence base is small and additional studies could materially alter the synthesis
Invest more effort in targeted unpublished-evidence searches and follow strong leads carefully.
If selective publication is particularly plausible
Prioritize sources that can reveal research independently of journal publication, such as appropriate registries, regulatory sources, repositories, or other records relevant to the field.
If a specific unpublished study is known to exist and appears important
Spend more effort locating its results or obtaining information from investigators than you would on a speculative lead.
If successive appropriate searches yield little or no unique eligible evidence
Consider whether the declining marginal yield, together with the remaining risk of bias, justifies stopping.
If further searching is infeasible but potentially important evidence could remain missing
Stop when necessary, but report the limitation and consider its implications when interpreting the findings rather than implying that the search was exhaustive.
For many reviews, this produces a staged strategy. Begin with the unpublished-evidence sources that are most strongly justified by the study design and field. Expand when those searches reveal evidence that important material remains missing. Deprioritize increasingly speculative routes when carefully selected additional methods stop producing meaningful information.
If your concern is specifically that unfavorable or statistically non-significant findings have disappeared from the visible literature, it may be more useful to focus on ways of locating unpublished negative or null studies than to broaden the search indiscriminately.
The final standard is defensibility. You should be able to explain why particular sources were searched, why certain leads received additional effort, why you eventually stopped, and what uncertainty remains because unpublished evidence may still exist.
07 · A Quick Checklist
Before you stop searching for unpublished evidence
Before ending the search, check:
Have you considered how strongly publication or non-reporting bias could affect this particular review question?
Have you identified the unpublished-evidence sources most appropriate to the study designs, field, and topic rather than searching sources indiscriminately?
Could one or a few additional studies plausibly change the synthesis, interpretation, precision, or confidence in the evidence?
Have you followed up known or strongly suspected eligible studies that lack accessible results?
If appropriate, have you searched relevant registries, repositories, grey-literature sources, reference lists, or other field-specific sources?
Have successive additional search methods stopped producing substantial unique eligible evidence?
Have you recorded unsuccessful searches and contact attempts rather than silently dropping unresolved leads?
Can you justify your stopping point in methodological terms rather than solely by saying that time ran out?
Will your review acknowledge the possible effect of evidence that could not be located or retrieved?
