Should You Screen for Relevance Before Quality or Quality Before Relevance?

Which should come first: relevance or quality?

You find a paper that looks methodologically excellent, but it studies the wrong population. Another addresses your question almost perfectly, yet its design has important limitations. Which one should you assess first?

The confusion comes partly from treating relevance and quality as two competing standards for inclusion. They answer different questions. Relevance asks whether the study belongs in the evidence base defined by your review question and eligibility criteria. Quality appraisal asks how much confidence you should place in the methods, findings, or inferences of a study that is potentially relevant.

The order matters because detailed critical appraisal can take substantial time. More importantly, assessing methodological quality before establishing eligibility can encourage reviewers to spend effort evaluating studies that could never answer the review question.

Usually, establish relevance and eligibility first

Relevance and quality answer different methodological questions

Relevance asks whether the study belongs in your review

During study selection, you are essentially asking whether a record or report satisfies the review's predefined eligibility criteria. Depending on the question, those criteria may concern population, phenomenon or intervention, comparator, outcome, setting, study design, publication characteristics, or other explicitly justified features.

A paper can therefore be rigorous research and still be irrelevant to your review. An excellent randomized trial of university students does not become relevant to a review restricted to primary-school pupils merely because the trial itself was well conducted.

This is why deciding which papers actually belong in the review should be anchored to the review question and eligibility criteria rather than to a general impression that a paper is "good."

Quality asks how trustworthy the relevant evidence is

Once a study is eligible, critical appraisal examines features of its design, conduct, analysis, and reporting that affect how its evidence should be interpreted. In systematic reviews, this may involve a formal risk-of-bias assessment rather than a single generic "quality score."

JBI, for example, describes study selection and critical appraisal as separate processes. Its guidance places selection according to prespecified inclusion criteria before critical appraisal and describes appraisal as an assessment of methodological quality and the extent to which bias has been minimized. Cochrane reviews likewise distinguish decisions about eligibility from risk-of-bias assessment.

Why relevance usually comes first

There is little methodological value in performing a detailed risk-of-bias assessment on a paper that clearly falls outside the review's eligibility criteria. If your question concerns a particular population and the study examines an entirely different population, a meticulous appraisal of randomization, missing data, measurement, or confounding does not make that study eligible.

The sequencing also makes screening more efficient. Searches often retrieve many records that can be excluded from their titles and abstracts because they clearly fail one or more eligibility criteria. Potentially eligible reports then proceed to full-text assessment. Detailed appraisal is reserved for the evidence that survives this selection process.

PRISMA 2020 reflects this conceptual separation in its reporting requirements. It asks reviewers to report the process used to determine whether studies met inclusion criteria separately from the methods used to assess risk of bias in included studies.

Eligibility is not the same as superficial relevance

"Screen relevance first" should not be interpreted as "keep anything that sounds related." Relevance needs an operational definition derived from your eligibility criteria. A title containing your topic keyword is not sufficient evidence that a paper belongs in the review.

At the title and abstract stage, the information available may only permit a provisional decision. If the record appears potentially eligible and there is insufficient information to exclude it confidently, retrieving the full report is generally safer than guessing. The uncertainty created by an uninformative abstract is therefore a screening problem, not evidence that the study is low quality.

Quality does not always determine whether an eligible study is excluded

This distinction is particularly important. Finding methodological limitations does not automatically mean that an otherwise eligible study must disappear from the review.

JBI guidance explicitly requires review authors to specify whether and how critical-appraisal results will be used to exclude studies. If studies below a methodological threshold will be excluded, the decision rules should be stated explicitly. JBI also notes that whether low-methodological-quality studies are excluded is a decision for the review team rather than an automatic consequence of appraisal.

In other review approaches, studies at high risk of bias may remain in the evidence base while their limitations influence interpretation, synthesis, sensitivity analyses, or judgments about certainty. The correct treatment therefore depends on the methodology and protocol rather than on a universal rule that "poor quality equals exclude."

What if quality can be judged immediately?

Sometimes a glaring methodological problem is visible while you are screening for eligibility. You do not need to pretend you have not noticed it. The important issue is whether that problem is itself part of a predefined eligibility rule.

If your criteria exclude certain study designs, for example, identifying the design during screening is an eligibility decision. That is different from conducting a full critical appraisal of an otherwise eligible study and deciding afterward that you dislike its methodological quality.

This distinction also explains why relevance and quality may both matter without being interchangeable. One determines whether the evidence addresses the intended question; the other helps determine how that evidence should be trusted and used.

The exact sequence depends on the type of review

Not every literature review follows the same appraisal procedure. Systematic reviews commonly include formal critical appraisal or risk-of-bias assessment. Scoping reviews are often conducted for broader mapping purposes, and critical appraisal may not always be undertaken. JBI's scoping-review guidance treats evidence selection and critical appraisal as distinct stages and recognizes critical appraisal as a methodological decision rather than an inevitable feature of every scoping review.

Narrative, integrative, qualitative, mixed-methods, and other review traditions may also handle methodological appraisal differently. You should therefore follow the methodological standards appropriate to the type of review you are conducting rather than mechanically importing a sequence from another review design.

How relevance-first screening works in practice

Common mistakes when combining relevance and quality decisions

Use a staged screening process rather than one overall impression

Before screening begins, define what makes a study eligible and decide separately how methodological appraisal will affect the review. Then apply those decisions consistently.

Whatever sequence you use, document it. PRISMA 2020 asks systematic-review authors to describe both their study-selection process and their risk-of-bias assessment methods, including how many reviewers were involved. If you exclude reports at full-text assessment, keeping explicit reasons also makes the process more transparent; recording why studies were excluded becomes particularly important at this stage.

If your protocol specifies independent screening, follow that process rather than allowing one reviewer's informal quality impression to become an unrecorded exclusion rule. JBI guidance for systematic reviews recommends independent study selection by two or more reviewers and independent critical appraisal, with disagreements resolved through consensus or a further reviewer.

Before mixing relevance and quality judgments, check your process

Questions about relevance and quality screening

First determine whether the evidence belongs, then judge its limitations

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.