Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

Part 2 · Guide 784 of 899

How Do You Avoid Choosing Sources Based on Whether You Agree With Their Conclusions?

A literature review becomes unreliable when sources are selected because their conclusions feel convincing or compatible with the researcher's position. Separate decisions about relevance and methodological quality from whether you agree with what a study found.

By Manuel B. Garcia
784
Avoiding Agreement-Based Source Selection Guide 784 of 899
01 · The Question

Are You Selecting Evidence or Selecting Conclusions?

Finding a paper that supports your argument feels useful. Finding one that contradicts it can feel like a problem to solve. That difference in reaction is precisely where source selection can become methodologically vulnerable.

The problem is not that researchers have prior beliefs. The problem begins when agreement quietly becomes an inclusion criterion: supportive studies are retained, cited, or discussed prominently, while inconvenient studies receive more demanding scrutiny, disappear during screening, or are dismissed for reasons that would not have disqualified a favorable result.

How can you prevent the conclusion of a study from determining whether it gets a fair hearing in your review?

02 · The Short Answer

Decide What Qualifies Before You Know Whether You Like the Answer

In Brief

Avoid choosing sources based on agreement by defining relevance and eligibility independently of study conclusions, searching broadly enough to retrieve conflicting evidence, and applying the same selection and appraisal rules regardless of the direction of the findings.

This does not mean every eligible source deserves equal weight. Selection asks whether evidence belongs in the review; appraisal and synthesis ask how much confidence that evidence deserves. Keeping those decisions separate makes agreement harder to disguise as methodology.

03 · What You Need to Know

Separate Relevance, Inclusion, and Evidential Weight

Agreement is not an eligibility criterion

Whether you find a conclusion persuasive should ordinarily have no role in deciding whether a study satisfies the substantive eligibility criteria of a review. For systematic reviews, Cochrane recommends specifying eligibility criteria in advance, based on characteristics relevant to the review question such as populations, interventions or exposures, comparators, outcomes where appropriate, and eligible study designs.

That principle matters beyond systematic reviews. Even in a narrative review, you can decide beforehand what makes a source relevant: the question it addresses, population studied, construct measured, research design, period, setting, or another defensible characteristic. The more those decisions depend on what the paper concluded, the more easily the literature can become a curated argument rather than an examination of evidence.

Eligibility Does this source meet the defensible criteria for addressing the review question?
Agreement Do I personally accept, prefer, or expect the conclusion reached by the source?

The second question should not answer the first.

Do not confuse selection with appraisal

A paper can be relevant and methodologically weak. It can also be relevant, methodologically strong, and deeply inconvenient to your argument. Those possibilities are easier to handle when source selection and critical appraisal remain conceptually distinct.

JBI describes study selection in systematic reviews as applying pre-specified inclusion criteria and treats critical appraisal as a separate process for examining methodological quality and risk of bias. This distinction helps prevent an improvised judgment such as “I don't trust this result” from quietly becoming “this study isn't relevant.”

If an eligible study has serious methodological weaknesses, identify those weaknesses during appraisal and account for them appropriately in synthesis. Do not retroactively redefine relevance merely because the study produced an undesirable result.

Searches can favor conclusions before screening even begins

Selective reviewing can occur long before you consciously accept or reject a paper. Your search vocabulary, databases, citation networks, language restrictions, date restrictions, and starting papers can shape which intellectual communities become visible.

Cochrane recommends thorough, objective, and reproducible searching across appropriate sources for systematic reviews and warns that narrow searching can retrieve an unrepresentative subset of the available evidence. The exact search requirements vary with review type, but the underlying problem travels well: you cannot fairly consider evidence your search strategy systematically prevents you from seeing.

On contested topics, different scholarly communities may use different terminology for related phenomena. A search built entirely from the vocabulary of one perspective may therefore look neutral while retrieving a skewed literature. When reviewing a politically or socially controversial literature fairly, inspect whether your terminology gives relevant competing research a reasonable opportunity to appear.

Watch what happens when you encounter an inconvenient paper

Agreement-based selection often appears not as an explicit decision but as asymmetric skepticism. A supportive study may be accepted after reading its abstract. A conflicting study suddenly prompts questions about sample size, measurement, journal quality, author motives, generalizability, statistical choices, and every limitation detectable before coffee.

Those questions may be entirely legitimate. The diagnostic question is whether you ask comparable questions when a study reaches the conclusion you prefer.

Watch Out

If a methodological flaw disqualifies a study only when its result conflicts with your position, the problem may not be the methodological flaw. State the criterion clearly and test whether you would apply it to a study reaching the opposite conclusion.

Do not replace conclusion bias with author or journal bias

Researchers sometimes avoid judging findings directly but use proxies that produce much the same selection pattern. They may exclude a paper because of the author's reputation, disciplinary affiliation, theoretical tradition, institution, or perceived political orientation.

Source characteristics can sometimes matter. Conflicts of interest, for example, may warrant attention, and publication venue can provide useful contextual information. But none of these characteristics automatically establishes whether a particular study is methodologically sound or whether its evidence is relevant.

Likewise, journal prestige should not become a substitute for evaluating research. Peer review is a screening and improvement process, not a guarantee that every published conclusion is correct.

Pre-specification constrains convenient rule changes

For systematic reviews, protocol development provides an important safeguard because eligibility criteria and key methods can be specified before study selection proceeds. PRISMA 2020 then asks authors to report inclusion and exclusion criteria, information sources, search strategies, and the methods used to determine whether studies met the criteria.

Pre-specification does not make judgment disappear. Eligibility criteria sometimes require clarification after researchers encounter unforeseen cases. What matters is that changes have substantive reasons rather than being reverse-engineered around unwanted findings, and that important deviations are documented where appropriate.

Independent screening can expose asymmetric judgment

Where the review methodology warrants it and resources permit, independent screening by more than one reviewer can reduce the influence of one person's discretionary judgments. JBI guidance for systematic reviews recommends independent study selection by two or more reviewers, with disagreements resolved through consensus or a third reviewer.

This safeguard is useful precisely because eligibility decisions can involve interpretation. If reviewers repeatedly disagree about papers reaching a particular conclusion, that pattern may reveal an ambiguous criterion worth clarifying.

Include first, then determine what the evidence deserves

Including a study does not require endorsing it. Once a study qualifies for consideration, critical appraisal can address whether its design, conduct, analysis, and reporting introduce important risks of bias.

This is where methodological quality should be evaluated independently of ideological agreement. A weak study does not become strong because you agree with it. A strong study does not become weak because you dislike its implications.

After appraisal, studies also need not receive equal influence in the synthesis. Differences in methodological credibility, precision, directness, and relevance can legitimately affect interpretation. The important distinction is that those judgments should be evidence-based rather than conclusion-based.

04 · A Practical Example

What Agreement-Based Selection Looks Like in Practice

Hypothetical Example

A finding that complicates your argument

Suppose you are reviewing research on a contested educational intervention. Your working expectation is that the intervention improves academic performance. Your search retrieves twelve studies that appear relevant.

Initial reaction Nine studies report favorable findings, while three report null or unfavorable findings. You immediately notice limitations in the three conflicting papers.
Apply the eligibility criteria All twelve meet the previously defined population, intervention, outcome, period, and study-design requirements. The direction of the result therefore provides no basis for excluding the three studies.
Appraise all studies consistently You apply the same design-appropriate appraisal questions to all twelve. Some favorable studies also contain important limitations that were less noticeable when you first read them.
Interpret rather than erase disagreement You discover that the conflicting studies examined different implementation conditions. Their results may therefore identify heterogeneity rather than simply constitute evidence to be discarded.
Synthesize what remains Your review can still conclude that the evidence generally favors the intervention if the evidence supports that interpretation, but it now specifies the conditions and uncertainty revealed by the conflicting studies.

The inconvenient papers did not have to overturn the conclusion to matter. Their inclusion made the conclusion more accurately conditional on the evidence.

05 · What Researchers Often Get Wrong

Selection Bias Can Hide Behind Reasonable-Sounding Decisions

Misconception

Can I Select Only the Best Studies?

Possibly, if your review methodology defines defensible quality requirements and applies them consistently. The danger lies in deciding which studies count as “best” after seeing their conclusions or using quality language to rationalize a preferred evidence set.

Misconception

Should I Include Every Paper That Disagrees With Me?

No. Disagreement does not make a source relevant. Apply the same eligibility criteria to supportive, conflicting, mixed, and null findings. Deliberately including irrelevant opposition would distort the literature in another direction.

Misconception

If a Study Has Serious Limitations, Shouldn't I Just Exclude It?

That depends on your review methodology and predetermined rules. Some reviews exclude studies meeting specified appraisal thresholds, while others include them and account for risk of bias during synthesis. What should not change opportunistically is the rule itself because of the study's conclusion.

Misconception

Is Citing Both Sides Enough to Prevent Cherry-Picking?

No. You can cite opposing papers while still selecting them asymmetrically, choosing unusually weak examples from one position, or describing equivalent methodological problems differently. Fair source selection concerns the process producing the evidence set, not merely the presence of citations pointing in different directions.

Misconception

Does a Majority of Papers Determine Which Sources Matter?

No. Frequency of a conclusion and evidential credibility are different questions. A minority result may be methodologically important, while multiple papers may derive from overlapping datasets or share similar limitations. Evaluate what the studies contribute rather than treating publications as votes.

06 · What This Means for You

Make Yourself Defend the Rule, Not the Result

A practical safeguard is to formulate selection decisions in terms that make sense without revealing the study's conclusion. If your reason for including or excluding a source remains defensible after imagining that its result were reversed, you are on firmer methodological ground.

A simple decision framework

If a paper supports your expected conclusion
Check its eligibility and quality with the same care you would give a conflicting paper.
If a paper contradicts your expected conclusion
Ask first whether it meets the established eligibility criteria, not whether you can find a reason to reject its conclusion.
If you discover a legitimate exclusion criterion after screening has begun
Determine whether it should apply consistently to all comparable studies, document the change when appropriate, and reconsider previously screened sources affected by it.
If a relevant study is methodologically weak
Handle that weakness according to the review's appraisal and synthesis method rather than quietly converting methodological concern into irrelevance.
If only a few eligible studies challenge the dominant finding
Do not dismiss them because they are numerically unusual; evaluate what the minority findings actually contribute.

The objective is not to force yourself to agree with every paper. It is to ensure that disagreement does not determine which papers are permitted to become evidence in the first place.

07 · A Quick Checklist

Before Finalizing Your Source Selection

Before locking your evidence set, check:
Are my inclusion and exclusion criteria based on the review question rather than the direction of findings?
Would I apply each exclusion criterion if the paper reached the opposite conclusion?
Does my search terminology give relevant competing findings a reasonable chance of being retrieved?
Have I kept eligibility decisions separate from methodological appraisal?
Am I scrutinizing favorable studies as carefully as unfavorable ones?
Have important changes to my eligibility criteria been justified and documented where appropriate?
For a systematic review, can another researcher trace why studies were included or excluded?
Have I avoided using author identity, journal prestige, or theoretical affiliation as substitutes for assessing the actual study?
08 · Frequently Asked Questions

Questions About Choosing Sources Without Cherry-Picking

Is it cherry-picking if I cite only studies relevant to my argument?

Not necessarily. Literature reviews require relevance judgments. The problem arises when relevance is defined according to whether a study supports the desired conclusion rather than whether it addresses the question under review.

Should I decide inclusion criteria before searching?

For systematic reviews, eligibility criteria should be pre-specified in the protocol. Other review types may be more iterative, but establishing explicit selection principles early can still reduce opportunities to redefine relevance around findings you encounter later.

Can I exclude a study because it has high risk of bias?

Potentially, depending on the review methodology and pre-established decision rules. JBI guidance allows review teams to determine how appraisal results affect inclusion or synthesis, but recommends that criteria and decision rules be specified and justified rather than improvised after individual results are known.

What if I discover that my original criteria were poorly designed?

Methodological plans sometimes require legitimate revision. Explain the rationale, apply the revised criterion consistently to affected studies, and document important deviations where the review method requires it. A transparent correction is preferable to preserving an indefensible rule.

Does independent screening eliminate confirmation bias?

No. Multiple reviewers can share assumptions, and screening still requires judgment. Independent assessment can nevertheless expose disagreements and reduce dependence on one reviewer's preferences, which is why it is standard practice in rigorous systematic-review methodologies.

What if the studies I disagree with really are methodologically worse?

They may be. Demonstrate that through the same design-appropriate appraisal criteria used for comparable studies. The issue is not whether conflicting evidence can be weaker; it is whether weakness is established by methodology rather than inferred from disagreement.

09 · The Bottom Line

Your Conclusion Should Emerge From the Evidence Set

The Bottom Line

Do not decide whether a source belongs in your review according to whether you agree with what it found. Define relevance and eligibility independently of the conclusion, then evaluate eligible evidence using consistent methodological standards.

You may ultimately give different studies different evidential weight. That is not cherry-picking when the difference follows from defensible appraisal. The crucial sequence is simple: determine whether the study belongs, assess what it can support, and only then decide what the literature as a whole allows you to conclude.

10 · Sources and Further Reading

Sources and Further Reading

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes