Should You Decide Which Study Designs Matter Before Searching?

Should Study Design Be Decided Before You Search?

Suppose you are planning a literature review about an educational intervention. Should you look only for randomized controlled trials? Include quasi-experimental studies? What about observational studies, qualitative research, mixed-methods studies, or implementation evaluations?

The temptation is to postpone that decision until you see what the database contains. Another common approach is to assume that one design is inherently “best” and search only for that design.

Both approaches can cause problems. Study design should follow from the question you are trying to answer. Yet deciding which designs matter conceptually is not the same as requiring a particular design label in every database record you retrieve.

Decide Which Designs Can Answer the Question, but Search Carefully

The Question Should Determine Which Designs Matter

There is no universally appropriate study design for every research question

A study design is useful because of what it allows researchers to investigate, not because it occupies a fixed position in an abstract hierarchy.

If your question concerns the causal effect of an intervention, randomized trials may be especially informative when randomization is feasible and ethically appropriate. If your question concerns prevalence, diagnostic accuracy, prognosis, experiences, implementation, rare harms, or long-term real-world patterns, other designs may be necessary or better suited to the particular inference.

Cochrane requires eligible study designs to be predefined for intervention reviews and recommends focusing on concrete design features rather than relying only on design labels. It also states that authors should justify decisions to restrict a review to randomized trials or to include non-randomized studies according to the review question and potential for bias.

Start with the inference you need to make

Instead of beginning with “Which study design is best?”, ask what you need the evidence to tell you.

These are examples rather than universal prescriptions. Design suitability depends on the exact question and the inference you intend to make.

Study-design labels can be deceptively vague

Terms such as observational study, controlled study, longitudinal study, and even cohort study can conceal important methodological differences.

Cochrane therefore recommends defining eligibility in terms of specific design features rather than relying solely on labels, particularly when non-randomized studies are eligible.

For example, if a review requires a comparison group, say so. If intervention allocation must be randomized, specify that feature. If repeated measurement over time is necessary, make the temporal requirement explicit. This is more reproducible than assuming every author uses the same methodological vocabulary.

Do not choose a design merely because it sounds more rigorous

A randomized trial is highly useful for some causal questions, but randomization does not make a study relevant to every question. A trial may tell you whether an intervention changes an outcome under particular conditions while offering little insight into how participants experience the intervention, why implementation succeeds in one institution and fails in another, or how common a phenomenon is in a population.

The design must therefore fit the question. JBI guidance similarly links eligible study types to the review objective and methodology rather than treating study design as an isolated criterion.

This is one reason to define what counts as relevant before examining the results. Study design then becomes one justified dimension of relevance rather than a prestige filter applied after the fact.

Deciding eligible designs does not require using a design filter

This distinction is easy to miss. You may decide that only randomized trials are eligible but still use a search strategy designed for high sensitivity rather than simply adding the phrase “randomized controlled trial” to your keywords. Conversely, you may decide that several designs are eligible and assess the design during screening.

Search filters can be valuable, particularly validated methodological filters developed for particular databases and study types. But their appropriateness depends on the purpose of the search, the indexing available, and the balance between retrieving irrelevant records and missing eligible ones.

Sometimes you should search more broadly than the final eligible designs

Broad retrieval can be reasonable when study-design terminology is inconsistent, database indexing is uncertain, the evidence base is small, or the consequences of missing eligible studies are substantial.

You can then determine the actual design from the methods section during screening. Cochrane's reporting guidance explicitly emphasizes design features such as how groups were formed and whether allocation occurred at individual or cluster level, rather than relying on labels alone.

This distinction also helps prevent overly strict search criteria from hiding important evidence.

The decision may differ for exploratory and systematic searching

If you are conducting an exploratory search to understand an unfamiliar field, you may intentionally begin without strict design restrictions. Seeing the methodological diversity of the literature may itself be informative.

A systematic review is different. Prespecified eligibility criteria are a defining methodological feature, and PRISMA 2020 expects authors to report inclusion and exclusion criteria as well as the methods used to decide whether studies met them.

Even then, prespecification does not mean arbitrary restriction. The rationale for eligible study designs should be connected to the review objective and the kind of inference the synthesis is intended to support.

How the Same Topic Can Require Different Study Designs

If all three researchers automatically restricted their searches to randomized controlled trials, only the first question would be reasonably aligned with that decision. The other two could systematically lose the evidence most capable of answering what they actually want to know.

Common Mistakes When Choosing Eligible Study Designs

Choose Designs by Asking What Evidence Could Answer the Question

Before deciding which designs to include, write down the inference you need to make. Are you estimating an intervention effect, describing prevalence, understanding experiences, examining associations, studying change over time, or investigating implementation?

Then identify which methodological features allow studies to contribute credible evidence to that question. Only after that should you decide whether study-design restrictions belong in the database search itself.

Study design is only one boundary. Population, context, intervention or phenomenon, and outcomes may also determine eligibility. The next step is often deciding which populations matter before searching without making the search so restrictive that relevant evidence disappears.

Before Restricting by Study Design

Questions About Study Designs and Literature Searching

Choose Designs for the Question, Not for Their Reputation

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.