Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

Should a Replication Study Use the Same Population as the Original Study?

A replication study does not always need participants from the same population as the original research. Learn when population similarity is essential, when differences can be informative, and how participant selection affects the interpretation of replication findings.

1333
Population Selection in Replication Studies Guide 1333 of 1361
01 · The Question

Must a Replication Recruit the Same Type of Participants?

You want to replicate a published study, but the original participants may not be available to you. Perhaps the research was conducted with students at another university, patients from a particular hospital, or employees working in a different country.

Can you recruit participants from your own institution or community? Would the study still count as a direct replication, or would using a different population change its purpose?

These questions matter because a finding may depend on who participates. Differences in age, prior knowledge, socioeconomic circumstances, clinical characteristics, cultural context, or other attributes may influence the effect being investigated. Yet requiring every replication to recruit from the same location would unnecessarily restrict independent research.

02 · The Short Answer

Preserve Relevant Population Characteristics, Not Necessarily the Original Location

In Brief

A replication study does not necessarily need to recruit participants from the same institution, location, or population as the original study. For a direct replication, however, researchers should preserve the eligibility criteria and participant characteristics that are important to the original scientific claim. Deliberately changing the population can also be valuable when the purpose is to test whether a finding generalizes beyond its original context.

The appropriate choice depends on the intended inference and whether population differences could modify the effect. Researchers should distinguish a close test of the original finding from a test of its applicability to another population, and interpret results accordingly.

03 · What You Need to Know

How Population Selection Influences Replication Findings

What Does Population Mean in a Replication Study?

In research methodology, a population is the group of individuals or units about which a study seeks to make an inference. A sample consists of the individuals or units actually included in the research.

These concepts should not be confused with the recruitment site. A university, hospital, or organization may provide access to participants without defining the entire target population.

For example, a study conducted at one university may aim to investigate undergraduate students generally, students enrolled in a particular course, or only those meeting specific eligibility criteria. The intended population depends on the research question and sampling design, not simply the name of the institution.

A replication should therefore begin by distinguishing the original target population from the original sample and recruitment setting.

Target population The group to which the researchers intend their conclusions to apply.
Study sample The participants or units from whom the research data are actually collected.
Recruitment setting The institution, location, platform, or environment through which participants are identified and recruited.

These distinctions are essential because recruiting from another university does not necessarily mean changing the target population, while recruiting from the same university does not guarantee a comparable sample.

Why Population Similarity Matters in Direct Replication

A direct replication seeks to provide additional evidence about an earlier finding under closely corresponding research conditions. Participant characteristics may be part of those conditions.

Suppose an original study examines whether a particular instructional intervention improves programming performance among first-year students with no previous programming experience. Repeating the experiment with experienced software developers would change an important characteristic of the participants.

Prior programming knowledge could plausibly influence both the need for the intervention and the magnitude of its effect. A different result might therefore reflect population differences rather than a failure of the original finding to recur under comparable conditions.

For a close replication, preserving relevant eligibility criteria can reduce this ambiguity.

Nevertheless, matching every demographic characteristic is neither always feasible nor scientifically necessary. The relevant question is whether the characteristics are associated with the mechanisms, outcomes, or conditions under investigation.

The general principle of matching the original research design applies to population selection as well as interventions and measurement procedures.

Does Direct Replication Require Participants From the Same Institution?

No. Independent replication commonly involves recruiting new participants, and a different institution may provide an appropriate sample.

What matters is whether the new recruitment setting provides participants who are sufficiently comparable for the intended test.

For example, two universities may offer similar introductory programming courses with comparable prerequisites. A replication conducted at the second university could preserve the original eligibility criteria and instructional conditions even though the participants come from a different institution.

However, institutional differences may still matter. Curriculum structure, admission requirements, available technology, language of instruction, and prior student preparation could influence the effect.

Researchers should examine these possibilities rather than assume that institutional labels establish either equivalence or difference.

Which Participant Characteristics Should Be Preserved?

The relevant characteristics depend on the phenomenon being studied. They may include demographic attributes, educational background, baseline ability, health status, professional experience, language proficiency, or prior exposure to an intervention.

For example, age may be important in developmental research, baseline severity in clinical research, and prior knowledge in educational intervention studies.

The selection process should begin with the original study's inclusion and exclusion criteria. Researchers should then evaluate whether additional characteristics are plausible moderators of the effect.

A moderator is a variable associated with differences in the magnitude or direction of a relationship or treatment effect. Its relevance should be supported by theory, prior evidence, or a defensible substantive rationale.

Not every difference between samples is an effect modifier. Researchers should avoid treating demographic similarity as a substitute for identifying characteristics that actually matter to the research claim.

How Can Population Differences Affect Replication Results?

Population differences can influence replication findings through several pathways.

First, participants may respond differently to an intervention because of prior experience, baseline characteristics, or contextual conditions. This is a form of effect heterogeneity.

Second, different recruitment processes may introduce selection differences. Volunteers responding to an online advertisement, for example, may differ systematically from participants recruited through a required course activity.

Third, measurement properties may vary across populations. A questionnaire validated in one language or cultural context may not function identically in another.

Finally, population composition may affect the overall average effect even when subgroup-specific effects remain similar. If an intervention benefits beginners more than experienced participants, samples containing different proportions of beginners may produce different average effects.

These possibilities mean that a discrepancy between studies should not automatically be interpreted as evidence that one study was incorrect.

What Is the Difference Between Population Comparability and Generalizability?

Population comparability concerns whether two studies include sufficiently similar participants for a particular comparison. Generalizability concerns whether findings can be extended beyond the conditions and participants originally studied.

A replication using closely corresponding participants may provide a relatively focused test of the original finding. A study using a deliberately different population may provide evidence about the boundaries of that finding.

Research Objective Population Strategy Primary Interpretation
Close direct replication Preserve relevant eligibility and participant characteristics. Tests whether the finding is supported under comparable population conditions.
Replication in another institution Recruit comparable participants at a different site. Tests the finding with an independent sample while introducing possible site differences.
Population generalizability study Deliberately recruit a meaningfully different population. Examines whether the effect extends beyond the original population.
Effect-heterogeneity investigation Include populations or subgroups expected to differ in their responses. Investigates whether and how effects vary across participant characteristics.

These objectives can overlap, but they should not be treated as interchangeable. A new population may simultaneously provide independent evidence about a claim and information about its generalizability.

Researchers specifically interested in testing generalizability across populations and settings should define the intended target population and contextual contrasts explicitly.

Should a Replication Deliberately Use a Different Population?

Sometimes. Deliberately changing the population may be scientifically valuable when the original claim is intended to apply broadly or when theory suggests that an effect depends on participant characteristics.

For example, an instructional intervention initially evaluated among university students might be tested among adult learners in professional training. The new study could examine whether the intervention remains effective under different learner characteristics.

However, changing the population may introduce differences in prior knowledge, motivation, educational environment, and assessment relevance. These should be considered in the design rather than addressed only after observing the results.

Where the research question concerns generalizability, researchers should consider changing the population while preserving other research conditions to improve interpretability.

Changing multiple major features simultaneously may make it difficult to determine which differences account for divergent findings.

Can a Replication Use More Diverse Participants Than the Original Study?

Yes. Broader recruitment can increase the range of participant characteristics represented in the study and may help examine whether the original finding applies beyond a narrow sample.

Nevertheless, greater diversity does not automatically establish representativeness or generalizability.

A heterogeneous convenience sample may still be systematically selected. Moreover, an overall average effect can conceal meaningful differences between participant groups.

Researchers should define the population to which they wish to generalize, explain how participants are selected, and consider whether subgroup analyses or interaction tests are theoretically justified and sufficiently precise.

Broader recruitment is particularly informative when accompanied by a design capable of examining plausible effect modifiers.

How Should Researchers Evaluate Population Comparability?

Begin by comparing the original and replication studies on characteristics relevant to the research claim. These may include eligibility criteria, recruitment methods, baseline measures, and contextual factors.

A comparison table can help identify meaningful differences, but demographic summaries alone cannot establish equivalence.

Researchers should ask whether the differences could plausibly change treatment response, measurement interpretation, or selection into the study.

Where quantitative data are available, descriptive statistics can characterize differences between samples. Standardized differences may help describe imbalance, but no universal numerical cutoff establishes that two independently recruited populations are scientifically equivalent.

Substantive knowledge remains essential. A small difference in a critical effect modifier may matter more than a large difference in an irrelevant characteristic.

Does Using the Same Eligibility Criteria Guarantee Comparable Samples?

No. Eligibility criteria define who may participate, but they do not determine who actually enrolls.

Two studies using identical inclusion criteria may recruit participants with different baseline abilities, motivations, socioeconomic characteristics, or other attributes.

Recruitment channels, participation incentives, nonresponse, and institutional circumstances may influence sample composition.

Researchers should therefore report both eligibility criteria and the observed characteristics of enrolled participants. Where possible, they should also describe recruitment and participation patterns that may affect selection.

The procedural question of preserving recruitment and eligibility procedures is related to, but distinct from, establishing population comparability.

How Should Population Differences Be Considered in Statistical Analysis?

Population differences may be examined through descriptive comparisons, prespecified subgroup analyses, interaction models, or other approaches appropriate to the research design.

However, statistical adjustment does not automatically make two populations equivalent. Adjustment requires appropriate measurement, model assumptions, and sufficient information about relevant variables.

When evaluating effect modification, researchers should examine the uncertainty surrounding subgroup estimates and interaction effects. A statistically significant result in one subgroup and a nonsignificant result in another does not, by itself, demonstrate that the subgroup effects differ.

Researchers should also distinguish between estimating an effect in the replication sample and transporting an effect to another target population. The latter may require additional assumptions about selection, effect modifiers, and the relationship between the study sample and target population.

Methods for generalizing or transporting causal effects can be useful when their assumptions are defensible, but they cannot eliminate all uncertainty about unmeasured population differences.

How Should Researchers Interpret Different Results Across Populations?

Suppose the original study reports a positive effect, while a replication using another population estimates a smaller effect.

Several explanations are possible. The effect may genuinely differ across populations, the original estimate may have been imprecise, the replication may have insufficient precision, or methodological differences may have influenced the findings.

Researchers should compare effect estimates and uncertainty intervals rather than rely exclusively on whether each study crosses a statistical-significance threshold.

They should also examine whether the populations differ on plausible effect modifiers and whether the intervention, outcome, and comparison conditions remain sufficiently similar.

Watch Out

A different result in a different population does not automatically demonstrate that the original finding failed to replicate. It may instead reveal effect heterogeneity, measurement differences, sampling uncertainty, or other changes in the conditions under which the effect was estimated.

When several research conditions differ simultaneously, explaining differences in replication results becomes more difficult because competing explanations cannot easily be separated.

04 · A Practical Example

Replicating a Learning Intervention With a Different Student Population

Hypothetical Example

Testing an Automated Feedback Intervention Across Universities

An original randomized experiment evaluates automated formative feedback among 120 first-year undergraduate programming students with limited prior programming experience. The study compares automated explanatory feedback with basic correctness feedback over four weeks and measures programming performance afterward.

A replication team at another university can recruit 200 students, but its available population includes both beginners and students with substantial prior programming experience.

Identify the original population.

The original study concerns first-year programming students with limited previous experience. Prior programming knowledge is potentially relevant because it may influence how much students benefit from explanatory feedback.

Define the close-replication sample.

The team recruits students meeting eligibility criteria comparable to those used in the original study. The intervention, comparison, duration, and outcome procedures are preserved.

Evaluate the broader population.

Experienced students are not automatically included in the primary close-replication analysis. The researchers consider whether a separate prespecified comparison could examine the intervention among students with different prior experience.

Plan the analysis.

The primary analysis evaluates the original treatment contrast among comparable beginners. A separate analysis may investigate whether prior experience modifies the effect, provided the design and sample support a meaningful estimate.

Interpret the evidence.

The close-replication sample provides evidence about the original finding under relatively comparable participant conditions. The additional experienced-student sample addresses a different question about the effect's applicability beyond beginners.

Suppose the intervention produces a positive estimated effect among beginners but little estimated benefit among experienced students. Such a pattern could be consistent with effect modification by prior experience, but the strength of that conclusion would depend on the precision of the estimates and an appropriate test of the difference between groups.

The example illustrates why population selection should follow the intended scientific question. Recruiting a broader sample can be valuable, but it should not obscure which participants provide the closest test of the original finding.

05 · What Researchers Often Get Wrong

Common Misconceptions About Population Selection in Replication

Misconception

Must a Direct Replication Recruit From the Original Institution?

No. A different institution can provide an appropriate sample if relevant eligibility criteria and research conditions remain comparable. Institutional differences should be evaluated rather than treated as automatically disqualifying.

Misconception

Does Using the Same Eligibility Criteria Guarantee Population Equivalence?

No. Identical eligibility criteria can produce samples with different baseline characteristics because recruitment, participation, and selection processes may differ.

Misconception

Is a More Diverse Sample Automatically More Generalizable?

No. Diversity may improve coverage of relevant participant characteristics, but generalizability also depends on the target population, selection process, effect heterogeneity, and assumptions supporting inference beyond the observed sample.

Misconception

Does a Different Population Automatically Make the Study a Conceptual Replication?

Not necessarily. A study may preserve the same intervention, comparison, and outcome while recruiting comparable participants elsewhere. Deliberate changes to substantively important population characteristics may shift the study toward testing generalizability.

Misconception

Does a Different Result Prove That the Original Finding Was Population-Specific?

No. A discrepancy may reflect sampling uncertainty, methodological differences, or bias. Evidence of population-dependent effects requires an appropriate comparison and sufficient precision.

Misconception

Can Statistical Adjustment Eliminate All Population Differences?

No. Adjustment depends on measured variables and modeling assumptions. Unmeasured effect modifiers, selection mechanisms, and measurement differences may remain consequential.

06 · What This Means for You

Select Participants According to the Replication's Intended Inference

Before recruitment, identify the original study's target population, eligibility criteria, recruitment process, and participant characteristics that could plausibly influence the result.

Then decide whether the new study primarily seeks to reproduce the finding under comparable conditions or examine its applicability to a different population.

A simple decision framework

If your primary objective is close direct replication
Preserve relevant eligibility criteria and recruit participants whose characteristics support a comparable test.
If participants are available only at another institution
Evaluate relevant participant and institutional differences rather than assuming the new site prevents replication.
If your objective is to examine generalizability
Define the new target population and deliberately select characteristics that make the population contrast scientifically informative.
If you expect treatment effects to vary across participant groups
Specify plausible moderators, plan appropriate comparisons, and ensure sufficient precision for the intended analysis.
If important population differences are unavoidable
Document them, evaluate their likely implications, and qualify the interpretation of agreement or disagreement with the original result.

A practical population-comparison record should include the original and replication eligibility criteria, recruitment methods, baseline characteristics, relevant contextual differences, and anticipated effect modifiers.

Where population characteristics are central to the original claim, prioritize comparability. Where the research question concerns broader applicability, purposeful differences may be more informative than close matching.

Neither approach is inherently superior. Their value depends on whether the selected population allows the study to answer its intended question.

07 · A Quick Checklist

Before Selecting Participants for a Replication Study

Before finalizing the replication population, check:
Identify the original target population separately from the original sample and recruitment setting.
Retrieve the original inclusion criteria, exclusion criteria, and recruitment procedures.
Identify participant characteristics that could plausibly modify the effect.
Determine whether the objective is close replication, generalizability testing, or both.
Compare relevant characteristics of the original and proposed replication populations.
Evaluate potential selection differences introduced by recruitment methods and participation incentives.
Consider whether the original measurement instruments function comparably in the new population.
Prespecify important population comparisons or effect-modification analyses when appropriate.
Document population differences and their implications for interpreting replication results.
08 · Frequently Asked Questions

Frequently Asked Questions About Replication Study Populations

Can I replicate a study conducted in another country?

Yes. A replication in another country may be informative, but researchers should evaluate differences in language, institutions, cultural context, participant characteristics, and other factors relevant to the effect.

Can a replication use students from a different university?

Yes. Students from another university may provide a suitable sample if their relevant characteristics and research conditions are sufficiently comparable. The institution itself need not be identical.

Can a replication use participants of different ages?

Possibly. The implications depend on whether age is related to the phenomenon or intervention. In developmental research, age differences may be central, while in other contexts they may be less consequential.

Should a replication use the same sample size as the original study?

No. Sample size should be justified using the replication's intended analysis, plausible effect magnitude, and desired precision or power. Reproducing the original sample size is not inherently necessary.

Can a replication include participants who were excluded from the original study?

Yes, when the purpose includes testing broader applicability, but doing so changes the eligible population. Researchers should distinguish the close-replication comparison from analyses involving newly eligible participants.

Does a different population make the replication less valuable?

Not necessarily. A different population may provide valuable evidence about generalizability or effect heterogeneity. Its usefulness depends on whether the differences are intentional, scientifically justified, and appropriately interpreted.

Can researchers statistically control for differences between populations?

Some measured differences can be addressed using appropriate statistical methods, but valid adjustment requires assumptions and sufficient data. It cannot automatically account for unmeasured effect modifiers or selection mechanisms.

What if the replication produces a different result in the new population?

Compare effect estimates and uncertainty, examine relevant population characteristics, and assess other methodological differences. A discrepant result may suggest effect heterogeneity, but it does not establish that explanation by itself.

09 · The Bottom Line

Match the Population to the Scientific Question

The Bottom Line

A replication study does not need participants from the same institution or location as the original research, but a close direct replication should preserve population characteristics that are important to the original claim. Deliberately different populations can be appropriate when the objective is to examine generalizability or effect heterogeneity.

Population similarity should be judged through substantive relevance rather than superficial demographic matching. Define the intended population, document meaningful differences, and interpret the replication findings according to the conditions under which the new evidence was obtained.

10 · Sources and Further Reading

Authoritative Sources on Replication Populations and Generalizability

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes