03 · What You Need to Know
How Population Selection Influences Replication Findings
What Does Population Mean in a Replication Study?
In research methodology, a population is the group of individuals or units about which a study seeks to make an inference. A sample consists of the individuals or units actually included in the research.
These concepts should not be confused with the recruitment site. A university, hospital, or organization may provide access to participants without defining the entire target population.
For example, a study conducted at one university may aim to investigate undergraduate students generally, students enrolled in a particular course, or only those meeting specific eligibility criteria. The intended population depends on the research question and sampling design, not simply the name of the institution.
A replication should therefore begin by distinguishing the original target population from the original sample and recruitment setting.
Target population
The group to which the researchers intend their conclusions to apply.
Study sample
The participants or units from whom the research data are actually collected.
Recruitment setting
The institution, location, platform, or environment through which participants are identified and recruited.
These distinctions are essential because recruiting from another university does not necessarily mean changing the target population, while recruiting from the same university does not guarantee a comparable sample.
Why Population Similarity Matters in Direct Replication
A direct replication seeks to provide additional evidence about an earlier finding under closely corresponding research conditions. Participant characteristics may be part of those conditions.
Suppose an original study examines whether a particular instructional intervention improves programming performance among first-year students with no previous programming experience. Repeating the experiment with experienced software developers would change an important characteristic of the participants.
Prior programming knowledge could plausibly influence both the need for the intervention and the magnitude of its effect. A different result might therefore reflect population differences rather than a failure of the original finding to recur under comparable conditions.
For a close replication, preserving relevant eligibility criteria can reduce this ambiguity.
Nevertheless, matching every demographic characteristic is neither always feasible nor scientifically necessary. The relevant question is whether the characteristics are associated with the mechanisms, outcomes, or conditions under investigation.
The general principle of matching the original research design applies to population selection as well as interventions and measurement procedures.
Does Direct Replication Require Participants From the Same Institution?
No. Independent replication commonly involves recruiting new participants, and a different institution may provide an appropriate sample.
What matters is whether the new recruitment setting provides participants who are sufficiently comparable for the intended test.
For example, two universities may offer similar introductory programming courses with comparable prerequisites. A replication conducted at the second university could preserve the original eligibility criteria and instructional conditions even though the participants come from a different institution.
However, institutional differences may still matter. Curriculum structure, admission requirements, available technology, language of instruction, and prior student preparation could influence the effect.
Researchers should examine these possibilities rather than assume that institutional labels establish either equivalence or difference.
Which Participant Characteristics Should Be Preserved?
The relevant characteristics depend on the phenomenon being studied. They may include demographic attributes, educational background, baseline ability, health status, professional experience, language proficiency, or prior exposure to an intervention.
For example, age may be important in developmental research, baseline severity in clinical research, and prior knowledge in educational intervention studies.
The selection process should begin with the original study's inclusion and exclusion criteria. Researchers should then evaluate whether additional characteristics are plausible moderators of the effect.
A moderator is a variable associated with differences in the magnitude or direction of a relationship or treatment effect. Its relevance should be supported by theory, prior evidence, or a defensible substantive rationale.
Not every difference between samples is an effect modifier. Researchers should avoid treating demographic similarity as a substitute for identifying characteristics that actually matter to the research claim.
How Can Population Differences Affect Replication Results?
Population differences can influence replication findings through several pathways.
First, participants may respond differently to an intervention because of prior experience, baseline characteristics, or contextual conditions. This is a form of effect heterogeneity.
Second, different recruitment processes may introduce selection differences. Volunteers responding to an online advertisement, for example, may differ systematically from participants recruited through a required course activity.
Third, measurement properties may vary across populations. A questionnaire validated in one language or cultural context may not function identically in another.
Finally, population composition may affect the overall average effect even when subgroup-specific effects remain similar. If an intervention benefits beginners more than experienced participants, samples containing different proportions of beginners may produce different average effects.
These possibilities mean that a discrepancy between studies should not automatically be interpreted as evidence that one study was incorrect.
What Is the Difference Between Population Comparability and Generalizability?
Population comparability concerns whether two studies include sufficiently similar participants for a particular comparison. Generalizability concerns whether findings can be extended beyond the conditions and participants originally studied.
A replication using closely corresponding participants may provide a relatively focused test of the original finding. A study using a deliberately different population may provide evidence about the boundaries of that finding.
| Research Objective |
Population Strategy |
Primary Interpretation |
| Close direct replication |
Preserve relevant eligibility and participant characteristics. |
Tests whether the finding is supported under comparable population conditions. |
| Replication in another institution |
Recruit comparable participants at a different site. |
Tests the finding with an independent sample while introducing possible site differences. |
| Population generalizability study |
Deliberately recruit a meaningfully different population. |
Examines whether the effect extends beyond the original population. |
| Effect-heterogeneity investigation |
Include populations or subgroups expected to differ in their responses. |
Investigates whether and how effects vary across participant characteristics. |
These objectives can overlap, but they should not be treated as interchangeable. A new population may simultaneously provide independent evidence about a claim and information about its generalizability.
Researchers specifically interested in testing generalizability across populations and settings should define the intended target population and contextual contrasts explicitly.
Should a Replication Deliberately Use a Different Population?
Sometimes. Deliberately changing the population may be scientifically valuable when the original claim is intended to apply broadly or when theory suggests that an effect depends on participant characteristics.
For example, an instructional intervention initially evaluated among university students might be tested among adult learners in professional training. The new study could examine whether the intervention remains effective under different learner characteristics.
However, changing the population may introduce differences in prior knowledge, motivation, educational environment, and assessment relevance. These should be considered in the design rather than addressed only after observing the results.
Where the research question concerns generalizability, researchers should consider changing the population while preserving other research conditions to improve interpretability.
Changing multiple major features simultaneously may make it difficult to determine which differences account for divergent findings.
Can a Replication Use More Diverse Participants Than the Original Study?
Yes. Broader recruitment can increase the range of participant characteristics represented in the study and may help examine whether the original finding applies beyond a narrow sample.
Nevertheless, greater diversity does not automatically establish representativeness or generalizability.
A heterogeneous convenience sample may still be systematically selected. Moreover, an overall average effect can conceal meaningful differences between participant groups.
Researchers should define the population to which they wish to generalize, explain how participants are selected, and consider whether subgroup analyses or interaction tests are theoretically justified and sufficiently precise.
Broader recruitment is particularly informative when accompanied by a design capable of examining plausible effect modifiers.
How Should Researchers Evaluate Population Comparability?
Begin by comparing the original and replication studies on characteristics relevant to the research claim. These may include eligibility criteria, recruitment methods, baseline measures, and contextual factors.
A comparison table can help identify meaningful differences, but demographic summaries alone cannot establish equivalence.
Researchers should ask whether the differences could plausibly change treatment response, measurement interpretation, or selection into the study.
Where quantitative data are available, descriptive statistics can characterize differences between samples. Standardized differences may help describe imbalance, but no universal numerical cutoff establishes that two independently recruited populations are scientifically equivalent.
Substantive knowledge remains essential. A small difference in a critical effect modifier may matter more than a large difference in an irrelevant characteristic.
Does Using the Same Eligibility Criteria Guarantee Comparable Samples?
No. Eligibility criteria define who may participate, but they do not determine who actually enrolls.
Two studies using identical inclusion criteria may recruit participants with different baseline abilities, motivations, socioeconomic characteristics, or other attributes.
Recruitment channels, participation incentives, nonresponse, and institutional circumstances may influence sample composition.
Researchers should therefore report both eligibility criteria and the observed characteristics of enrolled participants. Where possible, they should also describe recruitment and participation patterns that may affect selection.
The procedural question of preserving recruitment and eligibility procedures is related to, but distinct from, establishing population comparability.
How Should Population Differences Be Considered in Statistical Analysis?
Population differences may be examined through descriptive comparisons, prespecified subgroup analyses, interaction models, or other approaches appropriate to the research design.
However, statistical adjustment does not automatically make two populations equivalent. Adjustment requires appropriate measurement, model assumptions, and sufficient information about relevant variables.
When evaluating effect modification, researchers should examine the uncertainty surrounding subgroup estimates and interaction effects. A statistically significant result in one subgroup and a nonsignificant result in another does not, by itself, demonstrate that the subgroup effects differ.
Researchers should also distinguish between estimating an effect in the replication sample and transporting an effect to another target population. The latter may require additional assumptions about selection, effect modifiers, and the relationship between the study sample and target population.
Methods for generalizing or transporting causal effects can be useful when their assumptions are defensible, but they cannot eliminate all uncertainty about unmeasured population differences.
How Should Researchers Interpret Different Results Across Populations?
Suppose the original study reports a positive effect, while a replication using another population estimates a smaller effect.
Several explanations are possible. The effect may genuinely differ across populations, the original estimate may have been imprecise, the replication may have insufficient precision, or methodological differences may have influenced the findings.
Researchers should compare effect estimates and uncertainty intervals rather than rely exclusively on whether each study crosses a statistical-significance threshold.
They should also examine whether the populations differ on plausible effect modifiers and whether the intervention, outcome, and comparison conditions remain sufficiently similar.
Watch Out
A different result in a different population does not automatically demonstrate that the original finding failed to replicate. It may instead reveal effect heterogeneity, measurement differences, sampling uncertainty, or other changes in the conditions under which the effect was estimated.
When several research conditions differ simultaneously, explaining differences in replication results becomes more difficult because competing explanations cannot easily be separated.