03 · What You Need to Know
How Measurement Decisions Affect Replication Validity and Comparability
Why Measurement Is Central to Replication
A replication study seeks to provide additional evidence about an earlier scientific claim. That claim usually depends on how the original researchers defined and measured their variables.
For example, a study reporting that an instructional intervention improves learning may have measured learning through a delayed knowledge test. Another study using self-reported confidence might investigate a related outcome, but it would not necessarily test the same empirical finding.
Measurement therefore affects the relationship between the theoretical construct and the observable evidence.
Nosek and Errington (2020) emphasize that replication should be evaluated according to whether the new study provides diagnostic evidence about the original claim. Preserving the original outcome measure can support this objective, particularly when the claim concerns a specific operationally defined effect.
However, reproducing the original instrument is not sufficient if its scores are unreliable, its administration differs materially, or its interpretation is inappropriate for the new population.
Distinguish Constructs, Instruments, and Observed Scores
A construct is an attribute or phenomenon that researchers intend to investigate, such as academic motivation, programming competence, anxiety, or scientific reasoning.
An instrument is the tool or procedure used to obtain evidence about that construct. It might be a questionnaire, performance assessment, observation schedule, interview protocol, or physiological measurement device.
An observed score is the result produced through the measurement procedure. Its meaning depends on the instrument, scoring rules, administration conditions, and intended interpretation.
Construct
The theoretical attribute or phenomenon being investigated.
Instrument
The tool or procedure used to collect evidence about the construct.
Observed score
The recorded result whose interpretation requires appropriate validity evidence.
Two instruments can claim to assess the same construct while emphasizing different dimensions of it. Conversely, the same instrument may produce scores with different interpretations across populations or contexts.
The Standards for Educational and Psychological Testing, developed jointly by AERA, APA, and NCME, emphasize that validity concerns evidence supporting interpretations and uses of test scores. It is not simply a permanent property attached to an instrument.
Why Direct Replication Usually Favors the Original Primary Measure
Using the original primary measure helps preserve the operational definition of the outcome and reduces one potential explanation for differences between studies.
For example, if the original study measures programming performance using a standardized set of coding tasks, reproducing those tasks and their scoring rubric provides a relatively close comparison.
Replacing the assessment with a multiple-choice knowledge test may change the cognitive processes being evaluated. A difference in results could then reflect the measurement change rather than a difference in the intervention's effect.
Direct replication therefore generally favors retaining the original primary instrument when it is available, appropriate, and sufficiently defensible.
This principle is part of the broader requirement to preserve the procedures that define the original scientific test.
Must Every Secondary Measure Also Be Reproduced?
Not necessarily. The importance of preserving a measure depends on its relationship to the original claim and the replication's objectives.
The primary outcome usually deserves the greatest attention because it defines the main result being evaluated. Secondary outcomes may also be important when the original conclusions depend on them or when they help interpret the intervention's effects.
Measures unrelated to the target finding need not always be reproduced. However, omitting a measure that captures a critical moderator, manipulation check, or potential source of bias may weaken interpretation.
Researchers should identify the original study's outcome hierarchy and distinguish measures essential to the replication from those included for supplementary purposes.
When Can an Alternative Instrument Be Appropriate?
An alternative instrument may be justified when the original is unavailable, inappropriate for the new population, technically obsolete, or supported by inadequate validity evidence.
However, replacing a measure should involve more than selecting another instrument with a similar name.
Researchers should examine whether the alternative assesses the same construct, covers comparable content, uses an appropriate response process, and supports the intended score interpretation.
They should also consider reliability, sensitivity to change, measurement range, administration burden, and possible floor or ceiling effects.
If the alternative measure changes the operational definition of the outcome, the study may be better characterized as a conceptual replication or a complementary test of the original claim.
The distinction between preserving the original test and introducing a different operationalization is central to deciding how closely a replication should match the original design.
Does Using the Same Questionnaire Guarantee Measurement Equivalence?
No. An instrument may function differently across languages, cultural contexts, age groups, educational backgrounds, or administration settings.
For example, a questionnaire item referring to classroom participation may carry different implications in educational systems where participation expectations differ substantially.
Participants may also interpret response categories differently or respond according to different social expectations.
These differences can affect score comparability even when the questionnaire wording remains unchanged.
Researchers should therefore distinguish identical administration from equivalent measurement.
What Is Measurement Invariance, and When Does It Matter?
Measurement invariance concerns whether a measurement model operates comparably across groups or conditions. It is particularly relevant when researchers compare scores from instruments intended to measure latent constructs.
In confirmatory factor analysis, researchers may examine increasingly restrictive forms of invariance, including configural, metric, and scalar invariance.
| Level |
General Meaning |
Typical Relevance |
| Configural invariance |
The same general factor structure is supported across groups. |
Provides an initial basis for comparing measurement models. |
| Metric invariance |
Corresponding factor loadings are constrained to equality. |
Supports certain comparisons involving relationships between latent constructs. |
| Scalar invariance |
Corresponding item intercepts, or relevant thresholds for categorical indicators, are constrained appropriately. |
Supports latent mean comparisons under the model's assumptions. |
| Strict invariance |
Residual variances are additionally constrained. |
Addresses a more restrictive form of measurement equivalence. |
These distinctions are not universal requirements for every replication. Their applicability depends on the instrument, data type, measurement model, and intended comparisons.
Putnick and Bornstein (2016) discuss measurement invariance conventions and reporting practices, including the need to consider how invariance evidence supports substantive interpretation.
Researchers should not assume that passing or failing a single numerical fit criterion provides a complete judgment about measurement equivalence.
Can the Original Instrument Be Translated?
Yes, but translation introduces methodological considerations beyond replacing words in one language with words in another.
A translated instrument should preserve the intended construct, item meaning, response processes, and scoring interpretation as far as possible.
Translation procedures may involve independent translation, reconciliation, expert review, cognitive interviewing, pilot testing, and psychometric evaluation. The appropriate process depends on the instrument and intended use.
Back-translation can be useful, but it does not independently establish conceptual equivalence or validity.
The International Test Commission's guidelines for translating and adapting tests emphasize the need to consider linguistic, cultural, technical, and psychometric issues throughout adaptation.
A translated measure may support a valuable replication, but the researcher should report the adaptation process and avoid assuming that scores are directly interchangeable with the original.
Should the Same Scoring Rules Be Preserved?
Generally, yes, when the purpose is to compare the original operational outcome directly.
Scoring decisions can affect the meaning, distribution, and interpretation of results. Examples include reverse coding, weighting items, calculating subscale scores, applying cutoffs, and handling missing responses.
Changing from a total score to a standardized score may preserve some comparisons under particular conditions, but it can also change the interpretation of effect estimates.
Similarly, converting a continuous outcome into categories can discard information and alter the statistical comparison.
Researchers should reproduce the original scoring rules when defensible, document any corrections, and distinguish original-score analyses from alternative scoring approaches.
Does Measurement Timing Need to Match?
Timing may be part of the operational definition of an outcome.
An immediate post-test may assess short-term performance, while a delayed test may provide evidence about retention. An intervention can produce different effects at these time points.
Therefore, replacing a seven-day follow-up with an immediate assessment is not merely an administrative adjustment.
Researchers should preserve theoretically relevant assessment intervals and consider whether timing differences change the phenomenon being measured.
Where additional assessments are useful, they can be included without necessarily replacing the original primary measurement time point.
Can Researchers Add Better Measures Without Replacing the Original?
Yes. Including the original primary measure alongside an additional validated instrument can be an informative strategy.
The original measure supports direct comparison, while the additional instrument may provide complementary evidence about construct validity, robustness, or broader theoretical implications.
However, adding measures can increase participant burden, introduce order effects, or change the assessment experience. Researchers should consider these consequences when designing the study.
Additional measures should have clearly specified purposes, and their analyses should be distinguished from the primary replication test.
This approach may help researchers improve measurement procedures while preserving direct replication, provided the added procedures do not materially alter the original conditions.
How Do Measurement Changes Affect Statistical Comparability?
Two studies using different outcome scales may produce effect estimates that cannot be compared directly without additional assumptions.
Standardized effect sizes can sometimes facilitate comparison across instruments, but standardization does not establish that the instruments measure the same construct.
Differences in reliability, score variability, sensitivity, and population composition may influence standardized estimates.
Researchers should therefore evaluate both substantive measurement equivalence and statistical comparability.
The choice of statistical analysis for the replication should reflect the outcome scale, estimand, and assumptions supported by the data.
What If the Original Measure Has Serious Validity Problems?
Preserving a measure with serious limitations may reproduce the original operational outcome without providing strong evidence about the intended theoretical construct.
For example, a questionnaire intended to assess actual competence may primarily capture self-confidence. Repeating it can test whether the original score pattern recurs, but it cannot automatically establish an effect on competence.
Researchers should distinguish replication of the original measured result from validation of the broader theoretical interpretation.
Where feasible, a study may retain the original measure for comparability while including a better-supported measure as a separate outcome.
If the original measure is fundamentally unsuitable, replacing it may be necessary, but the resulting study should be described according to the evidence it can actually provide.
Watch Out
Using the same instrument does not guarantee that the same construct is being measured, and using a different instrument does not automatically mean that the underlying theoretical claim has changed. The relevant issue is whether the measurement evidence supports the intended interpretation and comparison.