Can Researchers Refuse Payment When Survey Responses Appear Careless or Unusable?

If Survey Responses Look Careless, Do You Still Have to Pay?

You open a completed survey and something looks wrong.

The participant selected the same response down an entire matrix. An open-ended answer is gibberish. Completion took much less time than expected. An attention check failed. Perhaps the responses are internally inconsistent. From the researcher's perspective, the data may be useless.

Can you reject the participant and refuse payment?

Sometimes, but "I cannot use these data" is not by itself a sufficient ethical rule. Researchers need to distinguish demonstrable failure to perform the compensated task from unusual but legitimate responding, measurement error, misunderstanding, technical problems, and post hoc dissatisfaction with the data.

Unusable Data Do Not Automatically Mean Unpaid Participation

Bad Data and Bad-Faith Participation Are Not the Same Thing

Start With the Question Payment Is Supposed to Answer

What did the researcher promise to pay for?

If payment compensates participants for spending time completing a survey in good faith, then the relevant question is whether they meaningfully performed that task. It is not whether every response ultimately survives the researcher's data-cleaning procedure.

A response can be unusable for many reasons that do not imply misconduct. The participant may misunderstand a question, hold an unusual opinion, make an accidental selection, encounter a technical problem, interpret a scale differently from the researcher, or simply generate data that fail a later analytic criterion.

Compensation should therefore be connected to what participant payment was actually intended to compensate for, not retroactively redefined according to whether each observation improves the final dataset.

Data Exclusion and Payment Rejection Are Two Different Decisions

This distinction is fundamental in survey research.

A researcher may have legitimate methodological reasons to exclude a response from analysis. That does not automatically establish that the participant should not be paid.

The first is principally a data-quality decision. The second affects the participant directly and therefore needs its own ethical and procedural justification.

"Careless" Needs an Operational Definition

Researchers often recognize suspicious data intuitively. Intuition is a poor payment policy.

Possible indicators of low-quality responding include nonsensical open-ended text, impossible response patterns, straight-lining, duplicate participation, failed attention checks, contradictory eligibility information, or completion so rapid that meaningful engagement appears implausible.

Each indicator has limitations. Straight-lining can occur because a participant genuinely gives the same answer to related items. Fast completion does not prove inattention. A failed attention check can result from misunderstanding, language difficulty, accessibility issues, fatigue, or a poorly designed check.

The stronger approach is to define quality criteria prospectively and validate them methodologically rather than making payment depend on a researcher's impression after opening the dataset.

One Failed Attention Check Is Not a Universal Ethical Standard

Attention checks can be useful, but their evidentiary value depends on design.

A transparent instructed-response item, a comprehension check, an inconsistency index, and an obscure "gotcha" question are not methodologically equivalent. Nor is there a universal ethics rule stating that one failed check automatically cancels payment.

If payment consequences are attached to attention checks, researchers should be able to justify why those checks validly indicate failure to perform the compensated task, how many failures matter, whether participants can reasonably understand the requirement, and how accidental errors are handled.

Platform Rules Can Create Additional Payment Standards

Online participant platforms often have their own rules governing approval and rejection. Those rules are not universal research-ethics standards, but researchers using the platform must follow them.

For example, CloudResearch Connect instructs researchers to use rejection only for confirmed data-quality concerns. Its examples include nonsensical or irrelevant open-ended responses, straight-lining, failed attention checks, and clear indications that the participant did not attempt to complete the study as instructed.

The same guidance says researchers should not reject participants for researcher-side problems such as incorrect survey logic, technical errors, incorrect survey links, or removal for reasons unrelated to participant performance. Participants who made a legitimate effort should be approved and paid when the problem originated with the researcher.

That is a platform-specific rule, not a general rule that automatically applies to every survey. It nevertheless illustrates an ethically useful distinction between participant performance and researcher-side failure.

Researcher Error Should Not Become Participant Nonpayment

Suppose a survey contains a broken branch that sends participants to contradictory questions. Or the survey crashes after thirty minutes. Or an attention check is coded incorrectly.

The participant may produce incomplete or apparently inconsistent data, but the researcher created the problem.

Refusing payment in such circumstances shifts the cost of research design or technical failure onto the participant. Platform policies may expressly prohibit this, as CloudResearch Connect does for several researcher-side errors.

Even outside a platform, the broader fairness issue remains.

Unusual Responses Are Not Necessarily Careless Responses

Researchers should be particularly cautious when a response looks implausible because it conflicts with expectations.

A participant may genuinely select the lowest response on every item. Someone may report an unusual combination of demographic characteristics. A respondent may strongly disagree with every statement in a scale. An open-ended answer may be terse rather than elaborate.

Data cleaning should not become a process of paying only participants whose responses look psychologically plausible to the researcher.

Payment criteria should therefore focus on evidence of task performance rather than whether responses conform to anticipated distributions, theoretical expectations, or desirable findings.

Unusable for Analysis Does Not Necessarily Mean the Participant Failed

A response may be excluded because of a preregistered statistical criterion, missingness threshold, failed manipulation check, duplicate IP address, implausible completion time, or another methodological rule.

Some of those criteria may also provide evidence that the participant did not meaningfully complete the task. Others do not.

For example, failing a manipulation check can mean the experimental manipulation did not work for that participant. It does not necessarily mean the participant failed to follow instructions. Likewise, a response can become unusable because the researcher's measurement instrument performs poorly.

Researchers should therefore resist treating every analytic exclusion criterion as an automatic payment-exclusion criterion.

Prospective Rules Matter More Than Post Hoc Frustration

OHRP recommends that participants receive a detailed account of payment terms, including circumstances in which partial or no payment may occur.

This principle is especially valuable in online surveys because researchers can otherwise create payment criteria after seeing responses.

If researchers intend to condition payment on completing required questions, passing legitimate eligibility verification, avoiding duplicate participation, or satisfying particular quality criteria, those conditions should be considered during study design and ethics review where applicable.

Deliberate Deception Is Different From Ordinary Response Error

SACHRP recognizes that incentive payments can sometimes motivate individuals to make false statements about eligibility or other aspects of participation. Such deception can create safety and research-integrity concerns. It recommends reasonable measures to reduce the opportunity for deception, including objective verification where appropriate.

A participant deliberately fabricating eligibility information to obtain payment therefore presents a different case from someone who misunderstands one survey item.

Even then, payment consequences should follow the prospectively defined protocol, institutional requirements, and platform rules rather than becoming an improvised punishment.

Quality Control Should Be Designed Before Data Collection

The best time to decide what counts as inadequate survey participation is before the first response arrives.

That final step is important. Data quality criteria should not mysteriously become stricter after an inconvenient result appears. Reviewer 2 may already do enough of that for everyone.

Three Suspicious Survey Responses, Three Different Decisions

The same label, "unusable response," can therefore conceal three ethically different situations.

Common Mistakes When Refusing Payment for Survey Quality

Separate Data Cleaning From Payment Decisions

This approach protects both sides of the research transaction. Researchers retain legitimate tools for dealing with demonstrable nonperformance, while participants are not financially penalized merely because their data fail to look the way the researcher hoped.

Before Refusing Payment for a Survey Response

Frequently Asked Questions About Survey Quality and Payment

Judge Whether the Task Was Performed, Not Whether You Like the Data

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.