What Should the Literature Tell You Not to Repeat in Your Own Study?

What Has the Literature Already Taught You Not to Do?

Researchers often read previous studies looking for methods they can reuse. Which questionnaire did they administer? How large was the sample? What variables did they include? Which statistical test did they run?

That is useful, but it captures only half of what the literature can teach you.

Previous studies also show you where research repeatedly becomes difficult to interpret. Perhaps several studies use convenience samples while making claims about a much broader population. Perhaps everyone measures a complex construct with the same questionable proxy. Perhaps studies compare an intervention with an unrealistically weak alternative, collect outcomes too soon, ignore important contradictory evidence, or provide too little methodological detail for readers to understand what was actually done.

If you can see those problems before designing your study, repeating them is not inevitable. The literature should function partly as a record of mistakes, limitations, and unresolved methodological problems that your own research has an opportunity not to inherit.

Use Previous Research as a Warning System

Read the Literature for Problems, Not Just Precedents

Published Methods Are Evidence, Not Automatic Instructions

It is tempting to treat an established literature as a menu of accepted methodological choices. If five studies used the same scale, that scale begins to look like the obvious choice. If most studies recruited university students, another university sample can feel methodologically normal. If a familiar analysis appears repeatedly, reproducing it may seem safer than questioning it.

Frequency, however, does not establish methodological adequacy.

A practice may be common because it is defensible. It may also be common because researchers inherited it from earlier work, because it is inexpensive, because data are easy to obtain, or because a field has not yet resolved a methodological problem.

The useful question is therefore not simply “What did previous researchers do?” Ask: What did their choices allow them to conclude, and what remained difficult to conclude because of those choices?

Look for Problems That Recur Across Studies

A limitation reported once may be peculiar to one project. The same limitation appearing repeatedly deserves more attention.

Imagine that one study uses a narrow convenience sample. That constrains that study. If nearly every study in the literature draws from the same narrow population while making broader claims, you may be looking at a structural weakness in the evidence base.

The same reasoning applies to repeated reliance on self-report, short follow-up periods, poorly specified comparison conditions, inconsistent definitions, missing data, unvalidated measures, inadequate descriptions of interventions, or analyses that do not align well with the design.

NIH guidance on rigor and reproducibility provides an explicit example of this principle in biomedical research. Applicants are expected to assess the strengths and weaknesses of prior research serving as key support for a proposed project and describe how relevant weaknesses will be addressed. That requirement is specific to NIH contexts, but the methodological lesson is broader: previous research should be critically appraised before it becomes the foundation of the next study.

A Recurring Problem Can Appear at Several Levels

These problems differ substantially in severity. A minor reporting omission is not equivalent to a design feature that prevents the study from answering its research question. The literature review should help you distinguish them rather than producing a ceremonial catalogue of “limitations.”

Do Not Read Only the Limitations Section

Authors' stated limitations are useful, but they are not a complete methodological audit.

Researchers may overlook important weaknesses in their own studies, describe them gently, or emphasize limitations they consider most acceptable. Conversely, authors sometimes list generic limitations that have little bearing on the central inference.

Examine the methods and results yourself. Ask whether the sampling strategy matches the population claims, whether the measures correspond to the constructs, whether the comparison permits the intended inference, whether attrition or missing data could matter, and whether the analysis answers the question posed.

If you are working in a field with established critical-appraisal tools or risk-of-bias frameworks, those can help structure this evaluation. Use a tool appropriate to the study design rather than applying one checklist indiscriminately to every paper.

Separate a Genuine Weakness From an Unavoidable Trade-off

Not every limitation is evidence of poor research.

A laboratory experiment may sacrifice some ecological realism to achieve tighter control. A qualitative study may deliberately prioritize depth over population-level representativeness. A longitudinal design may provide stronger temporal information while increasing attrition risk. A short instrument may trade measurement breadth for feasibility.

The relevant question is whether the choice is appropriate for the study's purpose and whether the resulting limitation is handled honestly.

Your goal is not to design a study with no limitations. Such a creature remains mostly mythical, along with Reviewer 2 approving everything on the first round. The goal is to avoid weaknesses that previous research has already shown to be consequential and that your design has a reasonable opportunity to address.

Repeated Measurement Problems Deserve Particular Attention

If a literature repeatedly measures a construct in the same way, researchers can begin treating the measure and the construct as interchangeable.

Perhaps studies discuss “engagement” but measure only login frequency. Perhaps they discuss “learning” using self-reported perceptions of learning. Perhaps they discuss actual technology adoption while measuring behavioral intention.

These measures may still provide useful evidence. The problem occurs when the interpretation becomes broader than what the measure captures.

If measurement is a recurring weakness, do not automatically solve it by inventing a new questionnaire. First determine what outcome or construct you actually need to examine, then assess available measurement approaches and the evidence supporting their use.

Do Not Inherit an Irrelevant Comparison

Research traditions also inherit comparators.

A new intervention may repeatedly be tested against no intervention even after an effective alternative has become standard. Studies may compare naturally occurring groups without adequately addressing why those groups differ. Educational studies may contrast a highly structured innovation with poorly described “traditional teaching.”

If the comparator prevents previous research from answering the decision that now matters, repeating it reproduces the same limitation. Revisit what comparison the literature suggests your study actually needs.

Reporting Problems Can Become Scientific Problems

Poor reporting is sometimes dismissed as a writing issue that can be fixed after the research is finished. Often it cannot.

If researchers fail to record how participants were excluded, which outcomes were prespecified, how an intervention was implemented, how qualitative codes were developed, or how missing data were handled, the information may be impossible to reconstruct later.

The EQUATOR Network describes reporting guidelines as structured tools intended to ensure that research reports contain the information readers need to understand, replicate, use, or synthesize research. Different designs have different applicable guidelines, including CONSORT for randomized trials, STROBE for observational research, PRISMA for systematic reviews, and others.

Reporting guidelines are not substitutes for good design. A perfectly reported weak study remains weak. But consulting the appropriate guidance while planning can reveal information you will need to collect and preserve if the final study is to be transparent.

Do Not Correct a Weakness by Creating a Worse One

Suppose previous studies use small homogeneous samples. You respond by recruiting an extremely heterogeneous population but lack the sample size or design needed to examine meaningful differences within it. One problem has merely been exchanged for another.

Or perhaps previous studies rely on a brief measure, so you administer an exhaustive battery that creates severe participant burden and missing data. Again, the solution may undermine the study elsewhere.

Improvement should be systemic. Ask how the proposed correction affects feasibility, ethics, measurement quality, statistical precision, recruitment, participant burden, and interpretability.

Some Problems Should Change the Study; Others Should Change the Claim

You will not be able to eliminate every weakness identified in the literature.

When a limitation cannot reasonably be removed, you may be able to design around it, measure it, analyze its implications, or narrow the conclusion accordingly. For example, if access restricts you to one institution, you may not be able to solve the population limitation. You can, however, formulate the research question and eventual claims around the population you actually studied rather than implying universal applicability.

Knowing a limitation in advance should at least prevent you from being surprised by it in the discussion section.

Turning Repeated Limitations Into Better Design Decisions

You have not “fixed the literature.” Nor have you eliminated every possible weakness. You have used recurring problems to identify where the new study can avoid reproducing ambiguity that is already well documented.

Common Mistakes When Learning From Problems in Previous Research

Build a “Do Not Repeat” Record While You Review the Literature

As you read, keep a record separate from your summary of findings. Note recurring problems that affect how studies can be interpreted.

For each problem, record what it is, which studies exhibit it, why it matters, and whether your own project is vulnerable to the same problem. This turns criticism into design information.

This broad audit should produce a manageable set of concerns rather than an encyclopedic list. The next step is to decide which weaknesses deserve deliberate correction in your own design.

What Should Your Study Refuse to Inherit?

Questions About Avoiding Problems From Previous Studies

The Literature Is Also a Record of What Not to Reproduce

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.