03 · What You Need to Know
Separate Unfamiliarity From Methodological Weakness
Disciplines can differ in how they produce and evaluate knowledge
Disciplines do more than study different subjects. They can also develop different questions, evidentiary traditions, methodological preferences, vocabularies, publication practices, and assumptions about what constitutes a persuasive explanation.
These differences become particularly visible in interdisciplinary research. The National Academies defines interdisciplinary research in terms of integrating information, data, techniques, tools, perspectives, concepts, or theories from multiple disciplines or bodies of specialized knowledge. Current work on team science similarly notes that cross-disciplinary teams must integrate fields that can differ in their approaches, methodologies, and concerns.
Consequently, the fact that a study does not follow the conventions familiar to you is not itself evidence of poor quality.
Start with the question the study is trying to answer
Before judging a method, ask what the researchers are trying to know.
A study estimating the prevalence of a condition, a study exploring how participants experience that condition, a historical analysis of how the condition became defined, and an experiment testing a causal mechanism may all address the same broad topic while requiring very different evidence.
The quality question therefore begins with fit:
Identify the claim
What does the study ultimately claim to know or explain?
Identify the methodological tradition
What design and form of evidence are being used to support that claim?
Identify the relevant standards
What criteria does that methodological tradition use to distinguish stronger from weaker work?
Evaluate execution
Did the study meet those criteria convincingly?
Evaluate inference
Do the conclusions remain within what the design and evidence can reasonably support?
This sequence helps prevent a common error: rejecting the methodological family before evaluating how well the particular study used it.
Do not confuse a preferred method with a universal quality criterion
Researchers are often trained intensively in a limited set of methods. Those methods can gradually begin to feel like research itself rather than one family of approaches to particular questions.
For example, random assignment is extremely important when the research question requires certain kinds of causal inference, but its absence is not a generic defect in every observational, qualitative, historical, descriptive, or theoretical study. Statistical power matters when statistical estimation or hypothesis testing requires it, but the same calculation is not a universal criterion for every form of inquiry.
Similarly, a qualitative study should not be criticized simply because it lacks statistical generalizability if statistical population inference was never its methodological objective. That does not exempt it from scrutiny. Its sampling logic, analytic procedure, reflexivity, evidential support, transparency, and claims may instead require close examination according to the relevant qualitative tradition.
Watch Out
"This is not how we would do it in my field" is an observation about disciplinary difference, not yet a methodological criticism. The criticism begins when you can explain why a feature is inappropriate for the study's question, design, or inference.
Find the standards used by the methodological community itself
When you encounter an unfamiliar design, do not invent appraisal criteria from first principles if established guidance already exists. Look for methodological handbooks, reporting guidelines, professional standards, authoritative methods literature, and critical-appraisal tools appropriate to the study type.
The EQUATOR Network, for example, maintains a searchable collection of health-research reporting guidelines covering different study designs. Guidelines such as CONSORT, STROBE, PRISMA, and COREQ address different kinds of research and therefore ask for different information.
Reporting guidelines are not identical to quality-assessment instruments, and compliance with a reporting checklist does not prove that a study is methodologically sound. They can nevertheless help you understand what information a methodological community considers important to report.
Separate reporting quality from methodological quality
A poorly reported study can make soundness difficult to evaluate, but inadequate reporting and inadequate methodology are not exactly the same problem.
For example, a paper may fail to explain its sampling process clearly. You may therefore be unable to determine whether the sampling was appropriate. That uncertainty should not silently become a factual assertion that the sampling was inappropriate.
Conversely, excellent reporting can make a genuine methodological weakness very easy to see. Transparency is valuable precisely because it enables appraisal; it does not guarantee that what is transparently reported was well designed.
Distinguish disciplinary conventions from methodological requirements
Some differences you encounter are conventions rather than requirements for valid inference. Fields can differ in typical article structure, citation density, preferred terminology, acceptable manuscript length, common sample sizes, or the amount of methodological detail placed in the main text.
Other differences directly affect what conclusions the evidence can support. Confounding in a causal analysis, inappropriate measurement, unsupported extrapolation, selective outcome reporting, inadequate engagement with contradictory evidence, or analytic procedures inconsistent with the stated design can create substantive problems regardless of whether they are customary somewhere.
Your task is to determine which kind of difference you are looking at.
Disciplinary convention
A customary way of communicating or conducting research that may vary without automatically changing the validity of the inference.
Methodological requirement
A feature necessary for the particular design or inference to work as claimed.
Quality criterion
A principle used to judge how convincingly the study was designed, executed, analyzed, interpreted, or reported within its methodological context.
Do not assume the unfamiliar discipline has one standard either
Disciplines are internally heterogeneous. Quantitative, qualitative, computational, historical, theoretical, experimental, interpretive, and mixed-methods traditions can coexist within the same broad field.
Evaluation literature on interdisciplinary and transdisciplinary research has consequently emphasized variability in goals, criteria, and indicators rather than assuming that one disciplinary standard can simply be transferred across contexts.
Ask which methodological community is relevant to the particular paper, not merely which department might employ its authors.
Some principles travel better than particular methods
Although methods differ, broader questions can often guide appraisal across traditions. Is the research question clear enough to understand what is being investigated? Is the evidence appropriate to that question? Are important methodological decisions transparent? Are interpretations supported by the evidence presented? Are limitations relevant to the inference acknowledged? Could another knowledgeable researcher understand how the conclusions were reached?
The answers will look different across methods. "Adequate evidence" for an ethnography is not established in the same way as adequate evidence for a randomized trial. The broader principle travels; its implementation does not necessarily do so unchanged.
Watch for conceptual misreadings before methodological criticism
Sometimes the apparent methodological weakness begins earlier. You may have misunderstood what the study's central concept means.
Before criticizing how a construct was measured or operationalized, verify how that concept functions within the unfamiliar disciplinary tradition. A measure that appears incomplete under your definition may be entirely coherent under a narrower definition used by the study.
Likewise, familiar terminology may conceal a different disciplinary meaning. Methodological appraisal built on a conceptual misunderstanding is unlikely to end well.
Methodological pluralism does not mean methodological relativism
Respecting disciplinary difference does not require accepting every study on its own terms without criticism. A research tradition can contain disputed practices, weak applications, and methodological debates. Researchers within that field may be among the strongest critics of those practices.
The more defensible position is neither "my discipline's standards apply everywhere" nor "every discipline gets to define quality however it likes." Instead, evaluate the study using criteria appropriate to the knowledge claim and method, while remaining attentive to broader principles of research integrity, transparency, and evidential support.
When uncertainty remains, disciplinary expertise becomes evidence about context
If a methodological difference materially affects your review and you cannot determine whether it reflects accepted practice, a contested practice, or a genuine flaw, consultation can be more reliable than inference from unfamiliar literature alone.
Cross-disciplinary team-science guidance emphasizes both technical expertise and the ability to understand and use teammates' expertise. In interdisciplinary work, no individual researcher necessarily understands every technical detail contributed by every discipline. Structures for coordinating and verifying specialized contributions can therefore be important.
This is where disciplinary review of your interpretation may become necessary, particularly when the disputed judgment influences inclusion decisions, synthesis, or conclusions.
