Can You Trust a Paper When the Data or Code Are Unavailable?

If You Cannot Inspect the Data or Code, How Much Should You Trust the Paper?

You read a study with an important result and want to examine the analysis more closely. The paper provides no dataset. The analysis code is unavailable. Perhaps the data availability statement says that access is restricted, available only on request, or simply not available.

Does that mean you should distrust the findings?

Not automatically. Data and code can be unavailable for legitimate reasons, including participant privacy, legal restrictions, contractual conditions, intellectual-property constraints, or controlled-access requirements. In other cases, however, the absence of the materials needed to inspect or reproduce an analysis substantially limits what an independent researcher can verify.

The question is therefore not simply whether the files are public. It is what the lack of access prevents you from checking, why access is restricted, and what other evidence supports the credibility of the research.

Unavailable Data or Code Limits Verification, Not Automatically Validity

Availability and Reliability Are Related but Not Identical

Open Data and Code Make Some Claims More Verifiable

When appropriate research data and analytical code are available, other researchers may be able to inspect how variables were constructed, identify discrepancies, rerun analyses, explore whether reported results can be computationally reproduced, and reuse the materials for further research.

The current Transparency and Openness Promotion Guidelines maintained by the Center for Open Science treat data transparency and analytic code transparency as distinct research practices intended to increase the verifiability of empirical claims. Their framework distinguishes disclosure of whether materials are available from actual sharing and from stronger forms of independent certification.

This distinction is useful. Merely stating that data exist somewhere is not the same as making them accessible, and making files accessible is not the same as demonstrating that an analysis is correct.

These concepts overlap, but they should not be collapsed into one another. Public files can facilitate scrutiny without guaranteeing valid research. Conversely, restricted files can limit scrutiny without proving that the underlying research is invalid.

Ask Why the Data Are Unavailable

Not all non-availability has the same meaning.

Human-participant data may contain information that cannot responsibly be released publicly. Consent agreements, privacy protections, legal requirements, data-use agreements, Indigenous data governance, commercial restrictions, or third-party ownership can constrain sharing. In such circumstances, controlled access rather than unrestricted public release may be appropriate.

Recent research on clinical data-sharing statements illustrates this variation: studies may provide access through formal platforms, impose conditions, identify privacy or legal restrictions, or place a gatekeeper between applicants and the underlying data. The existence of conditions is therefore not equivalent to refusal to share.

Unavailable Code Can Matter Even When the Data Are Available

A dataset alone may not be enough to reconstruct an analysis. Researchers make many decisions between raw data and the final table: exclusions, transformations, recoding, derived variables, missing-data handling, model specification, software options, and visualization procedures can all affect results.

Analytic code can make those decisions more inspectable. The TOP framework therefore treats code transparency separately from data transparency.

If code is unavailable, ask whether the paper reports the analytical procedure in enough detail for a knowledgeable researcher to reconstruct it. For a simple analysis, that may be feasible from the methods alone. For a complicated computational pipeline involving many undocumented processing decisions, the absence of code can create a much larger verification gap.

Public Data Do Not Automatically Make a Paper Trustworthy

It is equally important not to reverse the mistake. A repository link is not a quality certificate.

A study can openly share its data and code yet still use a weak design, inappropriate measurement, questionable assumptions, unsuitable statistical models, or interpretations that extend beyond what the evidence supports. Code can faithfully reproduce a flawed analysis. Data can be perfectly accessible and still have serious measurement or sampling problems.

Open materials improve your opportunity to evaluate the research. They do not perform that evaluation for you.

Look at What Can Still Be Audited

If you cannot obtain the raw data or code, evaluate the transparency of the paper itself. Are sampling and recruitment adequately described? Are exclusions reported? Are variables and outcomes operationalized clearly? Are statistical models specified? Are effect estimates and uncertainty reported? Can you understand how the authors moved from observations to conclusions?

Also look for preregistration, registered protocols, analysis plans, supplementary materials, independent replication, robustness checks, or other records that may help establish what was planned and what was actually done.

None is a perfect substitute for access to the underlying materials. Together, however, they can change how much of the research process remains inspectable.

“Available on Request” Is Not the Same as Publicly Available

A statement that data are available from the authors upon reasonable request creates a possible access route, but it does not establish that the data are practically accessible to all qualified researchers.

Recent empirical work examining clinical research data-sharing statements found that conditional access and gatekeeper arrangements are common. That is another reason to distinguish a stated willingness to share from immediate repository access.

If access matters to your evaluation, request the materials. Document what you asked for and the response. Do not infer either cooperation or refusal before trying the stated process.

The Importance of Availability Depends on the Claim

For some research, computational verification is central. A result produced by a complex bespoke analysis may be difficult to scrutinize without code and data. For other work, the principal evidence may involve materials, observations, or procedures for which public release of a conventional dataset is less meaningful or inappropriate.

Disciplines also differ in established practices, ethical constraints, and infrastructure. TOP itself is modular and allows policies to vary in implementation, reflecting the fact that transparency standards must accommodate legitimate disciplinary differences.

Independent Evidence Becomes More Important When Direct Verification Is Limited

If an influential finding cannot be independently checked using the underlying materials, examine whether other research teams have produced independent evidence addressing the same claim.

Replication does not retroactively verify every detail of the inaccessible study. Still, convergence across genuinely independent studies can change how much your substantive conclusion depends on that single unverifiable analysis.

Conversely, if a paper makes an extraordinary or highly consequential claim and neither its underlying materials nor independent corroborating evidence are available, greater caution may be warranted.

Two Papers With Unavailable Data Can Deserve Different Assessments

The meaningful difference is not simply “open” versus “closed.” It is how much verification is possible, why access is limited, and whether the research provides credible alternative routes for scrutiny.

Common Mistakes When Data or Code Are Unavailable

Assess the Verification Gap Rather Than Applying a Binary Trust Test

When data or code are unavailable, identify what you would need those materials to verify and whether other evidence addresses the same concern.

When a Paper Does Not Provide Its Data or Code

Questions About Unavailable Research Data and Code

Unavailable Materials Create a Verification Limitation, Not an Automatic Verdict

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.