Data Management, Reproducibility, and Open Science

The Reproducibility Crisis: Scope and Causes

In 2011, researchers at Bayer HealthCare attempted to replicate the findings of sixty-seven published preclinical cancer studies as part of their internal drug discovery process. They were able to fully reproduce the results of fewer than one in four NASEM 2019. A similar effort by Amgen scientists produced comparable findings: researchers reported successful replication of only six of fifty-three landmark cancer biology studies they attempted to reproduce. These reports, along with a series of large-scale replication projects in psychology and other fields, crystallized what has come to be called the reproducibility crisis — a recognition that a substantial proportion of published scientific findings, when subjected to independent replication attempts, do not hold up.

The causes of the reproducibility crisis are multiple and interacting. Study design problems — underpowered samples, inadequate controls, inappropriate statistical tests — contribute. Publication bias — the tendency of journals to publish positive findings while leaving null results in the file drawer — inflates the apparent prevalence of effects that may not be real. Analytical flexibility — the range of defensible analytical choices that, when made post hoc and selectively reported, can produce statistical significance from noise — further inflates the false-positive rate. And, cutting across all of these specific problems, inadequate documentation and data-sharing practices mean that even results that are real and important often cannot be verified, extended, or corrected by other researchers, because the data and analytical code needed to do so were never made available.

Poor data management is both a cause of the reproducibility crisis and a barrier to its resolution. When researchers do not document their data collection procedures systematically, when file naming conventions are inconsistent or absent, when raw data are overwritten by processed versions, when analytical code is not retained alongside the data it was used to analyze — when, in short, the basic infrastructure of a research project is not organized with transparency in mind — the result is data that may be internally comprehensible to the researcher who collected it but opaque to anyone else, including future versions of that same researcher. The solution to this problem is not primarily technological; it is cultural and practical. It requires researchers to internalize habits of systematic documentation and organization as a routine part of their scientific practice Nosek et al. 2015.

FAIR Data Principles

The FAIR data principles, published in 2016 Wilkinson et al. 2016, articulate a framework for making research data as useful as possible beyond their original collection context. FAIR is an acronym standing for Findable, Accessible, Interoperable, and Reusable, and each of the four principles is supported by a set of specific, practical guidelines.

Findable means that data and their associated metadata are assigned a globally unique and persistent identifier — typically a digital object identifier (DOI) — and that the metadata are indexed in a searchable resource. A dataset that exists on a researcher’s hard drive is not findable; one that has been deposited in a public repository with a DOI and descriptive metadata can be discovered by other researchers through database searches. The findability principle also requires that metadata clearly identify the dataset they describe, using a standardized reference.

Accessible means that data and metadata can be retrieved using a standardized, open communications protocol. Accessibility does not necessarily mean that all data must be freely available without restriction; it means that the conditions of access — including any authentication requirements — are clearly specified and that the access mechanism is reliable and long-term. Metadata, importantly, should remain accessible even when the underlying data cannot be shared (as may be required for participant privacy reasons), because discoverable metadata helps other researchers know what data exist even when they cannot obtain a copy.

Interoperable means that data use a formal, accessible, shared, and broadly applicable language for knowledge representation — in practice, that data formats, variable names, and data structures follow community standards that allow data to be integrated with other datasets and processed by commonly available software tools. A dataset stored in a proprietary format that can only be opened by a specific version of a specific software application is not interoperable. A dataset stored in a plain-text, comma-separated values format with variable names that follow domain-specific conventions is substantially more interoperable, particularly if accompanied by a codebook that explains each variable in human-readable terms.

Reusable means that data are described with a plurality of accurate and relevant attributes that allow them to be used by others for purposes beyond the original collection context. This includes clear and accessible data usage licenses, detailed provenance information (describing how the data were collected and what quality-control procedures were applied), and community domain standards for metadata. Without this contextual information, a dataset may be technically accessible but practically unusable by someone unfamiliar with the original study’s methods.

Data Management Plans

A data management plan (DMP) is a structured document that describes how data generated in a research project will be collected, organized, stored, preserved, and shared. DMPs are now required by most major research funding agencies, including the National Institutes of Health and the National Science Foundation in the United States, as a condition of funding NIH 2023. But a DMP is valuable not only as a grant compliance requirement; it is a practical planning tool that forces researchers to think systematically about data management before data collection begins, when it is still possible to build good practices into the project’s infrastructure rather than trying to impose them retroactively.

A typical DMP addresses several key questions: What types of data will be generated, and in what formats? How will data be organized, named, and documented during the project? Where will data be stored during the active phase of the research, and what backup procedures will be in place? Who will have access to the data during and after the project? How long will data be retained after the project ends? In what repository will data be deposited for long-term access? What restrictions on sharing, if any, are required for legal, ethical, or proprietary reasons?

The answers to these questions will differ substantially across research contexts. A clinical study collecting protected health information about identifiable participants faces different data-sharing considerations than a genomics study generating sequences from de-identified samples, which faces different considerations than a computational project generating derived data products from publicly available sources. DMP templates specific to different funding agencies and research types are available from many institutions and through the DMPTool, an online platform that provides guidance and templates for a wide range of funding requirements.

Electronic Lab Notebooks and Documentation

The laboratory notebook is the foundational documentation instrument of experimental science. Traditionally kept in paper format, laboratory notebooks record experimental procedures, observations, results, and interpretations in sufficient detail to allow independent reproduction of the work. They serve as contemporaneous records that can be dated and verified, and they are the evidentiary foundation for priority claims in the event of disputes about who made a discovery first.

Electronic lab notebooks (ELNs) have become increasingly common as alternatives or complements to paper notebooks, offering advantages including legibility, searchability, automatic timestamping, multimedia integration (the ability to embed images, spreadsheets, and instrument data files directly into notebook entries), and cloud backup. Several dedicated ELN platforms are widely used in the life sciences: Benchling is popular in molecular biology and biotech settings for its integration with sequence analysis tools; RSpace offers institutional deployment options with strong access control and compliance features; LabArchives is widely used in academic settings. General-purpose tools including Microsoft OneNote and Notion are also used, though they lack some of the laboratory-specific features of dedicated ELN platforms Perkel 2020.

Whether electronic or paper, the guiding principle of laboratory notebook practice is the same: record enough information that someone else — or you, returning to the work after a year away — could understand exactly what was done and reproduce it. This means recording not only the procedures and results but also deviations from planned procedures, unexpected observations, failed experiments, instrument settings, lot numbers for reagents, and the reasoning behind methodological choices. The common practice of recording only successful experiments and omitting failures is both scientifically misleading and practically costly: failed experiments contain information that guides subsequent decisions, and their omission from the record means that the same mistakes are likely to be made again.

File Naming, Version Control, and Organizational Practices

Systematic file naming conventions and directory organization are unglamorous but important aspects of data management that pay large practical dividends over the life of a research project. A file named “data_final_FINAL_v3_USE_THIS_ONE.xlsx” communicates confusion about which version is authoritative and offers no information about the file’s content, date, or place in the project’s history. A file named “2024-09-15_mouse-behavior_OF-test_group-A_raw.csv” communicates clearly what the file contains, when it was created, and what stage of processing it represents.

Effective file naming conventions share several features: they use dates in ISO 8601 format (YYYY-MM-DD, which sorts chronologically), they are descriptive of content, they use underscores or hyphens rather than spaces (which cause problems in some computing environments), and they are applied consistently across the project. Directory structures should reflect the logical organization of the project — separating raw data from processed data, code from data, and figures from manuscripts — and should include a README file at each directory level explaining the contents.

Version control, most commonly implemented using the Git software and the GitHub hosting platform, provides a more powerful solution to the problem of tracking changes over time, particularly for code and text files. Git records the complete history of changes to a file or set of files, allowing a researcher to see exactly what changed between any two versions, to attribute changes to specific authors and dates, and to revert to any previous state of the project. For scientific computing, version control is rapidly becoming a professional expectation rather than an optional extra, and researchers who develop even basic proficiency with Git and GitHub are substantially better positioned than those who do not McKiernan et al. 2016.

Data Repositories and Sharing

The expectation that research data be shared has strengthened substantially in recent years, driven by a combination of funding mandates, journal requirements, and cultural change within the scientific community Tenopir et al. 2011. Studies that share their underlying data are cited more frequently than those that do not Piwowar et al. 2007, and data sharing enables the secondary analyses, reanalyses, and evidence syntheses that multiply the scientific value of a study beyond its original purpose.

Choosing the right repository for a given type of data requires familiarity with the landscape of discipline-specific and generalist repositories. In the life sciences, the NCBI family of databases — including GenBank for DNA sequences, the Sequence Read Archive (SRA) for raw next-generation sequencing data, and the Gene Expression Omnibus (GEO) for gene expression data — provide established, well-indexed homes for the most common data types in molecular biology and genomics. The Protein Data Bank (PDB) is the repository for macromolecular structure data. These discipline-specific repositories have the advantage of extensive domain-specific metadata standards and integration with the databases and tools researchers use to search and analyze data.

For data types without a discipline-specific repository, generalist repositories including Dryad, Zenodo, and Figshare provide open-access data hosting with DOI assignment and flexible metadata. Dryad focuses on data underlying published research articles and integrates with many journals to facilitate direct data submission. Zenodo, hosted by CERN, accepts a wide range of research outputs including datasets, software, and preprints. Figshare is particularly widely used for sharing supplementary figures, datasets, and other research outputs that accompany publications.

Code Sharing and Computational Reproducibility

As computational methods have become central to research across the life sciences — from genomic data analysis to image processing to mathematical modeling — the sharing of analytical code has become as important as the sharing of underlying data. A paper that reports results obtained through custom computational analysis but does not share the code used to produce those results cannot be reproduced or verified by other researchers, even if the underlying data are fully available Nosek et al. 2015.

Code should be shared in a form that is actually executable, not merely described. This means sharing the code itself (not a pseudocode description), along with the software environment needed to run it (specifying package versions), test data that allow users to verify the code produces the expected outputs, and documentation explaining how to run the analysis and what the outputs represent. Tools including Docker (for containerizing complete computational environments) and conda environments (for managing Python package dependencies) help ensure that shared code can be run on other systems without encountering the dependency problems that commonly prevent reproduction of computational analyses.

The Open Science Framework as an Integrated Hub

The Open Science Framework (OSF), developed and maintained by the Center for Open Science, provides a free, integrated platform for managing and sharing the full range of materials associated with a research project — preregistrations, protocols, data, code, preprints, and publications — in a single, versioned, citable repository. OSF integrates with external services including GitHub (for code), Google Drive (for documents), Dropbox (for files), and preprint servers including bioRxiv and PsyArXiv, allowing researchers to work within familiar tools while maintaining a centralized project record.

Using OSF as a project management tool from the beginning of a project — creating the project structure, linking external tools, registering study plans, and gradually uploading materials as they are generated — is a practical approach to open science that requires no large upfront investment but progressively builds a transparent, shareable record of the research. The habits of documentation, data organization, and code sharing described in this chapter are not additional burdens imposed on top of doing research; they are components of doing research well. Researchers who internalize them produce work that is more credible, more impactful, and more valuable to the scientific community — and that is more likely to withstand the scrutiny of replication.