Awesome Reproducible Research 
A curated list of reproducible research case studies, projects, tutorials, and media- Case studies - Ad-hoc reproductions - Theory papers - Theses and dissertations - Tool reviews - Courses - Development Resources - Literature tools - Scientific Data Management Systems - Books - Databases - Data Repositories - Exemplar Portals - Runnable Papers - Haibe-Kains lab - Pachter lab - Siepel lab - Journals - Ontologies - Minimal Standards - Organizations - Awesome Lists - Contribute - License
Case studies
The term "case studies" is used here in a general sense to describe any study of reproducibility. A _reproduction_ is an attempt to arrive at comparable results with identical data using computational methods described in a paper. A _refactor_ involves refactoring existing code into frameworks and other reproducibility best practices while preserving the original data. A _replication_ involves generating new data and applying existing methods to achieve comparable results. A _robustness test_ applies various protocols, workflows, statistical models or parameters to a given data set to study their effect on results, either as a follow-up to an existing study or as a "bake-off". A _census_ is a high-level tabulation conducted by a third party. A _survey_ is a questionnaire sent to practitioners. A _case narrative_ is an in-depth first-person account. An _independent discussion_ utilizes a secondary independent author to interpret the results of a study as a means to improve inferential reproducibility.|
Study |
Field |
Approach |
Size |
|
Medicine |
Census |
80 studies |
|
|
Cancer biology |
Refactor |
8 studies |
|
|
Biostatistics |
Census |
56 studies |
|
|
Genetics |
Reproduction |
18 studies |
|
|
Software engineering |
Replication |
4 companies |
|
|
Signal processing |
Census |
134 papers |
|
|
Biomedical sciences |
Survey |
23 PIs |
|
|
Bioinformatics |
Census |
100 studies |
|
|
Cancer biology |
Replication |
53 studies |
|
|
Computer science |
Census |
613 papers |
|
|
Psychology |
Replication |
100 studies |
|
|
Biomedical sciences |
Census |
100 papers |
|
|
Epidemiology |
Robustness test |
417 variables |
|
|
Economics |
Reproduction |
67 papers |
|
|
Biomedical sciences |
Census |
441 papers |
|
|
Science |
Survey |
1,576 researchers |
|
|
NLP |
Replication |
3 studies |
|
|
Cancer biology |
Replication |
9 studies |
|
|
Biomedical sciences |
Census |
318 journals |
|
|
Science |
Case narrative |
31 PIs |
|
|
Biological sciences
|
Survey |
704 PIs |
|
|
Bioinformatics |
Refactor |
1 study |
|
|
Economics |
Replication |
18 studies |
|
|
Machine learning |
Census |
30 studies |
|
|
Archaeology |
Case narrative |
1 survey |
|
|
Comparative toxicogenomics |
Census |
51,292 claims in 3,363 papers |
|
|
Artificial intelligence |
Census |
400 papers |
|
|
Economics |
Census |
203 papers |
|
|
Computational science |
Reproduction |
204 papers, 180 authors |
|
|
Psychology |
Survey |
2,056 psychologists |
|
|
Genomics |
Case narrative |
1 study |
|
|
Social sciences |
Replication |
21 papers |
|
|
Psychology |
Robustness test |
One data set, 29 analyst teams |
|
|
Medicine and health sciences |
Census |
30 papers |
|
|
Microbiome immuno oncology |
Replication |
1 paper |
|
|
Bioinformatics |
Refactor and test of robustness |
1 paper |
|
|
Biomedical Sciences |
Census |
149 papers |
|
|
Bioinformatics |
Synthetic replication & refactor |
1 paper |
|
|
Geosciences |
Survey, Reproduction |
146 scientists, 41 papers |
|
|
Reinforcement Learning |
Reproduction, case narrative |
1 paper |
|
|
Computational physics |
Census |
306 papers |
|
|
Science & Engineering |
Survey |
215 participants |
|
|
Nephrology |
Robustness test |
1 paper |
|
|
Social sciences & other |
Census |
810 Dataverse studies |
|
|
Social sciences & other |
Census, Survey |
2109 replication datasets |
|
|
GIScience/Geoinformatics |
Census, Survey |
32 papers, 22 participants |
|
|
Genomics |
Robustness test |
8 studies |
|
|
Geosciences |
Survey |
360 papers |
|
|
Deep learning |
Robustness test |
1 analysis |
|
|
Genomics |
Case narrative |
1 analysis |
|
|
Pharmacogenomics |
Case narrative |
2 analyses |
|
|
Biomedical sciences and Psychology |
Census |
127 registered reports |
|
|
All |
Census |
1,159,166 Jupyter notebooks |
|
|
Virology |
Census |
236 papers |
|
|
Anaesthesia |
Independent discussion |
1 study |
|
|
Psychology |
Replication |
1 paper |
|
|
Cell pharmacology |
Robustness test |
5 labs |
|
|
Machine learning |
Reproduction |
18 conference papers |
|
|
Experimental archaeology |
Replication |
1 theory |
|
|
Neurology |
Census |
202 papers |
|
|
Psychology |
Replication |
2 experiments |
|
|
Ecology and Evolution |
Census |
163 papers |
|
|
Neuroimaging |
Robustness test |
1 data set, 70 teams |
|
|
Psychology |
Replication |
1 experiment, 21 labs, 2,220 participants |
|
|
Psychology |
Census |
62 papers |
|
|
Oncology |
Census |
154 meta-analyses |
|
|
Bioinformatics |
Robustness test |
1 data set |
|
|
Neurobiology |
Census |
41 papers |
|
|
Genetics |
Census |
1799 papers |
|
|
Psychology |
Reproduction |
33 meta-analyses |
|
|
Biomedical science |
Census |
792 papers |
|
|
Ecology |
Census |
346 papers |
|
|
Physics |
Replication |
2 papers |
|
|
Reproductive endocrinology |
Census |
222 papers |
|
|
Biomedical sciences |
Census |
240 papers |
|
|
Environmental Modelling |
Census |
7500 papers |
|
|
Cardiology |
Census |
532 papers |
|
|
GIS |
Census |
75 papers |
|
|
Life Sciences |
Survey |
251 researchers |
|
|
Genetics |
Robustness test |
1 paper |
|
|
Life Sciences |
Census |
3377 articles |
|
|
Computational Biology |
Census |
622 papers |
|
|
Computational Biology |
Robustness test |
6 studies |
|
|
Computational Biology |
Survey |
214 researchers |
|
|
Differential expression |
Census |
2109 GEO submissions |
|
|
Computational biology |
Census |
186 papers |
|
|
Psychology |
Robustness test |
1 study |
|
|
Proteomics |
Refactor |
1 analysis |
|
|
Social science |
Robustness test |
73 teams |
|
|
Radiology |
Census |
62 studies |
|
|
Clinical ML |
Census |
511 ... (README truncated for length) Related StoriesI spent three weekends trying to secure an AI agent. Here's everything that didn't work — and the ~600 lines that did.
Dev.to
5 hours ago
Deep Dive: Autonomous Code Auditing, 40% Python Speedups & Stateful 1M-Token Agents with Gemini 4 Argon
Dev.to
8 hours ago
Someone hacked the freezers at 400 French supermarkets
Dev.to
10 hours ago
I built a 200-line tool that catches an AI agent exfiltrating data — while your test says PASS
Dev.to
14 hours ago
Your AI agent said it blocked the attack. I read the source of 5 tools that check if it's lying.
Dev.to
15 hours ago
Related Stories |