Reproducible research in computational science
Open accessAbstract
Computational science has led to exciting new developments, but the nature of the work has exposed limitations in our ability to evaluate published findings. Reproducibility has the potential to serve as a minimum standard for judging scientific claims when full independent replication of a study is not possible.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The difficulty in auditing spreadsheets makes them generally incompatible with the principles of computational reproducibility.
“The difficulty in auditing spreadsheets makes them generally incompatible with the principles of computational reproducibility [5].”
From Gene name errors: Lessons not learned (2021) · cites this paper as [5]
The paper's discussion of reproducibility (indexed as Abstract; no page given) says that analyses need available code and data, and that interactive software often fails to retain a record of users’ actions. This supports the concern about auditability, but the paper does not address spreadsheets specifically or say they are generally incompatible with computational reproducibility. (LLM, conf=0.84)
Experimental — accuracy not yet measuredReproducible analyses require preserving data, code, and runtime environment; all three are necessary and none is sufficient alone.
“The rough consensus reduces to three ingredients.”
From Designing reproducible large-language-model-assisted scientific analyses (2026) · cites this paper as [2]
The passage labeled Abstract defines the reproducibility standard in terms of making data and analysis code available. It does not identify preservation of the runtime environment as a third ingredient or say that all three are necessary and none is sufficient alone. (LLM, conf=0.87)
Experimental — accuracy not yet measuredThe authors develop a two-step attribution process which combines a transparent and reproducible machine learning approach to identifying studies on observed climate impacts with model-based assessments of detectable anthropogenic contributions to historical temperature and precipitation trends.
“We develop a two-step attribution process which combines a transparent and reproducible 39,40 machine learning approach to identifying studies on observed climate impacts with model-based assessments of detectable anthropogenic contributions to historical temperature and precipitation trends.”
From Machine-learning-based evidence and attribution mapping of 100,000 climate impact studies (2021) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551 · see citing works
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 455 · see citing works
Publish your computer code: it is good enough
Barnes, N. (2010) · Nature · doi:10.1038/467753a · Cited by 317 · see citing works
Accessible Reproducible Research
Mesirov, J. P. (2010) · Science · doi:10.1126/science.1179653 · Cited by 311 · see citing works
Reproducible Epidemiologic Research
Peng, R. D., Dominici, F., & Zeger, S. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj093 · Cited by 292 · see citing works
Self-citationReproducible research and Biostatistics
Peng, R. D. (2009) · Biostatistics · doi:10.1093/biostatistics/kxp014 · Cited by 244 · see citing works
Self-citationMaking scientific computations reproducible
Schwab, M., Karrenbach, N., & Claerbout, J. F. (2000) · Computing in Science & Engineering · doi:10.1109/5992.881708 · Cited by 232 · see citing works
Reproducible Research: Moving toward Research the Public Can Really Trust
Laine, C., Goodman, S. N., Griswold, M. E., & Sox, H. C. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-146-6-200703200-00154 · Cited by 213 · see citing works
Making Data Maximally Available
Hanson, B., Sugden, A. M., & Alberts, B. (2011) · Science · doi:10.1126/science.1203354 · Cited by 172 · see citing works
King, G. (1995) · PS Political Science & Politics · doi:10.2307/420301 · Cited by 145 · see citing works
Do economics journal archives promote replicable research?
McCullough, B. D., McGeary, K. A., & Harrison, T. D. (2008) · Canadian Journal of Economics/Revue canadienne d économique · doi:10.1111/j.1540-5982.2008.00509.x · Cited by 140 · see citing works
Geographic Information Systems
Chen, J. X. (2010) · Computing in Science & Engineering · doi:10.1109/mcse.2010.13 · Cited by 27 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [3]Madrid JP Macchetto D Bull AAS arXiv:0901.4552 2009 · arXiv
Cited by
Sorted by citations
phyloseq: An R Package for Reproducible Interactive Analysis and Graphics of Microbiome Census Data
McMurdie, P. J., & Holmes, S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0061217 · Cited by 21,013 · see citing works
Radiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688 · see citing works
Waste Not, Want Not: Why Rarefying Microbiome Data Is Inadmissible
McMurdie, P. J., & Holmes, S. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003531 · Cited by 2,891 · see citing works
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · see citing works
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · see citing works
An introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059 · see citing works
A standard protocol for reporting species distribution models
Zurell, D., Franklin, J., König, C., Bouchet, P. J., Dormann, C. F., Elith, J., Fandós, G., Feng, X., Guillera‐Arroita, G., Guisan, A., et al. (2020) · Ecography · doi:10.1111/ecog.04960 · Cited by 955 · see citing works
The conundrum of sharing research data
Borgman, C. L. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22634 · Cited by 910 · see citing works
Ten Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880 · see citing works
Model Transparency and Validation
Eddy, D. M., Hollingworth, W., Jaime, J., Tsevat, J., McDonald, K. M., & Wong, J. B. (2012) · Medical Decision Making · doi:10.1177/0272989x12454579 · Cited by 803 · see citing works
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760 · see citing works
A perspective on using partial least squares structural equation modelling in data articles
Ringle, C. M., Sarstedt, M., Sinkovics, N., & Sinkovics, R. R. (2023) · Data in Brief · doi:10.1016/j.dib.2023.109074 · Cited by 975 · see citing works
Best practices in data analysis and sharing in neuroimaging using MRI
Nichols, T. E., Das, S., Eickhoff, S. B., Evans, A. C., Glatard, T., Hanke, M., Kriegeskorte, N., Milham, M. P., Poldrack, R. A., Poline, J.-B., et al. (2017) · Nature Neuroscience · doi:10.1038/nn.4500 · Cited by 714 · see citing works
Building a Science of Individual Differences from fMRI
Dubois, J., & Adolphs, R. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.03.014 · Cited by 610 · see citing works
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · see citing works
NMRbox: A Resource for Biomolecular NMR Computation
Maciejewski, M. W., Schuyler, A. D., Gryk, M. R., Moraru, I. I., Romero, P. R., Ulrich, E. L., Eghbalnia, H. R., Livny, M., Delaglio, F., & Hoch, J. C. (2017) · Biophysical Journal · doi:10.1016/j.bpj.2017.03.011 · Cited by 520 · see citing works
pyGIMLi: An open-source library for modelling and inversion in geophysics
Rücker, C., Günther, T., & Wagner, F. M. (2017) · Computers & Geosciences · doi:10.1016/j.cageo.2017.07.011 · Cited by 520 · see citing works
Yaniv, Z. R., Lowekamp, B., Johnson, H. J., & Beare, R. J. (2017) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-017-0037-8 · Cited by 504 · see citing works
Moradi Kashkooli, F., Soltani, M., & Souri, M. (2020) · Journal of Controlled Release · doi:10.1016/j.jconrel.2020.08.012 · Cited by 514 · see citing works
Eddy, D. M., Hollingworth, W., Caro, J. J., Tsevat, J., McDonald, K. M., & Wong, J. B. (2012) · Value in Health · doi:10.1016/j.jval.2012.04.012 · Cited by 466 · see citing works
rfishbase: exploring, manipulating and visualizing FishBase data from R
Boettiger, C., Lang, D. T., & Wainwright, P. C. (2012) · Journal of Fish Biology · doi:10.1111/j.1095-8649.2012.03464.x · Cited by 444 · see citing works
O’Leary, S. J., Puritz, J. B., Willis, S. C., Hollenbeck, C. M., & Portnoy, D. S. (2018) · Molecular Ecology · doi:10.1111/mec.14792 · Cited by 436 · see citing works
Reproducibility vs. Replicability: A Brief History of a Confused Terminology
Pleßer, H. E. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00076 · Cited by 426 · see citing works
Osaba, E., Villar-Rodríguez, E., Ser, J. D., Nebro, A. J., Molina, D., LaTorre, A., Suganthan, P. N., Coello, C. A. C., & Herrera, F. (2021) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2021.100888 · Cited by 424 · see citing works
Convolutional Neural Networks for the Automatic Identification of Plant Diseases
Boulent, J., Foucher, S., Théau, J., & St-Charles, P.-L. (2019) · Frontiers in Plant Science · doi:10.3389/fpls.2019.00941 · Cited by 402 · see citing works
Prabhakar, P. K., Vatsa, S., Srivastav, P. P., & Pathak, S. S. (2020) · Food Research International · doi:10.1016/j.foodres.2020.109157 · Cited by 373 · see citing works
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369 · see citing works
Evolution of Translational Omics: Lessons Learned and the Path Forward
Micheel, C., Nass, S. J., & Omenn, G. S. (2013) · Cited by 365 · see citing works
Tximeta: Reference sequence checksums for provenance identification in RNA-seq
Love, M. I., Soneson, C., Hickey, P. F., Johnson, L. K., Pierce‐Ward, N. T., Shepherd, L. A., Morgan, M. T., & Patro, R. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007664 · Cited by 360 · see citing works
Exploration and Explanation in Computational Notebooks
Rule, A., Tabard, A., & Hollan, J. D. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173606 · Cited by 347 · see citing works
Huber, S. P., Zoupanos, S., Uhrin, M., Talirz, L., Kahle, L., Häuselmann, R., Gresch, D., Müller, T., Yakutovich, A. V., Andersen, C. W., et al. (2020) · Scientific Data · doi:10.1038/s41597-020-00638-4 · Cited by 347 · see citing works
A review in radiomics: Making personalized medicine a reality via routine imaging
Guiot, J., Vaidyanathan, A., Deprez, L., Zerka, F., Danthine, D., Frix, A.-N., Lambin, P., Bottari, F., Tsoutzidis, N., Miraglio, B., et al. (2021) · Medicinal Research Reviews · doi:10.1002/med.21846 · Cited by 344 · see citing works
Inflammatory Bowel Disease as a Model for Translating the Microbiome
Huttenhower, C., Kostic, A. D., & Xavier, R. J. (2014) · Immunity · doi:10.1016/j.immuni.2014.05.013 · Cited by 332 · see citing works
Reproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310 · see citing works
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293 · see citing works
Prospects and challenges of multi-omics data integration in toxicology
Canzler, S., Schor, J., Busch, W., Schubert, K. A., Rolle-Kampczyk, U. E., Seitz, H., Kamp, H., von Bergen�, M., Buesen, R., & Hackermüller, J. (2020) · Archives of Toxicology · doi:10.1007/s00204-020-02656-y · Cited by 291 · see citing works
Gilbert, K. J., Andrew, R. L., Bock, D. G., Franklin, M. T., Kane, N. C., Moore, J., Moyers, B. T., Renaut, S., Rennison, D. J., Veen, T., et al. (2012) · Molecular Ecology · doi:10.1111/j.1365-294x.2012.05754.x · Cited by 290 · see citing works
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298 · see citing works
Utilizing big data analytics for information systems research: challenges, promises and guidelines
Müller, O., Junglas, I. A., Brocke, J. V., & Debortoli, S. (2016) · European Journal of Information Systems · doi:10.1057/ejis.2016.2 · Cited by 282 · see citing works
OpenSim Moco: Musculoskeletal optimal control
Dembia, C. L., Bianco, N. A., Falisse, A. M. D., Hicks, J. L., & Delp, S. L. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008493 · Cited by 280 · see citing works
pcaExplorer: an R/Bioconductor package for interacting with RNA-seq principal components
Marini, F., & Binder, H. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2879-1 · Cited by 260 · see citing works
Performance-Based Screening of Porous Materials for Carbon Capture
Farmahini, A. H., Krishnamurthy, S., Friedrich, D., Brandani, S., & Sarkisov, L. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.0c01266 · Cited by 7 · see citing works
Pinder, J. W., Major, G. H., Baer, D. R., Terry, J., Whitten, J. E., Čechal, J., Crossman, J. D., Lizarbe, A. J., Jafari, S., Easton, C. D., et al. (2024) · Applied Surface Science Advances · doi:10.1016/j.apsadv.2023.100534 · Cited by 272 · see citing works
Patil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243 · see citing works
Self-citationPick, J. L., Nakagawa, S., & Noble, D. W. A. (2018) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13118 · Cited by 256 · see citing works
AquaCrop-OS: An open source version of FAO's crop water productivity model
Foster, T., Brozović, N., Butler, A. P., Neale, C. M. U., Raes, D., Steduto, P., Fereres, E., & Hsiao, T. C. (2016) · Agricultural Water Management · doi:10.1016/j.agwat.2016.11.015 · Cited by 252 · see citing works
Machine-learning-based evidence and attribution mapping of 100,000 climate impact studies
Callaghan, M., Schleussner, C.-F., Nath, S., Lejeune, Q., Knutson, T. R., Reichstein, M., Hansen, G., Theokritoff, E., Andrijevic, M., Brecha, R. J., et al. (2021) · Nature Climate Change · doi:10.1038/s41558-021-01168-6 · Cited by 251 · see citing works
Marwick, B. (2016) · Journal of Archaeological Method and Theory · doi:10.1007/s10816-015-9272-9 · Cited by 246 · see citing works
Reporting guidelines for simulation‐based research in social sciences
Rahmandad, H., & Sterman, J. D. (2012) · System Dynamics Review · doi:10.1002/sdr.1481 · Cited by 245 · see citing works
Revisiting the serotonin–aggression relation in humans: A meta-analysis.
Duke, A. A., Bègue, L., Bell, R., & Eisenlohr-Moul, T. A. (2013) · Psychological Bulletin · doi:10.1037/a0031544 · Cited by 243 · see citing works
On the Reuse of Scientific Data
Pasquetto, I. V., Randles, B. M., & Borgman, C. L. (2017) · Data Science Journal · doi:10.5334/dsj-2017-008 · Cited by 237 · see citing works
Most computational hydrology is not reproducible, so is it really science?
Hutton, C., Wagener, T., Freer, J. E., Han, D., Duffy, C., & Arheimer, B. (2016) · Water Resources Research · doi:10.1002/2016wr019285 · Cited by 233 · see citing works
Git can facilitate greater reproducibility and increased transparency in science
Ram, K. (2013) · Source Code for Biology and Medicine · doi:10.1186/1751-0473-8-7 · Cited by 226 · see citing works
Performing Skin Microbiome Research: A Method to the Madness
Kong, H. H., Andersson, B., Clavel, T., Common, J., Jackson, S. A., Olson, N. D., Segre, J. A., & Traidl‐Hoffmann, C. (2017) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2016.10.033 · Cited by 226 · see citing works
Open science, reproducibility, and transparency in ecology
Powers, S. M., & Hampton, S. E. (2018) · Ecological Applications · doi:10.1002/eap.1822 · Cited by 224 · see citing works
Yang, P., Fang, H., & Lin, J. (2018) · Journal of Data and Information Quality · doi:10.1145/3239571 · Cited by 222 · see citing works
MultiDIC: An Open-Source Toolbox for Multi-View 3D Digital Image Correlation
Solav, D., Moerman, K. M., Jaeger, A. M., Genovese, K., & Herr, H. (2018) · IEEE Access · doi:10.1109/access.2018.2843725 · Cited by 221 · see citing works
THE MEANING OF FAILED REPLICATIONS: A REVIEW AND PROPOSAL
Clemens, M. A. (2015) · Journal of Economic Surveys · doi:10.1111/joes.12139 · Cited by 215 · see citing works
Reproducibility and replicability of rodent phenotyping in preclinical studies
Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.01.003 · Cited by 215 · see citing works
Student success prediction in MOOCs
Gardner, J., & Brooks, C. (2018) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-018-9203-z · Cited by 210 · see citing works
Towards reproducible computational drug discovery
Schaduangrat, N., Lampa, S., Simeon, S., Gleeson, M. P., Spjuth, O., & Nantasenamat, C. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-0408-x · Cited by 208 · see citing works
Accidental, open and everywhere: Emerging data sources for the understanding of cities
Arribas‐Bel, D. (2013) · Applied Geography · doi:10.1016/j.apgeog.2013.09.012 · Cited by 201 · see citing works
van Atteveldt, W., & Peng, T. (2018) · Communication Methods and Measures · doi:10.1080/19312458.2018.1458084 · Cited by 198 · see citing works
Reproducible research can still be wrong: Adopting a prevention approach
Leek, J. T., & Peng, R. D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1421412111 · Cited by 193 · see citing works
Self-citationMandated data archiving greatly improves access to research data
Vines, T., Andrew, R. L., Bock, D. G., Franklin, M. T., Gilbert, K. J., Kane, N. C., Moore, J., Moyers, B. T., Renaut, S., Rennison, D. J., et al. (2013) · The FASEB Journal · doi:10.1096/fj.12-218164 · Cited by 188 · see citing works
A survey of tools for the analysis of quantitative PCR (qPCR) data
Pabinger, S., Rödiger, S., Kriegner, A., Vierlinger, K., & Weinhäusel, A. (2014) · Biomolecular Detection and Quantification · doi:10.1016/j.bdq.2014.08.002 · Cited by 189 · see citing works
Usbeck, R., Röder, M., Ngomo, A.-C. N., Baron, C., Both, A., Brümmer, M., Ceccarelli, D., Cornolti, M., Cherix, D., Eickmann, B., et al. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741626 · Cited by 191 · see citing works
An estimate of the science-wise false discovery rate and application to the top medical literature
Jager, L. R., & Leek, J. T. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt007 · Cited by 179 · see citing works
Shining Light into Black Boxes
Morin, A., Urban, J. M., Adams, P. D., Foster, I. T., Šali, A., Baker, D. A., & Piotr Sliz, A. (2012) · Science · doi:10.1126/science.1218263 · Cited by 190 · see citing works
Journal of Computational Neuroscience
(2013) · Journal of Computational Neuroscience · doi:10.1007/10827.1573-6873 · Cited by 190 · see citing works
Essential guidelines for computational method benchmarking
Weber, L. M., Saelens, W., Cannoodt, R., Soneson, C., Hapfelmeier, A., Gardner, P. P., Boulesteix, A.-L., Saeys, Y., & Robinson, M. D. (2019) · Genome Biology · doi:10.1186/s13059-019-1738-8 · Cited by 21 · see citing works
Vision dominates in perceptual language: English sensory vocabulary is optimized for usage
Winter, B., Perlman, M., & Majid, A. (2018) · Cognition · doi:10.1016/j.cognition.2018.05.008 · Cited by 181 · see citing works
High-Throughput Methods in the Discovery and Study of Biomaterials and Materiobiology
Yang, L., Pijuan‐Galito, S., Rho, H. S., Vasilevich, A. S., Eren, A. D., Ge, L., Habibović, P., Alexander, M. R., de Boer, J., Carlier, A., et al. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.0c00752 · Cited by 187 · see citing works
Punzo, V., Zheng, Z., & Montanino, M. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2021.103165 · Cited by 175 · see citing works
Tools and techniques for computational reproducibility
Piccolo, S., & Frampton, M. (2016) · GigaScience · doi:10.1186/s13742-016-0135-4 · Cited by 167 · see citing works
Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks
Rule, A., Birmingham, A., Zúñiga, C., Altıntaş, İ., Huang, S.-C., Knight, R. D., Moshiri, N., Nguyen, M. H., Rosenthal, S. B., Pérez, F., et al. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007007 · Cited by 169 · see citing works
Pasichnyi, O., Wallin, J., Levihn, F., Shahrokni, H., & Kordas, O. (2019) · Energy Policy · doi:10.1016/j.enpol.2018.11.051 · Cited by 168 · see citing works
A Review of Software for Spatial Econometrics in R
Bivand, R., Millo, G., & Piras, G. (2021) · Mathematics · doi:10.3390/math9111276 · Cited by 167 · see citing works
Shiny-phyloseq: Web application for interactive microbiome analysis with provenance tracking
McMurdie, P. J., & Holmes, S. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu616 · Cited by 166 · see citing works
Changing Trends in Computational Drug Repositioning
Yella, J., Yaddanapudi, S., Wang, Y., & Jegga, A. G. (2018) · Pharmaceuticals · doi:10.3390/ph11020057 · Cited by 166 · see citing works
Uhrin, M., Huber, S. P., Yu, J., Marzari, N., & Pizzi, G. (2020) · Computational Materials Science · doi:10.1016/j.commatsci.2020.110086 · Cited by 166 · see citing works
Winter, B. (2019) · Converging evidence in language and communication research · doi:10.1075/celcr.20 · Cited by 169 · see citing works
Wu, T.-C., Xu, K., Martínek, J., Young, R. R., Banchereau, R., George, J., Turner, J., Kim, K. I., Zurawski, S., Wang, X., et al. (2018) · Cancer Research · doi:10.1158/0008-5472.can-18-0413 · Cited by 166 · see citing works
Major, G. H., Avval, T. G., Moeini, B., Pinto, G., Shah, D., Jain, V., Carver, V., Skinner, W. M., Gengenbach, T. R., Easton, C. D., et al. (2020) · Journal of Vacuum Science & Technology A Vacuum Surfaces and Films · doi:10.1116/6.0000685 · Cited by 165 · see citing works
Stimfit: quantifying electrophysiological data with Python
Guzman, S. J., Schlögl, A., & Schmidt‐Hieber, C. (2014) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2014.00016 · Cited by 161 · see citing works
A systematic survey on multi-step attack detection
Navarro, J., Deruyver, A., & Parrend, P. (2018) · Computers & Security · doi:10.1016/j.cose.2018.03.001 · Cited by 159 · see citing works
Using the Jupyter Notebook as a Tool for Open Science: An Empirical Study
Randles, B. M., Pasquetto, I. V., Golshan, M. S., & Borgman, C. L. (2017) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2017.7991618 · Cited by 155 · see citing works
Rapid, accurate, precise and reproducible ligand–protein binding free energy prediction
Wan, S., Bhati, A. P., Zasada, S. J., & Coveney, P. V. (2020) · Interface Focus · doi:10.1098/rsfs.2020.0007 · Cited by 152 · see citing works
A Review of Multi‐Compartment Infectious Disease Models
Tang, L., Zhou, Y., Wang, L., Purkayastha, S., Zhang, L., He, J., Wang, F., & Song, P. X. ‐. (2020) · International Statistical Review · doi:10.1111/insr.12402 · Cited by 156 · see citing works
Reproducibility of computational workflows is automated using continuous analysis
Beaulieu‐Jones, B. K., & Greene, C. S. (2017) · Nature Biotechnology · doi:10.1038/nbt.3780 · Cited by 151 · see citing works
Reproducibility in Scientific Computing
Ivie, P., & Thain, D. (2018) · ACM Computing Surveys · doi:10.1145/3186266 · Cited by 153 · see citing works
Rethomics: An R framework to analyse high-throughput behavioural data
Geissmann, Q., Rodríguez, L. G., Beckwith, E. J., & Gilestro, G. F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0209331 · Cited by 135 · see citing works
CODATA-ICSTI Task Group on Data Citation Standards and Practices (2013) · Data Science Journal · doi:10.2481/dsj.osom13-043 · Cited by 148 · see citing works
Brokamp, C., Wolfe, C., Lingren, T., Harley, J. B., & Ryan, P. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx128 · Cited by 147 · see citing works
Exploring the impact of analysis software on task fMRI results
Bowring, A., Maumet, C., & Nichols, T. E. (2019) · Human Brain Mapping · doi:10.1002/hbm.24603 · Cited by 158 · see citing works
Philosophy of science and the replicability crisis
Romero, F. (2019) · Philosophy Compass · doi:10.1111/phc3.12633 · Cited by 152 · see citing works
Current Challenges in Plant Eco-Metabolomics
Peters, K., Worrich, A., Weinhold, A., Alka, O., Balcke, G. U., Birkemeyer, C., Bruelheide, H., Calf, O. W., Dietz, S., Dührkop, K., et al. (2018) · International Journal of Molecular Sciences · doi:10.3390/ijms19051385 · Cited by 139 · see citing works
From characterising three years of HRI to methodology and reporting recommendations
Baxter, P. E., Kennedy, J., Senft, E., Lemaignan, S., & Belpaeme, T. (2016) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2016.7451777 · Cited by 144 · see citing works
Legality and Ethics of Web Scraping
Krotov, V., Johnson, L., & Silva, L. (2020) · Communications of the Association for Information Systems · doi:10.17705/1cais.04724 · Cited by 138 · see citing works
Rethinking Reproducibility as a Criterion for Research Quality
Leonelli, S. (2018) · Research in the history of economic thought and methodology · doi:10.1108/s0743-41542018000036b009 · Cited by 136 · see citing works
- Authors
- Peng, R. D.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Peng, Roger D.
- ORCID
- Peng, Roger D.
- Venue
- Science
- Published
- 2011-12-02
- Volume
- 334
- Issue
- 6060
- Pages
- 1226-1227
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- PMC
- PMC3383002
- OpenAlex
- W2008985174
- Semantic Scholar
- e4625641d1072542315c14083befb43cd7096aba
- Cited by
- 1,449
- References
- 11
- Citations by year
- 2026: 162 · 2025: 75 · 2024: 63 · 2023: 74 · 2022: 85 · 2021: 121 · 2020: 139 · 2019: 162 · 2018: 138 · 2017: 113 · 2016: 106 · 2015: 82 · 2014: 51 · 2013: 46 · 2012: 29
- FWCI
- 98.43
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Environmental Health Sciences (R01ES019560)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- reproducible research, scientific computing, reproducibility, scientific claims, replication
- MeSH terms
- Computers, Publishing, Research, Reproducibility of Results, Computational Biology, Editorial Policies, Biomedical Research, Biostatistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
