Ten simple rules for building and maintaining a responsible data science workflow
Open accessThe DOI matches its Crossref record; 115 of the 116 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Contributors and beneficiaries of data-intensive research have become increasingly concerned about social and ethical risks from data science and machine learning applications
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Factors associated with COVID-19-related death using OpenSAFELY
Williamson, E. J., Walker, A. J., Bhaskaran, K., Bacon, S., Bates, C., Morton, C. E., Curtis, H. J., Mehrkar, A., Evans, D., Inglesby, P., et al. (2020) · Nature · doi:10.1038/s41586-020-2521-4 · Cited by 6,674
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
The Mythos of Model Interpretability
Lipton, Z. C. (2016) · arXiv · doi:10.48550/arxiv.1606.03490 · Cited by 5,422 · cited 1×
“In some settings, current hopes for interpretable and explainable AI may be unrealistic, and a rigorous internal and external validation of models may better achieve the goals of interpretability and fairness in a responsible data science workflow [49,50].”
BackgroundThe Sociology of Science: Theoretical and Empirical Investigations
Kaplan, N., Merton, R. K., & Storer, N. W. (1974) · Technology and Culture · doi:10.2307/3102264 · Cited by 4,005
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Doyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,071
Jupyter Notebooks – a publishing format for reproducible computational workflows
Thomas, K., Benjamin, R.-K., Fernando, P., Brian, G., Matthias, B., Jonathan, F., Kyle, K., Jessica, H., Jason, G., Sylvain, C., et al. (2016) · Positioning and Power in Academic Publishing: Players, Agents and Agendas · doi:10.3233/978-1-61499-649-1-87 · Cited by 3,063
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · cited 2×
“However, incentives in the current reward system of scientific research and of competitive industries are often misaligned with responsible data science and ML applications, favoring novel positive findings at the expense of model transparency, reproducibility, or bias evaluation [21,23,24,120].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · cited 2×
“By separating exploration from testing of predictions, preregistration brings researcher degrees of freedom to light [76] and helps protect researchers from biases that are otherwise hard to avoid [76,77], including possible selective reporting and overreporting of false positive results [74,75].”
BackgroundMerz, S. (2020) · Ethnic and Racial Studies · doi:10.1080/01419870.2020.1715454 · Cited by 1,870
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
A unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 1×
“This is understandable given that fairness criteria have been found to be incompatible and no method or metric can satisfy desirable fairness criteria simultaneously [40–42].”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Ten Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880 · cited 2×
“You can also integrate your computational notebooks with version-control software such as Git, GitHub, and GitLab [72,83] so that others can access your code and its history, and possibly collaborate and/or reproduce your research.”
BackgroundAuthor Sequence and Credit for Contributions in Multiauthored Publications
Tscharntke, T., Hochberg, M. E., Rand, T. A., Resh, V. H., & Krauss, J. (2007) · PLoS Biology · doi:10.1371/journal.pbio.0050018 · Cited by 579
Promoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540
Aequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577 · Cited by 496
A Second Chance to Get Causal Inference Right: A Classification of Data Science Tasks
Hernán, M. A., Hsu, J., & Healy, B. (2019) · CHANCE · doi:10.1080/09332480.2019.1579578 · Cited by 495
Operationalizing the CARE and FAIR Principles for Indigenous data futures
Carroll, S. R., Herczog, E., Hudson, M., Russell, K., & Stall, S. (2021) · Scientific Data · doi:10.1038/s41597-021-00892-0 · Cited by 483
Introducing the FAIR Principles for research software
Barker, M., Chue Hong, N. P., Katz, D. S., Lamprecht, A.-L., Martinez-Ortiz, C., Psomopoulos, F., Harrow, J., Castro, L. J., Gruenpeter, M., Martinez, P. A., et al. (2022) · Scientific Data · doi:10.1038/s41597-022-01710-x · Cited by 483
Assessing scientists for hiring, promotion, and tenure
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004089 · Cited by 462 · cited 1×
“Performance assessment of researchers for hiring, promotion, and tenure often considers metrics such as the number of published papers, citations, and Journal Impact Factor (JIF) [121].”
BackgroundAlgorithms of Oppression: How Search Engines Reinforce Racism
Noble, S. U. (2020) · New York University Press eBooks · doi:10.18574/nyu/9781479833641.001.0001 · Cited by 416
ETHICS GUIDELINES FOR TRUSTWORTHY AI
Mason, C. L. (2018) · Cited by 414
Gender-diverse teams produce more novel and higher-impact scientific ideas
Yang, Y., Tian, T. Y., Woodruff, T. K., Jones, B. F., & Uzzi, B. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2200841119 · Cited by 360
The EU Approach to Ethics Guidelines for Trustworthy Artificial Intelligence
Smuha, N. A. (2019) · Computer Law Review International · doi:10.9785/cri-2019-200402 · Cited by 356
Why Jupyter is data scientists’ computational notebook of choice
Perkel, J. M. (2018) · Nature · doi:10.1038/d41586-018-07196-1 · Cited by 352
Publishing: Credit where credit is due
Allen, L., Scott, J., Brand, A., Hlava, M., & Altman, M. (2014) · Nature · doi:10.1038/508312a · Cited by 348
Ten simple rules for responsible big data research
Zook, M., Barocas, S., boyd, D., Crawford, K., Keller, E., Gangadharan, S. P., Goodman, A., Hollander, R., Koenig, B. A., Metcalf, J., et al. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005399 · Cited by 343
Usage, costs, and benefits of continuous integration in open-source projects
Hilton, M., Tunnell, T., Huang, K., Marinov, D., & Dig, D. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970358 · Cited by 342
Implementing Reproducible Research
Stodden, V. (2014) · doi:10.1201/b16868 · Cited by 314 · cited 1×
“Both computational transparency and reproducibility are critical for data-informed computational research [15,20,67–73].”
BackgroundHow scientists fool themselves – and how they can stop
Nuzzo, R. (2015) · Nature · doi:10.1038/526182a · Cited by 296
Bit by Bit: Social Research in the Digital Age
Elgesem, D. (2020) · European Journal of Communication · doi:10.1177/0267323120922090 · Cited by 284
Reproducibility standards for machine learning in the life sciences
Heil, B. J., Hoffman, M. M., Markowetz, F., Lee, S.-I., Greene, C. S., & Hicks, S. C. (2021) · Nature Methods · doi:10.1038/s41592-021-01256-7 · Cited by 255
Open Science by Design: Realizing a Vision for 21st Century Research
Affairs, P. A. G., & National Academies of Sciences, E. (2018) · National Academies Press eBooks · doi:10.17226/25116 · Cited by 254
Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., & Larochelle, H. (2020) · arXiv · doi:10.48550/arxiv.2003.12206 · Cited by 232 · cited 1×
“…of Individual Prognosis Or Diagnosis-Artificial Intelligence), reproducibility checklists, such as “The Machine Learning Reproducibility Checklist” [99], “Model info sheets for detecting and preventing leakage” [48], and the “NLP Reproducibility Checklist,” and/or take part in the ML…”
Backgroundtestthat: Get Started with Testing
Wickham, H. (2011) · The R Journal · doi:10.32614/rj-2011-002 · Cited by 223
Fairlearn: Assessing and Improving Fairness of AI SystemsPreprint
Weerts, H., Dudík, M., Edgar, R., Jalali, A., Lutz, R., & Madaio, M. (2023) · TU/e Research Portal · doi:10.48550/arxiv.2303.16626 · Cited by 195
rmarkdown: Dynamic Documents for R
Allaire, J., Xie, Y., Dervieux, C., McPherson, J., Luraschi, J., Ushey, K., Atkins, A., Wickham, H., Cheng, J., Chang, W., et al. (2014) · CRAN: Contributed Packages · doi:10.32614/cran.package.rmarkdown · Cited by 195
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
Transparent and Reproducible Social Science Research
Christensen, G., Jeremy Freese, A., & Miguel, E. A. (2019) · University of California Press eBooks · doi:10.1525/9780520969230 · Cited by 171
Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks
Rule, A., Birmingham, A., Zuniga, C., Altintas, I., Huang, S.-C., Knight, R., Moshiri, N., Nguyen, M. H., Rosenthal, S. B., Pérez, F., et al. (2019) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007007 · Cited by 168
Goble, C., Cohen-Boulakia, S., Soiland-Reyes, S., Garijo, D., Gil, Y., Crusoe, M. R., Peters, K., & Schober, D. (2020) · Data Intelligence · doi:10.1162/dint_a_00033 · Cited by 167
2022 ACM Conference on Fairness Accountability and Transparency
(2022) · doi:10.1145/3531146 · Cited by 146
Davis, J. L., Williams, A., & Yang, M. W. (2021) · Big Data & Society · doi:10.1177/20539517211044808 · Cited by 143
Ten simple rules for writing Dockerfiles for reproducible data science
Nüst, D., Sochat, V., Marwick, B., Eglen, S. J., Head, T., Hirst, T., & Evans, B. D. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008316 · Cited by 142
Self-citationEthical guidelines for peer reviewers (English)
(2013) · doi:10.24318/cope.2019.1.9 · Cited by 136
Leakage and the Reproducibility Crisis in ML-based Science
Kapoor, S., & Narayanan, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.07048 · Cited by 133
Scientific workflows: Past, present and future
Atkinson, M., Gesing, S., Johan Montagnat, A., & Taylor, I. (2017) · Future Generation Computer Systems · doi:10.1016/j.future.2017.05.041 · Cited by 131
Transparent and Reproducible Social Science Research: How to Do Open Science
Christensen, G., Freese, J., & Miguel, E. A. (2019) · Cited by 129
Historical and impact analysis of API breaking changes: A large-scale study
Xavier, L., Brito, A., Hora, A., & Valente, M. T. (2017) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2017.7884616 · Cited by 127
A Global, Neutral Platform for Sharing Trial Data
Bierer, B. E., Li, R., Barnes, M., & Sim, I. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1605348 · Cited by 115
Responsible Research and Innovation
Zwart, H. A. E. (2016) · Radboud Repository (Radboud University) · Cited by 115
When and How to Make Breaking Changes: Policies and Practices in 18 Open Source Software Ecosystems
Bogart, C., Kästner, C., Herbsleb, J. D., & Thung, F. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3447245 · Cited by 115
Packaging Data Analytical Work Reproducibly Using R (and Friends)
Marwick, B., Boettiger, C., & Mullen, L. (2018) · The American Statistician · doi:10.1080/00031305.2017.1375986 · Cited by 108
Self-citationThe Turing Way: A Handbook for Reproducible Data Science
Sharan, M., Arnold, B., Gibson, S., Whitaker, K., Owoicho, P., Karoune, E., Ainsworth, R. E., Herterich, P., Carter, L., Plomp, E., et al. (2019) · doi:10.5281/zenodo.3233853 · Cited by 108
Self-citationHow to achieve trustworthy artificial intelligence for health.
Bærøe, K., Miyata-Sturm, A., & Henden, E. (2020) · Bulletin of the World Health Organization · doi:10.2471/blt.19.237289 · Cited by 103
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
Self-citationGender, Productivity, and Prestige in Computer Science Faculty Hiring Networks
Way, S. F., Larremore, D. B., & Clauset, A. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883073 · Cited by 90
Applying Code Inspection to Spreadsheet Testing
Panko, R. R. (1999) · Journal of Management Information Systems · doi:10.1080/07421222.1999.11518250 · Cited by 89
Model Multiplicity: Opportunities, Concerns, and Solutions
Black, E., Raghavan, M., & Barocas, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533149 · Cited by 85 · cited 1×
“…multiplicity (when for the same prediction task, there are multiple models that have equal accuracy but differ in their individual predictions and fairness properties) [47] and data leakage (when information related to the target variable in the test data is “leaked” to the training data) [48].”
BackgroundCreating and sharing reproducible research code the workflowr way
Blischak, J. D., Carbonetto, P., & Stephens, M. (2019) · F1000Research · doi:10.12688/f1000research.20843.1 · Cited by 77
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 75
Self-citationBuilding Software, Building Community: Lessons from the rOpenSci Project
Boettiger, C., Chamberlain, S., Hart, E., & Ram, K. (2015) · Journal of Open Research Software · doi:10.5334/jors.bu · Cited by 74
Principles for data analysis workflows
Stoudt, S., Vásquez, V., & Martinez, C. C. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008770 · Cited by 64
Self-citationTen simple rules on writing clean and reliable open-source scientific software
Hunter-Zinck, H., de Siqueira, A. F., Vásquez, V., Barnes, R. J., & Martinez, C. C. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009481 · Cited by 63
Reproducible Research with R and RStudio
Gandrud, C. (2020) · Chapman & Hall/CRC eBooks · doi:10.1201/9780429031854 · Cited by 63
Teaching Responsible Data Science: Charting New Pedagogical Territory
Lewis, A., & Stoyanovich, J. (2021) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00241-7 · Cited by 62
Interrogating Algorithmic Bias: From Speculative Fiction to Liberatory Design
Gaskins, N. R. (2022) · TechTrends · doi:10.1007/s11528-022-00783-0 · Cited by 60
‘Just What do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP
Rogers, A., Baldwin, T., & Leins, K. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.414 · Cited by 53
Packaging data analytical work reproducibly using R (and friends)Preprint
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · doi:10.7287/peerj.preprints.3192v1 · Cited by 50
Self-citationGendered behavior as a disadvantage in open source software development
Vedres, B., & Vásárhelyi, O. (2019) · EPJ Data Science · doi:10.1140/epjds/s13688-019-0202-z · Cited by 50
Srinivasan, R., & Chander, A. (2021) · Queue · doi:10.1145/3466132.3466134 · Cited by 49
Inclusion unlocks the creative potential of gender diversity in teams
Vedres, B., & Vásárhelyi, O. (2023) · Scientific Reports · doi:10.1038/s41598-023-39922-9 · Cited by 48
Understanding Potential Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. V. (2021) · doi:10.21428/2c646de5.c16a07bb · Cited by 44
Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP
Rogers, A., Baldwin, T., & Leins, K. (2021) · arXiv · doi:10.48550/arxiv.2109.06598 · Cited by 42
Computing Workflows for Biologists: A Roadmap
Shade, A., & Teal, T. K. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002303 · Cited by 40
The Practice of Reproducible Research: Case Studies and Lessons from the Data-Intensive Sciences
(2017) · University of California Press eBooks · doi:10.1525/9780520967779 · Cited by 40
Designing a Moral Compass for the Future of Computer Vision Using Speculative Analysis
Skirpan, M. W., & YEH, T. P. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.179 · Cited by 28
The Political Economy of AI: Towards Democratic Control of the Means of PredictionPreprint
Kasy, M. (2023) · SSRN Electronic Journal · doi:10.31235/osf.io/x7pcy · Cited by 28
Centralized scientific communities are less likely to generate replicable results
Danchev, V., Rzhetsky, A., & Evans, J. A. (2019) · eLife · doi:10.7554/elife.43094 · Cited by 27
Self-citationA Brief History of Free, Open Source Software and Its Communities
Gonzalez-Barahona, J. M. (2021) · Computer · doi:10.1109/mc.2020.3041887 · Cited by 24
Equity and Access in Algorithms Mechanisms and Optimization
(2023) · doi:10.1145/3617694 · Cited by 19
The Division of Labor in Teams: A Conceptual Framework and Application to Collaborations in Science
Haeussler, C., & Sauermann, H. (2016) · National Bureau of Economic Research · doi:10.3386/w22241 · Cited by 18
Trusted Research Environments (TRE) Green Paper
Hubbard, T., Reilly, G., Varma, S. D., & Seymour, D. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4594703 · Cited by 16
MIT Case Studies in Social and Ethical Responsibilities of Computing
(2021) · doi:10.21428/2c646de5 · Cited by 16
How to navigate authorship of scientific manuscripts
Pain, E. (2021) · Science · doi:10.1126/science.caredit.abj3459 · Cited by 15
Kerasidou, C., Malone, M., Daly, A., & Tava, F. (2023) · Journal of Medical Ethics · doi:10.1136/jme-2022-108696 · Cited by 15
Diagnosing Data Analytic Problems in the Classroom
Peng, R. D., Chen, A., Bridgeford, E., Leek, J. T., & Hicks, S. C. (2021) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2021.1971586 · Cited by 13
A Realistic Guide to Making Data Available Alongside Code to Improve ReproducibilityPreprint
Tierney, N. J., & Ram, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11626 · Cited by 12
Ten simple rules to cultivate belonging in collaborative data science research teams
Gaynor, K. M., Azevedo, T., Boyajian, C., Brun, J., Budden, A. E., Cole, A., Csik, S. R., DeCesaro, J. M., Do-Linh, H., Dudney, J. C., et al. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010567 · Cited by 10
Good Enough Practices in Scientific Computing
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005510 · Cited by 8
Eitzel, M. V. (2021) · Research Ideas and Outcomes · doi:10.3897/rio.7.e71553 · Cited by 8
Autoethnographic assessment of a manifesto for more trustworthy, relevant, and just models
Eitzel, M. V. (2023) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2023.105690 · Cited by 7
Reproducible Data Science with Python: An Open Learning Resource
Danchev, V. (2022) · Journal of Open Source Education · doi:10.21105/jose.00156 · Cited by 6
Self-citationUnit Testing Principles, Practices, and Patterns
Khorikov, V. (2020) · Cited by 5
Marshall, B. H. (2022) · Wiley eBooks · doi:10.1002/9781394320721 · Cited by 4
Self-citationResponsible Research and Innovation
Grinbaum, A. (2021) · J.B. Metzler eBooks · doi:10.1007/978-3-476-04901-8_90 · Cited by 3
(2018) · Digital Commons@Becker (Washington University School of Medicine) · doi:10.1371/journal
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Self-citation
Research works printed but not linked: 24
- C. O’Neil, 2016, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
- Association for Computational Linguistics, 2022, In: ACL Rolling Review [Internet].
- S Lau, 2023, Learning Data Science
- EOSC, 2024, In: European Open Science Cloud [Internet].
- R. Benjamin, 2019, Race After Technology: Abolitionist Tools for the New Jim Code
- R. Jeschke, 2019, Affordances: Science Fiction About Algorithmic Bias and Technological Resistance, In: Electronic Frontier Foundation [Internet].
- J. Tashea, 2022, 40 Futures: v1.00 Read_Me
- S Barocas, 2023, Fairness and Machine Learning: Limitations and Opportunities
- JD West, 2020, Calling Bullshit: The Art of Skepticism in a Data-Driven World
- US Census Bureau, 2019, A History of Census Privacy Protections
- B. Wolford, 2023, What is GDPR, the EU’s new data protection law?
- 2024, About large files on GitHub, In: GitHub Docs [Internet].
- S. Perez, 2022, Twitter alternative Hive shuts down its app to fix critical security issues, In: TechCrunch [Internet].
- 2019, Reproducibility and Replicability in Science.
- T. Lestang, 2021, Code reviews in academia
- J. VanderPlas, 2017, Reproducible Data Analysis in Jupyter
- H Wickham, 2023, Package “usethis.”, CRAN
- International Committee of Medical Journal Editors, 2023, Defining the Role of Authors and Contributors
- New York Times, 2023, Open: How we design and build digital products at The New York Times.
- Pew Research Center, 2023, Decoded: A behind-the-scenes blog about research methods at Pew Research Center
- Data Science by Design Collective, 2023, Our Environment
- Data Visualization Society, 2023, Nightingale: Journal of the Data Visualization Society
- Institute for Mathematical and Statistical Innovation, 2023, MathStatBites
- Royal Statistical Society, 2023, Real World Data Science
Cited by
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
“Applied not only to data but also to software, they ensure reproducibility by making workflows, tools, and dependencies openly accessible [26,314].”
The LISTEN principles for genetic sequence data governance and database engineering
Carlson, C. J., Granados, M., Phelan, A., Ramakrishnan, N., & Poisot, T. (2025) · Nature Genetics · doi:10.1038/s41588-025-02270-7 · Cited by 8
Big data, sound science, lasting impact: A framework for passive acoustic monitoring
Wall, C. C., McKenna, M. F., Hatch, L. T., Parijs, S. M. V., Bochenek, R., Dugan, P. J., Parcerisas, C., Ryan, J., Anderson, C. D., Becker, K. M., et al. (2025) · Big Earth Data · doi:10.1080/20964471.2025.2583505 · Cited by 7
Ethics for Responsible Data Research: Integrating Cybersecurity Perspectives in a Digital Era
Temara, S. (2024) · Advances in data mining and database management book series · doi:10.4018/979-8-3693-3848-3.ch005 · Cited by 6
Ethics For Responsible Data Research: Integrating Cybersecurity Perspectives In Digital EraPreprint
Temara, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5045045 · Cited by 2
Metadata
- Authors
- Stoudt, S., Jernite, Y., Marshall, B., Marwick, B., Sharan, M., Whitaker, K., & Danchev, V.
- Affiliations
- Bucknell University (US)
- Hugging Face (US)
- University of Washington (US)
- The Alan Turing Institute (GB)
- Queen Mary University of London (GB)
- Corresponding author
- Stoudt, Sara
- Danchev, Valentin
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2024-07-18
- Volume
- 20
- Issue
- 7
- Pages
- e1012232
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC11257324
- OpenAlex
- W4400770804
- Cited by
- 11
- References
- 104
- Citations by year
- 2026: 4 · 2025: 6 · 2024: 1
- FWCI
- 4.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Engineering and Physical Sciences Research Council (EP/T001569/1, EP/W006022/1, EP/X03870X/1)
- Alan Turing Institute
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- responsible data science, data science workflow, machine learning ethics, data-intensive research, reproducibility, responsible conduct of research
- MeSH terms
- Data Science, Humans, Computational Biology, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
![Fig 1 Components in a responsible data science workflow. For each stage in the data science life cycle, we identify associated practices of research ethics, risks, and responsibilities towards key stakeholders. This figure is inspired by [6,16].](https://figures.cochat.ai/paper-figures/29fc63dae0444014be620a5d2dbe0fa4/ten-simple-rules-for-building-and-maintaining-a-responsible-data-science-figure-1.jpg)