REFORMS: Consensus-based Recommendations for Machine-learning-based Science
Open accessThe DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Abstract
Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“For example, there is an active debate on how or if interpretability or explainability should be implemented in the context of different applications [see e.g., ( 141 )].”
Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., Egger, M., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040297 · Cited by 8,613
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · cited 1×
“Not using a held-out test set is a textbook error in ML ( 158 ).”
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · cited 3×
“Breiman’s proposition was that more scientific research should use algorithmic modeling ( 12 ).”
Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure
Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera‐Arroita, G., Hauenstein, S., Lahoz‐Monfort, J. J., Schröder, B., Thuiller, W., et al. (2017) · Ecography · doi:10.1111/ecog.02881 · Cited by 3,689
STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660
AUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499
Scientists rise up against statistical significance
Amrhein, V., Greenland, S., & McShane, B. (2019) · Nature · doi:10.1038/d41586-019-00857-9 · Cited by 2,983 · cited 1×
“As a result, reporting uncertainty is better than performing statistical tests alone ( 178 ).”
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · cited 2×
“The Transparency and Openness Promotion standards introduced by the Open Science Foundation ( 93 ) have a few sections that focus on computational reproducibility in scientific research.”
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Similarly, Mitchell et al. ( 138 ) introduced model cards to document details about the modeling and evaluation process, with a primary focus on natural language processing and computer vision.”
Self-citationBig Data and Machine Learning in Health Care
Beam, A. L., & Kohane, I. S. (2018) · JAMA · doi:10.1001/jama.2017.18391 · Cited by 2,506
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 2×
“This follows recent calls for more granular details about data, such as by Gebru et al. ( 98 ).”
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Machine Learning: An Applied Econometric Approach
Mullainathan, S., & Spiess, J. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.87 · Cited by 1,989 · cited 1×
“Machine learning (ML) methods are being widely adopted for scientific research ( 1 – 11 ).”
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
Machine learning phases of matter
Carrasquilla, J., & Melko, R. G. (2017) · Nature Physics · doi:10.1038/nphys4035 · Cited by 1,448 · cited 1×
“For instance, it is likely to be more useful for predictive modeling compared to research that uses ML methods for search tasks in vast and complex spaces, such as the search for new materials or new phases of matter ( 54 ).”
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 5×
“For example, the CLAIM checklist ( 35 ) provides best practices for reporting research on AI in medical imaging.”
InfluentialLeakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
Self-citationRoberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044
Self-citationExplainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040 · cited 1×
“For example, a model that is used in high-stakes settings like clinical decision-making may need to be interpretable or explainable ( 140 ), so it may be more appropriate to choose a model type that is likelier to have these properties.”
Constraints on Generality (COG): A Proposed Addition to All Empirical Papers
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617708630 · Cited by 986 · cited 3×
“Simons et al. ( 60 ) argue that making an explicit “constraints on generality” statement that identifies boundaries of the circum-stances where findings are expected to hold has several benefits, including helping to ensure that a study’s conclusions accurately reflect its evidence, increasing the…”
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Clinical Chemistry · doi:10.1373/clinchem.2015.246280 · Cited by 984
Ten Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880 · cited 1×
“Similarly, it is important to report the exact version of code used for running the experiments and producing the results in a paper ( 105 ).”
Plint, A. C., Moher, D., Morrison, A., Schulz, K., Altman, D. G., Hill, C., & Gaboury, I. (2006) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2006.tb00557.x · Cited by 872
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855
Leakage in data mining: Formulation, detection, and avoidance
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2382577.2382579 · Cited by 791
Establishment of Best Practices for Evidence for Prediction: A Review
Poldrack, R. A., Huckins, G., & Varoquaux, G. (2020) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.3671 · Cited by 738
Self-citationArtificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707 · cited 1×
“However, the rapid uptake of ML methods has been accompanied by concerns of validity, reproducibility, and generalizability ( 13 – 20 ).”
Machine learning and its applications to biology
Tarca, A. L., Carey, V. J., Chen, X.-W., Romero, R., & Drăghici, S. (2007) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0030116 · Cited by 656 · cited 1×
“In addition, many recent articles provide guidance on the value of ML methods for science ( 1 , 7 , 48 , 62 , 67 – 71 ).”
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551 · cited 1×
“Conversely, if the code and data for reproducing all results in a study are not available, then identifying the precise sources of errors in a study becomes hard ( 79 – 81 ).”
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540 · cited 1×
“Conversely, if the code and data for reproducing all results in a study are not available, then identifying the precise sources of errors in a study becomes hard ( 79 – 81 ).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · cited 1×
“External validity of claims about fitted models is more widely known as “domain generalization” or “robustness to distribution shift” and is a well-studied phenomenon in ML ( 186 , 187 ).”
How Is ChatGPT’s Behavior Changing Over Time?
Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507
Applications of machine learning in animal behaviour studies
Valletta, J. J., Torney, C., Kings, M., Thornton, A., & Madden, J. (2017) · Animal Behaviour · doi:10.1016/j.anbehav.2016.12.005 · Cited by 505
The Availability of Research Data Declines Rapidly with Article Age
Vines, T. H., Albert, A. Y. K., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., Gilbert, K. J., Moore, J.-S., Renaut, S., & Rennison, D. J. (2014) · Current Biology · doi:10.1016/j.cub.2013.11.014 · Cited by 482
Supervised Machine Learning for Population Genetics: A New Paradigm
Schrider, D. R., & Kern, A. D. (2018) · Trends in Genetics · doi:10.1016/j.tig.2017.12.005 · Cited by 453
Prediction and explanation in social systems
Hofman, J. M., Sharma, A., & Watts, D. J. (2017) · Science · doi:10.1126/science.aal3856 · Cited by 429
Self-citationIntegrating explanation and prediction in computational social science
Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., et al. (2021) · Nature · doi:10.1038/s41586-021-03659-0 · Cited by 371 · cited 4×
“Several responses to Breiman have argued for the value of merging the two cultures or moving iteratively between them ( 48 , 62 – 66 ).”
InfluentialSelf-citationAn empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
Challenges to the Reproducibility of Machine Learning Models in Health Care
Beam, A. L., Manrai, A. K., & Ghassemi, M. (2020) · JAMA · doi:10.1001/jama.2019.20866 · Cited by 366
Quality of reporting of observational longitudinal research.
Tooth, L. (2005) · American Journal of Epidemiology · doi:10.1093/aje/kwi042 · Cited by 363
Jacobs, A. Z., & Wallach, H. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445901 · Cited by 362 · cited 1×
“This is especially important because mismatches between variables and the constructs they are purported to represent can create fairness issues ( 120 ).”
Plonsky (2013) · doi:10.1017/s0272263113000399 · Cited by 347
Measuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322
Self-citationUncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300 · cited 1×
“In addition, communicating and reasoning about uncertainty is important, but uncertainty is currently under-reported ( 169 – 173 ).”
Machine Learning for Sociology
Molina, M., & Garip, F. (2019) · Annual Review of Sociology · doi:10.1146/annurev-soc-073117-041106 · Cited by 295
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293
Self-citationReproducibility in machine learning for health research: Still a ways to go
McDermott, M. B. A., Wang, S., Marinsek, N., Ranganath, R., Foschini, L., & Ghassemi, M. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.abb1655 · Cited by 273
Nijman, S., Leeuwenberg, A., Beekers, I., Verkouter, I., Jacobs, J., Bots, M., Asselbergs, F., Moons, K., & Debray, T. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.11.023 · Cited by 269 · cited 1×
“Extensive literature across multiple fields has established that research articles frequently provide insufficient information about the presence and handling of missing data ( 59 , 113 , 126 , 128 – 132 ).”
The Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
Self-citationYe, C., Fu, T., Hao, S., Zhang, Y., Wang, O., Jin, B., Xia, M., Liu, M., Zhou, X., Wu, Q., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9268 · Cited by 250 · cited 1×
“Filho et al. ( 166 ) found that a prominent paper on hypertension prediction ( 167 ) suffered from data leakage due to illegitimate features.”
Unrepresentative big surveys significantly overestimated US vaccine uptake
Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., & Flaxman, S. (2021) · Nature · doi:10.1038/s41586-021-04198-4 · Cited by 248 · cited 1×
“When the sampling frame is unrepresentative, increasing the sample size can shrink confidence intervals without decreasing bias in estimates, thus giving false confidence ( 123 ).”
Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., & Larochelle, H. (2020) · arXiv · doi:10.48550/arxiv.2003.12206 · Cited by 232 · cited 4×
“Pineau et al. ( 32 ) found that only around half of the papers submitted to NeurIPS 2018, a leading ML conference, contained the code and data needed to reproduce results.”
InfluentialNonreplicable publications are cited more than replicable ones
Serra-Garcia, M., & Gneezy, U. (2021) · Science Advances · doi:10.1126/sciadv.abd1705 · Cited by 177 · cited 1×
“If left unchecked, these flaws can lead to a feedback loop of overoptimism because nonreplicable findings are cited more than replicable ones ( 33 ).”
Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
Islam, R., Henderson, P., Gomrokchi, M., & Precup, D. (2017) · arXiv · doi:10.48550/arxiv.1708.04133 · Cited by 177 · cited 1×
“Islam et al. ( 149 ) show that because of variance in performance depending on hyperparameters, specific choices of hyperparameters can be mis-leading when interpreting results.”
Reproducible and reusable research: are journal data sharing policies meeting the mark?
Vasilevsky, N. A., Minnier, J., Haendel, M. A., & Champieux, R. E. (2017) · PeerJ · doi:10.7717/peerj.3208 · Cited by 165 · cited 1×
“Vasilevsky et al. ( 84 ) studied the data-sharing policies at 318 bio-medical journals.”
Han, S., Olonisakin, T. F., Pribis, J. P., Zupetic, J., Yoon, J. H., Holleran, K. M., Jeong, K., Shaikh, N., Rubio, D. M., & Lee, J. S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0183591 · Cited by 131
McCoy, L. G., Brenna, C. T. A., Chen, S. S., Vold, K., & Das, S. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.11.001 · Cited by 118
Let's (not) stick together: pairwise similarity biases cross-validation in activity recognition
Hammerla, N. Y., & Plötz, T. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2807551 · Cited by 91
Model Multiplicity: Opportunities, Concerns, and Solutions
Black, E., Raghavan, M., & Barocas, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533149 · Cited by 85 · cited 1×
“Black et al. ( 147 ) provide guidance on opportunities and concerns that arise from such issues.”
Macleod, M., Collings, A. M., Graf, C., Kiermer, V., Mellor, D., Swaminathan, S., Sweet, D., & Vinson, V. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2103238118 · Cited by 77
Reproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77
Computational Reproducibility via Containers in Psychology
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.892 · Cited by 69
Meaningless comparisons lead to false optimism in medical machine learning
DeMasi, O., Kording, K., & Recht, B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0184604 · Cited by 63
Liu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47 · cited 2×
“Liu and Salganik ( 25 ) describe their experiences ensuring computational reproducibility while editing a special issue in Socius on the Fragile Families Challenge.”
Self-citationJust What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP
Rogers, A., Baldwin, T., & Leins, K. (2021) · arXiv · doi:10.48550/arxiv.2109.06598 · Cited by 42
Predictive Multiplicity in Probabilistic Classification
Watson-Daniels, J., Parkes, D. C., & Ustun, B. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26227 · Cited by 27
Show Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · arXiv · doi:10.48550/arxiv.1909.03004 · Cited by 22
Overfitting to ‘predict’ suicidal ideation
Verstynen, T., & Kording, K. P. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01560-6 · Cited by 15
Cited by
Fleurence, R. L., Bian, J., Wang, X., Xu, H., Dawoud, D., Higashi, M., & Chhatwal, J. (2025) · Value in Health · doi:10.1016/j.jval.2024.10.3846 · Cited by 54
Why an overreliance on AI-driven modelling is bad for science
Narayanan, A., & Kapoor, S. (2025) · Nature · doi:10.1038/d41586-025-01067-2 · Cited by 21
Self-citationThe reanimation of pseudoscience in machine learning and its ethical repercussions
Andrews, M., Smart, A., & Birhane, A. (2024) · Patterns · doi:10.1016/j.patter.2024.101027 · Cited by 18
Vrba, J., Steinbach, J., Jirsa, T., Verde, L., De Fazio, R., Zeng, Y., Ichiji, K., Hájek, L., Sedláková, Z., Urbániová, Z., et al. (2025) · Journal of Voice · doi:10.1016/j.jvoice.2025.03.028 · Cited by 6
“Especially regarding reproducibility, to our best knowledge, this is the first paper on voice pathology detection combining SVD and ML methods, that is fully re-producible and conforms to the REFORMS practices [55].”
Metadata
- Authors
- Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J., Lones, M. A., Malik, M. M., Nanayakkara, P., Poldrack, R. A., Raji, I. D., Roberts, M., Salganik, M. J., Serra-Garcia, M., Stewart, B. M., Vandewiele, G., & Narayanan, A.
- Venue
- Science Advances
- Published
- 2024-05-03
- Volume
- 10
- Issue
- 18
- Pages
- eadk3452-eadk3452
- ISSN
- 2375-2548 (ISSN-L)
- OpenAlex
- W4396541948
- Cited by
- 129
- References
- 74
- FWCI
- 49.19
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.