Troubling Trends in Machine Learning Scholarship: Some ML papers suffer from flaws that could mislead the public and stymie future research.
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Flawed scholarship threatens to mislead the public and stymie future research by compromising ML’s intellectual foundations. Indeed, many of these problems have recurred cyclically throughout the history of AI and, more broadly, in scientific research. In 1976, Drew McDermott chastised the AI community for abandoning self-discipline, warning prophetically that "if we can’t criticize ourselves, someone else will save us the trouble." The current strength of machine learning owes to a large body of rigorous research to date, both theoretical and empirical. By promoting clear scientific thinking and communication, our community can sustain the trust and investment it currently enjoys.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Regarding the anthropomorphic language describing AI, its usage has been recognised as potentially detrimental in other AI sub-fields, including machine learning [87], brain-inspired AI [88] or art-generating AI [73].”
From Deanthropomorphising NLP: Can a language model be conscious? (2024) · cites this paper as [87]
The cited source says what the claim attributes to it.
In the citing paper
“Failing to identify the sources of empirical gains is a longstanding issue in ML and related fields [29, 16].”
From AI Agents That Matter (2024) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“Failing to do so or misrepresenting these comparisons can lead to follow-up work that rests on false assumptions (Lipton & Steinhardt, 2019;Leech et al., 2024).”
From Open Problems in Mechanistic Interpretability (2025) · cites this paper as Lipton & Steinhardt, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“has limited relevance to the proposed learning algorithm, but might confer a sense of theoretical depth to readers. The ubiquity of this issue is evidenced by the paper introducing the Adam optimizer [35]. In the course of introducing an optimizer with strong empirical performance, it also offers a theorem regarding co…”
BackgroundFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“In contrast, claims of human-level performance in [29] are better-qualified to refer to the ImageNet classification task (rather than object recognition more broadly).”
BackgroundVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“ive adversarial networks. This term first took root in deep learning in [79], which does address deconvolution, but was later over-generalized to refer to any neural architectures using upconvolutions [78, 50]. Such overloading of terminology can create lasting confusion. New machine learning papers referring to deconv…”
BackgroundDermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 2×
“Take, for example, the “dermatologist-level classification of skin cancer” reported in [21].”
BackgroundBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
(2010) · Cited by 8,261
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“instance, probing the behavior of neural networks led to identifying their susceptibility to adversarial perturbations [74].”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“insky describes mental processes such as consciousness, thinking, attention, emotion, and feeling that may not share “a single cause or origin”. Many terms in ML fall into this category. For example, [46] notes that interpretability holds no universally agreed-upon meaning, and often references disjoint methods and de…”
BackgroundSelf-citationLearning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422 · cited 1×
“ver clarifies, and some work suggests that this explanation of batch 2 normalization may be off the mark [65]. Nevertheless, the speculative explanation given in [33] has been repeated as fact, e.g. in [60], which states, “It is well-known that a deep neural network is very hard to optimize due to the internal-covariate…”
BackgroundReturn of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Platt, J. R. (1964) · Science · doi:10.1126/science.146.3642.347 · Cited by 2,907
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · cited 1×
“Notable examples of this include a demonstration that neural networks trained by stochastic gradient descent can fit randomly-assigned labels [81].”
BackgroundHuman-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817 · cited 1×
“t learning [30] and generative adversarial networks [51]. See [68] for more discussion of lapses in empirical rigor and resulting consequences. In contrast, many papers perform good ablation analyses [41, 45, 77, 82], and even retrospective attempts to isolate the source of gains can lead to new discoveries [10, 65].…”
BackgroundWhy Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010) · Journal of Machine Learning Research · Cited by 2,631
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 2×
“ines with suggestive definitions via anthropomorphic naming. Language understanding and reading comprehension, once grand challenges of AI, now refer to making accurate predictions on specific datasets [31]. 3.4.3 Suitcase Words Finally, we discuss the overuse of suitcase words in ML papers. Coined by Minsky in the 2007…”
BackgroundInfluentialPractical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259 · cited 1×
“roduction, a designated “Motivation” section clearly quarantines this discussion. This practice avoids confusing readers while allowing authors to express informal ideas. In another positive example, [3] presents practical guidelines for training neural networks. Here, the authors carefully convey uncertainty. Instead…”
BackgroundWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · cited 2×
“The current strength of machine learning owes to a large body of rigorous research to date, both theoretical [22, 7, 19] and empirical [34, 25, 5].”
BackgroundOn the Convergence of Adam and Beyond
Reddi, S. J., Kale, S., & Kumar, S. (2019) · arXiv · doi:10.48550/arxiv.1904.09237 · Cited by 1,771
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
Zeiler, M. D., Krishnan, D., Taylor, G. W., & Fergus, R. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539957 · Cited by 1,666 · cited 1×
“arning literature to refer to transpose convolutions (also called up-convolutions) as commonly found in auto-encoders and generative adversarial networks. This term first took root in deep learning in [79], which does address deconvolution, but was later over-generalized to refer to any neural architectures using upcon…”
BackgroundHow Does Batch Normalization Help Optimization?
Santurkar, S., Tsipras, D., Ilyas, A., & Madry, A. (2018) · arXiv · doi:10.48550/arxiv.1805.11604 · Cited by 1,092 · cited 2×
“normalization may be off the mark [65].”
BackgroundNeural Motifs: Scene Graph Parsing with Global Context
Zellers, R., Yatskar, M., Thomson, S., & Choi, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00611 · Cited by 1,052
Counterfactual reasoning and learning systems: the example of computational advertising
Bottou, L., Peters, J., Quiñonero-Candela, J., Charles, D. X., Chickering, D. M., Portugaly, E., Ray, D., Simard, P., & Snelson, E. (2013) · Journal of Machine Learning Research · Cited by 863
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827 · cited 2×
“We note that the broader interesting point in [18] that minima tend to have lower loss than saddle points is more clearly stated and empirically tested.”
BackgroundThe Loss Surfaces of Multilayer Networks
Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., & LeCun, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.0233 · Cited by 733
Return of the Devil in the Details: Delving Deep into Convolutional NetsPreprint
Chatfield, K., Simonyan, K., Vedaldi, A., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.3531 · Cited by 663 · cited 1×
“rigor and resulting consequences. In contrast, many papers perform good ablation analyses [41, 45, 77, 82], and even retrospective attempts to isolate the source of gains can lead to new discoveries [10, 65]. Furthermore, ablation is neither necessary nor sufficient for understanding a method, and can even be impractica…”
BackgroundThe emotion machine: commonsense thinking, artificial intelligence, and the future of the human mind
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5688 · Cited by 635 · cited 2×
“What sort of papers best serve their readers? We can enumerate desirable characteristics: these papers should (i) provide intuition to aid the reader’s understanding, but clearly distinguish it from stronger conclusions supported by evidence; (ii) describe empirical investigations that consider and rule out alternativ…”
BackgroundLearning Deconvolution Network for Semantic SegmentationPreprint
Noh, H., Hong, S., & Han, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.04366 · Cited by 613 · cited 1×
“ver clarifies, and some work suggests that this explanation of batch 2 normalization may be off the mark [65]. Nevertheless, the speculative explanation given in [33] has been repeated as fact, e.g. in [60], which states, “It is well-known that a deep neural network is very hard to optimize due to the internal-covariate…”
BackgroundA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 1×
“[11] studies a task designed for reading comprehension of news passages and finds that 73% of the questions can be answered by looking at a single sentence, while only 2% require looking at multiple sentences (the remaining 25% of examples were either ambiguous or contained coreference errors).”
BackgroundCertified Defenses for Data Poisoning Attacks
Steinhardt, J., Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1706.03691 · Cited by 479
Self-citationAlgorithmic Bias in Autonomous Systems
Danks, D., & London, A. J. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/654 · Cited by 472 · cited 1×
“of new suitcase words. In the fairness literature, where legal, philosophical, and statistical language are often overloaded, terms like bias become suitcase words that must be subsequently unpacked [17]. In common speech and as aspirational terms, suitcase words can serve a useful purpose. Perhaps the suitcase word r…”
BackgroundA Possibility for Implementing Curiosity and Boredom in Model-Building Neural Controllers
Schmidhuber, J. (1991) · From Animals to Animats · doi:10.7551/mitpress/3115.003.0030 · Cited by 421 · cited 1×
“n connotations without the need to argue for them. This often manifests in anthropomorphic characterizations of tasks (reading comprehension [31] and music composition [59]) and techniques (curiosity [66] and fear [48]). A number of papers name components of proposed models in a manner suggestive of human cognition, e…”
BackgroundNeural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Liang, C., Berant, J., Le, Q., Forbus, K. D., & Lao, N. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1003 · Cited by 381 · cited 1×
“t learning [30] and generative adversarial networks [51]. See [68] for more discussion of lapses in empirical rigor and resulting consequences. In contrast, many papers perform good ablation analyses [41, 45, 77, 82], and even retrospective attempts to isolate the source of gains can lead to new discoveries [10, 65].…”
BackgroundCovariate Shift by Kernel Mean Matching
Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borgwardt, K., & Schölkopf, B. (2008) · Dataset Shift in Machine Learning · doi:10.7551/mitpress/9780262170055.003.0008 · Cited by 378 · cited 1×
“a change in the distribution of model inputs. In fact, covariate shift refers to a specific type of shift where although the input distribution p(x) might change, the labeling function p(y|x) does not [27]. Moreover, due to the influence of [33], Google Scholar lists batch normalization as the first reference on searches…”
BackgroundQualitatively characterizing neural network optimization problems
Goodfellow, I. J., Vinyals, O., & Saxe, A. M. (2014) · arXiv · doi:10.48550/arxiv.1412.6544 · Cited by 341 · cited 1×
“n fit randomly-assignedlabels [81]. This paper questions the ability of learning-theoretic notions of model complexity to explain why neural networks can generalize to unseen data. In another example, [26] explored the loss surfaces of deep networks, revealing that straight-line paths in parameter space between initial…”
BackgroundArtificial intelligence meets natural stupidity
McDermott, D. (1976) · ACM SIGART Bulletin · doi:10.1145/1045339.1045340 · Cited by 323 · cited 2×
“In 1976, Drew Mc-Dermott [53] chastised the AI community for abandoning self-discipline, warning prophetically that “if we can’t criticize ourselves, someone else will save us the trouble”.”
BackgroundPractical recommendations for gradient-based training of deep architecturesPreprint
Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.5533 · Cited by 266 · cited 1×
“roduction, a designated “Motivation” section clearly quarantines this discussion. This practice avoids confusing readers while allowing authors to express informal ideas. In another positive example, [3] presents practical guidelines for training neural networks. Here, the authors carefully convey uncertainty. Instead…”
BackgroundImprovements that don't add up: ad-hoc retrieval results since 1998
Armstrong, T. G., Moffat, A., Webber, W., & Zobel, J. (2009) · Proceedings of the 18th ACM conference on Information and knowledge management · doi:10.1145/1645953.1646031 · Cited by 234 · cited 2×
“y for abandoning self-discipline, warning prophetically that “if we can’t criticize ourselves, someone else will save us the trouble”. Similar discussions recurred throughout the 80s, 90s, and aughts [13, 38, 2]. In other fields such as psychology, poor experimental standards have eroded trust in the discipline’s autho…”
BackgroundA MultiPath Network for Object Detection
Zagoruyko, S., Lerer, A., Lin, T.-Y., Pinheiro, P., Gross, S., Chintala, S., & Dollar, P. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.15 · Cited by 228
Mathiness in the Theory of Economic Growth
Romer, P. M. (2015) · American Economic Review · doi:10.1257/aer.p20151066 · Cited by 201 · cited 2×
“What sort of papers best serve their readers? We can enumerate desirable characteristics: these papers should (i) provide intuition to aid the reader’s understanding, but clearly distinguish it from stronger conclusions supported by evidence; (ii) describe empirical investigations that consider and rule out alternativ…”
BackgroundMOZER, M. C. (1994) · Connection Science · doi:10.1080/09540099408915726 · Cited by 187 · cited 1×
“This often manifests in anthropomorphic characterizations of tasks (reading comprehension [31] and music composition [59]) and techniques (curiosity [66] and fear [48]).”
BackgroundConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases
Stock, P., & Cisse, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_31 · Cited by 158
Learning in Implicit Generative Models
Mohamed, S., & Lakshminarayanan, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03483 · Cited by 157 · cited 1×
“Seeking to resolve the confusion and provide historical context, [58] distinguishes between prescribed and implicit generative models.”
BackgroundStatistics of Critical Points of Gaussian Fields on Large-Dimensional Spaces
Bray, A. J., & Dean, D. S. (2007) · Physical Review Letters · doi:10.1103/physrevlett.98.150201 · Cited by 156 · cited 1×
“As one piece of evidence, the work cites a statistical physics paper [9] on Gaussian random fields and states that in high dimensions “all local minima [of Gaussian random fields] are likely to have an error very close to that of the global minimum” (a similar statement appears in the related work of [12]).”
BackgroundBengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1709.08568 · Cited by 155 · cited 1×
““thought vectors” [36] and the “consciousness prior” [4].”
N-Rays: An Episode in the History and Psychology of Science
Nye, M. J. (1980) · Historical Studies in the Physical Sciences · doi:10.2307/27757473 · Cited by 124
A Thorough Examination of the CNN/Daily Mail Reading Comprehension TaskPreprint
Chen, D., Bolton, J., & Manning, C. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02858 · Cited by 122 · cited 1×
“[11] studies a task designed for reading comprehension of news passages and finds that 73% of the questions can be answered by looking at a single sentence, while only 2% require looking at multiple sentences (the remaining 25% of examples were either ambiguous or contained coreference errors).”
BackgroundWinner's Curse? On Pace, Progress, and Empirical Rigor.
Sculley, D., Snoek, J., Wiltschko, A. B., & Rahimi, A. R. O. (2018) · International Conference on Learning Representations · Cited by 117
How evaluation guides AI research
Coehn, P., & Howe, A. E. (1988) · AI Magazine · doi:10.1609/aimag.v9i4.952 · Cited by 86
Are All Languages Equally Hard to Language-Model?
Cotterell, R., Mielke, S. J., Eisner, J., & Roark, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2085 · Cited by 80 · cited 1×
“Understanding can also come from robustness checks (as in [15], which discovers that existing language models handle inflectional morphology poorly) as well as qualitative error analysis [40].”
BackgroundDoes mitigating ML's impact disparity require treatment disparity?
Lipton, Z. C., Chouldechova, A., & McAuley, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07076 · Cited by 75
Self-citationAre GANs Created Equal? A Large-Scale Study
Lučić, M., Kurach, K., Michalski, M., Gelly, S., & Bousquet, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10337 · Cited by 73 · cited 1×
“Similar evaluation issues have been observed for deep reinforcement learning [30] and generative adversarial networks [51].”
BackgroundGenerative Concatenative Nets Jointly Learn to Write and Classify ReviewsPreprint
Lipton, Z. C., Vikram, S., & McAuley, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.03683 · Cited by 51 · cited 1×
“Bending the term further, some discriminative models are now referred to as generative models on account of producing structured outputs [76], a mistake that we (ZL) make in [47].”
BackgroundSelf-citationStock, P., & Cissé, M. (2017) · arXiv.org · doi:10.48550/arxiv.1711.11443 · Cited by 48 · cited 1×
“misclassifying “Asians dressed in red” as ping-pong balls [73].”
BackgroundCombating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
Lipton, Z. C., Azizzadenesheli, K., Kumar, A., Li, L., Gao, J., & Deng, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01211 · Cited by 47 · cited 1×
“without the need to argue for them. This often manifests in anthropomorphic characterizations of tasks (reading comprehension [31] and music composition [59]) and techniques (curiosity [66] and fear [48]). A number of papers name components of proposed models in a manner suggestive of human cognition, e.g. “thought ve…”
BackgroundSelf-citationDeep Depth From FocusPreprint
Hazırbaş, C., Soyer, S. G., Staab, M. C., Leal-Taixé, L., & Cremers, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01085 · Cited by 18 · cited 1×
“upconvolution, or (iii) attempting to resolve the confusion, as in [28], which awkwardly refers to “upconvolution (deconvolution)”.”
BackgroundLearning Fast-Mixing Models for Structured PredictionPreprint
Steinhardt, J., & Liang, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.06668 · Cited by 9 · cited 1×
“culprits, inserted into papers to lend authoritativeness to empirical results, even when the theorem’s conclusions do not actually support the main claims of the paper. We (JS) are guilty of this in [70], where a discussion of “staged strong Doeblin chains” has limited relevance to the proposed learning algorithm, but…”
Self-citation
Research works printed but not linked: 15
- Bengio Y., Neural Networks: Tricks of the Trade, ed. G. Montavon, G. B. Orr, KR Müller, 437
- Choromanska A., 2015, Proceedings of the 18th International Conference on Artificial Intelligence and Statistics.
- Goodfellow I. J., 2015, Proceedings of the International Conference on Learning Representations.
- Henderson P., 2018, Proceedings of the 32nd Association for the Advancement of Artificial Intelligence Conference.
- Ioffe S., 2015, Proceedings of the 32nd International Conference on Machine Learning 37; http://proceedings.mlr.press/v37/ioffe15
- Kingma D. P., 2015, Proceedings of the 3rd International Conference on Learning Representations.
- Lipton Z. C., 2016, International Conference on Machine Learning Workshop on Human Interpretability.
- Lucic M., 2017, Proceedings of the 32nd Conference on Neural Information Processing Systems. arXiv Preprint arXiv:1711
- Melis G., 2018, Proceedings of the International Conference on Learning Representations.
- Reddi S. J., 2018, Proceedings of the International Conference on Learning Representations.
- Sculley D., 2018, Proceedings of the 6th International Conference on Learning Representations, Workshop Track.
- Steinhardt J., 2015, Proceedings of the 32nd International Conference on Machine Learning 37
- Steinhardt J., 2015, Proceedings of the 32nd International Conference on Machine Learning 37
- Szegedy C., 2013, International Conference on Learning Representations. arXiv Preprint arXiv:1312
- Zhang C., 2017, Proceedings of the International Conference on Learning Representations.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463
Dau, H. A., Bagnall, A., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., & Keogh, E. (2019) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2019.1911747 · Cited by 1,128
“While this is a fictional report, note that all the numbers presented in the table and figure are true values, based on reproducible experiments that we performed [16].”
BackgroundMethodCan You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
“Contributions In the spirit of calls for more rigorous understanding of existing methods (Lipton and Steinhardt, 2018; Sculley et al., 2018; Rahimi and Recht, 2017), this paper provides a benchmark for evaluating uncertainty that focuses not only on calibration in the i.”
BackgroundMethodAn overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2018) · NORA - Norwegian Open Research Archives · Cited by 1,012
Metadata
- Authors
- Lipton, Z. C., & Steinhardt, J.
- Affiliations
- Carnegie Mellon University (US)
- University of California, Berkeley (US)
- Venue
- Queue
- Published
- 2019-02
- Volume
- 17
- Issue
- 1
- Pages
- 45-77
- ISSN
- 1542-7730 (ISSN-L)
- 1542-7749
- OpenAlex
- W2881747041
- Cited by
- 231
- References
- 54
- Citations by year
- 2026: 30 · 2025: 14 · 2024: 17 · 2023: 20 · 2022: 21 · 2021: 35 · 2020: 46 · 2019: 34
- FWCI
- 159.63
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Queue (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Instituto de Ciencias del Mar y Limnología, Universidad Nacional Autónoma de México
- Topics
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- machine learning research, research integrity, scientific rigor, reproducibility, research critique, history of artificial intelligence, science communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.