Machine Learning that Matters
Open accessAbstract
Much of current machine learning (ML) research has lost its connection to problems of import to the larger world of science and society. From this perspective, there exist glaring limitations in the data sets we investigate, the metrics we employ for evaluation, and the degree to which results are communicated back to their originating domains. What changes are needed to how we conduct research to increase the impact that ML has? We present six Impact Challenges to explicitly focus the field?s energy and attention, and we discuss existing obstacles that must be addressed. We aim to inspire ongoing discussion and focus on ML that matters.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The choice of the objective function can misguide the inference task.
“The choice of the objective function can misguide the inference task. Risks are also linked to the selection of the objective functions used to express various inference or prediction tasks (Wagstaff, 2012)”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Wagstaff, 2012
Abstract metrics enable comparisons across domains but offer limited insights about the actual improvements for each problem domain.
“Abstract metrics enable comparisons across domains, but offer limited insights about the actual improvements for each problem domain.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Wagstaff, 2012
There is a lack of rigorous longitudinal evaluations of the contribution of social data to improve domain-relevant metrics.
“there is a lack of rigorous longitudinal evaluations of the contribution of social data to improve domain-relevant metrics, such as “dollars saved, lives preserved, time conserved, effort reduced, quality of living increased” (Wagstaff, 2012).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Wagstaff, 2012
If a benchmark allows shortcuts, accuracy on the benchmark does not translate to the real world.
“If a benchmark allows shortcuts, accuracy on the benchmark does not translate to the real world [32, 33, 52].”
From AI Agents That Matter (2024) · cites this paper as [52]
Cites
The changing science of machine learning
Langley, P. (2011) · Machine Learning · doi:10.1007/s10994-011-5242-y · Cited by 174 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · see citing works
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533083 · Cited by 317 · see citing works
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256 · see citing works
On the safety of machine learning: Cyber-physical systems, decision sciences, and data products
Varshney, K. R., & Alemzadeh, H. (2017) · Big Data · doi:10.1089/big.2016.0051 · Cited by 211 · see citing works
“that can be judged harmful or not [6].”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763 · see citing works
Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach
Bayram, F., & Ahmed, B. S. (2025) · ACM Computing Surveys · doi:10.1145/3708497 · Cited by 48 · see citing works
GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models
Xu, Z., Qiu, Y., Sun, L., Miao, F., Wu, F., Li, X., Wang, X., Lu, H., Zhang, Z., Hu, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06718 · Cited by 3 · see citing works
“Prior meta-science measurement studies have demonstrated the value of large-scale empirical analysis in exposing systemic issues, from reproducibility crises [16], [17] to evaluation pitfalls in machine learning [40], [41] and citation manipulation practices [42].”
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13 · see citing works
- Authors
- Wagstaff, K.
- Affiliations
- Jet Propulsion Laboratory (US)
- Corresponding author
- Wagstaff, Kiri
- ORCID
- Wagstaff, Kiri
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2012
- Pages
- 1851-1856
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1206.4656
- OpenAlex
- W1754293002
- Cited by
- 148
- References
- 1
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 12 · 2023: 13 · 2022: 18 · 2021: 16 · 2020: 8 · 2019: 15 · 2018: 6 · 2017: 11 · 2016: 9 · 2015: 10 · 2014: 10 · 2013: 13 · 2012: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Funders
- Jet Propulsion Laboratory
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Social Sciences › Decision Sciences › Information Systems and Management › Research, Science, and Academia
- Keywords
- research impact, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
