Preventing undesirable behavior of intelligent machines
The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Abstract
Making well-behaved algorithms: Machine learning algorithms are being used in an ever-increasing number of applications, and many of these applications affect quality of life. Yet such algorithms often exhibit undesirable behavior, from various types of bias to causing financial loss or delaying medical diagnoses. In standard machine learning approaches, the burden of avoiding this harmful behavior is placed on the user of the algorithm, who most often is not a computer scientist. Thomas et al. introduce a general framework for algorithm design in which this burden is shifted from the user to the designer of the algorithm. The researchers illustrate the benefits of their approach using examples in gender fairness and diabetes management. Science , this issue p. 999
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Machine Learning Applications in Cancer Prognosis and Prediction
Kourou, K., Exarchos, T. P., Exarchos, K. P., Karamouzis, M. V., & Fotiadis, D. I. (2015) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2014.11.005 · Cited by 3,338 · cited 1×
“Rather, the papers by Grabczewski and Duch [20] and subsequent machine learning researchers provided tools that researchers with the appropriate domain knowledge and medical expertise could apply properly to cancer treatment [22, 150, 151, 152].”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 2×
“• Equal opportunity requires that false negative rates are equal among groups [119, 116].”
BackgroundFalk, A., & Fischbacher, U. (2006) · Games and Economic Behavior · doi:10.1016/j.geb.2005.03.001 · Cited by 2,416
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 3×
“have adverse effects on the involved groups [115, 116, 25].”
BackgroundCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_5 · Cited by 1,706
Temporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Nonlinear model predictive control of glucose concentration in subjects with type 1 diabetes
Hovorka, R., Canonico, V., Chassin, L. J., Haueter, U., Massi-Benedetti, M., Federici, M. O., Pieber, T. R., Schaller, H. C., Schaupp, L., Vering, T., et al. (2004) · Physiological Measurement · doi:10.1088/0967-3334/25/4/010 · Cited by 1,274
Robust classification for imprecise environments
Provost, F., & Fawcett, T. (2001) · Machine Learning · doi:10.1023/a:1007601015854 · Cited by 1,250 · cited 1×
“Robust optimization algorithms have been proposed as a means for ensuring that machine learning algorithms, ranging from supervised learning algorithms [83] to reinforcement learning algorithms [84], are in some way “safe” to use.”
BackgroundKusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · cited 3×
“• Conditional use accuracy equality requires that precision (the probability that the model is correct when it predicts the label, +1) is the same for all groups, and that the negative predictive value (the probability that the model is correct when it predicts the label, −1) is the same for all groups [120].”
BackgroundInfluentialInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
The UVA/PADOVA Type 1 Diabetes Simulator: New Features
Man, C. D., Micheletto, F., Lv, D., Breton, M., Kovatchev, B., & Cobelli, C. (2014) · Journal of Diabetes Science and Technology · doi:10.1177/1932296813514502 · Cited by 791
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 2×
“, age correlates with savings, race correlates with name, and, in the United States, race correlates with zip code, models trained without access to a set of attributes can still effectively act unfairly with respect to those attributes [113, 114].”
BackgroundConcentration Inequalities and Model Selection
(2007) · Lecture Notes in Mathematics · doi:10.1007/978-3-540-48503-2 · Cited by 744
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728
Increased mortality of patients with diabetes reporting severe hypoglycemia
McCoy, R. G., Van Houten, H. K., Ziegenfuss, J. Y., Shah, N. D., Wermers, R. A., & Smith, S. A. (2012) · Diabetes Care · doi:10.2337/dc11-2054 · Cited by 539
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · cited 1×
“[128] found that “image search results for occupations slightly exaggerate gender stereotypes and portray the minority gender for an occupational [sic] less professionally.”
BackgroundBuilding Classifiers with Independency Constraints
Calders, T., Kamiran, F., & Pechenizkiy, M. (2009) · 2009 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2009.83 · Cited by 468 · cited 1×
“Balancing the training data and introducing noise by flipping the labels of some feature vectors and repeating some training data can improve demographic parity for näıve Bayes classifiers as well [125].”
BackgroundSuh, S. W., Gum, E. T., Hamby, A. M., Chan, P. H., & Swanson, R. A. (2007) · Journal of Clinical Investigation · doi:10.1172/jci30077 · Cited by 419 · cited 1×
“Hypoglycemia can cause symptoms ranging from palpitations, sweating, and hunger, to altered mental status, confusion, coma, and even death [33, 34, 69].”
BackgroundClassifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409 · cited 1×
“Kamiran and Calders [95] provide classification algorithms that aim to ensure that when, e.”
BackgroundFairness testing: testing software for discrimination
Galhotra, S., Brun, Y., & Meliou, A. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106277 · Cited by 405 · cited 2×
“In other words, if two contexts differ only in T , and are otherwise identical, this definition requires classifiers to predict the same outcome for both individuals [108, 121].”
BackgroundMethodSelf-citationThe reusable holdout: Preserving validity in adaptive data analysis
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · Science · doi:10.1126/science.aaa9375 · Cited by 307
Discrimination Aware Decision Tree Learning
Kamiran, F., Calders, T., & Pechenizkiy, M. (2010) · 2010 IEEE International Conference on Data Mining · doi:10.1109/icdm.2010.50 · Cited by 303
Auditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280
k-NN as an implementation of situation testing for discrimination discovery and prevention
Luong, B. T., Ruggieri, S., & Turini, F. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020488 · Cited by 263
Outcome measures for artificial pancreas clinical trials: a consensus report
Maahs, D. M., Buckingham, B. A., Castle, J. R., Cinar, A., Damiano, E. R., Dassau, E., DeVries, J. H., Doyle, F. J., Griffen, S. C., Haidar, A., et al. (2016) · Diabetes Care · doi:10.2337/dc15-2716 · Cited by 211
Time-uniform, nonparametric, nonasymptotic confidence sequences
Howard, S. R., Ramdas, A., McAuliffe, J., & Sekhon, J. (2021) · The Annals of Statistics · doi:10.1214/20-aos1991 · Cited by 190
Schmidt, S., & Nørgaard, K. (2014) · Journal of Diabetes Science and Technology · doi:10.1177/1932296814532906 · Cited by 170
Handling Conditional Discrimination
liobaite, I., Kamiran, F., & Calders, T. (2011) · 2011 IEEE 11th International Conference on Data Mining · doi:10.1109/icdm.2011.72 · Cited by 125
Mini-review: Impact of recurrent hypoglycemia on cognitive and brain function
McNay, E. C., & Cotero, V. E. (2010) · Physiology & Behavior · doi:10.1016/j.physbeh.2010.01.004 · Cited by 111
Allowing a wildfire to burn: estimating the effect on future fire suppression costs
Houtman, R. M., Montgomery, C. A., Gagnon, A. R., Calkin, D. E., Dietterich, T. G., McGregor, S., & Crowley, M. (2013) · International Journal of Wildland Fire · doi:10.1071/wf12157 · Cited by 85
Reddy, M., Pesl, P., Xenou, M., Toumazou, C., Johnston, D., Georgiou, P., Herrero, P., & Oliver, N. (2016) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2015.0413 · Cited by 70
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling
Ho, Y., & Wookey, S. (2020) · IEEE Access · doi:10.1109/access.2019.2962617 · Cited by 835
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646
Artificial intelligence: Who is responsible for the diagnosis?
Neri, E., Coppola, F., Miele, V., Bibbolino, C., & Grassi, R. (2020) · La radiologia medica · doi:10.1007/s11547-020-01135-9 · Cited by 240
Artificial Intelligence in Radiology—Ethical Considerations
Brady, A. P., & Neri, E. (2020) · Diagnostics · doi:10.3390/diagnostics10040231 · Cited by 150
Metadata
- Authors
- Thomas, P. S., Castro da Silva, B., Barto, A. G., Giguere, S., Brun, Y., & Brunskill, E.
- Venue
- Science
- Published
- 2019-11-22
- Volume
- 366
- Issue
- 6468
- Pages
- 999-1004
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2991598122
- Cited by
- 142
- References
- 41
- FWCI
- 32.75
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.