Systematic Poisoning Attacks on and Defenses for Machine Learning in Healthcare
The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Machine learning is being used in a wide range of application domains to discover patterns in large datasets. Increasingly, the results of machine learning drive critical decisions in applications related to healthcare and biomedicine. Such health-related applications are often sensitive, and thus, any security breach would be catastrophic. Naturally, the integrity of the results computed by machine learning is of great importance. Recent research has shown that some machine-learning algorithms can be compromised by augmenting their training datasets with malicious data, leading to a new class of attacks called poisoning attacks. Hindrance of a diagnosis may have life-threatening consequences and could cause distrust. On the other hand, not only may a false diagnosis prompt users to distrust the machine-learning algorithm and even abandon the entire system but also such a false positive classification may cause patient distress. In this paper, we present a systematic, algorithm-independent approach for mounting poisoning attacks across a wide range of machine-learning algorithms and healthcare datasets. The proposed attack procedure generates input data, which, when added to the training set, can either cause the results of machine learning to have targeted errors (e.g., increase the likelihood of classification into a specific class), or simply introduce arbitrary errors (incorrect classification). These attacks may be applied to both fixed and evolving datasets. They can be applied even when only statistics of the training dataset are available or, in some cases, even without access to the training dataset, although at a lower efficacy. We establish the effectiveness of the proposed attacks using a suite of six machine-learning algorithms and five healthcare datasets. Finally, we present countermeasures against the proposed generic attacks that are based on tracking and detecting deviations in various accuracy metrics, and benchmark their effectiveness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884 · cited 1×
“The first is a tree-based algorithm, i.e., BFTree (best-first decision tree) [27], [28].”
MethodPrivacy-preserving data mining
Agrawal, R., & Srikant, R. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335438 · Cited by 2,961
Instance-based learning algorithms
Aha, D. W., Kibler, D., & Albert, M. K. (1991) · Machine Learning · doi:10.1007/bf00153759 · Cited by 2,163
Scaling Up the Accuracy of Naive-Bayes Classifiers : a Decision-Tree Hybrid
Kohavi, R. (1996) · Knowledge Discovery and Data Mining · Cited by 2,141
Assessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088 · cited 1×
“2) Kappa statistic [47]: This statistic measures relative improvement over random predictors, i.e., it is a measure of how good the classifier is compared to a random classifier (the higher the Kappa statistic, the better the classifica- tion).”
BackgroundProbability Estimates for Multi-class Classification by Pairwise Coupling
Wu, T., Lin, C., & Weng, R. C. (2004) · Journal of Machine Learning Research · doi:10.5555/1005332.1016791 · Cited by 1,468
Poisoning Attacks against Support Vector Machines
Biggio, B., Nelson, B., & Laskov, P. (2012) · arXiv · doi:10.48550/arxiv.1206.6389 · Cited by 1,331
Improvements to Platt's SMO Algorithm for SVM Classifier Design
Keerthi, S. S., Shevade, S. K., Bhattacharyya, C., & Murthy, K. R. K. (2001) · Neural Computation · doi:10.1162/089976601300014493 · Cited by 1,318
Landwehr, N., Hall, M., & Frank, E. (2005) · Machine Learning · doi:10.1007/s10994-005-0466-3 · Cited by 1,214
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194
Probability Estimates for Multi-class Classification by Pairwise Coupling
Wu, T., Lin, C., Weng, R. C., & Singer, Y. (2003) · Cited by 989
Privacy-preserving distributed mining of association rules on horizontally partitioned data
Clifton, C., Kantarcioglu, M., Clifton, C., & Member, S. (2010) · Cited by 908
Dalvi, N., Domingos, P., Mausam, Sanghai, S., & Verma, D. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014066 · Cited by 887
Can machine learning be secure?
Barreno, M., Nelson, B., Sears, R., Joseph, A. D., & Tygar, J. D. (2006) · Proceedings of the 2006 ACM Symposium on Information, computer and communications security · doi:10.1145/1128817.1128824 · Cited by 836
The security of machine learning
Barreno, M., Nelson, B., Joseph, A. D., & Tygar, J. D. (2010) · Machine Learning · doi:10.1007/s10994-010-5188-5 · Cited by 817
Advances in Kernel Methods: Support Vector Learning
(1998) · The MIT Press eBooks · doi:10.7551/mitpress/1130.001.0001 · Cited by 761
Data Mining techniques for the detection of fraudulent financial statements
KIRKOS, E., SPATHIS, C., & MANOLOPOULOS, Y. (2007) · Expert Systems with Applications · doi:10.1016/j.eswa.2006.02.016 · Cited by 722
Predicting the Political Alignment of Twitter Users
Conover, M. D., Goncalves, B., Ratkiewicz, J., Flammini, A., & Menczer, F. (2011) · 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing · doi:10.1109/passat/socialcom.2011.34 · Cited by 588
A comparison of linear genetic programming and neural networks in medical data mining
Brameier, M., & Banzhaf, W. (2001) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.910462 · Cited by 505
Exploiting machine learning to subvert your spam filter
Nelson, B. A., Barreno, M., Chi, F. J., Joseph, A. D., Rubinstein, B. I. P., Saini, U., Sutton, C. A., Tygar, J. D., & Xia, K. (2008) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 481
Large-Scale Automatic Classification of Phishing Pages
Whittaker, C., Ryner, B., & Nazif, M. (2010) · Network and Distributed System Security Symposium · Cited by 468
A study of the effect of different types of noise on the precision of supervised learning techniques
Nettleton, D. F., Orriols-Puig, A., & Fornells, A. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-010-9156-z · Cited by 446
Security Evaluation of Pattern Classifiers under Attack
Biggio, B., Fumera, G., & Roli, F. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.57 · Cited by 411
Adaptive Intrusion Detection: A Data Mining Approach
Lee, W., Stolfo, S. J., & Mok, K. W. (2000) · Artificial Intelligence Review · doi:10.1023/a:1006624031083 · Cited by 347
Best-first Decision Tree Learning
Shi, H. (2007) · Research Commons (University of Waikato) · Cited by 337
ANTIDOTE: understanding and defending against poisoning of anomaly detectors
Rubinstein, B. I. P., Nelson, B., Huang, L., Joseph, A. D., Lau, S.-H., Rao, S., Taft, N., & Tygar, J. D. (2009) · Proceedings of the 9th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1644893.1644895 · Cited by 331 · cited 1×
“The attacks presented in [23] and [24] work in the feature space.”
BackgroundNightmare at test time: robust learning by feature deletion
Globerson, A., & Roweis, S. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143889 · Cited by 309
Links between perceptrons, MLPs and SVMs
Collobert, R., & Bengio, S. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015415 · Cited by 302
Static prediction games for adversarial learning problems
Brückner, M., Kanzow, C., & Scheffer, T. (2012) · publish.UP (University of Potsdam) · doi:10.5555/2503308.2503326 · Cited by 240
Online Anomaly Detection under Adversarial Impact
Kloft, M., & Laskov, P. (2010) · International Conference on Artificial Intelligence and Statistics · Cited by 177
Bagging Classifiers for Fighting Poisoning Attacks in Adversarial Classification Tasks
Biggio, B., Corona, I., Fumera, G., Giacinto, G., & Roli, F. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21557-5_37 · Cited by 146
Enabling Multilevel Trust in Privacy Preserving Data Mining
Li, Y., Chen, M., Li, Q., & Zhang, W. (2011) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.124 · Cited by 114
Detecting adversarial advertisements in the wild
Sculley, D., Otey, M. E., Pohl, M., Spitznagel, B., Hainsworth, J., & Zhou, Y. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020455 · Cited by 93
Hiding Intrusions: From the Abnormal to the Normal and Beyond
Tan, K. M. C., McHugh, J., & Killourhy, K. S. (2002) · Lecture notes in computer science · doi:10.1007/3-540-36415-3_1 · Cited by 83
Stealthy poisoning attacks on PCA-based anomaly detectors
Rubinstein, B. I. P., Nelson, B., Huang, L., Joseph, A. D., Lau, S.-H., Rao, S., Taft, N., & Tygar, J. D. (2009) · ACM SIGMETRICS Performance Evaluation Review · doi:10.1145/1639562.1639592 · Cited by 62
Cesa‐Bianchi, N., Shalev‐Shwartz, S., & Shamir, O. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2164053 · Cited by 49
Cited by
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., & Zhao, B. Y. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00031 · Cited by 1,368
Data Poisoning Attacks Against Federated Learning Systems
Tolpegin, V., Truex, S., Gursoy, M. E., & Liu, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58951-6_24 · Cited by 927
A Comprehensive Study of Security of Internet-of-Things
Mosenia, A., & Jha, N. K. (2017) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2016.2606384 · Cited by 806
Self-citationManipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00057 · Cited by 861
Metadata
- Authors
- Mozaffari-Kermani, M., Sur-Kolay, S., Raghunathan, A., & Jha, N. K.
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2015-11
- Volume
- 19
- Issue
- 6
- Pages
- 1893-1905
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W1968411139
- Cited by
- 383
- References
- 36
- FWCI
- 4.62
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.