Machine Learning Testing: Survey, Landscapes and Horizons
Open accessThe DOI matches its Crossref record; 257 of the 257 references it cites were found in a registry or an index.
Abstract
This paper provides a comprehensive survey of techniques for testing machine learning systems; Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Verification and validation of the behavior of artificial intelligence-powered systems is still an open issue, despite the many recent results [241].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [241]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Optimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
(2017) · Cited by 11,625
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
The MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 6,627
Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.17 · Cited by 6,601
Kernel Methods for Pattern Analysis
Shawe-Taylor, J., & Cristianini, N. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809682 · Cited by 6,587
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0043-6 · Cited by 6,260
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 1×
“To ensure a high coverage of ML testing related papers, we therefore also performed snowballing [74] on each of the related papers found by keyword searching.”
MethodReading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,149
A survey of decision tree classifier methodology
Safavian, S. R., & Landgrebe, D. (1991) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.97458 · Cited by 3,808
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
A comparison of event models for naive bayes text classification
McCallum, A., & Nigam, K. (1998) · Cited by 3,289
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Symbolic execution and program testing
King, J. C. (1976) · Communications of the ACM · doi:10.1145/360248.360252 · Cited by 3,031
The EU General Data Protection Regulation (GDPR)
Voigt, P., & von dem Bussche, A. (2017) · doi:10.1007/978-3-319-57959-7 · Cited by 3,001
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.41 · Cited by 2,982
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Hawkins, D. M. (2004) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci0342472 · Cited by 2,522
Drebin: Effective and Explainable Detection of Android Malware in Your Pocket
Arp, D., Spreitzenbarth, M., Hübner, M., Gascon, H., & Rieck, K. (2014) · Proceedings 2014 Network and Distributed System Security Symposium · doi:10.14722/ndss.2014.23247 · Cited by 2,331
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
Self-citationReluplex: An Efficient SMT Solver for Verifying Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_5 · Cited by 1,706
Non-Functional Requirements in Software Engineering
Chung, L., Nixon, B. A., Yu, E., & Mylopoulos, J. (2000) · Springer eBooks · doi:10.1007/978-1-4615-5269-7 · Cited by 1,690
(2000) · Cited by 1,467
Search‐based software test data generation: a survey
McMinn, P. (2004) · Software Testing, Verification and Reliability · doi:10.1002/stvr.294 · Cited by 1,368
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
Software Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124
Foundations of Machine Learning
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018) · doi:10.7551/mitpress/10290.001.0001 · Cited by 1,122
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
Self-citationInterpretable Machine Learning
Molnar, C. (2020) · Cited by 1,104
Foundations of Machine Learning
(2012) · Boosting · doi:10.7551/mitpress/8291.003.0006 · Cited by 1,074
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
Carlini, N., & Wagner, D. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140444 · Cited by 966
code2vec: Learning distributed representations of code
Alon, U., Zilberstein, M., Levy, O., & Yahav, E. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3290353 · Cited by 962
Interpretable machine learning
Parliamentary Office of Science and Technology, & Christie, L. (2020) · doi:10.58248/pn633 · Cited by 952
Search-based software engineering
Harman, M., & Jones, B. F. (2001) · Information and Software Technology · doi:10.1016/s0950-5849(01)00189-6 · Cited by 914
Self-citationA data-driven approach to predict the success of bank telemarketing
Moro, S., Cortez, P., & Rita, P. (2014) · Decision Support Systems · doi:10.1016/j.dss.2014.03.001 · Cited by 868
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845
Concentration Inequalities and Model Selection
(2007) · Lecture Notes in Mathematics · doi:10.1007/978-3-540-48503-2 · Cited by 744
Calibrating Probability with Undersampling for Unbalanced Classification
Pozzolo, A. D., Caelen, O., Johnson, R. A., & Bontempi, G. (2015) · 2015 IEEE Symposium Series on Computational Intelligence · doi:10.1109/ssci.2015.33 · Cited by 667 · cited 8×
“Performance-based Data Bug Detection.”
BackgroundInfluentialOn the Complexity of Best Arm Identification in Multi-Armed Bandit Models
Kaufmann, E., Cappé, O., & Garivier, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.4443 · Cited by 661
Bickel, P. J., Hammel, E. A., & O'Connell, J. W. (1975) · Science · doi:10.1126/science.187.4175.398 · Cited by 659
Fairness Constraints: Mechanisms for Fair Classification
Zafar, M. B., Valera, I., Rodriguez, M. G., & Gummadi, K. P. (2015) · arXiv · doi:10.48550/arxiv.1507.05259 · Cited by 645
MuJava: an automated class mutation system
Ma, Y.-S., Offutt, J., & Kwon, Y. R. (2005) · Software Testing, Verification and Reliability · doi:10.1002/stvr.308 · Cited by 627
Zhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611
Introduction to Software Testing
Ammann, P., & Offutt, J. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316771273 · Cited by 607
DeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588
Self-citationData Validation for Machine Learning
Breck, E. J., Zinkevich, M., Polyzotis, N., Whang, S. E., & Roy, S. (2019) · Cited by 510
Unsupervised Image-to-Image Translation Networks
Liu, M.-Y., Breuel, T. M., & Kautz, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00848 · Cited by 477
(1998) · Cited by 469
Metamorphic Testing: A New Approach for Generating Next Test Cases
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 448
A systematic review of search-based testing for non-functional system properties
Afzal, W., Torkar, R., & Feldt, R. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.12.005 · Cited by 447
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Guiding Deep Learning System Testing Using Surprise Adequacy
Kim, J., Feldt, R., & Yoo, S. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00108 · Cited by 430
Fruit recognition from images using deep learning
Mureşan, H., & Oltean, M. (2018) · Acta Universitatis Sapientiae, Informatica · doi:10.2478/ausi-2018-0002 · Cited by 426
Fairness testing: testing software for discrimination
Galhotra, S., Brun, Y., & Meliou, A. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106277 · Cited by 405
A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search
Harman, M., & McMinn, P. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.71 · Cited by 400
Self-citationTesting and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G., Xu, B., & Chen, T. Y. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 389
Measuring the VC-Dimension of a Learning Machine
Vapnik, V., Levin, E., & Cun, Y. L. (1994) · Neural Computation · doi:10.1162/neco.1994.6.5.851 · Cited by 388
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388
Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
Papernot, N., Faghri, F., Carlini, N., Goodfellow, I., Feinman, R., Kurakin, A., Xie, C., Sharma, Y., Brown, T., Roy, A., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.00768 · Cited by 381
DeepMutation: Mutation Testing of Deep Learning Systems
Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., Xie, C., Li, L., Liu, Y., Zhao, J., et al. (2018) · 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2018.00021 · Cited by 366
Self-citationTFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350
Dwork, C. (2011) · Encyclopedia of Cryptography and Security · doi:10.1007/978-1-4419-5906-5_752 · Cited by 339
Concolic testing for deep neural networks
Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., & Kroening, D. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238172 · Cited by 332
Software testability: the new verification
Voas, J. M., & Miller, K. W. (1995) · IEEE Software · doi:10.1109/52.382180 · Cited by 327
Virtual to Real Reinforcement Learning for Autonomous Driving
Pan, X., You, Y., Wang, Z., & Lu, C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.11 · Cited by 326 · cited 1×
“Autopilot-TensorFlow [265] Recorded to test the NVIDIA Dave model 45,568 robustness”
MethodEffects of sample size in classifier design
Fukunaga, K., & Hayes, R. R. (1989) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.31448 · Cited by 320
CUTE: a concolic unit testing engine for C
Sen, K., Marinov, D., & Agha, G. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1095430.1081750 · Cited by 315
On Formalizing Fairness in Prediction with Machine LearningPreprint
Gajane, P., & Pechenizkiy, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03184 · Cited by 314
Evaluating Robustness of Neural Networks with Mixed Integer Programming
Tjeng, V., Xiao, K., & Tedrake, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.07356 · Cited by 312
The ML test score: A rubric for ML production readiness and technical debt reduction
Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8258038 · Cited by 310
An empirical study on TensorFlow program bugs
Zhang, Y., Chen, Y., Cheung, S.-C., Xiong, Y., & Zhang, L. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213866 · Cited by 304
DLFuzz: differential fuzzing testing of deep learning systems
Guo, J., Jiang, Y., Zhao, Y., Chen, Q., & Sun, J. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264835 · Cited by 281
The Case for Process Fairness in Learning: Feature Selection for Fair Decision Making
Grgić-Hlača, N., Zafar, M. B., Gummadi, K. P., & Weller, A. (2016) · MPG.PuRe (Max Planck Society) · Cited by 271
Scalable End-to-End Autonomous Vehicle Testing via Rare-event SimulationPreprint
O’Kelly, M., Sinha, A., Namkoong, H., Duchi, J. C., & Tedrake, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00145 · Cited by 267
ActiveClean: interactive data cleaning for statistical modeling
Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994514 · Cited by 264
Testing vision-based control systems using learnable evolutionary algorithms
Abdessalem, R. B., Nejati, S., Briand, L. C., & Stifter, T. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180160 · Cited by 262
Neural-machine-translation-based commit message generation: how far are we?
Liu, Z., Xia, X., Hassan, A. E., Lo, D., Xing, Z., & Wang, X. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238190 · Cited by 247
Reachability Analysis of Deep Neural Networks with Provable Guarantees
Ruan, W., Huang, X., & Kwiatkowska, M. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/368 · Cited by 246
Automating large-scale data quality verification
Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., & Grafberger, A. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3229867 · Cited by 240
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594334 · Cited by 218
Feature-Guided Black-Box Safety Testing of Deep Neural Networks
Wicker, M., Huang, X., & Kwiatkowska, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-89960-2_22 · Cited by 212
MODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W.-C., Zhang, X., & Grama, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236082 · Cited by 211
Self-citationData Management Challenges in Production Machine Learning
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2017) · Proceedings of the 2017 ACM International Conference on Management of Data · doi:10.1145/3035918.3054782 · Cited by 208
Tuncali, C. E., Fainekos, G., Ito, H., & Kapinski, J. (2018) · 2018 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2018.8500421 · Cited by 207
Data Poisoning Attacks against Autoregressive Models
Alfeld, S., Zhu, X., & Barford, P. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10237 · Cited by 205
Metamorphic testing of driverless cars
Zhou, Z. Q., & Sun, L. (2019) · Communications of the ACM · doi:10.1145/3241979 · Cited by 200
On testing machine learning programs
Braiek, H. B., & Khomh, F. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110542 · Cited by 197
Constructing Subtle Faults Using Higher Order Mutation Testing
Jia, Y., & Harman, M. (2008) · 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation · doi:10.1109/scam.2008.36 · Cited by 195
Self-citationTesting advanced driver assistance systems using multi-objective search and neural networks
Ben Abdessalem, R., Nejati, S., Briand, L. C., & Stifter, T. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970311 · Cited by 195
Automated directed fairness testing
Udeshi, S., Arora, P., & Chattopadhyay, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238165 · Cited by 193
Software and Hardware Testing Using Combinatorial Covering Suites
Hartman, A. (2005) · Operations Research/Computer Science Interfaces Series · doi:10.1007/0-387-25036-0_10 · Cited by 192
CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190
On Detecting Adversarial Perturbations
Metzen, J. H., Genewein, T., Fischer, V., & Bischoff, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04267 · Cited by 188
Dwarakanath, A., Ahuja, M., Sikand, S., Rao, R. M., Bose, R. P. J. C., Dubash, N., & Podder, S. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213858 · Cited by 183
Testing autonomous cars for feature interaction failures using many-objective search
Abdessalem, R. B., Panichella, A., Nejati, S., Briand, L. C., & Stifter, T. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238192 · Cited by 180
cleverhans v0.1: an adversarial machine learning library.Preprint
Goodfellow, I., Papernot, N., & McDaniel, P. D. (2016) · arXiv (Cornell University) · Cited by 177
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171
DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems
Ma, L., Juefei-Xu, F., Xue, M., Li, B., Li, L., Liu, Y., & Zhao, J. (2019) · 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8668044 · Cited by 162
Self-citationA multi-objective approach to search-based test data generation
Lakhotia, K., Harman, M., & McMinn, P. (2007) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1276958.1277175 · Cited by 158
Self-citationEasy over hard: a case study on deep learning
Fu, W., & Menzies, T. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106256 · Cited by 155
Saxena, N. A., Huang, K., DeFilippis, E., Radanovic, G., Parkes, D. C., & Liu, Y. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314248 · Cited by 155
Self-citationDeepBillboard: systematic physical-world testing of autonomous driving systems
Zhou, H., Li, W., Kong, Z., Guo, J., Zhang, Y., Yu, B., Zhang, L., & Liu, C. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380422 · Cited by 153
Automatic testing and improvement of machine translation
Sun, Z., Zhang, J. M., Harman, M., Papadakis, M., & Zhang, L. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380420 · Cited by 152
Self-citationAdversarial Sample Detection for Deep Neural Network through Model Mutation Testing
Wang, J., Dong, G., Sun, J., Wang, X., & Zhang, P. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00126 · Cited by 151
Verification for Machine Learning, Autonomy, and Neural Networks SurveyPreprint
Xiang, W., Musau, P., Wild, A. A., Lopez, D. M., Hamilton, N. P., Yang, X., Rosenfeld, J. A., & Johnson, T. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.01989 · Cited by 151
From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis
Harman, M., & O'Hearn, P. (2018) · 2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2018.00009 · Cited by 151
Self-citationProperties of Machine Learning Applications for Use in Metamorphic Testing
Murphy, C., Kaiser, G. E., & Hu, L. (2008) · doi:10.7916/d8xk8pfd · Cited by 151
Metamorphic Relations for Enhancing System Understanding and Use
Zhou, Z. Q., Sun, L., Chen, T. Y., & Towey, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876433 · Cited by 148
An Empirical Study of Bugs in Machine Learning Systems
Thung, F., Wang, S., Lo, D., & Jiang, L. (2012) · 2012 IEEE 23rd International Symposium on Software Reliability Engineering · doi:10.1109/issre.2012.22 · Cited by 147
Chan, H., Sahiner, B., Wagner, R. F., & Petrick, N. (1999) · Medical Physics · doi:10.1118/1.598805 · Cited by 146
GUI testing: pitfalls and process
Memon, A. M. (2002) · Computer · doi:10.1109/mc.2002.1023795 · Cited by 145
Failing to Learn: Autonomously Identifying Perception Failures for Self-Driving Cars
Ramanagopal, M. S., Anderson, C., Vasudevan, R., & Johnson‐Roberson, M. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2857402 · Cited by 144
Banerjee, S. S., Jha, S., Cyriac, J., Kalbarczyk, Z. T., & Iyer, R. K. (2018) · 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2018.00066 · Cited by 142
Preventing undesirable behavior of intelligent machines
Thomas, P. S., Castro da Silva, B., Barto, A. G., Giguere, S., Brun, Y., & Brunskill, E. (2019) · Science · doi:10.1126/science.aag3311 · Cited by 142
Test Selection for Deep Learning Systems
Ma, W., Papadakis, M., Tsakmalis, A., Cordy, M., & Traon, Y. L. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3417330 · Cited by 136
Finkelstein, A., Harman, M., Mansouri, S. A., Ren, J., & Zhang, Y. (2009) · Requirements Engineering · doi:10.1007/s00766-009-0075-y · Cited by 130
Self-citationStructural Coverage Criteria for Neural Networks Could Be Misleading
Li, Z., Ma, X., Xu, C., & Cao, C. (2019) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier.2019.00031 · Cited by 130
TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
Odena, A., & Goodfellow, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10875 · Cited by 127
On the complexity of best-arm identification in multi-armed bandit models
KaufmannEmilie, CappéOlivier, & GarivierAurélien (2016) · Journal of Machine Learning Research · doi:10.5555/2946645.2946646 · Cited by 124
Sahiner, B., Chan, H., Petrick, N., Wagner, R. F., & Hadjiiski, L. (2000) · Medical Physics · doi:10.1118/1.599017 · Cited by 123
The state-based testing of object-oriented programs
Turner, C. D., & Robson, D. J. (2002) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.1993.366932 · Cited by 123
ML-Based Fault Injection for Autonomous Vehicles: A Case for Bayesian Fault Injection
Jha, S., Banerjee, S., Tsai, T., Hari, S. K. S., Sullivan, M. B., Kalbarczyk, Z. T., Keckler, S. W., & Iyer, R. K. (2019) · 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2019.00025 · Cited by 118
Search-based inference of polynomial metamorphic relations
Zhang, J., Chen, J., Hao, D., Xiong, Y., Xie, B., Zhang, L., & Mei, H. (2014) · Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/2642937.2642994 · Cited by 115
Self-citationTest generation via Dynamic Symbolic Execution for mutation testing
Zhang, L., Xie, T., Zhang, L., Tillmann, N., de Halleux, J., & Mei, H. (2010) · 2010 IEEE International Conference on Software Maintenance · doi:10.1109/icsm.2010.5609672 · Cited by 115
An empirical evaluation of the MC/DC coverage criterion on the HETE-2 satellite software
Dupuy, A., & Leveson, N. G. (2002) · IEEE/AIAA Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc.2000.886883 · Cited by 114
Robustness of Neural Networks: A Probabilistic and Practical Approach
Mangal, R., Nori, A. V., & Orso, A. (2019) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier.2019.00032 · Cited by 107
Measuring Neural Net Robustness with Constraints
Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., & Criminisi, A. (2016) · arXiv (Cornell University) · doi:10.17863/cam.26484 · Cited by 107
Input Prioritization for Testing Neural Networks
Byun, T., Sharma, V., Vijayakumar, A., Rayadurgam, S., & Cofer, D. D. (2019) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest.2019.000-6 · Cited by 106
Application of Metamorphic Testing to Supervised Classifiers
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2009) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2009.26 · Cited by 105
DeepSafe: A Data-Driven Approach for Assessing Robustness of Neural Networks
Gopinath, D., Katz, G., Păsăreanu, C. S., & Barrett, C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01090-4_1 · Cited by 105
Detecting Violations of Differential Privacy
Ding, Z., Wang, Y., Wang, G., Zhang, D., & Kifer, D. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243818 · Cited by 103
Pseudo-oracles for non-testable programs
Davis, M. D., & Weyuker, E. J. (1981) · Proceedings of the ACM '81 conference on - ACM 81 · doi:10.1145/800175.809889 · Cited by 100
A systematic review of software robustness
Shahrokni, A., & FELDT, R. H. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.06.002 · Cited by 97
Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic FailuresPreprint
Uesato, J., Kumar, A., Szepesvári, C., Erez, T., Ruderman, A., Anderson, K., Krishmamurthy, Dvijotham, Heess, N., & Kohli, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01647 · Cited by 97
A Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudı́k, M., Langford, J. C., & Wallach, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02453 · Cited by 95
Themis: automatically testing software for discrimination
Angell, R., Johnson, B., Brun, Y., & Meliou, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264590 · Cited by 94
Zhang, J. M., Wang, Z., Zhang, L., Hao, D., Zang, L., Cheng, S., & Zhang, L. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931038 · Cited by 92
Self-citationSecurity Risks in Deep Learning Implementations
Xiao, Q., Li, K., Zhang, D., & Xu, W. (2018) · 2018 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00027 · Cited by 88
Boosting operational DNN testing efficiency through conditioning
Li, Z., Ma, X., Xu, C., Cao, C., Xu, J., & Lü, J. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338930 · Cited by 85
Unit Tests for Stochastic OptimizationPreprint
Schaul, T., Antonoglou, I., & Silver, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6055 · Cited by 85
Automatic system testing of programs without test oracles
Murphy, C., Shen, K., & Kaiser, G. (2009) · Proceedings of the eighteenth international symposium on Software testing and analysis · doi:10.1145/1572272.1572295 · Cited by 84
State of the art: Dynamic symbolic execution for automated test generation
Chen, T., Zhang, X., Guo, S., Li, H., & Wu, Y. (2012) · Future Generation Computer Systems · doi:10.1016/j.future.2012.02.006 · Cited by 83
“Fairness Analysis” in Requirements Assignments
Finkelstein, A., Harman, M., Mansouri, S. A., Ren, J., & Zhang, Y. (2008) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2008.61 · Cited by 82
Self-citationPALM: Machine Learning Explanations For Iterative Debugging
Krishnan, S., & Wu, E. (2017) · Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/3077257.3077271 · Cited by 81
The Rademacher Complexity of Co-Regularized Kernel Classes
Rosenberg, D. S., & Bartlett, P. L. (2007) · QUT ePrints (Queensland University of Technology) · Cited by 80
Causal testing: understanding defects' root causes
Johnson, B., Brun, Y., & Meliou, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380377 · Cited by 78
The Methodology of N-Version Programming
Kuna, P. R. (2019) · Asian Journal of Computer Science and Technology · doi:10.51983/ajcst-2019.8.s3.2112 · Cited by 77
Analyzing Compositionality-Sensitivity of NLI Models
Nie, Y., Wang, Y., & Bansal, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016867 · Cited by 76
Towards Improved Testing For Deep Learning
Sekhon, J., & Fleming, C. H. (2019) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier.2019.00030 · Cited by 74
ActiveClean: An Interactive Data Cleaning Framework For Modern Machine Learning
Krishnan, S., Franklin, M. J., Goldberg, K., Wang, J., & Wu, E. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2899409 · Cited by 72
An Empirical Study on Real Bugs for Machine Learning Programs
Sun, X., Zhou, T., Li, G., Hu, J., Yang, H. M., & Li, B. (2017) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2017.41 · Cited by 72
MuNN: Mutation Analysis of Neural Networks
Shen, W., Wan, J., & Chen, Z. (2018) · 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c.2018.00032 · Cited by 70
AlphaClean: Automatic Generation of Data Cleaning PipelinesPreprint
Krishnan, S., & Wu, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11827 · Cited by 70
A Study of Oracle Approximations in Testing Deep Learning Libraries
Nejadgholi, M., & Yang, J. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00078 · Cited by 68
A Survey of Software Quality for Machine Learning Applications
Masuda, S., Ono, K., Yasue, T., & Hosokawa, N. (2018) · 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00061 · Cited by 68
AVFI: Fault Injection for Autonomous Vehicles
Jha, S., Banerjee, S. S., Cyriac, J., Kalbarczyk, Z. T., & Iyer, R. K. (2018) · 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W) · doi:10.1109/dsn-w.2018.00027 · Cited by 67
Androutsopoulos, K., Clark, D., Dan, H., Hierons, R. M., & Harman, M. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568314 · Cited by 66
Self-citationMurphy, C., Shen, K., & Kaiser, G. (2009) · 2009 International Conference on Software Testing Verification and Validation · doi:10.1109/icst.2009.19 · Cited by 65
Validating a Deep Learning Framework by Metamorphic Testing
Ding, J., Kang, X., & Hu, X. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.2 · Cited by 65
MISTIQUE: A System to Store and Query Model Intermediates for Model Diagnosis
Vartak, M., da Trindade, J. M. F., Madden, S. R., & Zaharia, M. (2018) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3183713.3196934 · Cited by 64
Symbolic Execution for Deep Neural NetworksPreprint
Gopinath, D., Wang, K., Zhang, M., Păsăreanu, C. S., & Sarfraz Khurshid, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10439 · Cited by 64
On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems
Nushi, B., Kamar, E., Horvitz, E., & Kossmann, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10633 · Cited by 63
Wegener, J., & Bühler, O. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24855-2_160 · Cited by 61
Systematic Testing of Convolutional Neural Networks for Autonomous DrivingPreprint
Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03309 · Cited by 60
Metamorphic Testing for Machine Translations: MT4MT
Sun, L., & Zhou, Z. Q. (2018) · Australasian Software Engineering Conference (ASWEC) · doi:10.1109/aswec.2018.00021 · Cited by 59
An Approach to Software Testing of Machine Learning Applications
Murphy, C., Kaiser, G. E., & Arias, M. (2007) · doi:10.7916/d8r49znr · Cited by 58
BoostClean: Automated Error Detection and Repair for Machine Learning
Krishnan, S., Franklin, M. J. M., Goldberg, K. Y., & Wu, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01299 · Cited by 54
DeepCruiser: Automated Guided Testing for Stateful Deep Learning SystemsPreprint
Du, X., Xie, X., Li, Y., Ma, L., Zhao, J., & Liu, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05339 · Cited by 52
Self-citationDetecting Adversarial Samples for Deep Neural Networks through Mutation TestingPreprint
Wang, J., Sun, J., Zhang, P., & Wang, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.05010 · Cited by 51
Albarghouthi, A., & Vinitsky, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287588 · Cited by 51
Automated Test Generation to Detect Individual Discrimination in AI ModelsPreprint
Agarwal, A., Lohia, P., Nagar, S., Dey, K., & Saha, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03260 · Cited by 50
Research works printed but not linked: 16
- Neter, 1996, Applied Linear Statistical Models, 4
- 10.1038/naturei4539 · doi:10.1038/naturei4539
- 2010, IEEE Standard Classification for Software Anomalies
- Kirk, 2014, Thoughtful Machine Learning: A Test-Driven Approach
- 1990, IEEE Standard Glossary of Software Engineering Terminology
- Biran, Explanation and justification in machine learning: A survey
- Goodfellow, Generative adversarial nets, 2672
- Kang, Model assertions for debugging machine learning
- 10.1145/3238147.3238172 · DOI · Paper page
- 10.1109/issre.2018.00021 · DOI · Paper page
- Japkowicz, Why question machine learning evaluation methods, 6
- Bichsel, DD-Finder: Finding differential privacy violations by sampling and optimization, 508
- McClure, 2017, TensorFlow Machine Learning Cookbook
- 10.1109/icse-companion.2019.00131 · DOI
- Belitskaya, Handwritten letters
- 2007, Report to the congress on credit scoring and its effects on the availability and affordability of credit
Cited by
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundSoftware Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap
Wagner, J., Triantafyllopoulos, A., Wierstorf, H., Schmitt, M., Burkhardt, F., Eyben, F., & Schuller, B. W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3263585 · Cited by 395
Metadata
- Authors
- Zhang, J. M., Harman, M., Ma, L., & Liu, Y.
- Affiliations
- University College London (GB)
- Meta (United Kingdom) (GB)
- Kyushu University (JP)
- Nanyang Technological University (SG)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2022-01-01
- Volume
- 48
- Issue
- 1
- Pages
- 1-36
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W3007157104
- Cited by
- 897
- References
- 256
- Citations by year
- 2026: 92 · 2025: 115 · 2024: 153 · 2023: 150 · 2022: 141
- FWCI
- 240.63
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- ERC (741278)
- JSPS KAKENHI (19K24348, 19H04086)
- Qdai-jump Research Program (01277)
- Nvidia
- National Research Foundation Singapore (NRF2018NCR-NCR005-0001)
- National Satellite of Excellence in Trustworthy Software System (NRF2018NCR-NSOE003-0001)
- NTU (NGF-2019-06-024)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- machine learning testing, correctness testing, robustness testing, fairness testing, test case generation, test evaluation, data testing, autonomous driving, machine translation, research trends
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.