The DOI matches its Crossref record; 255 of the 256 references it cites were found in a registry or an index.
Machine Learning Testing: Survey, Landscapes and Horizons
Open accessAbstract
This paper provides a comprehensive survey of techniques for testing machine learning systems; Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Verification and validation of the behavior of artificial intelligence-powered systems is still an open issue, despite the many recent results [241].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [241]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Optimization by Simulated Annealing
Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983) · Science · doi:10.1126/science.220.4598.671 · Cited by 45,134
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Research works printed but not linked: 17
- Neter, 1996, Applied Linear Statistical Models, 4
- 10.1038/naturei4539 · doi:10.1038/naturei4539
- 2010, IEEE Standard Classification for Software Anomalies
- Kirk, 2014, Thoughtful Machine Learning: A Test-Driven Approach
- 1990, IEEE Standard Glossary of Software Engineering Terminology
- Corbett-Davies, 2018, The measure and mismeasure of fairness: A critical review of fair machine learning
- Biran, Explanation and justification in machine learning: A survey
- Goodfellow, Generative adversarial nets, 2672
- Kang, Model assertions for debugging machine learning
- 10.1145/3238147.3238172 · DOI · Paper page
- 10.1109/issre.2018.00021 · DOI · Paper page
- Japkowicz, Why question machine learning evaluation methods, 6
- Bichsel, DD-Finder: Finding differential privacy violations by sampling and optimization, 508
- McClure, 2017, TensorFlow Machine Learning Cookbook
- 10.1109/icse-companion.2019.00131 · DOI
- Belitskaya, Handwritten letters
- 2007, Report to the congress on credit scoring and its effects on the availability and affordability of credit
Cited by
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundSoftware Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap
Wagner, J., Triantafyllopoulos, A., Wierstorf, H., Schmitt, M., Burkhardt, F., Eyben, F., & Schuller, B. W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3263585 · Cited by 395
Metadata
- Authors
- Zhang, J. M., Harman, M., Ma, L., & Liu, Y.
- Affiliations
- University College London (GB)
- Meta (United Kingdom) (GB)
- Kyushu University (JP)
- Nanyang Technological University (SG)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2022-01-01
- Volume
- 48
- Issue
- 1
- Pages
- 1-36
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W3007157104
- Cited by
- 897
- References
- 256
- Citations by year
- 2026: 92 · 2025: 115 · 2024: 153 · 2023: 150 · 2022: 141
- FWCI
- 240.63
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- ERC (741278)
- JSPS KAKENHI (19K24348, 19H04086)
- Qdai-jump Research Program (01277)
- Nvidia
- National Research Foundation Singapore (NRF2018NCR-NCR005-0001)
- National Satellite of Excellence in Trustworthy Software System (NRF2018NCR-NSOE003-0001)
- NTU (NGF-2019-06-024)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- machine learning testing, correctness testing, robustness testing, fairness testing, test case generation, test evaluation, data testing, autonomous driving, machine translation, research trends
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Machine Learning Testing: Survey, Landscapes and Horizons. https://app.cochat.ai/papers/10.1109/tse.2019.2962027
- The paper
- Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022). Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2019.2962027
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.