A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability
Open accessThe DOI matches its Crossref record; 206 of the 206 references it cites were found in a registry or an index.
Abstract
In the past few years, significant progress has been made on deep neural networks (DNNs) in achieving human-level performance on several long-standing tasks. With the broader deployment of DNNs on various applications, the concerns over their safety and trustworthiness have been raised in public, especially after the widely reported fatal incidents involving self-driving cars. Research to address these concerns is particularly active, with a significant number of papers released in the past few years. This survey paper conducts a review of the current research effort into making DNNs safe and trustworthy, by focusing on four aspects: verification, testing, adversarial attack and defence, and interpretability. In total, we survey 202 papers, most of which were published after 2017.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
(2016) · Cited by 11,679
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Clarke, E. M., Grümberg, O., & Long, D. (1996) · Cited by 6,813
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
Cousot, P., & Cousot, R. (1977) · Proceedings of the 4th ACM SIGACT-SIGPLAN symposium on Principles of programming languages - POPL '77 · doi:10.1145/512950.512973 · Cited by 6,279
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253
Proceedings of the 19th International Joint Conference on Artificial Intelligence
Pujol, J. M., Delgado, J., Ramon Sangüesa, A., & Flache, A. (2005) · International Joint Conference on Artificial Intelligence · Cited by 5,771
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Gradient [Simonyan et al., 2013].”
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
On Assessing ML Model Robustness: A Methodological Framework (Academic Track)
Awadid, A., & Robert, B. (2025) · OASIcs : OpenAccess Series in Informatics · doi:10.4230/oasics.saia.2024.1 · Cited by 4,702
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Proceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2015) · doi:10.1109/cvpr31182.2015 · Cited by 3,737
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05952 · Cited by 3,120
2017 IEEE International Conference on Computer Vision (ICCV)
(2017) · doi:10.1109/iccv37128.2017 · Cited by 3,056
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
Papernot, N., McDaniel, P., Wu, X., Jha, S., & Swami, A. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.41 · Cited by 2,982
2009 International Joint Conference on Artificial Intelligence
(2009) · doi:10.1109/jcai15434.2009 · Cited by 2,800
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
Universal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680
The information bottleneck method
Tishby, N., Pereira, F. C., & Bialek, W. (2000) · arXiv · doi:10.48550/arxiv.physics/0004057 · Cited by 2,626 · cited 2×
“Given two discrete real-valued variables X and Y with joint distribution p(x, y), the objective of the information bottleneck (IB) method [Tishby et al., 2000] is to figure out the stochastic encoder p(t|x), i.e., IB: min p(t|x) LIB = I(X;T )− βI(T ;Y ) (59a) subject to T ↔ X ↔ Y (59b) for β > 1,…”
MethodInternational Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Ensemble Adversarial Training: Attacks and Defenses
Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., & McDaniel, P. (2017) · arXiv · doi:10.48550/arxiv.1705.07204 · Cited by 2,176
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162
Learning regular sets from queries and counterexamples
Angluin, D. (1987) · Information and Computation · doi:10.1016/0890-5401(87)90052-6 · Cited by 2,114
Evasion attacks against machine learning at test time
Biggio, B., Corona, I., Maiorca, D., Nelson, B. A., Srndic, N., Laskov, P., Giacinto, G., & Roli, F. (2017) · CINECA IRIS Institutial Research Information System (University of Genoa) · Cited by 1,872
Conditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,833
Understanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
Xu, W., Evans, D., & Qi, Y. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23198 · Cited by 1,768
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks
Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_5 · Cited by 1,706
Counterexample-Guided Abstraction Refinement
Clarke, E., Grumberg, O., Jha, S., Lu, Y., & Veith, H. (2000) · Lecture Notes in Computer Science · doi:10.1007/10722167_15 · Cited by 1,702
International Conference on Learning Representations (ICLR)
Abbas, M. Z. A., Kivinen, J., & Raiko, T. (2016) · Cited by 1,694
(2002) · Choice Reviews Online · doi:10.5860/choice.40-1460 · Cited by 1,617
(2004) · IEEE Security & Privacy Magazine · doi:10.1109/msp.2004.9 · Cited by 1,518
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467 · cited 1×
“To sharpen the sensitive maps, SmoothGrad [Smilkov et al., 2017] randomly perturbs the input x with a small noise and averages the resulting explanation maps, i.e., ŵc(x) = 1 n ∑n i=1 wc(x+ gi), where gi ∼ N (0, σ2) is a Gaussian noise enabling random sampling around x.”
MethodUnderstanding Neural Networks Through Deep Visualization
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,358
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Software unit test coverage and adequacy
Zhu, H., Hall, P. A. V., & May, J. H. R. (1997) · ACM Computing Surveys · doi:10.1145/267580.267590 · Cited by 1,247
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security
(2018) · doi:10.1145/3243734 · Cited by 1,165
Zimmermann, U. (2003) · Public Integrity · doi:10.1080/15580989.2003.11770952 · Cited by 1,146
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097
Introduction to Software Testing
(2021) · Essentials of Software Testing · doi:10.1017/9781108974073.004 · Cited by 1,056
A unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029
Counterexample-guided abstraction refinement for symbolic model checking
Clarke, E., Grumberg, O., Jha, S., Lu, Y., & Veith, H. (2003) · Journal of the ACM · doi:10.1145/876638.876643 · Cited by 994
HotFlip: White-Box Adversarial Examples for Text Classification
Ebrahimi, J., Rao, A., Lowd, D., & Dou, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2006 · Cited by 922
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880
Safety Verification of Deep Neural Networks
Huang, X., Kwiatkowska, M., Wang, S., & Wu, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_1 · Cited by 818
Self-citationAI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation
Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., & Vechev, M. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00058 · Cited by 799
Thermometer Encoding: One Hot Way To Resist Adversarial Examples
Buckman, J., Roy, A., Raffel, C., & Goodfellow, I. (2018) · International Conference on Learning Representations · Cited by 702
Provable defenses against adversarial examples via the convex outer adversarial polytope
Wong, E., & Kolter, J. Z. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00851 · Cited by 634
Detecting Adversarial Samples from Artifacts
Feinman, R., Curtin, R. R., Shintre, S., & Gardner, A. B. (2017) · arXiv · doi:10.48550/arxiv.1703.00410 · Cited by 606
Differentiable Abstract Interpretation for Provably Robust Neural Networks
Mirman, M., Gehr, T., & Vechev, M. (2018) · International Conference on Machine Learning · Cited by 590
PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
Song, Y., Kim, T., Nowozin, S., Ermon, S., & Kushman, N. (2017) · arXiv · doi:10.48550/arxiv.1710.10766 · Cited by 562
Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks
Ehlers, R. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-68167-2_19 · Cited by 550
Evaluating the Robustness of Neural Networks: An Extreme Value Theory ApproachPreprint
Weng, T.-W., Zhang, H., Chen, P., Yi, J., Su, D., Gao, Y., Hsieh, C., & Daniel, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10578 · Cited by 544
Certified Defenses against Adversarial Examples
Raghunathan, A., Steinhardt, J., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09344 · Cited by 520
On the information bottleneck theory of deep learning*
Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. (2019) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ab3985 · Cited by 493
Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487
Certified Defenses for Data Poisoning Attacks
Steinhardt, J., Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1706.03691 · Cited by 479
Enhancing the reliability of out-of-distribution image detection in neural networks
Liang, S. (2025) · TIB Data Manager · doi:10.57702/uwcm8hhf · Cited by 476
Measuring Neural Net Robustness with Constraints
Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., & Criminisi, A. (2016) · neural information processing systems · doi:10.17863/cam.26484 · Cited by 472
Association for the Advancement of Artificial Intelligence
Crosby, M., & Petrick, R. P. A. (2014) · Cited by 461
Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
Ma, X., Li, B., Wang, Y., Erfani, S., Wijewickrema, S. N. R., Schoenebeck, G. R., Song, D. X., Houle, M. E., & Bailey, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02613 · Cited by 440
Guiding Deep Learning System Testing Using Surprise Adequacy
Kim, J., Feldt, R., & Yoo, S. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00108 · Cited by 430
Countering Adversarial Images using Input Transformations
Guo, C., Rana, M., Cissé, M., & van der Maaten, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00117 · Cited by 394
Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
Papernot, N., Faghri, F., Carlini, N., Goodfellow, I., Feinman, R., Kurakin, A., Xie, C., Sharma, Y., Brown, T., Roy, A., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.00768 · Cited by 381
Towards better understanding of gradient-based attribution methods for Deep Neural Networks
Ancona, M., Ceolini, E., Öztireli, C., & Groß, M. H. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000249929 · Cited by 368
DeepMutation: Mutation Testing of Deep Learning Systems
Ma, L., Zhang, F., Sun, J., Xue, M., Li, B., Juefei-Xu, F., Xie, C., Li, L., Liu, Y., Zhao, J., et al. (2018) · 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2018.00021 · Cited by 366
Structural Test Coverage Criteria for Deep Neural Networks
Sun, Y., Huang, X., Kroening, D., Sharp, J. J., Hill, M., & Ashmore, R. (2019) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3358233 · Cited by 363
Self-citationGenerative Adversarial Perturbations
Poursaeed, O., Katsman, I., Gao, B., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00465 · Cited by 352
Concolic testing for deep neural networks
Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., & Kroening, D. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238172 · Cited by 332
Self-citationLearning Confidence for Out-of-Distribution Detection in Neural Networks
DeVries, T., & Taylor, G. W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04865 · Cited by 326
2016 IEEE Symposium on Security and Privacy (SP)
(2016) · doi:10.1109/sp35280.2016 · Cited by 320
Efficient Neural Network Robustness Certification with General Activation Functions
Zhang, H., Weng, T.-W., Chen, P., Hsieh, C., & Daniel, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00866 · Cited by 316
Lang, J. (2018) · HAL · Cited by 312
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
Carlini, N., & Wagner, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07263 · Cited by 307
Certifying Some Distributional Robustness with Principled Adversarial Training
Sinha, A., Namkoong, H., Volpi, R., & Duchi, J. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10571 · Cited by 307
Deep Learning and Convolutional Neural Networks for Medical Image Computing
(2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-42999-1 · Cited by 289
Universal Adversarial Perturbations Against Semantic Image Segmentation
Metzen, J. H., Kumar, M. C., Brox, T., & Fischer, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.300 · Cited by 287
DLFuzz: differential fuzzing testing of deep learning systems
Guo, J., Jiang, Y., Zhao, Y., Chen, Q., & Sun, J. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264835 · Cited by 281
Spatially Transformed Adversarial Examples
Xiao, C., Zhu, J.-Y., Li, B., He, W., Liu, M., & Song, D. X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02612 · Cited by 275
Output Reachable Set Estimation and Verification for Multilayer Neural Networks
Xiang, W., Tran, H.-D., & Johnson, T. T. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2808470 · Cited by 274
General Data Protection Regulation
(2019) · Data Protection Law in the EU: Roles, Responsibilities and Liability · doi:10.1017/9781780688459.021 · Cited by 274
2019 IEEE Symposium on Security and Privacy (SP)
(2019) · doi:10.1109/sp37863.2019 · Cited by 267
Formal Security Analysis of Neural Networks using Symbolic Intervals
Wang, S., Pei, K., Whitehouse, J., Yang, J., & Jana, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.10829 · Cited by 265
Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
Hein, M., & Andriushchenko, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08475 · Cited by 263
An Abstraction-Refinement Approach to Verification of Artificial Neural Networks
Pulina, L., & Tacchella, A. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-14295-6_24 · Cited by 261
(2017) · arXiv (Cornell University) · Cited by 261
Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
Samangouei, P., Kabkab, M., & Chellappa, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06605 · Cited by 260
Reachability Analysis of Deep Neural Networks with Provable Guarantees
Ruan, W., Huang, X., & Kwiatkowska, M. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/368 · Cited by 246
Self-citationAn approach to reachability analysis for feed-forward ReLU neural networks
Lomuscio, A. R., & Maganti, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.07351 · Cited by 232
MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial ExamplesPreprint
Carlini, N., & Wagner, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08478 · Cited by 230
DeepRED – Rule Extraction from Deep Neural Networks
Zilke, J. R., Loza Mencía, E., & Janssen, F. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46307-0_29 · Cited by 228
Maximum Resilience of Artificial Neural Networks
Cheng, C.-H., Nührenberg, G., & Ruess, H. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-68167-2_18 · Cited by 227
Generalizable Data-Free Objective for Crafting Universal Adversarial Perturbations
Mopuri, K. R., Ganeshan, A., & Babu, R. V. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2861800 · Cited by 218
Feature-Guided Black-Box Safety Testing of Deep Neural Networks
Wicker, M., Huang, X., & Kwiatkowska, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-89960-2_22 · Cited by 212
Self-citationMODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W.-C., Zhang, X., & Grama, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236082 · Cited by 211
Proceedings of the 26th Symposium on Operating Systems Principles
(2017) · doi:10.1145/3132747 · Cited by 203
MagNet: a Two-Pronged Defense against Adversarial Examples
Meng, D., & Chen, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09064 · Cited by 194
Automated directed fairness testing
Udeshi, S., Arora, P., & Chattopadhyay, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238165 · Cited by 193
Verifying Properties of Binarized Deep Neural Networks
Narodytska, N., Kasiviswanathan, S., Ryzhyk, L., Sagiv, M., & Walsh, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12206 · Cited by 193
Extraction of rules from discrete-time recurrent neural networks
Omlin, C. W., & Giles, C. L. (1996) · Neural Networks · doi:10.1016/0893-6080(95)00086-0 · Cited by 187
Learning Universal Adversarial Perturbations with Generative Models
Hayes, J., & Danezis, G. (2018) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00015 · Cited by 187
Robust Deep Reinforcement Learning with Adversarial Attacks
Pattanaik, A., Tang, Z., Liu, S., Bommannan, G., & Chowdhary, G. (2018) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/rnct3583 · Cited by 187
NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles
Lu, J., Sibai, H., Fabry, E., & Forsyth, D. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.03501 · Cited by 186
Stochastic Activation Pruning for Robust Adversarial Defense
Dhillon, G. S., Azizzadenesheli, K., Lipton, Z. C., Bernstein, J., Kossaifi, J., Khanna, A., & Anandkumar, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01442 · Cited by 184
Mitigating Adversarial Effects Through Randomization
Xie, C., Wang, J., Zhang, Z., Ren, Z., & Yuille, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01991 · Cited by 181
Early Methods for Detecting Adversarial Images
Hendrycks, D., & Gimpel, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.00530 · Cited by 171
Sparse Adversarial Perturbations for Videos
Wei, X., Zhu, J., Yuan, S., & Su, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018973 · Cited by 170
Testing Deep Neural NetworksPreprint
Sun, Y., Huang, X., Kroening, D., Sharp, J., Hill, M., & Ashmore, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04792 · Cited by 167
Self-citationDeepStellar: model-based quantitative analysis of stateful deep learning systems
Du, X., Xie, X., Li, Y., Ma, L., Liu, Y., & Zhao, J. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338954 · Cited by 163
A Dual Approach to Scalable Verification of Deep Networks
Krishnamurthy, Dvijotham, Stanforth, R., Gowal, S., Mann, T., & Kohli, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.06567 · Cited by 162
Scalable Verified Training for Provably Robust Image Classification
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelović, R., Mann, T., & Kohli, P. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00494 · Cited by 161
Estimating Information Flow in Deep Neural NetworksPreprint
Goldfeld, Z., van den Berg, E., Greenewald, K. H., Melnyk, I., Nguyen, N. H., Kingsbury, B. E. D., & Polyanskiy, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05728 · Cited by 141
The Conference on Uncertainty in Artificial Intelligence (UAI 2017)
Belle, V. (2017) · Cited by 139
Universal Perturbation Attack Against Image Retrieval
Li, J., Ji, R., Liu, H., Hong, X., Gao, Y., & Tian, Q. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00500 · Cited by 139
Extracting Automata from Recurrent Neural Networks Using Queries and CounterexamplesPreprint
Weiss, G., Goldberg, Y., & Yahav, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09576 · Cited by 136
A game-based approximate verification of deep neural networks with provable guarantees
Wu, M., Wicker, M., Ruan, W., Huang, X., & Kwiatkowska, M. (2019) · Theoretical Computer Science · doi:10.1016/j.tcs.2019.05.046 · Cited by 127
Self-citationTensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
Odena, A., & Goodfellow, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10875 · Cited by 127
Towards Practical Verification of Machine Learning: The Case of Computer Vision SystemsPreprint
Pei, K., Zhu, L., Cao, Y., Yang, J., Vondrick, C., & Jana, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.01785 · Cited by 121
Universal Adversarial Perturbations for Speech Recognition Systems
Neekhara, P., Hussain, S., Pandey, P., Dubnov, S., McAuley, J., & Koushanfar, F. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1353 · Cited by 121
Rule extraction from neural networks via decision tree induction
Sato, M., & Tsukimoto, H. (2002) · IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222) · doi:10.1109/ijcnn.2001.938448 · Cited by 115
MuNN: Mutation Analysis of Neural Networks
Shen, W., Wan, J., & Chen, Z. (2018) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c.2018.00032 · Cited by 114
Su, T., Wu, K., Miao, W., Pu, G., He, J., Chen, Y., & Su, Z. (2017) · ACM Computing Surveys · doi:10.1145/3020266 · Cited by 109
Dimensionality Reduction as a Defense against Evasion Attacks on Machine Learning Classifiers.Preprint
Bhagoji, A. N., Cullina, D., & Mittal, P. (2017) · arXiv (Cornell University) · Cited by 106
Output Range Analysis for Deep Neural NetworksPreprint
Dutta, S., Jha, S., Sanakaranarayanan, S., & Tiwari, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.09130 · Cited by 104
Local Intrinsic Dimensionality I: An Extreme-Value-Theoretic Foundation for Similarity Applications
Houle, M. E. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-68474-1_5 · Cited by 97
Ruan, W., Wu, M., Sun, Y., Huang, X., Kroening, D., & Kwiatkowska, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/824 · Cited by 92
Self-citationTactics of Adversarial Attack on Deep Reinforcement Learning AgentsPreprint
Lin, Y.-C., Hong, Z.-W., Liao, Y.-H., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06748 · Cited by 91
Salay, R., & Czarnecki, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.01614 · Cited by 91
Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
Chu, L., Hu, X., Hu, J., Wang, L., & Pei, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220063 · Cited by 87
Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations
Tamagnini, P., Krause, J., Dasgupta, A., & Bertini, E. (2017) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/3077257.3077260 · Cited by 86
Runtime Monitoring Neuron Activation Patterns
Cheng, C., Nührenberg, G., & Yasuoka, H. (2019) · Design, Automation, and Test in Europe · doi:10.23919/date.2019.8714971 · Cited by 85
Combinatorial Testing for Deep Learning SystemsPreprint
Ma, L., Zhang, F., Xue, M., Li, B., Liu, Y., Zhao, J., & Wang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07723 · Cited by 81
Zhang, H., Zhang, P., & Hsieh, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015757 · Cited by 81
Inverting Convolutional Networks with Convolutional Networks.Preprint
Dosovitskiy, A., & Brox, T. (2015) · arXiv (Cornell University) · Cited by 75
DeepConcolic: Testing and Debugging Deep Neural Networks
Sun, Y., Huang, X., Kroening, D., Sharp, J. J., Hill, M., & Ashmore, R. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00051 · Cited by 74
Self-citation2018 IEEE Security and Privacy Workshops (SPW)
(2018) · doi:10.1109/spw45269.2018 · Cited by 73
Introduction to software testing
(2008) · Choice Reviews Online · doi:10.5860/choice.45-6834 · Cited by 65
Symbolic Execution for Deep Neural NetworksPreprint
Gopinath, D., Wang, K., Zhang, M., Păsăreanu, C. S., & Sarfraz Khurshid, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10439 · Cited by 64
DeepRoad: GAN-based Metamorphic Autonomous Driving System TestingPreprint
Zhang, M., Zhang, Y., Zhang, L., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.02295 · Cited by 62
Autonomous Systems – An Architectural Characterization
Sifakis, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-21485-2_21 · Cited by 62
Systematic Testing of Convolutional Neural Networks for Autonomous DrivingPreprint
Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03309 · Cited by 60
How AI detectives are cracking open the black box of deep learning
(2017) · AAAS Articles DO Group · doi:10.1126/science.aan7059 · Cited by 59
Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems
Cheng, C., Huang, C.-H., & Yasuoka, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01090-4_8 · Cited by 59
The Interpretation and Evaluation of Assurance Cases
Rushby, J. (2015) · Cited by 58
A Safety Framework for Critical Systems Utilising Deep Neural Networks
Zhao, X., Banks, A., Sharp, J. J., Robu, V., Flynn, D., Fisher, M., & Huang, X. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-54549-9_16 · Cited by 58
Self-citationSequential Attacks on Agents for Long-Term Adversarial GoalsPreprint
Tretschk, E., Oh, S. J., & Fritz, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.12487 · Cited by 55
Lower bounds on the robustness to adversarial perturbations
Peck, J., Roels, J., Goossens, B., & Saeys, Y. (2017) · Ghent University Academic Bibliography (Ghent University) · Cited by 51
Automated Test Generation to Detect Individual Discrimination in AI ModelsPreprint
Agarwal, A., Lohia, P., Nagar, S., Dey, K., & Saha, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03260 · Cited by 50
Ma, L., Juefei-Xu, F., Sun, J., Chen, C., Su, T., Zhang, F., Xue, M., Li, B., Li, L., Liu, Y., et al. (2018) · arXiv (Cornell University) · Cited by 46
Piecewise Linear Neural Networks verification: A comparative studyPreprint
Bunel, R., Turkaslan, I., Torr, P. H. S., Kohli, P., & Kumar, M. P. P. (2017) · arXiv (Cornell University) · Cited by 45
HyDiff: hybrid differential software analysis
Noller, Y., Păsăreanu, C. S., Böhme, M., Sun, Y., Nguyen, H. L., & Grunske, L. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380363 · Cited by 44
Self-citationTowards Dependability Metrics for Neural Networks
Cheng, C., Nührenberg, G., Huang, C.-H., Rueß, H., & Yasuoka, H. (2018) · ACM/IEEE International Conference on Formal Methods and Models for Co-Design (MEMOCODE) · doi:10.1109/memcod.2018.8556962 · Cited by 39
Formal Analysis of Deep Binarized Neural Networks
Narodytska, N. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/811 · Cited by 38
Coverage-Guided Fuzzing for Deep Neural Networks.Preprint
Xie, X., Ma, L., Juefei-Xu, F., Chen, H., Xue, M., Li, B., Liu, Y., Zhao, J., Yin, J., & See, S. C. W. (2018) · arXiv (Cornell University) · Cited by 33
Manifesting Bugs in Machine Learning Code: An Explorative Study with Mutation Testing
Cheng, D.-W., Cao, C., Xu, C., & Ma, X. (2018) · International Conference on Software Quality, Reliability and Security · doi:10.1109/qrs.2018.00044 · Cited by 32
New Theory Cracks Open the Black Box of Deep Learning
Wolchover, N. (2018) · The MIT Press eBooks · doi:10.7551/mitpress/11909.003.0037 · Cited by 27
Reasoning about Cognitive Trust in Stochastic Multiagent Systems
Huang, X., & Kwiatkowska, M. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11050 · Cited by 17
Self-citationAn entropy based approach for SSIM speed up
Bruni, V., & Vitulano, D. (2017) · Signal Processing · doi:10.1016/j.sigpro.2017.01.007 · Cited by 16
Ashmore, R., & Hill, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99229-7_33 · Cited by 15
Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics
USA, S. I. C. O. M. O. D. 2. C. I., Association for Computing Machinery-Digital Library, ACM Special Interest Group on Management of Data, & SIGs, A. (2017) · doi:10.1145/3077257 · Cited by 14
Cited by
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
Self-citationThe rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Self-citationGeneralized Out-of-Distribution Detection: A Survey
Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02117-4 · Cited by 884
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundInfluential
Metadata
- Authors
- Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Wu, M., & Yi, X.
- Venue
- Computer Science Review
- Publisher
- Elsevier BV
- Published
- 2020-08
- Volume
- 37
- Pages
- 100270
- ISSN
- 1574-0137 (ISSN-L)
- OpenAlex
- W3036286896
- Cited by
- 532
- References
- 206
- FWCI
- 26.23
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.