The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
Deep Learning-based Anomaly Detection in Cyber-physical Systems: Progress and Opportunities
Abstract
Anomaly detection is crucial to ensure the security of cyber-physical systems (CPS). However, due to the increasing complexity of CPSs and more sophisticated attacks, conventional anomaly detection methods, which face the growing volume of data and need domain-specific knowledge, cannot be directly applied to address these challenges. To this end, deep learning-based anomaly detection (DLAD) methods have been proposed. In this article, we review state-of-the-art DLAD methods in CPSs. We propose a taxonomy in terms of the type of anomalies, strategies, implementation, and evaluation metrics to understand the essential properties of current methods. Further, we utilize this taxonomy to identify and highlight new characteristics and designs in each CPS domain. Also, we discuss the limitations and open problems of these methods. Moreover, to give users insights into choosing proper DLAD methods in practice, we experimentally explore the characteristics of typical neural models, the workflow of DLAD methods, and the running performance of DL models. Finally, we discuss the deficiencies of DL approaches, our findings, and possible directions to improve DLAD methods and motivate future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
LOF: identifying density-based local outliers
Breunig, M. M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335388 · Cited by 5,207
LOF: identifying density-based local outliers
Breunig, M. M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000) · Proceedings of the 2000 ACM SIGMOD international conference on Management of data · doi:10.1145/342009.335388 · Cited by 4,149
False data injection attacks against state estimation in electric power grids
Liu, Y., Ning, P., & Reiter, M. K. (2011) · ACM Transactions on Information and System Security · doi:10.1145/1952982.1952995 · Cited by 2,428
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network
Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., & Pei, D. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330672 · Cited by 1,804
Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding
Hundman, K., Constantinou, V., Laporte, C., Colwell, I., & Soderstrom, T. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219845 · Cited by 1,771
Deep Learning for IoT Big Data and Streaming Analytics: A Survey
Mohammadi, M., Al-Fuqaha, A., Sorour, S., & Guizani, M. (2018) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2844341 · Cited by 1,471
Deep Learning for Anomaly Detection: A Survey
Chalapathy, R., & Chawla, S. (2019) · arXiv · doi:10.48550/arxiv.1901.03407 · Cited by 1,240
One-class svms for document classification
Manevitz, L. M., & Yousef, M. (2002) · doi:10.5555/944790.944808 · Cited by 1,207
MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks
Li, D., Chen, D., Jin, B., Shi, L., Goh, J., & Ng, S.-K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-30490-4_56 · Cited by 1,185
Cyber security in the Smart Grid: Survey and challenges
Wang, W., & Lu, Z. (2013) · Computer Networks · doi:10.1016/j.comnet.2012.12.017 · Cited by 1,123
A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder
Park, D., Hoshi, Y., & Kemp, C. C. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2801475 · Cited by 1,115
Forecasting Time Series With Complex Seasonal Patterns Using Exponential Smoothing
De Livera, A. M., Hyndman, R. J., & Snyder, R. D. (2011) · Journal of the American Statistical Association · doi:10.1198/jasa.2011.tm09771 · Cited by 1,049
In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning
Wang, X., Han, Y., Wang, C., Zhao, Q., Chen, X., & Chen, M. (2019) · IEEE Network · doi:10.1109/mnet.2019.1800286 · Cited by 1,030 · cited 1×
“Another solution is known as edge computing [103], which is used”
BackgroundMagNet: A Two-Pronged Defense against Adversarial Examples
Meng, D., & Chen, H. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134057 · Cited by 1,006
A reliability perspective of the smart grid
Moslehi, K., & Kumar, R. (2010) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2010.2046346 · Cited by 932
Malicious Data Attacks on the Smart Grid
Kosut, O., Jia, L., Thomas, R. J., & Tong, L. (2011) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2011.2163807 · Cited by 784
A survey of intrusion detection techniques for cyber-physical systems
Mitchell, R., & Chen, I.-R. (2014) · ACM Computing Surveys · doi:10.1145/2542049 · Cited by 768
A review of cyber security risk assessment methods for SCADA systems
Cherdantseva, Y., Burnap, P., Blyth, A., Eden, P., Jones, K., Soulsby, H., & Stoddart, K. (2016) · Computers & Security · doi:10.1016/j.cose.2015.09.009 · Cited by 606 · cited 1×
“Studies in terms of network security of SCADA systems are summarized with a focus on risk assessment techniques [20].”
BackgroundAnomaly Detection in Streams with Extreme Value Theory
Siffer, A., Fouque, P.-A., Termier, A., & Largouet, C. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098144 · Cited by 553
A survey of physics-based attack detection in cyber-physical systems
Giraldo, J., Urbina, D., Cardenas, A., Valente, J., Faisal, M., Ruths, J., Tippenhauer, N. O., Sandberg, H., & Candell, R. (2019) · ACM Computing Surveys · doi:10.1145/3203245 · Cited by 535
Multi-head CNN–RNN for multi-time series anomaly detection: An industrial case study
Canizo, M., Triguero, I., Conde, A., & Onieva, E. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.07.034 · Cited by 420
Anomaly Detection in Cyber Physical Systems Using Recurrent Neural Networks
Goh, J., Adepu, S., Tan, M., & Lee, Z. S. (2017) · 2017 IEEE 18th International Symposium on High Assurance Systems Engineering (HASE) · doi:10.1109/hase.2017.36 · Cited by 413
Anomaly Detection in Automobile Control Network Data with Long Short-Term Memory Networks
Taylor, A., Leblanc, S., & Japkowicz, N. (2016) · 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2016.20 · Cited by 400
False Data Injection Attacks in Electricity Markets
Xie, L., Mo, Y., & Sinopoli, B. (2010) · 2010 First IEEE International Conference on Smart Grid Communications · doi:10.1109/smartgrid.2010.5622048 · Cited by 384
Detecting Cyber Attacks in Industrial Control Systems Using Convolutional Neural Networks
Kravchik, M., & Shabtai, A. (2018) · Proceedings of the 2018 Workshop on Cyber-Physical Systems Security and PrivaCy · doi:10.1145/3264888.3264896 · Cited by 355
Anomaly Detection for a Water Treatment System Using Unsupervised Machine Learning
Inoue, J., Yamagata, Y., Chen, Y., Poskitt, C. M., & Sun, J. (2017) · 2017 IEEE International Conference on Data Mining Workshops (ICDMW) · doi:10.1109/icdmw.2017.149 · Cited by 344
Real-Time Sensor Anomaly Detection and Identification in Automated Vehicles
van Wyk, F., Wang, Y., Khojandi, A., & Masoud, N. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2906038 · Cited by 344
Fan, C., Xiao, F., Zhao, Y., & Wang, J. (2018) · Applied Energy · doi:10.1016/j.apenergy.2017.12.005 · Cited by 286
Multi-level Anomaly Detection in Industrial Control Systems via Package Signatures and LSTM Networks
Feng, C., Li, T., & Chana, D. (2017) · 2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2017.34 · Cited by 231
Anomaly detection in aircraft data using Recurrent Neural Networks (RNN)
Nanduri, A., & Sherry, L. (2016) · 2016 Integrated Communications Navigation and Surveillance (ICNS) · doi:10.1109/icnsurv.2016.7486356 · Cited by 230
Normalization as a Preprocessing Engine for Data Mining and the Approach of Preference Matrix
Al Shalabi, L., & Shaaban, Z. (2006) · 2006 International Conference on Dependability of Computer Systems · doi:10.1109/depcos-relcomex.2006.38 · Cited by 225
State of the art of cyber-physical systems security: An automatic control perspective
Zacchia Lun, Y., D’Innocenzo, A., Smarra, F., Malavolta, I., & Di Benedetto, M. D. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2018.12.006 · Cited by 211
Anomaly Detection Using Autoencoders in High Performance Computing Systems
Borghesi, A., Bartolini, A., Lombardi, M., Milano, M., & Benini, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019428 · Cited by 210
A deep learning approach for anomaly detection based on SAE and LSTM in mechanical equipment
Li, Z., Li, J., Wang, Y., & Wang, K. (2019) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-019-03557-w · Cited by 169
Assessing and augmenting SCADA cyber security: A survey of techniques
Nazir, S., Patel, S., & Patel, D. (2017) · Computers & Security · doi:10.1016/j.cose.2017.06.010 · Cited by 167
Bad data injection in smart grid: attack and defense mechanisms
Huang, Y., Esmalifalak, M., Nguyen, H., Zheng, R., Han, Z., Li, H., & Song, L. (2013) · IEEE Communications Magazine · doi:10.1109/mcom.2013.6400435 · Cited by 165
Packet-data anomaly detection in PMU-based state estimator using convolutional neural network
Basumallik, S., Ma, R., & Eftekharnejad, S. (2018) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2018.11.013 · Cited by 152
Dynamic Detection of False Data Injection Attack in Smart Grid using Deep Learning
Niu, X., Li, J., Sun, J., & Tomsovic, K. L. (2019) · IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT) · doi:10.1109/isgt.2019.8791598 · Cited by 149
Using LSTM encoder-decoder algorithm for detecting anomalous ADS-B messages
Habler, E., & Shabtai, A. (2018) · Computers & Security · doi:10.1016/j.cose.2018.07.004 · Cited by 148
A taxonomy of cyber-physical threats and impact in the smart home
Heartfield, R., Loukas, G., Budimir, S., Bezemskij, A., Fontaine, J. R. J., Filippoupolitis, A., & Roesch, E. B. (2018) · Computers & Security · doi:10.1016/j.cose.2018.07.011 · Cited by 147
Bridging the Skills Gap of Workers in Industry 4.0 by Human Performance Augmentation Tools
Ras, E., Wild, F., Stahl, C., & Baudet, A. (2017) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3056540.3076192 · Cited by 144
Integrity attacks on cyber-physical systems
Mo, Y., & Sinopoli, B. (2012) · International Conference on High Confidence Networked Systems · doi:10.1145/2185505.2185514 · Cited by 124
Distributed Framework for Detecting PMU Data Manipulation Attacks With Deep Autoencoders
Wang, J., Shi, D., Li, Y., Chen, J., Ding, H., & Duan, X. (2018) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2018.2859339 · Cited by 123
Towards Security Threats of Deep Learning Systems: A Survey
He, Y., Meng, G., Chen, K., Hu, X., & He, J. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3034721 · Cited by 122
Data Security Issues in Deep Learning: Attacks, Countermeasures, and Opportunities
Xu, G., Li, H., Ren, H., Yang, K., & Deng, R. H. (2019) · IEEE Communications Magazine · doi:10.1109/mcom.001.1900091 · Cited by 119
Program Analysis of Commodity IoT Applications for Security and Privacy
Celik, Z. B., Fernandes, E., Pauley, E., Tan, G., & McDaniel, P. D. (2019) · ACM Computing Surveys · doi:10.1145/3333501 · Cited by 118
Cyber Physical Security Analytics for Transactive Energy Systems
Zhang, Y., Krishnan, V., Pi, J., Kaur, K., Srivastava, A. K., Hahn, A., & Suresh, S. (2019) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2019.2928168 · Cited by 96
High-Performance Unsupervised Anomaly Detection for Cyber-Physical System Networks
Schneider, P., & Böttinger, K. (2018) · doi:10.1145/3264888.3264890 · Cited by 95
Mobile Edge Assisted Literal Multi-Dimensional Anomaly Detection of In-Vehicle Network Using LSTM
Zhu, K., Chen, Z., Peng, Y., & Zhang, L. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2907269 · Cited by 89
Wu, Z., Guo, Y., Lin, W., Yu, S., & Ji, Y. (2018) · Sensors · doi:10.3390/s18041096 · Cited by 87
Anomaly detection in discrete manufacturing using self-learning approaches
Lindemann, B., Fesenmayr, F., Jazdi, N., & Weyrich, M. (2019) · Procedia CIRP · doi:10.1016/j.procir.2019.02.073 · Cited by 86
Deep learning aided interval state prediction for improving cyber security in energy internet
Wang, H., Ruan, J., Ma, Z., Zhou, B., Fu, X., & Cao, G. (2019) · Energy · doi:10.1016/j.energy.2019.03.009 · Cited by 75
Unearthing Stealthy Program Attacks Buried in Extremely Long Execution Paths
Shu, X., Yao, D. D., & Ramakrishnan, N. (2015) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2810103.2813654 · Cited by 72
Self-citationLi, Y., Zhang, L., Lv, Z., & Wang, W. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3018259 · Cited by 71
Towards Automated Safety Vetting of PLC Code in Real-World Plants
Zhang, M., Chen, C.-Y., Kao, B.-C., Qamsane, Y., Shao, Y., Lin, Y., Shi, E., Mohan, S., Barton, K. L., Moyne, J. R., et al. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00034 · Cited by 70
Orpheus: Enforcing Cyber-Physical Execution Semantics to Defend Against Data-Oriented Attacks
Cheng, L., Tian, K., & Yao, D. D. (2017) · Annual Computer Security Applications Conference · doi:10.1145/3134600.3134640 · Cited by 69
Self-citationAhmed, C. M., Raman, M. R. G., & Mathur, A. P. (2020) · doi:10.1145/3384941.3409588 · Cited by 68
Khan, H. A., Sehatbakhsh, N., Nguyen, L. N., Prvulović, M., & Zajic, A. G. (2019) · Journal of Hardware and Systems Security · doi:10.1007/s41635-019-00074-w · Cited by 66
Peri, N., Khorramshahi, P., Rambhatla, S. S., Shenoy, V., Rawat, S., Chen, J.-C., & Chellappa, R. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00319 · Cited by 64
Causality reasoning about network events for detecting stealthy malware activities
Zhang, H., Yao, D. D., Ramakrishnan, N., & Zhang, Z. (2016) · Computers & Security · doi:10.1016/j.cose.2016.01.002 · Cited by 62
Self-citationImproving User Trust on Deep Neural Networks Based Intrusion Detection Systems
Amarasinghe, K., & Manic, M. (2018) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon.2018.8591322 · Cited by 47
NIPAD: a non-invasive power-based anomaly detection scheme for programmable logic controllers
Xiao, Y.-J., Xu, W., Jia, Z., Ma, Z., & Qi, D. (2017) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1601540 · Cited by 45
Self-citationAnomaly Detection as a Service: Challenges, Advances, and Opportunities
Yao, D. D., Shu, X., Cheng, L., & Stolfo, S. J. (2017) · Synthesis lectures on information security, privacy, and trust · doi:10.2200/s00800ed1v01y201709spt022 · Cited by 40
Self-citationDimensionality Reduction and Anomaly Detection for CPPS Data using Autoencoder
Eiteneuer, B., Hranisavljevic, N., & Niggemann, O. (2019) · IEEE International Conference on Industrial Technology (ICIT) · doi:10.1109/icit.2019.8755116 · Cited by 39
Haider, W., Hu, J., Xie, Y., Yu, X., & Wu, Q. (2017) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2017.2736555 · Cited by 37
Additive Explanations for Anomalies Detected from Multivariate Temporal Data
Giurgiu, I., & Schumann, A. (2019) · Proceedings of the 28th ACM International Conference on Information and Knowledge Management · doi:10.1145/3357384.3358121 · Cited by 27
DReAM: Deep Recursive Attentive Model for Anomaly Detection in Kernel Events
Ezeme, O. M., Mahmoud, Q. H., & Azim, A. (2019) · IEEE Access · doi:10.1109/access.2019.2897122 · Cited by 25
Hierarchical Attention-Based Anomaly Detection Model for Embedded Operating Systems
Ezeme, O. M., Mahmoud, Q. H., & Azim, A. (2018) · IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA) · doi:10.1109/rtcsa.2018.00035 · Cited by 24
Khanapuri, E. M., Chintalapati, T., Sharma, R., & Gerdes, R. (2019) · IEEE/ACM International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS) · doi:10.1109/sescps.2019.00014 · Cited by 24
False Data Injection Attack Detection in a Power Grid Using RNN
Deng, Q., & Sun, J. Q. (2018) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon.2018.8591079 · Cited by 22
Study of Autoencoder Neural Networks for Anomaly Detection in Connected Buildings
Legrand, A., Niepceron, B., Cournier, A., & Trannois, H. (2018) · doi:10.1109/gciot.2018.8620158 · Cited by 22
Deep Learning Based Forecasting of Critical Infrastructure Data
Zohrevand, Z., Glässer, U., Tayebi, M. A., Shahir, H. Y., Shirmaleki, M., & Shahir, A. Y. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133031 · Cited by 18
Narayanan, S. N., Joshi, A., & Bose, R. (2020) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2020.3034331 · Cited by 15
Wide and Recurrent Neural Networks for Detection of False Data Injection in Smart Grids
Wang, Y., Chen, D., Zhang, C., Chen, X., Huang, B., & Cheng, X. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23597-0_27 · Cited by 13
Anomaly Detection in Vehicle-to-Infrastructure Communications
Russo, M., Labonne, M., Olivereau, A., & Rmayti, M. (2018) · IEEE Vehicular Technology Conference · doi:10.1109/vtcspring.2018.8417863 · Cited by 11
DeepAnom: An Ensemble Deep Framework for Anomaly Detection in System Processes
Ezeme, O. M., Lescisin, M., Mahmoud, Q. H., & Azim, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-18305-9_58 · Cited by 6
Proceedings of the 9th ACM Cyber-Physical System Security Workshop
(2023) · doi:10.1145/3592538 · Cited by 1
Cited by
A Review of Anomaly Detection Strategies to Detect Threats to Cyber-Physical Systems
Jeffrey, N., Tan, Q., & Villar, J. R. (2023) · Electronics · doi:10.3390/electronics12153283 · Cited by 160
A Systematic Review of Data Quality in CPS and IoT for Industry 4.0
Goknil, A., Nguyen, P., Sen, S., Politaki, D., Niavis, H., Pedersen, K. J., Suyuthi, A., Anand, A., & Ziegenbein, A. (2023) · ACM Computing Surveys · doi:10.1145/3593043 · Cited by 84
A Systematic Analysis of Enhancing Cyber Security Using Deep Learning for Cyber Physical Systems
Gaba, S., Budhiraja, I., Kumar, V., Martha, S., Khurmi, J., Singh, A., Singh, K. K., Askar, S. S., & Abouhawwash, M. (2024) · IEEE Access · doi:10.1109/access.2023.3349022 · Cited by 62
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
Xu, Q., Ali, S., & Yue, T. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3582571 · Cited by 49
ADT: Time series anomaly detection for cyber-physical systems via deep reinforcement learning
Yang, X., Howley, E., & Schukat, M. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103825 · Cited by 47
Metadata
- Authors
- Luo, Y., Xiao, Y., Cheng, L., Peng, G., & Yao, D. (.
- Venue
- ACM Computing Surveys
- Published
- 2021-05-25
- Volume
- 54
- Issue
- 5
- Pages
- 1-36
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3164160032
- Cited by
- 268
- References
- 83
- FWCI
- 15.44
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.