Removing Backdoor-Based Watermarks in Neural Networks with Limited Data
Open accessAbstract
Deep neural networks have been widely applied and achieved great success in various fields. As training deep models usually consumes massive data and computational resources, trading the trained deep models is highly-demanded and lucrative nowadays. Unfortunately, the naive trading schemes typically involves potential risks related to copyright and trustworthiness issues, e.g., a sold model can be illegally resold to others without further authorization to reap huge profits. To tackle this problem, various watermarking techniques are proposed to protect the model intellectual property, amongst which the backdoor-based watermarking is the most commonly-used one. However, the robustness of these watermarking approaches is not well evaluated under realistic settings, such as limited in-distribution data availability and agnostic of watermarking patterns. In this paper, we benchmark the robustness of watermarking, and propose a novel backdoor-based watermark removal framework using limited data, dubbed WILD. The proposed WILD removes the watermarks of deep models with only a small portion of training data, and the output model can perform the same as models trained from scratch without watermarks injected. In particular, a novel data augmentation method is utilized to mimic the behavior of watermark triggers. Combining with the distribution alignment between the normal and perturbed (e.g., occluded) data in the feature space, our approach generalizes well on all typical types of trigger contents. The experimental results demonstrate that our approach can effectively remove the watermarks without compromising the deep model performance for the original task with the limited access to training data.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
After the aggregation phase, defense mechanisms can be employed to address any remaining injected backdoors in the global model.
“After the aggregation phase, defense mechanisms can be employed to address any remaining injected backdoors in the global model [251]-[255].”
From Data and model poisoning backdoor attacks on wireless federated learning, and the defense mechanisms: A comprehensive survey (2024) · cites this paper as [251]
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
Random Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617 · see citing works
DeepPose: Human Pose Estimation via Deep Neural Networks
Toshev, A., & Szegedy, C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.214 · Cited by 3,288 · see citing works
Face recognition: a convolutional neural-network approach
Lawrence, S., Giles, C. L., Tsoi, A. C., & Back, A. D. (1997) · IEEE Transactions on Neural Networks · doi:10.1109/72.554195 · Cited by 3,092 · see citing works
Deep Speech 2: End-to-End Speech Recognition in English and MandarinPreprint
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,191 · see citing works
Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H.-T., & Zhao, B. Y. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00031 · Cited by 1,486 · see citing works
Trojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270 · see citing works
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
Gu, T., Dolan-Gavitt, B., & Garg, S. (2017) · arXiv · doi:10.48550/arxiv.1708.06733 · Cited by 1,061 · see citing works
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
Liu, K., Dolan-Gavitt, B., & Garg, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00470-5_13 · Cited by 1,055 · see citing works
Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
Chen, X., Liu, C., Li, B., Lu, K., & Song, D. (2017) · arXiv · doi:10.48550/arxiv.1712.05526 · Cited by 1,039 · see citing works
Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D. C., & Nepal, . (2019) · Annual Computer Security Applications Conference · doi:10.1145/3359789.3359790 · Cited by 719 · see citing works
Embedding Watermarks into Deep Neural Networks
Uchida, Y., Nagai, Y., Sakazawa, S., & Satoh, S. (2017) · International Conference on Multimedia Retrieval · doi:10.1145/3078971.3078974 · Cited by 709 · see citing works
Protecting Intellectual Property of Deep Neural Networks with Watermarking
Zhang, J., Gu, Z., Jang, J., Wu, H., Stoecklin, M. P., Huang, H., & Molloy, I. (2018) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3196494.3196550 · Cited by 556 · see citing works
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering
Chen, B., Carvalho, W., Baracaldo, N., Ludwig, H., Edwards, B., Lee, T., Molloy, I., & Srivastava, B. (2018) · arXiv · doi:10.48550/arxiv.1811.03728 · Cited by 485 · see citing works
MLaaS: Machine Learning as a Service
Ribeiro, M., Grolinger, K., & Capretz, M. A. M. (2015) · IEEE International Conference on Machine Learning and Applications (ICMLA) · doi:10.1109/icmla.2015.152 · Cited by 395 · see citing works
Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by BackdooringPreprint
Adi, Y., Baum, C., Cissé, M., Benny Pinkas, B., & Keshet, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04633 · Cited by 395 · see citing works
Adversarial frontier stitching for remote neural network watermarking
Merrer, E. L., Pérez, P., & Trédan, G. (2019) · Neural Computing and Applications · doi:10.1007/s00521-019-04434-z · Cited by 342 · see citing works
Mitigating Poisoning Attacks on Machine Learning Models: A Data Provenance Based Approach
Baracaldo, N., Chen, B., Ludwig, H., & Safavi, J. A. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140450 · Cited by 108 · see citing works
REFIT: A Unified Watermark Removal Framework For Deep Learning Systems With Limited Data
Chen, X., Wang, W., Bender, C., Ding, Y., Jia, R., Li, B., & Song, D. (2021) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3433210.3453079 · Cited by 100 · see citing works
On the Robustness of Backdoor-based Watermarking in Deep Neural Networks
Shafieinejad, M., Lukas, N., Wang, J., Li, X., & Kerschbaum, F. (2021) · Proceedings of the 2021 ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3437880.3460401 · Cited by 98 · see citing works
Attacks on Digital Watermarks for Deep Neural Networks
Wang, T., & Kerschbaum, F. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682202 · Cited by 83 · see citing works
SentiNet: Detecting Physical Attacks Against Deep Learning SystemsPreprint
Chou, E., Tramèr, F., Pellegrino, G., & Boneh, D. (2018) · arXiv (Cornell University) · Cited by 76 · see citing works
Have You Stolen My Model? Evasion Attacks Against Deep Neural Network Watermarking TechniquesPreprint
Hitaj, D., & Mancini, L. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00615 · Cited by 39 · see citing works
Scaling Machine Learning as a Service.
Li, L. E., Chen, E., Hermann, J., Zhang, P., & Wang, L. (2016) · Cited by 23 · see citing works
Machine Learning-as-a-Service and Its Application to Medical Informatics
Tafti, A. P., LaRose, E. R., Badger, J. C., Kleiman, R., & Peissig, P. L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-62416-7_15 · Cited by 22 · see citing works
Flagged first, then by citations
4 of the 30 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Li, Y., Jiang, Y., Li, Z., & Xia, S.-T. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3182979 · Cited by 682 · see citing works
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., & Goldstein, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3162397 · Cited by 362 · see citing works
Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural NetworksPreprint
Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., & Ma, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05930 · Cited by 264 · see citing works
Wan, Y., Qu, Y., Ni, W., Xiang, Y., Gao, L., & Hossain, E. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3361451 · Cited by 152 · see citing works
“4) How to Remove the Injected Backdoor Effect After the Aggregation Phase: After the aggregation phase, defense mechanisms can be employed to address any remaining injected backdoors in the global model [251], [252], [253], [254], [255].”
MethodRIGA: Covert and Robust White-Box Watermarking of Deep Neural Networks
Wang, T., & Kerschbaum, F. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450000 · Cited by 131 · see citing works
Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models
Shan, S., Ding, W., Passananti, J., Wu, S. L., Zheng, H.-T., & Zhao, B. Y. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00207 · Cited by 117 · see citing works
A Systematic Review on Model Watermarking for Neural Networks
Boenisch, F. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.729663 · Cited by 91 · see citing works
SoK: How Robust is Image Classification Deep Neural Network Watermarking?
Lukas, N., Jiang, E., Li, X., & Kerschbaum, F. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833693 · Cited by 93 · see citing works
Xue, M., Zhang, Y., Wang, J., & Liu, W. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3133824 · Cited by 75 · see citing works
Deep Neural Backdoor in Semi-Supervised Learning: Threats and Countermeasures
Yan, Z., Wu, J., Li, G., Li, S., & Guizani, M. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3116431 · Cited by 46 · see citing works
Structural Watermarking to Deep Neural Networks via Network Channel Pruning
Zhao, X., Yao, Y., Wu, H., & Zhang, X. (2021) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs53200.2021.9648376 · Cited by 40 · see citing works
Lederer, I., Mayer, R. L., & Rauber, A. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270135 · Cited by 35 · see citing works
Backdoor Attack and Defense on Deep Learning: A Survey
Bai, Y., Xing, G., Wu, H., Rao, Z., Ma, C., Wang, S., Liu, X., Zhou, Y., Tang, J., Huang, K., et al. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3482723 · Cited by 36 · see citing works
Self-citationBackdoor Attacks Against Dataset Distillation
Liu, Y., Li, Z., Michael Backes, A., Shen, Y., & Zhang, Y. (2023) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2023.24287 · Cited by 31 · see citing works
Fine-tuning Is Not Enough: A Simple yet Effective Watermark Removal Attack for DNN Models
Guo, S., Zhang, T., Qiu, H., Zeng, Y., Xiang, T., & Liu, Y. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/500 · Cited by 33 · see citing works
Ownership Verification of DNN Architectures via Hardware Cache Side Channels
Lou, X., Guo, S., Li, J., & Zhang, T. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3184644 · Cited by 21 · see citing works
Chen, Y., Tian, J., Chen, X., & Zhou, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00785 · Cited by 19 · see citing works
AI Trojan Attack for Evading Machine Learning-Based Detection of Hardware Trojans
Pan, Z., & Mishra, P. (2023) · IEEE Transactions on Computers · doi:10.1109/tc.2023.3251864 · Cited by 13 · see citing works
DeepDIST: A Black-Box Anti-Collusion Framework for Secure Distribution of Deep Models
Cheng, H., Li, X., Wang, H., Zhang, X., Liu, X., Wang, M., & Li, F. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3284914 · Cited by 11 · see citing works
Self-citationDetect and Remove Watermark in Deep Neural Networks via Generative Adversarial Networks
Sun, S., Wang, H., Xue, M., Zhang, Y., Wang, J., & Liu, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91356-4_18 · Cited by 10 · see citing works
The Hidden Vulnerability of Watermarking for Deep Neural Networks.Preprint
Guo, S., Zhang, T., Qiu, H., Zeng, Y., Xiang, T., & Liu, Y. (2020) · arXiv (Cornell University) · Cited by 9 · see citing works
Secure Watermark for Deep Neural Networks with Multi-task LearningPreprint
Li, F., & Wang, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.10021 · Cited by 9 · see citing works
Self-citationTowards Practical Watermark for Deep Neural Networks in Federated LearningPreprint
Li, F., Wang, S., & Liew, A. W.-C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03167 · Cited by 9 · see citing works
Self-citationDeep neural networks watermark via universal deep hiding and metric learning
Ye, Z., Zhang, X., & Feng, G. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09469-5 · Cited by 8 · see citing works
Attention Distraction: Watermark Removal Through Continual Learning with Selective Forgetting
Zhong, Q., Zhang, L. Y., Hu, S., Gao, L., Zhang, J., & Xiang, Y. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9858930 · Cited by 7 · see citing works
Neural Dehydration: Effective Erasure of Black-box Watermarks from DNNs with Limited Data
Lu, Y., Li, W., Zhang, M., Pan, X., & Yang, M. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690334 · Cited by 6 · see citing works
SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)Preprint
Lukas, N., Jiang, E., Li, X., & Kerschbaum, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04974 · Cited by 6 · see citing works
A lightweight backdoor defense framework based on image inpainting
Wei, Y., Gao, H., Wang, Y., Gao, Y., & Liu, H. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.03.052 · Cited by 6 · see citing works
ChainMarks: Securing DNN Watermark with Cryptographic Chain
Choi, B., Wang, S., Choi, I., & Sun, K. (2025) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3708821.3736214 · Cited by 5 · see citing works
Robust Watermarking for Neural Network Models Using Residual Network
Wang, L., Wang, Z., Li, X., & Qin, C. (2022) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp55362.2022.9949601 · Cited by 5 · see citing works
When NAS Meets Watermarking: Ownership Verification of DNN Models via Cache Side Channels.Preprint
Lou, X., Guo, S., Zhang, T., Zhang, Y., & Liu, Y. (2021) · arXiv (Cornell University) · Cited by 5 · see citing works
Rule-Based Runtime Mitigation Against Poison Attacks on Neural Networks
Usman, M., Gopinath, D., Sun, Y.-C., & Păsăreanu, C. (2022) · Runtime Verification · doi:10.1007/978-3-031-17196-3_4 · Cited by 6 · see citing works
Detecting Digital Watermarking Image Attacks Using a Convolution Neural Network Approach
Alzahrani, A. (2022) · Security and Communication Networks · doi:10.1155/2022/4408336 · Cited by 3 · see citing works
RetractedMa, B., Wang, J., Wang, D., & Meng, B. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.09.036 · Cited by 3 · see citing works
Multipurpose Watermarking Approach for Copyright and Integrity of Steganographic Autoencoder Models
Wei, G., Chang, C., Bai, Y., Fan, Y., Tao, L., & Li, L. (2021) · Security and Communication Networks · doi:10.1155/2021/9936661 · Cited by 3 · see citing works
RetractedDeep Watermarking for Deep Intellectual Property Protection: A Comprehensive SurveyPreprint
Sun, Y., Liu, L., Yu, N., Liu, Y., Tian, Q. C., & Guo, D. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4697020 · Cited by 3 · see citing works
Membership Encoding for Black-Box Neural Network Watermarking
Zhang, H., Li, F., & Wang, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890687 · Cited by 2 · see citing works
Self-citationМенисов, А. Б., Lomako, A. G., & Dudkin, A. S. (2022) · Scientific and technical journal of information technologies mechanics and optics · doi:10.17586/2226-1494-2022-22-4-742-750 · Cited by 2 · see citing works
Wu, Y., Cheng, H., Guan, L., Liu, P., Chen, F., & Wang, M. (2023) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-dss-smartcity-dependsys60770.2023.00096 · Cited by 2 · see citing works
Yan, M., Su, Z., Wang, Y., Ran, X., Liu, Y., & Luan, T. H. (2023) · Globecom · doi:10.1109/globecom54140.2023.10437312 · Cited by 2 · see citing works
Backdoor Attacks Prediction in IIoT Network using Optimal Double Mask Region Convolution Model
Subramanian, N., Bushra, S. N., Shobana, G., & Radhika, S. (2024) · IETE Journal of Research · doi:10.1080/03772063.2023.2230174 · Cited by 1 · see citing works
Ural, Ö., & Yoshigoe, K. (2025) · IEEE Access · doi:10.1109/access.2025.3642198 · Cited by 1 · see citing works
Delete but Not Gone: Reactivation of Neural Network Watermarks
Nie, H., Xiao, J., Lu, Z. G., Shen, R., & Lu, S. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3793678 · Cited by 1 · see citing works
The Victim and The Beneficiary: Exploiting a Poisoned Model to Train a Clean Model on Poisoned Data
Zhu, Z., Wang, R., Zou, C., & Jing, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00021 · Cited by 2 · see citing works
An Approach to Generation Triggers for Parrying Backdoor in Neural Networks
Менисов, А. Б. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-19907-3_29
SDBC: A Novel and Effective Self-Distillation Backdoor Cleansing Approach
Ran, S., Zheng, B., & Sun, M. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8148-9_23
Beyond Robustness: On the Unforgeability Trade-offs in Statistical DNN Watermarking
Bui, P.-D., & Tran, T.-D. (2025) · International Conference on Cryptography and Information Security (VCRIS) · doi:10.1109/vcris68011.2025.11250570
Quantifying the Impact of Poisoning Attacks in Deep Neural Networks for Attack Classification
Spelter, F., Loevenich, J. F., Hürten, T., Adler, E., & Lopes, R. R. F. (2025) · IEEE Military Communications Conference (MILCOM) · doi:10.1109/milcom64451.2025.11310697
Formal Foundations of Trigger-Based Watermarking
Calzavara, S., Cazzaro, L., Lucchese, C., & Orlando, S. (2026) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp68448.2026.00088
- Authors
- Liu, X., Li, F., Wen, B., & Li, Q.
- Affiliations
- Tsinghua University (CN)
- Nanyang Technological University (SG)
- Beijing National Research Center for Information Science and Technology (CN)
- Venue
- 2020 25th International Conference on Pattern Recognition (ICPR)
- Publisher
- IEEE
- Published
- 2021-01-10
- Pages
- 10149-10156
- ISSN
- 1041-3278 (ISSN-L)
- OpenAlex
- W3158240034
- Cited by
- 49
- References
- 30
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 7 · 2023: 12 · 2022: 9 · 2021: 13 · 2020: 1
- FWCI
- 4.74
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Research Foundation (NRF) Singapore
- National Natural Science Foundation of China (61572278)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Steganography and Watermarking Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Physical Unclonable Functions (PUFs) and Hardware Security
- Keywords
- watermark removal, deep neural network, limited training data, data augmentation, distribution alignment, occlusion, watermark robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review