Cited by 364Privacy-Preserving Technologies in DataNetwork Security and Intrusion DetectionInternet of Things and AI

Federated Deep Learning for Cyber Security in the Internet of Things: Concepts, Applications, and Experimental Analysis

Open access
Ferrag, Mohamed Amine, Friha, Othmane, Maglaras, Leandros, Janicke, Helge, Shu, Lei·2021·IEEE Access

Abstract

In this article, we present a comprehensive study with an experimental analysis of federated deep learning approaches for cyber security in the Internet of Things (IoT) applications. Specifically, we first provide a review of the federated learning-based security and privacy systems for several types of IoT applications, including, Industrial IoT, Edge Computing, Internet of Drones, Internet of Healthcare Things, Internet of Vehicles, etc. Second, the use of federated learning with blockchain and malware/intrusion detection systems for IoT applications is discussed. Then, we review the vulnerabilities in federated learning-based security and privacy systems. Finally, we provide an experimental analysis of federated deep learning with three deep learning approaches, namely, Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), and Deep Neural Network (DNN). For each deep learning model, we study the performance of centralized and federated learning under three new real IoT traffic datasets, namely, the Bot-IoT dataset, the MQTTset dataset, and the TON_IoT dataset. The goal of this article is to provide important information on federated deep learning approaches with emerging technologies for cyber security. In addition, it demonstrates that federated deep learning approaches outperform the classic/centralized versions of machine learning (non-federated learning) in assuring the privacy of IoT device data and provide the higher accuracy in detecting attacks.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper surveys federated deep learning for IoT cyber security and adds an experimental comparison of centralized vs. federated DNN, CNN, and RNN intrusion detection models on three IoT traffic datasets. It concludes that federated learning improves privacy and can approach or outperform centralized learning, though its own reported centralized accuracies are often higher than the federated global-model results.

Key findings

  • The review covers federated learning-based security and privacy systems across Industrial IoT, edge computing, Internet of Drones, Internet of Healthcare Things, Internet of Vehicles, cloud computing, 5G-enabled IoT, mobile crowdsensing, and cyber-physical systems; it also discusses federated learning with blockchain and federated intrusion/malware detection.
  • In centralized experiments, the highest accuracies were RNN on Bot-IoT at 96.76%, DNN on MQTTset at 90.06%, and RNN on TON_IoT at 99.98%; the lowest were DNN on Bot-IoT at 95.76%, RNN on MQTTset at 89.29%, and CNN on TON_IoT at 98.87%.
  • In federated experiments, IID global models approximated centralized performance; Non-IID models struggled initially but after 50 rounds approached centralized performance. For Bot-IoT with CNN and K=15, the worst client had 01.00% accuracy at the first round, 52.98% at round 50, while the global model achieved 90.35%.
  • The paper states that federated deep learning approaches outperform classic/centralized machine learning in assuring IoT data privacy and provide higher accuracy in detecting attacks, although the experimental comparison text reports that federated global models “managed to approach” centralized performance.

Methods

  • Design: comprehensive survey plus experimental analysis training three deep learning IDS models—DNN, CNN, and RNN—under centralized and federated learning settings.
  • Data: three real IoT traffic datasets—Bot-IoT (69.3 GB PCAP, over 72 million records; CSV files used), MQTTset, and TON_IoT—with preprocessing including dropping selected features, one-hot encoding of `proto`, and Z-score normalization.
  • Federated setup and analysis: Sherpa.AI framework on Google Colaboratory with NumPy, Pandas, TensorFlow, and Keras; client distributions K=5, K=10, and K=15; IID and Non-IID data distributions; 50 federated rounds; metrics include accuracy, precision, recall, F1-score, and ROC curves.

Limitations

  • Paper-stated open challenges include: privacy can still be revealed by analyzing the global model; differential privacy can degrade performance or require high computing power; IoT network constraints such as bandwidth, interference, noise, and limited reliability; data bias and unequal data sizes across devices; and lack of FL-specific IoT frameworks.
  • Evident from the text: the experiments use pre-existing public datasets with engineered partitions to mimic federations because of a lack of FL-specific datasets; experiments were run on Google Colab; source code is available only upon request; and the conclusion that federated learning provides higher attack-detection accuracy is not fully supported by the paper’s own centralized-vs-federated accuracy results, which show federated global models approaching rather than consistently exceeding centralized models.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2021.3118642

Cites

176 of 177 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 50 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Federated deep learning approaches outperform classic/centralized machine learning (non-federated learning) in assuring the privacy of IoT device data.

    “In addition, it demonstrates that federated deep learning approaches outperform the classic/centralized versions of machine learning (non-federated learning) in assuring the privacy of IoT device data and provide the higher accuracy in detecting attacks.”
    Quote in the paperSubstantial claim
  • Findingown data

    Federated deep learning approaches provide higher accuracy in detecting attacks than classic/centralized machine learning (non-federated learning).

    “In addition, it demonstrates that federated deep learning approaches outperform the classic/centralized versions of machine learning (non-federated learning) in assuring the privacy of IoT device data and provide the higher accuracy in detecting attacks.”
    Quote in the paperSubstantial claim
  • Findingnone given

    Almost all surveys on federated learning for IoT applications present security and privacy countermeasures without focusing on an experimental analysis.

    “Almost all of the surveys on federated learning for IoT applications present security and privacy countermeasures without focusing on an experimental analysis.”
    Quote in the paperSubstantial claim
  • Findingnone given

    All related surveys did not cover the application of federated deep learning for cyber security in IoT applications with a focus on experimental analysis.

    “All these related surveys did not cover the application of federated deep learning for cyber security in IoT applications with focusing on experimental analysis.”
    Quote in the paperNot a substantial claim
  • Findingown data

    For the IID data distribution strategy, the federated deep learning global models were able to approximate the performance of the centralized learning models.

    “For the IID data distribution strategy, the federated deep learning global models were able to approximate the performance of the centralized learning models.”
    Quote in the paperSubstantial claim
  • May combine two claims. 20
  • Classified as probably not a substantial claim. 8
  • May combine two claims.

    Findingown data

    For the Non-IID data distribution strategy, the global models struggled to perform as well as in IID, but after 50 federated learning rounds the overall performance was pretty good.

    “For the non-IDI data distribution strategy, the global models struggled a bit to perform the same as in IID, which is quite normal since the data samples were randomly distributed for all clients, however after 50 FL runs, the overall performance was pretty good.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    For the TON_IoT dataset, the highest multiclass classification accuracy among centralized deep learning models was obtained by RNN at 99.98%, and the lowest by CNN at 98.87%.

    “The highest accuracy for the TON_IoT dataset was obtained using the RNN classifier which achieved 99.98%, while the lowest accuracy was obtained using the CNN classifier with 98.87%.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    For the Bot-IoT dataset, the highest multiclass classification accuracy among centralized deep learning models was obtained by RNN at 96.76%, and the lowest by DNN at 95.76%.

    “The highest accuracy for the Bot-IoT dataset was obtained using the RNN classifier which achieved 96.76%, while the lowest accuracy was obtained using the DNN classifier with 95.76%.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Premisecitation

    The CrowdSFL protocol can resist malicious miners, malicious workers, and malicious requesters.

    “The simulation results show that the proposed CrowdSFL protocol can resist the following malicious behaviors: Malicious miners, Malicious workers, and Malicious requesters.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [100] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The FedSky system can reduce significantly the computation time of the users as well as the latency of the system compared to the conventional FedAvg approach.

    “Compared to the conventional FedAvg approach [83], the proposed system can reduce significantly the computation time of the users as well as the latency of the system.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [82] Zhang, 2021, FedSky: An efficient and privacy-preserving scheme for federated mobile crowdsensing, IEEE Internet Things J.

      Not checked in the free run.

    • cites [83] McMahan, 2016, Communication-efficient learning of deep networks from decentralized data, arXiv:1602.05629

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Many existing systems already contain vulnerable IoT devices that are open to being compromised, which is harmful in sensitive tasks such as surveillance.

    “Accordingly, many existing systems already contain vulnerable IoT devices that are open to being compromised, which is furthermore harmful in sensitive tasks such as surveillance, as shown by the work of Ciuonzo et al. [59], which focused on the issue of distributed detection of a non-cooperative object in a wireless sensor network.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ciuonzo et al. [59] source

      Not checked in the free run.

    • Rests on an assumption: Many IoT devices are susceptible to attack through unsafe design, deployment, and configurations.
  • May combine two claims.

    Premisecitation

    The FedAGRU system provides less communication overhead with higher detection accuracy.

    “the results show that the FedAGRU system provides less communication overhead with higher detection accuracy.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [96] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In the absence of credible security defense systems implemented on IoT devices, they can be attacked by hackers and represent a large attack surface that is actively exploited.

    “In the absence of a credible security defense systems implemented on the IoT devices, they can be attacked by hackers [2] and are representing a large attack surface that is actively exploited.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Federated learning achieves great success and is widely used in many fields, including mobile edge network optimization, Google keyboard query suggestions and prediction, COVID-19 detection, vehicle communications, Internet of Drones, augmented reality, and intrusion detection.

    “The federated learning achieves great success and is widely used in many fields, e.g., mobile edge network optimization [18], Google keyboard query suggestions and prediction [19], [20], COVID-19 detection [21]- [23], vehicles communications [24], Internet of Drones [25], Augmented reality [26], Intrusion detection [27]- [29] etc.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [18] source

      Not checked in the free run.

    • cites [19] 2021

      Not checked in the free run.

    • cites [20] Hard, 2018, Federated learning for mobile keyboard prediction, arXiv:1811.03604

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The proposed permissioned blockchain-based mechanism can guarantee the quality of shared data as well as differential privacy.

    “The simulation results on two real-world data sets (i.e., Reuters dataset and 20 newsgroups dataset) show that the proposed system can guarantee the quality of shared data as well as differential privacy.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [106] source

      Not checked in the free run.

  • May combine two claims.

    Premiseown data

    The proposed IDS for industrial cyber physical systems achieves F-score = 98.14%, recall = 97.47%, precision = 98.85%, accuracy = 99.20%, which are better compared to three related works.

    “The performance evaluation under the gas pipeline system dataset show the following results: F-score = 98.14 %, recall = 97.47 %, precision = 98.85 %, accuracy = 99.20 %, which are better compared to three related works [57], [95], and [77]”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [92] source

      Not checked in the free run.

    • cites [57] source

      Not checked in the free run.

    • cites [95] Schneble, Attack detection using federated learning in medical cyber-physical systems

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Differential privacy techniques degrade model performance or require additional conditions that are not suitable for IoT networks, especially high computing power.

    “Although some techniques have been used to overcome this problem, including differential privacy [166], these approaches degrade model performance or require additional conditions that are not suitable for IoT networks, especially high computing power [167].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [166] source

      Not checked in the free run.

    • cites [167] source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    The three datasets TON_IoT, Bot-IoT, and MQTTset are chosen because they are built from heterogeneous data sources and collected from IoT and IIoT sensor telemetry datasets.

    “They are chosen specifically because they are build from heterogeneous data sources as well as collected from IoT and IIoT sensor telemetry datasets.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The combination of blockchain technology with federated deep learning reduces the threat of data leakage and enables data owners to have more control over access to stored and shared data.

    “This combination reduces the threat of data leakage and enables data owners to have more control over the access to stored and shared data.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationown data

    Federated deep learning approaches (CNN, RNN, and DNN) outperform classic/centralized versions of machine learning (non-federated learning) in assuring the privacy of IoT device data and provide higher accuracy in detecting attacks.

    “The results demonstrate that federated deep learning approaches can outperform the classic/centralized versions of machine learning (non-federated learning) in assuring the privacy of IoT device data and provides the highest accuracy in detecting attacks.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisecitation

    The proposed idea can not only identify shilling hackers but also improve the performance of recommendations significantly.

    “The performance evaluation on two real-world datasets, namely, MovieLens and Netflix, demonstrates that the proposed idea can not only identify shilling hackers but also improve the performance of recommendations significantly.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [131] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Federated deep learning-based security and privacy systems can be applied for several types of IoT applications, including Industrial Internet of Things, Edge Computing, Internet of Drones, Internet of Healthcare Things, Cloud Computing, 5G-enabled IoT, Internet of Vehicles, and Mobile Crowdsensing.

    “The study shows that the federated deep learning-based security and privacy systems can be applied for several types of IoT applications, including, Industrial Internet of Things, Edge Computing, Internet of Drones, Internet of Healthcare Things, Cloud Computing, 5G-enabled IoT, Internet of Vehicles, Mobile Crowdsensing, etc.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    Centralized intrusion detection approaches have privacy issues because they require data to be collected at a single entity, making it easier for an attacker to target a single location; if that single entity is compromised, all sensitive data will be breached.

    “First, and most importantly, privacy issues, since it requires data to be collected at a single entity, thus making it easier for an attacker to target a single location for all data, if that single entity is compromised, all sensitive data will be breached.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisecitation

    Communication channels and computational power are constrained in terms of capacity, and there are network issues such as bandwidth, interference, and noise.

    “In addition, communication channels and computational power are constrained in terms of capacity, as well as the presence of various network issues such as bandwidth, interference, and noise [167].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [167] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The TON_IoT dataset was introduced by the IoT Lab of UNSW Canberra Cyber, SEIT, UNSW Canberra at ADFA for the collection and analysis of mixed data sources from IoT and Industrial IoT (IIoT).

    “This dataset is introduced by the IoT Lab of the UNSW Canberra Cyber, the School of Engineering and Information technology (SEIT), UNSW Canberra at the Australian Defence Force Academy (ADFA) [150] for the collection and analysis of mixed data sources from IoT and Industrial IoT (IIoT).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [150] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Federated learning-based intrusion detection systems significantly decrease the previous issues with decent detection accuracy, and in many cases approached the performance of a centralized approach.

    “Federated learning-based intrusion detection systems, on the other hand, significantly decrease the previous issues with decent detection accuracy, and in many cases, it approached the performance of a centralized approach as we showed with our federated deep learning models.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premiseown data

    The proposed system can classify with a binary cross-entropy loss score of 0.47 and an accuracy of 95.55% on a SCADA dataset.

    “Throughout the performance evaluation on a SCADA dataset, the results show that the proposed system can classify with a binary cross-entropy loss score of 0.47 and an accuracy of 95.55 %.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [122] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The BoT-IoT dataset was produced at the Cyber Range Lab at UNSW Canberra as a result of building a real-life network environment integrating a mix of normal and botnet traffic.

    “The BoT-IoT dataset was produced at the Cyber Range Lab at UNSW Canberra as a result of building a real-life network environment integrating a mix of normal and botnet traffic [154], [160]- [164].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [154] source

      Not checked in the free run.

    • cites [160]-[164]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    If IoT devices have varying data sizes, the FL-based system may give more importance to the contributions of the populations.

    “For example, if IoT devices have varying data sizes, the FL-based system may give more importance to the contributions of the populations.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The MQTTset dataset was introduced by Vaccari et al. to address the lack of support for specific protocols that IoT environments are currently using.

    “2) MQTTset DATASET introduced by Vaccari et al. [156] to address the lack of support for specific protocols that IoT environments are currently using.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [156] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning

    Any pre-existing public dataset with engineered partitions can be used to mimic data federations.

    “However, due to the lack of FL-specific datasets, any pre-existing public dataset with engineered partitions can be used to mimic data federations, as employed in our experiment.”
    Quote in the paperNot a substantial claim
26 of 26
  • The paper relies on this without support.

    Many cyber security researchers have difficulty in finding the best learning type (centralized or federated) to test and evaluate their proposed security methods in IoT applications.

    “Therefore, many cyber security researchers have difficulty in finding the best learning type (i.e., centralized or federated learning) to test and evaluate their proposed security methods in IoT applications, and selecting an appropriate federated deep learning method is an essential issue in this field.”
  • The paper relies on this without support.

    Selecting an appropriate federated deep learning method is an essential issue in this field.

    “Therefore, many cyber security researchers have difficulty in finding the best learning type (i.e., centralized or federated learning) to test and evaluate their proposed security methods in IoT applications, and selecting an appropriate federated deep learning method is an essential issue in this field.”
  • The paper relies on this without support.

    Traditional centralized learning has issues such as privacy, latency, bandwidth, and connectivity.

    “However, there are some issues for traditional centralized learning such as privacy, latency, bandwidth, and connectivity.”
  • The paper relies on this without support.

    Many IoT devices are susceptible to attack through unsafe design, deployment, and configurations.

    “Many of those devices, however, are susceptible to attack through unsafe design, deployment, and configurations.”
  • The paper relies on this without support.

    Federated learning can suffer from machine learning attacks against the locally created models.

    “Since it is trained on sensitive user data, federated learning can suffer from machine learning attacks against the locally created models.”
  • The paper relies on this without support.

    FL-based tasks require the data distribution to be Non-IID and unbalanced, reflecting the properties of the real-world scenario.

    “FL-based tasks require the data distribution to be Non-Independent and Identically Distributed (Non-IID) and unbalanced, which reflects the properties of the real-world scenario.”
  • The paper relies on this without support.

    Adversaries can use cyber security attacks such as adversarial attacks or poisoning attacks to degrade the accuracy of a machine learning model or deduce if an IoT device has been involved in some mission from their local model updates.

    “These adversaries can use cyber security attacks such as adversarial attacks or poisoning attacks to degrading the accuracy of a machine learning model or deduce if an IoT device has been involved in some mission from their local model updates.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • FedAGRU is attributed to different cited authors

    Minor

    One passage attributes the proposed FedAGRU system to Cvitic et al. [29] [88], while an earlier passage says Cetin et al. [96] proposed FedAGRU.

  • Dataset citations do not match the numbered references

    Minor

    The dataset discussion cites [149] for TON_IoT, but reference 149 is titled "LEAF: A benchmark for federated settings"; the TON_IoT reference is numbered 150. The same one-number mismatch affects the cited Bot-IoT and MQTTset references.

  • TON_IoT figure lists six classes after stating five

    Minor

    The discussion says Fig 16 presents ROC curves for five TON_IoT classes, but the figure caption enumerates classes 0 through 5, including XSS as a sixth class.

  • Research questions are attributed to the learning-performance figure

    Minor

    The text points to Fig 17 for the research questions, but the caption identifies Fig 17 as learning performances.

  • MQTTset figure lists six classes after stating five

    Minor

    The discussion says Fig 15 presents ROC curves for five MQTTset classes, but the figure caption enumerates classes 0 through 5, including Malformed as a sixth class.

5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. McMahan, 2017, Communication-efficient learning of deep networks from decentralized data, Artificial Intelligence and Statistics, 1273
  2. 2021, Federated Learning: Collaborative Machine Learning Without Centralized Training Data
  3. Schneble, Attack detection using federated learning in medical cyber-physical systems, 1
  4. Goodfellow, Generative adversarial nets, 27, 1
  5. 2021, The MNIST Database of Handwritten Digits

Cited by

Sorted by citations

Authors
Ferrag, M. A., Friha, O., Maglaras, L., Janicke, H., & Shu, L.
Affiliations
  • Edith Cowan University (AU)
  • Badji Mokhtar-Annaba University (DZ)
  • De Montfort University (GB)
  • Nanjing Agricultural University (CN)
Corresponding author
Ferrag, Mohamed Amine
Venue
IEEE Access
Published
2021
Volume
9
Pages
138509-138542
ISSN
2169-3536 (ISSN-L)
OpenAlex
W3203932036
Cited by
342
References
174
Citations by year
2026: 53 · 2025: 94 · 2024: 80 · 2023: 70 · 2022: 44 · 2021: 1
FWCI
17.16
Type
journal article
Language
en
Open access
Locations
  • IEEE Access (publishedVersion, cc-by)
  • DMU Open Research Archive (De Montfort University) (submittedVersion, cc-by)
  • Australasian Journal of Paramedicine (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion, other-oa)
  • Greater South Information System
Funders
Cyber Security Cooperative Research Centre (CSCRC) and Edith Cowan University, Australia
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
  • Physical Sciences › Computer Science › Information Systems › Internet of Things and AI
Keywords
federated deep learning, Internet of Things security, IoT privacy, malware detection, intrusion detection, blockchain, Industrial Internet of Things, edge computing, Internet of Drones, Internet of Medical Things, Internet of Vehicles, Bot-IoT dataset, ToN-IoT, recurrent neural networks, convolutional neural networks, deep neural network

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review