Practical Secure Aggregation for Privacy-Preserving Machine Learning
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
We design a novel, communication-efficient, failure-robust protocol for secure aggregation of high-dimensional data. Our protocol allows a server to compute the sum of large, user-held data vectors from mobile devices in a secure manner (i.e. without learning each user's individual contribution), and can be used, for example, in a federated learning setting, to aggregate user-provided model updates for a deep neural network. We prove the security of our protocol in the honest-but-curious and active adversary settings, and show that security is maintained even if an arbitrarily chosen subset of users drop out at any time. We evaluate the efficiency of our protocol and show, by complexity analysis and a concrete implementation, that its runtime and communication overhead remain low even on large data sets and client pools. For 16-bit input values, our protocol offers $1.73 x communication expansion for 210 users and 220-dimensional vectors, and 1.98 x expansion for 214 users and 224-dimensional vectors over sending data in the clear.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although SMC guarantees that none of the parties shares anything with each other or with any third party, it can not prevent an adversary from learning some individual information”
From Federated Learning for Healthcare Informatics (2021) · cites this paper as [8]
In the citing paper
“By combining local inference with trusted aggregation, TDAI can enhance both privacy and robustness of anomaly detection pipelines [239].”
From Anomaly detection and root-cause identification in microservices: a survey (2026) · cites this paper as [239]
In the citing paper
“FL is still in its infancy and is still vulnerable to various challenges [148,149].”
From Smart Wearables for the Detection of Cardiovascular Diseases: A Systematic Literature Review (2023) · cites this paper as [148]
In the citing paper
“Therefore, many FL frameworks use secure aggregation [81], [82], [83], [84] algorithms in the aggregation process, at which point most of the poisoning attack defense methods mentioned above will be invalid.”
From Poisoning attacks in federated learning: A survey (2023) · cites this paper as [81]
In the citing paper
“Bonawitz et al. [62] introduce a secure aggregation protocol for protecting individual model updates.”
From Handling privacy-sensitive medical data with federated learning: Challenges and future directions (2023) · cites this paper as [62]
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
New directions in cryptography
Diffie, W., & Hellman, M. (1976) · IEEE Transactions on Information Theory · doi:10.1109/tit.1976.1055638 · Cited by 14,713
Communication-Efficient Learning of Deep Networks from Decentralized Data
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Arcas, B. A. Y. (2016) · arXiv · doi:10.48550/arxiv.1602.05629 · Cited by 14,215 · cited 1×
“maintains a private database of her text messages securely on her own mobile device, and a shared global model is trained under the coordination of a central server based upon highly processed, minimally scoped, ephemeral updates from users [20], [10].”
BackgroundSelf-citationShamir, A. (1979) · Communications of the ACM · doi:10.1145/359168.359176 · Cited by 13,756 · cited 1×
“We rely on Shamir’s t -out-of- n Secret Sharing [38], which allows a user to split a secret s into n shares, such that any t shares can be used to reconstruct s , but any set of at most t − 1 shares gives no information about s .”
Method(2016) · Cited by 11,679
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · cited 3×
“This is particular advantageous in the local privacy setting [13], which offers provable guarantees for the protection of individual training examples [14], [15] even when the data aggregator is not assumed to be trusted [16], [17].”
BackgroundMethodSelf-citationThe Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557
Dwork, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11787006_1 · Cited by 5,291 · cited 1×
“In such cases, secure aggregation composes well with differential privacy [12].”
BackgroundMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
The Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2013) · doi:10.1561/9781601988195 · Cited by 3,856
Goldreich, O., Micali, S., & Wigderson, A. (1987) · Proceedings of the nineteenth annual ACM conference on Theory of computing - STOC '87 · doi:10.1145/28395.28420 · Cited by 3,590 · cited 1×
“As noted in Section I, there is a long line of work showing how multiple parties can securely compute any function using generic secure MPC [23], [24], [25], [26], [27].”
BackgroundModel Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961 · cited 2×
“While secure aggregation alone may suffice for some applications, for other applications stronger guarantees may be needed, as indicated by the failures of ad-hoc anonymization techniques [6], [7], [8], and by the demonstrated capability to extract information about individual training data from fullytrained models (w…”
BackgroundCompleteness theorems for non-cryptographic fault-tolerant distributed computation
Ben-Or, M., & Wigderson, A. (1988) · Proceedings of the twentieth annual ACM symposium on Theory of computing - STOC '88 · doi:10.1145/62212.62213 · Cited by 2,524 · cited 1×
“As noted in Section I, there is a long line of work showing how multiple parties can securely compute any function using generic secure MPC [23], [24], [25], [26], [27].”
BackgroundRobust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372
Privacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356 · cited 3×
“…other applications stronger guarantees may be needed, as indicated by the failures of ad-hoc anonymization techniques [6], [7], [8], and by the demonstrated capability to extract information about individual training data from fully-trained models (which are essentially aggregates) [9], [10], [11].”
BackgroundRAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response
Erlingsson, Ú., Pihur, V., & Korolova, A. (2014) · Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2660267.2660348 · Cited by 1,838 · cited 1×
“This is particular advantageous in the local privacy setting [13], which offers provable guarantees for the protection of individual training examples [14], [15] even when the data aggregator is not assumed to be trusted [16], [17].”
BackgroundOur Data, Ourselves: Privacy Via Distributed Noise Generation
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., & Naor, M. (2006) · Lecture Notes in Computer Science · doi:10.1007/11761679_29 · Cited by 1,835
The dining cryptographers problem: Unconditional sender and recipient untraceability
Chaum, D. (1988) · Journal of Cryptology · doi:10.1007/bf00206326 · Cited by 1,408 · cited 1×
“Dining cryptographers networks, or DC-nets, are a type of communication network which provide anonymity by using pairwise blinding of inputs [29], [30], similarly to our secure aggregation protocol.”
MethodMultiparty Computation from Somewhat Homomorphic Encryption
Damgård, I., Pastro, V., Smart, N., & Zakarias, S. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-32009-5_38 · Cited by 1,331 · cited 1×
“As noted in Section I, there is a long line of work showing how multiple parties can securely compute any function using generic secure MPC [23], [24], [25], [26], [27].”
BackgroundHow to Generate Cryptographically Strong Sequences of Pseudorandom Bits
Blum, M., & Micali, S. (1984) · SIAM Journal on Computing · doi:10.1137/0213053 · Cited by 1,322
Theory and application of trapdoor functions
Yao, A. C.-C. (1982) · doi:10.1109/sfcs.1982.95 · Cited by 1,008
Theory and application of trapdoor functions
Yao, A. C. (1982) · 23rd Annual Symposium on Foundations of Computer Science (sfcs 1982) · doi:10.1109/sfcs.1982.45 · Cited by 968
Local Privacy and Statistical Minimax Rates
Duchi, J. C., Jordan, M. I., & Wainwright, M. J. (2013) · 2013 IEEE 54th Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2013.53 · Cited by 954 · cited 1×
“This is particular advantageous in the local privacy setting [13], which offers provable guarantees for the protection of individual training examples [14], [15] even when the data aggregator is not assumed to be trusted [16], [17].”
BackgroundAuthenticated Encryption: Relations among Notions and Analysis of the Generic Composition Paradigm
Bellare, M., & Namprempre, C. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-44448-3_41 · Cited by 879
Revisiting Distributed Synchronous SGD
Pan, X., Chen, J., Monga, R., Bengio, S., & Józefowicz, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.05800 · Cited by 548 · cited 1×
“…consists of finding parameters Θ that achieve small L f ( D , Θ ) , typically by iterating a variant of a minibatch stochastic gradient descent rule [21], [22]: where Θ t are the parameters after iteration t , D t ⊆ D is a randomly selected subset of the training examples, and η is a learning rate…”
BackgroundSecure Multiparty Computation Goes Live
Bogetoft, P., Christensen, D. L., Damgård, I., Geisler, M., Jakobsen, T., Krøigaard, M., Nielsen, J. D., Nielsen, J. B., Nielsen, K., Pagter, J., et al. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03549-4_20 · Cited by 539 · cited 1×
“To overcome the constraints of client devices, some previous work has suggested that clients distribute their trust across multiple non-colluding servers, and this has been deployed in real-world applications [11].”
BackgroundDifferentially private aggregation of distributed time-series with transformation and encryption
Rastogi, V., & Nath, S. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807247 · Cited by 534 · cited 1×
“Schemes based on threshold additively-homomorphic cryptosystems (e.g. the Paillier cryptosystem [35]) can handle client dropouts, but are either computationally expensive, or require additional trust assumptions.”
BackgroundThe Oracle Diffie-Hellman Assumptions and an Analysis of DHIES
Abdalla, M., Bellare, M., & Rogaway, P. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-45353-9_12 · Cited by 445 · cited 4×
“2ODH-ExpbG,M (k): (1) (G′,д,q,H )← G(k) (2) a ← Zq ;A← дa (3) b ← Zq ;B ← дb (4) if b = 1, s ← H (дab ), else s $← {0,1}k (5) MOa (·),Ob (·)(G′,д,q,H ,A,B,s)→ b ′ (6) Output 1 if b = b ′, 0 o/w. where Oa (X ) returnsH (Xa ) on anyX ̸= B (and an error on input B) and similarly Ob (X ) returns H (Xb ) on any X ̸= A.”
MethodInfluentialHigh-Throughput Semi-Honest Secure Three-Party Computation with an Honest Majority
Araki, T., Furukawa, J., Lindell, Y., Nof, A., & Ohara, K. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978331 · Cited by 405 · cited 1×
“recently presented a generic three-party computation protocol that achieves very high throughput [5].”
BackgroundI Have a DREAM! (DiffeRentially privatE smArt Metering)
Ács, G., & Castelluccia, C. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24178-9_9 · Cited by 388 · cited 5×
“This is particular advantageous in the local privacy setting [13], which offers provable guarantees for the protection of individual training examples [14], [15] even when the data aggregator is not assumed to be trusted [16], [17].”
BackgroundMethodInfluential(2010) · Cited by 291
Computational Differential Privacy
Mironov, I., Pandey, O., Reingold, O., & Vadhan, S. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03356-8_8 · Cited by 231
Duchi, J. C., Jordan, M. I., & Wainwright, M. J. (2014) · Journal of the ACM · doi:10.1145/2666468 · Cited by 231
Privacy-Preserving Stream Aggregation with Fault Tolerance
Chan, T. H. H., Shi, E., & Song, D. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-32946-3_15 · Cited by 202 · cited 1×
“[14] use an approach similar to ours, but in the exponent in some group”
MethodSecure Computation on the Web: Computing without Simultaneous Interaction
Halevi, S., Lindell, Y., & Pinkas, B. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22792-9_8 · Cited by 177 · cited 1×
“Halevi, Lindell and Pinkas [36] present a protocol that uses homomorphic encryption to securely compute the sum in just one round of interaction between the server and each of the clients (assuming a PKI is already in place).”
BackgroundDining Cryptographers Revisited
Golle, P., & Juels, A. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24676-3_27 · Cited by 168 · cited 1×
“Dining cryptographers networks, or DC-nets, are a type of communication network which provide anonymity by using pairwise blinding of inputs [29], [30], similarly to our secure aggregation protocol.”
MethodLanguage modeling for soft keyboards
Goodman, J., Venolia, G., Steury, K., & Parker, C. (2002) · Proceedings of the 7th international conference on Intelligent user interfaces · doi:10.1145/502716.502753 · Cited by 161 · cited 1×
“Such models are commonly used to to improve typing efficacy for a phone’s on-screen keyboard [19].”
MethodZero-knowledge using garbled circuits: how to prove non-algebraic statements efficiently
Jawurek, M., Kerschbaum, F., & Orlandi, C. (2013) · Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security - CCS '13 · doi:10.1145/2508859.2516662 · Cited by 146 · cited 1×
“Unfortunately, even using the best-known garbled circuit techniques [35], even one such proof would be more costly than the entire protocol.”
BackgroundAdvances in nowcasting influenza-like illness rates using search query logs
Lampos, V., Miller, A. C., Crossan, S., & Stefansen, C. (2015) · Scientific Reports · doi:10.1038/srep12760 · Cited by 143 · cited 1×
“Machine learned models trained on sensitive real-world data promise improvements to everything from medical screening [1] to disease outbreak discovery [2].”
BackgroundPrio: Private, Robust, and Scalable Computation of Aggregate Statistics
Corrigan-Gibbs, H., & Boneh, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06255 · Cited by 140 · cited 2×
“The recently presented Prio system of Gibbs and Boneh [16] is, from the perspective of the client devices, noninteractive, and the computation among the servers is very lightweight.”
MethodA Comprehensive Comparison of Multiparty Secure Additions with Differential Privacy
Goryczka, S., & Xiong, L. (2017) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2015.2484326 · Cited by 139 · cited 3×
“suggested by [18], are the most closely related to our scheme,”
BackgroundMethodPaparrizos, J., White, R. W., & Horvitz, E. (2016) · Journal of Oncology Practice · doi:10.1200/jop.2015.010504 · Cited by 134 · cited 1×
“Machine learned models trained on sensitive real-world data promise improvements to everything from medical screening [1] to disease outbreak discovery [2].”
BackgroundEfficient Constant Round Multi-party Computation Combining BMR and SPDZ
Lindell, Y., Benny Pinkas, B., Smart, N. P., & Yanai, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-662-48000-7_16 · Cited by 93
Wainwright, M. J., Jordan, M. I., & Duchi, J. C. (2012) · neural information processing systems · Cited by 86
Large-Scale Secure Computation: Multi-party Computation for (Parallel) RAM Programs
Boyle, E., Chung, K.-M., & Pass, R. (2015) · Lecture notes in computer science · doi:10.1007/978-3-662-48000-7_36 · Cited by 72 · cited 1×
“studied generic MPC at such scale, relying on a particular ORAM construction to help localize the computation and avoid broadcasts [12].”
BackgroundJansen, R., & Johnson, A. (2016) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2976749.2978310 · Cited by 71 · cited 2×
“Pairwise blinding using additive stream ciphers has been explored in previous work [15], [18], [33], [34], and deal with dropouts/ failures in different ways.”
BackgroundMethodPrivEx: Private Collection of Traffic Statistics for Anonymous Communication Networks
Elahi, T., Danezis, G., & Goldberg, I. (2014) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2660267.2660280 · Cited by 70 · cited 2×
“Pairwise blinding using additive stream ciphers has been explored in previous work [15], [18], [33], [34], and deal with”
BackgroundMethodProactively Accountable Anonymous Messaging in Verdict
Corrigan-Gibbs, H., Wolinsky, D. I., & Ford, B. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1209.4819 · Cited by 68 · cited 1×
“Recent research has examined increasing the efficiency of DC-nets protocols and allowing them to operate in the presence of malicious users [31].”
BackgroundPrivacy Aware LearningPreprint
Duchi, J. C., Jordan, M. I., & Wainwright, M. J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1210.2085 · Cited by 61
The IPS Compiler: Optimizations, Variants and Concrete Efficiency
Lindell, Y., Oxman, E., & Benny Pinkas, B. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22792-9_15 · Cited by 53 · cited 7×
“MPC protocols based on secret sharing, however, can extend to hundreds of users.”
BackgroundInfluentialPrivate and Dynamic Time-Series Data Aggregation with Trust Relaxation
Leontiadis, I., Elkhiyaoui, K., & Molva, R. A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12280-9_20 · Cited by 52 · cited 1×
“the Paillier cryptosystem [40, 49]) can handle client dropouts, but are either computationally expensive or require additional trust assumptions.”
BackgroundPUDA – Privacy and Unforgeability for Data Aggregation
Leontiadis, I., Elkhiyaoui, K., Önen, M., & Molva, R. A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-26823-1_1 · Cited by 41 · cited 1×
“[41] calls for a trusted dealer to set up the keys.”
BackgroundJoint Training of Deep Boltzmann MachinesPreprint
Goodfellow, I., Courville, A. C., & Bengio, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1212.2686 · Cited by 25
Riffle : an efficient communication system with strong anonymity
Kwon, Y. H. (2015) · Cited by 24
De-anonymizing South Korean Resident Registration Numbers Shared in Prescription Data
연구, 처. 데. 주. 익. 해., Sweeney, P. L., & Yoo, J. S. (2015) · Cited by 18
McCabe, K. E. (2013) · Journal of Intellectual Property Law · Cited by 5
Research works printed but not linked: 4
- Stephen Advokat. 1987. Publication Of Bork's Video Rentals Raises Privacy Issue. Chicago Tribune (1987).
- Michael Barbaro, Tom Zeller, and Saul Hansell. 2006. A face is exposed for AOL searcher no. 4417749. New York Times 9, 2008 (2006).
- Jakub KonečnỴ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016). · arXiv
- Elaine Shi, HTH Chan, Eleanor Rieffel, Richard Chow, and Dawn Song. 2011. Privacy-preserving aggregation of time-series data. In Annual Network & Distributed System Security Symposium (NDSS). Internet Society.
Cited by
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772
“[73], effectively handling dropping out participants at the cost of complexity of the protocol.”
BackgroundInfluentialSelf-citationFederated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322
“SMC provides formal privacy proof for these protocols [25].”
BackgroundInfluentialFederated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
“quires that the updates are also private to the server. Current works that aim to improve the privacy of federated learning typically build upon previous classical cryptographic protocols such as SMC [10, 42] and differential privacy [3, 8, 41, 76]. Bonawitz et al. [10] introduce an SMC protocol to protect individual…”
BackgroundFederated Learning: Strategies for Improving Communication Efficiency
Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016) · arXiv · doi:10.48550/arxiv.1610.05492 · Cited by 3,766
Self-citationFederated Learning in Mobile Edge Networks: A Comprehensive Survey
Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y.-C., Yang, Q., Niyato, D., & Miao, C. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986024 · Cited by 2,565
“Although both the encryption techniques presented in [161] and [79] can prevent the curious server from extracting information, they require multi-round communications and cannot preclude collusions between the server and participants.”
Background
Metadata
- Authors
- Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., & Seth, K.
- Affiliations
- Google (United States) (US)
- Cornell Tech (US)
- Venue
- Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2017-10-30
- Volume
- 2017
- Pages
- 1175-1191
- ISSN
- 1543-7221 (ISSN-L)
- OpenAlex
- W2767079719
- Cited by
- 3,956
- References
- 55
- Citations by year
- 2026: 701 · 2025: 712 · 2024: 643 · 2023: 564 · 2022: 400 · 2021: 436 · 2020: 299 · 2019: 152 · 2018: 42 · 2017: 7
- FWCI
- 62.37
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
- IACR Cryptology ePrint Archive (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Cryptography and Data Security
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Information Systems › Cloud Data Security Solutions
- Keywords
- secure aggregation, privacy-preserving machine learning, federated learning, high-dimensional data, communication efficiency, user dropout, honest-but-curious adversary, active adversaries, secure multi-party computation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.