Algorithm Auditing: Managing the Legal, Ethical, and Technological Risks of Artificial Intelligence, Machine Learning, and Associated Algorithms
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Algorithms are becoming ubiquitous. However, companies are increasingly alarmed about their algorithms causing major financial or reputational damage. A new industry is envisaged: auditing and assurance of algorithms with the remit to validate artificial intelligence, machine learning, and associated algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For this reason, researchers like Koshiyama et al. (2022) have proposed procedures for auditing the entire process whereby AI systems are designed and deployed.”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Koshiyama et al. (2022)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“9,10 As with a financial audit, an algo-rithm audit is required to deliver formal assurance that algorithms are legal, ethical, and safe.”
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325
The Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911
The Ethics of AI Ethics -- An Evaluation of Guidelines
Hagendorff, T. (2019) · arXiv · doi:10.48550/arxiv.1903.03425 · Cited by 1,907
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
On Evaluating Adversarial Robustness
Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Mądry, A., & Kurakin, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06705 · Cited by 524
Countering Adversarial Images using Input Transformations
Guo, C., Rana, M., Cissé, M., & van der Maaten, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00117 · Cited by 394
InterpretML: A Unified Framework for Machine Learning Interpretability
Nori, H., Jenkins, S., Koch, P., & Caruana, R. (2019) · arXiv · doi:10.48550/arxiv.1909.09223 · Cited by 322
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306
Algorithmic recourse: From counterfactual explanations to interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445899 · Cited by 274
A High-Level Overview of AI Ethics
Kazim, E., & Koshiyama, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3609292 · Cited by 269
Self-citationA high-level overview of AI ethics
Kazim, E., & Koshiyama, A. S. (2021) · Patterns · doi:10.1016/j.patter.2021.100314 · Cited by 269
Self-citationAn Overview of Privacy in Machine LearningPreprint
Cristofaro, E. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.08679 · Cited by 153
Generalization in Adaptive Data Analysis and Holdout Reuse
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · arXiv · doi:10.48550/arxiv.1506.02629 · Cited by 152
Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages
Robertson, R. E., Lazer, D., & Wilson, C. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186143 · Cited by 139
Algorithmic Recourse: from Counterfactual Explanations to Interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2020) · arXiv · doi:10.48550/arxiv.2002.06278 · Cited by 76
The dangers of faulty, biased, or malicious algorithms requires independent oversight
Shneiderman, B. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618211113 · Cited by 73
Adversarial Attacks on Neural Network Policies
Huang, S., Papernot, N., Goodfellow, I., Duan, Y., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02284 · Cited by 59
Quality Measures and Assurance for AI (Artificial Intelligence) Software
Rushby, J. (1988) · NASA Technical Reports Server (NASA) · Cited by 47
Verification of Non-Linear Specifications for Neural NetworksPreprint
Qin, C., Krishnamurthy, Dvijotham, O’Donoghue, B., Bunel, R., Stanforth, R., Gowal, S., Uesato, J., Świrszcz, G., & Kohli, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09592 · Cited by 45
Achieving Verified Robustness to Symbol Substitutions via Interval Bound PropagationPreprint
Huang, P.-S., Stanforth, R., Welbl, J., Dyer, C., Yogatama, D., Gowal, S., Dvijotham, K., & Kohli, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01492 · Cited by 43
AI auditing and impact assessment: according to the UK information commissioner’s office
Kazim, E., Denny, D. M. T., & Koshiyama, A. (2021) · AI and Ethics · doi:10.1007/s43681-021-00039-2 · Cited by 39
Self-citationThe interrelation between data and AI ethics in the context of impact assessments
Kazim, E., & Koshiyama, A. (2021) · AI and Ethics · doi:10.1007/s43681-020-00029-w · Cited by 39
Self-citationTransferring Knowledge across Learning Processes
Flennerhag, S., Moreno, P. G., Lawrence, N. D., & Damianou, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01054 · Cited by 38
Koshiyama, A., Firoozye, N., & Treleaven, P. (2020) · Proceedings of the First ACM International Conference on AI in Finance · doi:10.1145/3383455.3422539 · Cited by 38
Self-citationData-Informed Duties in AI Development
Pasquale, F. (2019) · Digital Commons at University of Maryland Carey Law (University of Maryland Francis King Carey School of Law) · Cited by 32
Sharma, S., Henderson, J., & Ghosh, J. (2019) · arXiv · doi:10.48550/arxiv.1905.07857 · Cited by 28
Towards better understanding of gradient-based attribution methods for Deep Neural NetworksPreprint
Ancona, M., Ceolini, E., Öztireli, C., et al. (2017) · Zurich Open Repository and Archive (University of Zurich) · doi:10.48550/arxiv.1711.06104 · Cited by 26
Algorithms in Future Capital MarketsPreprint
Koshiyama, A., Firoozye, N. B., & Treleaven, P. C. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3527511 · Cited by 15
Self-citationSoftware Certification Services: Encouraging Trust and Reasonable Expectations
Voas, J., & Miller, K. (2006) · IT Professional · doi:10.1109/mitp.2006.120 · Cited by 12
Human Centric AI: A Comment on the IEEE’s Ethically Aligned DesignPreprint
Kazim, E., & Koshiyama, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3575140 · Cited by 9
Self-citationSoftware product quality assurance
Ryan, J. R. (1982) · AFIPS National Computer Conference (NCC) · doi:10.1145/1500774.1500823 · Cited by 7
Research works printed but not linked: 59
- Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016, October). Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (pp. 308-318).
- Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., ... & De Freitas, N. (2016). Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems (pp. 3981-3989).
- Ateniese, G., Mancini, L. V., Spognardi, A., Villani, A., Vitali, D., & Felici, G. (2015). Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. International J of Security and Networks, 10(3), 137-150.
- Barabási, A. L. (2016). Network science. Cambridge university press.
- Barrett, C., & Tinelli, C. (2018). Satisfiability modulo theories. In Handbook of Model Checking (pp. 305-343). Springer, Cham.
- Bieker, F., Friedewald, M., Hansen, M., Obersteller, H., & Rost, M. (2016, September). A process for data protection impact assessment under the european general data protection regulation. In Annual Privacy Forum (pp. 21-37). Springer, Cham.
- Hall, P. (2019). An introduction to machine learning interpretability. O'Reilly Media, Incorporated.
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media.
- Hind, M., Mehta, S., Mojsilovic, A., Nair, R., Ramamurthy, K. N., Olteanu, A., & Varshney, K. R. (2018). Increasing trust in AI services through supplier's declarations of conformity. arXiv preprint arXiv:1808.07261. · arXiv
- Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017, October). Deep models under the GAN: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (pp. 603- 618). https://ico.org.uk/for-organisations/guide-to-data-protection/key-data-protection-themes/explaining-decisions-made- with-ai/
- Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I., & Tygar, J. D. (2011, October). Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence (pp. 43-58). ACM.
- Kaminski, M. E., & Malgieri, G. (2020, January). Multi-layered explanations from algorithmic impact assessments in the GDPR. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 68-79).
- Kamiran, F., & Calders, T. (2012). Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems, 33(1), 1-33.
- Kargupta, H., Datta, S., Wang, Q., & Sivakumar, K. (2005). Random-data perturbation techniques and privacy-preserving data mining. Knowledge and Information Systems, 7(4), 387-414.
- Buckman, J., Roy, A., Raffel, C., & Goodfellow, I. (2018, February). Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations.
- Bunel, R. R., Turkaslan, I., Torr, P., Kohli, P., & Mudigonda, P. K. (2018). A unified view of piecewise linear neural network verification. In Advances in Neural Information Processing Systems (pp. 4790-4799).
- Butterworth, M. (2018). The ICO and artificial intelligence: The role of fairness in the GDPR framework. Computer Law & Security Review, 34(2), 257-268.
- Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2), 153-163.
- Donini, M., Oneto, L., Ben-David, S., Shawe-Taylor, J. S., & Pontil, M. (2018). Empirical risk minimization under fairness constraints. In Advances in Neural Information Processing Systems (pp. 2791-2801).
- Dvijotham, K., Stanforth, R., Gowal, S., Mann, T. A., & Kohli, P. (2018, March). A Dual Approach to Scalable Verification of Deep Networks. In UAI (Vol. 1, p. 2).
- Dwork, C., & Naor, M. (2010). On the difficulties of disclosure prevention in statistical databases or the case for differential privacy. Journal of Privacy and Confidentiality, 2(1).
- Efron, B., & Hastie, T. (2016). Computer age statistical inference (Vol. 5). Cambridge University Press.
- Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015, August). Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 259-268).
- Finn, C., Abbeel, P., & Levine, S. (2017, August). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 1126-1135). JMLR. org.
- Floridi, L. (2018). Soft ethics, the governance of the digital and the General Data Protection Regulation. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2133), 20180081. Electronic copy available at: https://ssrn.com/abstract=3889612
- Fredrikson, M., Jha, S., & Ristenpart, T. (2015, October). Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (pp. 1322-1333).
- Fredrikson, M., Jha, S., & Ristenpart, T. (2015, October). Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (pp. 1322-1333).
- Ganju, K., Wang, Q., Yang, W., Gunter, C. A., & Borisov, N. (2018, January). Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (pp. 619-633).
- Goldsteen, A., Ezov, G., Shmelkin, R., Moffie, M., & Farkash, A. (2020). Data Minimization for GDPR Compliance in Machine Learning Models. arXiv preprint arXiv:2008.04113. · arXiv
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.
- Kim, H., Park, J., Bennis, M., & Kim, S. L. (2019). Blockchained on-device federated learning. IEEE Communications Letters.
- Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807. · arXiv
- Russell, S. J., & Norvig, P. (2016). Artificial intelligence: a modern approach. Malaysia; Pearson Education Limited.
- Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017, May). Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP) (pp. 3-18). IEEE.
- Steinhardt, J., Koh, P. W. W., & Liang, P. S. (2017). Certified defences for data poisoning attacks. In Advances in neural information processing systems (pp. 3517-3529).
- Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
- Tan, T. J. L., & Shokri, R. (2019). Bypassing backdoor detection algorithms in deep learning. arXiv preprint arXiv:1905.13409. · arXiv
- Taylor, S. (Ed.). (2014). Agent-based modeling and simulation. Springer.
- Tjeng, V., & Tedrake, R. (2017). Verifying neural networks with mixed integer programming. arXiv:1711.07356, 945-950. · arXiv
- Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016). Stealing machine learning models via prediction apis. In 25th USENIX Security Symposium (USENIX Security 16) (pp. 601-618).
- Treleaven, P., & Batrinca, B. (2017). Algorithmic regulation: automating financial compliance monitoring and regulation using AI and blockchain. Journal of Financial Transformation, 45, 14-21.
- Treleaven, P., Barnett, J., & Koshiyama, A. (2019). Algorithms: law and regulation. Computer, 52(2), 32-40.
- Zafar, M. B., Valera, I., Gomez-Rodriguez, M., & Gummadi, K. P. (2019). Fairness Constraints: A Flexible Approach for Fair Classification. J. Mach. Learn. Res., 20(75), 1-42.
- Zemel, R., Wu, Y., Swersky, K., Pitassi, T., & Dwork, C. (2013, February). Learning fair representations. In International Conference on Machine Learning (pp. 325-333).
- Zhang, B. H., Lemoine, B., & Mitchell, M. (2018, December). Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 335-340).
- Kusner, M. J., Loftus, J., Russell, C., & Silva, R. (2017). Counterfactual fairness. In Advances in neural information processing systems (pp. 4066-4076).
- Longo, L., Goebel, R., Lecue, F., Kieseberg, P., & Holzinger, A. (2020, August). Explainable Artificial Intelligence: Concepts, Applications, Research Challenges and Visions. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction (pp. 1-16). Springer, Cham.
- Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., & Zhang, G. (2018). Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12), 2346-2363.
- Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. In Advances in neural information processing systems (pp. 4765-4774).
- McGregor, L., Murray, D., and Ng, V. (2019). International human rights law as a framework for algorithmic accountability. International & Comparative Law Quarterly, 68(2), 309-343.
- Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019, May). Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy (SP) (pp. 691-706). IEEE.
- Meyer, M. (2014). Continuous integration and its tools. IEEE software, 31(3), 14-16.
- Moosavi-Dezfooli, S. M., Fawzi, A., & Frossard, P. (2016). Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2574-2582).
- Müller, R., Kornblith, S., & Hinton, G. E. (2019). When does label smoothing help?. In Advances in Neural Information Processing Systems (pp. 4694-4703).
- Neubauer, T., & Heurix, J. (2011). A methodology for the pseudonymization of medical data. International journal of medical informatics, 80(3), 190-204.
- Orekondy, T., Schiele, B., & Fritz, M. (2019). Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4954-4963).
- Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., & Weinberger, K. Q. (2017). On fairness and calibration. In Advances in Neural Information Processing Systems (pp. 5680-5689).
- Puiutta, E., & Veith, E. (2020). Explainable Reinforcement Learning: A Survey. arXiv preprint arXiv:2005.06247. · arXiv
- Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144).
Cited by
Auditing Large Language Models: A Three-Layered Approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023) · doi:10.2139/ssrn.4361607 · Cited by 292
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Fairness in recommender systems: research landscape and future directions
Deldjoo, Y., Jannach, D., Bellogin, A., Difonzo, A., & Zanzonelli, D. (2024) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09364-z · Cited by 228
Adopting artificial intelligence in sustainable business
Sipola, J., Saunila, M., & Ukko, J. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.139197 · Cited by 153
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“In many jurisdictions, this has meant drafting new legislation.”
Metadata
- Authors
- Koshiyama, A., Kazim, E., & Treleaven, P.
- Affiliations
- University College London (GB)
- Venue
- Computer
- Published
- 2022-04
- Volume
- 55
- Issue
- 4
- Pages
- 40-50
- ISSN
- 0018-9162 (ISSN-L)
- 1558-0814
- OpenAlex
- W4223475929
- Cited by
- 55
- References
- 13
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 17 · 2023: 20 · 2022: 3
- FWCI
- 13.61
- Type
- journal article
- Language
- en
- Open access
- green
- discovery.ucl.ac.uk
- Licence
- Locations
- Computer (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Cisco Research Center (2019-207109, 3696)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic auditing, artificial intelligence, machine learning, algorithmic risk, legal risk, ethical risks, reputational risk, financial risk
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.