Decision Making in Multiagent Systems: A Survey
The DOI matches its Crossref record; 263 of the 275 references it cites were found in a registry or an index.
Abstract
Intelligent transport systems, efficient electric grids, and sensor networks for data collection and analysis are some examples of the multiagent systems (MAS) that cooperate to achieve common goals. Decision making is an integral part of intelligent agents and MAS that will allow such systems to accomplish increasingly complex tasks. In this survey, we investigate state-of-the-art work within the past five years on cooperative MAS decision making models, including Markov decision processes, game theory, swarm intelligence, and graph theoretic models. We survey algorithms that result in optimal and suboptimal policies such as reinforcement learning, dynamic programming, evolutionary computing, and neural networks. We also discuss the application of these models to robotics, wireless sensor networks, cognitive radio networks, intelligent transport systems, and smart electric grids. In addition, we define key terms in the area and discuss remaining challenges that include incorporating big data advancements to decision making, developing autonomous, scalable and computationally efficient algorithms, tackling more complex tasks, and developing standardized evaluation metrics. While recent surveys have been published on this topic, we present a broader discussion of related models and applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The objective is to maximize the system's overall reward, while agents work together to accomplish a common task [54].”
From Leveraging 3GPP Features and Optimization Techniques for 5G NR-V2X Resource Allocation: A Survey (2025) · cites this paper as [54]
Citations
Cites
Items 201–263 of 263
Approximate solutions for factored Dec-POMDPs with many agents
Oliehoek, F. A., Whiteson, S., & Spaan, M. T. J. (2013) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/hjoq6869 · Cited by 26 · cited 1×
“A set of approximate inferences and heuristics including bootstrapping were used to find approximate solutions to Dec-POMDP in [96].”
MethodDynamic Programming Approximations for Partially Observable Stochastic Games
Kumar, A., & Zilberstein, S. (2009) · Cited by 24
Parallel Swarm Intelligence for VANETs Optimization
Toutouh, J., & Alba, E. (2012) · International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC) · doi:10.1109/3pgcic.2012.53 · Cited by 24
A Universal Criteria Catalog for Evaluation of Heterogeneous Agent Development Artifacts
Braubach, L., Pokahr, A., & Lamersdorf, W. (2008) · Cited by 23
Efficient distributed algorithm of dynamic task assignment for swarm robotics
de Mendonça, R. M., Nedjah, N., & Mourelle, L. D. M. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.05.117 · Cited by 22 · cited 1×
“It has been applied to dynamic task allocation [161], distributed localization problems [162], foraging tasks [163]–[165], collision free navigation [166], [167], and communication free flocking with minimal memory requirements [168].”
Particle swarm optimization applied to integrated demand response resources scheduling
Faria, P., Vale, Z., Soares, J., & Ferreira, J. (2011) · IEEE Symposium on Computational Intelligence Applications in Smart Grid (CIASG) · doi:10.1109/ciasg.2011.5953326 · Cited by 22
Reinforcement learning based routing in wireless mesh networks
Boushaba, M., Hafid, A. S., Belbekkouche, A., & Michel Gendreau, A. (2013) · Wireless Networks · doi:10.1007/s11276-013-0592-y · Cited by 21 · cited 1×
“used for routing algorithms [181]–[183], adaptive broadcasting [184], adaptive data collection [185], and traffic signal control [186]–[193].”
MethodMurray, C., & Gordon, G. J. (2006) · Defense Technical Information Center (DTIC) · doi:10.21236/ada462577 · Cited by 21
Characterization and Evaluation of Multi-agent System Architectural Styles
Davidsson, P., Johansson, S. J., & Svahnberg, M. (2006) · Lecture notes in computer science · doi:10.1007/11738817_11 · Cited by 20 · cited 1×
“Their evaluation criteria include domain specific performance metrics and domain invariant criteria such as time and space complexity, load balancing, fairness, resource utilization, communication overhead, robustness, and scalability [38].”
BackgroundStick-Breaking Policy Learning in Dec-POMDPsPreprint
Liu, M., Amato, C., Liao, X., Carin, L., & How, J. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00274 · Cited by 19 · cited 1×
“Approximate solutions have been proposed based on bounded policy iteration [90], Q-value function methods [91], multiagent A∗ [81], genetic algorithms [82], DP [92]–[94], and a Bayesian learning, stick-breaking policy algorithm [95].”
MethodAnalysis of an evolutionary reinforcement learning method in a multiagent domain
Metzen, J. H., Edgington, M., Kassahun, Y., & Kirchner, F. (2008) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/1402383.1402428 · Cited by 19
Survey of task allocation in multi Agent systems
Yong-lin, L. (2010) · Systems engineering and electronics · Cited by 19
A particle filtering based approach to approximating interactive POMDPs
Doshi, P., & Gmytrasiewicz, P. J. (2005) · Cited by 18
Li, B., & Si, J. (2010) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2050334 · Cited by 18
Rock–Scissors–Paper and evolutionarily stable strategies
Loertscher, S. (2012) · Economics Letters · doi:10.1016/j.econlet.2012.12.017 · Cited by 18 · cited 1×
“Stochastic games have been shown to have an evolutionary stable strategy under certain conditions [114].”
BackgroundCognitive security protocol for sensor based VANET using swarm intelligence
Muraleedharan, R., & Osadciw, L. A. (2009) · 2018 52nd Asilomar Conference on Signals, Systems, and Computers · doi:10.1109/acssc.2009.5470101 · Cited by 17 · cited 1×
“information dissemination [198]–[202].”
Exploiting the Rule Structure for Decision Making within the Independent Choice LogicPreprint
Poole, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1302.4978 · Cited by 17 · cited 2×
“IDs require the optimization of all parent nodes of a decision variable [34].”
BackgroundInvestigating Open Issues in Swarm Intelligence for Mitigating Security Threats in MANET
Krishnappa, P. K., & Babu, B. R. P. (2015) · International Journal of Electrical and Computer Engineering (IJECE) · doi:10.11591/ijece.v5i5.pp1194-1201 · Cited by 16 · cited 1×
“selection [216], for security protocols [215], [217], and node positioning and localization [218], [219].”
BackgroundSen, S. D., & Adams, J. A. (2014) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-014-9276-y · Cited by 16 · cited 1×
“IDs solved the problem of coalition formation for complex real-world missions by selecting a subset of coalition formation algorithms suitable for the problem at hand [175].”
BackgroundSolving decentralized POMDP problems using genetic algorithms
Eker, B., & Akın, H. L. (2012) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-012-9204-y · Cited by 16 · cited 2×
“Approximate solutions have been proposed based on bounded policy iteration [90], Q-value function methods [91], multiagent A∗ [81], genetic algorithms [82], DP [92]–[94], and a Bayesian learning, stick-breaking policy algorithm [95].”
MethodO’Grady, R., Pinciroli, C., Groß, R., Christensen, A. L., Francesco Mondada, C., Michael Bonani, C., & Dorigo, M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21283-3_21 · Cited by 15
Optimal control in Markov decision processes via distributed optimization
Fu, J., Han, S., & Topcu, U. (2015) · Proceedings of the IEEE Conference on Decision & Control, including the Symposium on Adaptive Processes · doi:10.1109/cdc.2015.7403398 · Cited by 15 · cited 1×
“Then, a distributed Newton method [66] or linear programming algorithm [67] find the optimal policy.”
MethodFlesch, J., Parthasarathy, T., Thuijsman, F., & Uyttendaele, P. (2012) · Dynamic Games and Applications · doi:10.1007/s13235-012-0059-2 · Cited by 14 · cited 1×
“Evolutionary stable strategy is a refinement of the Nash equilibrium which requires a strategy to be stable to any perturbations that may occur to the games as they evolve [112] and was extended to stochastic games [113].”
BackgroundThe Infinite Regionalized Policy Representation
Liu, M., Liao, X., & Carin, L. (2011) · International Conference on Machine Learning · Cited by 14
Approximate Solutions of Interactive Dynamic Influence Diagrams Using Model Clustering
Zeng, Y., Doshi, P., & Chen, Q. (2007) · VBN Forskningsportal (Aalborg Universitet) · Cited by 13
Coalition formation based on marginal contributions and the Markov process
Liao, S. S., Zhang, J.-D., Lau, R. Y. K., & Wu, T. (2013) · Decision Support Systems · doi:10.1016/j.dss.2013.09.019 · Cited by 13 · cited 1×
“butions [173].”
Least-squares temporal difference learning based on an extreme learning machine
Escandell-Montero, P., Martínez-Martínez, J. M., Martín‐Guerrero, J. D., Soria‐Olivas, E., & Gómez‐Sanchís, J. (2014) · Neurocomputing · doi:10.1016/j.neucom.2013.11.040 · Cited by 13 · cited 1×
“learning [48]–[50], policy search [51], and linear programming [52], [53] require models of the state transition and reward functions.”
BackgroundElsayed, S. M., Sarker, R. A., & Essam, D. (2015) · International Journal of Computational Intelligence and Applications · doi:10.1142/s146902681550025x · Cited by 13 · cited 2×
“Swarm intelligence is a popular approach to intrusion detection, evidenced by the recently published surveys [25]–[27].”
BackgroundAn Evaluation Method for Multi-Agent Systems
Bitonto, P. D., Laterza, M., Roselli, T., & Rossano, V. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13480-7_5 · Cited by 12 · cited 1×
“[259] developed a hier-”
Multi-agent Reinforcement Learning for Traffic Signal Control
Prabuchandran, K., Hemanth Kumar, A. N., & Bhatnagar, S. (2014) · NOT FOUND REPOSITORY (Indian Institute of Science Bangalore) · Cited by 12
Ni, J., Liu, L., Liu, C., Hu, X., & Li, S. (2017) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2017.7963044 · Cited by 11 · cited 1×
“For example, a Lyapunov-based voltage and frequency controller was designed for micro-grid systems that only requires local communication among neighbors [137] and a secondary voltage distributed controller based on input– output feedback linearization that requires sparse communication [138].”
BackgroundSolving Multi-agent Decision Problems Modeled as Dec-POMDP: A Robot Soccer Case Study
Aşık, O., & Akın, H. L. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39250-4_13 · Cited by 10 · cited 1×
“MDPs [150] and POMDP [151]–[155] have been used for robotics coordination including robot soccer [156].”
MethodA novel sleep scheduling scheme in green wireless sensor networks
Zhang, J., Xu, L., Zhou, S., & Ye, X. (2014) · The Journal of Supercomputing · doi:10.1007/s11227-014-1354-z · Cited by 10 · cited 1×
“for performance optimization [204], data and memory access control [205], and sleep scheduling [206].”
BackgroundA bee-inspired approach for information dissemination in VANETs
Medetov, S., Bakhouya, M., Gaber, J., Zine-Dine, K., & Wack, M. (2014) · doi:10.1109/icmcs.2014.6911290 · Cited by 10 · cited 1×
“information dissemination [198]–[202].”
Planning for weakly-coupled partially observable stochastic games
Guo, A., & Lesser, V. (2005) · International Joint Conference on Artificial Intelligence · Cited by 9
Metrics for Multiagent Systems
Lass, R. N., Sultanik, E. A., & Regli, W. C. (2009) · Springer eBooks · doi:10.1007/978-1-4419-0492-8_1 · Cited by 8 · cited 1×
“[258] distinguished between two metric categories: 1) effectiveness (e.”
BackgroundVehicular Ad Hoc Network Mobility Models Applied for Reinforcement Learning Routing Algorithm
Kulkarni, S. A., & Rao, G. R. (2010) · Communications in computer and information science · doi:10.1007/978-3-642-14825-5_20 · Cited by 8 · cited 1×
“used for routing algorithms [181]–[183], adaptive broadcasting [184], adaptive data collection [185], and traffic signal control [186]–[193].”
MethodCollings, J., & Kim, E. (2014) · IEEE Symposium on Swarm Intelligence (SIS) · doi:10.1109/sis.2014.7011805 · Cited by 8 · cited 1×
“Distributed implementations of ACO [126], [127], and PSO [128] have been developed to speedup convergence.”
MethodVehicular Adhoc Networks Handovers with Metaheuristic Algorithms
Wararkar, P., & Dorle, S. S. (2014) · doi:10.1109/icesc.2014.33 · Cited by 8
TYPES AND PRIORITIES OF MULTI-AGENT SYSTEM INTERACTIONS
Ngobye, M., GROOT, W. T. D., van der Weide, T. P., & C, J. C. (2010) · RePEc: Research Papers in Economics · Cited by 7
Distributed vs. Centralized Particle Swarm Optimization for Learning Flocking Behaviors
Navarro, I., Mario, E. D., & Martino, A. (2015) · doi:10.7551/978-0-262-33027-5-ch056 · Cited by 7 · cited 5×
“PSO has been widely applied in combination with support vector machines [225]–[227] and linear programming [228].”
MethodInfluentialWang, W., Huang, J., & Wen, C. (2016) · International Conference on Control, Automation, Robotics and Vision (ICARCV) · doi:10.1109/icarcv.2016.7838696 · Cited by 7 · cited 1×
“Distributed consensus tracking was achieved by designing: distributed adaptive controllers in weakly connected, directed graphs [140], a distributed optimal control algorithm [141], and nonlinear distributed impulsive control (control signals are given as impulses instead of continuously) with delayed”
MethodA Scalable Model-Free Recurrent Neural Network Framework for Solving POMDPs
Liu, Z., & Elhanany, I. (2007) · doi:10.1109/adprl.2007.368178 · Cited by 6
Hemmati, M., Yassine, A., & Shirmohammadi, S. (2015) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2015.170 · Cited by 6 · cited 1×
“Adopted models include POMDP, used to minimize network bandwidth congestion and fairly allocate resource [245], Bayesian games [246] and stochastic games with multiagent RL used for job, and resource scheduling in grid computing [247].”
MethodGPU Implementation of Food-Foraging Problem for Evolutionary Swarm Robotics Systems
Ohkura, K., Yasuda, T., Matsumura, Y., & Kadota, M. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-09952-1_22 · Cited by 5 · cited 1×
“It has been applied to dynamic task allocation [161], distributed localization problems [162], foraging tasks [163]–[165], collision free navigation [166], [167], and communication free flocking with minimal memory requirements [168].”
RLAB: Reinforcement Learning-Based Adaptive Broadcasting for Vehicular Ad-Hoc Networks
Hosseininezhad, S., Shirazi, G. N., & Leung, V. C. M. (2011) · IEEE Vehicular Technology Conference · doi:10.1109/vetecs.2011.5956659 · Cited by 5
Solving DEC-POMDPs by Expectation Maximization of Value Function
Song, Z., Liao, X., & Carin, L. (2016) · National Conference on Artificial Intelligence · Cited by 5
Probabilistic Planning for Decentralized Multi-Robot Systems.
Amato, C., Konidaris, G., Omidshafiei, S., Agha–mohammadi, A., How, J. P., & Kaelbling, L. P. (2015) · National Conference on Artificial Intelligence · Cited by 5
A game theoretical formulation for distributed optimization problems
Zhang, J., Qi, D., & Zhao, G. (2013) · International Conference on Control and Automation · doi:10.1109/icca.2013.6564947 · Cited by 4 · cited 1×
“In [251], potential games and cooperative control provided a theoretical framework to formulate distributed optimization problems.”
BackgroundKovács, D. L., Li, W., Fukuta, N., & Watanabe, T. (2012) · International Conference on Advanced Information Networking and Applications (AINA) · doi:10.1109/aina.2012.17 · Cited by 4 · cited 1×
“for performance optimization [204], data and memory access control [205], and sleep scheduling [206].”
BackgroundA Comprehensive Survey of Nature-Inspired Routing Protocols
Farooq, M. (2008) · Natural computing series · doi:10.1007/978-3-540-85954-3_2 · Cited by 4
Dynamic Metrics for Multi-agent Systems Using Aspect-Oriented Programming
Marir, T., Mokhati, F., Bouchelaghem-Seridi, H., & Benaissa, B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45889-2_5 · Cited by 4
Distributed lifelong reinforcement learning with sub-linear regret
Tutunov, R., Zini, J. E., Ammar, H. B., & Jadbabaie, A. (2017) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc.2017.8263978 · Cited by 4 · cited 1×
“Then, a distributed Newton method [66] or linear programming algorithm [67] find the optimal policy.”
MethodBiologically inspired multi-robot foraging
Alers, S., Claes, D., Tuyls, K., & Weiß, G. (2014) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/xdpk9778 · Cited by 4
Learning States Representations in POMDPPreprint
Contardo, G., Denoyer, L., Artières, T., & Gallinari, P. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6042 · Cited by 4 · cited 1×
“resentation space where policies are learned, in [88].”
BackgroundKnowledge-based recurrent neural networks in Reinforcement Learning
Dung, L. T., Komeda, T., & Takagi, M. (2007) · International Conference on Artificial Intelligence and Soft Computing · Cited by 3
Network Intrusion Detection Using Support Vector Machine Based on Particle Swarm Optimization
Wang, L., Dong, C., Hu, J., & Li, G. (2015) · Advances in engineering research/Advances in Engineering Research · doi:10.2991/asei-15.2015.125 · Cited by 3
Towards efficient online reinforcement learning using neuroevolution
Metzen, J. H., Kirchner, F., Edgington, M., & Kassahun, Y. (2008) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1389095.1389371 · Cited by 3
Swarm intelligence algorithm for optimality discovery in distributed constraint optimization
Hasegawa, K., & Noto, M. (2014) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2014.6974490 · Cited by 3 · cited 6×
“PSO has been widely applied in combination with support vector machines [225]–[227] and linear programming [228].”
MethodInfluentialYang, R., Zhang, H., Yan, H., & Yang, F. (2016) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon.2016.7793580 · Cited by 3 · cited 1×
“[145] proposed distributed output regularization using adaptive control in MAS with”
MethodMurray, C., & Gordon, G. J. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0130 · Cited by 2 · cited 1×
“POSG has been applied to multiple problems in robotics [148], [149].”
MethodSHP-VI Method of Solving DEC-POMDP Problem
Wan, X. P., & Li, S. Y. (2014) · Advanced materials research · doi:10.4028/www.scientific.net/amr.926-930.3245 · Cited by 1 · cited 1×
“Many algorithms have been proposed in the literature to provide exact and approximate solutions for the POMDP and its variants, including value iteration [71], [72], expectation maximization [73], [74],”
MethodYang, R., Zhang, H., Yan, H., & Yang, F. (2016) · IEEE Conference Proceedings · Cited by 1
Research works printed but not linked: 11
- 2015, First Workshop on Cognitive Architectures
- altman, 1999, Constrained Markov Decision Processes, 7
- amato, 2013, Scalable Bayesian reinforcement learning for multiagent POMDPs, Multidisciplinary Conf on Reinforcement Learning and Decision Making, 188
- sandhu, 2004, Wireless sensor networks for commercial lighting control: Decision making with multi-agent systems, Proc of the AAAI-04 Workshop on Sensor Networks, 10, 131
- strikos, 2007, A Full Approach for Intrusion Detection in Wireless Sensor Networks
- melo, 2012, Decentralized multiagent planning for balance control in smart grids, Proc 1st Int Workshop Inf Technol Energy Appl, 923, 21
- 2016, An Internet of Things That Will Number Ten Billions
- 2015, Gartner Says 6 4 Billion Connected “Things” Will Be in Use in 2016 Up 30 Percent From 2015
- lauri, 2013, Managing power flows in microgrids using multi-agent reinforcement learning, Proc Agent Technol Energy Syst, 1
- emery, 2005, Game Theoretic Control for Robot Teams
- teodorovi?, 2005, Bee colony optimization—A cooperative learning approach to complex transportation problems, Proc 16th Mini Eur Conf 10th Meeting Adv OR AI Methods Transport (EWGT), 51
Cited by
Blockchain for AI: review and open research challenges
Salah, K., Rehman, M. H. U., Nizamuddin, N., & Al-Fuqaha, A. (2019) · IEEE Access · doi:10.1109/access.2018.2890507 · Cited by 1,001
“ing inter-layer and intralayer operability across entire systems [50].”
BackgroundCooperative Heterogeneous Multi-Robot Systems: A Survey
Rizk, Y., Awad, M., & Tunstel, E. W. (2020) · ACM Computing Surveys · doi:10.1145/3303848 · Cited by 449
Self-citationAgbehadji, I. E., Awuzie, B. O., Ngowi, A. B., & Millham, R. C. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17155330 · Cited by 248
Baccour, E., Mhaisen, N., Abdellatif, A. A., Erbad, A., Mohamed, A., Hamdi, M., & Guizani, M. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3200740 · Cited by 186
“Other papers surveyed the single agent and multi-agents RL, such as [75], [76], [77], [78], [79].”
BackgroundTheoretical approaches to AI in supply chain optimization: Pathways to efficiency and resilience
Abaku, E. A., Edunjobi, T. E., & Odimarha, A. C. (2024) · International Journal of Science and Technology Research Archive · doi:10.53771/ijstra.2024.6.1.0033 · Cited by 188
“Multi-agent systems (MAS) represent another approach to decentralized decision-making and coordination, where autonomous agents interact with each other and their environment to achieve individual and collective objectives (Rizk et al ., 2018).”
Background
Metadata
- Authors
- Rizk, Y., Awad, M., & Tunstel, E. W.
- Affiliations
- American University of Beirut (LB)
- Venue
- IEEE Transactions on Cognitive and Developmental Systems
- Published
- 2018-09
- Volume
- 10
- Issue
- 3
- Pages
- 514-529
- ISSN
- 2379-8920 (ISSN-L)
- 2379-8939
- OpenAlex
- W2803208334
- Cited by
- 219
- References
- 275
- Citations by year
- 2026: 18 · 2025: 29 · 2024: 34 · 2023: 39 · 2022: 35 · 2021: 29 · 2020: 21 · 2019: 13 · 2018: 1
- FWCI
- 8.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Cognitive and Developmental Systems (publishedVersion)
- Funders
- National Council of Scientific Research in Lebanon
- American University of Beirut
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Physical Sciences › Engineering › Control and Systems Engineering › Distributed Control Multi-Agent Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- multi-agent systems, cooperative decision-making, Markov decision process, game theory, swarm intelligence, graph model, reinforcement learning, dynamic programming, evolutionary computation, artificial neural networks, robotics, wireless sensor networks, cognitive radio networks, intelligent transportation systems, big data, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.