The More You Know: Trust Dynamics and Calibration in Highly Automated Driving and the Effects of Take-Overs, System Malfunction, and System Transparency
Open accessThe DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.
Abstract
Objective: This paper presents a theoretical model and two simulator studies on the psychological processes during early trust calibration in automated vehicles. Background: The positive outcomes of automation can only reach their full potential if a calibrated level of trust is achieved. In this process, information on system capabilities and limitations plays a crucial role. Method: In two simulator experiments, trust was repeatedly measured during an automated drive. In Study 1, all participants in a two-group experiment experienced a system-initiated take-over, and the occurrence of a system malfunction was manipulated. In Study 2 in a 2 × 2 between-subject design, system transparency was manipulated as an additional factor. Results: Trust was found to increase during the first interactions progressively. In Study 1, take-overs led to a temporary decrease in trust, as did malfunctions in both studies. Interestingly, trust was reestablished in the course of interaction for take-overs and malfunctions. In Study 2, the high transparency condition did not show a temporary decline in trust after a malfunction. Conclusion: Trust is calibrated along provided information prior to and during the initial drive with an automated vehicle. The experience of take-overs and malfunctions leads to a temporary decline in trust that was recovered in the course of error-free interaction. The temporary decrease can be prevented by providing transparent information prior to system interaction. Application: Transparency, also about potential limitations of the system, plays an important role in this process and should be considered in the design of tutorials and human-machine interaction (HMI) concepts of automated vehicles.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This requires understanding how the system works and where its boundaries are (Körber et al., 2018), which can be achieved by user education (Körber et al., 2018) or by system transparency (Kraus et al., 2020).”
From Effects of Explainable Artificial Intelligence on trust and human behavior in a high-risk decision task (2023) · cites this paper as Kraus et al., 2020
In the citing paper
“Trust evolves and adapts over time with growing knowledge until, ideally, 'a user's trust in a system corresponds exactly with its objective capabilities and its actual performance' (Kraus et al., 2020, p. 719).”
From Effects of Explainable Artificial Intelligence on trust and human behavior in a high-risk decision task (2023) · cites this paper as Kraus et al., 2020
In the citing paper
“However, a high level of trust cannot be a goal in itself. Rather, an appropriate level of trust-so-called calibrated trust-that also corresponds with the competencies of the system should be achieved (e.g., Kraus et al., 2020).”
From Effects of Explainable Artificial Intelligence on trust and human behavior in a high-risk decision task (2023) · cites this paper as Kraus et al., 2020
In the citing paper
“and such trust may be dynamically adapted (Kraus et al., 2020).”
From Effects of Explainable Artificial Intelligence on trust and human behavior in a high-risk decision task (2023) · cites this paper as Kraus et al., 2020
In the citing paper
“Most commonly, the trust calibration was measured as the statistical diferences of trustworthiness perceptions of systems with high reliability and trustworthiness perceptions of systems with low-reliability groups, examined through analysis of variance (see, e.g., [4, 15, 26, 47, 58, 61, 68-70, 82, 84, 89, 90, 107, 117, 130, 140, 145]).”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [68]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The theory of planned behavior
Ajzen, I. (1991) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(91)90020-t · Cited by 90,587 · cited 3×
“In their conceptual model of trust in automation, Lee and See (2004) integrated concepts of trust from different domains with the theory of planned behavior (Ajzen, 1991; Ajzen & Fishbein, 1980; Fishbein & Ajzen, 1975).”
Background(1975) · PhilPapers (PhilPapers Foundation) · Cited by 36,737
(1980) · Medical Entomology and Zoology · Cited by 25,810
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 13×
“Otherwise, the interaction with automated vehicles will neither fully benefit from the advantages associated with automated driving (distrust) nor will it be safe for the driver or other road users (overtrust; Lee & See, 2004).”
BackgroundMethodInfluentialHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1×
“This situation is reflected by an absence of both disuse and misuse of the system (Parasuraman & Riley, 1997).”
BackgroundA model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction
Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y. C., de Visser, E. J., & Parasuraman, R. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811417254 · Cited by 2,051 · cited 3×
“In addition, it includes system, environmental, and individual factors influencing trust establishment (Hancock et al., 2011). A first key implication of the model is the prediction that trust is calibrated in accordance with information provided both prior to and during the interaction with an automated system (see a…”
BackgroundInfluentialFoundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799
Trust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459 · cited 4×
“If a malfunction did not corrupt trust to a degree that hinders further system usage, prolonged error-free operation was found to lead to remediation of trust in automated plant simulations (Lee & Moray, 1992) and automated robots (Desai, Kaniarasu, Medvedev, Steinfeld, & Yanco, 2013) over time.”
BackgroundMethodResultInfluentialSchmider, E., Ziegler, M., Danay, E., Beyer, L., & Bühner, M. (2010) · Methodology · doi:10.1027/1614-2241/a000016 · Cited by 1,290 · cited 1×
“MF– = no malfunction; LT = low transparency; MF+ = with malfunction; HT = high transparency. violation (Schmider et al., 2010) and thus are reported with corrected statistics. results”
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 6×
“In the same vein, it can be hypothesized that an automation that has limitations falls short of that expectation and should lead to a temporal reduction in trust in an automated system (e.g., Dzindolet et al., 2003; Madhavan & Wiegmann, 2007).”
BackgroundResultInfluentialContrast Analysis: Focused Comparisons in the Analysis of Variance.
Levin, J. R., Rosenthal, R., & Rosnow, R. L. (1987) · Journal of the American Statistical Association · doi:10.2307/2289419 · Cited by 1,251
MUIR, B. M., & MORAY, N. (1996) · Ergonomics · doi:10.1080/00140139608964474 · Cited by 993 · cited 4×
“Relevant beliefs for trust calibration include perceived predictability, dependability, faith, competence, and reliability of the system (Muir, 1994; Muir & Moray, 1996).”
BackgroundInfluentialFrom Here to Autonomy: Lessons Learned From Human–Automation Research
Endsley, M. R. (2017) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816681350 · Cited by 898
MUIR, B. M. (1994) · Ergonomics · doi:10.1080/00140139408964957 · Cited by 873 · cited 3×
“Relevant beliefs for trust calibration include perceived predictability, dependability, faith, competence, and reliability of the system (Muir, 1994; Muir & Moray, 1996).”
BackgroundTrust between humans and machines, and the design of decision aids
Muir, B. M. (1987) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(87)80013-5 · Cited by 762 · cited 4×
“Information fostering system transparency in advance or during the drive should facilitate trust calibration (Muir, 1987).”
BackgroundInfluential“Take over!” How long does it take to get the driver back into the loop?
Gold, C., Damböck, D., Lorenz, L., & Bengler, K. (2013) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213571433 · Cited by 754
Intention to use a fully automated car: Attitudes and a priori acceptability
Payre, W., Cestac, J., & Delhomme, P. (2014) · Transportation Research Part F: Traffic Psychology and Behaviour · doi:10.1016/j.trf.2014.04.009 · Cited by 695
Similarities and differences between human–human and human–automation trust: an integrative review
Madhavan, P., & Wiegmann, D. A. (2007) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220500337708 · Cited by 577 · cited 2×
“In the same vein, it can be hypothesized that an automation that has limitations falls short of that expectation and should lead to a temporal reduction in trust in an automated system (e.g., Dzindolet et al., 2003; Madhavan & Wiegmann, 2007).”
BackgroundMerritt, S. M., & Ilgen, D. R. (2008) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872008x288574 · Cited by 527 · cited 1×
“While in the beginning of the interaction—especially with new technologies—this assessment is vague, unstable, and prone to relatively fast dynamic changes as a function of new information (Merritt & Ilgen, 2008), it stabilizes in the course of interaction.”
BackgroundGold, C., Körber, M., Lechner, D., & Bengler, K. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816634226 · Cited by 440
Introduction matters: Manipulating trust in automation and reliance in automated driving
Körber, M., Baseler, E., & Bengler, K. (2018) · Applied Ergonomics · doi:10.1016/j.apergo.2017.07.006 · Cited by 418
Hergeth, S., Lorenz, L., Vilimek, R., & Krems, J. F. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720815625744 · Cited by 355 · cited 1×
“This is in line with earlier studies (e.g., Beggiato et al., 2015; Dzindolet et al., 2003; Hergeth et al., 2016; Lee & Moray, 1992).”
ResultGong, L. (2008) · Computers in Human Behavior · doi:10.1016/j.chb.2007.05.007 · Cited by 338
Beggiato, M., & Krems, J. F. (2013) · Transportation Research Part F: Traffic Psychology and Behaviour · doi:10.1016/j.trf.2012.12.006 · Cited by 327 · cited 1×
“A similar trust increase was found in studies assessing trust in adaptive cruise control (ACC) during the initial days (Kazi, Stanton, Walker, & Young, 2007) and weeks (Beggiato & Krems, 2013) of system interaction.”
BackgroundTrust and etiquette in high-criticality automated systems
Parasuraman, R., & Miller, C. A. (2004) · Communications of the ACM · doi:10.1145/975817.975844 · Cited by 311 · cited 1×
“It evolves and adapts over time along the accumulating knowledge of the user about a system (trust calibration, Lee & See, 2004; Parasuraman & Miller, 2004).”
BackgroundSituation Awareness-Based Agent Transparency
Chen, J. Y., Procci, K., Boyce, M., Wright, J., Garcia, A., & Barnes, M. (2014) · doi:10.21236/ada600351 · Cited by 293
Helldin, T., Falkman, G., Riveiro, M., & Davidsson, S. (2013) · Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications · doi:10.1145/2516540.2516554 · Cited by 286 · cited 1×
“Knowledge about situations leading to a TOR or entailing a higher risk for system malfunctions may reduce uncertainties and therefore help users to calibrate their trust more efficiently (e.g., Helldin et al., 2013) and use the system safely.”
BackgroundImpact of robot failures and feedback on real-time trust
Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A., & Yanco, H. (2013) · 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2013.6483596 · Cited by 234
Hergeth, S., Lorenz, L., & Krems, J. F. (2017) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816678714 · Cited by 221 · cited 1×
“This is in line with earlier studies (e.g., Beggiato et al., 2015; Dzindolet et al., 2003; Hergeth et al., 2016; Lee & Moray, 1992).”
ResultBeggiato, M., Pereira, M., Petzoldt, T., & Krems, J. (2015) · Transportation Research Part F: Traffic Psychology and Behaviour · doi:10.1016/j.trf.2015.10.005 · Cited by 215 · cited 1×
“This is in line with earlier studies (e.g., Beggiato et al., 2015; Dzindolet et al., 2003; Hergeth et al., 2016; Lee & Moray, 1992).”
ResultEvaluating Effects of User Experience and System Transparency on Trust in Automation
Yang, X. J., Unhelkar, V. V., Li, K., & Shah, J. A. (2017) · Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2909824.3020230 · Cited by 208
Automated diagnostic aids: The effects of aid reliability on users' trust and reliance
Wiegmann, D. A., Rich, A., & Zhang, H. (2001) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220110110306 · Cited by 176
Hartwich, F., Witzlack, C., Beggiato, M., & Krems, J. F. (2018) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2018.05.012 · Cited by 174
Trust and Reliance on an Automated Combat Identification System
Wang, L., Jamieson, G. A., & Hollands, J. G. (2009) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720809338842 · Cited by 154 · cited 1×
“This is in line with earlier studies showing that appropriate information about system functioning may lead to a facilitated trust calibration when system malfunctions occur (e.g., Dzindolet et al., 2003; Wang et al., 2009).”
ResultWalch, M., Mühl, K., Kraus, J., Stoll, T., Baumann, M. R. K., & Weber, M. (2017) · Human-computer interaction series · doi:10.1007/978-3-319-49448-7_10 · Cited by 145 · cited 1×
“…of system interaction. sysTem lImITATIons And Tors In highly automated driving, the automation can only perform the driving task in an array of predefined situations, and a foreknown approach to system limitations leads to a TOR (e.g., Gold, Körber, Lechner, & Bengler, 2016; Walch et al., 2017).”
BackgroundSelf-citationSystem reliability, performance and trust in adaptable automation
Chavaillaz, A., Wastell, D. G., & Sauer, J. (2015) · Applied Ergonomics · doi:10.1016/j.apergo.2015.07.012 · Cited by 139
Cooperation between drivers and automation: implications for safety
Hoc, J.-M., Young, M. S., & Blosseville, J.-M. (2009) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220802368856 · Cited by 135 · cited 1×
“This model synthesizes some central assumptions of the two models by Lee and See (2004) and Hoff and Bashir (2015) and advances these with several specifications.”
MethodThe impact of cognitive feedback on judgment performance and trust with decision aids
Seong, Y., & Bisantz, A. M. (2008) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2008.01.007 · Cited by 129 · cited 1×
“The concept of transparency can be defined as the extent to which the inner processes and decisions of an automation are made accessible (Seong & Bisantz, 2008).”
BackgroundMeasuring Individual Differences in the Perfect Automation Schema
Merritt, S. M., Unnerstall, J. L., Lee, D., & Huber, K. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720815581247 · Cited by 123
Traffic Augmentation as a Means to Increase Trust in Automated Driving Systems
Wintersberger, P., von Sawitzky, T., Frison, A.-K., & Riener, A. (2017) · Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter · doi:10.1145/3125571.3125600 · Cited by 89
Laboratory studies of trust between humans and machines in automated systems
Moray, N., & Inagaki, T. (1999) · Transactions of the Institute of Measurement and Control · doi:10.1177/014233129902100408 · Cited by 83 · cited 2×
“…prediction from the depicted trust calibration feedback loop, if a driver experiences a malfunction blindsided, the automated system should be perceived as less reliable and competent than expected, and trust should decline as an initial reaction (Madhavan & Wiegmann, 2007; Moray & Inagaki, 1999).”
BackgroundBehavioral Measurement of Trust in Automation: The Trust Fall
Miller, D., Johns, M., Mok, B., Gowda, N., Sirkin, D., Lee, K., & Ju, W. (2016) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213601422 · Cited by 71 · cited 1×
“Fifth, it would be interesting to include behavioral measures of trust and examine their relation to self-reported trust (e.g., Miller et al., 2016).”
BackgroundDesigner driving: drivers' conceptual models and level of trust in adaptive cruise control
Kazi, T. A., Stanton, N. A., Walker, G. H., & Young, M. S. (2007) · International Journal of Vehicle Design · doi:10.1504/ijvd.2007.014909 · Cited by 70 · cited 1×
“This in turn should diminish distrust and overtrust from the beginning of system use (e.g., Kazi et al., 2007; Muir, 1994).”
BackgroundHergeth, S., Lorenz, L., Krems, J. F., & Toenert, L. (2015) · Driving Assessment Conference 2015 · doi:10.17077/drivingassessment.1591 · Cited by 70 · cited 1×
“In case of the TORs, Study 1 provided first evidence that repeated TORs may be associated with more permanent trust reduction while in other research no permanent decrease was found (e.g., Hergeth et al., 2015).”
Walker, G. H., Stanton, N. A., & Salmon, P. (2016) · International Journal of Vehicle Design · doi:10.1504/ijvd.2016.074419 · Cited by 59
van den Beukel, A. P., van der Voort, M. C., & Eger, A. O. (2016) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2016.09.009 · Cited by 51
Molina, C., de Almeida, J. R., Vismari, L. F., Gonzalez, R. I. R., Naufal, J. K., & Camargo, J. B. (2017) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w.2017.14 · Cited by 47 · cited 2×
“To protect the vehicle from collision, excessive speed, or other hazardous conditions, such as a malfunction, is resolved through a fallback strategy, for example, leading to a safe halt in SAE level 4 (Emzivat et al., 2017; Molina et al., 2017).”
BackgroundForster, Y., Kraus, J., Feinauer, S., & Baumann, M. (2018) · Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications · doi:10.1145/3239060.3239070 · Cited by 36
Self-citationEffects of Attribute and Goal Framing on Automation Reliance and Compliance
Lacson, F. C., Wiegmann, D. A., & Madhavan, P. (2005) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120504900357 · Cited by 28
Emzivat, Y., Ibañez‐Guzmán, J., Martinet, P., & Roux, O. H. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995973 · Cited by 19
To See or Not to See: The Effect of Object Recognition on Users' Trust in "Automated Vehicles"
Ekman, F., Johansson, M., & Sochor, J. (2016) · Nordic Conference on Human-Computer Interaction · doi:10.1145/2971485.2971551 · Cited by 15
Research works printed but not linked: 5
- Bühner M., 2009, Statistik für Psychologen und Sozialwissenschaftler
- De Winter J., 2012, Proceedings of measuring behavior 2012, 47
- George D., 2003, Using SPSS for Windows step by step: A simple guide and reference
- IBM Corp, 2016, IBM SPSS statistics
- Madsen M., 2000, Proceedings of the 11th Australasian Conference on Information Systems (pp. 53–64). Melbourne, Australia: AIS
Cited by
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
“[104] presented a model of trust calibration and presented studies on trust dynamics in the early phases of user interaction with the system.”
BackgroundTrust and reliance on AI — An experimental study on the extent and costs of overreliance on AI
Klingbeil, A., Grützner, C., & Schreck, P. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108352 · Cited by 450
Explanations in Autonomous Driving: A Survey
Omeiza, D., Webb, H., Jirotka, M., & Kunze, L. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3122865 · Cited by 355
“This trust calibration is considered to be an important requirement for safe and efficient human-machine interaction [45], [38].”
BackgroundPsychological factors underlying attitudes toward AI tools
De Freitas, J., Agarwal, S., Schmitt, B., & Haslam, N. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01734-2 · Cited by 221
Metadata
- Authors
- Kraus, J., Scholz, D., Stiegemeier, D., & Baumann, M.
- Affiliations
- Universität Ulm (DE)
- Corresponding author
- Kraus, Johannes
- Venue
- Human Factors: The Journal of the Human Factors and Ergonomics Society
- Publisher
- SAGE Publications
- Published
- 2019-06-24
- Volume
- 62
- Issue
- 5
- Pages
- 718-736
- ISSN
- 0018-7208 (ISSN-L)
- 1547-8181
- OpenAlex
- W2953416276
- Cited by
- 276
- References
- 54
- Citations by year
- 2026: 47 · 2025: 50 · 2024: 61 · 2023: 54 · 2022: 27 · 2021: 26 · 2020: 10 · 2019: 1
- FWCI
- 20.12
- Type
- journal article
- Language
- en
- Open access
- green
- nbn-resolving.de
- Licence
- cc-by-nc-nd
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Human Factors The Journal of the Human Factors and Ergonomics Society (publishedVersion)
- PubMed (publishedVersion)
- OPen Access Repositorium der Universität Ulm (OPARU) (Ulm University) (submittedVersion, cc-by-nc-nd)
- Universität Ulm, Kommunikations- und Informationszentrum (kiz)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- trust in automated vehicles, trust calibration, highly automated driving, take-over, system malfunction, system transparency, human-machine interaction, trust repair, driver-vehicle interaction
- MeSH terms
- Automation, Automobile Driving, Calibration, Man-Machine Systems, Models, Theoretical, Models, Statistical, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.