Characterizing and Predicting Corrections in Spoken Dialogue Systems
Open accessAbstract
This article focuses on the analysis and prediction of corrections, defined as turns where a user tries to correct a prior error made by a spoken dialogue system. We describe our labeling procedure of various corrections types and statistical analyses of their features in a corpus collected from a train information spoken dialogue system. We then present results of machine-learning experiments designed to identify user corrections of speech recognition errors. We investigate the predictive power of features automatically computable from the prosody of the turn, the speech recognition process, experimental conditions, and the dialogue history. Our best-performing features reduce classification error from baselines of 25.70–28.99% to 15.72%.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
[43] utilised the TOOT train information corpus from 1999.
“For instance,[43] utilised the TOOT train information corpus from 1999.”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [43]
Section 2.1, “The TOOT Corpus” (page unavailable), says the authors used dialogues from TOOT, a system providing train information, and Section 5 says they analyzed corrections in that corpus. Section 2.1 cites Litman and Pan (1999) when describing why TOOT was collected, but does not establish that the corpus itself dates from 1999. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe selection of user-repair strategies is not random but is significantly influenced by the system's repair strategies.
“Importantly, the selection of user-repair strategies is not random but is significantly influenced by the system's repair strategies [43, 50, 60, 61].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Empirical methods for artificial intelligence
(1996) · Choice Reviews Online · doi:10.5860/choice.33-5144 · Cited by 625 · see citing works
How to find trouble in communication
Batliner, A. M., Fischer, K., Huber, R., Spilker, J., & Nöth, E. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00079-1 · Cited by 281 · see citing works
Learning Trees an ules with Set-val ed Features
Cohen, W. W. (1996) · Cited by 247 · see citing works
Exploring human error recovery strategies: Implications for spoken dialogue systems
Skantze, G. (2005) · Speech Communication · doi:10.1016/j.specom.2004.11.005 · Cited by 137 · see citing works
Word level confidence annotation using combinations of features
Zhang, R., & Rudnicky, A. I. (2001) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2001-496 · Cited by 93 · see citing works
Prosodic and other cues to speech recognition failures
Hirschberg, J., Litman, D. J., & Swerts, M. G. J. (2004) · Speech Communication · doi:10.1016/j.specom.2004.01.006 · Cited by 89 · see citing works
Self-citationCorrections in spoken dialogue systems
Swerts, M. G. J., Litman, D. J., & Hirschberg, J. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-344 · Cited by 82 · see citing works
Self-citationCharacterizing and recognizing spoken corrections in human-computer dialogue
Levow, G. (1998) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/980845.980969 · Cited by 80 · see citing works
Walker, M. A., Langkilde-Geary, I., Wright, J. H., Gorin, A. L., & Litman, D. J. (2000) · Cited by 78 · see citing works
Self-citationEmpirically Evaluating an Adaptable Spoken Dialogue System
Litman, D. J., & Pan, S. (1999) · CISM International Centre for Mechanical Sciences · doi:10.1007/978-3-7091-2490-1_6 · Cited by 67 · see citing works
Self-citationAutomatic detection of poor speech recognition at the dialogue level
Litman, D. J., Walker, M. A., & Kearns, M. S. (1999) · doi:10.3115/1034678.1034729 · Cited by 65 · see citing works
Self-citationBouwman, G., Sturm, J., & Boves, L. (1999) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1999.758170 · Cited by 51 · see citing works
Error Detection in Spoken Human-Machine Interaction
Krahmer, E., Swerts, M. G. J., Theune, M., & Weegels, M. F. (2001) · International Journal of Speech Technology · doi:10.1023/a:1009648614566 · Cited by 45 · see citing works
Self-citationHandling errors and determining confirmation strategies—An object-based approach
McTear, M., O’Neill, I., Hanna, P., & Liu, X. (2005) · Speech Communication · doi:10.1016/j.specom.2004.11.006 · Cited by 44 · see citing works
Modeling hyperarticulate speech during human-computer error resolution
Oviatt, S. L., Levow, G., MacEachern, M. R., & Kuhn, K. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607722 · Cited by 44 · see citing works
Prosodic cues to recognition errors
Hirschberg, J., Litman, D. J., & Swerts, M. G. J. (1999) · Cited by 42 · see citing works
Self-citationA boosting approach for confidence scoring
Moreno, P. J., Logan, B., & Raj, B. (2001) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2001-497 · Cited by 40 · see citing works
The Watson speech recognition engine
Sharp, R. D., Bocchieri, E., Castillo, C. M., Parthasarathy, S., Rath, C., Riley, M., & Rowland, J. R. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1997.604839 · Cited by 39 · see citing works
Problem detection in human–machine interactions based on facial expressions of users
Barkhuysen, P. N., Krahmer, E., & Swerts, M. G. J. (2004) · Speech Communication · doi:10.1016/j.specom.2004.10.004 · Cited by 36 · see citing works
Self-citationError-correction detection and response generation in a spoken dialogue system
Bulyko, I., Kirchhoff, K., Ostendorf, M., & Goldberg, J. (2005) · Speech Communication · doi:10.1016/j.specom.2004.09.009 · Cited by 36 · see citing works
Bell, L., & Gustafson, J. (1999) · Cited by 35 · see citing works
Error handling in a stochastic dialog system through confidence measures
Torres, F., Hurtado, L.-F., García, F., Sanchis, E., & Segarra, E. (2004) · Speech Communication · doi:10.1016/j.specom.2004.10.014 · Cited by 32 · see citing works
Evaluating spoken dialog systems for telecommunication services
Kamm, C., Narayanan, S. S., Dutton, D., & Ritenour, R. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-581 · Cited by 32 · see citing works
Effective error recovery strategies for multimodal form-filling applications
Sturm, J., & Boves, L. (2005) · Speech Communication · doi:10.1016/j.specom.2004.11.007 · Cited by 29 · see citing works
On the influence of hyperarticulated speech on recognition performance
Soltau, H., & Waibel, A. (1998) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1998-237 · Cited by 28 · see citing works
Specialized acoustic models for hyperarticulated speech
Soltau, H., & Waibel, A. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2000.862098 · Cited by 25 · see citing works
Transparency strategies to help users handle system errors
Karsenty, L., & Botherel, V. (2005) · Speech Communication · doi:10.1016/j.specom.2004.10.018 · Cited by 25 · see citing works
Acoustic and word lattice based algorithms for confidence scores
Falavigna, D., Gretter, R., & Riccardi, G. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-482 · Cited by 22 · see citing works
Ambiguity representation and resolution in spoken dialogue systems
Ammicht, E., Potamianos, A., & Fosler‐Lussier, E. (2001) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2001-523 · Cited by 21 · see citing works
Prodanov, P. J., & Drygajlo, A. (2005) · Speech Communication · doi:10.1016/j.specom.2004.10.015 · Cited by 20 · see citing works
User behaviors affecting speech recognition
Wade, E., Shriberg, E. E., & Price, P. J. (1992) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1992-290 · Cited by 15 · see citing works
Compensating for hyperarticulation by modeling articulatory properties
Soltau, H., Metze, F., & Waibel, A. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-288 · Cited by 14 · see citing works
Robust semantic confidence scoring
Guillevic, D., Gandrabur, S., & Normandin, Y. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-291 · Cited by 12 · see citing works
Shimojima, A., Katagiri, Y., Koiso, H., & Swerts, M. (1999) · TU/e Research Portal · Cited by 10 · see citing works
Self-citationExperiments in confidence scoring for word and sentence verification
Andorno, M., Laface, P., & Gemello, R. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-417 · Cited by 7 · see citing works
Error-tolerant spoken language understanding with confidence measuring
Wang, H.-M., & Lin, Y. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-483 · Cited by 5 · see citing works
Extracting information from spoken user input' a machine learning approach
Lendvai, P. (2004) · Research portal (Tilburg University) · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Cohen, W. 1996. Learning trees and rules with set-valued features. In 14th Conference of the American Association of Artificial Intelligence, AAAI, pages 709-716, Portland.
- Kirchhoff, Katrin. 2001. A comparison of classification techniques for the automatic detection of error corrections in human-computer dialogues. In Proceedings of the NAACL Workshop on Adaptation in Dialogue Systems, pages 33-40, Pittsburgh, PA.
- Talkin, D. 1995. A Robust algorithm for pitch tracking (RAPT). In W. B. Klein and K. K. Paliwal, editors, Speech Coding and Synthesis. Elsevier Science, Athens, pages 495-518.
Cited by
Sorted by citations
Adapting speaking after evidence of misrecognition: Local and global hyperarticulation
Stent, A. J., Huffman, M. K., & Brennan, S. E. (2007) · Speech Communication · doi:10.1016/j.specom.2007.07.005 · Cited by 82 · see citing works
From Annotator Agreement to Noise Models
Klebanov, B. B., & Beigman, E. (2009) · Computational Linguistics · doi:10.1162/coli.2009.35.4.35402 · Cited by 76 · see citing works
Lo, V. E.-W., & Green, P. (2013) · International Journal of Vehicular Technology · doi:10.1155/2013/924170 · Cited by 50 · see citing works
Computational Models of Miscommunication Phenomena
Purver, M., Hough, J., & Howes, C. (2018) · Topics in Cognitive Science · doi:10.1111/tops.12324 · Cited by 33 · see citing works
Degrees of grounding based on evidence of understanding
Roque, A., & Traum, D. R. (2008) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.3115/1622064.1622073 · Cited by 25 · see citing works
Improving a virtual human using a model of degrees of grounding
Roque, A. C., & Traum, D. R. (2009) · Cited by 22 · see citing works
The MATCH corpus: a corpus of older and younger users’ interactions with spoken dialogue systems
Georgila, K., Wolters, M., Moore, J. D., & Logie, R. H. (2010) · Language Resources and Evaluation · doi:10.1007/s10579-010-9118-8 · Cited by 21 · see citing works
Crowdsourcing for web genre annotation
Asheghi, N. R., Sharoff, S., & Markert, K. (2016) · Language Resources and Evaluation · doi:10.1007/s10579-015-9331-6 · Cited by 20 · see citing works
On what happens in gesture when communication is unsuccessful
Hoetjes, M., Krahmer, E., & Swerts, M. G. J. (2015) · Speech Communication · doi:10.1016/j.specom.2015.06.004 · Cited by 18 · see citing works
Self-citationSpoken Dialog Systems for Automated Survey Interviewing
Johnston, M. O., Ehlen, P., Conrad, F. G., Schober, M. F., Antoun, C., Stefanie, F., Hupp, A. L., Vickers, L., Yan, H. Y., & Zhang, C. (2013) · Cited by 15 · see citing works
Robot perception errors and human resolution strategies in situated human–robot dialogue
Schütte, N., Namee, B. M., & Kelleher, J. D. (2017) · Advanced Robotics · doi:10.1080/01691864.2016.1268973 · Cited by 15 · see citing works
Schnall, A., & Heckmann, M. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-565 · Cited by 15 · see citing works
Alghamdi, E., Halvey, M., & Nicol, E. (2024) · ACM Conversational User Interfaces 2024 · doi:10.1145/3640794.3665558 · Cited by 36 · see citing works
Learning about Voice Search for Spoken Dialogue Systems
Passonneau, R. J., Epstein, S. L., Ligorio, T., Gordon, J. B., & Bhutada, P. (2010) · Cited by 14 · see citing works
Opfermann, C., & Pitsch, K. (2017) · 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN) · doi:10.1109/roman.2017.8172319 · Cited by 12 · see citing works
Predicting concept types in user corrections in dialog
Stoyanchev, S., & Stent, A. J. (2009) · doi:10.3115/1626296.1626302 · Cited by 11 · see citing works
Designing and Evaluating a Reliable Corpus of Web Genres via Crowd-Sourcing
Asheghi, N. R., Sharoff, S., & Markert, K. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/47hyeu78sdtj · Cited by 10 · see citing works
Sodhi, S., Chio, E. K.-I., Jash, A., Ontañón, S., Apte, A., Kumar, A., Jeje, A., Kuzmin, D., Fung, H., Cheng, H.-T., et al. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467156 · Cited by 9 · see citing works
Detecting repetitions in spoken dialogue systems using phonetic distances
Lopes, J., Salvi, G., Skantze, G., Abad, A., Gustafson, J., Batista, F., Meena, R., & Trancoso, I. M. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-60 · Cited by 9 · see citing works
Learning to Balance Grounding Rationales for Dialogue Systems
Gordon, J. A., Passonneau, R. J., & Epstein, S. L. (2011) · Cited by 8 · see citing works
Computing the fundamental frequency variation spectrum in conversational spoken dialogue systems
Laskowski, K., Wölfel, M., Heldner, M., & Edlund, J. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2934193 · Cited by 8 · see citing works
Predicting Causes of Reformulation in Intelligent Assistants
Sano, S., Kaji, N., & Sassano, M. (2017) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w17-5536 · Cited by 8 · see citing works
Alloatti, F., Grasso, F., Ferrod, R., Siragusa, G., Di, L., & Cena, F. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2023.101603 · Cited by 7 · see citing works
The Dynamics of Action Corrections in Situated Interaction
Raux, A., & Nakano, M. (2010) · Cited by 7 · see citing works
Feature-space SVM adaptation for speaker adapted word prominence detection
Schnall, A., & Heckmann, M. (2018) · Computer Speech & Language · doi:10.1016/j.csl.2018.06.001 · Cited by 7 · see citing works
Rasenberg, M., Amha, A., Coler, M., van Koppen, M., van Miltenburg, E., de Rijk, L. E. M., Stommel, W., & Dingemanse, M. (2023) · Linguistics in the Netherlands · doi:10.1075/avt.00095.ras · Cited by 6 · see citing works
Audio-visual word prominence detection from clean and noisy speech
Heckmann, M. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2017.09.002 · Cited by 6 · see citing works
On the Role of Discourse Markers in Interactive Spoken Question Answering Systems
Vasilescu, I., Rosset, S., & Adda‐Decker, M. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/48tdjempainf · Cited by 6 · see citing works
Detecting Inappropriate Clarification Requests in Spoken Dialogue Systems
Liu, A. X., Sloan, R., Then, M.-V., Stoyanchev, S., Hirschberg, J., & Shriberg, E. E. (2014) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.3115/v1/w14-4331 · Cited by 6 · see citing works
Self-citationUser-Initiated Repetition-Based Recovery in Multi-Utterance Dialogue Systems
Nguyen, H. L., Renkens, V., Pelemans, J., Potharaju, S. P., Nalamalapu, A. K., & Akbacak, M. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1536 · Cited by 5 · see citing works
Steps Towards More Natural Human-Machine Interaction via Audio-Visual Word Prominence Detection
Heckmann, M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-15557-9_2 · Cited by 5 · see citing works
Evolutionary Reinforcement Learning of Spoken Dialogue Strategies
Toney, D. (2007) · ERA · Cited by 5 · see citing works
Hyperarticulation of Corrections in Multilingual Dialogue Systems
Kraljevski, I., & Hirschfeld, D. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-753 · Cited by 3 · see citing works
Schnall, A., & Heckmann, M. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280639 · Cited by 3 · see citing works
Autonomous Self-Assessment of Autocorrections: Exploring Text Message Dialogues
Baldwin, T., & Chai, J. Y. (2012) · Cited by 3 · see citing works
Towards a comprehensive repair framework for human-chatbot interaction
Asisof, A. (2022) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3514197.3549641 · Cited by 3 · see citing works
Jaramillo, C. M. Z., & Mesa, J. E. (2009) · DYNA · Cited by 3 · see citing works
Wizards' dialogue strategies to handle noisy speech recognition
Ligorio, T., Epstein, S. L., & Passonneau, R. J. (2010) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2010.5700871 · Cited by 3 · see citing works
Concept Type Prediction and Responsive Adaptation in a Dialogue System
Stoyanchev, S., & Stent, A. J. (2012) · Dialogue & Discourse · doi:10.5087/dad.2012.101 · Cited by 2 · see citing works
Comparing speaker independent and speaker adapted classification for word prominence detection
Schnall, A., & Heckmann, M. (2016) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2016.7846271 · Cited by 2 · see citing works
Data Mining to Support Human-Machine Dialogue for Autonomous Agents
Epstein, S. L., Passonneau, R. J., Ligorio, T., & Gordon, J. A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-27609-5_10 · Cited by 1 · see citing works
Impact of responsive and directive adaptation on local dialog processing
Stent, A. J., & Stoyanchev, S. (2009) · Cited by 1 · see citing works
The Impact of Word Alignment Accuracy on Audio-visual Word Prominence Detection
Heckmann, M., Mikias, P., & Kolossa, D. (2014) · doi:10.13140/2.1.2823.0408 · Cited by 1 · see citing works
Online adaptation for mobile device text input personalization
Chai, J. Y., & Baldwin, T. (2012) · ProQuest LLC eBooks · Cited by 1 · see citing works
Speech Driven by Artificial Larynx: Potential Advancement Using Synthetic Pitch Contours
Jian, H. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20684-4_30 · Cited by 1 · see citing works
Farkhadov, M., Петухова, Н. В., Eliseev, A., & Farkhadova, M. (2019) · IEEE International Conference on Application of Information and Communication Technologies (AICT) · doi:10.1109/aict47866.2019.8981770 · Cited by 1 · see citing works
Lo, E.-W. (2013) · Deep Blue (University of Michigan) · Cited by 1 · see citing works
Semantic tagging of medical narratives using SNOMED CT
Hina, S. (2013) · White Rose eTheses Online (University of Leeds, The University of Sheffield, University of York)
Naturalistic Dialogue Management for Noisy Speech Recognition
Passonneau, R. J., Epstein, S. L., & Ligorio, T. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2229964
Impact of Responsive and Directive Adaptation on Local
Stoyanchev, S. (2009)
A daily scrum meeting summarizer for agile software development teams
Park, S. S. (2007) · Libraries and Cultural Resources (University of Calgary) · doi:10.11575/prism/1255
Talking hands. Reference in speech, gesture, and sign
Hoetjes, M. (2015) · Research portal (Tilburg University) · doi:10.13140/rg.2.1.2956.3606
Jaramillo, C. M. Z., & Mesa, J. E. (2009) · DYNA · Cited by 3 · see citing works
Analysis and categorization of corrections in multilingual spoken dialogue system
Kraljevski, I., & Hirschfeld, D. (2019)
Schaub, L.-P., Hudeček, V., Stancl, D., Dušek, O., & Paroubek, P. (2021)
- Authors
- Litman, D., Hirschberg, J., & Swerts, M.
- Affiliations
- University of Pittsburgh (US)
- Columbia University (US)
- Tilburg University (NL)
- Venue
- Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2006-09
- Volume
- 32
- Issue
- 3
- Pages
- 417-438
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W2096643992
- Cited by
- 56
- References
- 37
- Citations by year
- 2024: 1 · 2023: 1 · 2022: 1 · 2021: 3 · 2019: 2 · 2018: 2 · 2017: 5 · 2016: 2 · 2015: 6 · 2014: 5 · 2013: 4 · 2012: 4
- FWCI
- 4.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Computational Linguistics (publishedVersion)
- Research portal (Tilburg University) (publishedVersion)
- D-Scholarship@Pitt (University of Pittsburgh) (acceptedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Universiteit Antwerpen
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- spoken dialogue systems, user corrections, speech recognition errors, prosody features, dialogue history
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
