CoChat
Sign Up
PreprintCited by 43Music and Audio ProcessingMultimodal Machine Learning ApplicationsSpeech Recognition and Synthesis

Listen, Think, and Understand

Open access
Gong, Yuan, Luo, Hongyin, Liu, Alexander H., Karlinsky, Leonid, Glass, James·2023·arXiv
DOI 10.48550/arxiv.2305.10790arXiv 2305.10790
Open this paper in CoChat — free accountOpen PDF

Abstract

The ability of artificial intelligence (AI) systems to perceive and comprehend audio signals is crucial for many applications. Although significant progress has been made in this area since the development of AudioSet, most existing models are designed to map audio inputs to pre-defined, discrete sound label sets. In contrast, humans possess the ability to not only classify sounds into general categories, but also to listen to the finer details of the sounds, explain the reason for the predictions, think about what the sound infers, and understand the scene and what action needs to be taken, if any. Such capabilities beyond perception are not yet present in existing audio models. On the other hand, modern large language models (LLMs) exhibit emerging reasoning ability but they lack audio perception capabilities. Therefore, we ask the question: can we build a model that has both audio perception and a reasoning ability? In this paper, we propose a new audio foundation model, called LTU (Listen, Think, and Understand). To train LTU, we created a new OpenAQA-5M dataset consisting of 1.9 million closed-ended and 3.7 million open-ended, diverse (audio, question, answer) tuples, and have used an autoregressive training framework with a perception-to-understanding curriculum. LTU demonstrates strong performance and generalization ability on conventional audio tasks such as classification and captioning. More importantly, it exhibits emerging audio reasoning and comprehension abilities that are absent in existing audio models. To the best of our knowledge, LTU is one of the first multimodal large language models that focus on general audio (rather than just speech) understanding.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.10790

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LTU-AS substantially improves the recent audio large language model LTU that only understands non-speech audio.

    “LTU-AS substantially improves the recent audio large language model LTU [5] that only understands non-speech audio.”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [5]

    The Introduction identifies LTU as a model for general audio understanding, and the Conclusion says it has limited ability to understand speech content. That supports the contrast with a speech-capable successor, but not the absolute claim that LTU only understands non-speech audio. The statement that LTU-AS improves LTU is the citing authors’ claim, not one attributed to this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The OpenAQA dataset used to train LTU is an audio question-answering dataset consisting of 1.9 million closed-ended and 3.7 million open-ended AQAs.

    “The closest one is the OpenAQA dataset used to train LTU [5], which is an audio question-answering dataset consisting of 1.9 million closed-ended and 3.7 million openended AQAs.”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [5]

  • The authors use approximately 3.7 million AQAs about non-speech audio events from the original OpenAQA dataset.

    “We use the approximately 3.7 million AQAs about non-speech audio events from the original OpenAQA dataset [5].”

    From Joint Audio and Speech Understanding (2023) · cites this paper as [5]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 71

  1. Jerome Abdelnour, Giampiero Salvi, and Jean Rouat. Clear: A dataset for compositional language and elementary acoustic reasoning. arXiv preprint arXiv:1811.10561, 2018. · arXiv
  2. Jerome Abdelnour, Jean Rouat, and Giampiero Salvi. Naaqa: A neural architecture for acoustic question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
  4. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
  6. Kai-Wei Chang, Wei-Cheng Tseng, Shang-Wen Li, and Hung-yi Lee. An exploration of prompt tuning on generative spoken language model for speech processing tasks. In INTERSPEECH, pp. 5005-5009. ISCA, 2022.
  7. Kai-Wei Chang, Yu-Kai Wang, Hua Shen, Iu-thing Kang, Wei-Cheng Tseng, Shang-Wen Li, and Hung-yi Lee. Speechprompt v2: Prompt tuning for speech classification tasks. arXiv preprint arXiv:2303.00733, 2023. · arXiv
  8. Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP, pp. 721-725, 2020a.
  9. Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts- at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646-650. IEEE, 2022.
  10. Kun Chen, Yusong Wu, Ziyue Wang, Xuan Zhang, Fudong Nian, Shengchen Li, and Xi Shao. Audio captioning based on transformer and pre-trained cnn. In DCASE, pp. 21-25, 2020b.
  11. Jason Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello. Look, listen, and learn more: Design choices for deep audio embeddings. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3852-3856. IEEE, 2019.
  12. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. arXiv preprint arXiv:2305.11834, 2023. · arXiv
  13. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.03378, 2023. · arXiv
  14. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 736-740. IEEE, 2020.
  15. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
  16. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, 2018.
  17. Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829-852, 2021.
  18. Frederic Font, Gerard Roma, and Xavier Serra. Freesound technical demo. In Proceedings of the 21st ACM international conference on Multimedia, pp. 411-412, 2013.
  19. Logan Ford, Hao Tang, Franc ¸ois Grondin, and James R Glass. A deep residual network for large- scale acoustic scene analysis. In INTERSPEECH, pp. 2568-2572, 2019.
  20. Heting Gao, Junrui Ni, Kaizhi Qian, Yang Zhang, Shiyu Chang, and Mark Hasegawa-Johnson. Wavprompt: Towards few-shot spoken language understanding with frozen language models. arXiv preprint arXiv:2203.15863, 2022. · arXiv
  21. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, pp. 776-780, 2017.
  22. Yuan Gong, Yu-An Chung, and James Glass. Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3292-3306, 2021a.
  23. Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. In Proc. Interspeech 2021, pp. 571-575, 2021b. doi: 10.21437/Interspeech.2021-698. · DOI · Paper page
  24. Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R Glass. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representations, 2022a.
  25. Yuan Gong, Jin Yu, and James Glass. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 151-155. IEEE, 2022b.
  26. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013. · arXiv
  27. Zheng Gu, Xinzhou Xu, Shuo Liu, and Björn Schuller. Zero-shot audio classification using syn- thesised classifiers and pre-trained models. In 2022 15th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1-6. IEEE, 2022.
  28. Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
  29. Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 366-370. IEEE, 2021.
  30. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. · arXiv
  31. Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Infor- mation Processing Systems, 35:28708-28720, 2022.
  32. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304.12995, 2023a. · arXiv
  33. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023b. · arXiv
  34. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019. · arXiv
  35. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers), pp. 119-132, 2019.
  36. Eungbeom Kim, Jinhee Kim, Yoori Oh, Kyungsu Kim, Minju Park, Jaeheon Sim, Jinwoo Lee, and Kyogu Lee. Improving audio-language learning with mixgen and multi-level test-time augmen- tation. arXiv preprint arXiv:2210.17143, 2022. · arXiv
  37. Yuma Koizumi, Yasunori Ohishi, Daisuke Niizumi, Daiki Takeuchi, and Masahiro Yasuda. Audio captioning using pre-trained large-scale language model guided by audio-based similar caption retrieval. arXiv preprint arXiv:2012.07331, 2020. · arXiv
  38. Qiuqiang Kong, Yong Xu, Wenwu Wang, and Mark D Plumbley. Audio set classification with attention model: A probabilistic perspective. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 316-320, 2018.
  39. Qiuqiang Kong, Changsong Yu, Yong Xu, Turab Iqbal, Wenwu Wang, and Mark D Plumbley. Weakly labelled audioset tagging with attention neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(11):1791-1802, 2019.
  40. Qiuqiang Kong, Yong Xu, Wenwu Wang, and Mark D Plumbley. Sound event detection of weakly labelled data with cnn-transformer and automatic threshold optimization. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 28:2450-2460, 2020.
  41. Anurag Kumar, Maksim Khadkevich, and Christian Fügen. Knowledge transfer from weakly labeled audio using convolutional neural network for sound events and scenes. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 326-330, 2018.
  42. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1354, 2021.
  43. Seungjun Lee, Haesang Yang, Hwiyong Choi, and Woojae Seong. Zero-shot single-microphone sound classification and localization in a building via the synthesis of unseen features. IEEE Transactions on Multimedia, 24:2339-2351, 2021.
  44. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059, 2021.
  45. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. · arXiv
  46. Samuel Lipping, Konstantinos Drossos, and Tuoams Virtanen. Crowdsourcing a dataset of audio captions. In Proceedings of the Detection and Classification of Acoustic Scenes and Events Work- shop (DCASE), 2019.
  47. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023. · arXiv
  48. Xinhao Mei, Xubo Liu, Qiushi Huang, Mark D Plumbley, and Wenwu Wang. Audio captioning transformer. arXiv preprint arXiv:2107.09817, 2021. · arXiv
  49. Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumb- ley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio cap- tioning dataset for audio-language multimodal research. arXiv preprint arXiv:2303.17395, 2023. · arXiv
  50. Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. Tut database for acoustic scene classi- fication and sound event detection. In 2016 24th European Signal Processing Conference (EU- SIPCO), pp. 1128-1132. IEEE, 2016.
  51. R Channing Moore, Daniel PW Ellis, Eduardo Fonseca, Shawn Hershey, Aren Jansen, and Manoj Plakal. Dataset balancing can hurt model performance. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
  52. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. · arXiv
  53. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023a. · arXiv
  54. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023b. · arXiv
  55. Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
  56. Mi Tian, Ajay Srinivasamurthy, Mark Sandler, and Xavier Serra. A study of instrument-wise on- set detection in beijing opera percussion ensembles. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pp. 2159-2163. IEEE, 2014.
  57. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
  58. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
  59. Yun Wang, Juncheng Li, and Florian Metze. A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31-35, 2019.
  60. Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. On decoder-only architecture for speech-to-text and large language model integration. arXiv preprint arXiv:2307.03917, 2023a. · arXiv
  61. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023b.
  62. Yuzhong Wu and Tan Lee. Reducing model complexity for dnn based large-scale audio classifica- tion. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 331-335, 2018.
  63. Huang Xie and Tuomas Virtanen. Zero-shot audio classification based on class label embeddings. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 264-267. IEEE, 2019.
  64. Huang Xie and Tuomas Virtanen. Zero-shot audio classification via semantic embeddings. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1233-1242, 2021.
  65. Huang Xie, Okko Räsänen, and Tuomas Virtanen. Zero-shot audio classification with factored linear and nonlinear acoustic-semantic projections. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 326-330. IEEE, 2021.
  66. Huang Xie, Okko Räsänen, Konstantinos Drossos, and Tuomas Virtanen. Unsupervised audio- caption aligning learns correspondences between individual sound events and textual phrases. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8867-8871. IEEE, 2022.
  67. Xuenan Xu, Heinrich Dinkel, Mengyue Wu, and Kai Yu. A crnn-gru based reinforcement learning approach to audio captioning. In DCASE, pp. 225-229, 2020.
  68. Xuenan Xu, Heinrich Dinkel, Mengyue Wu, Zeyu Xie, and Kai Yu. Investigating local and global information for automated audio captioning with transfer learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 905-909. IEEE, 2021.
  69. Changsong Yu, Karim Said Barsim, Qiuqiang Kong, and Bin Yang. Multi-level attention model for weakly supervised audio classification. In DCASE Workshop, 2018.
  70. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. arXiv preprint arXiv:2305.11000, 2023a. · arXiv
  71. Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init atten- tion. arXiv preprint arXiv:2303.16199, 2023b. · arXiv

Cited by

Sorted by citations

Authors
Gong, Y., Luo, H., Liu, A. H., Karlinsky, L., & Glass, J.
Affiliations
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • IBM (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2305.10790
arXiv
2305.10790
OpenAlex
W4377130946
Cited by
43
References
0
Citations by year
2025: 17 · 2024: 23 · 2023: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
listening comprehension, multimodal large language models, audio captioning, audio classification, audio question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review