CoChat
Sign Up
PreprintCited by 208Speech and Audio ProcessingSpeech Recognition and SynthesisTopic Modeling

The DOI matches its DataCite record.

Masked Autoencoders that Listen

Open access
Huang, Po-Yao, Xu, Hu, Li, Juncheng, Baevski, Alexei, Auli, Michael, Galuba, Wojciech, Metze, Florian, Feichtenhofer, Christoph·2022·arXiv
DOI 10.48550/arxiv.2207.06405arXiv 2207.06405
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2207.06405

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 70

  1. [1][1] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, 'Masked autoencoders are scalable vision learners,' arXiv preprint arXiv:2111.06377 , 2021. - · arXiv
  2. [2][2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, 'Attention is all you need,' in Proceedings of the 31st International Conference on Neural Information Processing Systems , ser. NIPS'17. USA: Curran Associates Inc., 2017, pp. 6000-6010. -
  3. [3][3] J. Devlin, M. Chang, K. Lee, and K. Toutanova, 'BERT: pre-training of deep bidirectional transformers for language understanding,' in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) . Association for Computational Linguistics, 2019, pp. 4171-4186. -
  4. [4][4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, 'Language models are few-shot learners,' in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. -
  5. [5][5] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, 'Roberta: A robustly optimized BERT pretraining approach,' CoRR , vol. abs/1907.11692, 2019. - · arXiv
  6. [6][6] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, 'Momentum contrast for unsupervised visual representation learning,' in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 . Computer Vision Foundation / IEEE, 2020, pp. 9726-9735. -
  7. [7][7] X. Chen, S. Xie, and K. He, 'An empirical study of training self-supervised vision transformers,' in 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 . IEEE, 2021, pp. 9620-9629. -
  8. [8][8] R. Paulus, C. Xiong, and R. Socher, 'A deep reinforced model for abstractive summarization,' arXiv , vol. abs/1705.04304, 2017. - · arXiv
  9. [9][9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' arXiv preprint arXiv:2010.11929 , 2020. - · arXiv
  10. [10][10] Y. Gong, Y. Chung, and J. R. Glass, 'AST: audio spectrogram transformer,' in Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021 . ISCA, 2021, pp. 571-575. -
  11. [11][11] A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun, 'Attention bottlenecks for multimodal fusion,' in Advances in Neural Information Processing Systems 34: Annual - Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , 2021, pp. 14 200-14 213. -
  12. [12][12] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, 'Audio set: An ontology and human-labeled dataset for audio events,' in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2017, pp. 776-780. -
  13. [13][13] K. J. Piczak, 'ESC: Dataset for Environmental Sound Classification,' in Proceedings of the 23rd Annual ACM Conference on Multimedia . ACM Press, 2015, pp. 1015-1018. -
  14. [14][14] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, 'Training dataefficient image transformers & distillation through attention,' in International Conference on Machine Learning . PMLR, 2021, pp. 10 347-10 357. -
  15. [15][15] J. Carreira and A. Zisserman, 'Quo vadis, action recognition? A new model and the kinetics dataset,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 . IEEE Computer Society, 2017, pp. 4724-4733. -
  16. [16][16] C. Feichtenhofer, A. Pinz, and R. P. Wildes, 'Spatiotemporal residual networks for video action recognition,' in NIPS , 2016. -
  17. [17][17] H. Bao, L. Dong, and F. Wei, 'Beit: BERT pre-training of image transformers,' CoRR , vol. abs/2106.08254, 2021. - · arXiv
  18. [18][18] Y. Gong, C.-I. Lai, Y.-A. Chung, and J. R. Glass, 'Ssast: Self-supervised audio spectrogram transformer,' ArXiv , vol. abs/2110.09784, 2021. - · arXiv
  19. [19][19] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, 'Swin transformer: Hierarchical vision transformer using shifted windows,' in 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 . IEEE, 2021, pp. 9992-10 002. -
  20. [20][20] C. Wei, H. Fan, S. Xie, C. Wu, A. L. Yuille, and C. Feichtenhofer, 'Masked feature prediction for self-supervised visual pre-training,' CoRR , vol. abs/2112.09133, 2021. - · arXiv
  21. [21][21] P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol, 'Extracting and composing robust features with denoising autoencoders,' in Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 , ser. ACM International Conference Proceeding Series, vol. 307. ACM, 2008, pp. 1096-1103. -
  22. [22][22] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol, 'Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,' J. Mach. Learn. Res. , vol. 11, pp. 3371-3408, 2010. -
  23. [23][23] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros, 'Context encoders: Feature learning by inpainting,' in 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 . IEEE Computer Society, 2016, pp. 2536-2544. -
  24. [24][24] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, 'Generative pretraining from pixels,' in Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 1691-1703. -
  25. [25][25] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, 'Zero-shot text-to-image generation,' in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 8821-8831. -
  26. [26][26] N. Dalal and B. Triggs, 'Histograms of oriented gradients for human detection,' in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego, CA, USA . IEEE Computer Society, 2005, pp. 886-893. -
  27. [27][27] K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 770-778, 2015. -
  28. [28][28] K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer, 'Efficient training of audio transformers with patchout,' CoRR , vol. abs/2110.05069, 2021. - · arXiv
  29. [29][29] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, 'Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection,' arXiv preprint arXiv:2202.00874 , 2022. - · arXiv
  30. [30][30] Y. Gong, Y. Chung, and J. Glass, 'Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2021. -
  31. [32][32] S. Schneider, A. Baevski, R. Collobert, and M. Auli, 'wav2vec: Unsupervised pre-training for speech recognition,' in Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019 . ISCA, 2019, pp. 3465-3469. -
  32. [33][33] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, 'wav2vec 2.0: A framework for self-supervised learning of speech representations,' in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. -
  33. [34][34] A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, 'data2vec: A general framework for self-supervised learning in speech, vision and language,' CoRR , vol. abs/2202.03555, 2022. - · arXiv
  34. [35][35] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, 'Hubert: Self-supervised speech representation learning by masked prediction of hidden units,' IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 3451-3460, 2021. -
  35. [36][36] B. Shi, W. Hsu, and A. Mohamed, 'Robust self-supervised audio-visual speech recognition,' CoRR , vol. abs/2201.01763, 2022. - · arXiv
  36. [37][37] S. Srivastava, Y . Wang, A. Tjandra, A. Kumar, C. Liu, K. Singh, and Y. Saraf, 'Conformer-based self-supervised learning for non-speech audio tasks,' arXiv preprint arXiv:2110.07313 , 2021. - · arXiv
  37. [38][38] A. Baade, P. Peng, and D. Harwath, 'Mae-ast: Masked autoencoding audio spectrogram transformer,' arXiv preprint arXiv:2203.16691 , 2022. - · arXiv
  38. [39][39] A. van den Oord, Y. Li, and O. Vinyals, 'Representation learning with contrastive predictive coding,' CoRR , vol. abs/1807.03748, 2018. - · arXiv
  39. [40][40] R. Arandjelovic and A. Zisserman, 'Objects that sound,' in Computer Vision - ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I , ser. Lecture Notes in Computer Science, vol. 11205. Springer, 2018, pp. 451-466. -
  40. [41][41] M. Patrick, P. Huang, I. Misra, F. Metze, A. Vedaldi, Y. M. Asano, and J. F. Henriques, 'Space-time crop & attend: Improving cross-modal video representation learning,' in 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 . IEEE, 2021, pp. 10 540-10 552. -
  41. [42][42] A. T. Liu, S. Yang, P. Chi, P. Hsu, and H. Lee, 'Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,' in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020 . IEEE, 2020, pp. 6419-6423. -
  42. [44][44] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino, 'Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation,' arXiv:2204.12260 , 2022. - · arXiv
  43. [45][45] Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer, 'Mvitv2: Improved multiscale vision transformers for classification and detection,' in CVPR , 2022. -
  44. [46][46] A. van den Oord, Y. Li, and O. Vinyals, 'Representation learning with contrastive predictive coding,' CoRR , vol. abs/1807.03748, 2018. - · arXiv
  45. [47][47] A. Baevski, M. Auli, and A. Mohamed, 'Effectiveness of self-supervised pre-training for speech recognition,' CoRR , vol. abs/1911.03912, 2019. - · arXiv
  46. [48][48] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, 'Specaugment: A simple data augmentation method for automatic speech recognition,' ArXiv , vol. abs/1904.08779, 2019. - · arXiv
  47. [49][49] J. B. Li, S. Qu, P. Huang, and F. Metze, 'Audiotagging done right: 2nd comparison of deep learning methods for environmental sound classification,' CoRR , vol. abs/2203.13448, 2022. - · arXiv
  48. [50][50] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. Weiss, and K. Wilson, 'Cnn architectures for large-scale audio classification,' in International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2017. -
  49. [51][51] S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, 'The benefit of temporally-strong labels in audio event classification,' in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 366-370. -
  50. [52][52] P. Warden, 'Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition,' ArXiv e-prints , Apr. 2018. -
  51. [53][53] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, 'SUPERB: Speech Processing Universal PERformance Benchmark,' in Proc. Interspeech 2021 , 2021, pp. 1194-1198. -
  52. [54][54] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, 'Voxceleb: Large-scale speaker verification in the wild,' Comput. Speech Lang. , vol. 60, 2020. -
  53. [55][55] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz et al. , 'The kaldi speech recognition toolkit,' in IEEE 2011 workshop on automatic speech recognition and understanding , no. CONF. IEEE Signal Processing Society, 2011. -
  54. [56][56] S. Yun, D. Han, S. Chun, S. J. Oh, Y. Yoo, and J. Choe, 'Cutmix: Regularization strategy to train strong classifiers with localizable features,' in 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 . IEEE, 2019, pp. 6022-6031. -
  55. [57][57] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz, 'mixup: Beyond empirical risk minimization,' in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. -
  56. [58][58] S. Verbitskiy, V. Berikov, and V. Vyshegorodtsev, 'Eranns: Efficient residual audio neural networks for audio pattern recognition,' arXiv preprint arXiv:2106.01621 , 2021. - · arXiv
  57. [59][59] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, 'Panns: Large-scale pretrained audio neural networks for audio pattern recognition,' IEEE ACM Trans. Audio Speech Lang. Process. , vol. 28, pp. 2880-2894, 2020. -
  58. [60][60] M. Tan and Q. V. Le, 'Efficientnet: Rethinking model scaling for convolutional neural networks,' in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 6105-6114. -
  59. [61][61] D. Griffin and J. Lim, 'Signal estimation from modified short-time fourier transform,' IEEE Transactions on Acoustics, Speech, and Signal Processing , vol. 32, no. 2, pp. 236-243, 1984. -
  60. [62][62] Y. Suzuki and H. Takeshima, 'Equal-loudness-level contours for pure tones,' The Journal of the Acoustical Society of America , vol. 116, no. 2, pp. 918-933, 2004. -
  61. [63][63] I. Loshchilov and F. Hutter, 'Decoupled weight decay regularization,' in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. -
  62. [64][64] --, 'SGDR: stochastic gradient descent with warm restarts,' in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. -
  63. [65][65] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, 'Deep networks with stochastic depth,' in Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV , ser. Lecture Notes in Computer Science, vol. 9908. Springer, 2016, pp. 646-661. -
  64. [66][66] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, 'Dropout: a simple way to prevent neural networks from overfitting,' J. Mach. Learn. Res. , vol. 15, no. 1, pp. 1929-1958, 2014. -
  65. [67][67] R. Müller, S. Kornblith, and G. E. Hinton, 'When does label smoothing help?' in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , 2019, pp. 4696-4705. -
  66. [68][68] B. Lee and J. Chang, 'Packet loss concealment based on deep neural networks for digital speech transmission,' IEEE ACM Trans. Audio Speech Lang. Process. , vol. 24, no. 2, pp. 378-387, 2016. -
  67. [69][69] J. Lin, Y. Wang, K. Kalgaonkar, G. Keren, D. Zhang, and C. Fuegen, 'A time-domain convolutional recurrent network for packet loss concealment,' in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2021, pp. 7148-7152. -
  68. [70][70] Y. Chang, K. Lee, P. Wu, H. Lee, and W. H. Hsu, 'Deep long audio inpainting,' CoRR , vol. abs/1911.06476, 2019. - · arXiv
  69. [71][71] A. Marafioti, N. Perraudin, N. Holighaus, and P. Majdak, 'A context encoder for audio inpainting,' IEEE ACM Trans. Audio Speech Lang. Process. , vol. 27, no. 12, pp. 2362-2372, 2019. -
  70. [72][72] B. Liu, J. Tao, Z. Wen, Y. Li, and D. Bukhari, 'A novel method of artificial bandwidth extension using deep architecture,' in INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015 . ISCA, 2015, pp. 2598-2602.

Cited by

Sorted by citations

Items 101–200 of 208

Related

See all

Metadata

Authors
Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., & Feichtenhofer, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
28708-28720
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2207.06405
arXiv
2207.06405
OpenAlex
W7133185223
Cited by
206
References
0
Citations by year
2026: 41 · 2025: 51 · 2024: 65 · 2023: 48 · 2022: 1
FWCI
29.33
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
masked autoencoder, audio, audio representation learning, audio denoising

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review