The Voice Conversion Challenge 2016
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
This paper describes the Voice Conversion Challenge 2016 devised by the authors to better understand different voice conversion (VC) techniques by comparing their performance on a common dataset. The task of the challenge was speaker conversion, i.e., to transform the voice identity of a source speaker into that of a target speaker while preserving the linguistic content. Using a common dataset consisting of 162 utterances for training and 54 utterances for evaluation from each of 5 source and 5 target speakers, 17 groups working in VC around the world developed their own VC systems for every combination of the source and target speakers, i.e., 25 systems in total, and generated voice samples converted by the developed systems. These samples were evaluated in terms of target speaker similarity and naturalness by 200 listeners in a controlled environment. This paper summarizes the design of the challenge, its result, and a future plan to share views about unsolved problems and challenges faced by the current VC techniques.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A benefit of using voice conversion is it can help those with speech disorders, which is useful for rehabilitation medicine (Abe et al., 1990; Toda et al., 2016).”
From Audio deepfakes: A survey (2023) · cites this paper as Toda et al., 2016
In the citing paper
“Voice conversion challenge 2016 database (Toda et al., 2016): The original dataset created by the challenge organisers was derived from the DAPS (Device and Produced Speech) Dataset (Mysore, 2015). It contains 216 utterances (162 for training and 54 for testing) per speaker from 10 speakers.”
From Deepfake: definitions, performance metrics and standards, datasets, and a meta-review (2024) · cites this paper as Toda et al., 2016
Citations
Cites
Kawahara, H., Masuda-Katsuse, I., & de Cheveigné, A. (1999) · Speech Communication · doi:10.1016/s0167-6393(98)00085-5 · Cited by 1,831 · cited 1×
“Various sophisticated techniques, such as harmonic plus noise model (HNM) [15] and STRAIGHT [16], have been often used to extract high quality speech features from a speech waveform, and also to generate a speech wave-form from the converted speech features.”
MethodVoice Conversion Based on Maximum-Likelihood Estimation of Spectral Parameter Trajectory
Toda, T., Black, A. W., & Tokuda, K. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2007.907344 · Cited by 937
Self-citationContinuous probabilistic transform for voice conversion
Stylianou, Y., Cappe, O., & Moulines, E. (1998) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.661472 · Cited by 931
Voice conversion through vector quantization
Abe, M., Nakamura, S., Shikano, K., & Kuwabara, H. (2003) · ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1988.196671 · Cited by 415 · cited 1×
“VC research has a relatively long history from the late 1980s onwards [11].”
BackgroundApplying the harmonic plus noise model in concatenative speech synthesis
Stylianou, Y. (2001) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.890068 · Cited by 325
Voice conversion using deep Bidirectional Long Short-Term Memory based Recurrent Neural Networks
Sun, L., Kang, S., Li, K., & Meng, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178896 · Cited by 257
Mel-generalized cepstral analysis - a unified approach to speech spectral estimation
Tokuda, K., Kobayashi, T., Masuko, T., & Imai, S. (1994) · 3rd International Conference on Spoken Language Processing (ICSLP 1994) · doi:10.21437/icslp.1994-275 · Cited by 237
Spectral Mapping Using Artificial Neural Networks for Voice Conversion
Desai, S., Black, A. W., Yegnanarayana, B., & Prahallad, K. (2010) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2010.2047683 · Cited by 223
Voice Conversion Using Deep Neural Networks With Layer-Wise Generative Training
Chen, L.-H., Ling, Z.-H., Liu, L.-J., & Dai, L.-R. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2353991 · Cited by 210
Self-citationStatistical Voice Conversion Techniques for Body-Conducted Unvoiced Speech Enhancement
Toda, T., Nakagiri, M., & Shikano, K. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2012.2205241 · Cited by 182
Self-citation
Cited by
Items 101–155 of 155
Submission from SRCB for Voice Conversion Challenge 2020
Ma, Q., Liu, R., Xue, W., Lu, C., & Chen, X. (2020) · doi:10.21437/vcc_bc.2020-18 · Cited by 6
Perturbation AUTOVC: Voice Conversion From Perturbation and Autoencoder Loss
Park, H.-Y., Lee, Y. H., & Chun, C. (2023) · IEEE Access · doi:10.1109/access.2023.3341434 · Cited by 7
“CycleGAN-VC is a method that does not rely on additional modules and parallel data that exhibited better performance in the Voice Conversion Challenge 2016 (VCC 2016) than those used by a GMM-based method trained using twice as many parallel datasets [43].”
MethodLi, Y., Lee, K. A., Yuan, Y., Li, H., & Yang, Z. (2018) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipa.2018.8659628 · Cited by 6
Improving Robustness of One-Shot Voice Conversion with Deep Discriminative Speaker Encoder
Du, H., & Xie, L. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-2132 · Cited by 8
Using a Manifold Vocoder for Spectral Voice and Style Conversion
Dinh, T., Kain, A. B., & Tjaden, K. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1176 · Cited by 5
“To evaluate vocoding quality, we used the voice conversion challenge (VCC) 2016 database [24], which features 5 male and 5 female speakers.”
MethodReplay Speech Detection Based on Dual-Input Hierarchical Fusion Network
Hu, C., Zhou, R., & Yuan, Q. (2023) · Applied Sciences · doi:10.3390/app13095350 · Cited by 6
MOSPC: MOS Prediction Based on Pairwise Comparison
Wang, K., Zhao, Y., Dong, Q., Ko, T., & Wang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.132 · Cited by 15
Synthetic Speech Spoofing Detection Based on Online Hard Example Mining
Hu, C., Zhou, R., & Yuan, Q. (2023) · IEEE Access · doi:10.1109/access.2023.3311849 · Cited by 4
“Attackers can create spoofed speech using methods such as mimicry, replay attacks [2], and synthetic speech attacks, which involve text-to-speech (TTS) [3], [4] and voice conversion (VC) [5], [6], to impersonate real users and deceive ASV systems and humans.”
BackgroundVoice Conversion and Spoofed Voice Detection from Parallel English and Urdu Corpus using Cyclic GANs
Saleem, S., Dilawari, A., Ghani Khan, M. U., & Husnain, M. (2019) · 2019 International Conference on Robotics and Automation in Industry (ICRAI) · doi:10.1109/icrai47710.2019.8967385 · Cited by 5
RAMP+: Retrieval-Augmented MOS Prediction With Prior Knowledge Integration
Wang, H., Zhao, S.-W., Zheng, X., Zhou, J., Wang, X., & Qin, Y. (2025) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslpro.2025.3552957 · Cited by 9
Dictionary update for NMF-based voice conversion using an encoder-decoder network
Hsu, C.-C., Hwang, H.-T., Wu, Y.-C., Tsao, Y., & Wang, H. (2016) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp.2016.7918382 · Cited by 4
“The proposed SC system was evaluated on a parallel English corpus from the Voice Conversion Challenge 2016 [17].”
MethodPersonalizing TTS Voices for Progressive Dysarthria
Zhao, Y., Song, M., Yue, Y., & Kuruvilla-Dugdale, M. (2021) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi50953.2021.9508522 · Cited by 6
Yang, J., Zhou, Y., & Huang, H. (2023) · Speech Communication · doi:10.1016/j.specom.2023.05.004 · Cited by 7
CCLCap-AE-AVSS: Cycle consistency loss based capsule autoencoders for audio–visual speech synthesis
Ghosh, S., Jana, N. D., Si, T., Mallik, S., & Shah, M. A. (2024) · Journal of Intelligent Systems · doi:10.1515/jisys-2023-0171 · Cited by 5
“However, these models often grappled with the challenge of faithfully retaining the intricate rhythms of prosody and the innate authenticity that characterizes human speech [31].”
Crossed-Time Delay Neural Network for Speaker Recognition
Chen, L., Liang, Y., Shi, X., Zhou, Y., & Wu, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67832-6_1 · Cited by 4
Self-citationVoice Reenactment with F0 and timing constraints and adversarial learning of conversions
Bous, F., Benaroya, L., Obin, N., & Röebel, A. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909854 · Cited by 4
“Primarily applied to identity conversion, VC has considerably gained in popularity and in quality thanks to neural VC [6]–[8].”
BackgroundAuthenticity at Risk: Key Factors in the Generation and Detection of Audio Deepfakes
Martínez-Serrano, A., Montero-Ramírez, C., & Peláez-Moreno, C. (2025) · Applied Sciences · doi:10.3390/app15020558 · Cited by 4
“In 2016 The Voice Conversion Challenge 2016 [9] was released to compare different voice conversion techniques using a common dataset.”
Shape-adaptive image compression using lossy shape coding, SA-prediction, and SA-deblocking
Chen, L., Ding, J., & Lee, Y.-C. (2016) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2016.7820772 · Cited by 3
Self-citationNon-parallel Many-to-many Singing Voice Conversion by Adversarial Learning
Hu, J., Yu, C., & Guan, F. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023357 · Cited by 3
The Interpretable and Transferable Adversarial Attack against Synthetic Speech Detectors
Deng, J., Ye, D., Li, J., Liu, Z., Tang, L., & Zhang, Y. (2025) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3727341 · Cited by 2
“Synthetic speech techniques commonly used by attackers include text-to-speech [10, 27, 33] and speech conversion (VC) [23, 32].”
Serenade: A Singing Style Conversion Framework Based on Audio Infilling
Violeta, L. P., Huang, W.-C., & Toda, T. (2025) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco63237.2025.11226227 · Cited by 6
Self-citationChange Your Singer: A Transfer Learning Generative Adversarial Framework for Song to Song Conversion
Daher, R., Zein, M. K., Zini, J. E., Awad, M., & Asmar, D. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206878 · Cited by 2
“We collected a dataset of non-parallel aligned song segments of 3 sec each on average to be consistent with the Voice Conversion Challenge 2016 (VCC 2016) dataset [22] used by [9].”
MethodSR-MT: a metamorphic method to test the robustness of speech recognition software
Wang, F., Ben, K., & Zhang, X. (2022) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/3524846.3527339 · Cited by 3
“Referring to the International Speech Conversion Challenge [21], naturalness and sharpness were used as subjective evaluation indexes.”
MethodEffects of Training Data Variety in Generating Glottal Pulses from Acoustic Features with DNNs
Airaksinen, M., & Alku, P. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-363 · Cited by 2
“The Voice Conversion Challenge (VCC) 2016 database [27] is a multi-speaker corpus consisting of 5 male and 5 female speakers.”
BackgroundIdentification of Synthetic Spoofed Speech with Deep Capsule Network
Mao, T., Yan, D., Gong, Y., & Wang, R. (2022) · Communications in computer and information science · doi:10.1007/978-981-19-0523-0_17 · Cited by 2
Correlation Loss for MOS Prediction of Synthetic Speech
Hu, B., & Li, Q. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980182 · Cited by 4
“BVCC is a MOS dataset that contains 7106 English synthetic samples from Blizzard Challenges [17-22], Voice Conversion Challenges [23-25], and ESPnet-TTS [26].”
BackgroundLatent-Based Neural Net for Non-Intrusive Speech Quality Assessment
Cumlin, F., Schüldt, C., & Chatterjee, S. (2023) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco58844.2023.10289840 · Cited by 4
“VCC2016 dataset consists of 26028 speech clips from 20 different systems [18].”
BackgroundMethodDevelopment of a computationally efficient voice conversion system on mobile phones
Gao, S., Wu, X., Xiang, C., & Huang, D. (2019) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2018.23 · Cited by 2
“…conversion systems only perform simple modifications of prosodic features, e.g., normalizing the global mean and variance of log-scaled f 0 values [8, 14], because prosody is a supra-segmental characteristic not conveyed on segmental basis but through larger units, which is more challenging [15].”
BackgroundSemi-Supervised Voice Conversion with Amortized Variational Inference
Stephenson, C., Keskin, G., Thomas, A., & Elibol, O. H. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1840 · Cited by 2
“Due the broad applicability and inherent difficulty of the problem, the design of VC systems has been an area of consistent interest for decades and continues to see active research [1, 2].”
BackgroundAn Evaluation of Postfiltering for Deep Learning Based Speech Synthesis with Limited Data
Lőrincz, B., Nuțu, M., Stan, A., & Giurgiu, M. (2020) · IEEE International Conference on Intelligent Systems · doi:10.1109/is48319.2020.9199932 · Cited by 2
“The postfiltering uses the same set of acoustic features extracted with WORLD, and a baseline network architecture as the one described in [33].”
MethodVoice Privacy with Smart Digital Assistants in Educational Settings
Niknazar, M., Vempaty, A., & Kokku, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80421-3_31 · Cited by 2
PROSODY-VC: Voice style transfer with speech prosody
Kalita, A. K., & Patra, R. (2023) · IEEE Guwahati Subsection Conference (GCON) · doi:10.1109/gcon58516.2023.10183559 · Cited by 1
“Methods Speech Corpus VCC 2018 [21] LibriSpeech MCD WER MCD WER proposed 6.”
BackgroundOthmane, I. B., Martino, J. D., & Ouni, K. (2019) · International Journal of Speech Technology · doi:10.1007/s10772-019-09643-4 · Cited by 1
“An extensive subjective evaluation was done during the Voice Conversion Challenge 2016 (Toda et al. 2016; Wester et al. 2016) with different submitted systems, such as those using a vocoder free VC framewor (Kobayashi et al. 2016).”
MethodLiu, J., Cui, Z., & Zhou, Y. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2024.109355 · Cited by 1
Voice Conversion Based on Unified Dictionary with Clustered Features between Non-parallel Corpus
Jin, H., & Yu, Y. (2018) · International Conference on Network and Information Systems for Computers (ICNISC) · doi:10.1109/icnisc.2018.00052 · Cited by 1
Survey Paper:Introduction to Generative Adversarial Networks in Speech Imitation
Guha, R., Sharma, V., Yadav, Y., & Bhat, A. (2020) · International Conference on Intelligent Computing and Control Systems (ICICCS) · doi:10.1109/iciccs48265.2020.9120898 · Cited by 1
“For verified benchmarks, various challenges such as Voice conversion Challenge [16, 20] are organized.”
BackgroundVillavicencio, F., Yamagishi, J., Bonada, J., & Espic, F. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-305
“It contains five source and five different target speakers, resulting in twenty five conversion pairs, all of them requested by the task of the challenge (see [34] for further information of the VCC2016 task) .”
BackgroundSelf-citationHigh quality voice conversion based on ISODATA clustering algorithm
Li, Y., Zuo, Y., Yang, Z., & Shao, X. (2017) · International Conference on Intelligent Systems and Knowledge Engineering (ISKE) · doi:10.1109/iske.2017.8258822
“Personification of synthesized speech is another important application for many automated systems which use synthesized speech as a human computer interaction tool, for example, we expect that machine can speak with a desired voice or read out email in the sender’s voice [2-7].”
BackgroundNon-parallel Voice Conversion Using Generative Adversarial Networks
蓮沼, 勇., Hirayama, C., Kobayashi, M., & Nagao, T. (2018) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2018.00283
Non-parallel dictionary learning for voice conversion using non-negative Tucker decomposition
Takashima, Y., Nakashika, T., Takiguchi, T., & Ariki, Y. (2019) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-019-0160-1 · Cited by 1
Semi-supervised voice conversion with amortized variational inferencePreprint
Stephenson, C., Keskin, G., Thomas, A., & Elibol, O. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00067
“Due the broad applicability and inherent difficulty of the problem, the design of VC systems has been an area of consistent interest for decades and continues to see active research [1, 2].”
BackgroundInvestigating Deep Neural Structures and their Interpretability in the Domain of Voice Conversion
Broughton, S. J., Jalal, A., & Moore, R. K. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1730
Exploring Source Separation as a Countermeasure for Voice Conversion Spoofing Attack
Hemavathi, R., Thoshith, S., & Kumaraswamy, R. (2021) · Transactions on computer systems and networks · doi:10.1007/978-981-16-1681-5_24
Hiding Sensitive Information in Desensitized Voice Sequences
Huang, H., Zhang, J., Chen, D., & Yan, H. (2023) · International Conference on Data Security and Privacy Protection (DSPP) · doi:10.1109/dspp58763.2023.10405307
“A. Setup 1) Dataset: VCC 2016 dataset, where VCC means Voice Conversion Challenge [12]–[14], which is what it was designed for, is employed for training of F D and generation of data for our evaluation.”
MethodNVCGAN: Leveraging Generative Adversarial Networks for Robust Voice Conversion
Zhang, G., Liu, J., Bi, W., Dongye, G., Zhang, L., Jing, M., & Yu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5666-7_28 · Cited by 1
Multi-Perspective Transfer Learning for Automatic MOS Prediction of Low Resource Language
Yin, P., Liu, R., Bao, F., & Gao, G. (2024) · International Conference on Asian Language Processing · doi:10.1109/ialp63756.2024.10661179
“…languages [10], [11] such as English, and their remarkable performance cannot be separated from the support of large-scale data, such as VCC2016 [12], VCC2018 [13], BVCC [2], BC2019 [14] and ASV2019 [15] et al. However, for low-resource languages, there is no large-scale MOS prediction data,…”
BackgroundDetecting Spoofed Noisy Speeches via Activation-Based Residual Blocks for Embedded Systems
Zhan, J., Peng, S., Jiang, W., Wang, X., & Liu, J. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3437331
Progress and Challenges in DNN-Based Objective Quality Assessment of Synthesized Speech
Cooper, E. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249252
“The Voice Conversion Challenge (VCC), which began in 2016 [13], is a similar challenge for voice conversion that also shares listening test data.”
An, W., Li, R., Li, H., & Ge, H. (2026) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-026-11071-x · Cited by 1
User-Customizable Voice Anonymization Through Personalized Style Transfer
Putri, W. R., Lu, C.-S., & Wang, J.-C. (2025) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb65343.2025.11410889
“These models were pre-trained on the VCC 2016 [54] and VCC 2018 [28] datasets, which include extensive MOS rating scores evaluated by many participants.”
Wang, Z., Xia, X., Huang, C., & Xie, L. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463480 · Cited by 5
“The Voice Conversion Challenge (VCC) series [1, 2, 3] has fostered reproducible comparisons across competing systems by providing common datasets, evaluation protocols, and baseline implementations.”
Instrument-to-Instrument Timbre Conversion Using MaskCycleGAN
Lai, W.-H., & Hsieh, C.-W. (2026) · International Conference on Information Networking · doi:10.1109/icoin68469.2026.11480919
“Compared to voice conversion, which has received significant attention and has been the subject of the Voice Conversion Challenge held in 2016, 2018, and 2020 [3]-[5], research on instrument-to-instrument timbre conversion is relatively recent.”
A Hybrid Attention-based Speaker Conditioning with dynamic loss tuning for Voice Conversion
Kalita, A. K., & Patra, R. (2026) · Circuits Systems and Signal Processing · doi:10.1007/s00034-026-03737-6
Hu, C.-H., Yasuda, Y., & Toda, T. (2026) · APSIPA Transactions on Signal and Information Processing · doi:10.1108/atsip-01-2026-0007
Self-citation
Metadata
- Authors
- Toda, T., Chen, L.-H., Saito, D., Villavicencio, F., Wester, M., Wu, Z., & Yamagishi, J.
- Affiliations
- Nagoya University (JP)
- University of Science and Technology of China (CN)
- The University of Tokyo (JP)
- National Institute of Informatics (JP)
- University of Edinburgh (GB)
- Venue
- Interspeech 2016
- Publisher
- ISCA
- Published
- 2016-09-08
- Pages
- 1632-1636
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2473388484
- Cited by
- 156
- References
- 40
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 12 · 2023: 16 · 2022: 18 · 2021: 20 · 2020: 21 · 2019: 25 · 2018: 14 · 2017: 11 · 2016: 8
- FWCI
- 10.42
- Type
- conference paper
- Language
- en
- Open access
- Licence
- other-oa
- Locations
- Interspeech (USB) (publishedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
- Funders
- Engineering and Physical Sciences Research Council (EP/J002526/1, EP/I031022/1)
- Japan Society for the Promotion of Science (26280060)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Health Professions › Speech and Hearing › Voice and Speech Disorders
- Keywords
- voice conversion, naturalness evaluation, listener evaluation, speech synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.