The DOI matches its Crossref record.
Perplexity—a measure of the difficulty of speech recognition tasks
Open accessAbstract
Using counterexamples, we show that vocabulary size and static and dynamic branching factors are all inadequate as measures of speech recognition complexity of finite state grammars. Information theoretic arguments show that perplexity (the logarithm of which is the familiar entropy) is a more appropriate measure of equivalent choice. It too has certain weaknesses which we discuss. We show that perplexity can also be applied to languages having no obvious statistical description, since an entropy-maximizing probability assignment can be found for any finite-state grammar. Table I shows perplexity values for some well-known speech recognition tasks. Perplexity Vocabulary Dynamic Phone Word size branching factor IBM-Lasers 2.14 21.11 1000 1000 IBM-Raleigh 1.69 7.74 250 7.32 CMU-AIX05 1.52 6.41 1011 35
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Texygen: A Benchmarking Platform for Text Generation Models
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., & Yu, Y. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210080 · Cited by 408
A Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251
Deep Learning in Protein Structural Modeling and Design
Gao, W., Mahajan, S. P., Sulam, J., & Gray, J. J. (2020) · Patterns · doi:10.1016/j.patter.2020.100142 · Cited by 227
Using natural language processing to analyse text data in behavioural science
Feuerriegel, S., Maarouf, A., Bär, D., Geissler, D., Schweisthal, J., Pröllochs, N., Robertson, C. E., Rathje, S., Hartmann, J., Mohammad, S. M., et al. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-024-00392-z · Cited by 140
Green AI: do deep learning frameworks have different costs?
Georgiou, S., Kechagia, M., Sharma, T., Sarro, F., & Zou, Y. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510221 · Cited by 102
Neuronal parts list and wiring diagram for a visual system
Matsliah, A., Yu, S.-C., Kruk, K., Bland, D., Burke, A. T., Gager, J., Hebditch, J., Silverman, B., Willie, K. P., Willie, R., et al. (2024) · Nature · doi:10.1038/s41586-024-07981-1 · Cited by 85
Oviatt, S. L., Cohen, P. R., & Wang, M. (1994) · Speech Communication · doi:10.1016/0167-6393(94)90079-5 · Cited by 76
Pakhomov, S. V. S., Smith, G. E., Chacon, D., Feliciano, Y., Graff-Radford, N., Caselli, R., & Knopman, D. S. (2010) · Cognitive and Behavioral Neurology · doi:10.1097/wnn.0b013e3181c5dde3 · Cited by 64
Hosseini, E. A., Schrimpf, M., Zhang, Y., Bowman, S., Zaslavsky, N., & Fedorenko, E. (2024) · Neurobiology of Language · doi:10.1162/nol_a_00137 · Cited by 63
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
“…on some data distribution D is to measure perplexity , which is defined as the exponential of the average negative loglikelihood per token (Jelinek et al., 2005; Brown et al., 1992), that is: where | D | is the number of documents in the dataset, y j is the j -th document in D , N j is…”
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
Cao, J., Chen, S., Zhang, W., Lo, H. C., Li, Y., & Cheung, S.-C. (2025) · Proceedings of the 16th International Conference on Internetware · doi:10.1145/3755881.3755901 · Cited by 2
Does Few-Shot Learning Help LLM Performance in Code Synthesis?
Xu, D., Xie, T., Xia, B., Li, H., Bai, Y., Sun, Y., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2412.02906 · Cited by 3
“Because Pass@1 is a discrete metric, we optimize a more granular surrogate objective: the perplexity of generating the ground truth code, P ERPLEXITY (T ARGET ), as described in Equation 1, where perplexity is defined as P P ( y ; (Jelinek et al., 1977).”
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“…of sequences S = { s 1 , . . . , s N } is perplexity (PPL), which is defined as the exponential of the average negative log-likelihood per token [53, 17, 12]: i =1 log P ( s j,i | s j, 1 , . . . , s j,i − 1 ) where | S | is the number of sequences, s j is the j’th sequence in S and s j,i is the…”
Metadata
- Authors
- Jelinek, F., Mercer, R. L., Bahl, L. R., & Baker, J. K.
- Venue
- The Journal of the Acoustical Society of America
- Publisher
- Acoustical Society of America (ASA)
- Published
- 1977-12-01
- Volume
- 62
- Issue
- S1
- Pages
- S63-S63
- ISSN
- 0001-4966 (ISSN-L)
- 1520-8524
- OpenAlex
- W2058373514
- Cited by
- 398
- References
- 0
- Type
- conference abstract
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
