The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
A bit of progress in language modeling
Open accessAbstract
In the past several years, a number of different language modeling improvements over simple trigram models have been found, including caching, higher-order n -grams, skipping, interpolated Kneser?Ney smoothing, and clustering. We present explorations of variations on, or of the limits of, each of these techniques, including showing that sentence mixture models may have more potential. While all of these techniques have been studied separately, they have rarely been studied in combination. We compare a combination of all techniques together to a Katz smoothed trigram model with no count cutoffs. We achieve perplexity reductions between 38 and 50% (1 bit of entropy), depending on training data size, as well as a word error rate reduction of 8.9%. Our perplexity reductions are perhaps the highest reported compared to a fair baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
THE POPULATION FREQUENCIES OF SPECIES AND THE ESTIMATION OF POPULATION PARAMETERS
GOOD, I. J. (1953) · Biometrika · doi:10.1093/biomet/40.3-4.237 · Cited by 3,437
(1992) · doi:10.5555/176313.176316 · Cited by 2,892
An empirical study of smoothing techniques for language modeling
Chen, S. F., & Goodman, J. (1999) · Computer Speech & Language · doi:10.1006/csla.1999.0128 · Cited by 2,066
Self-citationA statistical approach to machine translation
Brown, P. F., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty, J., Mercer, R. L., & Roossin, P. S. (1990) · doi:10.5555/92858.92860 · Cited by 1,655
Estimation of probabilities from sparse data for the language model component of a speech recognizer
Katz, S. (1987) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1987.1165125 · Cited by 1,637
2007 IEEE International Conference on Acoustics, Speech, and Signal Processing
(2007) · 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07 · doi:10.1109/icassp.2007.366148 · Cited by 1,520
Scatter/Gather: A Cluster-based Approach to Browsing Large Document Collections
Cutting, D. R., Karger, D. R., Pedersen, J. O., & Tukey, J. W. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130362 · Cited by 1,419
Generalized Iterative Scaling for Log-Linear Models
Darroch, J. N., & Ratcliff, D. (1972) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177692379 · Cited by 1,206
Distributional clustering of English words
Pereira, F., Tishby, N., & Lee, L. (1993) · Proceedings of the 31st annual meeting on Association for Computational Linguistics - · doi:10.3115/981574.981598 · Cited by 974
A stochastic parts program and noun phrase parser for unrestricted text
Church, K. W. (1988) · Proceedings of the second conference on Applied natural language processing - · doi:10.3115/974235.974260 · Cited by 957
Scatter/Gather: a cluster-based approach to browsing large document collections
Cutting, D. R., Karger, D. R., Pedersen, J. O., & Tukey, J. W. (1992) · Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '92 · doi:10.1145/133160.133214 · Cited by 953
Interpolated estimation of Markov source parameters from sparse data
Jelinek, F. (1980) · Medical Entomology and Zoology · Cited by 823
Two decades of statistical language modeling: where do we go from here?
Rosenfeld, R. (2000) · Proceedings of the IEEE · doi:10.1109/5.880083 · Cited by 735
On structuring probabilistic dependences in stochastic language modelling
Ney, H., Essen, U., & Kneser, R. (1994) · Computer Speech & Language · doi:10.1006/csla.1994.1001 · Cited by 517
A cache-based natural language model for speech recognition
Kuhn, R., & De Mori, R. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.56193 · Cited by 495
The SPHINX-II speech recognition system: an overview
Huang, X., Alleva, F., Hon, H.-W., Hwang, M.-Y., Lee, K.-F., & Rosenfeld, R. (1993) · Computer Speech & Language · doi:10.1006/csla.1993.1007 · Cited by 368
Exploiting latent semantic information in statistical language modeling
Bellegarda, J. R. (2000) · Proceedings of the IEEE · doi:10.1109/5.880084 · Cited by 349
A Gaussian Prior for Smoothing Maximum Entropy Models
Chen, S. F., & Rosenfeld, R. (1999) · doi:10.21236/ada360974 · Cited by 345
Immediate-head parsing for language models
Charniak, E. (2001) · Proceedings of the 39th Annual Meeting on Association for Computational Linguistics - ACL '01 · doi:10.3115/1073012.1073029 · Cited by 293
Adaptive Statistical Language Modeling: A Maximum Entropy Approach
Rosenfeld, R. (2018) · doi:10.1184/r1/6603032 · Cited by 269
A spelling correction program based on a noisy channel model
Kernighan, M. D., Church, K. W., & Gale, W. A. (1990) · Proceedings of the 13th conference on Computational linguistics - · doi:10.3115/997939.997975 · Cited by 262
Classes for fast maximum entropy training
Goodman, J. (2002) · 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221) · doi:10.1109/icassp.2001.940893 · Cited by 219
Self-citationImproved clustering techniques for class-based statistical language modelling
Kneser, R., & Ney, H. (1993) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1993-229 · Cited by 193
Exploiting Syntactic Structure for Language ModelingPreprint
Chelba, C. I., & Jelinek, F. (1998) · Cited by 163
Modeling long distance dependence in language: topic mixtures versus dynamic cache models
Iyer, R. M., & Mari Ostendorf, E. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.736328 · Cited by 157
Aggregate and mixed-order Markov models for statistical language processingPreprint
Saul, L. K., & Pereira, F. M. B. (1997) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9706007 · Cited by 148
A dynamic language model for speech recognition
Jelinek, F., Mérialdo, B., Roukos, S., & Strauss, M. (1991) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/112405.112464 · Cited by 126
Expoiting Syntactic Structure for Language ModelingPreprint
Chelba, C. I., & Jelinek, F. (1998) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/9811022 · Cited by 122
A Bit of Progress in Language Modeling Extended VersionPreprint
Goodman, J. (2001) · Cited by 113
Self-citation1993 International conference on acoustics, speech and signal processing
(1992) · Signal Processing · doi:10.1016/0165-1684(92)90105-6 · Cited by 107
Variable n-grams and extensions for conversational speech language modeling
Siu, M.-H., & Mari Ostendorf, E. (2000) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.817454 · Cited by 91
Using a stochastic context-free grammar as a language model for speech recognition
Jurafsky, D., Wooters, C., Segal, J., Stolcke, A., Fosler, E., Tajchaman, G., & Morgan, N. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1995.479396 · Cited by 89
Whole-sentence exponential language models: a vehicle for linguistic-statistical integration
Rosenfeld, R., Chen, S. F., & Zhu, X. (2001) · Computer Speech & Language · doi:10.1006/csla.2000.0159 · Cited by 88
Niesler, T., Whittaker, E. W. D., & Woodland, P. C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1998.674396 · Cited by 66
Language model size reduction by pruning and clustering
Goodman, J., & Gao, J. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-490 · Cited by 46
Self-citationMulti-class composite N-gram based on connection direction
Yamamoto, H., & Sagisaka, Y. (1999) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1999.758180 · Cited by 31
Exploiting syntactic structure for language modeling
Chelba, C., & Jelinek, F. (1998) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/980845.980882 · Cited by 30
Objective methods for evaluating synthetic intonation
Clark, R. A. J., & Dusterhoff, K. E. (1999) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1999-426 · Cited by 28
Efficient sampling and feature selection in whole sentence maximum entropy language models
Chen, S. F., & Rosenfeld, R. (1999) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1999.758184 · Cited by 28
Probabilistic models of short and long distance word dependencies in running text
Kupiec, J. (1989) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/100964.101006 · Cited by 27
Combining Statistical and Syntactic Methods in Recognizing Handwritten Sentences
Srihari, R. K., & Baltus, C. M. (1992) · Cited by 20
Cited by
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
“Some early [BB01, Goo01] work found power-law scalings between performance and dataset size.”
BackgroundSukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
(2005) · Cited by 823
Neural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821
Continuous space language models
Schwenk, H. (2007) · Computer Speech & Language · doi:10.1016/j.csl.2006.09.003 · Cited by 516
Metadata
- Authors
- Goodman, J. T.
- Venue
- Computer Speech & Language
- Publisher
- Elsevier BV
- Published
- 2001-10
- Volume
- 15
- Issue
- 4
- Pages
- 403-434
- ISSN
- 0885-2308 (ISSN-L)
- OpenAlex
- W2111305191
- Cited by
- 469
- References
- 43
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
