The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
A bit of progress in language modeling
Open accessAbstract
In the past several years, a number of different language modeling improvements over simple trigram models have been found, including caching, higher-order n -grams, skipping, interpolated Kneser?Ney smoothing, and clustering. We present explorations of variations on, or of the limits of, each of these techniques, including showing that sentence mixture models may have more potential. While all of these techniques have been studied separately, they have rarely been studied in combination. We compare a combination of all techniques together to a Katz smoothed trigram model with no count cutoffs. We achieve perplexity reductions between 38 and 50% (1 bit of entropy), depending on training data size, as well as a word error rate reduction of 8.9%. Our perplexity reductions are perhaps the highest reported compared to a fair baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
THE POPULATION FREQUENCIES OF SPECIES AND THE ESTIMATION OF POPULATION PARAMETERS
GOOD, I. J. (1953) · Biometrika · doi:10.1093/biomet/40.3-4.237 · Cited by 3,437
(1992) · doi:10.5555/176313.176316 · Cited by 2,892
An empirical study of smoothing techniques for language modeling
Chen, S. F., & Goodman, J. (1999) · Computer Speech & Language · doi:10.1006/csla.1999.0128 · Cited by 2,066
Self-citationA statistical approach to machine translation
Brown, P. F., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty, J., Mercer, R. L., & Roossin, P. S. (1990) · doi:10.5555/92858.92860 · Cited by 1,655
Estimation of probabilities from sparse data for the language model component of a speech recognizer
Katz, S. (1987) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1987.1165125 · Cited by 1,637
2007 IEEE International Conference on Acoustics, Speech, and Signal Processing
(2007) · 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07 · doi:10.1109/icassp.2007.366148 · Cited by 1,520
Scatter/Gather: A Cluster-based Approach to Browsing Large Document Collections
Cutting, D. R., Karger, D. R., Pedersen, J. O., & Tukey, J. W. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130362 · Cited by 1,419
Generalized Iterative Scaling for Log-Linear Models
Darroch, J. N., & Ratcliff, D. (1972) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177692379 · Cited by 1,206
Cited by
Extensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., & Khudanpur, S. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947611 · Cited by 1,568
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244303322533223 · Cited by 1,566
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
“Some early [BB01, Goo01] work found power-law scalings between performance and dataset size.”
BackgroundSukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
(2005) · Cited by 823
Neural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821
Neural Probabilistic Language Models
Bengio, Y., Schwenk, H., Senécal, J.-S., Morin, F., & Gauvain, J.-L. (2006) · Studies in Fuzziness and Soft Computing · doi:10.1007/10985687_6 · Cited by 540
Continuous space language models
Schwenk, H. (2007) · Computer Speech & Language · doi:10.1016/j.csl.2006.09.003 · Cited by 516
A hierarchical Bayesian language model based on Pitman-Yor processes
Teh, Y. W. (2006) · Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL - ACL '06 · doi:10.3115/1220175.1220299 · Cited by 489
Pulse: Mining Customer Opinions from Free Text
Gamon, M., Aue, A., Corston-Oliver, S., & Ringger, E. (2005) · Lecture Notes in Computer Science · doi:10.1007/11552253_12 · Cited by 446
Probabilistic finite-state machines - part I
Vidal, E., Thollard, F., de la Higuera, C., Casacuberta, F., & Carrasco, R. C. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.147 · Cited by 301
The Effects of Feature‐Label‐Order and Their Implications for Symbolic Learning
Ramscar, M., Yarlett, D., Dye, M., Denny, K., & Thorpe, K. (2010) · Cognitive Science · doi:10.1111/j.1551-6709.2009.01092.x · Cited by 300
Li, H.-L., Pang, Y.-H., & Liu, B. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab829 · Cited by 269
Effects of Language Modeling and its Personalization on Touchscreen Typing Performance
Fowler, A., Partridge, K., Chelba, C., Bi, X., Ouyang, T., & Zhai, S. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702503 · Cited by 127
Unlimited vocabulary speech recognition with morph language models applied to Finnish
Hirsimäki, T., Creutz, M., Siivola, V., Kurimo, M., Virpioja, S., & Pylkkönen, J. (2006) · Computer Speech & Language · doi:10.1016/j.csl.2005.07.002 · Cited by 124
On Growing and Pruning Kneser–Ney Smoothed $ N$-Gram Models
Siivola, V., Hirsimaki, T., & Virpioja, S. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2007.896666 · Cited by 80
Augmenting Self-attention with Persistent Memory
Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., & Joulin, A. (2019) · arXiv · doi:10.48550/arxiv.1907.01470 · Cited by 75
On Compression-Based Text Classification
Marton, Y., Wu, N., & Hellerstein, L. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-31865-1_22 · Cited by 59
Metadata
- Authors
- Goodman, J. T.
- Venue
- Computer Speech & Language
- Publisher
- Elsevier BV
- Published
- 2001-10
- Volume
- 15
- Issue
- 4
- Pages
- 403-434
- ISSN
- 0885-2308 (ISSN-L)
- OpenAlex
- W2111305191
- Cited by
- 469
- References
- 43
- FWCI
- 15.02
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.