CoChat
Sign Up
Cited by 28Speech Recognition and SynthesisNatural Language Processing TechniquesSpeech and dialogue systems

TheanoLM — An Extensible Toolkit for Neural Network Language Modeling

Open access
Enarvi, Seppo, Kurimo, Mikko·2016·Interspeech 2016
DOI 10.21437/interspeech.2016-618
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a new tool for training neural network language models (NNLMs), scoring sentences, and generating text. The tool has been written using Python library Theano, which allows researcher to easily extend it and tune any aspect of the training process. Regardless of the flexibility, Theano is able to generate extremely fast native code that can utilize a GPU or multiple CPU cores in order to parallelize the heavy numerical computations. The tool has been evaluated in difficult Finnish and English conversational speech recognition tasks, and significant improvement was obtained over our best back-off n-gram models. The results that we obtained in the Finnish task were compared to those from existing RNNLM and RWTHLM toolkits, and found to be as good or better, while training times were an order of magnitude shorter.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21437/interspeech.2016-618

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The accuracy of the first-pass recognition exceeded earlier results on both data sets due to the new neural network acoustic models.

    “The accuracy of the first-pass recognition exceeded our earlier results on both data sets [11], [49], due to the new neural network acoustic models.”

    From Automatic Speech Recognition With Very Large Conversational Finnish and Estonian Vocabularies (2017) · cites this paper as [11]

    The Conclusions (p. 4) say the Finnish task improved on an earlier record and attribute the improved baseline to newly collected acoustic data. They do not report that first-pass accuracy exceeded earlier results on both data sets. The Data section (p. 3) describes a triphone HMM acoustic model for Finnish, not a new neural network acoustic model. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The best previously published results are 48.4% WER in the Finnish task and 52.7% WER in the Estonian task.

    “The best previously published results are 48.4 % WER in the Finnish task [11] and 52.7 % WER in the Estonian task [49].”

    From Automatic Speech Recognition With Very Large Conversational Finnish and Estonian Vocabularies (2017) · cites this paper as [11]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. [1][1] H. Schwenk and J. L. Gauvain, 'Connectionist language modeling for large vocabulary continuous speech recognition,' in Proceedings of the 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , vol. 1, May 2002, pp. I-765-I-768. - [11] Y. Nesterov, 'A method of solving a convex programming problem with convergence rate O (1 /k 2 ) ,' Soviet Mathematics Doklady , vol. 27, no. 2, pp. 372-376, 1983. - [12] J. Duchi, E. Hazan, and Y. Singer, 'Adaptive subgradient methods for online learning and stochastic optimization,' The Journal of Machine Learning Research , vol. 12, pp. 2121-2159, Jul. 2011. - [13] M. D. Zeiler, 'ADADELTA: an adaptive learning rate method,' Computing Research Repository , vol. abs/1212.5701, 2012. [Online]. Available: http://arxiv.org/abs/1212.5701 - · arXiv
  2. [3][3] T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur, 'Recurrent neural network based language model,' in Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTERSPEECH) , Sep. 2010, pp. 1045-1048. [Online]. Available: http://www.isca-speech.org/archive/interspeech\_2010/i10\_1045.html -
  3. [4][4] S. Hochreiter and J. Schmidhuber, 'Long shortterm memory,' Neural Computation , vol. 9, no. 8, pp. 1735-1780, Nov. 1997. [Online]. Available: http://dx.doi.org/10.1162/neco.1997.9.8.1735 - · DOI · Paper page
  4. [5][5] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin, 'A neural probabilistic language model,' The Journal of Machine Learning Research , vol. 3, pp. 1137-1155, Mar. 2003. [Online]. Available: http://dl.acm.org/citation.cfm?id=944919.944966 - [16] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, 'Dropout: A simple way to prevent neural networks from overfitting,' The Journal of Machine Learning Research , vol. 15, no. 1, pp. 1929-1958, Jan. 2014. [Online]. Available: http://dl.acm.org/citation.cfm?id=2627435.2670313 - [17] T. Hirsimäki, J. Pylkkönen, and M. Kurimo, 'Importance of high-order n-gram models in morph-based speech recognition,' IEEE Transactions on Audio, Speech, and Language Processing , vol. 17, no. 4, pp. 724-732, 2009. - [18] M. Kurimo, S. Enarvi, O. Tilk, M. Varjokallio, A. Mansikkaniemi, and T. Alumäe, 'Modeling under-resourced languages for speech recognition,' Language Resources and Evaluation (LRE) , 2016. -
  5. [6][6] H. Schwenk, 'CSLM - a modular open-source continuous space language modeling toolkit,' in Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH) , Aug. 2013, pp. 1198-1202. [Online]. Available: http://www.isca-speech.org/archive/interspeech\_2013/i13\_1198.html - [19] R. Kneser and H. Ney, 'Forming word classes by statistical clustering for statistical language modelling,' in Contributions to Quantitative Linguistics , R. Köhler and B. Rieger, Eds. Springer Netherlands, 1993, pp. 221-226. [Online]. Available: http://dx.doi.org/10.1007/978-94-011-1769-2\_15 - · DOI
  6. [7][7] F. Morin and Y. Bengio, 'Hierarchical probabilistic neural network language model,' in Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics (AISTATS) . Society for Artificial Intelligence and Statistics, 2005, pp. 246-252. [Online]. Available: http://www.iro.umontreal.ca/~lisa/pointeurs/hierarchical-nnlm-aistats05.pdf -
  7. [8][8] M. Sundermeyer, R. Schlüter, and H. Ney, 'rwthlm - the RWTH Aachen University neural network language modeling toolkit,' in Proceedings of the 15th Annual Conference of the International Speech Communication Association (INTERSPEECH) , Sep. 2014, pp. 2093-2097. -
  8. [9][9] T. Mikolov, S. Kombrink, A. Deoras, L. Burget, and J. ˇ Cernocký, 'RNNLM - recurrent neural network language modeling toolkit,' in Proceedings of the 2011 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) . IEEE Signal Processing Society, Dec. 2011, iEEE Catalog No.: CFP11SRW-USB. [Online]. Available: http://www.fit.vutbr.cz/research/view\_pub.php?id=10087 -
  9. [10][10] F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. J. Goodfellow, A. Bergeron, N. Bouchard, and Y. Bengio, 'Theano: new features and speed improvements,' NIPS 2012 Workshop on Deep Learning and Unsupervised Feature Learning, 2012. - [20] T. Mikolov, S. Kombrink, L. Burget, J. ˇ Cernocký, and S. Khudanpur, 'Extensions of recurrent neural network language model,' in Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , May 2011, pp. 5528-5531. - [21] A. Venkataraman and W. Wang, 'Techniques for effective vocabulary selection,' in Proceedings of the 8th European Conference on Speech Communication and Technology (EUROSPEECH) , Sep. 2003. [Online]. Available: http://www.isca-speech.org/archive/eurospeech\_2003/e03\_0245.html - [22] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, 'The Kaldi speech recognition toolkit,' in Proceedings of the 2011 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) . IEEE Signal Processing Society, Dec. 2011, iEEE Catalog No.: CFP11SRW-USB. [23] S. Kombrink, T. Mikolov, M. Karafiát, and L. Burget, 'Recurrent neural network based language modeling in meeting recognition,' in Proceedings of the 12th Annual Conference of the International Speech Communication Association (INTERSPEECH) , Aug. 2011, pp. 2877-2880. [Online]. Available: http://www.isca-speech.org/archive/interspeech\_2011/i11\_2877.html

Cited by

Sorted by citations

Authors
Enarvi, S., & Kurimo, M.
Affiliations
Aalto University (FI)
ORCID
Kurimo, Mikko
Venue
Interspeech 2016
Publisher
ISCA
Published
2016-09-08
Pages
3052-3056
ISSN
2308-457X (ISSN-L)
DOI
10.21437/interspeech.2016-618
OpenAlex
W2345764277
Cited by
28
References
1
Citations by year
2025: 1 · 2024: 1 · 2023: 3 · 2021: 3 · 2020: 5 · 2019: 5 · 2018: 2 · 2017: 6 · 2016: 2
FWCI
2.69
Type
conference paper
Language
en
Open access
Locations
  • Interspeech (USB) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Aaltodoc (Aalto University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
neural language models, Theano, conversational speech recognition, English speech recognition, text generation, sentence scoring, GPU acceleration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review