The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Spoken language understanding using long short-term memory neural networks
Abstract
Neural network based approaches have recently produced record-setting performances in natural language understanding tasks such as word labeling. In the word labeling task, a tagger is used to assign a label to each word in an input sequence. Specifically, simple recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have shown to significantly outperform the previous state-of-the-art - conditional random fields (CRFs). This paper investigates using long short-term memory (LSTM) neural networks, which contain input, output and forgetting gates and are more advanced than simple RNN, for the word labeling task. To explicitly model output-label dependence, we propose a regression model on top of the LSTM un-normalized scores. We also propose to apply deep LSTM to the task. We investigated the relative importance of each gate in the LSTM by setting other gates to a constant and only learning particular gates. Experiments on the ATIS dataset validated the effectiveness of the proposed models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“LSTM [30] [31] has some advanced properties compared to the simple RNN.”
(2010) · Cited by 8,261
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“In recent years, neural network based approaches have demonstrated outstanding performance in a variety of natural language processing tasks [1] [2] [3] [4] [5] [6] [7] [8].”
BackgroundNatural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Learning to forget: Continual prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 4,823
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Self-citationGenerating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
Extensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., & Khudanpur, S. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947611 · Cited by 1,568
Serial Order: A Parallel Distributed Processing Approach
Jordan, M. I. (1997) · Advances in Psychology · doi:10.1016/s0166-4115(97)80111-2 · Cited by 1,242
(2005) · Cited by 823
The ATIS spoken language systems pilot corpus
Hemphill, C. T., Godfrey, J. J., & Doddington, G. R. (1990) · Proceedings of the workshop on Speech and Natural Language - HLT '90 · doi:10.3115/116580.116613 · Cited by 790
LSTM recurrent networks learn simple context-free and context-sensitive languages
Gers, F. A., & Schmidhuber, E. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.963769 · Cited by 722
(2014) · International Conference on Learning Representations · Cited by 634
An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories
Williams, R. J., & Peng, J. (1990) · Neural Computation · doi:10.1162/neco.1990.2.4.490 · Cited by 577
Context dependent recurrent neural network language model
Mikolov, T., & Zweig, G. (2012) · 2012 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424228 · Cited by 560
Self-citationContinuous space language models
Schwenk, H. (2007) · Computer Speech & Language · doi:10.1016/j.csl.2006.09.003 · Cited by 516
Strategies for training large scale neural network language models
Mikolov, T., Deoras, A., Povey, D., Burget, L., & Cernocky, J. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163930 · Cited by 509
Fast and Robust Neural Network Joint Models for Statistical Machine Translation
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., & Makhoul, J. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1129 · Cited by 481 · cited 1×
“In recent years, neural network based approaches have demonstrated outstanding performance in a variety of natural language processing tasks [1] [2] [3] [4] [5] [6] [7] [8].”
BackgroundMesnil, G., He, X., Deng, L., & Bengio, Y. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-596 · Cited by 413
Evaluation of spoken language systems: the ATIS domain
Price, P. J. (1990) · Proceedings of the workshop on Speech and Natural Language - HLT '90 · doi:10.3115/116580.116612 · Cited by 385
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2021068 · Cited by 346
An Introduction to Computational Networks and the Computational Network Toolkit
Yu, D., Eversole, A., Seltzer, M., Yao, K., Kuchaiev, O., Zhang, Y., Seide, F. T. B., Huang, Z., Guenter, B. K., Wang, H., et al. (2014) · Cited by 332
Self-citationConvolutional neural network based triangular CRF for joint intent detection and slot filling
Xu, P., & Sarikaya, R. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707709 · Cited by 326
Recurrent neural networks for language understanding
Yao, K., Zweig, G., Hwang, M.-Y., Shi, Y., & Yu, D. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-569 · Cited by 315
Self-citationNatural Language Processing (Almost) from Scratch
CollobertRonan, WestonJason, BottouLéon, KarlenMichael, KavukcuogluKoray, & KuksaPavel (2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078186 · Cited by 306
Generative and discriminative algorithms for spoken language understanding
Raymond, C., & Riccardi, G. (2007) · Interspeech 2007 · doi:10.21437/interspeech.2007-448 · Cited by 295
Koutník, J., Greff, K., Gomez, F. J., & Schmidhuber, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.3511 · Cited by 262
How to Construct Deep Recurrent Neural Networks
Pascanu, R., Gülçehre, Ç., Cho, K., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6026 · Cited by 228
Deep Neural Network Language Models
Arısoy, E., Sainath, T. N., Kingsbury, B. E. D., & Ramabhadran, B. (2012) · Cited by 174
Structured Output Layer neural network language model
Le, H.-S., Oparin, I., Allauzen, A., Gauvain, J., & Yvon, F. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947610 · Cited by 152
The application of semantic classification trees to natural language understanding
Kühn, R., & Mori, R. D. (1995) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.391397 · Cited by 135
Discriminative spoken language understanding using word confusion networks
Henderson, M., Gašić, M., Thomson, B., Tsiakoulis, P., Yu, K., & Young, S. J. (2012) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424218 · Cited by 123
Recurrent conditional random field for language understanding
Yao, K., Peng, B., Zweig, G., Yu, D., Li, X., & Gao, F. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854368 · Cited by 112
Self-citationA generalized LSTM-like training algorithm for second-order recurrent neural networks
Monner, D. D., & Reggia, J. A. (2011) · Neural Networks · doi:10.1016/j.neunet.2011.07.003 · Cited by 110
A data-driven spoken language understanding system
He, Y., & Young, S. J. (2004) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2003.1318505 · Cited by 106
Spoken language understanding: a survey
Mori, R. D. (2007) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2007.4430139 · Cited by 67
Combining statistical and knowledge-based spoken language understanding in conditional models
Wang, Y., Acero, A., Mahajan, M. V., & Lee, J. S. Y. (2006) · International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics (COLING/ACL) · doi:10.3115/1273073.1273186 · Cited by 29
Cited by
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874
A Network-based End-to-End Trainable Task-oriented Dialogue System
Wen, T.-H., Vandyke, D., Mrkšić, N., Gasic, M., Rojas Barahona, L. M., Su, P.-H., Ultes, S., & Young, S. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1042 · Cited by 778
A Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Goo, C.-W., Gao, G., Hsu, Y.-K., Huo, C.-L., Chen, T.-C., Hsu, K.-W., & Chen, Y.-N. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2118 · Cited by 497
Multi-Domain Joint Semantic Frame Parsing Using Bi-Directional RNN-LSTM
Hakkani-Tür, D., Tur, G., Celikyilmaz, A., Chen, Y.-N., Gao, J., Deng, L., & Wang, Y.-Y. (2016) · Interspeech 2016 · doi:10.21437/interspeech.2016-402 · Cited by 463
Metadata
- Authors
- Yao, K., Peng, B., Zhang, Y., Yu, D., Zweig, G., & Shi, Y.
- Venue
- 2014 IEEE Spoken Language Technology Workshop (SLT)
- Publisher
- IEEE
- Published
- 2014-12
- Pages
- 189-194
- OpenAlex
- W2024632416
- Cited by
- 317
- References
- 41
- FWCI
- 13.83
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.