The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding
Abstract
Semantic slot filling is one of the most challenging problems in spoken language understanding (SLU). In this paper, we propose to use recurrent neural networks (RNNs) for this task, and present several novel architectures designed to efficiently model past and future temporal dependencies. Specifically, we implemented and compared several important RNN architectures, including Elman, Jordan, and hybrid variants. To facilitate reproducibility, we implemented these networks with the publicly available Theano neural network toolkit and completed experiments on the well-known airline travel information system (ATIS) benchmark. In addition, we compared the approaches on two custom SLU data sets from the entertainment and movies domains. Our results show that the RNN-based models outperform the conditional random field (CRF) baseline by 2% in absolute error reduction on the ATIS benchmark. We improve the state-of-the-art by 0.5% in the Entertainment domain, and 6.7% for the movies domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationBidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 13×
“As in [15], we describe here the two most common RNN architectures in the literature: the Elman [16] and Jordan [17] models.”
BackgroundInfluentialLearning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Self-citationSpeech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
(2012) · Cited by 8,009
Error bounds for convolutional codes and an asymptotically optimum decoding algorithm
Viterbi, A. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1054010 · Cited by 6,759
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Self-citationContext-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · cited 1×
“It has made strong impacts on applications ranging from automatic speech recognition [8], [69], [70] to image recognition [10].”
BackgroundSelf-citationLinguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Self-citationBoosTexter: A Boosting-based System for Text Categorization
Schapire, R. E., & Singer, Y. (2000) · Machine Learning · doi:10.1023/a:1007649029923 · Cited by 2,188
Learning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009
Self-citationDeep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,759
Self-citationExtensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., & Khudanpur, S. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947611 · Cited by 1,568
Maximum Entropy Markov Models for Information Extraction and Segmentation
McCallum, A., Freitag, D., & Pereira, F. C. N. (2000) · Cited by 1,315
Semantic Compositionality through Recursive Matrix-Vector Spaces
Socher, R., Huval, B., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,259
Serial Order: A Parallel Distributed Processing Approach
Jordan, M. I. (1997) · Advances in Psychology · doi:10.1016/s0166-4115(97)80111-2 · Cited by 1,242
Theano: A CPU and GPU Math Compiler in Python
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., & Bengio, Y. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-003 · Cited by 1,052
Self-citationA Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval
Shen, Y., He, X., Gao, J., Deng, L., & Mesnil, G. (2014) · Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management · doi:10.1145/2661829.2661935 · Cited by 661
Self-citationSequence-discriminative training of deep neural networks
Veselý, K., Ghoshal, A., Burget, L., & Povey, D. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-548 · Cited by 657
Context dependent recurrent neural network language model
Mikolov, T., & Zweig, G. (2012) · 2012 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424228 · Cited by 560
Self-citationChunking with support vector machines
Kudo, T., & Matsumoto, Y. (2001) · Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001 - NAACL '01 · doi:10.3115/1073336.1073361 · Cited by 520
Spoken Language Understanding: Systems for Extracting Semantic Information from Speech
Tür, G., & Mori, R. D. (2011) · Cited by 510
Automatic Speech Recognition: A Deep Learning Approach
Yu, D., & Deng, L. (2015) · Signals and Communication Technology · doi:10.1007/978-1-4471-5779-3 · Cited by 474
Self-citationMesnil, G., He, X., Deng, L., & Bengio, Y. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-596 · Cited by 413
Self-citationSemantic Parsing for Single-Relation Question Answering
Yih, W.-T., He, X., & Meek, C. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2105 · Cited by 410
Self-citationConvolutional neural network based triangular CRF for joint intent detection and slot filling
Xu, P., & Sarikaya, R. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707709 · Cited by 326
Spoken language understanding using long short-term memory neural networks
Yao, K., Peng, B., Zhang, Y., Yu, D., Zweig, G., & Shi, Y. (2014) · 2014 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2014.7078572 · Cited by 317
Self-citationRecurrent neural networks for language understanding
Yao, K., Zweig, G., Hwang, M.-Y., Shi, Y., & Yu, D. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-569 · Cited by 315
Self-citationGenerative and discriminative algorithms for spoken language understanding
Raymond, C., & Riccardi, G. (2007) · Interspeech 2007 · doi:10.21437/interspeech.2007-448 · Cited by 295
Spoken Language Understanding: Systems for Extracting Semantic Information from Speech
(2011) · Wiley eBooks · doi:10.1002/9781119992691 · Cited by 291
Unsupervised and Transfer Learning Challenge: a Deep Learning Approach
Mesnil, G., Dauphin, Y., Glorot, X., Rifai, S., Bengio, Y., Goodfellow, I., Lavoie, E., Muller, X., Desjardins, G., Warde-Farley, D., et al. (2011) · International Conference on Machine Learning · Cited by 254
Self-citationConversational interfaces: advances and challenges
Zue, V. W., & Glass, J. R. (2000) · Proceedings of the IEEE · doi:10.1109/5.880078 · Cited by 245
Kingsbury, B., Sainath, T. N., & Soltau, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-3 · Cited by 220
Optimizing SVMs for complex call classification
Haffner, P., Tur, G., & Wright, J. H. (2003) · 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03). · doi:10.1109/icassp.2003.1198860 · Cited by 214
Self-citationModeling Interestingness with Deep Neural Networks
Gao, J., Pantel, P., Gamon, M., He, X., & Deng, L. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1002 · Cited by 165
Self-citationWang, Y., Deng, L., & Acero, A. (2005) · IEEE Signal Processing Magazine · doi:10.1109/msp.2005.1511821 · Cited by 158
Self-citationDeep belief nets for natural language call-routing
Sarikaya, R., Hinton, G. E., & Ramabhadran, B. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947649 · Cited by 147
Towards deeper understanding: Deep convex networks for semantic utterance classification
Tür, G., Deng, L., Hakkani-Tür, D. Z., & He, X. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6289054 · Cited by 143
Self-citationSu, H., Li, G., Yu, D., & Seide, F. T. B. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638951 · Cited by 139
Self-citationThe application of semantic classification trees to natural language understanding
Kühn, R., & Mori, R. D. (1995) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.391397 · Cited by 135
Discriminative spoken language understanding using word confusion networks
Henderson, M., Gašić, M., Thomson, B., Tsiakoulis, P., Yu, K., & Young, S. J. (2012) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424218 · Cited by 123
Training neural network language models on very large corpora
Schwenk, H., & Gauvain, J. (2005) · doi:10.3115/1220575.1220601 · Cited by 123
Peng, J., Bo, L., & Xu, J. (2009) · Cited by 122
Use of kernel deep convex networks and end-to-end learning for spoken language understanding
Deng, L., Tür, G., He, X., & Hakkani-Tür, D. Z. (2012) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424224 · Cited by 115
Self-citationRecurrent conditional random field for language understanding
Yao, K., Peng, B., Zweig, G., Yu, D., Li, X., & Gao, F. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854368 · Cited by 112
Self-citationA data-driven spoken language understanding system
He, Y., & Young, S. J. (2004) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2003.1318505 · Cited by 106
Hidden understanding models of natural language
Miller, S., Bobrow, R., Ingria, R. J. P., & Schwartz, R. M. (1994) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/981732.981736 · Cited by 91
A speech understanding system based on statistical representation of semantics
Pieraccini, R., Tzoukermann, E., Gorelov, Z., Gauvain, J.-L., Levin, E., Lee, C.-H., & Wilpon, J. G. (1992) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1992.225939 · Cited by 84
Discriminative models for spoken language understanding
Wang, Y., & Acero, A. (2006) · Interspeech (USB) · doi:10.21437/interspeech.2006-608 · Cited by 66
Deep belief network based semantic taggers for spoken language understanding
Deoras, A., & Sarikaya, R. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-623 · Cited by 65
Sentence simplification for spoken language understanding
Tür, G., Hakkani-Tür, D. Z., Heck, L. P., & Parthasarathy, S. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947636 · Cited by 56
Self-citationSemantic Frame‐Based Spoken Language Understanding
Wang, Y., Deng, L., & Acero, A. (2011) · Wiley eBooks · doi:10.1002/9781119992691.ch3 · Cited by 49
Self-citationSequential Labeling Using Deep-Structured Conditional Random Fields
Yu, D., Wang, S., & Deng, L. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2075990 · Cited by 46
Self-citationSpeech-Centric Information Processing: An Optimization-Oriented Approach
He, X., & Deng, L. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2236631 · Cited by 44
Self-citationA conversational movie search system based on conditional random fields
Liu, J., Cyphers, S., Pasupat, P., Mcgraw, I. C., & Glass, J. R. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-563 · Cited by 38
Zero-Shot Learning and Clustering for Semantic Utterance ClassificationPreprint
Dauphin, Y., Tür, G., Hakkani-Tür, D. Z., et al. (2013) · arXiv (Cornell University) · Cited by 15
Self-citation
Cited by
Ahmad, Z., Shahid Khan, A., Wai Shiang, C., Abdullah, J., & Ahmad, F. (2021) · Transactions on Emerging Telecommunications Technologies · doi:10.1002/ett.4150 · Cited by 1,288
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1547 · Cited by 1,028
From Eliza to XiaoIce: Challenges and opportunities with social chatbots
Shum, H.-Y., He, X.-D., & Li, D. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700826 · Cited by 749
“…AI technologies, e.g., natural language understanding (Bengio et al., 2001; Mikolov et al., 2013; Sutskever et al., 2014; Bahdanau et al., 2015; Mesnil et al., 2015), speech recognition and synthesis (Hinton et al., 2012; Deng et al., 2013; Xiong et al., 2016; Qian et al., 2014; van den Oord…”
BackgroundSelf-citationMultimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511
Self-citationSlot-Gated Modeling for Joint Slot Filling and Intent Prediction
Goo, C.-W., Gao, G., Hsu, Y.-K., Huo, C.-L., Chen, T.-C., Hsu, K.-W., & Chen, Y.-N. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2118 · Cited by 497
Metadata
- Authors
- Mesnil, G., Dauphin, Y., Yao, K., Bengio, Y., Deng, L., Hakkani-Tur, D., He, X., Heck, L., Tur, G., Yu, D., & Zweig, G.
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2015-03
- Volume
- 23
- Issue
- 3
- Pages
- 530-539
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W2137871902
- Cited by
- 461
- References
- 62
- FWCI
- 23.11
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.