The DOI matches its DataCite record; 54 of the 58 references it cites were found in a registry or an index.
Hierarchical Multiscale Recurrent Neural Networks
Open accessAbstract
Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural networks, which can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that our proposed multiscale architecture can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence modelling.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Self-citation(2014) · Cited by 33,419
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Self-citationAuto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,050
Self-citationOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Self-citationGenerating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180
Self-citationTheano: A Python framework for fast computation of mathematical expressionsPreprint
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.02688 · Cited by 2,176
Self-citationCourbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1602.02830 · Cited by 2,106
Self-citationDeep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910
Self-citationCharacter-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D. A., & Rush, A. M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,590
Bursty and Hierarchical Structure in Streams
Kleinberg, J. M. (2003) · Data Mining and Knowledge Discovery · doi:10.1023/a:1024940629314 · Cited by 1,372
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · Cited by 1,266
(1994) · Aston Publications Explorer (Aston University) · Cited by 1,245
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472618 · Cited by 1,161
Self-citationLearning long-term dependencies in NARX recurrent neural networks
Lin, T.-N., Horne, B. G., Peter Tiňo, B., & Giles, C. L. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.548162 · Cited by 816
A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion
Sordoni, A., Bengio, Y., Vahabi, H., Lioma, C., Simonsen, J. G., & Nie, J. (2015) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2806416.2806493 · Cited by 501
Self-citationLearning Complex, Extended Sequences Using the Principle of History Compression
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.2.234 · Cited by 398
Grid Long Short-Term MemoryPreprint
Kalchbrenner, N., Danihelka, I., & Graves, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.01526 · Cited by 382
A Clockwork RNNPreprint
Koutník, J., Greff, K., Gomez, F. J., & Schmidhuber, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.3511 · Cited by 352
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation
Chung, J., Cho, K., & Bengio, Y. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1160 · Cited by 335
Self-citationHierarchical Recurrent Neural Networks for Long-Term Dependencies
Hihi, S. E., & Bengio, Y. (1995) · Cited by 319
Self-citationRecurrent Highway NetworksPreprint
Zilly, J., Srivastava, R., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000233981 · Cited by 314
Zoneout: Regularizing RNNs by Randomly Preserving Hidden ActivationsPreprint
Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N. R., Goyal, A., Bengio, Y., Courville, A. C., & Pal, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01305 · Cited by 284
Self-citationCharacter-based Neural Machine TranslationPreprint
Ling, W., Trancoso, I. M., Dyer, C., & Black, A. W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04586 · Cited by 267
Unconstrained On-line Handwriting Recognition with Recurrent Neural Networks
Graves, A., Liwicki, M., Horst Bunke, A., Schmidhuber, J., & Fernández, S. (2007) · Bern Open Repository and Information System (University of Bern) · Cited by 222
IAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard
Liwicki, M., & Bunke, H. (2005) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2005.132 · Cited by 212
Architectural Complexity Measures of Recurrent Neural NetworksPreprint
Zhang, S., Wu, Y., Che, T., Lin, Z., Memisevic, R., Salakhutdinov, R. R., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08210 · Cited by 150
Self-citationA Recurrent Latent Variable Model for Sequential DataPreprint
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., & Bengio, Y. (2026) · arXiv (Cornell University) · doi:10.5281/zenodo.19983308 · Cited by 149
Self-citationAdaptive weighing of context models for lossless data compression
Mahoney, M. V. (2005) · Cited by 127
On Multiplicative Integration with Recurrent Neural NetworksPreprint
Wu, Y., Zhang, S., Zhang, Y., Bengio, Y., & Salakhutdinov, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.06630 · Cited by 113
Self-citationSegmental Recurrent Neural NetworksPreprint
Kong, L., Dyer, C., & Smith, N. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06018 · Cited by 113
Strategic Attentive Writer for Learning Macro-ActionsPreprint
Vezhnevets, A. S., Vezhnevets, Mnih, V., Agapiou, J., Osindero, S., Graves, A., Vinyals, O., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04695 · Cited by 111
Sequence labelling in structured domains with hierarchical recurrent neural network
Fernández, S., Graves, A., & Schmidhuber, J. (2007) · Cited by 87
One-Shot Adaptation of Supervised Deep Convolutional ModelsPreprint
Hoffman, J., Tzeng, E., Donahue, J., Jia, Y., Saenko, K., & Darrell, T. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6204 · Cited by 67
Regularizing RNNs by Stabilizing ActivationsPreprint
Krueger, D., & Memisevic, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08400 · Cited by 59
Regularization and nonlinearities for neural language models: when are they needed?Preprint
Pachitariu, M., & Sahani, M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.5650 · Cited by 46
Surprisal-Driven Feedback in Recurrent NetworksPreprint
Rocki, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.06027 · Cited by 15
Recurrent Memory Array StructuresPreprint
Rocki, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.03085 · Cited by 12
Cited by
Sorted by citations
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Discovering Event Structure in Continuous Narrative Perception and Memory
Baldassano, C., Chen, J., Zadbood, A., Pillow, J. W., Hasson, U., & Norman, K. A. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.041 · Cited by 952
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972
Dynamic Neural Networks: A Survey
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117837 · Cited by 848
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation
Luo, Y., Chen, Z., & Yoshioka, T. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054266 · Cited by 881
Metadata
- Authors
- Chung, J., Ahn, S., & Bengio, Y.
- Affiliations
- Université de Montréal (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1609.01704
- OpenAlex
- W2510842514
- Cited by
- 386
- References
- 58
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 16 · 2023: 30 · 2022: 24 · 2021: 58 · 2020: 68 · 2019: 83 · 2018: 60 · 2017: 35 · 2016: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Keywords
- hierarchical representation, temporal dependencies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
