Hierarchical Multiscale Recurrent Neural Networks
Open accessAbstract
Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural networks, which can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that our proposed multiscale architecture can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence modelling.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Self-citation(2014) · Cited by 33,419 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Self-citationDeep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 5,037 · see citing works
Self-citationOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · see citing works
Self-citationGenerating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · see citing works
Theano: A Python framework for fast computation of mathematical expressionsPreprint
Al-Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., Belopolsky, A., et al. (2016) · arXiv.org · doi:10.48550/arxiv.1605.02688 · Cited by 2,438 · see citing works
Self-citationListen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226 · see citing works
BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Courbariaux, M., Bengio, Y., & David, J.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.00363 · Cited by 2,180 · see citing works
Self-citationCourbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1602.02830 · Cited by 2,106 · see citing works
Self-citationDeep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104 · see citing works
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
Bengio, Y., Léonard, N., & Courville, A. (2013) · arXiv · doi:10.48550/arxiv.1308.3432 · Cited by 1,910 · see citing works
Self-citationCharacter-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D. A., & Rush, A. M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10362 · Cited by 1,546 · see citing works
Bursty and Hierarchical Structure in Streams
Kleinberg, J. M. (2003) · Data Mining and Knowledge Discovery · doi:10.1023/a:1024940629314 · Cited by 1,315 · see citing works
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · Cited by 1,266 · see citing works
(1994) · Aston Publications Explorer (Aston University) · Cited by 1,245 · see citing works
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472618 · Cited by 1,129 · see citing works
Self-citationLearning long-term dependencies in NARX recurrent neural networks
Lin, T.-N., Horne, B. G., Peter Tiňo, B., & Giles, C. L. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.548162 · Cited by 788 · see citing works
A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion
Sordoni, A., Bengio, Y., Vahabi, H., Lioma, C., Simonsen, J. G., & Nie, J. (2015) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2806416.2806493 · Cited by 482 · see citing works
Self-citationLearning Complex, Extended Sequences Using the Principle of History Compression
Schmidhuber, J. (1992) · Neural Computation · doi:10.1162/neco.1992.4.2.234 · Cited by 398 · see citing works
Hierarchical Recurrent Neural Networks for Long-Term Dependencies
Hihi, S. E., & Bengio, Y. (1995) · Cited by 319 · see citing works
Self-citationA Character-level Decoder without Explicit Segmentation for Neural Machine Translation
Chung, J., Cho, K., & Bengio, Y. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1160 · Cited by 310 · see citing works
Self-citationGrid Long Short-Term MemoryPreprint
Kalchbrenner, N., Danihelka, I., & Graves, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.01526 · Cited by 275 · see citing works
A Clockwork RNNPreprint
Koutník, J., Greff, K., Gomez, F. J., & Schmidhuber, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.3511 · Cited by 262 · see citing works
Recurrent Highway NetworksPreprint
Zilly, J., Srivastava, R., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000233981 · Cited by 243 · see citing works
Unconstrained On-line Handwriting Recognition with Recurrent Neural Networks
Graves, A., Liwicki, M., Horst Bunke, A., Schmidhuber, J., & Fernández, S. (2007) · Bern Open Repository and Information System (University of Bern) · Cited by 222 · see citing works
Zoneout: Regularizing RNNs by Randomly Preserving Hidden ActivationsPreprint
Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N. R., Goyal, A., Bengio, Y., Courville, A. C., & Pal, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01305 · Cited by 214 · see citing works
Self-citationIAM-OnDB - an on-line English sentence database acquired from handwritten text on a whiteboard
Liwicki, M., & Bunke, H. (2005) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2005.132 · Cited by 212 · see citing works
Character-based Neural Machine TranslationPreprint
Ling, W., Trancoso, I. M., Dyer, C., & Black, A. W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04586 · Cited by 197 · see citing works
Adaptive weighing of context models for lossless data compression
Mahoney, M. V. (2005) · Cited by 127 · see citing works
A Recurrent Latent Variable Model for Sequential DataPreprint
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., & Bengio, Y. (2026) · arXiv (Cornell University) · doi:10.5281/zenodo.19983308 · Cited by 119 · see citing works
Self-citationArchitectural Complexity Measures of Recurrent Neural NetworksPreprint
Zhang, S., Wu, Y., Che, T., Lin, Z., Memisevic, R., Salakhutdinov, R., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08210 · Cited by 114 · see citing works
Self-citationSegmental Recurrent Neural NetworksPreprint
Kong, L., Dyer, C., & Smith, N. A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06018 · Cited by 101 · see citing works
On Multiplicative Integration with Recurrent Neural NetworksPreprint
Wu, Y., Zhang, S., Zhang, Y., Bengio, Y., & Salakhutdinov, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.06630 · Cited by 97 · see citing works
Self-citationSequence labelling in structured domains with hierarchical recurrent neural networks
Fernández, S., Graves, A., & Schmidhuber, J. (2007) · Cited by 87 · see citing works
Strategic Attentive Writer for Learning Macro-ActionsPreprint
-, A., Vezhnevets, Mnih, V., Agapiou, J., Osindero, S., Graves, A., Vinyals, O., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04695 · Cited by 78 · see citing works
One-Shot Adaptation of Supervised Deep Convolutional ModelsPreprint
Hoffman, J., Tzeng, E., Donahue, J., Jia, Y., Saenko, K., & Darrell, T. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6204 · Cited by 67 · see citing works
Regularization and nonlinearities for neural language models: when are they needed?Preprint
Pachitariu, M., & Sahani, M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.5650 · Cited by 40 · see citing works
Regularizing RNNs by Stabilizing ActivationsPreprint
Krueger, D., & Memisevic, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08400 · Cited by 40 · see citing works
Surprisal-Driven Feedback in Recurrent NetworksPreprint
Rocki, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.06027 · Cited by 6 · see citing works
Recurrent Memory Array StructuresPreprint
Rocki, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.03085 · Cited by 4 · see citing works
Flagged first, then by citations
5 of the 58 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Discovering Event Structure in Continuous Narrative Perception and Memory
Baldassano, C., Chen, J., Zadbood, A., Pillow, J. W., Hasson, U., & Norman, K. A. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.041 · Cited by 952 · see citing works
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972 · see citing works
Dynamic Neural Networks: A Survey
Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3117837 · Cited by 848 · see citing works
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation
Luo, Y., Chen, Z., & Yoshioka, T. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054266 · Cited by 825 · see citing works
Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi
Zheng, Y., Zhang, Y., Qian, K., Zhang, G., Liu, Y., Wu, C., & Yang, Z. (2019) · Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3307334.3326081 · Cited by 557 · see citing works
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 860 · see citing works
Neural Machine Translation in Linear Time
Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · see citing works
A Decade Survey of Content Based Image Retrieval Using Deep Learning
Dubey, S. R. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3080920 · Cited by 365 · see citing works
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00461 · Cited by 330 · see citing works
Widar3.0: Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi
Zhang, Y., Zheng, Y., Qian, K., Zhang, G., Liu, Y., Wu, C., & Yang, Z. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3105387 · Cited by 316 · see citing works
Simple Recurrent Units for Highly Parallelizable Recurrence
Lei, T., Zhang, Y., Wang, S. I., Dai, H., & Artzi, Y. (2017) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1477 · Cited by 315 · see citing works
Breaking the Softmax Bottleneck: A High-Rank RNN Language ModelPreprint
Yang, Z., Dai, Z., Salakhutdinov, R., & Cohen, W. W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.03953 · Cited by 291 · see citing works
HSA-RNN: Hierarchical Structure-Adaptive RNN for Video Summarization
Zhao, B., Li, X., & Lu, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00773 · Cited by 243 · see citing works
Ordered Neurons: Integrating Tree Structures into Recurrent Neural NetworksPreprint
Shen, Y., Tan, S., Sordoni, A., & Courville, A. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09536 · Cited by 222 · see citing works
A Realistic Dataset and Baseline Temporal Model for Early Drowsiness Detection
Ghoddoosian, R., Galib, M., & Athitsos, V. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00027 · Cited by 219 · see citing works
Recurrent Batch NormalizationPreprint
Cooijmans, T., Ballas, N., Laurent, C., Gülçehre, Ç., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.09025 · Cited by 218 · see citing works
Compressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · see citing works
Learning to Compose and Reason with Language Tree Structures for Visual Grounding
Hong, R., Liu, D., Mo, X., He, X., & Zhang, H. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2911066 · Cited by 196 · see citing works
Mansouri, M. M., Trabelsi, M., Nounou, H. N., & Nounou, M. N. (2021) · IEEE Access · doi:10.1109/access.2021.3110947 · Cited by 179 · see citing works
Deep Learning for Distributed Channel Feedback and Multiuser Precoding in FDD Massive MIMO
Sohrabi, F., Attiah, K. M., & Yu, W. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3055202 · Cited by 169 · see citing works
Skip RNN: Learning to Skip State Updates in Recurrent Neural NetworksPreprint
Campos, V., Jou, B., Giró-i-Nieto, X., Torres, J., & Chang, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.06834 · Cited by 169 · see citing works
Document-Level N-ary Relation Extraction with Multiscale Representation Learning
Jia, R., Wong, C., & Poon, H. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1370 · Cited by 168 · see citing works
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement LearningPreprint
Oh, J., Singh, S., Lee, H., & Kohli, P. (2017) · International Conference on Machine Learning · doi:10.48550/arxiv.1706.05064 · Cited by 292 · see citing works
Action-Agnostic Human Pose Forecasting
Chiu, H.-K., Adeli, E., Wang, B., Huang, D.-A., & Niebles, J. C. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00156 · Cited by 163 · see citing works
Learning Structured Representation for Text Classification via Reinforcement Learning
Zhang, T., Huang, M., & Li, Z. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12047 · Cited by 150 · see citing works
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential DataPreprint
Hsu, W.-N., Zhang, Y., & Glass, J. R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.07902 · Cited by 148 · see citing works
Neural Language Modeling by Jointly Learning Syntax and LexiconPreprint
Shen, Y., Lin, Z., Huang, C.-W., & Courville, A. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02013 · Cited by 147 · see citing works
Learning to Compose Task-Specific Tree Structures
Choi, J., Yoo, K. M., & Lee, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11975 · Cited by 8 · see citing works
Recurrent Independent MechanismsPreprint
Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., & Schölkopf, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10893 · Cited by 164 · see citing works
Self-citationCanine : Pre-training an Efficient Tokenization-Free Encoder for Language Representation
Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00448 · Cited by 161 · see citing works
Retrospective Encoders for Video Summarization
Zhang, K., Grauman, K. L., & Sha, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_24 · Cited by 133 · see citing works
Learning Multimodal Graph-to-Graph Translation for Molecular OptimizationPreprint
Jin, W., Yang, K., Barzilay, R., & Jaakkola, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01070 · Cited by 129 · see citing works
Discovering Subsequence Patterns for Next POI Recommendation
Zhao, K., Zhang, Y., Yin, H., Wang, J., Zheng, K., Zhou, X., & Xing, C. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/445 · Cited by 127 · see citing works
Learning to Attend, Copy, and Generate for Session-Based Query Suggestion
Dehghani, M., Rothe, S., Alfonseca, E., & Fleury, P. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133010 · Cited by 125 · see citing works
An Analysis of Neural Language Modeling at Multiple ScalesPreprint
Merity, S., Keskar, N. S., & Socher, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08240 · Cited by 125 · see citing works
Lifelong Sequential Modeling with Personalized Memorization for User Response Prediction
Ren, K., Qin, J., Fang, Y., Zhang, W., Zheng, L., Bian, W., Zhou, G., Xu, J., Yu, Y. H., Zhu, X., et al. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331230 · Cited by 121 · see citing works
Deep Neural Decision TreesPreprint
Yang, Y., Morillo, I. G., & Hospedales, T. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.06988 · Cited by 120 · see citing works
Switching between internal and external modes: A multiscale learning principle
Honey, C. J., Newman, E. L., & Schapiro, A. C. (2017) · Network Neuroscience · doi:10.1162/netn_a_00024 · Cited by 110 · see citing works
Multiplicative LSTM for sequence modellingPreprint
Krause, B., Murray, I., Renals, S., & Lu, L. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07959 · Cited by 232 · see citing works
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPPreprint
Mielke, S. J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W. Y., Sagot, B., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10508 · Cited by 193 · see citing works
DeePr-ESN: A deep projection-encoding echo-state network
Ma, Q., Shen, L., & Cottrell, G. W. (2019) · Information Sciences · doi:10.1016/j.ins.2019.09.049 · Cited by 111 · see citing works
Linder, J., Bogard, N., Rosenberg, A. B., & Seelig, G. (2020) · Cell Systems · doi:10.1016/j.cels.2020.05.007 · Cited by 107 · see citing works
Deep learning in electron microscopy
Ede, J. M. (2020) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abd614 · Cited by 105 · see citing works
Unsupervised Recurrent Neural Network Grammars
Kim, Y., Rush, A. M., Yu, L., Kuncoro, A., Dyer, C., & Melis, G. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1114 · Cited by 102 · see citing works
Yu, A. W., Lee, H., & Le, Q. V. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1172 · Cited by 96 · see citing works
Do latent tree learning models identify meaningful structure in sentences?
Williams, A., Drozdov, A., & Bowman, S. R. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00019 · Cited by 92 · see citing works
Revisiting Character-Based Neural Machine Translation with Capacity and Compression
Cherry, C. A., Foster, G. F., Bapna, A., Fırat, O., & Macherey, W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1461 · Cited by 91 · see citing works
Constructing and Forgetting Temporal Context in the Human Cerebral Cortex
Chien, H.-Y. S., & Honey, C. J. (2020) · Neuron · doi:10.1016/j.neuron.2020.02.013 · Cited by 86 · see citing works
Shi, C., Zhang, Z., Zhang, W., Zhang, C., & Xu, Q. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3176642 · Cited by 86 · see citing works
Modeling Recurrence for Transformer
Hao, J., Wang, X., Yang, B., Wang, L., Zhang, J., & Tu, Z. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1122 · Cited by 85 · see citing works
Explicit Sparse Transformer: Concentrated Attention Through Explicit SelectionPreprint
Zhao, G., Lin, J., Zhang, Z., Ren, X., Su, Q., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11637 · Cited by 131 · see citing works
NAOMI: Non-Autoregressive Multiresolution Sequence ImputationPreprint
Liu, Y., Yu, R., Zheng, S., Zhan, E., & Yue, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10946 · Cited by 84 · see citing works
Encoding word order in complex embeddingsPreprint
Wang, B., Zhao, D., Lioma, C., Li, Q., Zhang, P., & Simonsen, J. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.12333 · Cited by 85 · see citing works
Chinese Medical Question Answer Matching Using End-to-End Character-Level Multi-Scale CNNs
Zhang, S., Zhang, X., Wang, H., Cheng, J., Li, P., & Ding, Z. (2017) · Applied Sciences · doi:10.3390/app7080767 · Cited by 80 · see citing works
Condensed Memory Networks for Clinical Diagnostic Inferencing
Prakash, A., Zhao, S., Hasan, S. A., Datla, V. V., Lee, K., Qadir, A., Liu, J., & Farri, O. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10964 · Cited by 39 · see citing works
Unsupervised Automatic Speech Recognition: A Review
Aldarmaki, H., Ullah, A., Ram, S., & Zaki, N. (2021) · Speech Communication · doi:10.1016/j.specom.2022.02.005 · Cited by 75 · see citing works
Role of Deep Learning in Wireless Communications
Yu, W., Sohrabi, F., & Jiang, T. (2022) · IEEE BITS the Information Theory Magazine · doi:10.1109/mbits.2022.3212978 · Cited by 75 · see citing works
Augmenting Self-attention with Persistent Memory
Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., & Joulin, A. (2019) · arXiv · doi:10.48550/arxiv.1907.01470 · Cited by 75 · see citing works
Bai, Y., Liu, M.-D., Ding, L., & Ma, Y. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.117461 · Cited by 73 · see citing works
Multi-Scale Self-Attention for Text Classification
Guo, Q., Qiu, X., Liu, P., Xue, X., & Zhang, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6290 · Cited by 77 · see citing works
Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural NetworksPreprint
Mendis, C., Renda, A., Amarasinghe, S. P., & Carbin, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07412 · Cited by 70 · see citing works
Long-term Forecasting using Higher Order Tensor RNNsPreprint
Yu, R., Zheng, S., Anandkumar, A., & Yue, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00073 · Cited by 102 · see citing works
Fast activation maximization for molecular sequence design
Linder, J., & Seelig, G. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04437-5 · Cited by 72 · see citing works
Variational Temporal Deep Generative Model for Radar HRRP Target Recognition
Guo, D., Chen, B., Chen, W., Wang, C., Liu, H., & Zhou, M. (2020) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2020.3027470 · Cited by 63 · see citing works
Learning Simpler Language Models with the Differential State Framework
Ororbia, A. G., Mikolov, T., & Reitter, D. (2017) · Neural Computation · doi:10.1162/neco_a_01017 · Cited by 62 · see citing works
Convolutional Multitimescale Echo State Network
Ma, Q., Chen, E., Lin, Z., Yan, J., Yu, Z., & Ng, W. W. Y. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2919648 · Cited by 59 · see citing works
Time Series Anomaly Detection with Multiresolution Ensemble Decoding
Shen, L., Yu, Z., Ma, Q., & Kwok, J. T.-Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i11.17152 · Cited by 60 · see citing works
Phonetic Temporal Neural Model for Language Identification
Tang, Z., Wang, D., Chen, Y., Li, L., & Abel, A. K. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2764271 · Cited by 60 · see citing works
Character-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56 · see citing works
Wang, Y., Lee, H.-Y., & Lee, L.-S. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462002 · Cited by 54 · see citing works
BP-Transformer: Modelling Long-Range Context via Binary PartitioningPreprint
Ye, Z., Guo, Q., Gan, Q., Qiu, X., & Zhang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04070 · Cited by 61 · see citing works
Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence LabellingPreprint
Liu, H., Zhu, Z., Li, X., & Satheesh, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00096 · Cited by 50 · see citing works
DReLUs: Dual Rectified Linear Units
Godin, F., Degrave, J., Dambre, J., & Neve, W. D. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.09.006 · Cited by 51 · see citing works
Dynamic Neural Network Structure: A Review for its Theories and Applications
Guo, J., Chen, C. L. P., Liu, Z., & Yang, X. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3377194 · Cited by 47 · see citing works
Na, X., Ren, W., & Xu, X. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104229 · Cited by 46 · see citing works
Temporal Difference Variational Auto-EncoderPreprint
Gregor, K., Papamakarios, G., Besse, F., Buesing, L., & Weber, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03107 · Cited by 46 · see citing works
Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationPreprint
Luo, Y., Chen, Z., & Yoshioka, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06379 · Cited by 46 · see citing works
Bridging the gap between emotion and joint action
Bieńkiewicz, M. M. N., Smykovskyi, A., Olugbade, T., Janaqi, S., Camurri, A., Bianchi‐Berthouze, N., Björkman, M., & Bardy, B. G. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.08.014 · Cited by 47 · see citing works
Chen, Y.-C., Huang, S.-F., Lee, H.-Y., Wang, Y., & Shen, C.-H. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2922832 · Cited by 42 · see citing works
Online Credit Payment Fraud Detection via Structure-Aware Hierarchical Recurrent Neural Network
Lin, W., Sun, L., Zhong, Q., Liu, C., Feng, J., Ao, X., & Yang, H. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/505 · Cited by 42 · see citing works
A New Training Pipeline for an Improved Neural Transducer
Zeyer, A., Merboldt, A., Schlüter, R., & Ney, H. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1855 · Cited by 42 · see citing works
Modeling multivariate clinical event time-series with recurrent temporal mechanisms
Lee, J. M., & Hauskrecht, M. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102021 · Cited by 41 · see citing works
Na, X., Ren, W., Liu, M., & Han, M. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3157830 · Cited by 41 · see citing works
Takanobu, R., Huang, M., Zhao, Z., Li, F.-L., Chen, H., Zhu, X., & Nie, L. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/612 · Cited by 41 · see citing works
Hierarchical Multiscale Recurrent Neural Networks for Detecting Suicide Notes
Schoene, A. M., Turner, A. P., de Mel, G. R., & Dethlefs, N. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3057105 · Cited by 41 · see citing works
Maybe Deep Neural Networks are the Best Choice for Modeling Source CodePreprint
Karampatsis, R.-M., & Sutton, C. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.05734 · Cited by 41 · see citing works
Examining the Limits of Predictability of Human Mobility
Kulkarni, V., Mahalunkar, A., Garbinato, B., & Kelleher, J. D. (2019) · Entropy · doi:10.3390/e21040432 · Cited by 41 · see citing works
Interpreting Multivariate Shapley Interactions in DNNs
Zhang, H., Xie, Y., Zheng, L., Zhang, D., & Zhang, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17299 · Cited by 45 · see citing works
DVAE++: Discrete Variational Autoencoders with Overlapping TransformationsPreprint
Vahdat, A., Macready, W. G., Bian, Z., Khoshaman, A. H., & Andriyash, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04920 · Cited by 38 · see citing works
Tang, X., Chen, Y., Ma, Y., Yang, W., Zhou, H., & Huang, J. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.107910 · Cited by 37 · see citing works
Narayanswamy, G., Liu, Y., Yang, Y., Ma, C., Liu, X., McDuff, D., & Patel, S. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00773 · Cited by 36 · see citing works
3G structure for image caption generation
Yuan, A., Li, X., & Lu, X. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.10.059 · Cited by 35 · see citing works
SyntaxNet Models for the CoNLL 2017 Shared TaskPreprint
Alberti, C., Andor, D., Bogatyy, I., Collins, M., Gillick, D., Kong, L., Koo, T., Ma, J., Omernick, M., Petrov, S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.04929 · Cited by 35 · see citing works
Deep Independently Recurrent Neural Network (IndRNN)Preprint
Li, S., Li, W., Cook, C., & Gao, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06251 · Cited by 35 · see citing works
A Multiscale Interactive Recurrent Network for Time-Series Forecasting
Chen, D., Chen, L., Zhang, Y., Wen, B., & Cheng-hu, Y. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3055951 · Cited by 34 · see citing works
Long Short-Term Memory with Dynamic Skip Connections
Gui, T., Zhang, Q., Zhao, L., Lin, Y., Peng, M., Gong, J., & Huang, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016481 · Cited by 34 · see citing works
Recurrent Neural Network for Text Classification with Hierarchical Multiscale Dense Connections
Zhao, Y., Shen, Y., & Yao, J. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/757 · Cited by 34 · see citing works
A dedicated hardware accelerator for real-time acceleration of YOLOv2
Xu, K., Wang, X., Liu, X., Cao, C., Li, H., Peng, H., & Wang, D. (2020) · Journal of Real-Time Image Processing · doi:10.1007/s11554-020-00977-w · Cited by 34 · see citing works
Two-Timescale End-to-End Learning for Channel Acquisition and Hybrid Precoding
Hu, Q., Cai, Y., Kang, K., Yu, G., Hoydis, J., & Eldar, Y. C. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2021.3126050 · Cited by 34 · see citing works
- Authors
- Chung, J., Ahn, S., & Bengio, Y.
- Affiliations
- Université de Montréal (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2016
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1609.01704
- OpenAlex
- W2510842514
- Cited by
- 386
- References
- 58
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 16 · 2023: 30 · 2022: 24 · 2021: 58 · 2020: 68 · 2019: 83 · 2018: 60 · 2017: 35 · 2016: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Keywords
- hierarchical representation, temporal dependencies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
