Recurrent Memory Networks for Language Modeling
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Recurrent Neural Networks (RNNs) have obtained excellent result in many natural language processing (NLP) tasks.However, understanding and interpreting the source of this success remains a challenge.In this paper, we propose Recurrent Memory Network (RMN), a novel RNN architecture, that not only amplifies the power of RNN but also facilitates our understanding of its internal functioning and allows us to discover underlying patterns in data.We demonstrate the power of RMN on language modeling and sentence completion tasks.On language modeling, RMN outperforms Long Short-Term Memory (LSTM) network on three large German, Italian, and English dataset.Additionally we perform indepth analysis of various linguistic dimensions that RMN captures.On Sentence Completion Challenge, for which it is essential to capture sentence coherence, our RMN obtains 69.2% accuracy, surpassing the previous state of the art by a large margin. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · cited 3×
“However, training simple RNNs is difficult because of the vanishing and exploding gradient problems (Bengio et al., 1994; Pascanu et al., 2013). A simple and effective solution for exploding gradients is gradient clipping proposed by Pascanu et al. (2013). To address the more challenging problem of vanishing gradients…”
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · cited 2×
“There have been few recent attempts to understand the components of an LSTM from an empirical point of view: Greff et al. (2015) carry out a large-scale experiment of eight LSTM variants.”
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“Recurrent Neural Networks (RNNs) (Elman, 1990; Mikolov et al., 2010) are remarkably powerful models for sequential data.”
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · cited 1×
“The MB takes the hidden state of the LSTM and compares it to the most recent inputs using an attention mechanism (Gregor et al., 2015; Bahdanau et al., 2014; Graves et al., 2014).”
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2013) · arXiv · doi:10.48550/arxiv.1312.3005 · Cited by 1,051
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845 · cited 10×
“It has been demonstrated that RNNs can retain input information over a long period.”
Influential
Cited by
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · arXiv · doi:10.48550/arxiv.1601.06733 · Cited by 1,198
Recent Advances in Recurrent Neural Networks
Salehinejad, H., Sankar, S., Barfett, J., Colak, E., & Valaee, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01078 · Cited by 562
LSTM with sentence representations for document-level sentiment classification
Rao, G., Huang, W., Feng, Z., & Cong, Q. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.04.045 · Cited by 326
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213
“BART, 22 T5,GPT-2, 25 BiomedGPT 26 Perplexity 20,27 Likelihoodofthemodelgeneratingthereferencetext.”
InfluentialKeep Me Updated! Memory Management in Long-term Conversations
Bae, S., Kwak, D., Kang, S., Lee, M. Y., Kim, S., Jeong, Y., Kim, H., Lee, S.-W., Park, W., & Sung, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.276 · Cited by 29
Metadata
- Authors
- Tran, K., Bisazza, A., & Monz, C.
- Venue
- Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2016
- Pages
- 321-331
- OpenAlex
- W2963077125
- Cited by
- 84
- References
- 11
- FWCI
- 7.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Recurrent Memory Networks for Language Modeling. https://app.cochat.ai/papers/10.18653/v1/n16-1036
- The paper
- Tran, K., Bisazza, A., & Monz, C. (2016). Recurrent Memory Networks for Language Modeling. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. https://doi.org/10.18653/v1/n16-1036
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.