CoChat
Sign Up
PreprintCited by 1,020Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

Open access
Kumar, Ankit, Irsoy, Ozan, Ondruska, Peter, Iyyer, Mohit, Bradbury, James, Gulrajani, Ishaan, Zhong, Victor, Paulus, Romain +1 more·2015·arXiv
DOI 10.48550/arxiv.1506.07285arXiv 1506.07285
Open this paper in CoChat — free accountOpen PDF

Abstract

Most tasks in natural language processing can be cast into question answering (QA) problems over language input. We introduce the dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers. Questions trigger an iterative attention process which allows the model to condition its attention on the inputs and the result of previous iterations. These results are then reasoned over in a hierarchical recurrent sequence model to generate answers. The DMN can be trained end-to-end and obtains state-of-the-art results on several types of tasks and datasets: question answering (Facebook's bAbI dataset), text classification for sentiment analysis (Stanford Sentiment Treebank) and sequence modeling for part-of-speech tagging (WSJ-PTB). The training for these different tasks relies exclusively on trained word vector representations and input-question-answer triplets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1506.07285

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • References [61,83] are earlier works that attempt to unify multiple NLP tasks into a QA framework.

    “References [61,83] are earlier works that attempt to unify multiple NLP tasks into a QA framework.”

    From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [61]

  • In the majority of previous work, the prompting function consists of applying a template with an input slot and an answer slot and filling the input slot with the input text.

    “In the majority of previous work [61,83,105,117], this function consists of a two step process: (1) Apply a template, which is a textual string that has two slots: an input slot [X] for input x and an answer slot [Z] for an intermediate generated answer text z that will later be mapped into y.(2) Fill slot [X] with the input text x.”

    From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [61]

    The Introduction (p. 1) gives examples of inputs, questions, and answers, while the Question Module (p. 2) describes encoding questions with a recurrent network. Neither passage describes a two-slot textual template, filling an input slot, or what the majority of previous work does; searches for those terms found no supporting passage in our copy. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Learning to Compose Neural Networks for Question Answering. arXiv preprint arXiv:1601.01705 , 2016. · arXiv
  2. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. CoRR , abs/1409.0473, 2014. · arXiv
  3. Bordes, A., Glorot, X., Weston, J., and Bengio, Y. Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing. AISTATS , 2012.
  4. Bowman, S. R., Potts, C., and Manning, C. D. Recursive neural networks for learning logical semantics. CoRR , abs/1406.1827, 2014. · arXiv
  5. Chen, X. and Zitnick, C. L. Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654 , 2014. · arXiv
  6. Cho, K., van Merrienboer, B., Bahdanau, D., and Bengio, Y. On the properties of neural machine translation: Encoder-decoder approaches. CoRR , abs/1409.1259, 2014a. · arXiv
  7. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP , 2014b.
  8. Chung, J., G¨ ulc ¸ehre, C ¸ ., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR , abs/1412.3555, 2014. · arXiv
  9. Dusek, J. A. and Eichenbaum, H. The hippocampus and memory for orderly stimulusrelations. Proceedings of the National Academy of Sciences , 94(13):7109-7114, 1997.
  10. Eichenbaum, H. and Cohen, N. J. From Conditioning to Conscious Recollection: Memory Systems of the Brain (Oxford Psychology) . Oxford University Press, 1 edition, 2004. ISBN 0195178041.
  11. Elman, J. L. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning , 7(2-3):195-225, 1991.
  12. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. CoRR , abs/1410.5401, 2014. · arXiv
  13. Hermann, K. M., Koˇ cisk´ y, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. In NIPS , 2015.
  14. Howard, Marc W. and Kahana, Michael J. A distributed representation of temporal context. Journal of Mathematical Psychology , 46(3):269 - 299, 2002.
  15. Irsoy, O. and Cardie, C. Modeling compositionality with multiplicative recurrent neural networks. In ICLR , 2015.
  16. Iyyer, M., Boyd-Graber, J., Claudino, L., Socher, R., and Daum´ e III, H. A neural network for factoid question answering over paragraphs. In EMNLP , 2014.
  17. Joulin, A. and Mikolov, T. Inferring algorithmic patterns with stack-augmented recurrent nets. In NIPS , 2015.
  18. Kaiser, L. and Sutskever, I. Neural GPUs Learn Algorithms. arXiv preprint arXiv:1511.08228 , 2015. · arXiv
  19. Kalchbrenner, N., Grefenstette, E., and Blunsom, P. A convolutional neural network for modelling sentences. In ACL , 2014.
  20. Kim, Y. Convolutional neural networks for sentence classification. In EMNLP , 2014.
  21. Kingma, P. and Ba, Jimmy. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014. · arXiv
  22. Le, Q.V. and Mikolov., T. Distributed representations of sentences and documents. In ICML , 2014.
  23. Malinowski, M. and Fritz, M. A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input. In NIPS , 2014.
  24. Mikolov, T. and Zweig, G. Context dependent recurrent neural network language model. In SLT , pp. 234-239. IEEE, 2012. ISBN 978-1-4673-5125-6.
  25. Passos, A., Kumar, V., and McCallum, A. Lexicon infused phrase embeddings for named entity resolution. In Conference on Computational Natural Language Learning . Association for Computational Linguistics, June 2014.
  26. Pennington, J., Socher, R., and Manning, C. D. Glove: Global vectors for word representation. In EMNLP , 2014.
  27. Socher, R., Gershman, S., Perotte, A., Sederberg, P., Blei, D., and Norman, K. A bayesian analysis of dynamics in free recall. In NIPS . 2009.
  28. Socher, R., Huang, E. H., Pennington, J., Ng, A. Y., and Manning, C. D. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS , 2011.
  29. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , 2013.
  30. Søgaard, A. Semisupervised condensed nearest neighbor for part-of-speech tagging. In ACL-HLT , 2011.
  31. Stollenga, M. F. and J. Masci, F. Gomez, J. Schmidhuber. Deep Networks with Internal Selective Attention through Feedback Connections. In NIPS , 2014.
  32. Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In NIPS , 2014.
  33. Tai, K. S., Socher, R., and Manning, C. D. Improved semantic representations from tree-structured long shortterm memory networks. In ACL , 2015.
  34. Weston, J., Bordes, A., Chopra, S., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR , abs/1502.05698, 2015a. · arXiv
  35. Weston, J., Chopra, S., and Bordes, A. Memory networks. In ICLR , 2015b.
  36. Yang, Z., He, X., Gao, J., Deng, L., and Smola, A. Stacked attention networks for image question answering. arXiv preprint arXiv:1511.02274 , 2015. · arXiv
  37. Yates, A., Banko, M., Broadhead, M., Cafarella, M. J., Etzioni, O., and Soderland, S. Textrunner: Open information extraction on the web. In HLT-NAACL (Demonstrations) , 2007.

Cited by

Sorted by citations

Authors
Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., & Socher, R.
Affiliations
Salesforce (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2015
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1506.07285
arXiv
1506.07285
OpenAlex
W2131494463
Cited by
1,020
References
4
Citations by year
2025: 23 · 2024: 32 · 2023: 69 · 2022: 59 · 2021: 130 · 2020: 150 · 2019: 185 · 2018: 166 · 2017: 138 · 2016: 56 · 2015: 12
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
dynamic memory networks, question answering, episodic memory, sentiment analysis, text classification, part-of-speech tagging, word embeddings, end-to-end learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review