CoChat
Sign Up
Cited by 230Topic ModelingNatural Language Processing TechniquesReinforcement Learning in Robotics

Deep Reinforcement Learning for Sequence-to-Sequence Models

Open access
Keneshloo, Yaser, Shi, Tian, Ramakrishnan, Naren, Reddy, Chandan K.·2019·IEEE Transactions on Neural Networks and Learning Systems
DOI 10.1109/tnnls.2019.2929141
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent times, sequence-to-sequence (seq2seq) models have gained a lot of popularity and provide state-of-the-art performance in a wide variety of tasks, such as machine translation, headline generation, text summarization, speech-to-text conversion, and image caption generation. The underlying framework for all these models is usually a deep neural network comprising an encoder and a decoder. Although simple encoder-decoder models produce competitive results, many researchers have proposed additional improvements over these seq2seq models, e.g., using an attention-based model over the input, pointer-generation models, and self-attention models. However, such seq2seq models suffer from two common problems: 1) exposure bias and 2) inconsistency between train/test measurement. Recently, a completely novel point of view has emerged in addressing these two problems in seq2seq models, leveraging methods from reinforcement learning (RL). In this survey, we consider seq2seq problems from the RL point of view and provide a formulation combining the power of RL methods in decision-making with seq2seq models that enable remembering long-term memories. We present some of the most recent frameworks that combine the concepts from RL and deep neural networks. Our work aims to provide insights into some of the problems that inherently arise with current approaches and how we can address them with better RL models. We also provide the source code for implementing most of the RL models discussed in this paper to support the complex task of abstractive text summarization and provide some targeted experiments for these RL models, both in terms of performance and training time.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tnnls.2019.2929141

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LMs are particularly susceptible to hallucination, making it essential to penalize such behavior.

    “LMs are particularly susceptible to hallucination [77,113], making it essential to penalize such behavior.”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [77]

    The paper’s “Current RL-Based Model Issues” section discusses exposure bias and training challenges, while “Actor-Critic Models” discusses errors accumulating during sequence generation. Neither passage addresses whether language models are particularly susceptible to hallucination or whether hallucination should be penalized. Searches for hallucination, factual consistency, and related terms found no passage making that point. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 162

  1. I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks," in NeurIPS, 2014, pp. 3104-3112.
  2. S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, "Scheduled sampling for sequence prediction with recurrent neural networks," in NeurIPS, 2015, pp. 1171-1179.
  3. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al., "Google's neural machine translation system: Bridging the gap between human and machine translation," arXiv preprint arXiv:1609.08144, 2016. · arXiv
  4. A. Vaswani, S. Bengio, E. Brevdo, F. Chollet, A. N. Gomez, S. Gouws, L. Jones, Ł. Kaiser, N. Kalchbrenner, N. Parmar et al., "Tensor2tensor for neural machine translation," arXiv preprint arXiv:1803.07416, 2018. · arXiv
  5. D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," arXiv preprint arXiv:1409.0473, 2014. · arXiv
  6. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using rnn encoder-decoder for statistical machine translation," arXiv preprint arXiv:1406.1078, 2014. · arXiv
  7. S. Shen, Y. Cheng, Z. He, W. He, H. Wu, M. Sun, and Y. Liu, "Minimum risk training for neural machine translation," in ACL, vol. 1, 2016, pp. 1683-1692.
  8. A. M. Rush, S. Chopra, and J. Weston, "A neural attention model for abstractive sentence summarization," in EMNLP, 2015, pp. (379-389.
  9. S. Chopra, M. Auli, A. M. Rush, and S. Harvard, "Abstractive sentence summarization with attentive recurrent neural networks," in NAACL- HLT, 2016, pp. 93-98.
  10. R. Nallapati, B. Zhou, C. dos Santos, C. Gulcehre, and B. Xiang, "Abstractive text summarization using sequence-to-sequence rnns and beyond," in SIGNLL, 2016, pp. 280-290.
  11. A. See, P. J. Liu, and C. D. Manning, "Get to the point: Summarization with pointer-generator networks," in ACL, vol. 1, 2017, pp. 1073-1083.
  12. R. Paulus, C. Xiong, and R. Socher, "A deep reinforced model for abstractive summarization," arXiv preprint arXiv:1705.04304, 2017. · arXiv
  13. R. Nallapati, F. Zhai, and B. Zhou, "Summarunner: A recurrent neural network based sequence model for extractive summarization of documents." in AAAI, 2017, pp. 3075-3081.
  14. A. Graves, A.-r. Mohamed, and G. Hinton, "Speech recognition with deep recurrent neural networks," in ICASSP. IEEE, 2013, pp. 6645- 6649.
  15. D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, "End-to-end attention-based large vocabulary speech recognition," in ICASSP. IEEE, 2016, pp. 4945-4949.
  16. D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen et al., "Deep speech 2: End-to-end speech recognition in english and mandarin," in ICML, 2016, pp. 173-182.
  17. S. Ö. Arık, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. Li, J. Miller, A. Ng, J. Raiman et al., "Deep voice: Real- time neural text-to-speech," in ICML, 2017, pp. 195-204.
  18. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.
  19. J. L. Elman, "Finding structure in time," Cognitive science, vol. 14, no. 2, pp. 179-211, 1990.
  20. C.-Y. LIN, "Rouge: A package for automatic evaluation of summaries," in Proc. of Workshop on Text Summarization Branches Out, 2004.
  21. S. Banerjee and A. Lavie, "Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments," in Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65-72.
  22. R. Vedantam, C. Lawrence Zitnick, and D. Parikh, "Cider: Consensus- based image description evaluation," in CVPR, 2015, pp. 4566-4575.
  23. M. Ranzato, S. Chopra, M. Auli, and W. Zaremba, "Sequence level training with recurrent neural networks," arXiv preprint arXiv:1511.06732, 2015. · arXiv
  24. J. Su, J. Xu, X. Qiu, and X. Huang, "Incorporating discriminator in sentence generation: a gibbs sampling method," arXiv preprint arXiv:1802.08970, 2018. · arXiv
  25. F. Huszár, "How (not) to train your generative model: Scheduled sampling, likelihood, adversary?" arXiv preprint arXiv:1511.05101, 2015. · arXiv
  26. L. Yu, W. Zhang, J. Wang, and Y. Yu, "Seqgan: sequence generative adversarial nets with policy gradient," in AAAI, vol. 31, 2017, pp. 2852- 2858.
  27. K. Lin, D. Li, X. He, Z. Zhang, and M.-T. Sun, "Adversarial ranking for language generation," in NeurIPS, 2017, pp. 3155-3165.
  28. J. Guo, S. Lu, H. Cai, W. Zhang, Y. Yu, and J. Wang, "Long text generation via adversarial training with leaked information," arXiv preprint arXiv:1709.08624, 2017. · arXiv
  29. T. Che, Y. Li, R. Zhang, R. D. Hjelm, W. Li, Y. Song, and Y. Ben- gio, "Maximum-likelihood augmented discrete generative adversarial networks," arXiv preprint arXiv:1702.07983, 2017. · arXiv
  30. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in NeurIPS, 2014, pp. 2672-2680.
  31. Y. Zhang, Z. Gan, K. Fan, Z. Chen, R. Henao, D. Shen, and L. Carin, "Adversarial feature matching for text generation," in ICML, 2017, pp. 4006-4015.
  32. Z. Shi, X. Chen, X. Qiu, and X. Huang, "Towards diverse text generation with inverse reinforcement learning," arXiv preprint arXiv:1804.11258, 2018. · arXiv
  33. D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y. Bengio, "An actor-critic algorithm for sequence prediction," in ICLR, 2017.
  34. S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, "Self- critical sequence training for image captioning," in CVPR, 2017, pp. 7008-7024.
  35. R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning," in Reinforcement Learning. Springer, 1992, pp. 5-32.
  36. R. S. Sutton and A. G. Barto, "Reinforcement learning: An introduc- tion," Cambridge: MIT press, vol. 1, no. 1, 1998.
  37. S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," The Journal of Machine Learning Research, vol. 17, no. 1, pp. 1334-1373, 2016.
  38. I. Lenz, H. Lee, and A. Saxena, "Deep learning for detecting robotic grasps," The International Journal of Robotics Research, vol. 34, no. 4-5, pp. 705-724, 2015.
  39. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, p. 529, 2015.
  40. P. Li, L. Bing, and W. Lam, "Actor-critic based training framework for abstractive summarization," arXiv preprint arXiv:1803.11070, 2018. · arXiv
  41. K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, "A brief survey of deep reinforcement learning," arXiv preprint arXiv:1708.05866, 2017. · arXiv
  42. S. Team, "Deep learning for siri's voice: On-device deep mixture density networks for hybrid unit selection synthesis," Apple Machine Learning Journal, vol. 1, no. 4, 2017.
  43. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. Rush, "Opennmt: Open- source toolkit for neural machine translation," Proceedings of ACL, System Demonstrations, pp. 67-72, 2017.
  44. M. X. Chen, O. Firat, A. Bapna, M. Johnson, W. Macherey, G. Foster, L. Jones, N. Parmar, M. Schuster, Z. Chen et al., "The best of both worlds: Combining recent advances in neural machine translation," in ACL, 2018, pp. 76-86.
  45. J. Ling and A. Rush, "Coarse-to-fine attention models for document summarization," in Proceedings of the Workshop on New Frontiers in Summarization, 2017, pp. 33-42.
  46. Q. Zhou, N. Yang, F. Wei, and M. Zhou, "Selective encoding for abstractive sentence summarization," in ACL, vol. 1, 2017, pp. 1095- 1104.
  47. J. Tan, X. Wan, and J. Xiao, "Abstractive document summarization with a graph-based attentional neural model," in ACL, vol. 1, 2017, pp. 1171-1181.
  48. Y.-C. Chen and M. Bansal, "Fast abstractive summarization with reinforce-selected sentence rewriting," in ACL, vol. 1, 2018, pp. 675- 686.
  49. J. Lin, X. Sun, S. Ma, and Q. Su, "Global encoding for abstractive summarization," in ACL, 2018, pp. 163-169.
  50. W.-T. Hsu, C.-K. Lin, M.-Y. Lee, K. Min, J. Tang, and M. Sun, "A unified model for extractive and abstractive summarization using inconsistency loss," in ACL, 2018, pp. 132-141.
  51. Y. Xia, F. Tian, L. Wu, J. Lin, T. Qin, N. Yu, and T.-Y. Liu, "Delib- eration networks: Sequence generation beyond one-pass decoding," in NeurIPS, 2017, pp. 1782-1792.
  52. I. V. Serban, A. García-Durán, C. Gulcehre, S. Ahn, S. Chandar, A. Courville, and Y. Bengio, "Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus," in ACL, vol. 1, 2016, pp. 588-598.
  53. N. Mostafazadeh, I. Misra, J. Devlin, M. Mitchell, X. He, and L. Van- derwende, "Generating natural questions about an image," in ACL, vol. 1, 2016, pp. 1802-1813.
  54. Z. Yang, J. Hu, R. Salakhutdinov, and W. Cohen, "Semi-supervised qa with generative domain-adaptive nets," in ACL, vol. 1, 2017, pp. 1040-1050.
  55. X. Yuan, T. Wang, C. Gulcehre, A. Sordoni, P. Bachman, S. Zhang, S. Subramanian, and A. Trischler, "Machine comprehension by text-to- text neural question generation," in Proceedings of the 2nd Workshop on Representation Learning for NLP, 2017, pp. 15-25.
  56. C. Xiong, V. Zhong, and R. Socher, "Dynamic coattention networks for question answering," in ICLR, 2016.
  57. Z. Yang, X. He, J. Gao, L. Deng, and A. Smola, "Stacked attention networks for image question answering," in CVPR, 2016, pp. 21-29.
  58. O. Vinyals and Q. Le, "A neural conversational model," arXiv preprint arXiv:1506.05869, 2015. · arXiv
  59. J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan, "A diversity- promoting objective function for neural conversation models," in NAACL-HLT, 2016, pp. 110-119.
  60. I. V. Serban, A. Sordoni, Y. Bengio, A. C. Courville, and J. Pineau, "Building end-to-end dialogue systems using generative hierarchical neural network models." in AAAI, vol. 16, 2016, pp. 3776-3784.
  61. A. Bordes, Y.-L. Boureau, and J. Weston, "Learning end-to-end goal- oriented dialog," arXiv preprint arXiv:1605.07683, 2016. · arXiv
  62. V. Zhong, C. Xiong, and R. Socher, "Seq2SQL: Generating structured queries from natural language using reinforcement learning," 2018. [Online]. Available: https://openreview.net/forum?id=Syx6bz-Ab
  63. X. Xu, C. Liu, and D. Song, "Sqlnet: Generating structured queries from natural language without reinforcement learning," arXiv preprint arXiv:1711.04436, 2017. · arXiv
  64. R. Kiros, R. Salakhutdinov, and R. Zemel, "Multimodal neural lan- guage models," in ICML, 2014, pp. 595-603.
  65. R. Kiros, R. Salakhutdinov, and R. S. Zemel, "Unifying visual-semantic embeddings with multimodal neural language models," arXiv preprint arXiv:1411.2539, 2014. · arXiv
  66. X. Chen and C. L. Zitnick, "Mind's eye: A recurrent visual repre- sentation for image caption generation," in CVPR. IEEE, 2015, pp. 2422-2431.
  67. J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille, "Deep captioning with multimodal recurrent neural networks (m-rnn)," arXiv preprint arXiv:1412.6632, 2014. · arXiv
  68. H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt et al., "From captions to visual concepts and back," in CVPR. IEEE, 2015, pp. 1473-1482.
  69. J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venu- gopalan, K. Saenko, and T. Darrell, "Long-term recurrent convolutional networks for visual recognition and description," in CVPR, 2015, pp. 2625-2634.
  70. S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. Mooney, and K. Saenko, "Translating videos to natural language using deep recurrent neural networks," in NAACL-HLT, 2015, pp. 1494-1504.
  71. S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko, "Sequence to sequence-video to text," in ICCV, 2015, pp. 4534-4542.
  72. S. Venugopalan, L. A. Hendricks, R. Mooney, and K. Saenko, "Im- proving lstm-based video description with linguistic knowledge mined from text," in EMNLP, 2016, pp. 1961-1966.
  73. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. Lecun, "Overfeat: Integrated recognition, localization and detection using convolutional networks," in ICLR, 2014.
  74. R. Girshick, "Fast r-cnn," arXiv preprint arXiv:1504.08083, 2015. · arXiv
  75. A. Graves and N. Jaitly, "Towards end-to-end speech recognition with recurrent neural networks," in ICML, 2014, pp. 1764-1772.
  76. Y. Miao, M. Gowayyed, and F. Metze, "Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding," in IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015, pp. 167-174.
  77. H. Ze, A. Senior, and M. Schuster, "Statistical parametric speech synthesis using deep neural networks," in ICASSP. IEEE, 2013, pp. 7962-7966.
  78. Y. Fan, Y. Qian, F.-L. Xie, and F. K. Soong, "Tts synthesis with bidirectional lstm based recurrent neural networks," in ISCA, 2014.
  79. J. Ba, V. Mnih, and K. Kavukcuoglu, "Multiple object recognition with visual attention," arXiv preprint arXiv:1412.7755, 2014. · arXiv
  80. O. Vinyals, M. Fortunato, and N. Jaitly, "Pointer networks," in NeurIPS, 2015, pp. 2692-2700.
  81. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in NeurIPS, 2017, pp. 6000-6010.
  82. C. dos Santos and M. Gatti, "Deep convolutional neural networks for sentiment analysis of short texts," in COLING, 2014, pp. 69-78.
  83. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, "Recursive deep models for semantic compositionality over a sentiment treebank," in EMNLP, 2013, pp. 1631-1642.
  84. A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, "Supervised learning of universal sentence representations from natural language inference data," in EMNLP, 2017, pp. 670-680.
  85. S. Kim, J.-H. Hong, I. Kang, and N. Kwak, "Semantic sentence match- ing with densely-connected recurrent and co-attentive information," arXiv preprint arXiv:1805.11360, 2018. · arXiv
  86. B. Pan, Y. Yang, Z. Zhao, Y. Zhuang, D. Cai, and X. He, "Discourse marker augmented network with reinforcement learning for natural language inference," in ACL, vol. 1, 2018, pp. 989-999.
  87. N. FitzGerald, O. Täckström, K. Ganchev, and D. Das, "Semantic role labeling with neural network factors," in EMNLP, 2015, pp. 960-970.
  88. L. He, M. Lewis, and L. Zettlemoyer, "Question-answer driven seman- tic role labeling: Using natural language to annotate natural language," in EMNLP, 2015, pp. 643-653.
  89. D. Marcheggiani, A. Frolov, and I. Titov, "A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling," in CoNLL, 2017, pp. 411-420.
  90. M. Mintz, S. Bills, R. Snow, and D. Jurafsky, "Distant supervision for relation extraction without labeled data," in ACL, 2009, pp. 1003-1011.
  91. X. Huang et al., "Attention-based convolutional neural network for semantic relation extraction," in ICLR, 2016, pp. 2526-2536.
  92. P. Qin, W. Xu, and W. Y. Wang, "Robust distant supervision relation extraction via deep reinforcement learning," in ACL, vol. 1, 2018, pp. 2137-2147.
  93. C. Chen and V. Ng, "Chinese zero pronoun resolution with deep neural networks," in ACL, vol. 1, 2016, pp. 778-788.
  94. Q. Yin, Y. Zhang, W. Zhang, and T. Liu, "Chinese zero pronoun resolution with deep memory network," in EMNLP, 2017, pp. 1309- 1318.
  95. T. H. Trinh and Q. V. Le, "A simple method for commonsense reasoning," arXiv preprint arXiv:1806.02847, 2018. · arXiv
  96. Q. Yin, Y. Zhang, W. Zhang, T. Liu, and W. Y. Wang, "Deep reinforcement learning for chinese zero pronoun resolution," in ACL, 2018, pp. 569-578.
  97. P. Anderson, B. Fernando, M. Johnson, and S. Gould, "Spice: Semantic propositional image caption evaluation," in ECCV, 2016.
  98. A. Axelrod, X. He, and J. Gao, "Domain adaptation via pseudo in- domain data selection," in EMNLP, 2011, pp. 355-362.
  99. K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, "Teaching machines to read and comprehend," in NeurIPS, 2015, pp. 1693-1701.
  100. M. Grusky, M. Naaman, and Y. Artzi, "Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies," in NAACL-HLT, New Orleans, Louisiana, June 2018. [Online]. Available: https://summari.es/newsroom.pdf
  101. D. Graff, J. Kong, K. Chen, and K. Maeda, "English gigaword," Linguistic Data Consortium, Philadelphia, vol. 4, p. 1, 2003.
  102. P. Rajpurkar, R. Jia, and P. Liang, "Know what you don't know: Unanswerable questions for squad," arXiv preprint arXiv:1806.03822, 2018. · arXiv
  103. J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Chopra, A. Miller, A. Szlam, and J. Weston, "Evaluating prerequisite qualities for learning end-to-end dialog systems," arXiv preprint arXiv:1511.06931, 2015. · arXiv
  104. C. Danescu-Niculescu-Mizil and L. Lee, "Chameleons in imagined conversations: A new approach to understanding coordination of lin- guistic style in dialogs," in Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, 2011, pp. 76-87.
  105. P. Pasupat and P. Liang, "Compositional semantic parsing on semi- structured tables," in ACL, 2015, pp. 1470-1480.
  106. Y. Wang, J. Berant, and P. Liang, "Building a semantic parser overnight," in ACL, vol. 1, 2015, pp. 1332-1342.
  107. J. McAuley, R. Pandey, and J. Leskovec, "Inferring networks of substitutable and complementary products," in KDD. ACM, 2015, pp. 785-794.
  108. S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, "A large annotated corpus for learning natural language inference," in EMNLP, 2015.
  109. A. Williams, N. Nangia, and S. Bowman, "A broad-coverage challenge corpus for sentence understanding through inference," in NAACL-HLT, vol. 1, 2018, pp. 1112-1122.
  110. M. Palmer, D. Gildea, and P. Kingsbury, "The proposition bank: An annotated corpus of semantic roles," Computational linguistics, vol. 31, no. 1, pp. 71-106, 2005.
  111. J. R. Finkel, T. Grenager, and C. Manning, "Incorporating non-local information into information extraction systems by gibbs sampling," in ACL, 2005, pp. 363-370.
  112. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in European conference on computer vision. Springer, 2014, pp. 740-755.
  113. V. Ordonez, G. Kulkarni, and T. L. Berg, "Im2text: Describing images using 1 million captioned photographs," in NeurIPS, 2011, pp. 1143- 1151.
  114. J. Xu, T. Mei, T. Yao, and Y. Rui, "Msr-vtt: A large video description dataset for bridging video and language," in CVPR. IEEE, 2016, pp. 5288-5296.
  115. D. L. Chen and W. B. Dolan, "Collecting highly parallel data for paraphrase evaluation," in ACL, 2011, pp. 190-200.
  116. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
  117. W. Zaremba and I. Sutskever, "Reinforcement learning neural turing machines-revised," arXiv preprint arXiv:1505.00521, 2015. · arXiv
  118. T. Jie and P. Abbeel, "On a connection between importance sampling and the likelihood ratio policy gradient," in NeurIPS, 2010, pp. 1000- 1008.
  119. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, "Asynchronous methods for deep reinforcement learning," in ICML, 2016, pp. 1928-1937.
  120. R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare, "Safe and efficient off-policy reinforcement learning," in NeurIPS, 2016, pp. 1054-1062.
  121. M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Sil- ver, and K. Kavukcuoglu, "Reinforcement learning with unsupervised auxiliary tasks," arXiv preprint arXiv:1611.05397, 2016. · arXiv
  122. A. Gruslys, M. G. Azar, M. G. Bellemare, and R. Munos, "The reactor: A sample-efficient actor-critic architecture," arXiv preprint arXiv:1704.04651, 2017. · arXiv
  123. Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas, "Sample efficient actor-critic with experience replay," in ICLR, 2017.
  124. R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approxima- tion," in NeurIPS, 2000, pp. 1057-1063.
  125. G. Tucker, S. Bhupatiraju, S. Gu, R. E. Turner, Z. Ghahramani, and S. Levine, "The mirage of action-dependent baselines in reinforcement learning," ICLR Workshop, 2018.
  126. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, "High- dimensional continuous control using generalized advantage estima- tion," in ICLR, 2016.
  127. R. S. Sutton and A. G. Barto, "Reinforcement learning: An introduc- tion," MIT press, 2018.
  128. C. J. Watkins and P. Dayan, "Q-learning," Machine learning, vol. 8, no. 3-4, pp. 279-292, 1992.
  129. J. N. Tsitsiklis, "Asynchronous stochastic approximation and q- learning," Machine learning, vol. 16, no. 3, pp. 185-202, 1994.
  130. L.-J. Lin, "Self-improving reactive agents based on reinforcement learning, planning and teaching," Machine learning, vol. 8, no. 3-4, pp. 293-321, 1992.
  131. T. Schaul, J. Quan, I. Antonoglou, and D. Silver, "Prioritized experience replay," arXiv preprint arXiv:1511.05952, 2015. · arXiv
  132. H. V. Hasselt, "Double q-learning," in NeurIPS, 2010, pp. 2613-2621.
  133. S. Thrun and A. Schwartz, "Issues in using function approximation for reinforcement learning," in Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.
  134. Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, "Dueling network architectures for deep reinforcement learning," in ICML, 2016, pp. 1995-2003.
  135. H. Daumé, J. Langford, and D. Marcu, "Search-based structured prediction," Machine learning, vol. 75, no. 3, pp. 297-325, 2009.
  136. S. Ross, G. Gordon, and D. Bagnell, "A reduction of imitation learning and structured prediction to no-regret online learning," in Proceedings of the fourteenth international conference on artificial intelligence and statistics, 2011, pp. 627-635.
  137. A. Venkatraman, M. Hebert, and J. A. Bagnell, "Improving multi-step prediction of learned time series models." in AAAI, 2015, pp. 3024- 3030.
  138. V. Mnih, N. Heess, A. Graves et al., "Recurrent models of visual attention," in NeurIPS, 2014, pp. 2204-2212.
  139. L. Wang, J. Yao, Y. Tao, L. Zhong, W. Liu, and Q. Du, "A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization," arXiv preprint arXiv:1805.03616, 2018. · arXiv
  140. S. Narayan, S. B. Cohen, and M. Lapata, "Ranking sentences for extractive summarization with reinforcement learning," in NAACL- HLT, vol. 1, 2018, pp. 1747-1759.
  141. Z. Li, X. Jiang, L. Shang, and H. Li, "Paraphrase generation with deep reinforcement learning," arXiv preprint arXiv:1711.00279, 2017. · arXiv
  142. R. Pasunuru and M. Bansal, "Reinforced video captioning with entail- ment rewards," in EMNLP, 2017, pp. 979-985.
  143. Y. Zhou, C. Xiong, and R. Socher, "Improving end-to-end speech recognition with policy learning," arXiv preprint arXiv:1712.07101, 2017. · arXiv
  144. J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and J. Gao, "Deep reinforcement learning for dialogue generation," in EMNLP, 2016, pp. 1192-1202.
  145. M. Hu, Y. Peng, and X. Qiu, "Reinforced mnemonic reader for machine comprehension," CoRR, abs/1705.02798, 2017. · arXiv
  146. D. He, H. Lu, Y. Xia, T. Qin, L. Wang, and T. Liu, "Decoding with value networks for neural machine translation," in NeurIPS, 2017, pp. 177-186.
  147. L. Zhang, F. Sung, F. Liu, T. Xiang, S. Gong, Y. Yang, and T. M. Hospedales, "Actor-critic sequence training for image captioning," arXiv preprint arXiv:1706.09601, 2017. · arXiv
  148. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., "Mastering the game of go with deep neural networks and tree search," Nature, vol. 529, no. 7587, pp. 484-489, 2016.
  149. J. Li, W. Monroe, and D. Jurafsky, "Learning to decode for future success," arXiv preprint arXiv:1701.06549, 2017. · arXiv
  150. Y. Wu and B. Hu, "Learning to Extract Coherent Summary via Deep Reinforcement Learning," arXiv preprint arXiv:1804.07036, 2018. · arXiv
  151. J. Kreutzer, J. Uyheng, and S. Riezler, "Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning," in ACL, vol. 1, 2018, pp. 1777-1788.
  152. C. Liang, J. Berant, Q. Le, K. D. Forbus, and N. Lao, "Neural symbolic machines: Learning semantic parsers on freebase with weak supervision," in ACL, vol. 1, 2017, pp. 23-33.
  153. S. Yeung, O. Russakovsky, G. Mori, and L. Fei-Fei, "End-to-end learning of action detection from frame glimpses in videos," in CVPR, 2016, pp. 2678-2687.
  154. I. Higgins, A. Pal, A. Rusu, L. Matthey, C. Burgess, A. Pritzel, M. Botvinick, C. Blundell, and A. Lerchner, "Darla: Improving zero- shot transfer in reinforcement learning," in International Conference on Machine Learning, 2017, pp. 1480-1490.
  155. M. Norouzi, S. Bengio, N. Jaitly, M. Schuster, Y. Wu, D. Schuurmans et al., "Reward augmented maximum likelihood for neural structured prediction," in NeurIPS, 2016, pp. 1723-1731.
  156. N. Ding and R. Soricut, "Cold-start reinforcement learning with softmax policy gradient," in NeurIPS, 2017, pp. 2817-2826.
  157. B. Tan, Z. Hu, Z. Yang, R. Salakhutdinov, and E. Xing, "Connecting the dots between mle and rl for sequence generation," arXiv preprint arXiv:1811.09740, 2018. · arXiv
  158. B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse reinforcement learning," in AAAI. AAAI Press, 2008, pp. 1433-1438.
  159. M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, "Rain- bow: Combining improvements in deep reinforcement learning," arXiv preprint arXiv:1710.02298, 2017. · arXiv
  160. W. Dabney, G. Ostrovski, D. Silver, and R. Munos, "Implicit quantile networks for distributional reinforcement learning," arXiv preprint arXiv:1806.06923, 2018. · arXiv
  161. J. Pennington, R. Socher, and C. Manning, "Glove: Global vectors for word representation," in EMNLP, 2014, pp. 1532-1543.
  162. Naren Ramakrishnan is the Thomas L. Phillips Professor of Engineering at Virginia Tech. He directs the Discovery Analytics Center, a university-wide effort that brings together researchers from computer science, statistics, mathematics, and electrical and computer engineering to tackle knowledge discovery problems in important areas of national interest. His work has been featured in the Wall Street Jour- nal, Newsweek, Smithsonian Magazine, PBS/NoVA Next, Chronicle of Higher Education, and Popular Science, among other venues. Ramakrishnan serves on the editorial boards of IEEE Computer, ACM Transactions on Knowledge Discovery from Data, Data Mining and Knowledge Discovery, IEEE Trans- actions on Knowledge and Data Engineering, and other journals. He received his PhD in Computer Sciences from Purdue University.

Cited by

Sorted by citations

Authors
Keneshloo, Y., Shi, T., Ramakrishnan, N., & Reddy, C. K.
Affiliations
Virginia Tech (US)
ORCID
Venue
IEEE Transactions on Neural Networks and Learning Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2019
Volume
31
Issue
7
Pages
1-21
ISSN
  • 2162-237X (ISSN-L)
  • 2162-2388
DOI
10.1109/tnnls.2019.2929141
OpenAlex
W2803933017
Cited by
230
References
21
Citations by year
2026: 2 · 2025: 29 · 2024: 37 · 2023: 36 · 2022: 32 · 2021: 33 · 2020: 33 · 2019: 23 · 2018: 5
FWCI
13.96
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (IIS-1619028, IIS-1707498, IIS-1838730, DGE-1545362, IIS-1633363)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
deep reinforcement learning, sequence-to-sequence models, encoder-decoder architecture, exposure bias, abstractive summarization, long-term memory, machine translation, image captioning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review