Sequence-to-Sequence Learning as Beam-Search Optimization
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Sequence-to-Sequence (seq2seq) modeling has rapidly become an important generalpurpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks.Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions.In this work, we introduce a model and beamsearch training scheme, based on the work of Daumé III and Marcu (2005), that extends seq2seq to learn global sequence scores.This structured approach avoids classical biases associated with local training and unifies the training loss with the test-time usage, while preserving the proven model architecture of seq2seq and its efficient training approach.We show that our system outperforms a highlyoptimized attention-based seq2seq system and other baselines on three different sequence to sequence tasks: word ordering, parsing, and machine translation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
(2014) · Cited by 33,419
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In addition to demonstrating impressive results for machine translation (Bahdanau et al., 2015), roughly the same model and training have also proven to be useful for sen-”
BackgroundConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
(2010) · Cited by 8,261
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 1×
“, 2016), and they additionally underlie other text generation applications, such as image or video captioning (Venugopalan et al., 2015; Xu et al., 2015).”
Background
Cited by
Adversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1230 · Cited by 873
Zhao, T., Zhao, R., & Eskenazi, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1061 · Cited by 736
“, 2016a); (2) the second category aims to improve the encoder-decoder model itself, including decoding with beam search and its variations (Wiseman and Rush, 2016), encouraging responses that have long-term payoff (Li et al.”
BackgroundAn Actor-Critic Algorithm for Sequence Prediction
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1607.07086 · Cited by 423
Kao, I.-F., Zhou, Y., Chang, L.-C., & Chang, F.-J. (2020) · Journal of Hydrology · doi:10.1016/j.jhydrol.2020.124631 · Cited by 410
Neural Machine Translation: A Review
Stahlberg, F. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12007 · Cited by 397
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
Liang, C., Berant, J., Le, Q., Forbus, K. D., & Lao, N. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1003 · Cited by 381
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models
Chorowski, J., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-343 · Cited by 358
“Global normalization has yielded important benefits on many NLP tasks including parsing and translation [32, 33].”
BackgroundDeep Neural Solver for Math Word Problems
Wang, Y., Liu, X., & Shi, S. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1088 · Cited by 329
Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems
Madotto, A., Wu, C.-S., & Fung, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1136 · Cited by 302
Quasi-Recurrent Neural Networks
Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01576 · Cited by 299
Retrieval-based neural source code summarization
Zhang, J., Wang, X., Zhang, H., Sun, H., & Liu, X. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380383 · Cited by 275
Diverse Beam Search for Improved Description of Complex Scenes
Vijayakumar, A., Cogswell, M., Selvaraju, R., Sun, Q., Lee, S., Crandall, D., & Batra, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12340 · Cited by 219
“In an orthogonal line of work, (Wiseman and Rush 2016) directly learn to search in the exponential output space to fix the shortcomings of using seq2seq models.”
BackgroundA Minimal Span-Based Neural Constituency Parser
Stern, M., Andreas, J., & Klein, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1076 · Cited by 189
“Moreover, models with recurrent state require complex training procedures to benefit from anything other than greedy decoding (Wiseman and Rush, 2016).”
BackgroundEnd-to-End Neural Relation Extraction with Global Optimization
Zhang, M., Zhang, Y., & Fu, G. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1182 · Cited by 178
“We address this potential issue by building a structural neural model for end-to-end relation extraction, following a recent line of efforts on globally optimized models for neural structured prediction (Zhou et al., 2015; Watanabe and Sumita, 2015; Andor et al., 2016; Wiseman and Rush, 2016).”
MethodInfluentialOn Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
Wang, C., & Sennrich, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.326 · Cited by 118
Robust Text-to-SQL Generation with Execution-Guided Decoding
Wang, C., Tatwawadi, K., Brockschmidt, M., Huang, P.-S., Mao, Y., Polozov, O., & Singh, R. (2018) · arXiv · Cited by 64
Chen, Y., Li, H., Qi, G., Wu, T., & Wang, T. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3207477 · Cited by 41
Differentiable Scheduled Sampling for Credit Assignment
Goyal, K., Dyer, C., & Berg-Kirkpatrick, T. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2058 · Cited by 34
(2019) · doi:10.1609/aaai.v33i01.33016618 · Cited by 31
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
Wang, C., & Sennrich, R. (2020) · arXiv · doi:10.48550/arxiv.2005.03642 · Cited by 12
Efficient Fine-Tuning Large Language Models for Knowledge-Aware Response Planning
Nguyen, M., Kishan, K. C., Nguyen, T., Chadha, A., & Vu, T. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43415-0_35 · Cited by 8
Metadata
- Authors
- Wiseman, S., & Rush, A. M.
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 1296-1306
- OpenAlex
- W2963620441
- Cited by
- 481
- References
- 43
- FWCI
- 32.67
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.