CoChat
Sign Up
PreprintCited by 126Software Engineering ResearchTopic ModelingInformation Retrieval and Search Behavior

A Retrieve-and-Edit Framework for Predicting Structured Outputs

Open access
Hashimoto, Tatsunori B., Guu, Kelvin, Oren, Yonatan, Liang, Percy·2018·arXiv
DOI 10.48550/arxiv.1812.01194arXiv 1812.01194
Open this paper in CoChat — free accountOpen PDF

Abstract

For the task of generating complex outputs such as source code, editing existing outputs can be easier than generating complex outputs from scratch. With this motivation, we propose an approach that first retrieves a training example based on the input (e.g., natural language description) and then edits it to the desired output (e.g., code). Our contribution is a computationally efficient method for learning a retrieval model that embeds the input in a task-dependent way without relying on a hand-crafted metric or incurring the expense of jointly training the retriever with the editor. Our retrieve-and-edit framework can be applied on top of any base model. We show that on a new autocomplete task for GitHub Python code and the Hearthstone cards benchmark, retrieve-and-edit significantly boosts the performance of a vanilla sequence-to-sequence model on both tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1812.01194

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 40

  1. [1][1] M. Allamanis, D. Tarlow, A. Gordon, and Y. Wei. Bimodal modelling of source code and natural language. In International Conference on Machine Learning (ICML) , pages 2123-2132, 2015. -
  2. [2][2] G. Andrew, R. Arora, J. Bilmes, and K. Livescu. Deep canonical correlation analysis. In International Conference on Machine Learning (ICML) , pages 1247-1255, 2013. -
  3. [3][3] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR) , 2015. -
  4. [4][4] M. Balog, A. L. Gaunt, M. Brockschmidt, S. Nowozin, and D. Tarlow. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989 , 2016. - · arXiv
  5. [5][5] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from a continuous space. In Computational Natural Language Learning (CoNLL) , pages 10-21, 2016. -
  6. [6][6] S. Chaidaroon and Y. Fang. Variational deep semantic hashing for text documents. In ACM Special Interest Group on Information Retreival (SIGIR) , pages 75-84, 2017. -
  7. [7][7] C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005 , 2013. - · arXiv
  8. [8][8] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Empirical Methods in Natural Language Processing (EMNLP) , pages 1724-1734, 2014. -
  9. [9][9] T. R. Davidson, L. Falorsi, N. D. Cao, T. Kipf, and J. M. Tomczak. Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891 , 2018. - · arXiv
  10. [10][10] F. Feng, X. Wang, and R. Li. Cross-modal retrieval with correspondence autoencoder. In Proceedings of the 22Nd ACM International Conference on Multimedia , pages 7-16, 2014. -
  11. [12][12] J. Gu, Z. Lu, H. Li, and V. O. Li. Incorporating copying mechanism in sequence-to-sequence learning. In Association for Computational Linguistics (ACL) , 2016. -
  12. [13][13] J. Gu, Y. Wang, K. Cho, and V. O. Li. Search engine guided non-parametric neural machine translation. arXiv preprint arXiv:1705.07267 , 2017. - · arXiv
  13. [14][14] K. Guu, T. B. Hashimoto, Y. Oren, and P. Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics (TACL) , 0, 2018. -
  14. [15][15] S. A. Hayati, R. Olivier, P. Avvaru, P. Yin, A. Tomasic, and G. Neubig. Retrieval-based neural code generation. In Empirical Methods in Natural Language Processing (EMNLP) , 2018. -
  15. [16][16] A. Krizhevsky and G. E. Hinton. Using very deep autoencoders for content-based image retrieval. In 19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) , pages 489-494, 2011. -
  16. [17][17] N. Kushman and R. Barzilay. Using semantic unification to generate regular expressions from natural language. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL) , pages 826-836, 2013. -
  17. [18][18] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Generalizing image captions for image-text parallel corpus. In Association for Computational Linguistics (ACL) , pages 790-796, 2013. -
  18. [19][19] T. Lei, H. Joshi, R. Barzilay, T. Jaakkola, K. Tymoshenko, A. Moschitti, and L. Marquez. Semi-supervised question retrieval with gated convolutions. In North American Association for Computational Linguistics (NAACL) , pages 1279-1289, 2016. -
  19. [20][20] J. Li, W. Monroe, T. Shi, A. Ritter, and D. Jurafsky. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547 , 2017. - · arXiv
  20. [21][21] P. Liang, M. I. Jordan, and D. Klein. Learning programs: A hierarchical Bayesian approach. In International Conference on Machine Learning (ICML) , pages 639-646, 2010. -
  21. [22][22] W. Ling, E. Grefenstette, K. M. Hermann, T. Koˇ ciský, A. Senior, F. Wang, and P. Blunsom. Latent predictor networks for code generation. In Association for Computational Linguistics (ACL) , pages 599-609, 2016. -
  22. [23][23] C. Maddison and D. Tarlow. Structured generative models of natural source code. In International Conference on Machine Learning (ICML) , pages 649-657, 2014. -
  23. [24][24] R. Mason and E. Charniak. Domain-specific image captioning. In Computational Natural Language Learning (CoNLL) , pages 2-10, 2014. -
  24. [25][25] K. Papineni, S. Roukos, T. Ward, and W. Zhu. BLEU: A method for automatic evaluation of machine translation. In Association for Computational Linguistics (ACL) , 2002. -
  25. [26][26] M. Rabinovich, M. Stern, and D. Klein. Abstract syntax networks for code generation and semantic parsing. In Association for Computational Linguistics (ACL) , 2017. -
  26. [27][27] A. Severyn and A. Moschitti. Learning to rank short text pairs with convolutional deep neural networks. In ACM Special Interest Group on Information Retreival (SIGIR) , pages 373-382, 2015. -
  27. [28][28] L. Shao, S. Gouws, D. Britz, A. Goldie, B. Strope, and R. Kurzweil. Generating high-quality and informative conversation responses with sequence-to-sequence models. In Empirical Methods in Natural Language Processing (EMNLP) , pages 2210-2219, 2017. -
  28. [29][29] D. Shen, Q. Su, P. Chapfuwa, W. Wang, G. Wang, R. Henao, and L. Carin. Nash: Toward endto-end neural architecture for generative semantic hashing. In Association for Computational Linguistics (ACL) , pages 2041-2050, 2018. -
  29. [30][30] Y. Song, R. Yan, X. Li, D. Zhao, and M. Zhang. Two are better than one: An ensemble of retrieval- and generation-based dialog systems. arXiv preprint arXiv:1610.07149 , 2016. - · arXiv
  30. [31][31] N. Srivastava and R. R. Salakhutdinov. Multimodal learning with deep Boltzmann machines. In Advances in Neural Information Processing Systems (NIPS) , pages 2222-2230, 2012. -
  31. [32][32] C. J. Stone. Consistent nonparametric regression. Annals of Statistics , 5, 1977. -
  32. [33][33] E. Sumita and H. Iida. Experiments and prospects of example-based machine translation. In Association for Computational Linguistics (ACL) , 1991. -
  33. [34][34] W. Sun, A. Beygelzimer, H. Daume, J. Langford, and P. Mineiro. Contextual memory trees. arXiv preprint arXiv:1807.06473 , 2018. - · arXiv
  34. [35][35] M. Tan, C. dos Santos, B. Xiang, and B. Zhou. LSTM-based deep learning models for nonfactoid answer selection. arXiv preprint arXiv:1511.04108 , 2015. - · arXiv
  35. [36][36] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016. - · arXiv
  36. [37][37] Y. Wu, F. Wei, S. Huang, Z. Li, and M. Zhou. Response generation by context-aware prototype editing. arXiv preprint arXiv:1806.07042 , 2018. - · arXiv
  37. [38][38] J. Xu and G. Durrett. Spherical latent spaces for stable variational autoencoders. In Empirical Methods in Natural Language Processing (EMNLP) , 2018. -
  38. [39][39] F. Yan and K. Mikolajczyk. Deep correlation for matching images and text. In Computer Vision and Pattern Recognition (CVPR) , pages 3441-3450, 2015. -
  39. [40][40] R. Yan, Y. Song, and H. Wu. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In ACM Special Interest Group on Information Retreival (SIGIR) , pages 55-64, 2016. -
  40. [41][41] P. Yin and G. Neubig. A syntactic neural model for general-purpose code generation. In Association for Computational Linguistics (ACL) , pages 440-450, 2017.

Cited by

Sorted by citations

Authors
Hashimoto, T. B., Guu, K., Oren, Y., & Liang, P.
Affiliations
  • Massachusetts Institute of Technology (US)
  • Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Volume
31
Pages
10052-10062
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1812.01194
arXiv
1812.01194
OpenAlex
W2890397703
Cited by
126
References
0
Citations by year
2025: 5 · 2024: 7 · 2023: 13 · 2022: 21 · 2021: 33 · 2020: 26 · 2019: 20 · 2018: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
structured prediction, code generation, sequence-to-sequence models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review