CoChat

StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow

Open access
Yao, Ziyu, Weld, Daniel S., Chen, Wei-Peng, Sun, Huan · 2018 · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18
Cited by 76

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Abstract

Stack Overflow (SO) has been a great source of natural language questions and their code solutions (i.e., question-code pairs), which are critical for many tasks including code retrieval and annotation. In most existing research, question-code pairs were collected heuristically and tend to have low quality. In this paper, we investigate a new problem of systematically mining question-code pairs from Stack Overflow (in contrast to heuristically collecting them). It is formulated as predicting whether or not a code snippet is a standalone solution to a question. We propose a novel Bi-View Hierarchical Neural Network which can capture both the programming content and the textual context of a code snippet (i.e., two views) to make a prediction. On two manually annotated datasets in Python and SQL domain, our framework substantially outperforms heuristic methods with at least 15% higher F1 and accuracy. Furthermore, we present StaQC (Stack Overflow Question-Code pairs), the largest dataset to date of ~148K Python and ~120K SQL question-code pairs, automatically mined from SO using our framework. Under various case studies, we demonstrate that StaQC can greatly help develop data-hungry models for associating natural language with programming language

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3178876.3186081

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Research works printed but not linked: 45

  1. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code ICML. 2091--2100.
  2. Miltos Allamanis, Daniel Tarlow, Andrew Gordon, and Yi Wei. 2015. Bimodal modelling of source code and natural language ICML. 2123--2132.
  3. Alberto Bacchelli. 2013. Mining Challenge 2013: Stack Overflow. In The 10th Working Conference on Mining Software Repositories. to appear.
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR Vol. abs/1409.0473 (2014). [arxiv]1409.0473 http://arxiv.org/abs/1409.0473 · arXiv
  5. Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of Python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275 (2017). · arXiv
  6. Kyunghyun Cho, Bart van Merriënboer, Calar Gülc cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder--Decoder for Statistical Machine Translation. In EMNLP. Association for Computational Linguistics, Doha, Qatar, 1724--1734.
  7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement Vol. 20, 1 (1960), 37--46. · DOI · Paper page
  8. David R Cox. 1958. The regression analysis of binary sequences. Journal of the Royal Statistical Society. Series B (Methodological) (1958), 215--242.
  9. Lucas BL de Souza, Eduardo C Campos, and Marcelo de A Maia. 2014. Ranking crowd knowledge to assist software development Proceedings of the 22nd International Conference on Program Comprehension. ACM, 72--82.
  10. Fernanda Madeiral Delfim, Klérisson VR Paix ao, Damien Cassou, and Marcelo de Almeida Maia. 2016. Redocumenting APIs with crowd knowledge: a coverage analysis based on question types. Journal of the Brazilian Computer Society Vol. 22, 1 (2016), 9. · DOI
  11. Maarten Duijn, Adam Kuvcera, and Alberto Bacchelli. 2015. Quality questions need quality code: classifying code fragments on stack overflow Proceedings of the 12th Working Conference on Mining Software Repositories. IEEE Press, 410--413.
  12. Alessandra Giordani and Alessandro Moschitti. 2009. Semantic mapping between natural language questions and SQL queries via syntactic pairing. In International Conference on Application of Natural Language to Information Systems. Springer, 207--221.
  13. Alessandra Giordani and Alessandro Moschitti. 2010. Corpora for Automatically Learning to Map Natural Language Questions into SQL Queries. LREC.
  14. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 249--256.
  15. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org
  16. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. 1693--1701.
  17. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model ACL, Vol. 1. 2073--2083.
  18. Iman Keivanloo, Juergen Rilling, and Ying Zou. 2014. Spotting working code examples. In ICSE. ACM, 664--675. · DOI · Paper page
  19. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. arXiv preprint arXiv:1506.01057 (2015). · arXiv
  20. Edward Loper and Steven Bird. 2002. NLTK: The Natural Language Toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1 (ETMTNLP '02). Association for Computational Linguistics, Stroudsburg, PA, USA, 63--70. https://doi.org/10.3115/1118108.1118117 · DOI · Paper page
  21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality NIPS. 3111--3119.
  22. Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural networks over tree structures for programming language processing AAAI. · DOI · Paper page
  23. Seyed Mehdi Nasehi, Jonathan Sillito, Frank Maurer, and Chris Burns. 2012. What makes a good code example?: A study of programming Q&A in StackOverflow Software Maintenance (ICSM), 2012 28th IEEE International Conference on. IEEE, 25--34.
  24. Anh Tuan Nguyen and Tien N Nguyen. 2015. Graph-based statistical language model for code. In Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, 858--868. · DOI · Paper page
  25. Kamal Nigam and Rayid Ghani. 2000. Analyzing the effectiveness and applicability of co-training CIKM. ACM, 86--93.
  26. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation (t). In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. IEEE, 574--584.
  27. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research Vol. 12 (2011), 2825--2830.
  28. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global Vectors for Word Representation.. In EMNLP, Vol. Vol. 14. 1532--1543. · DOI · Paper page
  29. Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract Syntax Networks for Code Generation and Semantic Parsing ACL. · DOI · Paper page
  30. Mukund Raghothaman, Yi Wei, and Youssef Hamadi. 2016. SWIM: synthesizing what I mean: code search and idiomatic snippet synthesis ICSE. ACM, 357--367. · DOI · Paper page
  31. Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016. Data programming: Creating large training sets, quickly NIPS. 3567--3575.
  32. Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models.. In AAAI. 3776--3784. · DOI · Paper page
  33. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval CIKM. ACM, 101--110.
  34. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). · arXiv
  35. Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, and Michael Gamon. 2017. Building Natural Language Interfaces to Web APIs, In CIKM. · DOI
  36. Christian Szegedy, Alexander Toshev, and Dumitru Erhan. 2013. Deep neural networks for object detection. In NIPS. 2553--2561.
  37. Duyu Tang, Bing Qin, and Ting Liu. 2015. Document Modeling with Gated Recurrent Neural Network for Sentiment Classification. EMNLP. 1422--1432. · DOI · Paper page
  38. Christoph Treude, Ohad Barzilay, and Margaret-Anne Storey. 2011. How do programmers ask and answer questions on the web?: Nier track ICSE. IEEE, 804--807. · DOI · Paper page
  39. Venkatesh Vinayakarao, Anita Sarma, Rahul Purandare, Shuktika Jain, and Saumya Jain. 2017. Anne: Improving source code search using entity retrieval approach Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM, 211--220. · DOI
  40. Ellen M Voorhees et al. 1999. The TREC-8 Question Answering Track Report.. In Trec, Vol. Vol. 99. 77--82.
  41. Di Yang, Aftab Hussain, and Cristina Videira Lopes. 2016 a. From query to usable code: an analysis of stack overflow code snippets Proceedings of the 13th International Workshop on Mining Software Repositories. ACM, 391--402. · DOI · Paper page
  42. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016 b. Hierarchical attention networks for document classification Proceedings of NAACL-HLT. 1480--1489. · DOI · Paper page
  43. Annie T. T. Ying. 2015. Mining Challenge 2015: Comparing and combining different information sources on the Stack Overflow data set. In The 12th Working Conference on Mining Software Repositories. to appear.
  44. Zhi-Hua Zhou and Ming Li. 2005. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on knowledge and Data Engineering Vol. 17, 11 (2005), 1529--1541. · DOI · Paper page
  45. Meital Zilberstein and Eran Yahav. 2016. Leveraging a corpus of natural language descriptions for program similarity. In Proceedings of the 2016 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. ACM, 197--211. · DOI

Cited by

See all 76

Metadata

Authors
Yao, Z., Weld, D. S., Chen, W.-P., & Sun, H.
Venue
Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18
Publisher
ACM Press
Published
2018
Pages
1693-1703
OpenAlex
W2788306232
Cited by
76
References
11
FWCI
7.57
Type
conference paper
Language
en
Open access
gold
Funders
  • DARPA (FA8750-13-2-0019)
  • Army Research Office (W911NF-17-1-0412)
  • Fujitsu Laboratories of America
  • the University of Washington WRF/Cable Professorship
  • Ohio Supercomputer Center (PAS1197)
  • NSF (CNS-1513120)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow. https://app.cochat.ai/papers/10.1145/3178876.3186081
The paper
Yao, Z., Weld, D. S., Chen, W.-P., & Sun, H. (2018). StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow. Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18. https://doi.org/10.1145/3178876.3186081

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.