How practitioners perceive the relevance of software engineering research
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
The number of software engineering research papers over the last few years has grown significantly. An important question here is: how relevant is software engineering research to practitioners in the field? To address this question, we conducted a survey at Microsoft where we invited 3,000 industry practitioners to rate the relevance of research ideas contained in 571 ICSE, ESEC/FSE and FSE papers that were published over a five year period. We received 17,913 ratings by 512 practitioners who labelled ideas as essential, worthwhile, unimportant, or unwise. The results from the survey suggest that practitioners are positive towards studies done by the software engineering research community: 71% of all ratings were essential or worthwhile. We found no correlation between the citation counts and the relevance scores of the papers. Through a qualitative analysis of free text responses, we identify several reasons why practitioners considered certain research ideas to be unwise. The survey approach described in this paper is lightweight: on average, a participant spent only 22.5 minutes to respond to the survey. At the same time, the results can provide useful insight to conference organizers, authors, and participating practitioners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 1×
“Other systematic review studies, for example on fault prediction [20], analyzed papers published in an area over several years to review the current state-of-the-art.”
BackgroundSoftware developers' perceptions of productivity
Meyer, A. N., Fritz, T., Murphy, G. C., & Zimmermann, T. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635892 · Cited by 224 · cited 1×
“Other work has focussed on how practitioners perceived specific software enginerring concepts such as coupling [22], bad smells [23], and productivity [24].”
BackgroundSelf-citation
Cited by
Smart Contract Development: Challenges and Opportunities
Zou, W., Lo, D., Kochhar, P. S., Le, X.-B. D., Xia, X., Feng, Y., Chen, Z., & Xu, B. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2942301 · Cited by 727
Self-citationPractitioners' expectations on automated fault localization
Kochhar, P. S., Xia, X., Lo, D., & Li, S. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931051 · Cited by 346
Self-citationMeasuring Program Comprehension: A Large-Scale Field Study with Professionals
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2734091 · Cited by 326
Self-citationRefining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149
“In this study, we follow the card sorting process highlighted by previous studies [33, 49], which mainly consisted of two phases.”
MethodSelf-citationProgrammers Are Users Too: Human-Centered Methods for Improving Programming Tools
Myers, B. A., Ko, A. J., LaToza, T. D., & Yoon, Y. (2016) · Computer · doi:10.1109/mc.2016.200 · Cited by 144
Metadata
- Authors
- Lo, D., Nagappan, N., & Zimmermann, T.
- Venue
- Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2015-08-30
- Pages
- 415-425
- OpenAlex
- W2011044158
- Cited by
- 135
- References
- 2
- FWCI
- 12.86
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.