Why don't software developers use static analysis tools to find bugs?
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Using static analysis tools for automating code inspections can be beneficial for software engineers. Such tools can make finding bugs, or software defects, faster and cheaper than manual inspections. Despite the benefits of using static analysis tools to find bugs, research suggests that these tools are underused. In this paper, we investigate why developers are not widely using static analysis tools and how current tools could potentially be improved. We conducted interviews with 20 developers and found that although all of our participants felt that use is beneficial, false positives and the way in which the warnings are presented, among other things, are barriers to use. We discuss several implications of these results, such as the need for an interactive mechanism to help developers fix defects.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“industry reports show that it is difficult to get developers to adopt static analysis in their workflows [5, 11, 30, 34].”
From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [11]
Citations
Cites
(2005) · Texas ScholarWorks (Texas Digital Library) · Cited by 1,064
A few billion lines of code later: using static analysis to find bugs in the real world
Bessey, A., Block, K., Chelf, B., Chou, A., Fulton, B., Hallem, S., Henri-Gros, C., Kamsky, A., McPeak, S., & Engler, D. (2010) · Communications of the ACM · doi:10.1145/1646353.1646374 · Cited by 635
Using Static Analysis to Find Bugs
Ayewah, N., Pugh, W., Hovemeyer, D., Morgenthaler, J. D., & Penix, J. (2008) · IEEE Software · doi:10.1109/ms.2008.130 · Cited by 534
Experiences from Conducting Semi-structured Interviews in Empirical Software Engineering Research
Hove, S. E., & Anda, B. (2005) · 11th IEEE International Software Metrics Symposium (METRICS'05) · doi:10.1109/metrics.2005.24 · Cited by 345
Mylar: a degree-of-interest model for IDEs
Kersten, M., & Murphy, G. C. (2005) · Proceedings of the 4th international conference on Aspect-oriented software development · doi:10.1145/1052898.1052912 · Cited by 298
Johnson, S. C., & Hill, M. G. (1978) · Cited by 280
Static analysis tools as early indicators of pre-release defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062558 · Cited by 272
Secure Programming with Static Analysis
Chess, B., & West, J. (2007) · Cited by 224
Refactoring Tools: Fitness for Purpose
Murphy-Hill, E., & Black, A. P. (2008) · IEEE Software · doi:10.1109/ms.2008.123 · Cited by 155
Self-citationAn interactive ambient visualization for code smells
Murphy-Hill, E., & Black, A. P. (2010) · Proceedings of the 5th international symposium on Software visualization · doi:10.1145/1879211.1879216 · Cited by 147
Self-citation
Research works printed but not linked: 10
- 0, KLOCwork inSight
- 0, Microsoft Visual Studio
- 0, Google CodePro Analytix
- robertson, 2004, Impact of interruption style on enduser debugging, Proc CHI, 287
- lewis, 1982, Using the "Thinking Aloud" Method in Cognitive Interface Design
- 0, Eclipse
- gordon, 1998, Coding interview responses, Basic Interviewing Skills Waveland Pr Inc, 183
- 0, FindBugs
- 0, IntelliJ IDEA
- 0, Netbeans
Cited by
Deep Learning Based Vulnerability Detection: Are We There Yet?
Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3087402 · Cited by 790
“Although studies report some limited case studies on such on finding vulnerabilities in real-world projects, these case studies do not shed light on the false positives and false negatives [1].”
BackgroundLe Goues, C., Pradel, M., & Roychoudhury, A. (2019) · Communications of the ACM · doi:10.1145/3318162 · Cited by 410
“sis are becoming increasingly popular as quality gates in many companies.(7,23) Testing, at multiple levels of system ab-”
BackgroundAnalyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
Beller, M., Bholanath, R., McIntosh, S., & Zaidman, A. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.105 · Cited by 219
“Developers might be unaware of these benefits or an overload of warning messages generated by multiple ASATs might cause developers to avoid them [12].”
BackgroundResultHow developers engage with static analysis tools in different contexts
Vassallo, C., Panichella, S., Palomba, F., Proksch, S., Gall, H. C., & Zaidman, A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09750-5 · Cited by 161
Tricorder: Building a Program Analysis Ecosystem
Sadowski, C., Van Gogh, J., Jaspan, C., Soderberg, E., & Winter, C. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.76 · Cited by 145
“High false positive rates, confusing output, and poor integration into the developers’ workflow all contribute to the lack of use in everyday development activities [23], [27].”
BackgroundInfluentialHackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes
Votipka, D., Stevens, R., Redmiles, E. M., Hu, J., & Mazurek, M. L. (2018) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2018.00003 · Cited by 203
Root causing flaky tests in a large-scale industrial setting
Lam, W., Godefroid, P., Nath, S., Santhiar, A., & Thummalapenta, S. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330570 · Cited by 185
“However, to integrate these techniques within modern CI pipelines would require overcoming significant challenges of meeting performance, resource and time constraints, providing multi-language support, and controlling rates of false positives and false negatives [14, 15, 28, 35].”
BackgroundPredicting Defective Lines Using a Model-Agnostic Technique
Wattanakriengkrai, S., Thongtanunam, P., Tantithamthavorn, C., Hata, H., & Matsumoto, K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3023177 · Cited by 137
Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise
Floyd, B., Santander, T., & Weimer, W. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.24 · Cited by 125
An empirical study on the effectiveness of static C code analyzers for vulnerability detection
Lipp, S., Bănescu, S., & Pretschner, A. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534380 · Cited by 240
“This is a common criticism [30, 39, 43, 44] of static analyzers and has been addressed in many research papers [19, 46, 48, 52, 55, 59, 67].”
BackgroundImproving developer participation rates in surveys
Smith, E., Loftin, R., Murphy-Hill, E., Bird, C., & Zimmermann, T. (2013) · 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2013.6614738 · Cited by 129
Self-citationDeepLineDP: Towards a Deep Learning Approach for Line-Level Defect Prediction
Pornprasit, C., & Tantithamthavorn, C. K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3144348 · Cited by 165
“However, static analysis tools generate too many false positives warnings [26], and they may not be associated with post-release defects.”
BackgroundThe Effectiveness of Supervised Machine Learning Algorithms in Predicting Software Refactoring
Aniche, M. F., Maziero, E. G., Durelli, R. S., & Durelli, V. H. S. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3021736 · Cited by 155
How Open Source Projects Use Static Code Analysis Tools in Continuous Integration Pipelines
Zampetti, F., Scalabrino, S., Oliveto, R., Canfora, G., & Di Penta, M. (2017) · 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.2 · Cited by 122
A Systematic Evaluation of Static API-Misuse Detectors
Amann, S., Nguyen, H., Nadi, S., Nguyen, T., & Mezini, M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2827384 · Cited by 135
“Past studies show that developers rarely use analysis tools that produce many false positives [39], [40], [41].”
BackgroundAre Static Analysis Violations Really Fixed? A Closer Look at Realistic Usage of SonarQube
Marcílio, D., Bonifácio, R., Monteiro, E. B., Canedo, E. D., Luz, W., & Pinto, G. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00040 · Cited by 159
“Even though the use of ASATs provide several benefits, developers still face challenges when using them [2], [3].”
BackgroundMethodInfluentialContext is king: The developer perspective on the usage of static analysis tools
Vassallo, C., Panichella, S., Palomba, F., Proksch, S., Zaidman, A., & Gall, H. C. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330195 · Cited by 132
“Hard and tedious tasks, such as source code analysis, quality assessment, and debugging [22], provide an excellent opportunity for tool support that does not only make life of developers easier, but that also improves the quality of the result.”
BackgroundInfluentialUnderstanding Privacy-Related Questions on Stack Overflow
Tahaei, M., Vaniea, K. E., & Saphra, N. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376768 · Cited by 129
“De-veloper errors are a common source of vulnerabilities [24] with many causes ranging from APIs with poor developer support [1, 47] to static analysis tools that produce too many false positives [38].”
BackgroundThe Interactive Visualization Gap in Initial Exploratory Data Analysis
Batch, A., & Elmqvist, N. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2743990 · Cited by 115
An empirical characterization of bad practices in continuous integration
Zampetti, F., Vassallo, C., Panichella, S., Canfora, G., Gall, H. C., & Penta, M. D. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09785-8 · Cited by 136
A Survey of Performance Optimization for Mobile Applications
Hort, M., Kechagia, M., Sarro, F., & Harman, M. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3071193 · Cited by 91
“Static analysis can be used to support developers to find bugs and inspect code [269].”
A Qualitative Study of Dependency Management and Its Security Implications
Николаевич, П. И., Vu, D., & Massacci, F. (2020) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3372297.3417232 · Cited by 143
“, build or compile time [21, 43], integrated in an IDE [15], or into a code review [43].”
BackgroundInfluentialDo Developers Read Compiler Error Messages?
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.59 · Cited by 86
Self-citationOn Reliability of Patch Correctness Assessment
Le, X.-B. D., Bao, L., Lo, D., Xia, X., Li, S., & Păsăreanu, C. S. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00064 · Cited by 121
“idered. •We have included 35 professional developers in our user study. The number of professional developers included in this study is larger or similar to those considered in many prior work, e.g., [25, 28, 60]. Admittedly, it is possible that results differ for other groups of developers. To reduce this threat, we…”
ResultQuang, L. N., Wright, J. R., & Ali, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3004525 · Cited by 79
“[6], [19], [22], [29], and asked participants to rank their importance between six categories: should not exist, neutral, low importance, important, very important, and indispensable (Q30).”
BackgroundMethodResultInfluentialA Study of Visual Studio Usage in Practice
Amann, S., Proksch, S., Nadi, S., & Mezini, M. (2016) · 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) · doi:10.1109/saner.2016.39 · Cited by 77
“4) A discussion of opportunities to advance the research in IDEs and developer-assistance tools.”
BackgroundSurvey of Approaches for Handling Static Analysis Alarms
Muske, T., & Serebrenik, A. (2016) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2016.25 · Cited by 94
Evaluating how static analysis tools can reduce code review effort
Singh, D., Sekar, V. R., Stolee, K. T., & Johnson, B. (2017) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2017.8103456 · Cited by 102
“In fact, studies have already recognized this challenge, showing that static analysis tools are under-utilized, in part because the output is hard to understand and false positives are often abundant [17], which can deter developers from integrating the tool into their regular workflow [10], [18].”
BackgroundSelf-citationSmart Contract and DeFi Security Tools: Do They Meet the Needs of Practitioners?
Chaliasos, S., Charalambous, M. A., Zhou, L., Galanopoulou, R., Gervais, A., Mitropoulos, D., & Livshits, B. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623302 · Cited by 130
“Johnson et al. [26] and Witschey et al. [52] explored why security tools are underused despite their benefits.”
BackgroundSmart Contract Security: A Practitioners' Perspective
Wan, Z., Xia, X., Lo, D., Chen, J., Luo, X., & Yang, X. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00127 · Cited by 95
Just-in-time software vulnerability detection: Are we there yet?
Lomio, F., Iannone, E., Lucia, A. D., Palomba, F., & Lenarduzzi, V. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111283 · Cited by 88
Lent, R., Grochla, K., Gelenbe, E., & Czachórski, T. (2016) · Communications in computer and information science · doi:10.1007/978-3-319-47217-1 · Cited by 69
The Adoption of JavaScript Linters in Practice: A Case Study on ESLint
Tomasdottir, K. F., Aniche, M. F., & van Deursen, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2871058 · Cited by 88
Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far Are We?
Wen, C., Cai, Y., Zhang, B., Su, J., Xu, Z., Liu, D., Qin, S., Ming, Z., & Tian, C. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3653718 · Cited by 114
“As a result, thoroughly scrutinizing the sheer volume of bug warnings to uncover true bugs remains a laborious process for software practitioners due to the significant human effort and expertise required [31, 32].”
BackgroundAssisting Static Analysis with Large Language Models: A ChatGPT Experiment
Li, H., Yu, H., Zhai, Y., & Qian, Z. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613078 · Cited by 102
“In real-world applications, static analysis tools often generate a significant number of false positives, hindering their widespread adoption [2, 6, 21, 22].”
BackgroundAn empirical study of security warnings from static application security testing tools
Aloraini, B., Nagappan, M., Germán, D. M., Hayashi, S., & Higo, Y. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.110427 · Cited by 73
An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions
Petrović, G., Ivanković, M., Kurtz, B., Ammann, P., & Just, R. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00027 · Cited by 58
VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatements
Ding, Y., Suneja, S., Zheng, Y., Laredo, J., Morari, A., Kaiser, G. E., & Ray, B. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00114 · Cited by 66
A critical comparison on six static analysis tools: Detection, agreement, and precision
Lenarduzzi, V., Pecorelli, F., Saarimäki, N., Lujan, S., & Palomba, F. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111575 · Cited by 57
DLFinder: Characterizing and Detecting Duplicate Logging Code Smells
Li, Z., Chen, T.-H., Yang, J., & Shang, W. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00032 · Cited by 54
“Similar to some code smells studied in prior research [23], [42], developers may be reluctant to fix DP due to additional maintenance overheads but limited supports (i.”
ResultLenarduzzi, V., Saarimäki, N., & Taibi, D. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110750 · Cited by 76
Trust enhancement issues in program repair
Noller, Y., Shariffdeen, R., Gao, X., & Roychoudhury, A. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510040 · Cited by 52
“Our replication package with the survey and experiment artifacts is available on Zenodo [35].”
MethodDetecting false alarms from automatic static analysis tools: how far are we?
Kang, H. J., Aw, K. L., & Lo, D. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510214 · Cited by 80
“alarms that they produce, and among other reasons, this has led to resistance in adopting them in many software projects [19].”
BackgroundA Gamified Tool for Motivating Developers to Remove Warnings of Bug Pattern Tools
Arai, S., Sakamoto, K., Washizaki, H., & Fukazawa, Y. (2014) · International Workshop on Empirical Software Engineering in Practice (IWESEP) · doi:10.1109/iwesep.2014.17 · Cited by 66
“Although the effectiveness of bug pattern tools has already been evaluated in various research studies [2], [3], [4], existing tools sometimes yield so many warnings that developers tend to ignore all the warnings [5], [6].”
BackgroundAn automatically created novel bug dataset and its validation in bug prediction
Ferenć, R., Gyimesi, P., Gyimesi, G., Tóth, Z., & Gyimóthy, T. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110691 · Cited by 76
Error Message Readability and Novice Debugging Performance
Denny, P., Prather, J., & Becker, B. A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387384 · Cited by 50
“Inadequate explanation of the problem causing an error is a key reason why developers report finding messages unhelpful [26].”
BackgroundWhy and how JavaScript developers use linters
Tomasdottir, K. F., Aniche, M. F., & Arie Van Deursen, A. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115668 · Cited by 68
Oyetoyan, T. D., Milosheska, B., Grini, M., & Cruzes, D. S. (2018) · Lecture notes in business information processing · doi:10.1007/978-3-319-91602-6_6 · Cited by 49
How Developers Diagnose Potential Security Vulnerabilities with a Static Analysis Tool
Smith, J., Johnson, B., Murphy-Hill, E. R., Chu, B., & Lipford, H. R. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2810116 · Cited by 58
Self-citationA large-scale study of usability criteria addressed by static analysis tools
Nachtigall, M., Schlichtig, M., & Bodden, E. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534374 · Cited by 66
“[34] also support that developers would prefer fix previews instead of directly applying the fix.”
BackgroundInfluentialSorry to Bother You: Designing Bots for Effective Recommendations
Brown, C., & Parnin, C. (2019) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse.2019.00021 · Cited by 59
“Additionally, Johnson and colleagues found that tools’ inability to integrate into developer workflow is a key barrier for adoption [7].”
Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities
Guo, Z., Tan, T., Liu, S., Liu, X., Lai, W., Yang, Y., Li, Y., Chen, L., Dong, W., & Zhou, Y. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3329667 · Cited by 65
An Empirical Study on Detecting and Fixing Buffer Overflow Bugs
Ye, T., Zhang, L., Wang, L., & Li, X. (2016) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2016.21 · Cited by 43
“[13] studied the static analysis techniques in general, and found that false positives are the barriers to use static analysis techniques in practice.”
BackgroundCausal testing: understanding defects' root causes
Johnson, B., Brun, Y., & Meliou, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380377 · Cited by 78
“nsed to ACM. ACM ISBN 978-1-4503-7121-6/20/05...$15.00 https://doi.org/10.1145/3377811.3380377 test input [74] and a set of test-breaking changes [73], they do not help explain why the code is faulty [40]. To address this shortcoming of modern debugging tools, this paper presents CausalTesting, a novel technique for i…”
MethodInfluentialSelf-citationChallenges with Responding to Static Analysis Tool Alerts
Imtiaz, N., Rahman, A., Farhana, E., & Williams, L. A. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00049 · Cited by 42
“[10] found that false positives; the way in which the alerts are presented; and a lack of customizability are barriers to the use of static analysis tools.”
BackgroundInfluentialSpongeBugs: Automatically generating fix suggestions in response to static code analysis warnings
Marcílio, D., Furia, C. A., Bonifácio, R., & Pinto, G. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110671 · Cited by 46
“Crucially, SATs are prone to generating a high number of false positives [37].”
BackgroundCRYLOGGER: Detecting Crypto Misuses Dynamically
Piccolboni, L., Guglielmo, G. D., Carloni, L. P., & Sethumadhavan, S. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00010 · Cited by 59
“As discussed in [27], the main problem with static analysis is the high number of false positives, which requires the users to manually examine the results and determine the true positives.”
BackgroundLearning to reduce false positives in analytic bug detectors
Kharkar, A., Moghaddam, R. Z., Jin, M., Liu, X., Shi, X., Clement, C. B., & Sundaresan, N. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510153 · Cited by 80
“However, one of the barriers to consistent usage of static analyzers is their high false positive rate [11].”
BackgroundInfluentialTrustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmaps
Lo, D. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00010 · Cited by 40
“Another research, although not centered on AI4SE, underscores that many professionals avoid static analysis solutions due to their frequent false positives, among other reasons [56], [57].”
BackgroundGame of detections: how are security vulnerabilities discovered in the wild?
Hafiz, M., & Fang, M. (2015) · Empirical Software Engineering · doi:10.1007/s10664-015-9403-7 · Cited by 41
“Works by Johnson and colleagues (Johnson et al. 2013), Baca and colleagues (Baca et al. 2013), Rutar and colleagues (Rutar et al. 2004), and McGraw and Steven (2011) explored the usability challenges of static analysis tools.”
BackgroundAmi, A. S., Moran, K., Poshyvanyk, D., & Nadkarni, A. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00019 · Cited by 86
“Contemporary literature asserts that false positives are a major reason for practitioners to avoid using SASTs [25], [26] since “Developer Happiness is Key” [27], and argues that it is necessary to reduce false positives, in general.”
BackgroundTrautsch, A., Herbold, S., & Grabowski, J. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09880-1 · Cited by 53
“One of the problems identified for ASAT usage is the number of false positives (Johnson et al., 2013; Chris-takis and Bird, 2016; Vassallo et al., 2019) for which warning priorization (Kim and Ernst, 2007a,b) was proposed, sometimes as actionable warning identification (Heckman and Williams, 2009),…”
BackgroundListen to Developers! A Participatory Design Study on Security Warnings for Cryptographic APIs
Gorski, P. L., Acar, Y., Iacono, L. L., & Fahl, S. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376142 · Cited by 63
“High false positives rates [22] and bad warning message design [7, 14] are significant problems when working with warnings from program analyzers.”
BackgroundGoues, C. L., Pradel, M., Roychoudhury, A., & Chandra, S. (2021) · IEEE Software · doi:10.1109/ms.2021.3072577 · Cited by 46
An Empirical Study of Design Degradation: How Software Projects Get Worse over Time
Ahmed, I., Mannan, U. A., Gopinath, R., & Jensen, C. (2015) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2015.7321186 · Cited by 53
“Moreover, developers don’t want to have their workflow disrupted by tools that do not integrate well into their development process [19].”
BackgroundHow Do Developers Act on Static Analysis Alerts? An Empirical Study of Coverity Usage
Imtiaz, N., Murphy, T. B., & Williams, L. A. (2019) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2019.00040 · Cited by 38
“However, SATs come with several shortcomings which can make developers turn away from using the tools [33].”
BackgroundInfluentialUnderstanding stack overflow code quality: A recommendation of caution
Meldrum, S., Licorish, S. A., Owen, C. A., & Savarimuthu, B. T. R. (2020) · Science of Computer Programming · doi:10.1016/j.scico.2020.102516 · Cited by 49
Towards Better Utilizing Static Application Security Testing
Yang, J., Tan, L., Peyton, J., & Duer, K. A. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00014 · Cited by 35
“A high rate of false warnings makes developers lose interests in the detection results [11].”
BackgroundInfluentialBuilding and Validating a Scale for Secure Software Development Self-Efficacy
Votipka, D., Abrokwa, D., & Mazurek, M. L. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376754 · Cited by 54
“A secure development skill measure is also needed for use as a covariate when evaluating new security tools [64, 102, 113, 106, 117, 98, 60, 101], documentation [2], and APIs [1, 58, 116].”
MethodSiavvas, M., Tsoukalas, D., Jankovic, M., Kehagias, D. D., & Tzovaras, D. K. (2020) · Enterprise Information Systems · doi:10.1080/17517575.2020.1824017 · Cited by 43
“…(Zheng et al. 2006; Austin, Holmgreen, and Williams 2013; Holzmann 2017; Felderer et al. 2016), whereas the existence of quality issues (e.g. code smells, etc.) in security contributing commits (and vice versa) is also highly discussed (Mohammed et al. 2017; Johnson et al. 2013; Holzmann 2017).”
BackgroundDeepMerge: Learning to Merge Programs
Dinella, E., Mytkowicz, T., Svyatkovskiy, A., Bird, C., Naik, M., & Lahiri, S. K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3183955 · Cited by 58
On the use of fine-grained vulnerable code statements for software vulnerability assessment models
Le, T. H. M., & Babar, M. A. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528433 · Cited by 57
“Note that it is non-trivial to use static application security testing tools to automatically create bug/SV reports from vulnerable functions for current SV assessment techniques as these tools often have too many false positives [2, 32].”
BackgroundOn the Survival of Android Code Smells in the Wild
Habchi, S., Rouvoy, R., & Moha, N. (2019) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1109/mobilesoft.2019.00022 · Cited by 35
Tahaei, M., Vaniea, K. E., Beznosov, K., & Wolters, M. K. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445616 · Cited by 60
“But existing SATs suffer from usability issues such as interrupting work flow [22, 54, 88, 93], and poor integration with IDEs [54].”
InfluentialNeural-Based Test Oracle Generation: A Large-Scale Evaluation and Lessons Learned
Hossain, S. B., Filieri, A., Dwyer, M. B., Elbaum, S. G., & Visser, W. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616265 · Cited by 60
“Widely cited studies [12, 27, 44] have demonstrated that high false positives are a major barrier to the adoption of automated software testing in industry and tools that generate more than 10% false positives waste developers time causing developers to lose trust in them and gradually abandon them.”
BackgroundHow do Developers Really Feel About Bug Fixing? Directions for Automatic Program Repair
Winter, E., Bowes, D., Counsell, S., Hall, T., Haraldsson, S. O., Nowack, V., & Woodward, J. R. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3194188 · Cited by 49
“This confirms other research findings [1], [5] that the use of debugging tools is not widespread among professional developers, indicating that the take-up of tools is slow and uneven in industry, as well as research on barriers to adoption of static analysis tools [53].”
ResultKoç, U., Wei, S., Foster, J. S., Carpuat, M. J., & Porter, A. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00036 · Cited by 47
“Because developers must manually resolve each SA report, many regard this issue as a key barrier to using SA tools in practice [32].”
BackgroundEvaluating Automatic Program Repair Capabilities to Repair API Misuses
Kechagia, M., Mechtaev, S., Sarro, F., & Harman, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3067156 · Cited by 43
Siavvas, M., Kehagias, D. D., Tzovaras, D. K., & Gelenbe, E. (2021) · Software Quality Journal · doi:10.1007/s11219-021-09555-0 · Cited by 38
“This process, which is known as triaging (Walden & Doyle, 2012; Ruthruff, 2008), is highly time consuming and effort demanding, and is known to be the main reason that static analysis tools are underused in practice (Johnson et al., 2013).”
BackgroundInfluentialLe, T.-D. B., Thung, F., & Lo, D. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9484-y · Cited by 38
“, FindBugs (Hovemeyer and Pugh 2004) and Lint (Johnson 1978)) that typically return many false positive warnings, which potentially cause developers to distrust their outputs (Heckman and Williams 2011; Kim and Ernst 2007; Johnson et al. 2013; Ayewah and Pugh 2010).”
BackgroundMethodSemgrep*: Improving the Limited Performance of Static Application Security Testing (SAST) Tools
Bennett, G., Hall, T., Winter, E., & Counsell, S. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661262 · Cited by 32
“Johnson et al. [18] report tool overload as a barrier to using static analysis tools, which may be exacerbated by combining tools.”
BackgroundC-3PR: A Bot for Fixing Static Analysis Violations via Pull Requests
Carvalho, A., Luz, W., Marcílio, D., Bonifácio, R., Pinto, G., & Canedo, E. D. (2020) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054842 · Cited by 40
“For the first problem, Johnson and colleagues discuss that the user interface of static analysis tools does not clearly display the detected problems nor show how to solve them [5].”
BackgroundInfluentialDetecting Video Game-Specific Bad Smells in Unity Projects
Borrelli, A., Nardone, V., Lucca, G. A. D., Canfora, G., & Penta, M. D. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387454 · Cited by 32
“For example, Johnson et al. [27] and Beller et al. [18] have studied the usage of static analysis tools in software projects.”
MethodAccelerating Software Engineering Research Adoption with Analysis Bots
Beschastnikh, I., Lungu, M. F., & Zhuang, Y. (2017) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier.2017.17 · Cited by 37
Why Crypto-detectors Fail: A Systematic Evaluation of Cryptographic Misuse Detection Techniques
Ami, A. S., Cooper, N., Kafle, K., Moran, K., Poshyvanyk, D., & Nadkarni, A. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833582 · Cited by 56
A Qualitative Analysis of Android Taint-Analysis Results
Luo, L., Bodden, E., & Späth, J. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00020 · Cited by 48
“Developers Are Responsible”: What Ad Networks Tell Developers About Privacy
Tahaei, M., & Vaniea, K. E. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451805 · Cited by 54
“2953609 [23] Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge.”
BackgroundInfluentialTracking Static Analysis Violations over Time to Capture Developer Characteristics
Avgustinov, P., Baars, A. I., Henriksen, A. S., Lavender, G., Menzel, G., de Moor, O., Schäfer, M., & Tibble, J. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.62 · Cited by 39
Towards More Realistic Evaluation for Neural Test Oracle Generation
Liu, Z., Liu, K., Xia, X., & Yang, X. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598080 · Cited by 40
“In summary, we find that simply generating the oracle that no exception should be thrown can achieve 68.9% of TOGA’s performance in terms of BugFound and twice TOGA’s Precision.”
BackgroundInfluentialAnalyzing Quantum Programs with LintQ: A Static Analysis Framework for Qiskit
Paltenghi, M., & Pradel, M. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660802 · Cited by 61
“Because precision is crucial for practical adoption of static analyzers [11, 20, 28], we assess to what extent the analyses suffer from false positives.”
BackgroundAiding Code Change Understanding with Semantic Change Impact Analysis
Hanam, Q., Mesbah, A., & Holmes, R. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00031 · Cited by 39
“, test selection) may tolerate imprecision and noise, human code comprehension [17], [23] tasks are less tolerant of imprecision and noise.”
BackgroundLearning to recognize actionable static code warnings (is intrinsically easy)
Yang, X., Chen, J., Yedida, R., Yu, Z., & Menzies, T. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09948-6 · Cited by 42
“This high false alarm rate is one of the most significant barriers for developers to use these tools (Thung et al. 2015; Avgustinov et al. 2015; Johnson et al. 2013).”
BackgroundContext-based code smells prioritization for prefactoring
Sae‐Lim, N., Hayashi, S., & Saeki, M. (2016) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2016.7503705 · Cited by 29
Cesar: Visual representation of source code vulnerabilities
Assal, H., Chiasson, S., & Biddle, R. (2016) · IEEE Symposium on Visualization for Cyber Security (VIZSEC) · doi:10.1109/vizsec.2016.7739576 · Cited by 29
“[12] explored the different factors influencing developers’ decision to use SATs to uncover vulnerabilities in their code.”
BackgroundMethodResultInfluentialAI-Assisted Assessment of Coding Practices in Modern Code Review
Vijayvergiya, M., Salawa, M., Budiselić, I., Zheng, D., Lamblin, P., Ivanković, M., Carin, J., Lewko, M., Andonov, J., Petrović, G., et al. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3665664 · Cited by 70
“Johnson et al. [14] explore challenges developers face when trying to use static analysis.”
BackgroundIDE Plugins for Detecting Input-Validation Vulnerabilities
Baset, A., & Denning, T. (2017) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2017.37 · Cited by 28
Enriching Compiler Testing with Real Program from Bug Report
Zhong, H. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556894 · Cited by 41
“As another example, we have not detected any optimization bugs, since we did not try corresponding flags as the prior work [58, 85] did.”
BackgroundMethodResultInfluentialStudying the use of Java logging utilities in the wild
Chen, B., & Jiang, Z. M. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380408 · Cited by 35
“During the interview, we sometime asked follow-up questions based on participants’ answers on the fly [32].”
BackgroundContext‐based approach to prioritize code smells for prefactoring
Sae‐Lim, N., Hayashi, S., & Saeki, M. (2017) · Journal of Software Evolution and Process · doi:10.1002/smr.1886 · Cited by 28
“Therefore, we also include different cut points of impact analysis (5, 10, 20, 30, and 40) into our setting.”
Kellogg, M., Schäf, M., Taşiran, S., & Ernst, M. D. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416593 · Cited by 48
“• Bug-finding tools like the above four tools aim for low false positive rates (high precision, or high confidence that each reported warning is useful), even at the cost of false negatives (unsoundness) [40].”
Background
Metadata
- Authors
- Johnson, B., Song, Y., Murphy-Hill, E., & Bowdidge, R.
- Affiliations
- North Carolina State University (US)
- Google (United States) (US)
- Venue
- 2013 35th International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 672-681
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4233410239
- Cited by
- 568
- References
- 23
- Citations by year
- 2026: 58 · 2025: 58 · 2024: 66 · 2023: 73 · 2022: 65 · 2021: 53 · 2020: 46 · 2019: 41 · 2018: 27 · 2017: 38 · 2016: 21 · 2015: 14 · 2014: 5 · 2013: 3
- FWCI
- 11.51
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- static analysis tools, software defect detection, bug detection, false positives, developer behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.