One size does not fit all: a grounded theory and online survey study of developer preferences for security warning types
Open accessThe DOI matches its Crossref record; 65 of the 67 references it cites were found in a registry or an index.
Abstract
A wide range of tools exist to assist developers in creating secure software. Many of these tools, such as static analysis engines or security checkers included in compilers, use warnings to communicate security issues to developers. The effectiveness of these tools relies on developers heeding these warnings, and there are many ways in which these warnings could be displayed. Johnson et al. [46] conducted qualitative research and found that warning presentation and integration are main issues. We built on Johnson et al.'s work and examined what developers want from security warnings, including what form they should take and how they should integrate into their workflow and work context. To this end, we conducted a Grounded Theory study with 14 professional software developers and 12 computer science students as well as a focus group with 7 academic researchers to gather qualitative insights. To back up the theory developed from the qualitative research, we ran a quantitative survey with 50 professional software developers. Our results show that there is significant heterogeneity amongst developers and that no one warning type is preferred over all others. The context in which the warnings are shown is also highly relevant, indicating that it is likely to be beneficial if IDEs and other development tools become more flexible in their warning interactions with developers. Based on our findings, we provide concrete recommendations for both future research as well as how IDEs and other security tools can improve their interaction with developers.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In order to verify that participants had programming knowledge, we asked a brief prescreening question before proceeding with the study that focused on participants' ability to read and interpret a for-loop-which has been used in other user studies [7].”
From Do Users Write More Insecure Code with AI Assistants? (2023) · cites this paper as [7]
Citations
Cites
The Discovery of Grounded Theory: Strategies for Qualitative Research.
Wagner, H. R., Glaser, B. G., & Strauss, A. L. (1968) · Social Forces · doi:10.2307/2575405 · Cited by 46,361
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 1×
“The coding schemes were compared, and inter-coder agreement was calculated using Cohen’s kappa coefficient (κ) [24].”
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449
(2006) · Cited by 18,730
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646
Basics of Qualitative Research
Ozanne, J. L., Strauss, A., & Corbin, J. (1992) · Journal of Marketing Research · doi:10.2307/3172751 · Cited by 17,850 · cited 2×
“As suggested by qualitative research experts [25], we present an explanatory theory that is grounded in the data collected from our participants:”
(2014) · Cited by 15,747
The Research Act: A Theoretical Introduction to Sociological Methods
Fox, W. S., & Denzin, N. K. (1979) · Contemporary Sociology · doi:10.2307/2065439 · Cited by 6,284
Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013) · Sociological Methods & Research · doi:10.1177/0049124113500475 · Cited by 2,974 · cited 1×
“In order to obtain the inter-coder agreement, we followed the approach of [18].”
The SAGE Handbook of Grounded Theory
Bryant, A., & Charmaz, K. (2007) · SAGE · doi:10.4135/9781848607941 · Cited by 2,798 · cited 3×
“As suggested by Charmaz [19], we used the idea of theoretical sorting to develop a diagram out of our memos, themes and categories offering an initial analytic frame (see supplementary material).”
Research works printed but not linked: 8
- [n.d.]. Amazon Mechanical Turk (MTurk). https://www.mturk.com/ Accessed: August 2019.
- [n.d.]. Qualtrics. https://www.qualtrics.com Accessed: August 2019.
- [n.d.]. XING. https://www.xing.com/ Accessed: August 2019.
- NK Denzin. 1970. The Research Act in Sociology (London, Croom Helm). Denzin The Research Act in Sociology 1970 (1970).
- Don A Dillman. 2011. Mail and Internet surveys: The tailored design method-2007 Update with new Internet, visual, and mixed-mode guide. John Wiley & Sons.
- 10.5555/2486788.2486877 · doi:10.5555/2486788.2486877
- 10.5555/3155562.3155681 · doi:10.5555/3155562.3155681
- David L Morgan. 1996. Focus groups. Annual review of sociology 22, 1 (1996), 129--152.
Cited by
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259
“…the entry to the database • Incorrect if it does not add the entry to the database • Secure if there are no security mistakes • Insecure if it is vulnerable to SQL injections • Unknown if the security of the answer cannot be determined • N/A If there is no answer, the answer is incomplete, or it is”
Do you Really Code? Designing and Evaluating Screening Questions for Online Surveys with Programmers
Danilova, A., Naiakshina, A., Horstmann, S., & Smith, M. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00057 · Cited by 62
Self-citationBeyond Bot Detection: Combating Fraudulent Online Survey Takers✱
Zhang, Z., Zhu, S., Mink, J., Xiong, A., Song, L., & Wang, G. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512230 · Cited by 84
“Finally, to identify eligible participants, screening questions are often used before the survey [1, 5, 7, 14, 15].”
Background(2022) · doi:10.1145/3491102.3501957 · Cited by 47
Ortloff, A.-M., Fassl, M., Ponticello, A., Martius, F., Mertens, A., Krombholz, K., & Smith, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580766 · Cited by 65
Self-citation
Metadata
- Authors
- Danilova, A., Naiakshina, A., & Smith, M.
- Affiliations
- University of Bonn (DE)
- Venue
- Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2020-06-27
- Pages
- 136-148
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W3090957174
- Cited by
- 25
- References
- 67
- Citations by year
- 2026: 3 · 2025: 1 · 2024: 2 · 2023: 6 · 2022: 6 · 2021: 7
- FWCI
- 3.79
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Funders
- ERC (678341)
- Topics
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- security warnings, developer preferences, integrated development environment, static analysis, security tools, developer workflows, software security
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: One size does not fit all: a grounded theory and online survey study of developer preferences for security warning types. https://app.cochat.ai/papers/10.1145/3377811.3380387
- The paper
- Danilova, A., Naiakshina, A., & Smith, M. (2020). One size does not fit all: a grounded theory and online survey study of developer preferences for security warning types. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. https://doi.org/10.1145/3377811.3380387
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.