Enabling Data-Driven API Design with Community Usage Data: A Need-Finding Study
The DOI matches its Crossref record; 57 of the 59 references it cites were found in a registry or an index.
Abstract
APIs are becoming the fundamental building block of modern software and their usability is crucial to programming efficiency and software quality. Yet API designers find it hard to gather and interpret user feedback on their APIs. To close the gap, we interviewed 23 API designers from 6 companies and 11 open-source projects to understand their practices and needs. The primary way of gathering user feedback is through bug reports and peer reviews, as formal usability testing is prohibitively expensive to conduct in practice. Participants expressed a strong desire to gather real-world use cases and understand users' mental models, but there was a lack of tool support for such needs. In particular, participants were curious about where users got stuck, their workarounds, common mistakes, and unanticipated corner cases. We highlight several opportunities to address those unmet needs, including developing new mechanisms that systematically elicit users' mental models, building mining frameworks that identify recurring patterns beyond shallow statistics about API usage, and exploring alternative design choices made in similar libraries.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
The SAGE handbook of qualitative research
(2005) · Choice Reviews Online · doi:10.5860/choice.43-1330 · Cited by 10,183
The Sage Handbook of Qualitative Research
Ho, R. T. H., Ng, S. M., & Ho, D. Y. F. (2007) · Asian Journal of Social Psychology · doi:10.1111/j.1467-839x.2007.00237.x · Cited by 2,167
Usability Analysis of Visual Programming Environments: A ‘Cognitive Dimensions’ Framework
Green, T. R. G., & Petre, M. (1996) · Journal of Visual Languages & Computing · doi:10.1006/jvlc.1996.0009 · Cited by 1,251
Social coding in GitHub: transparency and collaboration in an open software repository
Dabbish, L., Stuart, C., Tsay, J., & Herbsleb, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145396 · Cited by 999 · cited 1×
“, which project to depend on [56], which developer to follow [11, 33], how to test a project [42], which project to contribute to [6, 44, 43].”
BackgroundSix Learning Barriers in End-User Programming Systems
Ko, A. J., Myers, B. A., & Aung, H. H. (2004) · 2004 IEEE Symposium on Visual Languages - Human Centric Computing · doi:10.1109/vlhcc.2004.47 · Cited by 549 · cited 2×
“Previous studies have shown that programmers at all levels, from novices to experts, often find it difficult to learn new APIs [27, 40, 47, 15, 49].”
BackgroundWhy eve and mallory love android: an analysis of android SSL (in)security
Fahl, S., Harbach, M., Muders, T., Baumgärtner, L., Freisleben, B., & Smith, M. (2012) · Proceedings of the 2012 ACM conference on Computer and communications security · doi:10.1145/2382196.2382205 · Cited by 490 · cited 1×
“Making mistakes in API usage could lead to severe program failures such as runtime exceptions [60] and security vulnerabilities [21, 18, 16].”
BackgroundGu, X., Zhang, H., Zhang, D., & Kim, S. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950334 · Cited by 485 · cited 1×
“Large-scale community data has driven big success stories in domains such as bug detection [24, 60] and code completion [25, 45, 3], but remains under-utilized in the context of human-centered API design and usability evaluation.”
BackgroundAn empirical study of cryptographic misuse in android applications
Egele, M., Brumley, D., Fratantonio, Y., & Kruegel, C. (2013) · Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security - CCS '13 · doi:10.1145/2508859.2516693 · Cited by 435 · cited 1×
“Making mistakes in API usage could lead to severe program failures such as runtime exceptions [60] and security vulnerabilities [21, 18, 16].”
BackgroundThe most dangerous code in the world: validating SSL certificates in non-browser software
Georgiev, M., Iyengar, S., Jana, S., Anubhai, R., Boneh, D., & Shmatikov, V. (2012) · Proceedings of the 2012 ACM conference on Computer and communications security · doi:10.1145/2382196.2382204 · Cited by 413 · cited 1×
“Making mistakes in API usage could lead to severe program failures such as runtime exceptions [60] and security vulnerabilities [21, 18, 16].”
Background
Research works printed but not linked: 7
- Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2016. Deepcoder: Learning to write programs. arXiv preprint arXiv:1611.01989 (2016). · arXiv
- Steven Clarke. 2005. Describing and measuring API usability with the cognitive dimensions. In Cognitive Dimensions of Notations 10th Anniversary Workshop. Citeseer, 131.
- 10.5555/2337223.2337255 · doi:10.5555/2337223.2337255
- Robyn Longhurst. 2003. Semi-structured interviews and focus groups. Key methods in geography 3 (2003), 143--156.
- 10.1109/WAPI.2017.2 · DOI
- 10.5555/2486788.2486804 · doi:10.5555/2486788.2486804
- 10.5555/2487085.2487146 · doi:10.5555/2487085.2487146
Cited by
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality
Schlagowski, R., Nazarenko, D., Can, Y. S., Gupta, K., Mertes, S., Billinghurst, M., & André, E. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581162 · Cited by 51
“[23, 48, 104]) and allows researchers to code and analyze freeform text such as interview transcripts or, in our case, open feedback that users provided at the end of the experiment.”
BackgroundPräzi: from package-based to call-based dependency networks
Hejderup, J., Beller, M., Τriantafyllou, Κ., & Gousios, G. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10071-9 · Cited by 27
“Following Zhang et al. (2020a)’s need-finding study on data-driven API design, we are investigating possibilities to mine program contexts and error-inducing patterns using Pr¨azi to extract API usage patterns beyond syntactic features and frequencies.”
BackgroundMethodHow statically-typed functional programmers write code
Lubin, J., & Chasins, S. E. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485532 · Cited by 18
“The grounded theory approach to data analysis has become increasingly popular in the software engineering community (Adolph et al., 2012, Coleman and O’Connor, 2007, Hoda et al., 2013, Jantunen and Gause, 2014, Pang et al., 2020, Zhang et al., 2020).”
MethodConcept-Annotated Examples for Library Comparison
Yan, L., Kim, M., Hartmann, B., Zhang, T., & Glassman, E. L. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545647 · Cited by 20
Self-citation
Metadata
- Authors
- Zhang, T., Hartmann, B., Kim, M., & Glassman, E. L.
- Affiliations
- Harvard University (US)
- University of California, Berkeley (US)
- University of California, Los Angeles (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3028676906
- Cited by
- 25
- References
- 59
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 6 · 2023: 2 · 2022: 5 · 2021: 9 · 2020: 1
- FWCI
- 3.22
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- National Science Foundation (CCF-1764077, CCF-1527923, CCF-1723773)
- Office of Naval Research (N00014-18-1-2037)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- API design, API usability, user feedback, mental models, bug reports, peer review, usage mining, open source software, corner cases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Enabling Data-Driven API Design with Community Usage Data: A Need-Finding Study. https://app.cochat.ai/papers/10.1145/3313831.3376382
- The paper
- Zhang, T., Hartmann, B., Kim, M., & Glassman, E. L. (2020). Enabling Data-Driven API Design with Community Usage Data: A Need-Finding Study. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376382
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.