A Theory of Robust API Knowledge
The DOI matches its Crossref record; 49 of the 50 references it cites were found in a registry or an index.
Abstract
Creating modern software inevitably requires using application programming interfaces (APIs). While software developers can sometimes use APIs by simply copying and pasting code examples, a lack of robust knowledge of how an API works can lead to defects, complicate software maintenance, and limit what someone can express with an API. Prior work has uncovered the many ways that API documentation fails to be helpful, though rarely describes precisely why. We present a theory of robust API knowledge that attempts to explain why, arguing that effective understanding and use of APIs depends on three components of knowledge: (1) the domain concepts the API models along with terminology, (2) the usage patterns of APIs along with rationale, and (3) facts about an API’s execution to support reasoning about its runtime behavior. We derive five hypotheses from this theory and present a study to test them. Our study investigated the effect of having access to these components of knowledge, finding that while learners requested these three components of knowledge when they were not available, whether the knowledge helped the learner use or understand the API depended on the tasks and likely the relevance and quality of the specific information provided. The theory and our evidence in support of its claims have implications for what content API documentation, tutorials, and instruction should contain and the importance of giving the right information at the right time, as well as what information API tools should compute, and even how APIs should be designed. Future work is necessary to both further test and refine the theory, as well as exploit its ideas for better instructional design.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Empirical Studies of Programming Knowledge
Soloway, E., & Ehrlich, K. (1984) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1984.5010283 · Cited by 877
Program comprehension during software maintenance and evolution
Von Mayrhauser, A., & Vans, A. M. (1995) · Computer · doi:10.1109/2.402076 · Cited by 575
Six Learning Barriers in End-User Programming Systems
Ko, A. J., Myers, B. A., & Aung, H. H. (2004) · 2004 IEEE Symposium on Visual Languages - Human Centric Computing · doi:10.1109/vlhcc.2004.47 · Cited by 549
Self-citationInformation Needs in Collocated Software Development Teams
Ko, A. J., DeLine, R., & Venolia, G. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.45 · Cited by 499 · cited 7×
“Execution facts elucidate the details of the API structure [28], how to use API features [12, 21, 30], how to understand dependencies and relations between API features [12, 13, 30], and details of how code relates to output [21, 22].”
BackgroundInfluentialSelf-citation(2009) · doi:10.1145/1518701.1518944 · Cited by 447
What Makes APIs Hard to Learn? Answers from Developers
Robillard, M. P. (2009) · IEEE Software · doi:10.1109/ms.2009.193 · Cited by 382
0897915046/International Conference on Software Engineering
(1992) · doi:10.1145/143062 · Cited by 374
A field study of API learning obstacles
Robillard, M. P., & DeLine, R. (2011) · Empirical Software Engineering · doi:10.1007/s10664-010-9150-8 · Cited by 352
Paradigm Shifts in Designed Instruction: From Behaviorism to Cognitivism to Constructivism.
Cooper, P. A. (1993) · Educational Technology archive · Cited by 307
Graph-based mining of multiple object usage patterns
Nguyen, T. T., Nguyen, H. A., Pham, N. H., Al-Kofahi, J. M., & Nguyen, T. N. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595767 · Cited by 292
What makes a good code example?: A study of programming Q&A in StackOverflow
Nasehi, S. M., Sillito, J., Maurer, F., & Burns, C. (2012) · 2012 28th IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2012.6405249 · Cited by 287
How developers search for code: a case study
Sadowski, C., Stolee, K. T., & Elbaum, S. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786855 · Cited by 238
Mica: A Web-Search Tool for Finding API Components and Examples
Stylos, J., & Myers, B. A. (2006) · Visual Languages and Human-Centric Computing (VL/HCC'06) · doi:10.1109/vlhcc.2006.32 · Cited by 224
A Unified Framework to Quantify the Credibility of Scientific Findings
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918787489 · Cited by 214
Automated API Property Inference Techniques
Robillard, M. P., Bodden, E., Kawrykow, D., Mezini, M., & Ratchford, T. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.63 · Cited by 199
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Self-citationStruggles of new college graduates in their first software development job
Begel, A., & Simon, B. (2008) · Proceedings of the 39th SIGCSE technical symposium on Computer science education · doi:10.1145/1352135.1352218 · Cited by 193
What do developers search for on the web?
Xia, X., Bao, L., Lo, D., Kochhar, P. S., Hassan, A. E., & Xing, Z. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9514-4 · Cited by 174
Uddin, G., & Robillard, M. P. (2015) · IEEE Software · doi:10.1109/ms.2014.80 · Cited by 158
Why, when, and what: Analyzing Stack Overflow questions by topic, type, and code
Allamanis, M., & Sutton, C. A. (2013) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624004 · Cited by 148
Improving API documentation usability with knowledge pushing
Dekel, U., & Herbsleb, J. D. (2009) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2009.5070532 · Cited by 141
On-demand Developer Documentation
Robillard, M. P., Marcus, A., Treude, C., Bavota, G., Chaparro, O., Ernst, N., Gerosa, M. A., Godfrey, M. W., Lanza, M., Linares‐Vásquez, M., et al. (2017) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2017.17 · Cited by 140
Measuring API documentation on the web
Parnin, C., & Treude, C. (2011) · Proceedings of the 2nd International Workshop on Web 2.0 for Software Engineering · doi:10.1145/1984701.1984706 · Cited by 121
A Model DisciplinePolitical Science and the Logic of Representations
Clarke, K. A., & Primo, D. M. (2012) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195382198.001.0001 · Cited by 115
Measuring programming experience
Feigenspan, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2012) · 2012 20th IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240511 · Cited by 106
Program comprehension as fact finding
LaToza, T. D., Garlan, D., Herbsleb, J. D., & Myers, B. A. (2007) · Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1287624.1287675 · Cited by 105
Automatic parameter recommendation for practical API usage
Zhang, C., Yang, J., Zhang, Y., Fan, J., Zhang, X., Zhao, J., & Ou, P. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227136 · Cited by 105
Investigating reading techniques for object-oriented framework learning
Basili, V. R., Lanubile, F., & Shull, F. J. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.881720 · Cited by 92
Understanding contexts: how explanatory theories can help
Davidoff, F. (2019) · Implementation Science · doi:10.1186/s13012-019-0872-8 · Cited by 85
Application Programming Interface Documentation: What Do Software Developers Want?
Meng, M., Steinhardt, S., & Schubert, A. (2018) · Journal of Technical Writing and Communication · doi:10.1177/0047281617721853 · Cited by 78
Sturm, C., Oh, A. H., Linxen, S., Abdelnour-Nocera, J., Dray, S. M., & Reinecke, K. (2015) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2702613.2702656 · Cited by 67
Visualizing API Usage Examples at Scale
Glassman, E. L., Zhang, T., Hartmann, B., & Kim, M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174154 · Cited by 65
Proceedings. 2005 IEEE Symposium on Visual Languages and Human-Centric Computing
(2005) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2005.2 · Cited by 64
Structures of Scientific Theories
Craver, C. F. (2002) · Wiley eBooks · doi:10.1002/9780470756614.ch4 · Cited by 62
Barriers Faced by Coding Bootcamp Students
Thayer, K., & Ko, A. J. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3106176 · Cited by 62
Self-citationWhat programmers really want: results of a needs assessment for SDK documentation
Nykaza, J., Messinger, R., Boehme, F., Norman, C. L., Mace, M., & Gordon, M. (2002) · ACM International Conference on Systems Documentation (SIGDOC) · doi:10.1145/584955.584976 · Cited by 60
Struggles of new college graduates in their first software development job
Begel, A., & Simon, B. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1352322.1352218 · Cited by 60
Revealing actual documentation usage in software maintenance through war stories
Lutters, W. G., & Seaman, C. B. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2007.02.013 · Cited by 59
Aldrich, J., SPLASH '15: Conference on Systems, P., Association for Computing Machinery-Digital Library, ACM Special Interest Group on Programming Languages, & SIGs, A. (2015) · doi:10.1145/2814270 · Cited by 46
When not to comment: questions and tradeoffs with API documentation for C++ projects
Head, A., Sadowski, C., Murphy-Hill, E. R., & Knight, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180176 · Cited by 31
The role of conceptual knowledge in API usability
Ko, A. J., & Riche, Y. (2011) · 2011 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2011.6070395 · Cited by 30
Self-citationTowards a Model of API Learning
Kelleher, C. L., & Ichinco, M. (2019) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2019.8818850 · Cited by 22
What is an End User Software Engineer?
Clarke, S. (2007) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/dagsemproc.07081.26 · Cited by 19
Discovering API usability problems at scale
Murphy-Hill, E., Sadowski, C., Head, A., Daughtry, J., Macvean, A., Jaspan, C., & Winter, C. (2018) · Proceedings of the 2nd International Workshop on API Usage and Evolution · doi:10.1145/3194793.3194795 · Cited by 15
An Exploratory Study of Web Foraging to Understand and Support Programming Decisions
Hsieh, J., Liu, M. X., Myers, B. A., & Kittur, A. (2018) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2018.8506517 · Cited by 15
App-directed learning: An exploratory study
Sillito, J., & Begel, A. (2013) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2013.6614736 · Cited by 10
Docio: Documenting API Input/Output Examples
Jiang, S., Armaly, A., McMillan, C., Zhi, Q., & Metoyer, R. (2017) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2017.13 · Cited by 8
Openness to the Unknown: The Role of Falsifiability in Search of Better Knowledge
Kageyama, Y. (2003) · Philosophy of the Social Sciences · doi:10.1177/0048393102250307 · Cited by 5
Research works printed but not linked: 6
- Steven Clarke . 2004. Measuring API usability. Dr. Dobb’s Journal , Special Windows/.NET Supplement (May 2004 ). Retrieved from http://www.drdobbs.com/windows/measuring-api-usability/184405654. Steven Clarke. 2004. Measuring API usability. Dr. Dobb’s Journal, Special Windows/.NET Supplement (May 2004). Retrieved from http://www.drdobbs.com/windows/measuring-api-usability/184405654.
- Ekwa Duala-Ekoko and Martin P . Robillard . 2010 . The Information Gathering Strategies of API Learners. Technical ReportTR-2010.6, School of Computer Science, McGill University . Retrieved from https://pdfs.semanticscholar.org/a7ff/4cc954744f761e8697be4e73aa25166a76c4.pdf. Ekwa Duala-Ekoko and Martin P. Robillard. 2010. The Information Gathering Strategies of API Learners. Technical ReportTR-2010.6, School of Computer Science, McGill University. Retrieved from https://pdfs.semanticscholar.org/a7ff/4cc954744f761e8697be4e73aa25166a76c4.pdf.
- Patrick Suppes . 1967 . What is a scientific theory? In Philosophy of Science Today, Sidney Morgenbesser (Ed.) , Chapter 6. 55 -- 67 . Patrick Suppes. 1967. What is a scientific theory? In Philosophy of Science Today, Sidney Morgenbesser (Ed.), Chapter 6. 55--67.
- Kyle Matthew Thayer. 2020. Practical Knowledge Barriers in Professional Programming. Thesis. Retrieved from https://digital.lib.washington.edu:443/researchworks/handle/1773/45471. Kyle Matthew Thayer. 2020. Practical Knowledge Barriers in Professional Programming. Thesis. Retrieved from https://digital.lib.washington.edu:443/researchworks/handle/1773/45471.
- 10.5555/2337223.2337321 · doi:10.5555/2337223.2337321
- Leonid_2. 2015 . English: A simple torus fading out to a wireframe structure. Leonid_2. 2015. English: A simple torus fading out to a wireframe structure.
Cited by
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Concept-Annotated Examples for Library Comparison
Yan, L., Kim, M., Hartmann, B., Zhang, T., & Glassman, E. L. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545647 · Cited by 20
Identifying Common Errors in Open-Ended Machine Learning Projects
Skripchuk, J., Shi, Y., & Price, T. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499397 · Cited by 20
Usability-Oriented Design of Liquid Types for Java
Gamboa, C., Canelas, P., Timperley, C. S., & Fonseca, A. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00132 · Cited by 19
How statically-typed functional programmers write code
Lubin, J., & Chasins, S. E. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485532 · Cited by 18
“For example, Ko and Myers (2003) develop and evaluate a theory of programming errors, and Thayer et al. (2021) do the same for a theory of API knowledge.”
BackgroundSelf-citation
Metadata
- Authors
- Thayer, K., Chasins, S. E., & Ko, A. J.
- Affiliations
- University of Washington (US)
- University of California, Berkeley (US)
- Venue
- ACM Transactions on Computing Education
- Published
- 2021-01-28
- Volume
- 21
- Issue
- 1
- Pages
- 1-32
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W3128009513
- Cited by
- 25
- References
- 50
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 6 · 2023: 7 · 2022: 6 · 2021: 3
- FWCI
- 4.97
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computing Education (publishedVersion)
- Funders
- National Science Foundation (1703304, 1735123, and 1836813.)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- application programming interfaces, API documentation, software learning, domain concepts, API usage patterns, API design, instructional design, software maintenance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.