A Theory of Robust API Knowledge
The DOI matches its Crossref record; 49 of the 50 references it cites were found in a registry or an index.
Abstract
Creating modern software inevitably requires using application programming interfaces (APIs). While software developers can sometimes use APIs by simply copying and pasting code examples, a lack of robust knowledge of how an API works can lead to defects, complicate software maintenance, and limit what someone can express with an API. Prior work has uncovered the many ways that API documentation fails to be helpful, though rarely describes precisely why. We present a theory of robust API knowledge that attempts to explain why, arguing that effective understanding and use of APIs depends on three components of knowledge: (1) the domain concepts the API models along with terminology, (2) the usage patterns of APIs along with rationale, and (3) facts about an API’s execution to support reasoning about its runtime behavior. We derive five hypotheses from this theory and present a study to test them. Our study investigated the effect of having access to these components of knowledge, finding that while learners requested these three components of knowledge when they were not available, whether the knowledge helped the learner use or understand the API depended on the tasks and likely the relevance and quality of the specific information provided. The theory and our evidence in support of its claims have implications for what content API documentation, tutorials, and instruction should contain and the importance of giving the right information at the right time, as well as what information API tools should compute, and even how APIs should be designed. Future work is necessary to both further test and refine the theory, as well as exploit its ideas for better instructional design.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Empirical Studies of Programming Knowledge
Soloway, E., & Ehrlich, K. (1984) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1984.5010283 · Cited by 877 · cited 2×
“API usage patterns demonstrate what is possible with an API [28], help developers understand how API code works together and how to coordinate API use [21, 22, 36], and encode common usage patterns and best practices [38, 43, 52].”
BackgroundProgram comprehension during software maintenance and evolution
Von Mayrhauser, A., & Vans, A. M. (1995) · Computer · doi:10.1109/2.402076 · Cited by 575 · cited 2×
“Our theory incorporates many previous findings: Domain concepts help developers seek and recognize relevant information [13, 21, 24, 52], understand the purpose of code [22, 36], and decide what purpose they want new code to fulfill [21, 50].”
BackgroundSix Learning Barriers in End-User Programming Systems
Ko, A. J., Myers, B. A., & Aung, H. H. (2004) · 2004 IEEE Symposium on Visual Languages - Human Centric Computing · doi:10.1109/vlhcc.2004.47 · Cited by 549 · cited 8×
“Our theory incorporates many previous findings: Domain concepts help developers seek and recognize relevant information [13, 21, 24, 52], understand the purpose of code [22, 36], and decide what purpose they want new code to fulfill [21, 50].”
BackgroundInfluentialSelf-citationInformation Needs in Collocated Software Development Teams
Ko, A. J., DeLine, R., & Venolia, G. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.45 · Cited by 499 · cited 7×
“Execution facts elucidate the details of the API structure [28], how to use API features [12, 21, 30], how to understand dependencies and relations between API features [12, 13, 30], and details of how code relates to output [21, 22].”
BackgroundInfluentialSelf-citation(2009) · doi:10.1145/1518701.1518944 · Cited by 447 · cited 1×
“When an example exactly matches their needs, this method can be effective at helping developers leverage an API [4].”
MethodWhat Makes APIs Hard to Learn? Answers from Developers
Robillard, M. P. (2009) · IEEE Software · doi:10.1109/ms.2009.193 · Cited by 382
0897915046/International Conference on Software Engineering
(1992) · doi:10.1145/143062 · Cited by 374
A field study of API learning obstacles
Robillard, M. P., & DeLine, R. (2011) · Empirical Software Engineering · doi:10.1007/s10664-010-9150-8 · Cited by 352 · cited 9×
“However, evidence also shows that it is essential for API usage patterns to explain usage rules and their rationale [13] and why the API was designed in the way it was [22, 28, 31, 38].”
BackgroundInfluentialParadigm Shifts in Designed Instruction: From Behaviorism to Cognitivism to Constructivism.
Cooper, P. A. (1993) · Educational Technology archive · Cited by 307
Graph-based mining of multiple object usage patterns
Nguyen, T. T., Nguyen, H. A., Pham, N. H., Al-Kofahi, J. M., & Nguyen, T. N. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595767 · Cited by 292 · cited 1×
“information like control structures [32].”
Research works printed but not linked: 6
- Steven Clarke . 2004. Measuring API usability. Dr. Dobb’s Journal , Special Windows/.NET Supplement (May 2004 ). Retrieved from http://www.drdobbs.com/windows/measuring-api-usability/184405654. Steven Clarke. 2004. Measuring API usability. Dr. Dobb’s Journal, Special Windows/.NET Supplement (May 2004). Retrieved from http://www.drdobbs.com/windows/measuring-api-usability/184405654.
- Ekwa Duala-Ekoko and Martin P . Robillard . 2010 . The Information Gathering Strategies of API Learners. Technical ReportTR-2010.6, School of Computer Science, McGill University . Retrieved from https://pdfs.semanticscholar.org/a7ff/4cc954744f761e8697be4e73aa25166a76c4.pdf. Ekwa Duala-Ekoko and Martin P. Robillard. 2010. The Information Gathering Strategies of API Learners. Technical ReportTR-2010.6, School of Computer Science, McGill University. Retrieved from https://pdfs.semanticscholar.org/a7ff/4cc954744f761e8697be4e73aa25166a76c4.pdf.
- Patrick Suppes . 1967 . What is a scientific theory? In Philosophy of Science Today, Sidney Morgenbesser (Ed.) , Chapter 6. 55 -- 67 . Patrick Suppes. 1967. What is a scientific theory? In Philosophy of Science Today, Sidney Morgenbesser (Ed.), Chapter 6. 55--67.
- Kyle Matthew Thayer. 2020. Practical Knowledge Barriers in Professional Programming. Thesis. Retrieved from https://digital.lib.washington.edu:443/researchworks/handle/1773/45471. Kyle Matthew Thayer. 2020. Practical Knowledge Barriers in Professional Programming. Thesis. Retrieved from https://digital.lib.washington.edu:443/researchworks/handle/1773/45471.
- 10.5555/2337223.2337321 · doi:10.5555/2337223.2337321
- Leonid_2. 2015 . English: A simple torus fading out to a wireframe structure. Leonid_2. 2015. English: A simple torus fading out to a wireframe structure.
Cited by
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Concept-Annotated Examples for Library Comparison
Yan, L., Kim, M., Hartmann, B., Zhang, T., & Glassman, E. L. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545647 · Cited by 20
Identifying Common Errors in Open-Ended Machine Learning Projects
Skripchuk, J., Shi, Y., & Price, T. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499397 · Cited by 20
“Learning to use APIs is a difficult but teachable skill that is rarely taught explicitly in CS classrooms [10, 22].”
BackgroundUsability-Oriented Design of Liquid Types for Java
Gamboa, C., Canelas, P., Timperley, C. S., & Fonseca, A. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00132 · Cited by 19
“In previous research, authors have employed user studies to understand how programmers write code [29, 30] and compare different programming language designs.”
How statically-typed functional programmers write code
Lubin, J., & Chasins, S. E. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485532 · Cited by 18
“For example, Ko and Myers (2003) develop and evaluate a theory of programming errors, and Thayer et al. (2021) do the same for a theory of API knowledge.”
BackgroundSelf-citationAn Exploratory Study of Sharing Strategic Programming Knowledge
Arab, M., LaToza, T. D., Liang, J. T., & Ko, A. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502070 · Cited by 16
“Programming languages can be difficult to learn [33]: APIs require immense domain knowledge to master [49], tools often pose steep learning curves [32], and developers regularly encounter novel problems which they must understand and solve creatively.”
BackgroundSelf-citationDeveloping Behavioral Concepts of Higher-Order Functions
Krishnamurthi, S., & Fisler, K. (2021) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3446871.3469739 · Cited by 13
“Thayer et al.’s theory of robust api knowledge [25] synthesizes many prior studies in discussing the importance of understanding api usage patterns, which include input/output examples.”
BackgroundBarriers in Front-End Web Development
Samudio, D. I., & LaToza, T. D. (2022) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc53370.2022.9833127 · Cited by 12
“Idioms have a direct connection with robust API knowledge [16]: their semantic roles act as rationale which recur among APIs, and their code fragments act as code patterns.”
BackgroundExploring Design Choices to Support Novices' Example Use During Creative Open-Ended Programming
Wang, W., Meur, A. L., Bobbadi, M., Akram, B., Barnes, T. M., Martens, C., & Price, T. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499374 · Cited by 12
“While the Snap! website [20] offers galleries of complete projects for students to browse, prior work has shown that novices [14] and experienced programmers [25] preferred using “snippet-sized” examples that teach an API usage pattern – how code can be organized to produce a certain behavior[26, 28].”
BackgroundImproving API Knowledge Discovery with ML: A Case Study of Comparable API Methods
Nam, D., Myers, B. A., Vasilescu, B., & Hellendoorn, V. J. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00161 · Cited by 12
“Learning a new API requires many types of knowledge [2] and there are many documented challenges to obtaining these [3].”
BackgroundPoitras, E., Crane, B., & Siegel, A. (2024) · Proceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 1 · doi:10.1145/3649165.3690111 · Cited by 9
“Prior research has shown that novice learners may have missing pieces of knowledge in relation to components of API documentation [11], and step-by-step usage of the AI code generator could potentially reduce cognitive load in searching for information to address gaps in understanding.”
Understanding Documentation Use Through Log Analysis: A Case Study of Four Cloud Services
Nam, D., Macvean, A., Myers, B. A., & Vasilescu, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642721 · Cited by 9
Understanding initial API comprehension
Heinonen, A., & Fagerholm, F. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00016 · Cited by 8
“Much of the existing research has sought to understand the types of information that programmers need about APIs [6], [11], [24], and the difficulties they face in acquiring this information [21].”
BackgroundMethodHow Domain Experts Use an Embedded DSL
Rennels, L., & Chasins, S. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3622851 · Cited by 7
“…may have encouraged this practice; however, existing literature on blank page syndrome and the use of templates suggests this behavior is prevalent across a variety of domains [Bloch 2006; Chasins 2019; Felleisen et al. 2018; Krishnamurthi and Nelson 2019; Nosál’ et al. 2017; Thayer et al. 2021].”
BackgroundSelf-citationConsidering Visualization Example Galleries
Yang, J., McNutt, A. M., & Battle, L. (2024) · 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00043 · Cited by 11
“Other work investigates the challenges using code examples for different user groups, including lack of API knowledge [56]; identifying relevant parts, integrating and adapting existing code [24], [62].”
BackgroundAPI usage templates via structural generalization
Mahmoud, M., Walker, R. J., & Denzinger, J. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.111974 · Cited by 5
Domain-Specific Theories of Teaching Computing: Do they Inform Practice?
Malmi, L., Sheard, J., Sinclair, J. E., Kinnunen, P., & Simon, N. P. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631810 · Cited by 5
“Thayer et al. [73] present a theory of robust API knowledge , seeking to identify core knowledge components related to APIs.”
BackgroundAnalysing the API learning process through the use of eye tracking
Sparmann, S., & Schulte, C. (2023) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3588015.3589194 · Cited by 4
“There has been a lot of research done on API learning related to strategies and obstacles involved [Duala-Ekoko and Robillard 2010, 2012; Hou and Li 2011; Robillard 2009; Robillard and DeLine 2011; Xia et al. 2017] and the role of knowledge for API learning [Ko and Riche 2011; Thayer et al. 2021].”
BackgroundMethodLiang, J. T., Barik, T., Nichols, J. W., Schoop, E., & Cheng, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.04452 · Cited by 4
How Do Computing Education Researchers Talk About Threats and Limitations?
Sanders, K., Vahrenhold, J., & McCartney, R. (2023) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3568813.3600114 · Cited by 3
“handling of missing responses to survey data [36], test assumptions not met [36], and failure to correct for multiple tests [77].”
BackgroundA Semantic Web-Enabled Approach for Dependency Management
Eghan, E. E., & Rilling, J. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500498 · Cited by 3
Harnessing Test-Oriented Knowledge Graphs for Enhanced Test Function Recommendation
Liu, K., Wu, J. T., Sun, Q., Yang, H., & Wan, R. (2024) · Electronics · doi:10.3390/electronics13081547 · Cited by 1
Generering av API-dokumentation med hjälp av stora språkmodeller – Mot självförklarande API:er
Jorelle, Y. (2024) · Aaltodoc (Aalto University) · Cited by 1
Facilitating API lookup for novices learning data wrangling using thumbnail graphics
Sundin, L., Sakr, N., Leinonen, J., & Cutts, Q. I. (2021) · Foundations of Data Science · doi:10.3934/fods.2021032
“This leads to what Thayer, et al. call robust API knowledge [39].”
BackgroundInfluentialFra informatikkstudent til utvikler
Wistner, H. J. (2021) · Duo Research Archive (University of Oslo)
Vizualizacija plač v javnem in zasebnem sektorju
Perčič, U. (2023) · Repository of the University of Ljubljana (University of Ljubljana)
Improving the accessibility of API documentation for non-technical users
Marinosci, G., & Shchadrynski, R. (2023) · DiVA (Linnaeus University)
The Economics of Climate Change Under Uncertainty and Engineering the Software for its Research
Rennels, L. (2024) · eScholarship (California Digital Library)
Empirical Analysis and Enhancement of Machine Learning Software Documentation
Thirimanne, S. P. (2024) · YorkSpace (York University)
Implementation of Domain-Driven Design to Improve Developer Understandability of Web APIs
Taufiqurrahman, A., Sabariah, M. K., Husen, J. H., & Lu, Y. (2025) · International Conference on Software Engineering and Computer Systems · doi:10.1109/icsecs65227.2025.11279024
“The concept of domain is one of the essential components that help developers understand the information in web APIs [7].”
InfluentialAutomating API Documentation from Crowdsourced Knowledge
Kou, B., Zhou, Z., Chen, M., & Zhang, T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773130
Demirtaş, M. A., Zheng, C., & Cunningham, K. I. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772574
“Thayer et al. [33] examine programming with APIs (application programming interfaces), including “libraries, frameworks, web services, and other reusable code.”
Influential
Metadata
- Authors
- Thayer, K., Chasins, S. E., & Ko, A. J.
- Affiliations
- University of Washington (US)
- University of California, Berkeley (US)
- Venue
- ACM Transactions on Computing Education
- Published
- 2021-01-28
- Volume
- 21
- Issue
- 1
- Pages
- 1-32
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W3128009513
- Cited by
- 25
- References
- 50
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 6 · 2023: 7 · 2022: 6 · 2021: 3
- FWCI
- 4.97
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computing Education (publishedVersion)
- Funders
- National Science Foundation (1703304, 1735123, and 1836813.)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- application programming interfaces, API documentation, software learning, domain concepts, API usage patterns, API design, instructional design, software maintenance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.