Understanding How Programmers Can Use Annotations on Documentation
Open accessAbstract
Modern software development requires developers to find and effectively utilize new APIs and their documentation, but documentation has many well-known issues. Despite this, developers eventually overcome these issues but have no way of sharing what they learned. We investigate sharing this documentation-specific information through annotations, which have advantages over developer forums as the information is contextualized, not disruptive, and is short, thus easy to author. Developers can also author annotations to support their own comprehension. In order to support the documentation usage behaviors we found, we built the Adamite annotation tool, which provides features such as multiple anchors, annotation types, and pinning. In our user study, we found that developers are able to create annotations that are useful to themselves and are able to utilize annotations created by other developers when learning a new API, with readers of the annotations completing 67% more of the task, on average, than the baseline.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The VEGA and VEGA-LITE editor [59] includes specialized affordances for those libraries, such as debugging [20] and performance [64] tools.
“For instance, the VEGA and VEGA-LITE editor [59] includes specialized affordances for those libraries, such as debugging [20] and performance [64] tools.”
From Considering Visualization Example Galleries (2024) · cites this paper as [20]
The Future Work section discusses a proposed IDE extension for Adamite and mentions debugging only in connection with developers’ notes. Searches for VEGA, VEGA-LITE, and debugging tools found no passage about specialized tools in the VEGA or VEGA-LITE editor; our copy therefore does not address the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cognitive Load Theory and the Format of Instruction
Chandler, P., & Sweller, J. (1991) · Cognition and Instruction · doi:10.1207/s1532690xci0804_2 · Cited by 2,894 · see citing works
Self‐Explanations: How Students Study and Use Examples in Learning to Solve Problems
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989) · Cognitive Science · doi:10.1207/s15516709cog1302_1 · Cited by 1,714 · see citing works
(2009) · doi:10.1145/1518701.1518944 · Cited by 447 · see citing works
A field study of API learning obstacles
Robillard, M., & DeLine, R. (2011) · Empirical Software Engineering · doi:10.1007/s10664-010-9150-8 · Cited by 415 · see citing works
What Makes APIs Hard to Learn? Answers from Developers
Robillard, M. P. (2009) · IEEE Software · doi:10.1109/ms.2009.193 · Cited by 385 · see citing works
Quality Control in Crowdsourcing Systems: Issues and Directions
Allahbakhsh, M., Benatallah, B., Ignjatović, A., Motahari‐Nezhad, H. R., Bertino, E., & Dustdar, S. (2013) · IEEE Internet Computing · doi:10.1109/mic.2013.20 · Cited by 384 · see citing works
Effective Discussion Through a Computer-Mediated Anchored Forum
Guzdial, M., & Turns, J. (2000) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0904_3 · Cited by 362 · see citing works
How software engineers use documentation: the state of the practice
Lethbridge, T. C., Singer, J., & Forward, A. (2003) · IEEE Software · doi:10.1109/ms.2003.1241364 · Cited by 340 · see citing works
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2734091 · Cited by 326 · see citing works
Asking and Answering Questions during a Programming Change Task
Sillito, J., Murphy, G. C., & Volder, K. D. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.26 · Cited by 320 · see citing works
GenderMag: A Method for Evaluating Software's Gender Inclusiveness
Burnett, M., Stumpf, S., Macbeth, J., Makri, S., Beckwith, L., Kwan, I., Peters, A. N., & Jernigan, W. (2016) · Interacting with Computers · doi:10.1093/iwc/iwv046 · Cited by 292 · see citing works
Myers, B. A., & Stylos, J. (2016) · Communications of the ACM · doi:10.1145/2896587 · Cited by 269 · see citing works
Self-citationAnswering questions about unanswered questions of Stack Overflow
Asaduzzaman, M., Mashiyat, A. S., Roy, C. K., & Schneider, K. A. (2013) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624015 · Cited by 222 · see citing works
Software Documentation Issues Unveiled
Aghajani, E., Nagy, C., Vega-Marquez, O. L., Linares-Vasquez, M., Moreno, L., Bavota, G., & Lanza, M. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00122 · Cited by 195 · see citing works
On the Comprehension of Program Comprehension
Maalej, W., Tiarks, R., Roehm, T., & Koschke, R. (2014) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2622669 · Cited by 180 · see citing works
Uddin, G., & Robillard, M. P. (2015) · IEEE Software · doi:10.1109/ms.2014.80 · Cited by 162 · see citing works
Chen, J.-C., & Huang, S.-J. (2009) · Journal of Systems and Software · doi:10.1016/j.jss.2008.12.036 · Cited by 162 · see citing works
Exploring the relationship between personal and public annotations
Marshall, C., & Brush, A. J. B. (2004) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/996350.996432 · Cited by 152 · see citing works
Rethinking SSL development in an appified world
Fahl, S., Harbach, M., Perl, H., Koetter, M., & Smith, M. (2013) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2508859.2516655 · Cited by 151 · see citing works
Easing Program Comprehension by Sharing Navigation Data
DeLine, R. A., Czerwinski, M. P., & Robertson, G. G. (2005) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2005.32 · Cited by 125 · see citing works
Successful classroom deployment of a social document annotation system
Zyto, S., Karger, D., Ackerman, M., & Mahajan, S. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208326 · Cited by 90 · see citing works
How do API documentation and static typing affect API usability?
Endrikat, S., Hanenberg, S., Robbes, R., & Stefik, A. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568299 · Cited by 86 · see citing works
LemonAid: selection-based crowdsourced contextual help for web applications
Chilana, P. K., Ko, A. J., & Wobbrock, J. O. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208620 · Cited by 79 · see citing works
Asking and answering questions about unfamiliar APIs: An exploratory study
Duala-Ekoko, E., & Robillard, M. P. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227187 · Cited by 77 · see citing works
Unakite: Scaffolding Developers' Decision-Making Using the Web
Liu, M. X., Hsieh, J., Hahn, N., Zhou, A., Deng, E., Burley, S., Taylor, C., Kittur, A., & Myers, B. (2019) · ACM Symposium on User Interface Software and Technology · doi:10.1145/3332165.3347908 · Cited by 77 · see citing works
Self-citationNykaza, J., Messinger, R., Boehme, F., et al. (2002) · ACM International Conference on Computer Documentation (SIGDOC) · doi:10.1145/584955.584976 · Cited by 61 · see citing works
Evaluating Cues for Resuming Interrupted Programming Tasks
Parnin, C., & DeLine, R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753342 · Cited by 60 · cited 4× · see citing works
“To-do Annotation Developers take notes on open tasks that they must work on, especially when interrupted [46], and occasionally take notes on tasks they must complete that are related to parts of the documentation they are reading (Section 4)”
BackgroundMethodInfluentialCodepourri: Creating visual coding tutorials using a volunteer crowd of learners
Gordon, M., & Guo, P. J. (2015) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357193 · Cited by 60 · see citing works
Individual differences in program comprehension strategies in unfamiliar programming systems
Ko, A. J., & Uttl, B. (2003) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/wpc.2003.1199201 · Cited by 46 · see citing works
A historical and contemporary study on annotations to derive key features for systems design
Agosti, M., Bonfiglio-Dosio, G., & Ferro, N. (2007) · International Journal on Digital Libraries · doi:10.1007/s00799-007-0010-0 · Cited by 42 · see citing works
Costs and benefits of structured information foraging
Kittur, A., Peters, A. M., Diriye, A., Telang, T., & Bove, M. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481415 · Cited by 40 · see citing works
From work to word: How do software developers describe their work?
Maalej, W., & Happel, H.-J. (2009) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2009.5069490 · Cited by 38 · see citing works
How developers use API documentation
Meng, M., Steinhardt, S. B., & Schubert, A. (2019) · Communication Design Quarterly · doi:10.1145/3358931.3358937 · Cited by 30 · see citing works
Creating Guided Code Explanations with chat.codes
Oney, S., Brooks, C. A., & Resnick, P. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274400 · Cited by 25 · see citing works
Vermette, L., Dembla, S., Wang, A. Y., McGrenere, J., & Chilana, P. K. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134737 · Cited by 24 · see citing works
Reading the documentation of invoked API functions in program comprehension
Dekel, U., & Herbsleb, J. D. (2009) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2009.5090040 · Cited by 24 · see citing works
To Reuse or Not To Reuse?: A Framework and System for Evaluating Summarized Knowledge
Liu, M. X., Kittur, A., & Myers, B. A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449240 · Cited by 23 · see citing works
Self-citationOn electronic annotation and its implementation
Fogli, D., Fresta, G., & Mussio, P. (2004) · Working Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/989863.989877 · Cited by 19 · see citing works
Methods for Investigating Mental Models for Learners of APIs
Horvath, A., Nagy, M., Voichick, F., Kery, M. B., & Myers, B. A. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3312897 · Cited by 10 · see citing works
Self-citationThe Long Tail: Understanding the Discoverability of API Functionality
Horvath, A., Myers, B., Grover, S., Dong, S., Zhou, E., Voichick, F., Kery, M. B., Shinju, S., Nam, D., & Nagy, M. (2019) · 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2019.8818681 · Cited by 9 · see citing works
Self-citationRequirements of API Documentation: A Case Study into Computer Vision Services
Cummaudo, A., Vasa, R., Grundy, J., & Abdelrazek, M. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3047088 · Cited by 9 · see citing works
NoteCoStruct: Powering Online Learners with Socially Scaffolded Note Taking and Sharing
Fang, J., Wang, Y., Yang, C.-L., et al. (2021) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451694 · Cited by 8 · see citing works
Chatterjee, P., Nishi, M. A., Damevski, K., Augustine, V., Pollock, L., & Kraft, N. A. (2017) · 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2017.7884638
Hong, L., & Chi, E. H. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518976
Flagged first, then by citations
Research works printed but not linked: 6
- C.R. Cook, J.C. Scholtz, and J.C. Spohrer. 1993. Empirical Studies of Programmers: Fifth Workshop : Papers Presented at the Fifth Workshop on Empirical Studies of Programmers, December 3-5, 1993, Palo Alto, CA. Ablex Publishing Corporation, Norwood, NJ, USA. https://books.google.com/books?id=rMmxq8q0CGYC
- Adam Fourney and Meredith Ringel Morris. 2013. Enhancing Technical Q&A Forums with CiteHistory. In ICWSM 2013. AAAI, Palo Alto, CA, USA, 1-10.
- Thomas D. LaToza, David Garlan, James D. Herbsleb, and Brad A. Myers. 2007. Program comprehension as fact fnding. In ESEC-FSE 2007. ACM, New York, NY, USA, 361-270.
- Seyed Mehdi Nasehi, Jonathan Sillito, Frank Maurer, and Chris Burns. 2012. What makes a good code example?: A study of programming Q&A in StackOverfow. In ICSM 2012. IEEE, New York, NY, USA, 25-34.
- Tobias Roehm, Rebecca Tiarks, Rainer Koschke, and Walid Maalej. 2012. How do API documentation and static typing afect API usability?. In ICSE 2012. ACM, New York, NY, USA, 632-542. https://doi.org/10.1109/ICSE.2012.6227188 · DOI
- Tianyi Zhang, Ganesha Upadhyaya, Anastasia Reinhardt, Hridesh Rajan, and Miryung Kim. 2018. Are Code Examples on an Online Q&A Forum Reliable? A Study of API Misuse on Stack Overfow. In ICSE 2018. ACM, New York, NY, USA, 886-896.
Cited by
Sorted by citations
Considering Visualization Example Galleries
Yang, J., McNutt, A. M., & Battle, L. (2024) · 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00043 · Cited by 5 · see citing works
“Studies have explored how documentation might be enhanced by supporting sustainable maintenance [22], annotation [21], automated creation [10], [17], [29], enhancements to the development environment [35], [47], among others.”
Background
- Authors
- Horvath, A., Liu, M. X., Hendriksen, R., Shannon, C., Paterson, E., Jawad, K., Macvean, A., & Myers, B. A.
- Affiliations
- Carnegie Mellon University (US)
- University of Pittsburgh (US)
- Tufts University (US)
- Google (United States) (US)
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-27
- Pages
- 1-16
- OpenAlex
- W3212201636
- Cited by
- 26
- References
- 45
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 8 · 2023: 5 · 2022: 5
- FWCI
- 5.37
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (CCF-2007482)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- API documentation, annotation, developer learning, knowledge sharing, developer forums, multiple anchors, annotation types
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
