End-user debugging strategies: A sensemaking perspective
Abstract
Despite decades of research into how professional programmers debug, only recently has work emerged about how end-user programmers attempt to debug programs. Without this knowledge, we cannot build tools to adequately support their needs. This article reports the results of a detailed qualitative empirical study of end-user programmers' sensemaking about a spreadsheet's correctness. Using our study's data, we derived a sensemaking model for end-user debugging and categorized participants' activities and verbalizations according to this model, allowing us to investigate how participants went about debugging. Among the results are identification of the prevalence of information foraging during end-user debugging, two successful strategies for traversing the sensemaking model, potential ties to gender differences in the literature, sensemaking sequences leading to debugging progress, and sequences tied with troublesome points in the debugging process. The results also reveal new implications for the design of spreadsheet tools to support end-user programmers' sensemaking during debugging.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
IFT has proven useful for explaining and predicting developer behaviors during software maintenance, in ways beneficial to tool design.
“IFT has already proven useful for explaining and predicting developer behaviors during software maintenance, in ways beneficial to tool design [12,16,17,25,26,27,33].”
From Foraging and navigations, fundamentally: developers' predictions of value and cost (2016) · cites this paper as [16]
Cites
Social Foundations of Thought and Action
Bandura, A. (2002) · SAGE Publications Ltd eBooks · doi:10.4135/9781446221129.n6 · Cited by 9,948 · see citing works
Gender differences in risk taking: a meta-analysis
Byrnes, J. P., Miller, D. C., & Schafer, W. D. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.3.367 · Cited by 3,723 · see citing works
Talking about tactile experiences
Obrist, M., Seah, S. A., & Subramanian, S. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466220 · Cited by 1,627 · see citing works
Gender differences in risk behaviour in financial decision-making: An experimental analysis
Powell, M., & Ansic, D. (1997) · Journal of Economic Psychology · doi:10.1016/s0167-4870(97)00026-3 · Cited by 1,369 · see citing works
Pirolli, P. L. T., & Card, S. K. (1999) · Psychological Review · doi:10.1037/0033-295x.106.4.643 · Cited by 1,354 · see citing works
Gender, race, and perceived risk: The 'white male' effect
Finucane, M. L., Slovic, P., Mertz, C. K., Flynn, J., & Satterfield, T. A. (2000) · Health, Risk & Society · doi:10.1080/713670162 · Cited by 1,092 · see citing works
Lowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887 · see citing works
Top 10 list [software development]
BOEHM, B. W., & Basili, V. R. (2001) · Computer · doi:10.1109/2.962984 · Cited by 820 · see citing works
Making Sense of Sensemaking 1: Alternative Perspectives
Klein, G. A., Moon, B., & Hoffman, R. R. (2006) · IEEE Intelligent Systems · doi:10.1109/mis.2006.75 · Cited by 812 · see citing works
The cost structure of sensemaking
Russell, D. M., Stefik, M., Pirolli, P. L. T., & Card, S. K. (1993) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/169059.169209 · Cited by 738 · see citing works
A small matter of programming: perspectives on end user computing
(1994) · Choice Reviews Online · doi:10.5860/choice.31-4963 · Cited by 599 · see citing works
Gender Differences in Self-Efficacy and Attitudes toward Computers
Busch, T. (1995) · Journal of Educational Computing Research · doi:10.2190/h7e1-xmm7-gu9b-3hwr · Cited by 575 · see citing works
Six Learning Barriers in End-User Programming Systems
Ko, A. J., Myers, B. A., & Aung, H. H. (2004) · doi:10.1109/vlhcc.2004.47 · Cited by 552 · see citing works
Storytelling alice motivates middle school girls to learn computer programming
Kelleher, C. L., Pausch, R. F., & Kiesler, S. B. (2007) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1240624.1240844 · Cited by 505 · see citing works
What we know about spreadsheet errors
Panko, R. R. (1998) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.1998040102 · Cited by 464 · see citing works
Nardi, B. A. (1993) · The MIT Press eBooks · doi:10.7551/mitpress/1020.001.0001 · Cited by 431 · see citing works
Cognitive and affective responses to advertising
(1989) · Choice Reviews Online · doi:10.5860/choice.26-5727 · Cited by 426 · see citing works
Sense-Making Methodology Reader: Selected Writings of Brenda Dervin
Dervin, B. L., Foreman‐Wernet, L., & Lauterbach, E. (2003) · Cited by 408 · see citing works
Ko, A. J., & Myers, B. A. (2004) · doi:10.1145/985692.985712 · Cited by 362 · see citing works
Where should the person stop and the information search interface start?
Bates, M. J. (1990) · Information Processing & Management · doi:10.1016/0306-4573(90)90103-9 · Cited by 343 · see citing works
Estimating the Numbers of End Users and End User Programmers
Scaffidi, C., Shaw, M., & Myers, B. A. (2005) · Proceedings/Proceedings -- IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2005.34 · Cited by 342 · see citing works
Gender Differences in Advanced Mathematical Problem Solving
Gallagher, A. M., Lisi, R. D., Holst, P. C., Lisi, A. V. M., Morely, M., & Laitusis, C. C. (2000) · Journal of Experimental Child Psychology · doi:10.1006/jecp.1999.2532 · Cited by 241 · see citing works
Factorial Validity of a Computer Self-Efficacy Scale and the Impact of Computer Training
Torkzadeh, G., & Koufteros, X. A. (1994) · Educational and Psychological Measurement · doi:10.1177/0013164494054003028 · Cited by 225 · see citing works
Burnett, M., Cook, C. R., & Rothermel, G. (2004) · Communications of the ACM · doi:10.1145/1015864.1015889 · Cited by 214 · see citing works
Self-citationTinkering and gender in end-user programmers' debugging
Beckwith, L., Kissinger, C., Burnett, M., Wiedenbeck, S., Lawrance, J., Blackwell, A. F., & Cook, C. R. (2006) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124808 · Cited by 178 · see citing works
Self-citationFisher, M., & Rothermel, G. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083242 · Cited by 164 · see citing works
Mental models and software maintenance
Littman, D., Pinto, J., Letovsky, S. I., & Soloway, E. M. (1986) · Cited by 163 · see citing works
Usability Issues in the Design of Novice Programming Systems
Pane, J. F., & Myers, B. A. (2018) · KiltHub Repository · doi:10.1184/r1/6626363 · Cited by 150 · see citing works
End-user software engineering with assertions in the spreadsheet paradigm
Burnett, M., Cook, C. R., Pendse, O., Rothermel, G., Summet, J. W., & Wallace, C. (2003) · doi:10.5555/776816.776828 · Cited by 124 · see citing works
Self-citationSelf-interruption on the computer
Jin, J., & Dabbish, L. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518979 · Cited by 121 · see citing works
The Effects of Auditor Gender and Task Complexity on Information Processing Efficiency
O’Donnell, E., & Johnson, E. N. (2001) · International Journal of Auditing · doi:10.1111/j.1099-1123.2001.00328.x · Cited by 118 · see citing works
Effectiveness of end-user debugging software features
Beckwith, L., Burnett, M., Wiedenbeck, S., Cook, C. R., Sorte, S., & Hastings, M. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055094 · Cited by 109 · see citing works
Self-citationMyers, B. A., Ko, A. J., & Burnett, M. (2006) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1125451.1125472 · Cited by 109 · see citing works
Self-citationHow self-efficacy and gender issues affect software adoption and use
Hartzel, K. S. (2003) · Communications of the ACM · doi:10.1145/903893.903933 · Cited by 93 · see citing works
Some Problems of Professional End User Developers
Segal, J. (2007) · Proceedings/Proceedings -- IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2007.17 · Cited by 87 · see citing works
Does continuous visual feedback aid debugging in direct-manipulation programming systems?
Wilcox, E., Atwood, J. W., Burnett, M., Cadiz, J. J., & Cook, C. (1997) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/258549.258721 · Cited by 81 · see citing works
Self-citationUsing information scent to model the dynamic foraging behavior of programmers in maintenance tasks
Lawrance, J., Bellamy, R., Burnett, M., & Rector, K. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357261 · Cited by 76 · see citing works
Self-citationInstructionless learning about a complex device: the paradigm and observations
Shrager, J., & Klahr, D. (1986) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(86)80075-x · Cited by 72 · see citing works
Furnas, G. W., & Russell, D. M. (2005) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1056808.1057113 · Cited by 67 · see citing works
Fisher, M., & Rothermel, G. (2005) · doi:10.1145/1083231.1083242 · Cited by 62 · see citing works
Testing vs. code inspection vs. what else?
Subrahmaniyan, N., Beckwith, L., Grigoreanu, V., Burnett, M., Wiedenbeck, S., Narayanan, V., Bucht, K., Drummond, R., & Fern, X. Z. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357153 · Cited by 57 · see citing works
Self-citationCan feature design reduce the gender gap in end-user software development environments?
Grigoreanu, V., Cao, J., Kulesza, T., Bogart, C., Rector, K., Burnett, M., & Wiedenbeck, S. (2008) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2008.4639077 · Cited by 50 · see citing works
Self-citationDebugging strategies and tactics in a multi-representation software environment
Romero, P., du Boulay, B., Cox, R., Lutz, R., & Bryant, S. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.07.005 · Cited by 47 · see citing works
Programming at runtime: Requirements & paradigms for nonprogrammer web application development
(2003) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/hcc.2003.1260198 · Cited by 46 · see citing works
EulerView: a non-hierarchical visualization component
Chiara, R. D., & Fish, A. (2007) · Proceedings/Proceedings -- IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2007.42 · Cited by 41 · see citing works
Strategies and behaviors of end-user programmers with interactive fault localization
Prabhakararao, S., Cook, C., Ruthruff, J. R., Creswick, E. R., Main, M., Durham, M., & Burnett, M. (2004) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/hcc.2003.1260197 · Cited by 40 · see citing works
Self-citationMining problem-solving strategies from HCI data
Fern, X. Z., Komireddy, C., Grigoreanu, V., & Burnett, M. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1721831.1721834 · Cited by 34 · see citing works
Self-citationSarbanes-Oxley: What About all the Spreadsheets?Preprint
Panko, R. R., & Ordway, N. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0804.0797 · Cited by 31 · see citing works
Males’ and Females’ Script Debugging Strategies
Grigoreanu, V., Brundage, J. A., Bahna, E., Burnett, M., ElRif, P., & Snover, J. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-00427-8_12 · Cited by 26 · see citing works
Self-citationIs this spreadsheet a tax evader? How HM Customs and Excise test spreadsheet applications
Butler, R. J. (2005) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2000.926737 · Cited by 26 · see citing works
Garbage in, Garbage out? An Empirical Look at Oracle Mistakes by End-User Programmers
Phalgune, A., Kissinger, C., Burnett, M., Cook, C., Beckwith, L., & Ruthruff, J. R. (2005) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2005.40 · Cited by 25 · see citing works
Self-citationDervin, B. L. (1983) · Public Relations Review · doi:10.1016/s0363-8111(83)80197-0 · Cited by 24 · see citing works
Spreadsheet DebuggingPreprint
Ayalew, Y., & Mittermeir, R. T. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0801.4280 · Cited by 24 · see citing works
Supporting User Hypotheses in Problem Diagnosis on the Web and Elsewhere
Wagner, E. J., & Lieberman, H. (2004) · Cited by 22 · see citing works
Gender Differences in End-User Debugging, Revisited: What the Miners Found
Grigoreanu, V., Beckwith, L., Fern, X. Z., Yang, S., Komireddy, C., Narayanan, V., Cook, C., & Burnett, M. (2006) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2006.24 · Cited by 20 · see citing works
Self-citationSupporting user hypotheses in problem diagnosis
Wagner, E. J., & Lieberman, H. (2004) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/964442.964450 · Cited by 19 · see citing works
IEEE Symposium on Visual Languages and Human-Centric Computing-TOC
(2007) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2007.6 · Cited by 15 · see citing works
Ioannidou, A., Repenning, A., & Webb, D. C. (2008) · Proceedings/Proceedings -- IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2008.4639057 · Cited by 13 · see citing works
Kissinger, C., Burnett, M., Stumpf, S., Subrahmaniyan, N., Beckwith, L., Yang, S., & Rosson, M. B. (2006) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/1133265.1133293
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
GenderMag: A Method for Evaluating Software's Gender Inclusiveness
Burnett, M., Stumpf, S., Macbeth, J., Makri, S., Beckwith, L., Kwan, I., Peters, A. N., & Jernigan, W. (2016) · Interacting with Computers · doi:10.1093/iwc/iwv046 · Cited by 292 · see citing works
Self-citationFrom Gender Biases to Gender-Inclusive Design
Vorvoreanu, M., Zhang, L., Huang, Y., Hilderbrand, C., Steine-Hanson, Z., & Burnett, M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300283 · Cited by 119 · see citing works
Self-citationFinding Gender-Inclusiveness Software Issues with GenderMag
Burnett, M., Peters, A. N., Hill, C., & Elarief, N. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858274 · Cited by 101 · see citing works
Self-citationGender-Inclusive HCI Research and Design: A Conceptual Review
Stumpf, S., Peters, A. N., Bardzell, S., Burnett, M., Busse, D. K., Cauchard, J. R., & Churchill, E. F. (2020) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000056 · Cited by 107 · see citing works
Self-citationSocial influences on secure development tool adoption
Xiao, S., Witschey, J., & Murphy-Hill, E. R. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531722 · Cited by 93 · see citing works
Empowering End Users in Debugging Trigger-Action Rules
Corno, F., Russis, L. D., & Roffarello, A. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300618 · Cited by 75 · see citing works
Spreadsheet Comprehension: Guesswork, Giving Up and Going Back to the Author
Srinivasa Ragavan, S., Sarkar, A., & Gordon, A. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445634 · Cited by 67 · see citing works
Yücel, Y., & Rızvanoğlu, K. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.05.029 · Cited by 65 · see citing works
The whats and hows of programmers' foraging diets
Piorkowski, D. J., Fleming, S. D., Kwan, I., Burnett, M. M., Scaffidi, C., Bellamy, R. K. E., & Jordahl, J. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466418 · Cited by 61 · see citing works
Self-citationPiorkowski, D., Fleming, S., Scaffidi, C., Bogart, C., Burnett, M., John, B., Bellamy, R., & Swart, C. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208608 · Cited by 50 · see citing works
Self-citationCombining Tiled and Textual Views of Code
Homer, M., & Noble, J. (2014) · IEEE Working Conference on Software Visualization (VISSOFT) · doi:10.1109/vissoft.2014.11 · Cited by 48 · see citing works
How Developers Diagnose Potential Security Vulnerabilities with a Static Analysis Tool
Smith, J., Johnson, B., Murphy-Hill, E. R., Chu, B., & Lipford, H. R. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2810116 · Cited by 48 · see citing works
What Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 41 · see citing works
Henley, A. Z., & Fleming, S. D. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557073 · Cited by 41 · see citing works
To fix or to learn? How production bias affects developers' information foraging during debugging
Piorkowski, D., Fleming, S. D., Scaffidi, C. P., Burnett, M., Kwan, I., Henley, A. Z., Macbeth, J., Hill, C., & Horvath, A. (2015) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2015.7332447 · Cited by 37 · see citing works
Self-citationForaging and navigations, fundamentally: developers' predictions of value and cost
Piorkowski, D., Henley, A. Z., Nabi, T., Fleming, S. D., Scaffidi, C., & Burnett, M. (2016) · Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2950290.2950302 · Cited by 38 · see citing works
Self-citationPedagogical Content Knowledge for Teaching Inclusive Design
Oleson, A., Mendez, C., Steine-Hanson, Z., Hilderbrand, C., Perdriau, C., Burnett, M., & Ko, A. J. (2018) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3230977.3230998 · Cited by 35 · see citing works
Self-citationDrosos, I., Sarkar, A., Xu, X., Negreanu, C., Rintel, S., & Tankelevitch, L. (2024) · Proceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work · doi:10.1145/3663384.3663389 · Cited by 30 · see citing works
Predator behavior in the wild web world of bugs: An information foraging theory perspective
Kuttal, S. K., Sarma, A., & Rothermel, G. (2013) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2013.6645244 · Cited by 29 · see citing works
My IoT Puzzle: Debugging IF-THEN Rules Through the Jigsaw Metaphor
Corno, F., Russis, L. D., & Roffarello, A. M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-24781-2_2 · Cited by 28 · see citing works
Showkat, D., & Grimm, C. M. (2018) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/urai.2018.8441766 · Cited by 25 · see citing works
A principled evaluation for a principled idea garden
Jernigan, W., Horvath, A., Lee, M., Burnett, M., Cuilty, T., Kuttal, S., Peters, A., Kwan, I., Bahmani, F., & Ko, A. (2015) · 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357222 · Cited by 28 · see citing works
Self-citationChattopadhyay, S., NELSON, N. M., Au, A., Morales, N., Sanchez, C. A., Pandita, R., & Sarma, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380330 · Cited by 23 · see citing works
Analysing Gender Differences in Building Social Goal Models: A Quasi-Experiment
Gralha, C., Goulão, M., & Araújo, J. (2019) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2019.00027 · Cited by 20 · see citing works
Remote monitoring of end-user created automations in field trials
Manca, M., Paternò, F., & Santoro, C. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03239-0 · Cited by 19 · see citing works
Investigating the intelligibility of a computer vision system for blind users
Ahmed, S., Balasubramanian, H., Stumpf, S., Morrison, C. P., Sellen, A. J., & Grayson, M. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3377325.3377508 · Cited by 17 · see citing works
How end-user programmers debug visual web-based programs: An information foraging theory perspective
Kuttal, S. K., Sarma, A., Burnett, M., Rothermel, G., Koeppe, I., & Shepherd, B. (2019) · Journal of Computer Languages · doi:10.1016/j.cola.2019.04.003 · Cited by 14 · see citing works
Self-citationCombining Models for Improved Fault Localization in Spreadsheets
Hofer, B., Hofler, A., & Wotawa, F. (2017) · IEEE Transactions on Reliability · doi:10.1109/tr.2016.2632151 · Cited by 13 · see citing works
General principles for a Generalized Idea Garden
Jernigan, W., Horvath, A., Lee, M. J., Burnett, M., Cuilty, T., Kuttal, S. K., Peters, A. N., Kwan, I., Bahmani, F., Ko, A. J., et al. (2017) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2017.04.005 · Cited by 13 · see citing works
Self-citationSpreadsheet quality assurance: a literature review
Poon, P., Lau, M. F., Yu, Y. T., & Tang, S.-F. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-023-2384-6 · Cited by 11 · see citing works
Are there gender differences when interacting with social goal models?
Gralha, C., Goulão, M., & Araújo, J. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09883-y · Cited by 9 · see citing works
Examining gender issues in perception and acceptance in web-based end-user development activities
Tzafilkou, K., & Protogeros, N. (2017) · Education and Information Technologies · doi:10.1007/s10639-017-9650-x · Cited by 9 · see citing works
Recording, Visualising and Understanding Developer Programming Behaviour
Schröer, M., & Koschke, R. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00066 · Cited by 9 · see citing works
Penney, S., Dodge, J. E., Anderson, A. W., Hilderbrand, C., Simpson, L., & Burnett, M. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3396047 · Cited by 8 · see citing works
Self-citationA Qualitative Study on Framework Debugging
Coker, Z., Widder, D. G., Goues, C. L., Bogart, C., & Sunshine, J. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00091 · Cited by 8 · see citing works
Meng, X., Zhao, S., Huang, Y., Zhang, Z., Eagan, J., & Subramanian, R. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557349 · Cited by 8 · see citing works
Spreadsheet practices and challenges in a large multinational conglomerate
Smith, J., Middleton, J., & Kraft, N. A. (2017) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2017.8103463 · Cited by 8 · see citing works
The Long Tail: Understanding the Discoverability of API Functionality
Horvath, A., Myers, B., Grover, S., Dong, S., Zhou, E., Voichick, F., Kery, M. B., Shinju, S., Nam, D., & Nagy, M. (2019) · 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2019.8818681 · Cited by 9 · see citing works
Developing digital competence - learning, teaching and supporting use of information technology
Kaasbøll, J. (2012) · Cited by 7 · see citing works
Andrao, M., Balducci, F., Breve, B., Cena, F., Desolda, G., Deufemia, V., Cristina Gena, A., Matera, M., Mattioli, A., Paternò, F., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34433-6_7 · Cited by 6 · see citing works
Burnett, M., Churchill, E. F., & Lee, M. J. (2015) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2702613.2727689 · Cited by 6 · see citing works
Self-citationChattopadhyay, S., NELSON, N. M., Nam, T., Calvert, M., & Sarma, A. (2018) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3195836.3195861 · Cited by 6 · see citing works
Toward Theory-Based End-User Software Engineering
Burnett, M., Kulesza, T., Oleson, A., Ernst, S., Beckwith, L., Cao, J., Jernigan, W., & Grigoreanu, V. (2017) · Springer eBooks · doi:10.1007/978-3-319-60291-2_10 · Cited by 6 · see citing works
Self-citationGraceful Language Extensions and Interfaces
Homer, M. (2014) · Figshare · doi:10.26686/wgtn.17008246.v1 · Cited by 1 · see citing works
Multiple-Representation Visual Compositional Dataflow Programming
Homer, M. (2023) · International Conference on Art, Science, and Engineering of Programming (<Programming>) · doi:10.1145/3594671.3594681 · Cited by 3 · see citing works
Towards an IDE to Support Programming as Problem-Solving.
Nelson, N., Sarma, A., & van der Hoek, A. (2017) · PPIG · Cited by 3 · see citing works
Lifting the Curtain on Merge Conflict Resolution: A Sensemaking Perspective
Brindescu, C., Gonzalez, Y. R., Sarma, A., & Jensen, C. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00057 · Cited by 3 · see citing works
A sensemaking analysis of API learning using React
Kelleher, C. L., & Brachman, M. (2022) · Journal of Computer Languages · doi:10.1016/j.cola.2022.101189 · Cited by 2 · see citing works
Understanding and Evaluating Developer Behaviour in Programming Tasks
Schröer, M., & Koschke, R. (2024) · doi:10.1145/3643796.3648450 · Cited by 2 · see citing works
Challenges of Enabling End-Users to Develop Systems with AI
Tetteroo, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34433-6_16 · Cited by 1 · see citing works
Reclaiming the Unexplored in Hybrid Visual Programming
Homer, M. (2024) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3689492.3690045 · Cited by 1 · see citing works
Klein, H. E., & Raturi, A. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706696 · Cited by 1 · see citing works
Newcomers’ experiences during debugging: A cognitive inclusivity perspective using GenderMag
Culas, F. M., Singh, A., Arankalle, A., Dhopade, P., & Blincoe, K. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107932 · Cited by 1 · see citing works
Bjørge, E. M., Jønsson, A., Kaasbøll, J., & Pinard, M. (2015) · IST-Africa Conference (IST-Africa) · doi:10.1109/istafrica.2015.7190570 · Cited by 1 · see citing works
Manesh, D., Luu, A., Khalid, M., Li, J., Okonkwo, C., Akanmu, A. A., Awolusi, I., Murzi, H., & Lee, S. W. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00014 · Cited by 1 · see citing works
SATIN 2 : How to support self-efficacy and diversity in end-user development
Palmquist, L., & Wennberg, P. (2013) · Epubl LTU
FxD: a functional debugger for dysfunctional spreadsheets
Drosos, I., Wilson, N., Gordon, A. D., Ragavan, S. S., & Williams, J. M. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00012
In-Line Compositional Visual Programming
Homer, M. (2024) · International Conference on Art, Science, and Engineering of Programming (<Programming>) · doi:10.1145/3660829.3660841
Newcomers' Experiences During Debugging: A Cognitive Inclusivity PerspectivePreprint
Culas, F. M., Singh, A., Arankalle, A., Dhopade, P., & Blincoe, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5251646
Leon, S., Desai, A. R., Kanth, S., Mahmoud, N., & Kuttal, S. K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30781-1_17
- Authors
- Grigoreanu, V., Burnett, M., Wiedenbeck, S., Cao, J., Rector, K., & Kwan, I.
- Affiliations
- Microsoft (United States) (US)
- Oregon State University (US)
- Drexel University (US)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2012-03
- Volume
- 19
- Issue
- 1
- Pages
- 1-28
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W1974307657
- Cited by
- 60
- References
- 59
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 5 · 2023: 5 · 2022: 2 · 2021: 3 · 2020: 6 · 2019: 8 · 2018: 4 · 2017: 6 · 2016: 3 · 2015: 4 · 2014: 5 · 2013: 3 · 2012: 2
- FWCI
- 14.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- Funders
- National Science Foundation (325273, 917366)
- Air Force Office of Scientific Research (FA9550-09-1-0213)
- Topics
- Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- end-user programming, debugging, sensemaking, spreadsheet software, information foraging, gender differences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
