Understanding understanding source code with functional magnetic resonance imaging
Open accessAbstract
Program comprehension is an important cognitive process that inherently eludes direct measurement. Thus, researchers are struggling with providing suitable programming languages, tools, or coding conventions to support developers in their everyday work. In this paper, we explore whether functional magnetic resonance imaging (fMRI), which is well established in cognitive neuroscience, is feasible to soundly measure program comprehension. In a controlled experiment, we observed 17 participants inside an fMRI scanner while they were comprehending short source-code snippets, which we contrasted with locating syntax errors. We found a clear, distinct activation pattern of five brain regions, which are related to working memory, attention, and language processing---all processes that fit well to our understanding of program comprehension. Our results encourage us and, hopefully, other researchers to use fMRI in future studies to measure program comprehension and, in the long run, answer questions, such as: Can we predict whether someone will be an excellent programmer? How effective are new languages and tools for program understanding? How should we train programmers?
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When comprehension of code was compared to detection of syntactic errors in code, activity in both fronto-parietal and putative language areas was observed.
“When comprehension of code was compared to detection of syntactic errors in code, activity in both fronto-parietal and putative language areas was observed (Siegmund et al., 2014, Siegmund, 2017).”
From Computer code comprehension shares neural resources with formal logical inference in the fronto-parietal network (2020) · cites this paper as Siegmund et al., 2014
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
The assessment and analysis of handedness: The Edinburgh inventory
Oldfield, R. C. (1971) · Neuropsychologia · doi:10.1016/0028-3932(71)90067-4 · Cited by 35,900 · see citing works
Co-Planar Stereotaxic Atlas of the Human Brain
Talairach, J., & Tournoux, P. (1988) · Medical Entomology and Zoology · Cited by 11,798 · see citing works
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102 · see citing works
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059 · see citing works
Kiczales, G. (1996) · ACM Computing Surveys · doi:10.1145/242224.242420 · Cited by 5,146 · see citing works
Binder, J. R., Desai, R. H., Graves, W. W., & Conant, L. L. (2009) · Cerebral Cortex · doi:10.1093/cercor/bhp055 · Cited by 4,060 · see citing works
Imaging Cognition II: An Empirical Review of 275 PET and fMRI Studies
Cabeza, R., & Nyberg, L. (2000) · Journal of Cognitive Neuroscience · doi:10.1162/08989290051137585 · Cited by 3,598 · see citing works
Bergmans, L., & Lopes, C. V. (1999) · Lecture notes in computer science · doi:10.1007/3-540-46589-8_17 · Cited by 2,951 · see citing works
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,339 · see citing works
A simple objective technique for measuring flexibility in thinking
Berg, E. A. (1948) · The Journal of General Psychology · doi:10.1080/00221309.1948.9918159 · Cited by 2,255 · see citing works
Towards a neural basis of auditory sentence processing
Friederici, A. D. (2002) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01839-8 · Cited by 2,231 · see citing works
Handedness and hemispheric language dominance in healthy humans
Knecht, S. (2000) · Brain · doi:10.1093/brain/123.12.2512 · Cited by 1,560 · see citing works
On Broca, brain, and binding: A new framework
Hagoort, P. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.07.004 · Cited by 1,411 · see citing works
Tarr, P. L., Ossher, H. L., Harrison, W., & Sutton, S. M. (1999) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/302405.302457 · Cited by 1,249 · see citing works
Training of Working Memory in Children With ADHD
Klingberg, T., Forssberg, H., & Westerberg, H. (2002) · Journal of Clinical and Experimental Neuropsychology · doi:10.1076/jcen.24.6.781.8395 · Cited by 1,225 · cited 1× · see citing works
“Thus, a test prior to programming education [48, 41] might reveal which students might struggle with learning programming.”
Engle, R. W., Kane, M. J., & Tuholski, S. W. (1999) · Models of Working Memory · doi:10.1017/cbo9781139174909.007 · Cited by 1,195 · see citing works
Driven to Distraction: Dual-Task Studies of Simulated Driving and Conversing on a Cellular Telephone
Strayer, D. L., & Johnston, W. A. (2001) · Psychological Science · doi:10.1111/1467-9280.00386 · Cited by 1,134 · see citing works
Elements of software science (Operating and programming systems series)
Halstead, M. H. (1977) · Cited by 1,037 · see citing works
Lesion analysis of the brain areas involved in language comprehension
Dronkers, N. F., Wilkins, D. P., Van Valin, R. D., Redfern, B. B., & Jaeger, J. J. (2004) · Cognition · doi:10.1016/j.cognition.2003.11.002 · Cited by 1,035 · see citing works
Turken, A. U., & Dronkers, N. F. (2011) · Frontiers in System Neuroscience · doi:10.3389/fnsys.2011.00001 · Cited by 1,012 · see citing works
Is working memory still working?
Baddeley, A. (2001) · American Psychologist · doi:10.1037/0003-066x.56.11.851 · Cited by 1,011 · see citing works
Dissociating Verbal and Spatial Working Memory Using PET
Smith, E. E., Jonides, J., & Koeppe, R. A. (1996) · Cerebral Cortex · doi:10.1093/cercor/6.1.11 · Cited by 962 · see citing works
Empirical Studies of Programming Knowledge
Soloway, E. M., & Ehrlich, K. (1984) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1984.5010283 · Cited by 875 · see citing works
Language, Thought, and Reality
(2019) · Catholic University of America Press eBooks · doi:10.2307/j.ctvt9k614.12 · Cited by 840 · see citing works
Activation of Extrastriate and Frontal Cortical Areas by Visual Words and Word-Like Stimuli
Petersen, S. E., Fox, P. T., Snyder, A. Z., & Raichle, M. E. (1990) · Science · doi:10.1126/science.2396097 · Cited by 821 · see citing works
Object-oriented metrics: measures of complexity
Henderson‐Sellers, B. (1995) · Prentice-Hall, Inc eBooks · Cited by 753 · see citing works
Awh, E., Jonides, J., Smith, E. E., Schumacher, E. H., Koeppe, R. A., & Katz, S. (1996) · Psychological Science · doi:10.1111/j.1467-9280.1996.tb00662.x · Cited by 705 · see citing works
Bottini, G., Corcoran, R., Sterzi, R., Paulesu, E., Schenone, P., Scarpa, P., Frackowiak, R. S. J., & Frith, D. (1994) · Brain · doi:10.1093/brain/117.6.1241 · Cited by 660 · see citing works
Stimulus structures and mental representations in expert comprehension of computer programs
Pennington, N. (1987) · Cognitive Psychology · doi:10.1016/0010-0285(87)90007-7 · Cited by 526 · see citing works
Prabhakaran, V., Smith, J. A., Desmond, J. E., Glover, G. H., & Gabrieli, J. D. E. (1997) · Cognitive Psychology · doi:10.1006/cogp.1997.0659 · Cited by 509 · see citing works
Granularity in software product lines
Kästner, C., Apel, S., & Kuhlemann, M. (2008) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/1368088.1368131 · Cited by 497 · see citing works
Self-citationWorking memory capacity — facets of a cognitive ability construct
Oberauer, K., Süß, H., Schulze, R., Wilhelm, O., & Wittmann, W. W. (2000) · Personality and Individual Differences · doi:10.1016/s0191-8869(99)00251-2 · Cited by 496 · see citing works
Selected Writings on Computing: A personal Perspective
Dijkstra, E. W. (1982) · Springer eBooks · doi:10.1007/978-1-4612-5695-3 · Cited by 457 · see citing works
Culture, language and personality
Sapir, E., & Mandelbaum, D. G. (1966) · Medical Entomology and Zoology · Cited by 450 · see citing works
Brain activity during reading The effects of exposure duration and task
Price, C. J., Wise, R. J. S., Watson, J. D. G., Patterson, K. E., Howard, D., & Frackowiak, R. S. J. (1994) · Brain · doi:10.1093/brain/117.6.1255 · Cited by 425 · see citing works
The brain basis of syntactic processes: functional imaging and lesion studies
Friederici, A. D., & Kotz, S. A. (2003) · NeuroImage · doi:10.1016/j.neuroimage.2003.09.003 · Cited by 414 · see citing works
Teaching objects-first in introductory computer science
Cooper, S., Dann, W. P., & Pausch, R. F. (2003) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/611892.611966 · Cited by 400 · see citing works
Brodmann’s Localisation in the Cerebral Cortex
Brodmann, K. (2005) · Springer eBooks · doi:10.1007/b138298 · Cited by 396 · see citing works
Syntactic/semantic interactions in programmer behavior: A model and experimental results
Shneiderman, B., & Mayer, R. E. (1979) · International Journal of Parallel Programming · doi:10.1007/bf00977789 · Cited by 383 · see citing works
Jones, P. W. (2001) · Oxford Music Online · doi:10.1093/gmo/9781561592630.article.20622 · Cited by 383 · see citing works
Jäncke, L., Shah, N. J., & Peters, M. (2000) · Cognitive Brain Research · doi:10.1016/s0926-6410(00)00028-8 · Cited by 376 · see citing works
EducationPaving the way for computational thinking
Guzdial, M. (2008) · Communications of the ACM · doi:10.1145/1378704.1378713 · Cited by 350 · see citing works
The mind of expert motor performance is cool and focused
MILTON, J. G., Solodkin, A., Hluštı́k, P., & Small, S. L. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.01.003 · Cited by 340 · see citing works
Exploratory experimental studies comparing online and offline programming performance
Sackman, H., Erikson, W. J., & Grant, E. E. (1968) · Communications of the ACM · doi:10.1145/362851.362858 · Cited by 335 · see citing works
Grodzinsky, Y., & Santi, A. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.09.001 · Cited by 334 · see citing works
Fiebach, C. J., Schlesewsky, M., Lohmann, G., von Cramon, D. Y., & Friederici, A. D. (2004) · Human Brain Mapping · doi:10.1002/hbm.20070 · Cited by 329 · see citing works
Kersten, M. A., & Murphy, G. C. (2005) · International Conference on Aspect-Oriented Software Development (AOSD) · doi:10.1145/1052898.1052912 · Cited by 299 · see citing works
Hierarchical artificial grammar processing engages Broca's area
Bahlmann, J., Schubotz, R. I., & Friederici, A. D. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.04.249 · Cited by 294 · see citing works
Robillard, M. P., & Murphy, G. C. (2002) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/581339.581390 · Cited by 288 · see citing works
Task-Dependent Modulation of Regions in the Left Inferior Frontal Cortex during Semantic Processing
Roskies, A. L., Fiez, J. A., Balota, D. A., Raichle, M. E., & Petersen, S. E. (2001) · Journal of Cognitive Neuroscience · doi:10.1162/08989290152541485 · Cited by 279 · see citing works
Natural programming languages and environments
Myers, B. A., Pane, J. F., & Ko, A. J. (2004) · Communications of the ACM · doi:10.1145/1015864.1015888 · Cited by 247 · see citing works
Cerebral activation during performance of a card sorting test
Nagahama, Y., Fukuyama, H., Yamauchi, H., Matsuzaki, S., Konishi, J., Shibasaki, H., & Kimura, J. (1996) · Brain · doi:10.1093/brain/119.5.1667 · Cited by 214 · see citing works
University of California Press
Muto, A. (1995) · University of California Press eBooks · doi:10.1525/9780520912274 · Cited by 212 · see citing works
Studying programmer behavior experimentally
Brooks, R. (1980) · Communications of the ACM · doi:10.1145/358841.358847 · Cited by 208 · see citing works
What artificial grammar learning reveals about the neurobiology of syntax
Petersson, K. M., Folia, V., & Peter Hagoort, A. (2010) · Brain and Language · doi:10.1016/j.bandl.2010.08.003 · Cited by 195 · see citing works
Models and theories of programming strategy
Davies, S. P. (1993) · International Journal of Man-Machine Studies · doi:10.1006/imms.1993.1061 · Cited by 172 · see citing works
An Eye Tracking Study on camelCase and under_score Identifier Styles
Sharif, B., & Maletic, J. I. (2010) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2010.41 · Cited by 144 · see citing works
Attention to One or Two Features in Left or Right Visual Field: A Positron Emission Tomography Study
Vandenberghe, R. R., Duncan, J., Dupont, P. J., Ward, R. D., Poline, J., Bormans, G. M., Michiels, J., Mortelmans, L., & Orban, G. A. (1997) · Journal of Neuroscience · doi:10.1523/jneurosci.17-10-03739.1997 · Cited by 137 · see citing works
Thought and Two Languages: The Impact of Bilingualism on Cognitive Development
Díaz, R. M. (1983) · Review of Research in Education · doi:10.2307/1167134 · Cited by 129 · see citing works
Determining language laterality by fMRI and dichotic listening
Bethmann, A., Tempelmann, C., Bleser, R. D., Scheich, H., & Brechmann, A. (2006) · Brain Research · doi:10.1016/j.brainres.2006.11.057 · Cited by 119 · see citing works
Self-citationMeasuring programming experience
Feigenspan, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2012) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240511 · Cited by 106 · see citing works
Self-citationUsing a behavioral theory of program comprehension in software engineering
Brooks, R. (1978) · doi:10.5555/800099.803210 · Cited by 97 · see citing works
Exploiting the map metaphor in a tool for software evolution
Griswold, W. G., Yuan, J. J., & Kato, Y. (2001) · doi:10.5555/381473.381501 · Cited by 95 · see citing works
Neural Correlates of Artificial Grammar Learning
Skosnik, P. D., Mirza, F., Gitelman, D. R., Parrish, T. B., Mesulam, M. M., & Reber, P. J. (2002) · NeuroImage · doi:10.1006/nimg.2002.1291 · Cited by 87 · see citing works
How do we tell truths that might hurt?
Dijkstra, E. W. (1982) · ACM SIGPLAN Notices · doi:10.1145/947923.947924 · Cited by 82 · see citing works
Some basic determinants of computer programming productivity
Chrysler, E. (1978) · Communications of the ACM · doi:10.1145/359511.359523 · Cited by 80 · see citing works
The teaching of software engineering
Shooman, M. L. (1983) · doi:10.1145/800038.801015 · Cited by 60 · see citing works
Chapter 2: Thought and Two Languages: The Impact of Bilingualism on Cognitive Development
Díaz, R. M. (1983) · Review of Research in Education · doi:10.3102/0091732x010001023 · Cited by 58 · see citing works
The neurobiology of syntax: beyond string sets
Petersson, K. M., & Peter Hagoort, A. (2012) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2012.0101 · Cited by 56 · see citing works
Using Sex Differences to Link Spatial Cognition and Program Comprehension
Fisher, M. L., Cox, A., & Zhao, L. (2006) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2006.72 · Cited by 53 · see citing works
Creativity as a pathway to computer science
Knobelsdorf, M., & Romeike, R. (2008) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1384271.1384347 · Cited by 52 · see citing works
Darcy, D. P., & Ma, M. (2005) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2005.261 · Cited by 48 · see citing works
Hanenberg, S., Kleinschmager, S., & Josupeit-Walter, M. (2009) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5316028 · Cited by 46 · see citing works
Guéhéneuc, Y. (2006) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1145/1188966.1188968 · Cited by 45 · see citing works
Subvocalization - Toward Hearing the Inner Thoughts of Developers
Parnin, C. (2011) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2011.49 · Cited by 39 · see citing works
Self-citationProgrammer information needs after memory failure
Parnin, C., & Rugaber, S. (2012) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240479 · Cited by 37 · see citing works
Self-citationToward measuring program comprehension with functional magnetic resonance imaging
Siegmund, J., Brechmann, A., Apel, S., Kästner, C., Liebig, J., Leich, T., & Saake, G. (2012) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2393596.2393624 · Cited by 28 · see citing works
Self-citationA Simple and Effective Measure for Complex Low-Level Dependencies
Beyer, D., & Fararooy, A. (2010) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2010.49 · Cited by 26 · see citing works
Dyslexia and learning computer programming
Powell, N. J., Moore, D., Gray, J. M., Finlay, J., & Reaney, J. (2004) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1007996.1008072 · Cited by 20 · see citing works
Revisiting Information Hiding: Reflections on Classical and Nonclassical Modularity
Ostermann, K., Giarrusso, P. G., Kästner, C., & Rendel, T. (2011) · Cited by 18 · see citing works
Self-citationHansen, M. E., Lumsdaine, A., & Goldstone, R. L. (2012) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/2384592.2384596 · Cited by 18 · see citing works
What Makes Code Hard to Understand?Preprint
Hansen, M., Goldstone, R. L., & Lumsdaine, A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.5257 · Cited by 17 · see citing works
Baniassad, E., & Fleissner, S. (2006) · doi:10.1145/1176617.1176625 · Cited by 9 · see citing works
Creating a cognitive metric of programming task difficulty
de Alwis, B., Murphy, G. C., & Minto, S. (2008) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/1370114.1370122 · Cited by 8 · see citing works
Wootton, S., & Horne, T. (2010) · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 87 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Coding as another language: a pedagogical approach for teaching computer science in early childhood
Bers, M. U. (2019) · Journal of Computers in Education · doi:10.1007/s40692-019-00147-3 · Cited by 248 · see citing works
Eye Movements in Code Reading: Relaxing the Linear Order
Busjahn, T., Bednarik, R., Begel, A., Crosby, M. E., Paterson, J. H., Schulte, C., Sharif, B., & Tamm, S. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.36 · Cited by 182 · see citing works
Stuck and Frustrated or in Flow and Happy: Sensing Developers' Emotions and Progress
Muller, S. C., & Fritz, T. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.334 · Cited by 150 · see citing works
Comprehension of Procedural Visual Business Process Models
Figl, K. (2017) · Business & Information Systems Engineering · doi:10.1007/s12599-016-0460-2 · Cited by 140 · see citing works
Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise
Floyd, B., Santander, T., & Weimer, W. R. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.24 · Cited by 136 · see citing works
Using (bio)metrics to predict code quality online
Muller, S. C., & Fritz, T. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884803 · Cited by 120 · see citing works
(2017) · doi:10.1145/3106237.3106268 · Cited by 117 · see citing works
Self-citationThe effect of poor source code lexicon and readability on developers' cognitive load
Fakhoury, S., Ma, Y., Arnaoudova, V., & Adesope, O. (2018) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3196321.3196347 · Cited by 116 · see citing works
Comprehension of computer code relies primarily on domain-general executive brain regions
Ivanova, A. A., Srikant, S., Sueoka, Y., Kean, H., Dhamala, R., O’Reilly, U.-M., Bers, M. U., & Fedorenko, E. G. (2020) · eLife · doi:10.7554/elife.58906 · Cited by 111 · see citing works
Eye Movements in Code Reading: Relaxing the Linear Order
Busjahn, T., Bednarik, R., Begel, A., Crosby, M. E., Paterson, J. H., Schulte, C., Sharif, B., & Tamm, S. (2016) · Cited by 111 · see citing works
Tracing software developers' eyes and interactions for change tasks
Kevic, K., Walters, B. M., Shaffer, T., Sharif, B., Shepherd, D. C., & Fritz, T. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786864 · Cited by 107 · see citing works
Towards a theory of software development expertise
Baltes, S., & Diehl, S. (2018) · ESEC/SIGSOFT FSE · doi:10.1145/3236024.3236061 · Cited by 106 · see citing works
Program Comprehension: Past, Present, and Future
Siegmund, J. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.35 · Cited by 93 · see citing works
Self-citationTowards an affordable brain computer interface for the assessment of programmers’ mental workload
Kosti, M. V., Georgiadis, K., Adamos, D. A., Laskaris, N., Spinellis, D. D., & Angelis, L. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.03.002 · Cited by 83 · see citing works
Program Comprehension and Code Complexity Metrics: An fMRI Study
Peitek, N., Apel, S., Parnin, C., Brechmann, A., & Siegmund, J. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00056 · Cited by 84 · see citing works
Self-citationA Look into Programmers’ Heads
Peitek, N., Siegmund, J., Apel, S., Kästner, C., Parnin, C., Bethmann, A., Leich, T., Saake, G., & Brechmann, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2863303 · Cited by 82 · see citing works
Self-citationWeber, B., Fischer, T., & Riedl, R. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110946 · Cited by 78 · see citing works
The Language of Programming: A Cognitive Perspective
Fedorenko, E. G., Ivanova, A. A., Dhamala, R., & Bers, M. U. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.04.010 · Cited by 75 · see citing works
Mining biometric data to predict programmer expertise and task difficulty
Lee, S., Hooshyar, D., Ji, H., Nam, K., & Lim, H. (2017) · Cluster Computing · doi:10.1007/s10586-017-0746-2 · Cited by 71 · see citing works
Understanding Programming Expertise
Crk, I., Kluthe, T., & Stefik, A. (2015) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2829945 · Cited by 71 · see citing works
How programmers read regular code: a controlled experiment using eye tracking
Jbara, A., & Feitelson, D. G. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9477-x · Cited by 68 · see citing works
Liu, Y.-F., Kim, J., Wilson, C., & Bedny, M. (2020) · eLife · doi:10.7554/elife.59340 · Cited by 64 · see citing works
Lee, S., Matteson, A. S., Hooshyar, D., Kim, S., Jung, J., Nam, G., & Lim, H. (2016) · IEEE International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe.2016.30 · Cited by 67 · see citing works
Understanding misunderstandings in source code
Gopstein, D., Iannacone, J., Yu, Y., DeLong, L. A., Zhuang, Y., Yeh, M. K. ‐., & Cappos, J. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106264 · Cited by 65 · see citing works
Identifying Experts in Software Libraries and Frameworks Among GitHub Users
Montandon, J. E., Silva, L. L., & Valente, M. T. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00054 · Cited by 63 · see citing works
The role of the insula in intuitive expert bug detection in computer code: an fMRI study
Castelhano, J., Duarte, I. C., Ferreira, C., Duraes, J. A., Madeira, H., & Castelo-Branco, M. S. (2018) · Brain Imaging and Behavior · doi:10.1007/s11682-018-9885-1 · Cited by 63 · see citing works
Gender Differences in Personality Traits of Software Engineers
Russo, D., & Stol, K.-J. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3003413 · Cited by 59 · see citing works
How does the degree of variability affect bug finding?
Melo, J., Brabrand, C., & Wąsowski, A. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884831 · Cited by 55 · see citing works
ActionNet: Vision-Based Workflow Action Recognition From Programming Screencasts
Zhao, D., Xing, Z., Chen, C., Xia, X., & Li, G. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00049 · Cited by 49 · see citing works
Project-Level Encoding for Neural Source Code Summarization of Subroutines
Bansal, A., Haque, S., & McMillan, C. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00032 · Cited by 45 · see citing works
Feitelson, D. G., Mizrahi, A., Noy, N., Shabat, A. B., Eliyahu, O., & Sheffer, R. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2976920 · Cited by 48 · see citing works
WAP: understanding the brain at software debugging
Duraes, J., Madeira, H., Castelhano, J., Duarte, C., & Branco, M. C. (2016) · 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2016.53 · Cited by 47 · see citing works
Simultaneous measurement of program comprehension with fMRI and eye tracking
Peitek, N., Siegmund, J., Parnin, C., Apel, S., Hofmeister, J. C., & Brechmann, A. (2018) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3239235.3240495 · Cited by 47 · see citing works
Self-citationLeveraging Biometric Data to Boost Software Developer Productivity
Fritz, T., & Muller, S. C. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.107 · Cited by 45 · see citing works
Fakhoury, S., Roy, D., Ma, Y., Arnaoudova, V., & Adesope, O. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09751-4 · Cited by 45 · see citing works
Can EEG Be Adopted as a Neuroscience Reference for Assessing Software Programmers’ Cognitive Load?
Medeiros, J., Couceiro, R., Duarte, G., Duraes, J. A., Castelhano, J., Duarte, I. C., Castelo-Branco, M. S., Madeira, H., Carvalho, P., & Teixeira, C. (2021) · Sensors · doi:10.3390/s21072338 · Cited by 43 · see citing works
On the Optimal Order of Reading Source Code Changes for Review
Baum, T., Schneider, K. A., & Bacchelli, A. (2017) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2017.28 · Cited by 42 · see citing works
Ivie: Lightweight Anchored Explanations of Just-Generated Code
Yan, L., Hwang, A., Wu, Z., & Head, A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642239 · Cited by 40 · see citing works
Krueger, R., Huang, Y., Liu, X., Santander, T., Weimer, W. R., & Leach, K. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380348 · Cited by 40 · see citing works
Yan, L., Xu, S., & Jia Liu, B. (2021) · Journal of Pacific Rim Psychology · doi:10.1177/18344909211010240 · Cited by 39 · see citing works
Huang, Y., Leach, K., Sharafi, Z., McKay, N., Santander, T., & Weimer, W. R. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409681 · Cited by 38 · see citing works
The introductory computer programming course is first and foremost a language course
Portnoff, S. R. (2018) · ACM Inroads · doi:10.1145/3152433 · Cited by 37 · see citing works
Distilling Neural Representations of Data Structure Manipulation using fMRI and fNIRS
Huang, Y., Liu, X., Krueger, R., Santander, T., Hu, X., Leach, K., & Weimer, W. R. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00053 · Cited by 37 · see citing works
Cognitive load measurement - A methodology to compare low cost commercial EEG devices
Das, R., Chatterjee, D., Das, D., Sinharay, A., & Sinha, A. (2014) · International Conference on Advances in Computing, Communications and Informatics (ICACCI) · doi:10.1109/icacci.2014.6968528 · Cited by 36 · see citing works
A Replication Study on Code Comprehension and Expertise using Lightweight Biometric Sensors
Fucci, D., Girardi, D., Novielli, N., Quaranta, L., & Lanubile, F. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00050 · Cited by 35 · see citing works
What Drives the Reading Order of Programmers?
Peitek, N., Siegmund, J., & Apel, S. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389279 · Cited by 34 · see citing works
Self-citationAn Empirical Study on the Patterns of Eye Movement during Summarization Tasks
Rodeghero, P., & McMillan, C. (2015) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2015.7321188 · Cited by 34 · see citing works
Studying the difference between natural and programming language corpora
Casalnuovo, C., Sagae, K., & Devanbu, P. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9669-7 · Cited by 33 · see citing works
Eye gaze and interaction contexts for change tasks – Observations and potential
Kevic, K., Walters, B. M., Shaffer, T., Sharif, B., Shepherd, D. C., & Fritz, T. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.03.030 · Cited by 33 · see citing works
Indentation: Simply a Matter of Style or Support for Program Comprehension?
Bauer, J., Siegmund, J., Peitek, N., Hofmeister, J. C., & Apel, S. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00033 · Cited by 33 · see citing works
Self-citationObserving and predicting knowledge worker stress, focus and awakeness in the wild
Soto, M., Satterfield, C., Fritz, T., Murphy, G. C., Shepherd, D. C., & Kraft, N. A. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102560 · Cited by 30 · see citing works
Toward an Objective Measure of Developers’ Cognitive Activities
Sharafi, Z., Huang, Y., Leach, K., & Weimer, W. R. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3434643 · Cited by 30 · see citing works
Peitek, N., Bergum, A., Rekrut, M., Mucke, J., Nadig, M., Parnin, C., Siegmund, J., & Apel, S. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549084 · Cited by 32 · see citing works
Self-citationNeed for Sleep: The Impact of a Night of Sleep Deprivation on Novice Developers’ Performance
Fucci, D., Scanniello, G., Romano, S., & Juristo, N. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2834900 · Cited by 31 · see citing works
Toward measuring program comprehension with functional magnetic resonance imaging
Siegmund, J., Brechmann, A., Apel, S., Kästner, C., Liebig, J., Leich, T., & Saake, G. (2012) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2393596.2393624 · Cited by 28 · see citing works
Self-citationVariability through the Eyes of the Programmer
Melo, J., Narcizo, F. B., Hansen, D. W., Brabrand, C., & Wąsowski, A. (2017) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2017.34 · Cited by 27 · see citing works
Biofeedback Augmented Software Engineering: Monitoring of Programmers' Mental Effort
Couceiro, R., Duarte, G., Duraes, J. A., Castelhano, J., Duarte, I. C., Teixeira, C., Castelo-Branco, M. S., Carvalho, P., & Madeira, H. (2019) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier.2019.00018 · Cited by 27 · see citing works
Spotting Problematic Code Lines using Nonintrusive Programmers' Biofeedback
Couceiro, R., Barbosa, R., Duraes, J. A., Duarte, G., Castelhano, J., Duarte, I. C., Teixeira, C., Laranjeiro, N., Medeiros, J., Carvalho, P., et al. (2019) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2019.00019 · Cited by 26 · see citing works
Eyes on Code: A Study on Developers’ Code Navigation Strategies
Sharafi, Z., Bertram, I., Flanagan, M., & Weimer, W. R. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3032064 · Cited by 25 · see citing works
The effects of visualization and interaction techniques on feature model configuration
Asadi, M., Soltani, S., Gašević, D., & Hatala, M. (2015) · Empirical Software Engineering · doi:10.1007/s10664-014-9353-5 · Cited by 24 · see citing works
Expert Programmers Have Fine-Tuned Cortical Representations of Source Code
Ikutani, Y., Kubo, T., Nishida, S., Hata, H., Matsumoto, K., Ikeda, K., & Nishimoto, S. (2020) · eNeuro · doi:10.1523/eneuro.0405-20.2020 · Cited by 24 · see citing works
Statistical errors in software engineering experiments
Ch., R. P. R., Dieste, Ó., Fonseca, E. R., & Juristo, N. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180161 · Cited by 22 · see citing works
Duran, R., Sorva, J., & Seppälä, O. (2021) · ACM Transactions on Computing Education · doi:10.1145/3469128 · Cited by 22 · see citing works
Information flow and cognition affect each other: Evidence from digital learning
Sharma, K., Mangaroska, K., van Berkel, N., Giannakos, M. N., & Kostakos, V. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102549 · Cited by 24 · see citing works
Debugging: The key to unlocking the mind of a novice programmer?
Lowe, T. (2019) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie43999.2019.9028699 · Cited by 21 · see citing works
Xu, S., Li, Y., & Jia Liu, B. (2021) · Cerebral Cortex · doi:10.1093/cercor/bhab182 · Cited by 21 · see citing works
Graafsma, I. L., Robidoux, S., Nickels, L., Roberts, M., Polito, V., Zhu, J. D., & Marinus, E. (2023) · Journal of Cognitive Psychology · doi:10.1080/20445911.2023.2166054 · Cited by 21 · see citing works
Measuring User Experience of Adaptive User Interfaces using EEG: A Replication Study
Gaspar-Figueiredo, D., Abrahão, S., Insfrán, E., & Vanderdonckt, J. M. (2023) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3593434.3593452 · Cited by 21 · see citing works
Brain Activity Measurement during Program Comprehension with NIRS
Ikutani, Y., & Uwano, H. (2014) · The International journal of networked and distributed computing · doi:10.2991/ijndc.2014.2.4.7 · Cited by 21 · see citing works
Peitek, N., Siegmund, J., Parnin, C., Apel, S., & Brechmann, A. (2018) · doi:10.1145/3216723.3216725 · Cited by 21 · see citing works
Self-citationSemantic Transfer in Programming Languages: Exploratory Study of Relative Novices
Tshukudu, E., & Cutts, Q. I. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387406 · Cited by 21 · see citing works
Endres, M., Fansher, M., Shah, P., & Weimer, W. R. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468583 · Cited by 19 · see citing works
Relating Reading, Visualization, and Coding for New Programmers: A Neuroimaging Study
Endres, M., Karas, Z., Hu, X., Kovelman, I., & Weimer, W. R. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00062 · Cited by 18 · see citing works
Real-Time Monitoring of Neural State in Assessing and Improving Software Developers' Productivity
Radevski, S., Hata, H., & Matsumoto, K. (2015) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2015.28 · Cited by 19 · see citing works
Understanding the impact of pair programming on the minds of developers
Busechian, S., Ivanov, V. V., Rogers, A., Sirazitdinov, I., Succi, G., Tormasov, A. G., & Yi, J. (2018) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3183399.3183413 · Cited by 16 · see citing works
Studying programming in the neuroage
Siegmund, J., Peitek, N., Brechmann, A., Parnin, C., & Apel, S. (2020) · Communications of the ACM · doi:10.1145/3347093 · Cited by 16 · see citing works
Self-citationHow Programmers Read Regular Code: A Controlled Experiment Using Eye Tracking
Jbara, A., & Feitelson, D. G. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.35 · Cited by 16 · see citing works
Feitelson, D. G. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10160-3 · Cited by 15 · see citing works
Software code complexity assessment using EEG features
Medeiros, J., Teixeira, C., Couceiro, R., Castelhano, J., Castelo-Branco, M. S., Duarte, G., Duarte, I. C., Duraes, J. A., Madeira, H., & Carvalho, P. (2019) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2019.8856283 · Cited by 15 · see citing works
EEG Activities During Program Comprehension: An Exploration of Cognition
Lin, Y., Liao, Y.-Z., Hu, X., & Wu, C.-C. (2021) · IEEE Access · doi:10.1109/access.2021.3107795 · Cited by 14 · see citing works
Using biometric data in software engineering: a systematic mapping study
Menzen, J. P., Farias, K., & Bischoff, V. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1734086 · Cited by 14 · see citing works
Program Comprehension in Virtual Reality
Dominic, J., Tubre, B., Houser, J. A., Ritter, C., Kunkel, D., & Rodeghero, P. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389287 · Cited by 14 · see citing works
Gaze-Driven Design Insights to Amplify Debugging Skills: A Learner-Centered Analysis Approach
Mangaroska, K., Sharma, K., Giannakos, M. N., Trætteberg, H., & Dillenbourg, P. (2018) · Journal of Learning Analytics · doi:10.18608/jla.2018.53.7 · Cited by 15 · see citing works
Stangl, F. J., & Riedl, R. (2024) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-58396-4_11 · Cited by 13 · see citing works
Initial evaluation of the brain activity under different software development situations
Ikramov, R., Ivanov, V. V., Masyagin, S., Shakirov, R., Sirazidtinov, I., Succi, G., Thapaliya, A., Tormasov, A. G., & Zufarova, O. (2019) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2019-215 · Cited by 13 · see citing works
Hands-on in Computer Programming Education
von Hausswolff, K. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3105735 · Cited by 13 · see citing works
Santos, A. R., Machado, I., de Almeida, E. S., Siegmund, J., & Apel, S. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9658-x · Cited by 13 · see citing works
Self-citationReal-time monitoring of neural state in assessing and improving software developers' productivity
Radevski, S., Hata, H., & Matsumoto, K. (2015) · doi:10.5555/2819321.2819339 · Cited by 12 · see citing works
Doukakis, S. (2019) · AIMS neuroscience · doi:10.3934/neuroscience.2019.3.175 · Cited by 12 · see citing works
Measuring software developers' perceived difficulty with biometric sensors
Muller, S. C. (2015) · doi:10.5555/2819009.2819206 · Cited by 12 · see citing works
Computer Programming as a Second Language
Pandža, N. B. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-41932-9_36 · Cited by 12 · see citing works
Artifact Removal from EEG Signals Recorded Using Low Resolution Emotiv Device
Sinha, A., Chatterjee, D., Das, R., Datta, S., Gavas, R. D., & Saha, S. K. (2015) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2015.256 · Cited by 12 · see citing works
Medeiros, J., Simões, M. A., Castelhano, J., Abreu, R., Couceiro, R., Henriques, J. M. O., Castelo-Branco, M. S., Madeira, H., Teixeira, C., & Carvalho, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0299108 · Cited by 11 · see citing works
Moving towards objective measures of program comprehension
Fakhoury, S. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3275426 · Cited by 11 · see citing works
Beller, M., Zaidman, A., & Карпов, А. Г. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.34 · Cited by 11 · see citing works
Karas, Z., Jahn, A., Weimer, W. R., & Huang, Y. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468579 · Cited by 11 · see citing works
Gonçales, L. J., Farias, K., Kupssinskü, L. S., & Segalotto, M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117354 · Cited by 11 · see citing works
User experience with adaptive user interfaces: Comparing performance and preferences
Gaspar-Figueiredo, D., Vanderdonckt, J. M., Abrahão, S., & Insfrán, E. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112598 · Cited by 10 · see citing works
Beller, M., Zaidman, A., & Карпов, А. Г. (2015) · doi:10.5555/2820282.2820317 · Cited by 10 · see citing works
EEG monitoring during software development
Calcagno, A., Coelli, S., Couceiro, R., Duraes, J. A., Amendola, C., Pirovano, I., Re, R., & Bianchi, A. M. (2020) · IEEE Mediterranean Electrotechnical Conference (MELECON) · doi:10.1109/melecon48756.2020.9140717 · Cited by 10 · see citing works
- Authors
- Siegmund, J., Kästner, C., Apel, S., Parnin, C., Bethmann, A., Leich, T., Saake, G., & Brechmann, A.
- Affiliations
- University of Passau (DE)
- Carnegie Mellon University (US)
- Georgia Institute of Technology (US)
- Leibniz Institute for Neurobiology (DE)
- Metop Research Institute, Germany
- Otto-von-Guericke-Universität Magdeburg (DE)
- Venue
- Proceedings of the 36th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2014-05-31
- Pages
- 378-389
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W2014232144
- Cited by
- 219
- References
- 87
- Citations by year
- 2026: 19 · 2025: 13 · 2024: 15 · 2023: 8 · 2022: 12 · 2021: 24 · 2020: 38 · 2019: 22 · 2018: 18 · 2017: 10 · 2016: 18 · 2015: 17 · 2014: 4 · 2012: 1
- FWCI
- 13.93
- Type
- conference paper
- Language
- en
- Open access
- green
- figshare.com
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- KiltHub Repository (submittedVersion)
- Figshare (submittedVersion, cc-by)
- KiltHub Repository
- Funders
- National Science Foundation
- Deutsche Forschungsgemeinschaft
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neuroscience, Education and Cognitive Function
- Keywords
- program comprehension, source code understanding, functional MRI, brain activation patterns, working memory, attention, language processing, programming languages, programmer training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
