CoChat
Sign Up
Cited by 38Software Engineering ResearchSoftware Engineering Techniques and PracticesUsability and User Interface Design

Foraging and navigations, fundamentally: developers' predictions of value and cost

Open access
Piorkowski, David, Henley, Austin Z., Nabi, Tahmid, Fleming, Scott D., Scaffidi, Christopher, Burnett, Margaret·2016·Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering
DOI 10.1145/2950290.2950302
Open this paper in CoChat — free accountOpen PDF

Abstract

Empirical studies have revealed that software developers spend 35%–50% of their time navigating through source code during development activities, yet fundamental questions remain: Are these percentages too high, or simply inherent in the nature of software development? Are there factors that somehow determine a lower bound on how effectively developers can navigate a given information space? Answering questions like these requires a theory that captures the core of developers' navigation decisions. Therefore, we use the central proposition of Information Foraging Theory to investigate developers' ability to predict the value and cost of their navigation decisions. Our results showed that over 50% of developers' navigation choices produced less value than they had predicted and nearly 40% cost more than they had predicted. We used those results to guide a literature analysis, to investigate the extent to which these challenges are met by current research efforts, revealing a new area of inquiry with a rich and crosscutting set of research challenges and open problems.

AI snapshot

Written from the paper’s full text.

TL;DR

Using Information Foraging Theory, the authors examined how accurately professional developers predict the value and cost of source-code navigation during a debugging task, and then analyzed whether recent software engineering tools address those prediction failures. They found frequent overestimation of value and underestimation of cost, and identified three open problem areas—value estimation, cost estimation, and scaling beyond one navigation—with six associated research challenges.

Key findings

  • Across 179 navigations by 10 Oracle developers, expectations were optimistic: about 84% expected necessary or sufficient information, but 63/179 (about 35%) delivered less value than expected; including 28/29 “desperation” navigations that yielded no value, about 51% ended in value disappointment. Only 5 navigations exceeded expected value.
  • Cost surprises occurred in 66/179 navigations (36.9%); between-patch costs dominated (over 25%), while within-patch cognitive difficulty was about 13%, surrounding-context needs about 25.7%, and time about 5.6% (categories could co-occur).
  • A literature analysis of 302 papers yielded 55 tools; 100% contributed to value/cost aspects, and 47/55 (85%) improved actual value or cost. However, only 4/55 (7%) aligned expected between-patch cost with actual cost, and 10/55 (18%) aligned expected within-patch cost; 52/55 (95%) had some expected-value support, mostly only one navigation away.
  • The authors frame three open problems—Value Estimation, Cost Estimation, and Scaling Up—and six research challenges: false advertising, synonyms, wrong-question cues, prey in pieces, endless paths, and disjoint topologies.

Methods

  • Think-aloud and retrospective interview study with 10 professional developers at Oracle (4.5–40 years experience; 2–19 years Java), each session up to 2 hours, with 20 minutes on a real jEdit bug (#3223) in Eclipse on Windows; no participant finished the task.
  • Navigations to Java methods were identified from screen recordings; participants were paused at to-method and away-from-method events to report expected and actual value/cost; qualitative coding had 86% inter-rater agreement for value codes and 81% for cost codes on 20% of data.
  • Literature analysis covered 302 papers from a 2013 survey repository plus FSE’14 and ICSE’14; 55 tools met criteria and were coded for support of actual value/cost and expected value/cost, with 90% and 81% inter-rater reliability.

Limitations

  • Paper-stated: retrospective recollections may bias responses; participants were new to the codebase, limiting generalization to other debugging situations; generalizability to other languages and IDEs is deferred; the literature sample may not represent all SE literature.
  • Paper-stated: cost expectations were not measured separately before navigation because participants did not verbalize them; the study instead measured whether actual cost exceeded expected cost after the fact.
  • Evident from the text: the empirical study used a small sample (10 developers), one bug/task, one codebase (jEdit), one IDE/language (Eclipse/Java), and a 20-minute task that no one completed, which constrains external validity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/2950290.2950302

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 32 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    63 of 179 navigations (about 35%) produced lower value than expected.

    “As Table 2's bottom row shows, 63 of participants' 179 navigations (about 35%) produced lower value than expected.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.
  • Findingown data

    About 51% of developers' navigations ended in some degree of disappointment in information value.

    “over 50% of developers' navigation choices produced less value than they had predicted”
    • Also stated in: § 4.1.1 Did They Get the Value They Expected? · plain
      “Thus, in total, about 51% of participants' navigations (highlighted cells in Table 2) ended in some degree of disappointment in the information value they received.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.
  • Findingown data

    Synonym cues sometimes led participants astray.

    “However, synonyms sometimes led our participants astray.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.
  • Findingown data

    Participants almost exclusively used lexical cues, mostly method names, to predict the value of a patch.

    “Participants almost exclusively used lexical cues-mostly method names-to predict the value of a patch to which they were considering navigating.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.
  • Findingown data

    Seven of ten participants (P1, P2, P4, P5, P8, P9, P10) ran into trouble foraging for methods that update jEdit's underlying model when a jEdit user performs an editing action.

    “For example, 7 of our 10 participants (P1, P2, P4, P5, P8, P9, P10) ran into trouble foraging for the methods that actually update jEdit's underlying model when a jEdit user performs an editing action.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.
  • May combine two claims. 35
  • Classified as probably not a substantial claim. 35

  • May combine two claims.

    Findingown data

    About 25% of navigations expected Sufficient information, and about 59% expected Necessary but not Sufficient.

    “The first row shows navigations in which participants expected to find everything they needed (Sufficient: about 25%), and the second shows navigations in which they expected to find at least something they needed (Necessary: about 59%).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.

  • May combine two claims.

    Findingown data

    The false advertising problem was very common among participants: P2, P4, P7, P8, and P9 all suffered instances of it.

    “The false advertising problem was very common among our participants: P2, P4, P7, P8, and P9 all suffered instances of it.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.

  • May combine two claims.

    Findingown data

    The dominant type of unexpected cost was between-patch, affecting over 25% of participants' navigations.

    “However, the dominant type of unexpected cost was betweenpatch, C b , affecting over 25% of participants' navigations.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Research Challenge #4 (Prey in pieces scattered among several patches) was encountered by P1, P3, P4, P5, P6, P7, P9, P10.

    “#4: Prey in pieces scattered among several patches P1, P3, P4, P5, P6, P7, P9, P10”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the above situation, participants had to resort to finding the relevant representations of GUI elements in the code by navigating through code, which was both costly and unfruitful.

    “In the above situation, participants had to resort to finding the relevant representations of GUI elements in the code by navigating through code, which was both costly and unfruitful.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Research Challenge #5 (Path too long, no end in sight) was encountered by P1, P2, P3, P4, P5, P8, P9.

    “#5: Path too long, no end in sight P1, P2, P3, P4, P5, P8, P9”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    jEdit uses an external library-the third topology-to automatically build menus based on the content of an XML properties file-the fourth topology disjoint from the others.

    “jEdit uses an external library-the third topology-to automatically build menus based on the content of an XML properties file-the fourth topology disjoint from the others.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The empirical study showed these problems to have significant effects on developers' productivity, with high percentages of expensive wasted developer effort.

    “Our empirical study showed these problems to have significant effects on developers' productivity, with high percentages of expensive wasted developer effort.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Tools helping to align E(C) with C were relatively rare-only 4/55 (7%) made any attempt to align E(Cb) with Cb, and only 10/55 (18%) worked to align E(Cw) with Cw.

    “The fourth result Table 7 reveals is that tools helping to align E(C) with C were relatively rare-only 4/55 (7%) made any attempt to align E(C b ) with C b , and only 10/55 (18%) worked to align E(C w ) with C w .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Documentation for the semantic differences between variables existed within comments in the code, but participants sometimes needed a combination of knowledge from several patches to understand the different variables.

    “Documentation for the semantic differences between the variables existed within comments in the code, but participants sometimes needed a combination of knowledge from several patches to understand the different variables.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.

  • May combine two claims.

    Findingown data

    After formulating the goal of finding code triggered by GUI actions, participants had to switch over to jEdit's code and start a fresh set of navigations, since there was no direct way to go from executing the action in jEdit to the code that handles that action.

    “after formulating that goal, they then had to switch over to jEdit's code and start a fresh set of navigations, since there was no direct way to go from executing the action in jEdit to the code that handles that action.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Three patterns repeatedly led to unexpectedly high between-patch costs: prey in pieces scattered among multiple patches, path long with no end in sight, and no available path.

    “For between-patch cost C b , we identified three patterns that the participants faced that repeatedly led to unexpectedly high costs: (1) the prey was in pieces scattered among multiple patches, (2) the path to the prey was long with no end in sight, and (3) sometimes there simply was no available path to the prey.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    The properties file defined the content of the menus and specified which methods to fire for each menu item.

    “The properties file defined the content of the menus and specified which methods to fire for each menu item.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All except one of these 47 focuses on decreasing cost Cb of navigating to a patch, but over half (29) focus also (or in one case, instead) on the cost Cw of navigating within that patch.

    “All except one of these 47 focuses on decreasing cost C b of navigating to a patch, but over half (29) focus also (or in one case, instead) on the cost C w of navigating within that patch.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Participants located the relevant code by using search tools, by investigating the stack trace, or by setting a breakpoint and stepping through code.

    “Instead, participants located the relevant code by using search tools, by investigating the stack trace, or by setting a breakpoint and stepping through code.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For variables like physicalLine, understanding their origin required navigating multiple call hierarchies, and the number of patches to investigate grew rapidly; all four participants eventually gave up.

    “With each additional variable, there was yet another call hierarchy to investigate, and the number of patches to investigate grew rapidly.Eventually, all four of these participants decided the cost was too high, and gave up.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Research Challenge #6 (No path across different topologies) was encountered by P1, P2, P3, P4, P5, P6, P7, P8, P9, P10.

    “#6: No path across different topologies P1, P2, P3, P4, P5, P6, P7, P8, P9, P10”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Searches in code frequently failed to turn up desired results, and developers spent substantial time organizing and reorganizing files and bookmarks in the IDE.

    “This result was consistent with an earlier study into the challenges of information foraging by developers, such as the fact that searches in code frequently failed to turn up desired results, and that developers spent substantial time organizing and reorganizing files and bookmarks in the IDE [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] 21. Ko, A., Myers B., Coblenz, M., Aung, H. 2006. An explora- tory study of how developers seek, relate, and collect rele- vant information during software maintenance tasks. IEEE Trans. Soft. Eng (TSE). 33, 971-987. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Software engineering researchers have produced tools that help reduce developers' navigation costs.

    “These and other findings about developers' foraging and navigations (e.g., [26,31,37,44]) have led software engineering researchers to produce tools that help to reduce developers' navigation costs as they look for the information they need [8,17,18,19,22,30,33,43].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [8] 8. DeLine, R., Khella, A., Czerwinski, M. and Robertson, G. 2005. Towards understanding programs through wear-based filtering. ACM Symp. Software Visualization (SoftVis), ACM, 183-192. source

      Not checked in the free run.

    • cites [17] 17. Henley, A. and Fleming, S. 2014. The Patchworks code editor: Toward faster navigation with less code arranging and fewer navigation mistakes. ACM Conf. Human Factors in Comp. Sys. (CHI)., 2511-2520. source

      Not checked in the free run.

    • cites [18] 18. Karrer, T., Kramer, J., Diehl, J., Hartmann, B., and Borchers, J. 2011. Stacksplorer: Call graph navigation helps increasing code maintenance efficiency. ACM Symp. User Interface Soft. and Technology (UIST), 217-224. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    According to Information Foraging Theory, developers make navigation decisions by predicting the value a navigation will bring and the cost they will incur, treating foraging as an optimization problem to maximize expected value per expected cost.

    “According to IFT, developers make their navigation decisions by predicting the value a navigation will bring and the cost they will incur if they take that navigation.”
    • Also stated in: § 2. BACKGROUND AND RELATED WORK 2.1 Information Foraging Theory · plain
      “IFT's central proposition, which says that the predator treats foraging as an optimization problem.”
    • Also stated in: § 2. BACKGROUND AND RELATED WORK 2.1 Information Foraging Theory · plain
      “max(E(V)/E(C))”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [34] 34. Pirolli, P. and Card, S. 1995. Information foraging in infor- mation access environments. ACM Conf. Human Factors in Comp. Sys. (CHI), 51-58. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The task was sufficiently complex that no participant finished it in the allotted time.

    “The task, by design, was sufficiently complex that no participant finished it in the allotted time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Prior empirical studies have observed navigation behavior during maintenance and shed light on kinds of information developers seek.

    “Prior empirical studies have observed the navigation behavior of developers during maintenance activities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Analysts recovering requirement traceability spent disproportionate time in low-value patches.

    “One study examined navigations of analysts through documents and code as they attempted to recover requirement traceability between use cases and Java classes, revealing that the subjects spent a disproportionate amount of time in low-value patches (i.e., more time than the value of those patches justified) [31].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] 31. Niu, N., Mahmoud, A., Chen, Z. and Bradshaw, G. 2013. Departures from optimality: Understanding human analyst's information foraging in assisted requirements tracing. ACM/IEEE Intl. Conf. Soft. Engr. (ICSE), 572-581.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Developers seek information about how to use APIs, about data or control flow, and about requirements.

    “These include the need for information about how to use APIs [10], about data or control flow [23,46], and about requirements [31].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [10] 10. Duala-Ekoko, E. and Robillard, M. P. 2012. Asking and answering questions about unfamiliar APIs: An exploratory study. ACM/IEEE Intl. Conf. Soft. Engr. (ICSE), 266-276.

      Not checked in the free run.

    • cites [23] 23. LaToza, T. D. and Myers, B. A. 2010. Developers ask reach- ability questions. ACM/IEEE Intl. Conf. Soft. Engr. (ICSE), 185-194. source

      Not checked in the free run.

    • cites [46] 46. Xuan, Q., Okano, A., Devanbu, P., and Filkov, V. 2014. Focus-shifting patterns of OSS developers and their congru- ence with call graphs. ACM Int'l. Symp. Found. Soft. Engr. (FSE), 401-412. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Developers seek information to answer specific questions, test mental hypotheses, develop context, or follow leads.

    “These studies have also revealed that developers seek this information to answer specific questions that they have [23,42], to test mental hypotheses about how an existing program works [4], to develop an overall sense of context [13], or to follow leads that simply look relevant to their current task [26].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [23] 23. LaToza, T. D. and Myers, B. A. 2010. Developers ask reach- ability questions. ACM/IEEE Intl. Conf. Soft. Engr. (ICSE), 185-194. source

      Not checked in the free run.

    • cites [42] 42. Sillito, J., Murphy, G. C., and De Volder, K. 2006. Questions programmers ask during software evolution tasks. ACM Symp. Found. Soft. Engr. (FSE), 23-34. source

      Not checked in the free run.

    • cites [4] 4. Brooks, R. 1983. Towards a theory of the comprehension of computer programs. Intl. J. Man-Machine Studies, 18(6), 543-554. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The study did not measure expected cost and actual cost separately because participants did not verbalize cost expectations before navigating.

    “Participants did not verbalize their expectations of cost before navigating so we did not measure E(C) and C separately.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    The paper's literature analysis revealed a new area of inquiry with a rich and crosscutting set of research challenges and open problems.

    “We used those results to guide a literature analysis, to investigate the extent to which these challenges are met by current research efforts, revealing a new area of inquiry with a rich and crosscutting set of research challenges and open problems.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Huge variations in designers' terminology across numerous application domains are an inherent property of the English language.

    “That paper showed how huge variations in designers' terminology across numerous application domains are an inherent property of the English language.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] 14. Furnas, G. W., Landauer, T. K., Gomez, L. M., and Dumais, S. T. 1987. The vocabulary problem in human-system com- munication. Comm. ACM, 30(11), 964-971. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Developers may be disappointed if their destination does not provide as much value as expected or if the cost of extracting or getting to information is higher than expected.

    “When they carry out these navigation decisions, they may be in for disappointments if (1) their destination does not provide as much value as expected or (2) the cost of extracting the information or getting to it is higher than expected.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Over 90% of relevant synonyms are unique to software engineering.

    “The synonyms problem may relate to Ge et al.'s observation that over 90% of relevant synonyms are unique to software engineering [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] 15. Ge, X. and Murphy-Hill, E. 2014. Manual refactoring chang- es with automated refactoring validation. ACM/IEEE Intl. Conf. Soft. Engr. (ICSE), 1095-1105. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Too little synonym agglomeration produces too many false negatives, but too much synonym agglomeration produces too many false positives.

    “Thus, too little synonym agglomeration produces too many false negatives, but too much synonym agglomeration produces too many false positives.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Searches online often lead to irrelevant code and supporting documents, except after developers determined the right search terms for specific subtopics.

    “Another study of how developers search online resources corroborated that searches often lead to irrelevant code and supporting documents, except after developers had determined the right search terms for specific subtopics [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] 2. Bajracharya, S. K. and Lopes, C. V. 2012. Analyzing and mining a code search engine usage log. Empirical Soft. Engr., 17(4-5), 424-466. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Little is known at the foundational level about how well developers can predict the value and cost of their navigations, and SE research has not systematically considered this.

    “Despite these gains, however, little is known at the foundational level of developer navigation-how well developers go about choosing where to navigate.”
    • Also stated in: § 2. BACKGROUND AND RELATED WORK 2.1 Information Foraging Theory · plain
      “Software engineering research has much to say about information professional developers seek (e.g., [26,42]), but has not systematically considered the question of how well developers can predict these values and costs, or the accompanying implications for SE tools.”
    • Also stated in: § 2.2 Prior Empirical Studies of Developers · plain
      “Little empirical work has investigated the specific question of developers' abilities to predict the value and cost of their navigations.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [26] 26. Lawrance, J., Bogart, C., Burnett, M., Bellamy, R., Rector, K., and Fleming, S. 2013. How programmers debug, revisited: An information foraging theory perspective. IEEE Trans. Soft. Eng. (TSE), 39(2), 197-215. source

      Not checked in the free run.

    • cites [42] 42. Sillito, J., Murphy, G. C., and De Volder, K. 2006. Questions programmers ask during software evolution tasks. ACM Symp. Found. Soft. Engr. (FSE), 23-34. source

      Not checked in the free run.

    • cites [31] 31. Niu, N., Mahmoud, A., Chen, Z. and Bradshaw, G. 2013. Departures from optimality: Understanding human analyst's information foraging in assisted requirements tracing. ACM/IEEE Intl. Conf. Soft. Engr. (ICSE), 572-581.

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    Inter-rater reliability was 86% agreement for value codes and 81% for cost codes on 20% of the data.

    “Their resulting inter-rater reliability was 86% agreement using the Jaccard index (the intersection of all applied codes over the union of all applied codes) on 20% of the data for the value codes, and 81% agreement on 20% of the data for the cost codes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The results are not expected to generalize to other kinds of debugging situations.

    “This is a common situation for new hires and when developers transfer to different development teams, but we do not expect it to generalize to other kinds of debugging situations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Participants' recollections of what happened later may have biased their responses.

    “However, participants' recollections of what happened later may have biased their responses.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Whyline builds a path just-in-time to bridge the gap between two topologies: from GUI output to its relevant source code, and SketchLink links sketches to code.

    “Whyline [20] builds a path just-in-time to bridge the gap between two topologies: from GUI output to its relevant source code, and SketchLink [3] links sketches to code.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [20] 20. Ko, A. and Myers, B. 2008. Debugging reinvented: Asking and answering why and why not questions about program behavior. ACM/IEEE Int'l Conf. Software Eng. (ICSE), 301- 310. source

      Not checked in the free run.

    • cites [3] 3. Baltes, S., Schmitz, P., and Diehl, S. 2014. Linking sketches and diagrams to source code artifacts. ACM Symp. Found. Soft. Engr. (FSE), 743-746. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    For the empirical study, inter-rater reliability was 81%-92% on all code sets, helping to assure construct validity.

    “For our empirical study, our inter-rater reliability was 81%-92% on all code sets, helping to assure construct validity (the extent to which a measure actually captures what is intended).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Two researchers independently coded the same 20% of the data, and calculated their level of agreement using the Jaccard index.

    “Specifically, two researchers independently coded the same 20% of the data, and calculated their level of agreement using the Jaccard index.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The codes cover every possible way to align value V with E(V) if V<E(V), and to align cost C with E(C) if C>E(C) for the two factors of C, namely Cb and Cw.

    “As the table shows, the codes cover every possible way to align value V with E(V) if V<E(V), and to align cost C with E(C) if C>E(C) for the two factors of C, namely C b and C w .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    A threat to external validity is that the sampling of current SE trends may not be complete.

    “However, a threat to external validity is that our sampling of current SE trends may not be complete.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Whyline does not scale to a program of jEdit's size, approaches like Chen/Grundy's do not support navigations per se, and few of the approaches located handle more than two disjoint topologies.

    “However, Whyline does not scale to a program of jEdit's size, approaches like Chen/Grundy's do not support navigations per se, and few of the approaches we have located handle more than two disjoint topologies.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [20] 20. Ko, A. and Myers, B. 2008. Debugging reinvented: Asking and answering why and why not questions about program behavior. ACM/IEEE Int'l Conf. Software Eng. (ICSE), 301- 310. source

      Not checked in the free run.

    • cites [5] 5. Chen, X. and Grundy, J. 2011. Improving automated docu- mentation to code traceability by combining retrieval tech- niques. IEEE/ACM Intl. Conf. Automated Software Engineer- ing (ASE), 223-232. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The analysis covered 302 papers from top publication venues, but these SE literature repositories still may not generalize across all SE literature.

    “Our analysis covered 302 papers from top publication ven-ues, but these SE literature repositories still may not generalize across all SE literature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Study design and conductnone given

    There is a question of generalizability to other programming languages and IDEs.

    “Additionally, there is a question of generalizability to other programming languages and IDEs, which we defer to future work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The researchers did not rely on raters' interpretations alone, but also used retrospective interviews to remind participants where they were and to gather participants' interpretations of which events mattered and why they did what they did.

    “To further help assure construct validity, we did not rely on our raters' interpretations alone, but also used retrospective interviews to remind participants where they were and also to gather participants' interpretations of which events mattered and why they did what they did.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    After achieving 90% interrater reliability on the first repository and 81% on the remaining two, the researchers divided up the coding of the remaining data.

    “After achieving 90% interrater reliability on the first repository and 81% on the remaining two, they divided up the coding of the remaining data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    MCIDiff and CloneTracker consider the possible set of clones, show the length and the number of instances of all code clones, not just nearby ones, to help the developer predict E(Cw) of handling any or all of these clones.

    “These two clone-tracking tools consider the possible set of clones, show the length and the number of instances of all code clones, not just nearby ones, to help the developer predict E(C w ) of handing any or all of these clones.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [28] 28. Lin, Y., Xing, Z., Xue, Y., Liu, Y., Peng, X., Sun, J., and Zhao, W. 2014. Detecting differences across multiple in- stances of code clones. ACM/IEEE Intl. Conf. Soft. Engr. (ICSE), 164-174. source

      Not checked in the free run.

    • cites [11] 11. Duala-Ekoko, E. and Robillard, M. 2007. Tracking code clones in evolving software. ACM/IEEE Intl. Conf. Soft. Engr. (ICSE). 158-167. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    The results further suggest a new area of inquiry for SE researchers: how large, feature-dense SE environments can support developers' ability to predict the value they will receive from a navigation path and the cost they must expend to receive that value.

    “The results further suggest a new area of inquiry for SE researchers: how large, feature-dense SE environments can support developers' ability to predict the value they will receive from a navigation path and the cost they must expend to receive that value.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationnone given

    Section 5.2's examples provide useful ideas toward ultimately addressing some of the research challenges of Section 4, but only a few help the developer align E(V) with V or E(C) with C for patches more than one click away.

    “Section 5.2's examples provide useful ideas toward ultimately addressing some of the research challenges of Section 4, but only a few help the developer align E(V) with V or E(C) with C for patches more than one click away.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    Table 5's summary of cost-related foraging research challenges contributing to these issues reveals a substantive and difficult open problem space analogous to the Value Estimation Problem.

    “Table 5's summary of costrelated foraging research challenges contributing to these issues reveals a substantive and difficult open problem space analogous to the Value Estimation Problem presented in Section 4.1:”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Developers' future productivity will depend on SE researchers' ability to make significant progress toward solving the open problems and challenges that were revealed by considering developer navigation at a foundational level.

    “Developers' future productivity will depend on SE researchers' ability to make significant progress toward solving the open problems and challenges that were revealed by considering developer navigation at a foundational level.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationown data

    This area of inquiry appears to be rich, challenging, and cross-cutting, with three open problem spaces involving (at least) six research challenges.

    “As Sections 4-5 showed and Table 8 summarizes, this area of inquiry appears to be rich, challenging, and cross-cutting, with three open problem spaces involving (at least) six research challenges.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    A few such tools make useful inroads in directions needed to help address the issues.

    “Fortunately, a few such tools make useful inroads in directions needed to help address the issues, as we pointed out in the literature analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Ferret allows developers to ask conceptual queries about a particular program element and shows the number of results for the query, allowing the developer to gain some idea of the sum of all the E(Cw)s they will incur.

    “Ferret allows developers to ask conceptual queries about a particular program element such as”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] 7. De Alwis, B. and Murphy, G. C. 2008. Answering conceptual queries with Ferret. ACM/IEEE Int'l. Conf. Software Engi- neering (ICSE). 21-30. source

      Not checked in the free run.

54 of 54

  • The paper relies on this without support.

    Participants' verbalizations during think-aloud and retrospective interviews accurately reflect their expected and actual value and cost of navigations.

    “The purpose was to collect participants' expected value before a navigation, and then the value they actually received.”

  • The paper relies on this without support.

    The jEdit debugging task and Oracle participants are representative of professional software developers' navigation in general.

    “Ten professional developers at Oracle participated in the study.”
    “The participants' task was to debug an actual bug (issue #3223) in the jEdit open source project.”

  • The paper relies on this without support.

    The ordinal scale and coding scheme adequately capture the value and cost of navigations.

    “To investigate the participants' assessments of a patch's value, we used an ordinal scale of measurement.”
    “we coded participants' responses to the retrospective interview questions (Fig. 3).”

  • The authors state this as an assumption.

    All value and cost foraging disappointments overlapped.

    “Conservatively assuming that all value and cost foraging disappointments overlapped”

  • The authors state this as an assumption.

    There is no overlap in value and cost foraging disappointments.

    “a worst-case summation that assumes no overlap in these disappointments”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Synonym examples name different participants

    Minor

    The discussion names P6 as a synonym-related example, while the summary table lists P7 instead. The two lists otherwise name the same participants.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. 10.5555/1177220 · DOI
  2. 10.5555/2337223.2337255 · DOI
  3. 10.5555/2486788.2486848 · DOI

Cited by

Sorted by citations

Related

Authors
Piorkowski, D., Henley, A. Z., Nabi, T., Fleming, S. D., Scaffidi, C., & Burnett, M.
Affiliations
  • Oregon State University (US)
  • University of Memphis (US)
ORCID
Venue
Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering
Publisher
ACM
Published
2016-11
Pages
97-108
DOI
10.1145/2950290.2950302
OpenAlex
W2546851909
Cited by
38
References
45
Citations by year
2026: 4 · 2025: 2 · 2024: 5 · 2023: 1 · 2022: 4 · 2021: 3 · 2020: 3 · 2019: 4 · 2018: 5 · 2017: 6 · 2016: 1
FWCI
4.53
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor, 365 d)
Locations
ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
Funders
National Science Foundation (1302113, 1302117, and 1314384)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
information foraging theory, software developers, source code navigation, navigation decision-making, value prediction, cost prediction, information space

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review