Using developer interaction data to compare expertise metrics
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
The expertise of a software developer is said to be a crucial factor for the development time required to complete a task. Even if this hypothesis is intuitive, research has not yet quantified the effect of developer expertise on development time. A related problem is that the design space for expertise metrics is large; out of the various automated expertise metrics proposed, we do not know which metric most reliably captures expertise. What prevents a proper evaluation of expertise metrics and their relation with development time is the lack of data on development tasks, such as their precise duration. Fortunately, this data is starting to become available in the form of growing developer interaction repositories. We show that applying MSR techniques to these developer interaction repositories gives us the necessary tools to perform such an evaluation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452 · cited 2×
“We claim that repositories of fine-grained developer interactions—recorded with tools such as Mylyn [1], Syde [2], or Spyware [3]—contain the data needed for a comparison of”
MethodExpertise Browser: a quantitative approach to identifying expertise
Mockus, A., & Herbsleb, J. D. (2003) · Proceedings of the 24th International Conference on Software Engineering. ICSE 2002 · doi:10.1109/icse.2002.1007994 · Cited by 383
Expertise browser: a quantitative approach to identifying expertise
Mockus, A., & Herbsleb, J. D. (2002) · Proceedings of the 24th international conference on Software engineering - ICSE '02 · doi:10.1145/581339.581401 · Cited by 199 · cited 2×
“Fritz et al. [12] expanded the DOI in a Degree of Knowledge (DOK), and compared it to Mockus and Herbsleb’s recommender, finding an improvement according to developer feedback.”
How Developers Drive Software Evolution
Girba, T., Kuhn, A., Seeberger, M., & Ducasse, S. (2006) · Eighth International Workshop on Principles of Software Evolution (IWPSE'05) · doi:10.1109/iwpse.2005.21 · Cited by 166
Syde: a tool for collaborative software development
Hattori, L., & Lanza, M. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2 · doi:10.1145/1810295.1810339 · Cited by 135 · cited 2×
“We claim that repositories of fine-grained developer interactions—recorded with tools such as Mylyn [1], Syde [2], or Spyware [3]—contain the data needed for a comparison of expertise metrics.”
MethodA degree-of-knowledge model to capture source code familiarity
Fritz, T., Ou, J., Murphy, G. C., & Murphy-Hill, E. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806856 · Cited by 123 · cited 1×
“[12] expanded the DOI in a Degree of Knowledge (DOK), and compared it to Mockus and Herbsleb’s recommender, finding an improvement according to developer feedback.”
ResultDetermining Implementation Expertise from Bug Reports
Anvik, J., & Murphy, G. C. (2007) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2007.7 · Cited by 112 · cited 1×
“Anvik and Murphy [10] use bug reports as the primary source of information for their expertise metric.”
BackgroundAgents to assist in finding help
Vivacqua, A. S., & Lieberman, H. A. (2000) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/332040.332408 · Cited by 98 · cited 2×
“Vivacgua and Lieberman [8] present Expert Finder, a system that locates experts on java APIs based on their usage”
BackgroundDoes a programmer's activity indicate knowledge of code?
Fritz, T., Murphy, G. C., & Hill, E. (2007) · Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1287624.1287673 · Cited by 96
SpyWare: a change-aware development toolset
Robbes, R., & Lanza, M. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368219 · Cited by 81 · cited 2×
“We claim that repositories of fine-grained developer interactions—recorded with tools such as Mylyn [1], Syde [2], or Spyware [3]—contain the data needed for a comparison of expertise metrics.”
MethodSelf-citationExpert recommendation with usage expertise
David, W. L., Schüler, D., Zimmermann, T., & Sillito, J. (2009) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2009.5306386 · Cited by 65 · cited 1×
“[9] also present a similar recommender.”
Evaluating expertise recommendations
McDonald, D. W. (2001) · Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work - GROUP '01 · doi:10.1145/500286.500319 · Cited by 63 · cited 1×
“McDonald [5] presents a recommendation system that uses Change History and Tech Support heuristic to perform its recommendations.”
MethodVersioning Systems for Evolution Research
Robbes, R., & Lanza, M. (2006) · International Workshop on Principles of Software Evolution · doi:10.1109/iwpse.2005.32 · Cited by 54 · cited 1×
“session in a commit, and not the actual programmer activity [4], making them unsuitable for our needs.”
BackgroundSelf-citationRefining code ownership with synchronous changes
Hattori, L., Lanza, M., & Robbes, R. (2010) · Empirical Software Engineering · doi:10.1007/s10664-010-9145-5 · Cited by 28 · cited 2×
“[13], takes into account the recency of changes.”
BackgroundSelf-citation
Cited by
Thongtanunam, P., McIntosh, S., Hassan, A. E., & Iida, H. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884852 · Cited by 138
“Indeed, several studies complement authorship data with the data recorded during developer IDE activities to compute developer expertise [13, 21, 42].”
BackgroundAutomatically Recommending Peer Reviewers in Modern Code Review
Zanjani, M. B., Kagdi, H., & Bird, C. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2500238 · Cited by 175
“We consider time as a proxy measure of effort, which is typically used in other domains [18].”
MethodTowards a theory of software development expertise
Baltes, S., & Diehl, S. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236061 · Cited by 83
“tise research in software engineering mainly focused on expert recommendation, utilizing information such as change history [53, 71, 78], usage history [66, 112], bug reports [3], or interaction data [34, 86]. Investigated aspects of software development expertise (SDExp) include programming experience [95], age [81],…”
BackgroundBuilding an Expert Recommender Chatbot
Cerezo, J., Kubelka, J., Robbes, R., & Bergel, A. (2019) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse.2019.00022 · Cited by 78
“Anvik and Murphy used data from bug reports [5]; Robbes and Roethlis-berger [19] used interaction data [14].”
MethodSelf-citationStudying Software Developer Expertise and Contributions in Stack Overflow and GitHub
Vadlamani, S. L., & Baysal, O. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00038 · Cited by 77
“However, in software engineering, it is much more difficult to find reliable and objective metrics to measure expertise, and this has been previously reported in the software engineering literature [2], [3].”
Background
Metadata
- Authors
- Robbes, R., & Rothlisberger, D.
- Affiliations
- University of Chile (CL)
- ORCID
- Robbes, Romain
- Venue
- 2013 10th Working Conference on Mining Software Repositories (MSR)
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 297-300
- OpenAlex
- W2067096279
- Cited by
- 36
- References
- 14
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 1 · 2022: 1 · 2021: 3 · 2020: 7 · 2019: 2 · 2018: 2 · 2017: 3 · 2016: 2 · 2015: 7 · 2014: 4 · 2013: 1
- FWCI
- 3.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion)
- Open Research (University of Surrey) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- developer expertise, development time, metric evaluation, software engineering tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.