Using developer interaction data to compare expertise metrics
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
The expertise of a software developer is said to be a crucial factor for the development time required to complete a task. Even if this hypothesis is intuitive, research has not yet quantified the effect of developer expertise on development time. A related problem is that the design space for expertise metrics is large; out of the various automated expertise metrics proposed, we do not know which metric most reliably captures expertise. What prevents a proper evaluation of expertise metrics and their relation with development time is the lack of data on development tasks, such as their precise duration. Fortunately, this data is starting to become available in the form of growing developer interaction repositories. We show that applying MSR techniques to these developer interaction repositories gives us the necessary tools to perform such an evaluation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Using task context to improve programmer productivity
Kersten, M., & Murphy, G. C. (2006) · Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1181775.1181777 · Cited by 452 · cited 2×
“We claim that repositories of fine-grained developer interactions—recorded with tools such as Mylyn [1], Syde [2], or Spyware [3]—contain the data needed for a comparison of”
MethodExpertise Browser: a quantitative approach to identifying expertise
Mockus, A., & Herbsleb, J. D. (2003) · Proceedings of the 24th International Conference on Software Engineering. ICSE 2002 · doi:10.1109/icse.2002.1007994 · Cited by 383
Expertise browser: a quantitative approach to identifying expertise
Mockus, A., & Herbsleb, J. D. (2002) · Proceedings of the 24th international conference on Software engineering - ICSE '02 · doi:10.1145/581339.581401 · Cited by 199 · cited 2×
“Fritz et al. [12] expanded the DOI in a Degree of Knowledge (DOK), and compared it to Mockus and Herbsleb’s recommender, finding an improvement according to developer feedback.”
How Developers Drive Software Evolution
Girba, T., Kuhn, A., Seeberger, M., & Ducasse, S. (2006) · Eighth International Workshop on Principles of Software Evolution (IWPSE'05) · doi:10.1109/iwpse.2005.21 · Cited by 166
Syde: a tool for collaborative software development
Hattori, L., & Lanza, M. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2 · doi:10.1145/1810295.1810339 · Cited by 135 · cited 2×
“We claim that repositories of fine-grained developer interactions—recorded with tools such as Mylyn [1], Syde [2], or Spyware [3]—contain the data needed for a comparison of expertise metrics.”
MethodA degree-of-knowledge model to capture source code familiarity
Fritz, T., Ou, J., Murphy, G. C., & Murphy-Hill, E. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806856 · Cited by 123 · cited 1×
“[12] expanded the DOI in a Degree of Knowledge (DOK), and compared it to Mockus and Herbsleb’s recommender, finding an improvement according to developer feedback.”
ResultDetermining Implementation Expertise from Bug Reports
Anvik, J., & Murphy, G. C. (2007) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2007.7 · Cited by 112 · cited 1×
“Anvik and Murphy [10] use bug reports as the primary source of information for their expertise metric.”
BackgroundAgents to assist in finding help
Vivacqua, A. S., & Lieberman, H. A. (2000) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/332040.332408 · Cited by 98 · cited 2×
“Vivacgua and Lieberman [8] present Expert Finder, a system that locates experts on java APIs based on their usage”
BackgroundDoes a programmer's activity indicate knowledge of code?
Fritz, T., Murphy, G. C., & Hill, E. (2007) · Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1287624.1287673 · Cited by 96
SpyWare: a change-aware development toolset
Robbes, R., & Lanza, M. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368219 · Cited by 81 · cited 2×
“We claim that repositories of fine-grained developer interactions—recorded with tools such as Mylyn [1], Syde [2], or Spyware [3]—contain the data needed for a comparison of expertise metrics.”
MethodSelf-citation
Cited by
Thongtanunam, P., McIntosh, S., Hassan, A. E., & Iida, H. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884852 · Cited by 138
“Indeed, several studies complement authorship data with the data recorded during developer IDE activities to compute developer expertise [13, 21, 42].”
BackgroundAutomatically Recommending Peer Reviewers in Modern Code Review
Zanjani, M. B., Kagdi, H., & Bird, C. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2500238 · Cited by 175
“We consider time as a proxy measure of effort, which is typically used in other domains [18].”
MethodTowards a theory of software development expertise
Baltes, S., & Diehl, S. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236061 · Cited by 83
“tise research in software engineering mainly focused on expert recommendation, utilizing information such as change history [53, 71, 78], usage history [66, 112], bug reports [3], or interaction data [34, 86]. Investigated aspects of software development expertise (SDExp) include programming experience [95], age [81],…”
BackgroundBuilding an Expert Recommender Chatbot
Cerezo, J., Kubelka, J., Robbes, R., & Bergel, A. (2019) · IEEE/ACM International Workshop on Bots in Software Engineering (BotSE) · doi:10.1109/botse.2019.00022 · Cited by 78
“Anvik and Murphy used data from bug reports [5]; Robbes and Roethlis-berger [19] used interaction data [14].”
MethodSelf-citationStudying Software Developer Expertise and Contributions in Stack Overflow and GitHub
Vadlamani, S. L., & Baysal, O. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00038 · Cited by 77
“However, in software engineering, it is much more difficult to find reliable and objective metrics to measure expertise, and this has been previously reported in the software engineering literature [2], [3].”
BackgroundLean Software Development in Action
Janes, A., & Succi, G. (2014) · Springer eBooks · doi:10.1007/978-3-642-00503-9 · Cited by 55
An empirical study of work fragmentation in software evolution tasks
Sanchez, H., Robbes, R., & González, V. M. (2015) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2015.7081835 · Cited by 41
“We investigated whether the duration of a task could be used to evaluate the accuracy of expertise metrics, with promising early results [19].”
BackgroundMethodSelf-citationEmpirical Analysis on CI/CD Pipeline Evolution in Machine Learning ProjectsPreprint
Rzig, D. E., Houerbi, A., Chavan, R. G., & Hassan, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.12199 · Cited by 27
Towards understanding how developers spend their effort during maintenance activities
Soh, Z., Khomh, F., Guéhéneuc, Y., & Antoniol, G. (2013) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2013.6671290 · Cited by 26
“Robbes and Röthlisberger [14] mined the interactions data for PDE and Mylyn and used different metrics to measure developers’ experience.”
BackgroundMethodInfluentialA Release-Aware Bug Triaging Method Considering Developers' Bug-Fixing Loads
Kashiwa, Y., & Ohira, M. (2020) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2019edp7152 · Cited by 23
“Previous studies have proposed methods to predict developers’ skills needed to fix given bugs [18]– [21] and to recommend developers qualified to fix given bugs [5], [13], [16], [22], [23].”
MethodLean Software Development in Action
Janes, A., & Succi, G. (2014) · Springer eBooks · doi:10.1007/978-3-642-00503-9_11 · Cited by 22
Metrics to quantify software developer experience: a systematic mapping
Brasil-Silva, R., & Siqueira, F. L. (2022) · ACM Symposium on Applied Computing · doi:10.1145/3477314.3507304 · Cited by 20
“Robbes and Rothlisberger [32] argue that a commit is only the final result of a development session and not the real activity of the developer.”
BackgroundInfluentialExamining the effects of developer familiarity on bug fixing
Wang, C., Li, Y., Chen, L., Huang, W.-C., Zhou, Y., & Xu, B. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110667 · Cited by 19
“studies [24, 25, 26] used the data gathered from developer IDE activities to complete the authorship data which can compute developer expertise.”
MethodThe Influence of Developer Quality on Software Fault-Proneness Prediction
Wu, Y., Yang, Y., Zhao, Y., Lu, H., Zhou, Y., & Xu, B. (2014) · IEEE International Conference on Software Security and Reliability Companion · doi:10.1109/sere.2014.14 · Cited by 16
Using Developer-Interaction Trails to Triage Change Requests
Zanjani, M. B., Kagdi, H., & Bird, C. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.16 · Cited by 14
“[38] used the interaction data from Mylyn to evaluate developer-expertise metrics based on time, which is the start time of a Mylyn-interaction session.”
MethodPlatform Independent Software Development Monitoring: Design of an Architecture
Bieliková, M., Polášek, I., Barla, M., Kuric, E., Rástočný, K., Tvarožek, J., & Lacko, P. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-04298-5_12 · Cited by 13
“Determining a developer’s expertise [13] in a software company allows for example managers and team leaders to look for specialists with desired abilities, form working teams or compare candidates for certain positions.”
BackgroundCynthia, S. T., Das, J. K., & Roy, B. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793377 · Cited by 11
“Thus, prior work recommends estimating developer experience using a broad set of contribution activities [5, 73].”
Ahasanuzzaman, M., Oliva, G. A., & Hassan, A. E. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10421-9 · Cited by 11
“RQ1: How accurately can KUREC recommend code reviewers in pull requests? We compare the performance of KUREC to that of four other baseline recommenders, namely: (i) a commit-frequency-based recommender (CF) [57], (ii) a review-frequency-based recommender (RF) [75], (iii) a modification-expertise-based recommender (ER…”
MethodResultNoise in Mylyn interaction traces and its impact on developers and recommendation systems
Soh, Z., Khomh, F., Guéhéneuc, Y., & Antoniol, G. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9529-x · Cited by 11
“In fact, some researchers computed the times spent on tasks by participants considering the start and end timestamps of the whole ITs, e.g., Robbes and Röthlisberger (2013), while others aggregated and cleaned the times spent for each event, e.g., Soh et al. (2013).”
BackgroundInfluentialSecurity expert recommender in software engineering
Bayati, S. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2892648 · Cited by 11
“Developer interactions to IDE log is used to define the developers’ expertise on a specific entity in SE project [13].”
MethodKnowledge-Oriented Models Based on Developer-Artifact and Developer-Developer Interactions
Lucas, E. M., Oliveira, T., Schneider, D., & Alencar, P. S. C. (2020) · IEEE Access · doi:10.1109/access.2020.3042429 · Cited by 10
“Robbes and Röthlisberger [43] infer the developers’ knowledge from their editing, selection, and commit interactions with artifacts over time.”
BackgroundPitfalls and guidelines for using time-based Git data
Flint, S. W., Chauhan, J., & Dyer, R. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10200-y · Cited by 9
Software Fault-Proneness Analysis based on Composite Developer-Module Networks
Lee, S.-Y., Wong, W. E., Li, Y., & Chu, W. C. (2021) · IEEE Access · doi:10.1109/access.2021.3128438 · Cited by 9
“Recently, discussions have risen with the impact of human activities on software quality [9], [10], [16], [20], [22], [26], [44]-[47], [72], [79], [103], [106], [118], [126], [127], [130].”
BackgroundMeasuring Developers' Expertise Based on Version Control Data
Fekete, A., Cserép, M., & Porkoláb, Z. (2021) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.23919/mipro52101.2021.9597103 · Cited by 8
Who Will be Interested in? A Contributor Recommendation Approach for Open Source Projects
Zhang, X., Wang, T., Yin, G., Yang, C., & Wang, H. (2017) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2017-067 · Cited by 8
“Robbes [7] used the interactions between developers to calculate their expertise.”
BackgroundThreats of Aggregating Software Repository Data
Robillard, M. P., Nassif, M., & McIntosh, S. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00009 · Cited by 8
“Hence, more recent work replaces or augments change-based EAs with activities derived from API usage [21], IDE builds [22], IDE interactions [23], [24], and participation in peer code review [25].”
BackgroundNoises in Interaction Traces Data and Their Impact on Previous Research Studies
Soh, Z., Drioul, T., Rappe, P.-A., Khomh, F., Guéhéneuc, Y., & Habra, N. (2015) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2015.7321209 · Cited by 8
“We present a preliminary study of the possible effects of the correction in Section V.”
BackgroundCode Ownership: The Principles, Differences, and Their Associations with Software Quality
Thongtanunam, P., & Tantithamthavorn, C. K. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00044 · Cited by 7
“Various data were used to approximate code ownership such as code commits [9, 10], lines of code [12, 34, 50], bug reports [6], code reviews [48], time [17, 41], organizational structure [35], and social coding platform like GitHub [15].”
MethodWork fragmentation in developer interaction data
Cruz, L. C., Sanchez, H., González, V. M., & Robbes, R. (2017) · Journal of Software: Evolution and Process · doi:10.1002/smr.1839 · Cited by 6
“We investigated whether the duration of a task could be used to evaluate the accuracy of expertise metrics, with promising early results [19].”
BackgroundMethodSelf-citationPoster: Discovering Code Dependencies by Harnessing Developer's Activity
Konôpka, M., Návrat, P., & Bieliková, M. (2015) · doi:10.5555/2819009.2819174 · Cited by 6
DRS: A Developer Risk Metric for Better Predicting Software Fault-Proneness
Lee, S.-Y., & Li, Y. (2015) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/tsa.2015.27 · Cited by 6
“It is also recognized that the performance of developers has an impact on software quality [4]–[8], [10], [22], [23], [32], [33], [35], [37], [40].”
BackgroundEffective assignment and assistance to software developers and reviewers
Zanjani, M. B. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2983960 · Cited by 5
Poster: Discovering Code Dependencies by Harnessing Developer's Activity
Konôpka, M., Návrat, P., & Bieliková, M. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.255 · Cited by 4
“Much of important knowledge about source code is not present in its contents but remembered by developers themselves [2, 5, 6].”
BackgroundMethodDeveloping analytics models for software project management
Choetkiertikul, M. (2018) · Research Online (University of Wollongong) · Cited by 4
Mining expertise of developers from software repositories
Otoom, A. F., Hammad, M., Hijazi, H., & Hammad, M. (2020) · International Journal of Computer Applications in Technology · doi:10.1504/ijcat.2020.106581 · Cited by 3
Recommending Open Source Software Projects to DevelopersPreprint
Bayati, S., Tripathi, A., & Bapna, R. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3642052 · Cited by 1
“Further, linear approach has been commonly used in the literature (Robbes and Röthlisberger 2013) because it performs well.”
MethodTackling DevOps and CI within ML projects
Rzig, D. E. (2024) · Deep Blue (University of Michigan) · doi:10.7302/25170 · Cited by 1
Improving Program Comprehension and Recommendation Systems Using Developers' Context
Soh, Z. (2015) · PolyPublie (École Polytechnique de Montréal)
Mining Java packages for developer profiles : an exploratory study
Ramadani, J., & Wagner, S. (2017) · OPUS Publication Server of the University of Stuttgart (University of Stuttgart) · doi:10.18419/opus-9020
dos Santos, E. W. (2014) · Institutional Repository of the Federal Technological University of Paraná (RIUT) (Federal University of Technology – Paraná)
Interaction-Based Creation and Maintenance of Continuously Usable Trace Links
Hübner, P. (2020) · heiDOK (Heidelberg University) · doi:10.11588/heidok.00027859
Um modelo para a estimação da dívida técnica baseado em produtividade
de Jesus, J. S. (2019) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.45.2019.tde-23012020-154212
Silva, R. B. (2020) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.3.2020.tde-20052021-130141
Understanding and Predicting Software Developer Expertise in Stack Overflow and GitHub
Vadlamani, S. (2021) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2021-14475
Integration of code comprehension support into the software development process
Fekete, A. (2024) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2024.019
A Survey of Development Approaches for StackOverflow-Style Platform using Cloud Computing
Madde, V., Mali, Y., Joshi, S., & Tathe, P. B. (2024) · Intelligent Systems and Machine Learning Conference (ISML) · doi:10.1109/isml60050.2024.11007435
Developer Interaction Across Experience Levels: A Study of Three Systems
Roychoudhuri, R., Karar, S., Datta, S., & Majumder, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-96-8309-3_4
Gestion des bibliothèques tierces dans un contexte de maintenance logicielle
Teyton, C. (2014) · theses.fr (ABES) · doi:10.70675/25c754d5z7b6cz4088zbabazf00f0642a027
Cynthia, S. T., Widyasari, R., Roy, B., Zhang, T., & Lo, D. (2026) · arXiv.org · doi:10.48550/arxiv.2607.21997 · Cited by 1
“However, commit-based measures have limitations, as they can reflect individual development styles [43].”
Metadata
- Authors
- Robbes, R., & Rothlisberger, D.
- Affiliations
- University of Chile (CL)
- ORCID
- Robbes, Romain
- Venue
- 2013 10th Working Conference on Mining Software Repositories (MSR)
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 297-300
- OpenAlex
- W2067096279
- Cited by
- 36
- References
- 14
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 1 · 2022: 1 · 2021: 3 · 2020: 7 · 2019: 2 · 2018: 2 · 2017: 3 · 2016: 2 · 2015: 7 · 2014: 4 · 2013: 1
- FWCI
- 3.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion)
- Open Research (University of Surrey) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Keywords
- developer expertise, development time, metric evaluation, software engineering tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.