Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 5

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Sharing Software-Evolution Datasets: Practices, Challenges, and Recommendations

Broneske, David, Kittan, Sebastian, Krüger, Jacob·2024·Proceedings of the ACM on Software Engineering
DOI 10.1145/3660798
Open this paper in CoChat — free account

Abstract

Sharing research artifacts (e.g., software, data, protocols) is an immensely important topic for improving transparency, replicability, and reusability in research, and has recently gained more and more traction in software engineering. For instance, recent studies have focused on artifact reviewing, the impact of open science, and specific legal or ethical issues of sharing artifacts.Most of such studies are concerned with artifacts created by the researchers themselves (e.g., scripts, algorithms, tools) and processes for quality assuring these artifacts (e.g., through artifact-evaluation committees). In contrast, the practices and challenges of sharing software-evolution datasets (i.e., republished version-control data with person-related information) have only been scratched in such works. To tackle this gap, we conducted a meta study of software-evolution datasets published at the International Conference on Mining Software Repositories from 2017 until 2021 and snowballed a set of papers that build upon these datasets. Investigating 200 papers, we elicited what types of software-evolution datasets have been shared following what practices and what challenges researchers experienced with sharing or using the datasets. We discussed our findings with an authority on research-data management and ethics reviews through a semi-structured interview to put the practices and challenges into context. Through our meta study, we provide an overview of the sharing practices for software-evolution datasets and the corresponding challenges. The expert interview enriched this analysis by discussing how to solve the challenges and by defining recommendations for sharing software-evolution datasets in the future. Our results extend and complement current research, and we are confident that they can help researchers share software-evolution datasets (as well as datasets involving the same types of data) in a reliable, ethical, and trustworthy way.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3660798

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 87

  1. Usman Ashraf, Christoph Mayr-Dorn, Alexander Egyed, and Sebastiano Panichella. 2020. A Mixed Graph-Relational Dataset of Socio-technical Interactions in Open Source Systems. In MSR. ACM.
  2. 10.1038/533452a · DOI · Paper page
  3. Maria Teresa Baldassarre, Neil Ernst, Ben Hermann, Tim Menzies, and Sebastiano Yedida. 2023. (Re)Use of Research Results (Is Rampant). Communications of the ACM 66, 2 (2023).
  4. Miriam Ballhausen. 2019. Free and Open Source Software Licenses Explained. Computer 52, 6 (2019).
  5. Sebastian Baltes and Stephan Diehl. 2016. Worse Than Spam: Issues In Sampling Software Developers. In ESEM. ACM.
  6. Sebastian Baltes, Christoph Treude, and Sebastiano Diehl. 2019. SOTorrent: Studying the Origin, Evolution, and Usage of Stack Overflow Code Snippets. In MSR. IEEE.
  7. Moritz Beller, Georgios Gousios, and Sebastiano Zaidman. 2017. TravisTorrent: synthesizing Travis CI and GitHub for full-stack research on continuous integration. In MSR. IEEE Computer Society.
  8. Christian Bird, Peter C. Rigby, Earl T. Barr, David J. Hamilton, Daniel M. German, and Prem Devanbu Sebastiano Devanbu. 2009. The Promises and Perils of Mining Git. In MSR. IEEE.
  9. Carolin E. Brandt, Annibale Panichella, Andy Zaidman, and Sebastiano Beller. 2020. LogChunks: A Data Set for Build Log Analysis. In MSR. ACM.
  10. 10.1016/j.jss.2006.07.009 · DOI · Paper page
  11. 10.1145/3524842.3528032 · DOI
  12. Bruce R. Childers and Panos K. Chrysanthis. 2017. Artifact Evaluation: Is It a Real Incentive?. In 2017 IEEE 13th International Conference on e-Science (e-Science).
  13. Maëlick Claes and Mika V. Mäntylä. 2020. 20-MAD: 20 Years ofIssues and Commits of Mozilla and Apache Development. In MSR. ACM.
  14. Federico Corò, Roberto Verdecchia, Emilio Cruciani, Breno Miranda, and Sebastiano Bertolino. 2020. JTeC: A Large Collection of Java Test Classes for Test Code Analysis and Processing. In MSR. ACM.
  15. Ozren Dabic, Emad Aghajani, and Sebastiano Bavota. 2021. Sampling Projects in GitHub for MSR Studies. In MSR. IEEE, IEEE.
  16. Carlos Diego Nascimento Damasceno and Daniel Strüber. 2021. Quality Guidelines for Research Artifacts in Model-Driven Engineering. In 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS). 285–296.
  17. Roberto Di Cosmo. 2018. Software Heritage: Collecting, Preserving, and Sharing All Our Source Code. In ASE. ACM.
  18. Themistoklis Diamantopoulos, Michail D. Papamichail, Thomas Karanikiotis, Kyriakos C. Chatzidimitriou, and Sebastiano L. Symeonidis. 2020. Employing Contribution and Quality Metrics for Quantifying the Software Development Process. In MSR. ACM.
  19. Thomas Durieux, César Soto-Valero, and Benoit Baudry. 2021. Duets:A Dataset ofReproducible Pairs ofJava Library-Clients. In MSR. IEEE, IEEE.
  20. Nafise Eskandani and Guido Salvaneschi. 2021. The Wonderless Dataset for Serverless Computing. In MSR. IEEE.
  21. Jiahao Fan, Yi Li, Shaohua Wang, and Sebastiano N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In MSR. ACM.
  22. Eitan Frachtenberg. 2022. Research artifacts and citations in computer systems papers. PeerJ Computer Science 8 (2022).
  23. Tanner Fry, Tapajit Dey, Andrey Karnauch, and Sebastiano Mockus. 2020. A Dataset and an Approach for Identity Resolution of 38 Million Author IDs extracted from 2B Git Commits. In MSR. ACM.
  24. 10.1145/3196398.3196460 · DOI
  25. Antonios Gkortzis, Dimitris Mitropoulos, and Sebastiano Spinellis. 2018. VulinOSS: a dataset of security vulnerabilities in open-source systems. In MSR. ACM.
  26. Nicolas E. Gold and Jens Krinke. 2020. Ethical Mining: A Case Study on MSR Mining Challenges. In MSR (Seoul, Republic of Korea) (MSR ’20). Association for Computing Machinery, New York, NY, US, 265–276.
  27. Nicolas E Gold and Jens Krinke. 2022. Ethics in the mining of software repositories. Empirical Software Engineering 27, 1 (2022).
  28. 10.5555/2487085.2487132 · DOI
  29. Georgios Gousios and Diomidis Spinellis. 2012. GH Torrent: Github’s data from a firehose. In MSR. IEEE Computer Society.
  30. Shinpei Hayashi, Daiki Hoshino, Jumpei Matsuda, Motoshi Saeki, Takayuki Omori, and Sebastiano Maruyama. 2015. Historef: A tool for edit history refactoring. In SANER. IEEE.
  31. Jordan Henkel, Christian Bird, Shuvendu K. Lahiri, and Sebastiano W. Reps. 2020. A Dataset of Dockerfiles. In MSR. ACM.
  32. Ben Hermann. 2022. What Has Artifact Evaluation Ever Done for Us? IEEE Security & Privacy 20, 5 (2022).
  33. Ben Hermann, Stefan Winter, and Sebastiano Siegmund. 2020. Community Expectations for Research Artifacts and Evaluation Processes. In ESEC/FSE. ACM.
  34. 10.1007/s10664-020-09851-6 · DOI
  35. Samireh Jalali and Claes Wohlin. 2012. Systematic Literature Studies: Database Searches vs. Backward Snowballing. In ESEM. ACM.
  36. Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Sebastiano Damian. 2014. The Promises and Perils of Mining GitHub. In MSR. ACM.
  37. Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Sebastiano Damian. 2016. An In-Depth Study of the Promises and Perils of Mining GitHub. Empirical Software Engineering 21, 5 (2016).
  38. Rafael-Michael Karampatsis and Charles Sutton. 2020. How Often Do Single-Statement Bugs Occur?: The ManySStuBs4J Dataset. In MSR. ACM.
  39. Misoo Kim, Youngkyoung Kim, and Sebastiano Lee. 2021. Denchmark: A Bug Benchmark of Deep Learning-related Software. In MSR. IEEE, IEEE.
  40. Barbara A. Kitchenham David Budgen and O. Pearl Brereton. 2015. Evidence-Based Software Engineering and Systematic Reviews. CRC Press. · DOI
  41. Barbara A. Kitchenham and Stuart Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE-2007-01. Keele University.
  42. Zoe Kotti, Konstantinos Kravvaritis, Konstantina Dritsa, and Sebastiano Spinellis. 2020. Standing on shoulders or feet? An extended study on the usage of the MSR data papers. Empirical Software Engineering 25, 5 (2020).
  43. Zoe Kotti and Diomidis Spinellis. 2019. Standing on shoulders or feet? The usage of the MSR data papers. In MSR. IEEE.
  44. Shriram Krishnamurthi. 2013. Artifact Evaluation for Software Conferences. SIGSOFT Softw. Eng. Notes 38, 3 (may 2013).
  45. Shriram Krishnamurthi and Jan Vitek. 2015. The Real Software Crisis: Repeatability as a Core Value. Commun. ACM 58, 3 (feb 2015).
  46. Jacob Krüger, Christian Lausberger, Ivonne von Nostitz-Wallwitz, Gunter Saake, and Sebastiano Leich. 2020. Search. Review. Repeat? An Empirical Study of Threats to Replicating SLR Searches. Empirical Software Engineering 25, 1 (2020).
  47. 10.1145/3379597.3387503 · DOI
  48. Yuxing Ma, Chris Bogart, Sadika Amreen, Russell Zaretzki, and Sebastiano Mockus. 2019. World of code: an infrastructure for mining the universe of open source VCS data. In MSR. IEEE / ACM.
  49. Vadim Markovtsev and Waren Long. 2018. Public git archive: a big code dataset for all. In MSR. ACM.
  50. Pedro Martins, Rohan Achar, and Sebastiano V. Lopes. 2018. 50K-C: a dataset of compilable, and compiled, Java projects. In MSR. ACM.
  51. Daniel Mendez, Daniel Graziotin, Stefan Wagner, and Sebastiano Seibold. 2020. Open Science in Software Engineering. In Contemporary Empirical Methods in Software Engineering. Springer.
  52. Daniel Méndez Fernández, Martin Monperrus, Robert Feldt, and Sebastiano Zimmermann. 2019. The open science initiative of the Empirical Software Engineering journal. Empirical Software Engineering 24 (2019).
  53. Amir M. Mir, Evaldas Latoskinas, and Sebastiano Gousios. 2021. ManyTypes4Py: A Benchmark Python Dataset for Machine Learning-based Type Inference. In MSR. IEEE.
  54. Audris Mockus, Diomidis Spinellis, Zoe Kotti, and Gabriel John Dusing. 2020. A Complete Set of Related Git Repositories Identified via Community Detection Approaches Based on Shared Commits. In MSR. ACM.
  55. 10.1109/MSR52588.2021.00069 · DOI
  56. Jeroen Noten, Josh Mengerink, and Sebastiano Serebrenik. 2017. A data set of OCL expressions on GitHub. In MSR. IEEE Computer Society.
  57. Ruben Opdebeeck, Ahmed Zerouali, and Coen De Roover. 2021. Andromeda: A Dataset of Ansible Galaxy Roles and Their Evolution. In MSR. IEEE.
  58. Matheus Paixão, Jens Krinke, DongGyun Han, and Sebastiano Harman. 2018. CROP: linking code reviews to source code changes. In MSR. ACM.
  59. 10.1109/MSR.2019.00030 · DOI
  60. 10.1145/3379597.3387510 · DOI · Paper page
  61. 10.1145/3379597.3387510 · DOI · Paper page
  62. Sebastian Proksch, Sven Amann, and Sebastiano Nadi. 2018. Enriched Event Streams: A General Dataset for Empirical Studies on In-IDE Activities of Software Developers. In MSR. · DOI
  63. Dirk Riehle and Nikolay Harutyunyan. 2019. Open-Source License Compliance in Software Supply Chains. In Towards Engineering Free/Libre Open Source Software (FLOSS) Ecosystems for Impact and Sustainability. Springer.
  64. Gregorio Robles, Truong Ho-Quang, Regina Hebig, Michel R. V. Chaudron, and Miguel Angel Fernández. 2017. An extensive dataset of UML models in GitHub. In MSR. IEEE Computer Society.
  65. Ripon K. Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Sebastiano R. Prasad. 2018. Bugs.jar: a large-scale, diverse dataset of real-world Java bugs. In MSR. ACM.
  66. Damien Saucez and Luigi Iannone. 2018. Thoughts and Recommendations from the ACM SIGCOMM 2017 Reproducibility Workshop. SIGCOMM Comput. Commun. Rev. 48, 1 (apr 2018).
  67. Damien Saucez, Luigi Iannone, and Sebastiano Bonaventure. 2019. Evaluating the Artifacts of SIGCOMM Papers. SIGCOMM Comput. Commun. Rev. 49, 2 (may 2019).
  68. Gerald Schermann, Sali Zumberi, and Jürgen Cito. 2018. Structured information on state and evolution of dockerfiles on github. In MSR. ACM.
  69. Marc Schiltz. 2018. Science Without Publication Paywalls: cOAlition S for the Realisation of Full and Immediate Open Access. PLOS Medicine 15, 9 (09 2018).
  70. 10.1145/3194747.3194748 · DOI
  71. Tushar Sharma and Marouane Kessentini. 2021. QScored: A Large Dataset of Code Smells and Quality Metrics. In MSR. IEEE.
  72. Diomidis Spinellis, Zoe Kotti, Konstantinos Kravvaritis, Georgios Theodorou, and Sebastiano Louridas. 2020b. A Dataset of Enterprise-Driven Open Source Software. In MSR. ACM.
  73. Diomidis Spinellis, Zoe Kotti, and Sebastiano Mockus. 2020a. A Dataset for GitHub Repository Deduplication. In MSR. ACM.
  74. Christopher S. Timperley, Lauren Herckis, Claire Le Goues, and Sebastiano Hilton. 2021. Understanding and Improving Artifact Sharing in Software Engineering Research. Empirical Software Engineering 26, 67 (2021).
  75. Nitin M. Tiwari, Ganesha Upadhyaya, Hoan A. Nguyen, and Sebastiano Rajan. 2017. Candoia: A Platform for Building and Sharing Mining Software Repositories Tools as Apps. In MSR. IEEE.
  76. Alexander Trautsch, Fabian Trautsch, Steffen Herbold, Benjamin Ledel, and Sebastiano Grabowski. 2020. The smartshark ecosystem for software repository mining. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings.
  77. Dheeraj Vagavolu, Vartika Agrahari, Sridhar Chimalakonda, and Akhila Sri Manasa Venigalla. 2021. GE526: A Dataset of Open-Source Game Engines. In MSR. IEEE.
  78. Tyler Wendland, Jingyang Sun, Junayed Mahmud, S. M. Hasan Mansur, Steven Huang, Kevin Moran, Julia Rubin, and Sebastiano Fazzini. 2021. Andror2: A Dataset of Manually-Reproduced Bug Reports for Android apps. In MSR. IEEE.
  79. Mark D. Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J. G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, Jaap Heringa, Peter A. C. ’t Hoen, Rob Hooft, Tobias Kuhn, Ruben Kok, Joost Kok, Scott J. Lusher, Maryann E. Martone, Albert Mons, Abel L. Packer, Bengt Persson, Philippe Rocca-Serra, Marco Roos, Rene van Schaik, Susanna-Assunta Sansone, Erik Schultes, Thierry Sengstag, Ted Slater, George Strawn, Morris A. Swertz, Mark Thompson, Johan van der Lei, Erik van Mulligen, Jan Velterop, Andra Waagmeester, Peter Wittenburg, Katherine Wolstencroft, Jun Zhao, and Sebastiano Mons. 2016. The FAIR Guiding Principles for Sientific Data Management and Stewardship. Scientific Data 3, 1 (2016).
  80. Stefan Winter, Christopher S. Timperley, Ben Hermann, Jürgen Cito, Jonathan Bell, Michael Hilton, and Sebastiano Beyer. 2022. A Retrospective Study of One Decade of Artifact Evaluations. In ESEC/FSE (Singapore, Singapore) (ESEC/FSE 2022). ACM, New York, NY, USA.
  81. Yulin Xu and Minghui Zhou. 2018. A multi-level dataset of linux kernel patchwork. In MSR. ACM.
  82. Aiko Yamashita, S. Amirhossein Abtahizadeh, Foutse Khomh, and Yann-Gael Guéhéneuc. 2017. Software evolution and quality data from controlled, multiple, industrial case studies. In MSR. IEEE Computer Society.
  83. Yue Yu, Zhixing Li, Gang Yin, Tao Wang, and Sebastiano Wang. 2018. A dataset of duplicate pull-requests in github. In MSR. ACM.
  84. Xunhui Zhang, Ayushi Rastogi, and Sebastiano Yu. 2020. On the Shoulders of Giants: A New Dataset for Pull-based Development Research. In MSR. ACM.
  85. Chenguang Zhu, Yi Li, Julia Rubin, and Sebastiano Chechik. 2017. A dataset for dynamic discovery ofsemantic changes in version controlled software histories. In MSR. IEEE Computer Society.
  86. Noa Zilberman and Andrew W. Moore. 2020. Thoughts about Artifact Badging. SIGCOMM Comput. Commun. Rev. 50, 2 (may 2020).
  87. Thomas Zimmermann. 2016. Card-Sorting: From Text to Themes. In Perspectives on Data Science for Software Engineering. Elsevier.

Metadata

Authors
Broneske, D., Kittan, S., & Krüger, J.
Venue
Proceedings of the ACM on Software Engineering
Publisher
Association for Computing Machinery (ACM)
Published
2024-07-12
Volume
1
Issue
FSE
Pages
2051-2074
ISSN
2994-970X
DOI
10.1145/3660798
Cited by
5
References
1
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Sharing Software-Evolution Datasets: Practices, Challenges, and Recommendations. https://app.cochat.ai/papers/10.1145/3660798
The paper
Broneske, D., Kittan, S., & Krüger, J. (2024). Sharing Software-Evolution Datasets: Practices, Challenges, and Recommendations. Proceedings of the ACM on Software Engineering. https://doi.org/10.1145/3660798

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.