The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Understanding and improving artifact sharing in software engineering research
Abstract
In recent years, many software engineering researchers have begun to include artifacts alongside their research papers. Ideally, artifacts, including tools, benchmarks, and data, support the dissemination of ideas, provide evidence for research claims, and serve as a starting point for future research. However, in practice, artifacts suffer from a variety of issues that prevent the realization of their full potential. To help the software engineering community realize the full potential of artifacts, we seek to understand the challenges involved in the creation, sharing, and use of artifacts. To that end, we perform a mixed-methods study including a survey of artifacts in software engineering publications, and an online survey of 153 software engineering researchers. By analyzing the perspectives of artifact creators, users, and reviewers, we identify several high-level challenges that affect the quality of artifacts including mismatched expectations between these groups, and a lack of sufficient reward for both creators and reviewers. Using Diffusion of Innovations (DoI) as an analytical framework, we examine how these challenges relate to one another, and build an understanding of the factors that affect the sharing and success of artifacts. Finally, we make recommendations to improve the quality of artifacts based on our results and existing best practices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999) · American Journal of Public Health · doi:10.2105/ajph.89.9.1322 · Cited by 7,217
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372
An introduction to implementation science for the non-specialist
Bauer, M. S., Damschroder, L., Hagedorn, H., Smith, J., & Kilbourne, A. M. (2015) · BMC Psychology · doi:10.1186/s40359-015-0089-9 · Cited by 1,611
ORGANIZATIONAL INNOVATION: REVIEW, CRITIQUE AND SUGGESTED RESEARCH DIRECTIONS*
Wolfe, R. A. (1994) · Journal of Management Studies · doi:10.1111/j.1467-6486.1994.tb00624.x · Cited by 1,432
Frambach, R. T., & Schillewaert, N. (2002) · Journal of Business Research · doi:10.1016/s0148-2963(00)00152-1 · Cited by 1,270
Positioning and Power in Academic Publishing: Players, Agents and Agendas
Loizides, F., & Schmidt, B. (2016) · ORCA Online Research @Cardiff (Cardiff University) · Cited by 1,027
Zerhouni, E. (2003) · Science · doi:10.1126/science.1091867 · Cited by 918
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Implementation of Electronic Data Interchange: An Innovation Diffusion Perspective
Premkumar, G., Ramamurthy, K., & Nilakanta, S. (1994) · Journal of Management Information Systems · doi:10.1080/07421222.1994.11518044 · Cited by 767
Quality in qualitative research
Kitto, S. C., Chesters, J., & Grbich, C. (2008) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2008.tb01595.x · Cited by 728 · cited 4×
“We collected data through multiple sources and interpreted our results by using multiple conceptual frames (Kitto et al. 2008).”
Influential
Research works printed but not linked: 11
- Association for Computing Machinery (2018) Artifact Review and Badging. (April 2018). https://www.acm.org/publications/policies/artifact-review-badging [Date Accessed: March 30th 2020]
- Association for Computing Machinery (2020) Software and Data Artifacts in the ACM Digital Library. (2020). https://www.acm.org/publications/artifacts [Date Accessed: July 29th 2020]
- Brooks A, Roper M, Wood M, Daly J, Miller J (2008) Replication’s role in software engineering. Springer, London, pp 365–379
- Carver JC (2010) Towards reporting guidelines for experimental replications: A proposal. In: International workshop on replication in empirical software engineering research (RESER ’10)
- Di Cosmo R, Zacchiroli S (2017) Software heritage: Why and how to preserve software source code. In: International conference on digital preservation (iPRES ’17)
- Creswell JW, Clark VLP (2017) Designing and conducting mixed methods research. Sage Publications
- Herckis L (2018) Passing the baton: Digital literacy and sustained implementation of elearning technologies. Current Issues in Emerging eLearning 5(1):4
- Rougier NP, Hinsen K, Alexandre F, Arildsen T, Barba LA, Benureau FCY, Titus Brown C, de Buyl P, Caglayan O, Davison AP, Delsuc M, Detorakis G, Diem AK, Drix D, Enel P, Girard B, Guest O, Hall MG, Henriques RN, Hinaut X, Jaron KS, Khamassi M, Klein A, Manninen T, Marchesi P, McGlinn D, Metzner C, Petchey OL, Plesser HE, Poisot T, Ram K, Ram Y, Roesch EB, Rossant C, Rostami V, Shifman A, Stachelek J, Stimberg M, Stollmeier F, Vaggi F, Viejo G, Vitay J, Vostinar AE, Yurchak R, Zito T (2017) Sustainable computational science: the ReScience initiative. PeerJ Comput Sci 3(12):1–8
- Stodden V (2009a) Enabling reproducible research: Open licensing for scientific innovation. Int J Commun Law Pol 13:1–55
- Teshima CG, Cleary B, Singer L (2013) A study of innovation diffusion through link sharing on stack overflow. In: Working conference on mining software repositories (MSR ’13), pp 81–84
- Wright RE, Palmar JC, Kavanaugh DC (1995) The importance of promoting stakeholder acceptance of educational innovations. Education 115(4):628–633
Cited by
Research artifacts in software engineering publications: Status and trends
Liu, M., Huang, X., He, W., Xie, Y., Zhang, J. M., Jing, X., Chen, Z., & Ma, Y. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112032 · Cited by 10
Guevara-Vega, C., Bernárdez, B., Cruz, M., Durán, A., Ruiz-Cortés, A., & Solari, M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112187 · Cited by 8
Metadata
- Authors
- Timperley, C. S., Herckis, L., Le Goues, C., & Hilton, M.
- Venue
- Empirical Software Engineering
- Published
- 2021-05-11
- Volume
- 26
- Issue
- 4
- ISSN
- 1382-3256
- 1573-7616
- Cited by
- 22
- References
- 45
- Language
- en
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Understanding and improving artifact sharing in software engineering research. https://app.cochat.ai/papers/10.1007/s10664-021-09973-5
- The paper
- Timperley, C. S., Herckis, L., Le Goues, C., & Hilton, M. (2021). Understanding and improving artifact sharing in software engineering research. Empirical Software Engineering. https://doi.org/10.1007/s10664-021-09973-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
