Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 22

The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.

Understanding and improving artifact sharing in software engineering research

Timperley, Christopher S., Herckis, Lauren, Le Goues, Claire, Hilton, Michael·2021·Empirical Software Engineering
DOI 10.1007/s10664-021-09973-5
Open this paper in CoChat — free account

Abstract

In recent years, many software engineering researchers have begun to include artifacts alongside their research papers. Ideally, artifacts, including tools, benchmarks, and data, support the dissemination of ideas, provide evidence for research claims, and serve as a starting point for future research. However, in practice, artifacts suffer from a variety of issues that prevent the realization of their full potential. To help the software engineering community realize the full potential of artifacts, we seek to understand the challenges involved in the creation, sharing, and use of artifacts. To that end, we perform a mixed-methods study including a survey of artifacts in software engineering publications, and an online survey of 153 software engineering researchers. By analyzing the perspectives of artifact creators, users, and reviewers, we identify several high-level challenges that affect the quality of artifacts including mismatched expectations between these groups, and a lack of sufficient reward for both creators and reviewers. Using Diffusion of Innovations (DoI) as an analytical framework, we examine how these challenges relate to one another, and build an understanding of the factors that affect the sharing and success of artifacts. Finally, we make recommendations to improve the quality of artifacts based on our results and existing best practices.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10664-021-09973-5

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 11

  1. Association for Computing Machinery (2018) Artifact Review and Badging. (April 2018). https://www.acm.org/publications/policies/artifact-review-badging [Date Accessed: March 30th 2020]
  2. Association for Computing Machinery (2020) Software and Data Artifacts in the ACM Digital Library. (2020). https://www.acm.org/publications/artifacts [Date Accessed: July 29th 2020]
  3. Brooks A, Roper M, Wood M, Daly J, Miller J (2008) Replication’s role in software engineering. Springer, London, pp 365–379
  4. Carver JC (2010) Towards reporting guidelines for experimental replications: A proposal. In: International workshop on replication in empirical software engineering research (RESER ’10)
  5. Di Cosmo R, Zacchiroli S (2017) Software heritage: Why and how to preserve software source code. In: International conference on digital preservation (iPRES ’17)
  6. Creswell JW, Clark VLP (2017) Designing and conducting mixed methods research. Sage Publications
  7. Herckis L (2018) Passing the baton: Digital literacy and sustained implementation of elearning technologies. Current Issues in Emerging eLearning 5(1):4
  8. Rougier NP, Hinsen K, Alexandre F, Arildsen T, Barba LA, Benureau FCY, Titus Brown C, de Buyl P, Caglayan O, Davison AP, Delsuc M, Detorakis G, Diem AK, Drix D, Enel P, Girard B, Guest O, Hall MG, Henriques RN, Hinaut X, Jaron KS, Khamassi M, Klein A, Manninen T, Marchesi P, McGlinn D, Metzner C, Petchey OL, Plesser HE, Poisot T, Ram K, Ram Y, Roesch EB, Rossant C, Rostami V, Shifman A, Stachelek J, Stimberg M, Stollmeier F, Vaggi F, Viejo G, Vitay J, Vostinar AE, Yurchak R, Zito T (2017) Sustainable computational science: the ReScience initiative. PeerJ Comput Sci 3(12):1–8
  9. Stodden V (2009a) Enabling reproducible research: Open licensing for scientific innovation. Int J Commun Law Pol 13:1–55
  10. Teshima CG, Cleary B, Singer L (2013) A study of innovation diffusion through link sharing on stack overflow. In: Working conference on mining software repositories (MSR ’13), pp 81–84
  11. Wright RE, Palmar JC, Kavanaugh DC (1995) The importance of promoting stakeholder acceptance of educational innovations. Education 115(4):628–633

Cited by

See all 22

Metadata

Authors
Timperley, C. S., Herckis, L., Le Goues, C., & Hilton, M.
Venue
Empirical Software Engineering
Publisher
Springer Science and Business Media LLC
Published
2021-05-11
Volume
26
Issue
4
ISSN
  • 1382-3256
  • 1573-7616
DOI
10.1007/s10664-021-09973-5
Cited by
22
References
45
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.