Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 10

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.

Research artifacts in software engineering publications: Status and trends

Liu, Mugeng, Huang, Xiaolong, He, Wei, Xie, Yibing, Zhang, Jie M., Jing, Xiang, Chen, Zhenpeng, Ma, Yun·2024·Journal of Systems and Software
DOI 10.1016/j.jss.2024.112032
Open this paper in CoChat — free account

Abstract

The Software Engineering (SE) community has been embracing the open science policy and encouraging researchers to disclose artifacts in their publications. However, the status and trends of artifact practice and quality remain unclear, lacking insights on further improvement. In this paper, we present an empirical study to characterize the research artifacts in SE publications. Specifically, we manually collect 1,487 artifacts from all 2,196 papers published in top-tier SE conferences (ASE, FSE, ICSE, and ISSTA) from 2017 to 2022. We investigate the common practices (e.g., URL location and format, storage websites), maintenance activities (e.g., last update time and URL validity), popularity (e.g., the number of stars on GitHub and characteristics), and quality (e.g., documentation and code smell) of these artifacts. Based on our analysis, we reveal a rise in publications providing artifacts. The usage of Zenodo for sharing artifacts has significantly increased. However, artifacts stored in GitHub tend to receive few stars, indicating a limited influence on real-world SE applications. We summarize the results and provide suggestions to different stakeholders in conjunction with current guidelines.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jss.2024.112032

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 2

  1. ACM, 2020
  2. Jones, 2021, Robustifying links to combat reference rot, Code4Lib J., 50

Metadata

Authors
Liu, M., Huang, X., He, W., Xie, Y., Zhang, J. M., Jing, X., Chen, Z., & Ma, Y.
Venue
Journal of Systems and Software
Publisher
Elsevier BV
Published
2024-07
Volume
213
Pages
112032
ISSN
0164-1212
DOI
10.1016/j.jss.2024.112032
Cited by
10
References
22
Language
en
Licence
Funders
  • National Key Research and Development Program of China (2021YFF0901100)
  • European Research Council (741278)
  • Peking University

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.