Publish or perish, but do not forget your software artifacts
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Open-science initiatives have gained substantial momentum in computer science, and particularly in software-engineering research. A critical aspect of open-science is the public availability of artifacts (e.g., tools), which facilitates the replication, reproduction, extension, and verification of results. While we experienced that many artifacts are not publicly available, we are not aware of empirical evidence supporting this subjective claim. In this article, we report an empirical study on software artifact papers (SAPs) published at the International Conference on Software Engineering (ICSE), in which we investigated whether and how researchers have published their software artifacts, and whether this had scientific impact. Our dataset comprises 789 ICSE research track papers, including 604 SAPs (76.6 %), from the years 2007 to 2017. While showing a positive trend towards artifact availability, our results are still sobering. Even in 2017, only 58.5 % of the papers that stated to have developed a software artifact made that artifact publicly available. As we did find a small, but statistically significant, positive correlation between linking to artifacts in a paper and its scientific impact in terms of citations, we hope to motivate the research community to share more artifacts. With our insights, we aim to support the advancement of open science by discussing our results in the context of existing initiatives and guidelines. In particular, our findings advocate the need for clearly communicating artifacts and the use of non-commercial, persistent archives to provide replication packages.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploring Network Structure, Dynamics, and Function Using NetworkX
Hagberg, A. A., Schult, D. A., & Swart, P. J. (2008) · Proceedings of the Python in Science Conference · doi:10.25080/tcwv9851 · Cited by 7,477 · cited 1×
“…standard graph clustering algorithms, Spectral Clustering (von Luxburg 2007) and Girvan Newman Clustering (Girvan and Newman 2002), on our dataset, relying on the Scipy, Scikit-Learn , and NetworkX libraries for the implementations (Virtanen et al. 2020; Pedregosa et al. 2011; Hagberg et al. 2008).”
MethodExperimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460
The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles
Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., Farley, A., West, J., & Haustein, S. (2018) · PeerJ · doi:10.7717/peerj.4375 · Cited by 1,223
An introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700
Bibliometrics: Tracking Research Impact by Selecting the Appropriate Metrics
Agarwal, A., Durairajanayagam, D., Tatagari, S., Esteves, S., Harlev, A., Henkel, R., Roychoudhury, S., Homa, S., Puchalt, N., Ramasamy, R., et al. (2016) · Asian Journal of Andrology · doi:10.4103/1008-682x.171582 · Cited by 639
Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002541 · Cited by 500 · cited 1×
“To address this issue, we adapted the average relative citation (arc) metric (Hutchins et al. 2016; Piwowar et al. 2018).”
MethodAre Students Representatives of Professionals in Software Engineering Experiments?
Salman, I., Misirli, A. T., & Juristo, N. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.82 · Cited by 304
Liu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47 · cited 2×
“Only 8 % performed a solid validation, even though processes (Liu and Salganik 2019) and automatic support (Giannoulatou et al. 2014) for artifacts exist.”
BackgroundBridging the gap: SE technology transfer into practice: study design and preliminary results
Diebold, P., & Vetrò, A. (2014) · Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2652524.2652552 · Cited by 12
Cited by
Understanding and improving artifact sharing in software engineering research
Timperley, C. S., Herckis, L., Le Goues, C., & Hilton, M. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09973-5 · Cited by 22
Guevara-Vega, C., Bernárdez, B., Cruz, M., Durán, A., Ruiz-Cortés, A., & Solari, M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112187 · Cited by 8
Sharing Software-Evolution Datasets: Practices, Challenges, and Recommendations
Broneske, D., Kittan, S., & Krüger, J. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660798 · Cited by 5
Self-citationResearch artifacts in secondary studies: A systematic mapping in software engineering
Huotala, A., Kuutila, M., & Mäntylä, M. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107830 · Cited by 1
Metadata
- Authors
- Heumüller, R., Nielebock, S., Krüger, J., & Ortmeier, F.
- Venue
- Empirical Software Engineering
- Published
- 2020-10-08
- Volume
- 25
- Issue
- 6
- Pages
- 4585-4616
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W3091970108
- Cited by
- 42
- References
- 11
- FWCI
- 19.97
- Type
- book chapter
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Publish or perish, but do not forget your software artifacts. https://app.cochat.ai/papers/10.1007/s10664-020-09851-6
- The paper
- Heumüller, R., Nielebock, S., Krüger, J., & Ortmeier, F. (2020). Publish or perish, but do not forget your software artifacts. Empirical Software Engineering. https://doi.org/10.1007/s10664-020-09851-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.