Community expectations for research artifacts and evaluation processes
The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Background. Artifact evaluation has been introduced into the software engineering and programming languages research community with a pilot at ESEC/FSE 2011 and has since then enjoyed a healthy adoption throughout the conference landscape. Objective. In this qualitative study, we examine the expectations of the community toward research artifacts and their evaluation processes. Method. We conducted a survey including all members of artifact evaluation committees of major conferences in the software engineering and programming language field since the first pilot and compared the answers to expectations set by calls for artifacts and reviewing guidelines. Results. While we find that some expectations exceed the ones expressed in calls and reviewing guidelines, there is no consensus on quality thresholds for artifacts in general. We observe very specific quality expectations for specific artifact types for review and later usage, but also a lack of their communication in calls. We also find problematic inconsistencies in the terminology used to express artifact evaluation’s most important purpose – replicability. Conclusion. We derive several actionable suggestions which can help to mature artifact evaluation in the inspected community and also to aid its introduction into other communities in computer science.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Cited by
On the reliability of coverage-based fuzzer benchmarking
Böhme, M., Szekeres, L., & Metzman, J. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510230 · Cited by 104
“Upload all artifacts to an open-access repository like Zenodo for long-term archival [29].”
BackgroundUnderstanding and improving artifact sharing in software engineering research
Timperley, C. S., Herckis, L., Le Goues, C., & Hilton, M. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09973-5 · Cited by 22
Research artifacts in software engineering publications: Status and trends
Liu, M., Huang, X., He, W., Xie, Y., Zhang, J. M., Jing, X., Chen, Z., & Ma, Y. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112032 · Cited by 10
Guevara-Vega, C., Bernárdez, B., Cruz, M., Durán, A., Ruiz-Cortés, A., & Solari, M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112187 · Cited by 8
Sharing Software-Evolution Datasets: Practices, Challenges, and Recommendations
Broneske, D., Kittan, S., & Krüger, J. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660798 · Cited by 5
Automated Table Reproduction via Code Generation
Baek, D., & Pradel, M. (2026) · arXiv · doi:10.48550/arxiv.2602.10046
Metadata
- Authors
- Hermann, B., Winter, S., & Siegmund, J.
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-07
- Pages
- 469-480
- OpenAlex
- W3099627437
- Cited by
- 40
- References
- 1
- FWCI
- 4.17
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.