The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Research artifacts for human-oriented experiments in software engineering: An ACM badges-driven structure proposal
Abstract
Context: The Open Science (OS) movement promotes the value of making public the research artifacts (datasets, analysis scripts, guidelines, etc.) used during empirical studies. OS is widely known in areas such as Medicine or Biology, where the process of sharing research artifacts is subject to strict protocols. Unfortunately, in Software Engineering (SE), this process is carried out in a non-systematic way, resulting in incomplete or inaccurate material shared by researchers, which hinders the reproducibility and replicability of empirical studies. Nevertheless, in recent years, it seems that the Empirical Software Engineering (ESE) community is embracing some of the proposed OS initiatives, such as the one proposed by the Association for Computing Machinery (ACM), which provides a badge system to evaluate the quality of a research artifact. This badge system has been adopted by several SE conferences as a method of assessing research artifacts. Aims: Focusing on human-oriented experiments (HOEs) in SE, whose research artifacts are more complex than those for computational experiments, this work applies Design Science Research (DSR) with a twofold purpose: (i) review the current status of HOEs research artifacts publication through evaluation of this practice in the most relevant ESE journals , and (ii) propose a structured outline for HOEs research artifacts driven by the aforementioned ACM badging policy. Method: Regarding the first purpose, we carried out a survey to analyze the current status of the publication of research artifacts considering relevant peer review journals and the quality of 106 research artifacts published in these journals with respect to the ACM badging policy. For the second purpose, an iterative process was carried out to review the content of 106 research artifacts research and their concordance with ACM badges, obtaining a structured scheme for HOEs research artifacts that has been validated through a detailed review of 12 research artifacts obtained from some of those of ACM badges in relevant SE conferences. In addition, we validated the proposal in the research artifacts of 2 of our own experiments. Results: Our survey reveals issues such as the 39,70% of journal studies making completely accessible their research artifacts; most of the analyzed research artifacts are incomplete; the most common repositories used in the ESE community to share the research artifacts are GitHub, institutional repositories , and Zenodo. On the other hand, the validated and structured research artifact outline consists of a list of ordered sections containing a set of artifacts, which can be mandatory or not to achieve a particular ACM badge. For its internal validation, several improvement iterations on the first release of the outline have been carried out based on the conference guidelines, the ACM badging policy, and other relevant proposals. Conclusions: Although the ESE community is making great efforts in standardization, review, and digital publishing related to OS, the availability and completeness of research artifacts can be improved. Our proposal for the elaboration of structured research artifact outline meets the requirements of HOEs in SE. Nevertheless, further research is needed not only to improve and externally validate it but also to disseminate its use among the research community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Design Science Research Methodology for Information Systems Research
Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007) · Journal of Management Information Systems · doi:10.2753/mis0742-1222240302 · Cited by 7,832
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Systematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239
The Logic of Scientific Discovery.
Rescher, N., Popper, K. R., & Hanson, N. R. (1960) · Philosophy and Phenomenological Research · doi:10.2307/2104331 · Cited by 2,475
Toward a new economics of science
Partha, D., & David, P. A. (1994) · Research Policy · doi:10.1016/0048-7333(94)01002-1 · Cited by 2,448
Experimentation in Software Engineering: An Introduction
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2011) · Lund University Publications (Lund University) · Cited by 2,214
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
The role of replications in Empirical Software Engineering
Shull, F. J., Carver, J. C., Vegas, S., & Juristo, N. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9060-1 · Cited by 315
Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., & Chatzigeorgiou, A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.10.006 · Cited by 305
Chenail, R. J. (2014) · The Qualitative Report · doi:10.46743/2160-3715/2011.1051 · Cited by 304
Open science is a research accelerator
Woelfle, M., Olliaro, P., & Todd, M. H. (2011) · Nature Chemistry · doi:10.1038/nchem.1149 · Cited by 260
A large-scale study on research code quality and execution
Trišović, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022) · Scientific Data · doi:10.1038/s41597-022-01143-6 · Cited by 167
Measuring and modeling programming experience
Siegmund, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2013) · Empirical Software Engineering · doi:10.1007/s10664-013-9286-4 · Cited by 147
Replicating software engineering experiments: addressing the tacit knowledge problem
Shull, F. J., Basili, V. R., Carver, J. C., Maldonado, J. C., Travassos, G. H., Mendonca, M., & Fabbri, S. (2003) · International Symposium on Empirical Software Engineering · doi:10.1109/isese.2002.1166920 · Cited by 143
An empirical evaluation of three defect-detection techniques
Kamsties, E., & Lott, C. M. (1995) · Lecture notes in computer science · doi:10.1007/3-540-60406-5_25 · Cited by 140
Reinventing Discovery: The New Era of Networked Science
Vella, L. (2012) · Issues in Science and Technology Librarianship · doi:10.29173/istl1552 · Cited by 114
2011 IEEE 19th International Requirements Engineering Conference
(2011) · doi:10.1109/re17302.2011 · Cited by 103
Open Science in Software Engineering
Mendez, D., Graziotin, D., Wagner, S., & Seibold, H. (2020) · Springer eBooks · doi:10.1007/978-3-030-32489-6_17 · Cited by 81
Proceedings 2021 IEEE International Conference on Software Maintenance and Evolution
(2021) · IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme52107.2021.00001 · Cited by 81
The European Organization for Nuclear Research
WEINTROUB, S. (1963) · Nature · doi:10.1038/197853a0 · Cited by 65
An empirical evaluation of defect detection techniques
Roper, M., Wood, M., & Miller, J. (1997) · Information and Software Technology · doi:10.1016/s0950-5849(97)00028-1 · Cited by 63
Empirical Software Engineering and Verification
Meyer, B., & Nordio, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-25231-0 · Cited by 61
The role and value of replication in empirical software engineering results
Shepperd, M., Ajienka, N., & Counsell, S. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.01.006 · Cited by 60
Fakhoury, S., Roy, D., Ma, Y., Arnaoudova, V., & Adesope, O. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09751-4 · Cited by 51
Publish or perish, but do not forget your software artifacts
Heumüller, R., Nielebock, S., Krüger, J., & Ortmeier, F. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09851-6 · Cited by 43
ACM SIGSOFT Empirical Standards.
Ralph, P., Baltes, S., Bianculli, D., Dittrich, Y., Felderer, M., FELDT, R. H., Filieri, A., Furia, C. A., Graziotin, D., He, P., et al. (2020) · Spiral (Imperial College London) · Cited by 40
Community expectations for research artifacts and evaluation processes
Hermann, B., Winter, S., & Siegmund, J. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409767 · Cited by 40
Fernández-Sáez, A. M., Genero, M., Caivano, D., & Chaudron, M. R. V. (2014) · Empirical Software Engineering · doi:10.1007/s10664-014-9354-4 · Cited by 39
The open science initiative of the Empirical Software Engineering journal
Méndez, D., Monperrus, M., FELDT, R. H., & Zimmermann, T. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09712-x · Cited by 35
Paradigms for experimentation and empirical studies in software engineering
Basili, V. R., & Selby, R. W. (1991) · Reliability Engineering & System Safety · doi:10.1016/0951-8320(91)90053-a · Cited by 33
A first look at developers’ live chat on Gitter
Shi, L., Chen, X., Yang, Y., Jiang, H., Jiang, Z., Niu, N., & Wang, Q. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468562 · Cited by 33
Badges to Acknowledge Open PracticesPreprint
Blohowiak, B. B., Cohoon, J., de‐Wit, L., Eich, E., Farach, F. J., Hasselman, F., Holcombe, A. O., Humphreys, M., Lewis, M., Nosek, B. A., et al. (n.d.) · OSF Preprints (OSF Preprints) · Cited by 32
Prechelt, L. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.22 · Cited by 29
Cruz, M., Bernárdez, B., Durán, A., Galindo, J. Á., & Ruiz–Cortés, A. (2019) · IEEE Access · doi:10.1109/access.2019.2952191 · Cited by 28
Self-citationThe impact of using a domain language for an agile requirements management
Urbieta, M., Antonelli, L., Rossi, G. H., & Leite, J. C. S. D. P. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106375 · Cited by 28
Thinking Like a Developer? Comparing the Attention of Humans with Neural Models of Code
Paltenghi, M., & Pradel, M. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678712 · Cited by 28
SmartCommit: a graph-based interactive assistant for activity-oriented commits
Shen, B., Zhang, W., Kästner, C., Zhao, H., Wei, Z., Liang, G., & Jin, Z. G. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468551 · Cited by 25
Content and structure of laboratory packages for software engineering experiments
Solari, M., Vegas, S., & Juristo, N. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.12.016 · Cited by 24
Self-citationRicca, F., Torchiano, M., Leotta, M., Tiso, A., Guerrini, G., & Reggio, G. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9563-8 · Cited by 23
Understanding and improving artifact sharing in software engineering research
Timperley, C. S., Herckis, L., Le Goues, C., & Hilton, M. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09973-5 · Cited by 22
Endres, M., Fansher, M., Shah, P., & Weimer, W. R. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468583 · Cited by 22
A retrospective study of one decade of artifact evaluations
Winter, S., Timperley, C. S., Hermann, B., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549172 · Cited by 21
Replication of Controlled Experiments in Empirical Software Engineering - A Survey
Almqvist, J. P. F. (2006) · Lund University Publications Student Papers (Lund University) · Cited by 19
The National Academies of Sciences, Engineering and Medicine
(2023) · Palgrave Macmillan UK eBooks · doi:10.1057/978-1-349-96073-6_2814 · Cited by 19
Variation factors in the design and analysis of replicated controlled experiments
Runeson, P., Stefik, A., & Andrews, A. K. A. (2013) · Empirical Software Engineering · doi:10.1007/s10664-013-9262-z · Cited by 18
ISPY: Automatic Issue-Solution Pair Extraction from Community Live Chats
Shi, L., Jiang, Z., Yang, Y., Chen, X., Zhang, Y., Mu, F., Jiang, H., & Wang, Q. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678894 · Cited by 17
An empirical study to evaluate the impact of mindfulness on helpdesk employees
Bernárdez, B., Panach, J. I., Parejo, J. A., Durán, A., Juristo, N., & Ruiz–Cortés, A. (2023) · Science of Computer Programming · doi:10.1016/j.scico.2023.102977 · Cited by 16
Self-citationA Pragmatic Documents Standard for an Experience Library: Roles,Documen, Contents and Structure
Conradi, R., Basili, V. R., Carver, J., Shull, F. J., & Travassos, G. H. (2001) · University Libraries (University of Maryland) · Cited by 14
Which abbreviations should be expanded?
Jiang, Y., Liu, H., Zhang, Y., Niu, N., Zhao, Y., & Zhang, L. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468616 · Cited by 14
Packaging Researcher Experience to Assist Replication of Experiments
Basili, V. R., Green, S., Laitenberger, O., Lanubile, F., Shull, F. J., Sørumgård, S., & Zelkowitz, M. V. (1996) · Cited by 12
Wang, Z., Zhang, H., Chen, T.-H. P., et al. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468613 · Cited by 12
Assessing the effectiveness of goal-oriented modeling languages: A family of experiments
Abrahão, S., Insfrán, E., González-Ladrón-de-Guevara, F., Fernández-Diego, M., Cano-Genoves, C., & de Oliveira, R. P. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.08.003 · Cited by 11
A study into the practice of reporting software engineering experiments
Revoredo, K. C., Djurica, D., & Mendling, J. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10007-3 · Cited by 11
Towards a Taxonomy of Replications in Empirical Software Engineering Research: A Research Proposal
de Magalhães, C. V. C., & da Silva, F. Q. B. (2013) · doi:10.1109/reser.2013.10 · Cited by 8
Learning-based extraction of first-order logic representations of API directives
Liu, M., Peng, X., Marcus, A., Treude, C., Bai, X., Lyu, G., Xie, J., & Zhang, X. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468618 · Cited by 8
Effect of requirements specification using native language on external software quality
Uyaguari, F., Guevara‐Vega, C., Quiña‐Mera, A., Uyaguari, A., & Acosta, C. (2023) · IET Software · doi:10.1049/sfw2.12124 · Cited by 3
Self-citationAn Empirical Study to Evaluate the Impact of Mindfulness at a Software Company (Laboratory Package)
Bernárdez, B., Panach, J. I., Parejo, J. A., Durán, A., Juristo, N., & Ruiz–Cortés, A. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5607551 · Cited by 3
Self-citationWonsil, J. (2021) · cIRcle (University of British Columbia) · doi:10.14288/1.0398221 · Cited by 2
Research works printed but not linked: 21
- ACM, 2020
- Elsevier, 2023
- EMSE, ., Empirical software engineering journal. https://bit.ly/3QeQLwx.
- EMSE journal, 2023
- Ferro, 2018, SIGIR initiative to implement ACM artifact review and badging, Vol. 52, 4
- ICSE, 2021
- IEEE, 2023
- IST, ., Information and software technology journal. https://bit.ly/3H0Zhss.
- IST Journal, 2023
- JSS, ., Journal of systems and software journal. https://bit.ly/3tlGGSy.
- JSS Journal, 2023
- Open Science IEEE Computer Society, 2020
- Pucci, J., Passos, L., García, R., Messias, R., Olivete, C., 2014. ExpTool : a Tool to Conduct , Package and Replicate Controlled Experiments in Software Engineering. In: Proceedings of the International Conference on Software Engineering Research and Practice. SERP.
- Santos, 2018, Analyzing families of experiments in SE: A systematic mapping study, IEEE Trans. Softw. Eng., 46, 566, 10.1109/TSE.2018.2864633 · DOI
- Santos, 2021, A procedure and guidelines for analyzing groups of software engineering replications, IEEE Trans. Softw. Eng., 47, 1742
- Siegmund, 2015, Views on internal and external validity in empirical software engineering, Vol. 1, 9
- Solari, M., Vegas, S., 2006. Classifying and analysing replication packages for software engineering experimentation. In: 7th International Conference on Product Focused Software Process Improvement (PROFES 2006)-Workshop Series in Empirical Software Engineering (WSESE). Amsterdam, Paises Bajos.
- Springer, 2023
- SSRN Elsevier, 2022
- TSE, ., IEEE transactions on software engineering journal. https://bit.ly/3Q84cff.
- Vegas, 2006, Analysis of the influence of communication between researchers on experiment replication, Vol. 2006, 28
Metadata
- Authors
- Guevara-Vega, C., Bernárdez, B., Cruz, M., Durán, A., Ruiz-Cortés, A., & Solari, M.
- Venue
- Journal of Systems and Software
- Publisher
- Elsevier BV
- Published
- 2024-12
- Volume
- 218
- Pages
- 112187
- ISSN
- 0164-1212
- Cited by
- 8
- References
- 60
- Language
- en
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
