Presenting software engineering results using structured abstracts: a randomised experiment
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
When conducting a systematic literature review, researchers usually determine the relevance of primary studies on the basis of the title and abstract. However, experience indicates that the abstracts for many software engineering papers are of too poor a quality to be used for this purpose. A solution adopted in other domains is to employ structured abstracts to improve the quality of information provided. This study consists of a formal experiment to investigate whether structured abstracts are more complete and easier to understand than non-structured abstracts for papers that describe software engineering experiments. We constructed structured versions of the abstracts for a random selection of 25 papers describing software engineering experiments. The 64 participants were each presented with one abstract in its original unstructured form and one in a structured form, and for each one were asked to assess its clarity (measured on a scale of 1 to 10) and completeness (measured with a questionnaire that used 18 items). Based on a regression analysis that adjusted for participant, abstract, type of abstract seen first, knowledge of structured abstracts, software engineering role, and preference for conventional or structured abstracts, the use of structured abstracts increased the completeness score by 6.65 (SE 0.37, p < 0.001) and the clarity score by 2.98 (SE 0.23, p < 0.001). 57 participants reported their preferences regarding structured abstracts: 13 (23%) had no preference; 40 (70%) preferred structured abstracts; four preferred conventional abstracts. Many conventional software engineering abstracts omit important information. Our study is consistent with studies from other disciplines and confirms that structured abstracts can improve both information content and readability. Although care must be taken to develop appropriate structures for different types of article, we recommend that Software Engineering journals and conferences adopt structured abstracts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789
Flesch, R. (1948) · Journal of Applied Psychology · doi:10.1037/h0057532 · Cited by 5,391 · cited 1×
“We also performed some initial analysis at this stage, in that values of the Flesch Reading Ease measure and Gunning Fog Index were recorded for each document (Flesch 1948; Flesch-Kincaid Readability Test 2006; Automated Readability Index 2006).”
MethodSystematic Reviews in the Social Sciences: A Practical Guide
Petticrew, M., & Roberts, H. (2006) · doi:10.1002/9780470754887 · Cited by 3,586
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394 · cited 2×
“Before starting on the study, we first developed a comprehensive research protocol detailing our plans (Budgen et al. 2007b).”
BackgroundMethodSelf-citation(2004) · doi:10.5555/998675.999432 · Cited by 656
Self-citationCross‐over Trials In Clinical Research
Senn, S. (2002) · doi:10.1002/0470854596 · Cited by 653
Structured abstracts: Do they improve the quality of information in abstracts?
Sharma, S., & Harrison, J. E. (2006) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2005.10.023 · Cited by 86 · cited 3×
“…1980s, also provides convincing empirical evidence in favour of improved relevant information content provided through the use of structured abstracts (Sharma and Harrison 2006) (a broader discussion of previous research related to structured abstracts can be found in Kitchenham et al. (2008)).”
BackgroundResultHartley, J., & Sydes, M. (1996) · Journal of Information Science · doi:10.1177/016555159602200103 · Cited by 31 · cited 1×
“Hartley’s work has suggested that not only is the distinction between structured and unstructured abstracts a significant issue, but also that the typographic layout employed provides an important factor that needs consideration (Hartley and Sydes 1996; Hartley 2000).”
BackgroundAddressing the Limitations of Structured Abstracts
Falagas, M., & Vergidis, P. I. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-141-7-200410050-00023 · Cited by 16
Cited by
Identifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 557
Robust Statistical Methods for Empirical Software Engineering
Kitchenham, B., Madeyski, L., Budgen, D., Keung, J., Brereton, P., Charters, S., Gibbs, S., & Pohthong, A. (2017) · Empirical Software Engineering · doi:10.1007/s10664-016-9437-5 · Cited by 257
Self-citationWhat’s up with software metrics? – A preliminary mapping study
Kitchenham, B. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2009.06.041 · Cited by 253
Self-citationWubineh, B. Z., Deriba, F. G., & Woldeyohannis, M. M. (2024) · Urologic Oncology: Seminars and Original Investigations · doi:10.1016/j.urolonc.2023.11.019 · Cited by 141
Empirical evidence about the UML: a systematic literature review
Budgen, D., Burn, A. J., Brereton, O. P., Kitchenham, B. A., & Pretorius, R. (2011) · Software: Practice and Experience · doi:10.1002/spe.1009 · Cited by 138
“As the quality of reporting in abstracts is often poor (see [ 8 ] ), we erred on the side of caution when applying the inclusion and exclusion criteria.”
MethodSelf-citation
Metadata
- Authors
- Budgen, D., Kitchenham, B. A., Charters, S. M., Turner, M., Brereton, P., & Linkman, S. G.
- Venue
- Empirical Software Engineering
- Published
- 2008-07-17
- Volume
- 13
- Issue
- 4
- Pages
- 435-468
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W1964386935
- Cited by
- 69
- References
- 9
- FWCI
- 18.65
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Presenting software engineering results using structured abstracts: a randomised experiment. https://app.cochat.ai/papers/10.1007/s10664-008-9075-7
- The paper
- Budgen, D., Kitchenham, B. A., Charters, S. M., Turner, M., Brereton, P., & Linkman, S. G. (2008). Presenting software engineering results using structured abstracts: a randomised experiment. Empirical Software Engineering. https://doi.org/10.1007/s10664-008-9075-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.