CoChat

Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science

Open access
Betz, Stefanie, Penzenstadler, Birgit, Duboc, Leticia, Chitchyan, Ruzanna, Kocak, Sedef Akinli, Brooks, Ian, Oyedeji, Shola, Porras, Jari, Seyff, Norbert, Venters, Colin C. · 2024 · ACM Transactions on Software Engineering and Methodology

The DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index; 9 of the 57 sources its claims cite were checked for support.

Abstract

To foster a sustainable society within a sustainable environment, we must dramatically reshape our work and consumption activities, most of which are facilitated through software. Yet, most software engineers hardly consider the effects on the sustainability of the IT products and services they deliver. This issue is exacerbated by a lack of methods and tools for this purpose. Despite the practical need for methods and tools that explicitly support consideration of the effects that IT products and services have on the sustainability of their intended environments, such methods and tools remain largely unavailable. Thus, urgent research is needed to understand how to design such tools for the IT community properly. In this article, we describe our experience using design science to create the Sustainability Awareness Framework (SusAF), which supports software engineers in anticipating and mitigating the potential sustainability effects during system development. More specifically, we identify and present the challenges faced during this process. The challenges that we have faced and addressed in the development of the SusAF are likely to be relevant to others who aim to create methods and tools to integrate sustainability analysis into their IT products and services development. Thus, the lessons learned in SusAF development are shared for the benefit of researchers and other professionals who design tools for that end.

Overview

CoChat Summary

TL;DR

This paper reports a decade-long design-science case study of developing the Sustainability Awareness Framework (SusAF) across 13 cycles, synthesizing lessons for building sustainability analysis tools in software engineering. It finds that SusAF broadened participants’ awareness of sustainability effects and identifies six lessons plus methodological limitations of design science for long-term, multidimensional sustainability impacts.

Key findings

  • The SusAF development produced six lessons learned: need for methods/tools; need for dimensional and temporal link-up; need for examples; need for efficiency in elicitation and analysis; need for an introductory format; and need to adapt delivery to need or context.
  • Industry workshops substantially broadened participants’ perception of sustainability effects: Datamatrix moved from 4 initially perceived effects to 153 potential effects; Jeppesen-Boeing went from 7 to 72 effects; Premier Park went from 2 to 144 effects, described as more than a 10-fold broadening.
  • In teaching, the question set and improved artifacts were associated with more and better-contextualized effects, including an 80% increase in potential effects identified in LUT diagrams from academic year 2017–18 to 2018–19.
  • Design science was useful for creating SusAF, but the paper identifies methodological concerns: timescale challenges for evaluating long-term sustainability impacts, scope and measurement challenges, and the anthropocentric focus of design science.

Methods

  • Design: a design-science research project using Hevner’s three-cycle view—relevance, rigor, and design cycles—implemented over 13 cycles spanning almost a decade of SusAF development.
  • Data and sample: 208 students from IT-related programs in the United States, Finland, Spain, Germany, and Sweden; 64 IT practitioners from 43 companies in Finland, India, the United States, Sweden, Germany, and Spain; plus an initial qualitative interview study with 13 RE practitioners from eight countries. Data included coursework, surveys, instructor/moderator reflections, SusADs, reports, and workshop feedback.
  • Analysis: each cycle was analyzed separately to identify improvements and lessons learned; the paper reports descriptive statistics and qualitative feedback, with expert-panel/Delphi-style rounds used to develop the SusAF question set.

Limitations

  • Paper-stated threats to validity include convenience sampling of students and companies, possible reactive bias among students, construct confounding, novelty effects, experimenter expectancies, limited external validity, and conclusion-validity concerns mitigated by participant diversity and shared facilitator processes.
  • Paper-stated design-science limitations include difficulty evaluating long-term sustainability impacts within usual design-science timescales, lack of agreed units and boundaries for measuring social, individual, and environmental impacts, and design science’s anthropocentric focus.
  • Evident from the text: this is a single-framework case study, and the authors state they cannot compare design science to any other method they have applied as extensively; long-term impact verification would require decades of data collection within the same context.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3649597

Cites

100 of 100 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Relation of previous work to datasets. All cycles in this figure are relevance cycles.
Fig. 1 Relation of previous work to datasets. All cycles in this figure are relevance cycles.CC-BY · From the PDF
Fig. 2 Relation of previous work to cycles. Blue columns refer to relevance cycles (with datasets), whereas green ones refer to rigor and design cycles.
Fig. 2 Relation of previous work to cycles. Blue columns refer to relevance cycles (with datasets), whereas green ones refer to rigor and design cycles.CC-BY · From the PDF
Fig. 3 SusAD: summarizing diagram of the Partneur workshop results.
Fig. 3 SusAD: summarizing diagram of the Partneur workshop results.CC-BY · From the PDF
Fig. 4 Application of the design science research cycles to the development of the SusAF, adapted from Hevner and Chatterjee [51].
Fig. 4 Application of the design science research cycles to the development of the SusAF, adapted from Hevner and Chatterjee [51].CC-BY · From the PDF
Fig. 5 Teaching and industry artifacts.
Fig. 5 Teaching and industry artifacts.CC-BY · From the PDF
Fig. 6 The design cycles we applied for the SusAF.
Fig. 6 The design cycles we applied for the SusAF.CC-BY · From the PDF
Fig. 7 Question sheet for the social dimension.
Fig. 7 Question sheet for the social dimension.CC-BY · From the PDF
Fig. 8 Alternative way to draw the SusAD (1).
Fig. 8 Alternative way to draw the SusAD (1).CC-BY · From the PDF
Fig. 9 Two pages from the workbook as thumbnails. The workbook is available for free [13].
Fig. 9 Two pages from the workbook as thumbnails. The workbook is available for free [13].CC-BY · From the PDF

Verification

Claims the paper makes

9 Valid · 3 Not valid · 34 Not checked · 136 Own results and interpretations

65 more claims

Create a free account to see all 182 claims and to verify the 48 references still open.

See all claims — free account
  • Probably not a substantial claim. 74
  • It may state two separate claims as one. 58
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 100 references

Research works printed but not linked: 17

  1. Ian F. Alexander and Ljerka Beus-Dukic. 2009. Discovering Requirements: How to Specify Products and Services. Wiley.
  2. Stefanie Betz Christoph Becker Ruzanna Chitchyan Leticia Duboc Steve Easterbrook Birgit Penzenstadler Norbert Seyff and Colin Venters. 2015. Sustainability debt: A metaphor to support sustainability design decisions. In Proceedings of the International Workshop on Requirements Engineering for Sustainable Systems.
  3. Peter Checkland. 1999. Systems Thinking Systems Practice: Includes a 30-Year Retrospective. John Wiley & Sons Ltd.
  4. R. Chitchyan S. Betz L. Duboc B. Penzenstadler S. Easterbrook C. Ponsard and C. Venters. 2015. Evidencing sustainability design through examples. In Proceedings of the 4th International Workshop on Requirements Engineering for Sustainable Systems.
  5. Emelie Engström Margaret-Anne D. Storey Per Runeson Martin Höst and Maria Teresa Baldassarre. 2019. A review of software engineering research from a design science perspective. CoRR abs/1904.12742 (2019). http://arxiv.org/abs/1904.12742 · arXiv
  6. M. L. Gibson R. Chitchyan C. C. Venters M. Palacin-Silva L. Duboc B. Penzenstadler S. Betz and N. Seyff. 2017. Mind the chasm: A UK FishEye lens view of sustainable software engineering. In Proceedings of the 6th International Workshop on Requirements Engineering for Sustainable Systems (RE4SuSy’17). 15–24. http://eprints.hud.ac.uk/id/eprint/32429/
  7. André Heuer Philipp Diebold and Torsten Bandyszak. 2014. Supporting technology transfer by providing recommendations for writing structured guidelines. CEUR Workshop Proceedings 1129 (2014) 47–56.
  8. Alan Hevner. 2007. A three cycle view of design science research. Scandinavian Journal of Information Systems 19 (2007) 87–92.
  9. IPCC. 2022. Climate Change 2022: Impacts Adaptation and Vulnerability: Contribution of Working Group II to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press.
  10. Samuel Mann. 2011. The Green Graduate: Educating Every Student as a Sustainable Practitioner. NZCER Press Wellington New Zealand.
  11. Helena Holmström Olsson Hiva Alahyari and Jan Bosch. 2012. Climbing the “stairway to heaven”—A multiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In Proceedings of the 2012 38th Euromicro Conference on Software Engineering and Advanced Applications. IEEE 392–399.
  12. Alexander Osterwalder Yves Pigneur and Tim Clark. 2010. Business Model Generation: A Handbook for Visionaries Game Changers and Challengers. John Wiley & Sons Hoboken NJ.
  13. Birgit Penzenstadler Mara Anoushka Nam Stephanie and Budzinski Brian. 2019. Exploratory case study on sustainability awareness with a startup for business models. In Proceedings of the 6th International Conference on ICT for Sustainability (ICT4S’19).
  14. Roger S. Pressman. 1994. Software Engineering: A Practitioner’s Approach (European ed.). McGraw-Hill.
  15. Thorsten Schoormann Dennis Behrens and Ralf Knackstedt. 2018. Design principles for leveraging sustainability in business modelling tools. In Proceedings of the European Conference on Information Systems. https://api.semanticscholar.org/CorpusID:56140270
  16. World Commission on Environment and Development. 1987. Our Common Future. Oxford University Press, Oxford. http://www.un-documents.net/wced-ocf.htm
  17. 10.5555/1941925 · doi:10.5555/1941925

Cited by

See all 27

Metadata

Authors
Betz, S., Penzenstadler, B., Duboc, L., Chitchyan, R., Kocak, S. A., Brooks, I., Oyedeji, S., Porras, J., Seyff, N., & Venters, C. C.
Affiliations
  • Furtwangen University (DE)
  • Chalmers University of Technology (SE)
  • La Salle, University Ramon Llull, Barcelona, Spain
  • University of Bristol (GB)
  • Vector Institute for Artificial Intelligence, Toronto Canada
  • University of the West of England (GB)
  • Lappeenranta-Lahti University of Technology (FI)
  • FHNW University of Applied Sciences and Arts Northwestern Switzerland (CH)
  • University of Huddersfield (GB)
Venue
ACM Transactions on Software Engineering and Methodology
Published
2024-06-03
Volume
33
Issue
5
Pages
1-39
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
OpenAlex
W4392593558
Cited by
23
References
100
Citations by year
2026: 12 · 2025: 8 · 2024: 3
FWCI
2.38
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
  • Repository of Futwangen University of Applied Sciences (Furtwangen University) (submittedVersion, cc-by)
  • Chalmers Research (Chalmers University of Technology) (submittedVersion)
  • UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
  • Huddersfield Research Portal (University of Huddersfield) (submittedVersion, cc-by)
  • Explore Bristol Research (submittedVersion, cc-by)
  • Bristol Research (University of Bristol) (submittedVersion, cc-by)
Funders
  • Departament de Recerca i Universitats of Catalunya (2021 SGR 01396)
  • EPSRC (EP/R007373/1)
Topics
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Engineering › Building and Construction › Sustainable Design and Development
Keywords
sustainability assessment, sustainability assessment tools, lessons learned, design science

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science. https://app.cochat.ai/papers/10.1145/3649597
The paper
Betz, S., Penzenstadler, B., Duboc, L., Chitchyan, R., Kocak, S. A., Brooks, I., Oyedeji, S., Porras, J., Seyff, N., & Venters, C. C. (2024). Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3649597

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.