Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science
Open accessThe DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index; 9 of the 57 sources its claims cite were checked for support.
Abstract
To foster a sustainable society within a sustainable environment, we must dramatically reshape our work and consumption activities, most of which are facilitated through software. Yet, most software engineers hardly consider the effects on the sustainability of the IT products and services they deliver. This issue is exacerbated by a lack of methods and tools for this purpose. Despite the practical need for methods and tools that explicitly support consideration of the effects that IT products and services have on the sustainability of their intended environments, such methods and tools remain largely unavailable. Thus, urgent research is needed to understand how to design such tools for the IT community properly. In this article, we describe our experience using design science to create the Sustainability Awareness Framework (SusAF), which supports software engineers in anticipating and mitigating the potential sustainability effects during system development. More specifically, we identify and present the challenges faced during this process. The challenges that we have faced and addressed in the development of the SusAF are likely to be relevant to others who aim to create methods and tools to integrate sustainability analysis into their IT products and services development. Thus, the lessons learned in SusAF development are shared for the benefit of researchers and other professionals who design tools for that end.
Overview
CoChat Summary
TL;DR
This paper reports a decade-long design-science case study of developing the Sustainability Awareness Framework (SusAF) across 13 cycles, synthesizing lessons for building sustainability analysis tools in software engineering. It finds that SusAF broadened participants’ awareness of sustainability effects and identifies six lessons plus methodological limitations of design science for long-term, multidimensional sustainability impacts.
Key findings
- The SusAF development produced six lessons learned: need for methods/tools; need for dimensional and temporal link-up; need for examples; need for efficiency in elicitation and analysis; need for an introductory format; and need to adapt delivery to need or context.
- Industry workshops substantially broadened participants’ perception of sustainability effects: Datamatrix moved from 4 initially perceived effects to 153 potential effects; Jeppesen-Boeing went from 7 to 72 effects; Premier Park went from 2 to 144 effects, described as more than a 10-fold broadening.
- In teaching, the question set and improved artifacts were associated with more and better-contextualized effects, including an 80% increase in potential effects identified in LUT diagrams from academic year 2017–18 to 2018–19.
- Design science was useful for creating SusAF, but the paper identifies methodological concerns: timescale challenges for evaluating long-term sustainability impacts, scope and measurement challenges, and the anthropocentric focus of design science.
Methods
- Design: a design-science research project using Hevner’s three-cycle view—relevance, rigor, and design cycles—implemented over 13 cycles spanning almost a decade of SusAF development.
- Data and sample: 208 students from IT-related programs in the United States, Finland, Spain, Germany, and Sweden; 64 IT practitioners from 43 companies in Finland, India, the United States, Sweden, Germany, and Spain; plus an initial qualitative interview study with 13 RE practitioners from eight countries. Data included coursework, surveys, instructor/moderator reflections, SusADs, reports, and workshop feedback.
- Analysis: each cycle was analyzed separately to identify improvements and lessons learned; the paper reports descriptive statistics and qualitative feedback, with expert-panel/Delphi-style rounds used to develop the SusAF question set.
Limitations
- Paper-stated threats to validity include convenience sampling of students and companies, possible reactive bias among students, construct confounding, novelty effects, experimenter expectancies, limited external validity, and conclusion-validity concerns mitigated by participant diversity and shared facilitator processes.
- Paper-stated design-science limitations include difficulty evaluating long-term sustainability impacts within usual design-science timescales, lack of agreed units and boundaries for measuring social, individual, and environmental impacts, and design science’s anthropocentric focus.
- Evident from the text: this is a single-framework case study, and the authors state they cannot compare design science to any other method they have applied as extensively; long-term impact verification would require decades of data collection within the same context.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



![Fig. 4 Application of the design science research cycles to the development of the SusAF, adapted from Hevner and Chatterjee [51].](https://figures.cochat.ai/paper-figures/8b1b8a785d804b8ebb7483c5b84d2b24/lessons-learned-from-developing-a-sustainability-awareness-framework-for-figure-4.png)




![Fig. 9 Two pages from the workbook as thumbnails. The workbook is available for free [13].](https://figures.cochat.ai/paper-figures/8be6dda91db14ac69ce537e10325a827/lessons-learned-from-developing-a-sustainability-awareness-framework-for-figure-9.png)
Verification
Claims the paper makes
9 Valid · 3 Not valid · 34 Not checked · 136 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, the students were not sure as to how to initiate the process of sustainability requirements elicitation [35].”
§ Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science > Evaluation.
What the sources say
- Partly supportedcites [35] sourceExperimental — accuracy not yet measured
The closest our copy comes is 7.2 Method, which says the approach 'is not self-explanatory', that sufficient time must be set aside to describe it, that roleplay helped because 'the students could see how to use the framework', and that the question sets made students 'feel more confident when conducting the sustainability effect analysis' — i.e. students needed guidance and gained confidence, rather than a reported finding that they were unsure how to initiate sustainability requirements elicitation. The actually reported student difficulties are different in kind: 6.6.2 lists questions unrelated to the system (15%), lack of interviewee knowledge (10%) and question wording (9%), with one student finding it 'difficult to get a conversation going', and 7.3 identifies finding adequate surrogate stakeholders as the big challenge. 6.6.3 even records students seeing the questions as a 'good way to start' and the SusAD instructions as clear (87% liked them), so the claim's specific assertion of uncertainty about initiating the process overstates what this source states. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Our observations corroborated other studies (e.g., [45,48,58]).”
§ Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science > 5.1 [Relevance] Cycle 1: Identified Need: Interview Study
What the sources say
- Not checked yetcites [45]
- Not checked yetcites [48] source
- Partly supportedcites [58] sourceExperimental — accuracy not yet measured
Our copy of Kirchherr et al. (2018) does report a widespread deficit of interest and awareness around sustainability/circular-economy thinking: Section 4.1 'Cultural Barriers' names 'Lacking consumer interest and awareness' as the most mentioned barrier among its 208 EU business and policy respondents, and Section 5 concludes the CE 'has not reached the mainstream yet'. That partly supports the citing sentence's idea of a corroborating study on limited understanding of sustainability. However, the paper's population is European consumers, businesses and policy-makers — it addresses neither students nor IT practitioners, and never discusses IT products or the relationship between sustainability and IT; searches for 'information technology / IT products / students' returned only reference-list entries (e.g. Barth et al., 2007 on higher education, cited for culture's role), not findings of this study. Moreover, Section 4 states explicitly that varying understandings of the CE concept among respondents was 'not the case', so an 'understanding' framing is not this paper's result. (LLM, conf=0.65)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“It is interesting to highlight that there was an 80% increase in the potential effects identified in LUT diagrams from the academic year 2017-18 to 2018-19 [34,35].”
§ Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science > Evaluation.
What the sources say
- Not checked yetcites [34] source
- Supportedcites [35] source
Section 6.4 'RQ4: identify more effects' states that with the question sheets there was 'a 10% increase in CSULB diagrams and an 80% increase in LUT diagrams' in the number of (potential) effects identified, comparing the spring 2019 SusADs with the spring 2018 baseline SusADs (Sects. 5.1-5.2 confirm the baseline was spring 2018 and the later round spring 2019, i.e. the 2017-18 to 2018-19 academic years at LUT and CSULB). The magnitude, the university (LUT), the artefact (LUT diagrams/SusADs) and the direction of change all match the citing sentence. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“IT practitioners do not commonly deal with effects that may unfold over time [8].”
§ Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science > 5.4 [Rigor and Design] Cycle 4: Development of Questions with Expert Panel State of the Art.
What the sources say
- Supportedcites [8] source
The Karlskrona Manifesto's Introduction (p. 2) states that 'software practice has a tendency to focus only on the immediate effects and tangible benefits of software products and platforms' and that 'the lack of long-term thinking in software engineering research and practice has been critiqued throughout the history of the discipline'. The 'We recognize that' section (p. 8) reinforces this: 'There is a tendency to focus on the immediate impacts of any new technology' and 'measuring success at one point in time fails to capture the effects that play out over multiple timescales'. This paraphrases the claim that practitioners do not commonly deal with effects unfolding over time; the only slight shift is 'IT practitioners' for the paper's 'software engineering research and practice'. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, presently, integration of sustainability analysis into software development is a very time consuming process.”
§ Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science > Implications for SE practice:
What the sources say
- Partly supportedcites [24] sourceExperimental — accuracy not yet measured
Section 4.2 (page unavailable) reports interviewees’ view that sustainability design requires extra work and an initial investment of time. Section 5.2 (page unavailable) says companies lack time and resources to commit to it. These passages support a time burden, but do not establish that integrating sustainability analysis is, as a general fact, a very time-consuming process. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
65 more claims
Create a free account to see all 182 claims and to verify the 48 references still open.
- Probably not a substantial claim. 74
- It may state two separate claims as one. 58
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Bets et al.'s [18], Lagos et al.'s [98], and Naumann et al.'s [136] studies identify several potential indicators that need a large-scale benchmark evaluation and the integration within the ISO standards, like the environmental ISO 14,000 series and the social responsibility ISO 26000.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science > 6.3 Threats to Validity
“This study as described in Section 5 has more than 208 student participants from IT-related undergraduate and postgraduate courses across five countries, which”
§ Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science > 4.2 Participants
“Overall, we had 208 students in several countries-the United States (CSULB), Finland (LUT), Spain (LaSalle), Germany (HFU), and Sweden (KTH, LiU):”
Citations
Cites
(2002) · Cited by 13,211
Design Science in Information Systems Research1
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004) · MIS Quarterly · doi:10.2307/25148625 · Cited by 10,707
The Delphi method as a research tool: an example, design considerations and applications
Okoli, C., & Pawlowski, S. D. (2004) · Information & Management · doi:10.1016/j.im.2003.11.002 · Cited by 4,771 · cited 1×
“More specifically, we used an adaptation of the Delphi method [63, 77] to get these experts to define a set of questions to help start and guide the discussion on the effects of IT products on their in situ sustainability [35].”
An overview of the Schwartz theory of basic values
Schwartz, S. H. (2012) · Online Readings in Psychology and Culture · doi:10.9707/2307-0919.1116 · Cited by 4,147 · cited 1×
“…the current body of knowledge as a further input to the question set, including the Flourishing Business Canvas model [107], the Schwartz model of human values [104], the visualization diagram [8], the Sustainability Goal Reference model [87], and SIA (Societal Impact Assessment) literature.”
Doughnut Economics: Seven Ways to Think Like a 21st Century Economist
Diethelm, J. (2016) · She Ji: The Journal of Design, Economics, and Innovation · doi:10.1016/j.sheji.2017.05.002 · Cited by 2,071
Barriers to the Circular Economy: Evidence From the European Union (EU)
Kirchherr, J., Piscicelli, L., Bour, R., Kostense-Smit, E., Muller, J., Huibrechtse-Truijens, A., & Hekkert, M. (2018) · Ecological Economics · doi:10.1016/j.ecolecon.2018.04.028 · Cited by 1,903
Scenario types and techniques: Towards a user's guide
Börjeson, L., Höjer, M., Dreborg, K.-H., Ekvall, T., & Finnveden, G. (2006) · Futures · doi:10.1016/j.futures.2005.12.002 · Cited by 1,421 · cited 1×
“The “scenario techniques” widely used in future studies [16, 17] were also identified as a relevant method to help analyze and coherently present the possible future effects.”
Design Research in Information Systems: Theory and Practice
Hevner, A., & Chatterjee, S. (2010) · Integrated Series in Information Systems · doi:10.1007/978-1-4419-5653-8 · Cited by 1,382
Current validity of the Delphi method in social sciences
Landeta, J. (2006) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2005.09.002 · Cited by 1,353 · cited 1×
“More specifically, we used an adaptation of the Delphi method [63, 77] to get these experts to define a set of questions to help start and guide the discussion on the effects of IT products on their in situ sustainability [35].”
Design Science Methodology for Information Systems and Software Engineering
Wieringa, R. J. (2014) · Springer eBooks · doi:10.1007/978-3-662-43839-8 · Cited by 1,214 · cited 2×
“Design science research is a research paradigm that generates knowledge and scientific evidence by means of systematically developing an innovative artifact to solve human problems in a particular context [50, 51, 118].”
Research works printed but not linked: 17
- Ian F. Alexander and Ljerka Beus-Dukic. 2009. Discovering Requirements: How to Specify Products and Services. Wiley.
- Stefanie Betz Christoph Becker Ruzanna Chitchyan Leticia Duboc Steve Easterbrook Birgit Penzenstadler Norbert Seyff and Colin Venters. 2015. Sustainability debt: A metaphor to support sustainability design decisions. In Proceedings of the International Workshop on Requirements Engineering for Sustainable Systems.
- Peter Checkland. 1999. Systems Thinking Systems Practice: Includes a 30-Year Retrospective. John Wiley & Sons Ltd.
- R. Chitchyan S. Betz L. Duboc B. Penzenstadler S. Easterbrook C. Ponsard and C. Venters. 2015. Evidencing sustainability design through examples. In Proceedings of the 4th International Workshop on Requirements Engineering for Sustainable Systems.
- Emelie Engström Margaret-Anne D. Storey Per Runeson Martin Höst and Maria Teresa Baldassarre. 2019. A review of software engineering research from a design science perspective. CoRR abs/1904.12742 (2019). http://arxiv.org/abs/1904.12742 · arXiv
- M. L. Gibson R. Chitchyan C. C. Venters M. Palacin-Silva L. Duboc B. Penzenstadler S. Betz and N. Seyff. 2017. Mind the chasm: A UK FishEye lens view of sustainable software engineering. In Proceedings of the 6th International Workshop on Requirements Engineering for Sustainable Systems (RE4SuSy’17). 15–24. http://eprints.hud.ac.uk/id/eprint/32429/
- André Heuer Philipp Diebold and Torsten Bandyszak. 2014. Supporting technology transfer by providing recommendations for writing structured guidelines. CEUR Workshop Proceedings 1129 (2014) 47–56.
- Alan Hevner. 2007. A three cycle view of design science research. Scandinavian Journal of Information Systems 19 (2007) 87–92.
- IPCC. 2022. Climate Change 2022: Impacts Adaptation and Vulnerability: Contribution of Working Group II to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press.
- Samuel Mann. 2011. The Green Graduate: Educating Every Student as a Sustainable Practitioner. NZCER Press Wellington New Zealand.
- Helena Holmström Olsson Hiva Alahyari and Jan Bosch. 2012. Climbing the “stairway to heaven”—A multiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In Proceedings of the 2012 38th Euromicro Conference on Software Engineering and Advanced Applications. IEEE 392–399.
- Alexander Osterwalder Yves Pigneur and Tim Clark. 2010. Business Model Generation: A Handbook for Visionaries Game Changers and Challengers. John Wiley & Sons Hoboken NJ.
- Birgit Penzenstadler Mara Anoushka Nam Stephanie and Budzinski Brian. 2019. Exploratory case study on sustainability awareness with a startup for business models. In Proceedings of the 6th International Conference on ICT for Sustainability (ICT4S’19).
- Roger S. Pressman. 1994. Software Engineering: A Practitioner’s Approach (European ed.). McGraw-Hill.
- Thorsten Schoormann Dennis Behrens and Ralf Knackstedt. 2018. Design principles for leveraging sustainability in business modelling tools. In Proceedings of the European Conference on Information Systems. https://api.semanticscholar.org/CorpusID:56140270
- World Commission on Environment and Development. 1987. Our Common Future. Oxford University Press, Oxford. http://www.un-documents.net/wced-ocf.htm
- 10.5555/1941925 · doi:10.5555/1941925
Cited by
Optimizing Large Language Models: Metrics, Energy Efficiency, and Case Study Insights
Khan, T., Motie, S., Koçak, S. A., & Raza, S. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00067 · Cited by 25
“For instance, intersection of sustainability and software engineering is well established research area [3], [7], [26] such as exploring strategies for creating eco-friendly solutions that maintain functionality while reducing energy consumption [24].”
Sustainable Software Engineering: Concepts, Challenges, and Vision
König, C., Lang, D. J., & Schaefer, I. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709352 · Cited by 25
With Great Power Comes Great Responsibility: The Role of Software Engineers
Betz, S., & Penzenstadler, B. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715112 · Cited by 15
Self-citationA 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13
Self-citationde Walle, A. V., & Aerts, G. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103358 · Cited by 6
Metadata
- Authors
- Betz, S., Penzenstadler, B., Duboc, L., Chitchyan, R., Kocak, S. A., Brooks, I., Oyedeji, S., Porras, J., Seyff, N., & Venters, C. C.
- Affiliations
- Furtwangen University (DE)
- Chalmers University of Technology (SE)
- La Salle, University Ramon Llull, Barcelona, Spain
- University of Bristol (GB)
- Vector Institute for Artificial Intelligence, Toronto Canada
- University of the West of England (GB)
- Lappeenranta-Lahti University of Technology (FI)
- FHNW University of Applied Sciences and Arts Northwestern Switzerland (CH)
- University of Huddersfield (GB)
- ORCID
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2024-06-03
- Volume
- 33
- Issue
- 5
- Pages
- 1-39
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4392593558
- Semantic Scholar
- 12fcd6110a8ed90fc001a166cef7a68bdf1875fc
- Cited by
- 23
- References
- 100
- Citations by year
- 2026: 12 · 2025: 8 · 2024: 3
- FWCI
- 2.38
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
- Repository of Futwangen University of Applied Sciences (Furtwangen University) (submittedVersion, cc-by)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
- Huddersfield Research Portal (University of Huddersfield) (submittedVersion, cc-by)
- Explore Bristol Research (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Funders
- Departament de Recerca i Universitats of Catalunya (2021 SGR 01396)
- EPSRC (EP/R007373/1)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Green IT and Sustainability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Engineering › Building and Construction › Sustainable Design and Development
- Keywords
- sustainability assessment, sustainability assessment tools, lessons learned, design science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science. https://app.cochat.ai/papers/10.1145/3649597
- The paper
- Betz, S., Penzenstadler, B., Duboc, L., Chitchyan, R., Kocak, S. A., Brooks, I., Oyedeji, S., Porras, J., Seyff, N., & Venters, C. C. (2024). Lessons Learned from Developing a Sustainability Awareness Framework for Software Engineering Using Design Science. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3649597
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.