Evaluation Beyond Usability: Validating Sustainable HCI Research
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
The evaluation of research artefacts is an important step to validate research contributions. Sub-disciplines of HCI often pursue primary goals other than usability, such as Sustainable HCI (SHCI), HCI for development, or health and wellbeing. For such disciplines, established evaluation methods are not always appropriate or sufficient, and new conventions for identifying, discussing, and justifying suitable evaluation methods need to be established. In this paper, we revisit the purpose and goals of evaluation in HCI and SHCI, and elicit five key elements that can provide guidance to identifying evaluation methods for SHCI research. Our essay is meant as a starting point for discussing current and improving future evaluation practice in SHCI; we also believe it holds value for other subdisciplines in HCI that encounter similar challenges while evaluating their research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Energy efficiency and consumption—The rebound effect—A survey
A. Greening, L., Greene, D. L., & Difiglio, C. (2000) · Energy Policy · doi:10.1016/s0301-4215(00)00021-5 · Cited by 2,077 · cited 2×
“An example for this is the rebound effect [38,48], an economic observation stating that an increased efficiency of technology is met with an increased rate of consumption due to higher demand.”
BackgroundDourish, P. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124855 · Cited by 860
The Dynamics of Social Practice
(2012) · The Dynamics of Social Practice: Everyday Life and How it Changes · doi:10.4135/9781446250655.n1 · Cited by 846
The design of eco-feedback technology
Froehlich, J., Findlater, L., & Landay, J. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753629 · Cited by 813 · cited 18×
“[34]) or social practice theory [88] can be most promising.”
BackgroundInfluentialStaying open to interpretation: engaging multiple meanings in design and evaluation
Sengers, P., & Gaver, B. (2006) · Proceedings of the 6th conference on Designing Interactive systems · doi:10.1145/1142405.1142422 · Cited by 620 · cited 2×
“Many of HCI’s evaluation processes developed and matured over time; indeed, it took decades for HCI to agree on common evaluation standards, and the discussion about many of those standards continues to this day [28,30,36,85,86].”
BackgroundMethodSustainably unpersuaded: how persuasion narrows our vision of sustainability
Brynjarsdottir, H., Håkansson, M., Pierce, J., Baumer, E., DiSalvo, C., & Sengers, P. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208539 · Cited by 483 · cited 8×
“Indeed, researchers in the SHCI community have acknowledged this difficulty for quite some time [14,15,19,26,34] and have yet to completely resolve it.”
BackgroundInfluentialNew theoretical approaches for human‐computer interaction
Rogers, Y. (2004) · Annual Review of Information Science and Technology · doi:10.1002/aris.1440380103 · Cited by 386 · cited 1×
“However, many of those frameworks are almost never put to practice, which has been described as the theory-practice gap [81,83,92,93].”
BackgroundFrom Mice to Men - 24 Years of Evaluation in CHI
Barkhuus, L., & Rode, J. A. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.2180963 · Cited by 180
Preist, C., Schien, D., & Blevis, E. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858378 · Cited by 133 · cited 1×
“Such methods need to estimate an impact, draw scenarios of future trajectories [75], or discuss the uncertainties [58].”
Cited by
Mapping the Landscape of Creativity Support Tools in HCI
Frich, J., MacDonald Vermeulen, L., Remy, C., Biskjaer, M. M., & Dalsgaard, P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300619 · Cited by 410
Self-citationA Decade of Sustainable HCI: Connecting SHCI to the Sustainable Development Goals
Hansson, L. Å. E. J., Cerratto Pargman, T., & Pargman, D. S. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445069 · Cited by 148
Evaluating Creativity Support Tools in HCI Research
Remy, C., MacDonald Vermeulen, L., Frich, J., Biskjaer, M. M., & Dalsgaard, P. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395474 · Cited by 96
“In that case, it can help to investigate the mechanisms affected by the tool, as suggested for the second ingredient of the evaluation recipe [127].”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Remy, C., Bates, O., Dix, A., Thomas, V., Hazas, M., Friday, A., & Huang, E. M.
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-14
- OpenAlex
- W2782309908
- Cited by
- 69
- References
- 11
- FWCI
- 4.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation Beyond Usability: Validating Sustainable HCI Research. https://app.cochat.ai/papers/10.1145/3173574.3173790
- The paper
- Remy, C., Bates, O., Dix, A., Thomas, V., Hazas, M., Friday, A., & Huang, E. M. (2018). Evaluation Beyond Usability: Validating Sustainable HCI Research. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3173790
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.