Judgment Call the Game: Using Value Sensitive Design and Design Fiction to Surface Ethical Concerns Related to Technology
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) technologies are complex socio-technical systems that, while holding much promise, have frequently caused societal harm. In response, corporations, non-profits, and academic researchers have mobilized to build responsible AI, yet how to do this is unclear. Toward this aim, we designed Judgment Call, a game for industry product teams to surface ethical concerns using value sensitive design and design fiction. Through two industry workshops, we found Judgment Call to be effective for considering technology from multiple perspectives and identifying ethical concerns. This work extends value sensitive design and design fiction to ethical AI and demonstrates the game's effective use in industry.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Speculative Everything: Design, Fiction, and Social Dreaming
(2014) · Choice Reviews Online · doi:10.5860/choice.51-5390 · Cited by 1,952
Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“For example, Sweeney discovered systemic bias in Google AdSense, an algorithmic advertising platform [21].”
BackgroundImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 1×
“Holstein and colleagues [12] studied the challenges machine learning practitioners in industry face as they develop fairer commercial AI products.”
BackgroundSweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477
A Survey of Value Sensitive Design Methods
Friedman, B., Hendry, D. G., & Borning, A. (2017) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000015 · Cited by 389 · cited 4×
“…interaction, participatory design, and futures studies literatures we identified several potentially useful methods, among them design fiction [2, 3, 8, 15, 19, 20, 25], futures workshops [14], scenario planning [23, 24], stakeholder identification [10, 11, 26], and value scenarios [10, 11, 17].”
BackgroundInfluential
Cited by
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction
Shelby, R., Rismani, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al. (2023) · Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3600211.3604673 · Cited by 279
“It can be strengthened by bringing in different standpoints and epistemolo-gies, such as feminism [73, 155], value-based design [17, 88, 116], de-sign justice perspectives [60], and post-colonial theories [126, 148].”
BackgroundToward situated interventions for algorithmic equity: lessons from the field
Katell, M., Young, M., Dailey, D., Herman, B., Guetler, V., Tam, A., Bintz, C., Raz, D., & Krafft, P. M. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372874 · Cited by 153
“Other methods that aim to empower stakeholders as designers include the use of metaphors [42], design fiction [3], and futuristic scenarios [38].”
BackgroundToward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55
Agent-Supported Foresight for AI Systemic Risks: AI Agents for Breadth, Experts for Judgment
Fröhling, L., Giaconia, A., Bogucka, E. P., & Quercia, D. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790712 · Cited by 2
Metadata
- Authors
- Ballard, S., Chappell, K. M., & Kennedy, K.
- Venue
- Proceedings of the 2019 on Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2019-06-18
- Pages
- 421-433
- OpenAlex
- W2953212529
- Cited by
- 98
- References
- 6
- FWCI
- 7.64
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.