Replay Enactments: Exploring Possible Futures through Historical Data
Open accessAbstract
As we design increasingly complex systems, we run up against fundamental limitations of human imagination. To support practice, it becomes essential to use authentic data and algorithms as design materials to augment designers' intuitions. Recent work has explored some dimensions of using data as a design material, suggesting the contours of a new space of design and prototyping methods. In this paper, we present Replay Enactments (REs, an extension of the User Enactments methods that uses data replay as a boundary object, making complex system behavior tangible to designers and stakeholders. We reflect on a set of case studies that have instantiated REs in diverse ways and discuss trade-offs between different ways of using data replays in design. We conclude by highlighting opportunities and challenges for future work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
HCI researchers can use CoAuthor to generate hypotheses through replay enactment.
“HCI researchers can use CoAuthor to formulate hypotheses via replay enactment [29].”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [29]
The Introduction describes replay enactments as a way for designers to explore possible system behaviors, and the Conclusions say they help designers play out the implications of design decisions. Our copy does not mention CoAuthor or say that researchers use it to formulate hypotheses through this method (searched: CoAuthor, formulate hypotheses, hypothes). (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Star, S. L. (1989) · Elsevier eBooks · doi:10.1016/b978-1-55860-092-8.50006-x · Cited by 1,492 · see citing works
Technology probes: inspiring design for and with families
Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B. B., Druin, A., Plaisant, C., Beaudouin-Lafon, M., Conversy, S., Evans, H., Hansen, H., et al. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642616 · Cited by 1,471 · see citing works
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
Self-citationRe-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · cited 9× · see citing works
“These data replays may be enacted either by human wizards who interpret and react to incoming data as it is replayed; by low-fidelity algorithms [35, 36] that approximate the way an eventual system might behave; or by high-fidelity algorithms [14, 17] representing the actual material with which designers will shape a…”
BackgroundInfluentialCognitive Tutor: Applied research in mathematics education
RITTER, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. T. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194060 · Cited by 495 · see citing works
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 2× · see citing works
“Data-driven products and services often encode assumptions reflecting the demographics of the design team, which can have unintended consequences when designing for a global context [18, 30, 33, 38].”
BackgroundA review of recent advances in learner and skill modeling in intelligent learning environments
Desmarais, M. C., & Baker, R. S. (2011) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9106-8 · Cited by 457 · see citing works
Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity
Holstein, K., McLaren, B. M., & Aleven, V. (2019) · Journal of Learning Analytics · doi:10.18608/jla.2019.62.3 · Cited by 429 · see citing works
Self-citationUX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391 · see citing works
Self-citationDesigning as reflective conversation with the materials of a design situation
Schon, D. A. (1992) · Research in Engineering Design · doi:10.1007/bf01580516 · Cited by 334 · see citing works
Elsden, C., Chatting, D. J., Durrant, A. C., Garbett, A., Nissen, B., Vines, J., & Kirk, D. S. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025503 · Cited by 247 · see citing works
The Encyclopedia of Human Computer Interaction
Papin‐Ramcharan, J. (2006) · Issues in Science and Technology Librarianship · doi:10.29173/istl2058 · Cited by 244 · see citing works
A fieldwork of the future with user enactments
Odom, W., Zimmerman, J., Davidoff, S., Forlizzi, J., Dey, A. K., & Lee, M. K. (2012) · Designing Interactive Systems Conference · doi:10.1145/2317956.2318008 · Cited by 176 · see citing works
Self-citationSchlesinger, A., O'Hara, K. P., & Taylor, A. S. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173889 · Cited by 142 · see citing works
Towards Algorithmic Experience
Alvarado, O., & Wærn, A. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173860 · Cited by 139 · see citing works
Toward human-centered algorithm design
Baumer, E. P. (2017) · Big Data & Society · doi:10.1177/2053951717718854 · Cited by 125 · see citing works
Rapidly Exploring Application Design Through Speed Dating
Davidoff, S., Lee, M. K., Dey, A. K., & Zimmerman, J. (2007) · Springer eBooks · doi:10.1007/978-3-540-74853-3_25 · Cited by 96 · see citing works
Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
Yang, Q., Cranshaw, J., Amershi, S., Iqbal, S. T., & Teevan, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300415 · Cited by 90 · see citing works
Escape from the Skinner Box: The case for contemporary intelligent learning environments
du Boulay, B. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12860 · Cited by 80 · see citing works
Speed Dating: Providing a Menu of Possible Futures
Zimmerman, J., & Forlizzi, J. (2017) · She ji · doi:10.1016/j.sheji.2017.08.003 · Cited by 76 · see citing works
Self-citationThe TA Framework: Designing Real-time Teaching Augmentation for K-12 Classrooms
An, P., Holstein, K., d'Anjou, B., Eggen, B., & Bakker, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376277 · Cited by 69 · cited 1× · see citing works
“As a teacher walks throughout the room while wearing Lumilo , they can see real-time indicators about students’ learning, metacognitive, and behavioral states, floating directly above students’ heads [2, 17].”
BackgroundSelf-citationMartínez‐Maldonado, R., Pardo, A., Mirriahi, N., et al. (2016) · Journal of Learning Analytics · doi:10.18608/jla.2015.23.3 · Cited by 65 · see citing works
Digital Systems and the Experience of Legacy
Gulotta, R., Kelliher, A., & Forlizzi, J. (2017) · Designing Interactive Systems Conference · doi:10.1145/3064663.3064731 · Cited by 39 · see citing works
Self-citationTaking software design seriously
Karat, J. (1991) · Cited by 34 · see citing works
Harpstead, E., Myers, B. A., & Aleven, V. (2013) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/2470654.2470667 · Cited by 34 · see citing works
Self-citationChoi, J. O., Forlizzi, J., Christel, M. G., et al. (2016) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/2967934.2968103 · Cited by 34 · see citing works
Self-citationHolstein, K., McLaren, B. M., & Aleven, V. (2017) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3027385.3027450 · Cited by 34 · see citing works
Self-citationOpening Up an Intelligent Tutoring System Development Environment for Extensible Student Modeling
Holstein, K., Yu, Z., Sewall, J., Popescu, O., McLaren, B. M., & Aleven, V. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93843-1_13 · Cited by 19 · see citing works
Self-citationCognitive Tutor Use in Chile: Understanding Classroom and Lab Culture
Ogan, A., Yarzebinski, E., Fernández, P., & Casas, I. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_32 · Cited by 16 · see citing works
Bringing the field into the lab
Newman, M., Ackerman, M. S., Kim, J.-W., et al. (2010) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1866029.1866048 · Cited by 15 · see citing works
Using extracted features to inform alignment-driven design ideas in an educational game
Harpstead, E., MacLellan, C. J., Aleven, V., & Myers, B. A. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557393 · Cited by 13 · see citing works
Self-citationA System-General Model for the Detection of Gaming the System Behavior in CTAT and LearnSphere
Paquette, L., Baker, R. S., & Moskal, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93846-2_47 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Rebecca Gulotta. 2018. Digital Systems and the Material of Legacy: Supporting meaningful interactions with Methods, Prototypes and Portfolios DIS '20, July 6-10, 2020, Eindhoven, Netherlands multigenerational data. Unpublished doctoral dissertation, Carnegie Mellon University.
- Erik Harpstead. 2017. Projective Replay Analysis: A Reflective Approach for Aligning an Educational Game to its Goals. Unpublished doctoral dissertation, Carnegie Mellon University.
- Kenneth Holstein. 2019. Designing Real-time Teacher Augmentation to Combine Strengths of Human and AI Instruction. Unpublished doctoral dissertation, Carnegie Mellon University.
- Jonas Löwgren. 2007. Interaction Design Considered as a Craft. In HCI Remixed, Thomas Erickson and David W McDonald (eds.). The MIT Press, Cambridge, MA. https://doi.org/10.7551/mitpress/7455.003.0041 · DOI
- Christopher J MacLellan, Erik Harpstead, Rony Patel, and Kenneth R Koedinger. 2016. The Apprentice Learner Architecture: Closing the loop between learning theory and educational data. In Proceedings of the 9th International Conference on Educational Data Mining (EDM '16). IEDMS, 151-158.
Cited by
Sorted by citations
Lee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345 · see citing works
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171 · see citing works
Self-citationKawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · see citing works
Self-citationDeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98 · see citing works
Self-citation
- Authors
- Holstein, K., Harpstead, E., Gulotta, R., & Forlizzi, J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2020 ACM Designing Interactive Systems Conference
- Publisher
- ACM
- Published
- 2020-07-03
- Pages
- 1607-1618
- OpenAlex
- W3039540048
- Cited by
- 19
- References
- 33
- Citations by year
- 2026: 2 · 2025: 3 · 2023: 4 · 2022: 6 · 2021: 4
- FWCI
- 2.56
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- data replay, boundary objects, complex systems, design prototyping, design intuition, stakeholder engagement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
