MixT: automatic generation of step-by-step mixed media tutorials
The DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index.
Abstract
Users of complex software applications often learn concepts and skills through step-by-step tutorials. Today, these tutorials are published in two dominant forms: static tutorials composed of images and text that are easy to scan, but cannot effectively describe dynamic interactions; and video tutorials that show all manipulations in detail, but are hard to navigate. We hypothesize that a mixed tutorial with static instructions and per-step videos can combine the benefits of both formats. We describe a comparative study of static, video, and mixed image manipulation tutorials with 12 participants and distill design guidelines for mixed tutorials. We present MixT, a system that automatically generates step-by-step mixed media tutorials from user demonstrations. MixT segments screencapture video into steps using logs of application commands and input events, applies video compositing techniques to focus on salient infor-mation, and highlights interactions through mouse trails. Informal evaluation suggests that automatically generated mixed media tutorials were as effective in helping users complete tasks as tutorials that were created manually.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Furnas, G. W. (1986) · ACM SIGCHI Bulletin · doi:10.1145/22339.22342 · Cited by 1,629
Furnas, G. W. (1986) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/22627.22342 · Cited by 890 · cited 1×
“This video composition effectively creates a focus-plus-context view that makes important regions easier to see at a given video resolution [7].”
Template Matching Techniques in Computer Vision: Theory and Practice
Brunelli, R. (2009) · doi:10.1002/9780470744055 · Cited by 522
Sikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346 · cited 1×
“Methods that analyze user action traces [8,3,10] often capture application-level semantics about specific tools and their purpose, while techniques that analyze screen recordings [18,20,4] use computer vision to identify GUI interactions such as clicking on an icon or button.”
Stencils-based tutorials: design and evaluation
Kelleher, C., & Pausch, R. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055047 · Cited by 187
Chronicle: capture, exploration, and playback of document workflow histories
Grossman, T., Matejka, J., & Fitzmaurice, G. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866054 · Cited by 155 · cited 4×
“Methods that analyze user action traces [8,3,10] often capture application-level semantics about specific tools and their purpose, while techniques that analyze screen recordings [18,20,4] use computer vision to identify GUI interactions such as clicking on an icon or button.”
InfluentialPause-and-play: automatically linking screencast video tutorials with applications
Pongnumkul, S., Dontcheva, M., Li, W., Wang, J., Bourdev, L., Avidan, S., & Cohen, M. F. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047213 · Cited by 144 · cited 6×
“Many efforts propose instructional aids that work in conjunction with the target application, including in-application step-by-step wizards [1,12,6] and Q&A forums [15], video-based tooltips [9], interactive video tutorials [18], command recommenders [14], and interface facades for mapping commands between application…”
InfluentialSelf-citationPrefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
Dixon, M., & Fogarty, J. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753554 · Cited by 137 · cited 1×
“Methods that analyze user action traces [8,3,10] often capture application-level semantics about specific tools and their purpose, while techniques that analyze screen recordings [18,20,4] use computer vision to identify GUI interactions such as clicking on an icon or button.”
CommunityCommands: command recommendations for software applications
Matejka, J., Li, W., Grossman, T., & Fitzmaurice, G. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622214 · Cited by 125 · cited 2×
“Many efforts propose instructional aids that work in conjunction with the target application, including in-application step-by-step wizards [1,12,6] and Q&A forums [15], video-based tooltips [9], interactive video tutorials [18], command recommenders [14], and interface facades for mapping commands between application…”
Self-citationToolClips: an investigation of contextual video assistance for functionality understanding
Grossman, T., & Fitzmaurice, G. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753552 · Cited by 125 · cited 3×
“studied the effectiveness of embedding short (10-25 second) video clips in applications [9].”
Animated Demonstrations for Learning Procedural Computer-Based Tasks
Palmiter, S., & Elkerton, J. (1993) · Human-Computer Interaction · doi:10.1207/s15327051hci0803_1 · Cited by 119
Palmiter, S. L., Elkerton, J., & Baggett, P. (1991) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(91)90019-4 · Cited by 111 · cited 4×
“Palmiter, Elkerton [16,17], and Harrison [11] studied the effect of animated demonstrations on learning and instruction recall.”
InfluentialSketch-sketch revolution: an engaging tutorial system for guided sketching and application learning
Fernquist, J., Grossman, T., & Fitzmaurice, G. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047245 · Cited by 98 · cited 3×
“Many efforts propose instructional aids that work in conjunction with the target application, including in-application step-by-step wizards [1,12,6] and Q&A forums [15], video-based tooltips [9], interactive video tutorials [18], command recommenders [14], and interface facades for mapping commands between application…”
DocWizards: a system for authoring follow-me documentation wizards
Bergman, L., Castelli, V., Lau, T., & Oblinger, D. (2005) · Proceedings of the 18th annual ACM symposium on User interface software and technology · doi:10.1145/1095034.1095067 · Cited by 96 · cited 2×
“Many efforts propose instructional aids that work in conjunction with the target application, including in-application step-by-step wizards [1,12,6] and Q&A forums [15], video-based tooltips [9], interactive video tutorials [18], command recommenders [14], and interface facades for mapping commands between application…”
Harrison, S. M. (1995) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '95 · doi:10.1145/223904.223915 · Cited by 83 · cited 4×
“Palmiter, Elkerton [16,17], and Harrison [11] studied the effect of animated demonstrations on learning and instruction recall.”
InfluentialGenerating photo manipulation tutorials by demonstration
Grabler, F., Agrawala, M., Li, W., Dontcheva, M., & Igarashi, T. (2009) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1576246.1531372 · Cited by 54 · cited 5×
“'s system for generating static tutorials from demonstration to include videos [8].”
InfluentialSelf-citationDelta: a tool for representing and comparing workflows
Kong, N., Grossman, T., Hartmann, B., Agrawala, M., & Fitzmaurice, G. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208549 · Cited by 52
Self-citationIP-QAT: in-product questions, answers, & tips
Matejka, J., Grossman, T., & Fitzmaurice, G. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047218 · Cited by 43
MeshFlow: interactive visualization of mesh construction sequences
Denning, J. D., Kerr, W. B., & Pellacini, F. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964961 · Cited by 40 · cited 2×
“Methods that analyze user action traces [8,3,10] often capture application-level semantics about specific tools and their purpose, while techniques that analyze screen recordings [18,20,4] use computer vision to identify GUI interactions such as clicking on an icon or button.”
Searching for software learning resources using application context
Ekstrand, M., Li, W., Grossman, T., Matejka, J., & Fitzmaurice, G. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047220 · Cited by 37 · cited 1×
“RELATED WORK Previous HCI research on instructional content falls into four main categories: 1) new tutorial formats and interfaces [1,6,9,12,14,15,18]; 2) automated methods for generating tutorials [3,8,10,18]; 3) studies evaluating the effectiveness of different instructional formats [8,9,11,16,17]; and 4) technique…”
Self-citationShowMeHow: translating user interface instructions between applications
Ramesh, V., Hsu, C., Agrawala, M., & Hartmann, B. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047212 · Cited by 27
Self-citation
Cited by
Crowdsourcing step-by-step information extraction to enhance existing how-to videos
Kim, J., Nguyen, P. T., Weir, S., Guo, P. J., Miller, R. C., & Gajos, K. Z. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2556986 · Cited by 130
AdapTutAR: An Adaptive Tutoring System for Machine Tasks in Augmented Reality
Huang, G., Qian, X., Wang, T., Patel, F., Sreeram, M., Cao, Y., Ramani, K., & Quinn, A. J. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445283 · Cited by 119
Learnersourcing Subgoal Labels for How-to Videos
Weir, S., Kim, J., Gajos, K. Z., & Miller, R. C. (2015) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2675133.2675219 · Cited by 111
Chi, P.-Y., Liu, J., Linder, J., Dontcheva, M., Li, W., & Hartmann, B. (2013) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2501988.2502052 · Cited by 75
Self-citationSelective Undo Support for Painting Applications
Myers, B. A., Lai, A., Le, T. M., Yoon, Y., Faulring, A., & Brandt, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702543 · Cited by 73
Metadata
- Authors
- Chi, P.-Y., Ahn, S., Ren, A., Dontcheva, M., Li, W., & Hartmann, B.
- Affiliations
- University of California, Berkeley (US)
- Adobe Systems (United States) (US)
- Venue
- Proceedings of the 25th annual ACM symposium on User interface software and technology
- Publisher
- ACM
- Published
- 2012-10-07
- Pages
- 93-102
- OpenAlex
- W2029044878
- Cited by
- 113
- References
- 22
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 13 · 2023: 15 · 2022: 6 · 2021: 7 · 2020: 6 · 2019: 3 · 2018: 7 · 2017: 3 · 2016: 3 · 2015: 12 · 2014: 12 · 2013: 7
- FWCI
- 6.97
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- software tutorial, video tutorials, video compositing, task completion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.