The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
Evaluation Strategies for HCI Toolkit Research
Open accessAbstract
Toolkit research plays an important role in the field of HCI, as it can heavily influence both the design and implementation of interactive systems. For publication, the HCI community typically expects toolkit research to include an evaluation component. The problem is that toolkit evaluation is challenging, as it is often unclear what 'evaluating' a toolkit means and what methods are appropriate. To address this problem, we analyzed 68 published toolkit papers. From our analysis, we provide an overview of, reflection on, and discussion of evaluation methods for toolkit contributions. We identify and discuss the value of four toolkit evaluation strategies, including the associated techniques that each employs. We offer a categorization of evaluation strategies for toolkit researchers, along with a discussion of the value, potential limitations, and trade-offs associated with each strategy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2014) · Cited by 15,747
The Computer for the 21st Century
Weiser, M. (1991) · Scientific American · doi:10.1038/scientificamerican0991-94 · Cited by 10,017
(1993) · Cited by 9,879
Design science in information systems research
(2004) · Journal of the Association for Information Systems · doi:10.5555/2017212.2017217 · Cited by 8,254
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 1×
“For instance, 30 fps is often used for realtime tracking systems [79].”
MethodContext-Aware Computing Applications
Schilit, B., Adams, N., & Want, R. (1994) · 1994 First Workshop on Mobile Computing Systems and Applications · doi:10.1109/wmcsa.1994.16 · Cited by 3,231
Bostock, M., Ogievetsky, V., & Heer, J. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.185 · Cited by 3,165
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Sketchpad: a man-machine graphical communication system
Sutherland, I. E. (1963) · Proceedings of the May 21-23, 1963, spring joint computer conference on - AFIPS '63 (Spring) · doi:10.1145/1461551.1461591 · Cited by 1,821
The DaCapo benchmarks: java benchmarking development and analysis
Blackburn, S. M., Garner, R., Hoffmann, C., Khang, A. M., McKinley, K. S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S. Z., et al. (2006) · Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications · doi:10.1145/1167473.1167488 · Cited by 1,600
A Practical Guide to Usability Testing
Dumas, J. F., & Redish, J. C. (1993) · Cited by 1,335
The context toolkit: aiding the development of context-enabled applications
Salber, D., Dey, A. K., & Abowd, G. D. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303126 · Cited by 1,189
(2017) · Research Methods in Human Computer Interaction · doi:10.1016/b978-0-12-805390-4.09991-x · Cited by 889
Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes
Wobbrock, J. O., Wilson, A. D., & Li, Y. (2007) · Proceedings of the 20th annual ACM symposium on User interface software and technology · doi:10.1145/1294211.1294238 · Cited by 822
Improving a human-computer dialogue
Molich, R., & Nielsen, J. (1990) · Communications of the ACM · doi:10.1145/77481.77486 · Cited by 815
Past, present, and future of user interface software tools
Myers, B., Hudson, S. E., & Pausch, R. (2000) · ACM Transactions on Computer-Human Interaction · doi:10.1145/344949.344959 · Cited by 808
prefuse: a toolkit for interactive information visualization
Heer, J., Card, S. K., & Landay, J. A. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055031 · Cited by 802
Cognitive walkthroughs: a method for theory-based evaluation of user interfaces
Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(92)90039-n · Cited by 797
Phidgets: easy development of physical interfaces through physical widgets
Greenberg, S., & Fitchett, C. (2001) · Proceedings of the 14th annual ACM symposium on User interface software and technology · doi:10.1145/502348.502388 · Cited by 652
Self-citationThe bubble cursor: enhancing target acquisition by dynamic resizing of the cursor's activation area
Grossman, T., & Balakrishnan, R. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055012 · Cited by 604
Usability evaluation considered harmful (some of the time)
Greenberg, S., & Buxton, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357074 · Cited by 598
Self-citationBuilding real-time groupware with GroupKit, a groupware toolkit
Roseman, M., & Greenberg, S. (1996) · ACM Transactions on Computer-Human Interaction · doi:10.1145/226159.226162 · Cited by 509
Self-citationIntelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471
Research contributions in human-computer interaction
Wobbrock, J. O., & Kientz, J. A. (2016) · Interactions · doi:10.1145/2907069 · Cited by 416
Proxemic interaction: designing for a proximity and orientation-aware environment
Ballendat, T., Marquardt, N., & Greenberg, S. (2010) · ACM International Conference on Interactive Tabletops and Surfaces · doi:10.1145/1936652.1936676 · Cited by 413
Self-citationSikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346
DENIM: finding a tighter fit between tools and practice for Web site design
Lin, J., Newman, M. W., Hong, J. I., & Landay, J. A. (2000) · Proceedings of the SIGCHI conference on Human Factors in Computing Systems · doi:10.1145/332040.332486 · Cited by 337
DiamondSpin: an extensible toolkit for around-the-table interaction
Shen, C., Vernier, F. D., Forlines, C., & Morris, M. R. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985714 · Cited by 336
Cooper, A. (2004) · Pearson Higher Education eBooks · Cited by 335
Protovis: A Graphical Toolkit for Visualization
Bostock, M., & Heer, J. (2009) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2009.174 · Cited by 321 · cited 2×
“website loading time [14], spatial resolution [33], framerate [28,51], GPU usage [51], memory allocation [13,51], load time [13], lines of source code [2,91], size of binary [2]).”
BackgroundMethodAutobiographical design in HCI research: designing and learning through use-it-yourself
Neustaedter, C., & Sengers, P. (2012) · Proceedings of the Designing Interactive Systems Conference · doi:10.1145/2317956.2318034 · Cited by 302
Reflective physical prototyping through integrated design, test, and analysis
Hartmann, B., Klemmer, S. R., Bernstein, M., Abdulla, L., Burr, B., Robinson-Mosher, A., & Gee, J. (2006) · Proceedings of the 19th annual ACM symposium on User interface software and technology · doi:10.1145/1166253.1166300 · Cited by 302
METHODOLOGY MATTERS: DOING RESEARCH IN THE BEHAVIORAL and SOCIAL SCIENCES
MCGRATH, J. E. (1995) · Readings in Human–Computer Interaction · doi:10.1016/b978-0-08-051574-8.50019-4 · Cited by 296
Toolkit design for interactive structured graphics
Bederson, B. B., Grosjean, J., & Meyer, J. (2004) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2004.44 · Cited by 290
The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies
Marquardt, N., Diaz-Marino, R., Boring, S., & Greenberg, S. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047238 · Cited by 276
Self-citationContext as a Dynamic Construct
Greenberg, S. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_09 · Cited by 264
Self-citationDART: a toolkit for rapid design exploration of augmented reality experiences
MacIntyre, B., Gandy, M., Dow, S., & Bolter, J. D. (2004) · Proceedings of the 17th annual ACM symposium on User interface software and technology · doi:10.1145/1029632.1029669 · Cited by 264
A toolkit for managing user attention in peripheral displays
Matthews, T., Dey, A. K., Mankoff, J., Carter, S., & Rattenbury, T. (2004) · Proceedings of the 17th annual ACM symposium on User interface software and technology · doi:10.1145/1029632.1029676 · Cited by 244
Evaluating user interface systems research
Olsen, D. R. (2007) · Proceedings of the 20th annual ACM symposium on User interface software and technology · doi:10.1145/1294211.1294256 · Cited by 234
HCI Research as Problem-Solving
Oulasvirta, A., & Hornbæk, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858283 · Cited by 222
iStuff: a physical user interface toolkit for ubiquitous computing environments
Ballagas, R., Ringel, M., Stone, M., & Borchers, J. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642705 · Cited by 214
Papier-Mache: toolkit support for tangible input
Klemmer, S. R., Li, J., Lin, J., & Landay, J. A. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985743 · Cited by 201
Cognitive Dimensions of Notations: Design Tools for Cognitive Technology
Blackwell, A. F., Britton, C., Cox, A., Green, T. R. G., Gurr, C., Kadoda, G., Kutar, M. S., Loomes, M., Nehaniv, C. L., Petre, M., et al. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44617-6_31 · Cited by 195
CrossY: a crossing-based drawing application
Apitz, G., & Guimbretière, F. (2004) · Proceedings of the 17th annual ACM symposium on User interface software and technology · doi:10.1145/1029632.1029635 · Cited by 190
Midas: fabricating custom capacitive touch sensors to prototype interactive objects
Savage, V., Zhang, X., & Hartmann, B. (2012) · Proceedings of the 25th annual ACM symposium on User interface software and technology · doi:10.1145/2380116.2380189 · Cited by 176
Physikit: Data Engagement Through Physical Ambient Visualizations in the Home
Houben, S., Golsteijn, C., Gallacher, S., Johnson, R., Bakker, S., Marquardt, N., Capra, L., & Rogers, Y. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858059 · Cited by 159
Self-citationHartmann, B., Abdulla, L., Mittal, M., & Klemmer, S. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240646 · Cited by 154
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
Dixon, M., & Fogarty, J. A. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753554 · Cited by 150
Toolkit to support intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2010) · Proceedings of the 12th ACM international conference on Ubiquitous computing · doi:10.1145/1864349.1864353 · Cited by 143
Employing patterns and layers for early-stage design and prototyping of cross-device user interfaces
Lin, J. W., & Landay, J. A. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357260 · Cited by 128
Hong, J., & Landay, J. A. (2000) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/354401.354412 · Cited by 118
Toolkits and interface creativity
Greenberg, S. (2006) · Multimedia Tools and Applications · doi:10.1007/s11042-006-0062-y · Cited by 117
Self-citationSavage, V., Chang, C., & Hartmann, B. (2013) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2501988.2501992 · Cited by 113
Ramakers, R., Anderson, F., Grossman, T., & Fitzmaurice, G. W. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858485 · Cited by 110
The implications of method placement on API learnability
Stylos, J., & Myers, B. A. (2008) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1453101.1453117 · Cited by 108
Providing integrated toolkit-level support for ambiguity in recognition-based interfaces
Mankoff, J. C., Hudson, S. E., & Abowd, G. D. (2000) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/332040.332459 · Cited by 106
Lee, J. C., Avrahami, D., Hudson, S. E., Forlizzi, J., Dietz, P., & Leigh, D. (2004) · Designing Interactive Systems Conference · doi:10.1145/1013115.1013139 · Cited by 103
The infrastructure problem in HCI
Edwards, W. K., Newman, M., & Poole, E. S. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753390 · Cited by 103
PolyChrome: A Cross-Device Framework for Collaborative Web Visualization
Badam, S. K., & Elmqvist, N. (2014) · ACM International Conference on Interactive Tabletops and Surfaces (ITS) · doi:10.1145/2669485.2669518 · Cited by 97
Wigdor, D., Benko, H., Pella, J., Lombardo, J., & Williams, S. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979173 · Cited by 96
Xiao, R., Harrison, C., & Hudson, S. E. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466113 · Cited by 89
OpenCapSense: A rapid prototyping toolkit for pervasive interaction using capacitive sensing
Große-Puppendahl, T., Berghöfer, Y., Braun, A., Wimmer, R., & Kuijper, A. (2013) · IEEE International Conference on Pervasive Computing and Communications · doi:10.1109/percom.2013.6526726 · Cited by 86
Support for context-aware intelligibility and control
Dey, A. K., & Newberger, A. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518832 · Cited by 83
PaperPulse: An Integrated Approach for Embedding Electronics in Paper Designs
Ramakers, R., Todi, K., & Luyten, K. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702487 · Cited by 75
Pfeiffer, M., Duente, T., & Rohs, M. (2016) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/2935334.2935348 · Cited by 73
Xiao, R., Harrison, C., & Hudson, S. E. (2013) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2468356.2479563 · Cited by 67
Gummy for multi-platform user interface designs
Meskens, J., Vermeulen, J., Luyten, K., & Coninx, K. (2008) · Working Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/1385569.1385607 · Cited by 63
Self-citationSeyed, T., Azazi, A., Chan, E., Wang, Y., & Maurer, F. (2015) · ACM International Conference on Interactive Tabletops and Surfaces (ITS) · doi:10.1145/2817721.2817750 · Cited by 61
The trouble with social computing systems research
Bernstein, M. S., Ackerman, M. S., Ed, H., & Miller, R. C. (2011) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1979742.1979618 · Cited by 53
The MAUI Toolkit: Groupware Widgets for Group Awareness
Hill, J., & Gutwin, C. A. (2004) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-004-5063-7 · Cited by 53
A Capacitive Sensing Toolkit for Pervasive Activity Detection and Recognition
Wimmer, R., Kranz, M., Boring, S., & Schmidt, A. (2007) · IEEE International Conference on Pervasive Computing and Communications · doi:10.1109/percom.2007.1 · Cited by 50
EagleSense: Tracking People and Devices in Interactive Spaces using Real-Time Top-View Depth-Sensing
Wu, C.-J., Houben, S., & Marquardt, N. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025562 · Cited by 49
Self-citationEvaluation of Prototypes and the Problem of Possible Futures
Salovaara, A., Oulasvirta, A., & Jacucci, G. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025658 · Cited by 48
Preece, J. J., & Rombach, H. D. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1073 · Cited by 47
Pineal: Bringing Passive Objects to Life with Embedded Mobile Devices
Ledo, D., Anderson, F., Schmidt, R., Oehlberg, L., Greenberg, S., & Grossman, T. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025652 · Cited by 47
Self-citationUser interfaces when and where they are needed: an infrastructure for recombinant computing
Newman, M., Izadi, S., Edwards, W. K., Sedivy, J. Z., & Smith, T. F. (2002) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/571985.572009 · Cited by 44
Kaufmann, B., & Buechley, L. (2010) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/1851600.1851652 · Cited by 36
Concepts, Values, and Methods for Technical Human–Computer Interaction Research
Hudson, S. E., & Mankoff, J. C. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0378-8_4 · Cited by 33
What makes a prototype novel? a knowledge contribution concern for interaction design research
Wiberg, M., & Stolterman, E. (2014) · Nordic Conference on Human-Computer Interaction · doi:10.1145/2639189.2639487 · Cited by 27
Distributed physical interfaces with shared phidgets
Marquardt, N., & Greenberg, S. (2007) · International Conference on Tangible and Embedded Interaction (TEI) · doi:10.1145/1226969.1226973 · Cited by 26
Self-citationPrefab layers and prefab annotations: extensible pixel-based interpretation of graphical interfaces
Dixon, M., Nied, A. C., & Fogarty, J. A. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647412 · Cited by 24
Xiangang, Q. (2024) · Springer eBooks · doi:10.1007/978-981-97-7874-4_875 · Cited by 22
D-Macs: building multi-device user interfaces by demonstrating, sharing and replaying design actions
Meskens, J., Luyten, K., & Coninx, K. (2010) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1866029.1866051 · Cited by 20
Wiethoff, A., Schneider, H., Küfner, J., Rohs, M., Butz, A. M., & Greenberg, S. (2013) · Creativity and Cognition · doi:10.1145/2466627.2466635 · Cited by 17
Self-citation
Cited by
Mapping the Landscape of Creativity Support Tools in HCI
Frich, J., MacDonald Vermeulen, L., Remy, C., Biskjaer, M. M., & Dalsgaard, P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300619 · Cited by 410
“Three of the most recent examples are a survey of evaluation strategies for prototyping toolkits [104], a recipe for evaluating sustainability in HCI [138], and a triangulation of different evaluation methods [132].”
BackgroundBrudy, F., Holz, C., Rädle, R., Wu, C.-J., Houben, S., Klokmose, C. N., & Marquardt, N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300792 · Cited by 304
Self-citationAshtari, N., Bunt, A., McGrenere, J., Nebeling, M., & Chilana, P. K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376722 · Cited by 273
ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Arawjo, I., Swoopes, C., Vaithilingam, P., Wattenberg, M., & Glassman, E. (2023) · arXiv · doi:10.48550/arxiv.2309.09128 · Cited by 146
AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications
Evangelista Belo, J. M., Lystbæk, M. N., Feit, A. M., Pfeuffer, K., Kán, P., Oulasvirta, A., & Grønbæk, K. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545651 · Cited by 117
Metadata
- Authors
- Ledo, D., Houben, S., Vermeulen, J., Marquardt, N., Oehlberg, L., & Greenberg, S.
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-19
- Pages
- 1-17
- OpenAlex
- W2775663585
- Cited by
- 315
- References
- 84
- FWCI
- 14.59
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.