The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
Evaluation Strategies for HCI Toolkit Research
Open accessAbstract
Toolkit research plays an important role in the field of HCI, as it can heavily influence both the design and implementation of interactive systems. For publication, the HCI community typically expects toolkit research to include an evaluation component. The problem is that toolkit evaluation is challenging, as it is often unclear what 'evaluating' a toolkit means and what methods are appropriate. To address this problem, we analyzed 68 published toolkit papers. From our analysis, we provide an overview of, reflection on, and discussion of evaluation methods for toolkit contributions. We identify and discuss the value of four toolkit evaluation strategies, including the associated techniques that each employs. We offer a categorization of evaluation strategies for toolkit researchers, along with a discussion of the value, potential limitations, and trade-offs associated with each strategy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2014) · Cited by 15,747
The Computer for the 21st Century
Weiser, M. (1991) · Scientific American · doi:10.1038/scientificamerican0991-94 · Cited by 10,017
(1993) · Cited by 9,879
Design science in information systems research
(2004) · Journal of the Association for Information Systems · doi:10.5555/2017212.2017217 · Cited by 8,254
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008
Context-Aware Computing Applications
Schilit, B., Adams, N., & Want, R. (1994) · 1994 First Workshop on Mobile Computing Systems and Applications · doi:10.1109/wmcsa.1994.16 · Cited by 3,231
Bostock, M., Ogievetsky, V., & Heer, J. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.185 · Cited by 3,165
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Sketchpad: a man-machine graphical communication system
Sutherland, I. E. (1963) · Proceedings of the May 21-23, 1963, spring joint computer conference on - AFIPS '63 (Spring) · doi:10.1145/1461551.1461591 · Cited by 1,821
The DaCapo benchmarks: java benchmarking development and analysis
Blackburn, S. M., Garner, R., Hoffmann, C., Khang, A. M., McKinley, K. S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S. Z., et al. (2006) · Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications · doi:10.1145/1167473.1167488 · Cited by 1,600
Cited by
Mapping the Landscape of Creativity Support Tools in HCI
Frich, J., MacDonald Vermeulen, L., Remy, C., Biskjaer, M. M., & Dalsgaard, P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300619 · Cited by 410
Brudy, F., Holz, C., Rädle, R., Wu, C.-J., Houben, S., Klokmose, C. N., & Marquardt, N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300792 · Cited by 304
Self-citationAshtari, N., Bunt, A., McGrenere, J., Nebeling, M., & Chilana, P. K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376722 · Cited by 273
ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Arawjo, I., Swoopes, C., Vaithilingam, P., Wattenberg, M., & Glassman, E. (2023) · arXiv · doi:10.48550/arxiv.2309.09128 · Cited by 146
AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications
Evangelista Belo, J. M., Lystbæk, M. N., Feit, A. M., Pfeuffer, K., Kán, P., Oulasvirta, A., & Grønbæk, K. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545651 · Cited by 117
Metadata
- Authors
- Ledo, D., Houben, S., Vermeulen, J., Marquardt, N., Oehlberg, L., & Greenberg, S.
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-19
- Pages
- 1-17
- OpenAlex
- W2775663585
- Cited by
- 315
- References
- 84
- FWCI
- 14.59
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation Strategies for HCI Toolkit Research. https://app.cochat.ai/papers/10.1145/3173574.3173610
- The paper
- Ledo, D., Houben, S., Vermeulen, J., Marquardt, N., Oehlberg, L., & Greenberg, S. (2018). Evaluation Strategies for HCI Toolkit Research. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3173610
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.