The think aloud method: a guide to user interface design
Abstract
Objectives Clinical use of computer systems has been hampered by their poorly designed user interfaces. The objective of this study was to design a user interface for a pediatric oncologists' computerized patient record with great consideration of their working behavior and of human computer interfacing principles so as to contribute to oncologists' efficiency and satisfaction in interaction with the system.Methods The think aloud method was used in combination with video recording to get a deep understanding of the way in which four pediatric oncologists searched through the paper-based patient record in preparing a patient visit. Protocol and video analyses was used to develop a cognitive task model reflecting pediatric oncologists' task behavior. This model was input for a prototype user interface, which was subsequently evaluated by eight other pediatric oncologists.Results The resulting computerized medical record system proved to meet pediatric oncologists' information needs and task behavior patterns. The design of the user interface minimized pediatric oncologists' work load and was highly efficient in supporting the pediatric oncologists in preparing their patient visits. The pediatric oncologists were very much satisfied with the computer system.Conclusions It is argued that early involvement of cognitive engineering methods in the system design process may be of great help in designing systems that fully support health care professionals' work practices. The think aloud method, if applied under prescribed conditions, is a valuable information source of human task-behavior and as such a useful technique for requirements analysis in designing clinical computer systems.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The think-aloud protocol was implemented to enable participants to voice their thoughts and issues.
“The think-aloud protocol was implemented to enable participants to voice their thoughts and issues [57].”
From Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study (2022) · cites this paper as [57]
Cites
Practical Statistics for Medical Research.
Walter, S., & Altman, D. G. (1992) · Biometrics · doi:10.2307/2532320 · Cited by 11,665 · see citing works
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317 · cited 1× · see citing works
“In this notion, verbalizing thoughts is viewed as a cognitive process on its own and thus it is argued leads to incomplete or disturbed accounts of the cognitive processes under consideration [ 44 ].”
Background(1993) · Cited by 9,879 · see citing works
Carayannis, E. G. (2013) · Springer eBooks · doi:10.1007/978-1-4614-3858-8_100431 · Cited by 4,818 · see citing works
Book Review: Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,361 · see citing works
Norman, D. A., & DRAPER, S. W. (1986) · CRC Press eBooks · doi:10.1201/b15703 · Cited by 3,154 · see citing works
Development of an instrument measuring user satisfaction of the human-computer interface
Chin, J. P., Diehl, V. A., & Norman, L. (1988) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/57167.57203 · Cited by 1,631 · see citing works
Thinking aloud: reconciling theory and practice
Boren, T., & Ramey, J. A. (2000) · IEEE Transactions on Professional Communication · doi:10.1109/47.867942 · Cited by 849 · see citing works
Cognitive and usability engineering methods for the evaluation of clinical information systems.
Kushniruk, A. W., & Patel, V. L. (2004) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2004.01.003 · Cited by 760 · see citing works
Galitz, W. O. (1996) · Cited by 680 · see citing works
Evaluation of health information systems—problems and challenges
Ammenwerth, E., Gräber, S., Herrmann, G., Bürkle, T., & König, J. (2003) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(03)00131-x · Cited by 414 · see citing works
Techniques for requirements elicitation
Goguen, J. A., & Linde, C. (2002) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/isre.1993.324822 · Cited by 409 · see citing works
Learning Issues for Intelligent Tutoring Systems
Psotka, J., Mandl, H., & Lesgold, A. (1988) · Cognitive Sciences · doi:10.1007/978-1-4684-6350-7 · Cited by 380 · see citing works
Kaplan, B. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00184-8 · Cited by 380 · see citing works
Graphical summary of patient status
Powsner, S. M. (1994) · The Lancet · doi:10.1016/s0140-6736(94)91406-0 · Cited by 216 · see citing works
Wiedenbeck, S. (1999) · Behaviour and Information Technology · doi:10.1080/014492999119129 · Cited by 183 · see citing works
Berg, M., Langenberg, C. J. M., Berg, I. V. D., & Kwakkernaat, J. (1998) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(98)00143-9 · Cited by 183 · see citing works
Cimino, J. J., Patel, V. L., & Kushniruk, A. W. (2002) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(02)00070-9 · Cited by 164 · see citing works
Cognitive engineering principles for enhancing human‐computer performance
Gerhardt‐Powals, J. (1996) · International Journal of Human-Computer Interaction · doi:10.1080/10447319609526147 · Cited by 157 · see citing works
Applying cognitive psychology to user-interface design
(1988) · Applied Ergonomics · doi:10.1016/0003-6870(88)90043-9 · Cited by 140 · see citing works
van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994) · Academic Press eBooks · Cited by 127 · see citing works
Reading the medical record. I. Analysis of physician's ways of reading the medical record
Nygren, E., & Henriksson, P. (1992) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(92)90053-a · Cited by 116 · see citing works
Sittig, D. F., Kuperman, G. J., & Fiskio, J. M. (1999) · PubMed · Cited by 100 · see citing works
Kushniruk, A. W., Kaufman, D. R., Patel, V. L., Lévesque, Y., & Lottin, P. (1996) · PubMed · Cited by 98 · see citing works
Eberts, R. E. (1994) · Prentice-Hall, Inc eBooks · Cited by 87 · see citing works
Studying the Human--Computer--Terminology Interface
Cimino, J. J., Patel, V. L., & Kushniruk, A. W. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080163 · Cited by 85 · see citing works
Interface design for health care environments: the role of cognitive science.
Patel, V. L., & Kushniruk, A. W. (1998) · PubMed · Cited by 84 · see citing works
Design should help use of patients' data
Wyatt, J. C., & Wright, P. (1998) · The Lancet · doi:10.1016/s0140-6736(97)08306-2 · Cited by 83 · see citing works
Beuscart-Zéphir, M.-C., Anceaux, F., Crinquette, V., et al. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00210-6 · Cited by 82 · see citing works
Helping clinicians to find data and avoid delays
Nygren, E., Wyatt, J. C., & Wright, P. (1998) · The Lancet · doi:10.1016/s0140-6736(97)08307-4 · Cited by 82 · see citing works
Tang, P. C., & Patel, V. L. (1994) · International Journal of Bio-Medical Computing · doi:10.1016/0020-7101(94)90017-5 · Cited by 78 · see citing works
Target Paper: Conception of the cognitive engineering design problem
Dowell, J. D., & Long, J. B. (1998) · Ergonomics · doi:10.1080/001401398187125 · Cited by 77 · see citing works
Kushniruk, A. W., & Patel, V. L. (1998) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(98)00106-3 · Cited by 70 · see citing works
Cognitive evaluation: How to assess the usability of information technology in healthcare
Beuscart-Zéphir, M.-C., Brender, J., Beuscart, R., & Ménager-Depriester, I. (1997) · Computer Methods and Programs in Biomedicine · doi:10.1016/s0169-2607(97)00030-8 · Cited by 58 · see citing works
Mental Models and Metaphors: Implications for the Design of Adaptive User-System Interfaces
Streitz, N. (1988) · Springer eBooks · doi:10.1007/978-1-4684-6350-7_8 · Cited by 56 · see citing works
Kushniruk, A. W., Patel, C., Patel, V. L., & Cimino, J. J. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(00)00133-7 · Cited by 53 · see citing works
A conception of the cognitive engineering design problem
Dowell, J. D., & Long, J. B. (1998) · UCL Discovery (University College London) · Cited by 52 · see citing works
Cognitive evaluation of the user interface and vocabulary of an outpatient information system.
Kushniruk, A. W., Patel, V. L., Cimino, J. J., et al. (1996) · PubMed · Cited by 46 · see citing works
Modelling work practices: input to the design of a physician's workstation.
Fafchamps, D., Young, C. Y., & Tang, P. C. (1991) · PubMed · Cited by 41 · see citing works
Physicians' workstations: integrated information management for clinicians.
Tang, P. C., Annevelink, J., Fafchamps, D., et al. (1991) · PubMed · Cited by 35 · see citing works
Kushniruk, A. W., & Patel, V. L. (1995) · PubMed · Cited by 35 · see citing works
Gadd, C. S., Baskaran, P., & Lobach, D. F. (1998) · PubMed · Cited by 34 · see citing works
Nygren, E., Johnson, M., & Henriksson, P. (1992) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(92)90054-b · Cited by 32 · see citing works
Clinical data systems, part 2: components and techniques
Wyatt, J. (1994) · The Lancet · doi:10.1016/s0140-6736(94)90409-x · Cited by 31 · see citing works
Representation still matters: Cognitive engineering and user interface design
Stary, C., & Peschl, M. F. (1998) · Behaviour and Information Technology · doi:10.1080/014492998119292 · Cited by 23 · see citing works
Zeng, Q., & Cimino, J. J. (1999) · PubMed · Cited by 18 · see citing works
Verbal Protocols as a Research Tool in Human Factors: Symposium Abstract
Deffner, G. (1990) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129003401718 · Cited by 8 · see citing works
Acceptability and usage patterns of an image analysis workstation.
Boxwala, A. A., Friedman, C. P., Fritsch, D., et al. (1997) · PubMed · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Jaspers, M. W. M. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.10.002 · Cited by 649 · see citing works
Self-citationDesign and Evaluation in eHealth: Challenges and Implications for an Interdisciplinary Field
Pagliari, C. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e15 · Cited by 275 · see citing works
A user-centred design framework for mHealth
Farao, J., Malila, B., Conrad, N., Mutsvangwa, T., Rangaka, M. X., & Douglas, T. S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237910 · Cited by 203 · see citing works
“Observations [42], the “think-aloud” protocol [43], and the post-study system usability questionnaire [44] (a 19-item survey instrument to assess user satisfaction) were used as data collection tools, with the latter used in the final iteration once improvements had been made.”
MethodSolving the Explainable AI Conundrum by Bridging Clinicians’ Needs and Developers’ Goals
Bienefeld, N., Boss, J. M., Lüthy, R., Brodbeck, D., Azzati, J., Blaser, M., Willms, J., & Keller, E. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00837-4 · Cited by 159 · see citing works
Crane, D., Garnett, C., Brown, J., West, R., & Michie, S. (2017) · Frontiers in Public Health · doi:10.3389/fpubh.2017.00039 · Cited by 82 · see citing works
Anthierens, S., Tonkin-Crine, S., Douglas, E., Fernandez-Vandellos, P., Krawczyk, J., Llor, C., Cals, J. W., Francis, N. A., Yardley, L., Coenen, S., et al. (2012) · BMC Family Practice · doi:10.1186/1471-2296-13-101 · Cited by 75 · see citing works
Olde Keizer, R. A. C. M., van Velsen, L., Moncharmont, M., Riche, B., Ammour, N., Del Signore, S., Zia, G., Hermens, H., & N’Dja, A. (2019) · Health and Technology · doi:10.1007/s12553-019-00320-9 · Cited by 57 · see citing works
Baretta, D., Perski, O., & Steca, P. (2019) · JMIR mHealth and uHealth · doi:10.2196/11636 · Cited by 52 · see citing works
“Think aloud is a method that requires participants to verbalize their thoughts and impressions while performing a task [23].”
MethodBogza, L.-M., Patry-Lebeau, C., Farmanova, E., Witteman, H. O., Elliott, J., Stolee, P., Hudon, C., & Giguere, A. M. C. (2020) · Journal of Medical Internet Research · doi:10.2196/17406 · Cited by 52 · see citing works
Moradian, S., Krzyzanowska, M. K., Maguire, R., Morita, P. P., Kukreti, V., Avery, J., Liu, G., Cafazzo, J., & Howell, D. (2018) · JMIR Cancer · doi:10.2196/10932 · Cited by 56 · see citing works
Badr, H., Lipnick, D., Diefenbach, M. A., Posner, M., Kotz, T., Miles, B., & Genden, E. (2016) · European Journal of Cancer Care · doi:10.1111/ecc.12396 · Cited by 55 · see citing works
Breeman, L. D., Keesman, M., Atsma, D. E., Chavannes, N. H., Janssen, V., van Gemert-Pijnen, L., Kemps, H., Kraaij, W., Rauwers, F., Reijnders, T., et al. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104364 · Cited by 37 · see citing works
Kuijpers, W., Groen, W. G., Oldenburg, H. S., Wouters, M. W., Aaronson, N. K., & van Harten, W. H. (2015) · JMIR Research Protocols · doi:10.2196/resprot.3796 · Cited by 32 · see citing works
Mendes-Santos, C., Nunes, F., Weiderpass, E., Santana, R., & Andersson, G. (2022) · JMIR Cancer · doi:10.2196/33550 · Cited by 19 · see citing works
“The think-aloud protocol was implemented to enable participants to voice their thoughts and issues [57].”
MethodFarri, O., Monsen, K. A., Pakhomov, S. V., Pieczkiewicz, D. S., Speedie, S. M., & Melton, G. B. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.08.009 · Cited by 19 · see citing works
Robertson, E. G., Wakefield, C. E., Cohn, R. J., O'Brien, T., Ziegler, D. S., & Fardell, J. E. (2018) · JMIR Research Protocols · doi:10.2196/resprot.9258 · Cited by 17 · see citing works
Babatunde, F. O., MacDermid, J., Grewal, R., Macedo, L., & Szekeres, M. (2020) · JMIR Human Factors · doi:10.2196/17088 · Cited by 12 · see citing works
“This enabled the researcher to capture the ongoing thought processes of the participants while going over the program and any difficulties encountered [69].”
Accessible Maps for the Future of Inclusive Ridesharing
Fink, P. D. S., Milne, H., Caccese, A., Alsamsam, M., Loranger, J., Colley, M., & Giudice, N. A. (2024) · Proceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications · doi:10.1145/3640792.3675736 · Cited by 8 · see citing works
“Participants were tasked with monitoring each ride as it arrived using a think-aloud method [24] to gain insight into the user’s task-interaction and satisfaction with the interface.”
MethodO'Grady, M., Kapoor, S., Gilmer, E., Neighbors, C., Conigliaro, J., Kwon, N., & Morgenstern, J. (2019) · ACI Open · doi:10.1055/s-0039-1684002 · Cited by 6 · see citing works
- Authors
- JASPERS, M., STEEN, T., BOS, C., & GEENEN, M.
- Affiliations
- Amsterdam UMC Location University of Amsterdam (NL)
- Corresponding author
- JASPERS, M
- ORCID
- Venue
- International Journal of Medical Informatics
- Publisher
- Elsevier BV
- Published
- 2004-11
- Volume
- 73
- Issue
- 11-12
- Pages
- 781-795
- ISSN
- 1386-5056 (ISSN-L)
- OpenAlex
- W2075016831
- Cited by
- 544
- References
- 49
- Citations by year
- 2026: 35 · 2025: 64 · 2024: 41 · 2023: 58 · 2022: 41 · 2021: 41 · 2020: 34 · 2019: 30 · 2018: 36 · 2017: 17 · 2016: 16 · 2015: 31 · 2014: 15 · 2013: 12 · 2012: 22
- FWCI
- 12.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Medical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Pure Amsterdam UMC (submittedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Keywords
- think-aloud, user interface design, pediatric oncology, electronic health records, clinical information systems, cognitive engineering, requirements analysis, clinical workflow, user workload, user satisfaction
- MeSH terms
- Ambulatory Care Facilities, Child, Equipment Design, Humans, Medical Oncology, Neoplasms, Netherlands, Pediatrics, Practice Patterns, Physicians', User-Computer Interface, Medical Records Systems, Computerized
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
