“Think aloud” and “Near live” usability testing of two complex clinical decision support tools
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Objectives Low provider adoption continues to be a significant barrier to realizing the potential of clinical decision support. “Think Aloud” and “Near Live” usability testing were conducted on two clinical decision support tools. Each was composed of an alert, a clinical prediction rule which estimated risk of either group A Streptococcus pharyngitis or pneumonia and an automatic order set based on risk. The objective of this study was to further understanding of the facilitators of usability and to evaluate the types of additional information gained from proceeding to “Near Live” testing after completing “Think Aloud”. Methods This was a qualitative observational study conducted at a large academic health care system with 12 primary care providers. During “Think Aloud” testing, participants were provided with written clinical scenarios and asked to verbalize their thought process while interacting with the tool. During “Near Live” testing participants interacted with a mock patient. Morae usability software was used to record full screen capture and audio during every session. Participant comments were placed into coding categories and analyzed for generalizable themes. Themes were compared across usability methods. Results “Think Aloud” and “Near Live” usability testing generated similar themes under the coding categories visibility, workflow, content, understand-ability and navigation. However, they generated significantly different themes under the coding categories usability, practical usefulness and medical usefulness. During both types of testing participants found the tool easier to use when important text was distinct in its appearance, alerts were passive and appropriately timed, content was up to date, language was clear and simple, and each component of the tool included obvious indicators of next steps. Participant comments reflected higher expectations for usability and usefulness during “Near Live” testing. For example, visit aids, such as automatically generated order sets, were felt to be less useful during “Near-Live” testing because they would not be all inclusive for the visit. Conclusions These complementary types of usability testing generated unique and generalizable insights. Feedback during “Think Aloud” testing primarily helped to improve the tools’ ease of use. The additional feedback from “Near Live” testing, which mimics a real clinical encounter, was helpful for eliciting key barriers and facilitators to provider workflow and adoption.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Technology Acceptance Model 3 and a Research Agenda on Interventions
Venkatesh, V., & Bala, H. (2008) · Decision Sciences · doi:10.1111/j.1540-5915.2008.00192.x · Cited by 8,159 · cited 1×
“Comments made during “Think Aloud” testing about visibility, content, understand-ability and navigation primarily helped to improve the tools ease of use, which is a key driver of technology adoption[21].”
BackgroundAn Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 1×
“[18] Those raw scores would correspond to the 95 th percentile rank for “Think Aloud” testing and the 91st for “Near Live” testing.”
MethodThe quality of health care delivered to adults in the United States
McGlynn, E. A., Asch, S. M., Adams, J., Keesey, J., Hicks, J., DeCristofaro, A., & Kerr, E. A. (2003) · New England Journal of Medicine · doi:10.1056/nejmsa022615 · Cited by 5,079
Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494 · cited 1×
“Primary care doctors receive about 77 total notifications per day[27] and spend nearly two hours on the EHR and desk work for every hour of face to face time with their patients[28].”
BackgroundThe Factor Structure of the System Usability Scale
Lewis, J. R., & Sauro, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02806-9_12 · Cited by 1,278
Effect of Clinical Decision-Support Systems: A Systematic Review
Bright, T. J., Wong, A., Dhurjati, R., Bristow, E., Bastian, L., Coeytaux, R. R., Samsa, G., Hasselblad, V., Williams, J. W., Musty, M. D., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-1-201207030-00450 · Cited by 1,200
Types of Unintended Consequences Related to Computerized Provider Order Entry
Campbell, E. M., Sittig, D. F., Ash, J. S., Guappone, K. P., & Dykstra, R. H. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2042 · Cited by 823
Estimating the number of subjects needed for a thinking aloud test
Nielsen, J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1065 · Cited by 476
On the dimensionality of the System Usability Scale: a test of alternative measurement models
Borsci, S., Federici, S., & Lauriola, M. (2009) · Cognitive Processing · doi:10.1007/s10339-009-0268-9 · Cited by 275 · cited 1×
“[15] The SUS is a widely used, validated instrument that measures subjective usability.”
MethodA Consensus Statement on Considerations for a Successful CPOE Implementation
Ash, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1204 · Cited by 239
Computerized Physician Order Entry: Helpful or Harmful?
Berger, R. G. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1411 · Cited by 223
Kaushal, R., Shojania, K. G., & Bates, D. W. (2003) · Archives of Internal Medicine · doi:10.1001/archinte.163.12.1409 · Cited by 30
Cited by
Usability of Mobile Applications: A Systematic Literature Study
Weichbroth, P. (2020) · IEEE Access · doi:10.1109/access.2020.2981892 · Cited by 276
Rundo, L., Pirrone, R., Vitabile, S., Sala, E., & Gambino, O. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103479 · Cited by 113
Fletcher, E., Burns, A., Wiering, B., Lavu, D., Shephard, E., Hamilton, W., Campbell, J. L., & Abel, G. (2023) · BMC Primary Care · doi:10.1186/s12875-023-01973-2 · Cited by 40
Helenason, J., Ekström, C., Falk, M., & Papachristou, P. (2024) · Scandinavian Journal of Primary Health Care · doi:10.1080/02813432.2023.2283190 · Cited by 33
“…however, the method allows for thorough and systematic observation of the actual thoughts, considerations and reflections made while using the cDss, both by visual observation and through the think aloud recordings, unrestrained by stressful time limits in the clinic or diagnostic concerns [29,30].”
MethodLuetke Lanfer, H., Reifegerste, D., Weber, W., Memenga, P., Baumann, E., Geulen, J., Klein, S., Müller, A., Hahne, A., & Weg-Remers, S. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-10785-8 · Cited by 30
Metadata
- Authors
- Richardson, S., Mishuris, R., O’Connell, A., Feldstein, D., Hess, R., Smith, P., McCullagh, L., McGinn, T., & Mann, D.
- Venue
- International Journal of Medical Informatics
- Publisher
- Elsevier BV
- Published
- 2017-10
- Volume
- 106
- Pages
- 1-8
- ISSN
- 1386-5056 (ISSN-L)
- OpenAlex
- W2653731937
- Cited by
- 114
- References
- 13
- FWCI
- 9.96
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.