CoChat

Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

Open access
Walji, Muhammad F., Kalenderian, Elsbeth, Piotrowski, Mark, Tran, Duong, Kookal, Krishna K., Tokede, Oluwabunmi, White, Joel M., Vaderhobli, Ram, Ramoni, Rachel, Stark, Paul C., Kimmes, Nicole S., Lagerweij, Maxim, Patel, Vimla L. · 2014 · International Journal of Medical Informatics
Cited by 69

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Abstract

Objective To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. Materials and methods Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35). Results The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. Discussion Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. Conclusion We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ijmedinf.2014.01.010

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Cited by

See all 69

Metadata

Authors
Walji, M. F., Kalenderian, E., Piotrowski, M., Tran, D., Kookal, K. K., Tokede, O., White, J. M., Vaderhobli, R., Ramoni, R., Stark, P. C., Kimmes, N. S., Lagerweij, M., & Patel, V. L.
Venue
International Journal of Medical Informatics
Publisher
Elsevier BV
Published
2014-05
Volume
83
Issue
5
Pages
361-367
ISSN
1386-5056 (ISSN-L)
OpenAlex
W1974928492
Cited by
69
References
11
FWCI
12.43
Type
journal article
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. https://app.cochat.ai/papers/10.1016/j.ijmedinf.2014.01.010
The paper
Walji, M. F., Kalenderian, E., Piotrowski, M., Tran, D., Kookal, K. K., Tokede, O., White, J. M., Vaderhobli, R., Ramoni, R., Stark, P. C., Kimmes, N. S., Lagerweij, M., & Patel, V. L. (2014). Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2014.01.010

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.