Evaluation of a flowchart-based EHR query system: A case study of RetroGuide
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Provision of query systems which are intuitive for non-experts has been recognized as an important informatics challenge. We developed a prototype of a flowchart-based analytical framework called RetroGuide that enables non-experts to formulate query tasks using a step-based, patient-centered paradigm inspired by workflow technology. We present results of the evaluation of RetroGuide in comparison to Structured Query Language (SQL) in laboratory settings using a mixed method design. We asked 18 human subjects with limited database experience to solve query tasks in RetroGuide and SQL, and quantitatively compared their test scores. A follow-up questionnaire was designed to compare both technologies qualitatively and investigate RetroGuide technology acceptance. The quantitative comparison of test scores showed that the study subjects achieved significantly higher scores using the RetroGuide technology. Qualitative study results indicated that 94% of subjects preferred RetroGuide to SQL because RetroGuide was easier to learn, it better supported temporal tasks, and it seemed to be a more logical modeling paradigm. Additional qualitative evaluation results, based on a technology acceptance model, suggested that a fully developed RetroGuide-like technology would be well accepted by users. Our study is an example of a structure validation study of a prototype query system, results of which provided significant guidance in further development of a novel query paradigm for EHR data. We discuss the strengths and weakness of our study design and results, and their implication for future evaluations of query systems in general.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483
Examining the technology acceptance model using physician acceptance of telemedicine technology
Hu, P. J., Chau, P. Y. K., Sheng, O. R. L., & Tam, K. Y. (1999) · Journal of Management Information Systems · doi:10.1080/07421222.1999.11518247 · Cited by 1,951
Natural language interfaces to databases – an introduction
Androutsopoulos, I., Ritchie, G. D., & Thanisch, P. (1995) · Natural Language Engineering · doi:10.1017/s135132490000005x · Cited by 768 · cited 1×
“We have also reviewed evaluation strategies for a related field of natural language interfaces to databases [34-36].”
BackgroundModeling end-users’ acceptance of a knowledge authoring tool
Del Fiol, G., Rocha, R. A., & Hulse, N. C. (2006) · Methods of Information in Medicine · doi:10.1055/s-0038-1634114 · Cited by 11
Self-citation
Cited by
The Technology Acceptance Model: Its past and its future in health care
Holden, R. J., & Karsh, B.-T. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.07.002 · Cited by 2,669
The unified theory of acceptance and use of technology (UTAUT): a literature review
Williams, M. D., Rana, N. P., & Dwivedi, Y. K. (2015) · Journal of Enterprise Information Management · doi:10.1108/jeim-09-2014-0088 · Cited by 1,339
A Systematic Review of the Technology Acceptance Model in Health Informatics
Rahimi, B., Nadri, H., Lotfnezhad Afshar, H., & Timpka, T. (2018) · Applied Clinical Informatics · doi:10.1055/s-0038-1668091 · Cited by 688
A Meta-analysis of the Unified Theory of Acceptance and Use of Technology (UTAUT)
Dwivedi, Y. K., Rana, N. P., Chen, H., & Williams, M. D. (2011) · IFIP Advances in Information and Communication Technology · doi:10.1007/978-3-642-24148-2_10 · Cited by 312
Metadata
- Authors
- Huser, V., Narus, S. P., & Rocha, R. A.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2010-02
- Volume
- 43
- Issue
- 1
- Pages
- 41-50
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2005787113
- Cited by
- 37
- References
- 4
- FWCI
- 7.99
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.