A Real-Time Automated Patient Screening System for Clinical Trials Eligibility in an Emergency Department: Design and Evaluation
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
BACKGROUND: One critical hurdle for clinical trial recruitment is the lack of an efficient method for identifying subjects who meet the eligibility criteria. Given the large volume of data documented in electronic health records (EHRs), it is labor-intensive for the staff to screen relevant information, particularly within the time frame needed. To facilitate subject identification, we developed a natural language processing (NLP) and machine learning-based system, Automated Clinical Trial Eligibility Screener (ACTES), which analyzes structured data and unstructured narratives automatically to determine patients' suitability for clinical trial enrollment. In this study, we integrated the ACTES into clinical practice to support real-time patient screening. OBJECTIVE: This study aimed to evaluate ACTES's impact on the institutional workflow, prospectively and comprehensively. We hypothesized that compared with the manual screening process, using EHR-based automated screening would improve efficiency of patient identification, streamline patient recruitment workflow, and increase enrollment in clinical trials. METHODS: The ACTES was fully integrated into the clinical research coordinators' (CRC) workflow in the pediatric emergency department (ED) at Cincinnati Children's Hospital Medical Center. The system continuously analyzed EHR information for current ED patients and recommended potential candidates for clinical trials. Relevant patient eligibility information was presented in real time on a dashboard available to CRCs to facilitate their recruitment. To assess the system's effectiveness, we performed a multidimensional, prospective evaluation for a 12-month period, including a time-and-motion study, quantitative assessments of enrollment, and postevaluation usability surveys collected from the CRCs. RESULTS: Compared with manual screening, the use of ACTES reduced the patient screening time by 34% (P<.001). The saved time was redirected to other activities such as study-related administrative tasks (P=.03) and work-related conversations (P=.006) that streamlined teamwork among the CRCs. The quantitative assessments showed that automated screening improved the numbers of subjects screened, approached, and enrolled by 14.7%, 11.1%, and 11.1%, respectively, suggesting the potential of ACTES in streamlining recruitment workflow. Finally, the ACTES achieved a system usability scale of 80.0 in the postevaluation surveys, suggesting that it was a good computerized solution. CONCLUSIONS: By leveraging NLP and machine learning technologies, the ACTES demonstrated good capacity for improving efficiency of patient identification. The quantitative assessments demonstrated the potential of ACTES in streamlining recruitment workflow and improving patient enrollment. The postevaluation surveys suggested that the system was a good computerized solution with satisfactory usability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The System Usability Scale: Past, Present, and Future
Lewis, J. R. (2018) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1455307 · Cited by 1,958 · cited 1×
“The SUS is a widely used and validated survey instrument and it has been applied to assess the usability of patient-oriented computerized programs in prior clinical studies [36-38].”
A 36-hospital time and motion study: how do medical-surgical nurses spend their time?
Hendrich, A., Chow, M. P., Skierczynski, B. A., & Lu, Z. (2008) · The Permanente Journal · doi:10.7812/tpp/08-021 · Cited by 530 · cited 1×
“The methodology has been used to evaluate the efficiency of clinical activities to reduce redundant work and improve workflow [29-31].”
Review of health information technology usability study methodologies
Yen, P.-Y., & Bakken, S. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2010-000020 · Cited by 390
Clinical research informatics: challenges, opportunities and definition for an emerging domain
Embi, P. J., & Payne, P. R. O. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3005 · Cited by 163
Ni, Y., Wright, J., Perentesis, J., Lingren, T., Deleger, L., Kaiser, M., Kohane, I., & Solti, I. (2015) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-015-0149-3 · Cited by 142 · cited 6×
“Systematic error analyses have been performed on retrospective data in our previous research to identify limitations of the ACTES [18,19].”
InfluentialSelf-citation
Cited by
Artificial intelligence in emergency medicine: A scoping review
Kirubarajan, A., Taher, A., Khan, S., & Masood, S. (2020) · JACEP Open · doi:10.1002/emp2.12277 · Cited by 127
Application of Information Technology to Clinical Trial Evaluation and Enrollment: A Review
von Itzstein, M. S., Hullings, M., Mayo, H., Beg, M. S., Williams, E. L., & Gerber, D. E. (2021) · JAMA Oncology · doi:10.1001/jamaoncol.2021.1165 · Cited by 36
Implementation of Machine Learning Pipelines for Clinical Practice: Development and Validation Study
Kanbar, L. J., Wissel, B., Ni, Y., Pajor, N., Glauser, T., Pestian, J., & Dexheimer, J. W. (2022) · JMIR Medical Informatics · doi:10.2196/37833 · Cited by 30
“The use of ACTES significantly improved the number of screened patients for the majority of trials and improved the number of approached patients and enrolled patients, with statistical significance in 2 of 7 trials [52].”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Ni, Y., Bermudez, M., Kennebeck, S., Liddy-Hicks, S., & Dexheimer, J.
- Venue
- JMIR Medical Informatics
- Publisher
- JMIR Publications Inc.
- Published
- 2019-07-24
- Volume
- 7
- Issue
- 3
- Pages
- e14185
- ISSN
- 2291-9694 (ISSN-L)
- OpenAlex
- W2978696208
- Cited by
- 61
- References
- 5
- FWCI
- 7.43
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.