EliIE: An open-source information extraction system for clinical trial eligibility criteria
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Objective: To develop an open-source information extraction system called Eligibility Criteria Information Extraction (EliIE) for parsing and formalizing free-text clinical research eligibility criteria (EC) following Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) version 5.0. Materials and Methods: EliIE parses EC in 4 steps: (1) clinical entity and attribute recognition, (2) negation detection, (3) relation extraction, and (4) concept normalization and output structuring. Informaticians and domain experts were recruited to design an annotation guideline and generate a training corpus of annotated EC for 230 Alzheimer’s clinical trials, which were represented as queries against the OMOP CDM and included 8008 entities, 3550 attributes, and 3529 relations. A sequence labeling–based method was developed for automatic entity and attribute recognition. Negation detection was supported by NegEx and a set of predefined rules. Relation extraction was achieved by a support vector machine classifier. We further performed terminology-based concept normalization and output structuring. Results: In task-specific evaluations, the best F1 score for entity recognition was 0.79, and for relation extraction was 0.89. The accuracy of negation detection was 0.94. The overall accuracy for query formalization was 0.71 in an end-to-end evaluation. Conclusions: This study presents EliIE, an OMOP CDM–based information extraction system for automatic structuring and formalization of free-text EC. According to our evaluation, machine learning-based EliIE outperforms existing systems and shows promise to improve.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
2016 Alzheimer's disease facts and figures
Alzheimer's Association (2016) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2016.03.001 · Cited by 5,069
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,542
McDonald, A. M., Knight, R. C., Campbell, M. K., Entwistle, V. A., Grant, A. M., Cook, J. A., Elbourne, D. R., Francis, D., Garcia, J., Roberts, I., et al. (2006) · Trials · doi:10.1186/1745-6215-7-9 · Cited by 946
NLTK: The Natural Language Toolkit
Bird, S. (2006) · Proceedings of the COLING/ACL on Interactive presentation sessions - · doi:10.3115/1225403.1225421 · Cited by 841 · cited 1×
“We then added other surface syntactic features to the CRF baseline system, such as part-of-speech tags and lemmas using the Natural Language Toolkit (NLTK) package.(57) To include domain knowledge to recognize entities, a UMLS-based feature was generated for the system.”
MethodValidation of a common data model for active safety surveillance research
Overhage, J. M., Ryan, P. B., Reich, C. G., Hartzema, A. G., & Stang, P. E. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000376 · Cited by 548
Characterizing treatment pathways at scale using the OHDSI network
Hripcsak, G., Ryan, P. B., Duke, J. D., Shah, N. H., Park, R. W., Huser, V., Suchard, M. A., Schuemie, M. J., DeFalco, F. J., Perotte, A., et al. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510502113 · Cited by 326
Bidirectional RNN for Medical Event Detection in Electronic Health Records
Jagannatha, A. N., & Yu, H. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1056 · Cited by 314 · cited 1×
“Word embeddings and deep learning techniques have shown great promise for the NER task for clinical text.(47,48) Unfortunately, the above systems were designed for clinical notes in EHRs or text in the literature.”
Electronic Health Record Adoption In US Hospitals: Progress Continues, But Challenges Persist
Adler-Milstein, J., DesRoches, C. M., Kralovec, P., Foster, G., Worzala, C., Charles, D., Searcy, T., & Jha, A. K. (2015) · Health Affairs · doi:10.1377/hlthaff.2015.0992 · Cited by 310
Cited by
Evaluating eligibility criteria of oncology trials using real-world data and AI
Liu, R., Rizzo, S., Whipple, S., Pal, N., Pineda, A. L., Lu, M., Arnieri, B., Lu, Y., Capra, W., Copping, R., et al. (2021) · Nature · doi:10.1038/s41586-021-03430-5 · Cited by 322
Artificial intelligence applied to clinical trials: opportunities and challenges
Askin, S., Burkhalter, D., Calado, G., & El Dakrouni, S. (2023) · Health and Technology · doi:10.1007/s12553-023-00738-2 · Cited by 285
Criteria2Query: a natural language interface to clinical databases for cohort definition
Yuan, C., Ryan, P. B., Ta, C., Guo, Y., Li, Z., Hardin, J., Makadia, R., Jin, P., Shang, N., Kang, T., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy178 · Cited by 170
Self-citationUnlu, O., Shin, J., Mailly, C. J., Oates, M. F., Tucci, M. R., Varugheese, M., Wagholikar, K., Wang, F., Scirica, B. M., Blood, A. J., et al. (2024) · doi:10.1101/2024.02.08.24302376 · Cited by 117
Campillos-Llanos, L., Valverde-Mateos, A., Capllonch-Carrión, A., & Moreno-Sandoval, A. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01395-z · Cited by 65
Metadata
- Authors
- Kang, T., Zhang, S., Tang, Y., Hruby, G. W., Rusanov, A., Elhadad, N., & Weng, C.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-04-01
- Volume
- 24
- Issue
- 6
- Pages
- 1062-1071
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2604410201
- Cited by
- 110
- References
- 8
- FWCI
- 3.11
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: EliIE: An open-source information extraction system for clinical trial eligibility criteria. https://app.cochat.ai/papers/10.1093/jamia/ocx019
- The paper
- Kang, T., Zhang, S., Tang, Y., Hruby, G. W., Rusanov, A., Elhadad, N., & Weng, C. (2017). EliIE: An open-source information extraction system for clinical trial eligibility criteria. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocx019
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.