Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

ACR: A Benchmark for Automatic Cohort Retrieval

Thai, Dung Ngoc, Ardulov, Victor, Mena, Jose Ulises, Tiwari, Simran, Erofeev, Gleb, Eskander, Ramy, Tarabishy, Karim, Parikh, Ravi B, Salloum, Wael·2024·arXiv
DOI 10.48550/arxiv.2406.14780arXiv 2406.14780
Open this paper in CoChat — free account

Abstract

Identifying patient cohorts is fundamental to numerous healthcare tasks, including clinical trial recruitment and retrospective studies. Current cohort retrieval methods in healthcare organizations rely on automated queries of structured data combined with manual curation, which are time-consuming, labor-intensive, and often yield low-quality results. Recent advancements in large language models (LLMs) and information retrieval (IR) offer promising avenues to revolutionize these systems. Major challenges include managing extensive eligibility criteria and handling the longitudinal nature of unstructured Electronic Medical Records (EMRs) while ensuring that the solution remains cost-effective for real-world application. This paper introduces a new task, Automatic Cohort Retrieval (ACR), and evaluates the performance of LLMs and commercial, domain-specific neuro-symbolic approaches. We provide a benchmark task, a query dataset, an EMR dataset, and an evaluation framework. Our findings underscore the necessity for efficient, high-quality ACR systems capable of longitudinal reasoning across extensive patient databases.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.14780

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Longitudinal data challenges in Cohort Retrieval: This example shows a patient medical journey depicting 3 related facts scattered in 3 documents written over the years. Cohort retrieval systems must possess longitudinal reasoning capacities to accurately answer the user query shown above.
Figure 1 Longitudinal data challenges in Cohort Retrieval: This example shows a patient medical journey depicting 3 related facts scattered in 3 documents written over the years. Cohort retrieval systems must possess longitudinal reasoning capacities to accurately answer the user query shown above.CC-BY-SA · From the PDF
Figure 2 Architecture of a generic ACR system: Given a query, large-scale reasoning is conducted over numerous patients with longitudinal EMRs. This involves text-based reasoning on document or chunks, followed by longitudinal reasoning over time.
Figure 2 Architecture of a generic ACR system: Given a query, large-scale reasoning is conducted over numerous patients with longitudinal EMRs. This involves text-based reasoning on document or chunks, followed by longitudinal reasoning over time.CC-BY-SA · From the PDF
Figure 3 F1-Score stratified by complexity determined by (a) experts, (b) cohort size, and (c) document count per patient ( N d ).
Figure 3 F1-Score stratified by complexity determined by (a) experts, (b) cohort size, and (c) document count per patient ( N d ).CC-BY-SA · From the PDF
Figure 4 A plot of hallucination ratio against gold cohort sizes for all queries of the Broad , Narrow , and Sparse types. It helps identify queries where models over-hallucinate, retrieving too many unqualified patients relative to actual qualified ones.
Figure 4 A plot of hallucination ratio against gold cohort sizes for all queries of the Broad , Narrow , and Sparse types. It helps identify queries where models over-hallucinate, retrieving too many unqualified patients relative to actual qualified ones.CC-BY-SA · From the PDF
Figure 5 Hallucination tendencies on every Zero-Result query ( x-axis ) measured as False Positives counts ( y-axis ).
Figure 5 Hallucination tendencies on every Zero-Result query ( x-axis ) measured as False Positives counts ( y-axis ).CC-BY-SA · From the PDF
Figure 6 Distribution of patients by the quantity of documents. Documents are not distributed equally, and there is large variance over the amount of information available, that needs to be processed, per patient.
Figure 6 Distribution of patients by the quantity of documents. Documents are not distributed equally, and there is large variance over the amount of information available, that needs to be processed, per patient.CC-BY-SA · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 61

  1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  2. M. A. Al-Garadi, Y.-C. Yang, S. Lakamana, J. Lin, S. Li, A. Xie, W. Hogg-Bremer, M. Torres, I. Banerjee, and A. Sarker. Automatic Breast Cancer Cohort Detection from Social Media for Studying Factors Affecting Patient Centered Outcomes, May 2020. URL https://www.medrxiv.org/content/10.1101/2020.05.17.20104778v1. Pages: 2020.05.17.20104778. · DOI
  3. K. Bairaktari, P. Langton, H. L. Nguyen, N. Smedemark-Margulies, and J. Ullman. Fair and Useful Cohort Selection, Apr. 2022. URL http://arxiv.org/abs/2009.02207. arXiv:2009.02207 [cs. · arXiv
  4. B. Birnbaum, N. Nussbaum, K. Seidl-Rathkopf, M. Agrawal, M. Estevez, E. Estola, J. Haimson, L. He, P. Larson, and P. Richardson. Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research, Jan. 2020. URL http://arxiv.org/abs/2001.09765. arXiv:2001.09765 [cs. · arXiv
  5. R. Carlson, S. Tu, N. Lane, T. Lai, C. Kemper, M. Musen, and E. Shortliffe. Computer-based screening of patients with HIV/AIDS for clinical-trial eligibility. The Online journal of current clinical trials, Doc No 179:[3347 words; 32 paragraphs, Apr. 1995.
  6. S. Chakrabarti, A. Sen, V. Huser, G. W. Hruby, A. Rusanov, D. J. Albers, and C. Weng. An Interoperable Similarity- based Cohort Identification Method Using the OMOP Common Data Model version 5.0. Journal of Healthcare Informatics Research, 1(1):1-18, June 2017. ISSN 2509-4971. doi: 10.1007/s41666-017-0005-6. · DOI
  7. S. R. Chamberlin, S. D. Bedrick, A. M. Cohen, Y. Wang, A. Wen, S. Liu, H. Liu, and W. R. Hersh. Evaluation of patient- level retrieval from electronic health record data for a cohort discovery task. JAMIA Open, 3(3):395-404, Oct. 2020. ISSN 2574-2531. doi: 10.1093/jamiaopen/ooaa026. URL https://doi.org/10.1093/jamiaopen/ooaa026. · DOI
  8. C.-Y. Chang, J. Yuan, S. Ding, Q. Tan, K. Zhang, X. Jiang, X. Hu, and N. Zou. Towards Fair Patient-Trial Matching via Patient-Criterion Level Fairness Constraint. AMIA Annual Symposium Proceedings, 2023:884-893, Jan. 2024. ISSN 1942-597X. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10785912/.
  9. E. Chang and J. Mostafa. Cohort Identification from Free-Text Clinical Notes Using SNOMED CT's Hierarchical Semantic Relations. AMIA ... Annual Symposium proceedings. AMIA Symposium, 2022:349-358, 2022. ISSN 1942-597X.
  10. H.-J. Dai, F.-D. Wang, C.-W. Chen, C.-H. Su, C.-S. Wu, and J. Jonnagaddala. Cohort selection for clinical trials using multiple instance learning. Journal of Biomedical Informatics, 107:103438, July 2020. ISSN 1532-0464. doi: 10.1016/ j.jbi.2020.103438. URL https://www.sciencedirect.com/science/article/pii/S1532046420300666. · DOI
  11. P. Deka, A. Jurek-Loughrey, et al. Evidence extraction to validate medical claims in fake news detection. In International Conference on Health Information Science, pages 3-15. Springer, 2022.
  12. M. V. Devarakonda, S. Mohanty, R. R. Sunkishala, N. Mallampalli, and X. Liu. Clinical Trial Recommendations Using Semantics-Based Inductive Inference and Knowledge Graph Embeddings, Sept. 2023. URL http://arxiv.org/ abs/2309.15979. arXiv:2309.15979 [cs, q-bio]. · arXiv
  13. N. J. Dobbins, T. Mullen, Ö. Uzuner, and M. Yetisgen. The leaf clinical trials corpus: a new resource for query generation from clinical trial eligibility criteria. Scientific Data, 9(1):490, 2022.
  14. Y. Fang, B. Idnay, Y. Sun, H. Liu, Z. Chen, K. Marder, H. Xu, R. Schnall, and C. Weng. Combining human and machine intelligence for clinical trial eligibility querying. Journal of the American Medical Informatics Association: JAMIA, 29(7):1161-1171, June 2022. ISSN 1527-974X. doi: 10.1093/jamia/ocac051. · DOI
  15. E. Farrand, O. Gologorskaya, H. Mills, L. Radhakrishnan, H. R. Collard, and A. J. Butte. Machine-Learning Algorithm to Improve Cohort Identification in Interstitial Lung Disease. American Journal of Respiratory and Critical Care Medicine, 207(10):1398-1401, May 2023. ISSN 1535-4970. doi: 10.1164/rccm.202211-2092LE. · DOI
  16. B. Fletcher, A. Gheorghe, D. Moore, S. Wilson, and S. Damery. Improving the recruitment activity of clinicians in randomised controlled trials: a systematic review. BMJ open, 2(1):e000496, 2012. ISSN 2044-6055. doi: 10.1136/bmjopen-2011-000496. · DOI
  17. Y. Gao, D. Dligach, L. Christensen, S. Tesch, R. Laffin, D. Xu, T. Miller, O. Uzuner, M. M. Churpek, and M. Afshar. A scoping review of publicly available language tasks in clinical natural language processing. Journal of the American Medical Informatics Association: JAMIA, 29(10):1797-1806, Sept. 2022. ISSN 1527-974X. doi: 10.1093/jamia/ocac127. · DOI · Paper page
  18. B. S. Glicksberg, R. Miotto, K. W. Johnson, K. Shameer, L. Li, R. Chen, and J. T. Dudley. Automated disease cohort selection using word embeddings from Electronic Health Records. In Biocomputing 2018, pages 145- 156. WORLD SCIENTIFIC, Oct. 2017. ISBN 978-981-323-552-6. doi: 10.1142/9789813235533_0014. URL https://www.worldscientific.com/doi/abs/10.1142/9789813235533_0014. · DOI
  19. M. Henzinger, C. Peale, O. Reingold, and J. H. Shen. Leximax Approximations and Representative Cohort Selection, May 2022. URL http://arxiv.org/abs/2205.01157. arXiv:2205.01157 [cs. · arXiv
  20. N. Hiob and S. Lessmann. Health Analytics: a systematic review of approaches to detect phenotype cohorts using electronic health records, July 2017. URL http://arxiv.org/abs/1707.07425. arXiv:1707.07425 [stat. · arXiv
  21. B. Idnay, C. Dreisbach, C. Weng, and R. Schnall. A systematic review on natural language processing systems for eligibility prescreening in clinical research. Journal of the American Medical Informatics Association: JAMIA, 29(1): 197-206, Dec. 2021. ISSN 1527-974X. doi: 10.1093/jamia/ocab228. · DOI
  22. B. Idnay, Y. Fang, C. Dreisbach, K. Marder, C. Weng, and R. Schnall. Clinical research staff perceptions on a natural language processing-driven tool for eligibility prescreening: An iterative usability assessment. International Journal of Medical Informatics, 171:104985, Mar. 2023. ISSN 1872-8243. doi: 10.1016/j.ijmedinf.2023.104985. · DOI
  23. Q. Jin, Z. Wang, C. S. Floudas, J. Sun, and Z. Lu. Matching Patients to Clinical Trials with Large Language Models. ArXiv, page arXiv:2307.15051v2, July 2023. ISSN 2331-8422. URL https://www.ncbi.nlm.nih.gov/pmc/ articles/PMC10418514/. · arXiv
  24. T. Kang, N. Elhadad, and C. Weng. Initial Readability Assessment of Clinical Trial Eligibility Criteria. AMIA Annual Symposium Proceedings, 2015:687-696, Nov. 2015. ISSN 1942-597X. URL https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC4765646/.
  25. T. Kang, S. Zhang, Y. Tang, G. W. Hruby, A. Rusanov, N. Elhadad, and C. Weng. EliIE: An open-source information extraction system for clinical trial eligibility criteria. Journal of the American Medical Informatics Association, 24 (6):1062-1071, Nov. 2017. ISSN 1067-5027, 1527-974X. doi: 10.1093/jamia/ocx019. URL https://academic. oup.com/jamia/article/24/6/1062/3098256. · DOI
  26. G. Karystianis, O. Florez-Vargas, T. Butler, and G. Nenadic. A rule-based approach to identify patient eligibility criteria for clinical trials from narrative longitudinal records. JAMIA open, 2(4):521-527, Dec. 2019. ISSN 2574-2531. doi: 10.1093/jamiaopen/ooz041. · DOI
  27. R. Kundra, H. Zhang, R. Sheridan, S. J. Sirintrapun, A. Wang, A. Ochoa, M. Wilson, B. Gross, Y. Sun, R. Madupuri, B. A. Satravada, D. Reales, E. Vakiani, H. A. Al-Ahmadie, A. Dogan, M. Arcila, A. Zehir, S. Maron, M. F. Berger, C. Viaplana, K. Janeway, M. Ducar, L. Sholl, S. Dogan, P. Bedard, L. F. Surrey, I. H. Sanchez, A. Syed, A. B. Rema, D. Chakravarty, S. Suehnholz, M. Nissan, G. V. Iyer, R. Murali, N. Bouvier, R. A. Soslow, D. Hyman, A. Younes, A. Intlekofer, J. J. Harding, R. D. Carvajal, P. J. Sabbatini, G. K. Abou-Alfa, L. Morris, Y. Y. Janjigian, M. M. Gallagher, T. A. Soumerai, I. K. Mellinghoff, A. A. Hakimi, M. Fury, J. T. Huse, A. Bagrodia, M. Hameed, S. Thomas, S. Gardos, E. Cerami, T. Mazor, P. Kumari, P. Raman, P. Shivdasani, S. MacFarland, S. Newman, A. Waanders, J. Gao, D. Solit, and N. Schultz. Oncotree: A cancer classification system for precision oncology. JCO Clinical Cancer Informatics, (5):221-230, 2021. doi: 10.1200/CCI.20.00108. URL https://doi.org/10.1200/CCI.20.00108. PMID: 33625877. · DOI
  28. W. Kusa, Ó. E. Mendoza, P. Knoth, G. Pasi, and A. Hanbury. Effective matching of patients to clinical trials using entity extraction and neural re-ranking. Journal of biomedical informatics, 144:104444, 2023.
  29. P. Lee, S. Bubeck, and J. Petro. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine, 388(13):1233-1239, 2023a.
  30. S. Liu, Y. Wang, A. Wen, L. Wang, N. Hong, F. Shen, S. Bedrick, W. Hersh, and H. Liu. CREATE: Cohort Retrieval Enhanced by Analysis of Text from Electronic Health Records using OMOP Common Data Model. JMIR Medical Informatics, 8(10):e17376, Oct. 2020a. ISSN 2291-9694. doi: 10.2196/17376. URL http: //arxiv.org/abs/1901.07601. arXiv:1901.07601 [cs. · DOI
  31. S. Liu, Y. Wang, A. Wen, L. Wang, N. Hong, F. Shen, S. Bedrick, W. Hersh, and H. Liu. Implementation of a cohort retrieval system for clinical data repositories using the observational medical outcomes partnership common data model: Proof-of-concept system validation. JMIR Medical Informatics, 8(10), Oct. 2020b. ISSN 2291-9694. doi: 10.2196/17376. URL http://www.scopus.com/inward/record.url?scp=85097452931&partnerID= 8YFLogxK. · DOI
  32. X. Liu, L. A. Finelli, G. L. Hersch, and I. Khalil. Attention-Based LSTM Network for COVID-19 Clinical Trial Parsing, Dec. 2020c. URL http://arxiv.org/abs/2012.10063. arXiv:2012.10063 [cs. · arXiv
  33. C. D. Manning. An introduction to information retrieval. Cambridge university press, 2009.
  34. R. Miotto and C. Weng. Case-based reasoning using electronic health records efficiently identifies eligible patients for clinical trials. Journal of the American Medical Informatics Association, 22(e1):e141-e150, 2015.
  35. Y. Ni, M. Bermudez, S. Kennebeck, S. Liddy-Hicks, J. Dexheimer, et al. A real-time automated patient screening system for clinical trials eligibility in an emergency department: design and evaluation. JMIR medical informatics, 7 (3):e14185, 2019.
  36. M. Nievas, A. Basu, Y. Wang, and H. Singh. Distilling Large Language Models for Matching Patients to Clinical Trials, Dec. 2023. URL http://arxiv.org/abs/2312.09958. arXiv:2312.09958 [cs. · arXiv
  37. B. E. Nye, J. DeYoung, E. Lehman, A. Nenkova, I. J. Marshall, and B. C. Wallace. Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations. AMIA Summits on Translational Science Proceedings, 2021: 485-494, May 2021. ISSN 2153-4063. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8378650/.
  38. B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models. ArXiv, abs/2309.00071, 2023. URL https://api.semanticscholar.org/CorpusID:261493986. · arXiv
  39. A. Sarker, M. A. Al-Garadi, Y.-C. Yang, J. Choi, A. A. Quyyumi, and G. S. Martin. Defining Patient-Oriented Natural Language Processing: A New Paradigm for Research and Development to Facilitate Adoption and Use by Medical Experts. JMIR medical informatics, 9(9):e18471, Sept. 2021. ISSN 2291-9694. doi: 10.2196/18471. · DOI
  40. L. Schwartzberg, E. S. Kim, D. Liu, and D. Schrag. Precision Oncology: Who, How, What, When, and When Not? American Society of Clinical Oncology Educational Book, Oct. 2018. doi: 10.1200/EDBK_174176. URL https: //ascopubs.org/doi/10.1200/EDBK_174176. Publisher: American Society of Clinical OncologyAlexandria, VA. · DOI
  41. Segura-Bedmar and P. Raez. Cohort selection for clinical trials using deep learning models. Journal of the American Medical Informatics Association, 26(11):1181-1188, Nov. 2019. ISSN 1527-974X. doi: 10.1093/jamia/ocz139. URL https://doi.org/10.1093/jamia/ocz139. · DOI
  42. S. Shekhar, S. Tiwari, T. Rensink, R. Eskander, and W. Salloum. Coupling symbolic reasoning with language modeling for efficient longitudinal understanding of unstructured electronic medical records. arXiv preprint arXiv:2308.03360, 2023. · arXiv
  43. C. Shivade, P. Raghavan, E. Fosler-Lussier, P. J. Embi, N. Elhadad, S. B. Johnson, and A. M. Lai. A review of approaches to identifying patient phenotype cohorts using electronic health records. Journal of the American Medical Informatics Association, 21(2):221-230, Mar. 2014. ISSN 1067-5027. doi: 10.1136/amiajnl-2013-001935. URL https://doi.org/10.1136/amiajnl-2013-001935. · DOI
  44. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022. · arXiv
  45. K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu. Mpnet: Masked and permuted pre-training for language understanding. arXiv preprint arXiv:2004.09297, 2020. · arXiv
  46. S. Soni and K. Roberts. Patient Cohort Retrieval using Transformer Language Models, Sept. 2020. URL http: //arxiv.org/abs/2009.05121. arXiv:2009.05121 [cs. · arXiv
  47. A. Stubbs and Ö. Uzuner. New approaches to cohort selection. Journal of the American Medical Informatics Association, 26(11):1161-1162, 2019.
  48. A. Stubbs, M. Filannino, E. Soysal, S. Henry, and Ö. Uzuner. Cohort selection for clinical trials: n2c2 2018 shared task track 1. Journal of the American Medical Informatics Association, 26(11):1163-1171, 2019.
  49. J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021. URL https://arxiv.org/abs/2104.09864. · arXiv
  50. B. Theodorou, C. Xiao, and J. Sun. TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network, July 2023. URL http://arxiv.org/abs/2307.09942. arXiv:2307.09942 [cs. · arXiv
  51. S. W. Tu, C. A. Kemper, N. M. Lane, R. W. Carlson, and M. A. Musen. A Methodology for Determining Patients' Eli- gibility for Clinical Trials. Methods of Information in Medicine, 32(4):317-325, 1993. ISSN 0026-1270, 2511-705X. doi: 10.1055/s-0038-1634933. URL http://www.thieme-connect.de/DOI/DOI?10.1055/s-0038-1634933. Publisher: Schattauer GmbH. · DOI
  52. M. Vassar and H. Matthew. The retrospective chart review: important methodological considerations. Journal of educational evaluation for health professions, 10, 2013.
  53. V. G. V. Vydiswaran, A. Strayhorn, X. Zhao, P. Robinson, M. Agarwal, E. Bagazinski, M. Essiet, B. E. Iott, H. Joo, P. Ko, D. Lee, J. X. Lu, J. Liu, A. Murali, K. Sasagawa, T. Wang, and N. Yuan. Hybrid bag of approaches to characterize selection criteria for cohort identification. Journal of the American Medical Informatics Association: JAMIA, 26(11):1172-1180, Nov. 2019. ISSN 1527-974X. doi: 10.1093/jamia/ocz079. · DOI
  54. C. Weng, X. Wu, Z. Luo, M. R. Boland, D. Theodoratos, and S. B. Johnson. EliXR: an approach to eligibility criteria extraction and representation. Journal of the American Medical Informatics Association, 18(Supplement 1):i116-i124, Dec. 2011. ISSN 1067-5027, 1527-974X. doi: 10.1136/amiajnl-2011-000321. URL https://academic.oup. com/jamia/article-lookup/doi/10.1136/amiajnl-2011-000321. · DOI
  55. A. Wilcox, D. Vawdrey, C. Weng, M. Velez, and S. Bakken. Research Data Explorer: Lessons Learned in Design and Development of Context-based Cohort Definition and Selection. AMIA Summits on Translational Science Pro- ceedings, 2015:194-198, Mar. 2015. ISSN 2153-4063. URL https://www.ncbi.nlm.nih.gov/pmc/articles/ PMC4525259/.
  56. C. Wong, S. Zhang, Y. Gu, C. Moung, J. Abel, N. Usuyama, R. Weerasinghe, B. Piening, T. Naumann, C. Bifulco, and H. Poon. Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology, Aug. 2023. URL http://arxiv.org/abs/2308.02180. arXiv:2308.02180 [cs. · arXiv
  57. R. Yang, D. Zhu, L. E. Howard, A. De Hoedt, S. B. Williams, S. J. Freedland, and Z. Klaassen. Identification of Patients With Metastatic Prostate Cancer With Natural Language Processing and Machine Learning. JCO clinical cancer informatics, 6:e2100071, Oct. 2022. ISSN 2473-4276. doi: 10.1200/CCI.21.00071. · DOI
  58. S. Yang, P. Varghese, E. Stephenson, K. Tu, and J. Gronsbell. Machine learning approaches for electronic health records phenotyping: a methodical review. Journal of the American Medical Informatics Association: JAMIA, 30(2): 367-381, Jan. 2023. ISSN 1527-974X. doi: 10.1093/jamia/ocac216. · DOI
  59. C. Yuan, P. B. Ryan, C. Ta, Y. Guo, Z. Li, J. Hardin, R. Makadia, P. Jin, N. Shang, T. Kang, and C. Weng. Criteria2Query: a natural language interface to clinical databases for cohort definition. Journal of the American Medical Informatics Association, 26(4):294-305, Apr. 2019. ISSN 1527-974X. doi: 10.1093/jamia/ocy178. URL https://academic. oup.com/jamia/article/26/4/294/5308980. · DOI · Paper page
  60. X. Zhang, C. Xiao, L. M. Glass, and J. Sun. DeepEnroll: Patient-Trial Matching with Deep Embedding and Entailment Prediction, Jan. 2020. URL http://arxiv.org/abs/2001.08179. arXiv:2001.08179 [cs. · arXiv
  61. Y. Zhu, H. Yuan, S. Wang, J. Liu, W. Liu, C. Deng, Z. Dou, and J.-R. Wen. Large Language Models for Information Retrieval: A Survey, Aug. 2023. URL http://arxiv.org/abs/2308.07107. arXiv:2308.07107 [cs]. · arXiv

Cited by

See all 1

Metadata

Authors
Thai, D. N., Ardulov, V., Mena, J. U., Tiwari, S., Erofeev, G., Eskander, R., Tarabishy, K., Parikh, R. B., & Salloum, W.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2406.14780
arXiv
2406.14780
References
1
Type
Preprint
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: ACR: A Benchmark for Automatic Cohort Retrieval. https://app.cochat.ai/papers/10.48550/arxiv.2406.14780
The paper
Thai, D. N., Ardulov, V., Mena, J. U., Tiwari, S., Erofeev, G., Eskander, R., Tarabishy, K., Parikh, R. B., & Salloum, W. (2024). ACR: A Benchmark for Automatic Cohort Retrieval. arXiv. https://doi.org/10.48550/arxiv.2406.14780

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.