CoChat
Sign Up
Cited by 63Information Retrieval and Search BehaviorMachine Learning in HealthcareBiomedical Text Mining and Ontologies

A large-scale dataset of patient summaries for retrieval-based clinical decision support systems

Open access
Zhao, Zhengyun, Jin, Qiao, Chen, Fangyuan, Peng, Tuorui, Yu, Sheng·2023·Scientific Data
DOI 10.1038/s41597-023-02814-8PMC PMC10728216
Open this paper in CoChat — free accountOpen PDF

Abstract

Retrieval-based Clinical Decision Support (ReCDS) can aid clinical workflow by providing relevant literature and similar patients for a given patient. However, the development of ReCDS systems has been severely obstructed by the lack of diverse patient collections and publicly available large-scale patient-level annotation datasets. In this paper, we collect a novel dataset of patient summaries and relations called PMC-Patients to benchmark two ReCDS tasks: Patient-to-Article Retrieval (ReCDS-PAR) and Patient-to-Patient Retrieval (ReCDS-PPR). Specifically, we extract patient summaries from PubMed Central articles using simple heuristics and utilize the PubMed citation graph to define patient-article relevance and patient-patient similarity. PMC-Patients contains 167k patient summaries with 3.1 M patient-article relevance annotations and 293k patient-patient similarity annotations, which is the largest-scale resource for ReCDS and also one of the largest patient collections. Human evaluation and analysis show that PMC-Patients is a diverse dataset with high-quality annotations. We also implement and evaluate several ReCDS systems on the PMC-Patients benchmarks to show its challenges and conduct several case studies to show the clinical utility of PMC-Patients.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41597-023-02814-8

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For MHB-Report, the paper leveraged the PMC-Patients dataset, the first large-scale, structured clinical case dataset extracted from case reports in PubMed Central.

    “For the MHB-Report dataset, we leveraged the PMC-Patients dataset (Zhao et al. 2023), the first large-scale, structured clinical case dataset extracted from case reports in PubMed Central (PMC).”

    From MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks (2026) · cites this paper as Zhao et al. 2023

    The Background & Summary says PMC-Patients contains patient summaries extracted from PMC case reports, and Technical Validation > Dataset characteristics describes it as the largest patient-summary dataset by number of patients. The source does not establish that it was the first large-scale, structured clinical case dataset; its claim of being first concerns large-scale patient-similarity annotations. The citing authors’ statement that they leveraged it for MHB-Report is their own account, not a claim this source must verify. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. [1]Sackett, D. L. Evidence-based medicine. In Seminars in perinatology, vol. 21, 3–5 (Elsevier, 1997). 9190027
  2. [2]Ely JW Osheroff JA Chambliss ML Ebell MH Rosenbaum ME Answering physicians’ clinical questions: obstacles and potential solutions Journal of the American Medical Informatics Association 2005 12 217 224 10.1197/jamia.M1608 15561792 PMC551553 · DOI
  3. [3]Roberts, K., Demner-Fushman, D., Voorhees, E. M. & Hersh, W. R. Overview of the trec 2016 clinical decision support track. In Voorhees, E. M. & Ellis, A. (eds.) Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA, November 15–18, 2016, vol. Special Publication 500-321 (National Institute of Standards and Technology (NIST), 2016).
  4. [4]Pan M An adaptive term proximity based rocchio’s model for clinical decision support retrieval BMC Medical Informatics and Decision Making 2019 19 1 11 10.1186/s12911-019-0986-6 31830960 PMC6907113 · DOI
  5. [5]Park B Afzal M Hussain J Abbas A Lee S Automatic identification of high impact relevant articles to support clinical decision making using attention-based deep learning Electronics 2020 9 1364 10.3390/electronics9091364 · DOI
  6. [6]Zhang Z An improved bm25 algorithm for clinical decision support in precision medicine based on co-word analysis and cuckoo search BMC Medical Informatics and Decision Making 2021 21 1 15 10.1186/s12911-021-01454-5 33653325 PMC7927407 · DOI
  7. [7]Zhang Z Lin X Wu S A hybrid algorithm for clinical decision support in precision medicine based on machine learning BMC bioinformatics 2023 24 1 18 36597033 PMC9811720
  8. [8]Gurulingappa, H., Toldo, L., Schepers, C., Bauer, A. & Megaro, G. Semi-supervised information retrieval system for clinical decision support. In Voorhees, E. M. & Ellis, A. (eds.) Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA, November 15–18, 2016, vol. Special Publication 500-321 (National Institute of Standards and Technology (NIST), 2016).
  9. [9]Sankhavara, J. Biomedical document retrieval for clinical decision support system. In Shwartz, V. et al. (eds.) Proceedings of ACL 2018, Melbourne, Australia, July 15-20, 2018, Student Research Workshop, 84–90 (Association for Computational Linguistics, 2018).
  10. [10]Shi, M.-X., Pan, T.-H., Chen, H.-H. & Huang, H.-H. Hybrid re-ranking for biomedical information retrieval at the trec 2021 clinical trials track. In Soboroff, I. & Ellis, A. (eds.) Proceedings of the Thirtieth Text REtrieval Conference, TREC 2021, online, November 15-19, 2021, vol. 500-335 of NIST Special Publication (National Institute of Standards and Technology (NIST), 2021).
  11. [11]Simpson, M. S., Voorhees, E. M. & Hersh, W. R. Overview of the trec 2014 clinical decision support track. In Voorhees, E. M. & Ellis, A. (eds.) Proceedings of The Twenty-Third Text REtrieval Conference, TREC 2014, Gaithersburg, Maryland, USA, November 19-21, 2014, vol. Special Publication 500-308 (National Institute of Standards and Technology (NIST), 2014).
  12. [12]Roberts, K., Simpson, M. S., Voorhees, E. M. & Hersh, W. R. Overview of the trec 2015 clinical decision support track. In Voorhees, E. M. & Ellis, A. (eds.) Proceedings of The Twenty-Fourth Text REtrieval Conference, TREC 2015, Gaithersburg, Maryland, USA, November 17-20, 2015, vol. Special Publication 500-319 (National Institute of Standards and Technology (NIST), 2015).
  13. [13]Buckley, C. & Voorhees, E. M. Retrieval evaluation with incomplete information. In Sanderson, M., Järvelin, K., Allan, J. & Bruza, P. (eds.) SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 25–29, 2004, 25–32 (ACM, 2004).
  14. [14]Seligson ND Recommendations for patient similarity classes: results of the amia 2019 workshop on defining patient similarity Journal of the American Medical Informatics Association 2020 27 1808 1812 10.1093/jamia/ocaa159 32885823 PMC7671612 · DOI
  15. [15]Plaza, L. & Daz, A. Retrieval of similar electronic health records using umls concept graphs. In International Conference on Application of Natural Language to Information Systems, 296–303 (Springer, 2010).
  16. [16]Arnold, C. W., El-Saden, S. M., Bui, A. A. & Taira, R. Clinical case-based retrieval using latent topic analysis. In AMIA annual symposium proceedings, vol. 2010, 26 (American Medical Informatics Association, 2010). PMC3041464 21346934
  17. [17]Johnson A Pollard T Mark R 2016 Mimic-iii clinical database PhysioNet 10.13026/C2XW26 · DOI · Paper page
  18. [18]Johnson A 2023 Mimic-iv PhysioNet 10.13026/6mm1-ek67 · DOI
  19. [19]Johnson AE Mimic-iii, a freely accessible critical care database Scientific data 2016 3 1 9 10.1038/sdata.2016.35 PMC4878278 27219127 · DOI · Paper page
  20. [21]Chen, T., et al. (eds.) Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part I, vol. 13185 of Lecture Notes in Computer Science, 95–110 (Springer, 2022).
  21. [22]Bruch, S., Gai, S. & Ingber, A. An analysis of fusion functions for hybrid retrieval. ACM Trans. Inf. Syst. 42 (2023).
  22. [23]Cormack, G. V., Clarke, C. L. & Buettcher, S. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, 758–759 (2009).
  23. [24]Robertson SE Zaragoza H The probabilistic relevance framework: Bm25 and beyond Found. Trends Inf. Retr. 2009 3 333 389 10.1561/1500000019 · DOI · Paper page
  24. [25]Xiao, S., Liu, Z., Zhang, P. & Muennighoff, N. C-pack: Packaged resources to advance general chinese embedding. Preprint at 10.48550/arXiv.2309.07597 (2023). · DOI · Paper page
  25. [26]Jin Q MedCPT: Contrastive Pre-trained Transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval Bioinformatics 2023 39 btad651 10.1093/bioinformatics/btad651 37930897 PMC10627406 · DOI · Paper page
  26. [28]Kumar, L. & Sarkar, S. Listbert: Learning to rank e-commerce products with listwise bert. Preprint at 10.48550/arXiv.2206.15198 (2022). · DOI
  27. [29]Vaswani, A. et al. Attention is all you need. In Guyon, I. et al. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach, CA, USA, 6000–6010 (2017).
  28. [31]Gu Y Domain-specific language model pretraining for biomedical natural language processing ACM Transactions on Computing for Healthcare (HEALTH) 2022 3 1 23 10.1145/3458754 · DOI · Paper page
  29. [32]Yasunaga, M., Leskovec, J. & Liang, P. Linkbert: Pretraining language models with document links. In Muresan, S., Nakov, P. & Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22–27, 2022, 8003–8016 (Association for Computational Linguistics, 2022).
  30. [33]Cohan, A., Feldman, S., Beltagy, I., Downey, D. & Weld, D. S. Specter: Document-level representation learning using citation-informed transformers. In Jurafsky, D., Chai, J., Schluter, N. & Tetreault, J. R. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5–10, 2020, 2270–2282 (Association for Computational Linguistics, 2020).
  31. [34]Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019 (OpenReview.net, 2019).
  32. [35]Jin, Q., Shin, A. & Lu, Z. Lader: Log-augmented dense retrieval for biomedical literature search. In Chen, H.-H. et al. (eds.) Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, 2092–2097 (ACM, 2023).
  33. [36]Teodoro D Information retrieval in an infodemic: the case of covid-19 publications Journal of medical Internet research 2021 23 e30161 10.2196/30161 34375298 PMC8451964 · DOI
  34. [38]Thakur, N., Reimers, N., Rücklí, A., Srivastava, A. & Gurevych, I. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Vanschoren, J. & Yeung, S. K. (eds.) Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual (2021).
  35. [39]Ren P Case report: Pathogenic myh9 c. 5797delc mutation in a patient with apparent thrombocytopenia and nephropathy Frontiers in Genetics 2021 12 705832 10.3389/fgene.2021.705832 34394193 PMC8355614 · DOI
  36. [40]Fernandez-Prado R Carriazo-Julio SM Torra R Ortiz A Perez-Gomez MV Myh9-related disease: it does exist, may be more frequent than you think and requires specific therapy Clinical kidney journal 2019 12 488 493 10.1093/ckj/sfz103 31384439 PMC6671427 · DOI
  37. [41]Horinouchi T Pathogenic evaluation of synonymous col4a5 variants in x-linked alport syndrome using a minigene assay Molecular genetics & genomic medicine 2020 8 e1342 10.1002/mgg3.1342 32543079 PMC7434753 · DOI
  38. [42]Troxell ML Houghton DC Atypical anti-glomerular basement membrane disease Clinical Kidney Journal 2016 9 211 221 10.1093/ckj/sfv140 26985371 PMC4792615 · DOI
  39. [43]Gutkin E Hussain SA Kim SH The successful treatment of chronic cholecystitis with spyglass cholangioscopy-assisted gallbladder drainage and irrigation through self-expandable metal stents Gut and liver 2012 6 136 10.5009/gnl.2012.6.1.136 22375186 PMC3286734 · DOI
  40. [44]Fukui T Biliary peritonitis caused by spontaneous bile duct rupture in the left triangular ligament of the liver after endoscopic sphincterotomy for choledocholithiasis Case Reports in Gastroenterology 2021 15 53 61 10.1159/000510932 33613164 PMC7879265 · DOI
  41. [45]Wang, M., Xing, Y., Gao, Q., Lv, Z. & Yuan, J. Mirizzi syndrome with an unusual aberrant hepatic duct fistula: a case report. International Medical Case Reports Journal 173–177 (2016). PMC4938132 27445508
  42. [46]Sweeny A Smith NA Serfin JA Hemorrhagic cholecystitis causing hemobilia and common bile duct obstruction Journal of Surgical Case Reports 2019 2019 rjz081 10.1093/jscr/rjz081 31001409 PMC6463387 · DOI
  43. [47]Roberts, K., Demner-Fushman, D., Voorhees, E. M., Bedrick, S. & Hersh, W. R. Overview of the trec 2020 precision medicine track. In Voorhees, E. M. & Ellis, A. (eds.) Proceedings of the Twenty-Ninth Text REtrieval Conference, TREC 2020, Virtual Event [Gaithersburg, Maryland, USA], November 16–20, 2020, vol. 1266 of NIST Special Publication (National Institute of Standards and Technology (NIST), 2020).
  44. [48]Myrdal, C. N. & Sundararajan, S. Response to ipilimumab/nivolumab rechallenge and braf inhibitor/mek inhibitor rechallenge in a patient with advanced metastatic melanoma previously treated with braf targeted therapy and immunotherapy. Case Reports in Oncological Medicine2020 (2020). PMC7334770 32670650
  45. [49]Martini DJ Response to single agent pd-1 inhibitor after progression on previous pd-1/pd-l1 inhibitors: a case series Journal for ImmunoTherapy of Cancer 2017 5 1 5 10.1186/s40425-017-0273-y 28807048 PMC5557522 · DOI

Cited by

Sorted by citations

Authors
Zhao, Z., Jin, Q., Chen, F., Peng, T., & Yu, S.
Affiliations
Tsinghua University (CN)
ORCID
Venue
Scientific Data
Publisher
Springer Science and Business Media LLC
Published
2023-12-18
Volume
10
Issue
1
Pages
909-909
ISSN
2052-4463 (ISSN-L)
DOI
10.1038/s41597-023-02814-8
PMC
PMC10728216
OpenAlex
W4389894040
Cited by
63
References
6
Citations by year
2026: 19 · 2025: 27 · 2024: 7 · 2023: 7 · 2022: 3
Type
data paper
Language
en
Open access
Licence
Locations
  • Scientific Data (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by-nc-sa)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Natural Science Foundation of China (12171270)
  • Natural Science Foundation of Beijing Municipality (Z190024)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
Keywords
patient summary, patient similarity, PubMed Central, benchmark dataset
MeSH terms
Humans, Patients, Decision Support Systems, Clinical, PubMed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review