Overview of the MEDIQA-Chat 2023 Shared Tasks on the Summarization & Generation of Doctor-Patient Conversations
Open accessAbstract
Automatic generation of clinical notes from doctor-patient conversations can play a key role in reducing daily doctors' workload and improving their interactions with the patients.MEDIQA-Chat 2023 aims to advance and promote research on effective solutions through shared tasks on the automatic summarization of doctor-patient conversations and on the generation of synthetic dialogues from clinical notes for data augmentation.Seventeen teams participated in the challenge and experimented with a broad range of approaches and models.In this paper, we describe the three MEDIQA-Chat 2023 tasks, the datasets, and the participants' results and methods.We hope that these shared tasks will lead to additional research efforts and insights on the automatic generation and evaluation of clinical notes.Section Header Conversation Summary MEDICATIONS Doctor: Are you still taking the Trizivir?Patient: Yes.Doctor: How much are you taking?Patient: I take one pill two times a day.Doctor: Are you taking any other medications?Patient: I take Ibuprofen for body aches from time to time but that's it.1.She is on Trizivir 1 tablet p.o. b.i.d. 2. Ibuprofen over-the-counter p.r.n.ROS Doctor: Have you had any anxiety attacks lately?Patient: No. Doctor: Have you felt depressed or had any mood swing problems?Patient: No. Doctor: Any phobias?Patient: No, not really.Doctor: Okay.PSYCHIATRIC: Normal; Negative for anxiety, depression, or phobias.FAM/SOCHX Doctor: Are you still working?Patient: No, I am retired
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2004) · Cited by 9,718 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005 · see citing works
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
Gliwa, B., Mochol, I., Biesek, M., & Wawer, A. (2019) · Proceedings of the 2nd Workshop on New Frontiers in Summarization · doi:10.18653/v1/d19-5409 · Cited by 424 · see citing works
DialogSum: A Real-Life Scenario Dialogue Summarization Dataset
Chen, Y., Liu, Y., Liang, C., & Zhang, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.449 · Cited by 183 · see citing works
Abacha, A. B., Shivade, C., & Demner‐Fushman, D. (2019) · BioNLP Workshop and Shared Task (BioNLP) · doi:10.18653/v1/w19-5039 · Cited by 119 · see citing works
Overview of the MEDIQA 2021 Shared Task on Summarization in the Medical Domain
Abacha, A. B., Mrabet, Y., Zhang, Y., Shivade, C., Langlotz, C. P., & Demner‐Fushman, D. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.8 · Cited by 67 · see citing works
Yim, W.-W., Fu, Y., Ben Abacha, A., Snider, N., Lin, T., & Yetisgen, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02487-3 · Cited by 64 · cited 1× · see citing works
“We encouraged the participants to apply the models developed for this task to generate additional data for tasks A and B. (Yim et al., 2023) collections.”
MethodSelf-citationAn Empirical Study of Clinical Note Generation from Doctor-Patient Encounters
Ben Abacha, A., Yim, W.-W., Fan, Y., & Lin, T. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.168 · Cited by 54 · see citing works
Self-citationHuman Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Moramarco, F., Papadopoulos Korfiatis, A., Perera, M., Juric, D., Flann, J., Reiter, E., Belz, A., & Savkov, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.394 · Cited by 40 · see citing works
Giorgi, J., Toma, A., Xie, R., Chen, S., An, K., Zheng, G., & Wang, B. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.36 · Cited by 31 · see citing works
Michalopoulos, G., Williams, K. B., Singh, G., & Lin, T. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.349 · Cited by 27 · see citing works
PriMock57: A Dataset Of Primary Care Mock Consultations
Papadopoulos Korfiatis, A., Moramarco, F., Sarac, R., & Savkov, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.65 · Cited by 23 · see citing works
User-Driven Research of Medical Note Generation Software
Knoll, T., Moramarco, F., Korfiatis, A. P., Young, R., Ruffini, C., Perera, M., Perstl, C., Reiter, E., Belz, A., & Savkov, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.29 · Cited by 23 · see citing works
Yim, W.-W., & Yetişgen, M. (2021) · doi:10.18653/v1/2021.nlpmc-1.2 · Cited by 19 · see citing works
Self-citationSummQA at MEDIQA-Chat 2023: In-Context Learning with GPT-4 for Medical Summarization
Mathur, Y., Rangreji, S., Kapoor, R., Palavalli, M., Bertsch, A., & Gormley, M. R. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.51 · Cited by 19 · see citing works
A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
Adams, G., Zucker, J. E., & Elhadad, N. (2023) · PubMed · doi:10.48550/arxiv.2303.03948 · Cited by 15 · see citing works
Self-citationAn Investigation of Evaluation Metrics for Automated Medical Note Generation
Abacha, A. B., Yim, W.-W., Michalopoulos, G., & Lin, T. (2023) · arXiv · doi:10.48550/arxiv.2305.17364 · Cited by 11 · see citing works
Self-citationTang, X., Tran, A., Tan, J. T. C., & Gerstein, M. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.58 · Cited by 11 · see citing works
Wang, J., Yao, Z., Mitra, A., Osebe, S., Yang, Z., & Yu, H. Q. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.49 · Cited by 9 · see citing works
Sharma, A., Feldman, D., & Jain, A. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.28 · Cited by 6 · see citing works
Zhang, B., Mishra, R., & Teodoro, D. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.57 · Cited by 6 · see citing works
Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation
Savkov, A., Moramarco, F., Korfiatis, A. P., Perera, M., Belz, A., & Reiter, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.10 · Cited by 5 · see citing works
Zhang, B., Mishra, R., & Teodoro, D. (2023) · medRxiv · doi:10.1101/2023.06.08.23291121 · Cited by 4 · see citing works
Milintsevich, K., & Agarwal, N. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.56 · Cited by 4 · see citing works
Suri, K., Saha, S., & Singh, A. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.50 · Cited by 3 · see citing works
clulab at MEDIQA-Chat 2023: Summarization and classification of medical dialogues
Özler, K. B., & Bethard, S. J. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.19 · Cited by 3 · see citing works
Alqahtani, A., Salama, R., Diab, M., & Youssef, A. S. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.55 · Cited by 3 · see citing works
Tang, X., Tran, A., Tan, J. T. C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05001 · Cited by 2 · see citing works
Teddysum at MEDIQA-Chat 2023: an analysis of fine-tuning strategy for long dialog summarization
Jeong, Y., Han, J.-H., Chae, K. M., Cho, Y., Seo, H., Lim, K., Choi, K., & Hahm, Y. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.42 · Cited by 2 · see citing works
Srivastava, D. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.54 · Cited by 2 · see citing works
Mishra, P., & Desetty, R. T. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.44 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Colin Grambow, Longxiang Zhang, and Thomas Schaaf. 2022. In-domain pre-training improves clinical note generation from doctor-patient conversations. In Pro- ceedings of the First Workshop on Natural Language Generation in Healthcare, pages 9-22, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
Cited by
Sorted by citations
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · see citing works
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37 · see citing works
Disambiguation of acronyms in clinical narratives with large language models
Kugic, A., Schulz, S., & Kreuzthaler, M. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae157 · Cited by 18 · see citing works
Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658966 · Cited by 17 · see citing works
Ionescu, B. E., Müller, H., Drăgulinescu, A., Yim, W.-W., Abacha, A. B., Snider, N., Adams, G., Yetişgen, M., Rückert, J., de Herrera, A. G. S., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42448-9_25 · Cited by 14 · see citing works
Self-citationHow Do Learners Receive Feedback on Note Writing? A Scoping Review
Hansen, A., Klute, R. M., Yadav, M., Bansal, S. K., & Bond, W. F. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005653 · Cited by 7 · see citing works
Zhang, B., Mishra, R., & Teodoro, D. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.57 · Cited by 6 · see citing works
Overview of the ImageCLEF 2024: Multimedia Retrieval in Medical Applications
Ionescu, B. E., Müller, H., Drăgulinescu, A., Rückert, J., Abacha, A. B., García, A., Bloch, L., Brüngel, R., Idrissi-Yaghir, A., Schäfer, H., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71908-0_7 · Cited by 8 · see citing works
Self-citationBrake, N., & Schaaf, T. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.25 · Cited by 6
Enhancing Medical Dialogue Summarization: A MediExtract Distillation Framework
Liu, X., Huang, M., Rusnachenko, N., Ive, J., Chang, J. M., & Zhang, J. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822640 · Cited by 4 · see citing works
Green, K., Asthana, S., Ponce, O. J., Downey, J., & Watson, J. (2025) · JMIR Research Protocols · doi:10.2196/68511 · Cited by 2 · see citing works
Zhang, C., Chen, T., Chen, J., Wang, H., Shi, J., Luo, Z., & Zhang, M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679671 · Cited by 2 · see citing works
Ionescu, B. E., Müller, H., Stanciu, D.-C., Andrei, A.-G., Radzhabov, A., Prokopchuk, Y., Stefan, L., Constantin, M. G., Dogariu, M., Kovalev, V. A., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_17 · Cited by 2 · see citing works
Self-citationNishio, M., Matsunaga, T., Matsuo, H., Nogami, M., Kurata, Y., Fujimoto, K., Sugiyama, O., Akashi, T., Aoki, S., & Murakami, T. (2023) · medRxiv · doi:10.1101/2023.12.01.23299267 · Cited by 2 · see citing works
Ahmed, S., & Sadeque, F. (2026) · JMIR Medical Informatics · doi:10.2196/82545 · Cited by 1 · see citing works
A Language Model for Pediatric Occupational Therapy Documentation: Model Development and Pilot Study
DiMaio, R., Tuinstra, T., Yu, T., Koshy, I., Wylie-Toal, B., & Tripp, B. (2026) · JMIR AI · doi:10.2196/73274 · Cited by 1 · see citing works
Ahmed, S., & Sadeque, F. (2025) · doi:10.2196/preprints.82545
Wang, Q., Yong, X. Y. X., Wang, Y., Zhongyan, L., Wang, X., & Gao, J. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115605
Breaking Memory Wall for Fast Edge LLM Inference Using Contextual Sparsity
Wei, Z., Zhou, Y., Zhao, J., Li, K., & Fu, X. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3712257
Contextual Learning Strategies for Dialogue Summarization Using Large Language Models
Penyasov, D. E., & Turalchuk, K. (2026) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661826700252
Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts
Veen, D. V., Uden, C. V., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerova, A., et al. (2023) · doi:10.21203/rs.3.rs-3483777/v1 · Cited by 892 · see citing works
- Authors
- Ben Abacha, A., Yim, W.-W., Adams, G., Snider, N., & Yetisgen, M.
- Affiliations
- Microsoft (United States) (US)
- Columbia University (US)
- ORCID
- Yim, Wen-wai
- Venue
- Proceedings of the 5th Clinical Natural Language Processing Workshop
- Published
- 2023
- Pages
- 503-513
- OpenAlex
- W4385571763
- Cited by
- 20
- References
- 30
- Citations by year
- 2026: 6 · 2025: 4 · 2024: 7 · 2023: 3
- FWCI
- 3.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Clinical Natural Language Processing Workshop (ClinicalNLP) (publishedVersion, cc-by)
- Funders
- Strong
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- shared task, automatic summarization, doctor-patient conversation, clinical note generation, synthetic dialogue generation, data augmentation, clinical notes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
