Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 39 of the 40 references it cites were found in a registry or an index; 22 of the 27 sources its claims cite were checked for support.

Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation

Open access
Yim, Wen-wai, Fu, Yujuan, Ben Abacha, Asma, Snider, Neal, Lin, Thomas, Yetisgen, Meliha · 2023 · Scientific Data

Abstract

Recent immense breakthroughs in generative models such as in GPT4 have precipitated re-imagined ubiquitous usage of these models in all applications. One area that can benefit by improvements in artificial intelligence (AI) is healthcare. The note generation task from doctor-patient encounters, and its associated electronic medical record documentation, is one of the most arduous time-consuming tasks for physicians. It is also a natural prime potential beneficiary to advances in generative models. However with such advances, benchmarking is more critical than ever. Whether studying model weaknesses or developing new evaluation metrics, shared open datasets are an imperative part of understanding the current state-of-the-art. Unfortunately as clinic encounter conversations are not routinely recorded and are difficult to ethically share due to patient confidentiality, there are no sufficiently large clinic dialogue-note datasets to benchmark this task. Here we present the Ambient Clinical Intelligence Benchmark ( aci-bench ) corpus, the largest dataset to date tackling the problem of AI-assisted note generation from visit dialogue. We also present the benchmark performances of several common state-of-the-art approaches.

Overview

CoChat Summary

TL;DR

The paper introduces aci-bench, a public benchmark corpus for generating clinical visit notes from doctor-patient dialogue, comprising 207 role-played encounter-note pairs across three note-generation modes. It also reports baseline results for retrieval, BART/BioBART, LED, and OpenAI models, finding that division-based generation helps fine-tuned models while GPT-4 achieves the best medical-concept score.

Key findings

  • aci-bench is presented as the largest public corpus for AI-assisted note generation from visit dialogue, with 207 encounters split into train (67), validation (20), and three test sets (40 each); it includes virtassist, virtscribe, and aci subsets.
  • In full-note test set 1, BART + FTSAMSum (Division) achieved the highest ROUGE scores: ROUGE-1 53.46, ROUGE-2 25.08, ROUGE-L 48.62; GPT-4 achieved the highest medcon score of 57.78, with ROUGE-1 51.76, ROUGE-2 22.58, ROUGE-L 45.97.
  • Division-based generation improved BART- and LED-based models over full-note generation, with reported increases of 1–14 points in full-note ROUGE and medcon, and around 20% average-score increases for objective_exam, objective_results, and assessment_and_plan divisions.
  • ASR versus human transcription and ASR versus ASR-corrected transcripts did not show remarkable note-generation performance differences, with approximately 1 F1 point differences across test sets and metrics for the evaluated BART + FTSAMSum (Division) setup.

Methods

  • The corpus was created from role-played doctor-patient encounters and expert-generated clinical notes, with three subsets: virtual assistant, virtual scribe, and ambient clinical intelligence; transcripts include human transcription and/or ASR versions depending on subset.
  • Notes were cleaned by annotating and removing unsupported note text, with four annotators and a reported partial span overlap agreement of 0.85 F1; notes were partitioned into four contiguous divisions: subjective, objective_exam, objective_results, and assessment_and_plan.
  • Baselines included transcript copy-and-paste, retrieval-based methods, BART/BioBART, LED, and OpenAI models; evaluation used ROUGE, BERTScore, BLEURT, and a UMLS concept-based medcon metric, at both full-note and division levels.

Limitations

  • The paper states the data is small and synthetically produced by medical annotators or patient actors in a single institution, so it may not statistically represent all health topics, speech variations, or note formats in the real world.
  • The paper states content creators were limited and based in a Nuance product demonstration studio in Eastern Massachusetts, with demographic details such as age, education, origin, and schooling not captured; note format variations were intentionally limited, and EMR structured-data incorporation was not modeled.
  • The paper notes ASR errors were tolerable under its automatic metrics but could matter clinically for particular medical facts; the text also makes evident that the benchmark uses role-played rather than real clinical encounters and that evaluation relies heavily on automatic metrics.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41597-023-02487-3

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Note division example. The same content in a clinical note can appear under different sections. As an example, in the left note, “past medical history” contents are written in the “history” portion of the note on the right. To separate the full note target into smaller text and minimize data sparsity problems if modeling by individual sections, notes are partitioned into separate subjective, objective_exam, objective_results, and assessment_and_plan continuous divisions. This also allows evaluation and generation at a higher granularity compared to a full note level.
Fig. 1 Note division example. The same content in a clinical note can appear under different sections. As an example, in the left note, “past medical history” contents are written in the “history” portion of the note on the right. To separate the full note target into smaller text and minimize data sparsity problems if modeling by individual sections, notes are partitioned into separate subjective, objective_exam, objective_results, and assessment_and_plan continuous divisions. This also allows evaluation and generation at a higher granularity compared to a full note level.CC BY · PubMed Central
Fig. 2 BERT subtoken lengths of concatenated gold/system summaries (test1 Text-davinci-003 system) for doctor-patient dialogue to clinical note generation task. As embedding-based models require encoding the concatenated reference and hypothesis, it would be difficult to fairly evaluate the corpus using current pretrained BERT models with a 512 limit.
Fig. 2 BERT subtoken lengths of concatenated gold/system summaries (test1 Text-davinci-003 system) for doctor-patient dialogue to clinical note generation task. As embedding-based models require encoding the concatenated reference and hypothesis, it would be difficult to fairly evaluate the corpus using current pretrained BERT models with a 512 limit.CC BY · PubMed Central

Verification

Claims the paper makes

12 Valid · 3 Not valid · 4 Not checked · 140 Own results and interpretations

65 more claims

Create a free account to see all 159 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 56
  • It may state two separate claims as one. 47
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 40 references

Research works printed but not linked: 5

  1. Yim, W., Yetisgen, M., Huang, J. & Grossman, M. Alignment annotation for clinic visit dialogue to clinical note sentence language generation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, 413–421 (European Language Resources Association, Marseille, France, 2020).
  2. Jurafsky, D. & Martin, J. H. Speech and language processing: An introduction to speech recognition, computational linguistics and natural language processing. Upper Saddle River, NJ: Prentice Hall (2008).
  3. Gliwa, B., Mochol, I., Biesek, M. & Wawer, A. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, 70–79, https://doi.org/10.18653/v1/D19-5409 (Association for Computational Linguistics, Hong Kong, China, 2019). · DOI · Paper page
  4. Soldaini, L. & Goharian, N. Quickumls: a fast, unsupervised approach for medical concept extraction. In MedIR workshop, sigir, 1–4 (2016).
  5. Hall, J. K. Teaching and researching: Language and culture (Routledge).

Cited by

See all 65

Metadata

Authors
Yim, W.-W., Fu, Y., Ben Abacha, A., Snider, N., Lin, T., & Yetisgen, M.
Affiliations
  • Microsoft (United States) (US)
  • University of Washington (US)
  • Nuance Communications (United States) (US)
Corresponding author
Yim, Wen-wai
Venue
Scientific Data
Published
2023-09-06
Volume
10
Issue
1
Pages
586-586
ISSN
2052-4463 (ISSN-L)
OpenAlex
W4386492483
Cited by
64
References
40
Citations by year
2026: 23 · 2025: 21 · 2024: 11 · 2023: 9
Type
data paper
Language
en
Open access
Licence
Locations
  • Scientific Data (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
clinical note generation, benchmark dataset, electronic health records, doctor-patient dialogue, generative models, evaluation metrics, AI-assisted documentation
MeSH terms
Artificial Intelligence, Health Facilities, Humans, Benchmarking, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation. https://app.cochat.ai/papers/10.1038/s41597-023-02487-3
The paper
Yim, W.-W., Fu, Y., Ben Abacha, A., Snider, N., Lin, T., & Yetisgen, M. (2023). Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation. Scientific Data. https://doi.org/10.1038/s41597-023-02487-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.