CoChat
Sign Up
Cited by 298Biomedical Text Mining and OntologiesArtificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

A critical assessment of using ChatGPT for extracting structured data from clinical notes

Open access
Huang, Jingwei, Yang, Donghan M., Rong, Ruichen, Nezafati, Kuroush, Treager, Colin, Chi, Zhikai, Wang, Shidan, Cheng, Xian +6 more·2024·npj Digital Medicine
DOI 10.1038/s41746-024-01079-8PMC PMC11063058
Open this paper in CoChat — free account

Abstract

Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT’s capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral “prompt engineering” process, leveraging OpenAI’s API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data. ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset, outperforming the performance of two traditional NLP methods. The performance is influenced by the design of the instructive prompt. Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules. Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time. In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively. Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training. The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.

AI snapshot

Written from the paper’s full text.

TL;DR

This study developed an LLM-based workflow using ChatGPT with spiral prompt engineering to extract structured data from clinical pathology notes. ChatGPT-3.5 achieved an average accuracy of 89% on lung cancer staging/histology extraction and 98.6%/100% accuracy on pediatric osteosarcoma grade/margin classification, while showing limitations in specialized pathology terminology and TNM rule interpretation.

Key findings

  • On 774 valid TCGA lung cancer pathology reports, ChatGPT-3.5 (gpt-3.5-turbo-16k) achieved average accuracy 0.89, with pT 0.87, pN 0.91, overall tumor stage 0.76, and histological diagnosis 0.99; average coverage was 0.95.
  • ChatGPT outperformed two traditional NLP methods on the same testing data: average accuracy 0.89 for ChatGPT vs. 0.51 for WordPiece tokenizer and 0.76 for a Bio_ClinicalBERT-based NER classifier.
  • In pediatric osteosarcoma data, ChatGPT accurately classified histology grade with 98.6% accuracy and margin status with 100% accuracy.
  • Prompt engineering improved training-data classification accuracy from 0.854 (prompt v1) to 0.865 (v2), 0.90 (v3), and 0.936 (v4); reproducibility testing showed 0.913 equivalence between April and May 2023 outputs and mean absolute certainty-degree error of 0.051.

Methods

  • The study used an LLM-based workflow with OpenAI API batch querying, systems engineering methodology, and a spiral “prompt engineering” process; scanned PDFs were converted to text with R package pdftools, and ChatGPT was instructed to use AJCC 7th edition staging rules and output structured JSON.
  • Lung cancer data included 1026 pathology reports from CDSA and TCGA; 78 valid CDSA reports were used for prompt development, and 774 valid TCGA reports were used for testing after excluding overlaps and invalid reports. Pediatric osteosarcoma data included 191 cases, with 148 having histology grade and 81 having margin status.
  • Performance was evaluated by comparing ChatGPT outputs with expert-curated structured data using accuracy, F1, Kappa, recall, precision, coverage, and confusion matrices, plus case-based error analysis, reproducibility testing, and comparison with keyword search and NER methods.

Limitations

  • The paper states that ChatGPT showed occasional “hallucination,” especially on blank or near-blank reports, and that most misclassifications were due to lack of highly specialized pathology terminology or erroneous interpretation of TNM staging rules.
  • The paper notes that ChatGPT models evolve over time, that GPT-4-Turbo performed better but was much more expensive, and that privacy concerns require careful handling for sensitive healthcare data.
  • Evident from the text: prompt development used only 78 training reports, overall tumor stage accuracy was lower than other attributes (0.76), and the evaluation is limited to lung cancer pathology reports plus one pediatric osteosarcoma dataset.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-024-01079-8

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 16 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    ChatGPT-3.5 achieved an overall accuracy of 89% for extracting pathological classifications from lung cancer pathology reports.

    “ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset”
    • Also stated in: § Overall Performance · plain
      “The average accuracy of all attributes is 0.89.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the pediatric osteosarcoma dataset, ChatGPT-3.5 classified grades with an accuracy of 98.6%.

    “In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively.”
    Quote in the paperSubstantial claim
  • Findingown data

    ChatGPT-3.5 outperformed two traditional NLP methods on the lung cancer dataset.

    “outperforming the performance of two traditional NLP methods.”
    • Also stated in: § Comparison with other NLP methods · plain
      “The results (Supplementary Table S1) show that the ChatGPT outperforms WordPiece tokenizer and NER Classifier.”
    Quote in the paperSubstantial claim
  • Findingnone given

    ChatGPT-3.5's performance is influenced by the design of the instructive prompt.

    “The performance is influenced by the design of the instructive prompt.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the pediatric osteosarcoma dataset, ChatGPT-3.5 classified margin status with an accuracy of 100%.

    “In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 20
  • Classified as probably not a substantial claim. 21

  • May combine two claims.

    Findingown data

    Among 56 cases where ChatGPT misclassified stage II as stage III, 22 were due to error propagation and 34 were due to incorrect rules.

    “Looking into the 56 cases where ChatGPT misclassified stage II as stage III, 22 cases were due to error propagation, and 34 were due to incorrect rules.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among pT misclassification cases, the majority were T1 misclassified as T2 (67 out of 753 or 8.9%) or T3 misclassified as T2 (12 out of 753, or 1.6%).

    “Among these misclassification cases, the majority were T1 misclassified as T2 (67 out of 753 or 8.9%) or T3 misclassified as T2 (12 out of 753, or 1.6%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ChatGPT accurately classified osteosarcoma grades with 98.6% accuracy and margin status with 100% accuracy.

    “ChatGPT accurately classified both grades (with a 98.6% accuracy rate) and margin status (100% accuracy), as shown in Supplementary Fig. 3.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among 15 missing report forms, ChatGPT responded 'unknown' as expected in only 5 cases, with the remaining 10 still converging to the hallucinated response.

    “Last, among the 15 missing report forms in a small dataset, ChatGPT responded “unknown” as expected in only 5 cases, with the remaining 10 still converging to the hallucinated response.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In most pT misclassification cases, ChatGPT extracted the correct tumor size information but used an incorrect rule to distinguish pT categories.

    “In most cases, ChatGPT extracted the correct tumor size information but used an incorrect rule to distinguish pT categories.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Responses to blank or near-blank reports reflect hallucination, though these instances can be detected and corrected due to convergence towards an 'attractor'.

    “the responses to blank or near-blank reports reflect this issue, though these instances can be detected and corrected due to convergence towards an “attractor”.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among histology mistakes, ChatGPT misclassified 3 as 'other' type and 3 cases of actual 'other' type as 2 adenocarcinomas and 1 squamous cell carcinoma.

    “Among the mistakes that ChatGPT made for histology diagnosis, ChatGPT misclassified 3 of them as “other” type and 3 cases of actual “other” type (neither adenocarcinomas nor squamous cell carcinomas) as 2 adenocarcinomas and 1 squamous cell carcinoma.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4-turbo performs better than GPT-3.5-Turbo in almost every aspect, increasing overall performance by over 5%.

    “The results show that GPT-4-turbo performs better in almost every aspect; overall, the GPT-4-turbo model increases performance by over 5%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The institutional review board at UT Southwestern approved this study and waived patient consent for using retrospective, de-identified EHR data.

    “The institutional review board at the University of Texas Southwestern Medical Center has approved this study where patient consent was waived for using retrospective, de-identified electronic health record data.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Following the AJCC guideline, individual pathologic T, N, and M stage components can be summarized into an overall pathologic staging score of Stage I, II, III, or IV.

    “Following the AJCC guideline, individual pathologic T, N, and M stage components can be summarized into an overall pathologic staging score of Stage I, II, III, or IV.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [16] Amin, M. B. et al. AJCC cancer staging manual. 1024: Springer 2017.

      Not checked in the free run.

    • cites [17] Goldstraw, P. et al. The IASLC Lung Cancer Staging Project: Proposals for the Revision of the TNM Stage Groupings in the Forthcoming (Seventh) Edition of the TNM Classification of Malignant Tumours. J. Thorac. Oncol. 2, 706–714 (2007). source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The CDSA dataset was used as training data for prompt development, while the TCGA dataset, after removing overlapping cases, served as test data for evaluating ChatGPT model performance.

    “The CDSA dataset was utilized as the “training” data for prompt development, while the TCGA dataset, after removing the overlapping cases with CDSA, served as the test data for evaluating the ChatGPT model performance.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Invalid reports were preserved to evaluate ChatGPT's handling of irregular inputs separately, and were not included in the testing data for accuracy performance assessment.

    “The invalid reports were preserved to evaluate ChatGPT’s handling of irregular inputs separately, and were not included in the testing data for accuracy performance assessment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    We used OpenAI's 'tiktoken' Python library to estimate token count to ensure compliance with context window limits.

    “We used OpenAI’s “tiktoken” Python library to estimate the token count to ensure compliance.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: All LLMs have limited context windows, constraining the input length of a query.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Machine learning methods, particularly deep learning, typically employ neural networks and first-generation transformer-based large language models (e.g., BERT).

    “Machine learning methods11–14, particularly deep learning, typically employ neural networks and the first generation of transformer-based large language models (e.g., BERT).”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [11] Crichton, G. et al. A neural network multi-task learning approach to biomedical named entity recognition. BMC Bioinforma. 18, 368 (2017). source

      Not checked in the free run.

    • cites [12] Wang, J. et al. Document-Level Biomedical Relation Extraction Using Graph Convolutional Network and Multihead Attention: Algorithm Development and Validation. JMIR Med Inf. 8, e17638 (2020). source

      Not checked in the free run.

    • cites [13] Liu, Y. et al. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019).

      Not checked in the free run.

    • cites [14] Rasmy, L. et al. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digit. Med. 4, 86 (2021). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The approach employed in this study effectively leverages the OpenAI API for batch queries of ChatGPT services across a large set of tasks with similar input data structures, including but not limited to pathology reports and EHR.

    “The approach employed in this study effectively leverages the OpenAI API for batch queries of ChatGPT services across a large set of tasks with similar input data structures, including but not limited to pathology reports and EHR.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To mimic a real-world setting, we processed all reports regardless of data quality to collect model responses.

    “To mimic a real-world setting, we processed all reports regardless of data quality to collect model responses.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    For performance evaluation, we only used valid reports providing meaningful text and excluded reports with near-empty content, poor scanning quality, and missing report forms.

    “For performance evaluation, we only used valid reports providing meaningful text and excluded the reports with near-empty content, poor scanning quality, and missing report forms, which were reported as irregular cases.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The training dataset underwent meticulous annotation by experienced medical professionals, adhering to the AJCC7 standards.

    “The training dataset underwent meticulous annotation by experienced medical professionals, adhering to the AJCC7 standards.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Augmented Language Models (ALMs) incorporate reasoning and external tools for interaction with the environment.

    “Augmented Language Models (ALMs)24, which incorporate reasoning and external tools for interaction with the environment, are emerging.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Mialon, G. et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023).

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationown data

    This study presents a general framework for using ChatGPT's API in batch queries to process large volumes of pathology reports for structured information extraction and estimation.

    “It presents a general framework for using ChatGPT’s API in batch queries to process large volumes of pathology reports for structured information extraction and estimation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Traditional automated extraction of structured data elements from medical notes has relied on medical NLP using rule-based or machine-learning approaches or a combination of both.

    “Traditionally, the automated extraction of structured data elements from medical notes has relied on medical natural language processing (NLP) using rule-based or machine-learning approaches or a combination of both9,10.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [9] Hao, T. et al. Health Natural Language Processing: Methodology Development and Applications. JMIR Med Inf. 9, e23898 (2021). source

      Not checked in the free run.

    • cites [10] Pathak, J., Kho, A. N. & Denny, J. C. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J. Am. Med. Inform. Assoc. 20, e206–e211 (2013). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The study compared ChatGPT-3.5 outputs with expert-curated structured data.

    “comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    In-context learning (few-shot prompting) can complement LLMs with task-specific knowledge to perform downstream tasks effectively.

    “Research shows that in-context learning (most influentially, few-shot prompting) can complement LLMs with task-specific knowledge to perform downstream tasks effectively24,25.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [24] Mialon, G. et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023).

      Not checked in the free run.

    • cites [25] Brown, T. et al. Language Models are Few-Shot Learners. Curran Associates, Inc. (2020).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Some staging records in TCGA structured data needed to be updated; physicians curated and updated those records.

    “Some staging records in the TCGA structured data needed to be updated; our physicians curated and updated those records.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    We evaluated performance by comparing ChatGPT output with expert-curated data elements from TCGA structured data using the testing data set.

    “We evaluated the performance of ChatGPT by comparing its output with expert-curated data elements provided in the TCGA structured data using the testing data set.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Adopting ChatGPT over traditional methods has advantages in cost-effectiveness and efficiency.

    “our study has demonstrated the advantages of adopting ChatGPT over traditional methods in terms of cost-effectiveness and efficiency.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    ChatGPT still has limitations in specific domain knowledge in lung cancer pathology and the capability of correcting understanding its terminology.

    “Both cases reveal that ChatGPT still has limitations in the specific domain knowledge in lung cancer pathology and the capability of correcting understanding its terminology.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The model incorporates 175 billion parameters and was trained on various public and authorized documents, demonstrating specific Artificial General Intelligence (AGI) capabilities.

    “The model incorporates 175 billion parameters and was trained on various public and authorized documents, demonstrating specific Artificial General Intelligence (AGI) capabilities5.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] OpenAi, GPT-4 Technical Report. arXiv:2303.08774: https://arxiv.org/pdf/2303.08774.pdf (2023).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Clinicians were actively formulating questions and evaluating the results.

    “In this process, clinicians were actively formulating questions and evaluating the results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    De-identified osteosarcoma pathology notes were analyzed by Microsoft Azure OpenAI covered by the Business Associate Agreement in this study.

    “de-identified Osteosarcoma pathology notes were analyzed by Microsoft Azure OpenAI covered by the Business Associate Agreement in this study.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: De-identified data can be analyzed without significant privacy concerns.

  • May combine two claims.

    Premisecitation

    A significant obstacle to developing traditional medical NLP algorithms is the limited existence of human-annotated datasets and the costs associated with new human annotation.

    “However, a significant obstacle to developing these traditional medical NLP algorithms is the limited existence of human-annotated datasets and the costs associated with new human annotation15.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] Wu, H. et al. A survey on clinical natural language processing in the United Kingdom from 2007 to 2022. npj Digit. Med. 5, 186 (2022). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    LLMs have a potential role in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.

    “The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Missing data were excluded from accuracy evaluation, and coverage rate was reported for predicted values as 'unknown' or empty output.

    “Missing data were excluded from the accuracy evaluation, and the coverage rate was reported for predicted values as ‘unknown’ or empty output.”
    Quote in the paperNot a substantial claim
33 of 33

  • The paper relies on this without support.

    Few studies have examined the potential of LLMs in reading and interpreting clinical notes, turning unstructured texts into structured, analyzable data.

    “To date, few studies have examined the potential of LLMs in reading and interpreting clinical notes, turning unstructured texts into structured, analyzable data.”

  • The paper relies on this without support.

    Medical domain knowledge needs to be integrated into model designs to enhance performance.

    “Medical domain knowledge needs to be integrated into model designs to enhance performance.”

  • The paper relies on this without support.

    Despite meticulous ground-truth labeling, relatively small corpus sizes often result in models with poor generalizability or make evaluations of generalizability impossible.

    “Despite meticulous ground-truth labeling, the relatively small corpus sizes often result in models with poor generalizability or make evaluations of generalizability impossible.”

  • The paper relies on this without support.

    Conventional AI systems have suffered from a lack of general knowledge and commonsense reasoning for decades.

    “For decades, conventional artificial intelligence (AI) systems (symbolic and neural networks) have suffered from a lack of general knowledge and commonsense reasoning.”

  • The paper relies on this without support.

    LLMs, like GPT, offer a promising alternative, potentially using commonsense reasoning and broad general knowledge to facilitate language processing.

    “LLMs, like GPT, offer a promising alternative, potentially using commonsense reasoning and broad general knowledge to facilitate language processing.”

  • The paper relies on this without support.

    The lung cancer cases in the dataset are predominantly non-metastatic.

    “As the lung cancer cases in our dataset are predominantly non-metastatic, the pathologic metastasis (pM) stage was not extracted.”

  • The paper relies on this without support.

    ChatGPT models (even with the same version) evolve over time.

    “Since ChatGPT models (even with the same version) evolve over time, it is important to evaluate the stability and reproducibility of ChatGPT.”

  • The paper relies on this without support.

    A simple prompt without special design would cause ChatGPT to produce unstructured answers varying in format, terminology, and granularity across many pathology reports.

    “However, this simple prompt would make ChatGPT produce unstructured answers varying in format, terminology, and granularity across the large number of pathology reports.”

  • The paper relies on this without support.

    The main obstacle to developing traditional medical NLP algorithms is the limited availability of annotated data and the costs for new human annotations.

    “The main obstacle to developing traditional medical NLP algorithms is the limited availability of annotated data and the costs for new human annotations.”

  • The paper relies on this without support.

    Providing examples is an effective way for humans to learn, and it should have similar effects for ChatGPT.

    “Providing examples is an effective way for humans to learn, and it should have similar effects for ChatGPT.”

  • The paper relies on this without support.

    Tumor size plays a critical role in classifying the pT stage.

    “Although tumor size is not a primary interest for diagnosis and clinical research, it plays a critical role in classifying the pT stage.”

  • The paper relies on this without support.

    All LLMs have limited context windows, constraining the input length of a query.

    “All LLMs have limited context windows, constraining the input length of a query.”

  • The paper relies on this without support.

    The output of an LLM depends on both the model and the prompt.

    “Since the output of an LLM depends on both the model and the prompt, mitigating hallucination can be achieved through improvements in GPT models and prompting strategies.”

  • The paper relies on this without support.

    De-identified data can be analyzed without significant privacy concerns.

    “Although it is a less concern to analyze public available de-identified data, like the lung cancer pathology notes used in this study, careful considerations are needed for secured healthcare data.”
14 of 14

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These LLMs harbor the potential to balance accuracy and efficiency when processing large-scale, complex, unstructured text datasets found in EHRs.

    “These LLMs harbor the potential to balance accuracy and efficiency when processing large-scale, complex, unstructured text datasets found in EHRs [19].”

    From Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review (2025) · cites this paper as [19]

  • Providing LLMs with example outputs for few-shot learning and chain-of-thought reasoning methods for prompting yielded higher classification performance compared to baseline zero-shot applications of LLMs for data extraction.

    “Huang et al [19] demonstrated that providing LLMs with example outputs for few-shot learning and chain-of-thought reasoning methods for prompting yielded higher classification performance compared to baseline zero-shot applications of LLMs for data extraction.”

    From Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review (2025) · cites this paper as [19]

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Vaswani, A. et al. Attention is all you need. Adv. Neural Info. Processing Syst. 30, (2017).
  2. Amin, M. B. et al. AJCC cancer staging manual. 1024: Springer 2017.

Cited by

Sorted by citations

Related

Authors
Huang, J., Yang, D. M., Rong, R., Nezafati, K., Treager, C., Chi, Z., Wang, S., Cheng, X., Guo, Y., Klesse, L. J., Xiao, G., Peterson, E. D., Zhan, X., & Xie, Y.
Affiliations
The University of Texas Southwestern Medical Center (US)
Corresponding author
  • Zhan, Xiaowei
  • Xie, Yang
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2024-05-01
Volume
7
Issue
1
Pages
106-106
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-024-01079-8
PMC
PMC11063058
OpenAlex
W4396553888
Semantic Scholar
a44b237b93664d32d09dae6c5de8d9b88e0f01a2
Cited by
298
References
23
Citations by year
2026: 109 · 2025: 146 · 2024: 43
FWCI
40.57
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. Department of Health & Human Services | NIH | National Institute of General Medical Sciences (R35GM136375, R01GM141519, R01GM140012)
  • U.S. Department of Health & Human Services | NIH | National Cancer Institute (P30CA142543, P50CA70907, U01CA249245)
  • U.S. Department of Health & Human Services | NIH | National Institute of Dental and Craniofacial Research (R01DE030656)
  • Cancer Prevention and Research Institute of Texas (RP230330)
  • Division of Intramural Research, National Institute of Allergy and Infectious Diseases (U01AI169298)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
ChatGPT, large language models, clinical information extraction, structured data extraction, clinical notes, prompt engineering, pediatric osteosarcoma, pathology terminology, TNM staging, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review