Cited by 298Biomedical Text Mining and OntologiesArtificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.

A critical assessment of using ChatGPT for extracting structured data from clinical notes

Open access

Huang, Jingwei, Yang, Donghan M., Rong, Ruichen, Nezafati, Kuroush, Treager, Colin, Chi, Zhikai, Wang, Shidan, Cheng, Xian, Guo, Yujia, Klesse, Laura J., Xiao, Guanghua, Peterson, Eric D., Zhan, Xiaowei, Xie, Yang·2024·npj Digital Medicine

DOI 10.1038/s41746-024-01079-8PMC PMC11063058

Open this paper in CoChat — free account

Abstract

Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT’s capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral “prompt engineering” process, leveraging OpenAI’s API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data. ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset, outperforming the performance of two traditional NLP methods. The performance is influenced by the design of the instructive prompt. Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules. Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time. In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively. Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training. The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.

AI snapshot

Written from the paper’s full text.

TL;DR

This study developed an LLM-based workflow using ChatGPT with spiral prompt engineering to extract structured data from clinical pathology notes. ChatGPT-3.5 achieved an average accuracy of 89% on lung cancer staging/histology extraction and 98.6%/100% accuracy on pediatric osteosarcoma grade/margin classification, while showing limitations in specialized pathology terminology and TNM rule interpretation.

Key findings

On 774 valid TCGA lung cancer pathology reports, ChatGPT-3.5 (gpt-3.5-turbo-16k) achieved average accuracy 0.89, with pT 0.87, pN 0.91, overall tumor stage 0.76, and histological diagnosis 0.99; average coverage was 0.95.
ChatGPT outperformed two traditional NLP methods on the same testing data: average accuracy 0.89 for ChatGPT vs. 0.51 for WordPiece tokenizer and 0.76 for a Bio_ClinicalBERT-based NER classifier.
In pediatric osteosarcoma data, ChatGPT accurately classified histology grade with 98.6% accuracy and margin status with 100% accuracy.
Prompt engineering improved training-data classification accuracy from 0.854 (prompt v1) to 0.865 (v2), 0.90 (v3), and 0.936 (v4); reproducibility testing showed 0.913 equivalence between April and May 2023 outputs and mean absolute certainty-degree error of 0.051.

Methods

The study used an LLM-based workflow with OpenAI API batch querying, systems engineering methodology, and a spiral “prompt engineering” process; scanned PDFs were converted to text with R package pdftools, and ChatGPT was instructed to use AJCC 7th edition staging rules and output structured JSON.
Lung cancer data included 1026 pathology reports from CDSA and TCGA; 78 valid CDSA reports were used for prompt development, and 774 valid TCGA reports were used for testing after excluding overlaps and invalid reports. Pediatric osteosarcoma data included 191 cases, with 148 having histology grade and 81 having margin status.
Performance was evaluated by comparing ChatGPT outputs with expert-curated structured data using accuracy, F1, Kappa, recall, precision, coverage, and confusion matrices, plus case-based error analysis, reproducibility testing, and comparison with keyword search and NER methods.

Limitations

The paper states that ChatGPT showed occasional “hallucination,” especially on blank or near-blank reports, and that most misclassifications were due to lack of highly specialized pathology terminology or erroneous interpretation of TNM staging rules.
The paper notes that ChatGPT models evolve over time, that GPT-4-Turbo performed better but was much more expensive, and that privacy concerns require careful handling for sensitive healthcare data.
Evident from the text: prompt development used only 78 training reports, overall tumor stage accuracy was lower than other attributes (0.76), and the evaluation is limited to lung cancer pathology reports plus one pediatric osteosarcoma dataset.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-024-01079-8

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 16 cited sources checked · 0 supported · 0 flagged

A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset”
- Also stated in: § Overall Performance · plain
  “The average accuracy of all attributes is 0.89.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
Load-bearingStrength:boosted· not yet calibratedn = 774
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“outperforming the performance of two traditional NLP methods.”
- Also stated in: § Comparison with other NLP methods · plain
  “The results (Supplementary Table S1) show that the ChatGPT outperforms WordPiece tokenizer and NER Classifier.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time.”
- Also stated in: § Reproducibility evaluation · hedged
  “Considering the evolutionary nature of ChatGPT models, we regard an output difference to a certain extent as reasonable and the overall ChatGPT 3.5 model as stable.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes

Show all 39 claims

It may state two separate claims as one. 20
Probably not a substantial claim. 21

None of the sources cited for this claim have been checked yet.
In the paper
“Following the AJCC guideline, individual pathologic T, N, and M stage components can be summarized into an overall pathologic staging score of Stage I, II, III, or IV.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Data and endpoints
What the sources say
- Not checked yetcites [16] Amin, M. B. et al. AJCC cancer staging manual. 1024: Springer 2017.
- Not checked yetcites [17] Goldstraw, P. et al. The IASLC Lung Cancer Staging Project: Proposals for the Revision of the TNM Stage Groupings in the Forthcoming (Seventh) Edition of the TNM Classification of Malignant Tumours. J. Thorac. Oncol. 2, 706–714 (2007). source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We used OpenAI’s “tiktoken” Python library to estimate the token count to ensure compliance.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Data processing, workflow, and prompt engineering
Strength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: All LLMs have limited context windows, constraining the input length of a query.
None of the sources cited for this claim have been checked yet.
In the paper
“Machine learning methods11–14, particularly deep learning, typically employ neural networks and the first generation of transformer-based large language models (e.g., BERT).”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Introduction
What the sources say
- Not checked yetcites [11] Crichton, G. et al. A neural network multi-task learning approach to biomedical named entity recognition. BMC Bioinforma. 18, 368 (2017). source
- Not checked yetcites [12] Wang, J. et al. Document-Level Biomedical Relation Extraction Using Graph Convolutional Network and Multihead Attention: Algorithm Development and Validation. JMIR Med Inf. 8, e17638 (2020). source
- Not checked yetcites [13] Liu, Y. et al. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
- Not checked yetcites [14] Rasmy, L. et al. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digit. Med. 4, 86 (2021). source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Augmented Language Models (ALMs)24, which incorporate reasoning and external tools for interaction with the environment, are emerging.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
What the sources say
- Not checked yetcites [24] Mialon, G. et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023).
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Traditionally, the automated extraction of structured data elements from medical notes has relied on medical natural language processing (NLP) using rule-based or machine-learning approaches or a combination of both9,10.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Introduction
What the sources say
- Not checked yetcites [9] Hao, T. et al. Health Natural Language Processing: Methodology Development and Applications. JMIR Med Inf. 9, e23898 (2021). source
- Not checked yetcites [10] Pathak, J., Kho, A. N. & Denny, J. C. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J. Am. Med. Inform. Assoc. 20, e206–e211 (2013). source
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Research shows that in-context learning (most influentially, few-shot prompting) can complement LLMs with task-specific knowledge to perform downstream tasks effectively24,25.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
What the sources say
- Not checked yetcites [24] Mialon, G. et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023).
- Not checked yetcites [25] Brown, T. et al. Language Models are Few-Shot Learners. Curran Associates, Inc. (2020).
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
Probably not a substantial claim.
In the paper
“The model incorporates 175 billion parameters and was trained on various public and authorized documents, demonstrating specific Artificial General Intelligence (AGI) capabilities5.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Data processing, workflow, and prompt engineering
What the sources say
- Not checked yetcites [5] OpenAi, GPT-4 Technical Report. arXiv:2303.08774: https://arxiv.org/pdf/2303.08774.pdf (2023).
Strength:boosted· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“de-identified Osteosarcoma pathology notes were analyzed by Microsoft Azure OpenAI covered by the Business Associate Agreement in this study.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
Strength:plain· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passes
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: De-identified data can be analyzed without significant privacy concerns.
In the paper
“However, a significant obstacle to developing these traditional medical NLP algorithms is the limited existence of human-annotated datasets and the costs associated with new human annotation15.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Introduction
What the sources say
- Not checked yetcites [15] Wu, H. et al. A survey on clinical natural language processing in the United Kingdom from 2007 to 2022. npj Digit. Med. 5, 186 (2022). source
Load-bearingStrength:plain· not yet calibratedCausal:direct· not yet calibrated
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
It may state two separate claims as one.

33 of 33

The paper relies on this without support.
Few studies have examined the potential of LLMs in reading and interpreting clinical notes, turning unstructured texts into structured, analyzable data.
“To date, few studies have examined the potential of LLMs in reading and interpreting clinical notes, turning unstructured texts into structured, analyzable data.”
The paper relies on this without support.
Medical domain knowledge needs to be integrated into model designs to enhance performance.
“Medical domain knowledge needs to be integrated into model designs to enhance performance.”
The paper relies on this without support.
Despite meticulous ground-truth labeling, relatively small corpus sizes often result in models with poor generalizability or make evaluations of generalizability impossible.
“Despite meticulous ground-truth labeling, the relatively small corpus sizes often result in models with poor generalizability or make evaluations of generalizability impossible.”
The paper relies on this without support.
Conventional AI systems have suffered from a lack of general knowledge and commonsense reasoning for decades.
“For decades, conventional artificial intelligence (AI) systems (symbolic and neural networks) have suffered from a lack of general knowledge and commonsense reasoning.”
The paper relies on this without support.
LLMs, like GPT, offer a promising alternative, potentially using commonsense reasoning and broad general knowledge to facilitate language processing.
“LLMs, like GPT, offer a promising alternative, potentially using commonsense reasoning and broad general knowledge to facilitate language processing.”
The paper relies on this without support.
The lung cancer cases in the dataset are predominantly non-metastatic.
“As the lung cancer cases in our dataset are predominantly non-metastatic, the pathologic metastasis (pM) stage was not extracted.”
The paper relies on this without support.
ChatGPT models (even with the same version) evolve over time.
“Since ChatGPT models (even with the same version) evolve over time, it is important to evaluate the stability and reproducibility of ChatGPT.”
The paper relies on this without support.
A simple prompt without special design would cause ChatGPT to produce unstructured answers varying in format, terminology, and granularity across many pathology reports.
“However, this simple prompt would make ChatGPT produce unstructured answers varying in format, terminology, and granularity across the large number of pathology reports.”
The paper relies on this without support.
The main obstacle to developing traditional medical NLP algorithms is the limited availability of annotated data and the costs for new human annotations.
“The main obstacle to developing traditional medical NLP algorithms is the limited availability of annotated data and the costs for new human annotations.”
The paper relies on this without support.
Providing examples is an effective way for humans to learn, and it should have similar effects for ChatGPT.
“Providing examples is an effective way for humans to learn, and it should have similar effects for ChatGPT.”
The paper relies on this without support.
Tumor size plays a critical role in classifying the pT stage.
“Although tumor size is not a primary interest for diagnosis and clinical research, it plays a critical role in classifying the pT stage.”
The paper relies on this without support.
All LLMs have limited context windows, constraining the input length of a query.
“All LLMs have limited context windows, constraining the input length of a query.”
The paper relies on this without support.
The output of an LLM depends on both the model and the prompt.
“Since the output of an LLM depends on both the model and the prompt, mitigating hallucination can be achieved through improvements in GPT models and prompting strategies.”
The paper relies on this without support.
De-identified data can be analyzed without significant privacy concerns.
“Although it is a less concern to analyze public available de-identified data, like the lung cancer pathology notes used in this study, careful considerations are needed for secured healthcare data.”

14 of 14

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Integrative Analysis of Complex Cancer Genomics and Clinical Profiles Using the cBioPortal
Gao, J., Aksoy, B. A., Dogrusoz, U., Dresdner, G., Gross, B., Sumer, S. O., Sun, Y., Jacobsen, A., Sinha, R., Larsson, E., et al. (2013) · Science Signaling · doi:10.1126/scisignal.2004088 · Cited by 15,160
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 2×
“HarnessinghallucinationisanongoingchallengeinAIresearch,withvarious methodsbeingexplored 5,27 .”
A spiral model of software development and enhancement
BOEHM, B. W. (1986) · Computer · doi:10.1109/2.59 · Cited by 4,985
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“This approachhas become a dominant method for using LLMs in real-world problem-solving 24 – 26 .”
The IASLC Lung Cancer Staging Project: Proposals for the Revision of the TNM Stage Groupings in the Forthcoming (Seventh) Edition of the TNM Classification of Malignant Tumours
Goldstraw, P., Crowley, J., Chansky, K., Giroux, D. J., Groome, P. A., Rami-Porta, R., Postmus, P. E., Rusch, V., & Sobin, L. (2007) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e31812f3c1a · Cited by 3,655
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00455-y · Cited by 886
PaLM 2 Technical Report
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
ChatGPT: the future of discharge summaries?
Patel, S. B., & Lam, K. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00021-3 · Cited by 728
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557
Using ChatGPT to write patient clinic letters
Ali, S. R., Dobbs, T. D., Hutchings, H. A., & Whitaker, I. S. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00048-1 · Cited by 375
Electronic health records-driven phenotyping: challenges, recent advances, and perspectives
Pathak, J., Kho, A. N., & Denny, J. C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002428 · Cited by 274
ChatGPT: friend or foe?
Health, T. L. D. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00023-7 · Cited by 257 · cited 1×
“Since ChatGPT ’ s release in November 2022, it has spurred many potential innovative applications in healthcare 19 – 23 .”
A neural network multi-task learning approach to biomedical named entity recognition
Crichton, G., Pyysalo, S., Chiu, B., & Korhonen, A. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1776-8 · Cited by 229
Will ChatGPT transform healthcare?
(2023) · Nature Medicine · doi:10.1038/s41591-023-02289-5 · Cited by 214
ChatGPT and antimicrobial advice: the end of the consulting infection doctor?
Howard, A., Hope, W., & Gerada, A. (2023) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(23)00113-5 · Cited by 202 · cited 1×
“Since ChatGPT ’ s release in November 2022, it has spurred many potential innovative applications in healthcare 19 – 23 .”
Augmented Language Models: a Survey
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07842 · Cited by 160
A survey on clinical natural language processing in the United Kingdom from 2007 to 2022
Wu, H., Wang, M., Wu, J., Francis, F., Chang, Y.-H., Shavick, A., Dong, H., Poon, M. T. C., Fitzpatrick, N., Levine, A. P., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00730-6 · Cited by 101
Health Natural Language Processing: Methodology Development and Applications
Hao, T., Huang, Z., Liang, L., Weng, H., & Tang, B. (2021) · JMIR Medical Informatics · doi:10.2196/23898 · Cited by 48
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. J. F. (2023) · arXiv · doi:10.48550/arxiv.2303.08896 · Cited by 38
Document-Level Biomedical Relation Extraction Using Graph Convolutional Network and Multihead Attention: Algorithm Development and Validation
Wang, J., Chen, X., Zhang, Y., Zhang, Y., Wen, J., Lin, H., Yang, Z., & Wang, X. (2020) · JMIR Medical Informatics · doi:10.2196/17638 · Cited by 24
Osteosarcoma Explorer: A Data Commons With Clinical, Genomic, Protein, and Tissue Imaging Data for Osteosarcoma Research
Yang, D. M., Zhou, Q., Cline, L. F., Cheng, X. F., Luo, D., Lai, H., Li, Y., Jin, K. W., Yao, B., Leavey, P. J., et al. (2023) · JCO Clinical Cancer Informatics · doi:10.1200/cci.23.00104 · Cited by 4
Self-citation
GPT-4 Technical Report
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Self-citation

Flagged first, then by citations

Research works printed but not linked: 2

Vaswani, A. et al. Attention is all you need. Adv. Neural Info. Processing Syst. 30, (2017).
Amin, M. B. et al. AJCC cancer staging manual. 1024: Springer 2017.

Sorted by citations

Invasion and metastasis in cancer: molecular insights and therapeutic targets
Li, Y., Liu, F., Cai, Q., Deng, L., Ouyang, Q., Zhang, X. H.-F., & Zheng, J. (2025) · Signal Transduction and Targeted Therapy · doi:10.1038/s41392-025-02148-4 · Cited by 314
A Review of Large Language Models in Medical Education, Clinical Decision Support, and Healthcare Administration
Vrdoljak, J., Boban, Z., Vilović, M., Kumrić, M., & Božić, J. (2025) · Healthcare · doi:10.3390/healthcare13060603 · Cited by 202
“5’s potential for extracting structured data from clinical notes [36].”
Influential
Implementing large language models in healthcare while balancing control, collaboration, costs and security
Dennstädt, F., Hastings, J., Putora, P. M., Schmerder, M., & Cihoric, N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01476-7 · Cited by 144
Accuracy, Consistency, and Hallucination of Large Language Models When Analyzing Unstructured Clinical Notes in Electronic Medical Records
Shah, S. V. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.25953 · Cited by 80
Reporting guideline for the use of Generative Artificial intelligence tools in MEdical Research: the GAMER Statement
Luo, X., Tham, Y. C., Giuffrè, M., Ranisch, R., Daher, M., Lam, K., Eriksen, A. V., Hsu, C.-W., Ozaki, A., de Moraes, F. Y., et al. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2025-113825 · Cited by 110

See all 298

Authors: Huang, J., Yang, D. M., Rong, R., Nezafati, K., Treager, C., Chi, Z., Wang, S., Cheng, X., Guo, Y., Klesse, L. J., Xiao, G., Peterson, E. D., Zhan, X., & Xie, Y.
Affiliations: The University of Texas Southwestern Medical Center (US)
Corresponding author: Zhan, Xiaowei
Xie, Yang
ORCID: Huang, Jingwei
Yang, Donghan M.
Rong, Ruichen
Nezafati, Kuroush
Chi, Zhikai
Wang, Shidan
Cheng, Xian
Guo, Yujia
Klesse, Laura J.
Xiao, Guanghua
Peterson, Eric D.
Zhan, Xiaowei
Xie, Yang
Venue: npj Digital Medicine
Publisher: Springer Science and Business Media LLC
Published: 2024-05-01
Volume: 7
Issue: 1
Pages: 106-106
ISSN: 2398-6352 (ISSN-L)
DOI: 10.1038/s41746-024-01079-8
PMC: PMC11063058
OpenAlex: W4396553888
Semantic Scholar: a44b237b93664d32d09dae6c5de8d9b88e0f01a2
Cited by: 298
References: 23
Citations by year: 2026: 109 · 2025: 146 · 2024: 43
FWCI: 40.57
Type: journal article
Language: en
Open access: gold
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0 (tdm)
creativecommons.org/licenses/by/4.0 (vor)
Locations: npj Digital Medicine (publishedVersion, cc-by)
SSRN Electronic Journal (acceptedVersion)
PubMed Central (publishedVersion, cc-by)
PubMed (publishedVersion)
DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders: U.S. Department of Health & Human Services | NIH | National Institute of General Medical Sciences (R35GM136375, R01GM141519, R01GM140012)
U.S. Department of Health & Human Services | NIH | National Cancer Institute (P30CA142543, P50CA70907, U01CA249245)
U.S. Department of Health & Human Services | NIH | National Institute of Dental and Craniofacial Research (R01DE030656)
Cancer Prevention and Research Institute of Texas (RP230330)
Division of Intramural Research, National Institute of Allergy and Infectious Diseases (U01AI169298)
Topics: Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords: ChatGPT, large language models, clinical information extraction, structured data extraction, clinical notes, prompt engineering, pediatric osteosarcoma, pathology terminology, TNM staging, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A critical assessment of using ChatGPT for extracting structured data from clinical notes

Abstract

Overview

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks