The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
A critical assessment of using ChatGPT for extracting structured data from clinical notes
Open accessAbstract
Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training. This study aims to evaluate ChatGPT’s capacity to extract information from free-text medical notes efficiently and comprehensively. We developed a large language model (LLM)-based workflow, utilizing systems engineering methodology and spiral “prompt engineering” process, leveraging OpenAI’s API for batch querying ChatGPT. We evaluated the effectiveness of this method using a dataset of more than 1000 lung cancer pathology reports and a dataset of 191 pediatric osteosarcoma pathology reports, comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data. ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset, outperforming the performance of two traditional NLP methods. The performance is influenced by the design of the instructive prompt. Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules. Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time. In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively. Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training. The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This study developed an LLM-based workflow using ChatGPT with spiral prompt engineering to extract structured data from clinical pathology notes. ChatGPT-3.5 achieved an average accuracy of 89% on lung cancer staging/histology extraction and 98.6%/100% accuracy on pediatric osteosarcoma grade/margin classification, while showing limitations in specialized pathology terminology and TNM rule interpretation.
Key findings
- On 774 valid TCGA lung cancer pathology reports, ChatGPT-3.5 (gpt-3.5-turbo-16k) achieved average accuracy 0.89, with pT 0.87, pN 0.91, overall tumor stage 0.76, and histological diagnosis 0.99; average coverage was 0.95.
- ChatGPT outperformed two traditional NLP methods on the same testing data: average accuracy 0.89 for ChatGPT vs. 0.51 for WordPiece tokenizer and 0.76 for a Bio_ClinicalBERT-based NER classifier.
- In pediatric osteosarcoma data, ChatGPT accurately classified histology grade with 98.6% accuracy and margin status with 100% accuracy.
- Prompt engineering improved training-data classification accuracy from 0.854 (prompt v1) to 0.865 (v2), 0.90 (v3), and 0.936 (v4); reproducibility testing showed 0.913 equivalence between April and May 2023 outputs and mean absolute certainty-degree error of 0.051.
Methods
- The study used an LLM-based workflow with OpenAI API batch querying, systems engineering methodology, and a spiral “prompt engineering” process; scanned PDFs were converted to text with R package pdftools, and ChatGPT was instructed to use AJCC 7th edition staging rules and output structured JSON.
- Lung cancer data included 1026 pathology reports from CDSA and TCGA; 78 valid CDSA reports were used for prompt development, and 774 valid TCGA reports were used for testing after excluding overlaps and invalid reports. Pediatric osteosarcoma data included 191 cases, with 148 having histology grade and 81 having margin status.
- Performance was evaluated by comparing ChatGPT outputs with expert-curated structured data using accuracy, F1, Kappa, recall, precision, coverage, and confusion matrices, plus case-based error analysis, reproducibility testing, and comparison with keyword search and NER methods.
Limitations
- The paper states that ChatGPT showed occasional “hallucination,” especially on blank or near-blank reports, and that most misclassifications were due to lack of highly specialized pathology terminology or erroneous interpretation of TNM staging rules.
- The paper notes that ChatGPT models evolve over time, that GPT-4-Turbo performed better but was much more expensive, and that privacy concerns require careful handling for sensitive healthcare data.
- Evident from the text: prompt development used only 78 training reports, overall tumor stage accuracy was lower than other attributes (0.76), and the evaluation is limited to lung cancer pathology reports plus one pediatric osteosarcoma dataset.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
39 Own result · 13 Own method · 9 Not checked yet · 9 Own interpretation · 2 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT-3.5 demonstrated the ability to extract pathological classifications with an overall accuracy of 89%, in lung cancer dataset”
- Also stated in: § Overall Performance · plain
“The average accuracy of all attributes is 0.89.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
boostedn = 774- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Overall Performance · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“outperforming the performance of two traditional NLP methods.”
- Also stated in: § Comparison with other NLP methods · plain
“The results (Supplementary Table S1) show that the ChatGPT outperforms WordPiece tokenizer and NER Classifier.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Comparison with other NLP methods · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The performance is influenced by the design of the instructive prompt.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In pediatric osteosarcoma dataset, ChatGPT-3.5 accurately classified both grades and margin status with accuracy of 98.6% and 100% respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our case analysis shows that most misclassifications were due to the lack of highly specialized pathology terminology, and erroneous interpretation of TNM staging rules.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Reproducibility shows the relatively stable performance of ChatGPT-3.5 over time.”
- Also stated in: § Reproducibility evaluation · hedged
“Considering the evolutionary nature of ChatGPT models, we regard an output difference to a certain extent as reasonable and the overall ChatGPT 3.5 model as stable.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Reproducibility evaluation · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the remaining 726 cases, 8.5% of pN was misclassified.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plainn = 726- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our case analysis shows that most misclassifications were due to a lack of knowledge of detailed pathology terminology or very specialized information in the current versions of ChatGPT models”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the remaining 726 cases, 23.6% of the overall stage was misclassified.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plainn = 726- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy of primary tumor features (pT), regional lymph node involvement (pN), overall tumor stage, and histological diagnosis are 0.87, 0.91, 0.76, and 0.99, respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Overall Performance
plainn = 774- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“processing over 900 pathology reports incurring a minimal monetary cost (less than $10 using GPT 3.5 Turbo and less than $30 using GPT-4 Turbo).”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, most of the misclassification cases were caused by ChatGPT interpreting the number of positive nodes rather than the locations of the positive nodes.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plaindirectn = 32- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy of primary tumor features (pT), regional lymph node involvement (pN), overall tumor stage, and histological diagnosis are 0.87, 0.91, 0.76, and 0.99, respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Overall Performance
plainn = 774- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mean absolute error between certainty degrees in the two experiments is 0.051.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Reproducibility evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments showed that, with the temperature set at 0 for blank reports, ChatGPT converged to a consistent, hallucinated response.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Analyzing irregularities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy of primary tumor features (pT), regional lymph node involvement (pN), overall tumor stage, and histological diagnosis are 0.87, 0.91, 0.76, and 0.99, respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Overall Performance
plainn = 774- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the lung cancer cases in our dataset are predominantly non-metastatic, the pathologic metastasis (pM) stage was not extracted.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Data and endpoints
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the remaining 753 cases, 12.6% of pT was misclassified.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plainn = 753- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, for nearly blank reports with a few random characters and poorly scanned reports, ChatGPT consistently converged to the same response with increased variance as noise increased.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Analyzing irregularities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the remaining 745 cases, 6 ( < 1%) of histology types were misclassified.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plainn = 745- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After adding these two instructions (prompt v2), the performance of the classification in the training data increased to 0.865.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Prompt engineering process and results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average accuracy for ChatGPT, WordPiece tokenizer, and NER Classifier are 0.89, 0.51, and 0.76, respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Comparison with other NLP methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, when a report was blank, the prompt contained only the instruction part. ChatGPT failed to recognize this situation in most cases and inappropriately generated a fabricated case.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Analyzing irregularities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the 34 cases affected by incorrect rules, ChatGPT mistakenly inferred tumor stage as stage III for 26 cases where pT is T3 and pN is N0, respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plaincorrelationaln = 34- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The accuracy of primary tumor features (pT), regional lymph node involvement (pN), overall tumor stage, and histological diagnosis are 0.87, 0.91, 0.76, and 0.99, respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Overall Performance
plainn = 774- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most of these misclassification cases were N1 misclassified as N2 (32 cases).”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plainn = 32- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average accuracy for ChatGPT, WordPiece tokenizer, and NER Classifier are 0.89, 0.51, and 0.76, respectively.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Comparison with other NLP methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After this modification (prompt v4), the performance of the classification in the training data increased to 0.936.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Prompt engineering process and results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Providing examples is an effective way for humans to learn, and it should have similar effects for ChatGPT.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we decided to use prompt v4 as the final model and apply it to the testing data and get the final classification accuracy of 0.89 in the testing data.”
- Also stated in: § Discussion · plain
“achieved an overall average accuracy of 89% in extracting and estimating lung cancer staging information and histology subtypes compared to pathologist-curated data.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Prompt engineering process and results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The classification accuracy of this prompt with multiple choice output format (prompt v1) in our training data could achieve 0.854.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Prompt engineering process and results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After this modification, the performance of the classification in the training data increased to 0.90.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Prompt engineering process and results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Tumor size plays a critical role in classifying the pT stage.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The rate of equivalence between ChatGPT estimations in April and May on key attributes of interest (pT, pN, tumor stage, and histological diagnosis) is 0.913.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Reproducibility evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Existing natural language processing (NLP) methods to convert free-text clinical notes into structured data often require problem-specific annotations and model training.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
What the sources say
- Not checked yetcites [15] Wu, H. et al. A survey on clinical natural language processing in the United Kingdom from 2007 to 2022. npj Digit. Med. 5, 186 (2022). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“this study demonstrates our effective methodology in “prompt engineering”.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study shows the feasibility of using ChatGPT to extract data from free text without extensive task-specific human annotation and model training.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The main obstacle to developing traditional medical NLP algorithms is the limited availability of annotated data and the costs for new human annotations.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study shows the feasibility of using ChatGPT to process large volumes of clinical notes for structured information extraction without requiring extensive task-specific human annotation and model training.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Considering the evolutionary nature of ChatGPT models, we regard an output difference to a certain extent as reasonable and the overall ChatGPT 3.5 model as stable.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Reproducibility evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“the most significant being the occasional “hallucination” phenomenon27,28, where the generated content is not faithful to the provided source content.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 21
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Looking into the 56 cases where ChatGPT misclassified stage II as stage III, 22 cases were due to error propagation, and 34 were due to incorrect rules.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plaindirectn = 56- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among these misclassification cases, the majority were T1 misclassified as T2 (67 out of 753 or 8.9%) or T3 misclassified as T2 (12 out of 753, or 1.6%).”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plainn = 753- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT accurately classified both grades (with a 98.6% accuracy rate) and margin status (100% accuracy), as shown in Supplementary Fig. 3.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Analyzing osteosarcoma data
plainn = 148- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Last, among the 15 missing report forms in a small dataset, ChatGPT responded “unknown” as expected in only 5 cases, with the remaining 10 still converging to the hallucinated response.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Analyzing irregularities
plainn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In most cases, ChatGPT extracted the correct tumor size information but used an incorrect rule to distinguish pT categories.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the responses to blank or near-blank reports reflect this issue, though these instances can be detected and corrected due to convergence towards an “attractor”.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the mistakes that ChatGPT made for histology diagnosis, ChatGPT misclassified 3 of them as “other” type and 3 cases of actual “other” type (neither adenocarcinomas nor squamous cell carcinomas) as 2 adenocarcinomas and 1 squamous cell carcinoma.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that GPT-4-turbo performs better in almost every aspect; overall, the GPT-4-turbo model increases performance by over 5%.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > ChatGPT-4 performance
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The institutional review board at the University of Texas Southwestern Medical Center has approved this study where patient consent was waived for using retrospective, de-identified electronic health record data.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Data processing, workflow, and prompt engineering
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Following the AJCC guideline, individual pathologic T, N, and M stage components can be summarized into an overall pathologic staging score of Stage I, II, III, or IV.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Data and endpoints
What the sources say
- Not checked yetcites [16] Amin, M. B. et al. AJCC cancer staging manual. 1024: Springer 2017.
- Not checked yetcites [17] Goldstraw, P. et al. The IASLC Lung Cancer Staging Project: Proposals for the Revision of the TNM Stage Groupings in the Forthcoming (Seventh) Edition of the TNM Classification of Malignant Tumours. J. Thorac. Oncol. 2, 706–714 (2007). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The CDSA dataset was utilized as the “training” data for prompt development, while the TCGA dataset, after removing the overlapping cases with CDSA, served as the test data for evaluating the ChatGPT model performance.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Data and endpoints
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The invalid reports were preserved to evaluate ChatGPT’s handling of irregular inputs separately, and were not included in the testing data for accuracy performance assessment.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Data and endpoints
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We used OpenAI’s “tiktoken” Python library to estimate the token count to ensure compliance.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Data processing, workflow, and prompt engineering
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: All LLMs have limited context windows, constraining the input length of a query.
None of the sources cited for this claim have been checked yet.
In the paper
“Machine learning methods11–14, particularly deep learning, typically employ neural networks and the first generation of transformer-based large language models (e.g., BERT).”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Introduction
What the sources say
- Not checked yetcites [11] Crichton, G. et al. A neural network multi-task learning approach to biomedical named entity recognition. BMC Bioinforma. 18, 368 (2017). source
- Not checked yetcites [12] Wang, J. et al. Document-Level Biomedical Relation Extraction Using Graph Convolutional Network and Multihead Attention: Algorithm Development and Validation. JMIR Med Inf. 8, e17638 (2020). source
- Not checked yetcites [13] Liu, Y. et al. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
- Not checked yetcites [14] Rasmy, L. et al. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digit. Med. 4, 86 (2021). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The approach employed in this study effectively leverages the OpenAI API for batch queries of ChatGPT services across a large set of tasks with similar input data structures, including but not limited to pathology reports and EHR.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To mimic a real-world setting, we processed all reports regardless of data quality to collect model responses.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Model evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For performance evaluation, we only used valid reports providing meaningful text and excluded the reports with near-empty content, poor scanning quality, and missing report forms, which were reported as irregular cases.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Model evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The training dataset underwent meticulous annotation by experienced medical professionals, adhering to the AJCC7 standards.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Comparison with other NLP methods
plainn = 78- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Augmented Language Models (ALMs)24, which incorporate reasoning and external tools for interaction with the environment, are emerging.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
What the sources say
- Not checked yetcites [24] Mialon, G. et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It presents a general framework for using ChatGPT’s API in batch queries to process large volumes of pathology reports for structured information extraction and estimation.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Traditionally, the automated extraction of structured data elements from medical notes has relied on medical natural language processing (NLP) using rule-based or machine-learning approaches or a combination of both9,10.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Introduction
What the sources say
- Not checked yetcites [9] Hao, T. et al. Health Natural Language Processing: Methodology Development and Applications. JMIR Med Inf. 9, e23898 (2021). source
- Not checked yetcites [10] Pathak, J., Kho, A. N. & Denny, J. C. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J. Am. Med. Inform. Assoc. 20, e206–e211 (2013). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“comparing the ChatGPT-3.5 (gpt-3.5-turbo-16k) outputs with expert-curated structured data.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Research shows that in-context learning (most influentially, few-shot prompting) can complement LLMs with task-specific knowledge to perform downstream tasks effectively24,25.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
What the sources say
- Not checked yetcites [24] Mialon, G. et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023).
- Not checked yetcites [25] Brown, T. et al. Language Models are Few-Shot Learners. Curran Associates, Inc. (2020).
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Some staging records in the TCGA structured data needed to be updated; our physicians curated and updated those records.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Model evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We evaluated the performance of ChatGPT by comparing its output with expert-curated data elements provided in the TCGA structured data using the testing data set.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Model evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our study has demonstrated the advantages of adopting ChatGPT over traditional methods in terms of cost-effectiveness and efficiency.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both cases reveal that ChatGPT still has limitations in the specific domain knowledge in lung cancer pathology and the capability of correcting understanding its terminology.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Results > Error analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The model incorporates 175 billion parameters and was trained on various public and authorized documents, demonstrating specific Artificial General Intelligence (AGI) capabilities5.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Data processing, workflow, and prompt engineering
What the sources say
- Not checked yetcites [5] OpenAi, GPT-4 Technical Report. arXiv:2303.08774: https://arxiv.org/pdf/2303.08774.pdf (2023).
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In this process, clinicians were actively formulating questions and evaluating the results.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Data processing, workflow, and prompt engineering
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“de-identified Osteosarcoma pathology notes were analyzed by Microsoft Azure OpenAI covered by the Business Associate Agreement in this study.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: De-identified data can be analyzed without significant privacy concerns.
In the paper
“However, a significant obstacle to developing these traditional medical NLP algorithms is the limited existence of human-annotated datasets and the costs associated with new human annotation15.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Introduction
What the sources say
- Not checked yetcites [15] Wu, H. et al. A survey on clinical natural language processing in the United Kingdom from 2007 to 2022. npj Digit. Med. 5, 186 (2022). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results underscore the potential role of LLMs in transforming unstructured healthcare data into structured formats, thereby supporting research and aiding clinical decision-making.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Missing data were excluded from the accuracy evaluation, and the coverage rate was reported for predicted values as ‘unknown’ or empty output.”
§ A critical assessment of using ChatGPT for extracting structured data from clinical notes > Methods > Model evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
Few studies have examined the potential of LLMs in reading and interpreting clinical notes, turning unstructured texts into structured, analyzable data.
“To date, few studies have examined the potential of LLMs in reading and interpreting clinical notes, turning unstructured texts into structured, analyzable data.”
The paper relies on this without support.
Medical domain knowledge needs to be integrated into model designs to enhance performance.
“Medical domain knowledge needs to be integrated into model designs to enhance performance.”
The paper relies on this without support.
Despite meticulous ground-truth labeling, relatively small corpus sizes often result in models with poor generalizability or make evaluations of generalizability impossible.
“Despite meticulous ground-truth labeling, the relatively small corpus sizes often result in models with poor generalizability or make evaluations of generalizability impossible.”
The paper relies on this without support.
Conventional AI systems have suffered from a lack of general knowledge and commonsense reasoning for decades.
“For decades, conventional artificial intelligence (AI) systems (symbolic and neural networks) have suffered from a lack of general knowledge and commonsense reasoning.”
The paper relies on this without support.
LLMs, like GPT, offer a promising alternative, potentially using commonsense reasoning and broad general knowledge to facilitate language processing.
“LLMs, like GPT, offer a promising alternative, potentially using commonsense reasoning and broad general knowledge to facilitate language processing.”
The paper relies on this without support.
The lung cancer cases in the dataset are predominantly non-metastatic.
“As the lung cancer cases in our dataset are predominantly non-metastatic, the pathologic metastasis (pM) stage was not extracted.”
The paper relies on this without support.
ChatGPT models (even with the same version) evolve over time.
“Since ChatGPT models (even with the same version) evolve over time, it is important to evaluate the stability and reproducibility of ChatGPT.”
The paper relies on this without support.
A simple prompt without special design would cause ChatGPT to produce unstructured answers varying in format, terminology, and granularity across many pathology reports.
“However, this simple prompt would make ChatGPT produce unstructured answers varying in format, terminology, and granularity across the large number of pathology reports.”
The paper relies on this without support.
The main obstacle to developing traditional medical NLP algorithms is the limited availability of annotated data and the costs for new human annotations.
“The main obstacle to developing traditional medical NLP algorithms is the limited availability of annotated data and the costs for new human annotations.”
The paper relies on this without support.
Providing examples is an effective way for humans to learn, and it should have similar effects for ChatGPT.
“Providing examples is an effective way for humans to learn, and it should have similar effects for ChatGPT.”
The paper relies on this without support.
Tumor size plays a critical role in classifying the pT stage.
“Although tumor size is not a primary interest for diagnosis and clinical research, it plays a critical role in classifying the pT stage.”
The paper relies on this without support.
All LLMs have limited context windows, constraining the input length of a query.
“All LLMs have limited context windows, constraining the input length of a query.”
The paper relies on this without support.
The output of an LLM depends on both the model and the prompt.
“Since the output of an LLM depends on both the model and the prompt, mitigating hallucination can be achieved through improvements in GPT models and prompting strategies.”
The paper relies on this without support.
De-identified data can be analyzed without significant privacy concerns.
“Although it is a less concern to analyze public available de-identified data, like the lung cancer pathology notes used in this study, careful considerations are needed for secured healthcare data.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These LLMs harbor the potential to balance accuracy and efficiency when processing large-scale, complex, unstructured text datasets found in EHRs [19].”
From Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review (2025) · cites this paper as [19]
In the citing paper
“Huang et al [19] demonstrated that providing LLMs with example outputs for few-shot learning and chain-of-thought reasoning methods for prompting yielded higher classification performance compared to baseline zero-shot applications of LLMs for data extraction.”
From Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review (2025) · cites this paper as [19]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Integrative Analysis of Complex Cancer Genomics and Clinical Profiles Using the cBioPortal
Gao, J., Aksoy, B. A., Dogrusoz, U., Dresdner, G., Gross, B., Sumer, S. O., Sun, Y., Jacobsen, A., Sinha, R., Larsson, E., et al. (2013) · Science Signaling · doi:10.1126/scisignal.2004088 · Cited by 15,160
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 2×
“HarnessinghallucinationisanongoingchallengeinAIresearch,withvarious methodsbeingexplored 5,27 .”
A spiral model of software development and enhancement
BOEHM, B. W. (1986) · Computer · doi:10.1109/2.59 · Cited by 4,985
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“This approachhas become a dominant method for using LLMs in real-world problem-solving 24 – 26 .”
Goldstraw, P., Crowley, J., Chansky, K., Giroux, D. J., Groome, P. A., Rami-Porta, R., Postmus, P. E., Rusch, V., & Sobin, L. (2007) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e31812f3c1a · Cited by 3,655
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
Research works printed but not linked: 2
- Vaswani, A. et al. Attention is all you need. Adv. Neural Info. Processing Syst. 30, (2017).
- Amin, M. B. et al. AJCC cancer staging manual. 1024: Springer 2017.
Cited by
Sorted by citations
Invasion and metastasis in cancer: molecular insights and therapeutic targets
Li, Y., Liu, F., Cai, Q., Deng, L., Ouyang, Q., Zhang, X. H.-F., & Zheng, J. (2025) · Signal Transduction and Targeted Therapy · doi:10.1038/s41392-025-02148-4 · Cited by 314
Vrdoljak, J., Boban, Z., Vilović, M., Kumrić, M., & Božić, J. (2025) · Healthcare · doi:10.3390/healthcare13060603 · Cited by 202
“5’s potential for extracting structured data from clinical notes [36].”
InfluentialDennstädt, F., Hastings, J., Putora, P. M., Schmerder, M., & Cihoric, N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01476-7 · Cited by 144
Shah, S. V. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.25953 · Cited by 80
Luo, X., Tham, Y. C., Giuffrè, M., Ranisch, R., Daher, M., Lam, K., Eriksen, A. V., Hsu, C.-W., Ozaki, A., de Moraes, F. Y., et al. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2025-113825 · Cited by 110
Metadata
- Authors
- Huang, J., Yang, D. M., Rong, R., Nezafati, K., Treager, C., Chi, Z., Wang, S., Cheng, X., Guo, Y., Klesse, L. J., Xiao, G., Peterson, E. D., Zhan, X., & Xie, Y.
- Affiliations
- The University of Texas Southwestern Medical Center (US)
- Corresponding author
- Zhan, Xiaowei
- Xie, Yang
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2024-05-01
- Volume
- 7
- Issue
- 1
- Pages
- 106-106
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11063058
- OpenAlex
- W4396553888
- Semantic Scholar
- a44b237b93664d32d09dae6c5de8d9b88e0f01a2
- Cited by
- 298
- References
- 23
- Citations by year
- 2026: 109 · 2025: 146 · 2024: 43
- FWCI
- 40.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Health & Human Services | NIH | National Institute of General Medical Sciences (R35GM136375, R01GM141519, R01GM140012)
- U.S. Department of Health & Human Services | NIH | National Cancer Institute (P30CA142543, P50CA70907, U01CA249245)
- U.S. Department of Health & Human Services | NIH | National Institute of Dental and Craniofacial Research (R01DE030656)
- Cancer Prevention and Research Institute of Texas (RP230330)
- Division of Intramural Research, National Institute of Allergy and Infectious Diseases (U01AI169298)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- ChatGPT, large language models, clinical information extraction, structured data extraction, clinical notes, prompt engineering, pediatric osteosarcoma, pathology terminology, TNM staging, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
