Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review

May combine two claims.

Advantages of LLMs included positive data extraction performance and reduced manual workload.

“Advantages of LLMs included positive data extraction performance and reduced manual workload.”

May combine two claims.

Our review highlighted the generalized performance of LLM-enabled data extraction across various text types in oncology, including histological and pathological classification, imaging report classification, and data extraction from postoperative surgery reports.

“For instance, our review highlighted the generalized performance of LLM-enabled data extraction across various text types in oncology, including histological and pathological classification [936], imaging report classification [814], and data extraction from postoperative surgery reports [5].”

• cites [936]Not checked yet

  Not checked in the free run.

• cites [814]Not checked yet

  Not checked in the free run.

• cites [5] Seong, Deep learning approach to detection of colonoscopic information from unstructured reports, BMC Med Inform Decis Mak sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Author teams belonged to institutions in the United Kingdom (n=1, 4%).

“the United Kingdom (n=1, 4%)”

May combine two claims.

In general, LLMs exhibited positive performance metrics compared to baseline human or statistical NLP approaches or were deemed feasible and acceptable in cross-sectional studies.

“In general, LLMs exhibited positive performance metrics compared to baseline human or statistical NLP approaches or were deemed feasible and acceptable in cross-sectional studies.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Author teams of included studies belonged to institutions in the United States (n=11, 46%).

“The author teams of these studies belonged to institutions in the United States (n=11, 46%)”

May combine two claims.

The literature search yielded 817 papers, of which 24 met the inclusion criteria.

“The literature search yielded 817 papers, of which 24 papers met the inclusion criteria (Figure 1).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Author teams belonged to institutions in Germany (n=2, 8%).

“Germany (n=2, 8%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Author teams belonged to institutions in China (n=4, 17%).

“China (n=4, 17%)”

May combine two claims.

The most common domains for moderate risk of bias included bias due to confounding (n=21, 87.5%) and bias in the selection of the reported result (n=21, 87.5%).

“The most common domains for moderate risk of bias included bias due to confounding (n=21, 87.5%) and bias in the selection of the reported result (n=21, 87.5%).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Author teams belonged to institutions in Spain (n=2, 8%).

“Spain (n=2, 8%)”

May combine two claims.

Across the included studies in this review, LLMs offer several benefits for data extraction in clinical oncology, though further benchmarking against representative datasets and classical machine learning or statistical NLP approaches is required to determine their superior performance.

“Across the included studies in this review, we found that LLMs offer several benefits for data extraction in clinical oncology, though further benchmarking against representative datasets and classical machine learning or statistical NLP approaches is required to determine their superior performance.”

May combine two claims.

The analysis reveals an increasing trend in the use of fine-tuning and prompt-engineering techniques in studies on LLMs, with 16 (67%) studies incorporating fine-tuning and 5 (21%) using prompt engineering.

“Our analysis reveals an increasing trend in the use of fine-tuning and prompt-engineering techniques in studies on LLMs, with 16 (67%) studies incorporating fine-tuning and 5 (21%) using prompt engineering.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Author teams belonged to institutions in Singapore (n=1, 4%).

“and Singapore (n=1, 4%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Author teams belonged to institutions in Korea (n=3, 12%).

“Korea (n=3, 12%)”

Probably not a substantial claim: may not pass S3 (load-bearing).

ROBINS-I risk of bias assessment exhibited moderate inter-rater concordance based on an κ score of 0.43.

“ROBINS-I risk of bias assessment exhibited moderate inter-rater concordance based on an κ score of 0.43.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs such as BERT and GPT may have zero-shot capabilities, meaning they can perform tasks without prior task-specific training.

“Due to their extensive training dataset, LLMs such as BERT and GPT may have zero-shot capabilities, meaning they can perform tasks without prior task-specific training [9].”

• cites [9] Truhn, Extracting structured information from unstructured histopathology reports using generative pre-trained transformer 4 (GPT-4), J Pathol sourceNot checked yet

  Not checked in the free run.

May combine two claims.

LLMs applied to data extraction in oncology can serve as useful automated tools to reduce the administrative burden of reviewing patient health records and increase time for patient-facing care.

“LLMs applied to data extraction in oncology can serve as useful automated tools to reduce the administrative burden of reviewing patient health records and increase time for patient-facing care.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Deep learning models have demonstrated superior performance across diverse NLP tasks, including data extraction.

“has been particularly effective at modeling the hierarchical structure of natural language and demonstrated superior performance across diverse NLP tasks, including but not limited to data extraction [3].”

• cites [3] Zhou, CancerBERT: a cancer domain-specific language model for extracting breast cancer phenotypes from electronic health records, J Am Med Inform Assoc sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

These models are often limited by their need for large-scale computational resources and training data.

“However, these models are often limited by their need for large-scale computational resources and training data [78].”

• cites [78]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Deep learning-based transformer models may require less time and fewer resources for manual annotation compared to classical machine learning models.

“Deep learning–based transformer models may require less time and fewer resources needed to manually annotate training datasets compared to classical machine learning models”

• cites [56]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Transformer models have gained attention for generative language competency and strong performance in question answering, sentence completion, and sentence classification compared to other deep learning models.

“has gained worldwide attention for its generative language competency and strong performance in question answering, sentence completion, and sentence classification tasks compared to other deep learning models [4].”

• cites [4] Devlin J Chang MW Lee K Toutanova K . BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. Preprint posted online onMay24, 2019. URL: http://arxiv.org/abs/1810.04805Not checked yet

  Not checked in the free run.

May combine two claims.

LLMs have the potential to be customized for specialized medical fields while also having broad out-of-the-box utility in general oncology.

“This underscores the potential of LLMs to be customized for specialized medical fields while also highlighting their broad “out-of-the-box” utility in general oncology.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The focus on breast and lung cancers indicates a targeted approach in which models are fine-tuned to address unique data extraction challenges such as cancer type-specific nomenclature and lexicons.

“The focus on these specific cancers indicates a targeted approach, where models are fine-tuned to address unique data extraction challenges, such as cancer type-specific nomenclature and lexicons.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cohen κ score was used to assess inter-rater concordance.

“Cohen κ score was used to assess inter-rater concordance.”

May combine two claims.

Multimodal LLMs capable of processing both image- and text-based inputs have recently emerged and serve as a new frontier for clinical decision support.

“However, we also highlight the recent emergence of multimodal LLMs capable of processing both image- and text-based inputs, serving as a new frontier for clinical decision support [40].”

• cites [40] Chen, Performance of multimodal artificial intelligence chatbots evaluated on clinical oncology cases, JAMA Netw Open sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Abstract and full-text screening were conducted in duplicate.

“We conducted abstract screening followed by full-text screening of articles in duplicate (KA and SA)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Screening conflicts were resolved through consensus discussion with a third reviewer.

“Screening conflicts were resolved through consensus discussion with a third reviewer (DC).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Risk of bias was assessed using ROBINS-I (Version 2) in duplicate.

“The risk of bias was assessed using ROBINS-I (Version 2) in duplicate (KA and SA), with conflicts resolved through consensus discussion with a third reviewer (DC).”

Classified as probably not a substantial claim.

The general human-like natural language competencies of LLMs enable out-of-the-box solutions to automatically adapt to and extract relevant information from varied data sources, including structured and unstructured EHR data.

“Whether dealing with structured data formats or the myriad forms of unstructured data present in EHRs, such as physician’s notes and diagnostic reports, the general human-like natural language competencies of LLMs enable these “out-of-the-box” solutions to automatically adapt to and extract relevant information from varied data sources.”

Classified as probably not a substantial claim. May combine two claims.

The broad natural language competency of LLMs encourages the design of specialized LLM applications for data extraction from unstructured clinical text, reducing the oncologists' time and effort spent in manually reviewing patient EHRs.

“The broad natural language competency of LLMs encourages the design of specialized LLM applications for data extraction from unstructured clinical text, reducing the oncologists’ time and effort spent in manually reviewing patient EHRs to extract key information to inform their clinical decision-making.”

May combine two claims.

This shift not only improves clinical efficiency and cost-effectiveness but also reduces the serious risks of burnout among clinical staff by mitigating some of the repetitive administrative tasks associated with data handling.

“This shift not only improves clinical efficiency and cost-effectiveness but also reduces the serious risks of burnout among clinical staff by mitigating some of the repetitive administrative tasks associated with data handling [1138].”

• cites [1138]Not checked yet

  Not checked in the free run.

May combine two claims.

The increasing use of fine-tuning and prompt engineering suggests a maturation in the application of LLMs in clinical settings, with research transitioning from baseline models for simple data extraction to optimization of existing models using novel adaptations and prompting methodologies tailored to medical data extraction.

“This progression suggests a maturation in the application of LLMs in clinical settings, where research has transitioned from developing baseline models for simple data extraction to the optimization of existing models using novel model adaptations and prompting methodologies tailored to the intricacies of medical data extraction.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Given the longitudinal nature of cancer care, the vast amount of clinical text associated with cancer patient EHRs necessitates the development of automated methods for data extraction from these clinical records into structured data.

“Given the longitudinal nature of cancer care, the vast amount of clinical text associated with cancer patient EHRs necessitates the development of automated methods for data extraction from these clinical records into structured data”

May combine two claims.

The predominance of pan-cancer clinical domains suggests that researchers leverage the versatility of LLMs to address broad oncological challenges across multiple cancer types, likely due to the generalizable nature of these models for various cancer data.

“This suggests that researchers leverage the versatility of LLMs to address broad oncological challenges across multiple cancer types, likely due to the generalizable nature of these models for various cancer data [42].”

• cites [42] Truhn, Large language models and multimodal foundation models for precision oncology, NPJ Precis Oncol sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

This scoping review followed the PRISMA-ScR reporting guideline.

“This scoping review followed the PRISMA-ScR reporting guideline.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Key study attributes were extracted from included full-text papers in duplicate.

“We extracted key study attributes from the included full-text papers in duplicate (KA and SA)”

Probably not a substantial claim: may not pass S3 (load-bearing).

Data extraction conflicts were resolved through consensus discussion with a third reviewer.

“Data extraction conflicts were resolved through consensus discussion with a third reviewer (DC).”

Probably not a substantial claim: may not pass S3 (load-bearing).

A minor disadvantage is the token limit of many LLMs, including both BERT and ChatGPT.

“Another minor disadvantage is the token limit of many LLMs, including both BERT and ChatGPT [204244].”

• cites 204244Not checked yet

  Not checked in the free run.

May combine two claims.

The potential of LLMs to enhance data extraction processes and improve clinical decision-making is significant yet tempered by current technological and methodological limitations.

“The potential to enhance data extraction processes and improve clinical decision-making is significant yet tempered by the current technological and methodological limitations.”

May combine two claims.

The token limit may complicate the extraction process, requiring models to be adapted to longer texts and resulting in reduced performance of these models.

“This limitation may complicate the extraction process, requiring models to be adapted to longer texts and resulting in reduced performance of these models [51].”

• cites [51] Liu, Lost in the middle: how language models use long contexts, Trans Assoc Comput Linguist sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The majority of included studies originated from high-resource settings, primarily the United States, which may limit the applicability of results to lower-resource or structurally different health care systems.

“Third, the majority of included studies originated from high-resource settings, primarily the United States, which may limit the applicability of results to lower-resource or structurally different health care systems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The risk of publication bias was not formally evaluated in this scoping review.

“while the risk of publication bias was not formally evaluated in our review”

Probably not a substantial claim: may not pass S3 (load-bearing).

Heterogeneity in study designs, datasets, and methodologies across included articles may affect the generalizability of findings in external contexts not evaluated under the same conditions as the original studies.

“Second, the heterogeneity in study designs, datasets, and methodologies across included articles may affect the generalizability of findings in external contexts not evaluated in the same conditions as the original studies.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

More recent journals that publish artificial intelligence research may not be indexed in the search databases yet, limiting the completeness of the search results in this scoping review.

“Fifth, more recent journals that publish artificial intelligence research may not be indexed in the search databases yet, limiting the completeness of the search results in this scoping review.”

May combine two claims.

This scoping review searched only one literature database, which may have resulted in omission of relevant studies from other sources and limited the comprehensiveness of the findings.

“Sixth, this scoping review searched only one literature database, which may have resulted in the omission of relevant studies from other sources and limited the comprehensiveness of the findings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The rapid evolution of LLM technologies means that newer advancements may not have been fully represented in the reviewed studies due to publication delays, leading to omission of recent models.

“First, the rapid evolution of LLM technologies means that newer advancements may not have been fully represented in the reviewed studies due to the delays in publication cycles, leading to the omission of recent models.”