Development and validation of a human-supervised AI-augmented living oncology evidence platform: a breast cancer pilot study
Open accessAbstract
Background: Oncologists face increasing difficulty staying current with rapidly evolving clinical data, guidelines, and regulatory updates. Building and maintaining an annotated clinical trial evidence library is time- and labor-intensive. To address this challenge, we developed and validated a living oncology evidence platform (Living-OEP) for breast cancer (BC), powered by an agentic artificial intelligence (AI) system that supports daily, human-conducted, AI-augmented systematic literature review (SLR). Materials and methods: The agentic AI system, incorporating GPT-4.1 and o3 (OpenAI) and Claude [Anthropic, PBC, San Francisco, CA] Sonnet-4 (Anthropic), was designed to emulate expert-led, Cochrane-compliant SLR workflows. Guided by a human-developed annotation manual, the system decomposes tasks, self-debugs, and validates outputs. Training data included 29 236 clinical trial abstracts across BC, lung, and prostate cancer, each annotated with four review and 32 extraction variables. Structured data were integrated with guideline-based treatment pathways, forming a real-time, evidence-linked OEP. Accuracy was benchmarked against 1997 human annotations. Living-OEP evidence quality was compared with AI chatbots (ChatGPT [OpenAI, San Francisco, CA], Perplexity [Perplexity AI, Inc., San Francisco, CA], Consensus [Consensus, Boston, MA], and OpenEvidence [OpenEvidence Inc., Miami, FL]) using six criteria across eight BC treatment scenarios. Results: The agentic AI review accuracy ranged from 95.1% to 97.2%. Extraction accuracy ranged from 51.1% to 99.4%, with three variables still undergoing iterative refinement. Compared with other AI tools, the Living-OEP provided more comprehensive and accurate evidence and linked all data to original publications and Food and Drug Administration labels. Conclusions: A human-conducted, AI-augmented living SLR integrated with guidelines and regulatory data can provide real-time evidence support. Future studies are required to evaluate its impact on physician workflows, clinical decision-making, and implementation in oncology practice.
AI snapshot
Written from the paper’s full text.
TL;DR
The authors developed and validated a human-supervised, agentic AI-augmented living oncology evidence platform (Living-OEP) for breast cancer, using daily Cochrane-compliant systematic literature review workflows. In this pilot, the agentic AI achieved high review accuracy and variable extraction accuracy, and the Living-OEP outperformed four comparator chatbots on descriptive evidence-quality criteria.
Key findings
- Review accuracy ranged from 95.1% to 97.2% in the abstract; detailed results reported population 99.82%, intervention/comparator 98.75%, outcomes 93.73%, and study design 98.03% after iterative refinement.
- Extraction accuracy ranged from 51.1% to 99.4%; examples included randomization 100.0%, phase 100.0%, clinical trial name 95.2%, clinical trial registration 100.0%, follow-up time 94.4%, sample size 90.4%, histology 99.4%, biomarker 87.8%, treatment path 61.8%, interventions 51.1%, and observation time 34.0%. Three variables—analysis type, subgroup names, and subgroup data—were not yet incorporated into the AI training pipeline and were annotated manually.
- The agentic AI annotated 1,997 publications in 7.25 h in a single continuous session, versus an estimated 174.7 h for human scientists, representing a 95.9% reduction in annotation time.
- The Living-OEP consistently demonstrated all six evaluation criteria in Table 2, whereas comparator chatbots mostly showed limited, rare, minimal, or no demonstration; 23% of evidence included in the Living-OEP derived from recently published, potentially practice-changing pivotal trials.
Methods
- Design: developmental and descriptive validation of the REAL-SLR methodology, combining an agentic AI system (GPT-4.1, o3, and Claude Sonnet-4) emulating Cochrane-compliant SLR workflows with a human-led data management system for quality control, evidence interpretation, study grouping, treatment-path assignment, and guideline/regulatory integration.
- Data and sample: training used 29,236 clinical trial abstracts across breast, lung, and prostate cancer (9,887 full manuscripts and 19,349 conference abstracts; 9,800 breast cancer abstracts), each annotated with four review variables and 32 extraction variables; validation benchmarked against 1,997 human-annotated breast cancer publications independent of the training dataset. All data were publicly accessible clinical trial data, with no confidential individual patient-level data.
- Comparison: eight clinically realistic breast cancer scenarios were evaluated against ChatGPT (GPT-4o), Perplexity, Consensus, and OpenEvidence using six criteria; responses were scored by a single trained researcher with a binary rubric, not blinded, and the specific comparison was determined post hoc as a descriptive validation.
Limitations
- Stated: the search strategy was limited to PubMed and English-language publications, potentially missing Embase, Cochrane CENTRAL, regional databases, and non-English evidence; training-data biases may be propagated or amplified; the platform primarily evaluates abstract-level evidence, including conference abstracts that may omit subgroup analyses, mature safety data, and methodological details.
- Stated: the Living-OEP has not been evaluated or approved by any regulatory authority as a clinical decision-making device; the analysis evaluates the evidence base retrospectively, and prospective validation of its “living” capabilities is planned; the chatbot comparison was not blinded, was assessed by a single researcher, did not permit inter-rater reliability assessment, and lacked formal statistical testing with confidence intervals and prespecified sample sizes.
- Evident from the text: comparator assessment dates differed (ChatGPT, Perplexity, and Consensus accessed 24 June 2025; OpenEvidence accessed 19 July 2025), although the authors state no substantive updates occurred in the interval; GPT-5 was available but GPT-4o was selected based on preliminary testing; several extraction variables remained below 80% accuracy and three variables were still manually annotated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
- Findingown data
Current accuracy for response measured is 91.5%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for OS measure is 93.6%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Study design classification achieved 98.03% accuracy.
“Study design classification achieved 98.03% accuracy after refinement of systematic review and meta-analysis identification.”
Quote in the paperSubstantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within 10 iterations, the system achieved 87.2% accuracy for clinical stage.
“Within 10 iterations, the system achieved 87.2%, 99.4%, 87.8%, 93.6%, 80.9%, 61.8%, 51.1%, and 74.5% accuracy for clinical stage, histology, biomarker, subpopulation, risk, treatment path, interventions, and intervention category, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within the first three iterations of training, the system achieved 90.4% accuracy for sample size.
“Within the first three iterations of training, the system achieved 100.0%, 100.0%, 95.2%, 100.0%, 94.4%, and 90.4% accuracy for the following study characteristic variables: randomization, phase, clinical trial name, clinical trial registration, follow-up time, and sample size, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within the first three iterations of training, the system achieved 94.4% accuracy for follow-up time.
“Within the first three iterations of training, the system achieved 100.0%, 100.0%, 95.2%, 100.0%, 94.4%, and 90.4% accuracy for the following study characteristic variables: randomization, phase, clinical trial name, clinical trial registration, follow-up time, and sample size, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for landmark primary progression is 84.2%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within 10 iterations, the system achieved 99.4% accuracy for histology.
“Within 10 iterations, the system achieved 87.2%, 99.4%, 87.8%, 93.6%, 80.9%, 61.8%, 51.1%, and 74.5% accuracy for clinical stage, histology, biomarker, subpopulation, risk, treatment path, interventions, and intervention category, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for quality-of-life measure(s) is 97.9%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for median survival is 83.6%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for median other progression is 64.4%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within the first three iterations of training, the system achieved 100.0% accuracy for randomization.
“Within the first three iterations of training, the system achieved 100.0%, 100.0%, 95.2%, 100.0%, 94.4%, and 90.4% accuracy for the following study characteristic variables: randomization, phase, clinical trial name, clinical trial registration, follow-up time, and sample size, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within 10 iterations, the system achieved 61.8% accuracy for treatment path.
“Within 10 iterations, the system achieved 87.2%, 99.4%, 87.8%, 93.6%, 80.9%, 61.8%, 51.1%, and 74.5% accuracy for clinical stage, histology, biomarker, subpopulation, risk, treatment path, interventions, and intervention category, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for other progression measures is 60.8%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Intervention/comparator classification achieved 98.75% accuracy.
“Intervention/comparator classification achieved 98.75% accuracy after iterative prompt and annotation manual refinement.”
Quote in the paperSubstantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Variables with accuracy <80% remain under active iterative development.
“Variables with accuracy <80% remain under active iterative development.”
Quote in the paperNot a substantial claim - Findingown data
Current accuracy for observation time is 34.0%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Eight iterations were required for the system to achieve above 80% accuracy for patient characteristic variables.
“Eight iterations were required for the system to achieve above 80% accuracy for patient characteristic variables.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
The agentic AI system identified study populations with 99.82% concordance with human annotations.
“The system identified study populations with high accuracy, achieving 99.82% concordance with human annotations based on iterative prompt refinement.”
Quote in the paperSubstantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for landmark survival is 91.0%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within 10 iterations, the system achieved 87.8% accuracy for biomarker.
“Within 10 iterations, the system achieved 87.2%, 99.4%, 87.8%, 93.6%, 80.9%, 61.8%, 51.1%, and 74.5% accuracy for clinical stage, histology, biomarker, subpopulation, risk, treatment path, interventions, and intervention category, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within the first three iterations of training, the system achieved 100.0% accuracy for clinical trial registration.
“Within the first three iterations of training, the system achieved 100.0%, 100.0%, 95.2%, 100.0%, 94.4%, and 90.4% accuracy for the following study characteristic variables: randomization, phase, clinical trial name, clinical trial registration, follow-up time, and sample size, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
The agentic AI review accuracy ranged from 95.1% to 97.2%.
“The agentic AI review accuracy ranged from 95.1% to 97.2%.”
Quote in the paperSubstantial claim - Findingown data
Within 10 iterations, the system achieved 93.6% accuracy for subpopulation.
“Within 10 iterations, the system achieved 87.2%, 99.4%, 87.8%, 93.6%, 80.9%, 61.8%, 51.1%, and 74.5% accuracy for clinical stage, histology, biomarker, subpopulation, risk, treatment path, interventions, and intervention category, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for primary progression measure is 89.4%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within the first three iterations of training, the system achieved 95.2% accuracy for clinical trial name.
“Within the first three iterations of training, the system achieved 100.0%, 100.0%, 95.2%, 100.0%, 94.4%, and 90.4% accuracy for the following study characteristic variables: randomization, phase, clinical trial name, clinical trial registration, follow-up time, and sample size, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Outcomes classification achieved 93.73% accuracy.
“Outcomes classification achieved 93.73% accuracy after refinement to distinguish study protocols from results-reporting abstracts.”
Quote in the paperSubstantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for landmark other progression is 76.6%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for quality-of-life data is 85.1%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for response data is 76.1%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Current accuracy for median primary progression is 65.8%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within 10 iterations, the system achieved 74.5% accuracy for intervention category.
“Within 10 iterations, the system achieved 87.2%, 99.4%, 87.8%, 93.6%, 80.9%, 61.8%, 51.1%, and 74.5% accuracy for clinical stage, histology, biomarker, subpopulation, risk, treatment path, interventions, and intervention category, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within 10 iterations, the system achieved 51.1% accuracy for interventions.
“Within 10 iterations, the system achieved 87.2%, 99.4%, 87.8%, 93.6%, 80.9%, 61.8%, 51.1%, and 74.5% accuracy for clinical stage, histology, biomarker, subpopulation, risk, treatment path, interventions, and intervention category, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
The agentic AI system annotated 1997 publications in 7.25 h in a single continuous session without interruption.
“The agentic AI system annotated 1997 publications in 7.25 h in a single continuous session without interruption.”
Quote in the paperSubstantial claim - Findingown data
Current accuracy for safety data is 78.7%.
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%), primary progression measure (89.4%), median primary progression (65.8%), landmark primary progression (84.2%), other progression measures (60.8%), median other progression (64.4%), landmark other progression (76.6%), response measured (91.5%), response data (76.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within the first three iterations of training, the system achieved 100.0% accuracy for phase.
“Within the first three iterations of training, the system achieved 100.0%, 100.0%, 95.2%, 100.0%, 94.4%, and 90.4% accuracy for the following study characteristic variables: randomization, phase, clinical trial name, clinical trial registration, follow-up time, and sample size, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
Within 10 iterations, the system achieved 80.9% accuracy for risk.
“Within 10 iterations, the system achieved 87.2%, 99.4%, 87.8%, 93.6%, 80.9%, 61.8%, 51.1%, and 74.5% accuracy for clinical stage, histology, biomarker, subpopulation, risk, treatment path, interventions, and intervention category, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Findingown data
The Living-OEP provided more comprehensive structured evidence outputs compared with ChatGPT, Perplexity, Consensus, and OpenEvidence in completeness and precision of evidence across all six clinical criteria in eight BC scenarios.
“The Living-OEP provided more comprehensive structured evidence outputs compared with ChatGPT, Perplexity, Consensus, and OpenEvidence in completeness and precision of evidence across all six clinical criteria in eight BC scenarios.”
Quote in the paperSubstantial claim - Findingown data
Twenty-three percent of evidence included in the Living-OEP derived from recently published, potentially practice-changing pivotal trials.
“Twenty-three percent of evidence included in the Living-OEP derived from recently published, potentially practice-changing pivotal trials, highlighting the system’s capacity to rapidly incorporate emerging data into the decision support framework.”
Quote in the paperSubstantial claim - Findingown data
Comparative analysis revealed clear advantages over leading AI chatbots in guideline concordance, completeness, and inclusion of pivotal new data.
“Comparative analysis revealed clear advantages over leading AI chatbots in guideline concordance, completeness, and inclusion of pivotal new data—factors that are critical for real-world clinical decision-making.”
Quote in the paperSubstantial claim - Interpretationown data
A human-conducted, AI-augmented living SLR integrated with guidelines and regulatory data can provide real-time evidence support.
“A human-conducted, AI-augmented living SLR integrated with guidelines and regulatory data can provide real-time evidence support.”
Quote in the paperSubstantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Rests on an assumption: The agentic AI system's accuracy on the 1997-publication validation set generalizes to daily use on new publications.
- Study design and conductreasoning
All evidence included within the Living-OEP undergoes human review and contextual reconciliation before integration.
“For this reason, all evidence included within the Living-OEP undergoes human review and contextual reconciliation before integration.”
Quote in the paperSubstantial claim - Study design and conductnone given
The present system primarily evaluates abstract-level evidence, including conference presentations that may precede full peer-reviewed publication.
“The present system primarily evaluates abstract-level evidence, including conference presentations that may precede full peer-reviewed publication.”
Quote in the paperNot a substantial claim - Study design and conductnone given
Every AI-generated annotation is quality-checked by a human scientist before inclusion in the Living-OEP database.
“Although several extraction variables have not yet achieved target accuracy through AI annotation, every AI-generated annotation is quality-checked by a human scientist before inclusion in the Living-OEP database.”
Quote in the paperSubstantial claim - Study design and conductreasoning
Certain extraction variables continue to require expert review and reconciliation across full-text publications, guidelines, regulatory documents, and trial registries.
“Because conference abstracts frequently omit subgroup analyses, mature safety data, and methodological details, certain extraction variables continue to require expert review and reconciliation across full-text publications, guidelines, regulatory documents, and trial registries.”
Quote in the paperSubstantial claim- Rests on an assumption: Conference abstracts frequently omit subgroup analyses, mature safety data, and methodological details.
- Study design and conductreasoning
This design does not permit inter-rater reliability assessment.
“We acknowledge that this design does not permit inter-rater reliability assessment and future studies should incorporate blinded, multiassessor evaluation with a validated scoring instrument.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Human oversight ensures accurate data presentation regardless of AI performance on individual variables.
“This human oversight ensures accurate data presentation regardless of AI performance on individual variables.”
Quote in the paperSubstantial claim - Interpretationown data
The Living-OEP demonstrates that human-conducted, AI-augmented daily SLR can deliver highly accurate, up-to-date, and evidence-linked clinical information support in oncology.
“Our Living-OEP demonstrates that human-conducted, AI-augmented daily SLR can deliver highly accurate, up-to-date, and evidence-linked clinical information support in oncology.”
Quote in the paperSubstantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Rests on an assumption: The agentic AI system's accuracy on the 1997-publication validation set generalizes to daily use on new publications.
- Interpretationreasoning
By combining daily, human-conducted, AI-augmented SLR with guideline and regulatory integration, the Living-OEP delivers real-time, evidence-linked clinical information support for oncology.
“By combining daily, human-conducted, AI-augmented SLR with guideline and regulatory integration, the Living-OEP delivers real-time, evidence-linked clinical information support for oncology.”
Quote in the paperSubstantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
- Rests on an assumption: The agentic AI system's accuracy on the 1997-publication validation set generalizes to daily use on new publications.
- Interpretationnone given
The integration of structured trial data, guidelines, and regulatory updates enables evidence-linked recommendations that are continuously updated.
“The integration of structured trial data, guidelines, and regulatory updates enables evidence-linked recommendations that are continuously updated.”
Quote in the paperSubstantial claim
- May combine two claims. 12
- Classified as probably not a substantial claim. 10
May combine two claims.
Findingnone givenCompared with other AI tools, the Living-OEP provided more comprehensive and accurate evidence and linked all data to original publications and FDA labels.
“Compared with other AI tools, the Living-OEP provided more comprehensive and accurate evidence and linked all data to original publications and Food and Drug Administration labels.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe agentic AI extraction accuracy ranged from 51.1% to 99.4%, with three variables still undergoing iterative refinement.
“Extraction accuracy ranged from 51.1% to 99.4%, with three variables still undergoing iterative refinement.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataHuman scientists review ∼80 abstracts per hour and extract data from approximately four abstracts per hour, with an inclusion rate of 30% following title/abstract screening.
“Based on institutional benchmarks, we estimated that human scientists review ∼80 abstracts per hour and extract data from approximately four abstracts per hour, with an inclusion rate of 30% following title/abstract screening. These estimates were derived from internal time-tracking data collected during the development of the annotation manual.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe Living-OEP consistently provided guideline-concordant, evidence-graded treatment recommendations with traceable source references, whereas all comparator chatbots failed to systematically identify guideline recommendation status and evidence levels for most scenarios tested.
“Notably, the Living-OEP consistently provided guideline-concordant, evidence-graded treatment recommendations with traceable source references, whereas all comparator chatbots failed to systematically identify guideline recommendation status and evidence levels for most scenarios tested.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe Living-OEP consistently included guideline-recommended treatments, pivotal new trials, and actionable clinical data, whereas other AI tools frequently omitted recent therapeutic advances or failed to specify guideline categories and evidence levels.
“The Living-OEP consistently included guideline-recommended treatments, pivotal new trials, and actionable clinical data, whereas other AI tools frequently omitted recent therapeutic advances or failed to specify guideline categories and evidence levels.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductnone givenThree variables (analysis type, subgroup names, and subgroup data) have yet to be incorporated into the AI training pipeline and are currently annotated manually by trained scientists within the Living-OEP.
“Three variables have yet to be incorporated into the AI training pipeline: analysis type, subgroup names, and subgroup data. These variables are currently annotated manually by trained scientists within the Living-OEP.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningAccuracy metrics are reported as point estimates; formal statistical testing with confidence intervals and prespecified sample sizes will be incorporated in future evaluations.
“Accuracy metrics are reported as point estimates. As this study represents a descriptive benchmarking validation, formal statistical testing with confidence intervals and prespecified sample sizes will be incorporated in future evaluations.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenThe comparison between the Living-OEP and AI chatbots was not blinded and was assessed by a single researcher.
“The comparison between the Living-OEP and AI chatbots was not blinded and was assessed by a single researcher.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningBecause the AI system is trained on human-generated annotations, any systematic biases present in the training data may be propagated and amplified.
“Because the AI system is trained on human-generated annotations, any systematic biases present in the training data may be propagated and amplified.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThe agentic AI system achieved high concordance with human annotations for key trial annotation tasks and dramatically reduced the time required for evidence integration.
“The agentic AI system achieved high concordance with human annotations for key trial annotation tasks and dramatically reduced the time required for evidence integration.”
Quote in the paperSubstantial claim- Rests on an assumption: Human annotations used as the benchmark are accurate.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningThis approach offers oncologists a scalable framework for structured oncology evidence synthesis and retrieval.
“This approach offers oncologists a scalable framework for structured oncology evidence synthesis and retrieval.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Study design and conductnone givenRestricting the search to English-language publications may underrepresent evidence from non-English-speaking regions.
“Additionally, restricting the search to English-language publications may underrepresent evidence from non-English-speaking regions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe current search strategy is limited to PubMed and may miss relevant evidence indexed in Embase, the Cochrane Central Register of Controlled Trials, or regional databases.
“The current search strategy is limited to PubMed and may miss relevant evidence indexed in Embase, the Cochrane Central Register of Controlled Trials, or regional databases.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenAlthough the Living-OEP is updated daily, this analysis evaluates the evidence base retrospectively.
“Although the Living-OEP is updated daily, this analysis evaluates the evidence base retrospectively.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe Living-OEP has not been evaluated or approved by any regulatory authority as a clinical decision-making device.
“It has not been evaluated or approved by any regulatory authority as a clinical decision-making device and its outputs should be interpreted within the context of individual patient circumstances and physician expertise.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningUnder these assumptions, human scientists would require an estimated 174.7 h to complete the same tasks, representing a 95.9% reduction in annotation time.
“Under these assumptions, human scientists would require an estimated 174.7 h to complete the same tasks, representing a 95.9% reduction in annotation time.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe Living-OEP is intended as an informational tool to support, not replace, clinical judgment.
“It is important to note that the Living-OEP is intended as an informational tool to support, not replace, clinical judgment.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductreasoningAlthough database searches included publications from 2019 onward, guidelines and regulatory approvals were cross-checked to include critical-path studies referenced in regulatory labels, guideline recommendations, and major conferences regardless of publication date.
“Although database searches included publications from 2019 onward, guidelines and regulatory approvals were cross-checked to include critical-path studies referenced in regulatory labels, guideline recommendations, and major conferences regardless of publication date.”
Quote in the paperSubstantial claim
The paper relies on this without support.
Conference abstracts frequently omit subgroup analyses, mature safety data, and methodological details.
“Because conference abstracts frequently omit subgroup analyses, mature safety data, and methodological details, certain extraction variables continue to require expert review and reconciliation across full-text publications, guidelines, regulatory documents, and trial registries.”
The paper relies on this without support.
Human annotations used as the benchmark are accurate.
“Accuracy was defined as the percentage of AI annotations exactly matching human annotations in both content and format.”
The paper relies on this without support.
The agentic AI system's accuracy on the 1997-publication validation set generalizes to daily use on new publications.
“The agentic AI system’s review and extraction accuracy was benchmarked against 1997 human-annotated BC publications independent of the training dataset.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Abstract extraction-accuracy minimum excludes observation time
MinorThe abstract states that extraction accuracy starts at 51.1%, whereas the Results report 34.0% accuracy for observation time.
“Extraction accuracy ranged from 51.1% to 99.4%, with three variables still undergoing iterative refinement.”
“Current accuracy values are as follows: OS measure (93.6%), median survival (83.6%), landmark survival (91.0%), observation time (34.0%),”
Abstract extraction-accuracy maximum excludes 100% results
MinorThe abstract caps extraction accuracy at 99.4%, but the Results report 100.0% accuracy for several study-characteristic variables.
“Extraction accuracy ranged from 51.1% to 99.4%, with three variables still undergoing iterative refinement.”
“Within the first three iterations of training, the system achieved 100.0%, 100.0%, 95.2%, 100.0%, 94.4%, and 90.4% accuracy for the following study characteristic variables: randomization, phase, clinical trial name, clinical trial registration, follow-up time, and sample size, respectively.”
More than three extraction variables remain under refinement
MinorThe abstract says three variables are still undergoing iterative refinement. The Results say all variables below 80% remain under active iterative development and list more than three, including treatment path, interventions, intervention category, and observation time.
“Extraction accuracy ranged from 51.1% to 99.4%, with three variables still undergoing iterative refinement.”
“Variables with accuracy <80% remain under active iterative development.”
Cites
Gennari, A., André, F., Barrios, C. H., Cortés, J., de Azambuja, E., DeMichele, A., Dent, R. A., Fenlon, D., Gligorov, J., Hurvitz, S. A., et al. (2021) · Annals of Oncology · doi:10.1016/j.annonc.2021.09.019 · Cited by 1,455 · see citing works
Self-citationEconomic burden of cancer across the European Union: a population-based cost analysis
Luengo-Fernández, R., Leal, J., Gray, A. M., & Sullivan, R. J. (2013) · The Lancet Oncology · doi:10.1016/s1470-2045(13)70442-x · Cited by 1,010 · see citing works
Medical Care Costs Associated with Cancer Survivorship in the United States
Mariotto, A. B., Enewold, L., Zhao, J., Zeruto, C. A., & Yabroff, K. R. (2020) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-19-1534 · Cited by 667 · see citing works
Health at a Glance: Europe 2018
OECD, & Union, E. (2018) · Health at a glance. Europe · doi:10.1787/health_glance_eur-2018-en · Cited by 525 · see citing works
Sentinel Lymph Node Biopsy in Early-Stage Breast Cancer: ASCO Guideline Update
Park, K. U., Somerfield, M. R., Anne, N., Brackstone, M., Conlin, A. K., Couto, H. L., Dengel, L. T., Eisen, A. F., Harvey, B. E., Hawley, J. R., et al. (2025) · Journal of Clinical Oncology · doi:10.1200/jco-25-00099 · Cited by 156 · see citing works
Akpınar, H. (2025) · BMC Oral Health · doi:10.1186/s12903-025-06294-7 · Cited by 26 · see citing works
Royce, M., Shah, M., Zhang, L., Cheng, J., Bonner, M. K., Pegues, M. A., Miller, C. P., Leu, L., Price, L., Qiu, J., et al. (2025) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-25-1388 · Cited by 21 · see citing works
Pramanik, A., Grieve, S., Liu, R., & Forsythe, A. (2026) · Value in Health · doi:10.1016/j.jval.2026.03.1392
Self-citationData from Medical Care Costs Associated with Cancer Survivorship in the United States
Mariotto, A. B., Enewold, L., Zhao, J., Zeruto, C. A., & Yabroff, K. R. (2024) · doi:10.1158/1055-9965.c.7392626
Flagged first, then by citations
Research works printed but not linked: 1
- Guy, 2020, Economic burden of cancer survivorship among adults in the United States, Cancer Epidemiol Biomarkers Prev, 29, 1304
- Authors
- Rugo, H., Forsythe, A., Flora, D., Glück, S., Grieve, S., Campden, R., Liu, R. J., Musat, M., Rege, J. M., Kaufman, P. A., Coleman, R. L., Tarantino, P., O’Shaughnessy, J., & Schwartzberg, L.
- Affiliations
- City of Hope (US)
- Zoo Miami (US)
- St. Elizabeth Healthcare (US)
- University of Vermont Medical Center (US)
- Texas Oncology (US)
- Harvard University (US)
- Renown Health (US)
- Corresponding author
- Forsythe, A.
- ORCID
- Venue
- ESMO Real World Data and Digital Oncology
- Publisher
- Elsevier BV
- Published
- 2026-09
- Volume
- 13
- Pages
- 100770
- ISSN
- 2949-8201 (ISSN-L)
- PMC
- PMC13563553
- OpenAlex
- W7204910734
- Cited by
- 0
- References
- 4
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- ESMO Real World Data and Digital Oncology (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- agentic AI, breast cancer, GPT-4, Claude Sonnet 4, treatment pathways
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-30.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-30.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
