Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.

Language models for data extraction and risk of bias assessment in complementary medicine

Open access

Abstract

Large language models (LLMs) have the potential to enhance evidence synthesis efficiency and accuracy. This study assessed LLM-only and LLM-assisted methods in data extraction and risk of bias assessment for 107 trials on complementary medicine. Moonshot-v1-128k and Claude-3.5-sonnet achieved high accuracy (≥95%), with LLM-assisted methods performing better (≥97%). LLM-assisted methods significantly reduced processing time (14.7 and 5.9 min vs. 86.9 and 10.4 min for conventional methods). These findings highlight LLMs' potential when integrated with human expertise.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01457-w

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Comparison of accuracy between Moonshot-v1-128k and Claude-3.5-sonnet in extracting data and assessing ROB. This figure compares the accuracy of two language models, Claude-3.5-sonnet and Moonshot-v1-128k, in data extraction and risk-of-bias (ROB) assessments across multiple domains in 107 RCTs. Claude-3.5-sonnet demonstrated higher overall accuracy in data extraction (96.2%, 95% CI: 95.8% to 96.5%) than Moonshot-v1-128k (95.1%, 95% CI: 94.7% to 95.5%), with a statistically significant difference of 1.1% (p < 0.001). In the ROB assessment, Claude-3.5-sonnet also achieved slightly higher accuracy (96.9% vs. 95.7%), though the difference was not statistically significant. The greatest difference in domain-specific accuracy was observed in the Baseline Characteristics for data extraction, where Claude-3.5-sonnet outperformed Moonshot-v1-128k.
Fig. 1 Comparison of accuracy between Moonshot-v1-128k and Claude-3.5-sonnet in extracting data and assessing ROB. This figure compares the accuracy of two language models, Claude-3.5-sonnet and Moonshot-v1-128k, in data extraction and risk-of-bias (ROB) assessments across multiple domains in 107 RCTs. Claude-3.5-sonnet demonstrated higher overall accuracy in data extraction (96.2%, 95% CI: 95.8% to 96.5%) than Moonshot-v1-128k (95.1%, 95% CI: 94.7% to 95.5%), with a statistically significant difference of 1.1% (p < 0.001). In the ROB assessment, Claude-3.5-sonnet also achieved slightly higher accuracy (96.9% vs. 95.7%), though the difference was not statistically significant. The greatest difference in domain-specific accuracy was observed in the Baseline Characteristics for data extraction, where Claude-3.5-sonnet outperformed Moonshot-v1-128k.CC BY-NC-ND · PubMed Central
Fig. 2 Comparison of accuracy and efficiency of conventional, LLM-only, and LLM-assisted methods in extracting data and assessing ROB using Moonshot-v1-128k. This figure presents a comparison of the accuracy (correct rate) and efficiency (time spent) of three methods for data extraction and risk of bias (ROB) assessment: conventional, LLM-only, and LLM-assisted. For data extractions, the conventional method had an estimated accuracy of 95.3% and took 86.9 min per RCT. The LLM-only method achieved an accuracy of 95.1% and took only 96 s per RCT, while the LLM-assisted method had the highest accuracy at 97.9% and took 14.7 min per RCT. For ROB assessments, the conventional method had an estimated accuracy of 90.0% and took 10.4 min per RCT. The LLM-only method achieved an accuracy of 95.7% and took only 42 s per RCT, while the LLM-assisted method had the highest accuracy at 97.3% and took 5.9 min per RCT. These results demonstrate that LLM-assisted methods can achieve higher accuracy than conventional methods while being substantially more efficient.
Fig. 2 Comparison of accuracy and efficiency of conventional, LLM-only, and LLM-assisted methods in extracting data and assessing ROB using Moonshot-v1-128k. This figure presents a comparison of the accuracy (correct rate) and efficiency (time spent) of three methods for data extraction and risk of bias (ROB) assessment: conventional, LLM-only, and LLM-assisted. For data extractions, the conventional method had an estimated accuracy of 95.3% and took 86.9 min per RCT. The LLM-only method achieved an accuracy of 95.1% and took only 96 s per RCT, while the LLM-assisted method had the highest accuracy at 97.9% and took 14.7 min per RCT. For ROB assessments, the conventional method had an estimated accuracy of 90.0% and took 10.4 min per RCT. The LLM-only method achieved an accuracy of 95.7% and took only 42 s per RCT, while the LLM-assisted method had the highest accuracy at 97.3% and took 5.9 min per RCT. These results demonstrate that LLM-assisted methods can achieve higher accuracy than conventional methods while being substantially more efficient.CC BY-NC-ND · PubMed Central
Fig. 3 Flow diagram of the study process. The study process began with the random selection of 107 randomized controlled trials (RCTs) on complementary and alternative medicine interventions. An investigator checked the PDF files of these RCTs to ensure their eligibility and completeness. Two large language models (LLMs)—Moonshot-v1-128k and Claude-3.5-sonnet—were employed, with tailored prompts iteratively tested and refined until both LLMs could achieve continuous, correct extractions and assessments across five consecutive RCTs. A single investigator then independently conducted extractions and risk-of-bias (ROB) assessments for each RCT using these finalized prompts. Simultaneously, four reviewers, after one month of training, used the lower-accuracy LLM outputs as initial references. Each reviewer independently checked and, if necessary, modified the LLM results to create four separate LLM-assisted assessments and extractions per RCT. To validate these results, two methodologists compared the LLM-only, LLM-assisted, and conventional manual assessments and reached a consensus to establish a benchmark reference. The study was conducted between November 2023 and September 2024, adhering to AAPOR guidelines and exempt from ethics approval by Lanzhou University’s Medical Ethics Review Committee due to its use of publicly available data.
Fig. 3 Flow diagram of the study process. The study process began with the random selection of 107 randomized controlled trials (RCTs) on complementary and alternative medicine interventions. An investigator checked the PDF files of these RCTs to ensure their eligibility and completeness. Two large language models (LLMs)—Moonshot-v1-128k and Claude-3.5-sonnet—were employed, with tailored prompts iteratively tested and refined until both LLMs could achieve continuous, correct extractions and assessments across five consecutive RCTs. A single investigator then independently conducted extractions and risk-of-bias (ROB) assessments for each RCT using these finalized prompts. Simultaneously, four reviewers, after one month of training, used the lower-accuracy LLM outputs as initial references. Each reviewer independently checked and, if necessary, modified the LLM results to create four separate LLM-assisted assessments and extractions per RCT. To validate these results, two methodologists compared the LLM-only, LLM-assisted, and conventional manual assessments and reached a consensus to establish a benchmark reference. The study was conducted between November 2023 and September 2024, adhering to AAPOR guidelines and exempt from ethics approval by Lanzhou University’s Medical Ethics Review Committee due to its use of publicly available data.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 35 references

Research works printed but not linked: 2

  1. Lim, C. E. D., Ng, R. W. C., Cheng, N. C. L., Zhang, G. S. & Chen, H. Acupuncture for polycystic ovarian syndrome. Cochrane Database Syst. Rev. 2019, CD007689 (2019).
  2. Chapter 5: Collecting data. https://training.cochrane.org/handbook/current/chapter-05.

Cited by

See all 86

Metadata

Authors
Lai, H., Liu, J., Bai, C., Liu, H., Pan, B., Luo, X., Hou, L., Zhao, W., Xia, D., Tian, J., Chen, Y., Zhang, L., Estill, J., Liu, J., Liao, X., Shi, N., Sun, X., Shang, H., Bian, Z., Yang, K., Huang, L., Ge, L., On behalf of ADVANCED Working Group, Li, H., Wang, Y., Zhang, H., Zhu, D., Peng, D., Wang, F., Li, Y., Tang, S., Liu, H., Li, Z., Yang, Z., Yu, X., & Qin, Y.
Affiliations
  • Lanzhou University (CN)
  • China Academy of Chinese Medical Sciences (CN)
  • Southern Medical University (CN)
  • Chinese Academy of Medical Sciences & Peking Union Medical College (CN)
  • Hong Kong Baptist University (HK)
  • University of Geneva (CH)
  • Sichuan University (CN)
  • Beijing University of Chinese Medicine (CN)
  • Children's Hospital of Chongqing Medical University (CN)
Corresponding author
  • Huang, Luqi
  • Ge, Long
Venue
npj Digital Medicine
Published
2025-01-31
Volume
8
Issue
1
Pages
74-74
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4407009455
Cited by
59
References
35
Citations by year
2026: 40 · 2025: 19
FWCI
37.74
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Natural Science Foundation of China
  • China Academy of Chinese Medical Sciences
  • Fundamental Research Funds for the Central Universities
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
large language models, data extraction, risk of bias assessment, complementary therapy, evidence synthesis, Claude 3.5 Sonnet, human-AI collaboration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Language models for data extraction and risk of bias assessment in complementary medicine. https://app.cochat.ai/papers/10.1038/s41746-025-01457-w
The paper
Lai, H., Liu, J., Bai, C., Liu, H., Pan, B., Luo, X., Hou, L., Zhao, W., Xia, D., Tian, J., Chen, Y., Zhang, L., Estill, J., Liu, J., Liao, X., Shi, N., Sun, X., Shang, H., Bian, Z., . . . Qin, Y. (2025). Language models for data extraction and risk of bias assessment in complementary medicine. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01457-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.