Epidemiology, Quality and Reporting Characteristics of Systematic Reviews of Traditional Chinese Medicine Interventions Published in Chinese Journals
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.
Abstract
BACKGROUND: Systematic reviews (SRs) of TCM have become increasingly popular in China and have been published in large numbers. This review provides the first examination of epidemiological characteristics of these SRs as well as compliance with the PRISMA and AMSTAR guidelines. OBJECTIVES: To examine epidemiological and reporting characteristics as well as methodological quality of SRs of TCM published in Chinese journals. METHODS: Four Chinese databases were searched (CBM, CSJD, CJFD and Wanfang Database) for SRs of TCM, from inception through Dec 2009. Data were extracted into Excel spreadsheets. The PRISMA and AMSTAR checklists were used to assess reporting characteristics and methodological quality, respectively. RESULTS: A total of 369 SRs were identified, most (97.6%) of which used the terms systematic review or meta-analysis in the title. None of the reviews had been updated. Half (49.8%) were written by clinicians and nearly half (47.7%) were reported in specialty journals. The impact factors of 45.8% of the journals published in were zero. The most commonly treated conditions were diseases of the circulatory and digestive disease. Funding sources were not reported for any reviews. Most (68.8%) reported information about quality assessment, while less than half (43.6%) reported assessing for publication bias. Statistical mistakes appeared in one-third (29.3%) of reviews and most (91.9%) did not report on conflict of interest. CONCLUSIONS: While many SRs of TCM interventions have been published in Chinese journals, the quality of these reviews is troubling. As a potential key source of information for clinicians and researchers, not only were many of these reviews incomplete, some contained mistakes or were misleading. Focusing on improving the quality of SRs of TCM, rather than continuing to publish them in great quantity, is urgently needed in order to increase the value of these studies.
Overview
CoChat Summary
TL;DR
This study surveyed systematic reviews of traditional Chinese medicine (TCM) interventions published in Chinese journals and found 369 reviews, with generally poor reporting and methodological quality. The authors conclude that many reviews were incomplete, some had mistakes, and quality improvement should take priority over publishing more reviews.
Key findings
- The search found 10,001 records, and after screening/exclusions the final sample was 369 TCM systematic reviews published in Chinese journals.
- Nearly all reviews (97.6%) used “systematic review” or “meta-analysis” in the title, but none had been updated from a previous review.
- Reporting and quality were weak: fewer than half reported assessing publication bias (43.6%), 91.9% did not report conflict of interest, and statistical mistakes appeared in 29.3% of reviews.
- Most reviews were about herbal interventions (84.3%); the most common conditions studied were circulatory system diseases (6.5%) and digestive system diseases (5.7%).
Methods
- Design: cross-sectional review of published systematic reviews, assessing epidemiological characteristics, PRISMA reporting compliance, and AMSTAR methodological quality.
- Data/sample: all eligible SRs of TCM interventions published in Chinese journals, identified by searching CBM, CSJD, CJFD, and Wanfang from inception through December 2009; 369 reviews were included.
- Main analysis: standardized data extraction into Excel and summary with descriptive statistics (frequency, median, IQR); screening was done by two researchers independently for titles/abstracts, with full-text screening and disagreement resolution by discussion.
Limitations
- Stated in the paper: the authors note that they included only systematic reviews published in Chinese journals, so reviews by Chinese investigators in international journals were not captured.
- Stated in the paper: because PRISMA had not yet been released when many included reviews were published, some poor reporting may reflect historical timing rather than current practice.
- Evident from the text: the study relies on descriptive assessment of published reports, so conclusions about quality are limited to what was reported, not necessarily what was actually done.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
2 Valid · 0 Not valid · 10 Not checked · 47 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Reviews of numerous medical specialties, as well as comprehensive reviews, have concluded that the quality of SR and meta-analysis (MA) reporting is generally poor [6], [7], [17].”
§ Epidemiology, Quality and Reporting Characteristics of Systematic Reviews of Traditional Chinese Medicine Interventions Published in Chinese Journals > Discussion
What the sources say
- Not checked yetcites [6] TQ Li, 2007, Assessment of Systematic Reviews and Meta-analyses on Traditional Chinese Medicine Published in Chinese Journals., Chin J Evid-based Med
- Not checked yetcites [7] JP Liu, 2007, Quality appraisal of systematic reviews or meta-analysis on traditional Chinese medicine published in Chinese journals., Zhongguo Zhong Xi Yi Jie He Za Zhi
- Supportedcites [17] D Moher, 2009, Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement., BMJ source
The Discussion says reporting quality in systematic reviews is still not optimal and cites assessments across several medical specialties. The introductory text also reports that an evaluation of 83 meta-analyses found reporting generally poor, with little improvement in a later update. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This instrument, developed on the biasis of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) [11] and Sack's Quality Assessment Checklist [12] is considered a validated tool in assessing the methodological quality of systematic reviews, and receives recognition from international agencies, including The Canadian Agency for Drugs and Technologies in Health (CADTH) [13].”
§ Epidemiology, Quality and Reporting Characteristics of Systematic Reviews of Traditional Chinese Medicine Interventions Published in Chinese Journals > Introduction
What the sources say
- Partly supportedcites [10] BJ Shea, 2007, Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews., BMC Med Res Methodol sourceExperimental — accuracy not yet measured
Our copy supports only that the resulting 11-item instrument 'was judged to have face and content validity' (Abstract, Results; Discussion, Strengths and Weaknesses) and that CADTH undertook an independent assessment with positive reviewer feedback — so the development basis (OQAQ + Sacks checklist) and the CADTH recognition are accurate. However, the same Discussion passage explicitly hedges validation: 'We recognize the need for further testing of AMSTAR. Additional studies are necessary with a focus on the reproducibility and construct validity of AMSTAR before strong recommendations can be made on its use,' and the Discussion refers to use 'after external validation'. Presenting AMSTAR as 'a validated tool' therefore overstates what this source claims. (LLM, conf=0.78)
- Not checked yetcites [11] AD Oxman, 1994, Checklists for review articles., BMJ source
- Not checked yetcites [12] H Sacks, 1987, Meta-analyses of randomized controlled trials., N Engl J Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“nearly half (46.1%) had not been cited and only 2.4% had been cited more than 15 times.”
§ Epidemiology, Quality and Reporting Characteristics of Systematic Reviews of Traditional Chinese Medicine Interventions Published in Chinese Journals > Results > Epidemiological Characteristics (Table 1)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A total of 369 SRs were identified”
- Also stated in: § Search · plain
“A total of 369 publications were included (Figure 1, Text S2).”
- Also stated in: § Discussion · plain
“Our study identified 369 systematic reviews of TCM interventions published since 1999, most of which evaluated herbal interventions.”
§ Epidemiology, Quality and Reporting Characteristics of Systematic Reviews of Traditional Chinese Medicine Interventions Published in Chinese Journals > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Search · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reviews were classified as either Herbal interventions (84.3%) or Non-Herbal interventions.”
§ Epidemiology, Quality and Reporting Characteristics of Systematic Reviews of Traditional Chinese Medicine Interventions Published in Chinese Journals > Results > Epidemiological Characteristics (Table 1)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
43 more claims
Create a free account to see all 59 claims and to verify the 27 references still open.
- It may state two separate claims as one. 8
- Probably not a substantial claim. 6
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Outside of these organizations, the existence of a protocol is infrequently reported in completed reviews [9, 10].”
From Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement (2015) · cites this paper as [9]
Experimental — accuracy not yet measured
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2010) · International Journal of Surgery · doi:10.1016/j.ijsu.2010.02.007 · Cited by 12,156 · cited 1×
“Reviews of numerous medical specialties, as well as comprehensive reviews, have concluded that the quality of SR and metaanalysis (MA) reporting is generally poor [6,7,17].”
BackgroundMoher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-10 · Cited by 4,374
Meta-Analyses of Randomized Controlled Trials
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A., & Chalmers, T. C. (1987) · New England Journal of Medicine · doi:10.1056/nejm198702193160806 · Cited by 1,250 · cited 1×
“This instrument, developed on the biasis of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) [11] and Sack’s Quality Assessment Checklist [12] is considered a validated tool in assessing the methodological quality of systematic reviews, and receives recognition from international agencies, including T…”
MethodEpidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891 · cited 5×
“Examination of changes in reporting quality of MAs published in Medline, Embase and the Cochrane Database of Systematic Reviews, suggested that quality had improved after the release of the QUORUM reporting guidelines [18,19].”
BackgroundInfluentialSystematic Reviews from Astronomy to Zoology: Myths and Misconceptions
Petticrew, M. (2001) · BMJ · doi:10.1136/bmj.322.7278.98 · Cited by 550
Systematic Reviews: Checklists for review articles
Oxman, A. D. (1994) · BMJ · doi:10.1136/bmj.309.6955.648 · Cited by 480
Jadad, A. R., & McQuay, H. J. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00062-3 · Cited by 205 · cited 1×
“In the last decade, a few studies have examined the reporting of systematic reviews, and found that the quality of reporting was generally poor [3,4,5].”
BackgroundA systematic evaluation of the quality of meta-analyses in the critical care literature
Delaney, A., Bagshaw, S. M., Ferland, A., Manns, B., Laupland, K. B., & Doig, C. J. (2005) · Critical Care · doi:10.1186/cc3803 · Cited by 142 · cited 1×
“Examination of changes in reporting quality of MAs published in Medline, Embase and the Cochrane Database of Systematic Reviews, suggested that quality had improved after the release of the QUORUM reporting guidelines [18,19].”
BackgroundSystematic reviews: when is an update an update?
Moher, D., & Tsertsvadze, A. (2006) · The Lancet · doi:10.1016/s0140-6736(06)68358-x · Cited by 110 · cited 1×
“Although it is well known that results from reviews are most useful when they are up-to-date [16], none of the reviews included in our SR reported being an update of a previous review.”
BackgroundShea, B., Moher, D., Graham, I., Pham, B., & Tugwell, P. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001008 · Cited by 110 · cited 1×
“In the last decade, a few studies have examined the reporting of systematic reviews, and found that the quality of reporting was generally poor [3,4,5].”
BackgroundJunhua, Z., Hongcai, S., Xiumei, G., Boli, Z., Yaozu, X., Hongbo, C., & Ming, R. (2007) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2007.7195 · Cited by 48
Nogueira, R. V. B., Vasconcelos, B. C. D. E., & Niederman, R. (2008) · BMC Oral Health · doi:10.1186/1472-6831-8-27 · Cited by 40 · cited 1×
“This instrument, developed on the biasis of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) [11] and Sack’s Quality Assessment Checklist [12] is considered a validated tool in assessing the methodological quality of systematic reviews, and receives recognition from international agencies, including T…”
MethodLiu, J., & Xia, Y. Q. (2007) · PubMed · Cited by 17
Zhang, Y. (2007) · Zhongguo xunzheng yixue zazhi · Cited by 12
Self-citationInvestigation into Nursing Staff's Cognition Status of Evidence-based Medicine
Zhang, L. (2010) · Liaoning Yixueyuan xuebao · Cited by 3
Research works printed but not linked: 3
- ZH Chen, 1999, Troditional Chinese herbs for precancerous lesion of esophagus: a meta analysis., TCM Res, 12, 20
- XP Wang, 2010, Investigation cognition about Evidence-based Medicine., Medical information, 8, 2537
- J Xiong, 2010, Assessing tool of methodology quality for systematic reviews and meta analysis: introduction of AMSTAR., Chin J Evid-based Med, 9, 44
Cited by
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
“Outside of these organizations, the existence of a protocol is infrequently reported in completed reviews [9,10].”
BackgroundPage, M. J., & Moher, D. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0663-8 · Cited by 769
Systematic review adherence to methodological or reporting quality
Pussegoda, K., Turner, L., Garritty, C., Mayhew, A., Skidmore, B., Stevens, A., Boutron, I., Sarkis-Onofre, R., Bjerre, L. M., Hróbjartsson, A., et al. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0527-2 · Cited by 322
“Additional reports eligible for inclusion were identified by members of the research team prior to the start of the project [2, 21, 22].”
The methodological and reporting quality of systematic reviews from China and the USA are similar
Tian, J., Zhang, J., Ge, L., Yang, K., & Song, F. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.12.004 · Cited by 248
Self-citationA systematic scoping review of adherence to reporting guidelines in health care literature
Thabane, L., Samaan, Mbuagbaw, L., Kosa, S., Debono, B., Dillenburg, Zhang, Fruci, V., Dennis, & Bawor (2013) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s43952 · Cited by 192
“Ma et al 74 and Willis et al 88 used the 27-item PRISMA checklist to assess the level of adherence.”
Metadata
- Authors
- Ma, B., Guo, J., Qi, G., Li, H., Peng, J., Zhang, Y., Ding, Y., & Yang, K.
- Affiliations
- Lanzhou University (CN)
- Corresponding author
- Yang, Kehu
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2011-05-25
- Volume
- 6
- Issue
- 5
- Pages
- e20185
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3102106
- OpenAlex
- W2059874779
- Cited by
- 56
- References
- 17
- Citations by year
- 2026: 1 · 2023: 3 · 2022: 1 · 2021: 3 · 2020: 5 · 2019: 1 · 2018: 4 · 2017: 6 · 2016: 4 · 2015: 6 · 2014: 12 · 2013: 3 · 2012: 7
- FWCI
- 1.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Lanzhou University Institutional Repository (submittedVersion, cc-by-nc)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Lanzhou University
- University of Alberta
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Complementary and alternative medicine › Traditional Chinese Medicine Clinical Research
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- traditional Chinese medicine, methodological quality, reporting quality, PRISMA, AMSTAR, Chinese periodicals, publication bias, conflicts of interest, funding disclosure, statistical errors, research quality assessment
- MeSH terms
- China, Humans, Medicine, Chinese Traditional, Publishing, Review Literature as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
