The DOI matches its Crossref record; 235 of the 236 references it cites were found in a registry or an index.
The inclusion of reports of randomised trials published in languages other than English in systematic reviews
Open accessAbstract
OBJECTIVE: To assemble a large dataset of language restricted and language inclusive systematic reviews, including both conventional medicinal (CM) and complementary and alternative medicine (CAM) interventions. To then assess the quality of these reports by considering and comparing different types of systematic reviews and their associated RCTs; CM and CAM interventions; the effect of language restrictions compared with language inclusions, and whether these results are influenced by other issues, including statistical heterogeneity and publication bias, in the systematic review process. DATA SOURCES: MEDLINE, EMBASE, the Cochrane Database of Systematic Reviews and the Centralised Information Service for Complementary Medicine. REVIEW METHODS: Three types of systematic reviews were included: language restricted; language inclusive/English language (EL) reviews that searched RCTs in languages other than English (LOE) but did not find any and, hence, could not include any, in the quantitative data synthesis; and systematic reviews that searched for RCTs in LOE and included them in the quantitative data synthesis. Fisher's exact test was applied to compare the three different types of systematic reviews with respect to their reporting characteristics and the systematic review quality assessment tool. The odds ratio of LOE trials versus EL trials was computed for each review and this information was pooled across the reviews to examine the influence that language of publication and type of intervention (CM, CAM) have on the estimates of intervention effect. Several sensitivity analyses were performed. RESULTS: The LOE RCTs were predominantly in French and German. Language inclusive/LOE systematic reviews were of the highest quality compared with the other types of reviews. The CAM reviews were of higher quality compared with the CM reviews. There were only minor differences in the quality of reports of EL RCTs compared with the eight other languages considered. However, there are inconsistent differences in the quality of LOE reports depending on the intervention type. The results, and those reported previously, suggest that excluding reports of RCTs in LOE from the analytical part of a systematic review is reasonable. Because the present research and previous efforts have not included every type of CM RCT and the resulting possibility of the uncertainty as to when bias will be present by excluding LOE, it is always prudent to perform a comprehensive search for all evidence. This result only applies to reviews investigating the benefits of CM interventions. This does not imply that systematic reviewers should neglect reports in LOE. We recommend that systematic reviewers search for reports regardless of the language. There may be merit in including them in some aspects of the review process although this decision is likely to depend on several factors, including fiscal and other resources being available. Language restrictions significantly shift the estimates of an intervention's effectiveness when the intervention is CAM. Here, excluding trials reported in LOE, compared with their inclusion, resulted in a reduced intervention effect. The present results do not appear to be influenced by statistical heterogeneity and publication bias. CONCLUSIONS: With the exception of CAM systematic reviews, the quality of recently published systematic reviews is less than optimal. Language inclusive/LOE systematic reviews appear to be a marker for a better quality systematic review. Language restrictions do not appear to bias the estimates of a conventional intervention's effectiveness. However, there is substantial bias in the results of a CAM systematic review if LOE reports are excluded from it.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There is mixed evidence for the potential impact of not including non-English-language trials: one study found that non-English trials were more likely to produce significant results at p <0.05 (Jüni et al., 2002), whereas an earlier review concluded that excluding non-English trials did not significantly affect the results of meta-analyses (Moher et al., 2003).”
From Self-help interventions to reduce self-stigma in people with mental health problems: A systematic literature review (2020) · cites this paper as Moher et al., 2003
In the citing paper
“In addition, it has been reported that language-restricted meta-analyses only minimally overestimate treatment effects (ϳ2% on average) compared with language-inclusive meta-analyses.112”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Therefore, language-restricted metaanalyses do not appear to lead to biased estimates of intervention effectiveness.112,113”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Over the last two decades, empirical evidence has accumulated from studies investigating different sources of bias related to specific SR tasks.”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
In the citing paper
“Some authors suggested that the impact of excluding non-English language studies may depend on the topic of the review and the quality of non-English language studies [29, 31, 32].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
3 retracted
Daya, S., Maheshwari, A., Siristatidis, C. S., Bhattacharya, S., & Gibreel, A. (2000) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001299 · Cited by 114
RetractedRice-based oral rehydration solution for treating diarrhoea
Fontaine, O., Gore, S., & Pierce, N. (1998) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001264 · Cited by 79
RetractedAndrogens versus placebo or no treatment for idiopathic oligo/asthenospermia
Vandekerckhove, P., Lilford, R. J., Vail, A., & Hughes, E. G. (1996) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000150 · Cited by 30
RetractedThe Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Eisenberg, D. M., Davis, R. B., Ettner, S. L., Appel, S., Wilkey, S., Van Rompay, M., & Kessler, R. C. (1998) · JAMA · doi:10.1001/jama.280.18.1569 · Cited by 6,658
Cited by
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group
Furlan, A. D., Pennick, V., Bombardier, C., & van Tulder, M. (2009) · Spine · doi:10.1097/brs.0b013e3181b1c99f · Cited by 1,663
Morrison, A., Polisena, J., Husereau, D., Moulton, K., Clark, M., Fiander, M., Mierzwinski-Urban, M., Clifford, T., Hutton, B., & Rabb, D. (2012) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462312000086 · Cited by 1,257
“Two reports did not search EMBASE (11;15).”
BackgroundInfluentialEvidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232
“Although exclusion of languages other than English is undesirable, it is considered to be a reasonable practice given limited time and resources and there is some evidence to suggest that it may not markedly bias review findings [9].”
BackgroundSelf-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039
“The funnel plot asymmetry is not statistically significant for inception, regulatory, abstract cohort studies and manuscript cohort studies (see Figure 6). However, there are reasons to suspect the existence of publication and reporting bias in studies of meeting abstracts. A large number of reports of full publicatio…”
Background
Metadata
- Authors
- Moher, D., Pham, B., Lawson, M., & Klassen, T.
- Affiliations
- University of Ottawa (CA)
- Venue
- Health Technology Assessment
- Published
- 2003-12
- Volume
- 7
- Issue
- 41
- Pages
- 1-90
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2119404638
- Cited by
- 500
- References
- 236
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 18 · 2023: 13 · 2022: 33 · 2021: 27 · 2020: 26 · 2019: 26 · 2018: 27 · 2017: 18 · 2016: 22 · 2015: 44 · 2014: 19 · 2013: 24 · 2012: 16
- FWCI
- 2.71
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Health Technology Assessment programme (96/52/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- randomized controlled trials, systematic review quality, complementary and alternative medicine, publication bias, statistical heterogeneity, language of publication, bias in systematic reviews, Cochrane Collaboration
- MeSH terms
- Complementary Therapies, United Kingdom, Humans, Language, Publishing, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The inclusion of reports of randomised trials published in languages other than English in systematic reviews. https://app.cochat.ai/papers/10.3310/hta7410
- The paper
- Moher, D., Pham, B., Lawson, M., & Klassen, T. (2003). The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technology Assessment. https://doi.org/10.3310/hta7410
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.