The inclusion of reports of randomised trials published in languages other than English in systematic reviews
Open accessThe DOI matches its Crossref record; 235 of the 236 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To assemble a large dataset of language restricted and language inclusive systematic reviews, including both conventional medicinal (CM) and complementary and alternative medicine (CAM) interventions. To then assess the quality of these reports by considering and comparing different types of systematic reviews and their associated RCTs; CM and CAM interventions; the effect of language restrictions compared with language inclusions, and whether these results are influenced by other issues, including statistical heterogeneity and publication bias, in the systematic review process. DATA SOURCES: MEDLINE, EMBASE, the Cochrane Database of Systematic Reviews and the Centralised Information Service for Complementary Medicine. REVIEW METHODS: Three types of systematic reviews were included: language restricted; language inclusive/English language (EL) reviews that searched RCTs in languages other than English (LOE) but did not find any and, hence, could not include any, in the quantitative data synthesis; and systematic reviews that searched for RCTs in LOE and included them in the quantitative data synthesis. Fisher's exact test was applied to compare the three different types of systematic reviews with respect to their reporting characteristics and the systematic review quality assessment tool. The odds ratio of LOE trials versus EL trials was computed for each review and this information was pooled across the reviews to examine the influence that language of publication and type of intervention (CM, CAM) have on the estimates of intervention effect. Several sensitivity analyses were performed. RESULTS: The LOE RCTs were predominantly in French and German. Language inclusive/LOE systematic reviews were of the highest quality compared with the other types of reviews. The CAM reviews were of higher quality compared with the CM reviews. There were only minor differences in the quality of reports of EL RCTs compared with the eight other languages considered. However, there are inconsistent differences in the quality of LOE reports depending on the intervention type. The results, and those reported previously, suggest that excluding reports of RCTs in LOE from the analytical part of a systematic review is reasonable. Because the present research and previous efforts have not included every type of CM RCT and the resulting possibility of the uncertainty as to when bias will be present by excluding LOE, it is always prudent to perform a comprehensive search for all evidence. This result only applies to reviews investigating the benefits of CM interventions. This does not imply that systematic reviewers should neglect reports in LOE. We recommend that systematic reviewers search for reports regardless of the language. There may be merit in including them in some aspects of the review process although this decision is likely to depend on several factors, including fiscal and other resources being available. Language restrictions significantly shift the estimates of an intervention's effectiveness when the intervention is CAM. Here, excluding trials reported in LOE, compared with their inclusion, resulted in a reduced intervention effect. The present results do not appear to be influenced by statistical heterogeneity and publication bias. CONCLUSIONS: With the exception of CAM systematic reviews, the quality of recently published systematic reviews is less than optimal. Language inclusive/LOE systematic reviews appear to be a marker for a better quality systematic review. Language restrictions do not appear to bias the estimates of a conventional intervention's effectiveness. However, there is substantial bias in the results of a CAM systematic review if LOE reports are excluded from it.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There is mixed evidence for the potential impact of not including non-English-language trials: one study found that non-English trials were more likely to produce significant results at p <0.05 (Jüni et al., 2002), whereas an earlier review concluded that excluding non-English trials did not significantly affect the results of meta-analyses (Moher et al., 2003).”
From Self-help interventions to reduce self-stigma in people with mental health problems: A systematic literature review (2020) · cites this paper as Moher et al., 2003
In the citing paper
“In addition, it has been reported that language-restricted meta-analyses only minimally overestimate treatment effects (ϳ2% on average) compared with language-inclusive meta-analyses.112”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Therefore, language-restricted metaanalyses do not appear to lead to biased estimates of intervention effectiveness.112,113”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Over the last two decades, empirical evidence has accumulated from studies investigating different sources of bias related to specific SR tasks.”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
In the citing paper
“Some authors suggested that the impact of excluding non-English language studies may depend on the topic of the review and the quality of non-English language studies [29, 31, 32].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
3 retracted
Items 201–235 of 235
Głowacki, Ł., & Smaill, F. (1994) · PubMed · Cited by 51
Browman, G. P. (1994) · Cancer Investigation · doi:10.3109/07357909409023052 · Cited by 48
O’Connor, G. T., Malenka, D. J., Olmstead, E. M., Johnson, P. S., & Hennekens, C. H. (1992) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(18)30830-4 · Cited by 47
Martin‐Hirsch, P. L., Lilford, R. J., & Jarvis, G. J. (1996) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/0301-2115(95)02359-3 · Cited by 46
Pouleur, H. G., & Buyse, M. E. (1995) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/s0022-5223(95)70243-1 · Cited by 45
Cohen, H. J., Silberman, H., Tornyos, K., & Bartolucci, A. (1984) · Blood · doi:10.1182/blood.v63.3.639.639 · Cited by 45
Meta-analysis of hydroxyethylrutosides in the treatment of chronic venous insufficiency.
Poynard, T., & Valterio, C. (1994) · PubMed · Cited by 44
Berry, E., Kelly, Hutton, J., Harris, Roderick, P., Boyce, Cullingworth, Gathercole, O’Connor, P., & Smith, M. A. (1999) · Health Technology Assessment · doi:10.3310/hta3180 · Cited by 43
Lloyd Jones, M., Hummel, S., Bansback, N., Orr, B., & Seymour, M. (2002) · Health Technology Assessment · doi:10.3310/hta5250 · Cited by 42
The Association between Cholesterol and Death from Injury
Cummings, P., & Psaty, B. M. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-10-199405150-00006 · Cited by 42
Garside, R., Round, A., Dalziel, K., Stein, K., & Royle, P. (2002) · Health Technology Assessment · doi:10.3310/hta6330 · Cited by 41
Suri, R., Wallis, C., Bush, A., Thompson, S., Normand, C., Flather, M., Grieve, R., Metcalfe, C., & Lees, B. (2002) · Health Technology Assessment · doi:10.3310/hta6340 · Cited by 40
Nicotine replacement therapies in smoking cessation
Silagy, C. (1994) · Biomedicine & Pharmacotherapy · doi:10.1016/0753-3322(94)90062-0 · Cited by 40
Midgette, A. S., O'Connor, G. T., Baron, J. A., & Bell, J. (1990) · Annals of Internal Medicine · doi:10.7326/0003-4819-113-12-961 · Cited by 39
Immunization as therapy for recurrent spontaneous abortion: A review and meta-analysis
Cowchock, S., & Smith, J. B. (1994) · Obstetrics and Gynecology · doi:10.1097/00006250-199404000-00027 · Cited by 39
Wake, B., Hyde, C., Bryan, S., Barton, P., Song, F., Fry-Smith, A., & Davenport, C. F. (2002) · Health Technology Assessment · doi:10.3310/hta6030 · Cited by 37
Interferon-α Treatment in Patients with Chronic Hepatitis C
Malaguarnera, M., Restuccia, S., Trovato, G. M., Siciliano, R., Motta, M., & Trovato, B. A. (1995) · Clinical Drug Investigation · doi:10.2165/00044011-199509030-00002 · Cited by 36
Low-dose aspirin: Lack of association with an increase in abruptio placentae or perinatal mortality
HAUTH, J., GOLDENBERG, R., RICHARDPARKERJR, C., CUTTER, G., & CLIVER, S. (1995) · Obstetrics & Gynecology · doi:10.1016/0029-7844(95)00042-p · Cited by 36
Figueredo, E. D., & Canosa, L. (1999) · Acta Anaesthesiologica Scandinavica · doi:10.1034/j.1399-6576.1999.430608.x · Cited by 35
An introduction to statistical methods for health technology assessment.
White, Ashby, & Brown (2000) · Health Technology Assessment · doi:10.3310/hta4080 · Cited by 31
Meta-Analysis of the Effect of Placebo on the Outcome of Medically Treated Reflux Esophagitis
Pace, F., Maconi, G., Molteni, P., Minguzzi, M., & Porro, G. B. (1995) · Scandinavian Journal of Gastroenterology · doi:10.3109/00365529509093245 · Cited by 30
Peters, J., Stevenson, M., Beverley, C., Lim, J. N. W., & Smith, S. L. (2002) · Health Technology Assessment · doi:10.3310/hta6050 · Cited by 30
Meunier, F., Paesmans, M., & Autier, P. (1994) · European Journal of Cancer Part B Oral Oncology · doi:10.1016/0964-1955(94)90091-4 · Cited by 29
A meta-analysis of the use of amoxycillin-clavulanic acid in surgical prophylaxis
Wilson, A. P. R., Shrimpton, S. B., & Jaderberg, M. (1992) · Journal of Hospital Infection · doi:10.1016/s0195-6701(05)80003-9 · Cited by 28
Di Mascio, R., Marchioli, R., & Tognoni, G. (1994) · PubMed · Cited by 26
Hooker, K. D., DiPiro, J. T., & Wynn, J. J. (1991) · PubMed · Cited by 25
An Analysis of Review Articles Published in Primary Care Journals
Silagy, C. (1993) · Family Practice · doi:10.1093/fampra/10.3.337 · Cited by 20
Meta-analysis of streptokinase and heparin in deep vein thrombosis
Ng, C. M., & Rivera, J. O. (1998) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/55.19.1995 · Cited by 19
Hansen, J. F. (1994) · Cardiovascular Drugs and Therapy · doi:10.1007/bf00877222 · Cited by 18
Avgerinos, A., Armonis, A., & Raptis, S. A. (1995) · Journal of Hepatology · doi:10.1016/0168-8278(95)80436-6 · Cited by 17
Righetti, E., Celani, M., Cantisani, T. A., Sterzi, R., Boysen, G., & Ricci, S. (2000) · doi:10.1002/14651858.cd000096 · Cited by 13
Blondel, B., & Bréart, G. (1992) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.1992.tb13723.x · Cited by 11
Prevention of Auditory Sequelae in Pediatric Bacterial Meningitis: A Meta‐Analysis
Yurkowski, P. J., & Plaisance, K. I. (1993) · Pharmacotherapy The Journal of Human Pharmacology and Drug Therapy · doi:10.1002/j.1875-9114.1993.tb04313.x · Cited by 10
Intercessory Prayer for ill Health: A Systematic Review
Roberts, L., Ahmed, I. I. K., Hall, S., Sargent, C., & Adams, C. E. (1998) · Complementary Medicine Research · doi:10.1159/000057115 · Cited by 9
Tholl, D. A., Miller, T. Q., Henderson, W. G., & Myers, T. F. (1994) · PubMed · Cited by 4
Cited by
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group
Furlan, A. D., Pennick, V., Bombardier, C., & van Tulder, M. (2009) · Spine · doi:10.1097/brs.0b013e3181b1c99f · Cited by 1,663
Morrison, A., Polisena, J., Husereau, D., Moulton, K., Clark, M., Fiander, M., Mierzwinski-Urban, M., Clifford, T., Hutton, B., & Rabb, D. (2012) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462312000086 · Cited by 1,257
“Two reports did not search EMBASE (11;15).”
BackgroundInfluentialEvidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232
“Although exclusion of languages other than English is undesirable, it is considered to be a reasonable practice given limited time and resources and there is some evidence to suggest that it may not markedly bias review findings [9].”
BackgroundSelf-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039
“The funnel plot asymmetry is not statistically significant for inception, regulatory, abstract cohort studies and manuscript cohort studies (see Figure 6). However, there are reasons to suspect the existence of publication and reporting bias in studies of meeting abstracts. A large number of reports of full publicatio…”
Background
Metadata
- Authors
- Moher, D., Pham, B., Lawson, M., & Klassen, T.
- Affiliations
- University of Ottawa (CA)
- Venue
- Health Technology Assessment
- Published
- 2003-12
- Volume
- 7
- Issue
- 41
- Pages
- 1-90
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2119404638
- Cited by
- 500
- References
- 236
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 18 · 2023: 13 · 2022: 33 · 2021: 27 · 2020: 26 · 2019: 26 · 2018: 27 · 2017: 18 · 2016: 22 · 2015: 44 · 2014: 19 · 2013: 24 · 2012: 16
- FWCI
- 2.71
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Health Technology Assessment programme (96/52/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- randomized controlled trials, systematic review quality, complementary and alternative medicine, publication bias, statistical heterogeneity, language of publication, bias in systematic reviews, Cochrane Collaboration
- MeSH terms
- Complementary Therapies, United Kingdom, Humans, Language, Publishing, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.