The inclusion of reports of randomised trials published in languages other than English in systematic reviews
Open accessThe DOI matches its Crossref record; 235 of the 236 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To assemble a large dataset of language restricted and language inclusive systematic reviews, including both conventional medicinal (CM) and complementary and alternative medicine (CAM) interventions. To then assess the quality of these reports by considering and comparing different types of systematic reviews and their associated RCTs; CM and CAM interventions; the effect of language restrictions compared with language inclusions, and whether these results are influenced by other issues, including statistical heterogeneity and publication bias, in the systematic review process. DATA SOURCES: MEDLINE, EMBASE, the Cochrane Database of Systematic Reviews and the Centralised Information Service for Complementary Medicine. REVIEW METHODS: Three types of systematic reviews were included: language restricted; language inclusive/English language (EL) reviews that searched RCTs in languages other than English (LOE) but did not find any and, hence, could not include any, in the quantitative data synthesis; and systematic reviews that searched for RCTs in LOE and included them in the quantitative data synthesis. Fisher's exact test was applied to compare the three different types of systematic reviews with respect to their reporting characteristics and the systematic review quality assessment tool. The odds ratio of LOE trials versus EL trials was computed for each review and this information was pooled across the reviews to examine the influence that language of publication and type of intervention (CM, CAM) have on the estimates of intervention effect. Several sensitivity analyses were performed. RESULTS: The LOE RCTs were predominantly in French and German. Language inclusive/LOE systematic reviews were of the highest quality compared with the other types of reviews. The CAM reviews were of higher quality compared with the CM reviews. There were only minor differences in the quality of reports of EL RCTs compared with the eight other languages considered. However, there are inconsistent differences in the quality of LOE reports depending on the intervention type. The results, and those reported previously, suggest that excluding reports of RCTs in LOE from the analytical part of a systematic review is reasonable. Because the present research and previous efforts have not included every type of CM RCT and the resulting possibility of the uncertainty as to when bias will be present by excluding LOE, it is always prudent to perform a comprehensive search for all evidence. This result only applies to reviews investigating the benefits of CM interventions. This does not imply that systematic reviewers should neglect reports in LOE. We recommend that systematic reviewers search for reports regardless of the language. There may be merit in including them in some aspects of the review process although this decision is likely to depend on several factors, including fiscal and other resources being available. Language restrictions significantly shift the estimates of an intervention's effectiveness when the intervention is CAM. Here, excluding trials reported in LOE, compared with their inclusion, resulted in a reduced intervention effect. The present results do not appear to be influenced by statistical heterogeneity and publication bias. CONCLUSIONS: With the exception of CAM systematic reviews, the quality of recently published systematic reviews is less than optimal. Language inclusive/LOE systematic reviews appear to be a marker for a better quality systematic review. Language restrictions do not appear to bias the estimates of a conventional intervention's effectiveness. However, there is substantial bias in the results of a CAM systematic review if LOE reports are excluded from it.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There is mixed evidence for the potential impact of not including non-English-language trials: one study found that non-English trials were more likely to produce significant results at p <0.05 (Jüni et al., 2002), whereas an earlier review concluded that excluding non-English trials did not significantly affect the results of meta-analyses (Moher et al., 2003).”
From Self-help interventions to reduce self-stigma in people with mental health problems: A systematic literature review (2020) · cites this paper as Moher et al., 2003
In the citing paper
“In addition, it has been reported that language-restricted meta-analyses only minimally overestimate treatment effects (ϳ2% on average) compared with language-inclusive meta-analyses.112”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Therefore, language-restricted metaanalyses do not appear to lead to biased estimates of intervention effectiveness.112,113”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Over the last two decades, empirical evidence has accumulated from studies investigating different sources of bias related to specific SR tasks.”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
In the citing paper
“Some authors suggested that the impact of excluding non-English language studies may depend on the topic of the review and the quality of non-English language studies [29, 31, 32].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
3 retracted
Daya, S., Maheshwari, A., Siristatidis, C. S., Bhattacharya, S., & Gibreel, A. (2000) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001299 · Cited by 114
RetractedRice-based oral rehydration solution for treating diarrhoea
Fontaine, O., Gore, S., & Pierce, N. (1998) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001264 · Cited by 79
RetractedAndrogens versus placebo or no treatment for idiopathic oligo/asthenospermia
Vandekerckhove, P., Lilford, R. J., Vail, A., & Hughes, E. G. (1996) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000150 · Cited by 30
RetractedThe Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Eisenberg, D. M., Davis, R. B., Ettner, S. L., Appel, S., Wilkey, S., Van Rompay, M., & Kessler, R. C. (1998) · JAMA · doi:10.1001/jama.280.18.1569 · Cited by 6,658
Cited by
Items 101–200 of 499
Grootenboer, N., van Sambeek, M. R. H. M., Arends, L. R., Hendriks, J. M., Hunink, M. G. M., & Bosch, J. (2010) · British journal of surgery · doi:10.1002/bjs.7134 · Cited by 93
Ruptured Abdominal Aortic Aneurysms: Endovascular Repair versus Open Surgery—Systematic Review
Visser, J. J., van Sambeek, M. R. H. M., Hamza, T. H., Hunink, M. G. M., & Bosch, J. L. (2007) · Radiology · doi:10.1148/radiol.2451061204 · Cited by 95
Bond, M., Pitt, M., Akoh, J. A., Moxham, T., Hoyle, M., & Anderson, R. (2009) · Health Technology Assessment · doi:10.3310/hta13380 · Cited by 91
Vitamin K to prevent fractures in older women: systematic review and economic evaluation
Stevenson, M., Lloyd-Jones, M., & Papaioannou, D. (2009) · Health Technology Assessment · doi:10.3310/hta13450 · Cited by 90
Guerra, P. H., Nobre, M. R. C., da Silveira, J. A. C., & Taddei, J. A. D. A. C. (2013) · Clinics · doi:10.6061/clinics/2013(09)14 · Cited by 111
“However, this limitation does not usually represent a significant bias in terms of estimating the effects of conventional interventions (48).”
BackgroundGrossi, U., Tanna, G. L. D., Heinrich, H. S., Taylor, S. A., Knowles, C. H., & Scott, S. M. (2018) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.15039 · Cited by 92
“literature searches limited to the English language publications do not affect the quality of systematic reviews.(50) Only peer‐reviewed publications reporting primary data were eli-”
Cooke, M. R., Marsh, J. L., Clark, M., Nakash, R., Jarvis, R., Hutton, J. L., Szczepura, A. K., Wilson, S., & Lamb, S. E. (2009) · Health Technology Assessment · doi:10.3310/hta13130 · Cited by 95
Mulder, R. L., Kremer, L. C. M., van Santen, H. M., Ket, J. L., van Trotsenburg, A. S. P., Koning, C. C. E., Meeteren, A. Y. N. S., Caron, H. N., Neggers, S. J. C. M. M., & van Dalen, E. C. (2009) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2009.06.004 · Cited by 102
“However, systematic reviews about prognostic studies are a rather new field and it is unclear what the additional value of studies in other languages than English is.(54) The association between the maximal CRT dose and the prevalence of GHD as identified in the meta-regression analysis could also be the result of con…”
BackgroundChase, D., Rosten, C., Turner, S., Hicks, N., & Milne, R. (2009) · Health Technology Assessment · doi:10.3310/hta13590 · Cited by 89
Jia, Y., Huang, H., & Gagnier, J. J. (2017) · Quality of Life Research · doi:10.1007/s11136-017-1542-4 · Cited by 91
“Also, it appears that systematic reviews and meta-analyses of clinical trials in conventional medicine do not have biased estimates of treatment effect when they exclude non-English language studies [165, 166].”
BackgroundWorster, A., Balion, C. M., Hill, S., Santaguida, P. L., Ismaila, A., McKelvie, R. S., Reichert, S. M., McQueen, M., Booker, L., & Raina, P. S. (2007) · Clinical Biochemistry · doi:10.1016/j.clinbiochem.2007.08.008 · Cited by 89
“3% of the 4338 screened met all of the inclusion criteria, the probability of any of the omitted non-English studies meeting the same criteria is low and inclusion is unlikely to impact on the results [42].”
BackgroundO'Shea, L., Mitchell, A., Picchioni, M., & Dickens, G. (2013) · Aggression and Violent Behavior · doi:10.1016/j.avb.2012.11.016 · Cited by 77
Raggio, D. P., Hesse, D., Lenzi, T. L., Guglielmi, C. A. B., & Braga, M. M. (2012) · International Journal of Paediatric Dentistry · doi:10.1111/ipd.12013 · Cited by 106
Challenges in Systematic Reviews of Complementary and Alternative Medicine Topics
Shekelle, P. G., Morton, S. C., Suttorp, M. J., Buscemi, N., & Friesen, C. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-12_part_2-200506211-00003 · Cited by 88
“Thus, all else being equal, the exclusion of non–English-language reports may result in a lower estimate of intervention effect for CAM interventions than would be seen if these reports were included (10).”
Hummel, S., Simpson, E., Hemingway, P., Stevenson, M., & Rees, A. (2010) · Health Technology Assessment · doi:10.3310/hta14470 · Cited by 97
Screening for pancreatic cancer in familial high-risk individuals: A systematic review
Lu, C. (2015) · World Journal of Gastroenterology · doi:10.3748/wjg.v21.i28.8678 · Cited by 94
Prevention of Poststroke Depression: Does Prophylactic Pharmacotherapy Work?
Salter, K. L., Foley, N. C., Zhu, L., Jutai, J. W., & Teasell, R. W. (2012) · Journal of Stroke and Cerebrovascular Diseases · doi:10.1016/j.jstrokecerebrovasdis.2012.03.013 · Cited by 93
“7/75 #6 mos Fluoxetine (10-40), nortriptyline (25-100) 3 mos HRSD17, DSM-IV NR At end of treatment, 1 minor depression was reported in the nortriptyline group, 3 in the fluoxetine group, and 5 in the placebo group Rasmussen et al(20) (2003) 70/67 72/68 50/50.”
BackgroundPregnancy and Neonatal Outcomes of Aboriginal Women: A Systematic Review and Meta-Analysis
Shah, P. S., Zao, J. H., Al-Wassia, H., & Shah, V. S. (2010) · Women s Health Issues · doi:10.1016/j.whi.2010.08.005 · Cited by 86
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85
Pilgrim, H., Lloyd-Jones, M., & Rees, A. (2009) · Health Technology Assessment · doi:10.3310/hta13100 · Cited by 102
Meadows, E., & Saux, N. L. (2004) · BMC Public Health · doi:10.1186/1471-2458-4-50 · Cited by 83
“2003 indicate that the exclusion of trials in languages other than English (LOE) does not bias measures of effectiveness- however, both are cautionary, advocating language inclusive search strategies [18,19].”
BackgroundAcedillo, R. R., Shah, M., Devereaux, P. J., Li, L., Iansavichus, A. V., Walsh, M. W., & Garg, A. X. (2013) · Annals of Surgery · doi:10.1097/sla.0000000000000244 · Cited by 93
Attar, W. S. A. A., Khaledi, E. H., Bakhsh, J. M., Faude, O., Ghulam, H. S. H., & Sanders, R. H. (2022) · Journal of physiotherapy · doi:10.1016/j.jphys.2022.05.019 · Cited by 83
Casado-Robles, C., Viciana, J., Guijarro-Romero, S., & Mayorga-Vega, D. (2022) · Sports Medicine - Open · doi:10.1186/s40798-021-00407-6 · Cited by 106
Colbourn, T., Asseburg, C., Bojke, L., Philips, Z., Claxton, K., Ades, A. E., & Gilbert, R. E. (2007) · Health Technology Assessment · doi:10.3310/hta11290 · Cited by 83
Clegg, A. J., Loveman, E., Gospodarevskaya, E., Harris, P. S., Bird, A., Bryant, J., Scott, D. A., Davidson, P., Little, P. S., & Coppin, R. (2010) · Health Technology Assessment · doi:10.3310/hta14280 · Cited by 82
Tselios, K., Sheane, B. J., Gladman, D. D., & Urowitz, M. B. (2015) · The Journal of Rheumatology · doi:10.3899/jrheum.150460 · Cited by 84
Few systematic reviews exist documenting the extent of bias: a systematic review
Tricco, A. C., Tetzlaff, J., Sampson, M., Fergusson, D. A., Cogo, E., Horsley, T., & Moher, D. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.10.017 · Cited by 93
Self-citationMoher, D., Soeken, K. L., Sampson, M., Ben‐Porat, L., & Berman, B. (2002) · BMC Pediatrics · doi:10.1186/1471-2431-2-3 · Cited by 89
Self-citationEfficacy of ω-3 Fatty Acids in Preventing Age-Related Macular Degeneration
Hodge, W., Schachter, H., Barnes, D., Pan, Y., Lowcock, E. C., Zhang, L., Sampson, M., Morrison, A., Tran, K. T., Miguelez, M., et al. (2006) · Ophthalmology · doi:10.1016/j.ophtha.2006.02.043 · Cited by 85
Quality of systematic reviews in pediatric oncology – A systematic review
Lundh, A., Knijnenburg, S. L., Jørgensen, A. W., van Dalen, E. C., & Kremer, L. C. M. (2009) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2009.08.010 · Cited by 78
Efficacy of Directional Preference Management for Low Back Pain: A Systematic Review
Surkitt, L. D., Ford, J. J., Hahne, A. J., Pizzari, T., & McMeeken, J. M. (2012) · Physical Therapy · doi:10.2522/ptj.20100251 · Cited by 79
Abdullah, G., Rossy, D., Ploeg, J., Davies, B., Higuchi, K. S., Sikora, L., & Stacey, D. (2014) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12060 · Cited by 97
Li, T., Vedula, S. S., Scherer, R. W., & Dickersin, K. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-5-201203060-00009 · Cited by 83
Autoantibody testing in children with newly diagnosed type 1 diabetes mellitus
Dretzke, J., Cummins, C., Sandercock, J., Fry-Smith, A., Barrett, T. G., & Burls, A. (2004) · Health Technology Assessment · doi:10.3310/hta8220 · Cited by 77
Mills, H., Mulfinger, N., Raeder, S., Rüsch, N., Clements, H., & Scior, K. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2019.112702 · Cited by 73
“earlier review concluded that excluding non-English trials did not significantly affect the results of meta-analyses (Moher et al., 2003).”
BackgroundHowell, D., Fitch, M. I., Bakker, D. A., Green, E., SUSSMAN, J. S., Mayo, S. J., Mohammed, S., Lee, C., & Doran, D. M. (2013) · Value in Health · doi:10.1016/j.jval.2012.10.017 · Cited by 78
“[79] Moher D, Pham B, Lawson ML, Klassen TP.”
BackgroundPeterson, M. C., Syndergaard, T., Bowler, J., & Doxey, R. S. (2011) · International Journal of Cardiology · doi:10.1016/j.ijcard.2011.06.042 · Cited by 72
“discussed this problemthoroughly in apaperon inclusionof English only literature in systematic reviews [103].”
BackgroundNeedleman, I. G., Chin, S., O'Brien, T., Petrie, A., & Donos, N. (2012) · Journal Of Clinical Periodontology · doi:10.1111/j.1600-051x.2011.01836.x · Cited by 79
“Conflicting evidence exists on the impact of language bias (Jüni et al. 2002, Moher et al. 2003).”
BackgroundOsho, O., Owoeye, O. B. A., & Armijo-Olivo, S. L. (2017) · Journal of Aging and Physical Activity · doi:10.1123/japa.2016-0326 · Cited by 97
Sutton, D. A., Côté, P., Wong, J. J., Varatharajan, S., Randhawa, K. A., Yu, H., Southerst, D., Shearer, H. M., van der Velde, G., Nordin, M. C., et al. (2014) · The Spine Journal · doi:10.1016/j.spinee.2014.06.019 · Cited by 87
“Other systematic reviews reported similar results when studying the effect of language restrictions in conventional medicine [80–83].”
ResultGuerra, P. H., Nobre, M. R. C., da Silveira, J. A. C., & Taddei, J. A. A. C. (2014) · Preventive Medicine · doi:10.1016/j.ypmed.2014.01.005 · Cited by 81
“observed that in spite of the strengths of studies published in French and German, language restrictions are not a source of bias in the estimation of the effectiveness of conventional interventions (Moher et al., 2003).”
ResultBurns, M. J., & O'Connor, A. M. C. (2007) · Vaccine · doi:10.1016/j.vaccine.2007.10.014 · Cited by 69
Jacobs, M., Macefield, R. C., Elbers, R. G., Sitnikova, K., Korfage, I. J., Smets, E. M. A., Henselmans, I., Henegouwen, M. I. V. B., de Haes, J. C. J. M., Blazeby, J. M., et al. (2013) · Quality of Life Research · doi:10.1007/s11136-013-0576-5 · Cited by 72
“Third, we omitted non-English studies, because we believe the potential impact of this selection criterion to be limited [68].”
MethodEfficacy and safety of acupuncture for idiopathic Parkinson’s disease: a systematic review
Lam, Y. C., Kum, W. F., Durairajan, S. S. K., Lu, J. H., Man, S. C., Xu, M., Zhang, X. F., Huang, X. Z., & Li, M. (2008) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2007.0011 · Cited by 69
“Because publication bias is more common in Chinese trials as mentioned in previous studies, 30,31 there is a general lack of well-designed research to evaluate the efficacy and safety of this therapy to treat specific conditions.”
Aldrigui, J. M., Jabbar, N. S., Bonecker, M., Braga, M. M., & Wanderley, M. T. (2014) · Community Dentistry and Oral Epidemiology · doi:10.1111/cdoe.12053 · Cited by 69
Bruls, V. E. J., Bastiaenen, C. H. G., & de Bie, R. A. (2015) · Pain · doi:10.1097/j.pain.0000000000000117 · Cited by 71
“Table 4 (continued) Study F. Putative prognostic factors G. Outcome measures H. Positive findings* (multivariate results) I. Strength of association (95% CI) distress, somatization, kinesiophobia, catastrophizing, social support, job insecurity, coworker support, supervisor support, quantitative job demands, skill dis…”
BackgroundInfluentialBankhead, C. R., Brett, J., Bukach, C., Webster, P., Stewart-Brown, S. L., Munafo, M. R., & Austoker, J. (2003) · Health Technology Assessment · doi:10.3310/hta7420 · Cited by 76
Williamson, I. G., Benge, S., Barton, S. J., Petrou, S., Letley, L. H., Fasey, N., Abangma, G., Dakin, H., & Little, P. S. (2009) · Health Technology Assessment · doi:10.3310/hta13370 · Cited by 66
Kai, J., Ulph, F., Cullinan, T., & Qureshi, N. (2009) · Health Technology Assessment · doi:10.3310/hta13570 · Cited by 72
Serum folate levels in schizophrenia: A meta-analysis
Wang, D., Zhai, J.-X., & Liu, D. (2015) · Psychiatry Research · doi:10.1016/j.psychres.2015.11.045 · Cited by 67
“Some studies investigated this issue by comparison on included English and non-English language trails, and got conflicting results (Gregoire et al., 1995; Moher et al., 1996, 2003; Juni et al., 2002).”
Safety of transcranial direct current stimulation: Evidence based update 2016
Godinho, M. M., Junqueira, D. R., Castro, M. L., Loke, Y. K., Golder, S., & Neto, H. P. (2017) · Brain stimulation · doi:10.1016/j.brs.2017.07.001 · Cited by 76
“The language limit of the search to the English language might or might not be another source of bias [5,6].”
BackgroundMichaels, J. A., Campbell, W., King, B., MacIntyre, J., Palfreyman, S. J., Shackley, P., & Stevenson, M. (2009) · Health Technology Assessment · doi:10.3310/hta13560 · Cited by 66
Hadi, M. A., Alldred, D. P., Briggs, M., Munyombwe, T., & Closs, S. J. (2014) · Clinical Journal of Pain · doi:10.1097/ajp.0000000000000063 · Cited by 76
Fleeman, N., McLeod, C., Bagust, A., Beale, S., Boland, A., Dundar, Y., Jorgensen, A., Payne, K., Pirmohamed, M., Pushpakom, S. P., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14030 · Cited by 72
Complementary and alternative medicine in obstetrics
Anderson, F. W. J., & Johnson, C. T. (2005) · International Journal of Gynecology & Obstetrics · doi:10.1016/j.ijgo.2005.07.009 · Cited by 74
“Additionally, this survey is limited to the studies in the English language, which can limit a full assessment of the spectrum of CAM in obstetrics [71].”
BackgroundSensory retraining of the leg after stroke: systematic review and meta-analysis
Chia, F. S., Kuys, S. S., & Choy, N. L. (2019) · Clinical Rehabilitation · doi:10.1177/0269215519836461 · Cited by 70
Meta-Analysis of Trials of Streptococcal Throat Treatment Programs to Prevent Rheumatic Fever
LENNON, D. R., Kerdemelidis, M., & Arroll, B. (2009) · The Pediatric Infectious Disease Journal · doi:10.1097/inf.0b013e3181a8e12a · Cited by 65
Hay, A. D., Redmond, N. M., Costelloe, C., Montgomery, A. A., Fletcher, M. E., Hollinghurst, S. P., & Peters, T. (2009) · Health Technology Assessment · doi:10.3310/hta13270 · Cited by 67
Shepherd, J., Rogers, G., Anderson, R., Main, C., Coon, J. T., Hartwell, D., Liu, Z., Loveman, E., Green, C., Pitt, M., et al. (2008) · Health Technology Assessment · doi:10.3310/hta12190 · Cited by 68
DANIELS, J. P., Gray, J. W., Pattison, H. M., Roberts, T. E., Edwards, E. A., Milner, P., Spicer, L., King, E., Hills, R. K., Gray, R. G., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13420 · Cited by 75
Esber, A. L., Miguel, R. D. V., Cherpes, T. L., Klebanoff, M. A., Gallo, M. F., & Turner, A. N. (2015) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiv017 · Cited by 64
“While this restriction may have led to the exclusion of potentially relevant findings, prior research illustrates that excluding non–English-language reports does not lead to significant differences in the summary estimates reported in meta-analyses [43].”
ResultMcLean, G., Murray, E., Band, R., Moffat, K. R., Hanlon, P., Bruton, A., Thomas, M., Yardley, L., & Mair, F. S. (2016) · BMC Pulmonary Medicine · doi:10.1186/s12890-016-0248-7 · Cited by 63
“Finally, we only considered studies published in peerreviewed journals in English as evidence suggests that limiting studies in this way does not introduce significant bias [24] but does save considerable resource.”
BackgroundRoffey, D. M., Wai, E. K., Bishop, P., Kwon, B. K., & Dagenais, S. (2010) · The Spine Journal · doi:10.1016/j.spinee.2010.03.025 · Cited by 82
Comparison of case note review methods for evaluating quality and safety in health care
Lock, C. A., Wilson, J. A., Steen, N., Eccles, M., Mason, H., Carrie, S., Clarke, R. W., Kubba, H., Raine, C. H., Zarod, A., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14130 · Cited by 63
O'Connor, A. M. C., Denagamage, T., Sargeant, J. M., Rajić, A., & McKean, J. D. (2008) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2008.06.017 · Cited by 63
Robson, S., Kelly, T., Howel, Deverill, M., Hewison, J., Lie, M., Stamp, E., Armstrong, N., & May, C. (2009) · Health Technology Assessment · doi:10.3310/hta13530 · Cited by 62
Coleman, T. J., Agboola, S., Leonardi‐Bee, J., Taylor, M. J., McEwen, A., & McNeill, A. D. (2010) · Health Technology Assessment · doi:10.3310/hta14490 · Cited by 74
Non-occupational postexposure prophylaxis for HIV: a systematic review
Bryant, J., Baxter, L., & Hird, S. (2009) · Health Technology Assessment · doi:10.3310/hta13140 · Cited by 62
Taylor, R. S., & Elston, J. (2009) · Health Technology Assessment · doi:10.3310/hta13080 · Cited by 63
Screening for hyperglycaemia in pregnancy: a rapid update for the National Screening Committee
Waugh, N. R., Royle, P., Clar, C., Henderson, R., Cummins, E., Hadden, D. R., Lindsay, R. S., & Pearson, D. (2010) · Health Technology Assessment · doi:10.3310/hta14450 · Cited by 67
Mindfulness-based stress reduction in Parkinson’s disease: a systematic review
McLean, G., Lawrence, M., Simpson, R., & Mercer, S. W. (2017) · BMC Neurology · doi:10.1186/s12883-017-0876-4 · Cited by 59
“Finally, we only considered studies published in peer-reviewed journals in English, as evidence suggests that limiting studies in this way does not introduce significant bias but is associated with considerable resource savings [21].”
BackgroundMcKenna, C., McDaid, C., Suekarran, S., Hawkins, N., Claxton, K., Light, K., Chester, M., Cleland, J. G. F., Woolacott, N. F., & Sculpher, M. (2009) · Health Technology Assessment · doi:10.3310/hta13240 · Cited by 66
Wong, J. J., Côté, P., Ameis, A., Varatharajan, S., Varatharajan, T., Shearer, H. M., Brison, R. J., Sutton, D. A., Randhawa, K. A., Yu, H., et al. (2015) · European Spine Journal · doi:10.1007/s00586-015-3891-4 · Cited by 71
“We only searched the English literature; however, systematic reviews of clinical trials investigating the impact of language restriction found that it does not lead to bias as most large trials are published in English [77–81].”
BackgroundThornley, P., Evaniew, N., Riediger, M., Winemaker, M., Bhandari, M., & Ghert, M. A. (2015) · CMAJ Open · doi:10.9778/cmajo.20150012 · Cited by 58
Hurwitz, E. L., Randhawa, K. A., Torres, P., Yu, H., Verville, L., Hartvigsen, J., Côté, P., & Haldeman, S. (2017) · European Spine Journal · doi:10.1007/s00586-017-5393-z · Cited by 73
“Previous systematic reviews of clinical trials have found that English language restrictions do not appear to bias expected results [63, 64].”
BackgroundPharmacologic Treatments for Sleep Disorders in Children: A Systematic Review
McDonagh, M. S., Holmes, R. S., & Hsu, F. C. (2019) · Journal of Child Neurology · doi:10.1177/0883073818821030 · Cited by 64
Danazumi, M. S., Lightbody, N., & Dermody, G. (2024) · Osteoporosis International · doi:10.1007/s00198-024-07052-1 · Cited by 75
Cahill, J., Barkham, M., Hardy, G. E., Gilbody, S. M., Richards, D. A., Bower, P., Audin, K., & Connell, J. (2008) · Health Technology Assessment · doi:10.3310/hta12240 · Cited by 61
Alfageh, B. H., Wang, Z., Mongkhon, P., Besag, F., Alhawassi, T. M., Brauer, R., & Wong, I. C. K. (2019) · Pediatric Drugs · doi:10.1007/s40272-019-00333-x · Cited by 68
Licina, A., Silvers, A. J., & Stuart, R. L. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01431-5 · Cited by 58
“likely to result in publication bias [32].”
Klassen, T. P., Pham, B., Lawson, M. L., & Moher, D. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.08.020 · Cited by 59
“We used an interobserver reliability study to standardize ourselves in using these instruments [8].”
MethodSelf-citationKhan, K. S., Zamora, J., Lamont, R., van Geijn, H. P., Svare, J., Santos-Jorge, C., Jacquemyn, Y., Husslein, P. W., Helmer, H., Dudenhausen, J. W., et al. (2010) · The Journal of Maternal-Fetal & Neonatal Medicine · doi:10.3109/14767050903572182 · Cited by 61
Murphy, C. P., Carson, C. A., Smith, B. A., Chapman, B., Marrotte, J., McCann, M., Primeau, C. A., Sharma, P., & Parmley, E. J. (2018) · Zoonoses and Public Health · doi:10.1111/zph.12515 · Cited by 68
POTTER, J. F., Mistri, A. K., Brodie, F. G., Tchernova, J., Wilson, E., Jagger, C., James, M. A., Ford, G. A., & Robinson, T. G. (2009) · Health Technology Assessment · doi:10.3310/hta13090 · Cited by 58
The effect of hawthorn (Crataegus spp.) on blood pressure: A systematic review
Cloud, A., Vilcins, D., & McEwen, B. J. (2019) · Advances in Integrative Medicine · doi:10.1016/j.aimed.2019.09.002 · Cited by 67
Makarounas‐Kirchmann, K., Glover-Koudounas, S., & Ferrari, P. (2009) · Clinical Therapeutics · doi:10.1016/j.clinthera.2009.08.010 · Cited by 65
“В то же время, ранее было продемонстрировано, что языковые ограничения при проведении систематических обзоров не приводят к систематической ошибке в оценке лекарственной эффективности [33].”
A systematic review of the effects of acupuncture on xerostomia and hyposalivation
Assy, Z., & Brand, H. S. (2018) · BMC Complementary and Alternative Medicine · doi:10.1186/s12906-018-2124-x · Cited by 69
“However, the results of systematic reviews of complementary and alternative medicine do substantially alter when language restrictions are used [55, 56].”
BackgroundTegen, D., Damtie, D., & Hailegebriel, T. (2020) · Journal of Parasitology Research · doi:10.1155/2020/8884064 · Cited by 66
“Articles written in English were searched from online public databases, namely, PubMed/ MEDLINE, ScienceDirect, Web of Science, Google Scholar, Hinari, WorldCat, and Cochrane Library [22], using core search terms and phrases: “prevalence,” “intestinal protozoan parasite,” “associated factors,” and “Ethiopia.””
MethodTyrer, P. J., Oliver, P., Romeo, R., Knapp, M. R. J., Huline‐Dickens, S., Bouras, N., Ahmed, M. Z., Cooray, S., Deb, S. S., Murphy, D., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13210 · Cited by 56
Spinal Manipulation or Mobilization for Radiculopathy: A Systematic Review
Leininger, B. D., Brønfort, G., Evans, R., & Reiter, T. (2010) · Physical Medicine and Rehabilitation Clinics of North America · doi:10.1016/j.pmr.2010.11.002 · Cited by 65
Côté, P., Hartvigsen, J., Axén, I., Leboeuf‐Yde, C., Corso, M., Shearer, H. M., Wong, J. J., Marchand, A.-A., Cassidy, J. D., French, S., et al. (2021) · Chiropractic & Manual Therapies · doi:10.1186/s12998-021-00362-9 · Cited by 57
Oliveira, C., Barbosa, B., Couto, J. G., Bravo, I., Khine, R., & McNair, H. A. (2022) · Radiography · doi:10.1016/j.radi.2022.04.009 · Cited by 61
Bodalia, P. N., Grosso, A., Sofat, R., MacAllister, R. J., Smeeth, L. L., Dhillon, S., Casas, J. P., Wonderling, D. E., & Hingorani, A. D. (2013) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.12083 · Cited by 57
Haby, M. M., Chapman, E., Clark, R., Barreto, J. O. M., Revéiz, L., & Lavis, J. N. (2015) · Implementation Science · doi:10.1186/s13012-016-0472-9 · Cited by 57
Piper, S., Shearer, H. M., Côté, P., Wong, J. J., Yu, H., Varatharajan, S., Southerst, D., Randhawa, K. A., Sutton, D. A., Stupar, M., et al. (2015) · Manual Therapy · doi:10.1016/j.math.2015.08.011 · Cited by 65
“studies are published in English (Moher et al., 1996; Sutton et al., 2000; Juni et al., 2002; Moher et al., 2003; Morrison et al., 2012).”
The effectiveness of sub-group specific manual therapy for low back pain: A systematic review
Slater, S. L., Ford, J. J., Richards, M. C., Taylor, N. F., Surkitt, L. D., & Hahne, A. J. (2012) · Manual Therapy · doi:10.1016/j.math.2012.01.006 · Cited by 59
Bhurke, S., Kwok, C. S., Pang, C. L., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., Harvey, I. M., & Song, F. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.022 · Cited by 61
“(2003) [18] examined 42 meta-analyses to investigate whether different estimates of treatment effect were obtained in meta-analyses restricted to English-language studies as compared with those without this restriction.”
BackgroundHemingway, H., Henriksson, M., Chen, R., Damant, J., Fitzpatrick, N. K., Abrams, K. R., Hingorani, A. D., Janzon, M., Shipley, M. J., Feder, G. S., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14090 · Cited by 55
Rabi, S., Evaniew, N., Sprague, S. A., Bhandari, M., & Slobogean, G. P. (2015) · World Journal of Orthopedics · doi:10.5312/wjo.v6.i10.838 · Cited by 51
Metadata
- Authors
- Moher, D., Pham, B., Lawson, M., & Klassen, T.
- Affiliations
- University of Ottawa (CA)
- Venue
- Health Technology Assessment
- Published
- 2003-12
- Volume
- 7
- Issue
- 41
- Pages
- 1-90
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2119404638
- Cited by
- 500
- References
- 236
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 18 · 2023: 13 · 2022: 33 · 2021: 27 · 2020: 26 · 2019: 26 · 2018: 27 · 2017: 18 · 2016: 22 · 2015: 44 · 2014: 19 · 2013: 24 · 2012: 16
- FWCI
- 2.71
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Health Technology Assessment programme (96/52/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- randomized controlled trials, systematic review quality, complementary and alternative medicine, publication bias, statistical heterogeneity, language of publication, bias in systematic reviews, Cochrane Collaboration
- MeSH terms
- Complementary Therapies, United Kingdom, Humans, Language, Publishing, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.