Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61602-x · Cited by 21,320
“Such instruments have been developed by other groups and were the subject of a recent systematic review (28).”
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.008 · Cited by 17,893
“Such instruments have been developed by other groups and were the subject of a recent systematic review [ 28 ].”
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Stang, A. (2010) · European Journal of Epidemiology · doi:10.1007/s10654-010-9491-z · Cited by 17,396
“It has been criticized that the use of summary scores involve inherent weighting of component items including items that may not be related to the validity of the study findings [2].”
BackgroundInfluentialShamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Elm, E. V., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · BMJ · doi:10.1136/bmj.39335.541782.ad · Cited by 10,766
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2014) · International Journal of Surgery · doi:10.1016/j.ijsu.2014.07.013 · Cited by 9,909
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Italian Journal of Public Health · doi:10.2427/5768 · Cited by 3,374
Assessing Bias in Studies of Prognostic Factors
Hayden, J. A., van der Windt, D. A., Cartwright, J. L., Côté, P., & Bombardier, C. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-4-201302190-00009 · Cited by 3,287
“Similar to our original review of quality assessment tools used in prognosis systematic reviews, they reported a lack of suitable tools (34).”
BackgroundResultGRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Preventive Medicine · doi:10.1016/j.ypmed.2007.08.012 · Cited by 2,698
Prevalence of obstructive sleep apnea in the general population: A systematic review
Senaratna, C. V., Perret, J. L., Lodge, C. J., Lowe, A. J., Campbell, B. E., Matheson, M. C., Hamilton, G. S., & Dharmage, S. C. (2017) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2016.07.002 · Cited by 2,581
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550
Tracking of childhood overweight into adulthood: a systematic review of the literature
Singh, A. S., Mulder, C., Twisk, J. W. R., Van Mechelen, W., & Chinapaw, M. J. M. (2008) · Obesity Reviews · doi:10.1111/j.1467-789x.2008.00475.x · Cited by 2,494
Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS)
Downes, M. J., Brennan, M. L., Williams, H. C., & Dean, R. S. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011458 · Cited by 2,376
Zeng, X., Zhang, Y., Kwong, J. S. W., Zhang, C., Li, S., Sun, F., Niu, Y., & Du, L. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12141 · Cited by 2,082
“Of them, some are recommended and used, some are used without recommendation, and some are eliminated (8, 26), (32), (37–39, 46, 47, 78).”
BackgroundMethodvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Epidemiology · doi:10.1097/ede.0b013e3181577654 · Cited by 1,991
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
Raghu, G., Rochwerg, B., Zhang, Y., Garcia, C. A. C., Azuma, A., Behr, J., Brozek, J. L., Collard, H. R., Cunningham, W., Homma, S., et al. (2015) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201506-1063st · Cited by 1,916
Ma, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020) · Military Medical Research · doi:10.1186/s40779-020-00238-8 · Cited by 1,648
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Bulletin of the World Health Organization · doi:10.2471/blt.07.045120 · Cited by 1,632
“Such instruments have been developed by other groups and were the subject of a recent systematic review (28).”
BackgroundSalvagioni, D. A. J., Melanda, F. N., Mesas, A. E., González, A. D., Gabani, F. L., & Andrade, S. M. D. (2017) · PLOS ONE · doi:10.1371/journal.pone.0185781 · Cited by 1,508
Ardern, C. L., Webster, K. E., Taylor, N. F., & Feller, J. A. (2011) · British Journal of Sports Medicine · doi:10.1136/bjsm.2010.076364 · Cited by 1,383
“and combined with items specifically developed to be applicable to this research area.”
BackgroundLong-term mental health of war-refugees: a systematic literature review
Bogic, M., Njoku, A., & Priebe, S. (2015) · BMC International Health and Human Rights · doi:10.1186/s12914-015-0064-9 · Cited by 1,166
“As already stated, at present there is no consensus on how to assess either the quality of observational studies or the impact of the study quality on the meta-analysis results [24, 25].”
BackgroundImpact of HIV‐related stigma on treatment adherence: systematic review and meta‐synthesis
Katz, I. T., Ryu, A. E., Onuegbu, A. G., Psaros, C., Weiser, S. D., Bangsberg, D. R., & Tsai, A. C. (2013) · Journal of the International AIDS Society · doi:10.7448/ias.16.3.18640 · Cited by 1,126
“[42].”
Pluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.01.009 · Cited by 1,119
“Quantitative observational: Checklists for appraising the methodological quality of quantitative observational studies were systematically reviewed (Sanderson et al., 2007).”
Hong, Q. N., Gonzalez‐Reyes, A., & Pluye, P. (2018) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12884 · Cited by 1,107
Schünemann, H. J., Cuello, C., Akl, E. A., Mustafa, R. A., Meerpohl, J. J., Thayer, K., Morgan, R. L., Gartlehner, G., Kunz, R., Katikireddi, S. V., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.012 · Cited by 911
Self-citationThe Complications of Myopia: A Review and Meta-Analysis
Haarman, A. E. G., Enthoven, C. A., Tideman, J. W. L., Tedja, M. S., Verhoeven, V. J. M., & Klaver, C. C. W. (2020) · Investigative Opthalmology & Visual Science · doi:10.1167/iovs.61.4.49 · Cited by 881
“We assessed the quality of all studies using the criteria proposed by Sanderson et al. 19 The variables examined included the definitions of the exposures (any, low, moderate, and high myopia), definitions of the outcome variables (MMD, RD, cataract, and OAG), number of participants, age ranges, sex prevalence, study…”
The relationship between professional burnout and quality and safety in healthcare: a meta-analysis
Salyers, M. P., Bonfils, K. A., Luther, L., Firmin, R. L., White, D. A., Adams, E. L., & Rollins, A. L. (2017) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3886-9 · Cited by 843
“34, 35 We tested and refined the initial rating system on several studies before rating the full sample.”
MethodThe effect of osteoarthritis definition on prevalence and incidence estimates: a systematic review
Pereira, D., Peleteiro, B., Araújo, J., Branco, J., Santos, R. A., & Ramos, E. (2011) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2011.08.009 · Cited by 843
Cook, D. A., & Reed, D. A. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000786 · Cited by 781
Advanced maternal age and adverse pregnancy outcomes: A systematic review and meta-analysis
Lean, S. C., Derricott, H., Jones, R. L., & Heazell, A. E. P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0186287 · Cited by 737
“Risk of bias was assessed for each of the domains as low, high or unclear using pre-specified rules [18] Data from included studies were extracted by two independent investigators (SL and AH) using a standard extraction form.”
BackgroundMethodMolendijk, M. L., Spinhoven, P., Polak, M., Bus, B. A. A., Penninx, B. W. J. H., & Elzinga, B. M. (2014) · Molecular Psychiatry · doi:10.1038/mp.2013.105 · Cited by 702
Petti, S., Glendor, U., & Andersson, L. (2018) · Dental Traumatology · doi:10.1111/edt.12389 · Cited by 698
Psychosocial work environment and stress-related disorders, a systematic review
Nieuwenhuijsen, K., Bruinvels, D., & Frings-Dresen, M. (2010) · Occupational Medicine · doi:10.1093/occmed/kqq081 · Cited by 696
Hyperuricemia and coronary heart disease: A systematic review and meta‐analysis
Kim, S. Y., Guevara, J. P., Kim, K. M., Choi, H. K., Heitjan, D. F., & Albert, D. A. (2010) · Arthritis Care & Research · doi:10.1002/acr.20065 · Cited by 671
Myopia as a Risk Factor for Open-Angle Glaucoma: A Systematic Review and Meta-Analysis
Marcus, M. W., de Vries, M. M., Montolio, F. G. J., & Jansonius, N. M. (2011) · Ophthalmology · doi:10.1016/j.ophtha.2011.03.012 · Cited by 658
Harris, P. E., Cooper, K. L., Relton, C., & Thomas, K. J. (2012) · International Journal of Clinical Practice · doi:10.1111/j.1742-1241.2012.02945.x · Cited by 642
Hall, A. M., Ferreira, P. H., Maher, C. G., Latimer, J., & Ferreira, M. L. (2010) · Physical Therapy · doi:10.2522/ptj.20090245 · Cited by 641
Karr, J. E., Areshenkoff, C. N., Rast, P., Hofer, S. M., Iverson, G. L., & Garcia-Barrera, M. A. (2018) · Psychological Bulletin · doi:10.1037/bul0000160 · Cited by 637
Educational attainment and obesity: a systematic review
Cohen, A. K., Rai, M., Rehkopf, D. H., & Abrams, B. (2013) · Obesity Reviews · doi:10.1111/obr.12062 · Cited by 596
Barriers to vaccination in Latin America: A systematic literature review
Guzman-Holst, A., DeAntonio, R., Prado-Cohrs, D., & Juliao, P. (2020) · Vaccine · doi:10.1016/j.vaccine.2019.10.088 · Cited by 594
Mazzaferro, V., Bhoori, S., Sposito, C., Bongini, M., Langer, M., Miceli, R., & Mariani, L. (2011) · Liver Transplantation · doi:10.1002/lt.22365 · Cited by 593
Meta-Analysis of Antibiotics and the Risk of Community-Associated Clostridium difficile Infection
Brown, K. A., Khanafer, N., Daneman, N., & Fisman, D. N. (2013) · Antimicrobial Agents and Chemotherapy · doi:10.1128/aac.02176-12 · Cited by 587
A systematic review on the clustering and co-occurrence of multiple risk behaviours
Meader, N., King, K., Moe-Byrne, T., Wright, K., Graham, H., Petticrew, M., Power, C., White, M., & Sowden, A. J. (2016) · BMC Public Health · doi:10.1186/s12889-016-3373-6 · Cited by 578
“A recent systematic review of quality assessment tools [12] was consulted and the University of Wales College of Medicine [13] tool for the critical appraisal of observational studies was judged to be the most appropriate based on methodological relevance and ease of use.”
MethodFatima, Y., Doi, S. A. R., & Mamun, A. A. (2015) · Obesity Reviews · doi:10.1111/obr.12245 · Cited by 577
“The validity of our quality assessment tool is comparable with other available tools, as we have given attention to factors related to information bias, selection bias, design specific bias, confounding and the statistical analysis (24).”
MethodSystematic Review and Meta-Analysis of Practice Facilitation Within Primary Care Settings
Baskerville, N. B., Liddy, C., & Hogg, W. (2012) · The Annals of Family Medicine · doi:10.1370/afm.1312 · Cited by 571
Lanctôt, N., & Guay, S. (2014) · Aggression and Violent Behavior · doi:10.1016/j.avb.2014.07.010 · Cited by 568
Mikocka-Walus, A., Knowles, S. R., Keefer, L., & Graff, L. (2016) · Inflammatory Bowel Diseases · doi:10.1097/mib.0000000000000620 · Cited by 567
Hart, L. M., Granillo, M. T., Jorm, A. F., & Paxton, S. J. (2011) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.03.004 · Cited by 557
Dewa, C. S., Loong, D., Bonato, S., & Trojanowski, L. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015141 · Cited by 526
“To assess the risk of bias in observational studies, Sanderson et al 26 recommend the use of a transparent checklist that concentrates on the ‘few, principal, and potential sources of bias in a study’s findings’.”
MethodDekkers, O. M., Vandenbroucke, J. P., Cevallos, M., Renehan, A. G., Altman, D. G., & Egger, M. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002742 · Cited by 525
Prevalence of Multiple Forms of Sexting Behavior Among Youth: A Systematic Review and Meta-analysis
Madigan, S., Ly, A., Rash, C. L., Van Ouytsel, J., & Temple, J. R. (2018) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2017.5314 · Cited by 519
Quality of life in inflammatory bowel disease: a systematic review and meta-analyses—part I
Knowles, S. R., Graff, L. A., Wilding, H., Hewitt, C., Keefer, L., & Mikocka-Walus, A. (2018) · Inflammatory Bowel Diseases · doi:10.1093/ibd/izx100 · Cited by 503
Oral language deficits in familial dyslexia: A meta-analysis and review.
Snowling, M. J., & Melby-Lervåg, M. (2016) · Psychological Bulletin · doi:10.1037/bul0000037 · Cited by 490
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486
Epidemiology of Pediatric Functional Abdominal Pain Disorders: A Meta-Analysis
Korterink, J. J., Diederen, K., Benninga, M. A., & Tabbers, M. M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0126982 · Cited by 481
“Because there is currently no gold-standard quality assessment tool for epidemiologic studies, [15] we composed a new assessment tool based on a scale for quantitative studies[16] and on a guideline for evaluating prevalence studies.”
MethodWhat Competencies Should Undergraduate Engineering Programs Emphasize? A Systematic Review
Passow, H. J., & Passow, C. H. (2017) · Journal of Engineering Education · doi:10.1002/jee.20171 · Cited by 472
The Association between Time Spent Outdoors and Myopia in Children and Adolescents
Sherwin, J. C., Reacher, M. H., Keogh, R. H., Khawaja, A. P., Mackey, D. A., & Foster, P. J. (2012) · Ophthalmology · doi:10.1016/j.ophtha.2012.04.020 · Cited by 469
“We id not produce a tally of “positive” and “negative” studie, because his can miss important sources of heterogeneity and obscure the ole of bias.9 ata Extraction and Assessment of Study Quality o appropriately report the systematic review and meta-analysis e were guided by the MOOSE checklist.10 For each study, the…”
Mycoplasma genitalium Infection and Female Reproductive Tract Disease: A Meta-analysis
Lis, R., Rowhani-Rahbar, A., & Manhart, L. E. (2015) · Clinical Infectious Diseases · doi:10.1093/cid/civ312 · Cited by 460
“components, some of which may not directly affect the validity of the study [6].”
BackgroundJohnson, P. I., Sutton, P., Atchley, D. S., Koustas, E., Lam, J., Sen, S., Robinson, K. A., Axelrad, D. A., & Woodruff, T. J. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307893 · Cited by 449
“Although there is existing guidance for assessing risk of bias of human obser vational studies (Viswanathan et al. 2012; Wells 2014), there is no universally accepted tool (Sanderson et al. 2007).”
BackgroundMethodAssessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448
“note that the default strategy in systematic reviews should be to consider including observational studies for evidence of benefit and the decision rests on the answer to two questions: (1) are there gaps in the trial evidence for the review questions under consideration? and (2) will observational studies provide val…”
BackgroundInfluentialMargulis, A. V., Pladevall, M., Riera-Guardia, N., Varas-Lorenzo, C., Hazell, L., Berkman, N., Viswanathan, M., & Perez-Gutthann, S. (2014) · Clinical Epidemiology · doi:10.2147/clep.s66677 · Cited by 441
“Also, it has been argued that quality summary scores may mask variations in quality by domain and use an unclear and often implicit weighting scheme.(3,5,13,36) The interrater reliability or agreement of the NOS has been previously evaluated.”
Liu, G., Sharma, M., Tan, N., & Barnabas, R. V. (2018) · AIDS · doi:10.1097/qad.0000000000001765 · Cited by 421
“Study quality assessment We assessed the risk of bias (low, moderate, high) in studies using modified criteria for nonrandomized observational studies across five domains: study population, detection of outcome and exposure, bias related to study design, statistical analysis, and disclosure of conflict of interest [19…”
Von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Gaceta Sanitaria · doi:10.1157/13119325 · Cited by 417
Quality of life in inflammatory bowel disease: a systematic review and meta-analyses—part II
Knowles, S. R., Keefer, L., Wilding, H., Hewitt, C., Graff, L. A., & Mikocka-Walus, A. (2018) · Inflammatory Bowel Diseases · doi:10.1093/ibd/izy015 · Cited by 416
Carreira, H., Williams, R., Müller, M., Harewood, R., Stanway, S., & Bhaskaran, K. (2018) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djy177 · Cited by 416
The prevalence of insomnia in the general population in China: A meta-analysis
Cao, X.-L., Wang, S.-B., Zhong, B.-L., Zhang, L., Ungvari, G. S., Ng, C. H., Li, L., Chiu, H. F. K., Lok, G. K. I., Lu, J.-P., et al. (2017) · PLOS ONE · doi:10.1371/journal.pone.0170772 · Cited by 409
“To date, there have been no well-defined tools for assessing quality of observational epidemiological studies [9].”
MethodLong-Term Unemployment and Suicide: A Systematic Review and Meta-Analysis
Milner, A., Page, A., & LaMontagne, A. D. (2013) · PLoS ONE · doi:10.1371/journal.pone.0051333 · Cited by 407
“The quality of studies was assessed before inclusion in the systematic review and meta-analysis, based on published recommendations [12].”
MethodIozzino, L., Ferrari, C., Large, M., Nielssen, O., & de Girolamo, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0128536 · Cited by 401
“As observed by Sanderson and colleagues [19], there is no single tool that is suitable for assessing the quality of all types of observational studies.”
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiolog…”
BackgroundCombined oral contraceptives: venous thrombosis
de Bastos, M., Stegeman, B. H., Rosendaal, F. R., Van Hylckama Vlieg, A., Helmerhorst, F. M., Stijnen, T., & Dekkers, O. M. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010813.pub2 · Cited by 399
Patient-centred communication is associated with positive therapeutic alliance: a systematic review
Pinto, R. Z., Ferreira, M. L., Oliveira, V. C., Franco, M. R., Adams, R., Maher, C. G., & Ferreira, P. H. (2012) · Journal of Physiotherapy · doi:10.1016/s1836-9553(12)70087-5 · Cited by 389
Suicide by occupation: Systematic review and meta-analysis
Milner, A., Spittal, M. J., Pirkis, J., & LaMontagne, A. D. (2013) · British Journal of Psychiatry · doi:10.1192/bjp.bp.113.128405 · Cited by 387
“26 Those deemed to be of acceptable quality can be found in online Table DS2 (as mentioned above, the grouping of occupations into the ISCO major (skill level) group category and aggregate skill-level groups can be found in Table DS1).”
Current global trends in the incidence of pediatric-onset inflammatory bowel disease
Sýkora, J., Pomahačová, R., Kreslová, M., Cvalínová, D., Štych, P., & Schwarz, J. (2018) · World Journal of Gastroenterology · doi:10.3748/wjg.v24.i25.2741 · Cited by 380
Knott, C., Bell, S., & Britton, A. (2015) · Diabetes Care · doi:10.2337/dc15-0710 · Cited by 375
“For instance, while a wide range of instruments have thus far been devised for the assessment of nonrandomized studies, each comprises assessment criteria that are disparate in both number and nature (56).”
BackgroundThe prevalence of fatigue after stroke: A systematic review and meta-analysis
Cumming, T. B., Packer, M., Kramer, S. F., & English, C. (2016) · International Journal of Stroke · doi:10.1177/1747493016669861 · Cited by 374
“This tool was designed specifically for case–control observational studies and has been through a process of robust development.(20) The tool was modified to allow different types of quantitative research designs to be analyzed, and the major areas of potential bias to be emphasized.”
MethodLiu, M., Wu, L., & Yao, S. (2016) · British Journal of Sports Medicine · doi:10.1136/bjsports-2015-095084 · Cited by 364
Füzéki, E., Engeroff, T., & Banzer, W. (2017) · Sports Medicine · doi:10.1007/s40279-017-0724-0 · Cited by 363
“Domains were identified based on a systematic review of assessment tools [14], and specific items were formulated based on Ariëns et al.”
MethodPhenomenology of depression in older compared with younger adults: Meta-analysis
Hegeman, J. M., Kok, R. M., van der Mast, R. C., & Giltay, E. J. (2012) · British Journal of Psychiatry · doi:10.1192/bjp.bp.111.095950 · Cited by 363
“Quality assessment Quality assessment of observational studies in meta-analyses is not usual and there is no consensus regarding the method used.(9,10) In the present study, the quality of the included studies was assessed by two authors (J.”
MethodMCINTYRE, S., TAITZ, D., KEOGH, J., GOLDSMITH, S., BADAWI, N., & BLAIR, E. (2013) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.12017 · Cited by 362
“Two recent systematic reviews have attempted to identify optimal tools to assess the risk of bias in reports of observational research.(35,36) Neither review could identify an optimal tool, but both recommended a transparent checklist over a scale, based on weighted and summed scores from individual sources.”
BackgroundToo, L. S., Spittal, M. J., Bugeja, L., Reifels, L., Butterworth, P., & Pirkis, J. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.08.054 · Cited by 360
“Quality assessment In a review of tools for quality assessment of observation studies, several minimum requirements for a quality assessment tool to be considered ‘good’ were recommended (Sanderson et al., 2007).”
BackgroundSchwingshackl, L., Knüppel, S., Schwedhelm, C., Hoffmann, G., Missbach, B., Stelmach-Mardas, M., Dietrich, S., Eichelmann, F., Kontopanteils, E., Iqbal, K., et al. (2016) · Advances in Nutrition · doi:10.3945/an.116.013052 · Cited by 360
“The rationale of the single grading system for each NutriGrade item and its detailed content was established after reviewing available guidelines on how to report systematic reviews: the Meta-analysis Of Observational Studies in Epidemiology checklist (9), the Cochrane Handbook (2), the Preferred Reporting Items for S…”
Westlake, S. L., Colebatch, A. N., Baird, J., Kiely, P., Quinn, M., Choy, E., Ostor, A. J. K., & Edwards, C. J. (2010) · Rheumatology · doi:10.1093/rheumatology/kep366 · Cited by 357
Koustas, E., Lam, J., Sutton, P., Johnson, P. I., Atchley, D. S., Sen, S., Robinson, K. A., Axelrad, D. A., & Woodruff, T. J. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307177 · Cited by 349
Quansah, R., Armah, F. A., Essumang, D. K., Luginaah, I., Clarke, E., Marfoh, K., Cobbina, S. J., Nketiah-Amponsah, E., Namujju, P. B., Obiri, S., et al. (2015) · Environmental Health Perspectives · doi:10.1289/ehp.1307894 · Cited by 349
Moskalewicz, A., & Oremus, M. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.013 · Cited by 336
A Meta-Analysis of Injuries in Senior Men’s Professional Rugby Union
Williams, S., Trewartha, G., Kemp, S., & Stokes, K. (2013) · Sports Medicine · doi:10.1007/s40279-013-0078-1 · Cited by 333
“Additionally, while a recent review of tools for assessing the quality of observational studies stated that qualitative checklists were more appropriate than quantitative scales, and that the STROBE statement was a suitable starting point [61], it should be noted that the STROBE statement was not designed to evaluate…”
BackgroundSmoking as a risk factor for lung cancer in women and men: a systematic review and meta-analysis
O’Keeffe, L. M., Taylor, G., Huxley, R. R., Mitchell, P., Woodward, M., & Peters, S. A. E. (2018) · BMJ Open · doi:10.1136/bmjopen-2018-021611 · Cited by 332
Abou Chakra, C. N., Pepin, J., Sirard, S., & Valiquette, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0098400 · Cited by 328
“for observational studies [65–67].”
Real-world data: towards achieving the achievable in cancer care
Booth, C. M., Karim, S., & Mackillop, W. J. (2019) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-019-0167-7 · Cited by 327
Barriers to Screening for Intimate Partner Violence
Sprague, S., Madden, K., Simunovic, N., Godin, K., Pham, N. K., Bhandari, M., & Goslings, J. C. (2012) · Women & Health · doi:10.1080/03630242.2012.690840 · Cited by 324
“2004 ; Sanderson, Tatt, & Higgins 2007 ; Kelley, Clark, Brown, & Sitzia 2003 ), and we found that none of the previously developed checklists were appropriate for this study.”
Lagerveld, S. E., Bültmann, U., Franche, R. L., van Dijk, F. J. H., Vlasveld, M. C., van der Feltz-Cornelis, C. M., Bruinvels, D. J., Huijs, J. J. J. M., Blonk, R. W. B., van der Klink, J. J. L., et al. (2010) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-009-9224-x · Cited by 324
“In accordance to the study of Sanderson [30] who applied the ICF to a population of workers with mental health problems, work functioning and work participation can be situated in the ICF domains ‘Activities’ (e.g., limitations in work activities or performance) and ‘Partici-pation’ (e.g., absenteeism).”
Influential
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.