Methodological index for non‐randomized studies ( MINORS ): development and validation of a new instrument
Abstract
BACKGROUND: Because of specific methodological difficulties in conducting randomized trials, surgical research remains dependent predominantly on observational or non-randomized studies. Few validated instruments are available to determine the methodological quality of such studies either from the reader's perspective or for the purpose of meta-analysis. The aim of the present study was to develop and validate such an instrument. METHODS: After an initial conceptualization phase of a methodological index for non-randomized studies (MINORS), a list of 12 potential items was sent to 100 experts from different surgical specialties for evaluation and was also assessed by 10 clinical methodologists. Subsequent testing involved the assessment of inter-reviewer agreement, test-retest reliability at 2 months, internal consistency reliability and external validity. RESULTS: The final version of MINORS contained 12 items, the first eight being specifically for non-comparative studies. Reliability was established on the basis of good inter-reviewer agreement, high test-retest reliability by the kappa-coefficient and good internal consistency by a high Cronbach's alpha-coefficient. External validity was established in terms of the ability of MINORS to identify excellent trials. CONCLUSIONS: MINORS is a valid instrument designed to assess the methodological quality of non-randomized surgical studies, whether comparative or non-comparative. The next step will be to determine its external validity when used in a large number of studies and to compare it with other existing instruments.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RCTs were appraised using the Cochrane RoB 2 tool and observational studies using the methodological index for non-randomized studies (MINORS).
“RCTs were appraised using the Cochrane RoB 2 tool and observational studies using methodological index for non-randomized studies (MINORS).13,14”
From Perioperative Analgesia and Postoperative Nausea and Vomiting Management in Otoplasty: A Systematic Review (2026) · cites this paper as [14]
Cites
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623 · see citing works
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252 · see citing works
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373 · see citing works
Statistics notes: Cronbach's alpha: Table 1
Bland, J. M., & Altman, D. G. (1997) · BMJ · doi:10.1136/bmj.314.7080.572 · Cited by 5,300 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Validation of an index of the quality of review articles
Oxman, A. D., & Guyatt, G. H. (1991) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(91)90160-b · Cited by 788 · see citing works
Randomised trials in surgery: problems and possible solutions
McCulloch, P. G., Taylor, I., Sasako, M., Lovett, B., & Griffin, D. (2002) · BMJ · doi:10.1136/bmj.324.7351.1448 · Cited by 761 · see citing works
Feinstein, A. R. (1987) · doi:10.2307/j.ctt1xp3vbc · Cited by 538 · see citing works
Treatment of open fractures of the shaft of the tibia
Bhandari, M., Guyatt, G. H., Swiontkowski, M. F., & Schemitsch, E. H. (2001) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.83b1.10986 · Cited by 386 · see citing works
Statistics Notes: Validating scales and indexes
Bland, J. M. (2002) · BMJ · doi:10.1136/bmj.324.7337.606 · Cited by 239 · see citing works
Meta-Analyses in Orthopaedic Surgery
Bhandari, M., Morrow, F., Kulkarni, A. V., & Tornetta, P. (2001) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-200101000-00003 · Cited by 190 · see citing works
A generalized concordance correlation coefficient for continuous and categorical data
King, T. S., & Chinchilli, V. M. (2001) · Statistics in Medicine · doi:10.1002/sim.845 · Cited by 169 · see citing works
Ten-year audit of randomized trials in digestive surgery from Europe
Slim, K., Haugh, M., Fagniez, P. L., et al. (2000) · British journal of surgery · doi:10.1046/j.1365-2168.2000.01561.x · Cited by 16 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Zheng, Z., Peng, F., Xu, B., Zhao, J., Liu, H., Peng, J., Li, Q., Jiang, C., Zhou, Y., Liu, S., et al. (2020) · Journal of Infection · doi:10.1016/j.jinf.2020.04.021 · Cited by 2,074 · see citing works
Zeng, X., Zhang, Y., Kwong, J. S. W., Zhang, C., Li, S., Sun, F., Niu, Y., & Du, L. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12141 · Cited by 2,082 · see citing works
Ma, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020) · Military Medical Research · doi:10.1186/s40779-020-00238-8 · Cited by 1,648 · see citing works
Ardern, C. L., Taylor, N. F., Feller, J. A., & Webster, K. E. (2014) · British Journal of Sports Medicine · doi:10.1136/bjsports-2013-093398 · Cited by 1,465 · see citing works
Kim, S. Y., Park, J. E., Lee, Y. J., Seo, H.-J., Sheen, S.-S., Hahn, S., Jang, B.-H., & Son, H.-J. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.016 · Cited by 1,165 · see citing works
Decoster, L., Van Puyvelde, K., Mohile, S., Wedding, U., Basso, U., Colloca, G., Rostoft, S., Overcash, J., Wildiers, H., Steer, C., et al. (2015) · Annals of Oncology · doi:10.1093/annonc/mdu210 · Cited by 794 · see citing works
Psychological factors affecting the outcome of total hip and knee arthroplasty: a systematic review
Vissers, M. M., Bussmann, J. B., Verhaar, J. A. N., Busschbach, J. J. V., Bierma-Zeinstra, S. M. A., & Reijman, M. (2012) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2011.07.003 · Cited by 473 · see citing works
Liu, M., Wu, L., & Ming, Q. (2015) · PLOS ONE · doi:10.1371/journal.pone.0134804 · Cited by 374 · see citing works
Lee, Y., Doumouras, A. G., Yu, J., Brar, K., Banfield, L., Gmora, S., Anvari, M., & Hong, D. (2019) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2018.10.017 · Cited by 371 · see citing works
Dedeyne, L., Deschodt, M., Verschueren, S., Tournoy, J., & Gielen, E. (2017) · Clinical Interventions in Aging · doi:10.2147/cia.s130794 · Cited by 315 · see citing works
Risk factors for chronic depression—a systematic review
Hölzel, L., Härter, M., Reese, C., & Kriston, L. (2011) · Journal of Affective Disorders · doi:10.1016/j.jad.2010.03.025 · Cited by 285 · see citing works
Thakur, B., Dubey, P., Benitez, J., Torres, J. P., Reddy, S., Shokar, N., Aung, K., Mukherjee, D., & Dwivedi, A. K. (2021) · Scientific Reports · doi:10.1038/s41598-021-88130-w · Cited by 284 · see citing works
The Role of Self-Efficacy on the Prognosis of Chronic Musculoskeletal Pain: A Systematic Review
Martinez-Calderon, J., Zamora-Campos, C., Navarro-Ledesma, S., & Luque-Suarez, A. (2018) · The Journal of Pain · doi:10.1016/j.jpain.2017.08.008 · Cited by 268 · see citing works
Mathur, M. B., Epel, E., Kind, S., Desai, M., Parks, C. G., Sandler, D. P., & Khazeni, N. (2016) · Brain, Behavior, and Immunity · doi:10.1016/j.bbi.2016.02.002 · Cited by 252 · see citing works
Farrah, K., Young, K., Tunis, M. C., & Zhao, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1172-8 · Cited by 246 · see citing works
Prevalence of vitamin D inadequacy in athletes: a systematic-review and meta-analysis
Farrokhyar, F., Tabasinejad, R., Dao, D., Peterson, D., Ayeni, O. R., Hadioonzadeh, R., & Bhandari, M. (2015) · Sports Medicine · doi:10.1007/s40279-014-0267-6 · Cited by 249 · see citing works
Endometriosis and obstetrics complications: A systematic review and meta-analysis
Zullo, F., Spagnolo, E., Saccone, G., Acunzo, M., Xodo, S., Ceccaroni, M., & Berghella, V. (2017) · Fertility and Sterility · doi:10.1016/j.fertnstert.2017.07.019 · Cited by 227 · see citing works
Statins for infection and sepsis: a systematic review of the clinical evidence.
Falagas, M. E., Makris, G. C., Matthaiou, D. K., & Rafailidis, P. I. (2008) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkn019 · Cited by 166 · see citing works
Masoudian, P., Nasr, A., de Nanassy, J., Fung-Kee-Fung, K., Bainbridge, S. A., & El Demellawy, D. (2016) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2015.11.020 · Cited by 157 · see citing works
Silva Junior, F. J. G. D., Sales, J. C. E. S., Monteiro, C. F. D. S., Costa, A. P. C., Campos, L. R. B., Miranda, P. I. G., Monteiro, T. A. D. S., Lima, R. A. G., & Lopes-Junior, L. C. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-039426 · Cited by 115 · see citing works
Knight, S. R., Ng, N., Tsanas, A., Mclean, K., Pagliari, C., & Harrison, E. M. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00525-1 · Cited by 104 · see citing works
Markers of liver regeneration—the role of growth factors and cytokines: a systematic review
Hoffmann, K., Nagel, A. J., Tanabe, K., Fuchs, J., Dehlke, K., Ghamarnejad, O., Lemekhova, A., & Mehrabi, A. (2020) · BMC Surgery · doi:10.1186/s12893-019-0664-8 · Cited by 94 · see citing works
Chong, E., Crowe, L., Mentor, K., Pandanaboyana, S., & Sharp, L. (2023) · Supportive Care in Cancer · doi:10.1007/s00520-022-07468-7 · Cited by 86 · see citing works
Bariatric surgery improves nonalcoholic fatty liver disease: systematic review and meta-analysis
Zhou, H., Luo, P., Li, P., wang, G., Yi, X., Fu, Z., Sun, X., Cui, B., Zhu, L., & Zhu, S. (2022) · Obesity Surgery · doi:10.1007/s11695-022-06011-1 · Cited by 79 · see citing works
Van Damme, L., & De Waele, J. J. (2018) · Critical Care · doi:10.1186/s13054-018-2103-0 · Cited by 74 · see citing works
Pipek, L. Z., Baptista, C. G., Nascimento, R. F. V., Taba, J. V., Suzuki, M. O., do Nascimento, F. S., Martines, D. R., Nii, F., Iuamoto, L. R., Carneiro-D’Albuquerque, L. A., et al. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237740 · Cited by 71 · see citing works
Bartels, H. C., Rogers, A. C., O’Brien, D., McVey, R., Walsh, J., & Brennan, D. J. (2018) · Obstetrics & Gynecology · doi:10.1097/aog.0000000000002865 · Cited by 67 · see citing works
Evidence on the Use of Mobile Apps During the Treatment of Breast Cancer: Systematic Review
Cruz, F. O. A. M., Vilela, R. A., Ferreira, E. B., Melo, N. S., & Reis, P. E. D. D. (2019) · JMIR mHealth and uHealth · doi:10.2196/13245 · Cited by 66 · see citing works
Cho, Y., Zhang, H., Harris, M. R., Gong, Y., Smith, E. L., & Jiang, Y. (2021) · Journal of Medical Internet Research · doi:10.2196/24638 · Cited by 53 · see citing works
Mocanu, V., Dang, J. T., Switzer, N., Skubleny, D., Shi, X., de Gara, C., Birch, D. W., & Karmali, S. (2018) · Obesity Surgery · doi:10.1007/s11695-017-3024-8 · Cited by 44 · see citing works
Lee, Y., Tian, C., Lovrics, O., Soon, M. S., Doumouras, A. G., Anvari, M., & Hong, D. (2020) · Surgery for Obesity and Related Diseases · doi:10.1016/j.soard.2020.05.012 · Cited by 44 · see citing works
Complications after prominent ear correction: a systematic review of the literature.
Sadhra, S. S., Motahariasl, S., & Hardwicke, J. T. (2017) · Journal of Plastic, Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2017.05.033 · Cited by 40 · see citing works
The Effect of Mindfulness-Based Interventions on Tinnitus Distress. A Systematic Review
Rademaker, M. M., Stegeman, I., Ho-Kang-You, K. E., Stokroos, R. J., & Smit, A. L. (2019) · Frontiers in Neurology · doi:10.3389/fneur.2019.01135 · Cited by 39 · see citing works
“Items were scored as 0 (not reported), 1 (reported but inadequate) or 2 (reported and adequate) (32).”
BackgroundJoss, D., & Teicher, M. H. (2021) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-021-00240-4 · Cited by 35 · see citing works
Murtha-Lemekhova, A., Fuchs, J., Ghamarnejad, O., Nikdad, M., Probst, P., & Hoffmann, K. (2021) · Scientific Reports · doi:10.1038/s41598-021-92888-4 · Cited by 36 · see citing works
Emile, S. H., Khan, S. M., & Barsoum, S. H. (2021) · Updates in Surgery · doi:10.1007/s13304-020-00857-9 · Cited by 34 · see citing works
Nawar, K., Mohammad, A., Johns, E. J., & Abdulla, M. H. (2022) · Journal of Human Hypertension · doi:10.1038/s41371-022-00658-0 · Cited by 32 · see citing works
Carletto, S., Cavalera, C., Sadowski, I., Rovaris, M., Borghi, M., Khoury, B., Ostacoli, L., & Pagnini, F. (2020) · Psychosomatic Medicine · doi:10.1097/psy.0000000000000819 · Cited by 29 · see citing works
Mathew, S. T., Po, S. S., & Thadani, U. (2018) · Heart Rhythm · doi:10.1016/j.hrthm.2017.10.004 · Cited by 29 · see citing works
Lee, C. S. C., Ng, K.-H., Chan, P. C. K., & Peng, X. (2022) · Hong Kong Journal of Occupational Therapy · doi:10.1177/15691861211073826 · Cited by 28 · see citing works
Tonet, E., Campana, R., Caglioni, S., Gibiino, F., Fiorio, A., Chiaranda, G., Zagnoni, S., Casella, G., & Campo, G. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10071508 · Cited by 26 · see citing works
Clinical benefit of neoadjuvant anti‐PD‐1/PD‐L1 utilization among different tumors
Li, Z., Wu, X., Zhao, Y., Xiao, Y., Zhao, Y., Zhang, T., Li, H., Sha, F., Wang, Y., Deng, L., et al. (2021) · MedComm · doi:10.1002/mco2.61 · Cited by 23 · see citing works
Systematic reviews and meta‐analyses in coloproctology: interpretation and potential pitfalls
Wille‐Jørgensen, P., & Renehan, A. G. (2008) · Colorectal Disease · doi:10.1111/j.1463-1318.2007.01421.x · Cited by 19 · see citing works
Wen, J., Syed, B., Nadora, D., How‐Volkman, C., Bernstein, E., Truong, A., Akhtar, M., Razick, A., Puglisi, J., & Frezza, E. (2025) · Endocrinology, Diabetes & Metabolism · doi:10.1002/edm2.70045 · Cited by 17 · see citing works
Kumar, R., Nuñez, N. A., Prokop, L. J., Veldic, M., Betcher, H., & Singh, B. (2021) · Journal of Clinical Psychopharmacology · doi:10.1097/jcp.0000000000001475 · Cited by 16 · see citing works
Heringer, L. C., Machado de Lima, M., Rotta, J. M., & Botelho, R. V. (2020) · World Neurosurgery · doi:10.1016/j.wneu.2019.11.041 · Cited by 15 · see citing works
Metabolomic biomarkers of polycystic ovary syndrome related-obesity: a review of the literature
Vonica, C. L., Farcas, A. D., Roman, G., Muresan, A. A., Fodor, A., Cernea, S., & Georgescu, C. E. (2020) · Revista Romana de Medicina de Laborator · doi:10.2478/rrlm-2020-0017 · Cited by 5 · see citing works
Dankner, M., Dubé, L.-R., Sorin, M., Stein, A. J. B., Nowakowski, A., Park, C. L., Magrill, J., Lazaratos, A.-M., Romero, J. M., Batist, G., et al. (2025) · Current Oncology · doi:10.3390/curroncol32110624 · Cited by 3 · see citing works
Rico-González, M., Martín-Moya, R., Giles-Girela, F. J., Ardigò, L. P., & González-Fernández, F. T. (2025) · Journal of Functional Morphology and Kinesiology · doi:10.3390/jfmk10030254 · Cited by 3 · see citing works
Alnufaei, A. H., AlSubaie, F. K., Aljabri, R. A., Aloqayli, S. Z., Alanshmi, S., Alotaibi, S. M., Ajeeb, Z. B., AlGhamdi, L. Y., Kattan, M. E., Alahmadi, B. H., et al. (2026) · Plastic and Reconstructive Surgery - Global Open · doi:10.1097/gox.0000000000007347
- Authors
- Slim, K., Nini, E., Forestier, D., Kwiatkowski, F., Panis, Y., & Chipponi, J.
- Affiliations
- Centre Jean Perrin (FR)
- Hôpital Lariboisière (FR)
- Venue
- ANZ Journal of Surgery
- Publisher
- Wiley
- Published
- 2003-09
- Volume
- 73
- Issue
- 9
- Pages
- 712-716
- ISSN
- 1445-1433 (ISSN-L)
- 1445-2197
- OpenAlex
- W2017044005
- Cited by
- 7,603
- References
- 14
- Citations by year
- 2026: 752 · 2025: 1014 · 2024: 988 · 2023: 896 · 2022: 846 · 2021: 814 · 2020: 634 · 2019: 444 · 2018: 336 · 2017: 247 · 2016: 200 · 2015: 146 · 2014: 93 · 2013: 52 · 2012: 45
- FWCI
- 5.48
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ANZ Journal of Surgery (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- minors, methodological quality assessment, non-randomized studies, observational studies, surgical research, instrument development, inter-reviewer agreement, test-retest reliability, internal consistency, generalizability, meta-analysis
- MeSH terms
- Clinical Trials as Topic, Humans, Research Design, Reproducibility of Results, Peer Review, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
