Interrater and intrarater reliability in the diagnosis and staging of endometriosis
Open accessAbstract
OBJECTIVE: To estimate the interrater and intrarater reliability of endometriosis diagnosis and severity of disease among gynecologic surgeons viewing operative digital images. METHODS: The study population comprised a random sample (n=148 [36%]) of women who participated in the Endometriosis: Natural History, Diagnosis and Outcomes study. Four academic expert and four local, specialized expert surgeons reviewed the images, diagnosed the presence or absence of endometriosis for each woman, and rated severity using the revised American Society for Reproductive Medicine (ASRM) criteria. Interrater-level and intrarater-level agreement were calculated for both endometriosis diagnosis and staging. RESULTS: The interrater reliability for endometriosis diagnosis among the eight surgeons was substantial: Fleiss κ=0.69 (95% confidence interval [CI] 0.64-0.74). Surgeons agreed on revised ASRM endometriosis staging criteria after experienced assessment in a majority of cases (mean 61%, range 52-75%) with moderate interrater reliability: Fleiss κ=0.44 (95% CI 0.41-0.47). The intrarater reliability for experienced assessment compared with computer-assisted revised ASRM staging was almost perfect (mean weighted κ=0.95, range 0.89-0.99). CONCLUSION: Substantial reliability was found for revised ASRM endometriosis diagnosis, whereas moderate reliability was observed for staging. Almost perfect reliability was observed for surgeons' rating of disease severity compared with computerized-assisted, checklist-based staging. Findings suggest that reliability in endometriosis diagnosis is not greatly altered by location or composition of surgeons, supporting the conduct of multisite studies or compilation of endometriosis data across clinical centers. Although surgeons appear to be skilled at assessing endometriosis stage intuitively, how staging of disease burden correlates with clinical outcomes remains to be developed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our previous work showed high agreement for endometriosis diagnosis among 8 expert surgeons viewing operative digital images (κ=0.69) or digital images in addition to operative reports (κ=0.88).
“Our previous work showed high agreement for endometriosis diagnosis among 8 expert surgeons viewing operative digital images (κ=0.69) or digital images in addition to operative reports (κ=0.88).26,27”
From Examining the co-occurrence of endometriosis and polycystic ovarian syndrome (2023) · cites this paper as [26]
The Results report that eight surgeons diagnosing endometriosis from digital images had overall agreement of Fleiss’s κ=0.69. The Materials and Methods say the images were free of clinical information; this paper does not report the claimed κ=0.88 for images combined with operative reports. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Statistical Methods for Rates and Proportions
Fleiss, J. L., Levin, B., & Paik, M. C. (2003) · Wiley Series in Probability and Statistics · doi:10.1002/0471445428 · Cited by 18,646 · see citing works
Statistical methods for assessing agreement between two methods of clinical measurement
Bland, J. M., & Altman, D. G. (2010) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2009.10.001 · Cited by 5,220 · see citing works
Revised American Society for Reproductive Medicine classification of endometriosis: 1996
American Society for Reproductive Medicine (1997) · Fertility and Sterility · doi:10.1016/s0015-0282(97)81391-x · Cited by 3,018 · see citing works
Revised American Fertility Society Classification of Endometriosis: 1985
(1985) · Fertility and Sterility · doi:10.1016/s0015-0282(16)48430-x · Cited by 1,803 · see citing works
ESHRE guideline for the diagnosis and treatment of endometriosis
Kennedy, S., Bergqvist, A., Chapron, C., D’Hooghe, T., Dunselman, G., Greb, R., Hummelshoj, L., Prentice, A., & Saridogan, E. (2005) · Human Reproduction · doi:10.1093/humrep/dei135 · Cited by 1,584 · see citing works
Endometriosis fertility index: the new, validated endometriosis staging system
Adamson, G. D., & Pasta, D. J. (2010) · Fertility and Sterility · doi:10.1016/j.fertnstert.2009.09.035 · Cited by 483 · see citing works
Incidence of endometriosis by study population and diagnostic method: the ENDO study
Buck Louis, G. M., Hediger, M. L., Peterson, C. M., Croughan, M., Sundaram, R., Stanford, J., Chen, Z., Fujimoto, V. Y., Varner, M. W., Trumble, A., et al. (2011) · Fertility and Sterility · doi:10.1016/j.fertnstert.2011.05.087 · Cited by 309 · see citing works
Self-citationClassification of Endometriosis
The American Fertility Society (1979) · Fertility and Sterility · doi:10.1016/s0015-0282(16)44409-2 · Cited by 305 · see citing works
Extracts from "Clinical Evidence": Endometriosis
Farquhar, C. M. (2000) · BMJ · doi:10.1136/bmj.320.7247.1449 · Cited by 173 · see citing works
6 Classification of endometriosis
Canis, M. J., de Jolinières, J. B., Wattiez, A., Pouly, J. L., Mage, G., MANHES, H., & Bruhat, M. A. (1993) · Baillière s Clinical Obstetrics and Gynaecology · doi:10.1016/s0950-3552(05)80462-6 · Cited by 119 · see citing works
Endometriosis classification: an update
Adamson, G. D. (2011) · Current Opinion in Obstetrics & Gynecology · doi:10.1097/gco.0b013e328348a3ba · Cited by 119 · see citing works
Chen, M. R. (2006) · Fertility and Sterility · doi:10.1016/j.fertnstert.2006.08.014 · Cited by 103 · see citing works
The revised American Fertility Society classification of endometriosis: reproducibility of scoring
Rock, J. A., & Zoladex Endometriosis Study Group (1995) · Fertility and Sterility · doi:10.1016/s0015-0282(16)57556-6 · Cited by 97 · see citing works
Haas, D., Chvátal, R., Habelsberger, A., et al. (2011) · Fertility and Sterility · doi:10.1016/j.fertnstert.2011.01.135 · Cited by 92 · see citing works
The reproducibility of the revised American Fertility Society classification of endometriosis
Hornstein, M. D., Gleason, R. E., Orav, E. J., Haas, S. T., Friedman, A. J., Rein, M. S., Hill, J. A., & Barbieri, R. L. (1993) · Fertility and Sterility · doi:10.1016/s0015-0282(16)55921-4 · Cited by 92 · see citing works
Non-invasive diagnosis of endometriosis: the goal or own goal?
Somigliana, E., Vercellini, P., Viganò, P., Benaglia, L., Crosignani, P. G., & Fedele, L. (2010) · Human Reproduction · doi:10.1093/humrep/deq141 · Cited by 55 · see citing works
The current staging system for endometriosis: does it help?
Roberts, C. P., & Rock, J. A. (2003) · Obstetrics and Gynecology Clinics of North America · doi:10.1016/s0889-8545(02)00056-6 · Cited by 50 · see citing works
Recognising, understanding and managing endometriosis
Fraser, I. S. (2008) · Journal of Human Reproductive Sciences · doi:10.4103/0974-1208.44112 · Cited by 46 · see citing works
Ultrasonographic staging: a new staging system for deep endometriosis
Coccia, M. E., & Rizzello, F. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.05951.x · Cited by 40 · see citing works
Interobserver variability in the diagnosis of minimal and mild endometriosis
Buchweitz, O., Wülfing, P., & Malik, E. (2005) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2005.02.002 · Cited by 30 · see citing works
Classification of endometriosis. The American Fertility Society.
(1979) · PubMed · Cited by 30 · see citing works
Intra-observer and inter-observer variability in scoring laparoscopic diagnosis of pelvic adhesions
Corson, S. L., Batzer, F. R., Gocial, B., et al. (1995) · Human Reproduction · doi:10.1093/humrep/10.1.161 · Cited by 27 · see citing works
Weijenborg, P. T. M., ter Kuile, M. M., & Jansen, F. W. (2006) · Fertility and Sterility · doi:10.1016/j.fertnstert.2006.06.052 · Cited by 18 · see citing works
Gynaecological laparoscopy: ‘see and treat’ should be the gold standard
Ball, E., Koh, C. H., Janik, G. M., & Davis, C. (2008) · Current Opinion in Obstetrics & Gynecology · doi:10.1097/gco.0b013e32830002bb · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Schliep, K., Chen, Z., Stanford, J., Xie, Y., Mumford, S., Hammoud, A., Boiman Johnstone, E., Dorais, J., Varner, M., Buck Louis, G., et al. (2017) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13711 · Cited by 39 · see citing works
Self-citationAdiposity and endometriosis severity and typology
Byun, J., Peterson, C. M., Backonja, U., Taylor, R. N., Stanford, J. B., Allen-Brady, K. L., Smith, K. R., Louis, G. M. B., & Schliep, K. C. (2020) · Journal of Minimally Invasive Gynecology · doi:10.1016/j.jmig.2020.01.002 · Cited by 27 · see citing works
“Endometriosis Severity and Typology Diagnosis and Definition All participating surgeons of the ENDO study had surgical training in the diagnosis and staging of endometriosis [1, 15].”
BackgroundResultSelf-citationExamining the co-occurrence of endometriosis and polycystic ovarian syndrome
Schliep, K. C., Ghabayen, L., Shaaban, M., Hughes, F. R., Pollack, A. Z., Stanford, J. B., Brady, K. A., Kiser, A., & Peterson, C. M. (2023) · AJOG Global Reports · doi:10.1016/j.xagr.2023.100259 · Cited by 9 · see citing works
“Never pregnant 155 (33) 71 (43) 12 (29) 10 (42) 62 (26)”
Self-citation
- Authors
- Schliep, K. C., Stanford, J. B., Chen, Z., Zhang, B., Dorais, J. K., Boiman Johnstone, E., Hammoud, A. O., Varner, M. W., Louis, G. M. B., & Peterson, C. M.
- Affiliations
- University of Utah (US)
- National Institutes of Health (US)
- ORCID
- Venue
- Obstetrics & Gynecology
- Published
- 2012-07
- Volume
- 120
- Issue
- 1
- Pages
- 104-112
- ISSN
- 0029-7844 (ISSN-L)
- PMC
- PMC8495766
- OpenAlex
- W2077088582
- Cited by
- 33
- References
- 27
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2 · 2023: 1 · 2022: 2 · 2021: 3 · 2020: 1 · 2019: 3 · 2018: 5 · 2017: 1 · 2016: 3 · 2015: 3 · 2014: 2 · 2013: 3 · 2012: 2
- FWCI
- 3.74
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Obstetrics and Gynecology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (Z01 HD008793, Z01 HD008791)
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (Z01HD008829, ZIAHD008829, L50 HD099868)
- Topics
- Health Sciences › Medicine › Reproductive Medicine › Endometriosis Research and Treatment
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Health Sciences › Medicine › Obstetrics and Gynecology › Endometrial and Cervical Cancer Treatments
- Keywords
- endometriosis, endometriosis diagnosis, endometriosis staging, inter-rater reliability, intra-rater reliability, disease severity assessment, gynecologic surgeons
- MeSH terms
- Adult, Endometriosis, Female, Humans, Severity of Illness Index, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
