Inter-rater and intra-rater reliability and agreement of echocardiographic diagnosis of rheumatic heart disease using the World Heart Federation evidence-based criteria
Open accessAbstract
Objective: Different definitions have been used for screening for rheumatic heart disease (RHD). This led to the development of the 2012 evidence-based World Heart Federation (WHF) echocardiographic criteria. The objective of this study is to determine the intra-rater and inter-rater reliability and agreement in differentiating no RHD from mild RHD using the WHF echocardiographic criteria. Methods: A standard set of 200 echocardiograms was collated from prior population-based surveys and uploaded for blinded web-based reporting. Fifteen international cardiologists reported on and categorised each echocardiogram as no RHD, borderline or definite RHD. Intra-rater and inter-rater reliability was calculated using Cohen’s and Fleiss’ free-marginal multirater kappa (κ) statistics, respectively. Agreement assessment was expressed as percentages. Subanalyses assessed reproducibility and agreement parameters in detecting individual components of WHF criteria. Results: Sample size from a statistical standpoint was 3000, based on repeated reporting of the 200 studies. The inter-rater and intra-rater reliability of diagnosing definite RHD was substantial with a kappa of 0.65 and 0.69, respectively. The diagnosis of pathological mitral and aortic regurgitation was reliable and almost perfect, kappa of 0.79 and 0.86, respectively. Agreement for morphological changes of RHD was variable ranging from 0.54 to 0.93 κ. Conclusions: The WHF echocardiographic criteria enable reproducible categorisation of echocardiograms as definite RHD versus no or borderline RHD and hence it would be a suitable tool for screening and monitoring disease progression. The study highlights the strengths and limitations of the WHF echo criteria and provides a platform for future revisions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In a study involving a team of cardiologists, inter-rater agreement as measured by Cohen's κ was as high as κ=0.97.
“In the same study referenced in section Internal Consistency of Human decision-making Processes, involving a team of cardiologists33, the values of inter-rater agreement, as measured by Cohen’s κ, were as high as κ=0.97.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [33]
The secondary endpoints section reports an inter-rater κ of 0.97 for detecting restricted aortic-valve leaflet motion, so the value is real. But the statistical analysis section explicitly says inter-rater reliability was calculated using Fleiss’ free-marginal multi-rater kappa; Cohen’s kappa was used for intra-rater reliability, contrary to the claim. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredDetecting some conditions was much harder even for experienced doctors, resulting in lower intra-rater reliability κ∈[0.4,0.85].
“However, detecting some conditions was much harder even for experienced doctors, resulting in a lower intra-rater reliability κ∈[0.4,0.85].”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [33]
Table 2 (page unavailable in our copy) reports lower intra-rater κ values for some rheumatic-heart-disease features, including 0.37 for chordal thickening and 0.41 for excessive leaflet motion. The limitations section says the readers were cardiologists experienced in RHD. The claimed 0.4–0.85 range is not the reported range of median intra-rater values: some are below 0.4, while pathological mitral regurgitation reaches 0.92. (LLM, conf=0.83)
Experimental — accuracy not yet measuredThe average values for inter-rater agreement are similar to the real-world values reported in the previously mentioned study, with E[κ]=0.49 and E[pa]=0.75.
“We also note that the average values for the inter-rater agreement are similar to the real-world values reported in the previously mentioned study33, with E[κ]=0.49 and E[pa]=0.75.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [33]
The Primary endpoint: RHD section reports a mean inter-rater κ of 0.49. Table 2 reports inter-rater proportion of agreement of 66.29%, not 75%; its approximately 75% figure (74.75%) is for intra-rater agreement. (LLM, conf=0.90)
Experimental — accuracy not yet measuredIn a study involving cardiological diagnostic tests, intra-rater agreement of cardiologists was consistently above κ=0.8 for some medical tests.
“In a study involving a series of cardiological diagnostic tests33, intra-rater agreement of cardiologists was found to be consistently above κ=0.8 for some medical tests.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [33]
In a more challenging diagnostic task, inter-rater reliability was on average κ=0.49, and the proportion of agreement between experts was pa=66.29%.
“It was found that in a more challenging diagnostic task, inter-rater reliability was, on average, κ=0.49. The proportion of agreement between experts was also reported at pa=66.29%.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [33]
The lower intra-rater reliability was especially noticeable when disease prevalence was very high or very low, as is the case in the financial fraud scenario.
“This was especially noticeable when the disease prevalence was very high or very low, as is the case in our financial fraud scenario.”
From A benchmarking framework and dataset for learning to defer in human-AI decision-making (2025) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 27
- Watkins DA, Johnson CO, Colquhoun SM, et al. Global, regional, and national burden of rheumatic heart disease, 1990-2015. N Engl J Med Overseas Ed 2017;377:713-22. · DOI
- Zühlke L, Engel ME, Karthikeyan G, et al. Characteristics, complications, and gaps in evidence-based interventions in rheumatic heart disease: the global rheumatic heart disease registry (the remedy study). Eur Heart J 2015;36:1115-22. · DOI
- Kassem AS, el-Walili TM, Zaher SR, et al. Reversibility of mitral regurgitation following rheumatic fever: clinical profile and echocardiographic evaluation. Indian J Pediatr 1995;62:717-23. · DOI
- Tompkins DG, Boxerbaum B, Liebman J. Long-term prognosis of rheumatic fever patients receiving regular intramuscular benzathine penicillin. Circulation 1972;45:543-51. · DOI
- Roberts KV, Brown ADH, Maguire GP, et al. Utility of auscultatory screening for detecting rheumatic heart disease in high-risk children in Australia's Northern Territory. Med J Aust 2013;199:196-9. · DOI
- Marijon E, Ou P, Celermajer DS, et al. Prevalence of rheumatic heart disease detected by echocardiographic screening. N Engl J Med 2007;357:470-6. · DOI
- Reményi B, Wilson N, Steer A, et al. World Heart Federation criteria for echocardiographic diagnosis of rheumatic heart disease-an evidence-based guideline. Nat Rev Cardiol 2012;9:297-309. · DOI
- Gewitz MHet al. Revision of the Jones criteria for the diagnosis of acute rheumatic fever in the era of Doppler echocardiography: a scientific statement from the American Heart Association. Circulation 2015;131:1806-18.
- Saxena A. Echocardiographic diagnosis of chronic rheumatic valvular lesions. Global Heart 2013;8:203-12. · DOI
- Roberts K, Maguire G, Brown A, et al. Echocardiographic screening for rheumatic heart disease in high and low risk Australian children. Circulation 2014;129:1953-61. · DOI
- Roberts KV, Maguire GP, Brown A, et al. Rheumatic heart disease in indigenous children in northern Australia: differences in prevalence and the challenges of screening. Med J Aust 2015;203. · DOI
- Webb RH, Gentles TL, Stirling JW, et al. Valvular regurgitation using portable echocardiography in a healthy student population: implications for rheumatic heart disease screening. J Am Soc Echocardiogr 2015;28:981-8. · DOI
- Clark BC, Krishnan A, McCarter R, et al. Using a low-risk population to estimate the specificity of the World Heart Federation criteria for the diagnosis of rheumatic heart disease. J Am Soc Echocardiogr 2016;29. · DOI
- Lu JC, Sable C, Ensing GJ, et al. Simplified rheumatic heart disease screening criteria for handheld echocardiography. J Am Soc Echocardiogr 2015;28:463-9. · DOI
- Engelman D, Kado JH, Reményi B, et al. Focused cardiac ultrasound screening for rheumatic heart disease by briefly trained health workers: a study of diagnostic accuracy. The Lancet Global Health 2016;4:e386-94. · DOI
- Nascimento BR, Nunes MCP, Lopes ELV, et al. Rheumatic heart disease echocardiographic screening: approaching practical and affordable solutions. Heart 2016;102:658-64. · DOI · Paper page
- Grayburn PA, Bhella P. Grading severity of mitral regurgitation by echocardiography: science or art? JACC Cardiovasc Imaging 2010;3:244-6. · DOI
- Biner S, Rafique A, Rafii F, et al. Reproducibility of proximal isovelocity surface area, vena contracta, and regurgitant jet area for assessment of mitral regurgitation severity. JACC: Cardiovascular Imaging 2010;3:235-43. · DOI
- Kottner J, Audigé L, Brorson S, et al. Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology 2011;64:96-106. · DOI
- Webb RH, Wilson NJ, Lennon DR, et al. Optimising echocardiographic screening for rheumatic heart disease in New Zealand: not all valve disease is rheumatic. Cardiol Young 2011;21:436-43. on July 11, 2023 by guest. Protected by copyright. http://heartasia.bmj.com/ Heart Asia: first published as 10.1136/heartasia-2019-011233 on 24 June 2019. Downloaded from Original research · DOI
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74. · DOI · Paper page
- de Vet HCW, Mokkink LB, Terwee CB, et al. Clinicians are right not to like Cohen's. BMJ 2013;346:f2125. · DOI
- Ooms EA, Zonderland HM, Eijkemans MJC, et al. Mammography: interobserver variability in breast density assessment. The Breast 2007;16:568-76. · DOI
- Stoler MH, Schiffman M. Interobserver reproducibility of cervical cytologic and histologic interpretations realistic estimates from the ASCUS-LSIL triage study. JAMA 2001;285:1500-5.
- Webb RH, Culliford-Semmens N, Sidhu K, et al. Normal echocardiographic mitral and aortic valve thickness in children. Heart Asia 2017;9:70-5. · DOI
- Ploutz M, Lu JC, Scheel J, et al. Handheld echocardiographic screening for rheumatic heart disease by non-experts. Heart 2016;102:35-9. · DOI
- Engelman D, Okello E, Beaton A, et al. Evaluation of computer-based training for health workers in echocardiography for RhD. Global Heart 2017;12:17-23. on July 11, 2023 by guest. Protected by copyright. http://heartasia.bmj.com/ Heart Asia: first published as 10.1136/heartasia-2019-011233 on 24 June 2019. Downloaded from · DOI
Cited by
Sorted by citations
A benchmarking framework and dataset for learning to defer in human-AI decision-making
Alves, J. V., Leitão, D., Jesus, S., Sampaio, M. O. P., Liébana, J., Saleiro, P., Figueiredo, M. A. T., & Bizarro, P. (2025) · Scientific Data · doi:10.1038/s41597-025-04664-y · Cited by 10 · see citing works
- Authors
- Remenyi, B., Carapetis, J., Stirling, J. W., Ferreira, B., Kumar, K., Lawrenson, J., Marijon, E., Mirabel, M., Mocumbi, A. O., Mota, C., Paar, J., Saxena, A., Scheel, J., Viali, S., Vijayalakshmi, I. B., Wheaton, G. R., Zuhlke, L., Sidhu, K., Dimalapang, E., Gentles, T. L., & Wilson, N. J.
- Affiliations
- The Kids Research Institute Australia (AU)
- Starship Children's Health (NZ)
- Amrita Institute of Medical Sciences and Research Centre (IN)
- Stellenbosch University (ZA)
- Inserm (FR)
- Universidade Federal de Minas Gerais (BR)
- Children's National (US)
- Sri Jayadeva Institute of Cardiovascular Sciences and Research (IN)
- Women's and Children's Hospital (AU)
- University of Cape Town (ZA)
- Auckland City Hospital (NZ)
- University of Auckland (NZ)
- ORCID
- Venue
- Heart Asia
- Publisher
- BMJ
- Published
- 2019-06
- Volume
- 11
- Issue
- 2
- Pages
- e011233
- ISSN
- 1759-1104 (ISSN-L)
- PMC
- PMC6591009
- OpenAlex
- W2955035813
- Cited by
- 29
- References
- 0
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 5 · 2023: 6 · 2022: 2 · 2021: 7 · 2020: 5
- FWCI
- 3.22
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Heart Asia (publishedVersion)
- PubMed Central (publishedVersion)
- UWA Profiles and Research Repository (University of Western Australia) (publishedVersion, other-oa)
- PubMed (publishedVersion)
- UWA Profiles and Research Repository (UWA) (submittedVersion, other-oa)
- HighWire Press Open Archive (submittedVersion)
- Funders
- Lowitja Institute
- Medical Research Council (MR/S005242/1)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Streptococcal Infections and Treatments
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Cardiac Valve Diseases and Treatments
- Keywords
- rheumatic heart disease, echocardiographic diagnosis, inter-rater reliability, intra-rater reliability, diagnostic agreement, mitral regurgitation, aortic regurgitation, disease progression monitoring
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
