Inter-rater agreement and reliability of thoracic ultrasonographic findings in feedlot calves, with or without naturally occurring bronchopneumonia
Open accessAbstract
Background: Thoracic ultrasonography (TUS) can be used to assess the extent and severity of lung lesions associated with bronchopneumonia (BP) in feedlot cattle. Hypothesis/Objectives: To assess inter-rater agreement and reliability of TUS findings in feedlot cattle, with or without naturally occurring BP. Animals: Feedlot steers with (n = 210) or without (n = 107) clinical signs of BP that were assessed by TUS in a previous case-control study. Methods: A random sample of 50 TUS videos (16-s duration) were scored by 6 raters with various levels of TUS expertise. Lung consolidation, comet tail artifacts, pleural irregularity and effusion were scored. Inter-rater agreement was assessed using raw percentage of agreement (Pa), Cohen's and Fleiss’ Kappa (κ), and Gwet agreement coefficient (AC1). Intra-class correlation (ICC) was determined for variables with continuous measurements (mixed factorial design). Results: Median (interquartile range [IQR]) Pa were 0.84 (0.80-0.89), 0.82 (0.80-0.87), 0.62 (0.53-0.67), and 0.82 (0.75-0.86) for presence of lung consolidation, comet tails, pleural irregularity, and pleural effusion, respectively. For the same lesions, Fleiss κ (95% confidence intervals [CI]) were 0.67 (0.49-0.86), 0.56 (0.33-0.80), 0.20 (−0.05 to 0.44), and 0.36 (0.10-0.61), respectively. AC1 were 0.68 (0.51-0.86), 0.73 (0.58-0.89), 0.21 (−0.01 to 0.44), and 0.71 (0.51-0.92), respectively. Moderate reliability was found among raters for all quantitative variables (ICC ranged from 0.52 to 0.70). Conclusions and Clinical Importance: Inter-rater agreement was good for presence of lung consolidation, comet tails and pleural effusion (based on Pa and AC1) but was slight to poor for pleural irregularity.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Complementary diagnostic methods such as laboratory testing, imaging, molecular diagnostics, and biomarker tracking can offer additional support but are usually prohibitively expensive.
“Complementary diagnostic methods such as laboratory testing, imaging, molecular diagnostics, and biomarker tracking can offer additional support but are usually prohibitively expensive [26,27,28,29,30,31,32,33,34,35].”
From Strategies for Bovine Respiratory Disease (BRD) Diagnosis and Prognosis: A Comprehensive Overview (2024) · cites this paper as [32]
The Introduction says thoracic ultrasonography can help confirm lung lesions in feedlot cattle, supporting the imaging portion of the claim. It does not say that imaging or the other listed methods are usually prohibitively expensive, or discuss laboratory testing, molecular diagnostics, or biomarker tracking in that context (searched for cost and expense). (LLM, conf=0.86)
Experimental — accuracy not yet measuredStatistical analyses using Cohen's kappa and Fleiss' kappa revealed only slight to moderate agreement among raters after training.
“Moreover, statistical analyses using Cohen’s kappa and Fleiss’ kappa revealed only slight to moderate agreement among raters after training [32].”
From Strategies for Bovine Respiratory Disease (BRD) Diagnosis and Prognosis: A Comprehensive Overview (2024) · cites this paper as [32]
The Rater selection section says all six raters received 30 minutes of training, and the Results report agreement measured with Cohen’s and Fleiss’ κ. But the claim’s ‘only slight to moderate’ characterization omits lung consolidation: its Fleiss’ κ was 0.67 and its median Cohen’s κ was 0.68, values the paper classifies as good agreement. Page numbers are not available in our copy. (LLM, conf=0.90)
Experimental — accuracy not yet measuredTUS is a dependable method for assessing lung consolidation, even when performed by individuals with minimal experience.
“Research indicates that TUS is a dependable method for assessing lung consolidation, even when performed by individuals with minimal experience [32,86,146,147].”
From Strategies for Bovine Respiratory Disease (BRD) Diagnosis and Prognosis: A Comprehensive Overview (2024) · cites this paper as [32]
The Rater selection section includes two beginners in thoracic ultrasound, and the Discussion reports good agreement for detecting the presence of lung consolidation across raters. But the Discussion specifies that raters interpreted videos rather than performed TUS examinations; it also reports only moderate reliability for measuring consolidation depth or area. The paper therefore supports dependable detection during video interpretation, not the broader claim about TUS performed by minimally experienced individuals. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-61 · Cited by 1,041 · cited 1× · see citing works
“Gwet's AC1 provides a chance-corrected agreement coefficient, in line with the percentage level of agreement.(13,14) This agreement measure is useful for interpreting tests with high raw agreement percentages but low κ values due to κ paradoxes (which can occur when a reported anomaly has low prevalence(15)).”
ResultOn-Farm Use of Ultrasonography for Bovine Respiratory Disease
Ollivett, T. L., & Buczinski, S. (2016) · Veterinary Clinics of North America: Food Animal Practice · doi:10.1016/j.cvfa.2015.09.001 · Cited by 177 · see citing works
Self-citationTimsit, E., Dendukuri, N., Schiller, I., & Buczinski, S. (2016) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2016.11.006 · Cited by 112 · see citing works
Self-citationUltrasonography of the Bovine Respiratory System and Its Practical Application
Babkine, M., & Blond, L. (2009) · Veterinary Clinics of North America: Food Animal Practice · doi:10.1016/j.cvfa.2009.07.001 · Cited by 57 · see citing works
Buczinski, S., Faure, C., Jolivet, S., & Abdallah, A. (2016) · New Zealand Veterinary Journal · doi:10.1080/00480169.2016.1153439 · Cited by 39 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 18
- [1]Wolfger B, Timsit E, White BJ, et al. A systematic review of bovine respiratory disease diagnosis focused on diagnostic confirmation, early detection, and prediction of unfavorable outcomes in feedlot cattle. Vet Clin North Am Food Anim Pract. 2015; 31: 351– 365. 26210764
- [3]Mang AV, Buczinski S, Booker CW, et al. Evaluation of a computer‐aided lung auscultation system for diagnosis of bovine respiratory disease in feedlot cattle. JVet Intern Med. 2015; 29: 1112– 1116. 26059327 PMC4895372
- [5]Rademacher RD, Buczinski S, Tripp HM, et al. Systematic thoracic ultrasonography in acute bovine respiratory disease of feedlot steers: impact of lung consolidation on diagnosis and prognosis in a case‐control study. Bov Pract. 2014; 48: 1– 10.
- [6]Mayo PH, Beaulieu Y, Doelken P, et al. American College of Chest Physicians/La Societe de Reanimation de Langue Francaise statement on competence in critical care ultrasonography. Chest. 2009; 135: 1050– 1060. 19188546
- [7]Kottner J, Audige L, Brorson S, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. JClin Epidemiol. 2011; 64: 96– 106. 21130355
- [8]Timsit E, Hallewell J, Booker C, et al. Prevalence and antimicrobial susceptibility of Mannheimia haemolytica, Pasteurella multocida, and Histophilus somni isolated from the lower respiratory tract of healthy feedlot cattle and those diagnosed with bovine respiratory disease. Vet Microbiol. 2017; 208: 118– 125. 28888626
- [9]Buczinski S, Forté G, Bélanger AM. Short communication: ultrasonographic assessment of the thorax as a fast technique to assess pulmonary lesions in dairy calves with bovine respiratory disease. JDairy Sci. 2013; 96: 4523– 4528. 23628251
- [10]Burn CC, Weir AA. Using prevalence indices to aid interpretation and comparison of agreement ratings between two or more observers. Vet J. 2011; 188: 166– 170. 20570535
- [12]Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20: 37.
- [15]Walsh P, Thornton J, Asato J, et al. Approaches to describing inter‐rater reliability of the overall clinical appearance of febrile infants and toddlers in the emergency department. PeerJ. 2014; 2: e651. 25401054 PMC4230550
- [19]Ollivett TL, Caswell JL, Nydam DV, et al. thoracic ultrasonography and bronchoalveolar lavage fluid analysis in holstein calves with subclinical lung lesions. JVet Intern Med. 2015; 29: 1728– 1734. 26332345 PMC4895683
- [21]Love WJ, Lehenbauer TW, Van Eenennaam AL, et al. Sensitivity and specificity of on‐farm scoring systems and nasal culture to detect bovine respiratory disease complex in preweaned dairy calves. JVet Diagn Invest. 2016; 28: 119– 128. 26796957
- [22]Teixeira AG, McArt JA, Bicalho RC. Thoracic ultrasound assessment of lung consolidation at weaning in Holstein dairy heifers: reproductive performance and survival. JDairy Sci. 2017; 100: 2985– 2991. 28215891
- [23]Hoehler FK. Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. JClin Epidemiol. 2000; 53: 499– 503. 10812322
- [24]Nazerian P, Volpicelli G, Vanni S, et al. Accuracy of lung ultrasound for the diagnosis of consolidations when compared to chest computed tomography. Am J Emerg Med. 2015; 33: 620– 625. 25758182
- [25]Ellington LE, Gilman RH, Chavez MA, et al. Lung ultrasound as a diagnostic tool for radiographically‐confirmed pneumonia in low resource settings. Respir Med. 2017; 128: 57– 64. 28610670 PMC5480773
- [26]Rademacher RMBS, Edmonds M, Tripp HT, Johnson E. Systematic thoracic ultrasonography in acute bovine respiratory disease of feedlot steers: impact of lung consolidation on diagnosis and prognosis in a case‐control study. Bov Prac. 2014; 41: 1– 10.
- [27]Rabeling B, Rehage J, Dopfer D, et al. Ultrasonographic findings in calves with respiratory disease. Vet Rec. 1998; 143: 468– 471. 9829303
Cited by
Sorted by citations
Bovine Respiratory Disease Diagnosis
Buczinski, S., & Pardon, B. (2020) · Veterinary Clinics of North America: Food Animal Practice · doi:10.1016/j.cvfa.2020.03.004 · Cited by 89 · see citing works
Self-citationStrategies for Bovine Respiratory Disease (BRD) Diagnosis and Prognosis: A Comprehensive Overview
Kamel, M. S., Davidson, J. L., & Verma, M. S. (2024) · Animals · doi:10.3390/ani14040627 · Cited by 76 · see citing works
“Moreover, statistical analyses using Cohen’s kappa and Fleiss’ kappa revealed only slight to moderate agreement among raters after training [32].”
MethodBuczinski, S., Boccardo, A., & Pravettoni, D. (2021) · Animals · doi:10.3390/ani11113244 · Cited by 15 · see citing works
Self-citation
- Authors
- Buczinski, S., Buathier, C., Bélanger, A. M., Michaux, H., Tison, N., & Timsit, E.
- Affiliations
- Université de Montréal (CA)
- University of Calgary (CA)
- Corresponding author
- Buczinski, S.
- Venue
- Journal of Veterinary Internal Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-09-01
- Volume
- 32
- Issue
- 5
- Pages
- 1787-1792
- ISSN
- 0891-6640 (ISSN-L)
- 1939-1676
- PMC
- PMC6189347
- OpenAlex
- W2888560867
- Cited by
- 45
- References
- 7
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 7 · 2023: 5 · 2022: 5 · 2021: 6 · 2020: 6 · 2019: 3
- FWCI
- 3.03
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Veterinary Internal Medicine (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Immunology and Microbiology › Microbiology › Microbial infections and disease research
- Health Sciences › Medicine › Critical Care and Intensive Care Medicine › Ultrasound in Clinical Applications
- Health Sciences › Veterinary › Small Animals › Animal health and immunology
- Keywords
- thoracic ultrasound, bronchopneumonia, feedlot steers, interobserver agreement, diagnostic reliability, lung consolidation, comet-tail artifact, pleural effusion, rater expertise, Fleiss' kappa, intra-class correlation, video-based scoring, bovine respiratory disease, herd health surveillance
- MeSH terms
- Animals, Bronchopneumonia, Cattle, Cattle Diseases, Male, Ultrasonography, Reproducibility of Results, Observer Variation, Case-Control Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
