Cited by 30Microbial infections and disease researchAnimal health and immunologyAnimal Behavior and Welfare Studies

Evaluation of inter-rater agreement of the clinical signs used to diagnose bovine respiratory disease in individually housed veal calves

Open access
Berman, J., Francoz, D., Abdallah, A., Dufour, S., Buczinski, S.·2021·Journal of Dairy Science

Abstract

In dairy calves raised for veal, typical clinical signs of bovine respiratory disease (BRD) are ocular discharge, nasal discharge, ear droop or head tilt, abnormal respiration, cough, and increased rectal temperature. Despite the existence of several clinical scoring systems, there are few studies on the variability of human recognition of individual BRD clinical signs. The objective of this study was therefore to assess the inter-rater agreement of BRD clinical signs in veal calves. We hypothesized that BRD clinical signs were not detected equally between veterinarians, technicians, and producers of the veal industry and that some clinical signs have higher inter-rater agreement than others. During 2017-2018, we prospectively recorded 524 videos of physical examinations of random veal calves from 48 different batches in Québec, Canada. A researcher, not involved in the inter-rater assessment, classified each video as presence/absence of each BRD clinical sign except rectal temperature. For each of the 5 clinical signs, 15 videos with and 15 videos without the clinical signs were randomly selected to avoid kappa paradoxes. Those 30 videos were then presented in a random order to experienced raters of BRD in veal calves: 6 veterinarians, 6 technicians, and 6 producers. The raters assessed the clinical signs using scores based on the Wisconsin and California scoring system with modifications (0 = absent, 1 = mild, 2 = moderate, 3 = severe for nasal discharge, ocular discharge, and ear droop or head tilt; and 0 = absent, 1 = moderate, 2 = severe for abnormal respiration and induced cough). We used median percentage agreement (Pa), median Cohen's kappa (κ), and Gwet's agreement coefficient 1 (AC1) to assess inter-rater agreement. The effect of scale combination was also tested to determine the optimal combination (4-scale 0/1/2/3 vs. 3-scale 0/1/2 vs. 2-scale 0/1,2,3; 0,1/2,3; or 0/1,2). The differences of inter-rater agreement between veterinarians, technicians, and producers were estimated by a Wilcoxon rank-sum test. The 2-scale combination (0,1/2,3 or 0/1,2) had the highest inter-rater agreement for all clinical signs. With this combination, induced cough was the clinical sign with the highest inter-rater agreement (Pa = 0.93; κ = 0.79; AC1 = 0.87) and abnormal respiration was the sign with the lowest inter-rater agreement (Pa = 0.77; κ = 0.20; AC1 = 0.74). According to Pa and AC1 values, the 2-scale inter-rater agreement of the 5 clinical signs was good (value > 0.6). According to κ, only ear droop or head tilt and induced cough had a substantial 2-scale inter-rater agreement (κ > 0.6). In general, the 2-scale inter-rater agreement was better among veterinarians than among technicians and producers, except for the ear droop/head tilt, where agreement was better among producers. We concluded that with severity scores assessed on a scale of 2 (0,1/2,3 or 0/1,2), the inter-rater agreement of BRD clinical signs was variable according to the sign in veal calves. BRD clinical signs were not detected equally between veterinarians, technicians, and producers of the veal industry. Future research could determine if this discrepancy could be improved by standardization training.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3168/jds.2021-20503

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Incorporation of rectal temperature measurements with discharge, ear position, breathing, and cough indicators has potential to refine the process for straightforward and reproducible screening.

    “The incorporation of rectal temperature measurements with these indicators has the potential to refine the process for straightforward and reproducible screening [68].”

    From Strategies for Bovine Respiratory Disease (BRD) Diagnosis and Prognosis: A Comprehensive Overview (2024) · cites this paper as [68]

    The Conclusions propose that induced cough, ear droop or head tilt, and rectal temperature could be used to design a simple, repeatable screening score. They do not propose adding discharge or abnormal breathing to that score; the Discussion reports that abnormal breathing had the lowest inter-rater agreement. (LLM, conf=0.89)

    Experimental — accuracy not yet measured
  • Pre-awareness of evaluation criteria by raters may have introduced bias.

    “Pre-awareness of the evaluation criteria by raters may have introduced bias.”

    From Strategies for Bovine Respiratory Disease (BRD) Diagnosis and Prognosis: A Comprehensive Overview (2024) · cites this paper as [68]

    The Selection and Training of the Raters section says raters received explanations of the scoring criteria before viewing the videos and chose categories for targeted clinical signs. This supports their pre-awareness, but the paper does not say that it may have introduced bias. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • Of the combined scales, cough and ear position were the most reliable indicators.

    “Of these, cough and ear position were the most reliable indicators [68].”
  • Scoring consistency was highest using a combination of two scales for assessing discharge, ear position, breathing, and cough.

    “Scoring consistency was highest using a combination of two scales for assessing discharge, ear position, breathing, and cough [68].”
  • The possibility of rater interactions during simultaneous on-farm assessment could influence individual scores.

    “Additionally, the possibility of rater interactions during simultaneous on-farm assessment could influence individual scores [68].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 42 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Berman, J., Francoz, D., Abdallah, A., Dufour, S., & Buczinski, S.
Affiliations
  • Université de Montréal (CA)
  • Zagazig University (EG)
Corresponding author
Berman, J.
Venue
Journal of Dairy Science
Published
2021-11
Volume
104
Issue
11
Pages
12053-12065
ISSN
0022-0302 (ISSN-L)
OpenAlex
W3196867335
Cited by
30
References
42
Citations by year
2026: 4 · 2025: 7 · 2024: 6 · 2023: 5 · 2022: 6 · 2021: 2
FWCI
3.35
Type
journal article
Language
en
Open access
Locations
  • Journal of Dairy Science (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Funders
  • Ministère de l'Agriculture, des Pêcheries et de l'Alimentation
  • Fonds Québécois de la Recherche sur la Nature et les Technologies
Topics
  • Life Sciences › Immunology and Microbiology › Microbiology › Microbial infections and disease research
  • Health Sciences › Veterinary › Small Animals › Animal health and immunology
  • Health Sciences › Veterinary › Small Animals › Animal Behavior and Welfare Studies
Keywords
bovine respiratory disease, veal calves, interobserver agreement, clinical signs, clinical scoring systems, ocular discharge, rhinorrhea, rectal temperature, individual housing, dairy cattle, video-based diagnosis, veterinarians, technicians, producers, kappa statistic, Gwet's AC1
MeSH terms
Red Meat, Animals, Canada, Cattle, Cattle Diseases, Quebec, Respiratory Tract Diseases

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review