CoChat

Clinical Scores in Veterinary Medicine: What Are the Pitfalls of Score Construction, Reliability, and Validation? A General Methodological Approach Applied in Cattle

Open access

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.

Abstract

Clinical scores are commonly used for cattle. They generally contain a mix of categorical and numerical variables that need to be assessed by scorers, such as farmers, animal caretakers, scientists, and veterinarians. This article examines the key concepts that need to be accounted for when developing the test for optimal outcomes. First, the target condition or construct that the scale is supposed to measure should be defined, and if possible, an adequate proxy used for classification should be determined. Then, items (e.g., clinical signs) of interest that are either caused by the target condition (reflective items) or that caused the target condition (formative items) are listed, and reliable items (inter and intra-rater reliability) are kept for the next step. A model is then developed to determine the relative weight of the items associated with the target condition. A scale is then built after validating the model and determining the optimal threshold in terms of sensitivity (ability to detect the target condition) and specificity (ability to detect the absence of the target condition). Its robustness to various scenarios of the target condition prevalence and the impact of the relative cost of false negatives to false positives can also be assessed to tailor the scale used based on specific application conditions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/ani11113244

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Treemap of the relative weights of different clinical signs included in various scoring systems used to detect bovine respiratory disease complex in calves. The size of the square is proportional to the specific clinical sign weight in the Wisconsin calf respiratory score (A) [7], California score (B) [8], Québec modified California score (C) [9], and Québec veal calf respiratory score (D) [10]. The map clearly shows the variable signs included and the relative impact on the total score of the same clinical sign.
Figure 1 Treemap of the relative weights of different clinical signs included in various scoring systems used to detect bovine respiratory disease complex in calves. The size of the square is proportional to the specific clinical sign weight in the Wisconsin calf respiratory score (A) [7], California score (B) [8], Québec modified California score (C) [9], and Québec veal calf respiratory score (D) [10]. The map clearly shows the variable signs included and the relative impact on the total score of the same clinical sign.CC BY · PubMed Central
Figure 2 General framework to establish and validate a clinical scoring system in veterinary medicine. Our 6-step framework is depicted with an initial definition of the target condition (1), the potential items of interest (2), and their reliability (3). The modeling approach then determines the adequate repartition between and within item weights (4). The accuracy of the score is determined in step (5). The optimal threshold selection is then adapted to the specific conditions where the test is intended to be used (6). Score users also need to be adequately trained.
Figure 2 General framework to establish and validate a clinical scoring system in veterinary medicine. Our 6-step framework is depicted with an initial definition of the target condition (1), the potential items of interest (2), and their reliability (3). The modeling approach then determines the adequate repartition between and within item weights (4). The accuracy of the score is determined in step (5). The optimal threshold selection is then adapted to the specific conditions where the test is intended to be used (6). Score users also need to be adequately trained.CC BY · PubMed Central
Figure 3 General framework to determine the specific items to be included for a specific construct or disease. Different items can be selected to assess a specific construct or determine a target condition. Reflective items are generally a reflection of the target condition or part of the description of the condition. In this example, cough or rectal temperature could be considered as part of the description of signs of respiratory disease (or the construct can cause them). By contrast, formative items can be considered as more causal items. They can cause the construct. In this example, diurnal variation of external temperature could be associated with an increased risk of bovine respiratory disease complex (BRD) but is not actually caused by BRD. For some items, it may be more difficult to determine both their formative and reflective roles. For example, the BCS could either be formative (e.g., calves with a low BCS could be at higher risk of being sick) or reflective (long disease duration may lead to a low BCS).
Figure 3 General framework to determine the specific items to be included for a specific construct or disease. Different items can be selected to assess a specific construct or determine a target condition. Reflective items are generally a reflection of the target condition or part of the description of the condition. In this example, cough or rectal temperature could be considered as part of the description of signs of respiratory disease (or the construct can cause them). By contrast, formative items can be considered as more causal items. They can cause the construct. In this example, diurnal variation of external temperature could be associated with an increased risk of bovine respiratory disease complex (BRD) but is not actually caused by BRD. For some items, it may be more difficult to determine both their formative and reflective roles. For example, the BCS could either be formative (e.g., calves with a low BCS could be at higher risk of being sick) or reflective (long disease duration may lead to a low BCS).CC BY · PubMed Central
Figure 4 Determination of the optimal threshold based on a specific score. This figure represents the California respiratory score threshold for detecting ultrasonographic lung consolidation (target condition defined as a maximal consolidation depth ≥ 1 cm) in 608 pre-weaned calves from 39 different dairy herds [9]. The accuracy of the score is presented in the y-axis with the sum of sensitivity (Se) and specificity (Sp) of the score depending on the threshold (≥ threshold to define a positive score, < for a negative score). The dots represent the observed value of the score, and the 95% confidence interval band was obtained after 1000 bootstrap estimates. Even though the threshold of ≥ 4 is numerically associated with the highest accuracy (dashed blue line), there is no evidence that this threshold is better than any threshold between 2 and 7 (dotted red lines) due to the confidence interval width.
Figure 4 Determination of the optimal threshold based on a specific score. This figure represents the California respiratory score threshold for detecting ultrasonographic lung consolidation (target condition defined as a maximal consolidation depth ≥ 1 cm) in 608 pre-weaned calves from 39 different dairy herds [9]. The accuracy of the score is presented in the y-axis with the sum of sensitivity (Se) and specificity (Sp) of the score depending on the threshold (≥ threshold to define a positive score, < for a negative score). The dots represent the observed value of the score, and the 95% confidence interval band was obtained after 1000 bootstrap estimates. Even though the threshold of ≥ 4 is numerically associated with the highest accuracy (dashed blue line), there is no evidence that this threshold is better than any threshold between 2 and 7 (dotted red lines) due to the confidence interval width.CC BY · PubMed Central
Figure 5 Misclassification cost-term (MCT) analysis of a specific score to determine lung consolidation. The same dataset as in Figure 3 was used to determine the misclassification cost term using: (A) the original prevalence of the target condition in the dataset (36% of calves were detected with ultrasonographic lung consolidation depth ≥ 1 cm); (B) a prevalence of 50%; and (C) a prevalence of 10% based on internal resampling. Different false negative (FN): false positive (FP) cost ratios (r) are indicated. In the first scenario, r was considered as 1:1, meaning that FN and FP had the same impact. Two scenarios where FN calves were considered more costly than FP calves with 3:1 and 5:1 r values. Finally, two scenarios considered that false-positive calves were more costly than FN cases with 1:3 and 1:5 r values. The 5 MCT curves are then derived to determine the score threshold robustness for various settings. The optimal threshold minimizes the y-axis value.
Figure 5 Misclassification cost-term (MCT) analysis of a specific score to determine lung consolidation. The same dataset as in Figure 3 was used to determine the misclassification cost term using: (A) the original prevalence of the target condition in the dataset (36% of calves were detected with ultrasonographic lung consolidation depth ≥ 1 cm); (B) a prevalence of 50%; and (C) a prevalence of 10% based on internal resampling. Different false negative (FN): false positive (FP) cost ratios (r) are indicated. In the first scenario, r was considered as 1:1, meaning that FN and FP had the same impact. Two scenarios where FN calves were considered more costly than FP calves with 3:1 and 5:1 r values. Finally, two scenarios considered that false-positive calves were more costly than FN cases with 1:3 and 1:5 r values. The 5 MCT curves are then derived to determine the score threshold robustness for various settings. The optimal threshold minimizes the y-axis value.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 39 references

Research works printed but not linked: 1

  1. Harrell, F.E. (2019). Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis, Springer. [2nd ed.].

Cited by

See all 20

Metadata

Authors
Buczinski, S., Boccardo, A., & Pravettoni, D.
Affiliations
  • Université de Montréal (CA)
  • University of Milan (IT)
Corresponding author
Buczinski, Sébastien
Venue
Animals
Publisher
MDPI AG
Published
2021-11-13
Volume
11
Issue
11
Pages
3244
ISSN
2076-2615 (ISSN-L)
OpenAlex
W3212240198
Cited by
15
References
38
Citations by year
2026: 3 · 2025: 5 · 2024: 2 · 2023: 4 · 2022: 1
FWCI
0.35
Type
journal article
Language
en
Open access
Locations
  • Animals (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Archivio Istituzionale della Ricerca (Universita Degli Studi Di Milano) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Life Sciences › Agricultural and Biological Sciences › Animal Science and Zoology › Animal health and immunology
  • Life Sciences › Agricultural and Biological Sciences › Animal Science and Zoology › Animal Behavior and Welfare Studies
Keywords
clinical scoring systems, cattle, scale development, reliability statistics, validation, inter-rater reliability, intra-rater reliability, item weighting, sensitivity and specificity, threshold optimization, prevalence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Clinical Scores in Veterinary Medicine: What Are the Pitfalls of Score Construction, Reliability, and Validation? A General Methodological Approach Applied in Cattle. https://app.cochat.ai/papers/10.3390/ani11113244
The paper
Buczinski, S., Boccardo, A., & Pravettoni, D. (2021). Clinical Scores in Veterinary Medicine: What Are the Pitfalls of Score Construction, Reliability, and Validation? A General Methodological Approach Applied in Cattle. Animals. https://doi.org/10.3390/ani11113244

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.