CoChat
Sign Up
Cited by 32Reliability and Agreement in MeasurementDental Health and Care UtilizationOral microbiology and periodontitis research

Examiner training and reliability in two randomized clinical trials of adult dental caries.

Open access
Banting, David W., Amaechi, Bennett T., Bader, James D., Blanchard, Peter, Gilbert, Gregg H., Gullion, Christina M., Holland, Jan Carlton, Makhija, Sonia K. +4 more·2011·Journal of public health dentistry
DOI 10.1111/j.1752-7325.2011.00278.xPMC PMC3683245
Open this paper in CoChat — free account

Abstract

This report describes the training of dental examiners participating in two dental caries clinical trials and reports the inter- and intra-examiner reliability scores from the initial standardization sessions. Intra-examiner reliability was notably higher than inter-examiner reliability for all measures and dental caries classifications employed.

Integrity signals

Registry

Crossref · 10.1111/j.1752-7325.2011.00278.x

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

1 of 16 cited sources checked · 1 supported · 0 flagged

  • Premisecitation

    Only a small number of trials or epidemiologic studies have reported detailed reliability data for examinations including non-cavitated dental caries.

    “Only a small number of trials or epidemiologic studies have reported detailed reliability data for examinations that include non-cavitated dental caries (18–24).”
    Quote in the paperNot a substantial claim1 of 7 sources checked
    • cites [18] Ismail, 1992, Prevalence of non-cavitated and cavitated carious lesions in a random sample of 7-9-year-old schoolchildren in Montreal, Quebec, Community Dent Oral Epidemiol. source
    • cites [19] Nyvad, 1999, Reliability of a new caries diagnostic system differentiating between active and inactive caries lesions, Caries Res. source
    • cites [20] Fyffe, 2000, Effect of diagnostic threshold on the validity and reliability of epidemiological caries diagnosis using the Dundee Selectable Threshold Method for caries diagnosis (DSTM), Community Dent Oral Epidemiol. source
    • cites [21] Assaf, 2006, Effect of different diagnostic thresholds on dental caries calibration, J Public Health Dent. source
    • cites [22] Assaf, 2006, Effect of different diagnostic thresholds on dental caries calibration - a 12 month evaluation, Community Dent Oral Epidemiol. source
    • cites [23] Kühnisch, 2008, Occlusal caries detection in permanent molars according to WHO basic methods, ICDAS II and laser fluorescence measurements, Community Dent Oral Epidemiol. source
    • cites [24] Braga, 2009, Feasibility of the International Caries Detection and Assessment System (ICDAS-II) in epidemiological surveys and comparability with standard World Health Organization criteria, Caries Res. source
  • Findingown data

    Intra-examiner reliability was notably higher than inter-examiner reliability for all measures and caries classification systems.

    “Intra-examiner reliability was notably higher than inter-examiner reliability for all measures and dental caries classification systems employed.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Weighted intra-examiner kappa for the S versus D1 versus D2/D3 classification ranged from 0.65 to 0.77.

    “The weighted intra-examiner kappa values for the S vs. D1 vs. D2/D3 classification of dental caries ranged from 0.65 to 0.77”
    Quote in the paperSubstantial claim
  • Findingown data

    Intra-examiner agreement was highest for the S/D1 versus D2/D3 comparison.

    “Like inter-examiner agreement, intra-examiner agreement was best when comparing the S/D1 vs. D2/D3 calls (Table 4).”
    Quote in the paperNot a substantial claim
  • Findingnone given

    Examiner performance was approximately the same across all caries classifications.

    “Examiner performance was about the same for all of the classifications of dental caries.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 17
  • Classified as probably not a substantial claim. 8

  • May combine two claims.

    Findingown data

    Weighted inter-examiner kappas were generally higher than unweighted kappas and ranged from 0.34 to 0.53.

    “Weighted kappas were generally higher than the corresponding unweighted kappas, ranging from 0.34 to 0.53.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the standardization sessions, the prevalence of coronal dental caries (D1, D2, or D3) detected by the study examiners was low, ranging from 1.4% to 13.5% of examined surfaces, and most lesions were non-cavitated D1.

    “The overall prevalence of coronal dental caries (D1, D2 and D3 lesions) detected by the study examiners was low, ranging from 1.4% to 13.5% of the surfaces examined, and the majority of these were non-cavitated (D1) (Table 2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Intra-examiner agreement ranged from 0.86 to 1.00, with only examiner #1 at the X-ACT session below 0.90.

    “Intra-examiner agreement was excellent, ranging from 0.86 to 1.00, with only examiner # 1 at the X-ACT session falling below 0.90.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Analysis of numerically coded scores showed some examiner variability: examiner #3 at PACS west coast scored lower and examiner #1 at X-ACT scored higher than the others.

    “Analysis of the numerically coded scores revealed some evidence of examiner variability, with one examiner (# 3 at the PACS west coast session) scoring somewhat lower and one examiner (#1 at the X-ACT session) scoring somewhat higher than the others.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The 0.60 ICC threshold was exceeded by all examiner pairs except those involving examiner #3 at the PACS west coast session; those pairs ranged 0.41-0.47, while other pairs ranged 0.67-0.87.

    “The acceptable ICC threshold of 0.60 was surpassed (range 0.67–0.87) for all pairs of examiners except those involving examiner #3 at the PACS west coast session (range 0.41–0.47).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Intra-examiner kappa values varied markedly across examiners and were highest for the two PACS east coast examiners.

    “These kappas varied markedly across examiners, but were highest for the two PACS east coast examiners.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Examiner #3 was given additional training and a second standardization session but was ultimately replaced during the study.

    “This examiner was subsequently provided additional training and underwent a second standardization session but was ultimately replaced during the study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Examiner reliability was highest for the S/D1 versus D2/D3 classification and lowest for the three-level S versus D1 versus D2/D3 classification.

    “One fairly consistent pattern that emerged from our analyses was that examiner reliability (whether expressed as intra- or inter-examiner agreement) was highest for comparisons of the two-level S/D1 vs. D2/D3 classification and lowest for comparisons of the three-level S vs. D1 vs. D2/D3 classification.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Acceptable inter-examiner reliability thresholds were set at greater than 0.90 for overall agreement, 0.60 for ICC and 2x2 unweighted kappa, and 0.65 for weighted kappa; intra-examiner thresholds were 0.70 for unweighted kappa and 0.75 for weighted kappa.

    “For inter-examiner analyses, we considered the acceptable values to be greater than 0.90 for overall agreement, 0.60 for ICC and 2×2 unweighted kappa and 0.65 for weighted kappa. Corresponding thresholds for intra-examiner kappa statistics were estimated to be higher, with the minimum acceptable level set to 0.70 for unweighted kappa and 0.75 for weighted kappa.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: There are no definitive criteria for acceptable levels of examiner agreement and reliability.

  • May combine two claims.

    Premisecitation

    The maximum possible kappa agreement is often below unity because kappa values are influenced by outcome prevalence and examiner bias.

    “The maximum possible agreement is often less than unity because kappa values are influenced by the prevalence of the outcome under study and by the amount of bias present between examiners(12, 13).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [12] Byrt, 1993, Bias, prevalence and kappa, J Clin Epidemiol. source
    • cites [13] Sim, 2005, The kappa statistic in reliability studies: use, interpretation, and sample size requirements, Phys Ther. source

  • May combine two claims.

    Interpretationreasoning

    Comparisons between this study and other reported studies are difficult because those studies used different diagnostic systems, had higher caries prevalence, examined children and young adults, and did not always report outcome classification.

    “Comparisons between this study and those studies are difficult because in the other reported studies, different diagnostic systems were used, the prevalence of dental caries was quite high compared to this study, the other studies examined children and young adult subjects whereas we examined only adult subjects and, in the other studies, the outcome classification was not always reported”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The four-level S/D1/D2/D3 caries classification used in PACS and X-ACT is categorical, so the ICC is a valid measure of examiner reliability.

    “We contend that the four-level (S, D1, D2 and D3) caries classification system used in the PACS and X-ACT studies is categorical and, therefore, the ICC is a valid measure of examiner reliability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Several investigators consider ICC superior to weighted kappa for multi-level outcome measures.

    “In fact, several investigators consider the ICC to be superior to a weighted kappa when there are multi-level outcome measures (9, 10).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [9] Fleiss, 1973, The equivalence of weighted kappa and the intraclass correlation coefficient as a measure of reliability, Educ Psychol Meas. source
    • cites [10] Maclure, 1987, Misinterpretation and misuse of the kappa statistic, Am J Epidemiol. source

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation

    Some investigators argue kappa is interpretable only when the outcome is binary.

    “Also, some investigators argue that the kappa statistic is interpretable only when the outcome is binary (15).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Kraemer, 2002, Mediators and moderators of treatment effects in randomized clinical trials, Arch Gen Psychiatry. source

  • May combine two claims.

    Study design and conductreasoning

    Surfaces scored as missing (M) or unable to score (Y) by any examiner were removed from analysis because determining missing due to caries is unreliable.

    “surfaces that were scored as missing (M) or unable to score (Y) by any of the examiners were removed from the analysis because the determination of “missing due to caries” is an unreliable measure”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Eight of the 24 subjects in each standardization session returned for a repeat examination at least 24 hours later to assess intra-examiner reliability.

    “In addition, we scheduled eight of those subjects to return for a repeat examination at least 24 hours later to assess intra-examiner reliability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Fleiss suggests ICC values between 0.40 and 0.75 represent fair to good reliability.

    “Fleiss suggests that ICC values between 0.40 and 0.75 represent fair to good reliability (11).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11] Fleiss, 1986

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    The examiners' calls were not independent because all examiners underwent a common training program, and totally random guessing would be unlikely.

    “However, we maintain that the examiners’ calls are not independent because they have all undergone a common training program and that totally random guessing would be an unlikely event (14).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14] Uebersax, 1987, Diversity of decision-making models and the measurement of interrater agreement, Psychol Bull. source
    • The authors state this as an assumption: The kappa statistic assumes that examiners' calls are independent and that an uncertain examiner will guess, producing chance agreement.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Weighted kappa is most appropriate for ordered classifications but should be interpreted cautiously because it behaves more like a measure of association than an index of agreement.

    “Weighted kappa is most appropriately used in the assessment of reliability for ordered classifications but this measure should be interpreted with caution not only in light of the weighting issue but because it behaves more like a measure of association than an index of agreement (16).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] Graham, 1993, The analysis of ordinal agreement data: beyond weighted kappa, J Clin Epidemiol. source

  • May combine two claims.

    Interpretationown data

    Inter-examiner kappa values for PACS and X-ACT examiners tend toward the lower end of values reported in the literature, while intra-examiner kappa values compare more favorably.

    “Nevertheless, the inter-examiner reliability kappa values for the PACS and X-ACT examiners tend toward the lower end of those reported in the literature although the intra-examiner kappa values compare more favorably (Table 5).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Examiners worked in separate operatories and were blinded to the calls made by other examiners.

    “Examiners worked in separate operatories and were blinded to the calls made by other examiners.”
    Quote in the paperSubstantial claim
21 of 21

  • The paper relies on this without support.

    Dental caries patterns are mostly bilateral and tend to be more alike within a mouth than between different mouths.

    “Since the pattern of dental caries is largely bilateral and tends to be more similar within mouths than between mouths, we used half-mouth evaluations with pre-specified, contra-lateral quadrants to allow for a larger number of different subjects to be examined.”

  • The paper relies on this without support.

    Methods for judging whether a caries lesion is active or inactive are rudimentary.

    “No attempt was made to distinguish between active and inactive lesions because the validity and reliability of methods for determining caries activity are rudimentary.”

  • The paper relies on this without support.

    There are no definitive criteria for acceptable levels of examiner agreement and reliability.

    “No definitive criteria exist for what are deemed to be acceptable levels of examiner agreement and reliability.”

  • The authors state this as an assumption.

    The kappa statistic assumes that examiners' calls are independent and that an uncertain examiner will guess, producing chance agreement.

    “Furthermore, the kappa statistic is predicated on the assumption that examiners’ calls are independent and, when not certain of a call, an examiner will simply guess thus producing chance agreement.”

  • The paper relies on this without support.

    The S/D1 versus D2/D3 caries classification reflects the traditional North American approach to caries detection.

    “The S/D1 vs. D2/D3 classification of dental caries reflects the traditional North American approach to caries detection.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The lesion code weighted Kappas in this study (0.77-0.82) were similar to or higher than those in a similar adult study (0.42-0.83).

    “The lesion code weighted Kappas in the current study (0.77–0.82) were similar or higher than a similar study conducted in adults (0.42–0.83).16”

    From Training Protocol and Calibration of the International Caries Detection and Assessment System in a School-Based Clinical Trial of Elementary School-Age Children (2025) · cites this paper as [16]

    The cited paper is an adult dental-caries study, and its Abstract > Results gives 0.42–0.83 for weighted kappas and observed-to-maximum kappa ratios taken together. Its Results section reports weighted inter-examiner kappas of 0.34–0.53, so 0.42–0.83 is not the range for weighted kappas alone. No page numbers are available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • One section says all ICCs exceeded 0.60, another says some were below

    Major

    The Discussion states that ICC levels exceeded 0.60 for all pairs of examiners. But the Results section says the acceptable ICC threshold was surpassed for all pairs except those involving examiner #3 at the PACS west coast session, whose range was 0.41-0.47.

  • Inter-examiner unweighted kappa ranges differ

    Minor

    The abstract gives 0.35 as the upper end of the inter-examiner unweighted kappa range, while the results give 0.57.

  • Excluded-examiner subset exceeds the stated full weighted-kappa range

    Minor

    The reported maximum weighted kappa after excluding examiner #3 is 0.62, which exceeds the stated maximum of 0.53 for all examiner pairs. Removing pairs cannot introduce a value outside the original range.

3 of 3

Cites

Flagged first, then by citations

1 of the 28 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. Ismail AI Banting D Eggertsson H Ekstrand K Ferreira-Zandoná A Longbottom C Pitts NB Reich E Ricketts D Selwitz R Sohn S Topping GVA Zero D Rational and Evidence for the International Caries Detection and Assessment System (ICDAS II) Stookey GK Clinical Models Workshop: Remin-Demin, Precavitation, Caries Indiana University School of Dentistry 2005 161 21
  2. Dunn, 2004, Statistical evaluation of measurement errors

Cited by

Sorted by citations

Related

Authors
Banting, D. W., Amaechi, B. T., Bader, J. D., Blanchard, P., Gilbert, G. H., Gullion, C. M., Holland, J. C., Makhija, S. K., Papas, A., Ritter, A. V., Singh, M. L., & Vollmer, W. M.
Affiliations
  • Western University (CA)
  • The University of Texas at San Antonio Health Science Center (US)
  • University of North Carolina at Chapel Hill (US)
  • University of Alabama (US)
  • Kaiser Permanente (US)
  • Tufts University (US)
Corresponding author
Banting, David W.
ORCID
Venue
Journal of public health dentistry
Publisher
Wiley
Published
2011-08-17
Volume
71
Issue
4
Pages
335-344
ISSN
0022-4006 (ISSN-L)
DOI
10.1111/j.1752-7325.2011.00278.x
PMC
PMC3683245
OpenAlex
W2102709672
Cited by
32
References
28
Citations by year
2025: 1 · 2024: 1 · 2023: 3 · 2022: 1 · 2021: 2 · 2020: 2 · 2019: 2 · 2017: 1 · 2016: 3 · 2015: 4 · 2014: 3 · 2013: 6 · 2012: 1
FWCI
4.09
Type
journal article
Language
en
Open access
Licence
doi.wiley.com/10.1002/tdm_license_1 (tdm, 1476 d)
Locations
  • Journal of Public Health Dentistry (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
National Institute of Dental and Craniofacial Research (U01DE018038)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Health Sciences › Dentistry › Periodontics › Dental Health and Care Utilization
  • Health Sciences › Dentistry › Periodontics › Oral microbiology and periodontitis research
Keywords
dental caries, examiner training, inter-examiner reliability, intra-rater reliability, caries classification, non-cavitated caries, dentin caries, enamel caries, kappa statistic, clinical trial data collection
MeSH terms
Adult, Calibration, Crowns, Dental Caries, Dental Enamel, Dental Restoration, Permanent, Dentin, DMF Index, Humans, Pit and Fissure Sealants, Research Design, Teaching, Teaching Materials, Reproducibility of Results, Observer Variation, Randomized Controlled Trials as Topic, Root Caries

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review