CoChat
Sign Up
Cited by 141Face Recognition and PerceptionFace recognition and analysisBiometric Identification and Security

The DOI matches its Crossref record; 39 of the 40 references it cites were found in a registry or an index; 24 of the 71 sources its claims cite were checked for support.

Error Rates in Users of Automatic Face Recognition Software

Open access
White, David, Dunn, James D., Schmid, Alexandra C., Kemp, Richard I.·2015·PLOS ONE
DOI 10.1371/journal.pone.0139827PMC PMC4605725
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, wide deployment of automatic face recognition systems has been accompanied by substantial gains in algorithm performance. However, benchmarking tests designed to evaluate these systems do not account for the errors of human operators, who are often an integral part of face recognition solutions in forensic and security settings. This causes a mismatch between evaluation tests and operational accuracy. We address this by measuring user performance in a face recognition system used to screen passport applications for identity fraud. Experiment 1 measured target detection accuracy in algorithm-generated 'candidate lists' selected from a large database of passport images. Accuracy was notably poorer than in previous studies of unfamiliar face matching: participants made over 50% errors for adult target faces, and over 60% when matching images of children. Experiment 2 then compared performance of student participants to trained passport officers-who use the system in their daily work-and found equivalent performance in these groups. Encouragingly, a group of highly trained and experienced "facial examiners" outperformed these groups by 20 percentage points. We conclude that human performance curtails accuracy of face recognition systems-potentially reducing benchmark estimates by 50% in operational settings. Mere practise does not attenuate these limits, but superior performance of trained examiners suggests that recruitment and selection of human operators, in combination with effective training and mentorship, can improve the operational accuracy of face recognition systems.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

In a realistic passport face-recognition candidate-list task, untrained users made over 50% errors for adult targets and over 60% for child targets. Trained passport review officers performed no better than students, but specialist facial examiners were about 20 percentage points more accurate.

Key findings

  • Experiment 1: untrained students’ overall accuracy was 45.1% for adult targets, 41.1% for adolescent targets, and 39.0% for child targets, with significant effects of target age.
  • Experiment 2: facial examiners (M = 69.1%) substantially outperformed facial review staff (M = 48.1%) and students (M = 47.6%); review staff and students did not differ significantly.
  • Employment duration was not correlated with accuracy for facial review staff (Spearman’s rho = -.248, p > 0.05) or facial examiners (rho = .000, p > 0.05), despite individual accuracy ranging from 20% to 85%.
  • A post-hoc ethnicity analysis found higher accuracy for non-Caucasian targets (n = 133, M = 52.3%) than Caucasian targets (n = 587, M = 48.1%), opposite to typical other-race effects.

Methods

  • Experiment 1 tested 36 undergraduate students (after exclusions) on 306 trials each, using 1,224 target identities divided into four sets; candidate lists showed the top eight matches from a passport database of more than 1,000,000 images, with child, adolescent, and adult targets.
  • Experiment 2 tested 24 facial review passport officers, 7 specialist facial examiners, and 48 untrained students/general-public participants on 180 trials each, using 720 target identities and an 18-second response limit.
  • Analyses used ANOVAs and planned t-tests for correct rejections, hits, misses, and misidentifications, plus correlations between accuracy and employment duration and post-hoc item/subject analyses of target ethnicity.

Limitations

  • The paper states that targets appeared on 50% of trials, unlike operational settings where targets are very rare (e.g., some reports estimate 1 in 400 passport applications are fraudulent), so operational accuracy may be substantially poorer than reported.
  • The paper states that the data cannot isolate why facial examiners performed better, and that the ethnicity analysis was post-hoc rather than planned.
  • Evident from the text: the facial examiner group was very small (n = 7), and the authors note that the Australian Passport Office did not use formal selection processes for these roles at the time.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0139827

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

32 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Verification

Claims the paper makes

24 of 71 cited sources checked · 13 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Leading commercial off-the-shelf algorithms in one-to-many identification returned matching images of the target identity in the top ten ranked matches for between 80% and 97% of targets in a challenging FRVT test.

    “Leading algorithms performed with a high level of accuracy, returning matching images of the target identity in the top ten ranked matches for between 80% and 97% of targets, depending on the algorithm.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [1] Grother PJ, Ngan M. Face Recognition Vendor Test (FRVT). NIST Interagency Report (8009). 2014. Available: http://biometrics.nist.gov/cs_links/face/frvt/frvt2013/NIST_8009.pdf source
  • Findingcitation

    Trained facial examiners have outperformed control participants both here and in recent studies.

    “Notwithstanding, trained facial examiners have outperformed control participants both here and in recent studies [20, 21].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [20] K Norell, 2015, The effect of image quality and forensic expertise in facial image comparisons, J Forensic Sci source
    • cites [21] D White, 2015, Perceptual expertise in forensic facial image comparison, P Roy Soc Lon B Biol source
  • Interpretationreasoning

    Estimates of operational FR system accuracy, in the conditions tested here, can be computed by halving scores attained on recent benchmark tests.

    “Thus, estimates of operational FR system accuracy, in the conditions we have tested here, can be computed by halving scores attained on recent benchmark tests [1].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [1] Grother PJ, Ngan M. Face Recognition Vendor Test (FRVT). NIST Interagency Report (8009). 2014. Available: http://biometrics.nist.gov/cs_links/face/frvt/frvt2013/NIST_8009.pdf source

      Section 4.6 reports accuracy measurements from operational datasets, and the 'Magnitude of accuracy gains' section describes a possible twofold reduction in miss rate when candidate rank increases from 1 to 50. Neither passage says that operational accuracy can be estimated by halving benchmark-test scores. Searches for 'benchmark,' 'half,' and comparisons between operational and benchmark accuracy did not find that conversion in our copy; page numbers were not supplied. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
  • Premisecitation

    In recent studies, facial examiners were more accurate in one-to-one facial image comparison compared to control groups.

    “In recent studies, facial examiners were more accurate in one-to-one facial image comparison compared to control groups [20, 21] and so we expected that examiners would outperform the other groups.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [20] K Norell, 2015, The effect of image quality and forensic expertise in facial image comparisons, J Forensic Sci source
    • cites [21] D White, 2015, Perceptual expertise in forensic facial image comparison, P Roy Soc Lon B Biol source
  • Interpretationcitation

    Face aging caused poorer performance compared to similar tasks where images were taken on the same day.

    “To account for this, and consistent with recent work [17, 23] we suggest that face aging caused poorer performance compared to similar tasks where images were taken on the same day [12].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [17] D White, 2014, Passport Officers’ errors in face matching, PLoS ONE source

      The Photo-to-Photo Test section says age-related changes affect matching accuracy, and its Results report poor performance when matching newer images to older photos. The Glasgow Face Matching Test section describes a same-day-image task, but the paper does not establish that aging caused poorer performance than on that task; the tasks were not directly compared for this effect. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [23] AM Megreya, 2013, Matching Face Images Taken on the Same Day or Months Apart: the Limitations of Photo ID, Appl Cognitive Psych source
    • cites [12] V Bruce, 1999, Verification of face identities from images captured on video, J Exp Psychol Appl source
  • It may state two separate claims as one. 28
  • Probably not a substantial claim. 13

  • It may state two separate claims as one.

    Interpretationcitation

    Face age only affected accuracy in target present trials, both here and in previous work, and so does not account for the equivalent error rates observed in target absent arrays.

    “However, face age only affected accuracy in target present trials, both here and in previous work [17, 23], and so does not account for the equivalent error rates we observed in target absent arrays.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [17] D White, 2014, Passport Officers’ errors in face matching, PLoS ONE source

      The Photo-to-Photo Test describes older photos and official ID photos as its comparison conditions. Its Results report a significant effect of photo type on mismatch trials—the equivalent of target-absent trials—contrary to the claim that the effect was confined to target-present trials. The citing authors’ own observation of equivalent error rates in their arrays is not attributed to this source. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [23] AM Megreya, 2013, Matching Face Images Taken on the Same Day or Months Apart: the Limitations of Photo ID, Appl Cognitive Psych source

  • It may state two separate claims as one.

    Premisecitation

    Benchmarking tests of automatic face recognition systems do not account for the errors of human operators, causing a mismatch between evaluation tests and operational accuracy.

    “However, benchmarking tests designed to evaluate these systems do not account for the errors of human operators, who are often an integral part of face recognition solutions in forensic and security settings. This causes a mismatch between evaluation tests and operational accuracy.”
    • Also stated in: § Introduction · plain
      “Importantly however, this method only estimates accuracy of the machine component of the total system, ignoring the impact of the decisions made by the human operators—who monitor image galleries containing the highest ranking matches [1].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [1] Grother PJ, Ngan M. Face Recognition Vendor Test (FRVT). NIST Interagency Report (8009). 2014. Available: http://biometrics.nist.gov/cs_links/face/frvt/frvt2013/NIST_8009.pdf source

      Section 5.7 assumes reviewers always find a mate when present and never incorrectly select a nonmate; the report therefore does not incorporate human adjudication errors into that model. The 'Accuracy by age group' passage also says reviewer adjudication accuracy remains poorly quantified. These passages support the concern about omitted human errors, but the report does not establish the asserted mismatch between benchmark results and operational accuracy. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Similar groups of professional examiners have been shown to have superior face identification abilities.

    “In recent studies, similar groups of professional examiners have been shown to have superior face identification abilities [20, 21].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [20] K Norell, 2015, The effect of image quality and forensic expertise in facial image comparisons, J Forensic Sci source
    • cites [21] D White, 2015, Perceptual expertise in forensic facial image comparison, P Roy Soc Lon B Biol source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    Previous studies show that some people perform very well in face matching tasks while others perform very poorly, and that these individual differences are stable across tests.

    “This would be consistent with previous studies showing that some people perform very well in face matching tasks while others perform very poorly [15, 33, 34] and that these individual differences are stable across tests [33–35].”
    Quote in the paperNot a substantial claim4 of 4 sources checked
    • cites [15] AM Burton, 2010, The Glasgow Face Matching Test, Behav Res Methods source
    • cites [33] AM Megreya, 2006, Unfamiliar faces are not faces: Evidence from a matching task, Mem Cognition source
    • cites [34] R Russell, 2009, Super-recognizers: People with extraordinary face recognition ability, Psychon B Rev source
    • cites [35] JB Wilmer, 2010, Human face recognition ability is specific and highly heritable, P Natl Acad Sci USA source

      The Results report substantial variation in Cambridge Face Memory Test scores and test–retest and alternate-forms reliability; Materials and Methods also report correlations with other face tests. These findings support individual differences that persist across testing, but the paper primarily measures face recognition after studying faces, not performance on face-matching tasks specifically. Page numbers are unavailable in our copy. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. It may state two separate claims as one.

    Premisecitation

    Previous research has shown that face aging makes unfamiliar face matching more difficult, increasing error by 20% when images are separated by more than a year.

    “Previous research has shown that face aging makes unfamiliar face matching more difficult, increasing error by 20% when images are separated by more than a year [17, 22, 23] and so we expected error rates to be higher than in similar test using images taken on the same day [12].”
    Quote in the paperSubstantial claim2 of 4 sources checked
    • cites [17] D White, 2014, Passport Officers’ errors in face matching, PLoS ONE source

      The Photo-to-Photo Test introduction (page unavailable) says age-related changes in appearance have a large effect on matching accuracy. Its Results report poorer matching for some older ID photos, but the paper does not report a 20% increase in errors or establish a threshold of more than one year; it tested photos taken two years apart and ID photos of uncertain age. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [22] JP Davis, 2009, CCTV on trial: Matching video images with the defendant in the dock, Appl Cognitive Psych source
    • cites [23] AM Megreya, 2013, Matching Face Images Taken on the Same Day or Months Apart: the Limitations of Photo ID, Appl Cognitive Psych source

      The General Discussion (page unavailable) reports that hit rates fell by about 20 percentage points when unfamiliar-face images were taken on different dates, and says age-related changes could increase mistaken identification. The Stimuli & procedure section (page unavailable) gives a mean separation of 17.2 months, not a condition restricted to images more than a year apart. The reported change is approximately 20 percentage points in accuracy, rather than a 20% relative increase in error. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [12] V Bruce, 1999, Verification of face identities from images captured on video, J Exp Psychol Appl source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    Recent work has shown that providing feedback on the accuracy of face matching decisions and performing face matching decisions in groups can improve accuracy in face matching tasks.

    “Alternatively, recent work has shown that providing feedback on the accuracy of face matching decisions [36, 37] and performing face matching decisions in groups [38] can improve accuracy in face matching tasks.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [36] D White, 2014, Feedback training for facial image comparison, Psychon B Rev source
    • cites [37] HM Alenezi, 2013, The Effect of Feedback on Face‐Matching Accuracy, Appl Cognitive Psych source

      The Discussion reports that feedback prevented a decline in face-matching accuracy relative to no feedback. But it also says feedback did not improve accuracy above initial baseline levels, so the unqualified claim that it can improve accuracy overstates this paper’s finding. The separate claim about group decisions is attributed to [38], not this source. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [38] AJ Dowsett, Unfamiliar face matching: Pairs out‐perform individuals and provide a route to training, Brit J Psychol

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    Facial Review staff perform many face matching decisions per day but check FR candidate lists as part of eligibility checks; they typically receive a small amount of training when commencing employment.

    “Facial review staff perform many face matching decisions per day, but check FR candidate lists as part of a series of eligibility checks that make up the majority of processing. These passport officers have typically received a small amount of training when commencing employment, and perform a similar role to passport officers tested in previous studies of face matching ability [17].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [17] D White, 2014, Passport Officers’ errors in face matching, PLoS ONE source

      The Participants section describes passport officers whose main responsibility was assessing eligibility, who routinely made face comparisons, and of whom all but three had completed a short training module. The Discussion says they made many face-matching decisions in their daily workflow. The paper does not describe Facial Review staff checking FR candidate lists or establish that training occurred when employment commenced. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • It may state two separate claims as one.

    Premisecitation

    The human face undergoes significant anatomical change during childhood, and this change is quantitatively and qualitatively different from changes occurring during adulthood.

    “The human face undergoes significant anatomical change during childhood, and the change is both quantitatively [24] and qualitatively [25] different to changes occurring during adulthood.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [24] VF Ferrario, 1999, Soft tissue facial growth and development as assessed by the three-dimensional computerized mesh diagram analysis, Am J Orthod Dentofac source
    • cites [25] MM Chakravarty, 2011, Automated analysis of craniofacial morphology using magnetic resonance images, PLoS ONE source

      The Results (Voxel-by-voxel analysis of deformation fields) quantify significant age-related facial changes in adolescents. They do not compare the magnitude of those changes with changes during adulthood. The citing sentence assigns the qualitative comparison to a different citation [25], so that part is not attributed to this paper. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • It may state two separate claims as one.

    Findingcitation

    Studies reported here and in previous work show large and stable individual differences in face matching accuracy that are unrelated to length of professional experience.

    “Studies reported here and in previous work show large and stable individual differences in face matching accuracy that are unrelated to length professional experience [15, 17, 32].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [15] AM Burton, 2010, The Glasgow Face Matching Test, Behav Res Methods source

      The General Discussion (page unavailable) reports large interindividual variation in face-matching performance, and A TEST FOR FACE MATCHING (page unavailable) reports high split-half reliability (r = .81). The paper does not establish stability over time or test whether accuracy is unrelated to length of professional experience; its finding of no correlation with age does not address experience. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [17] D White, 2014, Passport Officers’ errors in face matching, PLoS ONE source
    • cites [32] RJ Sternberg, 1996, The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games

  • It may state two separate claims as one.

    Findingown data

    In Experiment 1, participants made over 50% errors for adult target faces and over 60% errors when matching images of children.

    “Accuracy was notably poorer than in previous studies of unfamiliar face matching: participants made over 50% errors for adult target faces, and over 60% when matching images of children.”
    • Also stated in: § Results · plain
      “Overall percent correct was 45.1% (SD = 11.7%) for adult target images, 41.1% (SD = 11.7%) for adolescent target images and 39.0% (SD = 10.6%) for child target images.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingown data

    The mean time interval between target image and previous application image in the candidate list array was 6.2 years for Child applicants, 6.3 years for Adolescents, and 9.7 years for Adults.

    “For Child applicants the mean was 6.2 years (SD = 1.7) for Adolescents it was 6.3 years (SD = 1.6) and for Adults it was 9.7 years (SD = 2.8).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    For correct rejections in Experiment 1, there was a marginally significant effect of applicant age, with fewer correct rejections for adolescent compared to child and adult applicants, and no difference between child and adult applicants.

    “For correct rejections, there was a marginally significant effect of Applicant Age [F (2,70) = 2.95, p = .059, ηp 2 = .078], with planned comparisons confirming that participants made fewer correct rejections for adolescent compared to child [t (35) = 3.25, p < 0.05, Cohen’s d = 0.39] and adult applicants [t (35) = 3.96, p < 0.05, Cohen’s d = 0.37] but no difference between for child and adult”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    For misidentifications in Experiment 1, the main effect of applicant age was significant; children and adolescents were misidentified more often than adult applicants, while the difference between child and adolescent applicants was non-significant.

    “For misidentifications, the difference between child and adolescent applicants was non-significant [t (35) = 1.33, p > 0.05, Cohen’s d = 0.22]. Children [t (35) = 4.75, p < 0.05, Cohen’s d = 0.79] and adolescents [t (35) = 3.63, p < 0.05, Cohen’s d = 0.62] were misidentified more often than adult applicants.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    For hits in Experiment 1, participants were less likely to find younger target identities when present in a candidate list; the main effect of applicant age was significant, with fewer hits for child compared to adolescent applicants and for adolescent compared to adult applicants.

    “For hits, participants were less likely to find younger target identities when present in a candidate list. The main effect of Applicant Age was significant [F (2,70) = 27.15, p < 0.05, ηp 2 = .437], with planned comparisons confirming that participants made fewer hits for child compared to adolescent applicants [t (35) = 4.12, p < 0.05, Cohen’s d = 0.69] and for adolescent compared to adult”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    For misses in Experiment 1, the main effect of applicant age was significant, with more misses in child compared to adolescent and adult arrays, but no difference between adolescents and adults.

    “The main effect of applicant age was also significant for misses [F (2,70) = 6.70, p < 0.05, ηp 2 = .161] and misidentifications [F (2,70) = 11.54, p < 0.01, ηp 2 = .248]. Planned comparisons show more misses in child compared to adolescent [t (35) = 3.11, p < 0.05, Cohen’s d = 0.55] and adult arrays [t (35) = 2.90, p < 0.05, Cohen’s d = 0.56] but no difference between adolescents and adults (t <”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Premisecitation

    Identification accuracy of face recognition can be quite poor when image capture conditions are not optimal and when images of a face are captured several years apart.

    “Identification accuracy can be quite poor in cases where image capture conditions are not optimal [7] and where images of a face are captured several years apart [8].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [7] AJ O’Toole, 2012, Comparing face recognition algorithms to humans on challenging tasks, ACM Appl Percept source
    • cites [8] K Ricanek, 2006, MORPH: A Longitudinal Image Database of Normal Adult Age-Progression, IEEE 7th International Conference on Automatic Face and Gesture Recognition source

  • It may state two separate claims as one.

    Premisecitation

    Facial examiners perform more detailed facial image comparisons, receive more training, and spend more time each day performing face matching tasks than facial review staff.

    “Facial examiners perform more detailed facial image comparisons, receive more training and spend more time each day performing face matching tasks than facial review staff.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19] Guidelines and recommendations for facial comparison training to competency. Facial Identification Scientific Working Group. 2011. Available: www.fiswg.org/document.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Previous studies show poorer accuracy when matching faces that are of a different ethnicity to the viewer.

    “Previous studies show poorer accuracy when matching faces that are of a different ethnicity to the viewer [16].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16] AM Megreya, 2011, The other race effect does not rely on memory: Evidence from a matching task, Q J Exp Psychol source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductown data

    Six participants were excluded from Experiment 1 because they did not finish in the allocated time of 1 hour, and final analyses were based on 36 participants.

    “Six participants were excluded from our analysis because they did not finish the experiment in the allocated time of 1 hour and so final analyses were based on data from 36 participants (27 females).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data

    For adult target faces, participants were incorrect on over half of all trials, suggesting that previous studies of unfamiliar face matching may overestimate human accuracy in a variety of modern applied settings.

    “For adult target faces, participants were incorrect on over half of all trials, suggesting that previous studies of unfamiliar face matching may overestimate human accuracy in a variety of modern applied settings.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Study design and conductreasoning

    The database of passport application images was a random sample of consenting Australian citizens and its demographic composition was reflective of Australian demographics.

    “The database was a random sample of consenting Australian citizens and so demographic composition was reflective of Australian demographics (see S1 Data for item ethnicity and age).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Premisecitation

    Guidelines for training in facial comparison exist, but have not yet been subject to empirical validation.

    “Whilst guidelines for training in facial comparison do exist [19, 39, 40], these have not yet been subject to empirical validation.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [19] Guidelines and recommendations for facial comparison training to competency. Facial Identification Scientific Working Group. 2011. Available: www.fiswg.org/document.
    • cites [39] NA Spaun, 2009, Advances in Biometrics
    • cites [40] R Heyer, 2013, Forensic confirmation bias: the case of facial image comparison, J Appl Res Mem Cog source

  • Probably not a substantial claim.

    Premisecitation

    Accuracy of face recognition algorithms is sensitive to the demographic composition of both training and test image databases.

    “Previous studies show that accuracy of face recognition algorithms is sensitive to the demographic composition of both training and test image databases [28, 29].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [28] PJ Phillips, 2011, An other-race effect for face recognition algorithms, ACM Appl Percept

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [29] B Klare, 2012, Face Recognition Performance: Role of Demographic Information, IEEE T Inf Foren Sec source

  • Probably not a substantial claim.

    Study design and conductreasoning

    To derive reliable measures of perceptual performance in the limited time available, target images appeared in candidate lists on half of trials.

    “To derive reliable measures of perceptual performance in the limited time we had to test passport officers, target images appeared in candidate lists on half of trials.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data

    Differences between facial forensic examiners and standard experimental groups are not limited to quantitative differences in overall accuracy; qualitative differences also point towards differences in the cognitive processing they deploy.

    “Further, differences between facial forensic examiners and standard experimental groups are not limited to quantitative differences in overall accuracy–qualitative differences also point towards differences in the cognitive processing they deploy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    Facial Examiners are a smaller set of highly trained staff who perform more detailed forensic-comparison of facial images.

    “On the other hand, Facial Examiners were part of a smaller set of highly trained staff who perform more detailed forensic-comparison of facial images.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] Guidelines and recommendations for facial comparison training to competency. Facial Identification Scientific Working Group. 2011. Available: www.fiswg.org/document.

  • It may state two separate claims as one.

    Study design and conductnone given

    The task used real passport application images and was designed to model face matching decisions made by FR operators in a passport issuance workflow.

    “This task used real passport application images and was designed to model face matching decisions made by FR operators in a passport issuance workflow.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 720 target identities selected are representative of the daily work of passport officers.

  • It may state two separate claims as one.

    Interpretationown data

    Accelerated growth of facial structure in childhood and adolescence increases task difficulty for humans, as previously shown for machine recognition.

    “This suggests that accelerated growth of facial structure in childhood and adolescence [24, 25] increases task difficulty for humans; as has previously been shown for machine recognition [27].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [24] VF Ferrario, 1999, Soft tissue facial growth and development as assessed by the three-dimensional computerized mesh diagram analysis, Am J Orthod Dentofac source
    • cites [25] MM Chakravarty, 2011, Automated analysis of craniofacial morphology using magnetic resonance images, PLoS ONE source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [27] H Ling, 2007, A study of face recognition as people age, IEEE 11th International Conference on Computer Vision source

  • It may state two separate claims as one.

    Interpretationreasoning

    Error rates were especially poor for images of adolescents and children, despite the shorter average time interval between target and array images for children and adolescents compared to adults.

    “Error rates were especially poor for images of adolescents and children, despite the shorter average time interval between target and array images for children (6.2 years) and adolescents (6.3 years) compared to adults (9.7 years).”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    In systems implementing one-to-many algorithm checks, human users are required to make final identity decisions, and so the accuracy of the system as a whole is heavily constrained by human accuracy.

    “In systems implementing one-to-many algorithm checks, it is human users that are required to make final identity decisions, and so the accuracy of the system as a whole is heavily constrained by human accuracy.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data

    Examiners were equally likely to miss matching faces in the candidate list compared to other groups, but were less likely to select non-matching faces when the target was present.

    “Thus, examiners were equally likely to miss matching faces in the candidate list compared to other groups, but were less likely to select non-matching faces when the target was present.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning

    These qualitative differences offer important insights into the nature of expertise that can be used to inform development of training methods and workflow systems for facial image comparison.

    “These qualitative differences offer important insights into the nature of expertise that can be used to inform development of training methods and workflow systems for facial image comparison.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    Expertise of examiners is most evident at longer exposure durations and when comparing high-quality images.

    “For instance, expertise of examiners is most evident at longer exposure durations [20] and when comparing high-quality images [21].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [20] K Norell, 2015, The effect of image quality and forensic expertise in facial image comparisons, J Forensic Sci source
    • cites [21] D White, 2015, Perceptual expertise in forensic facial image comparison, P Roy Soc Lon B Biol source

      Checked before — its earlier verdict is held back until a new check runs.

33 of 33

  • The paper relies on this without support.

    The 50% target prevalence used in the experiments does not materially change the human error rates relative to the very low target prevalence in operational forensic and security settings.

    “Because we had limited time to conduct tests, and our aim was to provide reliable measures of perceptual matching ability on this task, we inserted matching identities on 50% of candidate lists. This was a necessary divergence from the target prevalence rate in many forensic and security applications, where targets appear in candidate lists very rarely [2, 18].”

  • The paper relies on this without support.

    The 720 target identities selected are representative of the daily work of passport officers.

    “As before, we included an equal number of child, adolescent and adult target identities to ensure that stimuli were representative of daily work of passport officers.”

  • The paper relies on this without support.

    Trained passport officers use this software in their daily work and make an average of 60 candidate list checks per day.

    “More importantly, trained passport officers who use this software in their daily work—making an average of 60 candidate list checks per day—were no better on the task than untrained student participants.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • None of these studies included facial examiners.

    “However, none of these studies included facial examiners.”

    From Do professional facial image comparison training courses work? (2019) · cites this paper as [10]

    Experiment 2 (Method) explicitly includes seven specialist Facial Examiners as participants. Its Results compare their performance with facial review staff and students. Thus, if this paper is one of “these studies,” the claim that none included facial examiners is contrary to our copy. (LLM, conf=0.89)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • These studies all show no improvement with more professional experience, suggesting that on-the-job learning in these particular roles does not help develop expertise.

    “These studies all show no improvement with more professional experience, suggesting that on-the-job learning in these particular roles does not help develop expertise.”

    From Do professional facial image comparison training courses work? (2019) · cites this paper as [10]

    Experiment 2 Results report no relationship between years employed and face-matching accuracy for either facial review staff or facial examiners. But the General Discussion says examiners may have developed better strategies through feedback and working with specialists, so the finding does not establish that on-the-job learning does not help develop expertise. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Participants in these studies were facial reviewers-non-specialist facial image comparison practitioners who perform fast, high-volume face identifications (e.g. at border control).

    “Instead, participants in these studies were facial reviewers–non-specialist facial image comparison practitioners who perform fast, high-volume face identifications (e.g. at border control).”

    From Do professional facial image comparison training courses work? (2019) · cites this paper as [10]

    The Introduction says Facial Review staff make many face-matching decisions per day and receive only a small amount of training. But Experiment 2 > Method also includes specialist Facial Examiners and untrained controls, so not all participants were facial reviewers. The staff worked in passport issuance; the paper does not identify border control as their setting or say their decisions were fast. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Infant and child facial photographs are ineffective for use in real-world identification.

    “Taken together, and in combination with previous work (White et al., 2015), our results suggest that such low levels of accuracy mean infant and child facial photographs are ineffective for use in real-world identification”

    From Unfamiliar face matching with photographs of infants and children (2018) · cites this paper as White et al., 2015

    Experiment 1 Results report 39.0% accuracy for child target images, and its Discussion describes errors for children as especially high. But the study’s child applicants were aged 6–13 (Experiment 1 Method); it did not test infants. The statement that the photographs are ineffective for real-world identification is the citing authors’ synthesis, not a conclusion this source establishes for both infants and children. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Only two studies to date have investigated the performance of typical adults when matching children's faces, and difficulties were apparent in both.

    “although only two studies to date have investigated the performance of typical adults when matching children’s faces, difficulties were apparent in both.”

    From Enhanced Matching of Children’s Faces in “Super-Recognisers” But Not High-Contact Controls (2020) · cites this paper as White et al., 2015

    Experiment 1 Results report lower accuracy for child than adult faces, and its Discussion describes error rates for children as especially poor. The paper does not establish that only two studies had investigated typical adults matching children’s faces, or that difficulties were found in both studies. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • Inconsistent p-value: F (1,76) = 3.92, p = .052

    Minor

    The paper reports p = .052 here; recomputing from the test statistic and its degrees of freedom gives p = .051. The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .052 · recomputed p = .051

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upF(1, 76) = 3.92reported p = .052 · recomputed p = .051

    “F (1,76) = 3.92, p = .052”

  • ConsistentF(2, 70) = 19.49reported p < .05 · recomputed p < .001

    “F (2,70) = 19.49, p < .05”

  • Consistentt(35) = -6.87reported p < .05 · recomputed p < .001

    “t (35) = -6.87, p < .05”

  • Consistentt(35) = -4.06reported p < .05 · recomputed p < .001

    “t (35) = -4.06, p < .05”

  • Consistentt(35) = -1.91reported p = .064 · recomputed p = .064

    “t (35) = -1.91, p = .064”

  • ConsistentF(2, 70) = 2.95reported p = .059 · recomputed p = .059

    “F (2,70) = 2.95, p = .059”

  • Consistentt(35) = 3.25reported p < .05 · recomputed p = .003

    “t (35) = 3.25, p < .05”

  • Consistentt(35) = 3.96reported p < .05 · recomputed p < .001

    “t (35) = 3.96, p < .05”

  • Consistentt(35) = 0.71reported p > .05 · recomputed p = .482

    “t (35) = 0.71, p > .05”

  • ConsistentF(2, 70) = 27.15reported p < .05 · recomputed p < .001

    “F (2,70) = 27.15, p < .05”

  • Consistentt(35) = 4.12reported p < .05 · recomputed p < .001

    “t (35) = 4.12, p < .05”

  • Consistentt(35) = 3.02reported p < .05 · recomputed p = .005

    “t (35) = 3.02, p < .05”

  • ConsistentF(2, 70) = 6.7reported p < .05 · recomputed p = .002

    “F (2,70) = 6.70, p < .05”

  • ConsistentF(2, 70) = 11.54reported p < .01 · recomputed p < .001

    “F (2,70) = 11.54, p < .01”

  • Consistentt(35) = 3.11reported p < .05 · recomputed p = .004

    “t (35) = 3.11, p < .05”

  • Consistentt(35) = 2.9reported p < .05 · recomputed p = .006

    “t (35) = 2.90, p < .05”

  • Consistentt(35) = 1.33reported p > .05 · recomputed p = .192

    “t (35) = 1.33, p > .05”

  • Consistentt(35) = 4.75reported p < .05 · recomputed p < .001

    “t (35) = 4.75, p < .05”

  • Consistentt(35) = 3.63reported p < .05 · recomputed p < .001

    “t (35) = 3.63, p < .05”

  • Consistentt(29) = 3.04reported p < .05 · recomputed p = .005

    “t (29) = 3.04, p < .05”

  • Consistentt(53) = 4.01reported p < .05 · recomputed p < .001

    “t (53) = 4.01, p < .05”

  • ConsistentF(2, 76) = 3.37reported p < .05 · recomputed p = .040

    “F (2,76) = 3.37, p < .05”

  • Consistentt(53) = 2.81reported p < .05 · recomputed p = .007

    “t (53) = 2.81, p < .05”

  • Consistentt(29) = 2.02reported p = .053 · recomputed p = .053

    “t (29) = 2.02, p = .053”

  • Consistentt(70) = 0.22reported p > .05 · recomputed p = .827

    “t (70) = .220, p > .05”

  • ConsistentF(2, 76) = 7.82reported p < .05 · recomputed p < .001

    “F (2,76) = 7.82, p < .05”

  • Consistentt(29) = 3.43reported p < .05 · recomputed p = .002

    “t (29) = 3.43, p < .05”

  • ConsistentF(2, 76) = 6.94reported p < .05 · recomputed p = .002

    “F (2,76) = 6.94, p < .05”

  • Consistentt(29) = 3.12reported p < .05 · recomputed p = .004

    “t (29) = 3.12, p < .05”

  • Consistentt(53) = 3.94reported p < .05 · recomputed p < .001

    “t (53) = 3.94, p < .05”

  • Consistentt(726) = 2.45reported p < .05 · recomputed p = .015

    “t (726) = 2.45, p < .05”

  • ConsistentF(1, 76) = 10.64reported p < .05 · recomputed p = .002

    “F (1,76) = 10.64, p < .05”

32 of 32

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Target-age effect also appears in target-absent trials

    Minor

    The discussion says poorer performance for younger targets was evident only in target-present trials, but the results report significantly fewer correct rejections for adolescents than for adults in target-absent trials.

1 of 1

Citations

Cites

Flagged first, then by citations

1 of the 40 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Metadata

Authors
White, D., Dunn, J. D., Schmid, A. C., & Kemp, R. I.
Affiliations
  • UNSW Sydney (AU)
  • The University of Sydney (AU)
Corresponding author
White, David
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2015-10-14
Volume
10
Issue
10
Pages
e0139827
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0139827
PMC
PMC4605725
OpenAlex
W2204202707
Semantic Scholar
7eb6a5dd7cd6aca995503bd387528932b6af7169
Cited by
141
References
40
Citations by year
2026: 5 · 2025: 10 · 2024: 13 · 2023: 14 · 2022: 14 · 2021: 17 · 2020: 14 · 2019: 15 · 2018: 17 · 2017: 10 · 2016: 12
FWCI
6.40
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
Australian Research Council (LP110100448, LP130100702)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
Keywords
face recognition, operator error, passport application, forensic face recognition, benchmarking, examiner training, target detection
MeSH terms
Adolescent, Adult, Algorithms, Ethnicity, Face, Female, Humans, Image Processing, Computer-Assisted, Male, Middle Aged, Software, Software Design, Databases, Factual

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review