Do professional facial image comparison training courses work?

In the paper

“Passport officers made errors on more than 50% of decisions—the same rate as untrained novices.”

What the sources say

• Supported

  Experiment 2 Results report 48.1% accuracy for facial review staff, or 51.9% errors, compared with 47.6% accuracy for untrained students; the difference was not significant. The claim applies to the facial review officers, not the specialist facial examiners, who performed substantially better. (LLM, conf=0.92)

  cites [10] D White, 2015, Error rates in users of automatic face recognition software, PLoS One source

In the paper

“Facial examiners are a specialist group of facial image comparison practitioners who conduct slow, systematic, feature-based comparisons, often as part of criminal investigations (see [20, 22, 23]).”

What the sources say

• ?Not checked — no full text of the sourcecites [20] A Towler, 2017, Evaluating the feature comparison strategy for forensic face identification, Journal of Experimental Psychology: Applied source
• Supported

  The Introduction describes trained forensic facial examiners making identity judgements used in criminal investigations and trials. The Discussion describes their professional training as emphasizing feature-by-feature comparison and characterizes their expertise as controlled and effortful analysis, supporting the paraphrase of slow, systematic comparison. (LLM, conf=0.90)

  cites [22] D White, 2015, Perceptual expertise in forensic facial image comparison, Proceedings of the Royal Society of London B: Biological Sciences source
• Partly supported

  The paper describes forensic facial examiners as extensively trained professionals whose image comparisons are rigorous and time-consuming, and whose decisions can support prosecutions and court testimony. Materials and Methods says they used their laboratory tools and methods. Our copy does not establish that their comparisons are specifically feature-based, so that part of the attributed description is unsupported. (LLM, conf=0.84)

  cites [23] PJ Phillips, 2018, Face recognition accuracy in forensic examiners, super-recognisers and algorithms, Proceedings of the National Academy of Sciences sourceExperimental — accuracy not yet measured

In the paper

“In a live photo-to-person identity verification task for example, passport officers accepted 14% of fraudulent ID documents and rejected 6% of valid IDs [9].”

What the sources say

• Supported

  The Person-to-Photo Test Results report that passport officers wrongly accepted 14% of fraudulent photos and wrongly rejected 6% of valid photos. The Participants and stimuli section identifies this as a live identification task with ID-card bearers. (LLM, conf=0.95)

  cites [9] D White, 2014, Passport officers’ errors in face matching, PLoS ONE source

In the paper

“In addition, Course D did not produce the qualitative changes associated with expertise in facial image comparison: increased diagnosticity of facial feature similarity ratings and a reduced inversion effect [20, 22].”

What the sources say

• ?Not checked — no full text of the sourcecites [20] A Towler, 2017, Evaluating the feature comparison strategy for forensic face identification, Journal of Experimental Psychology: Applied source
• Partly supported

  The Discussion (section 3; page unavailable) reports that image inversion impaired forensic examiners less than students, supporting the reduced-inversion-effect example of expertise. It does not report facial-feature similarity ratings or evaluate Course D. The Course D finding appears to be the citing authors’ own result, not a result attributed to this source. (LLM, conf=0.86)

  cites [22] D White, 2015, Perceptual expertise in forensic facial image comparison, Proceedings of the Royal Society of London B: Biological Sciences sourceExperimental — accuracy not yet measured

In the paper

“More generally, training is also accepted as evidence of expertise in court [11], allowing ‘trained experts’ to present highly persuasive identification evidence at trial [12].”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [11] Daubert v. Merrell Dow Pharmaceuticals, Inc.: Supreme Court of the United States; 1993.
• Supported

  The citation [11] attaches to the first clause, about courts accepting training as evidence of expertise. The paper’s Abstract (page not provided) says that a common legal exception permits opinion evidence based on specialised knowledge acquired through training, study, or experience, allowing forensic scientists to testify as experts. The later claim about persuasive identification evidence is attached to [12], not this source. (LLM, conf=0.92)

  cites [12] A Towler, 2018, Are forensic scientists experts?, Journal of Applied Research in Memory and Cognition source

In the paper

“Differences in the availability of ongoing on-the-job training and opportunities for deliberate practice (see [29]) may explain why facial examiners tend to outperform facial reviewers [10, 23].”

What the sources say

• ?Not checked — no full text of the sourcecites [29] KA Ericsson, 1993, The role of deliberate practice in the acquisition of expert performance, Psychological Review source
• Supported

  The citation to this source supports the comparison between facial examiners and facial reviewers; the parenthetical citation [29] is attached to the proposed explanation. Experiment 2 Results reports significantly fewer misidentifications among examiners than reviewers, and the General Discussion reports that examiners made 20% fewer errors as a group. (LLM, conf=0.93)

  cites [10] D White, 2015, Error rates in users of automatic face recognition software, PLoS One source
• Supported

  The Results (Accuracy; page unavailable) report higher median accuracy for facial examiners (AUC 0.93) than facial reviewers (0.87), supporting the performance comparison attributed to this source. The paper also says the difference between face-specialist groups was not statistically significant. The proposed training-and-practice explanation is separately directed to reference [29], not this paper. (LLM, conf=0.88)

  cites [23] PJ Phillips, 2018, Face recognition accuracy in forensic examiners, super-recognisers and algorithms, Proceedings of the National Academy of Sciences source

In the paper

“Similarly, mentors provide examiners with feedback on the accuracy of their decisions and feature-based analysis, which may also facilitate learning [19].”

What the sources say

• Partly supported

  The Experiment 1 section describes trial-by-trial feedback on face-matching decisions, and the General Discussion reports that feedback training improved accuracy and may have directed attention to diagnostic facial features. The paper does not describe mentors providing feedback to examiners on their feature-based analyses; that setting and form of feedback are not established by the study. (LLM, conf=0.88)

  cites [19] D White, 2014, Feedback training for facial image comparison, Psychonomic Bulletin and Review sourceExperimental — accuracy not yet measured

In the paper

“The Glasgow Face Matching Test (GFMT) is a standardised psychometric test of face matching ability [6].”

What the sources say

• Supported

  The Abstract describes the GFMT as a test of unfamiliar face matching and says the authors provide normative data from large subject samples. The Test Construction section specifies its fixed set of face pairs, supporting its description as a standardised test of face-matching ability. (LLM, conf=0.92)

  cites [6] AM Burton, 2010, The Glasgow Face Matching Test, Behavior Research Methods source

In the paper

“However, previous studies have examined the relationship between the length of time employed in a professional role and face identification accuracy to understand the role of on-the-job learning [9, 10, 27, 28].”

What the sources say

• Supported

  The Person-to-Photo Test Results say the authors tested whether passport officers’ duration of professional experience predicted face-matching performance; they found no relationship. The Discussion interprets the finding in terms of whether experience alone improves accuracy, matching the claim’s on-the-job learning rationale. (LLM, conf=0.94)

  cites [9] D White, 2014, Passport officers’ errors in face matching, PLoS ONE source
• Supported

  In Experiment 2 Results (page not specified), the authors correlate years employed at the Australian Passport Office with face-matching accuracy and report no significant relationship. The General Discussion considers how workplace experience might affect performance, consistent with the claim’s on-the-job-learning framing. (LLM, conf=0.93)

  cites [10] D White, 2015, Error rates in users of automatic face recognition software, PLoS One source
• Supported

  The Background states that the study correlated face-matching performance with occupational experience measures, including years of on-the-job experience. The Conclusions report that yearly experience had no detectable effect on identity-verification performance. Page numbers are not available in the supplied passages. (LLM, conf=0.94)

  cites [27] MH Papesh, 2018, Photo ID verification remains challenging despite years of practice, Cognitive Research: Principles and Implications source
• ?Not checked — no full text of the sourcecites [28] BE Wirth, 2017, An easy game for frauds? Effects of professional experience and time pressure on passport-matching performance, Journal of Experimental Psychology: Applied source

In the paper

“These studies all show no improvement with more professional experience, suggesting that on-the-job learning in these particular roles does not help develop expertise.”

What the sources say

• Supported

  The Discussion states that, across all experiments, length of employment as a passport officer did not predict face-matching accuracy. It interprets this as strong evidence that experience alone does not improve accuracy in that role, supporting the claim’s suggestion about on-the-job learning. (LLM, conf=0.92)

  cites [9] D White, 2014, Passport officers’ errors in face matching, PLoS ONE source
• Partly supported

  Experiment 2 Results report no relationship between years employed and face-matching accuracy for either facial review staff or facial examiners. But the General Discussion says examiners may have developed better strategies through feedback and working with specialists, so the finding does not establish that on-the-job learning does not help develop expertise. (LLM, conf=0.85)

  cites [10] D White, 2015, Error rates in users of automatic face recognition software, PLoS One sourceExperimental — accuracy not yet measured
• Supported

  The Conclusions report that notaries and bank tellers performed similarly to students despite years of on-the-job identity-verification experience, and that weekly and yearly experience showed no effect. The paper also reports a similar finding for passport officials and concludes that passive occupational experience does not benefit unfamiliar face matching; it distinguishes this from dedicated training. (LLM, conf=0.92)

  cites [27] MH Papesh, 2018, Photo ID verification remains challenging despite years of practice, Cognitive Research: Principles and Implications source
• ?Not checked — no full text of the sourcecites [28] BE Wirth, 2017, An easy game for frauds? Effects of professional experience and time pressure on passport-matching performance, Journal of Experimental Psychology: Applied source

In the paper

“Performance on the casework test was also notably low, with Course D trainees achieving just 72% accuracy after training—comparable to novice control participants who achieved 71%.”

In the paper

“Genuine trainees (n = 32) showed small but significant improvements in accuracy on a modified version of the GFMT and the inversion test, but not on the casework test.”

In the paper

“providing no evidence that Courses A and B improve face identification accuracy.”

• Also stated in: § Online training course evaluation · boosted

  “These analyses provide between ‘very strong’ and ‘extreme’ evidence in support of the null hypothesis, i.e. that Courses A and B do not improve face identification accuracy.”

In the paper

“All courses include practical exercises and provide feedback on the accuracy of facial comparison decisions to trainees (see [19]).”

In the paper

“The improvements observed in Course D were small, inconsistent across tests and not accompanied by qualitative markers of expertise in facial examination.”

In the paper

“This is especially important because our results show that 93% of Course C trainees believed they had improved as a result of training, despite this training not improving accuracy.”

In the paper

“The casework test is representative of forensic facial examination casework and was supplied by Course D’s host agency (see Methods for details).”

In the paper

“The FISWG guidelines do not make reference to empirical research supporting the validity of the recommended training practices, so adherence to the guidelines provides no evidence that a particular training course is effective.”

In the paper

“This paper provides the first comprehensive evaluation of the effectiveness of professional facial image comparison training courses.”

In the paper

“This is despite short courses being used routinely by government agencies around the world and complying with FISWG training guidelines (see S1 Appendix).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites S1 Appendix

In the paper

“The courses evaluated here were representative of accepted practice in facial image comparison training, and so results are likely to be indicative of the effectiveness of professional training courses more generally.”

In the paper

“These qualitative differences appear to reflect a greater reliance on feature-level details when comparing face images.”

In the paper

“This suggests that people do not have insight into the effectiveness of facial image comparison training, and that perceived benefit of training is unrelated to its actual benefit.”

In the paper

“Course D trainees had not previously seen the images in the casework test.”

In the paper

“However, these courses have not been empirically validated so it is unknown if they improve accuracy.”

• Also stated in: § Background · hedged

  “The only published evaluation of a professional facial image comparison training course was conducted in the 1970’s by psychologists Woodhead, Baddeley, and Simmonds [4].”

What the sources say

• ?Not checked — no full text of the sourcecites [4] MM Woodhead, 1979, On training people to recognize faces, Ergonomics source

In the paper

“For all tests, we split items into two sets of equal difficulty using accuracy data from previous studies so that equivalent versions could be administered at pre- and post-test.”

In the paper

“We created two equally difficult 20-item versions (10 match, 10 non-match) of the High-to-High and High-to-Low image quality tests. Allocation of each version to pre- and post-training was counterbalanced across participants.”

In the paper

“Some trainees’ data files were corrupt and could not be analysed.”

In the paper

“Test order was randomised for each participant.”

In the paper

“Two versions were randomly assigned to the upright and inverted blocks at pre- and post-training for each participant.”

In the paper

“The order of the GBU-unlimited and GBU-4 seconds was counterbalanced across participants, but held constant pre- to post-training.”

In the paper

“Following Towler, White and Kemp [17], the GFMT was split into two equally difficult 20-item versions (10 match, 10 non-match). Allocation of each version to the pre- and post-training was counterbalanced across participants.”

What the sources say

• ?Not checked — no full text of the sourcecites [17] A Towler, 2014, Evaluating training methods for facial image comparison: The face shape strategy does not work, Perception source

In the paper

“In an effort to equate motivation between the groups, the highest-performing control participant across the pre- and post-training tests received an additional £50 Amazon voucher.”

In the paper

“Task order was randomised for each participant, but held constant pre- to post-training.”

In the paper

“We created four equally difficult 30-item versions (15 match, 15 non-match) using human performance data from O’Toole, An, Dunlop, and Natu [41]. Allocation of each version to the pre- or post-training and GBU-unlimited and GBU-4 seconds was counterbalanced across participants.”

What the sources say

• ?Not checked — no full text of the sourcecites [41] AJ O’Toole, 2012, Comparing face recognition algorithms to humans on challenging tasks, ACM Transactions on Applied Perception source

In the paper

“Second, facial examiners’ training also includes extensive on-the-job mentoring from more senior analysts, which may be critical in developing their expertise.”

What the sources say

• Not checked yet

  Checked before — its earlier verdict is held back until a new check runs.

  cites [25] AJ Dowsett, 2014, Unfamiliar face matching: Pairs out-perform individuals and provide a route to training, British Journal of Psychology source

In the paper

“The test had not been previously seen by trainees. We created two equally difficult versions of the test, consisting of 24 trials each (12 match, 12 non-match).”

In the paper

“One version of the casework test was always administered at pre-training, and always as the last test, as it formed the basis of a pre-existing feedback training module.”

In the paper

“The two equally difficult halves of the GFMT used in the previous evaluations were adapted to exclude identities used in the Cambridge Face Memory Test—Australian (see [42]). One version of this modified GFMT contained 17-items (9 match, 8 non-match) and the other contained 18-items (10 match, 8 non-match). Test version order was randomly allocated to pre- and post-training for each participant.”

What the sources say

• ?Not checked — no full text of the sourcecites [42] E McKone, 2011, Face ethnicity and measurement reliability affect face recognition performance in developmental prosopagnosia: Evidence from the Cambridge Face Memory Test–Australian, Cognitive Neuropsychology source

In the paper

“To equate the groups’ experience as much as possible, control participants completed the pre-training tests on Day 1 and mock training on Days 2 and 3.”

In the paper

“We removed 10 image pairs from the model image set that were used in a pre-existing facial feature analysis exercise during training (see S3 Appendix for details of this exercise). We then created four equally difficult versions, each containing 12 trials (6 match, 6 non-match).”

In the paper

“Training courses were classified as having met each criterion either ‘not at all’, ‘briefly’ or ‘in detail’ by two raters. Disagreement between the raters was resolved by joint review of the material.”

In the paper

“Our approach to evaluating the effectiveness of training courses was not designed to capture these sources of learning.”

In the paper

“Ninety Australian undergraduate psychology students (Mean age = 19 years, SD = 3 years) participated in return for course credit. They were randomly allocated to one of three conditions, such that 30 participants were in each.”