CoChat
Sign Up
Cited by 19Face recognition and analysisBiometric Identification and SecurityFace and Expression Recognition

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 16 of the 45 sources its claims cite were checked for support.

Calibration of score based likelihood ratio estimation in automated forensic facial image comparison

Open access
Rodriguez, Andrea Macarulla, Geradts, Zeno, Worring, Marcel·2022·Forensic Science International
DOI 10.1016/j.forsciint.2022.111239
Open this paper in CoChat — free accountOpen PDF

Abstract

Forensic facial image comparison lacks a methodological standardization and empirical validation. We aim to address this problem by assessing the potential of machine learning to support the human expert in the courtroom. To yield valid evidence in court, decision making systems for facial image comparison should not only be accurate, they should also provide a calibrated confidence measure. This confidence is best conveyed using a score-based likelihood ratio. In this study we compare the performance of different calibrations for such scores. The score, either a distance or a similarity, is converted to a likelihood ratio using three types of calibration following similar techniques as applied in forensic fields such as speaker comparison and DNA matching, but which have not yet been tested in facial image comparison. The calibration types tested are: naive, quality score based on typicality, and feature-based. As transparency is essential in forensics, we focus on state-of-the-art open software and study their power compared to a state-of-the-art commercial system. With the European Network of Forensic Science Institutes (ENFSI) Proficiency tests as benchmark, calibration results on three public databases namely Labeled Faces in the Wild, SC Face and ForenFace show that both quality score and feature based calibration outperform naive calibration. Overall, the commercial system outperforms open software when evaluating these Likelihood Ratios. In general, we conclude that calibration implemented before likelihood ratio estimation is recommended. Furthermore, in terms of performance the commercial system is preferred over open software. As open software is more transparent, more research on open software is urged for.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This paper evaluates score-based likelihood ratio (SLR) calibration for automated forensic facial image comparison, comparing naive, quality-score, and feature-based calibration on public face datasets and ENFSI proficiency tests. It finds that quality-score and feature-based calibration generally outperform naive calibration, while the commercial FaceVACS system outperforms open-source Deepface/Dlib.

Key findings

  • Both quality-score and feature-based calibration outperformed naive calibration in Cllr and ECE terms; feature calibration improved over quality-score calibration in the reported Cllr results.
  • The commercial system FaceVACS generally outperformed the open-source system Dlib when evaluating likelihood ratios; for example, in Table 3 for 2011 with no filters, Cllr was 0.11 for FaceVACS versus 0.22 for Dlib, and average participants were 0.40.
  • Calibration performance varied strongly by ENFSI test year: 2020 was poor for both automated systems (Dlib 2.59, FaceVACS 2.74 with no filters; average participants 0.78), attributed in the text to identical twins in the test images.
  • Among the three statistical calibration methods tested—Isotonic Regression, Kernel Density Estimation, and Logistic Regression—none clearly stood out, though Isotonic Regression seemed to perform slightly better.

Methods

  • Design: compare SLR calibration types for automated facial image comparison, using ENFSI Proficiency Face Comparison tests as benchmark and comparing automated systems against forensic experts.
  • Data: public databases Labeled Faces in the Wild (13,233 images, 5,749 subjects), SC Face (4,160 images, 130 subjects), and ForenFace (mostly videos, 97 subjects); ENFSI proficiency tests from 2011–2013 and 2017–2020.
  • Scorers and calibration: open-source Deepface/Dlib and commercial FaceVACS 5.5.2; calibration types were naive, quality-score based on typicality, and feature-based; WSV/BSV distributions were fitted using Isotonic Regression, Kernel Density Estimation, or Logistic Regression; validation used Cllr, Cllr min, and ECE plots.

Limitations

  • The paper states that further work is needed to recommend which calibration method suits which use case, and that no calibration method clearly outperformed the others.
  • The paper states that automated system performance is less reliable with occlusions, and that when faces were not detected they were excluded from Cllr/ECE plots; adding non-detections as inconclusive LR = 1 would lower Cllr performance.
  • The paper states that ENFSI test images were often easier than normal casework, especially mostly frontal images with little pose variation, and that only one image per identity was available, so suspect-anchored or trace-anchored calibration was not the use case.
  • Evident from the text: results are highly year- and test-composition-dependent, with specific difficulties such as twins, children, age variation, pose, and quality, which limits generalizability beyond these ENFSI tests.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.forsciint.2022.111239

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

16 of 45 cited sources checked · 9 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    In facial image comparison, studies did not take into account how different calibration characteristics such as features affect the results.

    “Although the process of calibration has been studied and analysed in speaker comparison works such as [28] or [31], [3] and in facial image comparison in [40] those studies in facial image comparison did not take into account how different calibration characteristics such as features affect the results.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [3] Ali, 2014 source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [40] Rodriguez, 2020, Likelihood ratios for deep neural networks in face comparison, J. Forensic Sci. source

      The Results (ENFSI Test 2011) compare three calibration methods and report differences in their results. The Discussion also tests calibration datasets with different sizes and image characteristics and reports an effect on results. The paper therefore does examine how differing calibration choices affect outcomes, although these passages do not establish that it tested feature selection specifically. Page numbers are unavailable in our copy. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    Forensic facial image comparison lacks a standard method for interpreting the comparison between trace and suspect images.

    “Forensic facial image comparison lacks a methodological standardization and empirical validation.”
    • Also stated in: § 1. Introduction · plain
      “No standard method is, however, available for that task.”
    • Also stated in: § 1. Introduction · plain
      “Although the use of automatic systems is increasingly studied in the field of facial image comparison, for legal deployment it lacks standardization and validation.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [16] Jacquet, 2020, Automated face recognition in forensic science: Review and perspectives, Forensic Sci. Int. source
    • cites [45] 2017

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    Compared to equal error rate or ROC curves, Cllr, Cllr Min and ECE plots provide a better representation of both the discrimination power of a model and its calibration performance.

    “Compared to equal error rate or ROC curves, these metrics provide a better representation of both the discrimination power of the model and its calibration performance.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [16] Jacquet, 2020, Automated face recognition in forensic science: Review and perspectives, Forensic Sci. Int. source

      The ECE and cost-of-log-likelihood-ratio section describes ECE plots showing a model before and after calibration, while the DET/EER section says DET plots show discrimination but not calibration. However, it identifies minCllr specifically as a measure of discrimination, not of both characteristics, and our copy does not make the stated comparison with ROC curves. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [36] Ramos, 2013, Reliable support: measuring calibration of likelihood ratios, Forensic Sci. Int. source
  • Premisecitation

    Cllr and ECE plots are the best indicators of both discrimination and calibration performances for this work according to the studied literature.

    “In conclusion, for this work and according to the studied literature, the best indicators of both discrimination and calibration performances are Cllr and the ECE plots (explained in 3.3.1)[36,38] which give a good view of both the calibration and discrimination power of the forensic experts and the automated systems.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [36] Ramos, 2013, Reliable support: measuring calibration of likelihood ratios, Forensic Sci. Int. source
    • cites [38] Ramos, 2017, Likelihood ratio data to report the validation of a forensic fingerprint evaluation method, Data Brief source

      The Validation report’s Discriminating power and Calibration sections use ECE plots and Cllr-derived measures to assess those properties. Its Accuracy section uses Cllr with an ECE plot. The paper does not call Cllr and ECE the best indicators, and its reported comparison concerns fingerprint evaluation methods rather than the performance of forensic experts. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Premisecitation

    There is a lack of a study on the impact of adverse conditions on calibration in facial image comparison.

    “There is a lack of a similar study in case of facial image comparison.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [48] Zadora, 2010, Evaluation of glass samples for forensic purposes-an application of likelihood ratios and an information–theoretical approach, Chemom. Intell. Lab. Syst. source

      Section 3.4.1 examines how mismatched glass databases affect calibration, so this paper is an example of the kind of study the citing sentence calls “similar.” It does not address facial image comparison or establish that such a study is lacking in that field. (LLM, conf=0.80)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim. 33
  • It may state two separate claims as one. 22

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Premisecitation

    Automated systems for facial image comparison, especially deep-learning systems, combined with score-based likelihood ratio estimation have great potential to help the forensic expert in the evaluative process.

    “As explored in [40] and [3], automated systems for facial image comparison (especially when based on deep learning) combined with score-based likelihood ratio estimation have a great potential to help the forensic expert in the evaluative process [16].”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [40] Rodriguez, 2020, Likelihood ratios for deep neural networks in face comparison, J. Forensic Sci. source
    • cites [3] Ali, 2014

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [16] Jacquet, 2020, Automated face recognition in forensic science: Review and perspectives, Forensic Sci. Int. source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Current face recognition systems already reach very high levels of accuracy in public non-forensic benchmarks.

    “Current face recognition systems [8,24,43], already reach very high levels of accuracy in public non-forensic benchmarks, and it is expected that in the coming years they will keep improving.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [8] Deng, J., Guo, J., Xue, N., Zafeiriou, S., 2019. Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4685–4694. 10.1109/CVPR.2019.00482. source
    • cites [24] Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L., 2017. Sphereface: deep hypersphere embedding for face recognition. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6738–6746. source
    • cites [43] Schroff, F., Kalenichenko, D., Philbin, J., 2015. Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 815–823. 10.1109/CVPR.2015.7298682. source

  • Probably not a substantial claim.

    Premisecitation

    No study has yet shown the impact of variations in the choice of the relevant population for automatic face recognition.

    “According to [16], no study has yet shown the impact of variations in the choice of the relevant population for automatic face recognition.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [16] Jacquet, 2020, Automated face recognition in forensic science: Review and perspectives, Forensic Sci. Int. source

  • Probably not a substantial claim.

    Premisecitation

    Facial image comparison has proven to have potential to help the forensic expert if the likelihood ratio estimation method is properly standardized and validated.

    “And facial image comparison has proven to have potential to help the forensic expert if the likelihood ratio estimation method is properly standardized and validated [40].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [40] Rodriguez, 2020, Likelihood ratios for deep neural networks in face comparison, J. Forensic Sci. source

      The Conclusion (page not available) says the software can help forensic experts, particularly with full-frontal images. The paper also describes validating its likelihood-ratio estimates against expert results and discusses guidelines that standardize forensic reasoning. It does not establish that proper standardization and validation of the estimation method are the condition under which facial image comparison generally helps experts; its demonstrated benefit is narrower. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    For face images, a solution for data scarcity could be Generative Adversarial Networks that generate realistic fake face images.

    “For face images this would imply that a solution for data scarcity could be Generative Adversarial Networks (GAN) that generate realistic fake face images [18].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [18] Karras, T., Aila, T., Laine, S., Lehtinen, J., 2018. Progressive growing of gans for improved quality, stability, and variation. In: Proceedings of the International Conference on Learning Representations. source

      The Abstract (p. 1) reports high-quality generated CELEBA images, and §6.3 (p. 8) describes high-resolution images produced by the GAN. The paper does not assess whether those images remedy data scarcity; §7 (p. 9) also cautions that the results fall short of true photorealism. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim.

    Premisecitation

    The performance of facial image comparison highly depends on the quality of the data that a model is built with.

    “Given that the performance of facial image comparison highly depends on the quality of the data that a model is built with, the author in [34] suggests to use images having similar conditions to the real life facial image comparisons.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [34] Peng, Y., 2019. Face recognition at a distance: low-resolution and alignment problems. Ph.D. thesis. UT. Netherlands. 10.3990/1.9789036547116. source

  • Probably not a substantial claim. It may state two separate claims as one.

    Premiseown data

    In the literature, such calibrations are performed normally as suspect-anchored and trace-anchored, however this type of calibration was not the use case in this study due to only having one sample of each identity in the comparison tests.

    “In the literature, such calibrations are performed normally as suspect-anchored and trace-anchored [16,39], however this type of calibration was not the use case in this study due to only having one sample of each identity in the comparison tests.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [16] Jacquet, 2020, Automated face recognition in forensic science: Review and perspectives, Forensic Sci. Int. source

      The Trace-anchored approach and Source (suspect)-anchored approach sections describe both methods, supporting their identification as approaches in the literature. But the paper discusses them as ways to model score variability, not as the normal forms of calibration; its Suspect-independent/Generic approach section says a generic approach is used in most cases when only one or no reference image is available. The statement about the citing study’s samples is not attributed to this source. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [39] Ramos, 2017, From biometric scores to forensic likelihood ratios source

  • It may state two separate claims as one.

    Findingown data

    For the years 2017 and 2018, the Cllr is approximately 1, indicating the power decision of a random algorithm.

    “For the years 2017 and 2018 the Cllr is approximately 1, which indicates the power decision of a random algorithm.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingnone given

    Most of the images (especially in the year 2017) are frontal with little pose variation, which facilitates the task for the automated system.

    “Most of the images (especially in the year 2017) are frontal with little pose variation, which facilitates the task for the automated system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Findingnone given

    The automated system performance (both FaceVACS and Deepface) is less reliable if there are occlusions.

    “As future work, it would be convenient to indicate that the automated system performance (both FaceVACS and Deepface) is less reliable if there are occlusions.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The Isotonic Calibrator seemed to outperform a bit with respect to Logistic Regression and Calibration, although calibrating with any of the three would turn out to be similar to Cllr.

    “The calibrator chosen for the results in the table was Isotonic Calibrator, although calibrating with any of the three would turn out to be similar to Cllr, the Isotonic seemed to outperform a bit with respect to Logistic Regression and Calibration.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    Both the confusion score and labeled filters improved the Cllr with respect to applying generic calibration.

    “With respect to the three calibration methods chosen, although both the confusion score and labeled filters improved the Cllr with respect to applying generic calibration, also further research is needed to help the investigator to determine which method would suit best for each use case.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    When the size of the image is above 0.3 megapixels, the similarity of matched pairs is close to 1, and close to 0 in the case of mismatches.

    “When the size of the image (measured in megapixels) is above 0.3, the similarity of matched pairs is close to 1, and close to 0 in the case of mismatches.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    As resolution of the images decreases, similarity for matched images also decreases and similarity for mismatches rises for some cases.

    “As resolution of the images decreases, similarity for matched images also decreases and similarity for mismatches rises for some cases.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingreasoning

    For the year 2019, the Cllr is quite high, indicating poor performance, but it is in the same interval as the forensic experts.

    “For the year 2019, the Cllr is quite high, which indicates a poor performance, but it is in the same interval as the forensic experts.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingnone given

    Human participants managed to have their Cllr below one in year 2011 (except 2 participants) and about two thirds of them had a Cllr below 1 in year 2018.

    “On the other hand, human participants managed to have their cllr below one in year 2011 (except 2 participants) and about two thirds of them had a cllr below 1 in year 2018.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given

    In the ECE plots, both FaceVACs and Face Recognition seem to make fewer errors in the prosecution priors than in the defense priors.

    “Regarding the ECE plots, both FaceVACs and Face Recognition seem to make less errors in the prosecution priors than in the defense priors.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingnone given

    For the rest of the years 2011, 2012 and 2013, both FaceVACs and Face recognition present results comparable to the best performing experts.

    “For the rest of the years 2011, 2012 and 2013, both the commercial software FaceVACs and the open software Face recognition present results comparable to the best performing experts.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The commercial software FaceVACs outperforms both the open software face recognition and the experts for full frontal images.

    “As we can see in the Cllr and ECE plots, the commercial software FaceVACs outperforms both the open software face recognition and the experts for full frontal images.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Calibration in speaker recognition is based on features namely duration of audio, distance, language, and gender.

    “Calibration in speaker recognition is based on features namely duration of audio, distance, language, and gender [4].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Brummer, 2010

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may state two separate claims as one.

    Premisecitation

    Discrimination is robust independently from the dataset the system was calibrated with, but calibration itself is highly dependent on the conditions.

    “The discrimination is robust independently from the dataset the system was calibrated with, but calibration itself is highly dependent on the conditions [31].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31] Nandwana, M.K., Ferrer, L., McLaren, M., Castan, D., Lawson, A., 2019. Analysis of Critical Metadata Factors for the Calibration of Speaker Recognition Systems. In: Proceedings of the Interspeech 2019, pp. 4325–4329. 10.21437/Interspeech.2019–1808. source

  • It may state two separate claims as one.

    Premisecitation

    Minimizing Cllr implies an improvement of both discrimination and calibration performance of the automated system.

    “Minimizing the value of Cllr implies an improvement of both discrimination and calibration performance of the automated system [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Ramos, 2017, From biometric scores to forensic likelihood ratios source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    The use of simulated data gives a big improvement in data scarcity situations, but the testing of the validity of simulated databases for the operational use of systems in a real setup is still controversial.

    “The references indicate that the use of simulated data gives a big improvement in data scarcity situations, but the testing of the validity of simulated databases for the operational use of systems in a real setup is still controversial.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [46] Villalba, J., Brümmer, N., 2011. Towards fully bayesian speaker recognition: integrating out the between-speaker covariance. In: Proceedings of the Interspeech 2011, ISCA. 10.21437/interspeech.2011–142. source
    • cites [47] Zadora, 2013

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning

    For facial image comparison, ECE methods for evaluation are adequate for detection if the performance of both the automated model and the forensic participant are affected in the same way by the chosen calibration population.

    “For facial image comparison this implies that ECE methods for evaluation are adequate for detection if the performance of both the automated model and the forensic participant are affected in the same way by the chosen calibration population.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    The PAV LRs curve shows the best possible ECE in terms of calibration, and it is a measure of discriminating power.

    “This shows the best possible ECE in terms of calibration, and it is a measure of discriminating power.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Ramos, 2017, From biometric scores to forensic likelihood ratios source

  • Probably not a substantial claim.

    Premisecitation

    The Likelihood Ratio is a possible solution for standardization in forensic facial image comparison.

    “The Likelihood Ratio (LR) comes as a possible solution [26,36] for standardization, expressing the decision as the ratio of the probability given the evidence of a match against the probability of a non-match.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [26] Meuwly, 2017, A guideline for the validation of likelihood ratio methods used for forensic evidence evaluation, Forensic Sci. Int. source
    • cites [36] Ramos, 2013, Reliable support: measuring calibration of likelihood ratios, Forensic Sci. Int. source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    In speaker comparison, the effect of duration, distance, language, and gender by using a variety of datasets makes a difference in the calibration results.

    “In particular, in the work of [31] the effect of duration, distance, language, and gender in speaker comparison by using a variety of datasets makes a difference in the calibration results.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] Nandwana, M.K., Ferrer, L., McLaren, M., Castan, D., Lawson, A., 2019. Analysis of Critical Metadata Factors for the Calibration of Speaker Recognition Systems. In: Proceedings of the Interspeech 2019, pp. 4325–4329. 10.21437/Interspeech.2019–1808. source

  • Probably not a substantial claim.

    Premisecitation

    The accuracy of an LR method should always be better than the neutral reference.

    “This neutral method is taken as baseline performance: the accuracy should always be better than the neutral reference.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] Ramos, 2017, From biometric scores to forensic likelihood ratios source

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationnone given

    The expert cannot be replaced by this tool, but becomes more efficient because the computer can help to reduce the amount of information to be managed by doing appropriate filtering.

    “The expert cannot be replaced by this tool, but becomes more efficient because the computer can help to reduce the amount of information to be managed by doing appropriate filtering.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    For our work, measuring with Cllr and ECE plot covers the warning that validation of LR systems should check both accuracy and calibration, because the cost will increase for those systems that provide wrong assessments or low discrimination power (LR close to one).

    “For our work, measuring with Cllr and ECE plot has this warning covered, because looking at both Eqs. 3 and 4, the cost will increase for those systems that provide wrong assessments or low discrimination power (LR close to one).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Premisecitation

    Adverse conditions greatly affect calibration performance.

    “They find this impact greatly affects calibration performance.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [48] Zadora, 2010, Evaluation of glass samples for forensic purposes-an application of likelihood ratios and an information–theoretical approach, Chemom. Intell. Lab. Syst. source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    Integration of advanced machine-learning algorithms for the compensation of adverse conditions into forensic evaluation helps.

    “The analysis of [48] shows that integration of advanced machine-learning algorithms for the compensation of adverse conditions into forensic evaluation helps in this direction.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] Zadora, 2010, Evaluation of glass samples for forensic purposes-an application of likelihood ratios and an information–theoretical approach, Chemom. Intell. Lab. Syst. source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation

    On the Labeled Faces in the Wild benchmark, Deepface-supported models achieved accuracy scores of 99.65% for FaceNet-512, 99.41% for ArcFace, 99.38% for Dlib, and 98.78% for VGG-Face, whereas human beings could have just 97.53%.

    “According to [44], the supported models FaceNet-512 got 99.65%; ArcFace got 99.41%; Dlib got 99.38% and VGG-Face got 98.78%; accuracy scores on Labeled Faces in the Wild benchmark whereas human beings could have just 97.53%.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] Serengil, S.I., Ozpinar, A., 2020. Lightface: A hybrid deep face recognition framework. In: Proceedings of the 2020 Innovations in Intelligent Systems and Applications Conference (ASYU), IEEE. pp. 23–27. 10.1109/ASYU50717.2020.9259802. source

  • Probably not a substantial claim.

    Premisecitation

    The reference curve represents the comparative performance of a neutral LR method, defined as the one which always delivers LR = 1 for each forensic case in the set of LR values.

    “This curve represents the comparative performance of a so-called neutral LR method, defined as the one which always delivers LR = 1 for each forensic case in the set of LR values.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] Ramos, 2017, From biometric scores to forensic likelihood ratios source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    In [3], on average 59.2% of the cases using pseudo-traces were more effective than the generic approach.

    “In their results, on average 59,2% of the cases using this approach were more effective than the generic approach.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3] Ali, 2014

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may state two separate claims as one.

    Premisenone given

    The probability functions of within-source variability and between-source variability are unknown, so it is necessary to obtain them empirically.

    “As the probability functions of WSV and BSV are unknown, it is necessary to obtain them empirically.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Premisecitation

    Validation of Likelihood Ratio in the forensic field should take into account not only accuracy but also its calibration, i.e., the system capacity to make strong assessments.

    “As indicated, an important point made by [10] is that validation of Likelihood Ratio in the forensic field should take into account not only accuracy (if it is right or wrong assessment of match-mismatch) but also its calibration, i.e., the system capacity to make strong assessments.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Dror, 2020, (Mis)use of scientific measurements in forensic science, Forensic Sci. Int.: Synerg. source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisenone given

    A Cllr value of one indicates the system makes a random selection.

    “A value of one indicates the system makes a random selection.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    The work of [28] studies the impact in forensic voice comparison of lack of calibration and of mismatched conditions between the known-speaker recording and the relevant-population sample recordings.

    “The work of [28] studies the impact in forensic voice comparison of lack of calibration and of mismatched conditions between the known-speaker recording and the relevant-population sample recordings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] Morrison, 2017, The impact in forensic voice comparison of lack of calibration and of mismatched conditions between the known-speaker recording and the relevant-population sample recordings, Forensic Sci. Int. source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisereasoning

    The solid curve in an ECE plot should always be lower than the reference curve, for all represented values of the prior log-odds.

    “Therefore, the solid curve in an ECE plot should be always be lower than the reference curve, for all represented values of the prior log-odds.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Ramos, 2017, From biometric scores to forensic likelihood ratios source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given

    The poor Cllr in 2020 could be due to that year having identical twins in the ENFSI tests, which confused the algorithm and led it to classify as matches what should have been mismatches.

    “This could be due to that year having identical twins in the ENFSI tests, which confused the algorithm and led it to classify as matches what should have been mismatches.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    In [36], the problem of incorrect selection of databases is put forward.

    “In [36], the problem of incorrect selection of databases is put forward.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] Ramos, 2013, Reliable support: measuring calibration of likelihood ratios, Forensic Sci. Int. source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation

    A problem that could arise when calibrating is data scarcity.

    “A problem that could arise when calibrating [46,47] is data scarcity.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [46] Villalba, J., Brümmer, N., 2011. Towards fully bayesian speaker recognition: integrating out the between-speaker covariance. In: Proceedings of the Interspeech 2011, ISCA. 10.21437/interspeech.2011–142. source
    • cites [47] Zadora, 2013

  • It may state two separate claims as one.

    Interpretationown data

    The automated system had difficulty differentiating twin faces and gives them a high similarity score, making the calibration prone to error.

    “The automated system had difficulty differentiating these faces and gives them a high similarity score, making the calibration prone to error.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationreasoning

    In 2019, the comparison of faces was among children, so both the algorithm and the experts had difficulties with the images.

    “This year the comparison of faces was among children so both the algorithm and the experts had difficulties with the images.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given

    This study takes a step further the usability of automated facial image comparison systems in the forensic field.

    “This study takes a step further the usability of automated facial image comparison systems in the forensic field.”
    Quote in the paperNot a substantial claim
46 of 46

  • The paper relies on this without support.

    The three calibration types tested (naive, quality score based on typicality, and feature-based) have not yet been tested in facial image comparison.

    “The calibration types tested are: naive, quality score based on typicality, and feature-based.”
    “but which have not yet been tested in facial image comparison.”

  • The paper relies on this without support.

    The higher the number of comparisons and the more similar the dataset is to the tested data, the better the calibration will be.

    “Intuitively, the higher the number of comparisons and the more similar the dataset is to the tested data, the better the calibration will be.”

  • The paper relies on this without support.

    The choice of one calibration method (Isotonic Regression, Kernel Density Estimation, Logistic Regression) does not seem to have a correlation with the performance of the different models of Likelihood Ratio estimation.

    “The choice of one method or another doesn't seem to have a correlation with the performance of the different models of Likelihood Ratio estimation.”

  • The paper relies on this without support.

    The quality of images presented in the ENFSI tests are easier for the automated system than the material that is normally handled in cases.

    “However, the quality of images presented in the tests are easier for the automated system than the material that is normally handled in cases.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported naive 2020 Cllr is not approximately four

    Major

    In the naive-calibration discussion, the paper describes the 2020 Cllr as approaching 4. Table 3 instead lists unfiltered values of 2.59 for Dlib and 2.74 for FaceVACS.

1 of 1

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Jilani, 2020, FIC test 2020 -Planning and implementation of the ENFSI-DIWG proficiency test concerning facial image comparisons, ENFSI Intern report
  2. Morrison, G.S., Ochoa, F., Thiruvaran, T., 2012. Database selection for forensic voice comparison, In: Proceedings of the Odyssey 2012-The Speaker and Language Recognition Workshop.

Cited by

Sorted by citations

Related

Metadata

Authors
Rodriguez, A. M., Geradts, Z., & Worring, M.
Affiliations
Netherlands Forensic Institute (NL)
ORCID
Venue
Forensic Science International
Publisher
Elsevier BV
Published
2022-05
Volume
334
Pages
111239
ISSN
0379-0738 (ISSN-L)
DOI
10.1016/j.forsciint.2022.111239
OpenAlex
W4220707461
Semantic Scholar
08f8e8df5d49f7cb2c0a47ba52c54e4d5168f39c
Cited by
19
References
37
Citations by year
2026: 4 · 2025: 6 · 2024: 4 · 2023: 5
FWCI
1.04
Type
journal article
Language
en
Open access
Licence
Locations
  • Forensic Science International (publishedVersion)
  • PubMed (publishedVersion)
  • UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
likelihood ratio estimation, score calibration, confidence calibration, Labeled Faces in the Wild, expert decision support
MeSH terms
Machine Learning, Calibration, Forensic Medicine, Humans, Software, Forensic Sciences

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review