The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Reliable support: Measuring calibration of likelihood ratios
Abstract
Calculation of likelihood ratios (LR) in evidence evaluation still presents major challenges in many forensic disciplines: for instance, an incorrect selection of databases, a bad choice of statistical models, low quantity and bad quality of the evidence are factors that may lead to likelihood ratios supporting the wrong proposition in a given case. However, measuring performance of LR values is not straightforward, and adequate metrics should be defined and used. With this objective, in this work we describe the concept of calibration, a property of a set of LR values. We highlight that some desirable behavior of LR values happens if they are well calibrated. Moreover, we propose a tool for representing performance, the Empirical Cross-Entropy (ECE) plot, showing that it can explicitly measure calibration of LR values. We finally describe some examples using speech evidence, where the usefulness of ECE plots and the measurement of calibration is shown.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A well calibrated likelihood ratio will be accurate with its own predictions.
“A well calibrated LR will be accurate with its own predictions [36].”
From Calibration of score based likelihood ratio estimation in automated forensic facial image comparison (2022) · cites this paper as [36]
In [36], the problem of incorrect selection of databases is put forward.
“In [36], the problem of incorrect selection of databases is put forward.”
From Calibration of score based likelihood ratio estimation in automated forensic facial image comparison (2022) · cites this paper as [36]
Compared to equal error rate or ROC curves, Cllr, Cllr Min and ECE plots provide a better representation of both the discrimination power of a model and its calibration performance.
“Compared to equal error rate or ROC curves, these metrics provide a better representation of both the discrimination power of the model and its calibration performance.”
From Calibration of score based likelihood ratio estimation in automated forensic facial image comparison (2022) · cites this paper as [36]
Cllr and ECE plots are the best indicators of both discrimination and calibration performances for this work according to the studied literature.
“In conclusion, for this work and according to the studied literature, the best indicators of both discrimination and calibration performances are Cllr and the ECE plots (explained in 3.3.1)[36,38] which give a good view of both the calibration and discrimination power of the forensic experts and the automated systems.”
From Calibration of score based likelihood ratio estimation in automated forensic facial image comparison (2022) · cites this paper as [36]
The Likelihood Ratio is a possible solution for standardization in forensic facial image comparison.
“The Likelihood Ratio (LR) comes as a possible solution [26,36] for standardization, expressing the decision as the ratio of the probability given the evidence of a match against the probability of a non-match.”
From Calibration of score based likelihood ratio estimation in automated forensic facial image comparison (2022) · cites this paper as [36]
Citations
Cites
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2005) · doi:10.1002/047174882x · Cited by 13,791
Dawid, A. P. (1982) · Journal of the American Statistical Association · doi:10.1080/01621459.1982.10477856 · Cited by 625 · cited 3×
“Mainly because of the property described above, calibration has been dubbed reliability in the context of Bayesian probabilistic assessment [14,11].”
BackgroundThe Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604 · cited 6×
“In order to measure the accuracy of a validation set of posterior probabilities from the forecaster, the following procedure has been proposed [11]: LC 1⁄4 1 N p X”
BackgroundMethodInfluential
Flagged first, then by citations
Research works printed but not linked: 28
- Evett, 2011, Expressing evaluative opinions: a position statement, Sci. Justice, 51, 1, 10.1016/j.scijus.2011.01.002 · DOI
- Royall, 2000, On the probability of observing statistical misleading evidence, J. Am. Stat. Assoc., 95, 760, 10.1080/01621459.2000.10474264 · DOI
- Curran, 2002, Assessing uncertainty in DNA evidence caused by sampling effects, Sci. Justice, 42, 29, 10.1016/S1355-0306(02)71794-2 · DOI
- Saunders, 2012, The effect of uncertainty about the background population on the forensic value of the evidence
- Zadora, 2010, Evaluation of glass samples for forensic purposes – an application of likelihood ratio model and information-theoretical approach, Chemometr. Intell. Lab. Syst., 102, 62, 10.1016/j.chemolab.2010.03.007 · DOI · Paper page
- Gilbert, 2010, DNA's identity crisis, Nature, 464, 347, 10.1038/464347a · DOI
- Gill, 2000, An investigation of the rigor of interpretation rules for STRs derived from less than 100 pg of DNA, Forensic Sci. Int., 112, 17, 10.1016/S0379-0738(00)00158-4 · DOI
- Savage, 1971, The elicitation of personal probabilities and expectations, J. Am. Stat. Assoc., 66, 783, 10.1080/01621459.1971.10482346 · DOI
- Gneiting, 2007, Strictly proper scoring rules, prediction and estimation, J. Am. Stat. Assoc., 102, 359, 10.1198/016214506000001437 · DOI · Paper page
- Lindley, 1979, On the reconciliation of probability assessments, J. Roy. Stat. Soc. A, 142, 146, 10.2307/2345078 · DOI
- Garthwaite, 2005, Statistical methods for eliciting probability distributions, J. Am. Stat. Assoc., 100, 680, 10.1198/016214505000000105 · DOI
- O’Hagan, 2004, Dicing with the unknown, Significance, 1, 132, 10.1111/j.1740-9713.2004.00050.x · DOI
- Brümmer, 2006, Application independent evaluation of speaker detection, Comput. Speech Lang., 20, 230, 10.1016/j.csl.2005.08.001 · DOI
- N. Brümmer, Measuring, refining and calibrating speaker and language information extracted from speech, Ph.D. Thesis, School of Electrical Engineering, University of Stellenbosch, Stellenbosch, South Africa, available at: http://sites.google.com/site/nikobrummer/, 2010.
- D. Ramos, Forensic evaluation of the evidence using automatic speaker recognition systems, Ph.D. Thesis, Depto. de Ingenieria Informatica, Escuela Politecnica Superior, Universidad Autonoma de Madrid, Madrid, Spain, available at: http://atvs.ii.uam.es, 2007.
- D. Ramos, J. Gonzalez-Rodriguez, G. Zadora, C. Aitken, Information-theoretical assessment of the performance of likelihood ratio computation methods, J. Forensic Sci., in press.
- Cook, 1998, A model for case assessment and interpretation, Sci. Justice, 38, 151, 10.1016/S1355-0306(98)72099-4 · DOI
- Cook, 1998, A hierarchy of propositions: deciding which level to address in casework, Sci. Justice, 38, 231, 10.1016/S1355-0306(98)72117-3 · DOI
- Evett, 1995, Avoiding the transposed conditional, Sci. Justice, 35, 127, 10.1016/S1355-0306(95)72645-4 · DOI
- Evett, 1996, Statistical Analysis of STR Data, 6, 79, 10.1007/978-3-642-80029-0_20 · DOI
- Gonzalez-Rodriguez, 2007, Emulating DNA. Rigorous quantification of evidential weight in transparent and testable forensic speaker recognition, IEEE Transactions on Audio, Speech Lang. Process., 15, 2072, 10.1109/TASL.2007.902747 · DOI
- D. Meuwly, Reconaissance de Locuteurs en Sciences Forensiques: L’apport d’une Approache Automatique, Ph.D. Thesis, IPSC-Universite de Lausanne, 2001.
- Meuwly, 2007, Forensic individualisation from biometric data, Sci. Justice, 46, 205, 10.1016/S1355-0306(06)71600-8 · DOI
- Egli, 2007, Evidence evaluation in fingerprint comparison and automated fingerprint identification systems – modelling within finger variability, Forensic Sci. Int., 167, 189, 10.1016/j.forsciint.2006.06.054 · DOI
- N. Egli, Interpretation of partial fingermarks using an automated fingerprint identification system, Ph.D. Thesis, Institute de Police Scientifique, Ecole de Sciences Criminelles, 2009.
- Neumann, 2012, Quantifying the weight of evidence from a forensic fingerprint comparison: a new paradigm, J. Roy. Stat. Soc. Ser. A: Stat. Soc., 175, 371, 10.1111/j.1467-985X.2011.01027.x · DOI
- Hepler, 2012, Score-based likelihood ratios for handwriting evidence, Forensic Sci. Int., 219, 129, 10.1016/j.forsciint.2011.12.009 · DOI
- D. Ramos, J. Franco-Pedroso, J. Gonzalez-Rodriguez, Calibration and weight of the evidence by human listeners: the ATVS-UAM submission to NIST human-aided speaker recognition 2010, in: Proc. of ICASSP 2011, Pague, Czeck Republic, 2011, pp. 5908–5911. · DOI
Cited by
Sorted by citations
Calibration of score based likelihood ratio estimation in automated forensic facial image comparison
Rodriguez, A. M., Geradts, Z., & Worring, M. (2022) · Forensic Science International · doi:10.1016/j.forsciint.2022.111239 · Cited by 19
Metadata
- Authors
- Ramos, D., & Gonzalez-Rodriguez, J.
- Venue
- Forensic Science International
- Publisher
- Elsevier BV
- Published
- 2013-07
- Volume
- 230
- Issue
- 1-3
- Pages
- 156-169
- ISSN
- 0379-0738
- Cited by
- 118
- References
- 3
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
