Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies
Open accessThe DOI matches its Crossref record; the references it cites are not checked yet.
Abstract
BACKGROUND: Verbal autopsy (VA) is an important method for obtaining cause of death information in settings without vital registration and medical certification of causes of death. An array of methods, including physician review and computer-automated methods, have been proposed and used. Choosing the best method for VA requires the appropriate metrics for assessing performance. Currently used metrics such as sensitivity, specificity, and cause-specific mortality fraction (CSMF) errors do not provide a robust basis for comparison. METHODS: We use simple simulations of populations with three causes of death to demonstrate that most metrics used in VA validation studies are extremely sensitive to the CSMF composition of the test dataset. Simulations also demonstrate that an inferior method can appear to have better performance than an alternative due strictly to the CSMF composition of the test set. RESULTS: VA methods need to be evaluated across a set of test datasets with widely varying CSMF compositions. We propose two metrics for assessing the performance of a proposed VA method. For assessing how well a method does at individual cause of death assignment, we recommend the average chance-corrected concordance across causes. This metric is insensitive to the CSMF composition of the test sets and corrects for the degree to which a method will get the cause correct due strictly to chance. For the evaluation of CSMF estimation, we propose CSMF accuracy. CSMF accuracy is defined as one minus the sum of all absolute CSMF errors across causes divided by the maximum total error. It is scaled from zero to one and can generalize a method's CSMF estimation capability regardless of the number of causes. Performance of a VA method for CSMF estimation by cause can be assessed by examining the relationship across test datasets between the estimated CSMF and the true CSMF. CONCLUSIONS: With an increasing range of VA methods available, it will be critical to objectively assess their performance in assigning cause of death. Chance-corrected concordance and CSMF accuracy assessed across a large number of test datasets with widely varying CSMF composition provide a robust strategy for this assessment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For verbal autopsy, the results of the Population Health Metrics Research Consortium verbal autopsy studies [91,92] provide a potential strategy for assessing misclassification uncertainty.”
From Modeling causes of death: an integrated approach using CODEm (2012) · cites this paper as [92]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists are queued
They are fetched later and appear here then.
References 47 · Cited by 96
Research works printed but not linked: 2
- Chandramohan D, Maude GH, Rodrigues LC, Hayes RJ: Verbal autopsies for adult deaths: issues in their development and validation. Int J Epidemiol. 1994, 23: 213-222. 10.1093/ije/23.2.213. · DOI · Paper page
- Rosenberg M: Logic of Survey Analysis. 1968, New York, NY: Basic Books, 9
Metadata
- Authors
- Murray, C. J., Lozano, R., Flaxman, A. D., Vahdatpour, A., & Lopez, A. D.
- Affiliations
- University of Washington (US)
- Corresponding author
- Murray, Christopher JL
- Venue
- Population Health Metrics
- Published
- 2011-08-04
- Volume
- 9
- Issue
- 1
- Pages
- 28-28
- ISSN
- 1478-7954 (ISSN-L)
- PMC
- PMC3160921
- OpenAlex
- W2164561875
- Cited by
- 96
- References
- 47
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 2 · 2023: 5 · 2022: 9 · 2021: 11 · 2020: 5 · 2019: 8 · 2018: 5 · 2017: 2 · 2016: 5 · 2015: 9 · 2014: 12 · 2013: 2 · 2012: 1
- FWCI
- 22.66
- Type
- journal article
- Language
- en
- Update notices
- erratum (2014-04-10)
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/2.0 (tdm)
- Locations
- Population Health Metrics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Funders
- Bill and Melinda Gates Foundation
- Topics
- Social Sciences › Demography › Insurance, Mortality, Demography, Risk Management
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Pathology and Forensic Medicine › Autopsy Techniques and Outcomes
- Keywords
- verbal autopsy, cause of death assignment, performance indicators, cause-specific mortality fractions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies. https://app.cochat.ai/papers/10.1186/1478-7954-9-28
- The paper
- Murray, C. J., Lozano, R., Flaxman, A. D., Vahdatpour, A., & Lopez, A. D. (2011). Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies. Population Health Metrics. https://doi.org/10.1186/1478-7954-9-28
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.