CoChat

Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies

Open access
Murray, Christopher JL, Lozano, Rafael, Flaxman, Abraham D, Vahdatpour, Alireza, Lopez, Alan D · 2011 · Population Health Metrics

The DOI matches its Crossref record; the references it cites are not checked yet.

Abstract

BACKGROUND: Verbal autopsy (VA) is an important method for obtaining cause of death information in settings without vital registration and medical certification of causes of death. An array of methods, including physician review and computer-automated methods, have been proposed and used. Choosing the best method for VA requires the appropriate metrics for assessing performance. Currently used metrics such as sensitivity, specificity, and cause-specific mortality fraction (CSMF) errors do not provide a robust basis for comparison. METHODS: We use simple simulations of populations with three causes of death to demonstrate that most metrics used in VA validation studies are extremely sensitive to the CSMF composition of the test dataset. Simulations also demonstrate that an inferior method can appear to have better performance than an alternative due strictly to the CSMF composition of the test set. RESULTS: VA methods need to be evaluated across a set of test datasets with widely varying CSMF compositions. We propose two metrics for assessing the performance of a proposed VA method. For assessing how well a method does at individual cause of death assignment, we recommend the average chance-corrected concordance across causes. This metric is insensitive to the CSMF composition of the test sets and corrects for the degree to which a method will get the cause correct due strictly to chance. For the evaluation of CSMF estimation, we propose CSMF accuracy. CSMF accuracy is defined as one minus the sum of all absolute CSMF errors across causes divided by the maximum total error. It is scaled from zero to one and can generalize a method's CSMF estimation capability regardless of the number of causes. Performance of a VA method for CSMF estimation by cause can be assessed by examining the relationship across test datasets between the estimated CSMF and the true CSMF. CONCLUSIONS: With an increasing range of VA methods available, it will be critical to objectively assess their performance in assigning cause of death. Chance-corrected concordance and CSMF accuracy assessed across a large number of test datasets with widely varying CSMF composition provide a robust strategy for this assessment.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1478-7954-9-28

Cites

Statistics

Figures

Figure 1 Kappa versus total absolute CSMF error for method 1 for 500 iterations of experiment with varying true CSMFs. This graph shows why kappa should not be used as a metric for CSMF accuracy.
Figure 1 Kappa versus total absolute CSMF error for method 1 for 500 iterations of experiment with varying true CSMFs. This graph shows why kappa should not be used as a metric for CSMF accuracy.CC BY · PubMed Central
Figure 2 Estimated CSMF versus true CSMF for causes A, B, and C using method 1 for 500 iterations of experiment with varying true CSMFs.
Figure 2 Estimated CSMF versus true CSMF for causes A, B, and C using method 1 for 500 iterations of experiment with varying true CSMFs.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists are queued

They are fetched later and appear here then.

References 47 · Cited by 96

Research works printed but not linked: 2

  1. Chandramohan D, Maude GH, Rodrigues LC, Hayes RJ: Verbal autopsies for adult deaths: issues in their development and validation. Int J Epidemiol. 1994, 23: 213-222. 10.1093/ije/23.2.213. · DOI · Paper page
  2. Rosenberg M: Logic of Survey Analysis. 1968, New York, NY: Basic Books, 9

Metadata

Authors
Murray, C. J., Lozano, R., Flaxman, A. D., Vahdatpour, A., & Lopez, A. D.
Affiliations
University of Washington (US)
Corresponding author
Murray, Christopher JL
Venue
Population Health Metrics
Published
2011-08-04
Volume
9
Issue
1
Pages
28-28
ISSN
1478-7954 (ISSN-L)
OpenAlex
W2164561875
Cited by
96
References
47
Citations by year
2026: 5 · 2025: 4 · 2024: 2 · 2023: 5 · 2022: 9 · 2021: 11 · 2020: 5 · 2019: 8 · 2018: 5 · 2017: 2 · 2016: 5 · 2015: 9 · 2014: 12 · 2013: 2 · 2012: 1
FWCI
22.66
Type
journal article
Language
en
Update notices
erratum (2014-04-10)
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/2.0 (tdm)
Locations
  • Population Health Metrics (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by)
Funders
Bill and Melinda Gates Foundation
Topics
  • Social Sciences › Demography › Insurance, Mortality, Demography, Risk Management
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Health Sciences › Medicine › Pathology and Forensic Medicine › Autopsy Techniques and Outcomes
Keywords
verbal autopsy, cause of death assignment, performance indicators, cause-specific mortality fractions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies. https://app.cochat.ai/papers/10.1186/1478-7954-9-28
The paper
Murray, C. J., Lozano, R., Flaxman, A. D., Vahdatpour, A., & Lopez, A. D. (2011). Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies. Population Health Metrics. https://doi.org/10.1186/1478-7954-9-28

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.