Discordant bioinformatic predictions of antimicrobial resistance from whole-genome sequencing data of bacterial isolates: an inter-laboratory study
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Antimicrobial resistance (AMR) poses a threat to public health. Clinical microbiology laboratories typically rely on culturing bacteria for antimicrobial-susceptibility testing (AST). As the implementation costs and technical barriers fall, whole-genome sequencing (WGS) has emerged as a ‘one-stop’ test for epidemiological and predictive AST results. Few published comparisons exist for the myriad analytical pipelines used for predicting AMR. To address this, we performed an inter-laboratory study providing sets of participating researchers with identical short-read WGS data from clinical isolates, allowing us to assess the reproducibility of the bioinformatic prediction of AMR between participants, and identify problem cases and factors that lead to discordant results. We produced ten WGS datasets of varying quality from cultured carbapenem-resistant organisms obtained from clinical samples sequenced on either an Illumina NextSeq or HiSeq instrument. Nine participating teams (‘participants’) were provided these sequence data without any other contextual information. Each participant used their choice of pipeline to determine the species, the presence of resistance-associated genes, and to predict susceptibility or resistance to amikacin, gentamicin, ciprofloxacin and cefotaxime. We found participants predicted different numbers of AMR-associated genes and different gene variants from the same clinical samples. The quality of the sequence data, choice of bioinformatic pipeline and interpretation of the results all contributed to discordance between participants. Although much of the inaccurate gene variant annotation did not affect genotypic resistance predictions, we observed low specificity when compared to phenotypic AST results, but this improved in samples with higher read depths. Had the results been used to predict AST and guide treatment, a different antibiotic would have been recommended for each isolate by at least one participant. These challenges, at the final analytical stage of using WGS to predict AMR, suggest the need for refinements when using this technology in clinical settings. Comprehensive public resistance sequence databases, full recommendations on sequence data quality and standardization in the comparisons between genotype and resistance phenotypes will all play a fundamental role in the successful implementation of AST prediction using WGS in clinical microbiology laboratories.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Identification of acquired antimicrobial resistance genes
Zankari, E., Hasman, H., Cosentino, S., Vestergaard, M., Rasmussen, S., Lund, O., Aarestrup, F. M., & Larsen, M. V. (2012) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dks261 · Cited by 4,980 · cited 1×
“All participants used one or a combination of three AMR databases in their analysis, and these were card [35], Resfinder [18] and argannot [38].”
MethodKraken: ultrafast metagenomic sequence classification using exact alignments
Wood, D. E., & Salzberg, S. L. (2014) · Genome Biology · doi:10.1186/gb-2014-15-3-r46 · Cited by 4,521 · cited 1×
“For species identification, seven participants used a combination of command line tools Kraken [28], Kraken- HLL [29], mash [30], Centrifuge [31] and Kmerid (https:// github.”
MethodMash: fast genome and metagenome distance estimation using MinHash
Ondov, B. D., Treangen, T. J., Melsted, P., Mallonee, A. B., Bergman, N. H., Koren, S., & Phillippy, A. M. (2016) · Genome biology · doi:10.1186/s13059-016-0997-x · Cited by 3,373
ARG-ANNOT, a new bioinformatic tool to discover antibiotic resistance genes in bacterial genomes
Gupta, S. K., Padmanabhan, B. R., Diene, S. M., Lopez-Rojas, R., Kempf, M., Landraud, L., & Rolain, J.-M. (2014) · Antimicrobial Agents and Chemotherapy · doi:10.1128/aac.01310-13 · Cited by 1,476
Centrifuge: rapid and sensitive classification of metagenomic sequences
Kim, D., Song, L., Breitwieser, F. P., & Salzberg, S. L. (2016) · Genome Research · doi:10.1101/gr.210641.116 · Cited by 1,438 · cited 1×
“For species identification, seven participants used a combination of command line tools Kraken [28], Kraken- HLL [29], mash [30], Centrifuge [31] and Kmerid (https:// github.”
MethodEllington, M. J., Ekelund, O., Aarestrup, F. M., Canton, R., Doumith, M., Giske, C., Grundman, H., Hasman, H., Holden, M. T. G., Hopkins, K. L., et al. (2017) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2016.11.012 · Cited by 603
Self-citationPrediction of Susceptibility to First-Line Tuberculosis Drugs by DNA Sequencing
The CRyPTIC Consortium and the 100, 0. G. P. (2018) · New England Journal of Medicine · doi:10.1056/nejmoa1800474 · Cited by 500 · cited 1×
“Studies have reported good concordance between genotype and phenotype on datasets they have been applied to [9, 22, 25], but rarely address the factors underlying situations where different methods may produce discordant results and how this discordance should be resolved.”
Background
Cited by
ResFinder 4.0 for predictions of phenotypes from genotypes
Bortolaia, V., Kaas, R. S., Ruppe, E., Roberts, M. C., Schwarz, S., Cattoir, V., Philippon, A., Allesoe, R. L., Rebelo, A. R., Florensa, A. F., et al. (2020) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkaa345 · Cited by 3,373
Kim, J. I., Maguire, F., Tsang, K. K., Gouliouris, T., Peacock, S. J., McAllister, T. A., McArthur, A. G., & Beiko, R. G. (2022) · Clinical Microbiology Reviews · doi:10.1128/cmr.00179-21 · Cited by 256
Good microbes, bad genes? The dissemination of antimicrobial resistance in the human microbiome
Crits-Christoph, A., Hallowell, H. A., Koutouvalis, K., & Suez, J. (2022) · Gut Microbes · doi:10.1080/19490976.2022.2055944 · Cited by 156
Metadata
- Authors
- Doyle, R. M., O'Sullivan, D. M., Aller, S. D., Bruchmann, S., Clark, T., Coello Pelegrin, A., Cormican, M., Diez Benavente, E., Ellington, M. J., McGrath, E., Motro, Y., Phuong Thuy Nguyen, T., Phelan, J., Shaw, L. P., Stabler, R. A., van Belkum, A., van Dorp, L., Woodford, N., Moran-Gilad, J., Huggett, J. F., & Harris, K. A.
- Venue
- Microbial Genomics
- Publisher
- Microbiology Society
- Published
- 2020-02-01
- Volume
- 6
- Issue
- 2
- ISSN
- 2057-5858 (ISSN-L)
- OpenAlex
- W3005670610
- Cited by
- 120
- References
- 7
- FWCI
- 5.60
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Discordant bioinformatic predictions of antimicrobial resistance from whole-genome sequencing data of bacterial isolates: an inter-laboratory study. https://app.cochat.ai/papers/10.1099/mgen.0.000335
- The paper
- Doyle, R. M., O'Sullivan, D. M., Aller, S. D., Bruchmann, S., Clark, T., Coello Pelegrin, A., Cormican, M., Diez Benavente, E., Ellington, M. J., McGrath, E., Motro, Y., Phuong Thuy Nguyen, T., Phelan, J., Shaw, L. P., Stabler, R. A., van Belkum, A., van Dorp, L., Woodford, N., Moran-Gilad, J., . . . Harris, K. A. (2020). Discordant bioinformatic predictions of antimicrobial resistance from whole-genome sequencing data of bacterial isolates: an inter-laboratory study. Microbial Genomics. https://doi.org/10.1099/mgen.0.000335
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.