Cited by 1,644Single-cell and spatial transcriptomicsGenomics and Phylogenetic StudiesGene expression and cancer classification

SoupX removes ambient RNA contamination from droplet-based single-cell RNA sequencing data

Open access
Young, Matthew D, Behjati, Sam·2020·GigaScience

Published version of a preprint

Abstract

Background: Droplet-based single-cell RNA sequence analyses assume that all acquired RNAs are endogenous to cells. However, any cell-free RNAs contained within the input solution are also captured by these assays. This sequencing of cell-free RNA constitutes a background contamination that confounds the biological interpretation of single-cell transcriptomic data. Results: We demonstrate that contamination from this "soup" of cell-free RNAs is ubiquitous, with experiment-specific variations in composition and magnitude. We present a method, SoupX, for quantifying the extent of the contamination and estimating "background-corrected" cell expression profiles that seamlessly integrate with existing downstream analysis tools. Applying this method to several datasets using multiple droplet sequencing technologies, we demonstrate that its application improves biological interpretation of otherwise misleading data, as well as improving quality control metrics. Conclusions: We present SoupX, a tool for removing ambient RNA contamination from droplet-based single-cell RNA sequencing experiments. This tool has broad applicability, and its application can improve the biological utility of existing and future datasets.

AI snapshot

Written from the paper’s full text.

TL;DR

SoupX is a method and R package that estimates and removes ambient cell-free RNA contamination from droplet-based scRNA-seq by using empty droplets to define a background profile and estimating a contamination fraction. Across species-mixing, PBMC, kidney tumour, and fetal liver datasets, correction improved marker specificity and biological interpretation.

Key findings

  • Ambient RNA contamination was ubiquitous: in 10X species-mixing data, ∼1% of transcripts were cross-species contamination, providing a lower bound, with true contamination later estimated at ∼2%; background profiles correlated with cell aggregates at Pearson 0.71–0.96 (median 0.86).
  • SoupX reduced cross-species contamination by a factor of ≥2 and usually an order of magnitude, while correct-species expression was effectively unchanged.
  • In PBMCs, automated contamination estimation gave 6% and an IG-gene-based estimate gave ∼5%; correction increased marker fold-change contrast and improved LYZ specificity for mononuclear phagocytes.
  • In kidney tumour data, correction removed haemoglobin expression from non-red-blood cells, identified collagen expression in T/MNP clusters as contamination, increased cross-batch entropy, and yielded 2 fewer clusters; in fetal liver, HBB was confirmed as erythroid-specific and GYPA expression in EI macrophages as genuine.

Methods

  • SoupX has three steps: estimate the ambient mRNA profile from empty droplets with <Nemp UMIs (best correlation at Nemp < 10), estimate or manually set the contamination fraction using genes/cells assumed to have no endogenous expression or an automated marker-based procedure, and correct counts via a multinomial likelihood, optionally redistributing cluster-level corrected counts.
  • Datasets included 10X and DropSeq species-mixing experiments, PBMC data, a kidney tumour dataset (7 tumours across 10 channels in the Results; the Introduction describes 12 kidney tumour biopsies), and 40 channels of human fetal liver data.
  • Analyses compared corrected and uncorrected data using Seurat tSNE representations, cluster membership, marker gene fold changes, marker sensitivity, and cross-batch entropy.

Limitations

  • The paper states that best results require a user-specified set of genes and cells where no endogenous expression can be assumed; when this is not possible, the automated method requires clustering and may not always have an appropriate gene set.
  • The paper notes that a constant contamination fraction within a channel is assumed in most experiments, and that manual contamination setting can be used if estimates are deemed inaccurate or overcorrection is desired.
  • Evident from the text: the method assumes empty-droplet profiles represent the background and that background composition is shared across cells; evaluation is limited to the datasets analysed, and the paper has no formal limitations section.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/gigascience/giaa151

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

15 of 16 cited sources checked · 10 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Violations of the single-cell-per-droplet assumption are common in practice.

    “Violations of this assumption, which may distort the interpretation of scRNA-seq data, are common in practice.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [11] Gayoso, 2017, DoubletDetection: Identifying technical error in single-cell RNA-sequencing data

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [12] Ilicic, 2016, Classification of low quality cells from single-cell RNA-seq data, Genome Biol source

      The Background (page unavailable) says some scRNA-seq capture sites, including droplets, may contain multiple cells and that these cells can lead to misinterpretation. The Discussion (page unavailable) reports up to 30% multiple-cell captures in a Fluidigm study, but that figure concerns microfluidic capture sites, not droplets; the paper does not establish that single-cell-per-droplet violations are common. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [13] Wolock, 2019, Scrublet: Computational identification of cell doublets in single-cell transcriptomic data, Cell Syst source
  • Premisecitation

    Cell-free RNAs contained within the input solution are captured by droplet-based scRNA-seq assays.

    “However, any cell-free RNAs contained within the input solution are also captured by these assays.”
    • Also stated in: § Introduction · boosted
      “It is recognized that these contaminating non-endogenous RNAs are present even within datasets of the highest quality [2].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [2] Zheng, 2017, Massively parallel digital transcriptional profiling of single cells, Nat Commun source
  • Findingcitation

    A similar cross-species contamination effect is seen in Drop-Seq-based species-mixing data.

    “A similar effect is seen in the Drop-Seq–based species-mixing data (Supplementary Fig. S3).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [14] Macosko, 2015, Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets, Cell source
  • Findingcitation

    In 10X species-mixing data, droplets containing human or mouse cells show that approximately 1% of observed transcripts are cross-species contamination.

    “Droplets containing human (top right) and mouse (bottom right) cells show that ∼1% of observed transcripts are cross-species contamination.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [2] Zheng, 2017, Massively parallel digital transcriptional profiling of single cells, Nat Commun source
  • Findingown data

    The batch-mixing entropy of the kidney tumour data is increased after background correction.

    “This analysis shows that the batch-mixing entropy is increased after background correction, indicating better mixing between samples (Fig. 4E).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [20] Haghverdi, 2018, Batch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors, Nat Biotechnol source

      The Results section on the pancreas data set reports higher batch-mixing entropy after MNN batch correction than in uncorrected data. The Online Methods section also specifies that entropy was assessed in pancreas data sets. Neither passage supports the claim that entropy increased in kidney tumour data after background correction; no page numbers are available in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • May combine two claims. 11
  • Classified as probably not a substantial claim. 7
  • May combine two claims.

    Study design and conductcitation

    The species-mixing datasets allow direct identification of contaminating mRNAs and testing of SoupX's accuracy.

    “We first investigate 2 “species mixing” datasets run on the Chromium 10X [2] and DropSeq [14] platforms, which allow us to directly identify contaminating mRNAs and test our method’s accuracy.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [2] Zheng, 2017, Massively parallel digital transcriptional profiling of single cells, Nat Commun source
    • cites [14] Macosko, 2015, Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets, Cell source
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    DecontX relies on accurate clustering of the data to estimate and remove the background without the need for gene counts from empty droplets.

    “Finally, DecontX [23] relies on accurate clustering of the data to estimate and remove the background without the need for gene counts from empty droplets.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [23] Yang, 2003, Decontamination of ambient RNA in single-cell RNA-seq with DecontX, Genome Biol source
  • May combine two claims.

    Premisecitation

    Solid-tissue experiments are frequently highly contaminated with red blood cells and red cell lysis is used to prepare the samples.

    “For example, solid-tissue experiments are frequently highly contaminated with red blood cells and red cell lysis is used to prepare the samples [15].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [15] Young, 2018, Single-cell transcriptomes from human kidneys reveal the cellular identity of renal tumors, Science source

      The paper’s tissue-processing description (Abstract; page not provided) says kidney samples were made into viable single-cell solutions, but does not mention red-blood-cell contamination or red-cell lysis. Searches for red blood cells, erythrocyte lysis, and lysis buffer found no passage addressing either part of the claim. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
  • May combine two claims.

    Interpretationcitation

    CellBender's deep generative model comes with a heavy computational cost compared to other tools, and its output provides an imputed cell profile rather than raw counts with the background subtracted off.

    “This deep generative model comes with a heavy computational cost compared to other tools, and the output of the model (which in effect estimates mg, c for each cell type) provides an imputed cell profile rather than raw counts with the background “subtracted off,” which SoupX provides.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [22] Fleming, 2019, CellBender remove-background: a deep generative model for unsupervised removal of background noise from scRNA-seq datasets, bioRxiv source

      Section 3.3.2 describes the output as a model-inferred, background-free expression-rate estimate that is subsequently quantized, supporting the distinction from simply subtracting background counts from raw counts. Section 3.2.1 reports runtimes of 30 minutes to several hours on a GPU, but does not compare computational cost with other tools; that comparative claim is unsupported. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    CellBender uses a deep generative model to estimate shared expression patterns likely to represent distinct cell types while simultaneously removing contamination.

    “Cell Bender [22] uses a deep generative model to estimate shared expression patterns likely to represent distinct cell types while simultaneously removing contamination.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [22] Fleming, 2019, CellBender remove-background: a deep generative model for unsupervised removal of background noise from scRNA-seq datasets, bioRxiv source
  • May combine two claims.

    Premisecitation

    SoupOrCell uses the identification of conflicting genotypes to identify ambient RNA contamination, limiting its application to mixed-genotype experiments.

    “SoupOrCell [21] uses the identification of conflicting genotypes to identify ambient RNA contamination, limiting its application to mixed-genotype experiments.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [21] Heaton, 2020, souporcell: Robust clustering of single-cell RNA-seq data by genotype without reference genotypes, Nat Methods source
  • May combine two claims.

    Interpretationreasoning

    DecontX can be applied when empty droplet counts are not available but its results are potentially heavily dependent on the accuracy of the clustering provided.

    “This allows DecontX to be applied when empty droplet counts are not available but also means that the results are potentially heavily dependent on the accuracy of the clustering provided.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [23] Yang, 2003, Decontamination of ambient RNA in single-cell RNA-seq with DecontX, Genome Biol source
  • May combine two claims.

    Findingown data

    In the species-mixing experiment, any value of Nemp < 100 produces a good correlation between bg and true background expression, with the best correlation when Nemp < 10.

    “The species-mixing experiment allows us to compare how accurately bg recapitulates the true background expression found within each cell, revealing that any value of Nemp < 100 produces a good correlation, with the best correlation given when Nemp < 10 (Supplementary Fig. S2).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    In kidney tumour data, applying SoupX and re-analysing with the same clustering parameters resulted in many cells changing cluster and 2 fewer clusters identified compared to uncorrected data.

    “Applying SoupX and re-analysing the kidney tumour data revealed that, in contrast to the PBMC data, many cells changed cluster and with the same clustering parameters 2 fewer clusters were identified in the corrected data (Fig. 4B).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The aggregate expression profile of all droplets containing cells and all droplets with ≤10 UMIs were highly correlated in the 10X species experiment, and high correlation was found in all other datasets considered (Pearson correlation 0.71-0.96, median 0.86).

    “These 2 profiles were highly correlated in the 10X species experiment (Fig. 2B) with a high correlation found in all other datasets considered (Pearson correlation 0.71–0.96, median 0.86; Supplementary Table S2).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Droplets with ≤10 UMIs contain only ambient mRNAs.
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Running SoupX does not add appreciably to the computational cost of standard single-cell analyses.

    “The output of the SoupX package is a corrected table of counts, which can be used as input for standard workflows, and running SoupX does not add appreciably to the computational cost of standard single-cell analyses.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Comparing cluster membership before and after correction revealed the same number of clusters was identified, but some cells changed which cluster they belonged to.

    “Comparing cluster membership before and after correction revealed that the same number of clusters was identified, but some cells changed which cluster they belonged to (Fig. 3C).”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Genes for which mg,c = 0 for a given cell are strong negative markers of the cell type c.

    “The genes for which mg, c = 0 for a given cell are those genes that are strong negative markers of the cell type c.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationnone given

    SoupX is a method for quantifying the extent of ambient mRNA contamination and purifying the true, cell-specific signal from the observed mixture of cellular and exogenous mRNAs.

    “We present SoupX, a method for quantifying the extent of ambient mRNA contamination whilst purifying the true, cell-specific signal from the observed mixture of cellular and exogenous mRNAs.”
    • Also stated in: § Results · plain
      “We present a method, SoupX, for quantifying the extent of the contamination and estimating "background-corrected" cell expression profiles that seamlessly integrate with existing downstream analysis tools.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    SoupX produces a modified table of counts that can be used in place of the original count matrix in any downstream analysis tool.

    “SoupX produces a modified table of counts, which can be used in place of the original count matrix in any downstream analysis tool.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Selecting an appropriate set of genes to estimate contamination will not always be possible.

    “However, we recognize that selecting an appropriate set of genes to estimate contamination will not always be possible.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    SoupX has broad applicability.

    “This tool has broad applicability”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    SoupX can be applied generally, is computationally inexpensive, and does not depend heavily on accurate pre-annotation of input data.

    “By contrast, SoupX can be applied generally, is computationally inexpensive, and does not depend heavily on accurate pre-annotation of input data.”
    Quote in the paperSubstantial claim
18 of 18
  • The paper relies on this without support.

    Ambient mRNA contamination is increased in necrotic or complex samples.

    “with increased contamination in necrotic or complex samples”
  • The authors state this as an assumption.

    Droplets with fewer than Nemp UMIs unambiguously do not contain cells.

    “To estimate the background expression profile we consider all droplets with <Nemp UMIs, which we assume unambiguously do not contain cells.”
  • The authors state this as an assumption.

    The relative abundance of genes that make up the background does not differ between cells.

    “We assume that the relative abundance of genes that make up the background does not differ between cells”
  • The authors state this as an assumption.

    There is a combination of genes and cells for which mg,c = 0 exists.

    “To proceed, we assume that there is a combination of genes and cells for which mg, c = 0 exists.”
  • The authors state this as an assumption.

    If it can be shown for any cell c in cluster P that mg,c > 0, then mg,c > 0 for all c ∈ P.

    “If it can be shown for any cell c in cluster P that mg, c > 0, then it is assumed that mg, c > 0 for all c ∈ P (see Supplementary Fig. S1).”
  • The authors state this as an assumption.

    For each strong marker, mg,c = 0 for all cells in clusters where the gene is not a marker.

    “For each strong marker, it is assumed that mg, c = 0 for all cells in clusters where the gene is not a marker and the contamination fraction is estimated (Supplementary Fig. S1).”
  • The authors state this as an assumption.

    Inaccurate estimates of the contamination fraction will have no preferred value while true estimates will cluster around the true value.

    “To obtain a final value, it is assumed that inaccurate estimates will have no preferred value while true estimates will cluster around the true value.”
  • The authors state this as an assumption.

    Droplets with ≤10 UMIs contain only ambient mRNAs.

    “which we assumed to contain only ambient mRNAs”
  • The authors state this as an assumption.

    For species-mixing genes/cells (human transcripts in mouse cells and vice versa), mg,c = 0.

    “For these genes/cells it is assumed that mg, c = 0 and the contamination fraction is calculated using Equation 4.”
  • The authors state this as an assumption.

    SoupX assumes a constant contamination fraction within a channel.

    “SoupX assumes a constant contamination fraction within a channel.”
  • The paper relies on this without support.

    In most experiments there is less power to determine cell-specific contamination fractions.

    “In most experiments there is less power to determine cell-specific contamination fractions and so SoupX assumes a constant contamination fraction within a channel.”
  • The authors state this as an assumption.

    Some genes can be assumed to be unexpressed in some cells (mg,c = 0).

    “To confirm the accuracy of this estimate, we also calculated the background contamination rate using a set of genes that could be assumed to be unexpressed in some cells (i.e., where mg, c = 0).”
  • The paper relies on this without support.

    Haemoglobin genes are highly specific to red blood cells.

    “together with the knowledge that haemoglobin genes are highly specific to red blood cells.”
  • The paper relies on this without support.

    The ambient mRNA expression profile is experiment specific.

    “Because the ambient mRNA expression profile is experiment specific, we reasoned that background contamination likely creates batch effects.”
  • The paper relies on this without support.

    Red blood cells are the only cells that produce haemoglobin under normal physiological conditions.

    “Furthermore, red blood cells are the only cells that produce haemoglobin under normal physiological conditions, so for the set of haemoglobin genes, it is safe to assume that there is no cell endogenous expression for cells that are not red blood cells.”
  • The authors state this as an assumption.

    For haemoglobin genes, there is no cell endogenous expression for cells that are not red blood cells.

    “Furthermore, red blood cells are the only cells that produce haemoglobin under normal physiological conditions, so for the set of haemoglobin genes, it is safe to assume that there is no cell endogenous expression for cells that are not red blood cells.”
  • The authors state this as an assumption.

    Inaccurate estimates of the contamination fraction are not strongly correlated (i.e., there is no preferred, incorrect estimate).

    “By using all high-quality marker genes identified in the data to independently estimate the contamination fraction, this method estimates the true contamination fraction by assuming that inaccurate estimates of the contamination fraction are not strongly correlated (i.e., there is no preferred, incorrect estimate).”
  • The authors state this as an assumption.

    For a chosen set of genes and cells, the only source of expression for these genes is background contamination.

    “The best case for applying SoupX occurs when the user can specify a set of genes and cells where there is no cell endogenous expression, i.e., a set of genes and cells where it is safe to assume that the only source of expression for these genes is from background contamination.”
18 of 18

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Ambient RNA contamination may distort transcriptomic profiles, leading to the underrepresentation of specific cell types or misclassification of cellular states.

    “This contamination may distort transcriptomic profiles, leading to the underrepresentation of specific cell types or misclassification of cellular states[34, 37].”

    From Comparative analysis of nuclei isolation methods for brain single-nucleus RNA sequencing (2026) · cites this paper as [37]

    The Introduction (page not available) says ambient RNA can confound biological interpretation. In “Ambient RNA Confounds Interpretation in Complex Experiments” (page not available), correction changed many cells’ cluster assignments, and the authors describe how contaminating expression could lead to false conclusions about cells. These passages support distorted profiles and possible misclassification, but do not establish underrepresentation of specific cell types. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • Ambient RNA contamination was removed using SoupX v1.6.2 with default parameters, except for Miedema samples where tfidfMin was adjusted to 0.8.

    “Ambient RNA contamination was removed using the SoupX v1.6.2 package67 with default parameters, except for samples from Miedema et al.14 where tfidfMin was adjusted to 0.8 (default 1) due to the lower cellular complexity of the dataset (prior FACS).”
  • Accurate chromatin accessibility analysis depends on precise peak calling, because calling peaks across the entire cell population can inadvertently exclude peaks specific to smaller subpopulations.

    “Accurate chromatin accessibility analysis depends on precise peak calling, as calling peaks across the entire cell population can inadvertently exclude those specific to smaller subpopulations (85).”

    From Molecular and cellular processes disrupted in the early postnatal Down syndrome prefrontal cortex (2026) · cites this paper as [85]

    The SoupX Method and Discussion describe ambient RNA contamination in single-cell RNA sequencing and the use of gene markers and clustering to estimate it. Searches for chromatin accessibility, peak calling, and subpopulation-specific peaks found no passage addressing the claimed risk of calling peaks across an entire cell population. (LLM, conf=0.76)

    Experimental — accuracy not yet measured
  • Ambient RNA contamination can cause cell-type-specific marker gene transcripts to be detectable in other cell populations, blending populations together.

    “Ambient RNA contamination can lead to cell-type-specific marker gene transcripts being detectable also in other cell populations, which can blend different cell populations together21.”

    From Best practices for single-cell analysis across modalities (2023) · cites this paper as [21]

  • Removing ambient RNA as an initial quality-control step improves downstream analyses for many tissues.

    “one should consider removing ambient RNA as an initial analysis step in quality control to improve downstream analyses for many tissues21–23.”

    From Best practices for single-cell analysis across modalities (2023) · cites this paper as [21]

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Gayoso, 2017, DoubletDetection: Identifying technical error in single-cell RNA-sequencing data

Cited by

Sorted by citations

Authors
Young, M. D., & Behjati, S.
Affiliations
  • Wellcome Sanger Institute (GB)
  • University of Cambridge (GB)
Corresponding author
Young, Matthew D
Venue
GigaScience
Published
2020-12
Volume
9
Issue
12
ISSN
2047-217X (ISSN-L)
OpenAlex
W3114078084
Cited by
1,619
References
30
Citations by year
2026: 358 · 2025: 388 · 2024: 347 · 2023: 211 · 2022: 149 · 2021: 86 · 2020: 54 · 2019: 24 · 2018: 1
FWCI
35.00
Type
journal article
Language
en
Open access
Locations
  • GigaScience (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
Funders
Wellcome Trust
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
Keywords
cell-free RNA, background correction, single-cell transcriptomics, quality control, contamination quantification
MeSH terms
Base Sequence, RNA, Sequence Analysis, RNA, Gene Expression Profiling, Single-Cell Analysis, Transcriptome

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-14.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-14.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review