CoChat
Sign Up
PreprintCited by 1,210Single-cell and spatial transcriptomicsGene expression and cancer classificationBioinformatics and Genomic Networks

The DOI matches its Crossref record; 105 of the 105 references it cites were found in a registry or an index; 54 of the 64 sources its claims cite were checked for support.

Confronting false discoveries in single-cell differential expression

Open access
Squair, Jordan W., Gautier, Matthieu, Kathe, Claudia, Anderson, Mark A., James, Nicholas D., Hutson, Thomas H., Hudelle, Rémi, Qaiser, Taha, Matson, Kaya J. E., Barraud, Quentin, Levine, Ariel J., La Manno, Gioele, Skinnider, Michael A., Courtine, Grégoire·2021·Nature Communications
DOI 10.1101/2021.03.12.435024PMC PMC8479118
Open this paper in CoChat — free accountOpen PDF

Abstract

Differential expression analysis in single-cell transcriptomics enables the dissection of cell-type-specific responses to perturbations such as disease, trauma, or experimental manipulation. While many statistical methods are available to identify differentially expressed genes, the principles that distinguish these methods and their performance remain unclear. Here, we show that the relative performance of these methods is contingent on their ability to account for variation between biological replicates. Methods that ignore this inevitable variation are biased and prone to false discoveries. Indeed, the most widely used methods can discover hundreds of differentially expressed genes in the absence of biological differences. Our results suggest an urgent need for a paradigm shift in the methods used to perform differential expression analysis in single-cell data.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper benchmarks 14 differential expression methods on 18 ground-truth datasets and finds that pseudobulk methods, which aggregate cells within biological replicates, outperform single-cell methods. Methods that ignore between-replicate variation are biased toward highly expressed genes and can produce hundreds of false discoveries even without biological differences.

Key findings

  • In 18 ground-truth datasets, all six top-performing DE methods aggregated cells within biological replicates to form pseudobulks; pseudobulk versus single-cell differences were highly significant and robust to concordance methodology.
  • Single-cell DE methods were biased toward highly expressed genes: false positives were more highly expressed, abundant ERCC spike-ins were incorrectly called DE, and a compendium of 46 scRNA-seq datasets showed a systematic preference for highly expressed genes.
  • Failing to account for biological replicates produced false discoveries: simulations identified hundreds of DE genes with no perturbation; random splits of unstimulated controls in 14 datasets produced excess false positives; spatial transcriptomics identified thousands of DE genes within each spinal cord region.
  • In injured mouse spinal cord snRNA-seq (19,237 nuclei), RNAscope validated 5 of 6 genes called by edgeR-LRT versus 3 of 13 called by Wilcoxon rank-sum (p < 0.05, χ² test); GLMMs converged to pseudobulk performance in datasets of 500 or more cells and took an average of 13.5 h per cell type versus minutes for pseudobulk methods.

Methods

  • Literature review of 500 recent scRNA-seq studies, of which 205 conducted DE analysis between biological conditions, identified 14 methods: 7 single-cell methods, 6 pseudobulk methods, and 1 linear mixed model.
  • The study used 18 ground-truth datasets from four publications with matched bulk and scRNA-seq in the same purified cells, perturbations, and laboratories; concordance was measured by area under the concordance curve (AUCC) and rank correlation, with GO term enrichment, a 46-dataset scRNA-seq compendium, Splatter simulations, 14 control-group datasets, spatial transcriptomics, and spinal cord injury snRNA-seq plus RNAscope.
  • Additional analyses disabled pseudobulk aggregation, created pseudo-replicates by shuffling replicates, compared variance between biological replicates and pseudo-replicates, and evaluated Poisson or negative binomial GLMMs.

Limitations

  • The paper does not include a dedicated limitations section. It states that the enormous computational requirements of GLMMs prevented evaluation in the full ground-truth datasets, so downsampled datasets of 25–1,000 cells were used; the ground-truth resource was limited to 18 datasets from four publications.
  • Evident from the text: simulations used parameters estimated from the Kang et al. dataset and may not capture all biological or technical heterogeneity; the in vivo RNAscope validation screened selected putatively DE genes rather than all DE genes; and human-versus-mouse variability conclusions rely on the assembled compendium datasets.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1101/2021.03.12.435024

Cites

105 of 105 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 A systematic benchmark of differential expression in single-cell transcriptomics. a Schematic overview of the eighteen ground-truth datasets analyzed in this study. b Statistical methods for DE analysis employed in 500 recent scRNA-seq papers. Grey bars represent DE analysis methods included in this study. “Other” includes methods used in two or fewer studies. Inset pie chart shows the total proportion of recent scRNA-seq papers that employed DE analysis methods included in this study. Source data are provided as a Source Data file. c Area under the concordance curve (AUCC) for fourteen DE methods in the eighteen ground-truth datasets shown in a. d Mean difference in the AUCC (∆ AUCC) between the fourteen DE methods shown in c. Asterisks indicate comparisons with a two-tailed t-test p-value less than 0.05. e AUCC of GO term enrichment, as evaluated using gene set enrichment analysis46, in the eighteen ground-truth datasets shown in a. f Rank and statistical significance of the GO term GO:0043330 (“response to exogenous dsRNA”) in GSEA analyses of mouse bone marrow mononuclear cells stimulated with poly-I:C, a type of synthetic dsRNA, for four h, using the output of fourteen DE methods. Source data are provided as a Source Data file.
Fig. 1 A systematic benchmark of differential expression in single-cell transcriptomics. a Schematic overview of the eighteen ground-truth datasets analyzed in this study. b Statistical methods for DE analysis employed in 500 recent scRNA-seq papers. Grey bars represent DE analysis methods included in this study. “Other” includes methods used in two or fewer studies. Inset pie chart shows the total proportion of recent scRNA-seq papers that employed DE analysis methods included in this study. Source data are provided as a Source Data file. c Area under the concordance curve (AUCC) for fourteen DE methods in the eighteen ground-truth datasets shown in a. d Mean difference in the AUCC (∆ AUCC) between the fourteen DE methods shown in c. Asterisks indicate comparisons with a two-tailed t-test p-value less than 0.05. e AUCC of GO term enrichment, as evaluated using gene set enrichment analysis46, in the eighteen ground-truth datasets shown in a. f Rank and statistical significance of the GO term GO:0043330 (“response to exogenous dsRNA”) in GSEA analyses of mouse bone marrow mononuclear cells stimulated with poly-I:C, a type of synthetic dsRNA, for four h, using the output of fourteen DE methods. Source data are provided as a Source Data file.CC BY · PubMed Central
Fig. 2 Single-cell DE methods are biased towards highly expressed genes. a Schematic illustration of the creation of ‘pseudobulks’ from single-cell data. Top, biological replicate from which each cell was obtained. Bottom, simulated gene expression matrix. Read counts for each gene are aggregated across cells of a given type within each biological replicate. b Mean AUCCs across eighteen ground-truth datasets after dividing the transcriptome into terciles of lowly, moderately, or highly expressed genes. c Mean expression levels of the 100 top-ranked false-positive genes from each DE method. d Spearman correlation between the mean expression of 80 ERCC spike-ins expressed in at least three cells and the –log10 p-value of differential expression assigned by each DE method. e Scatterplots of mean ERCC expression vs. –log10 p-value for exemplary single-cell and pseudobulk DE methods. Trend lines and shaded areas show local polynomial regression and the 95% confidence interval, respectively. f Mean expression levels of the 200 top-ranked genes from each DE method in a collection of 46 scRNA-seq datasets.
Fig. 2 Single-cell DE methods are biased towards highly expressed genes. a Schematic illustration of the creation of ‘pseudobulks’ from single-cell data. Top, biological replicate from which each cell was obtained. Bottom, simulated gene expression matrix. Read counts for each gene are aggregated across cells of a given type within each biological replicate. b Mean AUCCs across eighteen ground-truth datasets after dividing the transcriptome into terciles of lowly, moderately, or highly expressed genes. c Mean expression levels of the 100 top-ranked false-positive genes from each DE method. d Spearman correlation between the mean expression of 80 ERCC spike-ins expressed in at least three cells and the –log10 p-value of differential expression assigned by each DE method. e Scatterplots of mean ERCC expression vs. –log10 p-value for exemplary single-cell and pseudobulk DE methods. Trend lines and shaded areas show local polynomial regression and the 95% confidence interval, respectively. f Mean expression levels of the 200 top-ranked genes from each DE method in a collection of 46 scRNA-seq datasets.CC BY · PubMed Central
Fig. 3 DE analysis of single-cell data must account for biological replicates. a Schematic illustration of the experiment shown in b, in which the aggregation procedure was disabled and pseudobulk DE methods were applied to individual cells. b Left, AUCC of the original fourteen DE methods, plus six pseudobulk methods applied to individual cells, in the eighteen ground-truth datasets. Right, Spearman correlation between ERCC mean expression and –log10 p-value assigned by six pseudobulk DE methods, before and after disabling the aggregation procedure. c Schematic illustration of the experiment shown in d, in which the replicate associated with each cell was shuffled to produce ‘pseudo-replicates.’ d Left, AUCC of the original fourteen DE methods, plus six pseudobulk methods applied to pseudo-replicates, in the eighteen ground-truth datasets. Right, Spearman correlation between ERCC mean expression and –log10 p-value assigned by six pseudobulk DE methods, before and after shuffling replicates to produce pseudo-replicates. e Variance of gene expression in pseudobulks formed from biological replicates and pseudo-replicates in mouse bone marrow mononuclear cells stimulated with poly-I:C. Shuffling the replicate associated with each cell produced a systematic decrease in the variance of gene expression. Right, pie chart shows the proportion of genes with increased or decreased variance in pseudo-replicates, as compared to biological replicates. f Decreases in the variance of gene expression in pseudo-replicates as compared to biological replicates across 46 scRNA-seq datasets. Points show the mean variance in biological replicates; arrowheads show the mean variance in pseudo-replicates. g Left, expression of the gene c in biological replicates (points) and pseudo-replicates (arrowheads) from unstimulated cells and cells stimulated with poly-I:C, with the range of possible pseudo-replicate expression values shown as a density. Right, mean (horizontal line) and variance (shaded area) of Txnrd3 expression in biological replicates (left) and pseudo-replicates (right). P-values were calculated by edgeR-LRT.
Fig. 3 DE analysis of single-cell data must account for biological replicates. a Schematic illustration of the experiment shown in b, in which the aggregation procedure was disabled and pseudobulk DE methods were applied to individual cells. b Left, AUCC of the original fourteen DE methods, plus six pseudobulk methods applied to individual cells, in the eighteen ground-truth datasets. Right, Spearman correlation between ERCC mean expression and –log10 p-value assigned by six pseudobulk DE methods, before and after disabling the aggregation procedure. c Schematic illustration of the experiment shown in d, in which the replicate associated with each cell was shuffled to produce ‘pseudo-replicates.’ d Left, AUCC of the original fourteen DE methods, plus six pseudobulk methods applied to pseudo-replicates, in the eighteen ground-truth datasets. Right, Spearman correlation between ERCC mean expression and –log10 p-value assigned by six pseudobulk DE methods, before and after shuffling replicates to produce pseudo-replicates. e Variance of gene expression in pseudobulks formed from biological replicates and pseudo-replicates in mouse bone marrow mononuclear cells stimulated with poly-I:C. Shuffling the replicate associated with each cell produced a systematic decrease in the variance of gene expression. Right, pie chart shows the proportion of genes with increased or decreased variance in pseudo-replicates, as compared to biological replicates. f Decreases in the variance of gene expression in pseudo-replicates as compared to biological replicates across 46 scRNA-seq datasets. Points show the mean variance in biological replicates; arrowheads show the mean variance in pseudo-replicates. g Left, expression of the gene c in biological replicates (points) and pseudo-replicates (arrowheads) from unstimulated cells and cells stimulated with poly-I:C, with the range of possible pseudo-replicate expression values shown as a density. Right, mean (horizontal line) and variance (shaded area) of Txnrd3 expression in biological replicates (left) and pseudo-replicates (right). P-values were calculated by edgeR-LRT.CC BY · PubMed Central
Fig. 4 False discoveries in single-cell DE. a Schematic illustration of simulation experiments. Single-cell RNA-seq datasets were simulated with varying degrees of heterogeneity between replicates. Replicates were then randomly assigned to either a ‘treatment’ or ‘control’ group, and DE analysis was performed between groups. b Number of DE genes detected in stimulation experiments with varying degrees of heterogeneity between replicates by a representative single-cell DE method, a representative pseudobulk method, and the same pseudobulk method applied to pseudo-replicates. Points and error bars show the mean and standard deviation across ten independent simulations. c Number of DE genes detected by the tests shown in b for genes divided into deciles by the magnitude of the change in variance between biological replicates and pseudo-replicates (∆-variance). d Volcano plots showing DE between T cells from random groups of unstimulated controls drawn from Kang et al.5 using a representative pseudobulk method, edgeR-LRT, applied to biological replicates or pseudo-replicates. Discarding information about biological replicates leads to the appearance of false discoveries. e Number of DE genes detected in comparisons of random groups of unstimulated controls from 14 scRNA-seq studies with at least six control samples. f Number of DE genes detected within spinal cord regions from control mice profiled by spatial transcriptomics24 using a representative pseudobulk method, edgeR-LRT (points), or a representative single-cell method, the Wilcoxon rank-sum test (arrowheads). g Mean change in variance between biological replicates and pseudo-replicates for 18 human and 20 mouse scRNA-seq datasets.
Fig. 4 False discoveries in single-cell DE. a Schematic illustration of simulation experiments. Single-cell RNA-seq datasets were simulated with varying degrees of heterogeneity between replicates. Replicates were then randomly assigned to either a ‘treatment’ or ‘control’ group, and DE analysis was performed between groups. b Number of DE genes detected in stimulation experiments with varying degrees of heterogeneity between replicates by a representative single-cell DE method, a representative pseudobulk method, and the same pseudobulk method applied to pseudo-replicates. Points and error bars show the mean and standard deviation across ten independent simulations. c Number of DE genes detected by the tests shown in b for genes divided into deciles by the magnitude of the change in variance between biological replicates and pseudo-replicates (∆-variance). d Volcano plots showing DE between T cells from random groups of unstimulated controls drawn from Kang et al.5 using a representative pseudobulk method, edgeR-LRT, applied to biological replicates or pseudo-replicates. Discarding information about biological replicates leads to the appearance of false discoveries. e Number of DE genes detected in comparisons of random groups of unstimulated controls from 14 scRNA-seq studies with at least six control samples. f Number of DE genes detected within spinal cord regions from control mice profiled by spatial transcriptomics24 using a representative pseudobulk method, edgeR-LRT (points), or a representative single-cell method, the Wilcoxon rank-sum test (arrowheads). g Mean change in variance between biological replicates and pseudo-replicates for 18 human and 20 mouse scRNA-seq datasets.CC BY · PubMed Central
Fig. 5 True and false discoveries in the injured mouse spinal cord. a Synapses and projections of Vglut2ON neurons in the mouse lumbar spinal cord before and after SCI. Experiments were repeated in n = 3 mice. b Top, chronophotography of mice before and after SCI. Bottom left, principal component analysis of gait parameters for each condition (small circles). Large circles show the average per group. Bottom right, bar plot showing the average scores on principal component 1 (PC1), which quantify the locomotor performance of injured and uninjured mice. c snRNA-seq experimental design. Inset demonstrates persistent amplitude at increasing frequency of the H-reflex in response to plantar stimulation, reflecting hyperexcitability. d Uniform manifold approximation and projection (UMAP) visualization of 19,237 nuclei, revealing the major cell types of the mouse lumbar spinal cord. e UMAP visualization colored by the AUC of cell type prioritizations, as calculated by Augur27. Endothelial cells are highlighted as the cell type with the highest AUC. f Pseudobulk expression levels in lumbar spinal cord endothelial cells by snRNA-seq (in counts per 10,000), left, and mean counts per Pecam1ON cell by RNA-scope, center, for Igfbp6, a gene identified as DE only by edgeR-LRT. Horizontal line and shaded area show the mean and standard deviation, respectively. Right, colocalization of Igfbp6 with the endothelial marker gene Pecam1 in RNAscope in situ hybridizations from injured and uninjured mouse spinal cords. g As in f, but for Prex2, a gene identified as DE only by the Wilcoxon rank-sum test. h Summary of in vivo screen results. Filled bars represent genes validated by RNAscope. i, Immunohistochemistry reveals Pecam1ON cells with atrophic features after SCI. Experiments were repeated in n = 3 mice. Source data are provided as a Source Data file.
Fig. 5 True and false discoveries in the injured mouse spinal cord. a Synapses and projections of Vglut2ON neurons in the mouse lumbar spinal cord before and after SCI. Experiments were repeated in n = 3 mice. b Top, chronophotography of mice before and after SCI. Bottom left, principal component analysis of gait parameters for each condition (small circles). Large circles show the average per group. Bottom right, bar plot showing the average scores on principal component 1 (PC1), which quantify the locomotor performance of injured and uninjured mice. c snRNA-seq experimental design. Inset demonstrates persistent amplitude at increasing frequency of the H-reflex in response to plantar stimulation, reflecting hyperexcitability. d Uniform manifold approximation and projection (UMAP) visualization of 19,237 nuclei, revealing the major cell types of the mouse lumbar spinal cord. e UMAP visualization colored by the AUC of cell type prioritizations, as calculated by Augur27. Endothelial cells are highlighted as the cell type with the highest AUC. f Pseudobulk expression levels in lumbar spinal cord endothelial cells by snRNA-seq (in counts per 10,000), left, and mean counts per Pecam1ON cell by RNA-scope, center, for Igfbp6, a gene identified as DE only by edgeR-LRT. Horizontal line and shaded area show the mean and standard deviation, respectively. Right, colocalization of Igfbp6 with the endothelial marker gene Pecam1 in RNAscope in situ hybridizations from injured and uninjured mouse spinal cords. g As in f, but for Prex2, a gene identified as DE only by the Wilcoxon rank-sum test. h Summary of in vivo screen results. Filled bars represent genes validated by RNAscope. i, Immunohistochemistry reveals Pecam1ON cells with atrophic features after SCI. Experiments were repeated in n = 3 mice. Source data are provided as a Source Data file.CC BY · PubMed Central

Verification

Claims the paper makes

88 Own result · 47 Own interpretation · 14 Partly supported · 9 No basis given · 7 Supported · 5 Couldn’t confirm · 4 No full text of the sources · 3 Own method · 1 Mixed support · accuracy not yet measured

Show all 138 claims

  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 21
  • The quote only partly states this claim. 2
40 of 40

  • The authors state this as an assumption.

    The closest possible approximation to the experimental ground truth can be obtained from matched bulk and scRNA-seq performed on the same population of purified cells, exposed to the same perturbations, and sequenced in the same laboratories.

    “We posited that the closest possible approximation to this ground truth could be obtained from matched bulk and scRNA-seq performed on the same population of purified cells, exposed to the same perturbations, and sequenced in the same laboratories.”

  • The paper relies on this without support.

    Measurements of gene expression in single cells are inherently sparse.

    “Measurements of gene expression in single cells are inherently sparse.”

  • The paper relies on this without support.

    Many factors that produce variability between biological replicates can be minimized in animal models, including genetic background, environment, intensity and timing of the biological perturbation, and sample processing.

    “Many of the factors that produce this variability between replicates can be minimized in animal models, including the genetic background, environment, intensity and timing of the biological perturbation, and sample processing.”

  • The paper relies on this without support.

    Sources of variation between biological replicates are inherently more difficult to control in experiments involving human subjects.

    “In contrast, these sources of variation are inherently more difficult to control in experiments involving human subjects.”

  • The paper relies on this without support.

    The pervasiveness of false discoveries in published analyses of single-cell data and the propensity for these false discoveries to affect the biological conclusions of a study have remained unclear.

    “Consequently, the pervasiveness of false discoveries in published analyses of single-cell data and the propensity for these false discoveries to affect the biological conclusions of a study have remained unclear.”

  • The paper relies on this without support.

    The 500 most recent entries in the curated database are representative of scRNA-seq studies published in the past two years.

    “we limited our analysis to the 500 most recently published studies”

  • The paper relies on this without support.

    The 18 ground truth datasets are representative of single-cell experiments for assessing DE methods.

    “which led us to compile a resource of eighteen ground truth datasets from four publications12–15”

  • The paper relies on this without support.

    The 46 published scRNA-seq datasets in the compendium are representative of single-cell RNA-seq studies generally.

    “We assembled a compendium of 46 published single-cell or single-nucleus RNA-seq studies (Supplementary Fig. 3), and performed DE analyses across this compendium to establish the generality of our conclusions.”

  • The paper relies on this without support.

    Randomly splitting a control group into artificial control and treatment groups creates a valid null distribution for DE analysis.

    “For each of these studies, we split the control group randomly into artificial ‘control’ and ‘treatment’ groups, and performed DE analysis.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 104 references

Research works printed but not linked: 1

  1. 10.1007/978-1-4939-3578-9$\_$19 · DOI

Cited by

Sorted by citations

Items 1,101–1,200 of 1,210

Related

See all

Metadata

Authors
Squair, J. W., Gautier, M., Kathe, C., Anderson, M. A., James, N. D., Hutson, T. H., Hudelle, R., Qaiser, T., Matson, K. J. E., Barraud, Q., Levine, A. J., La Manno, G., Skinnider, M. A., & Courtine, G.
Affiliations
  • University of British Columbia (CA)
  • Centre Hospitalier Universitaire Vaudois (CH)
  • National Institute of Neurological Disorders and Stroke (US)
Corresponding author
  • Skinnider, Michael A.
  • Courtine, Grégoire
ORCID
Venue
Nature Communications
Publisher
openRxiv
Published
2021-03-12
Volume
12
Issue
1
Pages
5692-5692
ISSN
2041-1723 (ISSN-L)
DOI
10.1101/2021.03.12.435024
PMC
PMC8479118
OpenAlex
W3202649987
Cited by
1,086
References
104
Citations by year
2026: 396 · 2025: 214 · 2024: 231 · 2023: 145 · 2022: 86 · 2021: 12
FWCI
34.62
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (submittedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • SERVAL (Université de Lausanne) (publishedVersion, cc-by)
  • IRIS (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
Funders
  • National Science Foundation
  • Wings for Life
  • Compute Canada
  • Western Canada Research Grid
  • Wyss Center for Bio and Neuroengineering
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (185728, 192558, 193445, 190495, 310030)
  • University of British Columbia
  • National Institutes of Health
  • Canadian Institutes of Health Research
  • National Supercomputing Center, Korea Institute of Science and Technology Information
  • National Institute of Neurological Disorders and Stroke
  • H2020 European Research Council (682999)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
single-cell differential expression, false discovery, biological replicates, comparison of statistical methods, differentially expressed genes, spinal cord injury, mouse spinal cord, cell type-specific responses, false positives
MeSH terms
Data Accuracy, Biological Variation, Population, Biological Variation, Individual, RNA-Seq, Animals, Gene Expression Regulation, Humans, Rabbits, Swine, Models, Statistical, Mice, Rats, Single-Cell Analysis, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review