CoChat
Sign Up
Cited by 710Single-cell and spatial transcriptomicsGene expression and cancer classificationSpatial and Panel Data Analysis

SpatialDE: identification of spatially variable genes

Open access
Svensson, Valentine, Teichmann, Sarah A, Stegle, Oliver·2018·Nature Methods
DOI 10.1038/nmeth.4636PMC PMC6350895

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Technological advances have made it possible to measure spatially resolved gene expression at high throughput. However, methods to analyze these data are not established. Here we describe SpatialDE, a statistical test to identify genes with spatial patterns of expression variation from multiplexed imaging or spatial RNA-sequencing data. SpatialDE also implements 'automatic expression histology', a spatial gene-clustering approach that enables expression-based tissue histology.

AI snapshot

Written from the paper’s full text.

TL;DR

SpatialDE is a Gaussian-process-based statistical test that identifies spatially variable genes from spatial transcriptomics, multiplexed imaging, and other spatially/temporally annotated expression data. It also implements automatic expression histology (AEH) to cluster genes into spatial expression patterns.

Key findings

  • In mouse olfactory bulb spatial transcriptomics, SpatialDE identified 67 SV genes (FDR < 0.05), with spatial dependencies explaining up to 70% of expression variance; this set overlapped only 40 genes with 3,497 conventional HVGs.
  • In breast cancer biopsies, SpatialDE identified 115 SV genes (FDR < 0.05) versus 3,503 HVGs, with 34 overlapping genes; SV genes were enriched for collagens (Reactome “Collagen formation”, P = 3.38 × 10⁻¹⁴).
  • In SeqFISH mouse hippocampus data (249 genes), SpatialDE identified 32 SV genes (FDR < 0.05), including linear (N = 5) and periodic (N = 8) patterns, versus 58 HVGs with 5 overlapping genes.
  • In MERFISH human osteosarcoma cell-line data (140 probes), SpatialDE identified 91 SV genes (65% of all genes, FDR < 0.05), including six of seven genes highlighted as differentially expressed between proliferating and resting subpopulations; negative control probes were not detected as spatially variable.

Methods

  • SpatialDE models each gene’s expression across spatial coordinates using a multivariate normal/Gaussian process model with a spatial covariance term and a non-spatial noise term; the fraction of spatial variance is FSV = 1 / (1 + δ).
  • Significance is assessed by comparing the fitted spatial model to a null model without spatial covariance, with P-values estimated from a χ² distribution and multiple testing adjusted by the Q-value method to control FDR; model selection uses BIC for linear, periodic, and squared exponential covariance functions.
  • AEH clusters spatially variable genes using a Gaussian mixture model with a spatial Gaussian process prior on cluster centroids, fitted by variational inference; applications used spatial transcriptomics (mouse olfactory bulb, breast cancer), SeqFISH (mouse hippocampus), MERFISH (human osteosarcoma), and frog development RNA-seq data.

Limitations

  • Stated: identification of periodic patterns has technical limitations, particularly in edge cases where noise can mask statistical significance for visually similar patterns.
  • Stated: future extensions could tailor SpatialDE to specific platforms, more explicitly model technical sources of variation, and incorporate tissue makeup or local differences in cell density.
  • Evident from the text: the model assumes normally distributed residual noise and independent observations across cells, requiring normalization steps such as Anscombe’s variance-stabilizing transformation and regression of log total counts before fitting.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/nmeth.4636

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 18 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured

  • Premisereasoning

    ANOVA-based methods for differential expression between groups of cells can only detect variations captured by differences between discrete groups.

    “Critically, such methods can only detect variations that are captured by differences between discrete groups.”
    Quote in the paperSubstantial claim3 of 5 sources checked
    • cites [2] Achim, K. et al. Nat. Biotechnol. 33, 503–509 (2015). source
    • cites [3] Satija, R., Farrell, J.A., Gennert, D., Schier, A.F. & Regev, A. Nat. Biotechnol. 33, 495–502 (2015). source

      The Online Methods section on embryonic subpopulations describes a likelihood-ratio test for differential expression, not ANOVA. Searches for “ANOVA” and for the claimed limitation on detecting variation between discrete groups found no passage in our copy that addresses that limitation; no page numbers are available. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [6] Ståhl, P.L. et al. Science 353, 78–82 (2016). source
    • cites [7] Shah, S., Lubeck, E., Zhou, W. & Cai, L. Neuron 92, 342–357 (2016). source

      The Experimental Procedure (Clustering; page unavailable) describes correlation-based clustering and random-forest classification, while the Results (Combinatorial expression patterns define fine clusters; page unavailable) discusses variation within the expression data. Neither passage addresses whether ANOVA-based differential-expression methods can detect only variation between discrete groups. Searches for “ANOVA” and that specific limitation found no relevant passage in our copy. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [10] Pettit, J.-B. et al. PLOS Comput. Biol. 10, e1003824 (2014). source

      The Results sections on the clustering method and model validation describe assigning voxels to clusters and comparing clustering approaches, not ANOVA-based differential-expression tests. Searches for “ANOVA,” differential expression between groups, and the claimed limitation found no passage in our copy that addresses it. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
  • Findingown data

    Significantly SV genes in breast cancer were enriched for collagens, which distinguish tissue substructure (Reactome "Collagen formation", P = 3.38 * 10-14).

    “Significantly SV genes were enriched for collagens, which distinguish tissue substructure13 (Reactome “Collagen formation”, P = 3.38 * 10-14, gProfiler14, Supp. Table 1).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [13] Seewaldt, V.L. Nature 490, 490–491 (2012). source
    • cites [14] Reimand, J. et al. Nucleic Acids Res. 44, W83–W89 (2016). source
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.
  • Findingown data

    These genes do not have high rank based on non-spatial HVG measures (including mean-CV2 relation or mean-dropout relation).

    “Furthermore, these genes do not have high rank based on non-spatial HVG measures (including mean-CV2 relation9 or mean-dropout relation15, Supp. Fig. 8).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [9] Brennecke, P. et al. Nat. Methods 10, 1093–1095 (2013). source
    • cites [15] Andrews, T.S. & Hemberg, M. bioRxiv Preprint at https://www.biorxiv.org/content/early/2016/10/20/065094 (2016).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.
  • Study design and conductcitation

    SpatialDE uses the Q-value method to adjust for multiple testing, thereby controlling the false discovery rate (FDR).

    “Unless stated otherwise, we use the Q-value method23 to adjust for multiple testing, thereby controlling the false discovery rate (FDR).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [23] Storey, J.D. & Tibshirani, R. Proc. Natl. Acad. Sci. USA 100, 9440–9445 (2003). source
  • Findingown data

    Applying automatic expression histology to mouse olfactory bulb identified five canonical expression patterns, clearly demarcating structures visible in the HE image.

    “Applying automatic expression histology identified five canonical expression patterns, clearly demarcating structures visible in the HE image (Fig. 2D, Supp. Fig. 4A).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.
  • May combine two claims. 9
  • Classified as probably not a substantial claim. 3

  • May combine two claims.

    Premisecitation

    Existing approaches for identifying highly variable genes (HVG) ignore spatial information and hence do not measure spatial variability.

    “However, existing approaches for identifying highly variable genes (HVG)9, as used for conventional scRNA-seq data, ignore spatial information and hence do not measure spatial variability (Fig. 1A).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [9] Brennecke, P. et al. Nat. Methods 10, 1093–1095 (2013). source

  • May combine two claims.

    Premisecitation

    SpatialDE is related to and generalizes previous approaches for detecting temporal and periodic gene expression patterns in time series.

    “SpatialDE is related to and generalizes previous approaches for detecting temporal19 and periodic gene expression patterns20 in time series.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [19] Kalaitzis, A.A. & Lawrence, N.D. BMC Bioinformatics 12, 180 (2011). source
    • cites [20] Durrande, N., Hensman, J., Rattray, M. & Lawrence, N.D. PeerJ Comput. Sci. 2, e50 (2016). source

  • May combine two claims.

    Findingown data

    The SV genes in mouse olfactory bulb included canonical marker genes highlighted by Ståhl et al., such as Penk, Doc2g, and Kctd12, and additional genes that define the granule cell layer.

    “These included canonical marker genes highlighted in the primary analysis by Ståhl et al6, such as Penk, Doc2g, and Kctd12, but also additional genes that define the granule cell layer of the bulb.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] Ståhl, P.L. et al. Science 353, 78–82 (2016). source
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • May combine two claims.

    Findingown data

    SpatialDE identified 115 SV genes (FDR < 0.05) in breast cancer biopsies, compared to 3,503 detected by HVG, with overlap of 34 genes.

    “SpatialDE identified 115 SV genes (FDR < 0.05, compared to 3,503 detected by HVG; overlap 34 genes), including seven genes with known disease relevance that were highlighted in the primary analysis (Supp. Fig. 6B-C).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • May combine two claims.

    Findingown data

    Automatic expression histology of the SV genes in breast cancer biopsy most clearly separated the adipocytic from the denser region of the tissue, but additionally identified a small region overlapping the tumor feature in the HE image.

    “Automatic expression histology of the SV genes in the breast cancer biopsy (Supp. Fig. 4B) most clearly separated the adipocytic from the denser region of the tissue, but additionally identified a small region overlapping the tumor feature in the HE image.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • May combine two claims.

    Findingown data

    Conventional clustering based on the expression profile of each spot ignores spatial information and does not establish relationships between genes defining cell types as in AEH.

    “However, this approach ignores spatial information and does not establish relationships between genes defining cell types as in AEH (Supp. Fig. 5).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • May combine two claims.

    Findingown data

    In MERFISH data from human osteosarcoma cell line, SpatialDE identified 91 genes as spatially variable (65% of all genes, FDR<0.05), compared to 29 HVG with overlap of 24 genes.

    “Consistent with these findings, our method identified a substantial proportion of the genes assayed as spatially variable (N=91, 65% of all genes, FDR<0.05, 29 genes HVG with overlap of 24 genes), including six of the seven genes highlighted as differentially expressed between proliferating and resting subpopulations (e.g. THBS1 and CENPF1, Supp Fig. 13C).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • May combine two claims.

    Findingown data

    SpatialDE identified 32 SV genes (FDR < 0.05) in multiplexed smFISH data of mouse hippocampus, compared to 58 HVG with overlap of 5 genes.

    “SpatialDE identified 32 SV genes (Fig. 2E, FDR < 0.05, 58 genes were detected as HVG, with an overlap of 5 genes) and again SpatialDE identified genes with different types of spatial variation, including linear (N=5) and periodic patterns (N=8, examples in Supp. Fig. 11).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    These genes (Mog, Myl14, Ndnf) were grouped into histological expression patterns by the AEH method.

    “These genes were grouped into histological expression patterns by the AEH method (Figure 2H, Supp. Fig. 2C).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Inspection revealed regularly dispersed regions potentially identifying a pattern associated with higher neuron density.

    “Other genes with periodic patterns, such as the vesicular glutamate transporter Slc17a7, were identified with shorter periods (1.1 mm), and inspection revealed regularly dispersed regions, potentially identifying a pattern associated with higher neuron density12, suggesting that periodic expression in tissues is of biological interest.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Takamori, S., Rhee, J.S., Rosenmund, C. & Jahn, R. Nature 407, 189–194 (2000). source
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    SpatialDE implements automatic expression histology, a spatial gene clustering approach that enables expression-based tissue histology.

    “SpatialDE also implements “automatic expression histology”, a spatial gene clustering approach that enables expression-based tissue histology.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

  • May combine two claims.

    Interpretationown data

    SpatialDE identifies spatially variable genes and allows biologically relevant features to be detected in tissue samples without a priori histological annotation.

    “Our results demonstrate that SpatialDE identifies spatially variable genes and allows biologically relevant features to be detected in tissue samples without a priori histological annotation.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The SpatialDE model assumes normally distributed residual noise and independent observations across cells.
    • The authors state this as an assumption: SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.
    • The authors state this as an assumption: The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.
12 of 12

  • The authors state this as an assumption.

    The SpatialDE model assumes normally distributed residual noise and independent observations across cells.

    “The SpatialDE model is based on the assumption of normally distributed residual noise and independent observations across cells.”

  • The authors state this as an assumption.

    SpatialDE assumes a squared exponential covariance function for the spatial covariance between cells.

    “SpatialDE uses the so called squared exponential covariance function to define Σ: (2) Σi,j=k(xi,xj)=exp⁡(−(|xi−xj(|)²)/(2·l²)), whereby the covariance between pairs of cells i and j is modelled to decay exponentially with the squared distance between them.”

  • The authors state this as an assumption.

    The null hypothesis of no spatial covariance is appropriate for testing significance of spatial variance.

    “the model likelihood of the fitted SpatialDE model is compared to the likelihood of a model that corresponds to the null hypothesis of no spatial covariance”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Length scale is described with incompatible units

    Minor

    The paper describes the length scale as the expected number of changes in a unit interval, but later assigns it values in micrometers. A number of changes per interval is not a length measured in micrometers.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Andrews, T.S. & Hemberg, M. bioRxiv Preprint at https://www.biorxiv.org/content/early/2016/10/20/065094 (2016).
  2. Bishop, C.M. Pattern Recognition and Machine Learning (Springer, 2006).
  3. Krige, D.G. J. S. Afr. Inst. Min. Metall. 52, 119–139 (1951).
  4. Lloyd, J.R., Duvenaud, D., Grosse, R., Tenenbaum, J.B. & Ghahramani, Z. in Proceedings of the Twenty-eighth AAAI Conference on Artificial Intelligence 1242–1250 (AAAI Press, 2014).

Cited by

Sorted by citations

Related

Authors
Svensson, V., Teichmann, S. A., & Stegle, O.
Affiliations
  • European Bioinformatics Institute (GB)
  • University of Cambridge (GB)
Corresponding author
  • Svensson, Valentine
  • Stegle, Oliver
ORCID
Venue
Nature Methods
Publisher
Springer Science and Business Media LLC
Published
2018-03-19
Volume
15
Issue
5
Pages
343-346
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
DOI
10.1038/nmeth.4636
PMC
PMC6350895
OpenAlex
W2953036966
Semantic Scholar
56345ac9378727f64aaf2aad0573a5268334cb82
Cited by
695
References
35
Citations by year
2026: 107 · 2025: 135 · 2024: 120 · 2023: 94 · 2022: 91 · 2021: 78 · 2020: 35 · 2019: 20 · 2018: 12 · 2017: 3
FWCI
13.16
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
  • Apollo (University of Cambridge) (submittedVersion)
  • Apollo
Funders
Wellcome Trust
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
Keywords
spatially variable genes, spatial gene expression, multiplexed imaging, gene clustering
MeSH terms
Breast Neoplasms, Female, Humans, Models, Biological, Gene Expression Regulation, Neoplastic, Protein Transport, Single-Cell Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-02.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-02.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review