CoChat
Sign Up
Cited by 235Gene Regulatory Network AnalysisSingle-cell and spatial transcriptomicsBioinformatics and Genomic Networks

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Evaluating methods of inferring gene regulatory networks highlights their lack of performance for single cell gene expression data

Open access
Chen, Shuonan, Mar, Jessica C.·2018·BMC Bioinformatics
DOI 10.1186/s12859-018-2217-zPMC PMC6006753
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: A fundamental fact in biology states that genes do not operate in isolation, and yet, methods that infer regulatory networks for single cell gene expression data have been slow to emerge. With single cell sequencing methods now becoming accessible, general network inference algorithms that were initially developed for data collected from bulk samples may not be suitable for single cells. Meanwhile, although methods that are specific for single cell data are now emerging, whether they have improved performance over general methods is unknown. In this study, we evaluate the applicability of five general methods and three single cell methods for inferring gene regulatory networks from both experimental single cell gene expression data and in silico simulated data. RESULTS: Standard evaluation metrics using ROC curves and Precision-Recall curves against reference sets sourced from the literature demonstrated that most of the methods performed poorly when they were applied to either experimental single cell data, or simulated single cell data, which demonstrates their lack of performance for this task. Using default settings, network methods were applied to the same datasets. Comparisons of the learned networks highlighted the uniqueness of some predicted edges for each method. The fact that different methods infer networks that vary substantially reflects the underlying mathematical rationale and assumptions that distinguish network methods from each other. CONCLUSIONS: This study provides a comprehensive evaluation of network modeling algorithms applied to experimental single cell gene expression data and in silico simulated datasets where the network structure is known. Comparisons demonstrate that most of these assessed network methods are not able to predict network structures from single cell expression data accurately, even if they are specifically developed for single cell methods. Also, single cell methods, which usually depend on more elaborative algorithms, in general have less similarity to each other in the sets of edges detected. The results from this study emphasize the importance for developing more accurate optimized network modeling methods that are compatible for single cell data. Newly-developed single cell methods may uniquely capture particular features of potential gene-gene relationships, and caution should be taken when we interpret these results.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12859-018-2217-z

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Study Workflow. Eight network reconstruction methods – including five general methods: partial correlation (Pcorr), Bayesian network (BN), GENIE3, ARACNE and CLR, and three single cell-specific methods: SCENIC, SCODE and PIDC – were applied to two single cell experimental datasets, and two simulated datasets that resemble single cell data. Evaluation of these methods was based on their ability to reconstruct a reference network, and this was assessed using PR, ROC curves, and other network analysis metrics
Fig. 1 Study Workflow. Eight network reconstruction methods – including five general methods: partial correlation (Pcorr), Bayesian network (BN), GENIE3, ARACNE and CLR, and three single cell-specific methods: SCENIC, SCODE and PIDC – were applied to two single cell experimental datasets, and two simulated datasets that resemble single cell data. Evaluation of these methods was based on their ability to reconstruct a reference network, and this was assessed using PR, ROC curves, and other network analysis metricsCC BY · PubMed Central
Fig. 2 ROC (top) and PR (bottom) curves for each method applied to the simulated datasets. The results obtained from the Sim1 dataset are shown on the left (a & c) and the Sim2 dataset is shown on the right (b & d). Diagonal black lines on the ROC curves are baselines indicating the prediction level equivalent to a random guess (a & b). ROC curves showed that when the threshold changes and more edges are detected, both false positive and true positive rates increased, but the speed of this increase might not be the same. The PR curves show that when the detection thresholds decreased, the number of detected edges increased, with a corresponding increase in recall (more true edges are detected) but decrease in precision (increasing the number of detected edges that are not in the reference network)
Fig. 2 ROC (top) and PR (bottom) curves for each method applied to the simulated datasets. The results obtained from the Sim1 dataset are shown on the left (a & c) and the Sim2 dataset is shown on the right (b & d). Diagonal black lines on the ROC curves are baselines indicating the prediction level equivalent to a random guess (a & b). ROC curves showed that when the threshold changes and more edges are detected, both false positive and true positive rates increased, but the speed of this increase might not be the same. The PR curves show that when the detection thresholds decreased, the number of detected edges increased, with a corresponding increase in recall (more true edges are detected) but decrease in precision (increasing the number of detected edges that are not in the reference network)CC BY · PubMed Central
Fig. 3 ROC (top) and PR (bottom) curves for each method applied to single cell experimental data. The results obtained from the ESC dataset are shown on the left side (a & c) and the HSC dataset is shown on the right side (b & d). Diagonal black lines on the ROC curves are baselines indicating the prediction level equivalent to a random guess (a & b). In (a) and (b), almost all the methods aligned with diagonal black lines in the ROC curve, suggesting the predictions are nearly equivalent to a random guess for all these methods. It is easier to observe the behavior of each method in PR curves. Although the ROC curves indicate similar performance across all methods, for the HSC dataset, the PR curve reveal that the methods have different prediction accuracy (when the total number of detected edges is small), as shown when the curve is close to the y-axis, and thus provides another aspect of evaluation
Fig. 3 ROC (top) and PR (bottom) curves for each method applied to single cell experimental data. The results obtained from the ESC dataset are shown on the left side (a & c) and the HSC dataset is shown on the right side (b & d). Diagonal black lines on the ROC curves are baselines indicating the prediction level equivalent to a random guess (a & b). In (a) and (b), almost all the methods aligned with diagonal black lines in the ROC curve, suggesting the predictions are nearly equivalent to a random guess for all these methods. It is easier to observe the behavior of each method in PR curves. Although the ROC curves indicate similar performance across all methods, for the HSC dataset, the PR curve reveal that the methods have different prediction accuracy (when the total number of detected edges is small), as shown when the curve is close to the y-axis, and thus provides another aspect of evaluationCC BY · PubMed Central
Fig. 4 AUROC (top) and AUPR (bottom) scores demonstrate consistently poor performance for most of the methods and datasets. In both panels, the horizontal red lines represent the line of a random guess and baseline, which are the same (= 0.5) for AUROC across datasets (a) but differ for AUPR (b), since the baseline indicates the value of precision when all the reference edges were recovered, and this depends on the total number of genes in the datasets and the number of edges in the reference networks
Fig. 4 AUROC (top) and AUPR (bottom) scores demonstrate consistently poor performance for most of the methods and datasets. In both panels, the horizontal red lines represent the line of a random guess and baseline, which are the same (= 0.5) for AUROC across datasets (a) but differ for AUPR (b), since the baseline indicates the value of precision when all the reference edges were recovered, and this depends on the total number of genes in the datasets and the number of edges in the reference networksCC BY · PubMed Central
Fig. 5 Intersection of reconstructed networks from general methods and reference networks outlines the ability of the different methods to identify the same true positives. These methods detected a core set of interactions in the learned networks for the ESC data (a), HSC data (b), and two simulated datasets, Sim1 (c) and Sim2 (d). In general, each method also detected edges that were unique to the method and the dataset, except for Pcorr in Sim1 (c, see text). Only a small set of edges in the reference networks were recovered by intersection of three methods, and also different methods detected edges that were unique. Moreover, we show that even after combining all the methods, there were still edges in the reference network that were not detected, as indicated in the white section for each panel
Fig. 5 Intersection of reconstructed networks from general methods and reference networks outlines the ability of the different methods to identify the same true positives. These methods detected a core set of interactions in the learned networks for the ESC data (a), HSC data (b), and two simulated datasets, Sim1 (c) and Sim2 (d). In general, each method also detected edges that were unique to the method and the dataset, except for Pcorr in Sim1 (c, see text). Only a small set of edges in the reference networks were recovered by intersection of three methods, and also different methods detected edges that were unique. Moreover, we show that even after combining all the methods, there were still edges in the reference network that were not detected, as indicated in the white section for each panelCC BY · PubMed Central
Fig. 6 Intersection of learned networks from single cell methods and the reference network highlights the differences between the edges that are uniquely detected by each method. Although single cell methods detected a common set of interactions, there was far more inconsistency in their detections, compared to the results of general methods. Each method, however, was able to detect some ‘correct’ edges that are unique to each method (as indicated from the overlap between each colored ellipse and the white ellipse, also see Results). Similarly, only a handful of edges were commonly detected by all three methods for the HSC and Sim2 datasets (b & d), and no single edge was commonly detected for the ESC and Sim1 datasets (a & c)
Fig. 6 Intersection of learned networks from single cell methods and the reference network highlights the differences between the edges that are uniquely detected by each method. Although single cell methods detected a common set of interactions, there was far more inconsistency in their detections, compared to the results of general methods. Each method, however, was able to detect some ‘correct’ edges that are unique to each method (as indicated from the overlap between each colored ellipse and the white ellipse, also see Results). Similarly, only a handful of edges were commonly detected by all three methods for the HSC and Sim2 datasets (b & d), and no single edge was commonly detected for the ESC and Sim1 datasets (a & c)CC BY · PubMed Central
Fig. 7 Investigating the similarity of the networks produced by the seven network methods. The PCA plots indicate how much each method is similar to each other in terms of edge detection ranking. We show that there is no consistency in the clustering of these methods, and any similarities amongst them vary based on the datasets. SCODE was notably a consistent outlier amongst all the methods
Fig. 7 Investigating the similarity of the networks produced by the seven network methods. The PCA plots indicate how much each method is similar to each other in terms of edge detection ranking. We show that there is no consistency in the clustering of these methods, and any similarities amongst them vary based on the datasets. SCODE was notably a consistent outlier amongst all the methodsCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 49 references

Cited by

Sorted by citations

See all 235

Related

Metadata

Authors
Chen, S., & Mar, J. C.
Affiliations
Albert Einstein College of Medicine (US)
Corresponding author
Mar, Jessica C.
ORCID
Venue
BMC Bioinformatics
Publisher
Springer Science and Business Media LLC
Published
2018-06-19
Volume
19
Issue
1
Pages
232-232
ISSN
1471-2105 (ISSN-L)
DOI
10.1186/s12859-018-2217-z
PMC
PMC6006753
OpenAlex
W2808743742
Cited by
235
References
49
Citations by year
2026: 14 · 2025: 19 · 2024: 24 · 2023: 36 · 2022: 35 · 2021: 44 · 2020: 38 · 2019: 19 · 2018: 6
FWCI
9.52
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Bioinformatics (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • NYSTEM Training Grant (C030292GG)
  • NYSTEM (C029154)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
Keywords
gene regulatory network inference, single-cell gene expression, gene-gene interaction
MeSH terms
Algorithms, Hematopoietic Stem Cells, Humans, Computational Biology, Gene Expression Profiling, Gene Regulatory Networks, Embryonic Stem Cells, Single-Cell Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review