Cited by 417Functional Brain Connectivity StudiesComplex Network Analysis TechniquesAdvanced Neuroimaging Techniques and Applications

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.

Graph Theoretical Analysis of Functional Brain Networks: Test-Retest Evaluation on Short- and Long-Term Resting-State Functional MRI Data

Open access
Wang, Jin-Hui, Zuo, Xi-Nian, Gohel, Suril, Milham, Michael P., Biswal, Bharat B., He, Yong·2011·PLoS ONE

Abstract

Graph-based computational network analysis has proven a powerful tool to quantitatively characterize functional architectures of the brain. However, the test-retest (TRT) reliability of graph metrics of functional networks has not been systematically examined. Here, we investigated TRT reliability of topological metrics of functional brain networks derived from resting-state functional magnetic resonance imaging data. Specifically, we evaluated both short-term (<1 hour apart) and long-term (>5 months apart) TRT reliability for 12 global and 6 local nodal network metrics. We found that reliability of global network metrics was overall low, threshold-sensitive and dependent on several factors of scanning time interval (TI, long-term>short-term), network membership (NM, networks excluding negative correlations>networks including negative correlations) and network type (NT, binarized networks>weighted networks). The dependence was modulated by another factor of node definition (ND) strategy. The local nodal reliability exhibited large variability across nodal metrics and a spatially heterogeneous distribution. Nodal degree was the most reliable metric and varied the least across the factors above. Hub regions in association and limbic/paralimbic cortices showed moderate TRT reliability. Importantly, nodal reliability was robust to above-mentioned four factors. Simulation analysis revealed that global network metrics were extremely sensitive (but varying degrees) to noise in functional connectivity and weighted networks generated numerically more reliable results in compared with binarized networks. For nodal network metrics, they showed high resistance to noise in functional connectivity and no NT related differences were found in the resistance. These findings provide important implications on how to choose reliable analytical schemes and network metrics of interest.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0021976

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Spatial similarity and TRT reliability patterns of S-AAL-based RSFC. Mean Pearson correlation matrices (a), consistency of overall patterns between mean matrices (b) and TRT reliability of individual connections as well as the relationship between short-term and long-term reliability (c) are illustrated. The mean correlation matrices exhibited high similarity from both visual inspection (a) and quantitative spatial correlation analyses (b). Further TRT reliability analyses revealed a portion of connections exhibiting fair to excellent reliability (c, also see Fig. 2). Moreover, short-term reliability was significantly (p<0.05) correlated with long-term reliability among connections (c). Functional connections linking inter-hemisphere homotopic regions, as highlighted by plus signs (+), showed high connectivity strength and many of them exhibited high reliability. TRT, test-retest; RSFC, resting-state functional connectivity; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas. Of note, the structural ROIs were listed as in Table S1.
Figure 1 Spatial similarity and TRT reliability patterns of S-AAL-based RSFC. Mean Pearson correlation matrices (a), consistency of overall patterns between mean matrices (b) and TRT reliability of individual connections as well as the relationship between short-term and long-term reliability (c) are illustrated. The mean correlation matrices exhibited high similarity from both visual inspection (a) and quantitative spatial correlation analyses (b). Further TRT reliability analyses revealed a portion of connections exhibiting fair to excellent reliability (c, also see Fig. 2). Moreover, short-term reliability was significantly (p<0.05) correlated with long-term reliability among connections (c). Functional connections linking inter-hemisphere homotopic regions, as highlighted by plus signs (+), showed high connectivity strength and many of them exhibited high reliability. TRT, test-retest; RSFC, resting-state functional connectivity; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas. Of note, the structural ROIs were listed as in Table S1.CC BY · PubMed Central
Figure 2 TRT reliability distribution of RSFC. Both short-term and long-term TRT reliability exhibit approximatively normal distribution for all ROI sets. The mean reliability was about 0.28 (short-term) and 0.24 (long-term) for both structural ROIs-based RSFC while relatively low values were observed for functional ROIs-based RSFC. Green dots indicate the critical values used in the present study to grade reliability. RSFC, resting-state functional connectivity; TRT, test-retest.
Figure 2 TRT reliability distribution of RSFC. Both short-term and long-term TRT reliability exhibit approximatively normal distribution for all ROI sets. The mean reliability was about 0.28 (short-term) and 0.24 (long-term) for both structural ROIs-based RSFC while relatively low values were observed for functional ROIs-based RSFC. Green dots indicate the critical values used in the present study to grade reliability. RSFC, resting-state functional connectivity; TRT, test-retest.CC BY · PubMed Central
Figure 3 Relationship between RSFC and TRT reliability. Scatter plots of mean connectivity strength against corresponding ICC values are depicted to show the relationship for both S-AAL (a) and F-DOS (b) based correlation matrices. The trend lines were obtained by linear least-square fit. Significant (p<0.05) positive correlations were found between positive RSFC and their corresponding ICC values for both ROIs sets and for both short-term and long-term scanning. In addition, significant negative correlations were also found for negative RSFC with their corresponding ICC values but only for F-DOS-based correlation matrices. These findings suggest higher reliability for stronger RSFC. Functional connections linking inter-hemisphere homotopic regions are highlighted by plus signs (+) for S-AAL but not for F-DOS because of the absence of direct correspondence. RSFC, resting-state functional connectivity; TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).
Figure 3 Relationship between RSFC and TRT reliability. Scatter plots of mean connectivity strength against corresponding ICC values are depicted to show the relationship for both S-AAL (a) and F-DOS (b) based correlation matrices. The trend lines were obtained by linear least-square fit. Significant (p<0.05) positive correlations were found between positive RSFC and their corresponding ICC values for both ROIs sets and for both short-term and long-term scanning. In addition, significant negative correlations were also found for negative RSFC with their corresponding ICC values but only for F-DOS-based correlation matrices. These findings suggest higher reliability for stronger RSFC. Functional connections linking inter-hemisphere homotopic regions are highlighted by plus signs (+) for S-AAL but not for F-DOS because of the absence of direct correspondence. RSFC, resting-state functional connectivity; TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).CC BY · PubMed Central
Figure 4 Spatial similarity and TRT reliability patterns of F-DOS-based RSFC. Mean Pearson correlation matrices (a), consistency of overall patterns between mean matrices (b) and TRT reliability of individual connections as well as the relationship between short-term and long-term reliability (c) are illustrated. The mean correlation matrices exhibited high similarity from both visual inspection (a) and quantitative spatial correlation analyses (b). Further TRT reliability analyses revealed many connections exhibiting fair to excellent reliability (c, also see Fig. 2). Moreover, a significant (p<0.05) correlation was found in the ICC matrices between short-term and long-term scans (c). No inter-hemisphere homotopic functional connections were highlighted because of the absence of direct inter-hemisphere correspondence for these ROIs. TRT, test-retest; RSFC, resting-state functional connectivity; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010). Of note, the functional ROIs were listed as in Table S3.
Figure 4 Spatial similarity and TRT reliability patterns of F-DOS-based RSFC. Mean Pearson correlation matrices (a), consistency of overall patterns between mean matrices (b) and TRT reliability of individual connections as well as the relationship between short-term and long-term reliability (c) are illustrated. The mean correlation matrices exhibited high similarity from both visual inspection (a) and quantitative spatial correlation analyses (b). Further TRT reliability analyses revealed many connections exhibiting fair to excellent reliability (c, also see Fig. 2). Moreover, a significant (p<0.05) correlation was found in the ICC matrices between short-term and long-term scans (c). No inter-hemisphere homotopic functional connections were highlighted because of the absence of direct inter-hemisphere correspondence for these ROIs. TRT, test-retest; RSFC, resting-state functional connectivity; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010). Of note, the functional ROIs were listed as in Table S3.CC BY · PubMed Central
Figure 5 TRT reliability of global network metrics as a function of sparsity threshold for S-AAL-based networks. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively. Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network anlysis; Weighted, weighted network analysis; TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas.
Figure 5 TRT reliability of global network metrics as a function of sparsity threshold for S-AAL-based networks. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively. Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network anlysis; Weighted, weighted network analysis; TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas.CC BY · PubMed Central
Figure 6 TRT reliability of summarized global network metrics (a) and metric-related differences in reliability (b). Areas under curves (AUCs) of each metrics were used to provide threshold-independent reliability estimation. Different metrics showed variable levels of reliability. Several of them were moderately reliable (e.g., lambda for S-AAL-based networks). Subsequent statistical analysis revealed significant differences in TRT reliability among the 12 global network metrics for S-AAL- but not for F-DOS-based networks. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively in (a). Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network analysis; Weighted, weighted network analysis; TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).
Figure 6 TRT reliability of summarized global network metrics (a) and metric-related differences in reliability (b). Areas under curves (AUCs) of each metrics were used to provide threshold-independent reliability estimation. Different metrics showed variable levels of reliability. Several of them were moderately reliable (e.g., lambda for S-AAL-based networks). Subsequent statistical analysis revealed significant differences in TRT reliability among the 12 global network metrics for S-AAL- but not for F-DOS-based networks. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively in (a). Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network analysis; Weighted, weighted network analysis; TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).CC BY · PubMed Central
Figure 7 TRT reliability of nodal metrics for S-AAL-based networks. Nodal reliability varied across nodal attributes and spatial locations. The full names of region's abbreviations were listed as in Table S1. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively. Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network analysis; Weighted, weighted network analysis; TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas.
Figure 7 TRT reliability of nodal metrics for S-AAL-based networks. Nodal reliability varied across nodal attributes and spatial locations. The full names of region's abbreviations were listed as in Table S1. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively. Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network analysis; Weighted, weighted network analysis; TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas.CC BY · PubMed Central
Figure 8 Boxplot of mean nodal TRT reliability for S-AAL- (a) and F-DOS- (b) based networks. Significant differences were found in the mean nodal reliability among the six nodal metrics examined with nodal degree showing the highest ICC values and least variances for both ROIs sets. TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).
Figure 8 Boxplot of mean nodal TRT reliability for S-AAL- (a) and F-DOS- (b) based networks. Significant differences were found in the mean nodal reliability among the six nodal metrics examined with nodal degree showing the highest ICC values and least variances for both ROIs sets. TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).CC BY · PubMed Central
Figure 9 Nodal TRT reliability of degree and its relationship with nodal degree centrality for S-AAL-based networks. (a) Nodal TRT reliability was mapped in anatomical space after average across scanning time interval, network type and network membership because of no effects of these factors on nodal reliability. (b) Nodal degree centrality (AUCs) was also mapped in anatomical space which was averaged across subjects and factors of scanning time interval, network type and network membership. Trend lines were further obtained by linear least-square fit to reveal the relationship between nodal degree centrality and their corresponding reliability after with (d) and without (c) correcting for the effects of regional size. Of note, the full names of region's abbreviations were listed as in Table S1. TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas; k, nodal degree; A, anterior; P, posterior; L, left; R, right.
Figure 9 Nodal TRT reliability of degree and its relationship with nodal degree centrality for S-AAL-based networks. (a) Nodal TRT reliability was mapped in anatomical space after average across scanning time interval, network type and network membership because of no effects of these factors on nodal reliability. (b) Nodal degree centrality (AUCs) was also mapped in anatomical space which was averaged across subjects and factors of scanning time interval, network type and network membership. Trend lines were further obtained by linear least-square fit to reveal the relationship between nodal degree centrality and their corresponding reliability after with (d) and without (c) correcting for the effects of regional size. Of note, the full names of region's abbreviations were listed as in Table S1. TRT, test-retest; S-AAL, structural ROIs from Anatomical Automatic Labeling atlas; k, nodal degree; A, anterior; P, posterior; L, left; R, right.CC BY · PubMed Central
Figure 10 The similarity between inter-scan ICC-based reliability and inter-scan Pearson correlation coefficients for S-AAL-based networks. The reliability and correlation analyses revealed highly consistent results (r>0.9 under most conditions), ruling out the possibility of linear scaling biases of network metrics across test and retest scans that will lead to low TRT reliability.
Figure 10 The similarity between inter-scan ICC-based reliability and inter-scan Pearson correlation coefficients for S-AAL-based networks. The reliability and correlation analyses revealed highly consistent results (r>0.9 under most conditions), ruling out the possibility of linear scaling biases of network metrics across test and retest scans that will lead to low TRT reliability.CC BY · PubMed Central
Figure 11 TRT reliability of global network metrics as a function of noise in RSFC for S-AAL-based networks. Global network metrics were sensitive to disturbances of RSFC and weighted network analysis generated numerically more stable results in comparison with binarized network analysis. The highlighted black border marks are the average reliability across metrics for binarized (square) and weighted (circle) network analysis, respectively. Of note, the sensitivity varied dramatically among metrics. Small-world parameters and network efficiency were extremely sensitive to even little noise in functional connectivity while assortativity, hierarchy, synchronization and modularity were relatively resistant to noise. TRT, test-retest; RSFC, resting-state functional connectivity.
Figure 11 TRT reliability of global network metrics as a function of noise in RSFC for S-AAL-based networks. Global network metrics were sensitive to disturbances of RSFC and weighted network analysis generated numerically more stable results in comparison with binarized network analysis. The highlighted black border marks are the average reliability across metrics for binarized (square) and weighted (circle) network analysis, respectively. Of note, the sensitivity varied dramatically among metrics. Small-world parameters and network efficiency were extremely sensitive to even little noise in functional connectivity while assortativity, hierarchy, synchronization and modularity were relatively resistant to noise. TRT, test-retest; RSFC, resting-state functional connectivity.CC BY · PubMed Central
Figure 12 TRT reliability of nodal network metrics as a function of noise in RSFC for S-AAL-based networks. Nodal network metrics were sensitive to disturbances of RSFC and no differences were observed in the resistance to noise in functional connectivity between binarized and weighted network analysis. The highlighted black border marks are the average reliability across metrics for binarized (square) and weighted (circle) network analysis, respectively. Of note, although sensitive, nodal network metrics showed strong tolerance of disturbances in RSFC. TRT, test-retest; RSFC, resting-state functional connectivity.
Figure 12 TRT reliability of nodal network metrics as a function of noise in RSFC for S-AAL-based networks. Nodal network metrics were sensitive to disturbances of RSFC and no differences were observed in the resistance to noise in functional connectivity between binarized and weighted network analysis. The highlighted black border marks are the average reliability across metrics for binarized (square) and weighted (circle) network analysis, respectively. Of note, although sensitive, nodal network metrics showed strong tolerance of disturbances in RSFC. TRT, test-retest; RSFC, resting-state functional connectivity.CC BY · PubMed Central
Figure 13 TRT reliability of global network metrics as a function of sparsity threshold for F-DOS-based networks. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively. Multiple network metrics showed modest reliability in certain threshold range. Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network anlysis; Weighted, weighted network analysis; TRT: test-retest; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).
Figure 13 TRT reliability of global network metrics as a function of sparsity threshold for F-DOS-based networks. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively. Multiple network metrics showed modest reliability in certain threshold range. Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network anlysis; Weighted, weighted network analysis; TRT: test-retest; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).CC BY · PubMed Central
Figure 14 TRT reliability of nodal metrics for F-DOS-based networks. Nodal reliability varied across nodal attributes and spatial locations. The full names of region's abbreviations were listed as in Table S3. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively. Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network analysis; Weighted, weighted network analysis; TRT, test-retest; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).
Figure 14 TRT reliability of nodal metrics for F-DOS-based networks. Nodal reliability varied across nodal attributes and spatial locations. The full names of region's abbreviations were listed as in Table S3. ICC values less than 0.25 were mapped to a single color of dark blue as well dark red color for ICC values greater than 0.75, respectively. Network (+/-), networks constructed using absolute both positive and negative correlations; Network (+), networks constructed using only positive correlations; Binarized, binarized network analysis; Weighted, weighted network analysis; TRT, test-retest; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010).CC BY · PubMed Central
Figure 15 Nodal TRT reliability of degree and its relationship with nodal degree centrality for F-DOS-based networks. (a) Nodal TRT reliability was mapped in anatomical space after average across scanning time interval, network type and network membership because of no effects of these factors on nodal reliability. (b) Nodal degree centrality (AUCs) was also mapped in anatomical space which was averaged across subjects and factors of scanning time interval, network type and network membership. Trend lines were further obtained by linear least-square fit to reveal the relationship between nodal degree centrality and their corresponding reliability after with (d) and without (c) correcting for the effects of regional size. Of note, the full names of region's abbreviations were listed as in Table S1. TRT, test-retest; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010); k, nodal degree; A, anterior; P, posterior; L, left; R, right.
Figure 15 Nodal TRT reliability of degree and its relationship with nodal degree centrality for F-DOS-based networks. (a) Nodal TRT reliability was mapped in anatomical space after average across scanning time interval, network type and network membership because of no effects of these factors on nodal reliability. (b) Nodal degree centrality (AUCs) was also mapped in anatomical space which was averaged across subjects and factors of scanning time interval, network type and network membership. Trend lines were further obtained by linear least-square fit to reveal the relationship between nodal degree centrality and their corresponding reliability after with (d) and without (c) correcting for the effects of regional size. Of note, the full names of region's abbreviations were listed as in Table S1. TRT, test-retest; F-DOS, functional ROIs from Dosenbach et al. (2006, 2010); k, nodal degree; A, anterior; P, posterior; L, left; R, right.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 74 references

Research works printed but not linked: 1

  1. R Salvador, 2005, Undirected graphs of frequency-dependent functional connectivity in whole brain networks., Phil Trans R Soc B 360, 937–946

Cited by

Sorted by citations

Metadata

Authors
Wang, J.-H., Zuo, X.-N., Gohel, S., Milham, M. P., Biswal, B. B., & He, Y.
Affiliations
  • Chinese Academy of Sciences (CN)
  • University of Medicine and Dentistry of New Jersey (US)
  • NYU Langone Health (US)
Venue
PLoS ONE
Published
2011-07-19
Volume
6
Issue
7
Pages
e21976
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2020617938
Cited by
416
References
74
Citations by year
2026: 21 · 2025: 24 · 2024: 29 · 2023: 35 · 2022: 30 · 2021: 33 · 2020: 27 · 2019: 19 · 2018: 27 · 2017: 36 · 2016: 19 · 2015: 40 · 2014: 27 · 2013: 25 · 2012: 19
FWCI
12.50
Type
journal article
Language
en
Open access
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Institutional Repository of Institute of Psychology, Chinese Academy of Sciences (Institute of Psychology, Chinese Academy of Sciences) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • Institutional Repository of Institute of Psychology, Chinese Academy of Sciences (Institute of Psychology, Chinese Academy of Sciences) (submittedVersion, cc-by-nc-sa)
Funders
National Natural Science Foundation of China (30870667, 81030028)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Complex Network Analysis Techniques
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced Neuroimaging Techniques and Applications
Keywords
test-retest reliability, functional brain networks, resting-state functional connectivity, graph theory, hub regions, association cortex, noise sensitivity, network topology
MeSH terms
Adult, Brain, Computer Simulation, Female, Humans, Magnetic Resonance Imaging, Male, Models, Neurological, Nerve Net, Rest, Statistics as Topic, Time Factors, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review