Mega‐analysis methods in ENIGMA : The experience of the generalized anxiety disorder working group
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
The ENIGMA group on Generalized Anxiety Disorder (ENIGMA‐Anxiety/GAD) is part of a broader effort to investigate anxiety disorders using imaging and genetic data across multiple sites worldwide. The group is actively conducting a mega‐analysis of a large number of brain structural scans. In this process, the group was confronted with many methodological challenges related to study planning and implementation, between‐country transfer of subject‐level data, quality control of a considerable amount of imaging data, and choices related to statistical methods and efficient use of resources. This report summarizes the background information and rationale for the various methodological decisions, as well as the approach taken to implement them. The goal is to document the approach and help guide other research groups working with large brain imaging data sets as they develop their own analytic pipelines for mega‐analyses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
SMITH, S., & NICHOLS, T. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.03.061 · Cited by 5,250
A multi-modal parcellation of human cerebral cortex
Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C. F., Jenkinson, M., et al. (2016) · Nature · doi:10.1038/nature18933 · Cited by 5,111 · cited 1×
“…parcellation schemes exist (Craddock, James, Holtzheimer, Hu, & Mayberg, 2012; Desikan et al., 2006; Destrieux, Fischl, Dale, & Halgren, 2010; Glasser et al., 2016; Ji et al., 2019; Maggioni, Tana, Arrigoni, Zucca, & Bianchi, 2014; Power et al., 2011; Schaefer et al., 2018; Tzourio-Mazoyer…”
Functional Network Organization of the Human Brain
Power, J. D., Cohen, A. L., Nelson, S. M., Wig, G. S., Barnes, K. A., Church, J. A., Vogel, A. C., Laumann, T. O., Miezin, F. M., Schlaggar, B. L., et al. (2011) · Neuron · doi:10.1016/j.neuron.2011.09.006 · Cited by 4,124 · cited 1×
“…et al., 2006; Destrieux, Fischl, Dale, & Halgren, 2010; Glasser et al., 2016; Ji et al., 2019; Maggioni, Tana, Arrigoni, Zucca, & Bianchi, 2014; Power et al., 2011; Schaefer et al., 2018; Tzourio-Mazoyer et al., 2002; Yeo et al., 2011), based on various macroscopic, microscopic, and functional…”
Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI
Schaefer, A., Kong, R., Gordon, E. M., Laumann, T. O., Zuo, X.-N., Holmes, A. J., Eickhoff, S. B., & Yeo, B. T. T. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhx179 · Cited by 3,657
Permutation inference for the general linear model
Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.01.060 · Cited by 3,563
Self-citationAutomatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature
Destrieux, C., Fischl, B., Dale, A., & Halgren, E. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.06.010 · Cited by 2,956
Tackling the widespread and critical impact of batch effects in high-throughput data
Leek, J. T., Scharpf, R. B., Bravo, H. C., Simcha, D., Langmead, B., Johnson, W. E., Geman, D., Baggerly, K., & Irizarry, R. A. (2010) · Nature Reviews Genetics · doi:10.1038/nrg2825 · Cited by 2,196
Multimodal population brain imaging in the UK Biobank prospective epidemiological study
Miller, K. L., Alfaro-Almagro, F., Bangerter, N. K., Thomas, D. L., Yacoub, E., Xu, J., Bartsch, A. J., Jbabdi, S., Sotiropoulos, S. N., Andersson, J. L. R., et al. (2016) · Nature Neuroscience · doi:10.1038/nn.4393 · Cited by 2,020 · cited 1×
“The UK Biobank, however, benefits from the fact that data collection is limited to only three sites, all of which use identical equipment (Miller et al., 2016).”
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Limitations of the Application of Fourfold Table Analysis to Hospital Data
Berkson, J. (1946) · Biometrics Bulletin · doi:10.2307/3002000 · Cited by 1,810 · cited 1×
“…data across participants and in the variation of the independent variable (i.e., it is a collider), adjustment for the undesired effect induces a false association (Berkson, 1946; Luque-Fernandez et al., 2019; Pearl, 2009, chapter 6), which can happen in either direction (positive or negative).”
BackgroundGorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., Flandin, G., Ghosh, S. S., Glatard, T., Halchenko, Y. O., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.44 · Cited by 1,775
Meta-analysis of individual participant data: rationale, conduct, and reporting
Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010) · BMJ · doi:10.1136/bmj.c221 · Cited by 1,701 · cited 1×
“Multiple studies have compared meta- and mega-analysis (Belias, Rovers, Reitsma, Debray, & IntHout, 2019; Riley et al., 2010; Simmonds, Stewart, & Stewart, 2015), suggesting superiority of megaanalyses with IPD when compared to meta-analyses in terms of higher statistical power and acceptable false…”
ResultPlasticity in gray and white: neuroimaging changes in brain structure during learning
Zatorre, R. J., Fields, R. D., & Johansen-Berg, H. (2012) · Nature Neuroscience · doi:10.1038/nn.3045 · Cited by 1,685
A whole brain fMRI atlas generated via spatially constrained spectral clustering
Craddock, R. C., James, G. A., Holtzheimer, P. E., Hu, X. P., & Mayberg, H. S. (2012) · Human Brain Mapping · doi:10.1002/hbm.21333 · Cited by 1,636
Harmonization of cortical thickness measurements across scanners and sites
Fortin, J.-P., Cullen, N., Sheline, Y. I., Taylor, W. D., Aselcioglu, I., Cook, P. A., Adams, P., Cooper, C., Fava, M., McGrath, P. J., et al. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.11.024 · Cited by 1,580 · cited 2×
“, 2017), cortical thickness measures (Fortin et al., 2018), rest and task-based functional MRI (Nielson et al.”
BackgroundMethodHan, X., Jovicich, J., Salat, D., van der Kouwe, A., Quinn, B., Czanner, S., Busa, E., Pacheco, J., Albert, M., Killiany, R., et al. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.02.051 · Cited by 1,508
Nosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Harmonization of multi-site diffusion tensor imaging data
Fortin, J.-P., Parker, D., Tunç, B., Watanabe, T., Elliott, M. A., Ruparel, K., Roalf, D. R., Satterthwaite, T. D., Gur, R. C., Gur, R. E., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.08.047 · Cited by 1,261 · cited 1×
“In brain imaging, the approach has been an effective method to harmonize diffusion tensor imaging data (Fortin et al., 2017), cortical thickness measures (Fortin et al., 2018), rest and task-based functional MRI (Nielson et al., 2018), and functional connectivity (Yu et al., 2018).”
MethodSelf-citationfor the ENIGMA-Major Depressive Disorder Working Group, Schmaal, L., Veltman, D. J., van Erp, T. G. M., Sämann, P. G., Frodl, T., Jahanshad, N., Loehrer, E., Tiemeier, H., Hofman, A., et al. (2016) · Molecular Psychiatry · doi:10.1038/mp.2015.69 · Cited by 1,172
Self-citationAn Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Ciric, R., Wolf, D. H., Power, J. D., Roalf, D. R., Baum, G. L., Ruparel, K., Shinohara, R. T., Elliott, M. A., Eickhoff, S. B., Davatzikos, C., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.03.020 · Cited by 1,042 · cited 1×
“In addition, for functional MRI, some sources of noise, such as movement and respiration, result in artifactual global signal changes, and so removal of the global signal is also an effective means of reducing artifacts (Ciric et al., 2017).”
BackgroundSelf-citationBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 1×
“For example, preregistration reduces problems that follow when negative results remain unreported (Rosenthal, 1979; Sterling, 1959), reduces the chances of selective reporting (Macleod et al., 2014) and maximizes transparency in analytic approaches, thereby facilitating replication (Simmons,…”
BackgroundMRIQC: Advancing the automatic prediction of image quality in MRI from unseen sites
Esteban, O., Birman, D., Schaer, M., Koyejo, O. O., Poldrack, R. A., & Gorgolewski, K. J. (2017) · PLOS ONE · doi:10.1371/journal.pone.0184661 · Cited by 920
The genetic architecture of the human cerebral cortex
Grasby, K. L., Jahanshad, N., Painter, J. N., Colodro-Conde, L., Bralten, J., Hibar, D. P., Lind, P. A., Pizzagalli, F., Ching, C. R. K., McMahon, M. A. B., et al. (2020) · Science · doi:10.1126/science.aay6690 · Cited by 876 · cited 1×
“…ENIGMA meta-analysis using data from 51,665 subjects identified 187 loci influencing cortical surface area and 12 others influencing thickness (Grasby et al., 2020); in another example, the recent UK Biobank analysis used 3,144 imagingderived traits (Elliott et al., 2018) to find 148…”
MethodSelf-citationCommon genetic variants influence human subcortical brain structures
Initiative, T. A. D. N., Hibar, D. P., The CHARGE Consortium, EPIGEN, IMAGEN, SYS, Stein, J. L., Renteria, M. E., Arias-Vasquez, A., Desrivières, S., et al. (2015) · Nature · doi:10.1038/nature14101 · Cited by 858
Self-citationThe ENIGMA Consortium: large-scale collaborative analyses of neuroimaging and genetic data
the Alzheimer’s Disease Neuroimaging Initiative, E. C., Thompson, P. M., Stein, J. L., Medland, S. E., Hibar, D. P., Vasquez, A. A., Renteria, M. E., Toro, R., Jahanshad, N., Schumann, G., et al. (2014) · Brain Imaging and Behavior · doi:10.1007/s11682-013-9269-5 · Cited by 838
Self-citationGenome-wide association studies of brain imaging phenotypes in UK Biobank
Elliott, L. T., Sharp, K., Alfaro-Almagro, F., Shi, S., Miller, K. L., Douaud, G., Marchini, J., & Smith, S. M. (2018) · Nature · doi:10.1038/s41586-018-0571-7 · Cited by 834
Ten simple rules for neuroimaging meta-analysis
Müller, V. I., Cieslik, E. C., Laird, A. R., Fox, P. T., Radua, J., Mataix-Cols, D., Tench, C. R., Yarkoni, T., Nichols, T. E., Turkeltaub, P. E., et al. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.11.012 · Cited by 805 · cited 1×
“…p-values or test statistics, tables with coordinates in relation to some standard (but not always the same) stereotaxic space, and different representations of the brain (volume-based or surface-based; Fox, Lancaster, Laird, & Eickhoff, 2014; Müller et al., 2018; Tahmasian et al., 2019).”
BackgroundA paradox in the interpretation of group comparisons.
Lord, F. M. (1967) · Psychological Bulletin · doi:10.1037/h0025105 · Cited by 627
Statistically controlling for confounding constructs is harder than you think
Westfall, J., & Yarkoni, T. (2016) · PLOS ONE · doi:10.1371/journal.pone.0152719 · Cited by 547 · cited 1×
“Moreover, controlling for poorly reliable measures may not completely remove their putative effects, leading to false conclusions about effects (J. Westfall & Yarkoni, 2016).”
BackgroundDickersin, K. (2003) · JAMA · doi:10.1001/jama.290.4.516 · Cited by 512
Jahanshad, N., Kochunov, P. V., Sprooten, E., Mandl, R. C., Nichols, T. E., Almasy, L., Blangero, J., Brouwer, R. M., Curran, J. E., de Zubicaray, G. I., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.04.061 · Cited by 393 · cited 1×
“Diffusion-weighted imaging also allows measurements that are robust to variations on processing pipelines, and workflows for ENIGMA have been developed (Jahanshad et al., 2013; Kochunov et al., 2015).”
MethodSelf-citationHuman brain mapping: A systematic comparison of parcellation methods for the human cerebral cortex
Arslan, S., Ktena, S. I., Makropoulos, A., Robinson, E. C., Rueckert, D., & Parisot, S. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.04.014 · Cited by 374 · cited 1×
“…Schaefer et al., 2018; Tzourio-Mazoyer et al., 2002; Yeo et al., 2011), based on various macroscopic, microscopic, and functional aspects of the brain, none of which is clearly superior to others for all possible investigations (Arslan et al., 2018; Brett, Johnsrude, & Owen, 2002; Messé, 2019).”
BackgroundGorgolewski, K. J., Alfaro-Almagro, F., Auer, T., Bellec, P., Capotă, M., Chakravarty, M. M., Churchill, N. W., Cohen, A. L., Craddock, R. C., Devenyi, G. A., et al. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005209 · Cited by 311
Novel genetic loci associated with hippocampal volume
Hibar, D. P., Adams, H. H. H., Jahanshad, N., Chauhan, G., Stein, J. L., Hofer, E., Renteria, M. E., Bis, J. C., Arias-Vasquez, A., Ikram, M. K., et al. (2017) · Nature Communications · doi:10.1038/ncomms13624 · Cited by 299
Self-citationKochunov, P., Jahanshad, N., Marcus, D., Winkler, A., Sprooten, E., Nichols, T. E., Wright, S. N., Hong, L. E., Patel, B., Behrens, T., et al. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.02.050 · Cited by 215 · cited 1×
“Diffusion-weighted imaging also allows measurements that are robust to variations on processing pipelines, and workflows for ENIGMA have been developed (Jahanshad et al., 2013; Kochunov et al., 2015).”
MethodSelf-citationIdentification of Anonymous MRI Research Participants with Face-Recognition Software
Schwarz, C. G., Kremers, W. K., Therneau, T. M., Sharp, R. R., Gunter, J. L., Vemuri, P., Arani, A., Spychalla, A. J., Kantarci, K., Knopman, D. S., et al. (2019) · New England Journal of Medicine · doi:10.1056/nejmc1908881 · Cited by 215 · cited 1×
“Such reidentification, however, has recently been demonstrated to be feasible (Schwarz et al., 2019), which now renders defacing mandatory for publicly available data.”
BackgroundD. Chambers, C., 1 Cardiff University Brain Research Imaging Centre, S. O. P., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014) · AIMS Neuroscience · doi:10.3934/neuroscience.2014.1.4 · Cited by 213
Reporting of conflicts of interest in meta-analyses of trials of pharmacological treatments
Roseman, M. (2011) · JAMA · doi:10.1001/jama.2011.257 · Cited by 145
Obscuring Surface Anatomy in Volumetric Imaging Data
Milchenko, M., & Marcus, D. (2013) · Neuroinformatics · doi:10.1007/s12021-012-9160-3 · Cited by 145
Kochunov, P., Jahanshad, N., Sprooten, E., Nichols, T. E., Mandl, R. C., Almasy, L., Booth, T., Brouwer, R. M., Curran, J. E., de Zubicaray, G. I., et al. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.03.033 · Cited by 133 · cited 1×
“In the context of ENIGMA, comparisons have likewise tended to favor mega-analyses (Boedhoe et al., 2019; Kochunov et al., 2014; Koshiyama et al., 2020).”
MethodSelf-citationPractical recommendations to conduct a neuroimaging meta-analysis for neuropsychiatric disorders
Tahmasian, M., Sepehry, A. A., Samea, F., Khodadadifar, T., Soltaninejad, Z., Javaheripour, N., Khazaie, H., Zarei, M., Eickhoff, S. B., & Eickhoff, C. R. (2019) · Human Brain Mapping · doi:10.1002/hbm.24746 · Cited by 126
A comparison of FreeSurfer-generated data with and without manual intervention
McCarthy, C. S., Ramprashad, A., Thompson, C., Botti, J.-A., Coman, I. L., & Kates, W. R. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00379 · Cited by 122
Everything Matters: The ReproNim Perspective on Reproducible Neuroimaging
Kennedy, D. N., Abraham, S. A., Bates, J. F., Crowley, A., Ghosh, S., Gillespie, T., Goncalves, M., Grethe, J. S., Halchenko, Y. O., Hanke, M., et al. (2019) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2019.00001 · Cited by 113
Zavaliangos-Petropulu, A., Nir, T. M., Thomopoulos, S. I., Reid, R. I., Bernstein, M. A., Borowski, B., Jack Jr., C. R., Weiner, M. W., Jahanshad, N., & Thompson, P. M. (2019) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2019.00002 · Cited by 101 · cited 1×
“ComBat has been used in ENIGMA studies (Hatton et al., in press; Villalón-Reina et al., in press), although it has been argued that it leads to similar results as random effects linear regression (Zavaliangos-Petropulu et al., 2019).”
ResultSelf-citationRefacing: Reconstructing Anonymized Facial Features Using GANS
Abramian, D., & Eklund, A. (2019) · 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) · doi:10.1109/isbi.2019.8759515 · Cited by 63
Adhikari, B. M., Jahanshad, N., Shukla, D., Turner, J., Grotegerd, D., Dannlowski, U., Kugel, H., Engelen, J., Dietsche, B., Krug, A., et al. (2019) · Brain Imaging and Behavior · doi:10.1007/s11682-018-9941-x · Cited by 53 · cited 1×
“The report is produced by the tool MRIQC, available, along with documentation that details all the metrics (many more than shown in the figure), at https:// mriqc.readthedocs.io functional MRI processing pipeline was proposed recently for use in ENIGMA meta-analyses (Adhikari et al., 2018).”
MethodSelf-citation
Cited by
Cardoner, N., Andero, R., Cano, M., Marin-Blasco, I., Porta-Casteràs, D., Serra-Blasco, M., Via, E., Vicent-Gil, M., & Portella, M. J. (2024) · Current Neuropharmacology · doi:10.2174/1570159x21666230703091435 · Cited by 45
“Mega-analysis methods [291] are now used to evaluate the specificity of structural alterations in stress-related disorders and hopefully will help reveal the potential relationship with core dimensions of psychopathology, rather than hermetic categories.”
MethodSelf-citationGroenewold, N. A., Bas-Hoogendam, J. M., Amod, A. R., Laansma, M. A., Van Velzen, L. S., Aghajani, M., Hilbert, K., Oh, H., Salas, R., Jackowski, A. P., et al. (2023) · Molecular Psychiatry · doi:10.1038/s41380-022-01933-9 · Cited by 36
Self-citationJahanshad, N., Lenzini, P., & Bijsterbosch, J. (2025) · Neuropsychopharmacology · doi:10.1038/s41386-024-01938-8 · Cited by 12
“This distributed analysis alternative to data sharing has been very successfully employed in ENIGMA working groups [46, 47].”
MethodSelf-citation
Metadata
- Authors
- Zugman, A., Harrewijn, A., Cardinale, E. M., Zwiebel, H., Freitag, G. F., Werwath, K. E., Bas‐Hoogendam, J. M., Groenewold, N. A., Aghajani, M., Hilbert, K., Cardoner, N., Porta‐Casteràs, D., Gosnell, S., Salas, R., Blair, K. S., Blair, J. R., Hammoud, M. Z., Milad, M., Burkhouse, K., Phan, K. L., Schroeder, H. K., Strawn, J. R., Beesdo‐Baum, K., Thomopoulos, S. I., Grabe, H. J., Van der Auwera, S., Wittfeld, K., Nielsen, J. A., Buckner, R., Smoller, J. W., Mwangi, B., Soares, J. C., Wu, M., Zunta‐Soares, G. B., Jackowski, A. P., Pan, P. M., Salum, G. A., Assaf, M., Diefenbach, G. J., Brambilla, P., Maggioni, E., Hofmann, D., Straube, T., Andreescu, C., Berta, R., Tamburo, E., Price, R., Manfro, G. G., Critchley, H. D., Makovac, E., Mancini, M., Meeten, F., Ottaviani, C., Agosta, F., Canu, E., Cividini, C., Filippi, M., Kostić, M., Munjiza, A., Filippi, C. A., Leibenluft, E., Alberton, B. A. V., Balderston, N. L., Ernst, M., Grillon, C., Mujica‐Parodi, L. R., van Nieuwenhuizen, H., Fonzo, G. A., Paulus, M. P., Stein, M. B., Gur, R. E., Gur, R. C., Kaczkurkin, A. N., Larsen, B., Satterthwaite, T. D., Harper, J., Myers, M., Perino, M. T., Yu, Q., Sylvester, C. M., Veltman, D. J., Lueken, U., Van der Wee, N. J. A., Stein, D. J., Jahanshad, N., Thompson, P. M., Pine, D. S., & Winkler, A. M.
- Venue
- Human Brain Mapping
- Publisher
- Wiley
- Published
- 2020-06-29
- Volume
- 43
- Issue
- 1
- Pages
- 255-277
- ISSN
- 1065-9471 (ISSN-L)
- 1097-0193
- OpenAlex
- W3037428841
- Cited by
- 80
- References
- 51
- FWCI
- 6.26
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.