Cited by 407Dementia and Cognitive Impairment ResearchAlzheimer's disease research and treatmentsMachine Learning in Healthcare

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.

Predicting Alzheimer’s disease progression using multi-modal deep learning approach

Open access
Lee, Garam, Nho, Kwangsik, Kang, Byungkon, Sohn, Kyung-Ah, Kim, Dokyoon, for Alzheimer’s Disease Neuroimaging Initiative, Weiner, Michael W., Aisen, Paul, Petersen, Ronald, Jack, Clifford R., Jagust, William, Trojanowki, John Q., Toga, Arthur W., Beckett, Laurel, Green, Robert C., Saykin, Andrew J., Morris, John, Shaw, Leslie M., Khachaturian, Zaven, Sorensen, Greg, Carrillo, Maria, Kuller, Lew, Raichle, Marc, Paul, Steven, Davies, Peter, Fillit, Howard, Hefti, Franz, Holtzman, Davie, Mesulam, M. Marcel, Potter, William, Snyder, Peter, Montine, Tom, Thomas, Ronald G., Donohue, Michael, Walter, Sarah, Sather, Tamie, Jiminez, Gus, Balasubramanian, Archana B., Mason, Jennifer, Sim, Iris, Harvey, Danielle, Bernstein, Matthew, Fox, Nick, Thompson, Paul, Schuff, Norbert, DeCArli, Charles, Borowski, Bret, Gunter, Jeff, Senjem, Matt, Vemuri, Prashanthi, Jones, David, Kantarci, Kejal, Ward, Chad, Koeppe, Robert A., Foster, Norm, Reiman, Eric M., Chen, Kewei, Mathis, Chet, Landau, Susan, Cairns, Nigel J., Householder, Erin, Taylor-Reinwald, Lisa, Lee, Virginia, Korecka, Magdalena, Figurski, Michal, Crawford, Karen, Neu, Scott, Foroud, Tatiana M., Potkin, Steven, Shen, Li, Faber, Kelley, Kim, Sungeun, Tha, Lean, Frank, Richard, Hsiao, John, Kaye, Jeffrey, Quinn, Joseph, Silbert, Lisa, Lind, Betty, Carter, Raina, Dolen, Sara, Ances, Beau, Carroll, Maria, Creech, Mary L., Franklin, Erin, Mintun, Mark A., Schneider, Stacy, Oliver, Angela, Schneider, Lon S., Pawluczyk, Sonia, Beccera, Mauricio, Teodoro, Liberty, Spann, Bryan M., Brewer, James, Vanderswag, Helen, Fleisher, Adam, Marson, Daniel, Griffith, Randall, Clark, David, Geldmacher, David, Brockington, John, Roberson, Erik, Love, Marissa Natelson, Heidebrink, Judith L., Lord, Joanne L., Mason, Sara S., Albers, Colleen S., Knopman, David, Johnson, Kris, Grossman, Hillel, Mitsis, Effie, Shah, Raj C., deToledo-Morrell, Leyla, Doody, Rachelle S., Villanueva-Meyer, Javier, Chowdhury, Munir, Rountree, Susan, Dang, Mimi, Duara, Ranjan, Varon, Daniel, Greig, Maria T., Roberts, Peggy, Stern, Yaakov, Honig, Lawrence S., Bell, Karen L., Albert, Marilyn, Onyike, Chiadi, D’Agostino, Daniel, Kielb, Stephanie, Galvin, James E., Cerbone, Brittany, Michel, Christina A., Pogorelec, Dana M., Rusinek, Henry, de Leon, Mony J., Glodzik, Lidia, De Santi, Susan, Womack, Kyle, Mathews, Dana, Quiceno, Mary, Doraiswamy, P. Murali, Petrella, Jeffrey R., Borges-Neto, Salvador, Wong, Terence Z., Coleman, Edward, Levey, Allan I., Lah, James J., Cella, Janet S., Burns, Jeffrey M., Swerdlow, Russell H., Brooks, William M., Arnold, Steven E., Karlawish, Jason H., Wolk, David, Clark, Christopher M., Apostolova, Liana, Tingus, Kathleen, Woo, Ellen, Silverman, Daniel H. S., Lu, Po H., Bartzokis, George, Smith, Charles D., Jicha, Greg, Hardy, Peter, Sinha, Partha, Oates, Elizabeth, Conrad, Gary, Graff-Radford, Neill R., Parfitt, Francine, Kendall, Tracy, Johnson, Heather, Lopez, Oscar L., Oakley, MaryAnn, Simpson, Donna M., Farlow, Martin R., Hake, Ann Marie, Matthews, Brandy R., Brosch, Jared R., Herring, Scott, Hunt, Cynthia, Porsteinsson, Anton P., Goldstein, Bonnie S., Martin, Kim, Makino, Kelly M., Ismail, M. Saleem, Brand, Connie, Mulnard, Ruth A., Thai, Gaby, Mc-Adams-Ortiz, Catherine, van Dyck, Christopher H., Carson, Richard E., MacAvoy, Martha G., Varma, Pradeep, Chertkow, Howard, Bergman, Howard, Hosein, Chris, Black, Sandra, Stefanovic, Bojana, Caldwell, Curtis, Hsiung, Ging-Yuek Robin, Feldman, Howard, Mudge, Benita, Assaly, Michele, Finger, Elizabeth, Pasternack, Stephen, Rachisky, Irina, Trost, Dick, Kertesz, Andrew, Bernick, Charles, Munic, Donna, Lipowski, Kristine, Weintraub, Masandra, Bonakdarpour, Borna, Kerwin, Diana, Wu, Chuang-Kuo, Johnson, Nancy, Sadowsky, Carl, Villena, Teresa, Turner, Raymond Scott, Johnson, Kathleen, Reynolds, Brigid, Sperling, Reisa A., Johnson, Keith A., Marshall, Gad, Yesavage, Jerome, Taylor, Joy L., Lane, Barton, Rosen, Allyson, Tinklenberg, Jared, Sabbagh, Marwan N., Belden, Christine M., Jacobson, Sandra A., Sirrel, Sherye A., Kowall, Neil, Killiany, Ronald, Budson, Andrew E., Norbash, Alexander, Johnson, Patricia Lynn, Obisesan, Thomas O., Wolday, Saba, Allard, Joanne, Lerner, Alan, Ogrocki, Paula, Tatsuoka, Curtis, Fatica, Parianne, Fletcher, Evan, Maillard, Pauline, Olichney, John, Carmichael, Owen, Kittur, Smita, Borrie, Michael, Lee, T.-Y., Bartha, Rob, Johnson, Sterling, Asthana, Sanjay, Carlsson, Cynthia M., Preda, Adrian, Nguyen, Dana, Tariot, Pierre, Burke, Anna, Trncic, Nadira, Fleisher, Adam, Reeder, Stephanie, Bates, Vernice, Capote, Horacio, Rainka, Michelle, Scharre, Douglas W., Kataki, Maria, Adeli, Anahita, Zimmerman, Earl A., Celmins, Dzintra, Brown, Alice D., Pearlson, Godfrey D., Blank, Karen, Anderson, Karen, Flashman, Laura A., Seltzer, Marc, Hynes, Mary L., Santulli, Robert B., Sink, Kaycee M., Gordineer, Leslie, Williamson, Jeff D., Garg, Pradeep, Watkins, Franklin, Ott, Brian R., Querfurth, Henry, Tremont, Geoffrey, Salloway, Stephen, Malloy, Paul, Correia, Stephen, Rosen, Howard J., Miller, Bruce L., Perry, David, Mintzer, Jacobo, Spicer, Kenneth, Bachman, David, Finger, Elizabether, Pasternak, Stephen, Rachinsky, Irina, Rogers, John, Drost, Dick, Pomara, Nunzio, Hernando, Raymundo, Sarrael, Antero, Schultz, Susan K., Ponto, Laura L. Boles, Shim, Hyungsub, Smith, Karen Ekstam, Relkin, Norman, Chaing, Gloria, Lin, Michael, Ravdin, Lisa, Smith, Amanda, Raj, Balebail Ashok, Fargher, Kristin·2019·Scientific Reports

Abstract

Alzheimer's disease (AD) is a progressive neurodegenerative condition marked by a decline in cognitive functions with no validated disease modifying treatment. It is critical for timely treatment to detect AD in its earlier stage before clinical manifestation. Mild cognitive impairment (MCI) is an intermediate stage between cognitively normal older adults and AD. To predict conversion from MCI to probable AD, we applied a deep learning approach, multimodal recurrent neural network. We developed an integrative framework that combines not only cross-sectional neuroimaging biomarkers at baseline but also longitudinal cerebrospinal fluid (CSF) and cognitive performance biomarkers obtained from the Alzheimer's Disease Neuroimaging Initiative cohort (ADNI). The proposed framework integrated longitudinal multi-domain data. Our results showed that 1) our prediction model for MCI conversion to AD yielded up to 75% accuracy (area under the curve (AUC) = 0.83) when using only single modality of data separately; and 2) our prediction model achieved the best performance with 81% accuracy (AUC = 0.86) when incorporating longitudinal multi-domain data. A multi-modal deep learning approach has potential to identify persons at risk of developing AD who might benefit most from a clinical trial or as a stratification approach within clinical trials.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-018-37769-z

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 An example using longitudinal data for MCI conversion prediction. Contrary to the experiment with baseline visit data, longitudinal data of individuals in all stages (CN, MCI, and AD) was used for training a classifier. Then, portion of longitudinal data was taken to the classifier to predict AD progression after Δt.
Figure 1 An example using longitudinal data for MCI conversion prediction. Contrary to the experiment with baseline visit data, longitudinal data of individuals in all stages (CN, MCI, and AD) was used for training a classifier. Then, portion of longitudinal data was taken to the classifier to predict AD progression after Δt.CC BY · PubMed Central
Figure 2 The number of subjects available in demographic data, neuroimaging data, cognitive performance, and CSF biomarkers over ∆t.
Figure 2 The number of subjects available in demographic data, neuroimaging data, cognitive performance, and CSF biomarkers over ∆t.CC BY · PubMed Central
Figure 3 Predictive performances with “proposed”, “baseline”, and “single modal”. Abbreviations: COG: cognition performance biomarkers; CSF: cerebrospinal fluid biomarkers; NeuroImg: MRI biomarkers.
Figure 3 Predictive performances with “proposed”, “baseline”, and “single modal”. Abbreviations: COG: cognition performance biomarkers; CSF: cerebrospinal fluid biomarkers; NeuroImg: MRI biomarkers.CC BY · PubMed Central
Figure 4 ROC curves from the “proposed”, “baseline”, and each “single modal” method. (a) 6 month prediction. (b) 12 month prediction. (c) 18 month prediction. (d) 24 month prediction.
Figure 4 ROC curves from the “proposed”, “baseline”, and each “single modal” method. (a) 6 month prediction. (b) 12 month prediction. (c) 18 month prediction. (d) 24 month prediction.CC BY · PubMed Central
Figure 5 Illustration of recurrent neural network. RNN is composed of input, memory state, and output, each of which has a weight parameter to be learned for a given task. The memory state (blue box) takes the input and computes the output based on the memory state from the previous step and the current input (left). Since the RNN has a feedback loop, variable-length input and output sequence can be represented as an “unfolded” sequence (right).
Figure 5 Illustration of recurrent neural network. RNN is composed of input, memory state, and output, each of which has a weight parameter to be learned for a given task. The memory state (blue box) takes the input and computes the output based on the memory state from the previous step and the current input (left). Since the RNN has a feedback loop, variable-length input and output sequence can be represented as an “unfolded” sequence (right).CC BY · PubMed Central
Figure 6 Overview of the proposed method. Our proposed method contains multiple GRU components that accept each modality of the dataset. At the first training step (blue dashed rectangle), each GRU component takes both time series or non-time series data to produce fixed-size feature vectors. And then the vectors are concatenated to form an input for the final prediction in the second training step (red dashed rectangle).
Figure 6 Overview of the proposed method. Our proposed method contains multiple GRU components that accept each modality of the dataset. At the first training step (blue dashed rectangle), each GRU component takes both time series or non-time series data to produce fixed-size feature vectors. And then the vectors are concatenated to form an input for the final prediction in the second training step (red dashed rectangle).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 41 references

Research works printed but not linked: 4

  1. Brodersen, K. H., Ong, C. S., Stephan, K. E. & Buhmann, J. M. In Pattern recognition (ICPR), 2010 20th international conference on. 3121–3124 (IEEE).
  2. Deng, L., Hinton, G. & Kingsbury, B. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. 8599–8603 (IEEE).
  3. Tang, D., Qin, B. & Liu, T. In Proceedings of the 2015 conference on empirical methods in natural language processing. 1422–1432.
  4. Srivastava, N., Mansimov, E., Salakhudinov, R. Unsupervised learning of video representations using LSTMs. In: 2015 International Conference on Machine Learning. 843–852 (2015).

Cited by

Sorted by citations

Metadata

Authors
Lee, G., Nho, K., Kang, B., Sohn, K.-A., Kim, D., Initiative, F. A. D. N., Weiner, M. W., Aisen, P., Petersen, R., Jack, C. R., Jagust, W., Trojanowki, J. Q., Toga, A. W., Beckett, L., Green, R. C., Saykin, A. J., Morris, J., Shaw, L. M., Khachaturian, Z., Sorensen, G., Carrillo, M., Kuller, L., Raichle, M., Paul, S., Davies, P., Fillit, H., Hefti, F., Holtzman, D., Mesulam, M. M., Potter, W., Snyder, P., Montine, T., Thomas, R. G., Donohue, M., Walter, S., Sather, T., Jiminez, G., Balasubramanian, A. B., Mason, J., Sim, I., Harvey, D., Bernstein, M., Fox, N., Thompson, P., Schuff, N., DeCArli, C., Borowski, B., Gunter, J., Senjem, M., Vemuri, P., Jones, D., Kantarci, K., Ward, C., Koeppe, R. A., Foster, N., Reiman, E. M., Chen, K., Mathis, C., Landau, S., Cairns, N. J., Householder, E., Taylor-Reinwald, L., Lee, V., Korecka, M., Figurski, M., Crawford, K., Neu, S., Foroud, T. M., Potkin, S., Shen, L., Faber, K., Kim, S., Tha, L., Frank, R., Hsiao, J., Kaye, J., Quinn, J., Silbert, L., Lind, B., Carter, R., Dolen, S., Ances, B., Carroll, M., Creech, M. L., Franklin, E., Mintun, M. A., Schneider, S., Oliver, A., Schneider, L. S., Pawluczyk, S., Beccera, M., Teodoro, L., Spann, B. M., Brewer, J., Vanderswag, H., Fleisher, A., Marson, D., Griffith, R., Clark, D., Geldmacher, D., Brockington, J., Roberson, E., Love, M. N., Heidebrink, J. L., Lord, J. L., Mason, S. S., Albers, C. S., Knopman, D., Johnson, K., Grossman, H., Mitsis, E., Shah, R. C., deToledo-Morrell, L., Doody, R. S., Villanueva-Meyer, J., Chowdhury, M., Rountree, S., Dang, M., Duara, R., Varon, D., Greig, M. T., Roberts, P., Stern, Y., Honig, L. S., Bell, K. L., Albert, M., Onyike, C., D’Agostino, D., Kielb, S., Galvin, J. E., Cerbone, B., Michel, C. A., Pogorelec, D. M., Rusinek, H., de Leon, M. J., Glodzik, L., De Santi, S., Womack, K., Mathews, D., Quiceno, M., Doraiswamy, P. M., Petrella, J. R., Borges-Neto, S., Wong, T. Z., Coleman, E., Levey, A. I., Lah, J. J., Cella, J. S., Burns, J. M., Swerdlow, R. H., Brooks, W. M., Arnold, S. E., Karlawish, J. H., Wolk, D., Clark, C. M., Apostolova, L., Tingus, K., Woo, E., Silverman, D. H. S., Lu, P. H., Bartzokis, G., Smith, C. D., Jicha, G., Hardy, P., Sinha, P., Oates, E., Conrad, G., Graff-Radford, N. R., Parfitt, F., Kendall, T., Johnson, H., Lopez, O. L., Oakley, M., Simpson, D. M., Farlow, M. R., Hake, A. M., Matthews, B. R., Brosch, J. R., Herring, S., Hunt, C., Porsteinsson, A. P., Goldstein, B. S., Martin, K., Makino, K. M., Ismail, M. S., Brand, C., Mulnard, R. A., Thai, G., Mc-Adams-Ortiz, C., van Dyck, C. H., Carson, R. E., MacAvoy, M. G., Varma, P., Chertkow, H., Bergman, H., Hosein, C., Black, S., Stefanovic, B., Caldwell, C., Hsiung, G.-Y. R., Feldman, H., Mudge, B., Assaly, M., Finger, E., Pasternack, S., Rachisky, I., Trost, D., Kertesz, A., Bernick, C., Munic, D., Lipowski, K., Weintraub, M., Bonakdarpour, B., Kerwin, D., Wu, C.-K., Johnson, N., Sadowsky, C., Villena, T., Turner, R. S., Johnson, K., Reynolds, B., Sperling, R. A., Johnson, K. A., Marshall, G., Yesavage, J., Taylor, J. L., Lane, B., Rosen, A., Tinklenberg, J., Sabbagh, M. N., Belden, C. M., Jacobson, S. A., Sirrel, S. A., Kowall, N., Killiany, R., Budson, A. E., Norbash, A., Johnson, P. L., Obisesan, T. O., Wolday, S., Allard, J., Lerner, A., Ogrocki, P., Tatsuoka, C., Fatica, P., Fletcher, E., Maillard, P., Olichney, J., Carmichael, O., Kittur, S., Borrie, M., Lee, T.-Y., Bartha, R., Johnson, S., Asthana, S., Carlsson, C. M., Preda, A., Nguyen, D., Tariot, P., Burke, A., Trncic, N., Fleisher, A., Reeder, S., Bates, V., Capote, H., Rainka, M., Scharre, D. W., Kataki, M., Adeli, A., Zimmerman, E. A., Celmins, D., Brown, A. D., Pearlson, G. D., Blank, K., Anderson, K., Flashman, L. A., Seltzer, M., Hynes, M. L., Santulli, R. B., Sink, K. M., Gordineer, L., Williamson, J. D., Garg, P., Watkins, F., Ott, B. R., Querfurth, H., Tremont, G., Salloway, S., Malloy, P., Correia, S., Rosen, H. J., Miller, B. L., Perry, D., Mintzer, J., Spicer, K., Bachman, D., Finger, E., Pasternak, S., Rachinsky, I., Rogers, J., Drost, D., Pomara, N., Hernando, R., Sarrael, A., Schultz, S. K., Ponto, L. L. B., Shim, H., Smith, K. E., Relkin, N., Chaing, G., Lin, M., Ravdin, L., Smith, A., Raj, B. A., & Fargher, K.
Affiliations
  • Geisinger Neuroscience Institute (US)
  • Indiana University Indianapolis (US)
  • Ajou University (KR)
  • Pennsylvania State University (US)
  • University of California, San Francisco (US)
  • Mayo Clinic (US)
  • University of Pennsylvania (US)
  • University of Southern California (US)
  • Brigham and Women's Hospital (US)
  • Washington University in St. Louis (US)
  • Prevent Alzheimer’s Disease 2020 (US)
  • Siemens (Germany) (DE)
  • Alzheimer's Association (US)
  • University of Pittsburgh (US)
  • Albert Einstein College of Medicine (US)
  • Alzheimer's Drug Discovery Foundation (US)
  • Northwestern University (US)
  • Brown University (US)
  • University of Washington (US)
  • University of California San Diego (US)
  • University of London (GB)
  • University of California, Los Angeles (US)
  • University of Utah (US)
  • Banner Alzheimer’s Institute (US)
  • University of California, Irvine (US)
  • Indiana University Bloomington (US)
  • National Institute on Aging (US)
  • Oregon Health & Science University (US)
  • University of Alabama at Birmingham (US)
  • Johns Hopkins University (US)
  • University of California, Berkeley (US)
  • Icahn School of Medicine at Mount Sinai (US)
  • Rush University Medical Center (US)
  • Baylor College of Medicine (US)
  • Columbia University Irving Medical Center (US)
  • New York University (US)
  • Duke Medical Center (US)
  • Emory University (US)
  • University of Kansas Medical Center (US)
  • University of Kentucky (US)
  • University of Rochester Medicine (US)
  • Yale University (US)
  • Jewish General Hospital (CA)
  • Sunnybrook Health Science Centre (CA)
  • University of British Columbia (CA)
  • St. Joseph's Hospital (CA)
  • Cleveland Clinic (US)
  • Palm Beach Neurology (US)
  • Georgetown University Medical Center (US)
  • Stanford University (US)
  • Boston University (US)
  • Howard University (US)
  • Case Western Reserve University (US)
  • University of California, Davis (US)
  • St Joseph's Health Care (CA)
  • Dent Neurologic Institute (US)
  • The Ohio State University (US)
  • Albany Medical College (US)
  • Hartford Hospital (US)
  • Dartmouth–Hitchcock Medical Center (US)
  • Wake Forest University Health Sciences (US)
  • Medical University of South Carolina (US)
  • Nathan Kline Institute for Psychiatric Research (US)
  • University of Iowa (US)
  • Cornell University (US)
  • University of South Florida (US)
Corresponding author
  • Sohn, Kyung-Ah
  • Kim, Dokyoon
ORCID
Venue
Scientific Reports
Published
2019-02-13
Volume
9
Issue
1
Pages
1952-1952
ISSN
2045-2322 (ISSN-L)
OpenAlex
W2914209001
Cited by
406
References
41
Citations by year
2026: 35 · 2025: 59 · 2024: 84 · 2023: 66 · 2022: 54 · 2021: 56 · 2020: 38 · 2019: 13
FWCI
32.00
Type
journal article
Language
en
Open access
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Scholarship@Western (Western University) (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • Scholarship@Western (Western University) (submittedVersion, cc-by)
  • Scholarship@Western (Western University) (submittedVersion)
  • IUScholarWorks (Indiana University) (submittedVersion, cc-by-nc-nd)
  • PMC
Funders
  • U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (NLM R01 LM012535)
  • U.S. Department of Health & Human Services | NIH | National Institute on Aging (NIA R03 AG054936)
  • National Research Foundation of Korea (NRF-2016R1D1A1B03933875)
  • Pennsylvania Department of Health (SAP 4100070267)
Topics
  • Health Sciences › Medicine › Psychiatry and Mental health › Dementia and Cognitive Impairment Research
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Alzheimer's disease research and treatments
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
Alzheimer's disease, mild cognitive impairment, MCI conversion prediction, disease progression, multimodal deep learning, recurrent neural networks, neuroimaging biomarkers, cerebrospinal fluid biomarkers, ADNI, clinical trial stratification
MeSH terms
Deep Learning, Aged, Aged, 80 and over, Alzheimer Disease, Brain, Cognition, Cross-Sectional Studies, Female, Forecasting, Humans, Male, Peptide Fragments, Biomarkers, Amyloid beta-Peptides, tau Proteins, Disease Progression, Area Under Curve, Neuroimaging, Cognitive Dysfunction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review