Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Background: Story recall is a simple and sensitive cognitive test that is commonly used to measure changes in episodic memory function in early Alzheimer disease (AD). Recent advances in digital technology and natural language processing methods make this test a candidate for automated administration and scoring. Multiple parallel test stimuli are required for higher-frequency disease monitoring. Objective: This study aims to develop and validate a remote and fully automated story recall task, suitable for longitudinal assessment, in a population of older adults with and without mild cognitive impairment (MCI) or mild AD. Methods: The “Amyloid Prediction in Early Stage Alzheimer’s disease” (AMYPRED) studies recruited participants in the United Kingdom (AMYPRED-UK: NCT04828122) and the United States (AMYPRED-US: NCT04928976). Participants were asked to complete optional daily self-administered assessments remotely on their smart devices over 7 to 8 days. Assessments included immediate and delayed recall of 3 stories from the Automatic Story Recall Task (ASRT), a test with multiple parallel stimuli (18 short stories and 18 long stories) balanced for key linguistic and discourse metrics. Verbal responses were recorded and securely transferred from participants’ personal devices and automatically transcribed and scored using text similarity metrics between the source text and retelling to derive a generalized match score. Group differences in adherence and task performance were examined using logistic and linear mixed models, respectively. Correlational analysis examined parallel-forms reliability of ASRTs and convergent validity with cognitive tests (Logical Memory Test and Preclinical Alzheimer’s Cognitive Composite with semantic processing). Acceptability and usability data were obtained using a remotely administered questionnaire. Results: Of the 200 participants recruited in the AMYPRED studies, 151 (75.5%)—78 cognitively unimpaired (CU) and 73 MCI or mild AD—engaged in optional remote assessments. Adherence to daily assessment was moderate and did not decline over time but was higher in CU participants (ASRTs were completed each day by 73/106, 68.9% participants with MCI or mild AD and 78/94, 83% CU participants). Participants reported favorable task usability: infrequent technical problems, easy use of the app, and a broad interest in the tasks. Task performance improved modestly across the week and was better for immediate recall. The generalized match scores were lower in participants with MCI or mild AD (Cohen d=1.54). Parallel-forms reliability of ASRT stories was moderate to strong for immediate recall (mean rho 0.73, range 0.56-0.88) and delayed recall (mean rho=0.73, range=0.54-0.86). The ASRTs showed moderate convergent validity with established cognitive tests. Conclusions: The unsupervised, self-administered ASRT task is sensitive to cognitive impairments in MCI and mild AD. The task showed good usability, high parallel-forms reliability, and high convergent validity with established cognitive tests. Remote, low-cost, low-burden, and automatically scored speech assessments could support diagnostic screening, health care, and treatment monitoring.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kok, J. S., Berg, I. J., & Scherder, E. J. A. (2013) · Dementia and Geriatric Cognitive Disorders Extra · doi:10.1159/000353441 · Cited by 20,589
The effects of acute exercise on cognitive performance: a meta-analysis
Chang, Y. K., Labban, J. D., Gapin, J. I., & Etnier, J. L. (2012) · Brain Research · doi:10.1016/j.brainres.2012.02.068 · Cited by 1,930 · cited 1×
“More importantly, test performance is measurably influenced by a range of state factors such as sleep [4], exercise [5], mood [6], and stress [7].”
BackgroundComparing effect sizes in follow-up studies: ROC Area, Cohen's d, and r.
Rice, M. E., & Harris, G. T. (2005) · Law and Human Behavior · doi:10.1007/s10979-005-6832-7 · Cited by 1,608 · cited 2×
“86, based on previously published equivalence tables (31).”
BackgroundThe Preclinical Alzheimer Cognitive Composite: Measuring Amyloid-Related Decline
Donohue, M. C., Sperling, R. A., Salmon, D. P., Rentz, D. M., Raman, R., Thomas, R. G., Weiner, M., & Aisen, P. S. (2014) · JAMA Neurology · doi:10.1001/jamaneurol.2014.803 · Cited by 860
COVID-19 mortality risk for older men and women
Yanez, N. D., Weiss, N. S., Romand, J.-A., & Treggiari, M. M. (2020) · BMC Public Health · doi:10.1186/s12889-020-09826-8 · Cited by 616 · cited 1×
“The continued drive toward remote assessment may be particularly important in older adults who are at a substantially increased risk in the pandemic [22].”
BackgroundDatasets: A Community Library for Natural Language Processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.emnlp-demo.21 · Cited by 343 · cited 1×
“The word error rate (WER) of the automatic transcript was calculated using the HuggingFace package [33], as the average number of errors per manual transcript word.”
MethodOptimizing the preclinical Alzheimer's cognitive composite with semantic processing: The PACC5
Papp, K. V., Rentz, D. M., Orlovsky, I., Sperling, R. A., & Mormino, E. C. (2017) · Alzheimer's & Dementia: Translational Research & Clinical Interventions · doi:10.1016/j.trci.2017.10.004 · Cited by 303
Self-citation
Cited by
Polk, S. E., Öhman, F., Hassenstab, J., König, A., Papp, K. V., Schöll, M., & Berron, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01583-5 · Cited by 73
Self-citationA remote speech-based AI system to screen for early Alzheimer's disease via smartphones
Fristed, E., Skirrow, C., Meszaros, M., Lenain, R., Meepegama, U., Cappa, S., Aarsland, D., & Weston, J. (2022) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1002/dad2.12366 · Cited by 47
Self-citationFristed, E., Skirrow, C., Meszaros, M., Lenain, R., Meepegama, U., Papp, K. V., Ropacki, M., & Weston, J. (2022) · Brain Communications · doi:10.1093/braincomms/fcac231 · Cited by 29
Self-citation
Metadata
- Authors
- Skirrow, C., Meszaros, M., Meepegama, U., Lenain, R., Papp, K. V., Weston, J., & Fristed, E.
- Venue
- JMIR Aging
- Publisher
- JMIR Publications Inc.
- Published
- 2022-09-30
- Volume
- 5
- Issue
- 3
- Pages
- e37090
- ISSN
- 2561-7605 (ISSN-L)
- OpenAlex
- W4298087975
- Cited by
- 35
- References
- 7
- FWCI
- 4.07
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study. https://app.cochat.ai/papers/10.2196/37090
- The paper
- Skirrow, C., Meszaros, M., Meepegama, U., Lenain, R., Papp, K. V., Weston, J., & Fristed, E. (2022). Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study. JMIR Aging. https://doi.org/10.2196/37090
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.