Machine learning and deep learning systems for automated measurement of “advanced” theory of mind: Reliability and validity in children and adolescents.
Open accessAbstract
Understanding individual differences in theory of mind (ToM; the ability to attribute mental states to others) in middle childhood and adolescence hinges on the availability of robust and scalable measures. Open-ended response tasks yield valid indicators of ToM but are labor intensive and difficult to compare across studies. We examined the reliability and validity of new machine learning and deep learning neural network automated scoring systems for measuring ToM in children and adolescents. Two large samples of British children and adolescents aged between 7 and 13 years (Sample 1: N = 1,135, Mage = 10.22 years, SD = 1.45; Sample 2: N = 1,020, Mage = 10.36 years, SD = 1.27) completed the silent film and strange stories tasks. Teachers rated Sample 2 children's social competence with peers. A single latent-factor explained variation in performance on both the silent film and strange stories task (in Sample 1 and 2) and test performance was sensitive to age-related differences and individual differences within each age-group. A deep learning neural network automated scoring system trained on Sample 1 exhibited interrater reliability and measurement invariance with manual ratings in Sample 2. Validity of ratings from the automated scoring system was supported by unique positive associations between ToM and teacher-rated social competence. The results demonstrate that reliable and valid measures of ToM can be obtained using the new freely available deep learning neural network…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Interrater reliability for the automated coding of the Silent Film task ranges from .80 to .93, mean Krippendorff's α = .86.
“Interrater reliability for this automatic coding ranges from .80 to .93, mean Krippendorff ' s α = .86 (i.e., Devine et al., 2023).”
From Longitudinal development of theory of mind in adolescence and its associations with fiction reading experience. (2025) · cites this paper as Devine et al., 2023
Cross-sectional studies have shown age-related change in theory of mind ability in middle childhood and early adolescence.
“a number of cross-sectional studies have shown age-related change in ToM ability in middle childhood and early adolescence (Banerjee et al., 2011; Devine et al., 2023; Osterhaus et al., 2016; Peterson & Wellman, 2019).”
From Longitudinal development of theory of mind in adolescence and its associations with fiction reading experience. (2025) · cites this paper as Devine et al., 2023
Both theory of mind ability and reading experience have been linked to individual differences in verbal ability, gender, and socioeconomic status.
“Both ToM ability and reading experience have for instance been linked to individual differences in verbal ability, gender, and socioeconomic status (e.g., Devine et al., 2023; Devine & Hughes, 2018; McGeown et al., 2016; Milligan et al., 2007; van der Kleij et al., 2022).”
From Longitudinal development of theory of mind in adolescence and its associations with fiction reading experience. (2025) · cites this paper as Devine et al., 2023
The ToM measures were selected on the basis of their demonstrable reliability and validity.
“Our measures of ToM both consisted of watching a short video clip, where participants were asked to explain the target ' s intentions, beliefs, and emotions, and they were selected on the basis of their demonstrable reliability and validity (i.e., Devine & Hughes, 2013; Devine et al., 2023).”
From Longitudinal development of theory of mind in adolescence and its associations with fiction reading experience. (2025) · cites this paper as Devine et al., 2023
Cites
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Effect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860 · see citing works
Theory of mind and prosocial behavior in childhood: A meta-analytic review.
Imuta, K., Henry, J. D., Slaughter, V., Selcuk, B., & Ruffman, T. (2016) · Developmental Psychology · doi:10.1037/dev0000140 · Cited by 474 · see citing works
Cotter, J., Granger, K., Backx, R., Hobbs, M., Looi, C. Y., & Barnett, J. H. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.11.014 · Cited by 369 · see citing works
Testing measurement invariance in longitudinal data with ordered-categorical measures.
Liu, Y., Millsap, R. E., West, S. G., Tein, J.-Y., Tanaka, R., & Grimm, K. J. (2017) · Psychological Methods · doi:10.1037/met0000075 · Cited by 351 · see citing works
Meta-Analysis of Theory of Mind and Peer Popularity in the Preschool and Early School Years
Slaughter, V., Imuta, K., Peterson, C. C., & Henry, J. D. (2015) · Child Development · doi:10.1111/cdev.12372 · Cited by 324 · see citing works
Rødgaard, E.-M., Jensen, K., Vergnes, J.-N., Soulières, I., & Mottron, L. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.1956 · Cited by 185 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
van der Kleij, S. W., Devine, R. T., Shapiro, L. R., Ricketts, J., & Apperly, I. (2025) · Developmental Psychology · doi:10.1037/dev0001965 · Cited by 7 · see citing works
Self-citation
- Authors
- Devine, R. T., Kovatchev, V., Grumley Traynor, I., Smith, P., & Lee, M.
- Affiliations
- University of Birmingham (GB)
- Venue
- Psychological Assessment
- Published
- 2023-02
- Volume
- 35
- Issue
- 2
- Pages
- 165-177
- ISSN
- 1040-3590 (ISSN-L)
- 1939-134X
- OpenAlex
- W4317779823
- Cited by
- 12
- References
- 7
- Citations by year
- 2026: 5 · 2025: 2 · 2024: 2 · 2023: 2 · 2020: 1
- FWCI
- 5.34
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Psychological Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- Wellcome Trust (215006/Z/18/Z)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Keywords
- theory of mind, automated grading, deep neural network, machine learning, inter-rater reliability, measurement invariance, social competence, children, adolescents, middle childhood, individual differences
- MeSH terms
- Deep Learning, Adolescent, Child, Humans, Motion Pictures, Reproducibility of Results, Theory of Mind, Social Skills
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
