Beyond accuracy: assessing the information quality of large language models for systematic literature reviews

Abstract

Systematic literature reviews (SLRs) are increasingly supported by large language models (LLMs), but existing evaluations rarely go beyond accuracy and overlook broader information quality dimensions. This study provides an Information Quality (IQ)-driven evaluation of LLM support for SLR tasks in communication research, focused on diagnosing failure modes and their implications for responsible use. Using 1,325 articles on cross-platform social media research, we design a seven-step evaluation pipeline that mirrors core SLR decisions and maps them onto contextual, intrinsic, and representational IQ dimensions. Two leading LLMs, GPT-4o and Claude 3.7 Sonnet, classify or extract information about article type and empirical status (empirical vs. non-empirical), data sources (e.g., media types and platform names), data collection methods, and the number and names of social media platforms. Validated human-coded annotations serve as ground truth, and mixed-method insights are reported from both quantitative performance analysis and qualitative error analysis. Both models achieve near-perfect accuracy for document type identification, yet their performance on tasks requiring boundary-setting judgments, methodological interpretation, or exact structured extraction is lower and more variable. Qualitative error analysis reveals that both models rely on surface-level cues rather than holistic reasoning; they apply narrow rules mechanically and fail to reconcile conflicting signals across different parts of a paper. The two models also exhibit task-specific error profiles rather than stable opposite decision biases: for example, GPT-4o is more conservative in digital trace detection but more expansive in platform-count extraction, while Claude 3.7 Sonnet shows stronger performance on grounded platform extraction. We conclude that current LLMs can reliably assist, but not replace, human judgment in SLRs. Effective deployment requires making model reasoning visible, designing prompts that account for the full range of methods in a field, and using models with complementary task-specific error profiles to flag ambiguous cases for human review.