CoChat

Let’s read the log: root cause analysis of railway test execution logs with large language models

Open access

The DOI matches its Crossref record; 45 of the 49 references it cites were found in a registry or an index; 9 of the 15 sources its claims cite were checked for support.

Abstract

Software quality assurance is pivotal in safety-critical domains such as railway systems, where failures could have catastrophic consequences. In this context, the train control and management system, which enables communication and control across multiple subsystems (such as doors and information panels) within a modern train, and its software must undergo rigorous validation. Alstom Rail Sweden AB employs a digital twin infrastructure to simulate and validate train control and management system software. While this setup improves system-level testing, root-cause analysis of test failures remains a manual, time-consuming bottleneck. In this study, we explore the potential of large language models to automate root cause analysis by interpreting test execution logs generated during digital twin-based testing. We benchmark nine state-of-the-art large language models: Aion−1.0, DeepSeek R1, DeepSeek V3 0324, Mistral Small 3.1 24B, GPT o3-mini, Gemini 2.5 Pro Experimental, QwQ 32B, Gemini 2.0 Flash Experimental, and Amazon Nova 2 Lite using zero-shot chain-of-thought prompting to assess their ability to reason about fault patterns in real-world industrial test execution logs. The logs were sourced from Alstom’s digital twin-based testing environment and captured complex operational behaviour typical of embedded, safety-critical systems. Our results showed that long-context LLMs tended to achieve higher accuracy than smaller models. Gemini 2.5 Pro Experimental achieved the best performance with 66.7% accuracy and produced coherent step-by-step explanations. However, coherent reasoning did not consistently translate into correct root-cause identification, suggesting that LLMs are currently better suited to support human-in-the-loop root cause analysis than to fully automate it, and motivating further work to improve prediction accuracy for log-based RCA.

Overview

CoChat Summary

TL;DR

This study benchmarks nine large language models with zero-shot chain-of-thought prompting for root cause analysis of railway TCMS test execution logs from Alstom’s digital twin testing environment. Gemini 2.5 Pro Experimental achieved the best accuracy at 66.7%, but long-context limitations and inconsistent predictions meant LLMs were not reliable enough to replace manual analysis in safety-critical contexts.

Key findings

  • Gemini 2.5 Pro Experimental achieved the highest accuracy: 10/15 correct predictions (66.7%) and an average reasoning score of 3.63. Gemini 2.0 Flash Experimental followed with 8/15 (53.3%) and Amazon Nova 2 Lite with 7/15 (46.7%), both with average reasoning scores of 3.23. DeepSeek R1, DeepSeek V3 0324, and Mistral Small 3.1 24B each achieved 5/15 (33.3%), while GPT o3-mini had the lowest accuracy at 1/15 (6.7%) despite an average reasoning score of 3.83 and 4.30 on its one correct prediction.
  • Long-context models generally outperformed smaller-context models. Logs ranged from 3,181 to 1,316,476 characters with a median of 468,468 characters; GPT o3-mini’s 200,000 context length likely prevented it from ingesting most log content. Logs exceeding an LLM’s context window led to unreliable predictions: log-8 (1,256,252 characters) and log-14 (446,776 characters) had 0/9 correct predictions, while log-1, log-10, and log-5 were easiest with 6/9 correct each.
  • Higher reasoning scores did not guarantee correct root causes. The Spearman correlation between number of correct predictions and reasoning scores was -0.2394 (p = 0.5351), indicating a weak relationship. The paper also notes a mismatch between the five-class prompt label space and the three-category ground truth present in the dataset, likely leading models to generate hallucinated classifications.

Methods

  • The authors benchmarked nine LLMs—Aion-1.0, DeepSeek R1, DeepSeek V3 0324, Mistral Small 3.1 24B, GPT o3-mini, Gemini 2.5 Pro Experimental, QwB 32B (QwQ-32B), Gemini 2.0 Flash Experimental, and Amazon Nova 2 Lite—on 15 Alstom digital twin test execution logs using zero-shot chain-of-thought prompting with a predefined list of possible root causes. Logs were preprocessed into JSON retaining test step, execution result, timestamp, and message content, and sensitive identifiers were obfuscated.
  • Ground truth root causes were manually defined with Alstom engineers. Root cause prediction was scored binary (1 for exact match, 0 otherwise), and accuracy was computed as correct predictions out of 15. Reasoning steps were evaluated by ten software engineering practitioners who rated 27 sampled LLM responses (21% of 125) on a 5-point Likert scale against an expert thinking-flow reference.
  • Models were accessed via the OpenRouter web GUI with default parameters (temperature=1.0, top-p=1.0, top-k=0). Responses were exported as JSON and parsed with Python scripts for root cause prediction and reasoning extraction.

Limitations

  • Stated: parts of the pipeline and ground-truth labeling were manual, potentially introducing human error or subjectivity; CoT logical validity was not formally verified; OpenRouter web GUI default parameters may change and threaten reproducibility; the dataset was limited to 15 logs from a single industrial partner and covered only three of five fault categories (“signal timeout” 7 logs, “expired bugfix” 4 logs, “test aborted” 4 logs); small dataset size limits statistical robustness; LLM non-determinism and single-run experiments are threats; only a subset of available models was evaluated; and inter-rater reliability was poor (Krippendorff’s α = 0.0714).
  • Evident from the text: the prompt used five root-cause categories while the dataset contained only three validated categories, creating a prompt-label misalignment that likely contributed to hallucinated classifications.
  • Evident from the text: zero-shot CoT may not provide sufficient guidance for complex and lengthy logs, and for categories such as “signal timeout,” a complete RCA may require propagation to other software artefacts not considered in this study.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11334-026-00657-1

Cites

45 of 49 references verified1 citation errors · 3 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Typical manual debugging of failed test execution by test engineers
Fig. 1 Typical manual debugging of failed test execution by test engineersCC-BY · From the PDF
Fig. 2 Experiment pipeline overview
Fig. 2 Experiment pipeline overviewCC-BY · From the PDF
Fig. 3 LLMs prediction accuracy and reasoning score
Fig. 3 LLMs prediction accuracy and reasoning scoreCC-BY · From the PDF
Fig. 4 Proposed semi-automatic failure debugging process when LLM assists RCA activity
Fig. 4 Proposed semi-automatic failure debugging process when LLM assists RCA activityCC-BY · From the PDF

Verification

Claims the paper makes

5 Valid · 4 Not valid · 4 Not checked · 60 Own results and interpretations

28 more claims

Create a free account to see all 73 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 19
  • It may state two separate claims as one. 29
  • The quote only partly states this claim. 5

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 49 references

Research works printed but not linked: 1

  1. Bucaioni A, Di Silvestro F, Singh I, Saadatmand M, Muccini H (2022) Model-based generation of test scripts across product variants: an experience report from the railway industry. J Softw Evol Process 34(11):2498 https://doi.org/10.1002/smr.2498 · DOI

Metadata

Authors
Hermawan, R., Bucaioni, A., Enoiu, E., Afzal, W., Saadatmand, M., Zaimovic, N., & Ibtasham, M. S.
Affiliations
  • Mälardalen University (SE)
  • RISE Research Institutes of Sweden (SE)
Corresponding author
  • Hermawan, Rahmanu
  • Bucaioni, Alessio
Venue
Innovations in Systems and Software Engineering
Published
2026-08-28
Volume
22
Issue
2
ISSN
  • 1614-5046 (ISSN-L)
  • 1614-5054
OpenAlex
W7138889259
Cited by
0
References
45
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Innovations in Systems and Software Engineering (publishedVersion, cc-by)
  • Research Square (submittedVersion, cc-by)
  • KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
Funders
  • Chips Joint Undertaking (101140216)
  • Clean Energy Transition Partnership (101069750)
  • VINNOVA (2023-01899)
  • ITEA (MONA LISA)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
root cause analysis, large language models, safety-critical software, software validation, chain-of-thought prompting, LLM benchmarking, human-in-the-loop analysis, embedded systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Let’s read the log: root cause analysis of railway test execution logs with large language models. https://app.cochat.ai/papers/10.1007/s11334-026-00657-1
The paper
Hermawan, R., Bucaioni, A., Enoiu, E., Afzal, W., Saadatmand, M., Zaimovic, N., & Ibtasham, M. S. (2026). Let’s read the log: root cause analysis of railway test execution logs with large language models. Innovations in Systems and Software Engineering. https://doi.org/10.1007/s11334-026-00657-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.