Let’s read the log: root cause analysis of railway test execution logs with large language models
Open accessThe DOI matches its Crossref record; 45 of the 49 references it cites were found in a registry or an index; 9 of the 15 sources its claims cite were checked for support.
Abstract
Software quality assurance is pivotal in safety-critical domains such as railway systems, where failures could have catastrophic consequences. In this context, the train control and management system, which enables communication and control across multiple subsystems (such as doors and information panels) within a modern train, and its software must undergo rigorous validation. Alstom Rail Sweden AB employs a digital twin infrastructure to simulate and validate train control and management system software. While this setup improves system-level testing, root-cause analysis of test failures remains a manual, time-consuming bottleneck. In this study, we explore the potential of large language models to automate root cause analysis by interpreting test execution logs generated during digital twin-based testing. We benchmark nine state-of-the-art large language models: Aion−1.0, DeepSeek R1, DeepSeek V3 0324, Mistral Small 3.1 24B, GPT o3-mini, Gemini 2.5 Pro Experimental, QwQ 32B, Gemini 2.0 Flash Experimental, and Amazon Nova 2 Lite using zero-shot chain-of-thought prompting to assess their ability to reason about fault patterns in real-world industrial test execution logs. The logs were sourced from Alstom’s digital twin-based testing environment and captured complex operational behaviour typical of embedded, safety-critical systems. Our results showed that long-context LLMs tended to achieve higher accuracy than smaller models. Gemini 2.5 Pro Experimental achieved the best performance with 66.7% accuracy and produced coherent step-by-step explanations. However, coherent reasoning did not consistently translate into correct root-cause identification, suggesting that LLMs are currently better suited to support human-in-the-loop root cause analysis than to fully automate it, and motivating further work to improve prediction accuracy for log-based RCA.
Overview
CoChat Summary
TL;DR
This study benchmarks nine large language models with zero-shot chain-of-thought prompting for root cause analysis of railway TCMS test execution logs from Alstom’s digital twin testing environment. Gemini 2.5 Pro Experimental achieved the best accuracy at 66.7%, but long-context limitations and inconsistent predictions meant LLMs were not reliable enough to replace manual analysis in safety-critical contexts.
Key findings
- Gemini 2.5 Pro Experimental achieved the highest accuracy: 10/15 correct predictions (66.7%) and an average reasoning score of 3.63. Gemini 2.0 Flash Experimental followed with 8/15 (53.3%) and Amazon Nova 2 Lite with 7/15 (46.7%), both with average reasoning scores of 3.23. DeepSeek R1, DeepSeek V3 0324, and Mistral Small 3.1 24B each achieved 5/15 (33.3%), while GPT o3-mini had the lowest accuracy at 1/15 (6.7%) despite an average reasoning score of 3.83 and 4.30 on its one correct prediction.
- Long-context models generally outperformed smaller-context models. Logs ranged from 3,181 to 1,316,476 characters with a median of 468,468 characters; GPT o3-mini’s 200,000 context length likely prevented it from ingesting most log content. Logs exceeding an LLM’s context window led to unreliable predictions: log-8 (1,256,252 characters) and log-14 (446,776 characters) had 0/9 correct predictions, while log-1, log-10, and log-5 were easiest with 6/9 correct each.
- Higher reasoning scores did not guarantee correct root causes. The Spearman correlation between number of correct predictions and reasoning scores was -0.2394 (p = 0.5351), indicating a weak relationship. The paper also notes a mismatch between the five-class prompt label space and the three-category ground truth present in the dataset, likely leading models to generate hallucinated classifications.
Methods
- The authors benchmarked nine LLMs—Aion-1.0, DeepSeek R1, DeepSeek V3 0324, Mistral Small 3.1 24B, GPT o3-mini, Gemini 2.5 Pro Experimental, QwB 32B (QwQ-32B), Gemini 2.0 Flash Experimental, and Amazon Nova 2 Lite—on 15 Alstom digital twin test execution logs using zero-shot chain-of-thought prompting with a predefined list of possible root causes. Logs were preprocessed into JSON retaining test step, execution result, timestamp, and message content, and sensitive identifiers were obfuscated.
- Ground truth root causes were manually defined with Alstom engineers. Root cause prediction was scored binary (1 for exact match, 0 otherwise), and accuracy was computed as correct predictions out of 15. Reasoning steps were evaluated by ten software engineering practitioners who rated 27 sampled LLM responses (21% of 125) on a 5-point Likert scale against an expert thinking-flow reference.
- Models were accessed via the OpenRouter web GUI with default parameters (temperature=1.0, top-p=1.0, top-k=0). Responses were exported as JSON and parsed with Python scripts for root cause prediction and reasoning extraction.
Limitations
- Stated: parts of the pipeline and ground-truth labeling were manual, potentially introducing human error or subjectivity; CoT logical validity was not formally verified; OpenRouter web GUI default parameters may change and threaten reproducibility; the dataset was limited to 15 logs from a single industrial partner and covered only three of five fault categories (“signal timeout” 7 logs, “expired bugfix” 4 logs, “test aborted” 4 logs); small dataset size limits statistical robustness; LLM non-determinism and single-run experiments are threats; only a subset of available models was evaluated; and inter-rater reliability was poor (Krippendorff’s α = 0.0714).
- Evident from the text: the prompt used five root-cause categories while the dataset contained only three validated categories, creating a prompt-label misalignment that likely contributed to hallucinated classifications.
- Evident from the text: zero-shot CoT may not provide sufficient guidance for complex and lengthy logs, and for categories such as “signal timeout,” a complete RCA may require propagation to other software artefacts not considered in this study.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
5 Valid · 4 Not valid · 4 Not checked · 60 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Likert scales o er established validity for measuring complex cognitive processes, such as diagnostic reasoning [40].”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > Score Description 1
What the sources say
- Could not verifycites [40] Kim E, Garg A, Peng K, Garg N (2025) Correlated errors in large language models https://arxiv.org/abs/2506.07962 sourceExperimental — accuracy not yet measured
The Data and methods section (p. 3) describes multiple-choice answers and numerical resume ratings, not Likert scales. The Conclusion (p. 9) says richer evaluation of complex reasoning remains future work. Searches for Likert-scale validity and diagnostic reasoning found no passage addressing the claim. (LLM, conf=0.76)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Another threat to conclusion validity was LLM non-determinism [45].”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > Threats to validity
What the sources say
- Supportedcites [45] Ouyang S, Zhang JM, Harman M, Wang M (2023) An empirical study of the non-determinism of ChatGPT in code generation. arXiv preprint https://doi.org/10.48550/arXiv.2308.02828. arXiv:2308.02828 source
The Introduction (p. 2) explicitly calls non-determinism in LLM-based code-generation experiments a significant threat to the validity of scientific conclusions. This supports the citing sentence’s characterization of LLM non-determinism as a threat to conclusion validity. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while the models often produced coherent step-by-step explanations, they did not reliably identify the correct root cause”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > Listing 5 Excerpt of LLM's reponse when incorrectly determining a root cause of aborted test
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Logs log-1, log-10, and log-5 were the easiest to predict, with 6 out of 9 models producing correct predictions for each.”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > 4 Experimental results and analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that when a log exceeded an LLM's context window, the model failed to reliably predict the root cause.”
- Also stated in: § 1 Introduction · plain
“while logs exceeding an LLM's context window led to unreliable predictions”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
28 more claims
Create a free account to see all 73 claims and to verify the 2 references still open.
- Probably not a substantial claim. 19
- It may state two separate claims as one. 29
- The quote only partly states this claim. 5
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > 5 Discussion
“For log-1 and log-5, which were medium and large logs per Table 1”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > 6 Conclusion and Future Work
“| Log-1 | 329,186 | 5.51 | Small |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > 5 Discussion
“The log les ranged from 3,181 to 1,316,476 characters, with a median of 468,468 characters.”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > 6 Conclusion and Future Work
“| Log-10 | 3,181 | 3.50 | Small | | Log-4 | 3,637 | 3.56 | Small | | Log-6 | 216,115 | 5.33 | Small | | Log-1 | 329,186 | 5.51 | Small | | Log-12 | 350,111 | 5.54 | Small | | Log-13 | 406,884 | 5.60 | Medium | | Log-14 | 446,776 | 5.65 | Medium | | Log-7 | 489,160 | 5.68 | Medium |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“From 125 LLM responses (not all models successfully provided reasoning steps), we selected a strati ed random sample of 27 responses (21% of total)”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > 3.3 Evaluation
Reported 21 · recomputed 21.6
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > Score Description 1
“Ten software engineering practitioners independently evaluated the reasoning steps in responses from 27 LLMs, irrespective of prediction correctness.”
§ Let's Read the Log: Root Cause Analysis of Railway Test Execution Logs with Large Language Models > 3.3 Evaluation
“From each stratum, we randomly and programmatically selected one representative log and compiled the step-by-step responses from all nine LLMs, yielding the 27 rating items.”
Citations
Cites
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448 · cited 1×
“This approach mit-igates imbalance while enabling correlation analysis between log size, LLM context length, and reasoning quality [31, 32].”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Likert Scale: Explored and Explained
Joshi, A., Kale, S., Chandel, S., & Pal, D. (2015) · British Journal of Applied Science & Technology · doi:10.9734/bjast/2015/14975 · Cited by 3,870 · cited 1×
“Likert scales offer established validity for measuring complex cognitive processes, such as diagnostic reasoning [33].”
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 3×
“We operationalised this design through an explicit task instruction, “Provide a step-by-step reasoning to analyse the failure" , which guided theLLMininferringrootcausesfromimplicitpatternswithin each log [12].”
InfluentialGu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15594 · Cited by 1,781
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 2×
“Prior work shows that LLMs often struggle to reliably select and integrate the most relevant evidence when multiple competing signals are present, particularly in long or complex contexts [36].”
The use of scoring rubrics: Reliability, validity and educational consequences
Jonsson, A., & Svingby, G. (2007) · Educational Research Review · doi:10.1016/j.edurev.2007.05.002 · Cited by 1,509
Introduction to Software Testing
Ammann, P., & Offutt, J. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316771273 · Cited by 607
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294
Research works printed but not linked: 1
- Bucaioni A, Di Silvestro F, Singh I, Saadatmand M, Muccini H (2022) Model-based generation of test scripts across product variants: an experience report from the railway industry. J Softw Evol Process 34(11):2498 https://doi.org/10.1002/smr.2498 · DOI
Metadata
- Authors
- Hermawan, R., Bucaioni, A., Enoiu, E., Afzal, W., Saadatmand, M., Zaimovic, N., & Ibtasham, M. S.
- Affiliations
- Mälardalen University (SE)
- RISE Research Institutes of Sweden (SE)
- Corresponding author
- Hermawan, Rahmanu
- Bucaioni, Alessio
- ORCID
- Venue
- Innovations in Systems and Software Engineering
- Published
- 2026-08-28
- Volume
- 22
- Issue
- 2
- ISSN
- 1614-5046 (ISSN-L)
- 1614-5054
- OpenAlex
- W7138889259
- Cited by
- 0
- References
- 45
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Innovations in Systems and Software Engineering (publishedVersion, cc-by)
- Research Square (submittedVersion, cc-by)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- Funders
- Chips Joint Undertaking (101140216)
- Clean Energy Transition Partnership (101069750)
- VINNOVA (2023-01899)
- ITEA (MONA LISA)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- root cause analysis, large language models, safety-critical software, software validation, chain-of-thought prompting, LLM benchmarking, human-in-the-loop analysis, embedded systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Let’s read the log: root cause analysis of railway test execution logs with large language models. https://app.cochat.ai/papers/10.1007/s11334-026-00657-1
- The paper
- Hermawan, R., Bucaioni, A., Enoiu, E., Afzal, W., Saadatmand, M., Zaimovic, N., & Ibtasham, M. S. (2026). Let’s read the log: root cause analysis of railway test execution logs with large language models. Innovations in Systems and Software Engineering. https://doi.org/10.1007/s11334-026-00657-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.