TeleEval-OS: Performance evaluations of large language models for telecommunications operations scheduling
The DOI matches its Crossref record; the references it cites are not checked yet.
Abstract
The rapid advancement of large language models (LLMs) has enabled their application across complex professional domains. In the telecommunications industry, operations scheduling integrates network monitoring, ticket management, risk assessment, and workforce coordination, requiring intelligent support due to its language-knowledge-decision coupling. However, the lack of standardized benchmarks limits LLM development in this field. To address these issues, we introduce TeleEval-OS, the first dedicated evaluation benchmark for telecommunications operations scheduling. TeleEval-OS comprehensively covers the four essential stages of dispatch workflows: intelligent ticket creation, resolution, ticket closure, and operational assessment. The benchmark includes 15 high-quality, manually annotated datasets with a total of 10.4K samples, spanning 13 representative real-world sub-tasks, such as similar-ticket recommendation, service intent classification, network fault ticket report generation, and risk indicator interpretation. To capture the spectrum of task complexity, we define a four-level evaluation hierarchy: basic natural language processing (NLP), domain-specific question answering (Q&A), structured report generation, and operational report analysis. We conduct a systematic evaluation of 14 representative LLMs under zero-shot and few-shot settings, such as GPT-4o, DeepSeek-V3, and Qwen-2.5-72B-Instruct. The results show that DeepSeek-V3 achieves the best performance on basic NLP and structured report generation tasks, while GPT-4o demonstrates superior capabilities in operational report analysis. These findings highlight the complementary strengths of LLMs across different task types. They also underscore the practical value of TeleEval-OS as the first dedicated benchmark for telecommunications operations scheduling, providing a unified framework for both future research and real-world deployment. Code is available at https://github.com/zjsllab/TeleEval-OS.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This change began with Software-Defined Networking (SDN), which separated the control plane from the data plane.This separation gave operators a clearer way to manage the control plane [46].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [46]
In the citing paper
“Even correct end states can be reached through unsafe intermediate states, such as loops, blackholes, or policy violations during rollout, because updates are applied under asynchronous convergence and partial deployment [11], [47], [46].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [46]
Citations
The citation lists are queued
They are fetched later and appear here then.
References 22
Research works printed but not linked: 17
- Achiam J Adler S Agarwal S et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023. · arXiv
- Liu A Feng B Xue B et al. DeepSeek-v3 technical report. arXiv preprint arXiv:2412.19437 2024. · arXiv
- Yang A Yang B Zhang B et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 2024. · arXiv
- Ye J, Wang G, Li Y, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical AI. Adv Neural Inf Process Syst 2025; 37: 94327–94427.
- Zhu K, Zhao Q, Chen H, et al. Promptbench: A unified library for evaluation of large language models. J Mach Learn Res 2024; 25: 1–22.
- Mou Y, Zhang S, Ye W. SG-bench: Evaluating LLM safety generalization across diverse tasks and prompt types. Adv Neural Inf Process Syst 2025; 37: 123032.
- Yue W Wang X Zhu W et al. TCMBench: A comprehensive benchmark for evaluating large language models in traditional Chinese medicine. CoRR 2024.
- Dai Y Feng D Huang J et al. LAiW: A Chinese legal large language models benchmark. In: Proceedings of the 31st International conference on computational linguistics 2025 pp.10738–10766.
- GLM T Zeng A Xu B et al. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools. arXiv preprint arXiv:2406.12793 2024. · arXiv
- Guo Z Jin R Liu C et al. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736 2023. · arXiv
- Xu L Li A Zhu L et al. SuperCLUE: A comprehensive Chinese large language model benchmark. arXiv preprint arXiv:2307.15020 2023. · arXiv
- Huang Y, Bai Y, Zhu Z, et al. C-eval: A multi-level multi-discipline Chinese evaluation suite for foundation models. Adv Neural Inf Process Syst 2023; 36: 62991–63010.
- Feng K Ding K Wang W et al. SciKnowEval: Evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098 2024. · arXiv
- Guo J Guo Y Li M et al. FLAME: Financial large-language model assessment and metrics evaluation. arXiv preprint arXiv:2501.06211 2025. · arXiv
- Yue S Chen W Wang S et al. DISC-LawLLM: Fine-tuning large language models for intelligent legal services. CoRR 2023.
- He Z Wang Z Liu X et al. TeleChat technical report. arXiv preprint arXiv:2401.03804 2024. · arXiv
- Zheng L Chiang W Sheng Y et al. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In: Proceedings of the 37th international conference on neural information processing systems (NeurIPS 2023) 2023 pp.1–29. Red Hook NY USA: Curran Associates Inc. Article no. 2020.
Metadata
- Authors
- Wang, Y., Wang, Y., Liang, J., Xu, Y., Liu, Y., Xu, Y., Jiang, Z., Li, Z., Lin, C., Li, F., Zhao, L., Xu, K., Song, Q., Li, X., & Wahab, A.
- Affiliations
- GuoChuang Cloud Technology Ltd, Hefei, 230031, Anhui, China
- University of Science and Technology of China (CN)
- University of Science and Technology of China
- Corresponding author
- Wang, Yingying
- Wahab, Abdul
- ORCID
- Venue
- Intelligent Data Analysis: An International Journal
- Publisher
- SAGE Publications
- Published
- 2025-11-03
- ISSN
- 1088-467X (ISSN-L)
- 1571-4128
- OpenAlex
- W4415819149
- Cited by
- 0
- References
- 22
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Intelligent Data Analysis (publishedVersion)
- Funders
- ”Pioneer” and ”Leading Goose” R&D Program of Zhejiang (2023C01029)
- Chinese Polar Environment Comprehensive Investigation and Assessment Programmes (2021ZD0110400)
- ”Leading Goose” R&D Program of Zhejiang (2023C01143)
- Innovation Program for Quantum Science and Technology (2021ZD0302900)
- China National Natural Science Foundation (62132018)
- Anhui Postdoctoral Scientific Research Program Foundation (2024A768)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, evaluation benchmark, structured report generation, DeepSeek-V3, GPT-4o
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: TeleEval-OS: Performance evaluations of large language models for telecommunications operations scheduling. https://app.cochat.ai/papers/10.1177/1088467x251389769
- The paper
- Wang, Y., Wang, Y., Liang, J., Xu, Y., Liu, Y., Xu, Y., Jiang, Z., Li, Z., Lin, C., Li, F., Zhao, L., Xu, K., Song, Q., Li, X., & Wahab, A. (2025). TeleEval-OS: Performance evaluations of large language models for telecommunications operations scheduling. Intelligent Data Analysis: An International Journal. https://doi.org/10.1177/1088467x251389769
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.