CoChat

TeleEval-OS: Performance evaluations of large language models for telecommunications operations scheduling

Wang, Yanyan, Wang, Yingying, Liang, Junli, Xu, Yin, Liu, Yunlong, Xu, Yiming, Jiang, Zhengwang, Li, Zhenhe, Lin, Can, Li, Fei, Zhao, Long, Xu, Kuang, Song, Qi, Li, Xiangyang, Wahab, Abdul · 2025 · Intelligent Data Analysis: An International Journal

The DOI matches its Crossref record; the references it cites are not checked yet.

Abstract

The rapid advancement of large language models (LLMs) has enabled their application across complex professional domains. In the telecommunications industry, operations scheduling integrates network monitoring, ticket management, risk assessment, and workforce coordination, requiring intelligent support due to its language-knowledge-decision coupling. However, the lack of standardized benchmarks limits LLM development in this field. To address these issues, we introduce TeleEval-OS, the first dedicated evaluation benchmark for telecommunications operations scheduling. TeleEval-OS comprehensively covers the four essential stages of dispatch workflows: intelligent ticket creation, resolution, ticket closure, and operational assessment. The benchmark includes 15 high-quality, manually annotated datasets with a total of 10.4K samples, spanning 13 representative real-world sub-tasks, such as similar-ticket recommendation, service intent classification, network fault ticket report generation, and risk indicator interpretation. To capture the spectrum of task complexity, we define a four-level evaluation hierarchy: basic natural language processing (NLP), domain-specific question answering (Q&A), structured report generation, and operational report analysis. We conduct a systematic evaluation of 14 representative LLMs under zero-shot and few-shot settings, such as GPT-4o, DeepSeek-V3, and Qwen-2.5-72B-Instruct. The results show that DeepSeek-V3 achieves the best performance on basic NLP and structured report generation tasks, while GPT-4o demonstrates superior capabilities in operational report analysis. These findings highlight the complementary strengths of LLMs across different task types. They also underscore the practical value of TeleEval-OS as the first dedicated benchmark for telecommunications operations scheduling, providing a unified framework for both future research and real-world deployment. Code is available at https://github.com/zjsllab/TeleEval-OS.

Overview

Integrity signals

Registry

Crossref · 10.1177/1088467x251389769

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

The citation lists are queued

They are fetched later and appear here then.

References 22

Research works printed but not linked: 17

  1. Achiam J Adler S Agarwal S et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023. · arXiv
  2. Liu A Feng B Xue B et al. DeepSeek-v3 technical report. arXiv preprint arXiv:2412.19437 2024. · arXiv
  3. Yang A Yang B Zhang B et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 2024. · arXiv
  4. Ye J, Wang G, Li Y, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical AI. Adv Neural Inf Process Syst 2025; 37: 94327–94427.
  5. Zhu K, Zhao Q, Chen H, et al. Promptbench: A unified library for evaluation of large language models. J Mach Learn Res 2024; 25: 1–22.
  6. Mou Y, Zhang S, Ye W. SG-bench: Evaluating LLM safety generalization across diverse tasks and prompt types. Adv Neural Inf Process Syst 2025; 37: 123032.
  7. Yue W Wang X Zhu W et al. TCMBench: A comprehensive benchmark for evaluating large language models in traditional Chinese medicine. CoRR 2024.
  8. Dai Y Feng D Huang J et al. LAiW: A Chinese legal large language models benchmark. In: Proceedings of the 31st International conference on computational linguistics 2025 pp.10738–10766.
  9. GLM T Zeng A Xu B et al. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools. arXiv preprint arXiv:2406.12793 2024. · arXiv
  10. Guo Z Jin R Liu C et al. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736 2023. · arXiv
  11. Xu L Li A Zhu L et al. SuperCLUE: A comprehensive Chinese large language model benchmark. arXiv preprint arXiv:2307.15020 2023. · arXiv
  12. Huang Y, Bai Y, Zhu Z, et al. C-eval: A multi-level multi-discipline Chinese evaluation suite for foundation models. Adv Neural Inf Process Syst 2023; 36: 62991–63010.
  13. Feng K Ding K Wang W et al. SciKnowEval: Evaluating multi-level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098 2024. · arXiv
  14. Guo J Guo Y Li M et al. FLAME: Financial large-language model assessment and metrics evaluation. arXiv preprint arXiv:2501.06211 2025. · arXiv
  15. Yue S Chen W Wang S et al. DISC-LawLLM: Fine-tuning large language models for intelligent legal services. CoRR 2023.
  16. He Z Wang Z Liu X et al. TeleChat technical report. arXiv preprint arXiv:2401.03804 2024. · arXiv
  17. Zheng L Chiang W Sheng Y et al. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In: Proceedings of the 37th international conference on neural information processing systems (NeurIPS 2023) 2023 pp.1–29. Red Hook NY USA: Curran Associates Inc. Article no. 2020.

Metadata

Authors
Wang, Y., Wang, Y., Liang, J., Xu, Y., Liu, Y., Xu, Y., Jiang, Z., Li, Z., Lin, C., Li, F., Zhao, L., Xu, K., Song, Q., Li, X., & Wahab, A.
Affiliations
  • GuoChuang Cloud Technology Ltd, Hefei, 230031, Anhui, China
  • University of Science and Technology of China (CN)
  • University of Science and Technology of China
Corresponding author
  • Wang, Yingying
  • Wahab, Abdul
Venue
Intelligent Data Analysis: An International Journal
Published
2025-11-03
ISSN
  • 1088-467X (ISSN-L)
  • 1571-4128
OpenAlex
W4415819149
Cited by
0
References
22
FWCI
0.00
Type
journal article
Language
en
Open access
closed
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
Intelligent Data Analysis (publishedVersion)
Funders
  • ”Pioneer” and ”Leading Goose” R&D Program of Zhejiang (2023C01029)
  • Chinese Polar Environment Comprehensive Investigation and Assessment Programmes (2021ZD0110400)
  • ”Leading Goose” R&D Program of Zhejiang (2023C01143)
  • Innovation Program for Quantum Science and Technology (2021ZD0302900)
  • China National Natural Science Foundation (62132018)
  • Anhui Postdoctoral Scientific Research Program Foundation (2024A768)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, evaluation benchmark, structured report generation, DeepSeek-V3, GPT-4o

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: TeleEval-OS: Performance evaluations of large language models for telecommunications operations scheduling. https://app.cochat.ai/papers/10.1177/1088467x251389769
The paper
Wang, Y., Wang, Y., Liang, J., Xu, Y., Liu, Y., Xu, Y., Jiang, Z., Li, Z., Lin, C., Li, F., Zhao, L., Xu, K., Song, Q., Li, X., & Wahab, A. (2025). TeleEval-OS: Performance evaluations of large language models for telecommunications operations scheduling. Intelligent Data Analysis: An International Journal. https://doi.org/10.1177/1088467x251389769

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.