Language Models as a Service: Overview of a New Paradigm and its Challenges
Open accessAbstract
Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major LMaaS, offering a synthesized overview of the licences and capabilities their interfaces offer.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The reason for non-determinism with temperature zero is still controversial, with hypotheses such as floating point, unreliable GPU calculations, and sparse MoE architecture failing to enforce per-sequence determinism.
“The reason for such non-determinism with the temperature being zero is still controversial [1], with different hypotheses such as floating point, unreliable GPU calculations, and its sparse MoE architecture failing to enforce per-sequence determinism [44, 54].”
From An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2025) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 2× · see citing works
“These behaviours are more prominent in larger models (Kaplan et al., 2020), which led to the development of even larger models.”
BackgroundAGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 1× · see citing works
“…literature between benchmarking via point-wise metrics on a large number of models and scenarios (Liang et al., 2022; Srivastava et al., 2022; Zhong et al., 2023), and evaluations where models are aggregated post-hoc based on latent factors that capture variations over comparable tasks…”
BackgroundIn ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT
Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.08979 · Cited by 76 · cited 6× · see citing works
“reasoning, symbols manipulation, and code generation (Chen et al., 2019; Kojima et al., 2022; Cohn and Hernandez-Orallo, 2023; Frieder et al., 2023; Ray, 2023; Rozière et al., 2023; Shen et al., 2023).”
BackgroundInfluential
Flagged first, then by citations
Cited by
Sorted by citations
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258 · see citing works
“The reason for such non-determinism with the temperature being zero is still controversial [1], with different hypotheses such as floating point, unreliable GPU calculations, and its sparse MoE architecture failing to enforce per-sequence determinism [44, 54].”
Resnik, D. B., & Hosseini, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00493-8 · Cited by 228 · see citing works
“ChatGPT, for example, is a highly opaque system that is yet to disclose its training data and it is unclear whether/when OpenAI would open up its technology to external scrutiny [28, 66, 130].”
Near to Mid-term Risks and Opportunities of Open-Source Generative AI
Eiras, F., Petrov, A., Vidgen, B., de Witt, C. S., Pizzati, F., Elkins, K., Mukhopadhyay, S., Bibi, A., Csaba, B., Steibel, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.17047 · Cited by 2 · see citing works
Self-citationAn Empirical Study of the Non-determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2023) · arXiv · doi:10.48550/arxiv.2308.02828
- Authors
- La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A. G., Shadbolt, N., & Wooldridge, M.
- Affiliations
- University of Oxford (GB)
- The Alan Turing Institute (GB)
- University of Leeds (GB)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2309.16573
- OpenAlex
- W4387225762
- Cited by
- 1
- References
- 3
- Citations by year
- 2025: 1
- FWCI
- 0.09
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- GCHQ
- Government of the United Kingdom
- Engineering and Physical Sciences Research Council (EP/S024050/1)
- Economic and Social Research Council (ES/W003473/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- model evaluation, benchmarking, replicability, reliability, trustworthiness, API interface, open-source models, AI accessibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
