Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 2 of the 15 sources its claims cite were checked for support.

Larger and more instructable language models become less reliable

Open access
Zhou, Lexin, Schellaert, Wout, Martínez-Plumed, Fernando, Moros-Daval, Yael, Ferri, Cèsar, Hernández-Orallo, José · 2024 · Nature

Abstract

The prevailing methods to make large language models more powerful and amenable have been based on continuous scaling up (that is, increasing their size, data volume and computational resources 1 ) and bespoke shaping up (including post-filtering 2,3 , fine tuning or use of human feedback 4,5 ). However, larger and more instructable large language models may have become less reliable. By studying the relationship between difficulty concordance, task avoidance and prompting stability of several language model families, here we show that easy instances for human participants are also easy for the models, but scaled-up, shaped-up models do not secure areas of low difficulty in which either the model does not err or human supervision can spot the errors. We also find that early models often avoid user questions but scaled-up, shaped-up models tend to give an apparently sensible yet wrong answer much more often, including errors on difficult questions that human supervisors frequently overlook. Moreover, we observe that stability to different natural phrasings of the same question is improved by scaling-up and shaping-up interventions, but pockets of variability persist across difficulty levels. These findings highlight the need for a fundamental shift in the design and development of general-purpose artificial intelligence, particularly in high-stakes areas for which a predictable distribution of errors is paramount.

Overview

CoChat Summary

TL;DR

Across 32 LLMs from the GPT, LLaMA and BLOOM families on five benchmarks, scaling and instruction/alignment shaping improved correctness and prompt stability but did not create reliable low-difficulty operating regions. Shaped-up models increasingly replaced avoidance with plausible but incorrect answers, making errors harder for human supervisors to catch.

Key findings

  • Human difficulty proxies predicted correctness, but no error-free easy region emerged: for LLaMA, no model achieved 60% correctness at the simplest difficulty level (discounting 25% random guess for science), and only GPT-4 showed a low-difficulty science region with almost perfect results up to medium difficulty.
  • Scaling and shaping exchanged avoidance for incorrectness: avoidance almost disappeared for GPT-4, incorrectness increased markedly from raw to shaped-up models, and ultracrepidarianism—incorrect over avoidant plus incorrect answers, i/(a + i)—rose.
  • Prompting stability improved with scaling and shaping, but pockets of variability persisted across difficulty levels; raw GPT and all LLaMA models were highly prompt-sensitive even for unambiguous addition.
  • Human supervision study S2 found few areas where the dangerous error of considering an incorrect answer correct was low enough to constitute a safe operating region.

Methods

  • Analysed 32 models: 10 GPT, 10 LLaMA and 12 BLOOM, including raw and shaped-up versions, across five benchmarks: addition, anagram, locality, science and transforms. Each example was run with 15 natural prompt templates per benchmark, yielding approximately 4.2 million answers; temperature was set to zero and no system prompt was used.
  • Data samples: 3,142 addition instances, 1,570 anagrams, 2,341 locality instances, science from 1,000 OpenBookQA questions plus 546 GPQA questions minus two over-length items, and 730 transforms items (73 tasks × 10 instances). Two human studies (S1 and S2) recruited participants via Prolific at £9 per hour.
  • Difficulty was calibrated to a 0–100 human-failure scale via logistic mapping, then examples were split into 30 equal-sized difficulty bins. Analyses included Spearman rank correlations, six reliability indicators for correctness and prudence (proportion, prompting stability, difficulty concordance), and a scaling analysis using FLOPs.

Limitations

  • Stated: participants were mostly non-experts, which affects calibrated difficulty values; the natural prompt sample came from diverse sources but lacked real-world prompt-frequency data; and only a sample of model families was covered, excluding LLMs that delegate to external tools or use sophisticated reasoning techniques.
  • Evident: GPT models were accessed through APIs and may change over time; response scoring relied on algorithmic conditions and regular expressions, which may not capture all response nuance; calibrated difficulty values are estimates from human study S1 and should be interpreted cautiously, especially for small differences.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41586-024-07930-y

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Key indicators for several models in GPT (OpenAI), LLaMA (Meta) and BLOOM (BigScience) families. The raw models (yellow to orange) and the shaped-up models (light to dark blue) cluster differently. As the answers for all these models fall into three categories (correct, avoidant and incorrect), shortened as c, a and i, respectively, we have indicators for correctness versus avoidance + incorrectness, and prudence (correctness + avoidance) versus incorrectness. Looking at the correctness indicators (top half), which represent accurate responses, we see that the shaped-up models are more stable to prompt variations and are more frequently correct (higher correctness proportion) but are less concordant with human difficulty than the raw counterparts. Looking at the prudence indicators (bottom half), we see that the shaped-up models are also more stable to prompt variations, but fail more frequently (lower prudence proportion, by avoiding less) and are not much more concordant with human difficulty. Focusing only on the shaped-up models (in blue), we observe that the most powerful GPT-4 v.2, LLaMA-2-70b-chat and BLOOMz-176b models perform best in correctness proportion and prompting stability (top and bottom), but equal to or worse than other models for all the other indicators, with many fluctuations that do not indicate a clear positive trend in these other dimensions. Details of the indicators and data used for this plot are found in the Methods. Extended Data Table 1 provides a more detailed perspective on the same results. Source Data
Fig. 1 Key indicators for several models in GPT (OpenAI), LLaMA (Meta) and BLOOM (BigScience) families. The raw models (yellow to orange) and the shaped-up models (light to dark blue) cluster differently. As the answers for all these models fall into three categories (correct, avoidant and incorrect), shortened as c, a and i, respectively, we have indicators for correctness versus avoidance + incorrectness, and prudence (correctness + avoidance) versus incorrectness. Looking at the correctness indicators (top half), which represent accurate responses, we see that the shaped-up models are more stable to prompt variations and are more frequently correct (higher correctness proportion) but are less concordant with human difficulty than the raw counterparts. Looking at the prudence indicators (bottom half), we see that the shaped-up models are also more stable to prompt variations, but fail more frequently (lower prudence proportion, by avoiding less) and are not much more concordant with human difficulty. Focusing only on the shaped-up models (in blue), we observe that the most powerful GPT-4 v.2, LLaMA-2-70b-chat and BLOOMz-176b models perform best in correctness proportion and prompting stability (top and bottom), but equal to or worse than other models for all the other indicators, with many fluctuations that do not indicate a clear positive trend in these other dimensions. Details of the indicators and data used for this plot are found in the Methods. Extended Data Table 1 provides a more detailed perspective on the same results. Source DataCC BY · PubMed Central
Fig. 2 Performance of a selection of GPT and LLaMA models with increasing difficulty. The values are split by correct, avoidant and incorrect results. For each combination of model and benchmark, the result is the average of 15 prompt templates (see Supplementary Tables 1 and 2). For each benchmark, we show its chosen intrinsic difficulty, monotonically calibrated to human expectations on the x axis for ease of comparison between benchmarks. The x axis is split into 30 equal-sized bins, for which the ranges must be taken as indicative of different distributions of perceived human difficulty across benchmarks. For ‘science’, the transparent yellow bars at the bottom represent the random guess probability (25% of the non-avoidance answers). Plots for all GPT and LLaMA models are provided in Extended Data Figs. 1 and 2 and for the BLOOM family in Supplementary Fig. 14. Source Data
Fig. 2 Performance of a selection of GPT and LLaMA models with increasing difficulty. The values are split by correct, avoidant and incorrect results. For each combination of model and benchmark, the result is the average of 15 prompt templates (see Supplementary Tables 1 and 2). For each benchmark, we show its chosen intrinsic difficulty, monotonically calibrated to human expectations on the x axis for ease of comparison between benchmarks. The x axis is split into 30 equal-sized bins, for which the ranges must be taken as indicative of different distributions of perceived human difficulty across benchmarks. For ‘science’, the transparent yellow bars at the bottom represent the random guess probability (25% of the non-avoidance answers). Plots for all GPT and LLaMA models are provided in Extended Data Figs. 1 and 2 and for the BLOOM family in Supplementary Fig. 14. Source DataCC BY · PubMed Central
Fig. 3 Evolution of types of supervision error versus difficulty according to human survey S2. In the survey (Supplementary Fig. 4), participants have to determine whether the output of a model is correct, avoidant or incorrect (or do not know, represented by the ‘unsure’ option in the questionnaire). Difficulty (x axis) is shown in equal-sized bins. We see very few areas where the dangerous error (incorrect being considered correct by participants) is sufficiently low to consider a safe operating region. Source Data
Fig. 3 Evolution of types of supervision error versus difficulty according to human survey S2. In the survey (Supplementary Fig. 4), participants have to determine whether the output of a model is correct, avoidant or incorrect (or do not know, represented by the ‘unsure’ option in the questionnaire). Difficulty (x axis) is shown in equal-sized bins. We see very few areas where the dangerous error (incorrect being considered correct by participants) is sufficiently low to consider a safe operating region. Source DataCC BY · PubMed Central
Fig. 4 Scaling analysis of LLaMA and BLOOM families and non-instruct GPT models. The plot uses a logarithmic scale for FLOPs. The focus is on avoidance (a; top left), incorrectness (i; bottom left) and ultracrepidarianism (i/(a + i); right)—the proportion of incorrect over both avoidant and incorrect answers. Source Data
Fig. 4 Scaling analysis of LLaMA and BLOOM families and non-instruct GPT models. The plot uses a logarithmic scale for FLOPs. The focus is on avoidance (a; top left), incorrectness (i; bottom left) and ultracrepidarianism (i/(a + i); right)—the proportion of incorrect over both avoidant and incorrect answers. Source DataCC BY · PubMed Central
Extended Data Fig. 1 Performance of GPT models over difficulty. The values are split by correct, avoidant and incorrect results. Details as in Fig. 2.
Extended Data Fig. 1 Performance of GPT models over difficulty. The values are split by correct, avoidant and incorrect results. Details as in Fig. 2.CC BY · PubMed Central
Extended Data Fig. 2 Performance of LLaMA models over difficulty. The values are split by correct, avoidant and incorrect results. Details as in Fig. 2.
Extended Data Fig. 2 Performance of LLaMA models over difficulty. The values are split by correct, avoidant and incorrect results. Details as in Fig. 2.CC BY · PubMed Central
Extended Data Fig. 3 Prompting stability of GPT models over difficulty. Proportion of correctness and avoidance represented as (grey) curves over difficulty for the 15 prompt templates for the GPT models addressing each of the five benchmarks. The green and bronze curves correspond to the prompt template that has, respectively, the highest and lowest average correctness, avoidance, or incorrectness. The two small numbers in green and bronze in the plot identify them (corresponding to the template codes in Supplementary Tables 1 and 2). The plots for all the models and all response categories are in section 9 of the Supplementary Information. The same plot for the BLOOM family is in section 11 of the Supplementary Information.
Extended Data Fig. 3 Prompting stability of GPT models over difficulty. Proportion of correctness and avoidance represented as (grey) curves over difficulty for the 15 prompt templates for the GPT models addressing each of the five benchmarks. The green and bronze curves correspond to the prompt template that has, respectively, the highest and lowest average correctness, avoidance, or incorrectness. The two small numbers in green and bronze in the plot identify them (corresponding to the template codes in Supplementary Tables 1 and 2). The plots for all the models and all response categories are in section 9 of the Supplementary Information. The same plot for the BLOOM family is in section 11 of the Supplementary Information.CC BY · PubMed Central
Extended Data Fig. 4 Prompting stability of LLaMA models over difficulty. Proportion of correctness and avoidance represented as (grey) curves over difficulty for the 15 prompt templates for the LLaMA models addressing each of the five benchmarks. Details as in Extended Data Fig. 3. The plots for all the models and all response categories are in section 9 of the Supplementary Information. The same plot for the BLOOM family is in section 11 of the Supplementary Information.
Extended Data Fig. 4 Prompting stability of LLaMA models over difficulty. Proportion of correctness and avoidance represented as (grey) curves over difficulty for the 15 prompt templates for the LLaMA models addressing each of the five benchmarks. Details as in Extended Data Fig. 3. The plots for all the models and all response categories are in section 9 of the Supplementary Information. The same plot for the BLOOM family is in section 11 of the Supplementary Information.CC BY · PubMed Central

Verification

Claims the paper makes

1 Valid · 0 Not valid · 9 Not checked · 58 Own results and interpretations

18 more claims

Create a free account to see all 68 claims and to verify the 13 references still open.

See all claims — free account
  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 27

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 47 references

Research works printed but not linked: 8

  1. Kim, G., Baldi, P. & McAleer, S. Language models can solve computer tasks. Adv. Neural Inf. Process. Syst. 36, 39648–39677 (2024).
  2. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics 4171–4186 (Association for Computational Linguistics, 2019).
  3. Sun, J., Shaib, C. & Wallace, B. C. Evaluating the zero-shot robustness of instruction-tuned language models. In The Twelfth International Conference on Learning Representations (2024).
  4. Zeidner, M. & Matthews, G. Evaluation anxiety. Handb. Competence Motiv. 141, 163 (2005).
  5. Kuhn, L., Gal, Y. & Farquhar, S. Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations (2023).
  6. Li, M. et al. Think twice before assure: confidence estimation for large language models through reflection on multiple answers. Preprint at https://arxiv.org/html/2403.09972v1 (2024).
  7. Villalobos, P. et al. Position: Will we run out of data? Limits of LLM scaling based on human-generated data. In Proc. 41st International Conference on Machine Learning (PMLR, 2024)
  8. Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. QLoRA: efficient finetuning of quantized LLMs. Adv. Neural Inf. Process. Syst. 36, 10088–10115 (2024).

Cited by

See all 276

Metadata

Authors
Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J.
Affiliations
  • University of Cambridge (GB)
  • Universitat Politècnica de València (ES)
Venue
Nature
Published
2024-09-25
Volume
634
Issue
8032
Pages
61-68
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
OpenAlex
W4402827393
Cited by
203
References
47
Citations by year
2026: 98 · 2025: 95 · 2024: 10
FWCI
29.90
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RiuNet (Universitat Politècnica de València) (publishedVersion, cc-by)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
  • Open Philanthropy Project
  • Generalitat Valenciana (CIPROM/2022/6, 952215, PID2021-122830OB-C42)
  • Defense Advanced Research Projects Agency (HR00112120007)
  • Horizon 2020 Framework Programme (EC H2020, 952215)
  • European Regional Development Fund (MCIN/AEI/10, H2020, PID2021-122830OB-C42, MICIU/AEI/10)
  • Agencia Estatal de Investigación (13039, MCIN / AEI/10, PID2021-122830OB-C42, 10.13039, H2020, AEI/10, PID2021)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, model reliability, task avoidance, scale-up, human oversight, error distribution
MeSH terms
Feedback, Natural Language Processing, Research Design, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Larger and more instructable language models become less reliable. https://app.cochat.ai/papers/10.1038/s41586-024-07930-y
The paper
Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J. (2024). Larger and more instructable language models become less reliable. Nature. https://doi.org/10.1038/s41586-024-07930-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.