The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index; 2 of the 15 sources its claims cite were checked for support.
Larger and more instructable language models become less reliable
Open accessAbstract
The prevailing methods to make large language models more powerful and amenable have been based on continuous scaling up (that is, increasing their size, data volume and computational resources 1 ) and bespoke shaping up (including post-filtering 2,3 , fine tuning or use of human feedback 4,5 ). However, larger and more instructable large language models may have become less reliable. By studying the relationship between difficulty concordance, task avoidance and prompting stability of several language model families, here we show that easy instances for human participants are also easy for the models, but scaled-up, shaped-up models do not secure areas of low difficulty in which either the model does not err or human supervision can spot the errors. We also find that early models often avoid user questions but scaled-up, shaped-up models tend to give an apparently sensible yet wrong answer much more often, including errors on difficult questions that human supervisors frequently overlook. Moreover, we observe that stability to different natural phrasings of the same question is improved by scaling-up and shaping-up interventions, but pockets of variability persist across difficulty levels. These findings highlight the need for a fundamental shift in the design and development of general-purpose artificial intelligence, particularly in high-stakes areas for which a predictable distribution of errors is paramount.
Overview
CoChat Summary
TL;DR
Across 32 LLMs from the GPT, LLaMA and BLOOM families on five benchmarks, scaling and instruction/alignment shaping improved correctness and prompt stability but did not create reliable low-difficulty operating regions. Shaped-up models increasingly replaced avoidance with plausible but incorrect answers, making errors harder for human supervisors to catch.
Key findings
- Human difficulty proxies predicted correctness, but no error-free easy region emerged: for LLaMA, no model achieved 60% correctness at the simplest difficulty level (discounting 25% random guess for science), and only GPT-4 showed a low-difficulty science region with almost perfect results up to medium difficulty.
- Scaling and shaping exchanged avoidance for incorrectness: avoidance almost disappeared for GPT-4, incorrectness increased markedly from raw to shaped-up models, and ultracrepidarianism—incorrect over avoidant plus incorrect answers, i/(a + i)—rose.
- Prompting stability improved with scaling and shaping, but pockets of variability persisted across difficulty levels; raw GPT and all LLaMA models were highly prompt-sensitive even for unambiguous addition.
- Human supervision study S2 found few areas where the dangerous error of considering an incorrect answer correct was low enough to constitute a safe operating region.
Methods
- Analysed 32 models: 10 GPT, 10 LLaMA and 12 BLOOM, including raw and shaped-up versions, across five benchmarks: addition, anagram, locality, science and transforms. Each example was run with 15 natural prompt templates per benchmark, yielding approximately 4.2 million answers; temperature was set to zero and no system prompt was used.
- Data samples: 3,142 addition instances, 1,570 anagrams, 2,341 locality instances, science from 1,000 OpenBookQA questions plus 546 GPQA questions minus two over-length items, and 730 transforms items (73 tasks × 10 instances). Two human studies (S1 and S2) recruited participants via Prolific at £9 per hour.
- Difficulty was calibrated to a 0–100 human-failure scale via logistic mapping, then examples were split into 30 equal-sized difficulty bins. Analyses included Spearman rank correlations, six reliability indicators for correctness and prudence (proportion, prompting stability, difficulty concordance), and a scaling analysis using FLOPs.
Limitations
- Stated: participants were mostly non-experts, which affects calibrated difficulty values; the natural prompt sample came from diverse sources but lacked real-world prompt-frequency data; and only a sample of model families was covered, excluding LLMs that delegate to external tools or use sophisticated reasoning techniques.
- Evident: GPT models were accessed through APIs and may change over time; response scoring relied on algorithmic conditions and regular expressions, which may not capture all response nuance; calibrated difficulty values are estimates from human study S1 and should be interpreted cautiously, especially for small differences.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
1 Valid · 0 Not valid · 9 Not checked · 58 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“F3b—improvement in prompt performance is not monotonic across difficulty levels.”
§ Larger and more instructable language models become less reliable > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the LLaMA family, no model achieves 60% correctness at the simplest difficulty level (discounting 25% random guess for ‘science’).”
§ Larger and more instructable language models become less reliable > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The percentage of avoidant answers rarely rises quicker than the percentage of incorrect ones.”
§ Larger and more instructable language models become less reliable > Results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“scaled-up, shaped-up models do not secure areas of low difficulty in which either the model does not err or human supervision can spot the errors”
§ Larger and more instructable language models become less reliable > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supplementary Table 8 shows the correlations between correctness and the proxies for human difficulty. Except for BLOOM for addition, all of them are high.”
§ Larger and more instructable language models become less reliable > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
18 more claims
Create a free account to see all 68 claims and to verify the 13 references still open.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 27
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“while its intrinsic fallibility, the tendency to produce confident but incorrect outputs, demands continuous human oversight.”
From Generative AI and Organizational Structure in the Knowledge Economy (2025) · cites this paper as Zhou et al. 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Second, they struggle with many aspects of reasoning (Shwartz, 2024;Zhou et al., 2024;Huckle and Williams, 2025).”
From There is no such thing as conscious artificial intelligence (2025) · cites this paper as Zhou et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“our result aligns with and extends recent findings that earlier LLMs were more likely to avoid answering difficult questions, whereas newer, larger, and more instructible models, instead of refusing to answer, often produced misleadingly authoritative yet flawed responses [41].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [41]
The cited source says what the claim attributes to it.
In the citing paper
“Zhou et al. (2024) document that more capable language models can exhibit equal or higher hallucination rates on certain task categories”
From Generative AI and Organizational Structure in the Knowledge Economy (2025) · cites this paper as Zhou et al. (2024)
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1×
“Although this prompt sensitivity has been analysed extensively 17–20 , it is poorly understood why an over-diligent system spouts a wrong answer for 100-digit addition instead of simply answering ‘I’m afraid I can’t do that’.”
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“Although this prompt sensitivity has been analysed extensively 17–20 , it is poorly understood why an over-diligent system spouts a wrong answer for 100-digit addition instead of simply answering ‘I’m afraid I can’t do that’.”
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Research works printed but not linked: 8
- Kim, G., Baldi, P. & McAleer, S. Language models can solve computer tasks. Adv. Neural Inf. Process. Syst. 36, 39648–39677 (2024).
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics 4171–4186 (Association for Computational Linguistics, 2019).
- Sun, J., Shaib, C. & Wallace, B. C. Evaluating the zero-shot robustness of instruction-tuned language models. In The Twelfth International Conference on Learning Representations (2024).
- Zeidner, M. & Matthews, G. Evaluation anxiety. Handb. Competence Motiv. 141, 163 (2005).
- Kuhn, L., Gal, Y. & Farquhar, S. Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations (2023).
- Li, M. et al. Think twice before assure: confidence estimation for large language models through reflection on multiple answers. Preprint at https://arxiv.org/html/2403.09972v1 (2024).
- Villalobos, P. et al. Position: Will we run out of data? Limits of LLM scaling based on human-generated data. In Proc. 41st International Conference on Machine Learning (PMLR, 2024)
- Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. QLoRA: efficient finetuning of quantized LLMs. Adv. Neural Inf. Process. Syst. 36, 10088–10115 (2024).
Cited by
Human-AI agency in the age of generative AI
Krakowski, S. (2025) · Information and Organization · doi:10.1016/j.infoandorg.2025.100560 · Cited by 162
Harmonizome 3.0: integrated knowledge about genes and proteins from diverse multi-omics resources
Diamant, I., Clarke, D. J. B., Evangelista, J. E., Lingam, N., & Ma’ayan, A. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae1080 · Cited by 194
Mirakhori, F., & Niazi, S. K. (2025) · Pharmaceuticals · doi:10.3390/ph18010047 · Cited by 216
Generalization bias in large language model summarization of scientific research
Peters, U., & Chin-Yee, B. (2025) · Royal Society Open Science · doi:10.1098/rsos.241776 · Cited by 126
AI hallucinations can’t be stopped — but these techniques can limit their damage
Jones, N. (2025) · Nature · doi:10.1038/d41586-025-00068-5 · Cited by 117
Metadata
- Authors
- Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J.
- Affiliations
- University of Cambridge (GB)
- Universitat Politècnica de València (ES)
- ORCID
- Venue
- Nature
- Published
- 2024-09-25
- Volume
- 634
- Issue
- 8032
- Pages
- 61-68
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC11446866
- OpenAlex
- W4402827393
- Cited by
- 203
- References
- 47
- Citations by year
- 2026: 98 · 2025: 95 · 2024: 10
- FWCI
- 29.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RiuNet (Universitat Politècnica de València) (publishedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- Open Philanthropy Project
- Generalitat Valenciana (CIPROM/2022/6, 952215, PID2021-122830OB-C42)
- Defense Advanced Research Projects Agency (HR00112120007)
- Horizon 2020 Framework Programme (EC H2020, 952215)
- European Regional Development Fund (MCIN/AEI/10, H2020, PID2021-122830OB-C42, MICIU/AEI/10)
- Agencia Estatal de Investigación (13039, MCIN / AEI/10, PID2021-122830OB-C42, 10.13039, H2020, AEI/10, PID2021)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, model reliability, task avoidance, scale-up, human oversight, error distribution
- MeSH terms
- Feedback, Natural Language Processing, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Larger and more instructable language models become less reliable. https://app.cochat.ai/papers/10.1038/s41586-024-07930-y
- The paper
- Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J. (2024). Larger and more instructable language models become less reliable. Nature. https://doi.org/10.1038/s41586-024-07930-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.