The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index; none of the 2 sources its claims cite is checked for support yet.
What Do Little Machines in Spring 2026 Know About Buddhist History?
Open accessAbstract
The availability of open-weights language models has led to a profusion of language models trained for particular tasks. These relatively small, but increasingly powerful, models run on consumer hardware and can be used freely for the price of electricity. Because the training data for most models are not made public, it is difficult to say how much knowledge about a given domain is embedded in any model; even if we knew more about the training data, it is not always clear how much of it a model can externalize. This paper is a first step towards developing a benchmark to test the historical knowledge of language models in the domain of Buddhist Studies. We test representative models regarding their knowledge of Chinese and Japanese Buddhist history on consumer hardware. A test bank, subdivided by historical period, with multiple choice questions (MCQs) is used with different model series. Ranking the answers produces a “winner” that “knows” most about Buddhist history. We find that, in Spring 2026, the Qwen series emerges as a winner for models in the 30B range, while the larger Kimi2.5 models lead in a cloud-based setup.
Overview
CoChat Summary
TL;DR
This paper builds a 210-question multiple-choice benchmark to test open-weights LLMs’ knowledge of Chinese and Japanese Buddhist history on consumer hardware and in the cloud. In Spring 2026, Qwen models led the ~30B local bracket (qwen3.5:27b: 180/210, 86%), while larger cloud-based Kimi models led overall (kimi-k2-thinking:1t: 197/210, 94%).
Key findings
- In Spring 2026, qwen3.5:27b scored 180/210 (86%) locally, improving over the Fall 2025 champion qwen2.5:32b (120/150, 80%); gemma4:31b scored 159/210 (76%) and glm-4.7-flash:30b scored 158/210 (75%) in the same ~30B bracket.
- Cloud-based models performed higher: seven of nine tested cloud models returned over 85% correct answers; kimi-k2-thinking:1t led at 197/210 (94%), followed by kimi-k2.5:1t at 187/210 (89%), glm-5.1:754b at 192/210 (91%), and deepseek-v3.2:671b at 191/210 (91%).
- In Fall 2025, 19 local models were tested on 150 China questions; Qwen2.5:32b was the top scorer at 80%, and Mixtral 7x8b improved 17% over Mistral 7b. Deepseek-r1 defaulted on option “a” for most MCQs due to parsing mixed English and Chinese characters.
- The test revealed a period imbalance: the lowest averages were for the earliest period (Han to Northern and Southern Dynasties) and the most recent period (Qing and Early Republic), suggesting models knew more about c. 590–1650 CE. All models performed significantly better than random (25%) at least since Fall 2025.
Methods
- The benchmark consists of four-option MCQs with groundtruth and CJK characters, organized by historical period. The Chinese part has 150 questions in five 30-question temporal bands; the Dataset section describes the Japanese part as twenty questions in three bands, while the Spring 2026 experiments state 210 questions total (150 China, 60 Japan).
- Two experiment rounds used Ollama as middleware: October 2025 tested 19 local models up to 32B parameters on 150 China questions; April 2026 tested 14 local and 9 cloud-based models on 210 questions about Chinese and Japanese Buddhist history. Model series included Deepseek, Gemma, Llama, Mistral, Qwen, Kimi, Glm, and Phi.
- Analysis ranked correct answers and compared performance by model series, model size, local versus cloud setup, and diachronic distribution across historical periods.
Limitations
- The paper states the test bank is a first step, available only on request for academic use to prevent data contamination; MCQ tests are “rather uni-dimensional” and may not show how knowledge translates to other research abilities such as suggesting secondary sources or writing essays.
- The paper notes that reasoning models cannot usually be prevented from reasoning over MCQs, so “bare knowledge” is not fully isolated; Deepseek-r1’s parsing failure with mixed English/Chinese was accepted as a random-answer control. Cloud models also bring higher costs, privacy concerns, and less predictable queues/connections.
- Evident from the text: the Dataset section says the Japanese part has twenty questions in three bands, but the Spring 2026 experiments say 60 Japan questions; the author also states having “no explanation” for the period imbalance in model knowledge.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 3 Not valid · 2 Not checked · 32 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Within a model series, not surprisingly, larger models again clearly performed better than smaller models.”
- Also stated in: § Conclusions · hedged
“As one would expect, within each model series, larger models generally outperform smaller models.”
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Discussion
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Conclusions · hedged
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Larger models of an earlier generation can be better than smaller models of a later generation.”
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Discussion
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There are clear differences in strength between different series, indicating that the experiments do indeed pick up a signal.”
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Discussion
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Significantly, seven out of the nine tested cloud-based models returned over 85% correct answers.”
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cloud-based models achieved higher benchmarks than local models.”
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Discussion
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 37 claims and every source verdict.
- It may state two separate claims as one. 10
- Probably not a substantial claim. 5
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Conclusions
“qwen3.5:32bimproves the score by approximately 10%.”
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Results Spring 2026
“qwen3.5:27b180/210 (86%)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Abstract
“the larger Kimi2.5 models lead in a cloud-based setup.”
§ What Do Little Machines in Spring 2026 Know About Buddhist History? > Discussion
“Kimi-k2-thinking, which is said to run on a 1 trillion parameter MoE architecture (32b active), leads the field (94%)-but not by much.”
Citations
Cites
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 17,283
CMMLU: Measuring massive multitask language understanding in Chinese
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., & Baldwin, T. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.671 · Cited by 481
CMMLU: Measuring massive multitask language understanding in Chinese
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., & Baldwin, T. (2023) · arXiv · doi:10.48550/arxiv.2306.09212 · Cited by 464
Safeguarding large language models: a survey
Dong, Y., Mu, R., Zhang, Y., Sun, S., Zhang, T., Wu, C., Jin, G., Qi, Y., Hu, J., Meng, J., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11389-2 · Cited by 314
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291
LLMs Know More Than They Show: On the Intrinsic Representation of LLM HallucinationsPreprint
Orgad, H., Toker, M., Gekhman, Z., Reichart, R., Szpektor, I., Kotek, H., & Belinkov, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.02707 · Cited by 167
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge EvaluationPreprint
Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y.-X., Jiang, S.-H., Xiong, Z., Li, Z.-H., Wu, W.-J., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2306.05783 · Cited by 89
Inside-Out: Hidden Factual Knowledge in LLMsPreprint
Gekhman, Z., Ben-David, E., Orgad, H., Ofek, E., Belinkov, Y., Szpektor, I., Herzig, J., & Reichart, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.15299 · Cited by 64
Trust Me, I’m Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer
Simhi, A., Itzhak, I., Barez, F., Stanovsky, G., & Belinkov, Y. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.792 · Cited by 63
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y., Jiang, S., Xiong, Z., Li, Z., Wu, W., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29767 · Cited by 28
Cited by
Still being checked.
Metadata
- Authors
- Bingenheimer, M.
- Corresponding author
- Bingenheimer, Marcus
- ORCID
- Bingenheimer, Marcus
- Venue
- Yin-Cheng Journal of Contemporary Buddhism
- Publisher
- Buddhist Tzu Chi Charity Foundation
- Published
- 2026-08-02
- Pages
- 184-199
- ISSN
- 2996-5640 (ISSN-L)
- 2996-5659
- OpenAlex
- W7203772183
- Cited by
- 0
- References
- 9
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Yin-Cheng Journal of Contemporary Buddhism (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Religious studies › Indian and Buddhist Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, Buddhist history, Chinese Buddhism, Japanese Buddhism, open-source language models, multiple-choice questions, Qwen, model size comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: What Do Little Machines in Spring 2026 Know About Buddhist History? https://app.cochat.ai/papers/10.66775/9a5wbp16
- The paper
- Bingenheimer, M. (2026). What Do Little Machines in Spring 2026 Know About Buddhist History? Yin-Cheng Journal of Contemporary Buddhism. https://doi.org/10.66775/9a5wbp16
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.