CoChat
Sign Up
Cited by 1Software Engineering ResearchTeaching and Learning ProgrammingNatural Language Processing Techniques

From Reasoning to Code: GRPO Optimization for Underrepresented Languages

Open access
PENNINO, FEDERICO, RAIMONDI, BIANCA, RONDELLI, MASSIMO, GURIOLI, ANDREA, GABBRIELLI, MAURIZIO·2026·Theory and Practice of Logic Programming
DOI 10.1017/s1471068426100489
Open this paper in CoChat — free accountOpen PDF

Abstract

Generating accurate and executable code using Large Language Models (LLMs) remains a significant challenge for underrepresented programming languages, such as Prolog and Lisp, due to the scarcity of public training data compared to high-resource languages like Python. This paper introduces a generalizable Reinforcement Learning (RL) approach that combines small-scale versions of the Qwen2.5-Coder model with Group Relative Policy Optimization (GRPO) to enable effective code generation through reasoning. To address the limitations of sparse datasets, we integrate execution-driven feedback directly into the RL loop, utilizing a reward system that exploits both logical correctness and structural formatting. Experimental results on GSM8K dataset demonstrate significant improvements in reasoning quality and code accuracy across underrepresented languages. These findings underscore the potential of our approach to benefit a wide range of programming languages lacking extensive training resources by leveraging symbolic reasoning and interpreter-based feedback.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper applies GRPO with execution-based rewards from a Prolog interpreter to small Qwen2.5-Coder models for underrepresented languages Prolog and Lisp. It finds that removing the KL penalty and adding a length-aware reward avoids “reasoning contraction” and produces large pass@4 gains on GSM8K, GSM-Symbolic, Rosetta Code, and Lisp.

Key findings

  • Standard GRPO with a KL penalty causes “reasoning contraction”: the 7B model’s one-shot GSM8K accuracy degrades from 0.814 to 0.672 over training, and its zero-shot score falls from a 500-step peak of 0.714 to 0.667 at 1500 steps.
  • Removing KL divergence and adding a length-aware reward raises Qwen2.5-Coder-7B one-shot GSM8K pass@4 to 0.893 and zero-shot to 0.882 at 1500 steps, outperforming PROPER (0.72), Llama 3.3 70B (0.839), and Qwen2.5-Coder-32B (0.873).
  • Gains transfer to GSM-Symbolic: p2 zero-shot pass@4 more than doubles from 0.298 to 0.673, and p1 one-shot rises from 0.754 to 0.803. On Rosetta Code, the 1.5B model reaches 0.20 pass@4 within 500 steps, and the 7B model peaks at 0.46 one-shot at 500 steps.
  • For Lisp, zero-shot pass@4 rises from 0.5716 to a peak of 0.8757; one-shot reaches 0.873 without KL. KL removal is notably less impactful for Lisp than for Prolog.

Methods

  • RL with GRPO on Qwen2.5-Coder-Instruct at 0.5B, 1.5B, 3B, and 7B parameters, using LoRA rank 32, TRL, vLLM, and 8×NVIDIA A100-SXM4 GPUs; training uses the official GSM8K training set and evaluation uses the official test split, with checkpoints at 500, 1000, and 1500 steps in zero-shot and one-shot settings.
  • Reward combines format signals for `<reasoning>`, `<code>`, and `<query>` tags (xmlcount, strict, soft) with correctness from executing generated Prolog via PySwip/SWI-Prolog in a sandbox with a 5 s timeout: success +1.0, logical error −1.0, syntax error −0.5, timeout/no output −0.1.
  • Evaluation uses pass@k and pass∧k with k = 4 on GSM8K, 20 Rosetta Code Prolog tasks, GSM-Symbolic p1/p2, and Lisp; ablations add a length reward equal to 1 when 0.009 < r_length(c) < 0.013 and remove the KL-divergence penalty.

Limitations

  • The paper states that GSM8K is arithmetic-focused and that future work could build a new training dataset to better capture non-numerical reasoning patterns.
  • The paper notes a persistent zero-shot versus one-shot gap, suggesting example-based prompting remains essential for correct predicate arity and recursion patterns in underrepresented languages.
  • Evident from the text: experiments are limited to the Qwen2.5-Coder family up to 7B and to Prolog/Lisp, and the Lisp results show that removing KL is less beneficial than in Prolog, so the approach’s generality across other languages or architectures is not established here.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1017/s1471068426100489

Cites

19 of 27 references verified0 citation errors · 8 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 14 cited sources checked · 0 supported · 1 flagged · accuracy not yet measured

  • Premisereasoning

    LLMs' proficiency at generating code in mainstream programming languages such as Python and JavaScript does not transfer to logic programming.

    “LLMs have achieved remarkable proficiency at generating code in mainstream programming languages such as Python and JavaScript, yet this proficiency does not transfer to logic programming.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Rozière et al. (2023) source

      Section 3.1.1 reports Python code-generation results, and Section 3.1.2 reports results across several other programming languages. The paper does not establish the claimed failure to transfer that proficiency to logic programming; the cited passages also do not specifically report JavaScript results. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Li et al . (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Hui et al. (2024) source

      The Multi-Programming Language section (page unavailable) reports strong code-generation results across eight mainstream languages, including Python and JavaScript. It does not evaluate logic programming or establish that proficiency fails to transfer to it; the Code Reasoning section instead evaluates reasoning about Python functions. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • Rests on an assumption: Modern code LLMs' ability to generate Prolog or Lisp is superficial and merely mimics syntax without understanding semantics.
  • Findingown data

    Optimized Qwen2.5-Coder-7B outperforms PROPER (0.72) on Prolog GSM8K.

    “remarkably outperforming PROPER (0.72), a Mistral 7B variant that underwent specialized supervised fine-tuning for Prolog code generation in GSM8K.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Yang et al. (2024) source

      The Setup (§4.1) describes fine-tuning a Mistral 7B model, and Table 1 reports PROPER accuracy on GSM8K. But Table 1 gives 70.2% (0.702), not 0.72; the paper does not evaluate Qwen2.5-Coder-7B or establish that it outperforms PROPER. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
  • Findingown data

    The 7B model's one-shot pass@4 on GSM8K (0.893) outperforms PROPER (0.72).

    “thus outperforming both PROPER, introduced by Yang et al. (2024), a variant of Mistral-7B, presented by Jiang et al. (2023), with specialized supervised fine-tuning (0.72)”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Yang et al. (2024) source

      The Conclusion identifies PROPER as the authors’ data-augmentation method, and the Setup says they fine-tuned a 7B Mistral model. But Table 1 reports 70.2% GSM8K accuracy for Mistral with PROPER, not 0.72; the paper also calls PROPER a method rather than a model variant. The cited paper does not establish the citing model’s 0.893 pass@4 result. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Jiang et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    The 3B model gains +0.436 zero-shot.

    “the 3B model gains + 0.436 zero-shot”
    Quote in the paperSubstantial claim
  • Findingown data

    Standard GRPO fine-tuning on GSM-Symbolic p2 one-shot setting reduced accuracy from 0.588 to 0.201 after 1500 steps.

    “in the one-shot setting on the p2 dataset, accuracy dropped from a baseline of 0.588 to 0.201 after 1500 steps.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 13
  • May combine two claims. 16
  • The quote only partly states this claim. 3

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Mapping math word problems to Prolog predicates yields higher accuracy than Python PoT for problems requiring constraint satisfaction or symbolic manipulation.

    “GSM8K-Prolog, presented by Yang et al. (2024), demonstrated that mapping math word problems to Prolog predicates yields higher accuracy than Python PoT for problems requiring constraint satisfaction or symbolic manipulation.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Yang et al. (2024) source

      Section 4.2 (page unavailable) reports that Python-code generation achieved 55.12% accuracy on GSM8K with Llama2, better than both Prolog and PROPER. The paper reports Prolog outperforming a chain-of-thought baseline, not Python, and does not report a separate accuracy comparison for constraint-satisfaction or symbolic-manipulation problems. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Experimental results on the GSM8K dataset demonstrate significant improvements in reasoning quality and code accuracy across underrepresented languages.

    “Experimental results on GSM8K dataset demonstrate significant improvements in reasoning quality and code accuracy across underrepresented languages.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The 7B model peaked early at 500 steps (0.46 one-shot), nearly doubling its zero-shot baseline.

    “The 7B model peaked early at 500 steps (0.46 one-shot), nearly doubling its zero-shot baseline.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Implementing a length-aware reward mechanism and removing the KL-divergence penalty effectively expanded the model's planning process.

    “By implementing a length-aware reward mechanism and removing the KL-divergence penalty, we effectively expanded the model's planning process.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The length-constrained reward formulation encourages reasoning traces within a target length range.

  • The quote only partly states this claim.

    Findingown data

    The 7B model's one-shot pass@4 on GSM8K (0.893) outperforms Qwen2.5-Coder-32B (0.873).

    “and Qwen2.5-Coder-32B(0.873)”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The 7B model's one-shot pass@4 on GSM8K (0.893) outperforms Llama 3.3 70B (0.839).

    “and substantially larger models including Llama 3.3 70B (0.839), from the model family described by Llama Team (2024)”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Llama Team (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • The quote only partly states this claim.

    Findingown data

    Optimized Qwen2.5-Coder-7B with length constraints and removal of KL divergence achieves a peak one-shot pass@4 score of 0.893 on Prolog GSM8K.

    “our optimized Qwen2.5-Coder-7Bconfigurationutilizing length constraints and the removal of KL divergence -achieves a peak oneshot score of 0.893”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The modifications caused a concurrent increase in both completion length and pass@k performance across zero-shot and one-shot settings.

    “which illustrates the concurrent increase in both completion length and pass@k performance across zero-shot and one-shot settings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The 7B model achieves a high one-shot baseline of 0.81, but its performance worsens during training, settling at 0.67 after 1500 steps.

    “while the 7B model achieves a high one-shot baseline of 0.81, its performance is worsening during training, settling at 0.67 after 1500 steps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Removing KL-divergence had a modest impact in Lisp (0.8712 to 0.8757) compared to transformative gains in Prolog.

    “the impact of removing the KL-divergence was notably more modest in Lisp (0.8712 to 0.8757) than the transformative gains observed in Prolog.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The 7B model without KL reaches 0.882 zero-shot and 0.893 one-shot.

    “the 7B model without KL reaches 0.882 (zero-shot) and 0.893 (one-shot)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The root cause of LLMs' failure on Prolog is distributional, not architectural: these languages are scarce in modern pre-training corpora, and scale cannot compensate for missing data.

    “The root cause is not architectural but distributional: these languages are scarce in modern pre-training corpora, and no amount of scale will compensate for data that does not exist.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Prolog and Lisp are scarce in modern pre-training corpora compared to high-resource languages such as Python.

  • May combine two claims.

    Premisecitation

    GRPO estimates the advantage of each generated output relative to a group of peer outputs sampled from the same prompt, reducing computational overhead and stabilizing learning in sparse-reward code-generation tasks.

    “GRPO estimates the advantage of each generated output relative to a group of peer outputs sampled from the same prompt, reducing computational overhead and stabilizing learning in the sparse-reward regime characteristic of code-generation tasks.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Shao et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The structured formatting strategy increases sample efficiency by penalizing structural deviations, allowing the model to quickly converge on a consistent and executable output format.

    “This structured formatting strategy increases sample efficiency by penalizing structural deviations, allowing the model to quickly converge on a consistent and executable output format.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Program-of-Thought prompting decouples reasoning from computation by having the LLM generate a program that is executed by an external interpreter, significantly boosting accuracy on arithmetic tasks.

    “This offloads calculation to a deterministic engine, significantly boosting accuracy on arithmetic tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chen et al .( 2021)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    By systematically modifying numerical values and contextual details while preserving the underlying logical structure, the GSM-Symbolic setup allows assessing whether models learned to extract and encode core reasoning patterns into Prolog predicates and rules or overfitted to specific problem templates.

    “By systematically modifying numerical values and contextual details while preserving the underlying logical structure, we can assess whether the models have learned to extract and encode the core reasoning patterns into Prolog predicates and rules, or whether their translations are overfitted to specific problem templates.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Baselines for Llama 3.3 70B and Qwen2.5-Coder-32B were evaluated using the same system prompts and XML-based output templates to ensure a fair comparison.

    “Baselines for Llama 3.3 70B and Qwen2.5-Coder32B were evaluated using the same system prompts and XML-based output templates to ensure a fair comparison.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    By embedding a Prolog interpreter directly into the reward loop, the authors obtain a dense, ground-truth correctness signal without needing hand-written tests, turning the interpreter from a development tool into a training supervisor.

    “By embedding a Prolog interpreter directly into the reward loop, we obtain a dense, ground-truth correctness signal without needing hand-written tests, thereby turning the interpreter from a development tool into a training supervisor.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: In domains such as mathematics and programming, the validity of an output can be objectively verified.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The GSM-Symbolic variants increase the complexity of reasoning in a controlled manner, with P2 involving multi-step comparisons across multiple entities while the base version focuses on single-step calculations.

    “These variants increase the complexity of reasoning in a controlled manner: for instance, P2 tasks may involve multi-step comparisons across multiple entities, whereas the base version focuses on single-step calculations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    GSM-Symbolic includes variants GSM-Symbolic-Plus-1 (P1) with one additional reasoning clause and GSM-Symbolic-Plus-2 (P2) with two additional clauses.

    “We included its variants: GSM-Symbolic-Plus-1 (P1), in which each question includes one additional reasoning clause, and GSM-Symbolic-Plus-2 (P2), with two additional clauses.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mirzadeh et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Correctly executing solutions demonstrate the model's ability to translate arithmetic word problems into well-structured predicates with appropriate variable bindings and queries.

    “Correctly executing solutions (green) demonstrate the model's ability to translate arithmetic word problems into well-structured predicates with appropriate variable bindings and queries.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The resulting models were appropriately prompted based on the learning hyperparameter to be consistent with the prompt setting.

    “The resulting models were appropriately prompted based on the learning hyperparameter to be consistent with the prompt setting.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The higher initial Lisp proficiency is likely due to a stronger representation in the model's pretraining data.

    “likely due to a stronger representation in its pretraining data.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Training used only the official GSM8K set, and all reported metrics were computed on the official test split to prevent train-test leakage.

    “Training used only the official GSM8K set, and all reported metrics are computed on the official test split to prevent train-test leakage.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GSM-Symbolic is a benchmark focused on symbolic reasoning over abstract mathematical expressions.

    “We also evaluated models on GSM-Symbolic, a benchmark focused on symbolic reasoning over abstract mathematical expressions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mirzadeh et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The methodology enhances genuine symbolic reasoning and cross-task robustness, enabling handling of structural variations rather than memorization.

    “These results demonstrate that our methodology enhances genuine symbolic reasoning and cross-task robustness, ensuring that the model can handle structural variations in logic problems rather than relying on simple memorization of patterns.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Evaluation metrics and experiment scripts were standardized to facilitate reproducibility and future benchmarking.

    “Evaluation metrics and experiment scripts were standardized to facilitate reproducibility and future benchmarking.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The interpreter-driven approach replaces scarce human data on which current approaches depend, providing a rigorous, automated supervisor that teaches the model to respect the syntax and logic of the target language.

    “This replaces the scarce human data on which current approaches depend, providing a rigorous, automated supervisor that teaches the model to respect the syntax and logic of the target language.”
    Quote in the paperSubstantial claim
28 of 28

  • The paper relies on this without support.

    Prolog and Lisp are scarce in modern pre-training corpora compared to high-resource languages such as Python.

    “due to the scarcity of public training data compared to high-resource languages like Python”
    “these languages are scarce in modern pre-training corpora”

  • The paper relies on this without support.

    In domains such as mathematics and programming, the validity of an output can be objectively verified.

    “This paradigm leverages the fact that, in domains such as mathematics and programming, the validity of an output can be objectively verified.”

  • The paper relies on this without support.

    Modern code LLMs' ability to generate Prolog or Lisp is superficial and merely mimics syntax without understanding semantics.

    “their ability to generate Prolog or Lisp is often superficial and merely mimics the syntax (indentation, parentheses) without understanding the semantics (unification, recursion, macro expansion).”

  • The paper relies on this without support.

    GSM8K's emphasis on structured, logical problem-solving makes it a valuable proxy for reasoning skills.

    “whose emphasis on structured, logical problem-solving makes it a valuable proxy for the reasoning skills.”

  • The paper relies on this without support.

    Publicly accessible Prolog code for supervised training is limited, significantly restricting models' initial exposure to Prolog's syntax.

    “A key challenge was the limited availability of publicly accessible Prolog code for supervised training, significantly restricting the models' initial exposure to Prolog's syntax.”

  • The paper relies on this without support.

    GSM8K's emphasis on structured, logical problem solving makes it a valuable proxy for evaluating and improving the reasoning skills necessary for Prolog code generation.

    “its emphasis on structured, logical problemsolving makes it a valuable proxy for evaluating and improving the reasoning skills necessary for Prolog code generation.”

  • The paper relies on this without support.

    The 20 Rosetta Code tasks represent core logic programming constructs, including recursion, list manipulation, and symbolic backtracking.

    “These 20 tasks were selected to represent core logic programming constructs, including recursion, list manipulation, and symbolic backtracking.”

  • The paper relies on this without support.

    The length-constrained reward formulation encourages reasoning traces within a target length range.

    “This formulation encourages reasoning traces within a target length range.”

  • The paper relies on this without support.

    Underrepresented languages often lack extensive training datasets.

    “underrepresented languages that often lack extensive training datasets”

  • The paper relies on this without support.

    Prolog is a highly specialized, low-resource language.

    “highly specialized, low-resource languages such as Prolog”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Correctness reward table omits the timeout reward

    Minor

    The reward description assigns −0.1 for a timeout or no output, but Table 1 lists only −1, −0.5, and 1 as possible correctness rewards.

  • Lisp baseline is lower than the Prolog baseline

    Minor

    The paper says the base 7B model had higher initial proficiency in Lisp, but its reported zero-shot Lisp score is 0.5716, below the zero-shot Prolog base score of 0.637.

2 of 2

Cites

Flagged first, then by citations

8 of the 23 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
PENNINO, F., RAIMONDI, B., RONDELLI, M., GURIOLI, A., & GABBRIELLI, M.
Affiliations
University of Bologna (IT)
ORCID
Venue
Theory and Practice of Logic Programming
Publisher
Cambridge University Press (CUP)
Published
2026-07-08
Pages
1-14
ISSN
  • 1471-0684 (ISSN-L)
  • 1475-3081
DOI
10.1017/s1471068426100489
OpenAlex
W4415110425
Cited by
1
References
23
Citations by year
2026: 1
FWCI
8.89
Type
journal article
Language
en
Open access
Licence
Locations
  • Theory and Practice of Logic Programming (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
code generation, Group Relative Policy Optimization, Prolog, LISP, Qwen2.5-Coder, reward function design, GSM8K, symbolic reasoning, low-resource languages

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-19.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-19.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review