CoChat
Sign Up
Cited by 5Natural Language Processing TechniquesTopic ModelingAI-based Problem Solving and Planning

Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages

Open access
Zuo, Max, Velez, Francisco Piedrahita, Li, Xiaochen, Littman, Michael, Bach, Stephen·2025·Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
DOI 10.18653/v1/2025.naacl-long.560
Open this paper in CoChat — free accountOpen PDF

Abstract

Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael Littman, Stephen Bach. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

AI snapshot

Written from the paper’s full text.

TL;DR

Planetarium is a benchmark for evaluating language models on translating natural language planning descriptions into PDDL, combining a formal PDDL equivalence algorithm with 145,918 text-to-PDDL pairs across 73 tasks. It finds that models often generate parseable and solvable PDDL that is still semantically incorrect, with GPT-4o zero-shot producing 96.1% parseable, 94.4% solvable, but only 24.8% correct outputs.

Key findings

  • Planetarium contains 145,918 text-to-PDDL pairs across 73 unique state combinations in Blocks World, Gripper, and a simplified Floor Tile domain, varying by abstractness and problem size.
  • The benchmark’s equivalence algorithm checks PDDL problem equivalence via scene graphs, goal fully specification, and graph isomorphism; it takes on average 12ms per example on an M2 Apple Silicon laptop with batch parallelization.
  • GPT-4o zero-shot generated 96.1% parseable and 94.4% solvable PDDL problems, but only 24.8% were semantically correct; fine-tuned Gemma 2 27B IT achieved the highest accuracy among tested models at 34.2%.
  • Error analysis found models often ignore the in-context domain and add `:typing` even though the Blocks World and Gripper domains used do not have typing, and performance correlated with a domain’s frequency in the wild.

Methods

  • The benchmark has two components: a PDDL equivalence algorithm that transforms problem files into scene graphs, fully specifies goal scenes, and checks isomorphism, and a procedurally generated dataset built from handcrafted templates for Blocks World, Gripper, and simplified Floor Tile.
  • Evaluation covered API-access GPT-4o and o1-mini plus open-weight Gemma 2 2B/9B/27B IT in zero-shot and fine-tuned settings, measuring parseable, solvable, and correct generated PDDL on the Planetarium test set.
  • Solvability was checked with specialized planners for Blocks World and Gripper, Fast Downward for Floor Tile, and VAL for plan validation; correctness was then determined by the PDDL equivalence algorithm.

Limitations

  • Stated: Planetarium currently supports only Blocks World, Gripper, and Floor Tile, and is restricted to the STRIPS subset of PDDL; extending to more expressive subsets would allow evaluation of non-deterministic, temporal, and numeric domains.
  • Stated: inaccurate evaluation of natural-language-to-planning-language translation could produce misleading or misaligned results if such systems become widespread.
  • Evident from the text: o1-mini was evaluated on a random sample of 250 problems per domain due to prohibitive costs, and fine-tuning truncated the longest 5% of the training dataset due to GPU memory constraints.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2025.naacl-long.560

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 22 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Match-based metrics cannot cover the vast space of equivalent programs.

    “but match-based metrics cannot cover the vast space of equivalent programs (Chen et al., 2021).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chen et al., 2021 source
  • Premisecitation

    Existing work uses plan validators to check if generated code can be solved with a traditional planner.

    “existing work uses “plan validators” to check if the generated code can be solved with a traditional planner”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Liu et al., 2023 source

      The Method section says LLM+P feeds LLM-generated PDDL into a classical planner to generate a plan, and the Experiment Setup says it feeds generated PDDL into FAST-DOWNWARD. Those passages support the planner component, but they do not describe using “plan validators” to check whether generated code can be solved. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Silver et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Guan et al., 2023 source

      Section 4.2 says PDDL validators check generated models for errors, and the planning experiments say a plan validator checks generated plans for unmet preconditions or goals. But the paper uses a classical planner to find plans; it does not say that plan validators check whether generated code can be solved by that planner. Page numbers are unavailable in our copy. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Study design and conductcitation

    The equivalence algorithm is efficient for three domains: Blocks World, Gripper, and a slightly simplified Floor Tile.

    “We show how to make this algorithm efficient for three domains: Blocks World, Gripper, and a slightly simplified Floor Tile (McDermott, 2000;Linares López et al., 2015).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites McDermott, 2000

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Linares López et al., 2015 source
  • Findingnone given

    Each task in Planetarium varies in two dimensions to assess difficulty: abstraction and size.

    “Each task varies in two dimensions to assess the difficulty of PDDL generation: abstraction and size.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    Instances with abstract descriptions or many propositions are particularly challenging.

    “Instances with abstract descriptions or many propositions are particularly challenging.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 19
  • May combine two claims. 17
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Chrpa et al. and Sievers et al. use a lifted graph representation of the planning domain, with Chrpa et al. specifically using graph isomorphism to check equivalence.

    “Chrpa et al. (2023) and Sievers et al. (2021) use a lifted graph representation of the planning domain, with the former specifically using graph isomorphism to check equivalence.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Chrpa et al. (2023) source
    • cites Sievers et al. (2021)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The open-weight models were fine-tuned using QLoRA with a rank of 16, adhering to the hyperparameter recommendations for small models provided by the original authors (Dettmers et al., 2023).

    “We fine-tuned the open-weight models using QLoRA with a rank of 16, adhering to the hyperparameter recommendations for small models provided by the original authors (Dettmers et al., 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Dettmers et al., 2023 source

      The Additional information section reports using LoRA rank 16 for small, medium, and large T5 models, so the rank has a basis in the paper. But its QLoRA finetuning hyperparameters specify rank 64, and the paper says rank does not affect final performance when adapters are used on all layers. It therefore does not provide a general rank-16 recommendation for small open-weight models. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    LLMs have limited ability to generate and validate plans on their own, even for simple planning tasks.

    “One of the main findings is that LLMs have limited ability to generate and validate plans on their own, even for simple planning tasks (Valmeekam et al., 2022;Silver et al., 2022;Valmeekam et al., 2023b).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Valmeekam et al., 2022 source
    • cites Silver et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Valmeekam et al., 2023b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Planetarium contains a dataset of 145,918 text-to-PDDL pairs derived from the International Planning Competition, across 73 unique state combinations.

    “along with a dataset of 145,918 text-to-PDDL pairs across 73 unique state combinations with varying levels of difficulty.”
    • Also stated in: § Benchmark Data for PDDL Generation. · plain
      “We present a dataset based on the International Planning Competition (IPC) (McDermott, 2000;Vallati et al., 2015;Taitler et al., 2024;Seipp et al., 2022), crafting 145,918 ground truth PDDL problems and corresponding text descriptions capturing a range of planning problems.”
    • Also stated in: § 4.2 The Dataset · plain
      “The Planetarium dataset includes 145,918 text-to-PDDL pairs derived from the Blocks World, Gripper, and Floor Tile domains (Seipp et al., 2022;McDermott, 2000).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites McDermott, 2000

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Vallati et al., 2015 source

      The 'Domain Models Introduced' section discusses IPC planning domains and PDDL features, but not a Planetarium dataset. Searches for Planetarium, 145,918 text-to-PDDL pairs, and 73 state combinations found no passage addressing the dataset or those figures. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Taitler et al., 2024 source

      The Classical Track history and motivation section discusses IPC planning domains and PDDL, but not Planetarium or a text-to-PDDL dataset. Searches for Planetarium, text-to-PDDL pairs, 145,918, and 73 state combinations found no passage establishing the claimed dataset size, composition, or derivation from the competition. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    The evaluation finds that while powerful models like GPT-4o can often generate valid PDDL problems, they are rarely correct.

    “We find that while powerful models like GPT-4o can often generate valid PDDL problems (in the sense that some plan solves them), they are rarely correct.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Fine-tuning improved performance across all open-weight models, with fine-tuned Gemma 2 27B IT achieving the highest accuracy of all tested models at 34.2%.

    “Fine-tuning improved performance across all open-weight models, with fine-tuned Gemma 2 27B IT achieving the highest accuracy of all tested models at 34.2%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In zero-shot setting, o1-mini and GPT-4o performed the best, followed closely by Gemma 27B IT.

    “In the zero-shot setting, o1-mini and GPT-4o performed the best, followed closely by Gemma 27B IT.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Planetarium currently only supports the Blocks World, Gripper, and Floor Tile domains.

    “Planetarium currently only supports the Blocks World, Gripper, and Floor Tile domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The simplified Floor Tile domain has 6 predicates and 9 actions compared to 10 predicates and 7 actions in the IPC version.

    “resulting in a domain with 6 predicates and 9 actions (compared to 10 predicates and 7 actions in the IPC version).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Planetarium is currently restricted to the STRIPS subset of PDDL.

    “This benchmark is also currently restricted to the STRIPS subset of PDDL.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4o's PDDL problem descriptions are 96.1% syntactically parseable, 94.4% solvable, and 24.8% semantically correct.

    “For example, 96.1% of the PDDL problem descriptions generated by GPT-4o are syntactically parseable, 94.4% are solvable, but only 24.8% are semantically correct”
    • Also stated in: § 6 Conclusion · plain
      “Models like GPT-4o can often produce valid, seemingly correct descriptions (94.4%) when, in reality, only a small fraction (24.8%) are genuinely correct.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The equivalence algorithm takes, on average, 12ms per example to compute on an M2 Apple Silicon laptop with batch parallelization.

    “We found our equivalence algorithm to take, on average, 12ms per example to compute on an M2 Apple Silicon laptop with batch parallelization.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Blocks World and Gripper domains do not have typing, yet when generating PDDL, these models confidently and consistently add object types.

    “our Blocks World and Gripper domains do not have typing, yet when generating PDDL, these models confidently and consistently add object types.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The data for Planetarium is available on HuggingFace and released under a Creative Commons CC-BY-4.0 license.

    “The data for Planetarium is available on Hugging-Face 1 and released under a Creative Commons CC-BY-4.0 license.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    All code to create the dataset and evaluate models on the benchmark is available on GitHub and released under a BSD 3 license.

    “In addition, all code to create the dataset and evaluate models on the benchmark is available on GitHub 2 and released under a BSD 3 license.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Using validators alone is insufficient to determine if PDDL generation is correct because an LLM can generate valid PDDL unrelated to the user's instructions and still be considered correct.

    “We argue using validators alone is insufficient to determine if PDDL generation is correct.This is because the LLM can generate valid PDDL that has nothing to do with the user's instructions and still be considered correct-a false positive.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Leaving planning to LLMs does not guarantee correctness, making them unreliable for critical applications.

    “In contrast, leaving the planning to LLMs does not guarantee correctness, making them unreliable for critical applications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    There are no rigorous techniques or benchmarks for evaluating the translation of natural language planning descriptions to PDDL.

    “Despite its promise, there are no rigorous techniques or benchmarks for evaluating the translation of natural language planning descriptions to PDDL.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    Existing evaluation methods for translating natural language planning descriptions to PDDL struggle to ensure semantic correctness and rely on simple or unrealistic datasets.

    “Existing evaluation methods struggle to ensure semantic correctness and rely on simple or unrealistic datasets.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Planetarium is a new benchmark for assessing the ability of LLMs to translate natural language descriptions of planning problems into PDDL.

    “Planetarium is a new benchmark for assessing the ability of LLMs to translate natural language descriptions of planning problems into PDDL.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    Many instances of PDDL can represent the same planning problem, but it is not always obvious when they are equivalent.

    “This is hard because many instances of PDDL can represent the same planning problem, but it is not always obvious when they are equivalent (Figure 1).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The expressiveness of problems defined in STRIPS is equivalent to those characterized by the set-theoretic definition of classical planning problems.

    “The expressiveness of problems defined in STRIPS is equivalent to those characterized by the set-theoretic definition (Definition 1) of classical planning problems (Ghallab et al., 2004).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ghallab et al., 2004

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    PDDL can represent various types of planning problems, including classical planning problems.

    “It can represent various types of planning problems, including classical planning problems (McDermott, 2000;Gerevini and Long, 2006).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites McDermott, 2000

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Gerevini and Long, 2006

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisereasoning

    Planetarium employs graph isomorphism but focuses on evaluating equivalence of PDDL problem instances rather than domains.

    “Planetarium also employs graph isomorphism, but our focus is on evaluating the equivalence of PDDL problem instances rather than domains.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The authors built specialized planners for Blocks World and Gripper problems that work on all problems in the dataset and generally all validly defined blocksworld and gripper domain problems except for a few invalid edge cases.

    “Instead, we built specialized planners for Blocks World and Gripper problems that work on all problems in our dataset and generally all validly defined blocksworld and gripper domain problems except for a few invalid edge cases (e.g., one block on top of two blocks at a time, holding two blocks, etc.).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Planetarium focuses on evaluating Planner-Augmented LLMs, determining whether an LLM can translate natural language descriptions of planning problems into correct PDDL representations.

    “In contrast, Planetarium focuses on evaluating “Planner-Augmented LLMs”.Here, the goal is to determine whether an LLM can successfully translate natural language descriptions of planning problems into the correct PDDL representations of those problems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    The paper formally defines planning problem equivalence and creates an algorithm for checking whether two PDDL problems satisfy this definition.

    “We formally define planning problem equivalence and create an algorithm for checking whether two PDDL problems satisfy this definition.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The specific domain files used in the research are obtained from pddl-generators (Seipp et al., 2022).

    “The specific domain files used in our research are obtained from pddl-generators (Seipp et al., 2022), which is distributed under the GNU General Public License.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Seipp et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Blocksworld, Gripper, and Floor Tile domains used in the study are sourced from the IPC (McDermott, 2000).

    “In our study, we use the Blocksworld, Gripper, and Floor Tile domains, which are sourced from the IPC (McDermott, 2000).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites McDermott, 2000

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Larger problems typically pose greater challenges for LLMs.

    “Larger problems typically pose greater challenges for LLMs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Equivalent(P_a, P_b, False) assumes that the domain files are shared between P_a and P_b.

    “Finally, to prove Property 4, we observe that Equivalent(P a , P b , False) assumes that the domain files are shared between P a and P b .”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The domain file for Floor Tile is a modified version of the one found from pddl-generators (Seipp et al., 2022).

    “The domain file for Floor Tile is a modified version of the one found from pddl-generators (Seipp et al., 2022)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Seipp et al., 2022 source
32 of 32

  • The authors state this as an assumption.

    The transition function γ is shared between the two planning problems being compared.

    “Our definition assumes that the transition function γ is shared between the two problems.”

  • The authors state this as an assumption.

    The entire PDDL domain is shared between the problems being compared.

    “We assume the entire PDDL domain is shared between the problems.”

  • The paper relies on this without support.

    PDDL uses the open-world assumption for goals, so goal propositions can leave implicit trivially true propositions.

    “Since PDDL uses the open-world assumption for goals, we have to identify all propositions that are true in all reachable goal states when starting from the initial state for each problem.”

  • The paper relies on this without support.

    Gripper, and to a lesser extent Blocks World, is often defined with :typing.

    “However, Gripper, and to a lesser extent, Blocks World, is often defined with :typing.”

  • The paper relies on this without support.

    If all blocks have their behavior above defined, then any block that does not have its bottom behavior specified must be on the table.

    “If all blocks have its behavior above it defined (they are either clear, have something on top of it, or is being held), then any block that does not have its bottom behavior specified, must be on the table (on-table).”

  • The paper relies on this without support.

    If all blocks have their behavior beneath defined, then any block that does not have its behavior above defined must be clear.

    “If all blocks have its behavior beneath it defined (they are either on top of something else, are on the table (on-table), or are being held), then any block that does not have its behavior above it defined must be clear.”

  • The paper relies on this without support.

    If there is only one room and all grippers are already specified, then any unspecified balls must be in the only room that exists.

    “If there is only one room, and if all grippers are already specified (either (free) or (carry ball gripper)), then any unspecified balls must be in the only room that exists.”

  • The paper relies on this without support.

    A tile cannot be unpainted.

    “A tile cannot be unpainted.”

  • The paper relies on this without support.

    All available colors are unchangeable and cannot be added.

    “All available colors (available-color color) are unchangeable and cannnot be added.”
9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Equivalence algorithm overwrites the first problem's scene graphs

    Minor

    Line 3 assigns the scene graphs for P b to the same variables used for P a on line 2. As written, this overwrites P a's graphs and never assigns the s i,b and g b variables used later in the algorithm.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. A. Gerevini and D. Long. 2006. Plan constraints and preferences in PDDL3. In ICAPS Workshop on Soft Constraints and Preferences in Planning.
  2. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
  3. Yunhao Yang, Jean-Raphael Gaglione, Cyrus Neary, and ufuk topcu. 2023a. Large language models for verifiable sequential decision-making in autonomous systems. In 2nd Workshop on Language and Robot Learning: Language as Grounding.

Cited by

Sorted by citations

Related

Authors
Zuo, M., Velez, F. P., Li, X., Littman, M., & Bach, S.
Affiliations
Brown University (US)
ORCID
Venue
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2025
Pages
11223-11240
DOI
10.18653/v1/2025.naacl-long.560
OpenAlex
W4411119285
Semantic Scholar
a5ff668294a0eb7216edda6c42e76ba6b18ebcb3
Cited by
5
References
44
Citations by year
2026: 4 · 2025: 1
FWCI
4.95
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • Cisco Systems
  • Office of Naval Research (N00014-20-1-2115, N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
planetarium, benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-19.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-19.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review