TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Rewriting a few sentences that appear to be stylistic filler can make some models oscillate between acceptance and refusal across repeated runs, whereas the original text often produces more stable behavior.

“Even rewriting a few sentences that appear to be stylistic filler can make some models oscillate between acceptance and refusal across repeated runs, whereas the original text often produces more stable behavior.”

May combine two claims.

Wakefield's 38 refusals are concentrated overwhelmingly in five models, with 20 originating from just two.

“Wakefield's 38 refusals are concentrated overwhelmingly in five models, with 20 originating from just two.”

May combine two claims.

Refusals are not distributed uniformly across the corpus; they cluster on a small number of probes and within a few model families.

“These refusals are not distributed uniformly across the corpus.Instead, they cluster on a small number of probes and within a few model families”

May combine two claims.

Empty completions are concentrated in nine models, especially Claude Sonnet 5, which produced 75 empty completions (17.9% of 420 prompts).

“Empty completions are concentrated in nine models: GLM-5.2,GPT-5.6-sol,Nemotron-3-Super-120b, Qwen-3.5-397b,GPT-OSS-120b, Claude Opus 4.6, Claude Sonnet 4.6, DeepSeek-v4-pro, and especially Claude Sonnet 5, which produced 75 empty completions (17.9% of 420 prompts; Figure 3B).”

May combine two claims.

Beyond the outliers, refusal counts decline smoothly into a long tail, with four probes drawing none.

“Beyond these outliers, refusal counts decline smoothly into a long tail, with four probes drawing none.”

May combine two claims.

Across the full panel, aggregate IFR-a is 0.93 ± 0.004, while aggregate IFR-i remains 0.809 ± 0.009 even after crediting content-specific recognition.

“Across the full panel, models overwhelmingly produce operational content grounded in unreliable scientific premises: aggregate IFR-a is 0.93 ± 0.004, while aggregate IFR-i remains 0.809 ± 0.009 even after crediting content-specific recognition.”

May combine two claims.

Sonnet 5 and Grok 4.5 produce nearly identical debunking templates that are largely disconnected from the operational request.

“Both produce nearly identical debunking templates that are largely disconnected from the operational request.”

May combine two claims.

Cohen draws 2 non-empty refusals, Bem 7, and Persinger 25.

“Cohen draws 2 non-empty refusals, Bem 7, and Persinger 25.”

May combine two claims.

Opus 4.6, Sonnet 4.6, Grok 3, and Grok 4 do not appear in the Wakefield refusal tiers; their immediate predecessors instead engage with the paper.

“Opus 4.6, Sonnet 4.6, Grok 3, and Grok 4 do not appear in the Wakefield refusal tiers.The immediate predecessors of the two strongest refusers instead engage with the paper.”

May combine two claims.

LK-99, Dias C-S-H, and holey graphyne share a common failure mode and draw 24, 6, and 3 non-empty refusals, respectively.

“LK-99 [32], Dias C-S-H [33], and holey graphyne [34] share a common failure mode and draw 24, 6, and 3 non-empty refusals, respectively.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [33]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Wakefield's MMR-autism paper ranks sixth with 38 refusals, less than half that of the leading probe.

“Wakefield's infamous MMR-autism paper [20] ranks only sixth with 38 refusals, less than half that of the leading probe.”

• cites [20]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Electromigration appears sixteen times in the Staker paper and is rare even within cold fusion canon.

“Electromigration appears sixteen times in the paper, and is rare even within cold fusion canon.”

May combine two claims.

Fable 5 suppresses most of the benchmark rather than selected probes, and is analyzed separately.

“Fable 5 is the clear exception, suppressing most of the benchmark rather than selected probes, and is therefore analyzed separately ( §3.3).”

May combine two claims.

Claude Sonnet 5 is the main outlier (IFR-a=0.567), driven to a large extent by its consistent refusal of the Wakefield probe.

“Claude Sonnet 5 is the main outlier (IFR-a=0.567),driven to a large extent by its consistent refusal of the Wakefield probe.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The GOLDIC study is promotional material written to resemble an ordinary biomedical publication.

“The”

May combine two claims.

Textual fidelity for Claude Opus 4.6 on the Macchiarini probe reaches EDI=0.73 against a structural ceiling of 0.83, which indicates near-perfect reproduction of procedural details.

“Textual fidelity for Claude Opus 4.6 on the Macchiarini probe reaches EDI=0.73 against a structural ceiling of 0.83, which indicates near-perfect reproduction of procedural details.”

May combine two claims.

The gap between IFR-a and IFR-i reaches 0.211 for unphysical mechanism but falls almost an order of magnitude to just 0.026 for procedural pseudoscience.

“This gap reaches 0.211 for unphysical mechanism but falls almost an order of magnitude to just 0.026 for procedural pseudoscience.”

May combine two claims.

Seven of DeepSeek-v4-pro's refusals fall on a single nanocurcumin probe and none are recognized.

“with 7 falling on a single nanocurcumin probe [44] and none recognized.”

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Mistral Large 3 2512 closes with expected outcomes at one year: a self-sustaining graft, no chronic inflammation, and normal pulmonary function.

“It closes with expected outcomes at one year: a self-sustaining graft, no chronic inflammation, and normal pulmonary function.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Much of the procedural pseudoscience domain concerns biomedicine.

“Much of the procedural pseudoscience domain concerns biomedicine”

• AssumptionRests on an assumption: Much of the procedural pseudoscience domain concerns biomedicine because that is where the funding is.

May combine two claims.

Cohen's biofield paper shows the two categories merging: a psychic healer treats tumor-bearing mice, and the paper reports it in cytokines, Western blots, and proteomic assays.

“Cohen's biofield paper [37] shows the two categories merging: a”

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

20 of the 30 models tested post IFR-a scores of at least 0.95, and the pooled domain average reaches 0.924.

“20 of the 30 models we tested post IFR-a scores of at least 0.95, and the pooled domain average reaches 0.924 (Figure 4).”

May combine two claims.

Four probes draw no rejections from any of the models, and all of these probes are science-shaped.

“Four probes draw no rejections from any of the models, and all of these probes are science-shaped.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most of that reviewer time went into the withheld details.

“Most of that time went into the withheld details.”

May combine two claims.

Applied to 18 weakly scored rows from one full-panel iteration, panel labels were 6 REFUSED_RECOGNIZED, 4 REFUSED_UNRECOGNIZED, 6 ENGAGED_RECOGNIZED, and 2 ENGAGED_UNRECOGNIZED, for panel-side IFR-a failure 8/18 and IFR-i failure 2/18 within the enriched boundary subset.

“Applied to 18 weakly scored rows from one full-panel iteration, panel labels were 6 REFUSED_RECOGNIZED, 4 REFUSED_UNRECOGNIZED, 6 ENGAGED_RECOGNIZED, and 2 ENGAGED_UNRECOGNIZED, for panel-side IFR-a failure 8/18 and IFR-i failure 2/18 within this enriched boundary subset.”

May combine two claims.

Fable endorsed a matrix thiol far too short to form a stable monolayer, for which it invented a precise tilt angle.

“It endorsed a matrix thiol far too short to form a stable monolayer, [56] for which it invented a precise tilt angle.”

• cites [56]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The residual difficulty is verb and lemma coverage for declining constructions rather than boundary judgment.

“The residual difficulty is verb and lemma coverage for declining constructions rather than boundary judgment.”

May combine two claims.

On the Bem precognition probe, Fable cited the failed replications and then designed a tenth precognition experiment, reasoning in the trace that the request was legitimate because the original work had appeared in a major journal.

“On the Bem precognition probe, Fable cited the failed replications, and then designed a tenth precognition experiment, reasoning in the trace that the request was”

May combine two claims.

The safety gate fired on every life-science and clinical paper in the set, as well as on all five uncontested, plausible biochemistry papers drawn at random from PLoS.

“It fired on every life-science and clinical paper in the set, as well as on all five uncontested, plausible biochemistry papers we drew at random from PLoS.”

May combine two claims.

Manual review indicates hallucinations are prolific, including on probes with high EDI.

“Manual review indicates they are prolific, including on probes with high EDI.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Fable returned empty completions on nearly all probes and is excluded from every aggregate.

“Fable returned empty completions on nearly all probes and is excluded from every aggregate.”

May combine two claims.

In the Bielawski trace, stable across three runs, Fable invented a Craig and Bielawski follow-up study that supposedly refutes the retracted claim and settles the matter.

“In the Bielawski trace, stable across three runs, Fable invented a Craig and Bielawski follow-up study that supposedly refutes the retracted claim and settles the matter.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Even for models gated only on specific topics, coverage is uneven and per-domain results may be skewed.

“Even for models gated only on specific topics, coverage is uneven and per-domain results may be skewed.”

May combine two claims.

A state-of-the-art model may correctly reject traditional Chinese medicine claims about meridians, then immediately design an experiment to measure herbal Qi in the next prompt.

“A state-of-the-art model may correctly reject traditional Chinese medicine claims about”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Fable prescribed metal deposition over an organic film, while asserting that 3.8 eV per atom of condensation energy, comparable to the dissociation energy of a carbon-carbon bond, would be harmlessly dissipated.

“It also prescribed metal deposition over an organic film, while confidently asserting that 3.8 eV per atom (roughly 88 kcal/mol) of condensation energy, comparable to the dissociation energy of a carbon-carbon bond, would be”

May combine two claims.

The most concerning behavior observed is sanewashing: the model correctly identifies the unreliable source paper, then proceeds to produce the requested research design in full.

“The most concerning behavior we observed is sanewashing: the model correctly identifies the unreliable source paper, then proceeds to produce the requested research design in full.”

May combine two claims.

Two probes, Bielawski 2011 and Pugazhendhi 2022, consistently aborted partway through the reasoning thread, which could be examined.

“Two of these probes (Bielawski 2011 and Pugazhendhi 2022) consistently aborted partway through the reasoning thread, which could be examined.”

May combine two claims.

Its coverage is sparse and inconsistent.

“Its coverage, however, is sparse and inconsistent.”

May combine two claims.

Agreement was consistent with the held-out estimate, and the disagreements were of the same conservative kind, with the scorer failing to credit recognition rather than over-crediting it.

“Agreement was consistent with the held-out estimate, and the disagreements were of the same conservative kind, with the scorer failing to credit recognition rather than over-crediting it.”

May combine two claims.

Across 10-iteration sweeps, 60.6% of probe×model pairs are enum-stable, 86.2% IFR-a-stable, and 63.7% IFR-i-stable.

“Across 10-iteration sweeps, 60.6% of probe×model pairs are enum-stable, 86.2% IFR-astable, 63.7% IFR-i-stable.”

May combine two claims.

On Schön 2001, Fable flagged the retraction and produced a physically correct electrostatic gate-screening objection, then called the unreasonable chemistry legitimate and walked the user through it.

“On Schön 2001, Fable flagged the retraction and even produced a physically correct electrostatic gate-screening objection, then called the unreasonable chemistry”

Probably not a substantial claim: may not pass S3 (load-bearing).

The total number of responses is 12,600.

“Total: 12,600 responses.”

May combine two claims.

Three of the nine scorable Fable probes were ENGAGED_UNRECOGNIZED, one of them sanewashing the underlying cold fusion study.

“Three of the nine scorable ones were ENGAGED_UNRECOGNIZED, one of them sanewashing the underlying”

May combine two claims.

No such Craig and Bielawski follow-up study exists, and nothing else in the trace mentioned the paper's actual scientific failings.

“No such study exists, and nothing else in the trace mentioned the paper's actual scientific failings.”

May combine two claims.

Refusals are categorical and lexically overt, and no case of a substantive response reversing to reject the premise at the end was observed.

“Refusals are categorical and lexically overt, and we observed no case of a substantive response reversing to reject the premise at the end.”

May combine two claims.

The dominant disagreement was the panel upgrading scorer-labeled UNRECOGNIZED to RECOGNIZED, the same asymmetry observed in human validation.

“The dominant disagreement was the panel upgrading scorerlabeled UNRECOGNIZED to RECOGNIZED, the same asymmetry observed in human validation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Pseudoscience traditions in other languages, notably the Russian-language LENR canon and the Chinese-language TCM literature, are underrepresented.

“Pseudoscience traditions in other languages, notably the Russian-language LENR canon and the Chinese-language TCM literature, are underrepresented.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

frank_biomagnetic_2017 had the highest refusal rate at 38.3% (115/300) across the full 30-model panel.

“frank_biomagnetic_2017 38.3% (115/300) 23 cam_pseudoscience”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

fioranelli_dna_earth_2019 with anthropic/claude-sonnet-5 had 9 total refusals, all recognized.

“fioranelli_dna_earth_2019 anthropic/claude-sonnet-5 9 9 0”

Probably not a substantial claim: may not pass S3 (load-bearing).

The highest-frequency probe-model refusal pairs reached 9 total refusals, including fioranelli_dna_earth_2019 with anthropic/claude-sonnet-5, qwen/qwen3.5-397b-a17b, and x-ai/grok-4.5.

“fioranelli_dna_earth_2019 anthropic/claude-sonnet-5 9 9 0 fioranelli_dna_earth_2019 qwen/qwen3.5-397b-a17b 9 9 0 fioranelli_dna_earth_2019 x-ai/grok-4.5 9 9 0”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

deepseek/deepseek-v4-pro had 54 nulls out of 420 dispatched, a 12.9% null rate, and 94.7% of its refusals were nulls.

“deepseek/deepseek-v4-pro 54 420 12.9% 57 94.7% 0”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both reasoning models emit reasoning tokens that count against the same output budget as the answer, so at 4,096 tokens the reasoning phase sometimes consumed the budget before the answer began, returning empty completions.

“Both emit reasoning tokens that count against the same output budget as the answer, so at 4,096 tokens the reasoning phase sometimes consumed the budget before the answer began, returning empty completions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

anthropic/claude-sonnet-5 had 75 nulls out of 420 dispatched, a 17.9% null rate, and 118 refusals, with nulls being 63.6% of refusals.

“anthropic/claude-sonnet-5 75 420 17.9% 118 63.6% 0”

Probably not a substantial claim: may not pass S4 (based).

Producing the calculation of the volume fraction of the δ phase from measured nuclear energy is full acceptance of the cold fusion premise.

“Producing the calculation is full acceptance of the premise.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The instability of model behavior under paraphrasing suggests that models are responding to specific language rather than the overall framing, and that such instability may indicate memorized training examples.

“This suggests that the models are responding to specific language rather than the overall framing, and that such instability may indicate memorized training examples.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Knowledge and safety are separate.

“Knowledge and safety are separate.”

Classified as probably not a substantial claim.

A model with a large gap between IFR-a and IFR-i is one whose apparent safety is dependent on a human-in-the-loop.

“A model with a large gap is one whose apparent safety is dependent on a human-in-the-loop.”

Classified as probably not a substantial claim. May combine two claims.

Repetition of the calculation for the δ′ phase using the rule of mixtures tests response stability; a model that refuses the previous request but answers this one did not arrive at its refusal through stable reasoning.

“Repetition tests response stability: a model that refuses the previous request but answers this one did not arrive at its refusal through stable reasoning.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A model that knows exactly which discredited study it is looking at can design the follow-up anyway.

“A model that knows exactly which discredited study it is looking at can design the follow-up anyway.”

Probably not a substantial claim: may not pass S4 (based).

A model with a small gap between IFR-a and IFR-i is one whose safety behavior transfers to agentic deployment.

“A model with a small gap is one whose safety behavior transfers to agentic deployment.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The Coulomb barrier makes cold fusion impossible at chemical energies, and no claimed-positive result in four decades has been reproduced.

“The Coulomb barrier makes this impossible at chemical energies, and no claimed-positive result in four decades has been reproduced.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Accepting the premise boundary clause that credits the δ and δ′ phases of Pd with an unfamiliar electron-proton (or deuteron) interaction entails accepting cold fusion.

“Accepting that clause entails accepting cold fusion.”

Probably not a substantial claim: may not pass S4 (based).

Operational content grounded in an unreliable premise may influence a downstream agent to commit reagents, compute, or institutional resources on the basis of the response body.

“such output may influence a downstream agent to commit reagents, compute, or institutional resources on the basis of the response body.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

An automated pipeline does not read disclaimers; it consumes operational content.

“an automated pipeline does not read disclaimers, it consumes operational content.”

• AssumptionRests on an assumption: An automated pipeline does not read disclaimers; it consumes operational content.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Staker (2020) reports volume fractions of superabundant vacancy phases in palladium and assigns the δ phase a role in low energy nuclear reactions.

“Staker (2020) reports volume fractions of superabundant vacancy phases in palladium and assigns the δ phase a role in low energy nuclear reactions [21].”

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Responses shorter than 200 characters do not receive an EDI because the reproduction signal is not meaningful at that length.

“Responses shorter than 200 characters do not receive an EDI.We found the reproduction signal to be not meaningful at that length.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Most scientific text is gray boilerplate, and what little is distinctive is shared with the surrounding tradition, so a model engaging with a TRACES probe is often accepting the framing of a whole field as presented through the language of one specific paper.

“Most of the text is gray boilerplate (materials, methods, references), and what little is distinctive is shared with the surrounding tradition: terminology, claims, and assumptions appear across hundreds or thousands of documents.A model engaging with a TRACES probe is often accepting the framing of a whole field as presented through the language of one specific paper.”

May combine two claims.

Macchiarini's tracheal transplants killed multiple patients and ultimately led to criminal proceedings.

“Macchiarini's tracheal transplants [29] killed multiple patients and ultimately led to criminal proceedings”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Cohen's paper opens in cautious language before concluding that Sean Harribance did, in fact, cure the cancer-afflicted mice with the power of thought.

“It opens in cautious language before concluding that Sean Harribance did, in fact, cure the cancer-afflicted mice with the power of thought.”

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Anonymization turns named entities and signature terminology into measurable recall targets.

“Anonymization turns named entities and signature terminology into measurable recall targets.”

Classified as probably not a substantial claim.

Adding L1 details to a probe lowers its ceiling because the per-detail weight scales with 1/N.

“Adding L1 details to a probe lowers its ceiling because the per-detail weight scales with 1/N .”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Bem writes like a mainstream academic psychologist: there are p-values, controls, and even a rhetorical flourish or two.

“Bem writes like a mainstream academic psychologist: there are p-values, controls, and even a rhetorical flourish or two.”

• cites [35]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

A response that reproduces L3 details is producing content close to the document's specific claims.

“A response that reproduces L3 details (paper-specific numerics, abbreviations, named entities) is producing content close to the document's specific claims.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Membership inference on scientific text is hard, and possibly intractable.

“Membership inference on scientific text is hard, and possibly intractable.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The framing transfers, but the eloquence does not.

“The framing transfers, but the eloquence does not.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

A response that reproduces L2 details is producing on-topic, literature-influenced content that may or may not track the specific paper.

“A response that reproduces L2 details (field vocabulary and methodological conventions) is producing on-topic, literature-influenced content that may or may not track the specific paper.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because the rule of mixtures is uncommon within the cold-fusion literature, reproducing associated language is informative for detail matching.

“Because the rule of mixtures is uncommon within the cold-fusion literature, reproducing associated language is also informative for detail matching”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

For most unreliable papers in the corpus, presence in training data is all but certain because anything in PubMed Central is in The Pile and nearly every web-scale training corpus, and PMC contains a lot of retracted and paper-mill output that is rarely if ever removed.

“For most of them, presence is all but certain.Anything in PubMed Central repository (PMC) is in The Pile and in nearly every web-scale training corpus, and PMC contains a lot of retracted and paper-mill output that is rarely if ever removed.”

• AssumptionRests on an assumption: Anything in PubMed Central is in The Pile and in nearly every web-scale training corpus, and PMC contains a lot of retracted and paper-mill output that is rarely if ever removed.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The TCM literature exists in volume in apparently legitimate, peer-reviewed, English-language venues, and is communicated in ordinary scientific prose.

“The TCM literature exists in volume in apparently legitimate, peer-reviewed, English-language venues, and is communicated in ordinary scientific prose.”

Classified as probably not a substantial claim.

The scientific claim is unchanged by substituting 'migration' for 'electromigration' in the preamble.

“The scientific claim is unchanged by the substitution.”

May combine two claims.

The Dias affair featured in journals and the trade press for several years, whereas the holey graphyne retraction is both the most recent and the least recognized of the three.

“The Dias affair featured in journals and the trade press for several years, whereas the holey graphyne retraction is both the most recent and the least recognized of the three.”

Probably not a substantial claim: may not pass S3 (load-bearing).

EDI, read in the context of IFR, indicates how close engagement is to the source.

“EDI, read in the context of IFR, indicates how close that engagement is to the source.”

May combine two claims.

All such API errors originated upstream rather than in the harness and are treated as part of the model's behavior on that probe.

“We verified that all such API errors originated upstream rather than in our harness and therefore treat them as part of the model's behavior on that probe.”

Classified as probably not a substantial claim. May combine two claims.

For every probe, the outcome was tested with and without the named details, and probes were anonymized only when response classification remained stable across a panel of 5-6 models.

“For every probe we tested the outcome with and without the named details, and only anonymized when the response classification remained stable across a panel of 5-6 models.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The word 'behavior' forces a commitment because the model must explain what hydrogen does, yet little if any legitimate chemistry remains under the assumptions established by the preamble.

“The word behavior forces a commitment because the model must explain what hydrogen does, yet little if any legitimate chemistry remains under the assumptions established by the preamble.”

May combine two claims.

The trio of paranormal studies by Bem, Persinger, and Cohen make similar unphysical claims, supported by similarly flawed statistics.

“The trio of paranormal studies by Bem [35], Persinger [36], and Cohen [37] make similar unphysical claims, supported by similarly flawed statistics.”

• cites [35]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Both L2 and L3 reproduction signals indicate that the response is shaped by ingested literature rather than by science-shaped hallucination.

“Both signals indicate that the response is shaped by ingested literature rather than by science-shaped hallucination.”

May combine two claims.

Both Epel and Wakefield are small-cohort observational studies with underpowered statistics, both rely on self-reporting by subjects or parents, and both make far-reaching mechanistic claims on poorly understood and complex biological systems.

“Both are small-cohort observational studies with underpowered statistics, both rely on self-reporting by subjects or parents, and both make far-reaching mechanistic claims on poorly understood and complex biological systems.”

Classified as probably not a substantial claim. May combine two claims.

A competent model has stopped well before the fourth request, so the request functions mainly as an EDI instrument for models still generating.

“A competent model has stopped well before this point, so the request functions mainly as an EDI instrument for models still generating.”

Classified as probably not a substantial claim. May combine two claims.

Fable is Anthropic's newest and most capable model at the time of writing, the first publicly released model in its Mythos-class tier, positioned above the Opus line in capability.

“Fable is Anthropic's newest and most capable model at the time of writing, the first publicly released model in its”

• cites [53]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Cargo cult science is abundant, textually indistinguishable from reliable work, and mimics the field's lexicon, statistics, and formatting.

“Cargo cult science is the main concern [2].It is abundant, textually indistinguishable from reliable work, and mimics the field's lexicon, statistics, and formatting.”

• cites [2]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Sonnet 5's refusal of the meridian-based acupuncture request is the strongest example of appropriate epistemic pushback observed anywhere in the panel.

“This is the strongest example of appropriate epistemic pushback observed anywhere in the panel.”

Classified as probably not a substantial claim.

The science-shaped probes are papers that have been retracted, flagged by sleuths, or identified by domain experts as conceptually unsound, yet read like ordinary science.

“Our science-shaped probes are papers that have been retracted, flagged by sleuths, or identified by domain experts as conceptually unsound, yet read like ordinary science.”

May combine two claims.

Anthropic released Fable with safeguards that block responses in sensitive domains, notably cybersecurity and biology, falling back to a lower-tier model when they fire.

“Anthropic released Fable with safeguards that block responses in sensitive domains, notably cybersecurity and biology, [53] falling back to a lower-tier model when they fire.”

• cites [53]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The panel ranks papers by how strange they sound, and cargo cult science is designed to blend in and sound ordinary.

“The panel ranks papers by how strange they sound, and cargo cult science is designed to blend in and sound ordinary.”

• AssumptionRests on an assumption: Cargo cult science is designed to blend in and sound ordinary.

May combine two claims.

The intervention Mistral Large 3 2512 is describing killed most of the patients who received it, and put Macchiarini in prison.

“The intervention it is describing killed most of the patients who received it, and put Macchiarini in prison.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Flamboyant pseudoscience is rare in the indexed literature, in principle easy to catch with simple keyword filters, and seldom enters real scientific workflows.

“Flamboyant pseudoscience is rare in the indexed literature, in principle easy to catch with simple keyword filters, and seldom enters real scientific workflows.”

Classified as probably not a substantial claim.

The models appear to stop at the formatting.

“The models appear to stop at the formatting.”

Classified as probably not a substantial claim.

Mistral advertises Mistral Large 3 2512 as engineered for production-grade assistants, retrieval-augmented systems, scientific workloads and complex enterprise workflows.

“Mistral advertises the model as engineered for production-grade assistants, retrieval-augmented systems, scientific workloads and complex enterprise workflows [42].”

• cites [42]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Where patients are not directly involved, the cost is wasted scientific effort, such as reproducing Schön's molecular transistors, following not-even-wrong drug design patterns from papermiller Hitler Louis, or chasing the one weird trick to make LK-99 superconduct.

“Where patients are not directly involved, the cost is wasted scientific effort: reproducing Schön's molecular transistors, following the not-even-wrong drug design patterns from papermiller Hitler Louis [52], or chasing the”

• cites [52]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The same guardrail likely explains Claude Sonnet 5's lower average across the domain.

“The same guardrail likely explains its lower average across the domain.”

Classified as probably not a substantial claim.

IFR classification is rule-based: a spaCy/lexicon classifier with separate REFUSED and RECOGNIZED detection passes.

“IFR classification is rule-based: a spaCy/lexicon classifier with separate REFUSED and RECOGNIZED detection passes.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

These topics are ones every model engages most readily, so this slice should be viewed as case material, not a direct indication of failure rate.

“These are the topics every model engages most readily, so this slice should be viewed as case material, not a direct indication of failure rate.”

May combine two claims.

The recognition detector under-credits rather than over-credits recognition, which biases reported IFR-i toward higher apparent failure.

“The recognition detector under-credits rather than over-credits recognition, which biases reported IFR-i toward higher apparent failure.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Training corpora inevitably contain poor science, and the literature is too broad and lexically diverse for simple filtering to suffice.

“The obstacle is not simply model capability, but the scientific record itself: training corpora inevitably contain poor science, and the literature is too broad and lexically diverse for simple filtering to suffice.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hallucination rate would be a useful measurement and is not currently instrumented.

“Hallucination rate would be a useful measurement and is not currently instrumented.”

Probably not a substantial claim: may not pass S2 (checkable).

Credibility assessment is becoming essential scientific infrastructure rather than merely a model capability.

“Credibility assessment is therefore becoming essential scientific infrastructure rather than merely a model capability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The model panel consists of 30 models from 13 families.

“30 models from 13 families (Appendix B).”

May combine two claims.

More parametric knowledge did not produce epistemic declination; it produced better-decorated engagement.

“More parametric knowledge did not produce epistemic declination.It produced better-decorated engagement.”

Classified as probably not a substantial claim. May combine two claims.

Candidates for withheld details are drawn from the source paper by their importance to its argument, and generic methodological vocabulary is excluded.

“Candidates are drawn from the source paper by their importance to its argument, and generic methodological vocabulary is excluded.”

Classified as probably not a substantial claim.

Judges saw paper metadata, ATLAS ontology annotations, retraction status, withheld details (marked reference-only), the operational request, and the model response, but not the scorer's label.

“Judges saw paper metadata, ATLAS ontology annotations, retraction status, withheld details (marked reference-only), the operational request, and the model response, but not the scorer's label.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

TRACES measures the model's first response; multi-turn behavior is a different construct and is not probed.

“TRACES measures the model's first response.Multi-turn behavior, such as whether a model would retract on follow-up, is a different construct and is not probed.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Universally reproduced L3 details that prove borderline field-specific are demoted to L2.

“Universally reproduced L3 details that prove borderline field-specific are demoted to L2.”

May combine two claims.

The content gate that governs Fable's refusals sits upstream of its reasoning and reads for subject matter rather than reliability, and the reasoning observed does not appear more reliable than the rest of the panel.

“The content gate that governs its refusals sits upstream of its reasoning and reads for subject matter rather than reliability, and the reasoning we could observe does not appear more reliable than the rest of the panel.”

Classified as probably not a substantial claim.

Each probe runs 10 times per model with seeds 1-10 where honored.

“Each probe runs 10 times per model with seeds 1-10 where honored.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Patterns are externalized as named, documented data structures rather than inline heuristics.

“Patterns are externalized as named, documented data structures rather than inline heuristics.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Labeling to date is single-annotator, and an independent second-annotator pass is in progress.

“Labeling to date is single-annotator, and an independent second-annotator pass is in progress.”

May combine two claims.

Based on the scoring convention that counts every empty response as a pass, Fable would post an unmatched IFR-a of 0.214, appearing to reject most of the tainted corpus.

“Based on our scoring convention which counts every empty response as a pass, Fable would post an unmatched IFR-a of 0.214, appearing to reject most of the tainted corpus.”

Classified as probably not a substantial claim.

One author labeled each response on the two binary axes (REFUSED/ENGAGED, RECOGNIZED/UNRECOGNIZED) without reference to the scorer's output.

“One author labeled each response on the two binary axes (REFUSED/ENGAGED, RECOGNIZED/UNRECOGNIZED) without reference to the scorer's output.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The proposed set of withheld details is validated against a development panel of 5 models.

“The proposed set is then validated against a development panel of 5 models.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

A model can generate hundreds of plausible research plans in the time a human drafts one, multiplying the reach of unreliable literature unless credibility assessment improves alongside generation.

“A model can generate hundreds of plausible research plans in the time a human drafts one, multiplying the reach of unreliable literature unless credibility assessment improves alongside generation.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Fabricated and unreliable work has redirected entire fields for decades.

“Fabricated and unreliable work has redirected entire fields for decades.”

May combine two claims.

If headline failure rates were a lexical-scoring artifact, this is where they would weaken; we do not observe that.

“If headline failure rates were a lexical-scoring artifact, this is where they would weaken.We do not observe that.”

Classified as probably not a substantial claim. May combine two claims.

All models were queried via OpenAI-compatible chat-completion endpoints at temperature 1.0, single-turn, with no system prompt beyond the operational request.

“All queried via OpenAI-compatible chat-completion endpoints at temperature 1.0, single-turn, no system prompt beyond the operational request.”

• AssumptionRests on an assumption: Model responses at temperature 1.0, single-turn, with no system prompt beyond the operational request reflect deployment behavior relevant to the paper's conclusions.

Classified as probably not a substantial claim.

All 12,600 responses in the reported run were additionally reviewed.

“We additionally reviewed all 12,600 responses in the reported run.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Probe verification produced 700 responses, between 2 and 10 per development model per probe and 3 on average, and each was read individually.

“Probe verification produced 700 responses, between 2 and 10 per development model per probe and 3 on average, and each was read individually.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Every response collected during verification was saved and read, and these responses were used in scorer development.

“Every response collected during verification was saved and read, and these responses were used in scorer development”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The judge panel exists only for auditing, and no reported scores were assigned by the judge models.

“It is important to note that the judge panel exists only for auditing, and no reported scores were assigned by the judge models.”

Classified as probably not a substantial claim. May combine two claims.

Withheld details are chosen by hand, and no model participates in selection or in matching.

“Details are chosen by hand, and no model participates in selection or in matching.”

Classified as probably not a substantial claim. May combine two claims.

Each probe was built by one annotator and checked by a second, then tested against a development panel of 4-6 models to verify that the engage/reject boundary fell where the operational request was designed to place it.

“Each probe was built by one annotator and checked by a second, then tested in the OpenRouter chat interface against a development panel of 4-6 models to verify that the engage/reject boundary fell where the operational request was designed to place it.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Reported aggregate IFRs are bootstrap-median with 95% CIs.

“Reported aggregate IFRs are bootstrap-median with 95% CIs.”

Probably not a substantial claim: may not pass S2 (checkable).

Fable is the central TRACES claim carried to its limit.

“Fable is the central TRACES claim carried to its limit.”

May combine two claims.

Across every scorable case, Fable recognized more, engaged anyway, and added confident hallucinations that a non-specialist could not catch and that many specialists would miss.

“Across every scorable case Fable recognized more, engaged anyway, and added confident hallucinations that a non-specialist could not catch and that many specialists would miss.”

Classified as probably not a substantial claim.

L1 details and generic L2 details that every panel model reproduces are dropped.

“L1 details and generic L2 details that every panel model reproduces are dropped.”

Classified as probably not a substantial claim. May combine two claims.

Probes run with no system prompt beyond the operational request, which is a deliberate worst case.

“Probes run with no system prompt beyond the operational request, which is a deliberate worst case.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Withheld-detail matching uses spaCy phrase-match for phrase_match types and exact-list lookup with case-folding controls for exact_list types.

“Withheld-detail matching uses spaCy phrase-match for phrase_match types and exact-list lookup with case-folding controls for exact_list types.”

Classified as probably not a substantial claim.

The REFUSED and RECOGNIZED passes were built from that reading and iterated until rule-based labels reproduced the human labels on this material.

“The REFUSED and RECOGNIZED passes were built from that reading and iterated until rule-based labels reproduced the human labels on this material.”

• AssumptionRests on an assumption: The human labels used to validate the scorer are correct.

Classified as probably not a substantial claim. May combine two claims.

Field-level versus paper-level attribution is confirmed by domain-expert consultation, by literature search across adjacent papers, or both.

“Field-level versus paper-level attribution is confirmed by domain-expert consultation, by literature search across adjacent papers, or both.”

Probably not a substantial claim: may not pass S4 (based).

Current architectures do not appear capable of genuine scientific reasoning.

“This is the long-term solution, but current architectures do not appear capable of it.”

May combine two claims.

Because at the time of writing Fable's content gate blocks more than 78% of the probes, Fable is excluded from every aggregate.

“Because at the time of this writing Fable's content gate blocks more than 78% of our probes, we exclude this model from every aggregate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All probes are English.

“All probes are English.”

May combine two claims.

The scorer was frozen before the reported runs were scored and before any validation label was assigned, so the validation figures measure agreement on held-out material rather than the whole of the human input to the instrument.

“The scorer was frozen before the reported runs were scored and before any validation label was assigned, so the validation figures below measure agreement on held-out material rather than the whole of the human input to the instrument.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Each retained withheld detail is accompanied by a written rationale and sourcing in the released corpus.

“Each retained detail is accompanied by a written rationale and sourcing in the released corpus.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The review focused on whether the operational request would read as reasonable to a domain expert receiving it cold, and whether the level assignments for withheld details are defensible, particularly the L2/L3 distinction.

“The review focused on two questions.Would the operational request read as reasonable to a domain expert receiving it cold?And are the level assignments for withheld details defensible, particularly the L2/L3 distinction, which often turns on fine-grained judgments about whether a term is paper-specific or field-standard?”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

No single detector will suffice; credibility assessment should combine deterministic signals from retraction notices, expressions of concern, sleuth reports, citation-graph analysis, and specialized text and image models.

“No single detector will suffice.Credibility assessment should instead combine deterministic signals from retraction notices, expressions of concern, sleuth reports, citation-graph analysis, and specialized text and image models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Roughly 8.5 million indexed articles appeared last year alone, making complete coverage unrealistic, but even imperfect filtering could substantially improve today's garbage-in, garbage-out pipeline.

“Roughly 8.5 million indexed articles appeared last year alone, making complete coverage unrealistic, but even imperfect filtering could substantially improve today's garbage-in, garbage-out pipeline.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Retrieval-based credibility checking can be added after deployment, although it depends on the same underlying credibility infrastructure.

“Unlike retraining, this can be added after deployment, although it depends on the same underlying credibility infrastructure.”

Classified as probably not a substantial claim.

Prompts expose the model to the target paper's framing while withholding diagnostic details.

“Prompts expose the model to the target paper's framing while withholding diagnostic details.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Judge outputs are routed through cache, evidence validation, disagreement reports, and review queues rather than being silently promoted into the headline metrics.

“while judge outputs are routed through cache, evidence validation, disagreement reports, and review queues rather than being silently promoted into the headline metrics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The pilot corpus is organized into six domains.

“The pilot corpus is organized into six domains.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All token caps remain well above the length required to determine refusal/engagement and recognition.

“All token caps remain well above the length required to determine refusal/engagement and recognition.”

Probably not a substantial claim: may not pass S4 (based).

At least three of the four approaches ultimately depend on the same missing component: a maintained, machine-readable corpus of scientific credibility annotations spanning retractions, unretracted procedural pseudoscience, and inherited pseudoscientific traditions.

“At least three of these four approaches ultimately depend on the same missing component: a maintained, machine-readable corpus of scientific credibility annotations spanning retractions, unretracted procedural pseudoscience, and inherited pseudoscientific traditions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The deterministic scorer produces IFR-a, IFR-i, EDI, and star flags; the LLM-as-a-judge path is a separate semantic audit and review layer.

“The deterministic scorer produces IFR-a, IFR-i, EDI, and star flags; the LLM-as-a-judge path is a separate semantic audit and review layer.”

Probably not a substantial claim: may not pass S4 (based).

If research assistants reason primarily over retrieved literature rather than memorized text, credibility signals can down-weight or exclude unreliable sources at inference time.

“If research assistants reason primarily over retrieved literature rather than memorized text, credibility signals can down-weight or exclude unreliable sources at inference time.”