Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models

Probably not a substantial claim: may not pass S3 (load-bearing).

Reporting follows the TRIPOD-LLM guideline for studies using large language models.

“Reporting follows the TRIPOD-LLM guideline for studies using large language models [11], and a completed checklist is provided as a supplementary file.”

• cites [11] sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The Lyons ophthalmic set was replaced by the registered Zandi set, because its vignette text is published only as an image.

“The Lyons ophthalmic set [16] was replaced by the registered Zandi set, because its vignette text is published only as an image.”

• cites [16] sourcePartly supported

  The Clinical vignette design section confirms that Lyons and colleagues developed an ophthalmic vignette set, so the citation identifies the set correctly. Our copy does not establish that its vignette text was published only as an image. The replacement with the Zandi set is the citing authors’ decision, not a finding attributed to Lyons. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Earlier vignette studies found high diagnostic accuracy for language models on Semigran-style cases, and one found their stated confidence reasonably calibrated by Brier score.

“Earlier vignette studies found high diagnostic accuracy for language models on Semigran-style cases, and one found their stated confidence reasonably calibrated by Brier score [5,6]”

• cites [5] sourcePartly supported

  The Results report 88% top-three diagnostic accuracy on 48 synthetic vignettes and good diagnostic calibration (Brier score 0.18). The Methods say the cases built on symptom-checker work, but the confidence scores were calculated from how often GPT-3 repeated a diagnosis across prompts; they were not confidence values stated by the model. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [6] sourcePartly supported

  The Methods (Overview; page unavailable) describe testing LLMs on 48 synthetic clinical vignettes, and the Results (page unavailable) report high diagnostic accuracy. The copy does not identify these as Semigran-style cases, and a search for Brier score and confidence calibration found no such assessment. Thus, it supports the accuracy portion but not the case provenance or calibration portion. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

A triage system using model probabilities works only if the probabilities are calibrated, meaning a stated probability matches how often the model is actually right.

“Such a system works only if the probabilities are calibrated, meaning that a stated probability matches how often the model is actually right [3].”

• cites [3] sourcePartly supported

  The paper defines calibration as agreement between estimated risk and observed event frequency (Main text, “How to assess calibration?”). It says calibration is especially important for decision-making and that poor calibration can lead to harmful decisions (Main text, “Discrimination is important, but are the risk estimates reliable?”; Conclusions). It does not establish the absolute claim that a probability-based triage system works only if its probabilities are calibrated. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

On the 30 uninformative inputs with a forced choice, Jev placed a mean 58% on its top diagnosis, compared with 37% for Opus 5 and 21% for Opus 5.5.

“On the 30 uninformative inputs with a forced choice, Jev placed a mean 58% on its top diagnosis, compared with 37% for Opus 5 and 21% for Opus 5.5.”

May combine two claims.

On the matched comparison, Jev was more order-sensitive than Opus 5.5 (excess TVD difference 0.015; 95% CI 0.005 to 0.025) and did not differ from Opus 5.

“On the matched comparison, Jev was more order-sensitive than Opus 5.5 (excess TVD difference 0.015; 95% CI 0.005 to 0.025) and did not differ from Opus 5.”

May combine two claims.

On the 80 ophthalmic complaints, top-1 accuracy did not differ between Jev and either comparator (only Jev correct in 3 cases vs 5 for Opus 5, P=.73; 3 vs 3 for Opus 5.5, P>.99).

“On the 80 ophthalmic complaints, top-1 accuracy did not differ between Jev and either comparator (only Jev correct in 3 cases vs 5 for Opus 5, P=.73; 3 vs 3 for Opus 5.5, P>.99).”

May combine two claims.

On DDXPlus, Opus 5 was better calibrated than both Jev and Opus 5.5.

“Opus 5 was better calibrated than both.”

May combine two claims.

Shuffling the option order changed all three models' outputs slightly more than rerunning them did (mean excess TVD 0.025 for Jev, 0.034 for Opus 5, 0.019 for Opus 5.5; all P<.001), but rarely changed the top diagnosis.

“Shuffling the option order changed all three models' outputs slightly more than rerunning them did (mean excess TVD 0.025 for Jev, 0.034 for Opus 5, 0.019 for Opus 5.5; all P<.001), but rarely changed the top diagnosis.”

May combine two claims.

Jev was better calibrated than Opus 5.5 on the ophthalmic complaints (ECE difference -0.081; 95% CI -0.124 to -0.018) and did not differ from Opus 5 (-0.057; 95% CI -0.095 to 0.015).

“Jev was better calibrated than Opus 5.5 (ECE difference -0.081; 95% CI -0.124 to -0.018) and did not differ from Opus 5 (-0.057; 95% CI -0.095 to 0.015).”

May combine two claims.

Jev did not differ from either comparator in order sensitivity.

“Jev did not differ from either comparator.”

May combine two claims.

Jev's and Opus 5's top-1 errors on DDXPlus were moderately correlated (phi 0.57), more so than Jev's and Opus 5.5's (phi 0.46).

“Jev's and Opus 5's top-1 errors on DDXPlus were moderately correlated (phi 0.57), more so than Jev's and Opus 5.5's (phi 0.46).”

May combine two claims.

Opus 5.5 reasoned before answering on 41.2% of DDXPlus cases and on 10 of 45 Semigran cases.

“Opus 5.5 reasoned before answering on 41.2% of DDXPlus cases (median 0 reasoning tokens; interquartile range 0 to 197) and on 10 of 45 Semigran cases.”

May combine two claims.

All three models were more accurate on the 5-descriptor version of each ophthalmic complaint than on the 3-descriptor version (Jev 82.5% vs 95.0%, P=.13; Opus 5 82.5% vs 100%, P=.02; Opus 5.5 80.0% vs 97.5%, P=.02).

“All three models were more accurate on the 5-descriptor version of each complaint than on the 3descriptor version (Figure 3): Jev 82.5% vs 95.0% (P=.13),Opus 5 82.5% vs 100% (P=.02), and Opus 5.5 80.0% vs 97.5% (P=.02).”

May combine two claims.

Jev's misses on ophthalmic complaints were harder misses: the true diagnosis fell outside its top 3 in 8.8% of cases (vs 3.8% and 2.5%), and it assigned exactly zero to the true diagnosis twice.

“Its misses were harder misses: the true diagnosis fell outside its top 3 in 8.8% of cases (vs 3.8% and 2.5%), and it assigned exactly zero to the true diagnosis twice.”

May combine two claims.

Given each option list with no patient information, Jev concentrated its probability on one diagnosis: viral upper respiratory illness at 95.6% for the Semigran list, upper respiratory tract infection at 63.4% for DDXPlus, and floaters at 32.4% for the ophthalmic list.

“Given each option list with no patient information, Jev concentrated its probability on one diagnosis (Figure 2): viral upper respiratory illness at 95.6% for the Semigran list, upper respiratory tract infection at 63.4% for DDXPlus, and floaters at 32.4% for the ophthalmic list.”

May combine two claims.

Across all outputs, 92.0% of Jev's probabilities were exactly zero and 98.8% fell on a 0.01 grid.

“Across all outputs, 92.0% of Jev's probabilities were exactly zero and 98.8% fell on a 0.01 grid.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Registered primary hypothesis H3 (irrelevant options) and secondary hypothesis H2 (paraphrases) were not run, because clinician-confirmed distractors and paraphrases were not prepared.

“Registered primary hypothesis H3 (irrelevant options) and secondary hypothesis H2 (paraphrases) were not run, because clinician-confirmed distractors and paraphrases were not prepared.”

Classified as probably not a substantial claim. May combine two claims.

The registered primary outcomes were order invariance, independence from irrelevant options, and calibration.

“The registered primary outcomes were order invariance, independence from irrelevant options, and calibration.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A sampled-answer arm for Opus 5.5 was dropped because its failures arose from reasoning exhausting a 50-token limit, and successful cases returned the same answer on all 20 samples.

“A sampled-answer arm for Opus 5.5 was dropped: its failures arose from reasoning exhausting a 50-token limit, and successful cases returned the same answer on all 20 samples.”

May combine two claims.

Our Semigran results agree with earlier vignette studies, and the set was too small and too easy to separate the models.

“our Semigran results agree, and the set was too small and too easy to separate the models.”

May combine two claims.

Prior independent tests of Jev on non-clinical tasks reported nondeterminism, overconfidence on some question types, and above-chance accuracy from answer options alone.

“In the two weeks after release, independent developers tested Jev on customer-support routing, phishing detection, and licensing-exam questions, reporting nondeterminism, overconfidence on some question types, and above-chance accuracy from the answer options alone [7][8][9].”

• cites [7] jujumilk3. jev-calibration-audit: independent API-only calibration audit of TypeSafe AI’s Jev decision model. GitHub. 2026. https://github.com/jujumilk3/jev-calibration-auditNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [8] TypeSafe’s Jev scores 62.6% asked once and 95% split five ways (pre-registered non-clinical evaluation). 2026. https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-evalNot checked yet

  Not checked in the free run.

• cites [9] Open Medical Jev: frozen open models on national medical licensing exams (GitHub, FeiLiuEM/open-medical-jev). 2026. https://github.com/FeiLiuEM/open-medical-jevNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Calibration was measured by expected calibration error (10 equal-width bins on top-1 confidence), multiclass Brier score, and log-loss after mixing each distribution with 1% uniform mass.

“Calibration was measured by expected calibration error (10 equal-width bins on top-1 confidence) [15], multiclass Brier score, and log-loss after mixing each distribution with 1% uniform mass (so that exact zeros are scored identically for every model).”

• cites [15] 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Two amendments made before any study data (scoring of log-loss, choice of Semigran input text) are included in the registration.

“Two amendments made before any study data (scoring of log-loss, choice of Semigran input text) are included in the registration.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Every response was stored with its timestamp, the provider's resolved model version, and the code version that produced it.

“Every response was stored with its timestamp, the provider's resolved model version, and the code version that produced it.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The registered rule that Claude answer without reasoning could not be met for Opus 5.5, which does not allow reasoning to be disabled.

“The registered rule that Claude answer without reasoning could not be met for Opus 5.5, which does not allow reasoning to be disabled.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Registered hypotheses on irrelevant options and paraphrases were not run.

“Registered hypotheses on irrelevant options and paraphrases were not run.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

An analysis error that counted the options-only calls as one extra case in paired comparisons was found and corrected before the results reported here; per-model accuracy and calibration were not affected.

“An analysis error that counted the options-only calls as one extra case in paired comparisons was found and corrected before the results reported here; per-model accuracy and calibration were not affected.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All random orders used a fixed seed.

“All random orders used a fixed seed.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All three models are commercial products that change; our results describe the versions tested.

“Finally, all three models are commercial products that change; our results describe the versions tested.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Each case was presented as a single message, whereas real triage is a conversation in which the clinician asks follow-up questions.

“And each case was presented as a single message, whereas real triage is a conversation in which the clinician asks follow-up questions.”

May combine two claims.

Jev 1.13 is a decision model that generates no text and returns a probability for every option in one pass.

“Jev 1.13 (TypeSafe AI, released September 2026) generates no text: given a case and a list of options, it returns a probability for every option in one pass, at a small fraction of an LLM's cost and latency.”

• cites [4] TypeSafe AI. Jev on OpenRouter (documentation). 2026. https://openrouter.ai/docs/guides/community/jev.mdNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

All code, data manifests, and results are version-controlled, and every result row records the commit that produced it.

“All code, data manifests, and results are version-controlled, and every result row records the commit that produced it.”

Classified as probably not a substantial claim.

Holm correction was applied across the registered primary comparisons.

“Holm correction was applied across the registered primary comparisons.”

May combine two claims.

On DDXPlus, Jev was overconfident while Opus 5.5 was underconfident; at similar calibration error, Jev's pattern is the more dangerous for triage, because underconfidence produces unnecessary referrals rather than missed ones.

“On DDXPlus, Jev was overconfident while Opus 5.5 was underconfident; at similar calibration error, Jev's pattern is the more dangerous for triage, because underconfidence produces unnecessary referrals rather than missed ones.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Jev cannot ask for clarification, and we did not test whether the language models' ability to do so would change their accuracy.

“Jev cannot ask for clarification, and we did not test whether the language models' ability to do so would change their accuracy.”

May combine two claims.

Its probabilities expressed uncertainty only when the question allowed abstention, and it assigned high confidence to option lists with no case attached.

“Its probabilities expressed uncertainty only when the question allowed abstention, and it assigned high confidence to option lists with no case attached.”

Classified as probably not a substantial claim. May combine two claims.

Paired comparisons between models used exact McNemar tests for binary outcomes and case-level bootstrap intervals for continuous ones.

“Paired comparisons between models used exact McNemar tests for binary outcomes and case-level bootstrap intervals (10,000 resamples; 2,000 for calibration error) for continuous ones.”

Classified as probably not a substantial claim.

The analysis plan, hypotheses, and violation thresholds were fixed before data collection.

“The analysis plan, hypotheses, and violation thresholds were fixed before data collection and registered on the Open Science Framework (osf.io/fpmg3)[10] on 26 September 2026, before the first study model call.”

• cites [10] Deobhakta A. Pre-registration: probabilistic coherence and calibration of a System One decision model (Jev) vs Claude Opus on clinical diagnosis. Open Science Framework. 2026. https://osf.io/fpmg3Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The study was pre-registered on OSF on 26 September 2026 before the first study model call.

“The analysis plan, hypotheses, and violation thresholds were fixed before data collection and registered on the Open Science Framework (osf.io/fpmg3)[10] on 26 September 2026, before the first study model call.”

• cites [10] Deobhakta A. Pre-registration: probabilistic coherence and calibration of a System One decision model (Jev) vs Claude Opus on clinical diagnosis. Open Science Framework. 2026. https://osf.io/fpmg3Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The developer markets Jev's probabilities as calibrated, the product of a training method aimed at calibration rather than fluent text.

“Its developer markets these probabilities as calibrated, the product of a training method aimed at calibration rather than fluent text [4].”

• cites [4] TypeSafe AI. Jev on OpenRouter (documentation). 2026. https://openrouter.ai/docs/guides/community/jev.mdNot checked yet

  Not checked in the free run.

May combine two claims.

In this pre-registered evaluation, a decision model marketed as calibrated matched frontier language models on short ophthalmic complaints, with better calibration than the registered comparator, but was substantially less accurate on multi-symptom synthetic patients, where it excluded the correct diagnosis outright in about 6% of cases.

“In this pre-registered evaluation, a decision model marketed as calibrated matched frontier language models on short ophthalmic complaints, with better calibration than the registered comparator, but was substantially less accurate on multi-symptom synthetic patients, where it excluded the correct diagnosis outright in about 6% of cases.”

Classified as probably not a substantial claim. May combine two claims.

All cases were written, not spoken, and none came from real patients; DDXPlus patients are synthetic, and our rendering of their structured findings is stilted.

“All cases were written, not spoken, and none came from real patients; DDXPlus patients are synthetic, and our rendering of their structured findings is stilted.”

May combine two claims.

On short, distinctive ophthalmic complaints, Jev was as accurate as two frontier language models and better calibrated than the registered comparator.

“On short, distinctive ophthalmic complaints, Jev was as accurate as two frontier language models and better calibrated than the registered comparator.”

May combine two claims.

Our results confirm run-to-run variation in a clinical setting (a third of repeated outputs differed) and show that option-list structure drives answers when the input is uninformative.

“Our results confirm run-to-run variation in a clinical setting (a third of repeated outputs differed) and show that option-list structure drives answers when the input is uninformative.”

May combine two claims.

The model can recognize an uninformative input, but its probabilities carry that signal only when the question allows it.

“The model can recognize an uninformative input, but its probabilities carry that signal only when the question allows it.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The ophthalmic set comprised 80 simulated complaints, a small sample of uncertain applicability to real patient calls.

“The ophthalmic set comprised 80 simulated complaints, a small sample of uncertain applicability to real patient calls”

Probably not a substantial claim: may not pass S3 (load-bearing).

Opus 5.5's resolved version carried no dated snapshot identifier.

“Opus 5.5's resolved version carried no dated snapshot identifier.”

May combine two claims.

None of the prior independent tests of Jev tested clinical diagnosis against a pre-specified plan, compared the model with frontier LLMs with reasoning both on and off, or examined how its probabilities fail.

“None of this work tested clinical diagnosis against a pre-specified plan, compared the model with frontier LLMs with reasoning both on and off, or examined how its probabilities fail.”

• cites [7] jujumilk3. jev-calibration-audit: independent API-only calibration audit of TypeSafe AI’s Jev decision model. GitHub. 2026. https://github.com/jujumilk3/jev-calibration-auditNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [8] TypeSafe’s Jev scores 62.6% asked once and 95% split five ways (pre-registered non-clinical evaluation). 2026. https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-evalNot checked yet

  Not checked in the free run.

• cites [9] Open Medical Jev: frozen open models on national medical licensing exams (GitHub, FeiLiuEM/open-medical-jev). 2026. https://github.com/FeiLiuEM/open-medical-jevNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A tie-breaking error that could register spurious top-1 changes under reordering was corrected before final analysis.

“A tie-breaking error that could register spurious top-1 changes under reordering was corrected before final analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Jev's 0.01 output resolution makes its noise and distance measures coarser than the comparators'.

“Jev's 0.01 output resolution makes its noise and distance measures coarser than the comparators'.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The registered order-sensitivity criterion compared the largest of five permutation distances with a single-pair noise threshold, which flags about 23% of cases from noise alone.

“The registered order-sensitivity criterion compared the largest of five permutation distances with a single-pair noise threshold, which flags about 23% of cases from noise alone.”