The economics of accuracy for medical reasoning with large language models

May combine two claims.

Fine-tuning and retrieval-augmented generation can improve domain-specific accuracy but require additional labeled data, technical skill, and infrastructure.

“While fine-tuning and retrieval-augmented generation (RAG) can improve domain-specific accuracy, these methods require additional labeled data, technical skill, and infrastructure.”

• cites [13] EJ Hu, 2022, Lora: Low-rank adaptation of large language models, ICLR sourcePartly supported

  Appendix A (p. 16) reports that fine-tuning improves accuracy on the tasks tested and discusses curating training examples. Related Works (p. 9) describes substantial hardware demands for conventional fine-tuning of GPT-3. These passages do not establish the claim about retrieval-augmented generation (RAG), or that both methods require labeled data, technical skill, and infrastructure. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [16] AJ Thirunavukarasu, 2023, Large language models in medicine, Nat Med sourceNot checked — no full text of the source

May combine two claims.

PCCR operates through natural language continuation instructions, making it compatible with any instruction-following model without specialized training or direct intervention in token generation, making it accessible over APIs.

“Unlike budget forcing through “Wait” token injection [18], PCCR operates through natural language continuation instructions, making it compatible with any instruction-following model without specialized training or direct intervention in the token generation process, making it accessible over APIs.”

• cites [18] Muennighoff N, Yang Z, Shi W, Li XL, Fei-Fei L, Hajishirzi H, et al. s1: Simple test-time scaling. In: Proceedings of the 2025 conference on empirical methods in natural language processing, 2025. 20286–332. https://doi.org/10.18653/v1/2025.emnlp-main.1025 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All analyses were performed on publicly available, de-identified benchmark datasets used in accordance with their respective licenses and terms of use.

“All analyses were performed on publicly available, de-identified benchmark datasets—MIRAGE [23], PubMedQA [24], and the medical licensing examination questions of Katz et al. [25]—used in accordance with their respective licenses and terms of use.”

• cites [23] Xiong G, Jin Q, Lu Z, Zhang A. Benchmarking retrieval-augmented generation for medicine. In: Findings of the association for computational linguistics ACL 2024, 2024. 6233–51. https://doi.org/10.18653/v1/2024.findings-acl.372 sourcePartly supported

  The Question-Only Retrieval section identifies MIRAGE as a medical QA benchmark, and Appendix B provides a link to it. The paper does not establish that MIRAGE is de-identified or that the citing authors used it in accordance with its license and terms of use. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [24] Jin Q, Dhingra B, Liu Z, Cohen W, Lu X. PubMedQA: a dataset for biomedical research question answering. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th International joint conference on natural language processing (EMNLP-IJCNLP), 2019. 2567–77. https://doi.org/10.18653/v1/d19-1259 sourcePartly supported

  The introduction (p. 2) identifies PubMedQA as a benchmark, and Data Collection (p. 3) says its questions and contexts come from PubMed article titles and abstracts. Our copy does not establish that PubMedQA was de-identified or that the citing authors' analyses complied with its license and terms of use. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [25] U Katz, 2024, GPT versus resident physicians — a benchmark based on official board scores, NEJM AI sourceNot checked — no full text of the source

May combine two claims.

Without context on the MIRAGE dataset, MedGemma 27B with extended reasoning (PCCR-1024) reached 80.1% which is higher than the 73.4% accuracy reported for GPT-4 without context and matched the performance of GPT-4 with access to extensive context [23].

“Similarly, without context on the MIRAGE datset, MedGemma 27B with extended reasoning (PCCR-1024) reached 80.1% which is higher than the 73.4% accuracy reported for GPT-4 without context and matched the performance of GPT-4 with access to extensive context [23].”

• cites [23] Xiong G, Jin Q, Lu Z, Zhang A. Benchmarking retrieval-augmented generation for medicine. In: Findings of the association for computational linguistics ACL 2024, 2024. 6233–51. https://doi.org/10.18653/v1/2024.findings-acl.372 sourceSupported

May combine two claims.

The two disagreements out of 200 pairs fell in opposite directions, consistent with random, borderline free-text adjudication rather than a systematic directional bias.

“The two disagreements out of 200 pairs fell in opposite directions, which is consistent with random, borderline free-text adjudication rather than a systematic directional bias.”

May combine two claims.

Chain-of-thought prompting has the expected effect on larger models (27B) but mixed performance for smaller models (1B-4B).

“First, we demonstrate that chain-of-thought prompting has the expected effect on larger models (27B) but mixed performance for smaller models (1B-4B)”

May combine two claims.

On the medical licensing examination, Gemma-3 4B reached 52.5% at PCCR-512, higher than MedGemma 4B's best performance (43.2% at PCCR-1024, p<0.002).

“Unlike on MIRAGE, Gemma-3 4B showed substantial improvement with PCCR on this dataset, reaching 52.5% at PCCR-512—higher than MedGemma 4B’s best performance (43.2% at PCCR-1024, p < 0.002).”

May combine two claims.

At the 27B size on MIRAGE, the fine-tuning advantage grew from +4.6% with P0 to +15.7% with P2 (Cohen's h=0.35), a 3.4-fold amplification.

“At the 27B size, the fine-tuning advantage grew from +4.6 % with direct answering (P0) to +15.7 % (Cohen’s h = 0.35) with self-consistency (P2), representing a 3.4-fold amplification of the fine-tuning benefit.”

May combine two claims.

For MedGemma 4B on PubMedQA with oracle context, P2 achieved 74.3%, recovering P0 performance and slightly exceeding it within confidence intervals.

“Remarkably, P2 recovered this performance entirely, achieving 74.3%—slightly exceeding P0 but within confidence intervals.”

May combine two claims.

MedGemma 4B on PubMedQA with oracle context dropped from 73.1% P0 to 56.2% P1, a 16.9 percentage point degradation (p<10^-5), the largest P0-to-P1 drop observed in any condition.

“MedGemma 4B exhibited anomalous behavior with P1 prompting: accuracy with context dropped from 73.1% (P0) to 56.2% (P1)—a 16.9 percentage point degradation (p < 10-5) that was the largest P0-to-P1 drop observed in any condition.”

May combine two claims.

For Gemma-3 4B, MedGemma 4B, and Gemma-3 27B, performance stays within CI95% from PCCR-512 onward, showing plateauing effects with extended PCCR reasoning.

“For Gemma-3 4B, MedGemma 4B, and Gemma-3 27B, we see some plateauing effects with extended PCCR reasoning where performance stays within CI95% from PCCR-512 onward.”

May combine two claims.

Without context, accuracy generally increases with token budget, with MedGemma 27B showing the largest gains.

“Without context (A, B, & C), accuracy generally increases with token budget, with MedGemma 27B showing the largest gains.”

May combine two claims.

Using PCCR, the 1B model reached comparable performance to P0 prompting for 4B models, but neither 1B nor 4B models could reach P0 performance of 27B models.

“Using PCCR, the 1B model reached comparable performance to P0 prompting for 4B models, but neither 1B nor 4B models could reach P0 performance of 27B models (Fig 1A).”

May combine two claims.

On PubMedQA with oracle context, P2 majority voting produced consistent accuracy improvements across all models, including smaller ones that showed minimal P2 benefit on MIRAGE.

“A notable finding emerged from the P2 results for PubMedQA (Table 4): majority voting produced consistent accuracy improvements with oracle context across all models, including smaller ones that showed minimal P2 benefit on MIRAGE.”

May combine two claims.

Without oracle context on PubMedQA, P2 provided minimal benefit for most models; MedGemma 27B showed essentially no change (43.3% P0 to 42.1% P2), and gains for other models were modest.

“Without oracle context, P2 provided minimal benefit for most models: MedGemma 27B showed essentially no change (43.3% P042.1% P2), and gains for other models were modest.”

May combine two claims.

With oracle context, accuracy peaks at low token budgets (128-256 tokens) and declines with extended reasoning.

“With oracle context (D), accuracy peaks at low token budgets (128–256 tokens) and declines with extended reasoning.”

May combine two claims.

Gemma-3 4B with PCCR-128 achieved 76.0% accuracy on PubMedQA with oracle context, exceeding MedGemma 27B with P0 (74.7%) by 1.3 percentage points (difference within CI95%).

“Gemma-3 4B with PCCR-128 achieved 76.0% accuracy, exceeding MedGemma 27B with P0 (74.7%) by 1.3 percentage points (difference within CI95%).”

May combine two claims.

Even with a stricter Bonferroni correction (p-value threshold at 0.002) 23 out of 25 tests remain significant and do not change the main conclusions.

“Note that even with a stricter Bonferroni correction (p-value threshold at 0.002) 23 out of 25 tests remain significant and do not change our main conclusions.”

May combine two claims.

For MedGemma 27B with oracle context, P1 achieves 77.4%, P2 achieves 75.2%, and PCCR-1024 achieves 70.1%.

“This distinction is illustrated by comparing MedGemma 27B across strategies with context: P1 achieves 77.4%, P2 achieves 75.2% (slight degradation within confidence intervals), while PCCR-1024 achieves only 70.1%.”

May combine two claims.

On MIRAGE, P2 gains were smaller for Gemma-3 27B (+2.8%, p<10^-2) and MedGemma 4B (+2.6%, p<10^-2).

“However, gains were smaller for Gemma-3 27B (+2.8 %, p < 10-2) and MedGemma 4B (+2.6 %, p < 10-2).”

May combine two claims.

On the medical licensing examination, MedGemma 27B reached 60.3% at PCCR-1024, the strongest performance with extended reasoning.

“MedGemma 27B again showed the strongest performance with extended reasoning, reaching 60.3% at PCCR-1024.”

May combine two claims.

All models perform below the passing score of 65% and below median physician scores (70%-80% depending on specialty).

“All models perform below the passing score of 65% and below the median physician scores which ranges from 70%-80% depending on the specialty [25].”

• cites [25] U Katz, 2024, GPT versus resident physicians — a benchmark based on official board scores, NEJM AI sourceNot checked — no full text of the source

• AssumptionRests on an assumption: The medical licensing examination passing score is 65%.

May combine two claims.

MedGemma 4B with PCCR-256 achieved 76.5% on PubMedQA with oracle context, matching MedGemma 27B P1 (77.4%) within confidence intervals.

“Similarly, MedGemma 4B with PCCR-256 achieved 76.5%, matching MedGemma 27B P1 (77.4%) within confidence intervals.”

May combine two claims.

For MedGemma 27B on PubMedQA with oracle context, accuracy peaked at P1 (77.4%) and declined monotonically with increasing PCCR budget, reaching 70.1% at PCCR-1024, a loss of 7.3 percentage points from the optimum (McNemar p<0.002, Cohen's h=0.16).

“For MedGemma 27B, accuracy peaked at P1 (77.4%) and declined monotonically with increasing PCCR budget, reaching 70.1% at PCCR-1024—a loss of 7.3 % from the optimum (McNemar’s test, p < 0.002, Cohen’s h = 0.16).”

May combine two claims.

For MedGemma 27B on PubMedQA with oracle context, P2 provided no additional benefit over P1 (77.4%) and showed slight degradation to 75.2% (-2.2%, within confidence intervals).

“For instance, MedGemma 27B, which already achieved strong performance with P1 (77.4%), P2 provided no additional benefit and showed slight degradation to 75.2% (-2.2 %, within confidence intervals).”

May combine two claims.

Without context, extended reasoning generally improved accuracy; with oracle context, extended reasoning beyond short budgets degraded performance.

“Without context, extended reasoning generally improved accuracy (Fig 1C). However, with oracle context, extended reasoning beyond short budgets degraded performance (Fig 1D).”

May combine two claims.

MedGemma models generate longer reasoning traces than their base counterparts, requiring fewer continuation rounds to meet a token budget.

“MedGemma models generate longer reasoning traces than their base counterparts (requiring fewer continuation rounds to meet a token budget; Fig 3)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The figure clearly shows the effect on accuracy from the availability of context vs. no-context (red vs. blue) and the effect on cost from fine-tuned vs. generic models (far-right vs. far-left).

“The figure clearly shows the effect on accuracy from the availability of context vs. no-context (red vs. blue) and the effect on cost from fine-tuned vs. generic models (far-right vs. far-left).”

May combine two claims.

The optimal solution configuration for Example 2 (no-context, high-accuracy application) is (27, 1, PCCR-512, 0) or MedGemma 27B + PCCR-512 at 1406.8$ per query and a latency of 30.51 seconds.

“The optimal solution configuration is (27, 1, PCCR-512, 0) or MedGemma 27B + PCCR-512 at 1406.8$ per query and a latency of 30.51 seconds.”

May combine two claims.

With context, the pattern reverses: base models show the largest P2 gains (+9.8 to +10.1% for Gemma-3 4B and 27B), while already high-performing configurations show ceiling effects.

“With context, the pattern reverses: base models show the largest P2 gains (+9.8 to +10.1% for Gemma-3 4B and 27B), while already high-performing configurations show ceiling effects.”

May combine two claims.

The difference in unique answers between correct and incorrect responses was significant (Mann-Whitney U=3,429,696, p<10^-5) with a large effect size (Cohen's d=0.90).

“This difference was significant (Mann-Whitney U = 3,429,696, p < 10-5) with a large effect size (Cohen’s d = 0.90).”

May combine two claims.

The same saturating-exponential form does not apply once context is available, where accuracy is non-monotonic in budget (the overthinking regime).

“the same form does not apply once context is available, where accuracy is non-monotonic in budget (the overthinking regime, modeled separately below).”

May combine two claims.

PCCR follows a saturating exponential relationship with reasoning budget, but this relationship is modulated by context availability.

“PCCR follows a saturating exponential relationship with reasoning budget, but critically, this relationship is modulated by context availability:”

May combine two claims.

When the majority-voted answer was correct, samples contained an average of 2.25 unique answers (SD=0.77); when incorrect, 2.91 unique answers (SD=0.65).

“When the majority-voted answer was correct, samples contained an average of 2.25 unique answers (SD = 0.77); when incorrect, samples contained 2.91 unique answers (SD = 0.65, Fig 2B).”

The quote only partly states this claim.

At the 27B scale, the accuracy benefit of medical fine-tuning increases from +4.6 percentage points with simple prompting (P0) to +13-16 percentage points (or +15.7 percentage points with self-consistency) with sophisticated prompting (P2, PCCR) on tasks without context.

“This captures our finding that fine-tuning benefit increases from +4.6 % (P0) to +13–16 % (P2, PCCR) at the 27B scale on tasks without context.”

• Also stated in: § 3.1 Summary of key empirical findings · plain

  “On MIRAGE, the fine-tuning benefit at the 27B scale increased from +4.6 percentage points with direct answering (P0) to +15.7 percentage points with self-consistency (P2)—a 3.4-fold amplification (Table 2).”

• Also stated in: § 3.1 Summary of key empirical findings · plain

  “The accuracy advantage of MedGemma over base Gemma models grew substantially when combined with self-consistency or extended reasoning.”

May combine two claims.

P2 matches or exceeds the best non-PCCR configuration for 4 of 5 models, while PCCR-1024 underperforms for 4 of 5 models.

“P2 matches or exceeds the best non-PCCR configuration for 4 of 5 models, while PCCR-1024 underperforms for 4 of 5 models.”

May combine two claims.

With high-quality context, short reasoning or self-consistency is optimal and small models become competitive with large ones.

“Second, with high-quality context, short reasoning or self-consistency is optimal and small models become competitive with large ones.”

May combine two claims.

The average P2 deviation from best is +2.1 (beneficial), while the average PCCR-1024 deviation is -4.5 (harmful).

“The average P2 deviation from best is + 2.1 (beneficial), while the average PCCR-1024 deviation is –4.5 (harmful).”

May combine two claims.

Gemma-3 4B with PCCR-128 and context achieves 76.0% accuracy, compared to MedGemma 27B with P1 and context at 77.4%-within confidence intervals at approximately 10% of the cost.

“Gemma-3 4B with PCCR-128 and context achieves 76.0% accuracy, compared to MedGemma 27B with P1 and context at 77.4%—within confidence intervals at approximately 10% of the cost.”

May combine two claims.

Fine-tuning incurs a notable cost when high-quality context can provide answers to questions, with fine-tuned candidate configurations being 10 more expensive with no substantial gain in accuracy.

“Note that fine-tuning incurs a notable cost when high-quality context can provide answers to questions, with fine-tuned candidate configurations being 10 more expensive with no substantial gain in accuracy.”

May combine two claims.

Self-consistency (P2) provides robust benefits with context across all model scales, including smaller models that show minimal P2 benefit without context.

“The key insight from our PubMedQA results is that P2 provides robust benefits with context across all model scales, including smaller models that show minimal P2 benefit without context.”

May combine two claims.

For MedGemma 4B on PubMedQA with oracle context, 13.2% of initially correct responses flipped to incorrect during PCCR continuation, a 7.3-fold higher flip rate than MedGemma 27B on MIRAGE.

“In contrast, for MedGemma 4B on PubMedQA with oracle context, where PCCR-1024 substantially underperformed PCCR-128 (63.8% vs. 75.2%), 13.2% of initially correct responses flipped to incorrect—a 7.3-fold higher flip rate.”

The quote only partly states this claim.

P2 benefit is concentrated in large, fine-tuned models without context.

“P2 benefit is concentrated in large, fine-tuned models”

Probably not a substantial claim: may not pass S4 (based).

There is a threshold below which chain-of-thought becomes harmful without context, corresponding to an assumed 8B parameters.

“where (corresponding to an assumed 8B parameters) is the threshold below which CoT becomes harmful without context.”

• AssumptionThe authors state this as an assumption: The threshold below which chain-of-thought becomes harmful without context corresponds to 8B parameters.

May combine two claims.

The pattern of iteration counts was consistent across budget levels, with the gap widening at higher thresholds.

“This pattern was consistent across budget levels, with the gap widening at higher thresholds.”

May combine two claims.

MedGemma 27B with P0 is comparable in accuracy to Gemma-3 27B with PCCR-256 but the Gemma-3 27B model is 2% of the cost-per-query.

“The effect of fine-tuning on estimated cost is evident in the first row of Table 21 where MedGemma 27B with P0 is comparable in accuracy of Gemma-3 27B with PCCR-256 but the Gemma-3 27B model is 2% of the cost-per-query.”

May combine two claims.

Fine-tuning provides substantial benefit without context, especially with sophisticated prompting.

“Fine-tuning provides substantial benefit, especially with sophisticated prompting”

May combine two claims.

With oracle context on PubMedQA, P2 achieves 63.1-75.2% accuracy while PCCR-1024 achieves 59-70.1% accuracy across models.

“With oracle context on PubMedQA, P2 achieves 63.1-75.2% accuracy while PCCR-1024 achieves 59-70.1% accuracy across models.”

May combine two claims.

Iteration counts varied substantially by model size: smaller models produced shorter reasoning traces per prompt, requiring more continuation rounds to meet budget thresholds.

“Iteration counts varied substantially by model size: smaller models produced shorter reasoning traces per prompt, requiring more continuation rounds to meet budget thresholds.”

May combine two claims.

Correct responses show higher concentration at lower variability (1-2 unique answers), while incorrect responses are skewed toward higher variability (3-4 unique answers).

“Correct responses show higher concentration at lower variability (1–2 unique answers), while incorrect responses are skewed toward higher variability (3–4 unique answers).”

May combine two claims.

Without context, P2 benefits are concentrated in larger fine-tuned models (MedGemma 27B shows +17.0% on MIRAGE), while smaller and base models show minimal or negative effects.

“Without context, P2 benefits are concentrated in larger fine-tuned models (MedGemma 27B shows +17.0 % on MIRAGE), while smaller and base models show minimal or negative effects.”

May combine two claims.

The accuracy gain from medical fine-tuning depends on both model scale and prompting strategy, with an interaction term capturing the amplification effect observed with sophisticated prompting.

“The accuracy gain from medical fine-tuning depends on both model scale and prompting strategy, with an interaction term capturing the amplification effect observed with sophisticated prompting:”

May combine two claims.

Without supporting context, extended reasoning (PCCR) and medical fine-tuning yield the largest gains, and MedGemma 27B with the largest reasoning budget is consistently the top configuration.

“First, without supporting context, extended reasoning (PCCR) and medical fine-tuning yield the largest gains, and MedGemma 27B with the largest reasoning budget is consistently the top configuration.”

May combine two claims.

Only 0.7% of incorrect responses showed complete agreement, while 59.3% produced 3 unique answers and 16.0% produced 4 unique answers.

“In contrast, only 0.7% of incorrect responses showed complete agreement, while 59.3% produced 3 unique answers and 16.0% produced 4 unique answers—indicating the model distributed probability mass across most available options.”

May combine two claims.

P2 and PCCR have similar token costs at high reasoning budgets (P2 1000 tokens, PCCR-1024 1024 tokens), but dramatically different effectiveness when context is available.

“Our results reveal that P2 and PCCR have similar token costs at high reasoning budgets (P21000 tokens, PCCR-10241024 tokens), but dramatically different effectiveness when context is available.”

May combine two claims.

Among correct responses, 15.3% showed complete agreement (1 unique answer) and 48.9% showed near-agreement (2 unique answers).

“Among correct responses, 15.3% showed complete agreement (1 unique answer) and 48.9% showed near-agreement (2 unique answers, Fig 2A).”

May combine two claims.

At PCCR-1024, Gemma-3 1B required an average of 6.9 iterations (SD=1.6), while MedGemma 27B required only 4.1 iterations (SD=0.9).

“At PCCR-1024, Gemma-3 1B required an average of 6.9 iterations (SD = 1.6), while MedGemma 27B required only 4.1 iterations (SD = 0.9).”

May combine two claims.

Decode throughput is memory-bandwidth-bound and therefore falls sharply with scale (Table 16), directly determining latency.

“Decode throughput is memory-bandwidth-bound and therefore falls sharply with scale (Table 16), directly determining latency (Section 2.8.3).”

May combine two claims.

The optimal solution configuration for Example 1 (high-accuracy RAG application) is (4, 0, PCCR-128, 1) or Gemma-3 4B + PCCR-128 + RAG at 114.76 per query and a latency of 2.76 seconds.

“The optimal solution configuration is (4, 0, PCCR-128, 1) or Gemma-3 4B + PCCR-128 + RAG at 114.76 per query and a latency of 2.76 seconds.”

May combine two claims.

With high-quality context, the accuracy of 4B models with short extended reasoning are comparable with that of fine-tuned 27B models-at approximately one-tenth the computational cost.

“With high-quality context, the accuracy of 4B models with short extended reasoning are comparable with that of fine-tuned 27B models—at approximately one-tenth the computational cost.”

May combine two claims.

The benefit of context interacted strongly with prompting strategy-a finding with important implications for system design.

“However, the benefit of context interacted strongly with prompting strategy—a finding with important implications for system design.”

May combine two claims.

No configuration tested reached the median physician score (70-80%), let alone the >90% level many would likely demand before trusting an automated clinical recommendation.

“no configuration we tested reached the median physician score (70–80%), let alone the > 90% level many would likely demand before trusting an automated clinical recommendation.”

May combine two claims.

With oracle context, up to 13.2% of answers that were correct after the initial response flipped to incorrect after further reasoning, versus 1.8% without context.

“With oracle context, up to 13.2% of answers that were correct after the initial response flipped to incorrect after further reasoning (versus 1.8% without context; Section 2.5).”

May combine two claims.

The identical continuation prompt improves accuracy when no context is available but harms it when high-quality context is present.

“the identical continuation prompt improves accuracy when no context is available but harms it when high-quality context is present”

May combine two claims.

Oracle context improved accuracy by 18-32 percentage points across models on PubMedQA, demonstrating the value of retrieval-augmented generation.

“Oracle context improved accuracy by 18–32 percentage points across models on PubMedQA, demonstrating the value of retrieval-augmented generation.”

May combine two claims.

On PubMedQA with oracle context, P2 achieved the highest or near-highest accuracy for most models, with base Gemma models showing +4.6 to +10.1 percentage point gains over P0.

“On PubMedQA with oracle context, P2 achieved the highest or near-highest accuracy for most models, with base Gemma models showing +4.6 to +10.1 percentage point gains over P0.”

May combine two claims.

No model configuration achieved the 65% passing threshold on the medical licensing examination, and all models performed below the median physician score of 70-80%.

“No model configuration achieved the 65% passing threshold on the medical licensing examination, and all models performed below the median physician score of 70–80% [25].”

• cites [25] U Katz, 2024, GPT versus resident physicians — a benchmark based on official board scores, NEJM AI sourceNot checked — no full text of the source

May combine two claims.

This pattern replicated across datasets, with Gemma-3 1B showing a 22.2 percentage point drop on PubMedQA without context (Table 4).

“This pattern replicated across datasets, with Gemma-3 1B showing a 22.2 percentage point drop on PubMedQA without context (Table 4).”

May combine two claims.

Our results reveal a fundamental bifurcation: without context, extended reasoning (PCCR) and fine-tuning provide the greatest benefits; with high-quality context, simpler prompting (P0, P1) or self-consistency (P2) is preferred, and extended reasoning becomes counterproductive.

“Our results reveal a fundamental bifurcation: without context, extended reasoning (PCCR) and fine-tuning provide the greatest benefits; with high-quality context, simpler prompting (P0, P1) or self-consistency (P2) is preferred, and extended reasoning becomes counterproductive.”

May combine two claims.

Gemma-3 4B with PCCR-128 reached 76.0%, approaching MedGemma 27B with P1 at 77.4%.

“Gemma-3 4B with PCCR-128 reached 76.0%, approaching MedGemma 27B with P1 at 77.4%.”

May combine two claims.

This pattern was consistent across all models, with overthinking penalties ranging from 7 to 13 percentage points.

“This pattern was consistent across all models, with overthinking penalties ranging from 7 to 13 percentage points.”

May combine two claims.

MedGemma 27B peaked at 77.4% with P1 and declined to 70.1% at PCCR-1024-a 7.3 percentage point loss from extended reasoning.

“MedGemma 27B peaked at 77.4% with P1 and declined to 70.1% at PCCR-1024—a 7.3 percentage point loss from extended reasoning.”

May combine two claims.

On MIRAGE, Gemma-3 1B dropped 6.9 percentage points and Gemma-3 4B dropped 3.9 percentage points when prompted to reason step-by-step (Table 2).

“On MIRAGE, Gemma-3 1B dropped 6.9 percentage points and Gemma-3 4B dropped 3.9 percentage points when prompted to reason step-by-step (Table 2).”

May combine two claims.

The best-performing configuration (MedGemma 27B with PCCR-1024) was below the 65% passing threshold and within the lowest quartile of practicing physicians.

“The best-performing configuration (MedGemma 27B with PCCR-1024) was below the 65% passing threshold and within the lowest quartile of practicing physicians.”

May combine two claims.

MedGemma 27B reports 87.7% on MedQA, comparable to frontier closed models at a fraction of the inference cost.

“MedGemma 27B reporting 87.7% on MedQA which is comparable to frontier closed models at a fraction of the inference cost [15].”

• cites [15] Sellergren A, Kazemzadeh S, Jaroensri T, Kiraly A, Traverse M, Kohlberger T. MedGemma technical report. 2025. https://arxiv.org/abs/250705201Not checked yet

  Not checked in the free run.

May combine two claims.

The base models (Gemma-3) and their medically fine-tuned variants (MedGemma) are derived from identical architectures and checkpoints, allowing direct measurement of medical fine-tuning without confounding by architectural differences.

“the base models (Gemma-3) and their medically fine-tuned variants (MedGemma) are derived from identical architectures and checkpoints; this shared lineage lets us measure the effect of medical fine-tuning directly, without confounding by architectural differences.”

Classified as probably not a substantial claim.

The confidence intervals are inflated to account for imperfect adjudication from the LLM-as-judge.

“We inflate the CIs to account for imperfect adjudication from the LLM-as-judge, as explained later.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

All experiments were conducted using consistent evaluation infrastructure to ensure reproducibility.

“All experiments were conducted using consistent evaluation infrastructure to ensure reproducibility.”

Classified as probably not a substantial claim. May combine two claims.

The focus on Gemma/MedGemma is a limitation: architecture-specific behaviors could influence findings, and the chain-of-thought threshold, overthinking penalty, and equivalence points may not transfer to other families.

“We note that this focus is also a limitation: architecture-specific behaviors could influence our findings, and the chain-of-thought threshold, overthinking penalty, and equivalence points we report may not transfer to other families (e.g., Llama, Mistral, Qwen).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The LLM-as-judge prompt included three few-shot examples of correct and incorrect evaluations to ensure reliability.

“To ensure reliability, the prompt included three few-shot examples of correct and incorrect evaluations.”

May combine two claims.

The medical licensing examination dataset was selected for its data-contamination profile: because these are recent, non-US board examinations originally administered in a non-English setting, the questions are substantially less likely to appear in model pretraining corpora than widely-circulated USMLE-style items.

“We selected this dataset specifically for its data-contamination profile: because these are recent, non-US board examinations originally administered in a non-English setting, the questions are substantially less likely to appear in model pretraining corpora than widely-circulated USMLE-style items.”

Classified as probably not a substantial claim.

Generations were evaluated using an LLM-as-judge which was validated with manual review with a set of 200 QA pairs.

“Generations were evaluated using a ‘LLM-as-judge’ which was validated with manual review with a set of 200 QA pairs.”

May combine two claims.

The dataset is treated as a lower-contamination-risk test of genuine generalization, though contamination cannot be fully excluded.

“We therefore treat this dataset as a lower-contamination-risk test of genuine generalization while acknowledging that we cannot fully exclude the possibility of contamination.”

May combine two claims.

Re-scoring every failed extraction as correct would raise the most-affected model's accuracy by at most 2 percentage points, which does not change the main conclusions.

“As an upper bound on any residual bias, re-scoring every failed extraction as correct would raise the most-affected model’s accuracy by at most 2 percentage points, which does not change our main conclusions.”

May combine two claims.

Self-consistency voting is particularly effective when models have access to relevant context but may produce variable reasoning chains, because voting filters out erroneous reasoning while preserving correct extractions from provided evidence.

“This pattern suggests that self-consistency voting is particularly effective when models have access to relevant context but may produce variable reasoning chains—voting filters out erroneous reasoning while preserving correct extractions from the provided evidence.”

May combine two claims.

MedGemma 4B's failures are high-variance rather than systematically wrong; voting filters out erroneous chains while preserving correct ones.

“The clean recovery under P2 (independent sampling plus majority voting) indicates that MedGemma 4B’s failures are high-variance rather than systematically wrong: voting filters out the erroneous chains while preserving correct ones, consistent with our general account of why P2 is robust to overthinking (Section 2.7).”

May combine two claims.

Chain-of-thought reasoning requires sufficient model size to execute coherently; smaller models may not benefit from it.

“These results suggest that chain-of-thought reasoning requires sufficient model size to execute coherently; smaller models may may not benefit from it.”

Classified as probably not a substantial claim. May combine two claims.

The largest of 10,000 CI95% intervals was chosen, and one CI range (the largest) per QA dataset applies to all values.

“We then used these 10,000 uncertainty-aware estimates of to compute 10,000 CI95% intervals from which we chose the largest interval. This expansion of CIs often made them comparable across the ranges of reported values (0.2-0.8) for a given n. Therefore, we report one CI range (the largest) per QA dataset that applies to all values.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We did not conduct an independent inter-annotator agreement or item-difficulty analysis of the source items; this is a limitation.

“We did not conduct an independent inter-annotator agreement or item-difficulty analysis of the source items; we note this as a limitation.”

Classified as probably not a substantial claim.

The Benjamini-Hochberg false discovery rate procedure was applied across the full family of McNemar tests (25 tests).

“we applied the Benjamini–Hochberg false discovery rate (FDR) procedure across the full family of McNemar tests (25 tests).”

May combine two claims.

Because the work uses only existing, publicly available, non-identifiable data and generates no new human-subjects data, it did not constitute human-subjects research and no IRB approval or informed consent was required.

“Because the work uses only existing, publicly available, non-identifiable data and generates no new human-subjects data, it did not constitute human-subjects research and no institutional review board approval or informed consent was required.”

May combine two claims.

When high-quality context is available, PCCR's extended sequential reasoning leads models to second-guess correct context-derived answers, an overthinking effect that compounds across continuation rounds.

“When high-quality context is available, PCCR’s extended sequential reasoning leads models to second-guess correct context-derived answers—an “overthinking” effect that compounds across continuation rounds.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We did not perform a temperature sensitivity sweep; this absence is a limitation.

“We did not perform a temperature sensitivity sweep; we note the absence of such an ablation as a limitation.”

May combine two claims.

When a model's reasoning is already reliable, the overhead of multiple samples provides no benefit and may introduce noise.

“This suggests that when a model’s reasoning is already reliable, the overhead of multiple samples provides no benefit and may introduce noise.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study involved no human subjects, no animal subjects, and no collection of new patient data.

“This study involved no human subjects, no animal subjects, and no collection of new patient data.”

Classified as probably not a substantial claim.

To account for residual labeling uncertainty, 10,000 bootstrapped samples were generated by pseudo-randomly flipping correct and incorrect answers with the corresponding error-rate of the LLM-as-judge (1%) for each model-prompt configuration for each dataset.

“To account for residual labeling uncertainty in the LLM-as-judge (Cohen’s = 0.99 with human raters), we generated 10,000 bootstrapped samples by pseudo-randomly flipping correct and incorrect answers with the corresponding error-rate of the LLM-as-judge (1%) for each model-prompt configuration for each dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The specific PCCR token budget values were a pragmatic design choice rather than pilot-optimized thresholds.

“The specific values were a pragmatic design choice rather than pilot-optimized thresholds.”

May combine two claims.

Self-consistency is most effective when the underlying model produces high-quality but variable reasoning chains, a condition that may be better satisfied by larger or fine-tuned models.

“This pattern indicates that self-consistency is most effective when the underlying model produces high-quality but variable reasoning chains—a condition that may be better satisfied by larger or fine-tuned models.”

May combine two claims.

P2's independent sampling produces some chains that correctly extract information from context and others that overthink; majority voting preferentially selects correct answers, providing robustness against overthinking.

“In contrast, P2’s independent sampling produces some chains that correctly extract information from context and others that overthink; majority voting preferentially selects the correct answers, providing a form of robustness against overthinking.”

Probably not a substantial claim: may not pass S4 (based).

PCCR ensures that the model explicitly revisits and reflects on its prior reasoning, potentially enabling error correction and consideration of alternative hypotheses.

“Finally, unlike simple extended generation, PCCR ensures that the model explicitly revisits and reflects on its prior reasoning, potentially enabling error correction and consideration of alternative hypotheses.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The physician performance figures used for comparison are aggregate, previously published statistics reported by Katz et al. and were not collected or re-identified by us.

“The physician performance figures used for comparison are aggregate, previously published statistics reported by Katz et al. [25] and were not collected or re-identified by us.”

• cites [25] U Katz, 2024, GPT versus resident physicians — a benchmark based on official board scores, NEJM AI sourceNot checked — no full text of the source

May combine two claims.

MedGemma 4B's P1 failures on PubMedQA with context stemmed from high-variance reasoning rather than systematic error; majority voting filtered out erroneous reasoning chains while preserving correct ones.

“This recovery suggests that MedGemma 4B’s P1 failures stemmed from high-variance reasoning rather than systematic error; majority voting may have filtered out the erroneous reasoning chains while preserving correct ones.”

May combine two claims.

The token cost of P2 relative to P1 yields no benefit but no harm, whereas similar token cost of PCCR-1024 yields substantial harm.

“The 5 token cost of P2 relative to P1 yields no benefit but also no harm, whereas the similar token cost of PCCR-1024 yields substantial harm.”

May combine two claims.

Extended reasoning is valuable when models must rely on parametric knowledge but counterproductive when relevant information is externally provided.

“This pattern suggests that extended reasoning is valuable when models must rely on parametric knowledge but counterproductive when relevant information is externally provided.”

Classified as probably not a substantial claim.

Only model-prompt configurations with accuracy values at least one expanded, full CI95% range away from each other were compared.

“To account for the imperfect adjudication of the LLM-as-judge when performing statistical tests, we only compare model-prompt configurations with accuracy values that are at least one expanded, full CI95% range away from each other.”

May combine two claims.

Model weights require approximately 0.65 bytes/parameter at INT4 and 2.0 bytes/parameter at FP16.

“Model weights require approximately 0.65 bytes/parameter at INT4 and 2.0 bytes/parameter at FP16 (S3 Appendix)”

• cites S3 AppendixNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

A 27B model occupies 17 GB at INT4 versus 54 GB at FP16-the difference between fitting on a single consumer GPU and requiring datacenter hardware.

“so, for example, a 27B model occupies 17 GB at INT4 versus 54 GB at FP16—the difference between fitting on a single consumer GPU and requiring datacenter hardware.”

May combine two claims.

At equivalent cost, P2 provides higher accuracy on average when context is available, making it the dominant strategy for RAG-based applications.

“At equivalent cost, P2 provides higher accuracy on average when context is available, making the dominant strategy for RAG-based applications.”

May combine two claims.

The baseline log fit is a two-parameter fit to three points; close agreement (mean absolute error 0.5 pp; all predictions within 1 pp of observed) should be read as an in-sample consistency check rather than evidence of extrapolation to unseen scales (pooled R2=0.95).

“This is a two-parameter fit to three points, so the close agreement (mean absolute error 0.5 pp; all predictions 1 pp of observed) should be read as an in-sample consistency check rather than as evidence of extrapolation to unseen scales (pooled R2 = 0.95).”

May combine two claims.

Our results suggest a fundamental bifurcation in optimal strategy based on context quality, informing general heuristics for system design.

“As highlighted by the two examples, our results suggest a fundamental bifurcation in optimal strategy based on context quality, informing general heuristics for system design:”

May combine two claims.

P2's five samples can be generated in parallel, while PCCR's continuation rounds are inherently sequential.

“A key practical consideration is that P2’s five samples can be generated in parallel, while PCCR’s continuation rounds are inherently sequential.”

May combine two claims.

P2 achieves its accuracy benefits with latency comparable to single sample P1, while PCCR incurs multiplicative latency from sequential continuation rounds.

“This means P2 achieves its accuracy benefits with latency comparable to single sample P1, while PCCR incurs multiplicative latency from sequential continuation rounds.”

May combine two claims.

Without context, P2 benefits require the model to produce multiple valid reasoning paths that converge on correct answers-a capability that depends on model scale and domain knowledge.

“Without context, P2 benefits require the model to produce multiple valid reasoning paths that converge on correct answers—a capability that depends on model scale and domain knowledge.”

Classified as probably not a substantial claim. May combine two claims.

The equations are a descriptive parameterization of observed relationships and are not predictive models validated on held-out models or datasets.

“We stress at the outset that the equations below are a descriptive parameterization of the relationships we observed and are not predictive models validated on held-out models or datasets.”

May combine two claims.

The study has not performed cross-family or held-out validation.

“We have not performed cross-family or held-out validation; we identify this as the most important next step (see Limitations and Future work).”

May combine two claims.

The relative multipliers between strategies (Table 14) are what our analysis depends on and are insensitive to the specific price basis.

“The relative multipliers between strategies (Table 14) are what our analysis depends on and are insensitive to the specific price basis.”

May combine two claims.

P2 (self-consistency) and PCCR operate through distinct mechanisms with different context interactions.

“Critically, our results reveal that P2 (self-consistency) and PCCR operate through distinct mechanisms with different context interactions:”

May combine two claims.

With context, even smaller models can extract correct answers from the provided evidence, but do so with variable reliability; P2 voting filters out erroneous extractions while preserving correct ones.

“With context, even smaller models can extract correct answers from the provided evidence, but do so with variable reliability; P2 voting filters out erroneous extractions while preserving correct ones.”

May combine two claims.

The run-to-run variability of P2 accuracy was small (0.5 pp) relative to reported 95% CIs, indicating P2 accuracies are stable across independent repetitions.

“This sampling variability was small (0.5 pp) relative to the reported 95% confidence intervals, indicating that the P2 accuracies reported throughout are stable across independent repetitions of the sampling procedure.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Good within-study fit reflects that the same qualitative patterns recur across all three datasets, which is supportive but non-independent evidence.

“Where the framework’s within-study fit is good, this reflects that the same qualitative patterns recur across all three datasets, which is supportive but non-independent evidence.”

May combine two claims.

The optimization problem defined by our framework is tractable and can be solved analytically or numerically to yield optimal configurations.

“We now demonstrate that the optimization problem defined by our framework is tractable and can be solved analytically or numerically to yield optimal configurations.”

May combine two claims.

The divergence between P2 and PCCR reflects the fundamental mechanistic difference: P2 samples independent reasoning chains and aggregates via voting, providing robustness against overthinking, while PCCR accumulates reasoning within a single chain, allowing errors to compound.

“This divergence reflects the fundamental mechanistic difference: P2 samples independent reasoning chains and aggregates via voting, providing robustness against overthinking, while PCCR accumulates reasoning within a single chain, allowing errors to compound.”

May combine two claims.

At very high query volumes, RAG cost may be comparable to the cost of token generation, but fine-tuning cost may remain high.

“At very high query volumes, may be comparable to the cost of token generation, , but may remain high.”

May combine two claims.

Agreement rate within P2 sampling functions as an implicit confidence measure; high agreement (1-2 unique answers) indicates reliable model knowledge and predicts correctness, while high variability (3-4 unique answers) signals uncertainty and predicts errors.

“These findings suggest that the agreement rate within P2 sampling functions as an implicit confidence measure. High agreement (1–2 unique answers) indicates reliable model knowledge and predicts correctness, while high variability (3–4 unique answers) signals uncertainty and predicts errors.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Increases or reductions in the query volume will result in corresponding linear changes to reference values for RAG and fine-tuning costs used in this analysis, but the relative difference between them would be maintained.

“Increases or reductions in the query volume will result in corresponding linear changes to reference values for and that were used in this analysis. However, the relative difference between and would be maintained.”

May combine two claims.

Several parameterizations are low-dimensional, so their in-sample fit should be read as a consistency check, not as evidence of out-of-sample generalization.

“Several parameterizations are deliberately low-dimensional (e.g., the baseline log fit uses two parameters for three model sizes), so their in-sample fit should be read as a consistency check, not as evidence of out-of-sample generalization.”

May combine two claims.

The transferable contribution is the framework's structure; the particular numbers likely are not generalizable.

“The transferable contribution is the framework’s structure in placing which variables matter, how they interact, and the method for estimating them. The particular numbers likely are not generalizable.”

May combine two claims.

Without context, models benefit from additional reasoning to access and organize parametric knowledge; with high-quality context, continued reasoning introduces opportunities to second-guess correct answers, with each iteration carrying a probability of harmful revision.

“Without context, models benefit from additional reasoning to access and organize parametric knowledge; initial responses are often uncertain and refinable. With high-quality context, initial responses are more likely to be correct context extractions; continued reasoning introduces opportunities to second-guess these correct answers, with each iteration carrying a probability of harmful revision.”

May combine two claims.

These results suggest that chain-of-thought reasoning requires sufficient model capacity to execute coherently; smaller models appear to "talk themselves out of" correct initial responses.

“These results suggest that chain-of-thought reasoning requires sufficient model capacity to execute coherently; smaller models appear to “talk themselves out of” correct initial responses.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The value of an accuracy improvement is entirely use-case-dependent and multiple-choice accuracy is at best a proxy for clinical utility.

“The value of an accuracy improvement is entirely use-case-dependent and multiple-choice accuracy is at best a proxy for clinical utility.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The evaluation used multiple-choice question-answering benchmarks, which do not fully capture the complexity of clinical reasoning.

“Our evaluation used multiple-choice question-answering benchmarks, which, while standard in the field, do not fully capture the complexity of clinical reasoning.”

May combine two claims.

The moderating variable for the overthinking effect is context availability rather than the continuation-prompt wording per se.

“so the moderating variable is context availability rather than the prompt wording per se”

Probably not a substantial claim: may not pass S3 (load-bearing).

The framework treats accuracy, cost, and latency as static properties of configurations.

“Our framework treats accuracy, cost, and latency as static properties of configurations.”

May combine two claims.

Our results suggest a more nuanced picture: scale remains important, but its benefits can be substantially replicated through alternative strategies.

“Our results suggest a more nuanced picture: scale remains important, but its benefits can be substantially replicated through alternative strategies.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The suboptimal performance on the medical licensing examination may be because its questions were released recently, from a country other than the US, and are therefore less similar to model training data, representing a more rigorous test of generalization.

“This may be because the medical licensing examination questions were released recently, from a country other than the US, and, therefore, are less likely to be similar to the model training data, representing a more rigorous test of generalization.”

May combine two claims.

The answer-flip failure mode is invisible to end-of-pipeline accuracy metrics and is particularly dangerous because it occurs precisely when reliable evidence is available.

“This failure mode that is invisible to end-of-pipeline accuracy metrics and particularly dangerous because it occurs precisely when reliable evidence is available.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper's specific findings-including the chain-of-thought threshold, overthinking penalty rates, and equivalence points-may not generalize to other architectures.

“our specific findings—including the chain-of-thought threshold, overthinking penalty rates, and equivalence points—may not generalize to other architectures.”

May combine two claims.

No controlled ablation varying the continuation-prompt phrasing while holding token budget fixed was run, so the contribution of the instruction's content cannot be fully separated from that of extended reasoning.

“Nonetheless, we did not run a controlled ablation that varies the continuation-prompt phrasing while holding token budget fixed, so we cannot fully separate the contribution of the instruction’s content from that of extended reasoning.”

May combine two claims.

The equivalence points identified establish equal accuracy-per-dollar on a multiple-choice proxy and not clinical readiness.

“The equivalence points we identify establish equal accuracy-per-dollar on a multiple-choice proxy and not clinical readiness.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.