Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis?

Using o4-mini as an LLM judge and BioLORD as a vendor-neutral retrieval-based judge reduces self-preference bias.

“Our primary evaluation uses o4-mini as an LLM judge, scoring freetext differential diagnoses against reference labels.To reduce potential self-preference bias, we also implement a vendor-neutral, retrieval-based judge: BioLORD (FRemyCompany/BioLORD-2023) (Remy et al., 2024).”

• cites Remy et al., 2024 sourceSupported

The o4-mini judge produced recurring discrepancies in the HMS subset where technically correct, granular predictions were penalized.

“Upon examining the evaluations provided by the o4-mini judge, we observed a recurring discrepancy in the HMS subset where technically correct, granular predictions were penalized.”

Similar refinements occurred in cases of Cryoglobulinemic vasculitis where agents correctly identified Hepatitis C as the underlying driver.

“Similar refinements occurred in cases of Cryoglobulinemic vasculitis where agents correctly identified Hepatitis C as the underlying driver.”

Mixed-Vendor MAC maintains better performance than all single-vendor configurations as depth increases on LIRICAL.

“Moreover, the Mixed-Vendor MAC maintains better performance than all single-vendor configurations as depth increases.”

On challenging datasets like MME and DiagnosisArena, heatmaps show low Jaccard similarity between the Single LLMs.

“On challenging datasets like MME and Diag-nosisArena, the heatmaps in Figure 4 show low Jaccard similarity between the Single LLMs.”

Supplementary analysis confirms the Mixed system achieves higher gains when the competing homogeneous teams exhibit low mutual overlap.

“Supplementary analysis of Single-Vendor MACs (see Appendix) confirms this mechanism: the Mixed system achieves higher gains when the competing homogeneous teams exhibit the low mutual overlap.”

The coverage metrics and Jaccard similarities together reveal a strong inverse correlation: the Mixed system's advantage is maximized precisely when the underlying models are most distinct.

“The coverage metrics in Figure 3 and the Jaccard similarities in Figure 4 together reveal a strong inverse correlation: the Mixed system's advantage is maximized precisely when the underlying models are most distinct.”

Mixed-Vendor system exceeds the strong o4-mini Single-Vendor MAC (35.76%) on DiagnosisArena.

“Crucially, although two of the three agents in the mixed team perform poorly in isolation, their distinct reasoning processes allow the Mixed-Vendor system to exceed the strong o4-mini Single-Vendor MAC (35.76%).”

On MME, Mixed-Vendor MAC maintains a clear lead at Recall@3, 5, and 10.

“This dominance extends consistently across all retrieval depths, with the Mixed-Vendor MAC maintaining a clear lead at Recall@3, 5, and 10.”

Scaling up to a multi-agent team of identical models does not guarantee improvement over the single-model baseline.

“Notably, our results reveal that scaling up to a multi-agent team of identical models does not guarantee improvement over the single-model baseline.”

Performance gains are inversely correlated with model similarity, peaking when models possess distinct areas of expertise that compensate for each other's weakness.

“(2) performance gains are inversely correlated with model similarity, peaking when models possess distinct areas of expertise that compensate for each other's weakness;”

In some cases, Single-Vendor MACs actually underperform their standalone base models, most prominently with Gemini on both LIRICAL and HMS, and Claude on HMS.

“In some cases, Single-Vendor MACs actually underperform their standalone base models, most prominently with Gemini on both LIRICAL and HMS, and Claude on HMS.”

Mixed-Vendor MAC consistently outperforms every Single-Vendor MAC baseline on RareBench.

“Crucially, it consistently outperforms every Single-Vendor MAC baseline.”

Mixed-Vendor configuration consistently outperforms all Single-Vendor teams, regardless of the supervisor chosen.

“As shown in Table 10, the Mixed-Vendor configuration consistently outperforms all Single-Vendor teams, regardless of the supervisor chosen.”

When comparing Mixed-Vendor MAC with Single-Vendor MACs in Figure 5, we observe an increase in average gains across HMS and LIRICAL.

“However, when comparing Mixed-Vendor MAC with Single-Vendor MACs in Figure 5, we observe an increase in average gains across HMS and LIRICAL.”

High inter-model similarity on HMS suggests that individual agents capture nearly the same information.

“Conversely, high inter-model similarity on HMS (dark blue cells) suggests that individual agents capture nearly the same information.”

The MAC framework inherently introduces higher computational overhead and latency than Single-LLM baselines.

“The MAC framework inherently introduces higher compu-tational overhead and latency than Single-LLM baselines.”

In Case HMS-12, the OpenAI agent (D1) individually correctly identified the gold label at Rank 1.

“Individually, the OpenAI agent (D1) correctly identifies the gold label at Rank 1.”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.

In the MME subset, the Jaccard similarity between models B and C increases from J = 0.565 in the single-model setting to J = 0.750 when utilizing the MAC framework.

“For example, in the MME subset, the Jaccard similarity between models B and C increases from J = 0.565 in the single-model setting to J = 0.750 when utilizing the MAC framework.”

∆Coverage peaks (32.97%) precisely where the cross-vendor Jaccard indices are at their lowest (J AB = 0.269, J AC = 0.233, J BC = 0.750).

“As illustrated in Figure 5, the ∆Coverage peaks (32.97%) precisely where the cross-vendor Jaccard indices are at their lowest (J AB = 0.269, J AC = 0.233, J BC = 0.750).”

By Round 3 in Case MME-14, the entire mixed-vendor team, including the previously biased OpenAI agent, converged on the correct diagnosis at Rank 1.

“By Round 3, the diverse reasoning paths allow the entire team-including the previously biased OpenAI agent-to converge on the correct diagnosis at Rank 1.”

On the BioLORD evaluation of Combined RareBench, Mixed-Vendor MAC achieved R@1 22.99, R@3 27.42, R@5 30.45, and R@10 33.68.

“Mixed-Vendor MAC 22.99 27.42 30.45 33.68”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.

Jaccard similarities between vendor-specific MAC teams are generally higher than those observed between their single-model counterparts.

“Interestingly, the Jaccard similarities between vendor-specific MAC teams are generally higher than those observed between their single-model counterparts.”

In Case HMS-12, vendor diversity led to the suppression of a correct hypothesis.

“HMS-12 illustrates a scenario where vendor diversity leads to the suppression of a correct hypothesis.”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.
• AssumptionRests on an assumption: The selected case narratives (MME-14 and HMS-12) are representative of the systems' general behavior.

The relationship between diversity and performance gain is most evident in the MME dataset.

“Finally, the relationship between diversity and performance gain is most evident in the MME dataset.”

By Round 2 in Case HMS-12, the mixed-vendor team reached a cohesive but incorrect consensus on Lipodystrophy.

“By Round 2, the team has reached a cohesive but incorrect consensus on Lipodystrophy.”

The Mixed-Vendor configuration is not consistently the most resource-intensive setup.

“However, our experiments revealed that the Mixed-Vendor configuration is not consistently the most resource-intensive setup.”

In Case HMS-12 mixed-vendor, the correct signal from D1 in Round 0 was progressively diluted.

“As the trajectory reveals, the correct signal from D1 in Round 0 is progressively diluted.”

In Case MME-14, the Single-Vendor MAC (OpenAI) failed to resolve a shared blind spot regarding thoracic cues across multiple rounds of internal reasoning.

“the Single-Vendor MAC (OpenAI) remains trapped in an”

In Case MME-14 mixed-vendor, the first agent (OpenAI) initially missed the diagnosis in Round 0.

“Although the first agent (OpenAI) initially misses the diagnosis in Round 0”

In Case MME-14, the mixed-vendor system identified the correct diagnosis (cerebrocostomandibular syndrome) that was missed by a single-vendor team.

“This case serves as an example of successful recovery, where the mixed-vendor system identifies a correct diagnosis that was missed by a single-vendor team.”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.
• AssumptionRests on an assumption: The selected case narratives (MME-14 and HMS-12) are representative of the systems' general behavior.

On the LIRICAL subset (n=370), Mixed-Vendor MAC achieved R@1 35.69, R@3 46.59, R@5 51.77, and R@10 57.22.

“Mixed-Vendor MAC 35.69 46.59 51.77 57.22”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.

On the HMS subset (n=88), Mixed-Vendor MAC achieved R@1 51.14, R@3 60.23, R@5 68.18, and R@10 77.27.

“Mixed-Vendor MAC 51.14 60.23 68.18 77.27”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.

On the BioLORD evaluation of the HMS subset (n=88), Mixed-Vendor MAC achieved R@1 48.28, R@3 57.47, R@5 64.37, and R@10 72.41.

“Mixed-Vendor MAC 48.28 57.47 64.37 72.41”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.

In Case HMS-12 mixed-vendor, the Gemini (D2) and Claude (D3) agents were both strongly biased toward metabolic and endocrine etiologies (Lipodystrophy and Cushing's).

“the Gemini (D2) and Claude (D3) agents are both strongly biased toward”

• Also stated in: § Gold Diagnosis: Focal Segmental Glomerulosclerosis (FSGS) · plain

  “metabolic and endocrine etiologies (Lipodystrophy and Cushing's), which also explain the patient's striae and dyslipidemia.”

On the BioLORD evaluation of the LIRICAL subset (n=370), Mixed-Vendor MAC achieved R@1 16.22, R@3 19.46, R@5 21.35, and R@10 23.51.

“Mixed-Vendor MAC 16.22 19.46 21.35 23.51”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.

The Mixed-Vendor MAC consistently maintains a relatively high Jaccard similarity with each individual constituent vendor.

“Furthermore, the Mixed-Vendor MAC (D) consistently maintains a relatively high Jaccard similarity with each individual constituent vendor.”

On Combined RareBench, Mixed-Vendor MAC achieved R@1 39.31, R@3 49.82, R@5 55.05, and R@10 61.35.

“Mixed-Vendor MAC 39.31 49.82 55.05 61.35”

• AssumptionRests on an assumption: The benchmark gold diagnoses used to score the systems are correct.

The observed drop in absolute recall on MME and LIRICAL compared to the LLM judge is primarily due to lexical sensitivity.

“The observed drop in absolute recall on MME and LIRICAL compared to the LLM judge is primarily due to lexical sensitivity.”

Existing mixed-vendor approaches in medicine mostly rely on independent model runs, combining models from different providers via voting or simple aggregation.

“However, existing mixed-vendor approaches in medicine mostly rely on independent model runs, combining models from different providers via voting or simple aggregation (Barabucci et al., 2024;Sorich et al., 2024).”

• cites Barabucci et al., 2024 sourceNot checked — no full text of the source
• cites Sorich et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

This saturation explains the low ∆Coverage in Figure 3: with little unique information to aggregate, consensus-building cannot significantly expand the solution space.

“This saturation explains the low ∆Coverage in Figure 3: with little unique information to aggregate, consensusbuilding cannot significantly expand the solution space.”

The HMS subset (n = 88) underwent independent clinical adjudication by a medical doctor across all experimental configurations.

“To ensure metric validity, the HMS subset (n = 88) underwent independent clinical adjudication by a medical doctor across all experimental configurations (Single-LLM, Single-Vendor MAC, and Mixed-Vendor MAC).”

Without sufficient cross-vendor diversity, the system cannot break past the reasoning limits of its most dominant constituent.

“This result shows that without sufficient cross-vendor diversity, the system cannot break past the reasoning limits of its most dominant constituent.”

When individual models struggle, diverse reasoning strategies provided by a mixed panel are critical for surfacing the correct diagnosis that homogeneous teams miss.

“This suggests that when individual models struggle, the diverse reasoning strategies provided by a mixed panel are critical for surfacing the correct diagnosis that homogeneous teams miss.”

Homogeneous teams remain limited by correlated failure modes that conversational debate alone cannot resolve.

“(3) homogeneous teams remain limited by correlated failure modes that conversational debate alone cannot resolve.”

In the absence of role-induced variance, when homogenous agents discuss and deliberate, they risk reinforcing shared hallucinations or collectively drifting away from a correct initial hypothesis.

“In this absence of role-induced variance, when homogenous agents discuss and deliberate, they risk reinforcing shared hallucinations or collectively drifting away from a correct initial hypothesis.”

The simple symmetric setup (no role-specialized prompts) ensures any observed performance differences come strictly from vendor diversity rather than artifacts of role assignment.

“This simple, symmetric setup allows us to vary the underlying LLMs while keeping the interaction pattern unchanged, ensuring that any observed performance differences come strictly from vendor diversity rather than artifacts of role assignment.”

Individual models possess orthogonal strengths, covering disjoint slices of the medical knowledge.

“This confirms that individual models possess orthogonal strengths, covering disjoint slices of the medical knowledge.”

Overlap analysis reveals that mixed-vendor teams pool complementary inductive biases, surfacing correct diagnoses that individual models or homogeneous teams collectively miss.

“Overlap analysis reveals the underlying mechanism: mixed-vendor teams pool complementary inductive biases, surfacing correct diagnoses that individual models or homogeneous teams collectively miss.”

• Also stated in: § 1 Introduction · plain

  “Analysis of solution space overlap reveals the mechanism driving this gain: while single-vendor teams remain highly correlated with their base model's priors, mixedvendor teams successfully pool complementary inductive biases to surface diagnoses that indi-1 vidual models miss.”

• Also stated in: § 7 Conclusion · boosted

  “Our analysis demonstrates that mixed-vendor systems expand the effective solution space by aggregating diverse inductive biases, recovering correct diagnoses missed by any single model while avoiding the correlated failures that limit homogeneous teams.”

Vendor diversity yields knowledge gains unattainable through homogeneous debate.

“Figure 5 shows that vendor diversity yields knowledge gains unattainable through homogeneous debate.”

This stability confirms that the performance gains stem from the diverse inductive biases of the doctor agents, rather than the specific capabilities of the aggregator.

“This stability confirms that the performance gains stem from the diverse inductive biases of the doctor agents, rather than the specific capabilities of the aggregator.”

Models from different vendors possess complementary inductive biases.

“Rather than having a universally superior model, existing research reveals that models from different vendors possess complementary inductive biases (Yang et al., 2024).”

• cites Yang et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The zero net gain against Single-Vendor MAC (OpenAI) on HMS occurs because the Mixed-Vendor system's ability to rescue cases missed by OpenAI is numerically offset by instances where the correct OpenAI-specific signal was suppressed by the incorrect majority (Gemini and Claude).

“The zero net gain against Single-Vendor MAC (Ope-nAI) on HMS occurs because the Mixed-Vendor system's ability to rescue cases missed by OpenAI is numerically offset by instances where the correct OpenAI-specific signal was suppressed by the incorrect majority (Gemini and Claude).”

This degradation likely stems from correlated failure modes.

“This degradation likely stems from correlated failure modes.”

The performance gain is primarily driven by the unique inductive biases of OpenAI (A) relative to the more closely correlated Gemini-Claude (B-C) pair.

“These values indicates that the performance gain is primarily driven by the unique inductive biases of OpenAI (A) relative to the more closely correlated Gemini-Claude (B-C) pair.”

This inverse correlation proves that the diagnostic system derives its most significant benefits from the aggregation of uncorrelated hypotheses.

“This inverse correlation proves that the diagnostic system derives its most significant benefits from the aggregation of uncorrelated hypotheses.”

Vendor diversity can introduce a consensus trap where a correct minority signal is discarded in favor of a cohesive but incorrect majority.

“While vendor diversity often facilitates the rescue of rare diagnoses, it can also introduce a”

• AssumptionRests on an assumption: The selected case narratives (MME-14 and HMS-12) are representative of the systems' general behavior.

Vendor diversity provides a robust reasoning path capable of amplifying correct minority signals that are otherwise discarded by homogeneous teams.

“This confirms that vendor diversity provides a robust reasoning path capable of amplifying correct minority signals that are otherwise discarded by homogeneous teams.”

• AssumptionRests on an assumption: The selected case narratives (MME-14 and HMS-12) are representative of the systems' general behavior.

This synthesis effect indicates that the mixed team successfully acts as an effective knowledge bridge, aggregating disjoint slices of clinical reasoning from diverse model families into a robust final consensus.

“This synthesis effect indicates that the mixed team successfully acts as an effective knowledge bridge, aggregating disjoint slices of clinical reasoning from diverse model families into a robust final consensus.”

Model heterogeneity is a key design principle for robust multi-agent reasoning in high-stakes domains.

“These findings highlight model heterogeneity as a key design principle for robust multi-agent reasoning in high-stakes domains.”

Pooling diverse inductive biases allows the team to surface a correct diagnosis missed by single-vendor baselines.

“where the pooling of diverse inductive biases allows the team to surface a correct diagnosis missed by single-vendor baselines.”

The performance gains of multi-agent diagnostic systems stem primarily from vendor diversity, rather than from scaling homogeneous agents.

“In this work, we show that the performance gains of multi-agent diagnostic systems stem primarily from vendor diversity, rather than from scaling homogeneous agents.”