MAIRA-2: Grounded Radiology Report Generation

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Grounded reporting may have utility to improve communication with patients when reviewing image findings.

“Grounded reporting may also have utility for non-radiology clinicians, where image grounding can support comprehension and a deeper engagement with the image beyond the text report; 2 and to improve communication with patients when reviewing image findings.3”

• cites [3] 3. Kalidindi S and Gandhi S. Workforce Crisis in Radiology in the UK and the Strategies to Deal With It: Is Artificial Intelligence the Saviour? Cureus 2023;15:e43866. sourceCould not verify

  The Review’s “Assistance in Reporting” section discusses AI-assisted detection, measurement, and reporting, but not grounded reporting or communication with patients while reviewing images. Searches for image grounding and patient communication found no passage addressing that proposed use. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The official MIMIC-CXR test split is highly enriched for abnormal cases, hence prior studies are more common.

“We note that the official MIMIC-CXR test split is highly enriched for abnormal cases, 26 hence prior studies are more common (Table 1).”

• cites [26] 26. Boecking B, Usuyama N, Bannur S, et al. MS-CXR: Making the Most of Text Semantics to Improve Biomedical Vision- Language Processing (version 0.1). 2022. doi: 10.13026/ B90J -VB87. url: https : // physionet . org / content / ms - cxr/0.1/. sourceCould not verify

  Section 4.1 (p. 9) describes the paper’s MIMIC-CXR training, validation, and held-out evaluation data, but does not give the abnormal-case distribution of the official test split. Section B.1 (p. 28) says normal scans are generally more common; it does not address whether the official test split is enriched for abnormal cases. Searches for the official split and abnormal-case prevalence did not establish the claim. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Grounded reporting differs from medical phrase grounding in that phrase grounding aims to ground a specified finding or phrase, typically assumed present within the image.

“Grounded reporting differs from the existing task of medical phrase grounding 3,[5][6][7] in that phrase grounding aims to ground a specified finding or phrase, typically assumed present within the image.”

• cites [3] 3. Kalidindi S and Gandhi S. Workforce Crisis in Radiology in the UK and the Strategies to Deal With It: Is Artificial Intelligence the Saviour? Cureus 2023;15:e43866. sourceCould not verify

  The Review’s “Assistance in Reporting” section discusses AI-assisted detection, segmentation, and reporting, but does not define medical phrase grounding or compare it with grounded reporting. Searches for the specified phrase and for grounding or localization in radiology images found no passage addressing that distinction. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [5][6][7]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This bidirectional approach differs from traditional factual verification approaches such as FActScore that assume a 'single' source of truth (e.g., Wikipedia), but has precedents in medical summarisation where both completeness and conciseness are important.

“This bidirectional approach differs from traditional factual verification approaches such as FActScore that assume a 'single' source of truth (e.g., Wikipedia), but has precedents in medical summarisation where both completeness and conciseness are important.45”

• cites [45] 45. Bernstein MH, Atalay MK, Dibble EH, et al. Can incorrect artificial intelligence (AI) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography. European Radiology 2023;33:8263-9. sourceCould not verify

  Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: The generated and ground-truth reports are assumed to consist of lists of sentences, each describing a single finding.

Probably not a substantial claim: may not pass S3 (load-bearing).

A variant of the grounded reporting task was explored in Tanida et al., where the model first located anatomical regions before generating region-level descriptions.

“A variant of this task was explored in Tanida et al. 8 , where the model first located anatomical regions before generating region-level descriptions.”

• cites [8] 8. Yildirim N, Richardson H, Wetscherek MT, et al. Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology. arXiv preprint arXiv:2402.14252 2024. sourceCould not verify

  Section 6.2 (p. 7) describes a draft-report prototype that links findings to image locations, but it does not describe a model locating anatomical regions before generating region-level descriptions. Searches for Tanida and that sequence found no passage addressing the attributed study. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The importance of the Indication section is already well-established.

“We do not explore dropping the Indication section here as its importance is already well-established.9,12”

• cites [9] 9. Yu F, Endo M, Krishnan R, et al. Evaluating progress in au- tomatic chest X-ray radiology report generation. Patterns 2023;4:100802. sourceCould not verify

  The Introduction and Experimental procedures discuss narrative reports, findings, impressions, and report-quality metrics, but do not establish the importance of the Indication section. Searches for “Indication section” and “indication” found no passage addressing that point. Page numbers were not available in our copy. (LLM, conf=0.70)

  Experimental — accuracy not yet measured
• cites [12] 12. Tu T, Azizi S, Driess D, et al. Towards Generalist Biomedical AI. NEJM AI 2024;1:AIoa2300138. sourcePartly supported

  The paper describes the Indication section as contextual information used with the chest X-ray to predict the Findings section (section labeled “8 Perspective on Generalist Biomedical AI”; page unavailable). Its “Side-by-side evaluation” section also says raters received the indication. These passages support its relevance to the task, but do not establish its importance or assess what happens when it is dropped. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

May combine two claims.

The qualitative analyses were predominantly with a single radiologist, limiting generalisability, especially as reporting styles can differ with geography.

“Further, although we conducted extensive qualitative analyses, these were predominantly with a single radiologist, limiting generalisability, especially as reporting styles can differ with geography.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

Classified as probably not a substantial claim.

The radiologist concluded that MAIRA-2 outputs were 'acceptable as a draft', alike 'the performance of a junior-to-mid level resident' that needs additional human expert review before signing-off.

“Overall, these findings led the radiologist to conclude that the MAIRA-2 outputs were 'acceptable as a draft', alike 'the performance of a junior-to-mid level resident' that needs to receive additional human expert review before signing-off on any one report.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

Dropping the prior study alone has a larger effect on clinical metrics such as CheXbert Macro F1-14 while dropping the Comparison predominantly impacts lexical metrics.

“Additional piece-wise ablations (Appendix D.6) show that dropping the prior study alone has a larger effect on clinical metrics such as CheXbert Macro F 1 -14 while dropping the Comparison predominantly impacts lexical metrics.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

Significant improvement is observed on RadGraph-F1 and CheXbert Macro F1-14.

“On existing clinical metrics, significant improvement is observed on the RadGraph-F 1 and on CheXbert Macro F 1 -14.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

On PadChest-GR, RadFact grounding precision and recall are 80% and 77%.

“On PadChest-GR RadFact grounding precision and recall are more balanced, with scores of 80% and 77%.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

On PadChest-GR, RadFact logical precision and recall are 56% and 51%.

“On PadChest-GR, RadFact logical precision and recall are 56% and 51%.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

RadFact provides for sentence-level granularity on model errors, and naturally supports both grounded and non-grounded reporting.

“Further, RadFact provides for sentence-level granularity on model errors, and naturally supports both grounded and non-grounded reporting.”

May combine two claims.

Qualitative review by a thoracic radiologist of twenty random cases from GR-Bench indicates that 14/20 reports (70%) required fewer than two corrections, and 123/135 generated sentences (91%) were acceptable as-is.

“Qualitative review by a thoracic radiologist of the text generated by MAIRA-2 on twenty random cases from GR-Bench (Figure 4) indicate that 14/20 reports (70%) required fewer than two corrections, and 123/135 generated sentences (91%) were acceptable as-is.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

Not using the prior study and Comparison during training produces a significant drop across all metrics compared to the MAIRA-2 baseline, and results in hallucinatory %Comparison mentions close to the background rate of 75% in this dataset.

“Not using the prior study and Comparison during training produces a significant drop across all metrics compared to the MAIRA-2 baseline as shown in Figure 5A, and results in hallucinatory %Comparison mentions close to the background rate of 75% in this dataset.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

The clinical implications of most errors were minor to none, with only two significant omissions observed.

“The 'clinical implications' of most errors were minor to none, with only two significant omissions observed.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

Extensive qualitative review with a radiologist indicates that MAIRA-2 produces reports which may be acceptable as a 'first draft' subject to consultant review, with the majority of generated sentences acceptable as-is.

“Extensive qualitative review with a radiologist, indicates that MAIRA-2 produces reports which may be acceptable as a 'first draft' subject to consultant review, with the majority of generated sentences acceptable as-is.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

Omissions were the most common error category (15 of 25 corrections), indicating model limitations include lower sensitivity on minor findings, occasional lack of internal consistency in reports, and lesser knowledge of device characteristics.

“With omissions being the most common error category (15 of 25 corrections), this analysis indicates model limitations include lower sensitivity on minor findings, occasional lack of internal consistency in reports, and lesser knowledge of device characteristics.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

The most commonly-observed error is missed finding, so work to improve recall is required.

“However, with the most commonly-observed error being that of missed finding, work to improve recall is required.”

• AssumptionRests on an assumption: The reference reports used as ground truth are accurate and complete.

May combine two claims.

RadFact does not rely on a fixed set of finding categories or a model specialised to a certain reporting style, instead operating via more flexible logical inference.

“Using the generalisation capabilities and reasoning faculties of LLMs, RadFact does not rely on a fixed set of finding categories or a model which is specialised to a certain reporting style, instead operating via more flexible logical inference.”

May combine two claims.

The notable exception is pneumonia, with recall of ≈ 3.4%, indicating that over 96% of mentions of pneumonia in the original reports have been removed by this processing.

“The notable exception is pneumonia, with recall of ≈ 3.4%, indicating that over 96% of mentions of pneumonia in the original reports have been removed by this processing.”

May combine two claims.

Table D.8 indicates limited impact with most metrics showing overlapping confidence intervals.

“Table D.8 indicates limited impact with most metrics showing overlapping confidence intervals.”

May combine two claims.

The Kendall rank correlation coefficient between the error counts in ReXVal and the logical F1-score of RadFact is 0.59 [0.51, 0.66] (0.62 [0.55, 0.68] for clinically significant errors).

“The Kendall rank correlation coefficient between the error counts in ReXVal and the logical F1-score of RadFact (computed as the harmonic mean between the logical precision and the logical recall) is 0.59 [0.51, 0.66] (0.62 [0.55, 0.68] for clinically significant errors).”

• AssumptionThe authors state this as an assumption: The generated and ground-truth reports are assumed to consist of lists of sentences, each describing a single finding.

May combine two claims.

For pathology-level CheXbert metrics, specificity is above 97% for all classes, indicating the conversion into sentence lists does not produce additional mentions of findings.

“For pathology-level CheXbert metrics, specificity is above 97% for all classes, indicating the conversion into sentence lists does not produce additional mentions of findings.”

Classified as probably not a substantial claim. May combine two claims.

To compute RadFact we need to process one such query per sentence in the report, in each direction. This results in, on average, six to seven queries per report.

“In practice, to compute RadFact we need to process one such query per sentence in the report, in each direction.This results in, on average, six to seven queries per report.”

• AssumptionThe authors state this as an assumption: The generated and ground-truth reports are assumed to consist of lists of sentences, each describing a single finding.

May combine two claims.

Training with the comparison section but dropping it at inference results in an even larger drop in ROUGE-L (-22.4) and an overall decrease in performance across all other metrics.

“When we train with the comparison section but drop it at inference time 'Infer:No Comp', we note an even larger drop in ROUGE-L (-22.4) in addition to an overall decrease in performance across all other metrics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Protocol differences produced fewer but larger boxes per finding in GR-1 compared to GR-Bench, especially for bilateral findings.

“Protocol differences produced, for example, fewer but larger boxes per finding in GR-1 compared to GR-Bench, especially for bilateral findings.”

May combine two claims.

Training without the comparison section results in a significant drop in ROUGE-L (-9.1%) and an increase in RadCliQ (+6.4), but no significant drop in Macro F1-14.

“When we train a model without the comparison section 'Train:No Comp', we observe a significant drop in lexical metrics (-9.1% drop in ROUGE-L) as well as an increase in RadCliQ (+6.4), but no significant drop in Macro F 1 -14.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This example has low RadFact logical precision and recall (0.0).

“This example has low RadFact logical precision and recall (0.0).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In terms of bounding box placement and size, the MAIRA-2 box for finding A was considered a bit too big, but acceptable; whereas the corresponding bounding boxes for the reference text (Finding 1) were described as a bit too narrow; especially on the right side.

“In terms of bounding box placement and size, the MAIRA-2 box for finding A was considered a bit too big, but acceptable; whereas the corresponding bounding boxes for the reference text (Finding 1) were described as a bit too narrow; especially on the right side.”

May combine two claims.

When the technique information is omitted during training 'Train:No Tech', the model can still figure out when to mention the lateral view (36.33%) but not as accurately as in MAIRA-2.

“when this information is omitted during training 'Train:No Tech', the model can still figure out when to mention the lateral view (36.33%) but not as accurately as in MAIRA-2.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This example has high grounding precision (1.0).

“This example has high grounding precision (1.0).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The selected example has moderate MAIRA-2 RadFact logical precision (0.78) and recall (0.75).

“The selected example has moderate MAIRA-2 RadFact logical precision (0.78) and recall (0.75).”

Probably not a substantial claim: may not pass S3 (load-bearing).

This example has moderate grounding precision (0.5).

“Figure E.9: This example has moderate grounding precision (0.5).”

Probably not a substantial claim: may not pass S3 (load-bearing).

MAIRA-2 outputs also did not include the report of the 'calcification of aortic knob'.

“MAIRA-2 outputs also did not include the report of the”

May combine two claims.

MAIRA-2 falsely generated a whole right lung box for a cleared pneumothorax (finding G), which would not match to the much narrower reference bounding box that is centered on the pleural thickening; thereby explaining the lower entailed box performance metrics.

“MAIRA-2 falsely generated a whole right lung box for a cleared pneumothorax (finding G), which would not match to the much narrower reference bounding box that is centered on the pleural thickening; thereby explaining the lower entailed box performance metrics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This example has low grounding precision (0.0).

“This example has low grounding precision (0.0).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

The consultant radiologist preferred the MAIRA-2 phrase of 'Median sternotomy sutures noted.', since it is more accurate in its indication of the sutures: sternotomy rather than the mediastinal.

“the consultant radiologist preferred the MAIRA-2 phrase of”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

This study presents an example of good logical precision, however, with lower spatial performance.

“As such, this study presents an example of good logical precision, however, with lower spatial performance.”

• Also stated in: § Handling of prior images · plain

  “this study presents an example of good logical precision, however, with lower spatial performance.”

May combine two claims.

Dropping the technique in the presence of the lateral view results in a large drop in ROUGE-L (-15.59%) and a substantial increase in %Lateral mentions.

“Moreover, when we drop the technique in the presence of the lateral view 'Infer:No Tech', we see a large drop in ROUGE-L (-15.59%) and a substantial increase of the %Lateral mentions,”

May combine two claims.

Our findings showed that data annotations were made to serve goals of completeness, disambiguating potentially inconclusive non-pathological findings, and improving language clarity.

“our findings showed that -under consideration of the study context (e.g., baseline scan) -data annotations were made to also serve goals of:”

May combine two claims.

All four study examples from Yang et al. represent mostly 'normal' patient cases that make little or no references to prior or lateral images.

“We find that all four study examples represent mostly”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Regarding finding 5, the term 'mitral' simply presents a type of 'intracardiac valve', and therefore finding F was considered acceptable.

“Regarding finding 5, the term”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In this instance identified requirements included the need to improve sensitivity to minor findings detection, to consider internal logic and consistency, and to expand on device knowledge.

“In this instance identified requirements included the need: to improve sensitivity to minor findings detection; to consider internal logic and consistency; and requirements to expand on device knowledge.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In review with a consultant radiologist, they agreed with these findings, however, they found that the corresponding MAIRA-2 bounding boxes for findings B and C were too big.

“In review with a consultant radiologist, they agreed with these findings, however, they found that the corresponding MAIRA-2 bounding boxes for findings B and C were too big.”

May combine two claims.

MAIRA-2 can generate grounded reports, and establishes new state-of-the-art in non-grounded report generation.

“MAIRA-2 can generate grounded reports, and establishes new state-of-the-art in non-grounded report generation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In radiologist review, the left lower lobe omission was indeed considered a missed subtle finding.

“In radiologist review, the left lower lobe omission was indeed considered a missed subtle finding.”

Classified as probably not a substantial claim. May combine two claims.

The example cases included a median of 2 pathologies per report (mean = 1.8, min = 0, max = 7) with no overlap or interaction between pathologies.

“However, they also acknowledged that the example cases were not very complex in that on average, they included a Median of 2 pathologies per report (Mean = 1.8, min = 0, max = 7) with no overlap or interaction between pathologies”

May combine two claims.

MAIRA-2 achieves RadFact logical precision above 50% with high grounding precision (68.8%, 80.2% respectively) and moderate spatial precision (33.5%, 37.1%).

“MAIRA-2achieves RadFact logical precision above 50% with high grounding precision (68.8%, 80.2% respectively) and moderate spatial precision (33.5%, 37.1%).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In this example study, MAIRA-2 outputs the Technique, which is in keeping with the reference report.

“In this example study, like Figure E.12, MAIRA-2 outputs the Technique, which is in keeping with the reference report.”

May combine two claims.

On the phrase grounding task, MAIRA-2 achieves competitive performance against baselines developed specifically for phrase grounding (MedRPG and TransVG) and appears to strongly outperform the multi-task ChEX model.

“On the phrase grounding task, MAIRA-2 achieves competitive performance against baselines developed specifically for phrase grounding (MedRPG and TransVG) and appears to strongly outperform the multi-task ChEX model.”

• cites MedRPGNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites ChEXNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites TransVGNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This example has moderate RadFact logic precision (0.78) and recall (0.64).

“This example has moderate RadFact logic precision (0.78) and recall (0.64).”

May combine two claims. The quote only partly states this claim.

In a radiologist review, two missed findings were identified: a calcified granuloma in the right lower lung and a slight compression fracture of T1, which can only be seen on the lateral view.

“In a radiologist review, we find two missed findings:”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The reference text states that the finding of 'linear scarring versus subsegmental atelectasis' exists bilaterally, whereas the MAIRA-2 outputs describe these as right-sided only.

“The reference text states that the finding of”

May combine two claims.

MAIRA-2 produces higher ROUGE-L scores and statistically equivalent CheXbert Micro F1 scores.

“MAIRA-2 produces higher ROUGE-L scores and statistically equivalent CheXbert Micro F 1 -14 scores.”

• cites LLaVA-RadNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

In this example, all generated MAIRA-2 phrases were evidenced by the reference text (RadFact logical precision: 1.0).

“In this example, all generated MAIRA-2 phrases were evidenced by the reference text (RadFact logical precision: 1.0).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

MAIRA-2 output includes the finding of a 'Calcification is noted within the aortic knob.', which is described in radiologist review as a plausible, borderline finding that was however not included in the reference text.

“MAIRA-2 output includes the finding of a”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Preferences were expressed for MAIRA-2 stating 'Lungs are clear', which is more concise than its counterpart; and for MAIRA-2's ordering of the findings from lungs to the cardiomediastinal/hilar structures and then the pleura, compared to the Med-Gemini findings that move between those structures.

“preferences were expressed for MAIRA-2 stating”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both generated candidates are near identical and match the reference text findings.

“Both generated candidates are near identical and match the reference text findings.”

The quote only partly states this claim.

Amongst the 25 corrections, 'omissions' of findings was the most prominent error type (n = 15) evidenced through either the addition of an entirely new finding phrase (n = 12) or as an adjunct to an existing phrase (n = 3).

“Amongst the 25 corrections (see Table F.1 to Table F.3 for the full list),”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

The MAIRA-2 output produces the exact same location description for the enteric tube as the reference text stating it to terminate 'in the left upper quadrant'; the Med-Gemini location of 'terminates within the stomach' is considered as more precise.

“While the MAIRA-2 output produces the exact same location description for the enteric tube as the reference text stating it to terminate”

Probably not a substantial claim: may not pass S3 (load-bearing).

Neither the reference text, nor MAIRA-2 and Med-Gemini phrases include any comparison information with the prior study.

“Neither the reference text, nor MAIRA-2 and Med-Gemini phrases include any comparison information with the prior study.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There is little difference between MAIRA-2 and Med-Gemini phrases, and the original reference text.

“there's little difference between MAIRA-2 and Med-Gemini phrases, and the original reference text.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Apart from the specific lines and tubes findings, the study in Figure E.11 is mostly normal.

“Apart from the specific lines and tubes findings, this study is mostly normal.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

One instance of incorrect progression information and two instances of incorrect location information were observed.

“We further observed one instance of incorrect progression information whereby the phrase stated”

Probably not a substantial claim: may not pass S3 (load-bearing).

The example study in Figure E.12 reports a 'normal'.

“This example study reports a”

Probably not a substantial claim: may not pass S3 (load-bearing).

Both candidate texts hallucinated the 4.5 cm measure of the endotracheal tube terminating above the carina.

“both candidate texts hallucinated the 4.5 cm measure of the endotracheal tube terminating above the carina”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The 20 example reports included 5 normal studies and encompassed in total 135 generated phrases, equating to a median of six phrases per report (min = 2, max = 13).

“The 20 example reports included 5 normal studies 1 and encompassed in total 135 generated phrases, equating to a Median of six phrases per report (min = 2, max = 13).”

May combine two claims.

In independent reviews with two radiologists, minor variances were surfaced in terms of findings missed or hallucinated, and preferences for their conciseness or ordering.

“In independent reviews with two radiologists, minor variances were surfaced in terms of findings missed or hallucinated, and preferences for their conciseness or ordering that are described in the Figure captions.”

May combine two claims.

Two of these misclassification cases stated about the image that 'The bone structures are intact', which contradicted subsequently generated phrases describing 'degenerative changes of the spine' and 'demineralized' bones.

“Two of these cases stated about the image that”

May combine two claims.

Added findings or text predominantly served to reduce ambiguity or need for clinical follow-up by explicitly stating the absence of a pathology, or clarifying the appearance of an otherwise potentially misinterpreted anatomical structure.

“(iii) to reduce ambiguity or need for clinical follow-up by explicitly stating the absence of a pathology (e.g.,”

Probably not a substantial claim: may not pass S3 (load-bearing).

One of the incorrect location cases pertained to the generated description of a pacemaker: 'Pacemakers are noted in the right atrium and ventricle', which was corrected to 'Pacemaker with lead noted in the right ventricle'.

“One of these cases pertained to the generated description of a pacemaker:”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Neither generated report findings missed any observations and both were assessed in radiologist reviews as equivalent from a clinical perspective.

“Neither generated report findings missed any observations and both were assessed in radiologist reviews as equivalent from a clinical perspective.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Like the reference text, MAIRA-2 outputs the phrase 'Frontal and lateral views of the chest were obtained.'

“Like the reference text, MAIRA-2 outputs the phrase”

May combine two claims.

These related to a missed pathology (an infiltrate in the lingula) that requires clinical action, and missed acute rib fractures in a patient case, which explained the 'chest pain' mentioned in the study indication.

“These related to a missed pathology (an infiltrate in the lingula) that requires clinical action -for example treatment with antibiotics; as well as missed acute rib fractures in a patient case, which explained the”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Amongst the total of 25 corrections, eleven presented 'edits' to existing finding phrases, two findings were 'deleted' and another 12 were 'added'.

“Amongst the total of 25 corrections, eleven presented”

May combine two claims.

The vast majority of these phrases (n = 123) and correspondingly 30% of all reports (6 out of 20) did not require any edits.

“The vast majority of these phrases (n = 123) -and correspondingly 30% of all reports (6 out of 20) -did not require any edits.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Amongst the remaining edits, the generated report findings were twice identified as being overspecific in their formulation.

“Amongst the remaining edits, the generated report findings were twice identified as being overspecific in their formulation”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The reporting style differs significantly between PadChest and MIMIC-CXR, which may impact the reliability of model-based metrics such as RadGraph-F1 that were developed for MIMIC-CXR.

“In addition, the reporting style differs significantly between PadChest and MIMIC-CXR, which may impact the reliability of model-based metrics such as RadGraph-F 1 that were developed for MIMIC-CXR.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The lower precision for PadChest-GR may be due to shorter reports in the dataset and the lower recall due to missing Indication sections in PadChest-GR, making it harder to report negatives.

“The lower precision for PadChest-GR may be due to shorter reports in the dataset and the lower recall due to missing Indication sections in PadChest-GR, making it harder to report negatives.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Grounded reporting may have utility for non-radiology clinicians, where image grounding can support comprehension and a deeper engagement with the image beyond the text report.

“Grounded reporting may also have utility for non-radiology clinicians, where image grounding can support comprehension and a deeper engagement with the image beyond the text report; 2 and to improve communication with patients when reviewing image findings.3”

• cites [2] 2. Fischetti C, Bhatter P, Frisch E, et al. The evolving impor- tance of artificial intelligence and radiology in medical trainee education. Academic Radiology 2022;29:S70-S75. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable).

The addition of grounding is a step towards real clinical impact in automated radiology report generation.

“The addition of grounding is a step towards real clinical impact in automated radiology report generation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The high RadFact scores suggest that MAIRA-2 may be producing higher-quality reports on IU-Xray than it does on MIMIC-CXR.

“In particular, the high RadFact scores suggest that MAIRA-2 may be producing higher-quality reports than it does on MIMIC-CXR, however this may also reflect that IU-Xray is an 'easier' dataset than MIMIC-CXR.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

IU-Xray may be an 'easier' dataset than MIMIC-CXR.

“In particular, the high RadFact scores suggest that MAIRA-2 may be producing higher-quality reports than it does on MIMIC-CXR, however this may also reflect that IU-Xray is an 'easier' dataset than MIMIC-CXR.”

Classified as probably not a substantial claim. May combine two claims.

We consider GR-Bench our benchmark for grounded reporting on USMix and report test results on a held-out portion of it.

“We consider GR-Bench our benchmark for grounded reporting on USMix and report test results on a held-out portion of it.”

Probably not a substantial claim: may not pass S4 (based).

There is no leakage between the training, validation and test patients in these datasets and those in Table 1.

“There is no leakage between the training, validation and test patients in these datasets and those in Table 1.”

May combine two claims.

The model generates fewer logically entailed sentences, but those which it generates are grounded correctly more often.

“the model generates fewer logically entailed sentences, but those which it generates are grounded correctly more often.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The evaluation sets were balanced with respect to the prevalence of prior mentions.

“The evaluation sets were balanced w.r.t. the prevalence of prior mentions.”

May combine two claims.

RadFact is not restricted to the six error types defined in ReXVal, and can perform entailment verification for any sentence that can potentially occur in a report, naturally leading to a lower alignment with ReXVal.

“In contrast, RadFact is not restricted to the six error types defined in ReXVal, and can perform entailment verification for any sentence that can potentially occur in a report, naturally leading to a lower alignment with ReXVal.”

• AssumptionThe authors state this as an assumption: The generated and ground-truth reports are assumed to consist of lists of sentences, each describing a single finding.

May combine two claims.

Using Llama3-70B as a backbone instead of GPT-4 provides multiple advantages: It is open-source and faster, making it more accessible to the research community and advantageous when evaluating large volumes.

“Using Llama3-70B as a backbone instead of GPT-4 -as in Chaves et al. 14 -provides multiple advantages: It is open-source and faster, making it more accessible to the research community and advantageous when evaluating large volumes.”

Classified as probably not a substantial claim.

We use IU-Xray as a held-out dataset to evaluate the generalisation ability of MAIRA-2.

“We use IU-Xray as a held-out dataset, hence evaluate the generalisation ability of MAIRA-2 here.”

Classified as probably not a substantial claim.

The performance drop observed in Table C.1 seems negligible relative to the gain in throughput.

“In this light, the performance drop observed in Table C.1 seems neglegible relative to the gain in throughput.”

May combine two claims.

To overcome the many-to-many challenge faced by Tanida et al., where a single sentence in a report can describe multiple findings and hence several regions, we design a dataset such that each sentence describes at most a single finding, enabling precise localisation.

“To overcome the many-to-many challenge faced by Tanida et al. 8 , where a single sentence in a report can describe multiple findings and hence several regions, we design a dataset such that each sentence describes at most a single finding, enabling precise localisation.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

This definition interprets a larger region as more specific than a smaller region contained within it, as the former makes stronger claims about where a finding is located.

“This definition interprets a larger region as more specific than a smaller region contained within it, as the former makes stronger claims about where a finding is located.”

• AssumptionThe authors state this as an assumption: The generated and ground-truth reports are assumed to consist of lists of sentences, each describing a single finding.

Classified as probably not a substantial claim. May combine two claims.

RadFact does not require a one-to-one mapping between generated and reference sentences, and there can be several pieces of evidence to support a logical inference.

“Note that RadFact does not require a one-to-one mapping between generated and reference sentences, and there can be several pieces of evidence to support a logical inference.”

• AssumptionThe authors state this as an assumption: The generated and ground-truth reports are assumed to consist of lists of sentences, each describing a single finding.

Classified as probably not a substantial claim. May combine two claims.

Pneumonia is a clinical interpretation of other findings, often described with speculative language such as '... opacity suggesting pneumonia', and the prompt directs the LLM to remove clinical speculation and interpretation.

“This is expected because pneumonia is a clinical interpretation of other findings, often described with speculative language such as '... opacity suggesting pneumonia', and the prompt directs the LLM to remove clinical speculation and interpretation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This low overlap in bounding boxes explains the lower box precision scores in this instance.

“This low overlap in bounding boxes explains the lower box precision scores in this instance.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Our findings representation Figure 4 conveys the likely effort required to correct errors for a specific data subset and different error types that occur that clarify model limitations and needed improvements.

“our findings representation Figure 4 conveys the likely”

Probably not a substantial claim: may not pass S3 (load-bearing).

The ablations were conducted on a slightly earlier version of MAIRA-2.

“Note that as in Figure 5 and Appendix D.5, we conducted these ablations on a slightly earlier version of MAIRA-2.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The phrase grounding in ChEX benefits from the synergies of multitask training, combining report generation and localisation tasks.

“the phrase grounding in ChEX benefits from the synergies of multitask training, combining report generation and localisation tasks.”

• cites ChEXNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

These goals suggest their inclusion as important evaluation criteria in future benchmarking efforts.

“suggesting their inclusion as important evaluation criteria in future benchmarking efforts.”

Classified as probably not a substantial claim. May combine two claims.

Because the final MAIRA-2 model was trained with a part of MS-CXR, we report results on the intersections of our new held-out test split and each of the splits from MedRPG and ChEX.

“Because the final MAIRA-2 model was trained with a part of MS-CXR, we report results on the intersections of our new held-out test split (176 phrases, 155 images) and each of the splits from MedRPG (138 phrases, 124 images) and ChEX (30 samples, 24 images), respectively.”

• cites MedRPGNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites ChEXNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

To enable fair comparison, we report comparative results on the intersections of our test set with their respective test subsets.

“To enable fair comparison, we therefore report comparative results on the intersections of our test set with their respective test subsets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MedRPG additionally employs a combination of contrastive and attention losses to better align image-and text-features.

“MedRPG additionally employs a combination of contrastive and attention losses to better align image-and text-features.”

• cites MedRPGNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Compared to MAIRA-2, these baselines directly regress bounding box coordinates using MLP heads.

“Compared to MAIRA-2, these baselines directly regress bounding box coordinates using MLP heads.”

• cites MedRPGNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites ChEXNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites TransVGNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

We chose to include step 3, the review of the reference findings sentences, as additional context to the image interpretation, because the web-UI did not provide the same image review resolution or functionality as DICOM viewers, nor was broader patient context available.

“We chose to include step 3, the review of the reference findings sentences, as additional context to the image interpretation -alike a peer reviewer perspectivesince the web-UI did not provide the same image review resolution or functionality that is provided by DICOM viewers, nor was any broader patient context available.”

• AssumptionRests on an assumption: The web-UI did not provide the same image review resolution or functionality as DICOM viewers, nor was broader patient context available.

Classified as probably not a substantial claim. May combine two claims.

Evaluating at a phrase-level enables higher-level aggregates and quantifications, and provides a record of concrete corrections.

“Whilst evaluating at a phrase-level also enables higher-level aggregates and quantifications, it has the added advantage to provide a record of concrete corrections.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Where generated MAIRA-2 findings and boxes are matching the reference, resulting grounding precision is high.

“Where generated MAIRA-2 findings and boxes are matching the reference, resulting grounding precision is high.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

These corrections can expand existing definitions of possible error types and their differentiation, and deepen understanding of what constitutes good quality in reporting outputs beyond common goals to reduce falsely classified, imprecise, or missed pathological findings.

“These can expand existing definitions of possible”

Probably not a substantial claim: may not pass S3 (load-bearing).

The synergy analysis was conducted on an earlier variant of MAIRA-2 as described in Appendix B.2.

“Note this analysis was conducted on an earlier variant of MAIRA-2 as described in Appendix B.2.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The presence of only 1 lead means that the text output should be restricted to only include one location and suggests that the MAIRA-2 model has little embedded knowledge of the device characteristics.

“The presence of only 1 lead means that the text output should be restricted to only include one location and suggest that the MAIRA-2 model has little embedded knowledge of the device characteristics.”

Classified as probably not a substantial claim. May combine two claims.

The pacemaker is singular and has only 1 lead and consequently cannot end in two locations.

“Notably, not only is the pacemaker singular, it has also only 1 lead and consequently cannot end in two locations.”

Classified as probably not a substantial claim.

A systematic, in-depth qualitative review of generated MAIRA-2 output was conducted for twenty randomly selected examples of the US Mix dataset with a thoracic radiologist.

“We conducted a systematic, in-depth qualitative review of generated MAIRA-2 output for twenty randomly selected examples of the US Mix dataset with a thoracic radiologist.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Neither model can plausibly predict the correct measurement from a chest X-ray as they have no information on scale.

“as neither model can plausibly predict the correct measurement from a chest X-ray as they have no information on scale.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The involvement of human domain experts as (qualitative) annotators is widely considered as effortful and costly and as less flexible or scalable for iterative model testing.

“The involvement of human domain experts as (qualitative) annotators is widely considered as effortful and costly; 61 and as less flexible or scalable for iterative model testing.”

• cites [61] 61. Zhu Q, Mathai TS, Mukherjee P, Peng Y, Summers RM, and Lu Z. Utilizing Longitudinal Chest X-Rays and Reports to Pre-fill Radiology Reports. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer. 2023:189-98. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

This evaluative exploration focused solely on assessing the generated report text.

“While the Demo UI has additional capability to show (and draw additional) grounding boxes, this evaluative exploration focused solely on assessing the generated report text.”