Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; 11 of the 22 sources its claims cite were checked for support.

Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study

Open access

Abstract

Background: Informed consent forms (ICFs) for clinical trials have become increasingly complex, often hindering participant comprehension and engagement due to legal jargon and lengthy content. The recent advances in large language models (LLMs) present an opportunity to streamline the ICF creation process while improving readability, understandability, and actionability. objectives: This study aims to evaluate the performance of the Mistral 8x22B LLM in generating ICFs with improved readability, understandability, and actionability. Specifically, we evaluate the model's effectiveness in generating ICFs that are readable, understandable, and actionable while maintaining the accuracy and completeness. Methods: We processed 4 clinical trial protocols from the institutional review board of UMass Chan Medical School using the Mistral 8x22B model to generate key information sections of ICFs. A multidisciplinary team of 8 evaluators, including clinical researchers and health informaticians, assessed the generated ICFs against human-generated counterparts for completeness, accuracy, readability, understandability, and actionability. Readability, Understandability, and Actionability of Key Information indicators, which include 18 binary-scored items, were used to evaluate these aspects, with higher scores indicating greater accessibility, comprehensibility, and actionability of the information. Statistical analysis, including Wilcoxon rank sum tests and intraclass correlation coefficient calculations, was used to compare outputs. Results: LLM-generated ICFs demonstrated comparable performance to human-generated versions across key sections, with no significant differences in accuracy and completeness (P>.10). The LLM outperformed human-generated ICFs in readability (Readability, Understandability, and Actionability of Key Information score of 76.39% vs 66.67%; Flesch-Kincaid grade level of 7.95 vs 8.38) and understandability (90.63% vs 67.19%; P=.02). The LLM-generated content achieved a perfect score in actionability compared with the human-generated version (100% vs 0%; P<.001). Intraclass correlation coefficient for evaluator consistency was high at 0.83 (95% CI 0.64-1.03), indicating good reliability across assessments. Conclusions: The Mistral 8x22B LLM showed promising capabilities in enhancing the readability, understandability, and actionability of ICFs without sacrificing accuracy or completeness. LLMs present a scalable, efficient solution for ICF generation, potentially enhancing participant comprehension and consent in clinical trials.

Overview

CoChat Summary

TL;DR

This mixed methods study evaluated Mistral 8x22B for generating key information sections of clinical trial informed consent forms (ICFs) from 4 research protocols. LLM-generated ICFs were comparable to human-generated ICFs in accuracy and completeness but scored higher in readability, understandability, and actionability.

Key findings

  • Accuracy and completeness were comparable across key sections, with no significant differences (P>.10); for example, benefits scored 3.0 for LLM vs 2.57 for human (P=.10), and alternatives were identical at 2.75 vs 2.75 (P≥.99).
  • LLM-generated ICFs outperformed human versions in understandability (90.63% vs 67.19%; P=.02) and actionability (100% vs 0%; P<.001), and had a higher combined RUAKI score (84.03% vs 61.82%; P=.008).
  • Readability favored the LLM but was not statistically significant: RUAKI readability 76.39% vs 66.67% (P=.26) and Flesch-Kincaid grade level 7.95 vs 8.38 (P=.77).
  • Evaluator consistency was good, with an intraclass correlation coefficient of 0.83 (95% CI 0.64-1.03).

Methods

  • Four clinical trial protocols from the UMass Chan Medical School IRB were processed with Mistral 8x22B to generate key information sections, yielding 8 ICFs total: 4 human-generated and 4 AI-generated.
  • Eight multidisciplinary evaluators, including clinical researchers, health informaticians, and physicians, assessed outputs; each protocol set was reviewed by 2 different evaluators.
  • Accuracy/completeness used a 3/2/1/0 scoring system for complete, incomplete, absent, or incorrect; readability, understandability, and actionability used the 18-item binary RUAKI indicator; analyses included Wilcoxon rank sum tests and intraclass correlation coefficient calculations.

Limitations

  • The paper states the small sample of only 4 clinical trial protocols limits generalizability and may have contributed to nonsignificant results; LLM performance is closely tied to the clarity and completeness of the source protocol; LLM text style may have been recognizable despite blinding; and the study evaluated a single LLM model.
  • The paper also notes that the 0% actionability score for human-generated ICFs reflects a structural template issue rather than a methodological flaw, and that prompt development required moderate initial time investment.
  • Evident from the text: all evaluators were also coauthors of the work, and no external participants were involved, which the paper frames as nonhuman subjects research but may still warrant consideration for evaluation independence.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/68139

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. Comparison of AI- and human-generated informed consent form performance: mean Readability, Understandability, and Actionability of Key Information scores with CIs and Wilcoxon signed rank test P values for readability, understandability, actionability, and Flesch-Kincaid grade level. AI: artificial intelligence.
Figure 1. Comparison of AI- and human-generated informed consent form performance: mean Readability, Understandability, and Actionability of Key Information scores with CIs and Wilcoxon signed rank test P values for readability, understandability, actionability, and Flesch-Kincaid grade level. AI: artificial intelligence.CC BY · PubMed Central

Verification

Claims the paper makes

9 Valid · 0 Not valid · 6 Not checked · 56 Own results and interpretations

22 more claims

Create a free account to see all 71 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 30
  • Probably not a substantial claim. 21
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 21 references

Cited by

Metadata

Authors
Shi, Q., Luzuriaga, K., Allison, J. J., Oztekin, A., Faro, J. M., Lee, J. L., Hafer, N., McManus, M., & Zai, A. H.
Affiliations
  • University of Massachusetts Chan Medical School (US)
  • University of Massachusetts Lowell (US)
Corresponding author
Shi, Qiming
Venue
JMIR Medical Informatics
Published
2025-02-13
Volume
13
Pages
e68139-e68139
ISSN
2291-9694 (ISSN-L)
OpenAlex
W4406055608
Cited by
23
References
21
Citations by year
2026: 15 · 2025: 8
FWCI
20.78
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Informatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health (UL1TR001453)
  • National Center for Advancing Translational Sciences
Topics
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
informed consent forms, large language models, clinical trial documentation, readability, comprehensibility, actionability, participant comprehension, research ethics committees, Flesch-Kincaid Grade Level, participant engagement
MeSH terms
Large Language Models, Clinical Trials as Topic, Humans, Informed Consent, Comprehension, Consent Forms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.