The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; 11 of the 22 sources its claims cite were checked for support.
Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study
Open accessAbstract
Background: Informed consent forms (ICFs) for clinical trials have become increasingly complex, often hindering participant comprehension and engagement due to legal jargon and lengthy content. The recent advances in large language models (LLMs) present an opportunity to streamline the ICF creation process while improving readability, understandability, and actionability. objectives: This study aims to evaluate the performance of the Mistral 8x22B LLM in generating ICFs with improved readability, understandability, and actionability. Specifically, we evaluate the model's effectiveness in generating ICFs that are readable, understandable, and actionable while maintaining the accuracy and completeness. Methods: We processed 4 clinical trial protocols from the institutional review board of UMass Chan Medical School using the Mistral 8x22B model to generate key information sections of ICFs. A multidisciplinary team of 8 evaluators, including clinical researchers and health informaticians, assessed the generated ICFs against human-generated counterparts for completeness, accuracy, readability, understandability, and actionability. Readability, Understandability, and Actionability of Key Information indicators, which include 18 binary-scored items, were used to evaluate these aspects, with higher scores indicating greater accessibility, comprehensibility, and actionability of the information. Statistical analysis, including Wilcoxon rank sum tests and intraclass correlation coefficient calculations, was used to compare outputs. Results: LLM-generated ICFs demonstrated comparable performance to human-generated versions across key sections, with no significant differences in accuracy and completeness (P>.10). The LLM outperformed human-generated ICFs in readability (Readability, Understandability, and Actionability of Key Information score of 76.39% vs 66.67%; Flesch-Kincaid grade level of 7.95 vs 8.38) and understandability (90.63% vs 67.19%; P=.02). The LLM-generated content achieved a perfect score in actionability compared with the human-generated version (100% vs 0%; P<.001). Intraclass correlation coefficient for evaluator consistency was high at 0.83 (95% CI 0.64-1.03), indicating good reliability across assessments. Conclusions: The Mistral 8x22B LLM showed promising capabilities in enhancing the readability, understandability, and actionability of ICFs without sacrificing accuracy or completeness. LLMs present a scalable, efficient solution for ICF generation, potentially enhancing participant comprehension and consent in clinical trials.
Overview
CoChat Summary
TL;DR
This mixed methods study evaluated Mistral 8x22B for generating key information sections of clinical trial informed consent forms (ICFs) from 4 research protocols. LLM-generated ICFs were comparable to human-generated ICFs in accuracy and completeness but scored higher in readability, understandability, and actionability.
Key findings
- Accuracy and completeness were comparable across key sections, with no significant differences (P>.10); for example, benefits scored 3.0 for LLM vs 2.57 for human (P=.10), and alternatives were identical at 2.75 vs 2.75 (P≥.99).
- LLM-generated ICFs outperformed human versions in understandability (90.63% vs 67.19%; P=.02) and actionability (100% vs 0%; P<.001), and had a higher combined RUAKI score (84.03% vs 61.82%; P=.008).
- Readability favored the LLM but was not statistically significant: RUAKI readability 76.39% vs 66.67% (P=.26) and Flesch-Kincaid grade level 7.95 vs 8.38 (P=.77).
- Evaluator consistency was good, with an intraclass correlation coefficient of 0.83 (95% CI 0.64-1.03).
Methods
- Four clinical trial protocols from the UMass Chan Medical School IRB were processed with Mistral 8x22B to generate key information sections, yielding 8 ICFs total: 4 human-generated and 4 AI-generated.
- Eight multidisciplinary evaluators, including clinical researchers, health informaticians, and physicians, assessed outputs; each protocol set was reviewed by 2 different evaluators.
- Accuracy/completeness used a 3/2/1/0 scoring system for complete, incomplete, absent, or incorrect; readability, understandability, and actionability used the 18-item binary RUAKI indicator; analyses included Wilcoxon rank sum tests and intraclass correlation coefficient calculations.
Limitations
- The paper states the small sample of only 4 clinical trial protocols limits generalizability and may have contributed to nonsignificant results; LLM performance is closely tied to the clarity and completeness of the source protocol; LLM text style may have been recognizable despite blinding; and the study evaluated a single LLM model.
- The paper also notes that the 0% actionability score for human-generated ICFs reflects a structural template issue rather than a methodological flaw, and that prompt development required moderate initial time investment.
- Evident from the text: all evaluators were also coauthors of the work, and no external participants were involved, which the paper frames as nonhuman subjects research but may still warrant consideration for evaluation independence.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
9 Valid · 0 Not valid · 6 Not checked · 56 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although many institutional review boards (IRBs) require investigators to develop documents written at the eighth-grade reading level [5]”
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Introduction
What the sources say
- Partly supportedcites [5] Hadden, Improving readability of informed consents for research at an academic medical institution, J Clin Trans Sci sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says consent forms are recommended to be written at a sixth- to eighth-grade reading level or lower. It does not say that many IRBs require investigators to write them at the eighth-grade level; the Materials and Methods describe this institution’s template as optional. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Studies have also shown that LLMs can effectively enhance the documentation of risks, benefits, and alternatives for common surgical procedures [17].”
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Introduction
What the sources say
- Supportedcites [17] Decker, Large language model-based chatbot vs surgeon-generated informed consent documentation for common procedures, JAMA Netw Open source
The Discussion reports that chatbot-generated risks, benefits, and alternatives were more complete and accurate overall than surgeon-generated versions for the common procedures studied. The improvement was driven mainly by benefits and alternatives; descriptions of surgical risks did not differ significantly. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To evaluate accuracy and completeness, we developed a scoring system based on recommendations from LeapFrog (VTech Group), The Joint Commission, the American College of Surgeons, and relevant available literature (Multimedia Appendix 2) [21-24].”
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Methods > Measurements of Accuracy and Completeness
What the sources say
- Not matchedcites [21] Quick Safety 21: informed consent: more than getting a signature. The Joint Commission. 2023. URL: https://www.jointcommission.org/resources/news-and-multimedia/newsletters/newsletters/quick-safety/quick-safety--issue-21-informed--consent-more-than-getting-a-signature/informed-consent-more-than-getting-a-signature/ [Accessed 31-01-2025]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [22] Informed consent. Leapfrog Ratings. URL: https://ratings.leapfroggroup.org/measure/hospital/2023/informed-consent [Accessed 31-01-2025]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [23] Samaan, Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery, Obes Surg sourceExperimental — accuracy not yet measured
The Methods > Question Grading section describes a scale that distinguishes accurate, comprehensive answers from correct but incomplete ones. It does not say the scale was based on recommendations from LeapFrog, The Joint Commission, or the American College of Surgeons, or on the cited literature. No page number is available in our copy. (LLM, conf=0.84)
- Not matchedcites [24] Informed consent. American College of Surgeons. Jul2023. URL: https://www.facs.org/for-patients/patient-resources/informed-consent [Accessed 22-01-2025]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“By adopting a Least-to-Most approach [30]—where each prompt contains a focused set of instructions and builds on the previous output—we achieved more consistent and reliable results.”
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Discussion > Lessons Learned
What the sources say
- Supportedcites [30] Zhou D Schärli N Hou L et al. Least-to-most prompting enables complex reasoning in large language models. arXiv. Preprint posted online onMay21, 2022. [doi: 10.48550/arXiv.2205.10625] source
Section 2 (p. 3) describes least-to-most prompting as solving subproblems sequentially and appending each generated answer to the next prompt. The citing authors’ statement that they achieved more consistent and reliable results describes their own outcome, not a result attributed to this source. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLMs demonstrated potential advantages in producing more readable, understandable, and actionable ICF content.”
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Discussion > Conclusions
boostedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
22 more claims
Create a free account to see all 71 claims and to verify the 3 references still open.
- It may state two separate claims as one. 30
- Probably not a substantial claim. 21
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Methods > Statistical Analysis
“Furthermore, we compared the RUAKI indicators between the 2 groups using the Wilcoxon rank sum test.”
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Results
“Figure: Figure 1. Comparison of AI- and human-generated informed consent form performance: mean Readability, Understandability, and Actionability of Key Information scores with CIs and Wilcoxon signed rank test P values for readability, understandability, actionability, and Flesch-Kincaid grade level.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Discussion > Principal Results
“While the human-generated content slightly outperformed the LLM in discussing alternatives, the LLM performed better in explaining the benefits.”
§ Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study > Results
“| Alternatives | 2.75 (0.46) | 2.75 (0.46) | ≥.99 |”
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709 · cited 1×
“An ICC below 0.5 indicates low reliability, between 0.5 and 0.74 indicates moderate reliability, from 0.75 to 0.9 suggests good reliability, and above 0.9 signifies excellent reliability [26].”
InfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380 · cited 1×
“By adopting a Least-to-Most approach [30]—where each prompt contains a focused set of instructions and builds on the previous output—we achieved more consistent and reliable results.”
Improving large language models for clinical named entity recognition via prompt engineering
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad259 · Cited by 314 · cited 1×
“LLMs show significant potential in health informatics, including tasks such as name entity extraction [11], patient trial matching [12,13], biomedical reasoning and classification [14], prediction of admissions [15], automation of administrative tasks [16], and so forth.”
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282
Patients’ recollection and understanding of informed consent: a literature review
Sherlock, A., & Brownie, S. (2014) · ANZ Journal of Surgery · doi:10.1111/ans.12555 · Cited by 191
Decker, H., Trang, K., Ramirez, J., Colley, A., Pierce, L., Coleman, M., Bongiovanni, T., Melton, G. B., & Wick, E. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36997 · Cited by 178 · cited 1×
“Studies have also shown that LLMs can effectively enhance the documentation of risks, benefits, and alternatives for common surgical procedures [17].”
Tripathi, S., Sukumaran, R., & Cook, T. S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad258 · Cited by 168 · cited 1×
“LLMs show significant potential in health informatics, including tasks such as name entity extraction [11], patient trial matching [12,13], biomedical reasoning and classification [14], prediction of admissions [15], automation of administrative tasks [16], and so forth.”
Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening
Ünlü, O., Shin, J., Mailly, C. J., Oates, M., Tucci, M. R., Varugheese, M., Wagholikar, K. B., Wang, F., Scirica, B. M., Blood, A. J., et al. (2024) · NEJM AI · doi:10.1056/aioa2400181 · Cited by 152
Cited by
Artificial intelligence across the cancer care continuum
Riaz, I. B., Khan, M. A., & Osterman, T. J. (2025) · Cancer · doi:10.1002/cncr.70050 · Cited by 40
Wang, X., Wang, Q., Ding, G., Wang, J., Tang, Y., & Feng, Y. (2025) · iScience · doi:10.1016/j.isci.2025.114082 · Cited by 25
“43 SNOMED CT expressions with rule-based logic achieved TNM classification accuracies of ∼ 72% (T), 78% (N), and 94% (M) against expert reference labels in a cohort of 718 patients.”
Scaling medical AI across clinical contexts
Li, M. M., Reis, B. Y., Rodman, A. M., Cai, T., Dagan, N., Balicer, R. D., Loscalzo, J., Kohane, I. S., & Žitnik, M. (2026) · Nature Medicine · doi:10.1038/s41591-025-04184-7 · Cited by 31
Compliance and factuality of large language models for clinical research document generation
Wang, Z., Gao, J., Danek, B. P., Theodorou, B., Shaik, R., Thati, S., Won, S., & Sun, J. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf174 · Cited by 10
Evaluating GPT-4’s ability to generate informed consent material for genetic testing
Petrou, E., Ormond, K. E., Stammbach, D., Ash, E., Buchholz, O., & Vayena, E. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00036-4 · Cited by 6
Metadata
- Authors
- Shi, Q., Luzuriaga, K., Allison, J. J., Oztekin, A., Faro, J. M., Lee, J. L., Hafer, N., McManus, M., & Zai, A. H.
- Affiliations
- University of Massachusetts Chan Medical School (US)
- University of Massachusetts Lowell (US)
- Corresponding author
- Shi, Qiming
- ORCID
- Venue
- JMIR Medical Informatics
- Publisher
- JMIR Publications Inc.
- Published
- 2025-02-13
- Volume
- 13
- Pages
- e68139-e68139
- ISSN
- 2291-9694 (ISSN-L)
- PMC
- PMC11841745
- OpenAlex
- W4406055608
- Semantic Scholar
- d464c898ca35eeba62ad4acc6b851e50d6bcb41d
- Cited by
- 23
- References
- 21
- Citations by year
- 2026: 15 · 2025: 8
- FWCI
- 20.78
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Informatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (UL1TR001453)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- informed consent forms, large language models, clinical trial documentation, readability, comprehensibility, actionability, participant comprehension, research ethics committees, Flesch-Kincaid Grade Level, participant engagement
- MeSH terms
- Large Language Models, Clinical Trials as Topic, Humans, Informed Consent, Comprehension, Consent Forms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study. https://app.cochat.ai/papers/10.2196/68139
- The paper
- Shi, Q., Luzuriaga, K., Allison, J. J., Oztekin, A., Faro, J. M., Lee, J. L., Hafer, N., McManus, M., & Zai, A. H. (2025). Transforming Informed Consent Generation Using Large Language Models: Mixed Methods Study. JMIR Medical Informatics. https://doi.org/10.2196/68139
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.