The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index; 11 of the 39 sources its claims cite were checked for support.
Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently
Open accessAbstract
Background: Recent advancements of Artificial Intelligence (AI) are rapidly transforming clinical research. While this technology offers exciting opportunities, it amplifies existing concerns regarding the need for transparent methodology that fosters patient engagement, and introduces new challenges. PCORI's Improving Methods portfolio has invested in methodological research to enhance rigor and transparency via patient-centered approaches in AI. Objective: This commentary outlines PCORI's approach to funding and promoting a portfolio of methodological research that aims to improve the conduct of patient-centered comparative clinical effectiveness research (CER), with a focus on AI methods. The paper highlights a growing portfolio of over 40 AI related projects, including a recent cohort leveraging large language models to augment research processes in CER. Discussion: PCORI's current portfolio of methods projects in AI illustrate timely opportunities for the clinical research informatics community to develop and assess AI applications that will further advance a robust, interoperable and ethical infrastructure for patient-centered CER. PCORI's requirement for ongoing, meaningful engagement of patients throughout the research lifecycle provides a blueprint for patient-centered AI by developing and applying models and methods designed to create value for patients and other healthcare partners.
Overview
CoChat Summary
TL;DR
PCORI’s commentary argues that patient-centered, transparent AI methods are essential for comparative clinical effectiveness research (CER) and describes how its methods portfolio is funding this work. As of spring 2025, PCORI reports funding more than 40 AI/ML methods projects, including 6 led by informaticians, plus 15 supplements funded in spring 2024 focused on AI and large language models (LLMs).
Key findings
- PCORI says its methods portfolio includes more than 40 AI/ML research projects as of spring 2025, including 6 projects led by informaticians.
- In spring 2024, PCORI funded supplements to fifteen projects aimed at advancing methodological research using AI, with emphasis on LLMs in patient-centered CER.
- The paper identifies several methodological applications already being funded: extracting information from unstructured data, developing computable phenotypes, identifying covariates, disease activity, mental health outcomes, non-medical drivers of health, and household/family relationships.
- The commentary highlights examples of rigorous evaluation in funded work, including assessment of bias, prediction errors, hallucinations, and alignment with human/expert responses.
Methods
- Commentary/perspective paper describing PCORI’s approach to funding and promoting methods research for patient-centered CER, with a focus on AI and LLMs.
- Uses examples from PCORI-funded projects and portfolio summaries; no primary study sample or dataset is reported.
- Main analysis is a narrative synthesis of PCORI’s portfolio, the methodological challenges of patient-centered AI, and areas for future exploration.
Limitations
- The paper does not report a primary empirical study, so no original effect estimates, sample sizes, or comparative outcomes are provided.
- The text itself makes evident that the discussion is portfolio-based and descriptive, not a systematic review or formal evaluation of the funded projects’ results.
- The paper notes that comprehensive evaluations employing a patient-centered approach remain limited in the broader literature.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
7 Valid · 4 Not valid · 11 Not checked · 9 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This issue of transparency/interpretability is even more pronounced with some of the powerful generative models, where technical details, model weights, and datasets are proprietary and undisclosed.6”
§ Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently > The need for transparent methods to support patient-centered AI
What the sources say
- Could not verifycites [6] Barrington, 2023, A bibliometric analysis of the rise of ChatGPT in medical research, Med Sci (Basel). sourceExperimental — accuracy not yet measured
The Discussion addresses transparency about where and how AI was used in a publication, not the interpretability of generative models. Although its Limitations section notes that LLaMA is open source, our copy does not say that other models’ technical details, weights, or datasets are proprietary and undisclosed. Searches for those terms and for model opacity found no supporting passage. (LLM, conf=0.73)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It is well understood that algorithms trained on biased datasets can lead to incorrect research conclusions that may disproportionately affect certain populations.5”
§ Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently > The need for transparent methods to support patient-centered AI
What the sources say
- Partly supportedcites [5] Jain, 2023 sourceExperimental — accuracy not yet measured
The Introduction says algorithms developed from biased data sets can result in care disparities for racial and ethnic minority groups. Results, Theme 6, adds that unrepresentative data can produce biased predictions. The paper supports the risk of disproportionate harm, but these passages discuss algorithm predictions and care, not incorrect research conclusions. (LLM, conf=0.84)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“While these models offer powerful foundational capabilities, adapting them to specific tasks and datasets without losing robustness is a significant hurdle, often resulting in significant variability in accuracy across different settings.7,9,10”
§ Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently > The need for transparent methods to support patient-centered AI
What the sources say
- Not checked — no full text of the sourcecites [7] Lee, 2023, Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine, N Engl J Med source
- Partly supportedcites [9] Hager, 2024, Evaluation and mitigation of the limitations of large language models in clinical decision-making, Nat Med sourceExperimental — accuracy not yet measured
The Results ('Current LLMs require extensive clinician supervision'; page not provided) report large changes in diagnostic accuracy when instructions change, supporting the robustness and accuracy-variability concerns. The paper does not show that adapting models to specific tasks or datasets causes this variability across different settings; its Discussion treats task-specific fine-tuning as a possibility for future work. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [10] Suchman, 2023, Chat generative pretrained transformer fails the multiple-choice American college of gastroenterology self-assessment test, Am J Gastroenterol source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As 1 example, Wu et al.25 developed and disseminated clinical LLMs such as GatorTron27 and GatorTronGPT28 to address limitations of traditional rule-based machine learning methods, which often lack scalability and transportability across data networks.”
§ Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently > PCORI’s contributions to improving use of AI methods in CER
What the sources say
- Not matchedcites [25]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [27] Yang, 2022, A large language model for electronic health records, NPJ Digit Med sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Partly supportedcites [28] Peng, 2023, A study of generative large language model for medical research and healthcare, NPJ Digit Med sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, in the spring of 2024, PCORI funded supplements to fifteen projects that aim to advance methodological research using AI, with a particular emphasis on LLMs in patient-centered CER.”
§ Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently > PCORI’s contributions to improving use of AI methods in CER
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
7 more claims
Create a free account to see all 31 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 12
- It may state two separate claims as one. 9
Citations
Cites
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · cited 1×
“26 As 1 example, Wu et al. 25 developed and disseminated clinical LLMs such as GatorTron 27 and GatorTronGPT 28 to address limitations of traditional rule-based machine learning methods, which often lack scalability and transportability across data networks.”
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408 · cited 1×
“26 As 1 example, Wu et al. 25 developed and disseminated clinical LLMs such as GatorTron 27 and GatorTronGPT 28 to address limitations of traditional rule-based machine learning methods, which often lack scalability and transportability across data networks.”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293
Are Emergent Abilities of Large Language Models a Mirage?
Schaeffer, R., Miranda, B., & Koyejo, S. (2023) · arXiv · doi:10.48550/arxiv.2304.15004 · Cited by 200
(2023) · doi:10.52202/075280-2425 · Cited by 200
Assessing the research landscape and clinical utility of large language models: a scoping review
Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02459-6 · Cited by 187 · cited 1×
“11,12 This gap in evidence underscores the need to augment existing CRI priorities with an emphasis on designing and assessing patient-centered uses of AI applications that strengthen the existing clinical research infrastructure for patient-centered CER.”
Clinical research informatics: challenges, opportunities and definition for an emerging domain
Embi, P. J., & Payne, P. R. O. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3005 · Cited by 163
Cited by
Yano, E. M. (2026) · Journal of Women s Health · doi:10.1177/15409996261489073
Metadata
- Authors
- Ou, J., & Holve, E.
- Affiliations
- Patient-Centered Outcomes Research Institute (US)
- Corresponding author
- Ou, Jinghua
- ORCID
- Holve, Erin
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-07-03
- Volume
- 8
- Issue
- 4
- Pages
- ooaf081-ooaf081
- ISSN
- 2574-2531 (ISSN-L)
- PMC
- PMC12296393
- OpenAlex
- W4412660375
- Cited by
- 1
- References
- 14
- Citations by year
- 2026: 1
- FWCI
- 0.06
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JAMIA Open (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Mental Health and Patient Involvement
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- artificial intelligence in clinical research, patient-centered outcomes research, large language models, research methodology development, patient engagement, research transparency, research informatics, AI ethics, interoperability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently. https://app.cochat.ai/papers/10.1093/jamiaopen/ooaf081
- The paper
- Ou, J., & Holve, E. (2025). Advancing methodological development of artificial intelligence in patient-centered comparative clinical effectiveness research: Patient-Centered Outcomes Research Institute’s unique contribution to research done differently. JAMIA Open. https://doi.org/10.1093/jamiaopen/ooaf081
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.