Leveraging AI Large Language Models for Writing Clinical Trial Proposals in Dermatology: Instrument Validation Study

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AI is limited by the data it is trained on and an inability to account for the nuanced contexts of individual research studies.

“However, AI is limited by the data it is trained on and an inability to account for the nuanced contexts of individual research studies [2].”

• cites [2] Askin, Artificial intelligence applied to clinical trials: opportunities and challenges, Health Technol (Berl) sourcePartly supported

  The Results and Discussion > Design sections identify incomplete or poor-quality datasets as limits on AI applications, supporting the data-dependent part of the claim. The paper also discusses model interpretation and validation, but does not say AI is unable to account for the nuanced contexts of individual research studies. Page numbers were not available in our copy. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S2 (checkable).

LLMs offer a transformative approach to drafting research proposals.

“LLMs offer a transformative approach to drafting research proposals [10].”

• cites [10] Markey, From RAGs to riches: utilizing large language models to write documents for clinical trials, Clin Trials sourcePartly supported

  The Discussion (page unavailable) reports strong potential for LLM-assisted clinical trial writing and substantial reductions in drafting time. The study assessed sections of clinical trial protocols, not research proposals generally, so the claim extends beyond the documents examined. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

ChatGPT-o1 and ChatGPT-o3-mini demonstrated the highest overall accuracy and comprehensiveness, delivering well-structured proposals with robust citations and high scientific rigor.

“ChatGPT-o1 and ChatGPT-o3-mini demonstrated the highest overall accuracy and comprehensiveness, delivering well-structured proposals with robust citations and high scientific rigor.”

May combine two claims.

Current LLMs were largely unable to score proposals and should not replace human review for quality control.

“Additionally, current LLMs were largely unable to score proposals and should not replace human review for quality control.”

May combine two claims.

All open access LLMs were highly efficient, generating proposals in seconds to minutes, with a minimum of 20 seconds for Llama 3.1 and a maximum of 1 minute and 37 seconds for ChatGPT-4o.

“All open access LLMs were highly efficient and ran in a matter of seconds to minutes (minimum of 20 seconds for Llama 3.1 and maximum of 1 minute and 37 seconds for ChatGPT-4o).”

• Also stated in: § Principal Findings · plain

  “LLMs take only minutes to generate proposals, whereas prior investigations into time commitment for generation of proposals by humans have reported estimates of 116 principal investigator hours, 55 coinvestigator hours, and 38 working days [56].”

May combine two claims.

The ChatGPT-o1 and DeepSeek proposals both received top scores of 5 for both accuracy and comprehensiveness from all LLMs, versus human averages of 4.3 (SD 2.2) and 3.3 (SD 1.9), respectively.

“Interestingly, the ChatGPT-o1 and DeepSeek proposals both received top scores of 5 for both accuracy and comprehensiveness from all LLMs versus human averages of 4.3 (SD 2.2) and 3.3 (SD 1.9), respectively.”

May combine two claims.

Human scorers rated ChatGPT-o1 as the most accurate and Llama 3.1 as the least accurate.

“Human scorers rated ChatGPT-o1 as the most accurate and Llama 3.1 as the least accurate.”

May combine two claims.

Llama 3.1 and Gemini Advanced were notably efficient, reliably delivering full proposals with rapid turnaround times, but occasionally produced less nuanced sections in preliminary data or limited discussion.

“Llama 3.1 and Gemini Advanced were notably efficient, reliably delivering full proposals with rapid turnaround times, but occasionally produced less nuanced sections in preliminary data or limited discussion.”

May combine two claims.

ChatGPT-4o and Claude Opus offered the most intuitive interfaces and were highly usable for researchers without computer science backgrounds, requiring minimal learning curves.

“When assessed for ease of use, ChatGPT-4o and Claude Opus offered the most intuitive interfaces and were highly usable for researchers (DC and MH) without computer science backgrounds.”

• Also stated in: § Principal Findings · plain

  “Regarding ease of use, ChatGPT-4o and Claude Opus feature intuitive interfaces and require minimal learning curves, making them ideal for researchers new to AI-powered tools.”

May combine two claims.

ChatGPT-o1 and Llama 3.1 were found to be the most and least comprehensive, respectively, by both human and LLM scorers.

“ChatGPT-o1 and Llama 3.1 were found to be the most and least comprehensive, respectively, by both human and LLM scorers.”

May combine two claims.

Other than Gemini Advanced, Claude Sonnet, and Llama 3.1, all the LLMs self-scored their own proposals with 5 for both accuracy and comprehensiveness, suggesting overlapping blind spots in LLM proposal generation and evaluation.

“Other than Gemini Advanced (who self-scored its written proposal with 3 for accuracy and comprehensiveness), Claude Sonnet, and Llama 3.1, all the LLMs self-scored their own proposals with 5 for both accuracy and comprehensiveness, suggesting overlapping “blind spots” in LLM proposal generation and evaluation.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

LLMs can assist in initial brainstorming of a clear researchable question and generating hypotheses based on existing literature.

“They can assist in initial brainstorming of a clear researchable question and generating hypotheses based on existing literature.”

Classified as probably not a substantial claim.

All references in the proposals were checked for verifiability, relevance, and reputable source quality.

“All references were checked for verifiability, relevance, and reputable source quality.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

All open access LLMs can propose data collection methods, define eligibility criteria based on study objectives, recommend appropriate statistical tests based on study design, and help draft proposal sections.

“Furthermore, all open access LLMs can propose data collection methods, define eligibility criteria based on study objectives, recommend appropriate statistical tests based on study design, and help draft proposal sections.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Rapid and frequent versioning of LLM platforms poses challenges for scientific reproducibility.

“Another important limitation is the rapid and frequent versioning of LLM platforms, which poses challenges for scientific reproducibility.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The use of LLMs in research proposal development is largely unexplored and perhaps underused.

“In contrast, the use of LLMs in research proposal development is largely unexplored, and thus, they are perhaps underused.”

Classified as probably not a substantial claim.

Each of the 11 blinded proposals (10 LLM-generated and 1 human-written) was systematically reviewed and scored by 6 independent physician evaluators with strong research backgrounds.

“To assess the outputs, each of the 11 blinded proposals (n=10, 90.9% LLM generated and n=1, 9.1% human written) was systematically reviewed and scored by 6 independent physician evaluators, all with strong research backgrounds.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Users must verify citations as the model may hallucinate or fabricate realistic-sounding but false information.

“Another limitation is that users must verify citations as the model may “hallucinate” or fabricate realistic-sounding but false information.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs allow for iterative refinements, enabling tailored outputs to meet specific requirements or needs.

“They also allow for iterative refinements, enabling tailored outputs to meet specific requirements or needs.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

These tools cannot entirely replace expert input but serve as powerful assistants, streamlining the development process and enhancing productivity.

“While these tools cannot entirely replace expert input, they serve as powerful assistants, streamlining the development process and enhancing productivity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Researchers are increasingly using LLMs in clinical trial design to improve patient selection, cohort composition, and recruitment.

“Researchers are increasingly using LLMs in clinical trial design to improve patient selection, cohort composition, and recruitment [3].”

• cites [3] Harrer, Artificial intelligence for clinical trial design, Trends Pharmacol Sci sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

LLMs are useful in literature review and can summarize existing studies related to the proposal topic and identify gaps in current knowledge.

“LLMs are useful in literature review and can summarize existing studies related to the proposal topic and identify gaps in current knowledge.”

Classified as probably not a substantial claim.

Each evaluator used a standardized Likert scale from 1 to 5 to rate each proposal for accuracy and comprehensiveness.

“Each evaluator used a standardized Likert scale ranging from 1 to 5 (1=“strongly disagree”; 5=“strongly agree”) to rate each proposal for capabilities and limitations in the LLMs’ accuracy and comprehensiveness (Table 1).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The gold standard human proposal was last, and question order likely played a role, with kinder grading of the LLM-derived proposals before reviewing the human-written proposal.

“Additionally, the “gold standard” (human proposal) was last, and question order likely played a role, with kinder grading of the LLM-derived proposals before reviewing the human-written proposal.”

Probably not a substantial claim: may not pass S4 (based).

All open access LLMs can aid in initial outlining and creation of research proposals.

“All open access LLMs can aid in initial outlining and creation of research proposals.”

Probably not a substantial claim: may not pass S2 (checkable).

LLMs offer powerful tools to assist humans in clinical trial proposal creation.

“LLMs offer powerful tools to assist humans in clinical trial proposal creation.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

While human verification is always required, LLMs can greatly improve time spent on initial proposal drafting and aid in mundane tasks associated with proposal writing, including proofreading and revisions, writing administrative sections, and optimizing citations.

“While human verification is always required, LLMs can greatly improve time spent on initial proposal drafting and aid in mundane tasks associated with proposal writing, including proofreading and revisions, writing administrative sections, and optimizing citations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

As models are updated, their performance and outputs can meaningfully change over time, making it difficult to reproduce results or maintain consistency in studies that rely on AI-generated content.

“As models are updated, their performance and outputs can meaningfully change over time, making it difficult to reproduce results or maintain consistency in studies that rely on AI-generated content.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Due to the training data cutoffs of AI models, researchers must manually incorporate the latest literature findings.

“Due to the training data cutoffs of AI models, researchers must manually incorporate the latest literature findings.”

Classified as probably not a substantial claim. May combine two claims.

Raters systematically fact-checked all proposal content, and only proposals with fully correct and verified factual information were rated highly.

“Raters systematically fact-checked all proposal content. Only proposals with fully correct and verified factual information (including cited data, statistics, and conclusions) were rated highly.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Had the human proposal been first, this would have highlighted missing components of LLM-derived proposals and likely led to harsher human grading of the latter.

“Had the human proposal been first, this would have highlighted missing components of LLM-derived proposals and likely led to harsher human grading of the latter.”