CoChat
Sign Up
Cited by 7Artificial Intelligence in Healthcare and EducationArtificial Intelligence ApplicationsEthics and Social Impacts of AI

Integrating large language models in biostatistical workflows for clinical and translational research

Open access
Grambow, Steven C., Desai, Manisha, Weinfurt, Kevin P., Lindsell, Christopher J., Pencina, Michael J., Rende, Lacey, Pomann, Gina-Maria·2025·Journal of Clinical and Translational Science
DOI 10.1017/cts.2025.10064PMC PMC12260977
Open this paper in CoChat — free accountOpen PDF

Abstract

Introduction: Biostatisticians increasingly use large language models (LLMs) to enhance efficiency, yet practical guidance on responsible integration is limited. This study explores current LLM usage, challenges, and training needs to support biostatisticians. Methods: A cross-sectional survey was conducted across three biostatistics units at two academic medical centers. The survey assessed LLM usage across three key professional activities: communication and leadership, clinical and domain knowledge, and quantitative expertise. Responses were analyzed using descriptive statistics, while free-text responses underwent thematic analysis. Results: Of 208 eligible biostatisticians (162 staff and 46 faculty), 69 (33.2%) responded. Among them, 44 (63.8%) reported using LLMs; of the 43 who answered the frequency question, 20 (46.5%) used them daily and 16 (37.2%) weekly. LLMs improved productivity in coding, writing, and literature review; however, 29 of 41 respondents (70.7%) reported significant errors, including incorrect code, statistical misinterpretations, and hallucinated functions. Key verification strategies included expertise, external validation, debugging, and manual inspection. Among 58 respondents providing training feedback, 44 (75.9%) requested case studies, 40 (69.0%) sought interactive tutorials, and 37 (63.8%) desired structured training. Conclusions: LLM usage is notable among respondents at two academic medical centers, though response patterns likely reflect early adopters. While LLMs enhance productivity, challenges like errors and reliability concerns highlight the need for verification strategies and systematic validation. The strong interest in training underscores the need for structured guidance. As an initial step, we propose eight core principles for responsible LLM integration, offering a preliminary framework for structured usage, validation, and ethical considerations.

AI snapshot

Written from the paper’s full text.

TL;DR

This cross-sectional survey of biostatisticians at two academic medical centers found substantial LLM use, especially for coding, writing, and communication, alongside frequent concerns about errors and reliability. The authors propose eight principles for responsible LLM integration and highlight strong demand for structured training and verification guidance.

Key findings

  • Among 208 eligible biostatisticians, 69 (33.2%) responded; 44/69 (63.8%) reported using LLMs, and among 43 who answered the frequency question, 20 (46.5%) used them daily and 16 (37.2%) weekly.
  • LLM use was highest for “writing, debugging, or documenting code” (31/40, 77.5%), followed by “editing and improving writing quality” (29/38, 76.3%) and “composing emails or other messages” (27/38, 71.1%).
  • Among 41 users answering about errors, 29 (70.7%) encountered incorrect answers that could have had significant consequences, including incorrect code, statistical misinterpretation, content fabrication, and inappropriate tone.
  • Among 58 respondents providing training feedback, 44 (75.9%) requested case studies, 40 (69.0%) interactive tutorials, and 37 (63.8%) structured training; the paper proposes eight core principles for responsible LLM use.

Methods

  • Cross-sectional survey conducted in September and October 2024 across three biostatistics units at Duke University School of Medicine and Stanford University School of Medicine; eligible population was 208 biostatisticians (162 staff, 46 faculty), administered via Qualtrics.
  • Survey assessed LLM use across three core competency domains: communication and leadership, clinical and domain knowledge, and quantitative expertise, with branching logic for non-users.
  • Quantitative analysis used frequency tables with counts and percentages in R/RStudio; free-text responses underwent thematic analysis combining human coding with ChatGPT-4o and Claude 3.5 Sonnet, plus word frequency analysis using R’s tidytext package.

Limitations

  • The paper states a low response rate of 33.2% raises response bias concerns, likely reflecting early adopters; the sample from two academic medical centers with established biostatistics groups may not generalize to all biostatistical practice settings.
  • The paper states reliance on self-reported interactions rather than direct analysis of LLM use, no systematic analysis of how different LLM models responded to the same statistical queries, and a focus on text-based LLMs rather than broader generative AI.
  • The paper notes rapid LLM advancements during the study and manuscript preparation period (September 2024–February 2025) may limit applicability; the table also notes the organizational support question was implemented as single-select though intended as multiple-select, which may undercount reported support.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1017/cts.2025.10064

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

21 of 31 cited sources checked · 11 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Data scientists follow similar interaction patterns to biostatisticians.

    “While data scientists follow similar interaction patterns, Chopra et al. [25] found that practitioners spend significant time on prompt preparation and validation”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [25] Chopra B, Singha A, Fariha A, et al. Conversational challenges in AI-powered data science: obstacles, needs, and design opportunities. 2023. doi: 10.48550/arXiv.2310.16164. source

      Section 4.1.2 (p. 8) reports that participants spent 64% of the measured time preparing prompts, but only the remaining 9%—24 seconds on average—validating code. Section 4.3.3 says validation took the least time of the activities measured, so describing time spent on both preparation and validation as significant overstates the validation finding. The comparison with biostatisticians is framing outside the claim attached to this citation. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    There is substantial variation in LLM performance when analyzing biomedical data.

    “Komandur et al. [28] found substantial variation in LLM performance when analyzing biomedical data”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [28] Komandur R, McDunn J, Nair N, et al. Artificial intelligence in biomedical data analysis: a comparative assessment of large language models for automated clinical trial interpretation and statistical evaluation. medRxiv. 2025; doi: 10.1101/2025.02.05.25321607. source
  • Premisecitation

    Domain-specialized systems achieved relevant, evidence-based answers in 24%-58% of clinical applications.

    “while domain-specialized systems achieved 24%–58%”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [26] Low YS, Jackson ML, Hyde RJ, et al. Answering real-world clinical questions using large language model-based systems. 2024. doi: 10.48550/arXiv.2407.00541. PMC12159471 40510193 source
  • Premisecitation

    General-purpose LLMs produced relevant, evidence-based answers in only 2%-10% of clinical applications.

    “Low et al. [26] found that general-purpose LLMs produced relevant, evidence-based answers in only 2%–10% of clinical applications”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [26] Low YS, Jackson ML, Hyde RJ, et al. Answering real-world clinical questions using large language model-based systems. 2024. doi: 10.48550/arXiv.2407.00541. PMC12159471 40510193 source

      The Relevance, Reliability, and Actionability section reports that general-purpose LLM responses met the relevant-and-evidence-based standard 2%–10% of the time. The study evaluated answers to 50 clinical questions; it does not establish that rate across clinical applications generally. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Findingown data

    Among the 43 respondents who answered the frequency question, 16 (37.2%) used LLMs weekly.

    “of the 43 who answered the frequency question, 20 (46.5%) used them daily and 16 (37.2%) weekly.”
    • Also stated in: § Survey participant characteristics and LLM usage · plain
      “Among those who answered the frequency question (43/69), 20 (46.5%) reported using them daily and 16 (37.2%) weekly.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 39
  • May combine two claims. 21
  • The quote only partly states this claim. 2

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    LLMs have capabilities in generating scientific text, guiding clinical trial design, optimizing code, synthesizing medical literature, and facilitating communication of technical findings.

    “These studies highlight LLMs’ capabilities in generating scientific text, guiding clinical trial design, optimizing code, synthesizing medical literature, and facilitating communication of technical findings [4–7].”
    Quote in the paperSubstantial claim1 of 4 sources checked
    • cites [4] Zhou H, Liu F, Gu B, et al. A survey of large language models in medicine: progress, application, and challenge. 2023. doi: 10.48550/arXiv.2311.05112. source

      Section 3.2 (page not provided) describes medical text generation and summarization, and section 4.6 describes adapting medical language for different audiences. Section 4.1 discusses processing medical literature and matching patients to clinical trials, but not guiding trial design. The passages do not establish scientific-text generation specifically or optimization of programming code; section 4.2’s “clinical coding” means assigning medical codes, not improving software. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [5] Thapa S, Adhikari S. ChatGPT, Bard, and large language models for biomedical research: opportunities and pitfalls. Ann Biomed Eng. 2023; 51(12): 2647– 2651. doi: 10.1007/s10439-023-03284-0. 37328703 source
    • cites [6] He K, Mao R, Lin Q, et al. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. 2023. doi: 10.48550/arXiv.2310.05694. source
    • cites [7] Hernandez AF, Lindsell CJ. The future of clinical trials: artificial to augmented to applied intelligence. JAMA. 2023; 330(21): 2061– 2063. doi: 10.1001/jama.2023.23822. 37950740 source

  • May combine two claims.

    Premisecitation

    LLMs can streamline research processes, enhance productivity, and allow experts to focus on higher-level analytical tasks.

    “Dell’Acqua et al. [1] and Peng et al. [2] demonstrated that these tools can streamline research processes, enhance productivity, and allow experts to focus on higher-level analytical tasks.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [1] Dell’Acqua F, McFowland E III, Mollick E, et al. Navigating the jagged technological frontier: field experimental evidence of the effects of AI on knowledge worker productivity and quality. Harvard Business School. 2023. (https://www.hbs.edu/faculty/Pages/item.aspx?num=64700) Working Paper 24-013. Accessed May 4, 2025. source
    • cites [2] Peng S, Kalliamvakou E, Cihon P, Demirer M. The impact of AI on developer productivity: evidence from GitHub copilot. 2023. doi: 10.48550/arXiv.2302.06590. source

      The Discussion reports that developers using GitHub Copilot completed a standardized programming task 55.8% faster, supporting the productivity claim. It does not establish that the tool streamlines research processes or frees experts for higher-level analytical tasks; the Discussion says how job tasks would change remains unclear. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    The framework established by Pomann et al. and validated by Slade et al. defines three core domains of biostatistical practice in collaborative research settings.

    “This study used a framework established by Pomann et al. [11] and recently validated by Slade et al. [12], defining three core domains of biostatistical practice in collaborative research settings.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [11] Pomann GM, Boulware LE, Cayetano SM, et al. Methods for training collaborative biostatisticians. J Clin Transl Sci. 2021; 5(1): e26. doi: 10.1017/cts.2020.518. PMC8057395 33948249 source
    • cites [12] Slade E, Brearley AM, Coles A, et al. Essential team science skills for biostatisticians on collaborative research teams. J Clin Transl Sci. 2023; 7(1): e243. doi: 10.1017/cts.2023.676. 38033706 PMC10685263 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Guidelines and practical strategies have been proposed to ensure ethical implementation and maintain scientific rigor.

    “Guidelines and practical strategies have been proposed to ensure ethical implementation and maintain scientific rigor [8–10].”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [8] Perlis RH, Fihn SD. Evaluating the application of large language models in clinical research contexts. JAMA Netw Open. 2023; 6(10): e2335924. doi: 10.1001/jamanetworkopen.2023.35924. 37782501 source
    • cites [9] Seckel E, Stephens BY, Rodriguez F. Ten simple rules to leverage large language models for getting grants. PLoS Comput Biol. 2024; 20(3): e1011863. doi: 10.1371/journal.pcbi.1011863. 38427611 PMC10906892 source
    • cites [10] Pencina MJ, Klotman ME. The crucial role of academic medical centers in preparing for the future of health AI. N C Med J. 2024; 85(4): 275– 277. doi: 10.18043/001c.120563. PMC13368198 39466091 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Integration of LLMs into data science workflows is complex, requiring navigation of challenges including potential errors and hallucinations, ethical considerations, and transparency.

    “However, their integration is complex, requiring careful navigation of multiple challenges: potential errors and hallucinations that necessitate robust verification strategies, ethical considerations around data privacy and bias mitigation, and ensuring transparency while balancing AI capabilities with human expertise [3].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [3] Cheong BC. Transparency and accountability in AI systems: safeguarding wellbeing in the age of algorithmic decision-making. Front Hum Dyn. 2024; 6: 1421273. doi: 10.3389/fhumd.2024.1421273. source

      The Introduction (p. 2) describes challenges in AI deployment involving bias, transparency and privacy; Data Protection and Privacy Laws (p. 6) discusses privacy risks and obstacles to transparency. The paper does not establish the more specific claim about integrating LLMs into data science workflows, hallucinations requiring verification, or balancing AI capabilities with human expertise. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    There is a lack of standardized validation frameworks and a challenge of ensuring reproducibility in LLM evaluation across healthcare applications.

    “emphasizing the lack of standardized validation frameworks and the challenge of ensuring reproducibility.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [29] Lee J, Park S, Shin J, Cho B. Analyzing evaluation methods for large language models in the medical field: a scoping review. BMC Med Inform Decis Mak. 2024; 24(1): 366. doi: 10.1186/s12911-024-02709-7. 39614219 PMC11606129 source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    LLM validation should be domain-specific.

    “Denecke et al. [27] emphasized that LLM validation should be domain-specific.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [27] Denecke K, May R, Rivera Romero O. Potential of large language models in health care: Delphi study. J Med Internet Res. 2024; 26: e52399. doi: 10.2196/52399. 38739445 PMC11130776 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Notable LLM releases during the study period included OpenAI's GPT-4o, GPT-4o with Canvas, o1, and o3; Anthropic's Claude 3.5 Sonnet and Claude Artifacts; Google's Gemini 2.0; and DeepSeek's open-source breakthroughs.

    “Notable releases included OpenAI’s GPT-4o, GPT-4o with Canvas, o1, and o3; Anthropic’s Claude 3.5 Sonnet and Claude Artifacts; Google’s Gemini 2.0; and DeepSeek’s open-source breakthroughs [30].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [30] DeepSeek-AI. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. 2025. doi: 10.48550/arXiv.2501.12948. source

      The baseline table in D.2 names GPT-4o, OpenAI o1, and Claude 3.5 Sonnet; A.1 describes DeepSeek-V3 as an open-source release, and D.2 describes DeepSeek-R1 as an open-source milestone. The cited paper does not substantiate the rest of the release list—GPT-4o with Canvas, o3, Claude Artifacts, or Gemini 2.0—or establish the citing study’s period. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Practitioners spend significant time on prompt preparation and validation.

    “Chopra et al. [25] found that practitioners spend significant time on prompt preparation and validation”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [25] Chopra B, Singha A, Fariha A, et al. Conversational challenges in AI-powered data science: obstacles, needs, and design opportunities. 2023. doi: 10.48550/arXiv.2310.16164. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLM outputs shift based on prompting strategies and model architecture.

    “demonstrating that outputs shift based on prompting strategies and model architecture.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [28] Komandur R, McDunn J, Nair N, et al. Artificial intelligence in biomedical data analysis: a comparative assessment of large language models for automated clinical trial interpretation and statistical evaluation. medRxiv. 2025; doi: 10.1101/2025.02.05.25321607. source

      Section 4.2 says subtle differences in prompt phrasing can significantly affect outputs. It also identifies differing model architectures as a possible influence on analytical outputs, but does not establish architecture as the cause of the observed shifts; the claim states that relationship more definitively than the paper does. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    During the study period and manuscript preparation (September 2024-February 2025), major advancements occurred across all leading companies, introducing new functionalities, model updates, and shifts in behavior.

    “during the study period and manuscript preparation (September 2024–February 2025), major advancements occurred across all leading companies, introducing new functionalities, model updates, and shifts in behavior.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [30] DeepSeek-AI. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. 2025. doi: 10.48550/arXiv.2501.12948. source

      The Introduction describes the release of DeepSeek-R1 models, and section C.2 describes changes in one model’s reasoning behavior during training. Neither addresses advancements across all leading companies during September 2024–February 2025. Searches for that period, company-wide updates, new functionalities, and behavior shifts found no passage establishing the claim. (LLM, conf=0.72)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    These developments introduced improved image processing, direct computer interaction, enhanced project management capabilities, and significantly improved code generation.

    “These developments introduced improved image processing, direct computer interaction, enhanced project management capabilities, and significantly improved code generation, further altering the landscape of LLM capabilities.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [30] DeepSeek-AI. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. 2025. doi: 10.48550/arXiv.2501.12948. source

      E.5 reports significant improvements on complex coding-reasoning problems, supporting the code-generation portion of the claim. The passages examined do not report improved image processing, direct computer interaction, or enhanced project management capabilities. Page numbers were not available in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Recent systematic reviews highlight notable improvements in LLMs' capabilities for code generation, model sophistication, and reliability.

    “Recent systematic reviews highlight notable improvements in LLMs’ capabilities for code generation, model sophistication, and reliability [21].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [21] Jiang J, Wang F, Shen J, Kim S, Kim S. A survey on large language models for code generation. arXiv. 2024; doi: 10.48550/arXiv.2406.00515. source

      The Introduction reports substantial improvement in code-generation performance, and §5.3.1 describes advances in model architectures. But §5.10 treats reliability as an ongoing concern, rather than documenting a notable improvement in it. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There are inconsistencies in LLM evaluation methodologies across healthcare applications.

    “Similarly, Lee et al. [29] highlighted inconsistencies in LLM evaluation methodologies across healthcare applications”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [29] Lee J, Park S, Shin J, Cho B. Analyzing evaluation methods for large language models in the medical field: a scoping review. BMC Med Inform Decis Mak. 2024; 24(1): 366. doi: 10.1186/s12911-024-02709-7. 39614219 PMC11606129 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among the 69 respondents, 56 (81.2%) held staff positions.

    “Among the 69 respondents, 56 (81.2%) held staff positions, and 57 (82.6%) identified as biostatisticians.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among 68 respondents reporting institutional affiliation, 25 (36.8%) were affiliated with Stanford QSU.

    “Of these, 68 reported their institutional affiliation: 29 (42.6%) were affiliated with DCRI Biostatistics, 25 (36.8%) with Stanford QSU, and 14 (20.6%) with Duke BERD.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The eligible population included 208 biostatisticians (162 staff, 46 faculty).

    “The eligible population included 208 biostatisticians (162 staff, 46 faculty).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among 68 respondents reporting institutional affiliation, 14 (20.6%) were affiliated with Duke BERD.

    “Of these, 68 reported their institutional affiliation: 29 (42.6%) were affiliated with DCRI Biostatistics, 25 (36.8%) with Stanford QSU, and 14 (20.6%) with Duke BERD.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among the 69 respondents, 57 (82.6%) identified as biostatisticians.

    “Among the 69 respondents, 56 (81.2%) held staff positions, and 57 (82.6%) identified as biostatisticians.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ChatGPT and Copilot remained the primary tools across all competency domains, with ChatGPT usage ranging from 60 to 62% and Copilot usage from 26 to 38% depending on the domain.

    “ChatGPT and Copilot remained the primary tools across all competency domains, with ChatGPT usage ranging from 60 to 62% and Copilot usage from 26 to 38% depending on the domain.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    LLMs improved productivity in coding, writing, and literature review.

    “LLMs improved productivity in coding, writing, and literature review;”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Among 68 respondents reporting institutional affiliation, 29 (42.6%) were affiliated with DCRI Biostatistics.

    “Of these, 68 reported their institutional affiliation: 29 (42.6%) were affiliated with DCRI Biostatistics, 25 (36.8%) with Stanford QSU, and 14 (20.6%) with Duke BERD.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Multimodal applications such as image generation and audio content creation were not widely adopted by biostatisticians during our study period.

    “they were not widely adopted by biostatisticians during our study period.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Verification approaches varied by task: quantitative analyses emphasized systematic testing and debugging, clinical knowledge required external verification of terminology and methodology, and communication tasks benefited from manual review and peer feedback.

    “These verification approaches varied by task. Quantitative analyses emphasized systematic testing and debugging, while clinical knowledge required external verification of terminology and methodology. Communication tasks benefited from manual review and peer feedback, particularly for tone and accuracy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Thematic analysis revealed statistical misinterpretation as a primary category of errors, where LLMs misinterpreted statistical results.

    “Statistical Misinterpretation: LLMs misinterpreted statistical results.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Five key themes emerged from the final survey question: productivity enhancement, reliability concerns, need for critical oversight, domain-specific limitations, and future potential.

    “In the final survey question, respondents were asked if they had any other thoughts or observations about using LLMs in biostatistical workflows. Five key themes emerged: Productivity Enhancement: Respondents noted efficiency gains: “I think it is very helpful for coding and things I would normally web search/look on stack overflow for I can just ask the LLM and get code.” Reliability Concerns:”
    • Also stated in: § Challenges and training needs · plain
      “Need for Critical Oversight: Respondents emphasized verification importance: “Please discourage others from sharing LLM-derived results until they have confirmed the accuracy… If I want to ask chatGPT I will do so myself. If I am asking a human chat group then I want a human response.” Domain-Specific Limitations: Users noted constraints in specialized applications: “For biostatistical/method”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    Thematic analysis identified four primary approaches to error identification: expertise and prior knowledge, external verification, testing and debugging, and manual review.

    “Among respondents who described how they identified errors that could have led to significant consequences, four primary approaches emerged: Expertise and Prior Knowledge: Respondents relied on existing knowledge: “I know the content that I teach and I know when the LLM is wrong.””
    • Also stated in: § Challenges and training needs · plain
      “External Verification: Users cross-referenced trusted sources: “If I am unsure of an answer… I usually do a web search as well or check some sources to try and see if it aligns with what the LLM gave me.””
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The qualitative analysis process began with parallel independent coding by a human analyst and the LLMs, followed by systematic integration and reconciliation.

    “The process began with parallel independent coding by a human analyst and the LLMs to identify themes, followed by systematic integration and reconciliation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    All LLM-generated content underwent thorough human review.

    “All LLM-generated content underwent thorough human review.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Human expertise was combined with ChatGPT-4o and Claude 3.5 Sonnet to qualitatively analyze free-text responses.

    “We combined human expertise with ChatGPT-4o and Claude 3.5 Sonnet to qualitatively analyze free-text responses.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Free-text responses underwent systematic preprocessing for word frequency analysis using R's tidytext package to provide quantitative validation of identified themes.

    “Free-text responses underwent systematic preprocessing for word frequency analysis using R’s tidytext package to provide quantitative validation of identified themes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The Duke University Health System Institutional Review Board determined that the study met the criteria for exemption from further IRB oversight.

    “The Duke University Health System Institutional Review Board determined that the study met the criteria for exemption from further IRB oversight (Pro00116592).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The manuscript was prepared using Quarto to enhance computational reproducibility.

    “Results were compiled, and the manuscript was prepared using Quarto [20] to enhance computational reproducibility.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] Quarto Development Team. Quarto: Open-source scientific and technical publishing system. Quarto Project. 2024. (https://quarto.org/) Accessed May 4, 2025.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Inconsistent tone and phrasing may undermine professional interactions.

    “Content fabrication threatens document integrity, particularly in formal research communications, while inconsistent tone and phrasing may undermine professional interactions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    We did not systematically analyze how different LLM models varied in their responses to the same statistical queries.

    “While we collected examples of effective prompts, we did not systematically analyze how different LLM models varied in their responses to the same statistical queries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Adoption is influenced not only by user preference but also by enterprise partnerships, licensing structures, and ongoing institutional caution and integration.

    “This suggests that adoption is influenced not only by user preference but also by enterprise partnerships, licensing structures, and ongoing institutional caution and integration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Our exclusive focus on text-based LLMs rather than broader generative AI represents a scope limitation.

    “Additionally, our exclusive focus on text-based LLMs rather than broader generative AI represents another scope limitation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Developments such as GitHub Copilot, ERNIE-Code, and autonomous AI software engineers like Devin illustrate the expanding role of LLMs and broader AI systems in software development.

    “Developments such as GitHub Copilot [22], ERNIE-Code [23], and autonomous AI software engineers like Devin [24] illustrate the expanding role of LLMs and broader AI systems in software development.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [22] GitHub. GitHub copilot: Your AI pair programmer. 2024. (https://github.com/features/copilot) Accessed May 4, 2025.

      Not checked in the free run.

    • cites [23] Chai Y, Wang S, Pang C, et al. ERNIE-code: Beyond English-centric cross-lingual pretraining for programming languages. In: Proceedings of the Association for Computational Linguistics: ACL 2023. Toronto, Canada: Association for Computational Linguistics; 2023: 10628– 10650. (https://aclanthology.org/2023.findings-acl.676) Accessed May 4, 2025. source
    • cites [24] Cognition AI. Devin: AI software engineer. 2024. (https://devin.ai/) Accessed May 4, 2025.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Real-world variability, model drift, or contextual reasoning may influence reliability of LLMs over time.

    “factors that may influence reliability over time.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] Bedi S, Liu Y, Orr-Ewing L, et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA. 2025; 333(4): 319– 328. doi: 10.1001/jama.2024.21700. 39405325 PMC11480901 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Many LLM evaluations focus on accuracy without addressing real-world variability, model drift, or contextual reasoning.

    “As Bedi et al. [31] noted, many LLM evaluations focus on accuracy without addressing real-world variability, model drift, or contextual reasoning”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] Bedi S, Liu Y, Orr-Ewing L, et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA. 2025; 333(4): 319– 328. doi: 10.1001/jama.2024.21700. 39405325 PMC11480901 source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Our findings reinforce the need for clear principles that support methodological integrity, scientific rigor, and ethical responsibility.

    “Our findings reinforce the need for clear principles that support methodological integrity, scientific rigor, and ethical responsibility.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    LLM usage is notable among respondents at two academic medical centers, though response patterns likely reflect early adopters.

    “LLM usage is notable among respondents at two academic medical centers, though response patterns likely reflect early adopters.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    As organizations formalize policies, training, and infrastructure around LLMs, broader adoption may extend beyond early users.

    “As organizations formalize policies, training, and infrastructure around LLMs, broader adoption may extend beyond early users.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The organizational context reveals an important gap: while nearly half of respondents received encouragement to use LLMs, few had access to formal training or verification protocols.

    “The organizational context surrounding these challenges reveals an important gap: while nearly half of the respondents received encouragement to use LLMs, few had access to formal training or verification protocols.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Our findings may reflect early adopters rather than typical usage patterns, limiting generalizability.

    “Thus, our findings may reflect early adopters rather than typical usage patterns, limiting generalizability.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Our reliance on self-reported interactions rather than direct analysis of LLM use presents a limitation.

    “Our reliance on self-reported interactions rather than direct analysis of LLM use presents another limitation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    This survey offers early insights into LLM usage among biostatisticians in academic medical centers, highlighting productivity benefits and integration challenges.

    “This survey offers early insights into LLM usage among biostatisticians in academic medical centers, highlighting productivity benefits and integration challenges.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Individuals already using or interested in LLMs may have been more likely to participate in the survey.

    “as individuals already using or interested in LLMs may have been more likely to participate in the survey.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Even when LLM tools are widely accessible, integration into daily biostatistical practice may lag substantially.

    “Importantly, even when LLM tools are widely accessible, integration into daily biostatistical practice may lag substantially.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Understanding current biostatistical workflows provides important context for interpreting clinical and translational research publications emerging today.

    “Moreover, understanding current biostatistical workflows provides an important context for interpreting clinical and translational research publications emerging today, many of which reflect practices from the same period captured in our survey.”
    Quote in the paperNot a substantial claim
50 of 50

  • The paper relies on this without support.

    Specialty-specific guidance is needed for integrating LLMs into biostatistical workflows.

    “However, specialty-specific guidance is still needed for integrating these tools into workflows.”

  • The paper relies on this without support.

    Insights from a survey of biostatisticians apply broadly to data scientists in healthcare research.

    “While the survey focused on biostatisticians, its insights apply broadly to data scientists in healthcare research.”

  • The paper relies on this without support.

    Biostatisticians increasingly use LLMs to enhance efficiency.

    “Biostatisticians increasingly use large language models (LLMs) to enhance efficiency, yet practical guidance on responsible integration is limited.”

  • The paper relies on this without support.

    Practical guidance on responsible integration of LLMs is limited.

    “Biostatisticians increasingly use large language models (LLMs) to enhance efficiency, yet practical guidance on responsible integration is limited.”

  • The paper relies on this without support.

    LLM performance is based on prompt design, computational constraints, and evolving training data.

    “Since LLM performance is based on prompt design, computational constraints, and evolving training data”
5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. 16. OpenAI. ChatGPT 4o teams edition. 2024. (https://www.openai.com/chatgpt) Accessed May 4, 2025.
  2. 17. Anthropic. Claude 3.5 sonnet professional edition. 2024. (https://www.anthropic.com/claude) Accessed May 4, 2025.
  3. 19. OpenAI. ChatGPT O1 preview teams edition. 2024. (https://www.openai.com/chatgpt) Accessed May 4, 2025.
  4. 18. Microsoft. Microsoft copilot [enterprise edition]. 2024. (https://www.microsoft.com/copilot) Accessed May 4, 2025.

Cited by

Sorted by citations

Related

Authors
Grambow, S. C., Desai, M., Weinfurt, K. P., Lindsell, C. J., Pencina, M. J., Rende, L., & Pomann, G.-M.
Affiliations
  • Duke University (US)
  • Stanford Medicine (US)
Corresponding author
Grambow, Steven C.
ORCID
Venue
Journal of Clinical and Translational Science
Publisher
Cambridge University Press (CUP)
Published
2025
Volume
9
Issue
1
Pages
e131-e131
ISSN
2059-8661 (ISSN-L)
DOI
10.1017/cts.2025.10064
PMC
PMC12260977
OpenAlex
W4410906353
Cited by
7
References
11
Citations by year
2026: 6 · 2025: 1
FWCI
1.70
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Clinical and Translational Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • University of Cambridge
  • National Institutes of Health
  • National Center for Advancing Translational Sciences
  • Duke Clinical Research Institute
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language models, biostatistics, LLM integration, biostatisticians, clinical research, translational research, hallucinations, verification strategies, responsible AI use, research productivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review