Generative Artificial Intelligence and Large Language Models in Primary Care Medical Education

May combine two claims.

The author's institution assembled the needed local LLM components for under $3,000, an amount affordable for academic medical centers and small projects.

“At my institution, we were able to assemble the necessary components for less than $3,000, which is well within the budgets of academic medical centers and small research projects.”

May combine two claims.

In multi-turn chats, every turn reprocesses all earlier tokens, so expenses can rise quickly.

“For multiturn conversations, each turn in the conversation involves processing all previously sent and received tokens.Consequently, these costs can quickly add up.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Generative AI may be applied in medical education for personalized teaching, simulation, feedback, evaluation, qualitative research support, and critical appraisal of literature.

“Additionally, I identify the potential applications of generative artificial intelligence to medical education, including personalized instruction, simulation, feedback, evaluation, augmentation of qualitative research, and performance of critical assessment of the existing scientific literature.”

May combine two claims.

LLMs are flexible and can be directed to perform tasks including writing, summarizing, logical inference, and programming.

“Importantly, LLMs appear to be flexible and can be instructed to conduct many tasks, such as text generation, summarization, logical inference, and computer programming.6,[8][9][10][11][12][17][18][19]”

• cites [6]Not checked yet

  Not checked in the free run.

• cites [8]Not checked yet

  Not checked in the free run.

• cites [9]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Generative AI and LLMs extend a long-running information-processing revolution whose origins lie in the 1947 transistor.

“Generative artificial intelligence and large language models are the continuation of a technological revolution in information processing that began with the invention of the transistor in 1947.”

Probably not a substantial claim: may not pass S4 (based).

Medical education carries high risk because wrongly taught material can go unnoticed for years and later harm a patient.

“Medical education is a high-risk activity: Information that is incorrectly taught to a student may go unrecognized for years until a relevant clinical situation appears in which that error can lead to patient harm.”

May combine two claims.

Local LLMs trade off performance against closed cloud models like GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro, which remain superior.

“Local LLMs have trade-offs in performance with closedsource cloud models such as OpenAI's GPT-4 Turbo, Anthropic's Claude 3 Opus, and Google's Gemini 1.5 Pro, which still provide superior performance.51”

• cites [51]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The transformer design removed recurrence from artificial neural networks while still encoding word relationships, which improved parallelism and training speed and allowed very large training sets.

“The transformer architecture eliminated recurrence from ANNs-without sacrificing the ability to encode sematic relationships between words-thereby allowing marked gains in parallelism and training time, enabling very large data sets to be used for training.6”

• cites [6]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

For LLMs to be widely deployed, they must be accurate, unbiased, affordable, and secure.

“Widespread deployment of LLMs requires them to be accurate, bias-free, affordable, and secure.”

Probably not a substantial claim: may not pass S2 (checkable).

LLM-based medical education tools should not be trusted until human experts trained to spot structural and systemic bias validate them, so bias is not passed to learners.

“Before LLM-based systems are trusted for medical education-even those including algorithmic components to mitigate bias-they will require validation by human experts who have training in recognizing structural and systemic biases to avoid unintentionally propagating biases to medical learners via AI.”

Probably not a substantial claim: may not pass S4 (based).

The principal constraints on generative AI in medical education are hallucination, bias, expense, and security.

“In this article, I discuss the principal limitations to the use of generative artificial intelligence in medical education-hallucination, bias, cost, and security-and suggest some approaches to confronting these problems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Larger models, in general, yield better performance.

“Larger models, in general, yield better performance.46”

• cites [46]Not checked yet

  Not checked in the free run.

May combine two claims.

Simulation is costly to run, and its traditional cost-effectiveness has been questioned.

“Simulation is nevertheless expensive to administer, and the cost-effectiveness of simulation as traditionally implemented has been questioned.56”

• cites [56]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

These technologies are expected to broadly influence society.

“These technologies, driven by transformer architectures for artificial neural networks, are poised to broadly influence society.”

May combine two claims.

These models can hold considerable world knowledge, but that knowledge is finite.

“Clearly these models can contain much knowledge about the world-but not an unlimited amount.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GPT-4, the top-performing model, costs 3 cents per 1,000 input tokens and 6 cents per 1,000 output tokens.

“For the highest performance model, GPT-4, the current price is 3 cents per thousand input tokens and 6 cents per thousand output tokens.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs are trained to perform next-word prediction.

“LLMs are trained to perform next-word prediction.16”

• cites [16]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Most LLMs have been deployed on cloud-based servers.

“This has meant that most LLMs have been run on servers using a cloud computing framework.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs store a statistical model of human language inside a neural network after processing enormous amounts of training text made possible by transformers.

“LLMs encode a statistical understanding of human languages within a neural network by analysis of vast volumes of training text enabled by transformer neural networks.16”

• cites [16]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

If LLMs were validated to give accurate, unbiased, hallucination-free information at least equal to human teachers, learners could use a personalized virtual tutor and progress at their own level.

“If LLMs could be validated to provide information that is known to be accurate, free of bias and hallucination-at least as good as human instructors-then students could interact with a personalized virtual tutor and could learn at a pace suitable to their current level.”

Probably not a substantial claim: may not pass S2 (checkable).

LLMs used in primary care and medical education need to provide correct information.

“For LLMs to be used in primary care and medical education, they must supply correct information.”

May combine two claims.

The GPT-3.5 model, having 175 billion parameters, is estimated to hold roughly 85 GB of information, about the same as all uncompressed Wikipedia text.

“For GPT-3.5, with 175 billion parameters, 9 the model likely encodes about 85 gigabytes of data, which is approximately the uncompressed equivalent of all text in Wikipedia.*”

• cites [9]Not checked yet

  Not checked in the free run.

• AssumptionThe authors state this as an assumption: During LLM training, parameters are usually represented as 16- or 32-bit floating-point numbers.
• AssumptionThe authors state this as an assumption: Reducing parameters to 4 bits each does not cause meaningful performance loss.
• AssumptionThe authors state this as an assumption: Each parameter carries roughly half a byte, which must represent both language structure and world knowledge.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Patient data handled by medical learners fall under HIPAA, and educational records fall under FERPA.

“Medical learners interact with patient data, which is protected under the Health Insurance Portability and Accountability Act (HIPAA).Likewise, educational records are protected by the Family Educational Rights and Privacy Act (FERPA).”

May combine two claims.

High-quality narrative feedback is hard to produce, and many efforts to improve it fail to improve, sustain, cost-effectively deliver, or generalize.

“Quality narrative feedback, however, is difficult to achieve, [58][59][60] and many interventions seeking to improve feedback quality do not yield improvement, lack sustainability, are not cost effective, or are not generalizable.[61][62][63]”

• cites [58]Not checked yet

  Not checked in the free run.

• cites [59]Not checked yet

  Not checked in the free run.

• cites [60]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Even with these difficulties, primary care educators must prepare to use generative AI and LLMs in student and resident training.

“Despite these problems, primary care medical educators have an imperative to prepare for the use of generative AI and LLMs in medical student and resident education.”

May combine two claims.

LLMs may reproduce biases from their training data, though developers are attempting to reduce this.

“LLMs, too, run the risk of perpetuating biases present in their training data, although measures are being taken by developers to mitigate them.16”

• cites [16]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

These tools will require validation to confirm accuracy and reduce the chance that unintended systemic bias reaches learners.

“Validation will be needed to ensure accuracy of these tools and to minimize the risk of unintentional, systemic bias being transmitted to medical learners.”