GPT-4 Technical Report

State-of-the-art large language models achieve more than 90% accuracy on popular benchmarks such as MMLU.

“LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1”

Generative pretrained transformers are a class of language models particularly powerful for rescoring decoded sequences.

“Generative pretrained transformers145 are a class of language models that are particularly powerful for this application.”

In practical applications like chatbots and agents, prompts are often much larger than generated responses such as summaries and code pieces, creating significant inference latency and memory utilization overhead.

“In practical applications like chatbots and agents, where prompts range from multi-turn conversations to extensive articles or codebases [1, 9, 10], prompts are often much larger than generated responses such as summaries and code pieces, thus creating significant inference latency and memory utilization overhead.”

LLM-infused search engines already include hedging language.

“First, LLM-infused search engines already include hedging language [56, 62, 73].”

The emergence of Large Language Models and Vision Large Language Models has revolutionized many AI-related research areas and applications.

“The emergence of Large Language Models (LLMs) [2,95,187,191] and the subsequent transition to combined language/vision models (Vision Large Language Models, VLLMs [2,9,14,123,124,196]) has revolutionized many AI-related research areas and applications.”

General-purpose LLMs such as GPT-4 have great in-context learning capability in financial-oriented tasks.

“The authors also found that the general-purpose LLMs such as GPT4[36] has great in-context learning capability in financial oriented tasks.”

GenAI systems can generate text, images, audio, code, and other modalities from natural-language prompts.

“Powered by large-scale deep learning models, these systems can generate text, images, audio, code, and other modalities from natural-language prompts [2].”

Details of the architecture, parameter counts, and training data of GPT-3.5-Turbo and GPT-4-Turbo are omitted or glancingly described in OpenAI's technical reports.

“details of the architecture, parameter counts, and training data of GPT-3.5-Turbo and GPT-4-Turbo are omitted or glancingly described in the company's technical reports [41].”

LLMs often exhibit a deep understanding of the semantic connections among existing knowledge to express new pieces of reasoning.

“More importantly, LLMs often exhibit a deep understanding of the semantic connections among existing knowledge to express new pieces of reasoning [1].”

GPT4 is known to have been trained with more parameters and has shown impressive performances across a variety of human tasks.

“This is consistent as GPT4 is known to have been trained with more parameters and has had shown to have made impressive performances across a variety of human tasks33,34.”

Natural language expressions of uncertainty allow uncertainty to be expressed seamlessly within natural language interactions rather than on the side or in onboarding materials that users might overlook.

“Third, this allows uncertainty to be expressed seamlessly within the natural language interactions of LLM-infused applications, rather than on the side or in onboarding materials [21, 74, 79] that users might overlook.”

Current findings suggest that LLMs are often overconfident.

“Current findings suggest that LLMs are often overconfident [26, 73, 113], which may give a false impression of their capabilities and exacerbate overreliance [35].”

GPT-4 was published in March 2023.

“GPT-4 was published in March of 2023 [12].”

Generative AI has demonstrated human-level performance on a broad range of tasks.

“Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive technology capable of achieving human-level performance on a broad range of tasks (1–5).”

MLLM (GPT-4o) is used to generate captions for each panel and summarize a story for each page.

“we utilize the powerful multimodal understanding ability of MLLM (GPT-4o [1]) to fed the original manga page and the corresponding XML file into MLLM, and prompt it to generate corresponding captions for each panel and summarize the corresponding story.”

Recent evidence suggests that models might infer and act upon intentions and goals, as a result of training or when prompted with a persona.

“Recent evidence suggests that models might infer and act upon intentions and goals [2,43,55], as a result of training or when prompted with a persona [15].”

LLMs, which could accept image and text inputs and produce text outputs, have shown promise in various aspects of medical care.

“However, large language models (LLMs), which could accept image and text inputs and produce text outputs, have shown promise in various aspects145 of medical care.”

The main enhancement of GPT-4 lies in its ability to carry out complex reasoning.

“This aligns with the assertion that the main enhancement of GPT-4 lies in its ability to carry out complex reasoning (OpenAI, 2023).”

Large language models have achieved notable successes, demonstrating significant potential in attaining human-like intelligence.

“In recent years, large language models (LLMs) have achieved notable successes, demonstrating significant potential in attaining human-like intelligence [5][6][7][8][9][10].”

Researchers propose building a predictable framework for training LLMs.

“In addition, researchers propose building a predictable framework for training LLMs [44].”

In the development of GPT-4, for open-domain hallucinations, OpenAI collected real-world ChatGPT data that were flagged as non-factual by users and used it together with additional labeled comparison data to train their reward models.

“Firstly, in tackling open-domain hallucinations, OpenAI collected real-world ChatGPT data that were flagged as non-factual by users and used it together with additional labeled comparison data to train their reward models.”

For closed-domain hallucinations, OpenAI utilized GPT-4 itself to generate and subsequently utilize synthetic data for reward model training.

“Secondly, for closed-domain hallucinations, OpenAI utilized GPT-4 itself to generate and subsequently utilize synthetic data for reward model training.”

For pre-trained GPT-4, the probability of generating an answer is often directly proportional to the probability of it being correct.

“For instance, [133] found that for pre-trained GPT-4, the probability of generating an answer is often directly proportional to the probability of it being correct.”

Reinforcement learning methods such as RLHF and RLAIF have proven effective in aligning LLM outputs with expected behaviors.

“Reinforcement learning methods, such as RLHF [37] and RLAIF [22], have proven effective in aligning LLM outputs with expected behaviors.”

Large language models have demonstrated exceptional language generation capabilities across various domains.

“Recently, large language models (LLMs) (Brown et al., 2020;Ouyang et al., 2022;OpenAI, 2023) have demonstrated exceptional language generation capabilities across various domains.”

The concept of calibration was introduced by Dawid and has been extensively studied in statistics, machine learning, and specifically for language models.

“As discussed in the introduction, the concept of calibration was introduced by Dawid [10] and has been extensively studied in statistics and machine learning and even specifically for LMs [7,17,38] and other related fields of deep learning and generative AI [21,28].”

The TruthfulQA dataset has become widely used for evaluating the factuality of LLMs.

“This dataset has become widely used for evaluating the factuality of LLMs [89, 146, 192, 220].”

Social bot detection methods assume the fundamental distinction is between human and bot; they cannot be directly applied to a platform where all participants are AI agents.

“However, these methods assume the fundamental distinction is between human and bot; they cannot be directly applied to a platform where all participants are AI agents powered by large language models 12 operating within multi-agent frameworks 10,30”

The uncertainty expressions inserted into AI responses preserve the realisticness of responses.

“Current LLMs already output expressions such as ' I'm not sure, but ' and ' I'm not sure, but my guess is ' [73], so we believe these insertions preserve the realisticness of responses.”

State-of-the-art LLMs are generally implemented using the Transformer architecture, a multi-layer feedforward network with attention mechanism.

“State-of-the-art LLMs [66, 67, 7, 52] are generally implemented using the Transformer architecture, which is a multi-layer feedforward network equipped with attention mechanism [71].”

The GPT-4 version of ChatGPT has been fine-tuned to reduce certain unwanted behaviours.

“In this case, the underlying LLM (GPT-4) has been fine-tuned to reduce certain unwanted behaviours 12 .”

General-purpose foundation models pre-trained on diverse multi-task data tend to outperform narrowly tailored and specialized solutions in natural language and computer vision.

“In natural language [1] and computer vision [39], general-purpose foundation models that are pre-trained on diverse multi-task data tend to outperform narrowly tailored and specialized solutions.”

The number of reported contributors increased from 25 for GPT-3 to 284 for GPT-4.

“The number of reported contributors increased from 25 for GPT-3 [14] to 284 for GPT-4 [12]”

GPT-4 finished training in August 2022.

“for example, GPT-4 finished training in August 2022 [12].”

OpenAI's technical report on GPT-4 presents benchmarks but its evaluation lacks scientific rigor: it fails to provide agreement between raters, details on criteria for assessment, and details on statistical analysis for a larger sample of essays.

“While OpenAI’s technical report on GPT-46 presents some benchmarks, their evaluation lacks scientific rigor: it fails to provide vital information like the agreement between raters, does not report on details regarding the criteria for assessment or to what extent and how a statistical analysis was conducted for a larger sample of essays.”

Proprietary models such as GPT-3.5 and GPT-4, and recent open-source models like LLaMA3, CodeQwen, and DeepSeek-Coder, were excluded because they do not fully comply with open-source data requirements.

“In accordance with our first criterion, we excluded proprietary models such as GPT-3.5 [64] and GPT-4 [1], along with recent open-source models like LLaMA3 [57], CodeQwen [8], and DeepSeek-Coder [22], which do not fully comply with open-source data requirements.”

In natural language and computer vision, general-purpose foundation models pre-trained on diverse multi-task data tend to outperform narrowly tailored and specialized solutions.

“In natural language [1] and computer vision [39], general-purpose foundation models that are pre-trained on diverse multi-task data tend to outperform narrowly tailored and specialized solutions.”

GPT-4's training involved the integration of training datasets from GSM8K and MATH to boost its mathematical reasoning skills.

“For instance, GPT-4's training involved the integration of training datasets from GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) to boost its mathematical reasoning skills, as evidenced by their technical report (OpenAI, 2023).”

The most critical problem of ChatGPT frequently mentioned by participants was hallucination.

“The most critical problem of ChatGPT that our participants frequently mentioned was its hallucination”

Specialized models like UniEval and Prometheus 2, and LLMs like GPT-4 can be used to judge the factuality of a passage.

“Specialized models like UniEval (Zhong et al., 2022) and Prometheus 2 (Kim et al., 2024), and LLMs like GPT-4 (OpenAI et al., 2024b) can be used to judge the”

In practice, the vocabulary size of LLMs typically ranges from 100K to 256K.

“In practice, the vocabulary size of LLMs typically ranges from 100K to 256K [44].”

In the case of RLHF models, the distribution is skewed toward more likely answers.

“However, in the case of RLHF models, the distribution is skewed toward more likely answers.”

Moltbook restricts posting to AI agents authenticated through the OpenClaw agent framework, an open-source system for deploying large language model agents with persistent identity and scheduled behaviors.

“Moltbook restricts posting to AI agents authenticated through the OpenClaw agent framework, an open-source system for deploying large language model agents 12 with persistent identity and scheduled behaviors.”

All experiments run on the same backbone LLM (GPT-4o-mini).

“All experiments run on the same backbone LLM (GPT-4o-mini) [38].”

When prompted, the model may produce convincing personalized scams given appropriate knowledge about the target.

“When prompted, the model may produce convincing personalized scams given appropriate knowledge about the target [21,43] (that was either given in the prompt or, importantly, the model acquired during the chat session).”

Retrieval-augmented models now cite their sources, possibly leading to overreliance on their factuality by users.

“Retrieval-augmented models now cite their sources, possibly leading to “overreliance” [43] on their factuality by users.”

ChatGPT and GPT-4 produce plausible utterances, even wrong ones, in a confident and authoritative tone.

“They also produce plausible utterances, even wrong ones, in a confident and authoritative tone [43].”

Enabling LLMs to write to some form of persistent storage is currently already investigated in many plugins and systems, including Bing Chat.

“Enabling these models to write to some form of persistent storage is currently already investigated in many plugins and systems [42], including Bing Chat [64].”

The evaluated LLMs include GPT-4o, GPT-4o-mini, Mixtral-8×22B, Llama-3.3-70B, and Qwen2.5-72B, covering proprietary and open-weight model families.

“Our selection includes GPT-4o and GPT-4o-mini (OpenAI), Mixtral-8×22B, Llama-3.3-70B, and Qwen2.5-72B,enabling analysis across proprietary and openweight model families.”

GPT-4o serves as a high-capacity proprietary judge model, frequently adopted as a reference in evaluation pipelines, with a 128K context window.

“GPT-4o (OpenAI, 2023) serves as a highcapacity proprietary judge model, frequently adopted as a reference in evaluation pipelines, with a 128K context window.”

GPT-4o-mini is a lower-cost, reduced-capacity variant used to study how evaluation reliability changes under constrained model capacity.

“GPT-4o-mini (OpenAI, 2023) is a lowercost, reduced-capacity variant used to study how evaluation reliability changes under constrained model capacity.”

Despite recent improvements in factual accuracy, the recurring issue of misinformation and potentially harmful consequences for patient care remains.

“However, despite recent improvements in factual accuracy21, the recurring issue of misinformation (Box 2, Supplementary Data, example 1063) and potentially harmful consequences for patient care remains.”

Dynamically updating LLMs can lead to improved performance compared to their predecessors.

“Although dynamically updating models can lead to improved performance compared to their predecessors5,21”

To date, the development of LLMs has been driven primarily by commercial companies such as OpenAI/Microsoft, Meta, and Google.

“Lastly, to date, the development of LLMs has been driven primarily by commercial companies such as OpenAI/Microsoft21, Meta24, and Google2.”

Higher-order relations play a central role in some cognitive theories of analogy.

“Such higher-order relations play a central role in some cognitive theories of analogy, 43”

Foundation model performance crucially depends on dataset and model size, as demonstrated by scaling law results.

“Foundation model performance crucially depends on dataset and model size, as demonstrated by scaling law results28–30.”

Pan-cancer detection performance scales with model and dataset size.

“We found that pan-cancer detection performance scales with model and dataset size (Fig. 2g), which is consistent with observations of prior foundation models in other domains28–30.”

LLMs can produce plausible sounding but inaccurate information, a phenomenon referred to as "AI hallucination".

“This focus can lead to the production of plausible sounding but inaccurate information, a phenomenon referred to as “AI hallucination” (6).”

OpenAI's assessment of GPT-4's performance in non-English languages yielded higher proficiencies in 24 out of 26 languages compared to the previous models' proficiency in English.

“OpenAI assessed GPT-4’s performance in non-English languages, which yielded higher proficiencies in 24 out of 26 languages as compared to the previous models’ proficiency in English [13].”

The results of this research validate OpenAI's assertion that GPT-4 has higher proficiency in non-English languages.

“Although OpenAI did not disclose the precise methodologies used to obtain these outcomes, the results of this research validate their assertion.”

GPT-4's superior correct response rate on hard-level questions compared with the average correct response rate of general examinees indicates the potential of language models such as GPT-4 to surpass human performance in highly specialized fields.

“Notably, GPT-4’s superior correct response rate on the challenging hard-level questions, compared with the average correct response rate of general examinees, indicates the potential of language models such as GPT-4 to surpass human performance in highly specialized fields [13].”