On the Societal Impact of Open Foundation Models

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For biosecurity risk (Gopal et al., 2023), the analyzed study provides no analysis of how a rogue individual or non-state actor could engage in bioterrorism without foundation models.

“The paper provides no analysis of how a rogue individual or non-state actor could engage in bioterrorism absent foundation models (or open foundation models).”

• cites Gopal et al., 2023 sourceDisputed

  The Evaluation section says the authors used online resources and scientific publications to outline several feasible paths to obtaining infectious 1918 influenza, depending on a person's skills and circumstances. That is an analysis of how an individual could obtain the agent without a foundation model, although the participant exercise itself compared two models rather than testing a no-model group. No page number is available in our copy. (LLM, conf=0.83)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: A study's failure to present evidence in a framework element means it does not show increased marginal societal risk from open foundation models.

May combine two claims.

The usability of foundation models strongly influences innovation, and factors beyond open release, such as model capabilities and inference API quality, shape usability.

“the usability of foundation models strongly influences innovation (Vipra & Korinek, 2023): factors beyond whether a model is released openly such as the capabilities of the model and the quality of potential inference APIs shape usability.”

• cites Vipra & Korinek, 2023 sourcePartly supported

  The Executive Summary (page unavailable) links growing model capabilities to usefulness across applications. The Intellectual property protections section (page unavailable) says some open-source models are easy to use only with particular hosting services, supporting the point that open release alone does not determine usability. Our copy does not specifically establish that usability strongly influences innovation or discuss the quality of inference APIs. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

May combine two claims.

Monocultures often yield poor societal resilience and are susceptible to widespread systemic risk, as illustrated by Meltdown and Spectre attacks due to widespread dependence on Intel and ARM-based microprocessors.

“Monocultures often yield poor societal resilience and are susceptible to widespread systemic risk: consider the Meltdown and Spectre attacks, which led to massive security risks because of the widespread dependence on Intel and ARM-based microprocessors (Kocher et al., 2018;Lipp et al., 2018;Staff, 2018).”

• cites Kocher et al., 2018 sourceSupported
• cites Lipp et al., 2018 sourcePartly supported

  The Abstract (p. 1) reports that Meltdown affects Intel microarchitectures and poses risks to millions of customers; the Discussion (p. 15) also identifies Spectre as a continuing security issue. But §6.4 (p. 13) says the authors could not reproduce Meltdown’s kernel-memory leak on the ARM CPUs they tested. The paper does not establish that dependence on both Intel and ARM processors caused the widespread risk, or make the broader claim about societal resilience. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Staff, 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Park et al. (2022) use foundation models without safety filters because their research aims to simulate human behavior including toxic speech, and most closed foundation models would suppress these outputs.

“Park et al. (2022) use foundation models without safety filters because their research aims to simulate human behavior (including toxic speech).Most closed foundation models would suppress these outputs.”

• cites Park et al. (2022) sourcePartly supported

  Section 2.3 says the authors use GPT-3 to simulate social behavior and regard its ability to generate troll behavior as useful for design. Section 4.3 identifies the base davinci engine and its settings, but does not say they chose a model without safety filters for that reason. The paper also does not say that most closed foundation models would suppress toxic outputs. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

Foundation model monocultures have been conjectured to lead to correlated failures and cultural homogenization.

“foundation model monocultures have been conjectured to lead to correlated failures (Bommasani et al., 2022) and cultural homogenization (Lee et al., 2022;Padmakumar & He, 2023).”

• cites Bommasani et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lee et al., 2022 sourceCould not verify

  Lee et al. is cited for cultural homogenization, not for correlated failures. The Abstract discusses plagiarism in generated text, and §6.2 discusses similarity within fine-tuning datasets, but neither addresses cultural homogenization caused by foundation model monocultures. Searches for cultural homogenization, monocultures, and cultural diversity found no passage on that claim. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites Padmakumar & He, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Understanding how biases propagate and are potentially amplified requires comparing data biases to model biases, which requires access to training data.

“to understand how biases propagate, and are potentially amplified, requires comparisons of data biases to model biases, which in turn requires access to the training data (Wang & Russakovsky, 2021).”

• cites Wang & Russakovsky, 2021 sourcePartly supported

  The Introduction (p. 1) defines bias amplification in terms of biases in training data, and Related Work (p. 2) explains why measuring it requires accounting for training correlations. But §5.3 (p. 7) measures amplification using several alternative base correlations, including uniform and labor-force-based values. The paper therefore does not support the unqualified claim that access to the training data is required. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For voice cloning (Ovadya & Whittlestone, 2019), the analyzed study discusses marginal risk of openness but makes little direct comparison to closed models or other enabling technologies.

“There is some discussion of the marginal risk of openness (e.g., related to reproducibility, modifiability, access ratchets), but minimal direct comparison to risks associated with closed models or other technologies that could similarly enable misuse.”

• cites Ovadya & Whittlestone, 2019 sourceSupported

• AssumptionRests on an assumption: A study's failure to present evidence in a framework element means it does not show increased marginal societal risk from open foundation models.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For disinformation (Musser, 2023), the analyzed study shows cost reduction for content generation but does not address demand, real-world effectiveness, or circumvention of defenses.

“The paper gives evidence of cost reduction for the generation of content, but does not discuss existing assessments of the demand for such content, whether it is actually effective in the real world, or if open foundation models can be used to circumvent existing defenses.”

• cites Musser, 2023 sourcePartly supported

  The Abstract reports modeled reductions in content-generation costs, supporting that part of the claim. But Section 5, “The Value of Monitoring when Public Models Are Accessible” (page unavailable), explicitly analyzes how switching to open-source models reduces the effect of monitoring controls, so the claim that the paper does not address circumvention of defenses is incorrect. Section 8.3 also briefly discusses research on the real-world effects of influence operations, although it does not establish the effectiveness of LLM-generated content. (LLM, conf=0.89)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: A study's failure to present evidence in a framework element means it does not show increased marginal societal risk from open foundation models.

Probably not a substantial claim: may not pass S3 (load-bearing).

Malware creation is a distinct cybersecurity risk that needs its own analysis, separate from automated vulnerability detection.

“for example, the creation of malware is another cybersecurity risk that requires separate analysis”

• cites Barrett et al., 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

For cybersecurity risk (Seger et al., 2023), the analyzed study offers no evidence that open foundation models have contributed to cybersecurity incidents by enabling malware creation.

“The paper does not provide any evidence that open foundation models have contributed to cybersecurity incidents by facilitating the creation of malware.”

• cites Seger et al., 2023 sourceSupported

• AssumptionRests on an assumption: A study's failure to present evidence in a framework element means it does not show increased marginal societal risk from open foundation models.

May combine two claims.

Dark web advertisements for tools claiming to facilitate automated vulnerability detection exist, but it is unclear if these products rely on open foundation models.

“Dark web advertisements for tools exist, claiming to facilitate automated vulnerability detection, but it is unclear if these products rely on open FMs (Amos, 2023).”

• cites Amos, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Training the Llama 2 series required 3.3 million GPU hours on NVIDIA A100-80GB GPUs; at February 2024 cloud rates of $1.8 per GPU hour, training this model would cost around $6 million.

“training the Llama 2 series of models required 3.3 million GPU hours on NVIDIA A100-80GB GPUs (Touvron et al., 2023b): at February 2024 cloud computing rates of $1.8/GPU hour (Lambda, 2024), training this model would cost around $6 million.”

• cites Touvron et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lambda, 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For NCII (Lakatos, 2023), the analyzed study shows tens of millions of unique visitors to NCII distribution sites but does not describe risks from preexisting tools such as face swapping.

“demonstrates that there are tens of millions of unique visitors to such sites, though it does not describe the risk of preexisting tools such as those for face swapping.”

• cites Lakatos, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: A study's failure to present evidence in a framework element means it does not show increased marginal societal risk from open foundation models.

May combine two claims.

Open foundation models used to create non-consensual intimate imagery require few resources to run; many prominent text-to-image models can run on an iPhone or MacBook.

“Open FMs used to create NCII require few resources to run-indeed, many prominent text-to-image models can run on an iPhone or Mac-Book.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For spear-phishing scams (Hazell, 2023), the analyzed study does not examine how open FMs could bypass defenses built into operating systems or email services.

“There is no analysis of how open FMs can be used to circumvent existing defenses built into operating systems or email services.”

• cites Hazell, 2023 sourceNot checked — no full text of the source

• AssumptionRests on an assumption: A study's failure to present evidence in a framework element means it does not show increased marginal societal risk from open foundation models.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Broader access, greater customizability, and local inference expand how foundation models are used to develop applications.

“Broader access, greater customizability, and local inference expand how foundation models are used to develop applications.”

May combine two claims.

Because open foundation models are generally more heavily customized, model usage becomes more fragmented and reduces potential for strong economies of scale.

“because open foundation models are generally more heavily customized, model usage becomes more fragmented and lessens the potential for strong economies of scale.”

Probably not a substantial claim: may not pass S4 (based).

Open foundation model developers generally lack access to user feedback and interaction logs that closed developers have for improving models over time.

“open foundation model developers generally do not have access to user feedback and interaction logs that closed model developers do for improving models over time.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Open foundation models allow greater diversity in defining acceptable model behavior, while closed models impose a monolithic view set by the developer.

“Open foundation models allow for greater diversity in defining what model behavior is acceptable, whereas closed foundation models implicitly impose a monolithic view that is determined unilaterally by the foundation model developer.”

Classified as probably not a substantial claim.

The framework retrospectively explains why research on the dangers of open foundation models has been contentious, because past studies implicitly analyze different subsets of the framework.

“It also helps retrospectively explain why the research on the dangers of open foundation models has been so contentious-past studies implicitly analyze risks for different subsets of our framework.”

May combine two claims.

Open foundation models offer substantial benefits, though with caveats, in innovation, competition, distribution of decision-making power, and transparency.

“Open foundation models present significant benefits, with some caveats, that span innovation, competition, the distribution of decision-making power, and transparency.”

May combine two claims.

For open foundation models, developers do not observe inference by default, which makes monitoring or moderation difficult, especially for local inference.

“In all cases, foundation model developers do not observe inference by default, making monitoring or moderation challenging, especially for local inference.”

Probably not a substantial claim: may not pass S4 (based).

The innovation benefits of open foundation models may be limited by comparative disadvantages in improving open models over time.

“However, the benefits of open foundation models for innovation may have limits due to potential comparative disadvantages in improving open foundation models over time.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Dedicated model hosts know which models are used, so developers may be able to coordinate with hosts to implement some monitoring or moderation.

“Since dedicated model hosts are aware of what models are being used, developers may be able to coordinate with hosts to implement certain forms of monitoring/moderation.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Open foundation models differ from closed ones in five ways: wider access, easier customization, local inference, inability to withdraw released weights, and weaker usage monitoring.

“Open foundation models are marked by broader access, greater customizabil-ity, the potential for local inference, an inability to rescind model access once released, and weaker monitoring.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

New defenses can be implemented or existing defenses modified to address the increase in overall risk.

“new defenses can be implemented or existing defenses can be modified to address the increase in overall risk.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Broad access to weights enables some forms of transparency.

“Broad access to weights enables some forms of transparency.”

Probably not a substantial claim: may not pass S4 (based).

Broader access and greater customizability facilitate scientific research.

“Broader access and greater customizability facilitate scientific research.”

May combine two claims.

Ensuring ongoing access to specific models is essential for scientific reproducibility, which has been undermined by closed model developers retiring models regularly.

“Ensuring ongoing access to specific models is essential for the scientific reproducibility of research, something that has been undermined to date by the business practice of closed model developers to retire models regularly (Kapoor & Narayanan, 2023).”

• cites Kapoor & Narayanan, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Greater customizability mitigates harms of monoculture and broader access reduces market concentration.

“Greater customizability mitigates the harms of monoculture and broader access reduces market concentration.”

May combine two claims.

Since open foundation models are more easily customized, they may yield more diverse downstream model behavior, thereby reducing the severity of homogeneous outcomes.

“Since open foundation models are more easily customized, they may yield more diverse downstream model behavior, thereby reducing the severity of homogeneous outcomes.”

May combine two claims.

Closed foundation models are often instrumented with safety measures by developers, and these measures can complicate or render some research impossible.

“since closed foundation models are often instrumented by safety measures by developers, these measures can complicate or render some research impossible.”

May combine two claims.

Model weights only make some types of transparency possible but do not guarantee such transparency will manifest.

“model weights only make some types of transparency (e.g.evaluations of risk) possible, but they do not guarantee such transparency will manifest.”

May combine two claims.

Open foundation models may increase competition in some regions of the AI supply chain but are unlikely to reduce market concentration in the highly concentrated upstream markets of computing and specialized hardware providers.

“while open foundation models may increase competition in some regions of the AI supply chain, they are unlikely to reduce market concentration in the highly concentrated upstream markets of computing and specialized hardware providers (Widder et al., 2023).”

• cites Widder et al., 2023 sourceNot checked — no full text of the source

May combine two claims.

Using a closed foundation model for vulnerability detection is not necessarily misuse, creating a nontrivial classification problem distinguishing legitimate and malicious uses.

“using a closed foundation model for vulnerability detection is not necessarily misuse, which introduces a nontrivial classification problem of distinguishing between legitimate and malicious uses of closed foundation models for automated vulnerability detection”

May combine two claims.

Existing defenses provide a baseline for addressing new risks introduced by open foundation models but do not fully clarify marginal risk.

“While existing defenses provide a baseline for addressing new risks introduced by open foundation models, they do not fully clarify the marginal risk.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The availability of other key assets, especially training data, would further accelerate scientific research.

“The availability of other key assets (especially training data) would further accelerate scientific research.”

May combine two claims.

Broader access to foundation models enables greater inclusion in scientific research, and model weights are essential for several forms of research across AI interpretability, security, and safety.

“Broader access to foundation models enables greater inclusion in scientific research, and model weights are essential for several forms of research across AI interpretability, security, and safety”

May combine two claims.

Misuse vectors often involve complex supply chains and foundation model capabilities are rapidly evolving, meaning the balance of power between attackers and defenders can be unstable.

“misuse vectors often involve complex supply chains and the capabilities of foundation models are rapidly evolving, meaning the balance of power between attackers and defenders can be unstable (Shevlane & Dafoe, 2020).”

• cites Shevlane & Dafoe, 2020 sourceNot checked — no full text of the source

May combine two claims.

While closed foundation models can be better monitored for misuse, it is not clear if such uses will be reliably identified.

“while closed foundation models can be better monitored for misuse, it is not clear if such uses will be reliability identified.”

• AssumptionThe authors state this as an assumption: Defenders will continue to have better access to state-of-the-art vulnerability detection tools, including those based on open foundation models.
• AssumptionThe authors state this as an assumption: Defenders will invest in using these tools to update their information security practices.
• AssumptionThe authors state this as an assumption: The offense-defense balance will not change dramatically as model capabilities improve.

Probably not a substantial claim: may not pass S1 (asserted).

Open foundation models present risks due to the inability to monitor, moderate, or revoke access.

“Technologists and policymakers have worried that open foundation models present risks, in particular, due to the inability to monitor, moderate, or revoke access.”

May combine two claims.

Broad access to model weights and greater customizability enable greater competition in downstream markets, helping reduce market concentration at the foundation model level from vertical cascading.

“Broad access to model weights and greater customizability further enable greater competition in downstream markets, helping to reduce market concentration at the foundation model level from vertical cascading.”

May combine two claims.

Model weights do not guarantee transparency on upstream resources used to build the foundation model nor on downstream impact.

“model weights do not guarantee transparency on the upstream resources used to build the foundation model (e.g.data sources, labor practices, energy expenditure) nor transparency on the downstream impact of the foundation model”

May combine two claims.

There are several legislative proposals to penalize the creation and distribution of digitally altered non-consensual intimate imagery, though their efficacy remains to be seen because channels for spreading such imagery can be anonymous or end-to-end encrypted.

“There are several legislative proposals to penalize the creation and distribution of digitally altered NCII, though given that channels for the spread of NCII can be anonymous or end-to-end encrypted, the efficacy of such legislation remains to be seen (Illinois General Assembly, 2023; Saliba, 2023;Reid, 2020;Kocsis, 2021;Hao, 2021;Siddique, 2023).”

• cites Illinois General Assembly, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Saliba, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Reid, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Many benefits of open foundation models depend on their meaningfully expanding consumer choice and reducing costs.

“many benefits hinge on open foundation models meaningfully expanding consumer choice and reducing costs.”

Probably not a substantial claim: may not pass S4 (based).

Even if future models can have robust technical safeguards, already-released models will continue to be misused.

“Even if future models can have robust technical safeguards, already-released models will continue to be misused.”

Probably not a substantial claim: may not pass S4 (based).

Compared with using tools like Photoshop, once a fine-tuned model is available, it is much easier for nonexperts to use these tools.

“Compared to using tools like Photoshop, once such a fine-tuned model is made available, it is much easier for nonexperts to use these tools.”

Probably not a substantial claim: may not pass S4 (based).

Even if downstream providers take steps to moderate digitally altered non-consensual intimate imagery, misuse can happen on smaller anonymous or end-to-end encrypted platforms or on the dark web by malicious users.

“Even if downstream providers take steps to moderate digitally altered NCII, misuse can happen on smaller (anonymous/end-to-end encrypted) platforms or on the dark web by malicious users.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

It is not clear how safety duties are split between an open model's developer and its user, and no settled norms exist.

“the division of responsibility for safety between the developer and user of an open foundation model is unclear and lacks established norms.”

May combine two claims.

Incomplete assessments do not necessarily indicate that prior studies are flawed, only that these studies alone are insufficient evidence to demonstrate increased marginal societal risk from open foundation models.

“incomplete assessments do not necessarily indicate that the analysis in prior studies is flawed, only that these studies, on their own, are insufficient evidence to demonstrate increased marginal societal risk from open foundation models.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Controversy around open foundation models stems from deep philosophical disagreement, disjointed concepts, and weak empirical evidence.

“Open foundation models are controversial due to fundamental philosophical disagreements, fragmented conceptual understanding, and poor empirical evidence.”

May combine two claims.

The risk assessment framework cannot by itself make release decisions because it does not weigh marginal benefits of open release against marginal risk or consider the opportunity cost of not releasing openly.

“it is not a complete framework for making release decisions since it does not provide a mechanism for trading the marginal benefits of openly releasing models against the marginal risk, nor does it look at the opportunity cost of not releasing a model openly.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The theoretical benefits of open foundation models include potential to catalyze innovation, distribute power, and foster competition.

“Significant theoretical benefits of open foundation models relate to their potential to catalyze innovation, distribute power, and foster competition.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

While developers of closed foundation models can enforce guardrails on text-to-image models for creating non-consensual intimate imagery, such guardrails on open foundation models can be easily circumvented.

“While developers of closed FMs can enforce guardrails on the use of their text-to-image models for creating NCII, such guardrails on open FMs can be easily circumvented.”

Classified as probably not a substantial claim.

Open foundation models have less clear safety expectations than closed foundation models.

“open foundation models have less clear safety expectations.”

May combine two claims.

Access to information may not be a major barrier to biosecurity attacks, and stronger interventions may lie downstream.

“access to information might not be a major barrier for carrying out such attacks-stronger interventions might lie downstream”

• cites Batalis, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The risk assessment framework can identify coordination opportunities but does not itself produce coordination among actors needed to reduce risks such as NCII.

“While the framework allows us to identify such opportunities, it does not automatically bring about the coordination of these actors.”

• AssumptionRests on an assumption: To reduce NCII risk, open model developers need to coordinate with social media and other downstream platforms.

Probably not a substantial claim: may not pass S4 (based).

If model quality differences are more direct causes of adoption of specific foundation models, the claimed benefits of open foundation models may not appear.

“If factors such as differences in model quality are the more direct causes of the adoption of specific foundation models, these purported benefits may not manifest.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Regardless of whether the model used for automated vulnerability detection is open or proprietary, signals and the ability to analyze them at machine scale and speed differentially support defenders because of better access to systems.

“Regardless of whether the model used for automated vulnerability detection is open or proprietary, signals and the ability to analyze them at machine scale and speed differentially supports defenders because of better access to the systems.”

• AssumptionThe authors state this as an assumption: Defenders will continue to have better access to state-of-the-art vulnerability detection tools, including those based on open foundation models.
• AssumptionThe authors state this as an assumption: Defenders will invest in using these tools to update their information security practices.
• AssumptionThe authors state this as an assumption: The offense-defense balance will not change dramatically as model capabilities improve.

Probably not a substantial claim: may not pass S4 (based).

Technical solutions for curtailing the use of already existing models to create non-consensual intimate imagery are hard or impossible.

“Technical solutions for curtailing the use of already existing models to create NCII are hard or impossible.”