Can large language models democratize access to dual-use biotechnology?
Open accessThe DOI matches its OpenAlex record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) such as those embedded in 'chatbots' are accelerating and democratizing research by providing comprehensible information and expertise from many different fields. However, these models may also confer easy access to dual-use technologies capable of inflicting great harm. To evaluate this risk, the 'Safeguarding the Future' course at MIT tasked non-scientist students with investigating whether LLM chatbots could be prompted to assist non-experts in causing a pandemic. In one hour, the chatbots suggested four potential pandemic pathogens, explained how they can be generated from synthetic DNA using reverse genetics, supplied the names of DNA synthesis companies unlikely to screen orders, identified detailed protocols and how to troubleshoot them, and recommended that anyone lacking the skills to perform reverse genetics engage a core facility or contract research organization. Collectively, these results suggest that LLMs will make pandemic-class agents widely accessible as soon as they are credibly identified, even to people with little or no laboratory training. Promising nonproliferation measures include pre-release evaluations of LLMs by third parties, curating training datasets to remove harmful concepts, and verifiably screening all DNA generated by synthesis providers or used by contract research organizations and robotic cloud laboratories to engineer organisms or viruses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Nipah Virus: Past Outbreaks and Future Containment
Soman Pillai, V., Krishna, G., & Valiya Veettil, M. (2020) · Viruses · doi:10.3390/v12040465 · Cited by 194
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2302.05733 · Cited by 114
Primate hemorrhagic fever-causing arteriviruses are poised for spillover to humans
Warren, C. J., Yu, S., Peters, D. K., Barbachano-Guerrero, A., Yang, Q., Burris, B. L., Worwa, G., Huang, I.-C., Wilkerson, G. K., Goldberg, T. L., et al. (2022) · Cell · doi:10.1016/j.cell.2022.09.022 · Cited by 42
Reverse genetics of influenza viruses
Neumann, G., Ozawa, M., & Kawaoka, Y. (2012) · Methods in Molecular Biology · doi:10.1007/978-1-61779-621-0_12 · Cited by 24
Plasmid-Based Reverse Genetics of Influenza A Virus
Perez, D. R., Seibert, B., Ferreri, L., Lee, C.-W., & Rajao, D. (2020) · Methods in Molecular Biology · doi:10.1007/978-1-0716-0346-8_4 · Cited by 19
The Quest to Overcome Gene Therapy’s Failures
Lewis, T. (2021) · Nature · doi:10.1038/d41586-021-02734-w · Cited by 2
Cited by
AI Deception: A Survey of Examples, Risks, and Potential Solutions
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2308.14752 · Cited by 243
Artificial intelligence risk management framework :
National Institute of Standards and Technology (US) (2024) · NIST Trustworthy and Responsible AI · doi:10.6028/nist.ai.600-1 · Cited by 274
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84
Future Shock: Generative AI and the International AI Policy and Governance Crisis
Leslie, D., & Perini, A. M. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.88b4cc98 · Cited by 33
Metadata
- Authors
- Soice, E. H., Rocha, R., Cordova, K., Specter, M., & Esvelt, K. M.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.03809
- OpenAlex
- W4379925078
- Cited by
- 25
- References
- 6
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Can large language models democratize access to dual-use biotechnology? https://app.cochat.ai/papers/10.48550/arxiv.2306.03809
- The paper
- Soice, E. H., Rocha, R., Cordova, K., Specter, M., & Esvelt, K. M. (2023). Can large language models democratize access to dual-use biotechnology? arXiv (Cornell University). https://doi.org/10.48550/arxiv.2306.03809
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.