Regularly updated benchmark sets for statistically correct evaluations of AlphaFold applications
Open accessPublished version of a preprint
Abstract
AlphaFold2 changed structural biology by providing high-quality structure predictions for all possible proteins. Since its inception, a plethora of applications were built on AlphaFold2, expediting discoveries in virtually all areas related to protein science. In many cases, however, optimism seems to have made scientists forget about data leakage, a serious issue that needs to be addressed when evaluating machine learning methods. Here we provide a rigorous benchmark set that can be used in a broad range of applications built around AlphaFold2/3.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
Accurate structure prediction of biomolecular interactions with AlphaFold 3
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. (2024) · Nature · doi:10.1038/s41586-024-07487-w · Cited by 15,861 · see citing works
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,523 · see citing works
Fast and accurate protein structure search with Foldseek
van Kempen, M., Kim, S. S., Tumescheit, C., Mirdita, M., Lee, J., Gilchrist, C. L. M., Söding, J., & Steinegger, M. (2022) · doi:10.1101/2022.02.07.479398 · Cited by 2,598 · see citing works
Fast and accurate protein structure search with Foldseek
van Kempen, M., Kim, S. S., Tumescheit, C., Mirdita, M., Lee, J., Gilchrist, C. L. M., Söding, J., & Steinegger, M. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01773-0 · Cited by 2,596 · see citing works
Using AlphaFold to predict the impact of single mutations on protein stability and function
Pak, M. A., Markhieva, K. A., Novikova, M. S., Petrov, D. S., Vorobyev, I. S., Maksimova, E. S., Kondrashov, F. A., & Ivankov, D. N. (2023) · PLOS One · doi:10.1371/journal.pone.0282689 · Cited by 322 · cited 1× · see citing works
“…AF2 structures to predict and/or assess various structural and physicochemical features of proteins in specific areas, such as protein stability [4], conformational heterogeneity [5], protein function [6], structure, [7] or interfaces [8] of protein complexes, domain definitions [9], or…”
SPEACH_AF: Sampling protein ensembles and conformational heterogeneity with Alphafold2
Stein, R. A., & Mchaourab, H. S. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1010483 · Cited by 258 · see citing works
Systematic discovery of protein interaction interfaces using AlphaFold and experimental validation
Lee, C. Y., Hubrich, D., Varga, J. K., Schäfer, C., Welzel, M., Schumbera, E., Djokic, M., Strom, J. M., Schönfeld, J., Geist, J. L., et al. (2024) · Molecular Systems Biology · doi:10.1038/s44320-023-00005-6 · Cited by 105 · cited 1× · see citing works
“…and physicochemical features of proteins in specific areas, such as protein stability [4], conformational heterogeneity [5], protein function [6], structure, [7] or interfaces [8] of protein complexes, domain definitions [9], or disordered regions that do not fold into unique structures [10].”
Systematic discovery of protein interaction interfaces using AlphaFold and experimental validation
Lee, C. Y., Hubrich, D., Varga, J. K., Schäfer, C., Welzel, M., Schumbera, E., Đokić, M., Strom, J. M., Schönfeld, J., Geist, J. L., et al. (2023) · doi:10.1101/2023.08.07.552219 · Cited by 105 · see citing works
AlphaFold2 reveals commonalities and novelties in protein structure space for 21 model organisms
Bordin, N., Sillitoe, I., Nallapareddy, V., Rauer, C., Lam, S. D., Waman, V. P., Sen, N., Heinzinger, M., Littmann, M., Kim, S., et al. (2023) · Communications Biology · doi:10.1038/s42003-023-04488-9 · Cited by 103 · see citing works
Conte, A. D., Mehdiabadi, M., Bouhraoua, A., Miguel Monzon, A., Tosatto, S. C. E., & Piovesan, D. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26582 · Cited by 86 · see citing works
Flagged first, then by citations
- Authors
- Dobson, L., Tusnády, G. E., & Tompa, P.
- Affiliations
- Institute of Molecular Life Sciences Research, Centre for Natural Sciences, Magyar Tudósok Körútja , Budapest ,
- Semmelweis University (HU)
- Venue
- Briefings in Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-03
- Volume
- 26
- Issue
- 2
- ISSN
- 1467-5463 (ISSN-L)
- 1477-4054
- PMC
- PMC11894802
- OpenAlex
- W4408333405
- Cited by
- 6
- References
- 11
- Citations by year
- 2026: 4 · 2025: 2
- FWCI
- 1.79
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Briefings in Bioinformatics (publishedVersion, cc-by-nc)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- NKFIH (K-146314, PD-146564)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Physical Sciences › Materials Science › Materials Chemistry › Enzyme Structure and Function
- Keywords
- AlphaFold2, AlphaFold3, protein structure prediction, benchmark set, data leakage
- MeSH terms
- Machine Learning, Algorithms, Models, Molecular, Protein Conformation, Proteins, Software, Protein Folding, Computational Biology, Benchmarking, Databases, Protein
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
