CoChat
Sign Up
PreprintCited by 335Protein Structure and DynamicsMachine Learning in BioinformaticsEnzyme Structure and Function

The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.

Benchmarking AlphaFold for protein complex modeling reveals accuracy determinants

Open access
Yin, Rui, Feng, Brandon Y., Varshney, Amitabh, Pierce, Brian G.·2021·Protein Science
DOI 10.1101/2021.10.23.465575PMC PMC9278006

Preprint — published as 10.1002/pro.4379 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

High resolution experimental structural determination of protein-protein interactions has led to valuable mechanistic insights, yet due to the massive number of interactions and experimental limitations there is a need for computational methods that can accurately model their structures. Here we explore the use of the recently developed deep learning method, AlphaFold, to predict structures of protein complexes from sequence. With a benchmark of 152 diverse heterodimeric protein complexes, multiple implementations and parameters of AlphaFold were tested for accuracy. Remarkably, many cases had highly accurate models generated as top-ranked predictions, greatly surpassing the performance of unbound protein-protein docking, whereas antibody-antigen docking was largely unsuccessful. While AlphaFold-generated accuracy predictions were able to discriminate near-native models, previously developed scoring protocols improved performance. Our study demonstrates that end-to-end deep learning can accurately model transient protein complexes, and identifies areas for improvement to guide future developments to reliably model any protein-protein interaction of interest.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2021.10.23.465575

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Figures

FIGURE 1 Transient protein–protein complex structure prediction success by AlphaFold, ColabFold and ZDOCK. End‐to‐end modeling using AlphaFold 24 and ColabFold 29 was performed on 152 complex test cases (details in Figure S1). AlphaFold failed to generate predictions for three complexes, thus AlphaFold predictions were obtained for 149 complexes; these 149 test cases were used to calculate success rates in this figure. Docking models were also generated with ZDOCK, 33 using unbound protein structures as input. All sets of models were assessed for near‐native predictions using CAPRI criteria for high, medium, and acceptable accuracy. (a) Complex prediction success of AlphaFold, ColabFold, and ZDOCK for the top 1 (T1) and top 5 (T5) models considered. AlphaFold and ColabFold models were ranked by AlphaFold pTM scores, and ZDOCK models were ranked by IRAD scores. 39 The percent success was calculated as the percentage of test cases with a given model accuracy from the top N models considered. Bars are colored according to the CAPRI quality classes. (b) Example of an accurately predicted complex structure (PDB code: 2X9A) by AlphaFold. This model has high accuracy by CAPRI criteria (I‐RMSD = 0.47) and has the highest pTM score (pTM = 0.77) of all five models generated for this complex. Structures are superposed by Phage G3P, with the model and the X‐ray structure chains are colored separately as indicated. For clarity, regions modeled by AlphaFold but unresolved in the X‐ray structure are not shown in the figure
FIGURE 1 Transient protein–protein complex structure prediction success by AlphaFold, ColabFold and ZDOCK. End‐to‐end modeling using AlphaFold 24 and ColabFold 29 was performed on 152 complex test cases (details in Figure S1). AlphaFold failed to generate predictions for three complexes, thus AlphaFold predictions were obtained for 149 complexes; these 149 test cases were used to calculate success rates in this figure. Docking models were also generated with ZDOCK, 33 using unbound protein structures as input. All sets of models were assessed for near‐native predictions using CAPRI criteria for high, medium, and acceptable accuracy. (a) Complex prediction success of AlphaFold, ColabFold, and ZDOCK for the top 1 (T1) and top 5 (T5) models considered. AlphaFold and ColabFold models were ranked by AlphaFold pTM scores, and ZDOCK models were ranked by IRAD scores. 39 The percent success was calculated as the percentage of test cases with a given model accuracy from the top N models considered. Bars are colored according to the CAPRI quality classes. (b) Example of an accurately predicted complex structure (PDB code: 2X9A) by AlphaFold. This model has high accuracy by CAPRI criteria (I‐RMSD = 0.47) and has the highest pTM score (pTM = 0.77) of all five models generated for this complex. Structures are superposed by Phage G3P, with the model and the X‐ray structure chains are colored separately as indicated. For clarity, regions modeled by AlphaFold but unresolved in the X‐ray structure are not shown in the figureCC BY · PubMed Central
FIGURE 2 Determinants of successful performance. (a) Prediction success of AlphaFold and ZDOCK, grouped by docking difficulty. Based on binding conformational changes as defined by BM5.5, 20 , 23 cases are categorized into “rigid,” “medium,” and “difficult” docking difficulty levels. To evaluate the success rate, all five models from AlphaFold and top five ZDOCK models were considered. (b) Prediction success of AlphaFold and ZDOCK grouped by complex category. To evaluate the success rate, all five models from AlphaFold and top 25 ZDOCK models were considered. The number of ZDOCK models was increased to 25 to allow for sufficient success rates to show its relative performance among the categories. (c) Prediction success of AlphaFold and ZDOCK grouped by protein source organism(s). To evaluate the success rate, all five models from AlphaFold and top 25 ZDOCK models were considered. Based on the source of subunit proteins in the complex structures, each case is classified as either “single, eukaryotic” (SE, denoting proteins from the same eukaryotic organism), “single, bacterial” (SB, denoting proteins from the same bacterial organism), “multiple, eukaryotic” (ME, denoting proteins from different eukaryotic organisms), or “multiple, mix” (MM, denoting proteins from mixed origins). Two additional protein source classes, corresponding to proteins from different bacterial organisms, and proteins of viral origin, were omitted from the success plot due to limited representation in each category (four and two cases in those classes, respectively). Bars are colored by model accuracy as indicated in (a). The horizontal stacked bars below each success rate plot denote the composition of the categories by class
FIGURE 2 Determinants of successful performance. (a) Prediction success of AlphaFold and ZDOCK, grouped by docking difficulty. Based on binding conformational changes as defined by BM5.5, 20 , 23 cases are categorized into “rigid,” “medium,” and “difficult” docking difficulty levels. To evaluate the success rate, all five models from AlphaFold and top five ZDOCK models were considered. (b) Prediction success of AlphaFold and ZDOCK grouped by complex category. To evaluate the success rate, all five models from AlphaFold and top 25 ZDOCK models were considered. The number of ZDOCK models was increased to 25 to allow for sufficient success rates to show its relative performance among the categories. (c) Prediction success of AlphaFold and ZDOCK grouped by protein source organism(s). To evaluate the success rate, all five models from AlphaFold and top 25 ZDOCK models were considered. Based on the source of subunit proteins in the complex structures, each case is classified as either “single, eukaryotic” (SE, denoting proteins from the same eukaryotic organism), “single, bacterial” (SB, denoting proteins from the same bacterial organism), “multiple, eukaryotic” (ME, denoting proteins from different eukaryotic organisms), or “multiple, mix” (MM, denoting proteins from mixed origins). Two additional protein source classes, corresponding to proteins from different bacterial organisms, and proteins of viral origin, were omitted from the success plot due to limited representation in each category (four and two cases in those classes, respectively). Bars are colored by model accuracy as indicated in (a). The horizontal stacked bars below each success rate plot denote the composition of the categories by classCC BY · PubMed Central
FIGURE 3 Assessing test case features associated with AlphaFold success. Protein complex and MSA feature values were computed for all cases, which are shown according to AlphaFold success (best AlphaFold model accuracy in the five models for that case). Features shown are interface buried surface area (BSA), MSA depth (N eff) for the ligand or receptor (minimum value of the two), total number of residues, and percent of total residues in the protein–protein interface. Statistical significance values (Wilcoxon rank‐sum test) were calculated between feature values for sets of cases with incorrect versus medium and incorrect versus high‐ CAPRI accuracy, as noted at top (ns: p > .05, *p ≤ .05, **p ≤ .01, ***p ≤ .001)
FIGURE 3 Assessing test case features associated with AlphaFold success. Protein complex and MSA feature values were computed for all cases, which are shown according to AlphaFold success (best AlphaFold model accuracy in the five models for that case). Features shown are interface buried surface area (BSA), MSA depth (N eff) for the ligand or receptor (minimum value of the two), total number of residues, and percent of total residues in the protein–protein interface. Statistical significance values (Wilcoxon rank‐sum test) were calculated between feature values for sets of cases with incorrect versus medium and incorrect versus high‐ CAPRI accuracy, as noted at top (ns: p > .05, *p ≤ .05, **p ≤ .01, ***p ≤ .001)CC BY · PubMed Central
FIGURE 4 The impact of MSA pairing on prediction accuracy. MSAs were generated using MMseqs2 using the “advanced” interface of ColabFold. 29 Pairing was performed in ColabFold on a total of 17 cases whose ligand and receptor proteins come from the same prokaryotic organism. Cases in the heatmap were sorted by the paired MSA depth (N eff; see Material and Methods for details) from the largest to the smallest values. Structural predictions were generated with the “advanced” interface of ColabFold, and RoseTTAFold 28 (through the Robetta server). All models were assessed for near‐native predictions within the top‐ranked (T1) and top 5 (T5) models using CAPRI criteria. Complex category: Enzyme‐inhibitor (EI), enzyme complex with a regulatory or accessory chain (ER), enzyme‐substrate (ES), others, receptor‐containing (OR); others, miscellaneous (OX)
FIGURE 4 The impact of MSA pairing on prediction accuracy. MSAs were generated using MMseqs2 using the “advanced” interface of ColabFold. 29 Pairing was performed in ColabFold on a total of 17 cases whose ligand and receptor proteins come from the same prokaryotic organism. Cases in the heatmap were sorted by the paired MSA depth (N eff; see Material and Methods for details) from the largest to the smallest values. Structural predictions were generated with the “advanced” interface of ColabFold, and RoseTTAFold 28 (through the Robetta server). All models were assessed for near‐native predictions within the top‐ranked (T1) and top 5 (T5) models using CAPRI criteria. Complex category: Enzyme‐inhibitor (EI), enzyme complex with a regulatory or accessory chain (ER), enzyme‐substrate (ES), others, receptor‐containing (OR); others, miscellaneous (OX)CC BY · PubMed Central
FIGURE 5 Association between AlphaFold predicted scores and docking model quality. (a) Average pLDDT and pTM per CAPRI criteria. Statistical significance (Wilcoxon rank‐sum test) between average pLDDT or pTM of incorrect versus acceptable, incorrect versus medium and incorrect versus high CAPRI criteria is indicated at the top (ns: p > .05, *p ≤ .05, **p ≤ .01, ***p ≤ .001). (b) Comparisons between pTM and calculated TM‐score and (c) between pTM and I‐RMSD are shown as scatter plots. All 5 models for 149 cases are shown as points, colored by model quality by CAPRI criteria. Linear regression is shown along with the 95% confidence interval (orange area), and Pearson's correlation coefficients and correlation p‐values are denoted in (b) and (c). In (c), the dashed green line indicates a possible pTM score cutoff (pTM = 0.8) for selection of accurate docking models, based on optimization of sensitivity and specificity for incorrect versus medium and high model discrimination
FIGURE 5 Association between AlphaFold predicted scores and docking model quality. (a) Average pLDDT and pTM per CAPRI criteria. Statistical significance (Wilcoxon rank‐sum test) between average pLDDT or pTM of incorrect versus acceptable, incorrect versus medium and incorrect versus high CAPRI criteria is indicated at the top (ns: p > .05, *p ≤ .05, **p ≤ .01, ***p ≤ .001). (b) Comparisons between pTM and calculated TM‐score and (c) between pTM and I‐RMSD are shown as scatter plots. All 5 models for 149 cases are shown as points, colored by model quality by CAPRI criteria. Linear regression is shown along with the 95% confidence interval (orange area), and Pearson's correlation coefficients and correlation p‐values are denoted in (b) and (c). In (c), the dashed green line indicates a possible pTM score cutoff (pTM = 0.8) for selection of accurate docking models, based on optimization of sensitivity and specificity for incorrect versus medium and high model discriminationCC BY · PubMed Central
FIGURE 6 Association between alternative scoring metrics and docking model quality. (a) Distributions of interface pLDDT (4 Å), IRAD, Rosetta cross‐interface binding energy, and number of interface residues for AlphaFold models grouped by CAPRI criteria. An interface pLDDT score of 0 was assigned to models without any interface contacts within the distance cutoff (4 Å). Constrained local minimization was performed using Rosetta FastRelax 65 to resolve unfavorable local geometries or clashes in models, and post‐relaxation models were scored with IRAD and the Rosetta 45 InterfaceAnalyzer protocol, with the latter used to calculate cross‐interface binding energy scores (based on the Rosetta REF15 energy function 66 ) and the number of interface residues. Statistical significance values (Wilcoxon rank‐sum test) between scores of incorrect versus acceptable, incorrect versus medium, incorrect versus high CAPRI criteria are indicated at the top of each plot (***p ≤ .001). Each point corresponds to one AlphaFold model, and all five AlphaFold models for 149 test cases are represented. (b) ROC curves among the scoring metrics for classifying incorrect versus high accuracy models by CAPRI criteria, with corresponding AUC values denoted in parentheses
FIGURE 6 Association between alternative scoring metrics and docking model quality. (a) Distributions of interface pLDDT (4 Å), IRAD, Rosetta cross‐interface binding energy, and number of interface residues for AlphaFold models grouped by CAPRI criteria. An interface pLDDT score of 0 was assigned to models without any interface contacts within the distance cutoff (4 Å). Constrained local minimization was performed using Rosetta FastRelax 65 to resolve unfavorable local geometries or clashes in models, and post‐relaxation models were scored with IRAD and the Rosetta 45 InterfaceAnalyzer protocol, with the latter used to calculate cross‐interface binding energy scores (based on the Rosetta REF15 energy function 66 ) and the number of interface residues. Statistical significance values (Wilcoxon rank‐sum test) between scores of incorrect versus acceptable, incorrect versus medium, incorrect versus high CAPRI criteria are indicated at the top of each plot (***p ≤ .001). Each point corresponds to one AlphaFold model, and all five AlphaFold models for 149 test cases are represented. (b) ROC curves among the scoring metrics for classifying incorrect versus high accuracy models by CAPRI criteria, with corresponding AUC values denoted in parenthesesCC BY · PubMed Central
FIGURE 7 Examples of antibody–antigen complex structure predictions by AlphaFold. (a) Native and top‐ranked AlphaFold model (pTM = 0.78) for PDB 4NIK (F5 antibody/human gankyrin complex). This model is of medium accuracy by CAPRI criteria (I‐RMSD = 1.52 Å). Modeled and X‐ray complex structures are colored as indicated and shown superposed by gankryin. Unresolved regions modeled by AlphaFold are not shown. (b) Native and top‐ranked AlphaFold model (pTM = 0.61) for PDB 6OAN (053054 antibody/P vivax DBP complex). This model is of medium accuracy by CAPRI criteria (I‐RMSD = 2.12 Å). Modeled and X‐ray structures are colored as indicated, shown superposed by DBP, and unresolved regions modeled by AlphaFold are not shown. (c) Native and top‐ranked AlphaFold model (pTM = 0.66) for PDB 3RJQ (A12 nanobody/HIV C186 gp120 complex), superposed by C186 gp120. This AlphaFold model does not have contacting residues between the proteins within a 5 Å distance cutoff. Structures are colored as indicated in the figure, and unresolved regions modeled by AlphaFold on C186 gp120 are shown in light green
FIGURE 7 Examples of antibody–antigen complex structure predictions by AlphaFold. (a) Native and top‐ranked AlphaFold model (pTM = 0.78) for PDB 4NIK (F5 antibody/human gankyrin complex). This model is of medium accuracy by CAPRI criteria (I‐RMSD = 1.52 Å). Modeled and X‐ray complex structures are colored as indicated and shown superposed by gankryin. Unresolved regions modeled by AlphaFold are not shown. (b) Native and top‐ranked AlphaFold model (pTM = 0.61) for PDB 6OAN (053054 antibody/P vivax DBP complex). This model is of medium accuracy by CAPRI criteria (I‐RMSD = 2.12 Å). Modeled and X‐ray structures are colored as indicated, shown superposed by DBP, and unresolved regions modeled by AlphaFold are not shown. (c) Native and top‐ranked AlphaFold model (pTM = 0.66) for PDB 3RJQ (A12 nanobody/HIV C186 gp120 complex), superposed by C186 gp120. This AlphaFold model does not have contacting residues between the proteins within a 5 Å distance cutoff. Structures are colored as indicated in the figure, and unresolved regions modeled by AlphaFold on C186 gp120 are shown in light greenCC BY · PubMed Central
FIGURE 8 AlphaFold‐Multimer modeling success for an expanded set of recently released antibody–antigen complex structures. A benchmark set of 100 recently released antibody–antigen complex structures was modeled with AlphaFold‐Multimer, and all five models from AlphaFold‐Multimer per test case were assessed for accuracy using CAPRI criteria. AlphaFold‐Multimer was run in ColabFold with MSA input and with the use of structural templates released before April 30, 2018, and models were ranked by pTM score. Success for top 1 and top 5 (T1, T5) ranked predictions is shown, colored by CAPRI model accuracy as indicated in the key on right (Hits). “Type” distinguishes complexes containing heavy‐light chain antibodies (“Antibody”) and single‐chain nanobodies (“Nanobody”).
FIGURE 8 AlphaFold‐Multimer modeling success for an expanded set of recently released antibody–antigen complex structures. A benchmark set of 100 recently released antibody–antigen complex structures was modeled with AlphaFold‐Multimer, and all five models from AlphaFold‐Multimer per test case were assessed for accuracy using CAPRI criteria. AlphaFold‐Multimer was run in ColabFold with MSA input and with the use of structural templates released before April 30, 2018, and models were ranked by pTM score. Success for top 1 and top 5 (T1, T5) ranked predictions is shown, colored by CAPRI model accuracy as indicated in the key on right (Hits). “Type” distinguishes complexes containing heavy‐light chain antibodies (“Antibody”) and single‐chain nanobodies (“Nanobody”).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 80 references

Research works printed but not linked: 3

  1. 10.1126/science.abj8754 · DOI · Paper page
  2. Guest JD , Vreven T , Zhou J , Moal I , Jeliazkov JR , Gray JJ , et al. An expanded benchmark for antibody-antigen docking and affinity prediction reveals insights into antibody recognition determinants. Structure. 2021. · DOI
  3. 10.1002/pro.687 · DOI

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Yin, R., Feng, B. Y., Varshney, A., & Pierce, B. G.
Affiliations
  • Institute for Bioscience and Biotechnology Research (US)
  • University of Maryland, College Park (US)
  • University of Maryland, Baltimore (US)
Corresponding author
Pierce, Brian G.
ORCID
Venue
Protein Science
Publisher
openRxiv
Published
2021-10-24
Volume
31
Issue
8
Pages
e4379-e4379
ISSN
0961-8368 (ISSN-L)
DOI
10.1101/2021.10.23.465575
PMC
PMC9278006
OpenAlex
W4285392187
Cited by
335
References
80
Citations by year
2026: 47 · 2025: 67 · 2024: 111 · 2023: 90 · 2022: 19
FWCI
23.42
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Protein Science (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University Libraries (University of Maryland) (submittedVersion)
  • Digital Repository at the University of Maryland (University of Maryland College Park)
Funders
  • National Institutes of Health (NIH-R01)
  • National Institute of General Medical Sciences (R01GM126299, R35GM144083)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Structural Biology › Enzyme Structure and Function
Keywords
AlphaFold, AlphaFold-Multimer, protein complex structure prediction, protein-protein interaction, immune complexes, protein-protein docking, benchmarking, multiple sequence alignment, deep learning
MeSH terms
Algorithms, Protein Binding, Protein Conformation, Proteins, Sequence Alignment, Molecular Docking Simulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review