Improved prediction of protein-protein interactions using AlphaFold2

May combine two claims.

AlphaFold2 can predict the structure of many heterodimeric protein complexes, although it is trained to predict the structure of individual protein chains.

“Here we show that AlphaFold216 (AF2) can predict the structure of many heterodimeric protein complexes, although it is trained to predict the structure of individual protein chains.”

• cites [16] 2021, Highly accurate protein structure prediction with AlphaFold, Nature sourceSupported

Classified as probably not a substantial claim.

The test set used 1661 protein complexes with known interfaces from a recent study.

“We used 1661 protein complexes with known interfaces from a recent study27 to test the developed pipeline.”

• cites [27] sourceCould not verify

  The Results section on grid cell representations in familiar environments reports recorded neuronal units, not protein complexes. Searches for “1661 protein complexes with known interfaces” and “protein complexes interface pipeline test set” found no passage in our copy addressing the claimed test set. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

May combine two claims.

No additional constraints are implemented in AF2 to pull two chains in contact, meaning that chain interactions and interface sizes are exclusively determined by the amount of inter-chain signals extracted by the predictor.

“Interestingly, no additional constraints are implemented in AF2 to pull two chains in contact, meaning that chain interactions (and subsequently interface sizes) are exclusively determined by the amount of inter-chain signals extracted by the predictor.”

• cites [16] 2021, Highly accurate protein structure prediction with AlphaFold, Nature sourceCould not verify

  The “MSA depth and cross-chain contacts” section discusses how contacts with other chains affect prediction accuracy and shows a homotrimer predicted without input stoichiometry. It does not say whether AF2 applies constraints to bring chains together or whether inter-chain signals exclusively determine interactions and interface sizes. Searches for those constraints, signals, and interface size found no passage addressing that claim. (LLM, conf=0.73)

  Experimental — accuracy not yet measured

May combine two claims.

Performing model relaxation did not increase performance in the AF2 paper and was therefore ignored to save computational cost.

“We note that performing model relaxation did not increase performance in the AF2 paper16 and was, therefore, ignored to save computational cost.”

• cites [16] 2021, Highly accurate protein structure prediction with AlphaFold, Nature sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AF2 has been shown to perform well for single chains without templates and has reported higher accuracy than template-based methods even when robust templates are available.

“Further, AF2 has been shown to perform well for single chains without templates and has reported higher accuracy than template-based methods even when robust templates are available16.”

• cites [16] 2021, Highly accurate protein structure prediction with AlphaFold, Nature sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

As the bound form of the proteins is used, this should represent an easy case for GRAMM-based docking, and performance drops significantly when unbound structures or models are used.

“As the bound form of the proteins is used, this should represent an easy case for GRAMM-based docking, and performance drops significantly when unbound structures or models are used53.”

• cites [53] 2016, DockQ: A Quality Measure for Protein-Protein Docking Models, PLoS One sourceCould not verify

  The Materials and Methods (Training set; page unavailable) describes docking models generated with SwarmDock, while the Results and Discussion evaluates DockQ as a quality measure. Searches for GRAMM and bound-versus-unbound docking performance found no passage supporting the claimed performance drop. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

AF2 model_1_ptm is a fine-tuned version of model_1 that predicts the TMscore and alignment errors.

“The second model, model_1_ptm, is a fine-tuned version of model_1 that predicts the TMscore52 and alignment errors16.”

• cites [16] 2021, Highly accurate protein structure prediction with AlphaFold, Nature sourcePartly supported

  The Methods > Training regimen section says the authors fine-tuned five models to add a pTM prediction objective. End-to-end structure prediction says the TM-score estimate is obtained from a pairwise error prediction. Our copy does not identify the fine-tuned model specifically as model_1_ptm or establish its relationship to a model named model_1. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [52] 2018, Contact Potential for Structure Prediction of Proteins and Protein Complexes from Potts Model, Biophys. J sourceNot checked — no full text of the source

May combine two claims.

The RoseTTAFold pipeline for complex modelling only generates MSAs for bacterial protein complexes, while the proteins in the test set are mainly Eukaryotic.

“The RoseTTAFold pipeline for complex modelling only generates MSAs for bacterial protein complexes, while the proteins in our test set are mainly Eukaryotic.”

• cites [17] sourceCould not verify

  The ‘Direct generation of protein-protein complex models’ section says RoseTTAFold was tested on paired sequence alignments and describes a model of a human complex, but it does not say that its pipeline generates MSAs only for bacterial complexes. The source also cannot establish the composition of the citing authors’ test set. Searches for bacterial-only MSA generation and eukaryotic complexes did not resolve either point. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The atom-atom contact energy AACE is used to score and rank all poses, as this has been shown to provide better results than shape-complementarity alone.

“The atom-atom contact energy AACE18 is used to score and rank all poses, as this has been shown to provide better results than shape-complementarity alone54.”

• cites [18] sourceCould not verify

  The Introduction discusses predicting inter-residue contacts, and §2.4.2 describes ranking structural matches by TM-score. Neither addresses AACE, docking-pose ranking, or a comparison with shape complementarity. Searches for those terms in our copy found no supporting passage; page numbers were not available. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [54] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

The fraction of correctly modelled sequences increases with larger Neff scores.

“It is clear that the fraction of correctly modelled sequences increases with larger Neff scores (Fig. 3c).”

The quote only partly states this claim.

The AlphaFold2 protocol together with optimised multiple sequence alignments generates models of acceptable quality (DockQ ≥ 0.23) for 63% of heterodimeric dimers.

“We find that the AlphaFold2 protocol together with optimised multiple sequence alignments, generate models with acceptable quality (DockQ ≥ 0.23) for 63% of the dimers.”

• Also stated in: § Findings and future prospects · plain

  “Using optimised MSAs with AF2, we can accurately predict the structure of heterodimeric complexes for an unprecedented SR of 62.7% on a large test set.”

May combine two claims.

The highest success rate is obtained mainly for helix interfaces (62%), followed by interfaces containing mainly sheets (59%), while the loop interface success rate is 53%.

“The highest SR is obtained mainly for helix interfaces (62%), followed by interfaces containing mainly sheets (59%). The loop interface SR of 53% is substantially lower than the others, suggesting that interfaces with more flexible structures are harder to predict.”

May combine two claims.

The success rates by kingdom are Eukarya 61%, Bacteria 73.7%, Archaea 84.5%, and Virus 60%.

“The SRs for each kingdom is; Eukarya 61%, Bacteria 73.7%, Archaea 84.5%, and Virus 60% (Supplementary Fig. 1b).”

May combine two claims.

At FPR 5%, the number of interface contacts and residues report TPRs of 49% and 42%, respectively, compared to 43% for average interface plDDT and 66% for pDockQ.

“At FPR 5%, the number of interface contacts and residues report TPRs of 49 and 42%, respectively, compared to 43% for the average interface plDDT and 66% for pDockQ.”

• AssumptionRests on an assumption: The 3989 E. coli proteins used as negatives are truly non-interacting.
• AssumptionRests on an assumption: The 1481 test set proteins used as positives are truly interacting.

May combine two claims.

AF-multimer achieves the best performance on the test set, with a success rate of 72.2% and median DockQ of 0.560.

“The recently developed AF-multimer28 has the best performance (SR = 72.2%, median = 0.560, Table 2).”

• cites [28] sourceNot checked — no full text of the source

May combine two claims.

Between five different initialisations for proteins without templates, the average difference in DockQ score is 0.03, and there is no deviation in success rate, so ranking did not improve the success rate.

“Between the five different initialisations, the average difference in the DockQ score is 0.03, and there is no deviation in SR, i.e., ranking did not improve the SR.”

May combine two claims.

On the development set, the best success rate is 33.3% for AF2 MSAs and 39.4% for AF2+paired MSAs.

“The best performance is 33.3% for the AF2 MSAs and 39.4% for the AF2+ paired MSAs (Table 1).”

May combine two claims.

The average success rate is 57.2% ± 0.0% and is similar for all five runs with different random seeds.

“The average SR (57.2% ± 0.0%) is similar for all five runs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The pipeline presented here has been deemed FoldDock and has since been applied.

“The computational cost to run all of this is ~5 days on an Nvidia A100 system and has since the development of the pipeline presented here, deemed FoldDock, been applied37.”

• cites [37] sourceNot checked — no full text of the source

May combine two claims.

For the ranked test set, the AUC is 0.93, meaning 31% of all models are acceptable at an error rate of 1% and 54% at an error rate of 10%.

“The AUC using the same metric for the ranked test set is 0.93, which means that 31% of all models are acceptable at an error rate of 1% and 54% at an error rate of 10% (Supplementary Table 2).”

May combine two claims.

Running five initialisations with random seeds and ranking models using pDockQ increases the success rate to 61.7% for AF2+paired and 62.7% for block diagonalization+paired MSAs.

“Further, running five initialisations with random seeds and ranking the models using the predicted DockQ score (pDockQ, Fig. 2c), increases the SR to 61.7% and 62.7% for the AF2 + paired and block diagonalization+paired MSAs, respectively (model variation and ranking, Fig. 2).”

May combine two claims.

On the test set, the best outcome using m1-10-1 gives a success rate of 57.8% for AF2+paired MSAs compared with 45.0% for AF2 MSAs alone.

“The best outcome using this modelling strategy results in an SR of 57.8% (856 out of 1481 correctly modelled complexes) for the AF2 + paired MSAs compared with 45.0% using the AF2 MSAs alone (Fig. 1, Table 2).”

May combine two claims.

pDockQ predicts DockQ scores in a continuous manner with an overall average error of 0.11 on the test set.

“This enables the prediction of the DockQ scores (pDockQ) in a continuous manner with an overall average error of 0.11 on the test set.”

May combine two claims.

Pairs with larger interfaces are easier to predict, as the success rate increases from 47% to 74% between the smallest and biggest tertiles.

“We divide the dataset by interface size, and find that pairs with larger interfaces are easier to predict, as the SR increases from 47 to 74% between the smallest and biggest tertiles (Fig. 3b).”

• AssumptionThe authors state this as an assumption: All residues in an interface contribute to the interaction energy.

May combine two claims.

On the combined set of 1481 interacting and 5694 non-interacting proteins, the AUC is 0.82 for average interface plDDT, 0.84 for number of interface contacts, 0.85 for number of interface residues, and 0.87 for pDockQ.

“On this combined set of 1481 interacting and 5694 non-interacting proteins, we obtain an AUC of 0.82 for the average interface plDDT and slightly higher (0.84 and 0.85) for the number of interface contacts and residues, respectively (Fig. 5a). pDockQ results in an ROC curve with an AUC of 0.87.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Pairing took 2 s.

“It took 7884 s for generating the AF2 MSAs, the single-chain searches took 338 s on average and the pairing 2 s.”

May combine two claims.

AF2 MSAs generated by supplying a concatenated protein sequence of the entire complex mainly contain gaps for one of the two query proteins in each row, as only single chains can obtain hits in the searched databases.

“The resulting MSAs will thus mainly contain gaps for one of the two query proteins in each row, as only single chains can obtain hits in the searched databases (Fig. 6).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Together there are 5694 non-interacting protein complexes.

“Together there are 5694 non-interacting protein complexes.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Fusing the MSAs took 3 s on average per tested complex.

“Fusing the MSAs took 3 s on average per tested complex.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Generating the AF2 MSAs took 7884 s.

“It took 7884 s for generating the AF2 MSAs, the single-chain searches took 338 s on average and the pairing 2 s.”

May combine two claims.

The pairing and fusing are negligible compared to searching, resulting in a speedup of 24 times for the hhblits searches.

“The pairing and fusing are thereby negligible compared to searching, resulting in a speedup of 24 times for the hhblits searches.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Folding using the m1-10-1 strategy took 191 s on average for the tested pairs.

“In comparison, folding using the m1-10-1 strategy took 191 s on average for these pairs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Single-chain searches took 338 s on average.

“It took 7884 s for generating the AF2 MSAs, the single-chain searches took 338 s on average and the pairing 2 s.”

Classified as probably not a substantial claim.

pDockQ was created by fitting a sigmoidal curve to the DockQ scores using the average interface plDDT multiplied with the logarithm of the number of interface contacts.

“This score is created by fitting a sigmoidal curve (Fig. 2c) using “curve_fit” from SciPy v.1.4.156, to the DockQ scores using the average interface plDDT multiplied with the logarithm of the number of interface contacts, with the following sigmoidal equation:”

• cites [56] sourceNot checked — no full text of the source

May combine two claims.

AF-multimer was trained using the same data as the test set, making a direct comparison difficult.

“This method was trained using the same data as the test set, which makes a direct comparison difficult.”

• cites [28] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some test-set structures failed to be modelled, resulting in a total of 1481 structures.

“Some structures failed to be modelled for various reasons (see limitations of data generation), resulting in a total of 1481 structures.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

AF2 MSAs could not be generated for three development-set complexes (1gg2, 2nqd, 2xwb) due to memory limitations, so they were disregarded, leaving 216 complexes.

“AF2 MSAs could not be generated for three of the complexes due to memory limitations (1gg2, 2nqd and 2xwb) using a computational node with 128 Gb RAM for the MSA generation and were thus disregarded, resulting in a total of 216 complexes.”

Classified as probably not a substantial claim.

Unbound chains in the development set share at least 97% sequence identity with their bound counterparts.

“Unbound chains share at least 97% sequence identity with the bound counterpart and, to facilitate comparisons, non-matching residues are deleted and renumbered to become identical to the unbound counterpart.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There may be some unknown selection bias in how the development and test sets were chosen.

“This leads us to believe that there may be some unknown selection bias in how the sets were chosen.”

Classified as probably not a substantial claim.

50 test-set complexes have overlapping PDB codes with the development set and were excluded.

“107 of the complexes in the test set lack beta carbons (Cβs), and 50 have overlapping PDB codes with the development set and were therefore excluded.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The development set protein pairs share less than 30% sequence identity.

“This set contains protein pairs, with each chain having at least 50 residues, sharing <30% sequence identity and no crystal packing artefacts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The development set protein pairs each have at least 50 residues.

“This set contains protein pairs, with each chain having at least 50 residues, sharing <30% sequence identity and no crystal packing artefacts.”

May combine two claims.

Both single chains in a complex are often predicted very well while their relative orientation may still be incorrect.

“Interestingly, the average plDDT of the entire complex only results in an AUC of 0.66, suggesting that both single chains in a complex are often predicted very well, while their relative orientation may still be incorrect.”

Probably not a substantial claim: may not pass S3 (load-bearing).

107 test-set complexes lack beta carbons.

“107 of the complexes in the test set lack beta carbons (Cβs), and 50 have overlapping PDB codes with the development set and were therefore excluded.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The higher performance in S. cerevisiae compared to H. sapiens suggests a similar relationship between higher and lower order organisms within the same kingdom.

“The higher performance in S. cerevisiae compared to H. sapiens suggests a similar relationship between higher and lower order organisms within the same kingdom.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

When folding, three complexes report a ValueError, leading to a final set of 1481 complexes.

“When folding, three of these (5AWF_D-5AWF_B, 2ZXE_B-2ZXE_A and 2ZXE_A-2ZXE_G) report “ValueError: Cannot create a tensor proto whose content is larger than 2GB”, leading to a final set of 1481 complexes.”

May combine two claims.

The pDockQ score discriminates between both model quality and binary interactions.

“The pDockQ score discriminates between both model quality and binary interactions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The development set protein pairs have no crystal packing artefacts.

“This set contains protein pairs, with each chain having at least 50 residues, sharing <30% sequence identity and no crystal packing artefacts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study only considers the structures of protein complexes in their heterodimeric state; alternative oligomeric states and larger biological assemblies are outside its scope.

“Here, we only consider the structures of protein complexes in their heterodimeric state, although each protein chain in these complexes may have homodimer configurations or other higher-order states.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In MSA generation from AF2, 20 MSAs report MergeMasterSlave errors, resulting in a total of 1484 AF2 MSAs.

“In the MSA generation from AF2, 20 MSAs report MergeMasterSlave errors regarding discrepancies in the number of match states, resulting in a total of 1484 AF2 MSAs.”

May combine two claims.

MSA co-evolutionary signal and correct identification of orthologous protein sequences have a strong impact on the outcome.

“This suggests that MSA co-evolutionary signal and, thereby, correct identification of orthologous protein sequences, has a strong impact on the outcome.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The test set complexes have a resolution between 1 and 5 Å.

“These complexes share <30% sequence identity, have a resolution between 1–5 Å and constitute unique pairs of PFAM domains (no single protein pair have PFAM domains matching that of any other pair).”

May combine two claims.

The same pipeline can identify if two proteins interact and the accuracy of their structure.

“Therefore, the same pipeline can identify if two proteins interact and the accuracy of their structure.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For RF, 26 complexes produced out of memory exceptions and were excluded, leaving 1455 complexes.

“For RF, 26 complexes produced out of memory exceptions during prediction using a GPU with 40 Gb RAM and were excluded from the RF analyses, leaving 1455 complexes.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

DSSP could only be run successfully for 1391 out of the 1481 protein complexes, and the rest were ignored in the analysis.

“DSSP could only be run successfully for 1391 out of the 1481 protein complexes, and we ignored the rest in the analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the mammalian proteins from Negatome, after exclusions, 1715 complexes remained.

“For the mammalian proteins from Negatome, seven out of 1733 single chains were redundant according to Uniprot (C4ZQ83, I0LJR4, I0LL25, K4CRX6, P62988, Q8NI70, Q8T3B2), 34 had no matching species in the MSA pairing, 106 produced out of memory exceptions during prediction using a GPU with 40 Gb RAM, 35 gave a tensor reshape error, and 65 complexes were homodimers, leaving 1715 complexes for this set”

May combine two claims.

No templates were used to build structures, as this would not assess the prediction accuracy of unknown structures or structures without sufficient matching templates.

“No templates were used to build structures, as this would not assess the prediction accuracy of unknown structures or structures without sufficient matching templates.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some AlphaFold-multimer complexes failed due to computational limitations, resulting in 1458 out of 1481 complexes successfully folded.

“Some complexes failed due to computational limitations, resulting in 1458 out of 1481 complexes successfully folded.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

65 structures in the second non-interacting set are homodimers and were excluded, resulting in 1705 structures.

“Some structures in this dataset are homodimers (65) and are therefore excluded, resulting in 1705 structures.”

Classified as probably not a substantial claim.

RF was compared with AF2 using the same inputs (the paired MSAs) for both the development and test datasets to provide a more fair comparison.

“We compare RF with AF2 using the same inputs (the paired MSAs) for both the development and test datasets to provide a more fair comparison, as AF2 searches many different databases to obtain as much evolutionary information as possible when generating its MSAs.”

Classified as probably not a substantial claim.

Five models initiated with different seeds were generated for each complex to account for non-determinism in AF2 structure prediction.

“Since structure prediction with AF2 is a non-deterministic process, we generate five models initiated with different seeds.”

• AssumptionRests on an assumption: AF2 structure prediction is a non-deterministic process.

Probably not a substantial claim: may not pass S3 (load-bearing).

The number of clusters obtained by clustering sequences at 62% identity has been used to indicate a Neff value for each MSA.

“To estimate the information in each MSA, we clustered sequences at 62% identity, as described in a previous study50. The number of clusters obtained in this way has been used to indicate a Neff value for each MSA.”

• cites [50] Zhang, Y. & Skolnick, J. Scoring function for automated assessment of protein structure template quality. Proteins 57, (2004). sourceNot checked — no full text of the source