Combination of deep neural network with attention mechanism enhances the explainability of protein contact prediction
Open accessPublished version of a preprint
Abstract
Deep learning has emerged as a revolutionary technology for protein residue‐residue contact prediction since the 2012 CASP10 competition. Considerable advancements in the predictive power of the deep learning‐based contact predictions have been achieved since then. However, little effort has been put into interpreting the black‐box deep learning methods. Algorithms that can interpret the relationship between predicted contact maps and the internal mechanism of the deep learning architectures are needed to explore the essential components of contact inference and improve their explainability. In this study, we present an attention‐based convolutional neural network for protein contact prediction, which consists of two attention mechanism‐based modules: sequence attention and regional attention. Our benchmark results on the CASP13 free‐modeling targets demonstrate that the two attention modules added on top of existing typical deep learning models exhibit a complementary effect that contributes to prediction improvements. More importantly, the inclusion of the attention mechanism provides interpretable patterns that contain useful insights into the key fold‐determining residues in proteins. We expect the attention‐based model can provide a reliable and practically interpretable technique that helps break the current bottlenecks in explaining deep neural networks for contact prediction. The source code of our method is available at https://github.com/jianlin-cheng/InterpretContactMap .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep learning with residue covariation significantly improved structure prediction.
“Deep learning with residue covariation significantly improved structure prediction.18,19,23–25,33”
From AlphaFold 2: Why It Works and Its Implications for Understanding the Relationships of Protein Sequence, Structure, and Function (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Hierarchical Attention Networks for Document Classification
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A. J., & Hovy, E. H. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1174 · Cited by 4,874 · see citing works
Instance Normalization: The Missing Ingredient for Fast StylizationPreprint
Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2016) · arXiv.org · doi:10.48550/arxiv.1607.08022 · Cited by 4,239 · see citing works
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378 · see citing works
On the momentum term in gradient descent learning algorithms
Ning Qian, A. (1999) · Neural Networks · doi:10.1016/s0893-6080(98)00116-6 · Cited by 2,318 · see citing works
HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment
Remmert, M., Biegert, A., Hauser, A., & Söding, J. (2012) · Nature Methods · doi:10.1038/nmeth.1818 · Cited by 2,288 · see citing works
Challenges in homology search: HMMER3 and convergent evolution of coiled-coil regions
Mistry, J., Finn, R. D., Eddy, S. R., Bateman, A. G., & Punta, M. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt263 · Cited by 1,622 · see citing works
Morcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434 · see citing works
Clustering huge protein sequence sets in linear time
Steinegger, M., & Söding, J. (2018) · Nature Communications · doi:10.1038/s41467-018-04964-5 · Cited by 875 · see citing works
Jones, D. T., Buchan, D. W. A., Cozzetto, D., & Pontil, M. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr638 · Cited by 760 · see citing works
Uniclust databases of clustered and deeply annotated protein sequences and alignments
Mirdita, M., von den Driesch, L., Galiez, C., Martin, M. J., Söding, J., & Steinegger, M. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw1081 · Cited by 710 · see citing works
Improved contact prediction in proteins: using pseudolikelihoods to infer Potts models
Ekeberg, M., Lövkvist, C., Lan, Y., Weigt, M., & Aurell, E. (2013) · Physical Review E · doi:10.1103/physreve.87.012707 · Cited by 661 · see citing works
Three key residues form a critical contact network in a protein folding transition state
Vendruscolo, M., Paci, E., Dobson, C. M., & Karplus, M. (2001) · Nature · doi:10.1038/35054591 · Cited by 429 · see citing works
CCMpred—fast and precise prediction of protein residue–residue contacts from correlated mutations
Seemayer, S., Gruber, M., & Söding, J. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu500 · Cited by 416 · see citing works
Serrano, L., & Martínez, J. C. (1999) · Nature Structural Biology · doi:10.1038/14896 · Cited by 331 · see citing works
Identification of diagnostic markers for tuberculosis by proteomic fingerprinting of serum
Agranoff, D., Fernández-Reyes, D., Papadopoulos, M. C., Rojas, S. A., Herbster, M., Loosemore, A., Tarelli, E., Sheldon, J., Schwenk, A., Pollok, R., et al. (2006) · The Lancet · doi:10.1016/s0140-6736(06)69342-2 · Cited by 263 · see citing works
Unifying features in protein-folding mechanisms
Gianni, S., Guydosh, N. R., Khan, F., Caldas, T. D., Mayor, U., White, G. W. N., DeMarco, M. L., Daggett, V., & Fersht, A. R. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1835776100 · Cited by 241 · see citing works
A breakdown of symmetry in the folding transition state of protein L
Kim, D. E., Fisher, C. L., & Baker, D. A. (2000) · Journal of Molecular Biology · doi:10.1006/jmbi.2000.3701 · Cited by 233 · see citing works
Deep‐learning contact‐map guided protein structure prediction in CASP13
Zheng, W., Li, Y., Zhang, C., Pearce, R., Mortuza, S. M., & Zhang, Y. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25792 · Cited by 208 · see citing works
Greener, J. G., Kandathil, S. M., & Jones, D. T. (2019) · Nature Communications · doi:10.1038/s41467-019-11994-0 · Cited by 178 · see citing works
DNCON2: improved protein contact prediction using two-level deep convolutional neural networks
Adhikari, B., Hou, J., & Cheng, J. J. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx781 · Cited by 168 · see citing works
Self-citationHou, J., Wu, T., Cao, R., & Cheng, J. J. (2019) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25697 · Cited by 162 · see citing works
Self-citationPredicting protein residue–residue contacts using deep networks and boosting
Eickholt, J., & Cheng, J. J. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts598 · Cited by 161 · see citing works
Self-citationFersht, A. R., Leatherbarrow, R. J., & Wells, T. N. C. (1987) · Biochemistry · doi:10.1021/bi00393a013 · Cited by 157 · see citing works
Hu, Y., Wang, Z., Hu, H., Wan, F., Chen, L., Xiong, Y., Wang, X., Zhao, D. Y., Huang, W., & Zeng, J. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz427 · Cited by 105 · see citing works
Chen, C., Hou, J., Shi, X., Yang, H. Q., Birchler, J. A., & Cheng, J. J. (2021) · BMC Bioinformatics · doi:10.1186/s12859-020-03952-1 · Cited by 92 · see citing works
Self-citationAnalysis of deep learning methods for blind protein contact prediction in CASP12
Wang, S., Sun, S., & Xu, J. (2017) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25377 · Cited by 90 · see citing works
Assessing the accuracy of contact predictions in CASP13
Shrestha, R., Fajardo, E. J., Gil, N., Fidelis, K. A., Kryshtafovych, A., Monastyrskyy, B., & Fiser, A. (2019) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25819 · Cited by 62 · see citing works
DeepDist: real-value inter-residue distance prediction with deep residual convolutional network
Wu, T., Guo, Z., Hou, J., & Cheng, J. J. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-03960-9 · Cited by 54 · see citing works
Self-citationCalculation of Mutational Free Energy Changes in Transition States for Protein Folding
Lindorff‐Larsen, K., Paci, E., Serrano, L. G., Dobson, C. M., & Vendruscolo, M. (2003) · Biophysical Journal · doi:10.1016/s0006-3495(03)74556-1 · Cited by 32 · see citing works
DeepDist: real-value inter-residue distance prediction with deep residual convolutional networkPreprint
Wu, T., Guo, Z., Hou, J., & Cheng, J. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.17.995910 · Cited by 25 · see citing works
Self-citationCharacterization of protein folding by a Φ-value calculation with a statistical-mechanical model
Wako, H., & Abe, H. (2016) · Biophysics and Physicobiology · doi:10.2142/biophysico.13.0_263 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [14]Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Paper presented at: Advances in Neural Information Processing Systems 2017.
- [26]Goodfellow IJ, Warde‐Farley D, Mirza M, Courville A, Bengio Y. Maxout networks. JMLR Workshop and Conference Proceedings. 2013; 28(3): 1319– 1327.
Cited by
Sorted by citations
Skolnick, J., Gao, M., Zhou, H., & Singh, S. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.1c01114 · Cited by 251 · see citing works
Deep learning methods for 3D structural proteome and interactome modeling
Lee, D., Xiong, D., Wierbowski, S., Li, L., Liang, S., & Yu, H. (2022) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2022.102329 · Cited by 26 · see citing works
- Authors
- Chen, C., Wu, T., Guo, Z., & Cheng, J.
- Affiliations
- University of Missouri (US)
- Corresponding author
- Cheng, Jianlin
- Venue
- Proteins: Structure, Function, and Bioinformatics
- Publisher
- Wiley
- Published
- 2021-02-16
- Volume
- 89
- Issue
- 6
- Pages
- 697-707
- ISSN
- 0887-3585 (ISSN-L)
- 1097-0134
- PMC
- PMC8089057
- OpenAlex
- W3128355392
- Cited by
- 44
- References
- 36
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 7 · 2023: 3 · 2022: 16 · 2021: 10
- FWCI
- 2.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proteins Structure Function and Bioinformatics (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Funders
- U.S. Department of Energy (DE‐SC0020400, DE‐SC0021303)
- National Institute of General Medical Sciences (R01GM093123)
- National Science Foundation (DBI1759934, IIS1763246)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Keywords
- attention mechanism, explainable deep learning, attention-based convolutional neural network, interpretability
- MeSH terms
- Deep Learning, Binding Sites, Humans, Protein Binding, Protein Conformation, Proteins, Research Design, Software, Sequence Alignment, Computational Biology, Benchmarking, Sequence Analysis, Protein, Databases, Protein, Protein Interaction Domains and Motifs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
