CoChat
Sign Up
PreprintCited by 5Protein Structure and DynamicsMachine Learning in BioinformaticsComputational Drug Discovery Methods

FoldToken: Learning Protein Language via Vector Quantization and Beyond

Open access
Gao, Zhangyang, Tan, Cheng, Wang, Jue, Huang, Yufei, Wu, Lirong, Li, Stan Z.·2024·arXiv
DOI 10.48550/arxiv.2403.09673arXiv 2403.09673
Open this paper in CoChat — free accountOpen PDF

Abstract

Is there a foreign language describing protein sequences and structures simultaneously? Protein structures, represented by continuous 3D points, have long posed a challenge due to the contrasting modeling paradigms of discrete sequences. We introduce \textbf{FoldTokenizer} to represent protein sequence-structure as discrete symbols. This innovative approach involves projecting residue types and structures into a discrete space, guided by a reconstruction loss for information preservation. We refer to the learned discrete symbols as \textbf{FoldToken}, and the sequence of FoldTokens serves as a new protein language, transforming the protein sequence-structure into a unified modality. We apply the created protein language on general backbone inpainting and antibody design tasks, building the first GPT-style model (\textbf{FoldGPT}) for sequence-structure co-generation with promising results. Key to our success is the substantial enhancement of the vector quantization module, Soft Conditional Vector Quantization (\textbf{SoftCVQ}).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.09673

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 47

  1. Alamdari, S., Thakkar, N., and van den Berg, R. a. Protein generation with evolutionary diffusion: sequence is all you need. bioRxiv, pp. 2023-09, 2023.
  2. Barrio-Hernandez, I., Yeo, J., Jänes, J., Wein, T., Varadi, M., Velankar, S., Beltrao, P., and Steinegger, M. Clustering predicted structures at the scale of the known protein universe. bioRxiv, pp. 2023-03, 2023.
  3. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurIPS, 33:1877-1901, 2020.
  4. Chen, T., Li, L., and Sun, Y. Differentiable product quanti- zation for end-to-end embedding compression. In ICML, pp. 1617-1626. PMLR, 2020.
  5. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. · arXiv
  6. Eguchi, R. R., Choe, C. A., and Huang, P.-S. Ig-vae: Gen- erative modeling of protein structure by direct 3d coor- dinate generation. PLoS computational biology, 18(6): e1010271, 2022.
  7. El-Nouby, A., Muckley, M. J., Ullrich, K., Laptev, I., Ver- beek, J., and Jégou, H. Image compression with prod- uct quantized masked image modeling. arXiv preprint arXiv:2212.07372, 2022. · arXiv
  8. Esser, P., Rombach, R., and Ommer, B. Taming transform- ers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
  9. Fan, H., Wang, Z., Yang, Y., and Kankanhalli, M. Continuous-discrete convolution for geometry-sequence modeling in proteins. In The Eleventh International Conference on Learning Representations, 2022.
  10. Gao, Z., Tan, C., Li, S., et al. Alphadesign: A graph protein design method and benchmark on alphafolddb. arXiv preprint arXiv:2202.01079, 2022. · arXiv
  11. Gao, Z., Tan, C., and Li, S. Z. Diffsds: A language diffusion model for protein backbone inpainting under geometric conditions and constraints. arXiv preprint arXiv:2301.09642, 2023a. · arXiv
  12. Gao, Z., Tan, C., and Li, S. Z. Knowledge-design: Push- ing the limit of protein deign via knowledge refinement. arXiv preprint arXiv:2305.15151, 2023b. · arXiv
  13. Gao, Z., Tan, C., and Li, S. Z. Pifold: Toward effective and efficient protein inverse folding. In International Conference on Learning Representations, 2023c. URL https://openreview.net/forum? id=oMsN9TYwJ0j.
  14. Horiguchi, S., Dohi, K., and Kawaguchi, Y. Streaming ac- tive learning for regression problems using regression via classification. arXiv preprint arXiv:2309.01013, 2023. · arXiv
  15. Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., and Rives, A. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
  16. Hu, B., Tan, C., Xia, J., Zheng, J., Huang, Y., Wu, L., Liu, Y., Xu, Y., and Li, S. Z. Learning complete protein representation by deep coupling of sequence and structure. bioRxiv, pp. 2023-07, 2023.
  17. Hu, B., Tan, C., Gao, B., Gao, Z., Wu, L., Xia, J., and Li, S. Z. Multimodal distillation of protein se- quence, structure, and function, 2024. URL https: //openreview.net/forum?id=O0dW800ukz.
  18. Ingraham, J. B., Baranov, M., Costello, Z., Barber, K. W., Wang, W., Ismail, A., Frappier, V., Lord, D. M., et al. Il- luminating protein space with a programmable generative model. Nature, pp. 1-9, 2023.
  19. Jin, W., Wohlwend, J., Barzilay, R., and Jaakkola, T. Iterative refinement graph neural network for anti- body sequence-structure co-design. arXiv preprint arXiv:2110.04624, 2021. · arXiv
  20. Jing, B., Eismann, S., Suriana, P., Townshend, R. J., and Dror, R. Learning from protein structure with geometric vector perceptrons. arXiv:2009.01411, 2020. · arXiv
  21. Juang, B.-H. and Gray, A. Multiple stage vector quantization for speech coding. In ICASSP, pp. 597-600. IEEE, 1982.
  22. Jégou, H., Douze, M., and Schmid, C. Product quantiza- tion for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117- 128, 2011. doi: 10.1109/TPAMI.2010. · DOI
  23. Kong, X., Huang, W., and Liu, Y. Conditional antibody design as 3d equivariant graph translation. arXiv preprint arXiv:2208.06073, 2022. · arXiv
  24. Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.-S. Autore- gressive image generation using residual quantization. In CVPR, pp. 11523-11532, 2022a.
  25. Lee, J. S., Kim, J., and Kim, P. M. Proteinsgm: Score-based generative modeling for de novo protein design. bioRxiv, pp. 2022-07, 2022b.
  26. Lin, Z., Akin, H., Rao, R., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
  27. Luo, S., Su, Y., Peng, X., Wang, S., Peng, J., and Ma, J. Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures. NeurIPS, 35:9754-9767, 2022.
  28. Martinez, J., Hoos, H. H., and Little, J. J. Stacked quantizers for compositional vector compression. arXiv preprint arXiv:1411.2173, 2014. · arXiv
  29. Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023. · arXiv
  30. Pintea, S. L., Lin, Y., Dijkstra, J., and van Gemert, J. C. A step towards understanding why classification helps regression. In ICCV, pp. 19972-19981, 2023.
  31. Samaga, Y. B., Raghunathan, S., and Priyakumar, U. D. Scones: self-consistent neural network for protein sta- bility prediction upon mutation. The Journal of Physical Chemistry B, 125(38):10657-10671, 2021.
  32. Shi, C., Wang, C., Lu, J., Zhong, B., and Tang, J. Pro- tein sequence and structure co-design with equivariant translation. arXiv preprint arXiv:2210.08761, 2022. · arXiv
  33. Song, Z., Zhao, Y., Shi, W., Yang, Y., and Li, L. Functional geometry guided protein sequence and backbone structure co-design. arXiv preprint arXiv:2310.04343, 2023. · arXiv
  34. Stewart, L., Bach, F., and Vert, J.-P. Regression as classifi- cation: Influence of task formulation on neural network features. In ICAIS, pp. 11563-11582. PMLR, 2023.
  35. Tan, C., Gao, Z., Xia, J., and Li, S. Z. Generative de novo protein design with global context. arXiv preprint arXiv:2204.10673, 2022. · arXiv
  36. Tan, C., Gao, Z., and Li, S. Z. Cross-gate mlp with pro- tein complex invariant embedding is a one-shot antibody designer. arXiv e-prints, pp. arXiv-2305, 2023a.
  37. Tan, C., Gao, Z., and Li, S. Z. Protein complex invariant embedding with cross-gate mlp is a one-shot antibody designer. arXiv preprint arXiv:2305.09480, 2023b. · arXiv
  38. Trippe, B. L., Yim, J., Tischer, D., Broderick, T., Baker, D., Barzilay, R., and Jaakkola, T. Diffusion probabilistic mod- eling of protein backbones in 3d for the motif-scaffolding problem. arXiv preprint arXiv:2206.04119, 2022. · arXiv
  39. Umerenkov, D., Shashkova, T. I., Strashnov, P. V., Nikolaev, F., Sindeeva, M., Ivanisenko, N. V., and Kardymon, O. L. Prostata: Protein stability assessment using transformers. bioRxiv, pp. 2022-12, 2022.
  40. Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
  41. Watson, J. L., Juergens, D., et al. De novo design of pro- tein structure and function with rfdiffusion. Nature, 620 (7976):1089-1100, 2023.
  42. Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021. · arXiv
  43. Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. · arXiv
  44. Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
  45. Zhang, Y., Gao, Z., Tan, C., and Li, S. Z. Efficiently predicting protein stability changes upon single-point mutation with large language models. arXiv preprint arXiv:2312.04019, 2023. · arXiv
  46. Zhang, Z., Xu, M., Jamasb, A., Chenthamarakshan, V., Lozano, A., Das, P., and Tang, J. Protein representa- tion learning by geometric structure pretraining. arXiv preprint arXiv:2203.06125, 2022. · arXiv
  47. Zheng, J., Li, S., Huang, Y., Gao, Z., Tan, C., Hu, B., Xia, J., Wang, G., and Li, S. Z. Mmdesign: Multi-modality transfer learning for generative protein design. arXiv preprint arXiv:2312.06297, 2023. · arXiv

Cited by

Sorted by citations

Authors
Gao, Z., Tan, C., Wang, J., Huang, Y., Wu, L., & Li, S. Z.
Affiliations
Institute for the Future (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.09673
arXiv
2403.09673
OpenAlex
W4392934751
Cited by
5
References
0
Citations by year
2025: 2 · 2024: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
Keywords
vector quantization, protein structure representation, antibody design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review