CoChat
Sign Up
PreprintCited by 1Protein Structure and DynamicsMachine Learning in BioinformaticsEnzyme Structure and Function

Protein Structure Tokenization: Benchmarking and New Recipe

Open access
Yuan, Xinyu, Wang, Zichen, Collins, Marcus, Rangwala, Huzefa·2025·arXiv
DOI 10.48550/arxiv.2503.00089arXiv 2503.00089
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent years have witnessed a surge in the development of protein structural tokenization methods, which chunk protein 3D structures into discrete or continuous representations. Structure tokenization enables the direct application of powerful techniques like language modeling for protein structures, and large multimodal models to integrate structures with protein sequences and functional texts. Despite the progress, the capabilities and limitations of these methods remain poorly understood due to the lack of a unified evaluation framework. We first introduce StructTokenBench, a framework that comprehensively evaluates the quality and efficiency of structure tokenizers, focusing on fine-grained local substructures rather than global structures, as typical in existing benchmarks. Our evaluations reveal that no single model dominates all benchmarking perspectives. Observations of codebook under-utilization led us to develop AminoAseed, a simple yet effective strategy that enhances codebook gradient updates and optimally balances codebook size and dimension for improved tokenizer utilization and quality. Compared to the leading model ESM3, our method achieves an average of 6.31% performance improvement across 24 supervised tasks, with sensitivity and utilization rates increased by 12.83% and 124.03%, respectively. Source code and model weights are available at https://github.com/KatarinaYuan/StructTokenBench

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.00089

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Impact of protein conformational diversity on AlphaFold predictions

    Saldaño, T., Escobedo, N., Marchetti, J., Zea, D. J., Mac Donagh, J., Velez Rueda, A. J., Gonik, E., Melani, A. G., Novomisky Nechcoff, J., Salas, M. N., et al. (2021) · doi:10.1101/2021.10.27.466189 · Cited by 196 · cited 5× · see citing works

    “3 shows that: (1) AminoAseed outperforms all models in Apo Holo and Fold Switching datasets, with an average relative gain of +12.83% over ESM3 across all metrics; (2) ESM3 performs second best; (3) VanillaVQ slightly trails ESM3 by -0.82%; (4) IF-based PSTs underperform most VQ-VAE-based PSTs because their training o…”
    Influential

Flagged first, then by citations

Research works printed but not linked: 56

  1. Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature , pp. 1-3, 2024.
  2. Ahdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O'Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., et al. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. Nature Methods , pp. 1-11, 2024.
  3. Athar, T., Al Balushi, K., and Khan, S. A. Recent advances on drug development and emerging therapeutic agents for alzheimer's disease. Molecular biology reports , 48(7): 5629-5645, 2021.
  4. Bengio, Y., L´ eonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013. · arXiv
  5. Boike, L., Henning, N. J., and Nomura, D. K. Advances in covalent drug discovery. Nature Reviews Drug Discovery , 21(12):881-898, 2022.
  6. Capel, H., Weiler, R., Dijkstra, M., Vleugels, R., Bloem, P., and Feenstra, K. A. Proteinglue multi-task benchmark suite for self-supervised protein modeling. Scientific Reports , 12(1):16047, 2022.
  7. Chakravarty, D. and Porter, L. L. Alphafold2 fails to predict protein fold switching. Protein Science , 31(6):e4353, 2022.
  8. Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., and Yang, K. K. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv , pp. 2021-11, 2021.
  9. Dauparas, J., Anishchenko, I., Bennett, N., Bai, H., Ragotte, R. J., Milles, L. F., Wicky, B. I., Courbet, A., de Haas, R. J., Bethel, N., et al. Robust deep learning-based protein sequence design using proteinmpnn. Science , 378 (6615):49-56, 2022.
  10. de Brevern, A. G. New assessment of a structural alphabet. In silico biology , 5(3):283-289, 2005.
  11. Ferruz, N., Schmidt, S., and H¨ ocker, B. Protgpt2 is a deep unsupervised language model for protein design. Nature communications , 13(1):4348, 2022.
  12. Gao, Z., Tan, C., Wang, J., Huang, Y., Wu, L., and Li, S. Z. Foldtoken: Learning protein language via vector quantization and beyond. arXiv preprint arXiv:2403.09673 , 2024. · arXiv
  13. Gaujac, B., Don` a, J., Copoiu, L., Atkinson, T., Pierrot, T., and Barrett, T. D. Learning the language of protein structure. arXiv preprint arXiv:2405.15840 , 2024. · arXiv
  14. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. · arXiv
  15. Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Mirdita, M., Steinegger, M., and Rost, B. Bilingual language model for protein sequence and structure. bioRxiv , pp. 2023-07, 2023.
  16. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022. · arXiv
  17. Hou, J., Adhikari, B., and Cheng, J. Deepsf: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics , 34(8):1295-1303, 2018.
  18. Huang, J., Rauscher, S., Nawrocki, G., Ran, T., Feig, M., De Groot, B. L., Grubm¨ uller, H., and MacKerell Jr, A. D. Charmm36m: an improved force field for folded and intrinsically disordered proteins. Nature methods , 14(1): 71-73, 2017.
  19. Jamasb, A. R., Morehead, A., Joshi, C. K., Zhang, Z., Didi, K., Mathis, S., Harris, C., Tang, J., Cheng, J., Li` o, P., et al. Evaluating representation learning on the protein structure universe. ArXiv , pp. arXiv-2406, 2024.
  20. Kessler, J. and Melief, C. Identification of t-cell epitopes for cancer immunotherapy. Leukemia , 21(9):1859-1874, 2007.
  21. Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013. · arXiv
  22. Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
  23. Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., and Moult, J. Critical assessment of methods of protein structure prediction (casp)-round xiv. Proteins: Structure, Function, and Bioinformatics , 89(12):1607-1617, 2021.
  24. Kucera, T., Oliver, C., Chen, D., and Borgwardt, K. Proteinshake: building datasets and benchmarks for deep learning on protein structures. Advances in Neural Information Processing Systems , 36, 2024.
  25. Li, M., Tan, Y., Ma, X., Zhong, B., Yu, H., Zhou, Z., Ouyang, W., Zhou, B., Hong, L., and Tan, P. Prosst: Protein language modeling with quantized structure and disentangled attention. biorxiv. 2024.
  26. Ligowski, L. and Rudnicki, W. An efficient implementation of smith waterman algorithm on gpu using cuda, for massively parallel scanning of sequence databases. In 2009 IEEE International Symposium on Parallel & Distributed Processing , pp. 1-8. IEEE, 2009.
  27. Likic, V. The needleman-wunsch algorithm for sequence alignment. Lecture given at the 7th Melbourne Bioinformatics Course, Bi021 Molecular Science and Biotechnology Institute, University of Melbourne , pp. 1-46, 2008.
  28. Lin, X., Chen, Z., Li, Y., Ma, Z., Fan, C., Cao, Z., Feng, S., Gao, Y. Q., and Zhang, J. Tokenizing foldable protein structures with machine-learned artificial amino-acid vocabulary. bioRxiv , pp. 2023-11, 2023a.
  29. Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science , 379(6637): 1123-1130, 2023b.
  30. Liu, A., Elaldi, A., Russell, N., and Viessmann, O. Bio2token: All-atom tokenization of any biomolecular structure with mamba. arXiv preprint arXiv:2410.19110 , 2024. · arXiv
  31. Liu, Z., Li, Y ., Han, L., Li, J., Liu, J., Zhao, Z., Nie, W., Liu, Y., and Wang, R. Pdb-wide collection of binding data: current status of the pdbbind database. Bioinformatics , 31(3):405-412, 2015.
  32. Loux, T., Wang, D., and Shakhnovich, E. More structures, less accuracy: Esm3's binding prediction paradox. bioRxiv , 2024. doi: 10.1101/2024.12.09.627585. URL https://www.biorxiv.org/content/ early/2024/12/09/2024.12.09.627585 . · DOI
  33. Lu, A. X., Yan, W., Yang, K. K., Gligorijevic, V., Cho, K., Abbeel, P., Bonneau, R., and Frey, N. Tokenized and continuous embedding compressions of protein sequence and structure. bioRxiv , pp. 2024-08, 2024.
  34. Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505 , 2023. · arXiv
  35. Min, H.-Y. and Lee, H.-Y. Molecular targeted therapy for anticancer treatment. Experimental & molecular medicine , 54(10):1670-1694, 2022.
  36. Mousavi, P., Della Libera, L., Duret, J., Ploujnikov, A., Subakan, C., and Ravanelli, M. Dasb-discrete audio and speech benchmark. arXiv preprint arXiv:2406.14294 , 2024. · arXiv
  37. Pearl, F. M., Bennett, C., Bray, J. E., Harrison, A. P., Martin, N., Shepherd, A., Sillitoe, I., Thornton, J., and Orengo, C. A. The cath database: an extended protein family resource for structural and functional genomics. Nucleic acids research , 31(1):452-455, 2003.
  38. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pp. 1-16. IEEE, 2020.
  39. Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, P., Canny, J., Abbeel, P., and Song, Y. Evaluating protein transfer learning with tape. Advances in neural information processing systems , 32, 2019.
  40. Robin, X., Haas, J., Gumienny, R., Smolinski, A., Tauriello, G., and Schwede, T. Continuous automated model evaluation (cameo)-perspectives on the future of fully automated evaluation of structure prediction methods. Proteins: Structure, Function, and Bioinformatics , 89 (12):1977-1986, 2021.
  41. Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 , 2020. · arXiv
  42. Stein, R. M., Yang, Y., Balius, T. E., O'Meara, M. J., Lyu, J., Young, J., Tang, K., Shoichet, B. K., and Irwin, J. J. Property-unmatched decoys in docking benchmarks. Journal of chemical information and modeling , 61(2): 699-714, 2021.
  43. Steinegger, M. and S¨ oding, J. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology , 35(11):1026-1028, 2017.
  44. Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv , pp. 2023-10, 2023.
  45. Sun, Z., Liu, Q., Qu, G., Feng, Y ., and Reetz, M. T. Utility of b-factors in protein science: interpreting rigidity, flexibility, and internal motion and engineering thermostability. Chemical reviews , 119(3):1626-1665, 2019.
  46. Townshend, R., Powers, A., Eismann, S., and Derry, A. Atom3d: tasks on molecules in three dimensions. arxiv 2021. arXiv preprint arXiv:2012.04035 , 2021. · arXiv
  47. Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.
  48. Van Kempen, M., Kim, S. S., Tumescheit, C., Mirdita, M., Lee, J., Gilchrist, C. L., S¨ oding, J., and Steinegger, M. Fast and accurate protein structure search with foldseek. Nature biotechnology , 42(2):243-246, 2024.
  49. Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems , 2017.
  50. Xu, J., Zhou, H., Gan, C., Zheng, Z., and Li, L. Vocabulary learning via optimal transport for neural machine translation. arXiv preprint arXiv:2012.15671 , 2020. · arXiv
  51. Xu, M., Zhang, Z., Lu, J., Zhu, Z., Zhang, Y., Chang, M., Liu, R., and Tang, J. Peer: a comprehensive and multitask benchmark for protein sequence understanding. Advances in Neural Information Processing Systems , 35: 35156-35173, 2022.
  52. Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 , 2023. · arXiv
  53. Zhang, B., Wang, H., Luo, C., Li, X., Liang, G., Ye, Y., Qi, X., and He, Y. Codebook transfer with part-of-speech for vector-quantized image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7757-7766, 2024a.
  54. Zhang, J., Meynard-Piganeau, B., Gong, J., Cheng, X., Luo, Y., Ly, H., Song, L., and Xing, E. Balancing locality and reconstruction in protein structure tokenizer. bioRxiv , 2024c. doi: 10.1101/2024.12.02.626366. URL https://www.biorxiv.org/content/ early/2024/12/06/2024.12.02.626366 . · DOI
  55. Zhang, J., Meynard-Piganeau, B., Gong, J., Cheng, X., Luo, Y., Ly, H., Song, L., and Xing, E. Balancing locality and reconstruction in protein structure tokenizer. bioRxiv , pp. 2024-12, 2024d.
  56. Zhang, Y. and Skolnick, J. Tm-align: a protein structure alignment algorithm based on the tm-score. Nucleic acids research , 33(7):2302-2309, 2005.

Cited by

Sorted by citations

Authors
Yuan, X., Wang, Z., Collins, M., & Rangwala, H.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2503.00089
arXiv
2503.00089
OpenAlex
W4415080914
Cited by
0
References
1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Physical Sciences › Materials Science › Materials Chemistry › Enzyme Structure and Function

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review