CoChat

Linguistically inspired roadmap for building biologically reliable protein language models

Open access
Vu, Mai Ha, Akbar, Rahmad, Robert, Philippe A., Swiatczak, Bartlomiej, Sandve, Geir Kjetil, Greiff, Victor, Haug, Dag Trygve Truslew · 2023 · Nature Machine Intelligence
Cited by 54

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.

Abstract

Language models trained on proteins can help to predict functions from sequences but provide little insight into the underlying mechanisms. Vu and colleagues explain how extracting the underlying rules from a protein language model can make them interpretable and help explain biological mechanisms. Deep neural-network-based language models (LMs) are increasingly applied to large-scale protein sequence data to predict protein function. However, being largely black-box models and thus challenging to interpret, current protein LM approaches do not contribute to a fundamental understanding of sequence–function mappings, hindering rule-based biotherapeutic drug development. We argue that guidance drawn from linguistics, a field specialized in analytical rule extraction from natural language data, can aid with building more interpretable protein LMs that are more likely to learn relevant domain-specific rules. Differences between protein sequence data and linguistic sequence data require the integration of more domain-specific knowledge in protein LMs compared with natural language LMs. Here, we provide a linguistics-based roadmap for protein LM pipeline choices with regard to training data, tokenization, token embedding, sequence embedding and model interpretation. Incorporating linguistic ideas into protein LMs enables the development of next-generation interpretable machine learning models with the potential of uncovering the biological mechanisms underlying sequence–function relationships.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-023-00637-1

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 49 references

Research works printed but not linked: 75

  1. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Vol. 1, 4171–4186 (Association for Computational Linguistics, 2019).
  2. Liu, Y. et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Preprint at https://doi.org/10.48550/arXiv.1907.11692 (2019). · DOI · Paper page
  3. Ofer, D., Brandes, N. & Linial, M. The language of proteins: NLP, machine learning & protein sequences. Comput. Struct. Biotechnol. J. 19, 1750–1758 (2021). · DOI · Paper page
  4. Hie, B. L., Yang, K. K. & Kim, P. S. Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins. Cell Syst. 13, 274–285 (2022). · DOI · Paper page
  5. Unsal, S. et al. Learning functional properties of proteins with language models. Nat. Mach. Intell. 4, 227–245 (2022). · DOI · Paper page
  6. Meier, J. et al. Language models enable zero-shot prediction of the effects of mutations on protein function. Adv. Neural Inf. Proc. Syst. 34, 29287–29303 (2021). · DOI · Paper page
  7. Wang, Y. et al. A high efficient biological language model for predicting protein–protein interactions. Cells 8, 122 (2019). · DOI · Paper page
  8. Xu, M. et al. PEER: a comprehensive and multi-task benchmark for protein sequence understanding. In International Conference of Learning Representations (ICLR, 2022).
  9. Burley, S. K. et al. in Protein Crystallography: Methods and Protocols (eds. Wlodawer, A., Dauter, Z. & Jaskolski, M.) 627–641 (Springer, 2017).
  10. Corrie, B. D. et al. iReceptor: a platform for querying and analyzing antibody/B-cell and T-cell receptor repertoire data across federated repositories. Immunol. Rev. 284, 24–41 (2018). · DOI · Paper page
  11. Ferruz, N. & Höcker, B. Controllable protein design with language models. Nat. Mach. Intell. 4, 521–532 (2022). · DOI · Paper page
  12. Akbar, R. et al. Progress and challenges for the machine learning-based design of fit-for-purpose monoclonal antibodies. mAbs 14, 2008790 (2022). · DOI · Paper page
  13. Villegas-Morcillo, A. et al. Unsupervised protein embeddings outperform hand-crafted sequence and structure features at predicting molecular function. Bioinformatics 37, 162–170 (2021). · DOI · Paper page
  14. Greiff, V. et al. Systems analysis reveals high genetic and antigen-driven predetermination of antibody repertoires throughout B cell development. Cell Rep. 19, 1467–1478 (2017). · DOI · Paper page
  15. Elhanati, Y. et al. Inferring processes underlying B-cell repertoire diversity. Philos. Trans. R. Soc. B 370, 20140243 (2015). · DOI · Paper page
  16. Kutuzov, A. & Kuzmenko, E. To lemmatize or not to lemmatize: how word normalisation affects ELMo performance in word sense disambiguation. In Proc. First NLPL Workshop on Deep Learning for Natural Language Processing 22–28 (Linköping University Electronic Press, 2019).
  17. Pan, Y., Li, X., Yang, Y. & Dong, R. Morphological word segmentation on agglutinative languages for neural machine translation. Preprint at https://doi.org/10.48550/arXiv.2001.01589 (2020). · DOI
  18. Schwartz, L. et al. Neural polysynthetic language modelling. Preprint at https://doi.org/10.48550/arXiv.2005.05477 (2020). · DOI
  19. Mielke, S. J. et al. Between words and characters: a brief history of open-vocabulary modeling and Tokenization in NLP. Preprint at https://doi.org/10.48550/arXiv.2112.10508 (2021). · DOI
  20. Doddapaneni, S., Ramesh, G., Khapra, M. M., Kunchukuttan, A. & Kumar, P. A primer on pretrained multilingual language models. Preprint at https://doi.org/10.48550/arXiv.2107.00676 (2021). · DOI
  21. Shin, S. et al. On the effect of pretraining corpora on in-context Learning by a large-scale language model. In Proc. 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 5168–5186 (Association for Computational Linguistics, 2022). · DOI · Paper page
  22. Leem, J., Mitchell, L. S., Farmery, J. H. R., Barton, J. & Galson, J. D. Deciphering the language of antibodies using self-supervised learning. Patterns 3, 100513 (2022). · DOI · Paper page
  23. Ruffolo, J. A., Gray, J. J. & Sulam, J. Deciphering antibody affinity maturation with language models and weakly supervised learning. In Machine Learning in Structural Biology Workshop, NeurIPS (2021)
  24. Agerri, R. et al. Give your Text Representation Models some Love: the Case for Basque. in Proceedings of the Twelfth Language Resources and Evaluation Conference 4781–4788 (European Language Resources Association, 2020).
  25. Liu, C.-L., Hsu, T.-Y., Chuang, Y.-S. & Lee, H.-Y. A study of cross-lingual ability and language-specific information in multilingual BERT. Preprint at https://doi.org/10.48550/arXiv.2004.09205 (2020). · DOI
  26. Lauscher, A., Ravishankar, V., Vulić, I. & Glavaš, G. From zero to hero: on the limitations of zero-shot language transfer with multilingual transformers. In Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) 4483–4499 (Association for Computational Linguistics, 2020). · DOI · Paper page
  27. de Vries, W., Wieling, M. & Nissim, M. Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages. In Proc. 60th Annual Meeting of the Association for Computational Linguistics (2022). · DOI
  28. Shuai, R. W., Ruffolo, J. A. & Gray, J. J. Generative language modeling for antibody design. In Machine Learning for Structural Biology Workshop, NeurIPS (2021) · DOI
  29. Ostrovsky-Berman, M., Frankel, B., Polak, P. & Yaari, G. Immune2vec: embedding B/T cell receptor sequences in ℝN using natural language processing. Front. Immunol. 12, 680687 (2021). · DOI
  30. Kao, W.-T. & Lee, H. Is BERT a cross-disciplinary knowledge learner? A surprising finding of pre-trained models’ transferability. In Findings of the Association for Computational Linguistics: EMNLP 2021 2195–2208 (Association for Computational Linguistics, 2021). · DOI
  31. Krishna, K., Bigham, J. & Lipton, Z. C. Does pretraining for summarization require knowledge transfer? In Findings of the Association for Computational Linguistics: EMNLP 2021 3178–3189 (Association for Computational Linguistics, 2021). · DOI
  32. Robert, P. A. et al. Unconstrained generation of synthetic antibody–antigen structures to guide machine learning methodology for antibody specificity prediction. Nat. Comput. Sci. 2, 845–865 (2022). · DOI
  33. Marcou, Q., Mora, T. & Walczak, A. M. High-throughput immune repertoire analysis with IGoR. Nat. Commun. 9, 561 (2018). · DOI · Paper page
  34. Weber, C. R. et al. immuneSIM: tunable multi-feature simulation of B- and T-cell receptor repertoires for immunoinformatics benchmarking. Bioinformatics 36, 3594–3596 (2020). · DOI
  35. Morris, T. P., White, I. R. & Crowther, M. J. Using simulation studies to evaluate statistical methods. Stat. Med. 38, 2074–2102 (2019). · DOI · Paper page
  36. Yang, K. K., Wu, Z., Bedbrook, C. N. & Arnold, F. H. Learned protein embeddings for machine learning. Bioinformatics 34, 2642–2648 (2018). · DOI · Paper page
  37. Brown, P. F., Pietra, V. J. D., Mercer, R. L., Pietra, S. A. D. & Lai, J. C. An estimate of an upper bound for the entropy of English. Comput. Linguist. 18, 10 (1992).
  38. Xu, J., Zhou, H., Gan, C., Zheng, Z. & Li, L. Vocabulary Learning via Optimal Transport for Neural Machine Translation. In Proc. 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing Vol. 1: Long Papers, 7361–7373 (Association for Computational Linguistics, 2021). · DOI
  39. Gage, P. A new algorithm for data compression. C Users J. 12, 23–38 (1994).
  40. Hofmann, V., Pierrehumbert, J. & Schütze, H. Superbizarre is not superb: derivational morphology improves BERT’s interpretation of complex words. In Proc. 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing Vol. 1, 3594–3608 (Association for Computational Linguistics, 2021). · DOI
  41. Littmann, M., Heinzinger, M., Dallago, C., Olenyi, T. & Rost, B. Embeddings from deep learning transfer GO annotations beyond homology. Sci. Rep. 11, 1160 (2021). · DOI · Paper page
  42. Madani, A. et al. Deep neural language modeling enables functional protein generation across families. Preprint at bioRxiv https://doi.org/10.1101/2021.07.18.452833 (2021). · DOI · Paper page
  43. Devi, G., Tendulkar, A. V. & Chakraborti, S. Protein word detection using text segmentation techniques. In BioNLP 2017 238–246 (Association for Computational Linguistics, 2017). · DOI
  44. Asgari, E., McHardy, A. C. & Mofrad, M. R. K. Probabilistic variable-length segmentation of protein sequences for discriminative motif discovery (DiMotif) and sequence embedding (ProtVecX). Sci. Rep. 9, 3577 (2019). · DOI · Paper page
  45. Strait, B. J. & Dewey, T. G. The Shannon information entropy of protein sequences. Biophys. J. 71, 148–155 (1996). · DOI · Paper page
  46. Asgari, E. & Mofrad, M. R. K. Continuous distributed representation of biological sequences for deep proteomics and genomics. PLoS ONE 10, e0141287 (2015). · DOI · Paper page
  47. Hofmann, V., Schütze, H. & Pierrehumbert, J. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. in Proc. 60th Annual Meeting of the Association for Computational Linguistics Vol. 2: Short Papers, 385–393 (Association for Computational Linguistics, 2022). · DOI
  48. Matthews, A., Neubig, G. & Dyer, C. Using morphological knowledge in open-vocabulary neural language models. In Proc. 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Vol. 1: Long Papers, 1435–1445 (Association for Computational Linguistics, 2018). · DOI
  49. Gutierrez-Vasques, X., Bentz, C., Sozinova, O. & Samardzic, T. From characters to words: the turning point of BPE merges. In Proc. 16th Conference of the European Chapter of the Association for Computational Linguistics Main Vol., 3454–3468 (Association for Computational Linguistics, 2021). · DOI
  50. Alva, V., Söding, J. & Lupas, A. N. A vocabulary of ancient peptides at the origin of folded proteins. eLife 4, e09410 (2015). · DOI · Paper page
  51. Kolodny, R., Nepomnyachiy, S., Tawfik, D. S. & Ben-Tal, N. Bridging themes: short protein segments found in different architectures. Mol. Biol. Evol. 38, 2191–2208 (2021). · DOI · Paper page
  52. Fernandez-Fuentes, N., Dybas, J. M. & Fiser, A. Structural characteristics of novel protein folds. PLoS Comput. Biol. 6, e1000750 (2010). · DOI
  53. Ferruz, N. et al. Identification and analysis of natural building blocks for evolution-guided fragment-based protein design. J. Mol. Biol. 432, 3898–3914 (2020). · DOI · Paper page
  54. Akbar, R. et al. A compact vocabulary of paratope-epitope interactions enables predictability of antibody-antigen binding. Cell Rep. 34, 108856 (2021). · DOI · Paper page
  55. Hiraoka, T., Takase, S., Uchiumi, K., Keyaki, A. & Okazaki, N. Optimizing Word Segmentation for Downstream Task. In Findings of the Association for Computational Linguistics: EMNLP 2020 1341–1351 (Association for Computational Linguistics, 2020). · DOI
  56. Welleck, S., Brantley, K., Daumé, H. III & Cho, K. Non-monotonic sequential text generation. In Proc. 36th International Conference on Machine Learning 6716–6726 (PMLR, 2019).
  57. Stern, M., Chan, W., Kiros, J. & Uszkoreit, J. Insertion transformer: flexible sequence generation via insertion operations. In Proc. 36th International Conference on Machine Learning 5976–5985 (PMLR, 2019).
  58. Gimona, M. Protein linguistics — a grammar for modular protein assembly? Nat. Rev. Mol. Cell Biol. 7, 68–73 (2006). · DOI
  59. Mikolov, T., Yih, W. & Zweig, G. Linguistic regularities in continuous space word representations. In Proc. 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 746–751 (Association for Computational Linguistics, 2013).
  60. Schluter, N. The word analogy testing caveat. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Vol. 2: Short Papers, 242–246 (Association for Computational Linguistics, 2018). · DOI
  61. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S. & Dean, J. Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Process. Syst. 26 (Curran Associates, Inc., 2013).
  62. Chen, C., Zhou, J., Wang, F., Liu, X. & Dou, D. Structure-aware protein self-supervised learning. Preprint at https://doi.org/10.48550/arXiv.2204.04213 (2022). · arXiv · doi:10.48550/arxiv.2204.04213
  63. Detlefsen, N. S., Hauberg, S. & Boomsma, W. Learning meaningful representations of protein sequences. Nat. Commun. 13, 1914 (2022). · DOI · Paper page
  64. Montague, R. Universal grammar. Theoria 36, 373–393 (1970). · DOI · Paper page
  65. McCoy, R. T., Frank, R. & Linzen, T. Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks. Trans. Assoc. Comput. Linguist. 8, 125–140 (2020). · DOI · Paper page
  66. Linzen, T. What can linguistics and deep learning contribute to each other? Response to Pater. Language 95, e99–e108 (2019). · DOI
  67. Clark, K., Khandelwal, U., Levy, O. & Manning, C. D. What does BERT look at? An analysis of BERT’s attention. In Proc. 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP 276–286 (Association for Computational Linguistics, 2019). · DOI · Paper page
  68. Adebayo, J., Muelly, M., Abelson, H. & Kim, B. Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International Conference on Learning Representations (2022).
  69. Linzen, T., Dupoux, E. & Goldberg, Y. Assessing the ability of LSTMS to learn syntax-sensitive dependencies. Trans. Assoc. Comput. Linguist. 4, 521–535 (2016). · DOI · Paper page
  70. Angluin, D. Learning regular sets from queries and counterexamples. Inf. Comput. 75, 87–106 (1987). · DOI · Paper page
  71. Weiss, G., Goldberg, Y. & Yahav, E. Extracting automata from recurrent neural networks using queries and counterexamples. In Proc. 35th International Conference on Machine Learning 5247–5256 (PMLR, 2018).
  72. Angluin, D. Computational learning theory: survey and selected bibliography. In Proc. 24th Annual ACM Symposium on Theory of Computing 351–369 (Association for Computing Machinery, 1992). · DOI · Paper page
  73. Eyraud, R. & Ayache, S. Distillation of weighted automata from recurrent neural networks using a spectral approach. Mach. Learn. https://doi.org/10.1007/s10994-021-05948-1 (2021). · DOI
  74. Wang, Q. et al. An empirical evaluation of rule extraction from recurrent neural networks. Neural Comput. 30, 2568–2591 (2018). · DOI · Paper page
  75. Sandve, G. K. & Greiff, V. Access to ground truth at unconstrained size makes simulated data as indispensable as experimental data for bioinformatics methods development and benchmarking. Bioinformatics 38, 4994–4996 (2022). · DOI

Cited by

See all 54

Metadata

Authors
Vu, M. H., Akbar, R., Robert, P. A., Swiatczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T.
Venue
Nature Machine Intelligence
Published
2023-04-06
Volume
5
Issue
5
Pages
485-496
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4362664122
Cited by
54
References
49
FWCI
4.03
Type
journal article
Language
en
Open access
green
Licence
Funders
  • University of Oslo | Livsvitenskap, Universitetet i Oslo
  • Norges Forskningsråd (311341, 300740)
  • Stiftelsen Kristian Gerhard Jebsen
  • Leona M. and Harry B. Helmsley Charitable Trust (2019PG-T1D011)
  • EC | Horizon 2020 Framework Programme (825821)
  • Kreftforeningen (215817)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Linguistically inspired roadmap for building biologically reliable protein language models. https://app.cochat.ai/papers/10.1038/s42256-023-00637-1
The paper
Vu, M. H., Akbar, R., Robert, P. A., Swiatczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023). Linguistically inspired roadmap for building biologically reliable protein language models. Nature Machine Intelligence. https://doi.org/10.1038/s42256-023-00637-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.