CoChat
Sign Up
Cited by 9Machine Learning in BioinformaticsMonoclonal and Polyclonal Antibodies Researchvaccines and immunoinformatics approaches

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Pre-training with a rational approach for antibody sequence representation

Open access
Gao, Xiangrui, Cao, Changling, He, Chenfeng, Lai, Lipeng·2024·Frontiers in Immunology
DOI 10.3389/fimmu.2024.1468599PMC PMC11537868

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Introduction: Antibodies represent a specific class of proteins produced by the adaptive immune system in response to pathogens. Mining the information embedded in antibody amino acid sequences can benefit both antibody property prediction and novel therapeutic development. However, antibodies possess unique features that should be incorporated using specifically designed training methods, leaving room for improvement in pre-training models for antibody sequences. Methods: In this study, we present a Pre-trained model of Antibody sequences trained with a Rational Approach for antibodies (PARA). PARA employs a strategy conforming to antibody sequence patterns and an advanced natural language processing self-encoding model structure. This approach addresses the limitations of existing protein pre-training models, which primarily utilize language models without fully considering the differences between protein sequences and language sequences. Results: We demonstrate PARA’s performance on several tasks by comparing it to various published pre-training models of antibodies. The results show that PARA significantly outperforms existing models on these tasks, suggesting that PARA has an advantage in capturing antibody sequence information. Discussion: The antibody latent representation provided by PARA can substantially facilitate studies in relevant areas. We believe that PARA’s superior performance in capturing antibody sequence information offers significant potential for both antibody property prediction and the development of novel therapeutics. PARA is available at https://github.com/xtalpi-xic .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fimmu.2024.1468599

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The training and prediction process of PARA. We engineered a pre-training task that is targeted at antibody sequences to yield a specialized model for antibody sequence analysis. The model serves two primary functions: firstly, it aids in predicting the masked regions of antibody sequences, which is instrumental in antibody engineering by narrowing down the vast number of potential sequence combinations. Secondly, the latent representations of antibody sequences obtained through PARA encoding can be utilized as input for downstream tasks, particularly beneficial when the dataset for such tasks is limited in size.
Figure 1 The training and prediction process of PARA. We engineered a pre-training task that is targeted at antibody sequences to yield a specialized model for antibody sequence analysis. The model serves two primary functions: firstly, it aids in predicting the masked regions of antibody sequences, which is instrumental in antibody engineering by narrowing down the vast number of potential sequence combinations. Secondly, the latent representations of antibody sequences obtained through PARA encoding can be utilized as input for downstream tasks, particularly beneficial when the dataset for such tasks is limited in size.CC BY · PubMed Central
Figure 2 Masking strategies for Antibody sequences. For presentation purposes, we have isolated the segment of the antibody heavy chain Fv region that includes the CDR-H3, indicated in red. The cyan color signifies the areas that will be masked during pre-training. With the PARA approach, we first mask portions of the CDR-H3, and then apply span masking to other regions.
Figure 2 Masking strategies for Antibody sequences. For presentation purposes, we have isolated the segment of the antibody heavy chain Fv region that includes the CDR-H3, indicated in red. The cyan color signifies the areas that will be masked during pre-training. With the PARA approach, we first mask portions of the CDR-H3, and then apply span masking to other regions.CC BY · PubMed Central
Figure 3 (A) Heavy and light chain matching model. We utilize different pre-trained models to extract the latent representations of heavy and light chains. During training, each training sample consists of one positive sample and multiple negative samples. We calculate their matching scores separately and compute the loss using infoNCE loss. (B) Antibody binding prediction model. We designed an antibody affinity prediction model, as illustrated in the figure. Firstly, we employed a pre-trained deep learning model to extract the latent representation of the antibody VH sequence. Subsequently, we selected only the latent representation of the CDR-H3 and reduced its dimensionality and flattened it using a single-layer multilayer perceptron (MLP) projection layer. Finally, this representation was fed into a two-layer MLP classification layer for a binary classification task, where antibodies with affinity were labeled as 1 and those without affinity were labeled as 0. We used cross-entropy as the loss function to optimize the model.
Figure 3 (A) Heavy and light chain matching model. We utilize different pre-trained models to extract the latent representations of heavy and light chains. During training, each training sample consists of one positive sample and multiple negative samples. We calculate their matching scores separately and compute the loss using infoNCE loss. (B) Antibody binding prediction model. We designed an antibody affinity prediction model, as illustrated in the figure. Firstly, we employed a pre-trained deep learning model to extract the latent representation of the antibody VH sequence. Subsequently, we selected only the latent representation of the CDR-H3 and reduced its dimensionality and flattened it using a single-layer multilayer perceptron (MLP) projection layer. Finally, this representation was fed into a two-layer MLP classification layer for a binary classification task, where antibodies with affinity were labeled as 1 and those without affinity were labeled as 0. We used cross-entropy as the loss function to optimize the model.CC BY · PubMed Central
Figure 4 Model performance in antibody sequence prediction. Top1 and Top3 denote the highest accuracy and the average of the top three accuracies, respectively. Specific subsequences of the antibody sequences were masked to test model robustness, with Top1 accuracy compared using slice notation, e.g., “10:-5” for masking from the 10th to the 5th position from the end. (A) PARA Mask ablation study evaluated recovery accuracy in the CDR3 region, with “*” indicating random masking and no “*” indicating PARA Mask. PARA 70 was selected as the standard configuration after showing superior performance at various masking ratios. (B) PARA outperformed other open-source models AntiBERTy and ABLang in CDR-H3 masking prediction, maintaining high accuracy even with truncated sequences. (C) All models showed high Top-1 and Top-3 accuracies in CDR-H1 and CDR-H2 regions, despite sequence truncation. (D) PARA and AntiBERTy outperformed ABLang in FR1 and FR2 region masking prediction. (E) PARA excelled in masking prediction accuracy in FR1, FR2, and FR3 regions. (F) PARA maintained high accuracy in FR1, FR2, FR3, and FR4 regions, even with extensive sequence masking, outperforming other models. PARA consistently achieved the best results in experiments (B–F), demonstrating optimal and stable performance under sequence truncation or extensive masking, unlike the significant performance decline in other models.
Figure 4 Model performance in antibody sequence prediction. Top1 and Top3 denote the highest accuracy and the average of the top three accuracies, respectively. Specific subsequences of the antibody sequences were masked to test model robustness, with Top1 accuracy compared using slice notation, e.g., “10:-5” for masking from the 10th to the 5th position from the end. (A) PARA Mask ablation study evaluated recovery accuracy in the CDR3 region, with “*” indicating random masking and no “*” indicating PARA Mask. PARA 70 was selected as the standard configuration after showing superior performance at various masking ratios. (B) PARA outperformed other open-source models AntiBERTy and ABLang in CDR-H3 masking prediction, maintaining high accuracy even with truncated sequences. (C) All models showed high Top-1 and Top-3 accuracies in CDR-H1 and CDR-H2 regions, despite sequence truncation. (D) PARA and AntiBERTy outperformed ABLang in FR1 and FR2 region masking prediction. (E) PARA excelled in masking prediction accuracy in FR1, FR2, and FR3 regions. (F) PARA maintained high accuracy in FR1, FR2, FR3, and FR4 regions, even with extensive sequence masking, outperforming other models. PARA consistently achieved the best results in experiments (B–F), demonstrating optimal and stable performance under sequence truncation or extensive masking, unlike the significant performance decline in other models.CC BY · PubMed Central
Figure 5 Amino acid distribution in CDR-H3 We obtained distributions from the CDR-H3 of the real antibody sequences, the CDR-H3 of the AntiBERTy predicted sequences, and the CDR-H3 of the PARA predicted sequences. In this test, we focused on the diversity of predictions. We tested IGHV1, IGHV2, IGHV3, IGHV4, IGHV5, IGHV6 and IGHV7 and present some of the results. Using data from the test set, we found that the CDR-H3 distributions derived from PARA are closer to the original distribution.
Figure 5 Amino acid distribution in CDR-H3 We obtained distributions from the CDR-H3 of the real antibody sequences, the CDR-H3 of the AntiBERTy predicted sequences, and the CDR-H3 of the PARA predicted sequences. In this test, we focused on the diversity of predictions. We tested IGHV1, IGHV2, IGHV3, IGHV4, IGHV5, IGHV6 and IGHV7 and present some of the results. Using data from the test set, we found that the CDR-H3 distributions derived from PARA are closer to the original distribution.CC BY · PubMed Central
Figure 6 auROC and auPRC for Heavy Light Chains Matching. We trained AntiBERTy and PARA as antibody sequences encoder on OAS antibody paired sequences with contrast learning, respectively. The matching model obtained by PARA performs better with the same training set and training method. The “Chance” line in the auROC plot indicates the baseline performance of a random classifier, with an AUC of 0.5, signifying that any predictive accuracy above this line is better than random guessing.
Figure 6 auROC and auPRC for Heavy Light Chains Matching. We trained AntiBERTy and PARA as antibody sequences encoder on OAS antibody paired sequences with contrast learning, respectively. The matching model obtained by PARA performs better with the same training set and training method. The “Chance” line in the auROC plot indicates the baseline performance of a random classifier, with an AUC of 0.5, signifying that any predictive accuracy above this line is better than random guessing.CC BY · PubMed Central
Figure 7 (A) Results of Models on HER2 Antibody Dataset. The logistic regression (LR) benchmark is derived from the methodology of Mason et al. To ensure the robustness of our model evaluation, we employed a repeated random sub-sampling validation approach, where the data was randomly partitioned into training, testing, and validation sets five separate times. The results tabulated represent the mean performance metrics across these five iterations. In addition to accuracy, we computed recall and precision metrics to provide a comprehensive analysis of the predictive capabilities of each pre-trained model under investigation. (B) Results of Models on HER2 Antibody Dataset. HER2 1/100 signifies training with only the top 1% of the original dataset to compare pre-trained models with a baseline model initiated from scratch in a data-constrained setting. The figure presents the average validation and test accuracy (Acc), recall, and precision metrics, calculated over five repeated validation trials, for each model. (C) Results of Models on Emibetuzumab mutants Dataset. The figure presents the average validation and test accuracy (Acc), recall, and precision metrics, calculated over five repeated validation trials, for each model.
Figure 7 (A) Results of Models on HER2 Antibody Dataset. The logistic regression (LR) benchmark is derived from the methodology of Mason et al. To ensure the robustness of our model evaluation, we employed a repeated random sub-sampling validation approach, where the data was randomly partitioned into training, testing, and validation sets five separate times. The results tabulated represent the mean performance metrics across these five iterations. In addition to accuracy, we computed recall and precision metrics to provide a comprehensive analysis of the predictive capabilities of each pre-trained model under investigation. (B) Results of Models on HER2 Antibody Dataset. HER2 1/100 signifies training with only the top 1% of the original dataset to compare pre-trained models with a baseline model initiated from scratch in a data-constrained setting. The figure presents the average validation and test accuracy (Acc), recall, and precision metrics, calculated over five repeated validation trials, for each model. (C) Results of Models on Emibetuzumab mutants Dataset. The figure presents the average validation and test accuracy (Acc), recall, and precision metrics, calculated over five repeated validation trials, for each model.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 8 references

Research works printed but not linked: 14

  1. Dondelinger, 2018, Understanding the significance and implications of antibody numbering and antigen-binding surface/residue definition, Front Immunol, 9, 10.3389/fimmu.2018.02278 · DOI · Paper page
  2. Leem, 2022, Deciphering the language of antibodies using self-supervised learning, Patterns, 3, 100535, 10.1016/j.patter.2022.100513 · DOI
  3. Ruffolo, 2021, Deciphering antibody affinity maturation with language models and weakly supervised learning. NeurIPS Workshop on Machine Learning in Structural Biology, Preprint at arXiv, 10.48550/arXiv.2112.07782 · DOI
  4. Olsen, 2022, Ablang: an antibody language model for completing antibody sequences, Bioinforma. Adv, 2, vbac046, 10.1093/bioadv/vbac046 · DOI · Paper page
  5. Steinegger, 2018, Clustering huge protein sequence sets in linear time, Nat Commun, 9, 2542, 10.1038/s41467-018-04964-5 · DOI · Paper page
  6. Rives, 2021, Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences, Proc Natl Acad Sci United States Am, 118, 10.1073/pnas.2016239118 · DOI · Paper page
  7. Vaswani, 2017, Attention is all you need, Adv Neural Inf Process Syst, 30, 10.48550/arXiv.1706.03762 · DOI · Paper page
  8. Shaw, 2018, Self-attention with relative position representations, 10.18653/v1/N18-2074 · DOI · Paper page
  9. Lin, 2017, Focal loss for dense object detection, 10.1109/ICCV.2017.324 · DOI · Paper page
  10. Shrivastava, 2016, Training region-based object detectors with online hard example mining, 10.1109/CVPR.2016.89 · DOI · Paper page
  11. Zhang, 2022, Unsupervised sentence representation via contrastive learning with mixing negatives, 10.1609/aaai.v36i10.21428 · DOI
  12. Marks, 2020, How repertoire data are changing antibody science, J Biol Chem, 295, 10.1074/jbc.REV120.010181 · DOI
  13. Makowski, 2022, Co-optimization of therapeutic antibody affinity and specificity using machine learning models that generalize to novel mutational space, Nat Commun, 13, 3788, 10.1038/s41467-022-31457-3 · DOI
  14. Wei, 2022, Masked feature prediction for self-supervised visual pre-training, 10.1109/CVPR52688.2022.01426 · DOI · Paper page

Metadata

Authors
Gao, X., Cao, C., He, C., & Lai, L.
Affiliations
  • XtalPi (China) (CN)
  • Beijing Institute of Technology (CN)
Corresponding author
  • Gao, Xiangrui
  • Cao, Changling
  • Lai, Lipeng
ORCID
Venue
Frontiers in Immunology
Publisher
Frontiers Media SA
Published
2024-10-23
Volume
15
Pages
1468599-1468599
ISSN
1664-3224 (ISSN-L)
DOI
10.3389/fimmu.2024.1468599
PMC
PMC11537868
OpenAlex
W4403701477
Cited by
9
References
8
Citations by year
2026: 1 · 2025: 4 · 2024: 1
FWCI
0.81
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Immunology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Immunology and Microbiology › Immunology › Monoclonal and Polyclonal Antibodies Research
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
Keywords
pre-trained models, Pará, therapeutic antibodies, protein language models
MeSH terms
Amino Acid Sequence, Antibodies, Humans, Natural Language Processing, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Pre-training with a rational approach for antibody sequence representation. https://app.cochat.ai/papers/10.3389/fimmu.2024.1468599
The paper
Gao, X., Cao, C., He, C., & Lai, L. (2024). Pre-training with a rational approach for antibody sequence representation. Frontiers in Immunology. https://doi.org/10.3389/fimmu.2024.1468599