The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Assessing the generalization capabilities of TCR binding predictors via peptide distance analysis
Open accessPublished version of a preprint
Abstract
Understanding the interaction between T Cell Receptors (TCRs) and peptide-bound Major Histocompatibility Complexes (pMHCs) is crucial for comprehending immune responses and developing targeted immunotherapies. While recent machine learning (ML) models show remarkable success in predicting TCR-pMHC binding within training data, these models often fail to generalize to peptides outside their training distributions, raising concerns about their applicability in therapeutic settings. Understanding and improving the generalization of these models is therefore critical to ensure real-world applications. To address this issue, we evaluate the effect of the distance between training and testing peptide distributions on ML model empirical risk assessments, using sequence-based and 3D structure-based distance metrics. In our analysis we use several state-of-the-art models for TCR-peptide binding prediction: Attentive Variational Information Bottleneck (AVIB), NetTCR-2.0 and -2.2, and ERGO II (pre-trained autoencoder) and ERGO II (LSTM). In this work, we introduce a novel approach for assessing the generalization capabilities of TCR binding predictors: the Distance Split (DS) algorithm. The DS algorithm controls the distance between training and testing peptides based on both sequence and structure, allowing for a more nuanced evaluation of model performance. We show that lower 3D shape similarity between training and test peptides is associated with a harder out-of-distribution task definition, which is more interesting when measuring the ability to generalize to unseen peptides. However, we observe the opposite effect when splitting using sequence-based similarity. These findings highlight the importance of using a distance-based splitting approach to benchmark models. This could then be used to estimate a confidence score on predictions on novel and unseen peptides, based on how different they are from the training ones. Additionally, our results may hint that employing 3D shape to complement sequence information could improve the accuracy of TCR-pMHC binding predictors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., et al. (2022) · doi:10.1101/2022.12.09.519842 · Cited by 2,369
De novo design of protein structure and function with RFdiffusion
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., et al. (2023) · Nature · doi:10.1038/s41586-023-06415-8 · Cited by 2,325
Identifying specificity groups in the T cell receptor repertoire
Glanville, J., Huang, H., Nau, A., Hatton, O., Wagar, L. E., Rubelt, F., Ji, X., Han, A., Krams, S. M., Pettus, C., et al. (2017) · Nature · doi:10.1038/nature22976 · Cited by 1,026
Flagged first, then by citations
Research works printed but not linked: 62
- B Alberts, 2017, Molecular biology of the cell, 10.1201/9781315735368 · DOI
- L Rowen, 1996, The complete 685-kilobase DNA sequence of the human beta T cell receptor locus, Science, 272, 1755, 10.1126/science.272.5269.1755 · DOI
- EW Hewitt, 2003, The MHC class I antigen presentation pathway: strategies for viral immune evasion, Immunology, 110, 163, 10.1046/j.1365-2567.2003.01738.x · DOI
- LC Eisenlohr, 2007, Rethinking peptide supply to MHC class I molecules, Nat Rev Immunol, 7, 403, 10.1038/nri2077 · DOI
- JE Smith-Garvin, 2009, T cell activation, Annu Rev Immunol, 27, 591, 10.1146/annurev.immunol.021908.132706 · DOI · Paper page
- D Feng, 2007, Structural evidence for a germline-encoded T cell receptor-major histocompatibility complex interaction ``codon’’, Nat Immunol, 8, 975, 10.1038/ni1502 · DOI
- J Rossjohn, 2015, T cell antigen receptor recognition of antigen-presenting molecules, Annu Rev Immunol, 33, 169, 10.1146/annurev-immunol-032414-112334 · DOI
- Q Qi, 2014, Diversity and clonal selection in the human T-cell repertoire, Proc Natl Acad Sci U S A, 111, 13139, 10.1073/pnas.1409155111 · DOI · Paper page
- SC Jameson, 2018, Understanding subset diversity in T cell memory, Immunity, 48, 214, 10.1016/j.immuni.2018.02.010 · DOI · Paper page
- KD Omilusik, 2019, Remembering to remember: T cell memory maintenance and plasticity, Curr Opin Immunol, 58, 89, 10.1016/j.coi.2019.04.009 · DOI
- Q He, 2019, Targeting cancers through TCR-peptide/MHC interactions, J Hematol Oncol, 12, 139, 10.1186/s13045-019-0812-8 · DOI
- AD Waldman, 2020, A guide to cancer immunotherapy: from T cell basic science to clinical practice, Nat Rev Immunol, 20, 651, 10.1038/s41577-020-0306-5 · DOI · Paper page
- SC Gilbert, 2012, T-cell-inducing vaccines - what’s the future, Immunology, 135, 19, 10.1111/j.1365-2567.2011.03517.x · DOI
- HE Heslop, 2013, T-cell therapy for viral infections, Hematol Am Soc Hematol Educ Program, 2013, 342, 10.1182/asheducation-2013.1.342 · DOI
- I Springer, 2021, Contribution of T cell receptor alpha and beta CDR3, MHC typing, V and J genes to peptide binding prediction, Front Immunol, 12, 664514, 10.3389/fimmu.2021.664514 · DOI
- A Montemurro, 2021, NetTCR-2.0 enables accurate prediction of TCR-peptide binding by using paired TCRα and β sequence data, Commun Biol, 4, 1060, 10.1038/s42003-021-02610-3 · DOI
- A Weber, 2021, TITAN: T-cell receptor specificity prediction with bimodal attention networks, Bioinformatics, 37, 10.1093/bioinformatics/btab294 · DOI
- F Grazioli, 2023, Attentive variational information bottleneck for TCR-peptide interaction prediction, Bioinformatics, 39, 10.1093/bioinformatics/btac820 · DOI
- I Springer, 2020, Prediction of specific TCR-peptide binding from large dictionaries of TCR-peptide pairs, Front Immunol, 11, 1803, 10.3389/fimmu.2020.01803 · DOI
- K Koyama, 2023, Attention network for predicting T-cell receptor-peptide binding can associate attention with interpretable protein structural properties, Front Bioinform, 3, 1274599, 10.3389/fbinf.2023.1274599 · DOI
- I Springer, 2021, Contribution of T cell receptor alpha and beta CDR3, MHC typing, v and j genes to peptide binding prediction, Front Immunol, 12, 664514, 10.3389/fimmu.2021.664514 · DOI
- R Vita, 2019, The Immune Epitope Database (IEDB): 2018 update, Nucleic Acids Res, 47, 10.1093/nar/gky1006 · DOI
- DV Bagaev, 2020, VDJdb in 2019: database extension, new analysis infrastructure and a T-cell receptor motif compendium, Nucleic Acids Res, 48, 10.1093/nar/gkz874 · DOI
- N Tickotsky, 2017, McPAS-TCR: a manually curated catalogue of pathology-associated T cell receptor sequences, Bioinformatics, 33, 2924, 10.1093/bioinformatics/btx286 · DOI
- KH Piepenbrink, 2009, Methods for quantifying T cell receptor binding affinities and thermodynamics, Methods Enzymol, 466, 359, 10.1016/S0076-6879(09)66015-8 · DOI
- F Grazioli, 2022, On TCR binding predictors failing to generalize to unseen peptides, Front Immunol, 13, 1014256, 10.3389/fimmu.2022.1014256 · DOI
- L Deng, 2023, Performance comparison of TCR-pMHC prediction tools reveals a strong data dependency, Front Immunol, 14, 1128326, 10.3389/fimmu.2023.1128326 · DOI
- S Mahajan, 2019, Benchmark datasets of immune receptor-epitope structural complexes, BMC Bioinformatics, 20, 490, 10.1186/s12859-019-3109-6 · DOI
- D Korpela, 2023, EPIC-TRACE: predicting TCR binding to unseen epitopes using attention and contextualized embeddings, Bioinformatics, 39, 10.1093/bioinformatics/btad743 · DOI
- P Moris, 2021, Current challenges for unseen-epitope TCR interaction prediction and a new perspective derived from image classification, Brief Bioinform, 22, 10.1093/bib/bbaa318 · DOI
- G Croce, 2024, Deep learning predictions of TCR-epitope interactions reveal epitope-specific chains in dual alpha T cells, Nat Commun, 15, 3211, 10.1038/s41467-024-47461-8 · DOI
- PJA Cock, 2009, Biopython: freely available Python tools for computational molecular biology and bioinformatics, Bioinformatics, 25, 1422, 10.1093/bioinformatics/btp163 · DOI · Paper page
- 2015, The PyMOL molecular graphics system, version 1.8
- M Shugay, 2018, VDJdb: a curated database of T-cell receptor sequences with known antigen specificity, Nucleic Acids Res, 46, 10.1093/nar/gkx760 · DOI
- V Jurtz, 2018, NetTCR: sequence-based prediction of TCR binding to peptide-MHC complexes using convolutional neural networks, BioRxiv, 433706
- R Wu, 2022, High-resolutionde novostructure prediction from primary sequence
- RA Wagner, 1974, The string-to-string correction problem, J ACM, 21, 168, 10.1145/321796.321811 · DOI
- S Henikoff, 1992, Amino acid substitution matrices from protein blocks, Proc Natl Acad Sci U S A, 89, 10915, 10.1073/pnas.89.22.10915 · DOI · Paper page
- PB Moore, 2022, The protein-folding problem: not yet solved, Science, 375, 507, 10.1126/science.abn9422 · DOI
- MF Jensen, 2024, NetTCR 2.2 - improved TCR specificity predictions by combining pan- and peptide-specific training strategies, loss-scaling and integration of sequence similarity, 10.7554/eLife.93934.2.sa3 · DOI
- M Mechelke, 2010, Robust probabilistic superposition and comparison of protein structures, BMC Bioinformatics, 11, 363, 10.1186/1471-2105-11-363 · DOI
- A Ballone, 2020, A new soaking procedure for X-ray crystallographic structural determination of protein-peptide complexes, Acta Crystallogr F Struct Biol Commun, 76, 501, 10.1107/S2053230X2001122X · DOI
- M van Breugel, 2022, Structural validation and assessment of AlphaFold2 predictions for centrosomal and centriolar proteins and their complexes, Commun Biol, 5, 312, 10.1038/s42003-022-03269-0 · DOI
- A Goswami, 2022, De novodesign of anti-variant COVID-19 Vaccine, 10.1101/2022.10.20.513049 · DOI · Paper page
- L Chang, 2023, Ranking peptide binders by affinity with AlphaFold**, Angewandte Chemie, 135, 10.1002/ange.202213362 · DOI
- LV Castorina, 2024, TIMED-Design: flexible and accessible protein sequence design with convolutional neural networks, Protein Eng Des Sel, 37, 10.1093/protein/gzae002 · DOI
- LV Castorina, 2023, PDBench: evaluating computational methods for protein-sequence design, Bioinformatics, 39, 10.1093/bioinformatics/btad027 · DOI
- H Berman, 2003, Announcing the worldwide Protein Data Bank, Nat Struct Biol, 10, 980, 10.1038/nsb1203-980 · DOI · Paper page
- CH Lee, 2020, Predicting cross-reactivity and antigen specificity of T cell receptors, Front Immunol, 11, 565096, 10.3389/fimmu.2020.565096 · DOI
- CH Coles, 2020, TCRs with distinct specificity profiles use different binding modes to engage an identical peptide-HLA complex, J Immunol, 204, 1943, 10.4049/jimmunol.1900915 · DOI
- GJ Kersh, 2001, Structural and functional consequences of altering a peptide MHC anchor residue, J Immunol, 166, 3345, 10.4049/jimmunol.166.5.3345 · DOI
- AT Nguyen, 2021, The pockets guide to HLA class I molecules, Biochem Soc Trans, 49, 2319, 10.1042/BST20210410 · DOI
- MI Barton, 2021, Effects of common mutations in the SARS-CoV-2 Spike RBD and its ligand, the human ACE2 receptor on binding affinity and kinetics, Elife, 10, 10.7554/eLife.70658 · DOI
- S Xue, 2024, Mutations in the SARS-CoV-2 spike receptor binding domain and their delicate balance between ACE2 affinity and antibody evasion, Protein Cell, 15, 403, 10.1093/procel/pwae007 · DOI
- Y Qi, 2020, DenseCPD: improving the accuracy of neural-network-based computational protein sequence design with DenseNet, J Chem Inf Model, 60, 1245, 10.1021/acs.jcim.0c00043 · DOI
- H Ji, 2024, Predicting TCR sequences for unseen antigen epitopes using structural and sequence features, Brief Bioinform, 25
- F Li, 2024, TCRcost: a deep learning model utilizing TCR 3D structure for enhanced of TCR-peptide binding, Front Genet, 15, 1346784, 10.3389/fgene.2024.1346784 · DOI
- P Bradley, 2023, Structure-based prediction of T cell receptor:peptide-MHC interactions, Elife, 12, 10.7554/eLife.82813 · DOI · Paper page
- M-DN Pham, 2024, epiTCR-KDA: knowledge distillation model on dihedral angles for TCR-peptide prediction
- O Carugo, 2007, Statistical validation of the root-mean-square-distance, a measure of protein structural proximity, Protein Eng Des Sel, 20, 33, 10.1093/protein/gzl051 · DOI
- A Postovskaya, 2024, tcrBLOSUM: an amino acid substitution matrix for sensitive alignment of distant epitope-specific TCRs, Brief Bioinform, 26, 10.1093/bib/bbae602 · DOI
- LV Castorina, 2025, From atoms to fragments: a coarse representation for functional and efficient protein design
Metadata
- Authors
- Castorina, L. V., Grazioli, F., Machart, P., Mösch, A., & Errica, F.
- Affiliations
- NEC (Germany) (DE)
- Corresponding author
- Castorina, Leonardo V.
- Venue
- PLOS One
- Publisher
- Public Library of Science (PLoS)
- Published
- 2025-05-20
- Volume
- 20
- Issue
- 5
- Pages
- e0324011
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC12091837
- OpenAlex
- W4410535766
- Cited by
- 7
- References
- 5
- Citations by year
- 2026: 6 · 2025: 1
- FWCI
- 2.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- NEC Laboratories Europe
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
- Keywords
- generalization, out-of-distribution evaluation, empirical risk, confidence estimation, benchmarking
- MeSH terms
- Machine Learning, Algorithms, Humans, Peptides, Protein Binding, Receptors, Antigen, T-Cell
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
