The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning
Open accessAbstract
Computer vision models suffer from a phenomenon known as catastrophic forgetting when learning novel concepts from continuously shifting training data. Typical solutions for this continual learning problem require extensive rehearsal of previously seen data, which increases memory costs and may violate data privacy. Recently, the emergence of large-scale pre-trained vision transformer models has enabled prompting approaches as an alternative to data-rehearsal. These approaches rely on a key-query mechanism to generate prompts and have been found to be highly resistant to catastrophic forgetting in the well-established rehearsal-free continual learning setting. However, the key mechanism of these methods is not trained end-to-end with the task sequence. Our experiments show that this leads to a reduction in their plasticity, hence sacrificing new task accuracy, and inability to benefit from expanded parameter capacity. We instead propose to learn a set of prompt components which are assembled with input-conditioned weights to produce input-conditioned prompts, resulting in a novel attention-based end-to-end key-query scheme. Our experiments show that we outperform the current SOTA method DualPrompt on established benchmarks by as much as 4.5% in average final accuracy. We also outperform the state of art by as much as 4.4% accuracy on a continual learning benchmark which contains both class-incremental and domain-incremental task shifts, corresponding to many practical settings. Our code is available at https://github.com/GT-RIPL/CODA-Prompt
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Li, Z., & Hoiem, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2773081 · Cited by 4,202
iCaRL: Incremental Classifier and Representation Learning
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., & Lampert, C. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.587 · Cited by 3,988
Continual Learning Through Synaptic Intelligence
Zenke, F., Poole, B., & Ganguli, S. (2017) · arXiv · doi:10.48550/arxiv.1703.04200 · Cited by 1,784
Moment Matching for Multi-Source Domain Adaptation
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., & Wang, B. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00149 · Cited by 1,621
Memory Aware Synapses: Learning What (not) to Forget
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., & Tuytelaars, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_9 · Cited by 1,590
Gradient Episodic Memory for Continual Learning
Lopez-Paz, D., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1706.08840 · Cited by 1,403
Large Scale Incremental Learning
Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., & Fu, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00046 · Cited by 1,313
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250
Learning a Unified Classifier Incrementally via Rebalancing
Hou, S., Pan, X., Loy, C. C., Wang, Z., & Lin, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00092 · Cited by 1,195
End-to-End Incremental Learning
Castro, F. M., Marín-Jiménez, M. J., Guil, N., Schmid, C., & Alahari, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01258-8_15 · Cited by 1,184
Efficient Lifelong Learning with A-GEM
Chaudhry, A., Ranzato, M., Rohrbach, M., & Elhoseiny, M. (2018) · arXiv · doi:10.48550/arxiv.1812.00420 · Cited by 852
Catastrophic forgetting, rehearsal and pseudorehearsal
ROBINS, A. (1995) · Connection Science · doi:10.1080/09540099550039318 · Cited by 833
Learning to Prompt for Continual Learning
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., & Pfister, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00024 · Cited by 778
Measuring Catastrophic Forgetting in Neural Networks
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11651 · Cited by 672
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion
Yin, H., Molchanov, P., Alvarez, J. M., Li, Z., Mallya, A., Hoiem, D., Jha, N. K., & Kautz, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00874 · Cited by 540
Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
Yin, W., Hay, J., & Roth, D. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1404 · Cited by 537
Experience Replay for Continual Learning
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. P., & Wayne, G. (2018) · arXiv · doi:10.48550/arxiv.1811.11682 · Cited by 529
Three scenarios for continual learning
van de Ven, G. M., & Tolias, A. S. (2019) · Lirias · doi:10.48550/arxiv.1904.07734 · Cited by 523
Brain-inspired replay for continual learning with artificial neural networks
van de Ven, G. M., Siegelmann, H. T., & Tolias, A. S. (2020) · Nature Communications · doi:10.1038/s41467-020-17866-2 · Cited by 474
Dark Experience for General Continual Learning: a Strong, Simple Baseline
Buzzega, P., Boschini, M., Porrello, A., Abati, D., & Calderara, S. (2020) · arXiv · doi:10.48550/arxiv.2004.07211 · Cited by 473
DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19809-0_36 · Cited by 472
Online continual learning in image classification: An empirical survey
Mai, Z., Li, R., Jeong, J., Quispe, D., Kim, H., & Sanner, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.10.021 · Cited by 410
Quantifying the Carbon Emissions of Machine Learning
Lacoste, A., Luccioni, A., Schmidt, V., & Dandres, T. (2019) · arXiv · doi:10.48550/arxiv.1910.09700 · Cited by 408 · cited 1×
“This could incur high financial [26] and environmental [33] costs if the replacement is frequent.”
BackgroundDyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
Douillard, A., Rame, A., Couairon, G., & Cord, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00907 · Cited by 357
Rainbow Memory: Continual Learning with a Memory of Diverse Samples
Bang, J., Kim, H., Yoo, Y., Ha, J.-W., & Choi, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00812 · Cited by 354
Continuous learning in single-incremental-task scenarios
Maltoni, D., & Lomonaco, V. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.03.010 · Cited by 309
On Tiny Episodic Memories in Continual Learning
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H. S., & Ranzato, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10486 · Cited by 298
FearNet: Brain-Inspired Model for Incremental Learning
Kemker, R., & Kanan, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10563 · Cited by 296
Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning
Ostapenko, O., Puscas, M., Klein, T., Jähnichen, P., & Nabi, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01158 · Cited by 283
Predicting the Computational Cost of Deep Learning Models
Justus, D., Brennan, J., Bonner, S., & McGough, A. S. (2018) · 2018 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2018.8622396 · Cited by 278
A Neural Dirichlet Process Mixture Model for Task-Free Continual LearningPreprint
Lee, S., Ha, J., Zhang, D., & Kim, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00689 · Cited by 257
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning
Smith, J., Hsu, Y.-C., Balloch, J., Shen, Y., Jin, H., & Kira, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00924 · Cited by 249
Self-citationOnline Continual Learning with Maximal Interfered RetrievalPreprint
Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Caccia, M., Lin, M., & Caccia, L. (2019) · arXiv (Cornell University) · Cited by 245
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
Hayes, T. L., & Kanan, C. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00118 · Cited by 222
(2019) · arXiv (Cornell University) · Cited by 214
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
Hsu, Y.-C., Liu, Y.-C., Ramasamy, A., & Kira, Z. (2018) · arXiv · doi:10.48550/arxiv.1810.12488 · Cited by 211 · cited 1×
“Class-incremental continual learning is challenging because the learner must support classification across all classes seen up to task n [25] (i.”
BackgroundSelf-citationOvercoming Catastrophic Forgetting With Unlabeled Data in the Wild
Lee, K., Lee, K., Shin, J., & Lee, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00040 · Cited by 203
Memory Efficient Experience Replay for Streaming Learning
Hayes, T. L., Cahill, N. D., & Kanan, C. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793982 · Cited by 200
Lifelong Learning via Progressive Distillation and Retrospection
Hou, S., Pan, X., Loy, C. C., Wang, Z., & Lin, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_27 · Cited by 190
Generative Models from the perspective of Continual Learning
Lesort, T., Caselles-Dupré, H., Garcia-Ortiz, M., Stoian, A., & Filliat, D. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851986 · Cited by 184
SS-IL: Separated Softmax for Incremental Learning
Ahn, H., Kwak, J., Lim, S., Bang, H., Kim, H., & Moon, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00088 · Cited by 177
Deep Generative Dual Memory Network for Continual LearningPreprint
Kamra, N., Gupta, U., & Liu, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10368 · Cited by 176
Functional Regularisation for Continual Learning with Gaussian ProcessesPreprint
Titsias, M. K., Schwarz, J., Matthews, A., Pascanu, R., & Teh, Y. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11356 · Cited by 168
Adversarial Continual Learning
Ebrahimi, S., Meier, F., Calandra, R., Darrell, T., & Rohrbach, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_23 · Cited by 162
Generative replay with feedback connections as a general strategy for continual learning
van de Ven, G. M., & Tolias, A. S. (2018) · Lirias · doi:10.48550/arxiv.1809.10635 · Cited by 161
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
Hayes, T. L., & Kanan, C. (2019) · arXiv · doi:10.48550/arxiv.1909.01520 · Cited by 150
CORe50: a New Dataset and Benchmark for Continuous Object Recognition
Lomonaco, V., & Maltoni, D. (2017) · arXiv · doi:10.48550/arxiv.1705.03550 · Cited by 135
R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning
Gao, Q., Zhao, C., Ghanem, B., & Zhang, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20050-2_25 · Cited by 128
A Closer Look at Rehearsal-Free Continual Learning *
Smith, J. S., Tian, J., Halbe, S., Hsu, Y.-C., & Kira, Z. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00239 · Cited by 108
Self-citationS-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental LearningPreprint
Wang, Y., Huang, Z., & Hong, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.12819 · Cited by 99
Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches
Lomonaco, V., Maltoni, D., & Pellegrini, L. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00131 · Cited by 95
Architecture Matters in Continual LearningPreprint
Mirzadeh, S. I., Chaudhry, A., Yin, D., Nguyen, T., Pascanu, R., Görür, D., & Farajtabar, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.00275 · Cited by 85
Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay
Choi, Y., El‐Khamy, M., & Lee, J. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00393 · Cited by 52
Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual LearnersPreprint
Li, D., Cao, G., Xu, Y., Cheng, Z., & Niu, Y. F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.04924 · Cited by 31
Incremental learning with self-organizing maps
Gepperth, A., & Karaoguz, C. (2017) · Workshop on Self-Organizing Maps · doi:10.1109/wsom.2017.8020021 · Cited by 12
Cited by
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
“Representative strategies include selecting the most relevant prompts from a prompt pool (L2P [458]), performing a weighted summation of the prompt pool with attention factors (CODAPrompt [402]), using explicitly task-sharing and task-specific prompts (DualPrompt [457]) or only task-specific prompts (S-Prompts [450]),…”
BackgroundMethodClass-Incremental Learning: A Survey
Zhou, D.-W., Wang, Q.-W., Qi, Z.-H., Ye, H.-J., Zhan, D.-C., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429383 · Cited by 376
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2023) · arXiv · doi:10.48550/arxiv.2312.10549 · Cited by 54
Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
Aleixo, E. L., Colonna, J. G., Cristo, M., & Fernandes, E. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.3966 · Cited by 54
Multimodal Parameter-Efficient Few-Shot Class Incremental Learning
D’Alessandro, M., Alonso, A., Calabrés, E., & Galar, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw60793.2023.00364 · Cited by 40
Metadata
- Authors
- Smith, J. S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., & Kira, Z.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 11909-11919
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386076680
- Cited by
- 334
- References
- 63
- FWCI
- 63.14
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.