CoChat
Sign Up
Cited by 2Domain Adaptation and Few-Shot LearningNeural Networks and ApplicationsAdvanced Neural Network Applications

A biologically inspired architecture with switching units can learn to generalize across backgrounds

Open access
Voina, Doris, Shea-Brown, Eric, Mihalas, Stefan·2023·Neural Networks
DOI 10.1016/j.neunet.2023.09.014PMC PMC10843013

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Humans and other animals navigate different environments effortlessly, their brains rapidly and accurately generalizing across contexts. Despite recent progress in deep learning, this flexibility remains a challenge for many artificial systems. Here, we show how a bio-inspired network motif can explicitly address this issue. We do this using a dataset of MNIST digits of varying transparency, set on one of two backgrounds of different statistics that define two contexts: a pixel-wise noise or a more naturalistic background from the CIFAR-10 dataset. After learning digit classification when both contexts are shown sequentially, we find that both shallow and deep networks have sharply decreased performance when returning to the first background - an instance of the catastrophic forgetting phenomenon known from continual learning. To overcome this, we propose the bottleneck-switching network or switching network for short. This is a bio-inspired architecture analogous to a well-studied network motif in the visual cortex, with additional "switching" units that are activated in the presence of a new background, assuming a priori a contextual signal to turn these units on or off. Intriguingly, only a few of these switching units are sufficient to enable the network to learn the new context without catastrophic forgetting through inhibition of redundant background features. Further, the bottleneck-switching network can generalize to novel contexts similar to contexts it has learned. Importantly, we find that - again as in the underlying biological network motif, recurrently connecting the switching units to network layers is advantageous for context generalization.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.neunet.2023.09.014

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing agent memory architectures lack selective forgetting mechanisms, causing either catastrophic forgetting at context boundaries or information overload within them.

    “Large language models deployed as autonomous agents face critical memory limitations, lacking selective forgetting mechanisms that lead to either catastrophic forgetting at context boundaries or information overload within them.”

    From FadeMem: Biologically-Inspired Forgetting for Efficient Agent Memory (2026) · cites this paper as [14]

    The Introduction (p. 2) reports catastrophic forgetting when visual-classification networks learn two backgrounds sequentially, and Section 3.3 (p. 5) finds that some contextual strategies do not prevent it. The paper does not establish that autonomous LLM agents lack selective forgetting mechanisms or experience information overload within context boundaries; those claims extend beyond its visual continual-learning setting. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 53

  1. [2][2] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems , volume 32, pages 9448-9458, 2019. -
  2. [3][3] S. Beery, G. V. Horn, and P. Perona. Recognition in terra incognita. In European Conference on Computer Vision (ECCV) , pages 472-489, 2018. -
  3. [5][5] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan. Domain separation networks. In Advances in Neural Information Processing Systems , pages 343-351, 2016. -
  4. [7][7] R. Chattopadhyay, J. Ye, S. Panchanathan, W. Fan, and I. Davidson. Multi-source domain adaptation and its application to early detection of fatigue. In In Proc. KDD , volume 6, pages 717-725, 2011. -
  5. [8][8] M. Choi, A. Torralba, and A. Willsky. Context models and out-of-context objects. Pattern Recognition Letters , 33(7):853-862, 2012. -
  6. [9][9] W. Chu, F. de la Torre, and J. Cohn. Selective transfer machine for personalized facial action unit detection. In Proceedings of the IEEE Conference on cOmputer Vision and Pattern Recognition , pages 3515-3522, 2013. -
  7. [10][10] G. Csurka. Domain adaptation for visual applications: A comprehensive survey. In G. Csurka, editor, Domain Adaptation in Computer Vision Applications. Advances in Computer Vision and Pattern Recognition . Springer, Cham, 2017. -
  8. [11][11] H. Daume. Frustratingly easy domain adaptation. In Proceedings of ACL. , pages 256-263, 2007. -
  9. [12][12] T. de Vries, I. Misra, C. Wang, and L. van der Maaten. Does object recognition work for everyone? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 52-59, 2019. -
  10. [13][13] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML: Proceedings of the 31st International Conference on International Conference on Machine Learning , volume 32, 2014. -
  11. [14][14] T. Draelos, N. Miner, C. Lamb, C. Vineyard, K. Carlson, C. James, and J. Aimone. Neurogenesis deep learning. In International Joint Conference on Neural Networks (IJCNN) , pages 526-533, 2017. -
  12. [15][15] L. Duan, I. Tsang, and D. Xu. Domain transfer multiple kernel learning. In IEEE Trans Pattern Anal Mach Intell. , volume 34, pages 465-479, 2012. -
  13. [16][16] L. Duan, D. Xu, and S. Chang. Exploiting web images for event recognition in consumer videos: a multiple source domain adaptation approach. In IEEE 2012 conference on computer vision and pattern recognition , pages 1338-1345, 2012. -
  14. [17][17] R. French. Pseudo-recurrent connectionist networks: An approach to the sensitivity-stability dilemma. In Connection Science , volume 9, pages 353-380, 1997. -
  15. [18][18] X. Glorot, A. Bordes, and Y. Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th international conference on machine learning , volume 27, pages 97-110, 2011. -
  16. [19][19] B. Gong, K. Grauman, and F. Sha. Connecting the dots with landmarks: Discriminatively learning domaininvariant features for unsupervised domain adaptation. In International Conference on Machine Learning , pages 222-230, 2013. -
  17. [20][20] G. Hinton and D. Plaut. Using fast weights to deblur old memories. In Proceedings of the Annual Conference of the Cognitive Science Society , pages 177-186, 1987. -
  18. [23][23] R. Kemker and C. Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference on Learning Representations (ICLR) , 2018. -
  19. [24][24] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. Rusu, and et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114(13): 3521-3526, 2017. -
  20. [26][26] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 2013. -
  21. [27][27] F. Li, S. Pan, O. Jin, Q. Yang, and X. Zhu. Cross-domain co-extraction of sentiment and topic lexicons. In Proceedings of the 50th annual meeting of the association for computational linguistics long papers , pages 410-419, 2012. -
  22. [29][29] Z. Li and D. Hoiem. Learning without forgetting. In ECCV 2016: Computer Vision , pages 614-629, 2016. -
  23. [30][30] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. Zitnic. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV) , 2014. -
  24. [31][31] M. Liu and O. Tuzel. Coupled generative adversarial networks. pages 469-477, 2016. -
  25. [32][32] M. Long, J. Wang, G. Ding, J. Sun, and P. Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the 2013 IEEE international conference on computer vision , pages 2200-2207, 2013. -
  26. [34][34] A. Mallya and S. Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7765-7773, 2018. -
  27. [35][35] A. Mallya, D. Davis, and S. Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. ECCV 2018: Computer Vision , pages 72-88, 2018. -
  28. [38][38] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. -
  29. [39][39] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. pages 1717-1724, 2014. -
  30. [40][40] S. Pan, I. Tsang, J. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. In IEEE Trans Neural Netw. , volume 22, pages 199-210, 2009. -
  31. [41][41] G. Parisi, R. Kemker, J. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural Networks , 113:57-71, 2019. -
  32. [42][42] R. Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review , 97(2):285-308, 1990. -
  33. [43][43] A. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: An astounding baseline for recognition,. In IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 512-519, 2014. -
  34. [45][45] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, and et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115(3):211-252, 2015. -
  35. [47][47] S. Sagawa and L. P. WK Pang, HB Tatsunori. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations , 2020. -
  36. [48][48] R. Shetty, M. Fritz, and B. Schiele. Adversarial scene editing: Automatic object removal from weak supervision. In Neural Information Processing Systems (NeurIPS) , 2018. -
  37. [49][49] Y. Shi and F. Sha. Information-theoretical learning of discriminative clusters for unsupervised domain adaptation. In Proceedings of the 29th international conference on machine learning , pages 1-8, 2012. -
  38. [50][50] H. Shin, J. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. In NIPS: Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 2994-3003, 2017. -
  39. [51][51] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR , abs/1409.1556, 2015. - · arXiv
  40. [52][52] B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops , pages 443-450. Springer, 2016. -
  41. [53][53] T. Tommasi, F. Orabona, and B. Caputo. Safety in numbers: learning categories from few examples with multi model knowledge transfer. In IEEE Conf Comput Vision Pattern Recog. , pages 3081-3088, 2010. -
  42. [54][54] A. Torralba and A. Efros. Unbiased look at dataset bias. CVPR , pages 1521-1528, 2011. -
  43. [55][55] D. Voina, S. Recanatesi, B. Hu, E. Shea-Brown, and S. Mihalas. Single circuit in v1 capable of switching contexts during movement using vip population as a switch. bioRxiv, 2020. -
  44. [56][56] A. Wang, A. Narayanan, and O. Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. In European Conference on Computer Vision (ECCV) , 2020. -
  45. [57][57] K. Weiss, T. M. Khoshgoftaar, and D. D. Wang. A survey of transfer learning. Journal of Big Data , 3(1):9, 2016. -
  46. [59][59] R. Xia, C. Zong, X. Hu, and E. Cambria. Feature ensemble plus sample selection: domain adaptation for sentiment classification. In IEEE Intell Syst. , volume 28, pages 10-18, 2013. -
  47. [61][61] T. Xiao, J. Zhang, K. Yang, Y. Peng, and Z. Zhang. Error-driven incremental learning in deep convolutional neural network for large-scale image classification. In Proceedings of the ACM International Conference on Multimedia , pages 177-186, 2014. -
  48. [62][62] T. Xiao, H. Li, W. Ouyang, and X. Wang. Learning deep feature representations with domain guided dropout for person re-identification. In In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1249-1258, 2016. -
  49. [63][63] Y. Yao and G. Doretto. Boosting for transfer learning with multiple sources. In Proceedings of the IEEE computer society conference on computer vision and pattern recognition , pages 1855-1862, 2010. -
  50. [64][64] J. Yoon, E. Yang, J. Lee, and S. Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations (ICLR) , 2018. -
  51. [67][67] F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In ICML: Proceedings of the 34th International Conference on Machine Learning , volume 70, 2017. -
  52. [68][68] G. Zhou, K. Sohn, and H. Lee. Online incremental feature learning with denoising autoencoders. In International Conference on Artificial Intelligence and Statistics , pages 1453-1461, 2012. -
  53. [69][69] Z. Zhu, L. Xie, and A. Yuille. Object recognition without and without objects. In International Joint Conference on Artificial Intelligence , 2017.

Cited by

Sorted by citations

Authors
Voina, D., Shea-Brown, E., & Mihalas, S.
Affiliations
University of Washington (US)
Corresponding author
Voina, Doris
ORCID
Voina, Doris
Venue
Neural Networks
Publisher
Elsevier BV
Published
2023-11
Volume
168
Pages
615-630
ISSN
0893-6080 (ISSN-L)
DOI
10.1016/j.neunet.2023.09.014
PMC
PMC10843013
OpenAlex
W4386813250
Cited by
2
References
6
Citations by year
2026: 1 · 2025: 1
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Neural Networks (acceptedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institutes of Health (5 R90 DA 033461-08)
  • University of Washington
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
catastrophic forgetting, continual learning, context generalization, switching unit, biologically inspired architecture, background invariance, recurrent connectivity
MeSH terms
Brain, Generalization, Psychological, Humans, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review