CoChat
Sign Up
PreprintCited by 0Memory Processes and InfluencesNeural Networks and ApplicationsMemory and Neural Mechanisms

Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture

Open access
Park, Sangjun, Bak, JinYeong·2023·arXiv
DOI 10.48550/arxiv.2310.03052arXiv 2310.03052
Open this paper in CoChat — free accountOpen PDF

Abstract

Making neural networks remember over the long term has been a longstanding issue. Although several external memory techniques have been introduced, most focus on retaining recent information in the short term. Regardless of its importance, information tends to be fatefully forgotten over time. We present Memoria, a memory system for artificial neural networks, drawing inspiration from humans and applying various neuroscientific and psychological theories. The experimental results prove the effectiveness of Memoria in the diverse tasks of sorting, language modeling, and classification, surpassing conventional techniques. Engram analysis reveals that Memoria exhibits the primacy, recency, and temporal contiguity effects which are characteristics of human memory.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.03052

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs have long-term (episodic) and working memory.

    “LLMs have long-term (episodic) and working memory [24, 25, 27].”

    From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [25]

    Section 3.1 (page unavailable) says that Memoria has working and long-term memory, and Appendix D.2 says it can be applied to LLMs. The paper does not establish that LLMs generally have both memory types, or explicitly identify its long-term memory as episodic. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Research works printed but not linked: 75

  1. Andoni, A., Indyk, P., Laarhoven, T., Razenshteyn, I., and Schmidt, L. Practical and optimal lsh for angular distance. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28.
  2. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/ paper files/paper/2015/file/ 2823f4797102ce1a1aec05359cc16dd9- Paper.pdf.
  3. Antony, J. W., Ferreira, C. S., Norman, K. A., and Wimber, M. Retrieval as a fast route to memory consolidation. Trends in cognitive sciences, 21(8):573-576, 2017.
  4. Atkinson, R. and Shiffrin, R. Human memory: A proposed system and its control processes. vol- ume 2 of Psychology of Learning and Motiva- tion, pp. 89-195. Academic Press, 1968. doi: https://doi.org/10.1016/S0079-7421(08)60422-3. URL https://www.sciencedirect.com/science/ article/pii/S0079742108604223. · DOI · Paper page
  5. Atkinson, R. C. and Juola, J. F. Search and decision pro- cesses in recognition memory., pp. xiii, 299-xiii, 299. Contemporary developments in mathematical psychol- ogy: I. Learning, memory and thinking. W. H. Freeman, Oxford, England, 1974. ISBN 0716708485.
  6. Atkinson, R. C., Hermann, D. J., and Wescourt, K. T. Search processes in recognition memory., pp. xi, 386-xi, 386. Theories in cognitive psychology: The Loyola Sympo- sium. Lawrence Erlbaum, Oxford, England, 1974. ISBN 0470812281.
  7. Baddeley, A. D. and Hitch, G. The recency effect: Implicit learning with explicit retrieval? Memory & Cognition, 21(2):146-155, 1993. doi: 10.3758/BF03202726. URL https://doi.org/10.3758/BF03202726. · DOI
  8. Banino, A., Badia, A. P., Köster, R., Chadwick, M. J., Zambaldi, V., Hassabis, D., Barry, C., Botvinick, M., Kumaran, D., and Blundell, C. Memo: A deep net- work for flexible combination of episodic memories. In International Conference on Learning Representa- tions, 2020. URL https://openreview.net/ forum?id=rJxlc0EtDr.
  9. Baranchuk, D., Babenko, A., and Malkov, Y. Revisiting the inverted indices for billion-scale approximate nearest neighbors. In Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (eds.), Computer Vision -ECCV 2018, pp. 209-224, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01258-8.
  10. Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. CoRR, abs/2004.05150, 2020. URL https://arxiv.org/ abs/2004.05150. · arXiv
  11. Bjork, R. Retrieval as a memory modifier: An interpretation of negative recency & related phenomena. Information Processing and Cognition: The Loyola Symposium, 01 1975.
  12. Bjork, R. A. Retrieval practice and the maintenance of knowledge., pp. 396-401. Practical aspects of memory: Current research and issues, Vol. 1: Memory in everyday life. John Wiley & Sons, Oxford, England, 1988. ISBN 0-471-91234-4 (Hardcover).
  13. Bjork, R. A. and Whitten, W. B. Recency-sensitive retrieval processes in long-term free recall. Cogni- tive Psychology, 6(2):173-189, 1974. ISSN 0010- 0285. doi: https://doi.org/10.1016/0010-0285(74)90009- 7. URL https://www.sciencedirect.com/ science/article/pii/0010028574900097. · DOI
  14. Braun, E. K., Wimmer, G. E., and Shohamy, D. Retroactive and graded prioritization of memory by reward. Nature Communications, 9(1):4886, 2018. doi: 10.1038/s41467- 018-07280-0. URL https://doi.org/10.1038/ s41467-018-07280-0. · DOI
  15. Brown, J. Some tests of the decay theory of im- mediate memory. Quarterly Journal of Experimen- tal Psychology, 10(1):12-21, 1958. doi: 10.1080/ 17470215808416249. URL https://doi.org/ 10.1080/17470215808416249. · DOI
  16. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901.
  17. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a- Paper.pdf.
  18. Bulatov, A., Kuratov, Y., and Burtsev, M. Recurrent memory transformer. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 11079-11091.
  19. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/ paper files/paper/2022/file/ 47e288629a6996a17ce50b90a056a0e1- Paper-Conference.pdf.
  20. Burtsev, M. S. and Sapunov, G. V. Memory trans- former. CoRR, abs/2006.11527, 2020. URL https: //arxiv.org/abs/2006.11527. · arXiv
  21. Chung, J., Gülc ¸ehre, C ¸., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014. URL http: //arxiv.org/abs/1412.3555. · arXiv
  22. Craik, F. I. and Lockhart, R. S. Levels of process- ing: A framework for memory research. Jour- nal of Verbal Learning and Verbal Behavior, 11(6):671-684, 1972. ISSN 0022-5371. doi: https://doi.org/10.1016/S0022-5371(72)80001-X. URL https://www.sciencedirect.com/science/ article/pii/S002253717280001X. · DOI · Paper page
  23. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. ACL 2019 -57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pp. 2978- 2988, 1 2019. doi: 10.48550/arxiv.1901.02860. URL https://arxiv.org/abs/1901.02860v3. · DOI
  24. Davis, T., Xue, G., Love, B. C., Preston, A. R., and Poldrack, R. A. Global neural pattern similarity as a common basis for categorization and recognition mem- ory. Journal of Neuroscience, 34(22):7472-7484, 2014. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.3376- 13.2014. URL https://www.jneurosci.org/ content/34/22/7472. · DOI
  25. Deese, J. and Kaufman, R. A. Serial effects in recall of unor- ganized and sequentially organized verbal material. Jour- nal of Experimental Psychology, 54(3):180-187, 1957. ISSN 0022-1015(Print). doi: 10.1037/h0040536. URL https://doi.org/10.1037/h0040536. · DOI
  26. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423. · DOI · Paper page
  27. Gerstner, W. and Kistler, W. M. Mathematical formula- tions of hebbian learning. Biological Cybernetics, 87 (5):404-415, 2002. doi: 10.1007/s00422-002-0353-y. URL https://doi.org/10.1007/s00422-002- 0353-y. · DOI
  28. Ginns, P. Integrating information: A meta-analysis of the spatial contiguity and temporal conti- guity effects. Learning and Instruction, 16 (6):511-525, 2006. ISSN 0959-4752. doi: https://doi.org/10.1016/j.learninstruc.2006.10.001. URL https://www.sciencedirect.com/ science/article/pii/S0959475206000806. · DOI · Paper page
  29. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. CoRR, abs/1410.5401, 2014. URL http: //arxiv.org/abs/1410.5401. · arXiv
  30. Graves, A., Wayne, G., Reynolds, M., Harley, T., Dani- helka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., Badia, A. P., Hermann, K. M., Zwols, Y., Ostrovski, G., Cain, A., King, H., Summerfield, C., Blunsom, P., Kavukcuoglu, K., and Hassabis, D. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):471-476, October 2016. ISSN 00280836. URL http://dx.doi.org/10.1038/nature20101. · DOI · Paper page
  31. Guo, R., Sun, P., Lindgren, E., Geng, Q., Simcha, D., Chern, F., and Kumar, S. Accelerating large-scale in- ference with anisotropic vector quantization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th In- ternational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3887-3896. PMLR, 13-18 Jul 2020a. URL https:// proceedings.mlr.press/v119/guo20h.html.
  32. Guo, R., Sun, P., Lindgren, E., Geng, Q., Simcha, D., Chern, F., and Kumar, S. Accelerating large-scale in- ference with anisotropic vector quantization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th In- ternational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3887-3896. PMLR, 13-18 Jul 2020b. URL https:// proceedings.mlr.press/v119/guo20h.html.
  33. Hassabis, D., Kumaran, D., Summerfield, C., and Botvinick, M. Neuroscience-inspired artificial intelligence. Neu- ron, 95(2):245-258, 2017. ISSN 0896-6273. doi: https://doi.org/10.1016/j.neuron.2017.06.011. URL https://www.sciencedirect.com/science/ article/pii/S0896627317305093. · DOI · Paper page
  34. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9: 1735-1780, 11 1997. ISSN 08997667. doi: 10.1162/NECO.1997.9.8.1735. URL https: //www.researchgate.net/publication/ 13853244 Long Short-term Memory. · DOI · Paper page
  35. Journé, A., Rodriguez, H. G., Guo, Q., and Moraitis, T. Hebbian deep learning without feedback. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https://openreview.net/ forum?id=8gd4M-Rj1.
  36. Kahana, M. J. Associative retrieval processes in free recall. Memory & Cognition, 24(1):103-109, 1996. doi: 10.3758/BF03197276. URL https://doi.org/ 10.3758/BF03197276. · DOI
  37. Kahana, M. J. Computational models of memory search. Annual Review of Psychology, 71(1):107-138, 2020. doi: 10.1146/annurev-psych-010418-103358. URL https://doi.org/10.1146/annurev-psych- 010418-103358. PMID: 31567043. · DOI
  38. Kiesel, J., Mestre, M., Shukla, R., Vincent, E., Adineh, P., Corney, D., Stein, B., and Potthast, M. SemEval- 2019 task 4: Hyperpartisan news detection. In Pro- ceedings of the 13th International Workshop on Seman- tic Evaluation, pp. 829-839, Minneapolis, Minnesota, USA, June 2019. Association for Computational Lin- guistics. doi: 10.18653/v1/S19-2145. URL https: //aclanthology.org/S19-2145. · DOI · Paper page
  39. Kim, T., Cochez, M., Franc ¸ois-Lavet, V., Neerincx, M., and Vossen, P. A machine with short-term, episodic, and semantic memory systems. In Proceed- ings of the Thirty-Seventh AAAI Conference on Artifi- cial Intelligence and Thirty-Fifth Conference on Inno- vative Applications of Artificial Intelligence and Thir- teenth Symposium on Educational Advances in Artifi- cial Intelligence, AAAI'23/IAAI'23/EAAI'23. AAAI Press, 2023. ISBN 978-1-57735-880-0. doi: 10.1609/aaai.v37i1.25075. URL https://doi.org/ 10.1609/aaai.v37i1.25075. · DOI · Paper page
  40. Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. CoRR, abs/2001.04451, 2020. URL https://arxiv.org/abs/2001.04451. · arXiv
  41. Kuriscak, E., Marsalek, P., Stroffek, J., and Toth, P. G. Biological context of hebb learning in artificial neural networks, a review. Neurocom- puting, 152:27-35, 2015. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2014.11.022. URL https://www.sciencedirect.com/science/ article/pii/S0925231214015239. · DOI
  42. Le, H., Tran, T., and Venkatesh, S. Self-attentive associative memory. In III, H. D. and Singh, A. (eds.), Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 5682-5691. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/ le20b.html.
  43. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl- main.703. URL https://aclanthology.org/ 2020.acl-main.703. · DOI · Paper page
  44. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmented genera- tion for knowledge-intensive nlp tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 9459-9474.
  45. Curran Associates, Inc., 2020b. URL https://proceedings.neurips.cc/ paper files/paper/2020/file/ 6b493230205f780e1bc26945df7481e5- Paper.pdf.
  46. Limbacher, T. and Legenstein, R. H-mem: Har- nessing synaptic plasticity with hebbian memory networks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21627-21637.
  47. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper files/paper/2020/file/ f6876a9f998f6472cc26708e27444456- Paper.pdf.
  48. Martins, P. H., Marinho, Z., and Martins, A. F. T. ∞-former: Infinite memory transformer. 9 2021. doi: 10.48550/ arxiv.2109.00301. URL https://arxiv.org/abs/ 2109.00301v3. · DOI
  49. Meng, Y. and Rumshisky, A. Context-aware neural model for temporal information extraction. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 527-536, Melbourne, Australia, July 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/P18-1049. URL https: //aclanthology.org/P18-1049. · DOI
  50. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: //openreview.net/forum?id=Byj72udxe.
  51. Morrissey, M. D., Insel, N., and Takehara-Nishiuchi, K. Generalizable knowledge outweighs incidental details in prefrontal ensemble code over time. eLife, 6:e22177, feb 2017. ISSN 2050-084X. doi: 10.7554/eLife.22177. URL https://doi.org/10.7554/eLife.22177. · DOI
  52. Murdock Jr., B. B. The serial position effect of free re- call. Journal of Experimental Psychology, 64(5):482-488, 1962. ISSN 0022-1015(Print). doi: 10.1037/h0045106. URL https://doi.org/10.1037/h0045106. · DOI · Paper page
  53. Nairne, J. S. and Pandeirada, J. N. Adaptive memory: Re- membering with a stone-age brain. Current Directions in Psychological Science, 17(4):239-243, 2008. doi: 10.1111/j.1467-8721.2008.00582.x. URL https:// doi.org/10.1111/j.1467-8721.2008.00582.x. · DOI
  54. Poo, M.-m., Pignatelli, M., Ryan, T. J., Tonegawa, S., Bon- hoeffer, T., Martin, K. C., Rudenko, A., Tsai, L.-H., Tsien, R. W., Fishell, G., Mullins, C., Gonc ¸alves, J. T., Shtrah- man, M., Johnston, S. T., Gage, F. H., Dan, Y., Long, J., Buzsáki, G., and Stevens, C. What is memory? the present state of the engram. BMC Biology, 14(1):40, 2016. doi: 10.1186/s12915-016-0261-6. URL https: //doi.org/10.1186/s12915-016-0261-6. · DOI
  55. Raaijmakers, J. G. and Shiffrin, R. M. Sam: A the- ory of probabilistic search of associative memory. volume 14 of Psychology of Learning and Moti- vation, pp. 207-262. Academic Press, 1980. doi: https://doi.org/10.1016/S0079-7421(08)60162-0. URL https://www.sciencedirect.com/science/ article/pii/S0079742108601620. · DOI
  56. Raaijmakers, J. G. and Shiffrin, R. M. Search of asso- ciative memory. Psychological Review, 88(2):93-134, 1981. doi: 10.1037/0033-295X.88.2.93. URL https: //doi.org/10.1037/0033-295X.88.2.93. · DOI · Paper page
  57. Rae, J., Hunt, J. J., Danihelka, I., Harley, T., Senior, A. W., Wayne, G., Graves, A., and Lillicrap, T. Scaling memory- augmented neural networks with sparse reads and writes. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29.
  58. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/ paper files/paper/2016/file/ 3fab5890d8113d0b5a4178201dc842ad- Paper.pdf.
  59. Rae, J., Dyer, C., Dayan, P., and Lillicrap, T. Fast para- metric learning with activation memorization. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4228-4237. PMLR, 10-15 Jul 2018. URL https:// proceedings.mlr.press/v80/rae18a.html.
  60. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long- range sequence modelling. In International Conference on Learning Representations, 2020. URL https:// openreview.net/forum?id=SylKikSYDH.
  61. Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Adler, T., Kreil, D., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need. In International Conference on Learning Representa- tions, 2021. URL https://openreview.net/ forum?id=tL89RnzIiCd.
  62. Roediger, H. L. Recall as a self-limiting process. Memory & Cognition, 6(1):54-63, 1978. doi: 10.3758/BF03197428. URL https://doi.org/10.3758/BF03197428. · DOI
  63. Roediger, H. L. and Butler, A. C. The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15(1):20-27, 2011. ISSN 1364-6613. doi: https://doi.org/10.1016/j.tics.2010.09.003. URL https://www.sciencedirect.com/science/ article/pii/S1364661310002081. · DOI · Paper page
  64. Rumelhart, D. E. and McClelland, J. L. Learning Inter- nal Representations by Error Propagation, pp. 318-362. 1987.
  65. Rundus, D. and Atkinson, R. C. Rehearsal processes in free recall: A procedure for direct observation. Journal of Verbal Learning and Verbal Behav- ior, 9(1):99-105, 1970. ISSN 0022-5371. doi: https://doi.org/10.1016/S0022-5371(70)80015-9. URL https://www.sciencedirect.com/science/ article/pii/S0022537170800159. · DOI
  66. Shiffrin, R. M. and Atkinson, R. C. Storage and retrieval processes in long-term memory. Psychological Review, 76(2):179-193, 1969. doi: 10.1037/h0027277. URL https://doi.org/10.1037/h0027277. · DOI · Paper page
  67. Shiffrin, R. M. and Raaijmakers, J. The SAM retrieval model: A retrospective and prospective., pp. 69-86. Es- says in honor of William K. Estes, Vol. 1: From learning theory to connectionist theory; Vol. 2: From learning pro- cesses to cognitive processes. Lawrence Erlbaum Asso- ciates, Inc, Hillsdale, NJ, US, 1992. ISBN 0-8058-1097-8 (Hardcover); 0-8058-0759-4 (Hardcover).
  68. Song, S., Miller, K. D., and Abbott, L. F. Competitive heb- bian learning through spike-timing-dependent synaptic plasticity. Nature Neuroscience, 3(9):919-926, 2000. doi: 10.1038/78829. URL https://doi.org/10.1038/ 78829. · DOI
  69. van de Ven, G. M., Siegelmann, H. T., and Tolias, A. S. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11(1):4069, 2020. doi: 10.1038/s41467-020-17866-2. URL https: //doi.org/10.1038/s41467-020-17866-2. · DOI · Paper page
  70. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30.
  71. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa- Paper.pdf.
  72. Waugh, N. C. and Norman, D. A. Primary memory. Psy- chological Review, 72(2):89-104, 1965. doi: 10.1037/ h0021797. · DOI
  73. Wu, Y., Rabe, M. N., Hutchins, D., and Szegedy, C. Memorizing transformers. In International Conference on Learning Representations, 2022. URL https:// openreview.net/forum?id=TrjbxzRcnf-.
  74. Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., and Ahmed, A. Big bird: Transformers for longer sequences. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17283-17297.
  75. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper files/paper/2020/file/ c8512d142a2d849725f31a9a7a361ab9- Paper.pdf.

Authors
Park, S., & Bak, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.03052
arXiv
2310.03052
OpenAlex
W4387427464
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Microsoft Research
  • National Research Foundation (2021M3A9E4080780)
  • Ministry of Science and ICT, South Korea
  • Microsoft Research Asia
  • Institute for Information and Communications Technology Promotion
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
Keywords
long-term memory, forgetting, primacy effect, recency effect, temporal contiguity, external memory, language modeling, classification, sorting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review