CoChat
Sign Up
Cited by 128Visual perception and processing mechanismsNeural dynamics and brain functionVisual Attention and Saliency Detection

A neural network trained for prediction mimics diverse features of biological neurons and perception

Open access
Lotter, William, Kreiman, Gabriel, Cox, David·2020·Nature Machine Intelligence
DOI 10.1038/s42256-020-0170-9PMC PMC8291226
Open this paper in CoChat — free account

Abstract

Recent work has shown that convolutional neural networks (CNNs) trained on image recognition tasks can serve as valuable models for predicting neural responses in primate visual cortex. However, these models typically require biologically-infeasible levels of labeled training data, so this similarity must at least arise via different paths. In addition, most popular CNNs are solely feedforward, lacking a notion of time and recurrence, whereas neurons in visual cortex produce complex time-varying responses, even to static inputs. Towards addressing these inconsistencies with biology, here we study the emergent properties of a recurrent generative network that is trained to predict future video frames in a self-supervised manner. Remarkably, the resulting model is able to capture a wide variety of seemingly disparate phenomena observed in visual cortex, ranging from single-unit response dynamics to complex perceptual motion illusions, even when subjected to highly impoverished stimuli. These results suggest potentially deep connections between recurrent predictive neural network models and computations in the brain, providing new leads that can enrich both fields.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-020-0170-9

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • PredNet units respond to illusory contours in the Kanizsa illusion.

    “there is evidence that PredNet units respond to illusory contours in the Kanizsa illusion, the perception of which depends on inferring the presence of objects from the wider context (Lotter et al. 2020).”

    From Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) · cites this paper as Lotter et al. 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 76

  1. [1]Yamins DLK, Hong H, Cadieu CF, Solomon EA, Seibert D, and DiCarlo JJ Proceedings of the National Academy of Sciences 111(23), 8619– 8624 (2014). PMC4060707 24812127
  2. [2]Yamins DLK and DiCarlo JJ Nature Neuroscience 19, 356– 365 (2016). 26906502
  3. [3]Khaligh-Razavi S-M and Kriegeskorte N PLOS Computational Biology 10, 1– 29 (2014). PMC4222664 25375136
  4. [4]Nayebi A, Bear D, Kubilius J, Kar K, Ganguli S, Sussillo D, DiCarlo JJ, and Yamins DLK In Advances in Neural Information Processing Systems, 5290– 5301. (2018).
  5. [6]Tang H, Schrimpf M, Lotter W, Moerman C, Paredes A, Ortega Caro J, Hardesty W, Cox D, and Kreiman G Proceedings of the National Academy of Sciences 115(35), 8835– 8840 (2018). PMC6126774 30104363
  6. [7]Kar K, Kubilius J, Schmidt K, Issa EB, and DiCarlo JJ Nature Neuroscience 22, 974– 983 (2019). 31036945 PMC8785116
  7. [8]Lotter W, Kreiman G, and Cox DD In International Conference on Learning Representations. (2017).
  8. [9]Rao RPN and Ballard DH Nature Neuroscience 2, 79– 87 (1999). 10195184
  9. [10]Friston K Philos Trans R Soc Lond B Biol Sci 360, 815– 836 (2005). 15937014 PMC1569488
  10. [11]Spratling MW Neural Computation 24, 60– 103 (2012). 22023197
  11. [12]Wen H, Han K, Shi J, Zhang Y, Culurciello E, and Liu Z In Proceedings of the 35th International Conference on Machine Learning, volume 80, 5266– 5275, (2018).
  12. [13]Geiger A, Lenz P, Stiller C, and Urtasun R International Journal of Robotics Research 32, 1231– 1237 (2013).
  13. [14]Softky WR In Advances in Neural Information Processing Systems, 809– 815. (1996).
  14. [15]Lotter W, Kreiman G, and Cox D In International Conference on Learning Representations. (2016).
  15. [16]Mathieu M, Couprie C, and LeCun Y In International Conference on Learning Representations. (2016).
  16. [17]Srivastava N, Mansimov E, and Salakhutdinov R In Proceedings of the 32nd International Conference on Machine Learning, volume 37, 843– 852. (2015).
  17. [18]Dosovitskiy A and Koltun V In International Conference on Learning Representations. (2017).
  18. [19]Finn C, Goodfellow IJ, and Levine S In Advances in Neural Information Processing Systems, 64– 72. (2016).
  19. [22]Luo Z, Peng B, Huang D-A, Alahi A, and Fei-Fei L In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7101– 7110, (2017).
  20. [23]Lee AX, Zhang R, Ebert F, Abbeel P, Finn C, and Levine S arXiv abs/1804.01523 (2018). · arXiv
  21. [24]Villegas R, Pathak A, Kannan H, Erhan D, Le QV, and Lee H Advances in Neural Information Processing Systems, 81– 91 (2019).
  22. [25]Villegas R, Yang J, Hong S, Lin X, and Lee H International Conference on Learning Representations (2017).
  23. [26]Denton E and Fergus R Proceedings of the 35th International Conference on Machine Learning, 1174– 1183 (2018).
  24. [27]Babaeizadeh M, Finn C, Erhan D, Campbell RH, and Levine S International Conference on Learning Representations (2018).
  25. [28]Wang Y, Gao Z, Long M, Wang J, and Yu PS arXiv abs/1804.06300 (2018). · arXiv
  26. [29]Finn C and Levine S In International Conference on Robotics and Automation, 2786– 2793, (2017).
  27. [30]Hsieh J-T, Liu B, Huang D-A, Fei-Fei LF, and Niebles JC In Advances in Neural Information Processing Systems, 517– 526. (2018).
  28. [31]Kalchbrenner N, van den Oord A, Simonyan K, Danihelka I, Vinyals O, Graves A, and Kavukcuoglu K In Proceedings of the 34th International Conference on Machine Learning, volume 70, 1771– 1779, (2017).
  29. [32]Qiu J, Huang G, and Lee TS In Advances in Neural Information Processing Systems, 2662– 2673. (2019).
  30. [33]Wang Y, Jiang L, Yang M-H, Li L-J, Long M, and Li F-F International Conference on Learning Representations (2019).
  31. [34]Wang Y, Long M, Wang J, Gao Z, and Yu PS In Advances in Neural Information Processing Systems, 879– 888. (2017).
  32. [35]Liu W, Luo W, Lian D, and Gao S The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6536– 6545 (2018).
  33. [36]Tandiya N, Jauhar A, Marojevic V, and Reed JH In 2018 IEEE International Conference on Communications Workshops (ICC Workshops), (2018).
  34. [37]Ebert F, Finn C, Dasari S, Xie A, Lee AX, and Levine S arXiv abs/1812.00568 (2018). · arXiv
  35. [38]Rao RPN and Sejnowski TJ In Advances in Neural Information Processing Systems, 164– 170. (2000).
  36. [39]Summerfield C, Egner T, Greene M, Koechlin E, Mangels J, and Hirsch J Science 314, 1311– 1314 (2006). 17124325
  37. [40]Bastos AM, Usrey WM, Adams RA, Mangun GR, Fries P, and Friston KJ Neuron 76, 695– 711 (2012). 23177956 PMC3777738
  38. [41]Kanai R, Komura Y, Shipp S, and Friston K Philos Trans R Soc Lond B Biol Sci 370 (2015). PMC4387510 25823866
  39. [42]Srinivasan MV, Laughlin SB, and Dubs A Proceedings of the Royal Society of London B: Biological Sciences 216, 427– 459 (1982). 6129637
  40. [44]Murray SO, Kersten D, Olshausen BA, Schrater P, and Woods DL Proceedings of the National Academy of Sciences 99, 15164– 15169 (2002). PMC137561 12417754
  41. [45]Spratling MW Journal of Neuroscience 30, 3531– 3543 (2010). 20203213 PMC6634102
  42. [46]Jehee JFM and Ballard DH PLOS Computational Biology 5, 1– 10 (2009). PMC2670540 19412529
  43. [47]Kumar S, Sedley W, Nourski KV, Kawasaki H, Oya H, Patterson RD III, M. A. H., Friston KJ, and Griffiths TD Journal of Cognitive Neuroscience 23, 3084– 3094 (2011). 21452943 PMC3821983
  44. [48]Zelano C, Mohanty A, and Gottfried JA Neuron 72, 178– 187 (2011). 21982378 PMC3190127
  45. [49]Mumford D Biological Cybernetics 66, 241– 251 (1992). 1540675
  46. [50]Hochreiter S and Schmidhuber J Neural Computation 9, 1735– 1780 (1997). 9377276
  47. [51]Shi X, Chen Z, Wang H, Yeung D, Wong W, and Woo W In Advances in Neural Information Processing Systems, 802– 810. (2015).
  48. [52]Hubel DH and Wiesel TN The Journal of Physiology 195, 215– 243 (1968). 4966457 PMC1557912
  49. [53]Nassi JJ, Lomber SG, and Born RT Journal of Neuroscience 33, 8504– 8517 (2013). 23658187 PMC3690087
  50. [54]Schmolesky MT, Wang Y, Hanes DP, Thompson KG, Leutgeb S, Schall JD, and Leventhal AG Journal of Neurophysiology 79, 3272– 3278 (1998). 9636126
  51. [55]Hung CP, Kreiman G, Poggio T, and DiCarlo JJ Science 310, 863– 866 (2005). 16272124
  52. [56]Meyer T and Olson CR Proceedings of the National Academy of Sciences 108, 19401– 19406 (2011). PMC3228439 22084090
  53. [57]Watanabe E, Kitaoka A, Sakamoto K, Yasugi M, and Tanaka K Frontiers in Psychology 9, 345 (2018). 29599739 PMC5863044
  54. [59]Lee TS and Nguyen M Proceedings of the National Academy of Sciences 98, 1907– 1911 (2001). PMC29355 11172049
  55. [60]Nijhawan R Nature 370, 256– 257 (1994). 8035873
  56. [61]Mackay DM Nature 181, 507– 508 (1958). 13517199
  57. [62]Eagleman DM and Sejnowski TJ Science 287, 2036– 2038 (2000). 10720334
  58. [63]Khoei MA, Masson GS, and Perrinet LU PLOS Computational Biology 13, 1– 31 (2017). PMC5268412 28125585
  59. [64]Hogendoorn H and Burkitt AN eNeuro 6 (2019). PMC6506824 31064839
  60. [65]Wojtach WT, Sung K, Truong S, and Purves D Proceedings of the National Academy of Sciences 105, 16338– 16343 (2008). PMC2566991 18852459
  61. [66]Zhu M and Rozell CJ PLOS Computational Biology 9, 1– 15 (2013). PMC3757072 24009491
  62. [67]Chalk M, Marre O, and Tkačik G Proceedings of the National Academy of Sciences 115, 186– 191 (2018). PMC5776796 29259111
  63. [68]Singer Y, Teramoto Y, Willmore BD, Schnupp JW, King AJ, and Harper NS eLife 7 (2018). PMC6108826 29911971
  64. [69]Hunsberger E and Eliasmith C arXiv abs/1611.05141 (2016). · arXiv
  65. [70]Boerlin M, Machens CK, and Denève S PLOS Computational Biology 9, 1– 16 (2013). PMC3828152 24244113
  66. [72]Nøkland A In Advances in Neural Information Processing Systems, 1037– 1045. (2016).
  67. [73]Lillicrap TP, Cownden D, Tweed DB, and Akerman CJ Nature Communications 7, 13276 (2016). PMC5105169 27824044
  68. [74]Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, and Bengio Y In Advances in Neural Information Processing Systems, 2672– 2680. (2014).
  69. [75]Kingma DP and Welling M In International Conference on Learning Representations. (2014).
  70. [77]Kingma DP and Ba J In International Conference on Learning Representations. (2015).
  71. [78]Adelson EH and Bergen JR J. Opt. Soc. Am. A 2, 284– 299 (1985). 3973762
  72. [79]McIntosh L, Maheswaranathan N, Nayebi A, Ganguli S, and Baccus S In Advances in Neural Information Processing Systems, 1369– 1377. (2016). 28729779 PMC5515384
  73. [80]Dura-Bernal S, Wennekers T, and Denham SL PLOS ONE 7, 1– 25 (2012). PMC3489785 23139765
  74. [81]Krizhevsky A, Sutskever I, and Hinton GE In Advances in Neural Information Processing Systems 25, 1097– 1105. (2012).
  75. [82]Simonyan K and Zisserman A International Conference on Learning Representations (2015).
  76. [83]He K, Zhang X, Ren S, and Sun J In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770– 778, (2016).

Cited by

Sorted by citations

Authors
Lotter, W., Kreiman, G., & Cox, D.
Affiliations
  • Harvard University (US)
  • Boston Children's Hospital (US)
Corresponding author
Lotter, William
ORCID
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2020-04-20
Volume
2
Issue
4
Pages
210-219
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-020-0170-9
PMC
PMC8291226
OpenAlex
W3016954138
Cited by
128
References
12
Citations by year
2026: 17 · 2025: 20 · 2024: 12 · 2023: 24 · 2022: 14 · 2021: 22 · 2020: 12 · 2019: 7
FWCI
9.61
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • ODNI | Intelligence Advanced Research Projects Activity (D16PC00002)
  • National Science Foundation (CCF-1231216, IIS 1409097)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
PredNet, visual cortex, recurrent neural networks, convolutional neural networks, neural response prediction, computational neuroscience

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review