CoChat
Sign Up
PreprintCited by 2Neural and Behavioral Psychology StudiesDecision-Making and Behavioral EconomicsCognitive Science and Education Research

A Unified Theory of Dual-Process Control

Open access
Moskovitz, Ted, Miller, Kevin, Sahani, Maneesh, Botvinick, Matthew M.·2022·arXiv
DOI 10.48550/arxiv.2211.07036arXiv 2211.07036
Open this paper in CoChat — free accountOpen PDF

Abstract

Dual-process theories play a central role in both psychology and neuroscience, figuring prominently in fields ranging from executive control to reward-based learning to judgment and decision making. In each of these domains, two mechanisms appear to operate concurrently, one relatively high in computational complexity, the other relatively simple. Why is neural information processing organized in this way? We propose an answer to this question based on the notion of compression. The key insight is that dual-process structure can enhance adaptive behavior by allowing an agent to minimize the description length of its own behavior. We apply a single model based on this observation to findings from research on executive control, reward-based learning, and judgment and decision making, showing that seemingly diverse dual-process phenomena can be understood as domain-specific consequences of a single underlying set of computational principles.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.07036

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The meta-learning framework offers the means to account for multiple computational costs instead of just a single one, such as a cost for implementing the default policy and one for deviating from it.

    “Importantly, their study highlights that the meta-learning framework offers the means to account for multiple computational costs instead of just a single one - in this case, a cost for implementing the default policy and one for deviating from it.”

    From Meta-learned models of cognition (2024) · cites this paper as Moskovitz, Miller, Sahani, and Botvinick (2022)

  • The Moskovitz et al. model captures behavioral phenomena from the cognitive control literature and known effects in decision-making and reinforcement learning tasks, thereby linking the three domains.

    “They demonstrate that this model is not only able to capture behavioral phenomena from the cognitive control literature but also known effects in decision-making and reinforcement learning tasks, thereby linking the three domains.”

    From Meta-learned models of cognition (2024) · cites this paper as Moskovitz, Miller, Sahani, and Botvinick (2022)

  • The Moskovitz et al. model learns a simple default policy that can be overwritten by a more complex one if necessary.

    “Their model learns a simple default policy - similar to the model of Binz and Schulz (2022a) - that can be overwritten by a more complex one if necessary.”

    From Meta-learned models of cognition (2024) · cites this paper as Moskovitz, Miller, Sahani, and Botvinick (2022)

  • Moskovitz et al. proposed a meta-learned model that can adjust computational resources as needed.

    “Moskovitz, Miller, Sahani, and Botvinick (2022) proposed a meta-learned model with such characteristics.”

    From Meta-learned models of cognition (2024) · cites this paper as Moskovitz, Miller, Sahani, and Botvinick (2022)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 54

  1. [1][1] James, W., Burkhardt, F., Bowers, F. & Skrupskelis, I. K. The principles of psychology , vol. 1 (Macmillan London, 1890). -
  2. [2][2] Diamond, A. Executive functions. Annual review of psychology 64 , 135-168 (2013). -
  3. [5][5] Perez, O. D. & Dickinson, A. A theory of actions and habits: The interaction of rate correlation and contiguity systems in free-operant behavior. Psychological Review 127 , 945 (2020). -
  4. [8][8] Lieder, F. & Griffiths, T. L. Strategy selection as rational metareasoning. Psychological review 124 , 762 (2017). -
  5. [9][9] Rougier, N. P., Noelle, D. C., Braver, T. S., Cohen, J. D. & O'Reilly, R. C. Prefrontal cortex and flexible cognitive control: Rules without symbols. Proceedings of the National Academy of Sciences 102 , 7338-7343 (2005). -
  6. [14][14] Miller, K. J., Shenhav, A. & Ludvig, E. A. Habits without values. Psychological review 126 , 292 (2019). -
  7. [16][16] De Neys, W. & Goel, V. Heuristics and biases in the brain: Dual neural pathways for decision making. In Neuroscience of decision making , 137-154 (Psychology Press, 2011). -
  8. [17][17] Miller, E. K. & Cohen, J. D. An integrative theory of prefrontal cortex function. Annual review of neuroscience 24 , 167-202 (2001). -
  9. [19][19] Lieberman, M. D. Social cognitive neuroscience: a review of core processes. Annu. Rev. Psychol. 58 , 259-289 (2007). -
  10. [20][20] O'Reilly, R. C., Nair, A., Russin, J. L. & Herd, S. A. How sequential interactive processing within frontostriatal loops supports a continuum of habitual to controlled processing. Frontiers in Psychology 11 , 380 (2020). -
  11. [25][25] Ritz, H., Leng, X. & Shenhav, A. Cognitive control as a multivariate optimization problem. Journal of Cognitive Neuroscience 1-23 (2022). -
  12. [28][28] Sutton, R. S. & Barto, A. G. Reinforcement learning: An introduction (MIT press, 2018). -
  13. [29][29] Kirk, R., Zhang, A., Grefenstette, E. & Rockt¨ aschel, T. A survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:2111.09794 (2021). - · arXiv
  14. [30][30] MacKay, D. J. Information Theory, Inference, and Learning Algorithms (Cambridge University Press, 2003). -
  15. [31][31] Hutter, M. Universal artificial intelligence: Sequential decisions based on algorithmic probability (Springer Science & Business Media, 2004). -
  16. [32][32] Feldman, J. The simplicity principle in perception and cognition. Wiley Interdisciplinary Reviews: Cognitive Science 7 , 330-340 (2016). -
  17. [33][33] Gr¨ unwald, P. D. The minimum description length principle (MIT press, 2007). -
  18. [34][34] Teh, Y. et al. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems 30 (2017). -
  19. [35][35] Tirumala, D. et al. Behavior priors for efficient reinforcement learning. arXiv preprint arXiv:2010.14274 (2020). - · arXiv
  20. [36][36] Kingma, D. P., Salimans, T. & Welling, M. Variational dropout and the local reparameterization trick. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M. & Garnett, R. (eds.) Advances in Neural Information Processing Systems , vol. 28 (Curran Associates, Inc., 2015). URL https://proceedings.neurips. cc/paper/2015/file/bc7316929fe1545bf0b98d114ee3ecb8-Paper.pdf . -
  21. [37][37] Hinton, G. E. & Van Camp, D. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory , 5-13 (1993). -
  22. [38][38] Molchanov, D., Ashukha, A. & Vetrov, D. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning , 2498-2507 (PMLR, 2017). -
  23. [39][39] Moskovitz, T., Kao, T.-C., Sahani, M. & Botvinick, M. Minimum description length control. In The Eleventh International Conference on Learning Representations (2023). URL https://openreview.net/forum?id=oX3tGygjW1q . -
  24. [43][43] Stroop, J. R. Studies of interference in serial verbal reactions. Journal of experimental psychology 18 , 643 (1935). -
  25. [44][44] Herd, S. A., Banich, M. T. & O'reilly, R. C. Neural mechanisms of cognitive control: An integrative model of stroop task performance and fmri data. Journal of cognitive neuroscience 18 , 22-32 (2006). -
  26. [51][51] Miller, K. J., Brody, C. D. & Botvinick, M. M. Identifying model-based and model-free patterns in behavior on multi-step tasks. bioRxiv (2016). -
  27. [56][56] Dickinson, A. Actions and habits: the development of behavioural autonomy. Philosophical Transactions of the Royal Society of London. B, Biological Sciences 308 , 67-78 (1985). -
  28. [61][61] Dickinson, A. Omission learning after instrumental pretraining. The Quarterly Journal of Experimental Psychology: Section B 51 , 271-286 (1998). -
  29. [64][64] Lai, L. & Gershman, S. J. Policy compression: An information bottleneck in action selection. In Psychology of Learning and Motivation , vol. 74, 195-232 (Elsevier, 2021). -
  30. [66][66] Binz, M., Gershman, S. J., Schulz, E. & Endres, D. Heuristics from bounded meta-learned inference. Psychological review (2022). -
  31. [68][68] Newell, B. R. & Shanks, D. R. Take the best or look at the rest? factors influencing' one-reason' decision making. Journal of Experimental Psychology: Learning, Memory, and Cognition 29 , 53 (2003). -
  32. [69][69] MacLeod, C. M. & Dunbar, K. Training and stroop-like interference: evidence for a continuum of automaticity. Journal of Experimental Psychology: Learning, Memory, and Cognition 14 , 126 (1988). -
  33. [70][70] Cohen, J., Dunbar, K. & Mcclelland, J. Cohen jd, mcclelland jl, dunbar k. on the control of automatic processes: a parallel distributed processing account of the stroop effect. psychol rev 97: 332-361. Psychological review 97 , 332-61 (1990). -
  34. [71][71] Riveland, R. & Pouget, A. A neural model of task compositionality with natural language instructions. bioRxiv (2022). -
  35. [72][72] O'Reilly, R. C., Herd, S. A. & Pauli, W. M. Computational models of cognitive control. Current opinion in neurobiology 20 , 257-261 (2010). -
  36. [77][77] Akam, T., Costa, R. & Dayan, P. Simple plans or sophisticated habits? state, transition and learning interactions in the two-step task. PLOS Computational Biology 11 , 1-25 (2015). URL https://doi.org/10.1371/journal. pcbi.1004648 . - · DOI
  37. [78][78] Wang, J. X. et al. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience 21 , 860-868 (2018). URL https://doi.org/10.1038/ s41593-018-0147-8 . -
  38. [79][79] Ashby, F. G., Ennis, J. M. & Spiering, B. J. A neurobiological theory of automaticity in perceptual categorization. Psychological review 114 , 632 (2007). -
  39. [81][81] Greenstreet, F. et al. Action prediction error: a value-free dopaminergic teaching signal that drives stable learning. bioRxiv (2022). URL https://www.biorxiv.org/content/early/2022/09/14/2022.09.12.507572 . https://www.biorxiv.org/content/early/2022/09/14/2022.09.12. 507572.full.pdf . -
  40. [84][84] Balcarras, M., Ardid, S., Kaping, D., Everling, S. & Womelsdorf, T. Attentional selection can be predicted by reinforcement learning of task-relevant stimulus features weighted by value-independent stickiness. Journal of cognitive neuroscience 28 , 333-349 (2016). -
  41. [92][92] Galashov, A. et al. Information asymmetry in kl-regularized rl. arXiv preprint arXiv:1905.01240 (2019). - · arXiv
  42. [93][93] Goyal, A. et al. Infobot: Transfer and exploration via the information bottleneck. In International Conference on Learning Representations (2018). -
  43. [94][94] Moskovitz, T., Arbel, M., Parker-Holder, J. & Pacchiano, A. Towards an understanding of default policies in multitask policy optimization. In CampsValls, G., Ruiz, F. J. R. & Valera, I. (eds.) Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , vol. 151 of Proceedings of Machine Learning Research , 10661-10686 (PMLR, 2022). URL https://proceedings.mlr.press/v151/moskovitz22a.html . -
  44. [95][95] Russin, J., O'Reilly, R. C. & Bengio, Y. Deep learning needs a prefrontal cortex. Work Bridging AI Cogn Sci 107 , 603-616 (2020). -
  45. [96][96] Mnih, V. et al. Asynchronous methods for deep reinforcement learning. In Balcan, M. F. & Weinberger, K. Q. (eds.) Proceedings of The 33rd International Conference on Machine Learning , vol. 48 of Proceedings of Machine Learning Research , 1928-1937 (PMLR, New York, New York, USA, 2016). URL https: //proceedings.mlr.press/v48/mniha16.html . -
  46. [97][97] Hinton, G., Vinyals, O. & Dean, J. Distilling the knowledge in a neural network (2015). URL http://arxiv.org/abs/1503.02531 . - · arXiv
  47. [98][98] Rusu, A. A. et al. Policy distillation (2015). URL https://arxiv.org/abs/ 1511.06295 . -
  48. [99][99] Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518 , 529-533 (2015). URL http://dx.doi.org/10.1038/nature14236 . - · DOI · Paper page
  49. [100][100] Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization (2014). URL http://arxiv.org/abs/1412.6980 . Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015. - · arXiv
  50. [101][101] Sutton, R. S., Precup, D. & Singh, S. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112 , 181-211 (1999). URL https://www.sciencedirect.com/science/ article/pii/S0004370299000521 . -
  51. [102][102] Tassa, Y. et al. Deepmind control suite (2018). URL https://arxiv.org/abs/ 1801.00690 . -
  52. [104][104] Song, H. F., Yang, G. R. & Wang, X.-J. Training excitatory-inhibitory recurrent neural networks for cognitive tasks: A simple and flexible framework. PLOS Computational Biology 12 , 1-30 (2016). URL https://doi.org/10. 1371/journal.pcbi.1004792 . -
  53. [106][106] Atkinson, A. On the measurement of inequality. Journal of Economic Inequality 6 , 277-283 (2008). -
  54. [108][108] Grunwald, P. A tutorial introduction to the minimum description length principle. arXiv preprint math/0406077 (2004). · arXiv

Cited by

Sorted by citations

Authors
Moskovitz, T., Miller, K., Sahani, M., & Botvinick, M. M.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • Gatsby Computational Neuroscience Unit (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2211.07036
arXiv
2211.07036
OpenAlex
W4309132691
Cited by
2
References
0
Citations by year
2025: 1 · 2024: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Cognitive Science and Education Research
Keywords
dual-process theory, computational complexity, description length, compression, adaptive behavior, executive function, reward learning, judgment and decision making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review