CoChat
Sign Up
PreprintCited by 71Reinforcement Learning in RoboticsBayesian Modeling and Causal InferenceLogic, Reasoning, and Knowledge

Causal Reasoning from Meta-reinforcement Learning

Open access
Dasgupta, Ishita, Wang, Jane, Chiappa, Silvia, Mitrovic, Jovana, Ortega, Pedro, Raposo, David, Hughes, Edward, Battaglia, Peter +2 more·2019·arXiv
DOI 10.48550/arxiv.1901.08162arXiv 1901.08162
Open this paper in CoChat — free accountOpen PDF

Abstract

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal causal reasoning algorithms also exist, in this paper we show that such reasoning can arise from model-free reinforcement learning, and suggest that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here. This work also offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1901.08162

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Meta-learning shifts most of the compute burden from inference time to training time, which is advantageous when training time is ample but fast answers are needed at run-time, and may therefore explain how people can perform intricate computations within a reasonable time-frame.

    “Meta-learning, on the other hand, 'shifts most of the compute burden from inference time to training time [which] is advantageous when training time is ample but fast answers are needed at run-time' (Dasgupta et al., 2019), and may therefore explain how people can perform such intricate computations within a reasonable time-frame.”

    From Meta-learned models of cognition (2024) · cites this paper as Dasgupta et al., 2019

  • Dasgupta et al. (2019) showed that meta-learning can be used to create models that draw causal inferences from observational data, select informative interventions, and make counterfactual predictions.

    “Dasgupta et al. (2019), however, showed that meta-learning can be used to create models that draw causal inferences from observational data, select informative interventions, and make counterfactual predictions.”

    From Meta-learned models of cognition (2024) · cites this paper as Dasgupta et al. (2019)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 39-48, 2016.
  2. M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems , pages 3981-3989, 2016.
  3. P. W. Battaglia, J. B. Hamrick, V . Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018. · arXiv
  4. A. P. Blaisdell, K. Sawa, K. J. Leising, and M. R. Waldmann. Causal reasoning in rats. Science , 311(5763):1020-1022, 2006.
  5. N. R. Bramley, P. Dayan, T. L. Griffiths, and D. A. Lagnado. Formalizing neuraths ship: Approximate algorithms for online causal learning. Psychological review , 124(3):301, 2017.
  6. K. Cho, B. Van Merri¨ enboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014. · arXiv
  7. Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. rl 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016. · arXiv
  8. L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 , 2018. · arXiv
  9. C. Finn, P. Abbeel, and S. Levine. Model-agnostic metalearning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 , 2017. · arXiv
  10. A. Forney, J. Pearl, and E. Bareinboim. Counterfactual datafusion for online reinforcement learners. In International Conference on Machine Learning , pages 1156-1164, 2017.
  11. Y. Ganin, T. Kulkarni, I. Babuschkin, S. M. Eslami, and O. Vinyals. Synthesizing programs for images using reinforced adversarial learning. arXiv preprint arXiv:1804.01118 , 2018. · arXiv
  12. N. D. Goodman, T. D. Ullman, and J. B. Tenenbaum. Learning a theory of causality. Psychological review , 118(1):110, 2011.
  13. A. Gopnik, D. M. Sobel, L. E. Schulz, and C. Glymour. Causal learning mechanisms in very young children: two-, three-, and four-year-olds infer causal relations from patterns of variation and covariation. Developmental psychology , 37(5): 620, 2001.
  14. A. Gopnik, C. Glymour, D. M. Sobel, L. E. Schulz, T. Kushnir, and D. Danks. A theory of causal learning in children: causal maps and bayes nets. Psychological review , 111(1):3, 2004.
  15. E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. arXiv preprint arXiv:1801.08930 , 2018. · arXiv
  16. M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart. arXiv preprint arXiv:1809.04474 , 2018. · arXiv
  17. T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, G. Dulac-Arnold, et al. Deep q-learning from demonstrations. arXiv preprint arXiv:1704.03732 , 2017. · arXiv
  18. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.
  19. A. Hyttinen, F. Eberhardt, and P . O. Hoyer. Experiment selection for causal discovery. The Journal of Machine Learning Research , 14(1):3041-3071, 2013.
  20. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012.
  21. D. A. Lagnado, T. Gerstenberg, and R. Zultan. Causal responsibility and counterfactuals. Cognitive science , 37(6): 1036-1073, 2013.
  22. F. Lattimore, T. Lattimore, and M. D. Reid. Causal bandits: Learning good interventions via causal inference. In Advances in Neural Information Processing Systems , pages 1181-1189, 2016.
  23. A. M. Leslie. The perception of causality in infants. Perception , 11(2):173-186, 1982.
  24. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR , abs/1602.01783, 2016. URL http://arxiv.org/ abs/1602.01783 . · arXiv
  25. J. Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, 2000.
  26. J. Pearl, M. Glymour, and N. P . Jewell. Causal inference in statistics: a primer . John Wiley & Sons, 2016.
  27. A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pages 1842-1850, 2016.
  28. R. Sen, K. Shanmugam, A. G. Dimakis, and S. Shakkottai. Identifying best interventions through online importance sampling. arXiv preprint arXiv:1701.02789 , 2017. · arXiv
  29. K. Shanmugam, M. Kocaoglu, A. G. Dimakis, and S. Vishwanath. Learning causal graphs with small interventions. In Advances in Neural Information Processing Systems , pages 3195-3203, 2015.
  30. P. Spirtes, C. N. Glymour, R. Scheines, D. Heckerman, C. Meek, G. Cooper, and T. Richardson. Causation, prediction, and search . MIT press, 2000.
  31. O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems , pages 3630-3638, 2016.
  32. J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to reinforcement learn. CoRR , abs/1611.05763, 2016. URL http://arxiv.org/abs/1611.05763 . · arXiv
  33. J. X. Wang, Z. Kurth-Nelson, D. Kumaran, D. Tirumala, H. Soyer, J. Z. Leibo, D. Hassabis, and M. Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature neuroscience , 21(6):860, 2018.

Cited by

Sorted by citations

Authors
Dasgupta, I., Wang, J., Chiappa, S., Mitrovic, J., Ortega, P., Raposo, D., Hughes, E., Battaglia, P., Botvinick, M., & Kurth-Nelson, Z.
Affiliations
Harvard University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1901.08162
arXiv
1901.08162
OpenAlex
W2909534157
Cited by
71
References
0
Citations by year
2026: 1 · 2025: 3 · 2024: 3 · 2023: 9 · 2022: 14 · 2021: 15 · 2020: 19 · 2019: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
Keywords
causal reasoning, meta-reinforcement learning, model-free reinforcement learning, recurrent neural networks, causal discovery, intervention, counterfactual prediction, causal inference from observational data, end-to-end learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review