Cultural reinforcement learning: a framework for modeling cumulative culture on a limited channel
Open accessAbstract
Humans' capacity for cumulative culture is remarkable: we can build up vast bodies of knowledge over generations. Communication, particularly via language, is a key component of this process. Previous work has described language as enabling posterior passing, where one Bayesian agent transmits a posterior distribution to the next. In practice, we cannot exactly copy our beliefs into the minds of others---we must communicate over the limited channel language provides. In this paper, we analyze cumulative culture as Bayesian reinforcement learning with communication over a rate-limited channel. We implement an agent that solves a crafting task and communicates to the next agent by approximating the optimal rate-distortion trade-off. Our model produces documented effects, such as the benefits of abstraction and selective social learning. It also suggests a new hypothesis: selective social learning can be harmful in tasks where initial exploration is required.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The approach of adding noise to the oracle policy could be seen as imposing an information rate limit between the oracle and learning agent.
“This approach could be seen as imposing an information rate limit [Prystawski et al., 2023] between the oracle and learning agent.”
From Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning (2024) · cites this paper as Prystawski et al., 2023
The “Rate-Limited Posterior Passing with Language” and “Transmission Policy” sections describe an information rate limit on messages passed from one agent to the next (page unavailable). They do not establish that adding noise to an oracle policy imposes such a limit between an oracle and a learning agent. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Predicting pragmatic reasoning in language games
Frank, M. C., & Goodman, N. D. (2012) · Science · doi:10.1126/science.1218633 · Cited by 868 · see citing works
Self-citationA rational analysis of rule-based concept learning
Goodman, N. D., Tenenbaum, J. B., Feldman, J., & Griffiths, T. L. (2008) · Cognitive Science · doi:10.1080/03640210701802071 · Cited by 469 · see citing works
Rational integration of noisy evidence and prior semantic expectations in sentence interpretation
Gibson, E., Bergen, L., & Piantadosi, S. T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1216438110 · Cited by 407 · see citing works
Hybrid social learning in human-algorithm cultural transmission
Brinkmann, L., Gezerli, D., Kleist, K. V., Müller, T. F., Rahwan, I., & Pescetelli, N. (2022) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0426 · Cited by 24 · see citing works
Flagged first, then by citations
Research works printed but not linked: 31
- Arimoto, S. (1972). An algorithm for computing the capacity of arbitrary discrete memoryless channels. IEEE Transac- tions on Information Theory, 18(1), 14-20.
- Arumugam, D., & Van Roy, B. (2022). Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learning. Advances in Neural Information Processing Sys- tems, 35.
- Bellman, R. (1957). A Markovian Decision Process. Journal of Mathematics and Mechanics, 679-684.
- Bellman, R., & Kalaba, R. (1959). On Adaptive Control Processes. IRE Transactions on Automatic Control, 4(2), 1-9.
- Beppu, A., & Griffiths, T. (2009). Iterated learning and the cultural ratchet. In Proceedings of the annual meeting of the cognitive science society (Vol. 31).
- Blahut, R. (1972). Computation of channel capacity and rate-distortion functions. IEEE Transactions on Informa- tion Theory, 18(4), 460-473.
- Boyd, R., Richerson, P. J., & Henrich, J. (2011). The cultural niche: Why social learning is essential for human adapta- tion. Proceedings of the National Academy of Sciences, 108(supplement 2), 10918-10925.
- Chopra, S., Tessler, M. H., & Goodman, N. D. (2019). The first crank of the cultural ratchet: Learning and transmitting concepts through language. In Proceedings of the annual meeting of the cognitive science society (Vol. 41, pp. 226- 232).
- Cover, T. M., & Thomas, J. A. (2012). Elements of Informa- tion Theory. John Wiley & Sons.
- Derex, M., Beugin, M.-P., Godelle, B., & Raymond, M. (2013). Experimental evidence for the influence of group size on cultural complexity. Nature, 503(7476), 389-391.
- Futrell, R., & Levy, R. (2017). Noisy-context surprisal as a human sentence processing cost model. In Proceedings of the 15th conference of the european chapter of the associ- ation for computational linguistics: Volume 1, long papers (pp. 688-698).
- Ghavamzadeh, M., Mannor, S., Pineau, J., & Tamar, A. (2015). Bayesian Reinforcement Learning: A Survey. Foundations and Trends® in Machine Learning, 8(5-6), 359-483.
- Henrich, J., & Gil-White, F. J. (2001). The evolution of prestige: Freely conferred deference as a mechanism for enhancing the benefits of cultural transmission. Evolution and human behavior, 22(3), 165-196.
- Jaksch, T., Ortner, R., & Auer, P. (2010). Near-Optimal Re- gret Bounds for Reinforcement Learning. Journal of Ma- chine Learning Research, 11(4).
- Levy, R. (2008). A noisy-channel model of human sentence comprehension under uncertain input. In Proceedings of the conference on empirical methods in natural language processing (pp. 234-243).
- Mesoudi, A., & Thornton, A. (2018). What is cumulative cultural evolution? Proceedings of the Royal Society B, 285(1880), 20180712.
- Morgan, T. J., Uomini, N. T., Rendell, L. E., Chouinard- Thuly, L., Street, S. E., Lewis, H. M., . . . others (2015). Experimental evidence for the co-evolution of hominin tool-making teaching and language. Nature communica- tions, 6(1), 6029.
- Osband, I., Russo, D., & Van Roy, B. (2013). (More) Efficient Reinforcement Learning via Posterior Sampling. Advances in Neural Information Processing Systems, 26, 3003-3011.
- Osband, I., & Van Roy, B. (2017). Why is posterior sam- pling better than optimism for reinforcement learning? In International conference on machine learning (pp. 2701- 2710).
- Puterman, M. L. (1994). Markov Decision Processes- Discrete Stochastic Dynamic Programming. New York, NY: John Wiley & Sons, Inc.
- Russo, D., Van Roy, B., Kazerouni, A., Osband, I., & Wen, Z. (2018). A Tutorial on Thompson Sampling. Foundations and Trends® in Machine Learning, 11(1), 1-96.
- Shafto, P., Wang, J., & Wang, P. (2021). Cooperative commu- nication as belief transport. Trends in Cognitive Sciences, 25(10), 826-828.
- Shannon, C. E. (1959). Coding theorems for a discrete source with a fidelity criterion. IRE Nat. Conv. Rec., March 1959, 4, 142-163.
- Smaldino, P. E., & Richerson, P. J. (2013). Human cumu- lative cultural evolution as a form of distributed computa- tion. In Handbook of human computation (pp. 979-992). Springer.
- Strens, M. J. (2000). A Bayesian Framework for Reinforce- ment Learning. In Proceedings of the seventeenth interna- tional conference on machine learning (pp. 943-950).
- Tennie, C., Call, J., & Tomasello, M. (2009). Ratcheting up the ratchet: on the evolution of cumulative culture. Philo- sophical Transactions of the Royal Society B: Biological Sciences, 364(1528), 2405-2415.
- Tessler, M. H., Madeano, J., Tsividis, P. A., Harper, B., Goodman, N. D., & Tenenbaum, J. B. (2021). Learning to solve complex tasks by growing knowledge culturally across generations. arXiv preprint arXiv:2107.13377. · arXiv
- Thompson, B., van Opheusden, B., Sumers, T., & Griffiths, T. (2022). Complex cognitive algorithms preserved by selec- tive social learning in experimental populations. Science, 376(6588), 95-98.
- Tomasello, M. (1999). The Cultural Origins of Human Cog- nition. Harvard University Press.
- Tomasello, M., Kruger, A. C., & Ratner, H. H. (1993). Cul- tural learning. Behavioral and Brain Sciences, 16(3), 495- 511.
- Zaslavsky, N., Hu, J., & Levy, R. P. (2020). A rate- distortion view of human pragmatic reasoning. arXiv preprint arXiv:2005.06641. · arXiv
Cited by
Sorted by citations
Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning
Cook, J., Lu, C., Hughes, E., Leibo, J. Z., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2406.00392 · Cited by 6 · see citing works
- Authors
- Prystawski, B., Arumugam, D., & Goodman, N. D.
- Affiliations
- Stanford University (US)
- Corresponding author
- Goodman, Noah D.
- ORCID
- Goodman, Noah D.
- Publisher
- Center for Open Science
- Published
- 2023-05-19
- OpenAlex
- W4377138583
- Cited by
- 2
- References
- 4
- Citations by year
- 2025: 1 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Cultural Studies › Language and cultural evolution
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
- Keywords
- cumulative culture, cultural transmission, Bayesian reinforcement learning, rate-distortion tradeoff, abstraction, exploration, language evolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
