CoChat
Sign Up
Cited by 4Reinforcement Learning in RoboticsArtificial Intelligence in GamesGame Theory and Applications

Learning to Deceive in Multi-agent Hidden Role Games

Open access
Aitchison, Matthew, Benke, Lyndon, Sweetser, Penny·2021·Communications in Computer and Information Science
DOI 10.1007/978-3-030-91779-1_5
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-030-91779-1_5

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Agents discover betrayal without explicit training.

    “Agents develop exploitative strategies (Baker et al., 2019), learn deception via theory-ofmind (Schulz et al., 2023), and discover betrayal without explicit training (Aitchison et al., 2021) -eluding threat models predicated on an identifiable adversary.”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Aitchison et al., 2021

    Section 6.3 (p. 9) reports that green agents developed an increasingly deceptive strategy despite having no explicit incentive to deceive. It does not describe this as betrayal: green agents concealed their role to survive blue’s hostility. Section 5 (p. 6) also says all agents were trained with deception modules, so the finding is about the absence of an explicit deception incentive, not the absence of training. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • In hidden-role games inspired by social deduction, reinforcement-learning agents learn to manipulate teammates' beliefs and betray them at opportune moments, despite no explicit training on deceptive behavior.

    “In hidden-role games inspired by social deduction, reinforcement-learning agents learn to manipulate teammates' beliefs and betray them at opportune moments, despite no explicit training on deceptive behavior (Aitchison et al., 2021).”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Aitchison et al., 2021

    The Introduction describes a hidden-role reinforcement-learning game inspired by social deception games. Section 6.3 reports that neutral green agents learned deceptive behavior without an explicit deception incentive, but red agents did not; the Conclusions say belief manipulation was effective with the paper’s deception model. Our copy does not report agents betraying teammates at opportune moments, and the claim incorrectly applies the absence of explicit deception training to the belief-manipulating agents generally. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 29

  1. [1][1] Almeshekah, M.H., Spafford, E.H.: Cyber security deception. In: Cyber deception, pp. 23–50. Springer (2016)
  2. [3][3] Bennett, P.G.: Hypergames: developing a model of conflict. Futures 12(6), 489–507 (1980)
  3. [5][5] Brown, N., Lerer, A., Gross, S., Sandholm, T.: Deep counterfactual regret minimization. In: International conference on machine learning. pp. 793–802. PMLR (2019)
  4. [6][6] Burda, Y., Edwards, H., Storkey, A., Klimov, O.: Exploration by random network distillation. arXiv preprint arXiv:1810.12894 (2018) · arXiv
  5. [7][7] Carroll, T.E., Grosu, D.: A game theoretic investigation of deception in network security. Security and Communi- cation Networks 4(10), 1162–1172 (2011)
  6. [8][8] Daniel, D.C., Herbig, K.L.: Strategic military deception: Pergamon policy studies on security affairs. Elsevier (2013)
  7. [10][10] Eger, M., Martens, C.: Practical specification of belief manipulation in games. In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. vol. 13 (2017)
  8. [11][11] Ettinger, D., Jehiel, P.: A theory of deception. American Economic Journal: Microeconomics 2(1), 1–20 (2010) 11 Learning to Deceive in Multi-Agent Hidden Role Games A PREPRINT
  9. [12][12] Floreano, D., Mitri, S., Magnenat, S., Keller, L.: Evolutionary Conditions for the Emergence of Communication in Robots. Current Biology 17(6), 514–519 (Mar 2007). https://doi.org/10.1016/j.cub.2007.01.058 · DOI
  10. [13][13] Foerster, J.N., Assael, Y.M., De Freitas, N., Whiteson, S.: Learning to communicate with deep multi-agent reinforcement learning. arXiv preprint arXiv:1605.06676 (2016) · arXiv
  11. [14][14] Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: Continual prediction with lstm. Neural Computation 12, 2451–2471 (2000). https://doi.org/10.1162/089976600300015015 · DOI
  12. [15][15] Gharesifard, B., Cortés, J.: Stealthy deception in hypergames under informational asymmetry. IEEE Transactions on Systems, Man, and Cybernetics: Systems 44(6), 785–795 (2013)
  13. [16][16] Gneezy, U.: Deception: The role of consequences. American Economic Review 95(1), 384–394 (2005)
  14. [17][17] Gutierrez, C.N., Bagchi, S., Mohammed, H., Avery, J.: Modeling deception in information security as a hypergame– a primer. In: Proceedings of the 16th Annual Information Security Symposium. p. 41. CERIAS-Purdue University (2015)
  15. [18][18] Ho, Y.C., Kastner, M., Wong, E.: Teams, signaling, and information theory. IEEE Transactions on Automatic Control 23(2), 305–312 (1978)
  16. [19][19] Iqbal, S., Sha, F.: Actor-attention-critic for multi-agent reinforcement learning. In: International Conference on Machine Learning. pp. 2961–2970. PMLR (2019)
  17. [20][20] Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J.Z., De Freitas, N.: Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In: International Conference on Machine Learning. pp. 3040–3049. PMLR (2019)
  18. [22][22] Kullback, S., Leibler, R.A.: On information and sufficiency. The annals of mathematical statistics 22(1), 79–86 (1951)
  19. [23][23] La, Q.D., Quek, T.Q., Lee, J., Jin, S., Zhu, H.: Deceptive attack and defense game in honeypot-enabled networks for the internet of things. IEEE Internet of Things Journal 3(6), 1025–1035 (2016)
  20. [24][24] Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., Graepel, T.: Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03037 (2017) · arXiv
  21. [25][25] Levine, T.R.: Truth-default theory (tdt) a theory of human deception and deception detection. Journal of Language and Social Psychology 33(4), 378–392 (2014)
  22. [26][26] MacNally, A.M., Lipovetzky, N., Ramirez, M., Pearce, A.R.: Action Selection for Transparent Planning. In: Proc. of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018) (2018)
  23. [27][27] Masters, P., Sardina, S.: Deceptive Path-Planning. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17). pp. 4368–4375. International Joint Conferences on Artificial Intelligence Organization (Aug 2017). https://doi.org/10.24963/ijcai.2017/610 · DOI
  24. [28][28] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013) · arXiv
  25. [29][29] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. vol. 32 (2019)
  26. [30][30] Samvelyan, M., Rashid, T., de Witt, C.S., Farquhar, G., Nardelli, N., Rudner, T.G.J., Hung, C.M., Torr, P.H.S., Foerster, J., Whiteson, S.: The starcraft multi-agent challenge. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 4, 2186–2188 (2 2019), http://arxiv.org/abs/1902.04043 · arXiv
  27. [31][31] Schmidhuber, J.: Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Transactions on Autonomous Mental Development 2(3), 230–247 (2010)
  28. [33][33] Serrino, J., Kleiman-Weiner, M., Parkes, D.C., Tenenbaum, J.B.: Finding friend and foe in multi-agent games. arXiv preprint arXiv:1906.02330 (2019) · arXiv
  29. [34][34] Strouse, D., Kleiman-Weiner, M., Tenenbaum, J., Botvinick, M., Schwab, D.: Learning to share and hide intentions using information regularization. arXiv preprint arXiv:1808.02093 (2018) · arXiv

Cited by

Sorted by citations

Authors
Aitchison, M., Benke, L., & Sweetser, P.
Affiliations
  • Australian National University (AU)
  • Defence Science and Technology Group (AU)
Corresponding author
Aitchison, Matthew
ORCID
Venue
Communications in Computer and Information Science
Publisher
Springer International Publishing
Published
2021
Pages
55-75
ISSN
  • 1865-0929 (ISSN-L)
  • 1865-0937
DOI
10.1007/978-3-030-91779-1_5
OpenAlex
W4205884528
Cited by
4
References
3
Citations by year
2026: 1 · 2024: 1 · 2023: 2
FWCI
0.35
Type
conference paper
Language
en
Open access
Licence
Locations
  • Communications in computer and information science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • ANU Open Research (Australian National University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
Keywords
deception, multi-agent reinforcement learning, werewolves, Avalon, Among Us

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review