The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Playing the System: Can Puzzle Players Teach us How to Solve Hard Problems?
Open accessAbstract
With nearly three billion players, video games are more popular than ever. Casual puzzle games are among the most played categories. These games capitalize on the players’ analytical and problem-solving skills. Can we leverage these abilities to teach ourselves how to solve complex combinatorial problems? In this study, we harness the collective wisdom of millions of players to tackle the classical NP-hard problem of multiple sequence alignment, relevant to many areas of biology and medicine. We show that Borderlands Science players propose solutions to multiple sequence alignment tasks that perform as well or better than standard approaches, while exploring a much larger area of the Pareto-optimal solution space. We also show the strategies of the players, although highly heterogeneous, follow a collective logic that can be mimicked with Behavioral Cloning with minimal performance loss, allowing the players’ collective wisdom to be leveraged for alignment of any sequences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In the first study52, we established that it is possible for reinforcement learning agents to learn player strategies for solving BLS puzzles”
From Improving microbial phylogeny with citizen science within a mass-market video game (2025) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Needleman, S. B., & Wunsch, C. D. (1970) · Journal of Molecular Biology · doi:10.1016/0022-2836(70)90057-4 · Cited by 11,288
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
Sherstinsky, A. (2020) · Physica D: Nonlinear Phenomena · doi:10.1016/j.physd.2019.132306 · Cited by 5,680
A Greedy Algorithm for Aligning DNA Sequences
Zhang, Z., Schwartz, S., Wagner, L., & Miller, W. (2000) · Journal of Computational Biology · doi:10.1089/10665270050081478 · Cited by 5,478
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455
Labeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,204
Research works printed but not linked: 28
- Maria Chatzou, Cedrik Magis, Jia-Ming Chang, Carsten Kemena, Giovanni Bussotti, Ionas Erb, and Cedric Notredame. 2016. Multiple sequence alignment modeling: methods and applications. Briefings in bioinformatics 17, 6 (2016), 1009–1023.
- Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. 2016. Learning transferable policies for monocular reactive mav control. In International Symposium on Experimental Robotics. Springer, 3–11.
- Chuong B Do, Mahathi SP Mahabhashyam, Michael Brudno, and Serafim Batzoglou. 2005. ProbCons: Probabilistic consistency-based multiple sequence alignment. Genome research 15, 2 (2005), 330–340.
- Sean Eddy. 1992. HMMER user’s guide. Department of Genetics, Washington University School of Medicine 2, 1 (1992), 13.
- Sean Eddy. 2022. 2022 Essential Facts About the Video Game Industry. Entertainment Software Association (2022).
- Christopher B Eiben, Justin B Siegel, Jacob B Bale, Seth Cooper, Firas Khatib, Betty W Shen, Barry L Stoddard, Zoran Popovic, and David Baker. 2012. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nature biotechnology 30, 2 (2012), 190–192.
- Reza Jafari Hamed Tabrizchi Marjan Kuchaki Rafsanjani and M.M. Javidi. 2020. A Distributed Deep Q-learning based Model for Solving the Multiple Sequence Alignment Problem. · DOI
- Matthias J Koepp, Roger N Gunn, Andrew D Lawrence, Vincent J Cunningham, Alain Dagher, Tasmin Jones, David J Brooks, Christopher J Bench, and PM Grasby. 1998. Evidence for striatal dopamine release during a video game. Nature 393, 6682 (1998), 266–268.
- Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation 1, 4 (1989), 541–551.
- Hong-Liang Li, Yi-He Pang, and Bin Liu. 2021. BioSeq-BLM: a platform for analyzing DNA, RNA and protein sequences based on biological language models. Nucleic acids research 49, 22 (2021), e129–e129.
- Ari Löytynoja and Nick Goldman. 2008. Phylogeny-aware gap placement prevents errors in sequence alignment and evolutionary analysis. science 320, 5883 (2008), 1632–1635.
- Umberto Lupo, Damiano Sgarbossa, and Anne-Florence Bitbol. 2022. Protein language models trained on multiple sequence alignments learn phylogenetic relationships. bioRxiv (2022).
- James N MacGregor and Tom Ormerod. 1996. Human performance on the traveling salesman problem. Perception & psychophysics 58, 4 (1996), 527–539.
- 10.1128/mSystems.00031-18 · doi:10.1128/msystems.00031-18 · Paper page
- Hieu V Nguyen and Li Bai. 2010. Cosine similarity metric learning for face verification. In Asian conference on computer vision. Springer, 709–720.
- Nam-phuong D Nguyen, Siavash Mirarab, Keerthana Kumar, and Tandy Warnow. 2015. Ultra-large alignments using phylogeny-aware profiles. Genome biology 16, 1 (2015), 1–15.
- Mohammad Norouzi David J Fleet and Russ R Salakhutdinov. 2012. Hamming distance metric learning. In Advances in neural information processing systems. 1061–1069.
- Benedict Paten, Javier Herrero, Kathryn Beal, Stephen Fitzgerald, and Ewan Birney. 2008. Enredo and Pecan: genome-wide mammalian consistency-based multiple alignment with paralogs. Genome research 18, 11 (2008), 1814–1828.
- Aluísio Pinheiro, Hildete Prisco Pinheiro, and Pranab Kumar Sen. 2012. The use of Hamming distance in bioinformatics. In Handbook of Statistics. Vol. 28. Elsevier, 129–162.
- Dean A Pomerleau. 1989. Alvinn: An autonomous land vehicle in a neural network. Technical Report. CARNEGIE-MELLON UNIV PITTSBURGH PA ARTIFICIAL INTELLIGENCE AND PSYCHOLOGY ….
- Ramchalam Kinattinkara Ramakrishnan, Jaspal Singh, and Mathieu Blanchette. 2018. Rlalign: a reinforcement learning approach for multiple sequence alignment. In 2018 IEEE 18th International Conference on Bioinformatics and Bioengineering (BIBE). IEEE, 61–66.
- Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. 2020. Transformer protein language models are unsupervised structure learners. Biorxiv (2020).
- J Redding, Jacob Schreiver, C Shrum, A Lauf, and R Yampolskiy. 2015. Solving NP-hard number matrix games with Wisdom of Artificial Crowds. In 2015 Computer Games: AI, Animation, Mobile, Multimedia, Educational and Serious Games (CGAMES). IEEE, 38–43.
- Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 627–635.
- Julie D Thompson, Desmond G Higgins, and Toby J Gibson. 1994. CLUSTAL W: improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice. Nucleic acids research 22, 22 (1994), 4673–4680.
- Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008.
- 10.1145/985692.985733 · DOI · Paper page
- Sheng Kung Michael Yi, Mark Steyvers, Michael D Lee, and Matthew J Dry. 2012. The wisdom of the crowd in combinatorial problems. Cognitive science 36, 3 (2012), 452–470.
Cited by
de Jong, S., Paananen, V., Tag, B., & van Berkel, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710946 · Cited by 32
Improving microbial phylogeny with citizen science within a mass-market video game
Sarrazin-Gendron, R., Ghasemloo Gheidari, P., Butyaev, A., Keding, T., Cai, E., Zheng, J., Mutalova, R., Mounthanyvong, J., Zhu, Y., Nazarova, E., et al. (2025) · Nature Biotechnology · doi:10.1038/s41587-024-02175-6 · Cited by 9
Self-citationProceedings of The ACM Collective Intelligence Conference
(2023) · doi:10.1145/3582269 · Cited by 4
Leadership Through Citizen Science Games
Schrier, K. “. (2024) · Journal of Leadership Studies · doi:10.1002/jls.21878 · Cited by 2
Player-Guided AI outperforms standard AI in Sequence Alignment Puzzles
Mutalova, R., Sarrazin-Gendron, R., Ghasemloo Gheidari, P., Cai, E., Richard, G., Caisse, S., Knight, R., Blanchette, M., Szantner, A., & Waldispühl, J. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615597 · Cited by 1
Self-citation
Metadata
- Authors
- Mutalova, R., Sarrazin-Gendron, R., Cai, E., Richard, G., Ghasemloo Gheidari, P., Caisse, S., Knight, R., Blanchette, M., Szantner, A., & Waldispühl, J.
- Affiliations
- McGill University (CA)
- School of Computer Science, McGill University, Canada
- Gearbox Studio Québec, Canada
- University of California San Diego (US)
- Massively Multiplayer Online Science, Switzerland
- ORCID
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-15
- OpenAlex
- W4366590983
- Cited by
- 6
- References
- 56
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 2
- FWCI
- 6.40
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc-sa)
- Funders
- McGill University
- Genome Canada
- Génome Québec
- University of California, San Diego
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
- Social Sciences › Sociology and Political Science › Digital Games and Media
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- puzzle games, multiple sequence alignment, NP-hard problems, combinatorial optimization, collective intelligence, crowdsourcing, behavior cloning, Pareto optimal solutions, player strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Playing the System: Can Puzzle Players Teach us How to Solve Hard Problems? https://app.cochat.ai/papers/10.1145/3544548.3581375
- The paper
- Mutalova, R., Sarrazin-Gendron, R., Cai, E., Richard, G., Ghasemloo Gheidari, P., Caisse, S., Knight, R., Blanchette, M., Szantner, A., & Waldispühl, J. (2023). Playing the System: Can Puzzle Players Teach us How to Solve Hard Problems? Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581375
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.