CoChat
Sign Up
Cited by 1Artificial Intelligence in GamesGenomics and Phylogenetic StudiesArtificial Intelligence Applications

Player-Guided AI outperforms standard AI in Sequence Alignment Puzzles

Open access
Mutalova, Renata, Sarrazin-Gendron, Roman, Ghasemloo Gheidari, Parham, Cai, Eddie, Richard, Gabriel, Caisse, Sebastien, Knight, Rob, Blanchette, Mathieu +2 more·2023·Proceedings of The ACM Collective Intelligence Conference
DOI 10.1145/3582269.3615597
Open this paper in CoChat — free accountOpen PDF

Abstract

Although Artificial Intelligence (AI) has gained widespread popularity across different fields, it is essential to recognize that AI systems, while impressive, do not consistently exhibit robust generalization, particularly for difficult problems such as the Multiple Sequence Alignment (MSA). In this study, we focus on bridging this performance gap by integrating human solutions into AI training. To illustrate these principles, we leverage data from Borderlands Science, a popular citizen science game in which small instances of the MSA problem are represented as puzzles. Our goal is to leverage the collective intelligence of human players to enhance the capabilities of AI agents.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper tests whether incorporating human player solutions into a reinforcement learning model improves AI performance on Borderlands Science sequence-alignment puzzles. It finds that a player-guided GAIL model outperforms a standard DQN model, producing higher scores, more Pareto-optimal solutions, and less noisy solutions.

Key findings

  • The standard DQN model was unable to achieve human-level performance and “barely passes the minimum required score,” whereas GAIL results were “at least close to the average human performance.”
  • GAIL generated a higher proportion of Pareto-optimal solutions than DQN, though this percentage decreased as puzzle complexity increased; GAIL still produced a larger proportion of solutions resembling real player solutions.
  • GAIL solutions showed less noise and closer agreement with player and Pareto-optimal solutions; for high-difficulty puzzles, GAIL performance was comparable to All-players solutions and “on the highest difficulty even surpasses.”
  • In one puzzle analysis, DQN scores centered around approximately 30 points, while GAIL showed a wider range extending beyond 30 points; both players and GAIL tended to achieve scores above 30 for that puzzle.

Methods

  • Data came from Borderlands Science puzzles built from American Gut Project sequences pre-aligned with PASTA. The study used a sample of 25,000 puzzles with over 1 million solutions; top human solutions were selected by proportional horizontal Pareto distance, excluding solutions with distance exceeding 0.7, which filtered out 53.4% of solutions.
  • DQN implementation used a transformer attention architecture with self-attention, feed-forward networks, sinusoid positional encoding, layer normalization, residual connections, dropout, and replay memory size 1000. Hyperparameters included learning rate 0.001, batch size 128, discount factor gamma = 1, epsilon = 0.8, and decrement iteration 5.
  • GAIL extended the DQN with a generator-discriminator setup: the base DQN agent generated trajectories, while the discriminator distinguished player demonstrations from agent trajectories. DQN was updated with Adam at learning rate 0.001; discriminator used binary cross-entropy loss and Adam at learning rate 0.0001; DQN was further updated using GAIL reward with discount factor 0.99.

Limitations

  • The paper states that Borderlands Science’s game design may limit AI transfer: agents may struggle with larger state and action spaces, the game may not capture the complexity and scalability of demanding AI tasks, and learned policies may not transfer to environments with different rules, actions, and dynamics.
  • The paper states that the game does not incentivize players to optimize the score-gaps trade-off or leave gaps unused; the authors addressed this by varying gap tokens and filtering near-Pareto solutions, but note that Pareto optimality as a proxy does not guarantee the absolute best MSA solution.
  • The paper states model limitations: DQN can be unstable, overestimate action values, converge poorly, and struggle with exploration-exploitation balance; GAIL may have difficulty generalizing to unseen or complex scenarios and may produce limited diversity or suboptimal new samples. The text also makes evident that evaluation centers on BLS puzzle scores and gap trade-offs rather than direct MSA accuracy on new sequences.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3582269.3615597

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 12 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    GAIL is capable of generating a higher proportion of optimal solutions.

    “In Figure 6 we observe that GAIL is capable of generating a higher proportion of optimal solutions which is define in the section Data.”
    Quote in the paperSubstantial claim
  • Findingown data

    Incorporating human-annotated information into the AI model improves its performance on puzzle tasks.

    “Our findings demonstrate that incorporating human-annotated information into the AI model improves its performance on puzzle tasks.”
    Quote in the paperSubstantial claim
  • Findingown data

    DQN is unable to achieve human-level performance in game score even with precise parameter tuning.

    “We show that classical Reinforcement learning methods such as DQN are unable to achieve human-level performance, even with precise parameter tuning.”
    Quote in the paperSubstantial claim
  • Findingown data

    Even with the decrease, GAIL still produces a larger proportion of solutions that closely resemble real solutions.

    “Nonetheless, even with the decrease, the GAIL algorithm still produces a larger proportion of solutions that closely resemble real solutions.”
    Quote in the paperSubstantial claim
  • Findingown data

    As puzzle complexity increases, the percentage of optimal solutions generated by GAIL tends to decrease.

    “However, it is worth noting that as the puzzle complexity increases, this percentage tends to decrease.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 9
  • May combine two claims. 10

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    53.4% of solutions were filtered out.

    “As a result, 53.4% of solutions were filtered out.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GAIL solutions demonstrate good agreement with the overall solutions and display significantly less noise compared to the DQN solution.

    “The GAIL solutions demonstrate good agreement with the overall solutions and display significantly less noise compared to the DQN solution.”
    • Also stated in: § 8 CONCLUSION · boosted
      “The GAIL solutions demonstrate good agreement with the all-players and Pareto-optimal solutions and display significantly less noise in comparison to the DQN solution.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Although players have a limited number of spaces, there is no incentive for them to leave any spaces unused.

    “Although they have a limited number of spaces, there is no incentive for them to leave any spaces unused.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The score distribution for DQN has a wide range and is close to normal, centered around approximately 30 points.

    “Figure 9 demonstrates that the score distribution for DQN has a wide range and exhibits a distribution that is close to normal, centered around approximately score equals 30 points.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GAIL can generate solutions close to the Pareto optimal front while effectively minimizing the number of noisy solutions compared to DQN.

    “The results on 10 indicate that the GAIL algorithm has the ability to generate solutions that closely to the Pareto optimal front, while effectively minimizing the number of solutions with noise in comparison to the DQN algorithm.”
    • Also stated in: § 8 CONCLUSION · boosted
      “The findings demonstrate that the GAIL algorithm can generate solutions that are close to Pareto optimal solutions and significantly reduce the number of noisy solutions compared to DQN.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Pareto optimality is a useful proxy for identifying good solutions to the MSA problem.

  • May combine two claims.

    Findingown data

    The average DQN solution exhibits a significant amount of noise and does not align well with the other solutions.

    “It is apparent that the average of the DQN solution exhibits a significant amount of noise and does not align well with the other solutions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For puzzles with high difficulty, GAIL performance is comparable to All-players solutions, and on the highest difficulty even surpasses them.

    “Also, for puzzles with high difficulty, the performance of our GAIL approach is comparable to All-players solutions, and on the highest difficulty even surpasses.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    MSA is NP-hard.

    “Similar to many classical bioinformatics problems, MSA is known to be NP-hard [38]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    The computational complexity of MSA increases exponentially with the number of sequences to be aligned.

    “which means that its computational complexity increases exponentially with the number of sequences to be aligned [38].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    RL algorithms such as DQN have been applied to Atari games, yielding impressive results but failing to surpass human-level performance.

    “RL algorithms, such as the Deep Q-Network, have been applied to Atari games which share with BLS a common aspect of utilizing gaming mechanics for problem-solving, yielding impressive results but failing to surpass human-level performance [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    GAIL may encounter difficulties in generalizing to unseen or complex scenarios, as it primarily learns from provided expert demonstrations, leading to limited diversity and suboptimal behaviour when generating new samples.

    “Additionally, GAIL may encounter difficulties in generalizing to unseen or complex scenarios, as it primarily learns from provided expert demonstrations, leading to limited diversity and suboptimal behaviour when generating new samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Incorporating player solutions effectively enhances the RL model, resulting in reduced noise and generation of compelling options that strike a balance between number of used gaps and game score.

    “Conclusion: By incorporating player solutions, we can effectively enhance the RL model, resulting in reduced noise and the generation of compelling options that strike a balance between the number of used gaps and the game score.”
    • Also stated in: § 6 DISCUSSION · boosted
      “Our results demonstrated that incorporating player solutions effectively enhanced the RL model, resulting in reduced noise and the generation of compelling options.”
    • Also stated in: § 8 CONCLUSION · plain
      “The incorporation of player solutions effectively enhances the AI standard model, reducing noise and generating compelling options that achieve a harmonious balance between used gaps and game score.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The score-gap trade-off is a relevant objective for evaluating MSA solution quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The Borderlands Science game may not fully capture the complexity and scalability required for demanding AI tasks, making it difficult to transfer learned policies to different game environments with diverse rules, actions, and dynamics.

    “The game may not fully capture the complexity and scalability required for demanding AI tasks, making it difficult to transfer learned policies to different game environments with diverse rules, actions, and dynamics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    A model trained using this approach can demonstrate performance levels close to humans, exhibiting similar degree of variability and quality in decision-making.

    “Our findings indicate that a model trained using this approach can demonstrate performance levels that are close to those of humans, exhibiting a similar degree of variability and quality in decision-making.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Average human performance is an appropriate benchmark for evaluating AI performance.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The use of Pareto optimality as a proxy for identifying good solutions to the MSA problem does not guarantee discovery of the absolute best solution for that problem.

    “Furthermore, it is important to note that the use of Pareto optimality as a proxy for identifying good solutions to the Multiple Sequence Alignment (MSA) problem does not guarantee the discovery of the absolute best solution for that problem.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Compared to humans, the player-guided AI model exhibited respectable variability and quality in decision-making, suggesting it successfully captured gameplay characteristics observed in human players.

    “Compared to humans, the player-guided AI model exhibited respectable variability and quality in decision-making, suggesting that it successfully captured the gameplay characteristics observed in human players.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    AI agents trained on Borderlands Science may face challenges in adapting to larger state and action spaces, which can impede performance and generalization.

    “AI agents trained on Borderlands Science may face challenges in adapting to larger state and action spaces, which can impede performance and generalization.”
    Quote in the paperNot a substantial claim
17 of 17

  • The paper relies on this without support.

    AI systems do not consistently exhibit robust generalization, particularly for difficult problems such as MSA.

    “Although Artificial Intelligence (AI) has gained widespread popularity across different fields, it is essential to recognize that AI systems, while impressive, do not consistently exhibit robust generalization, particularly for difficult problems such as the Multiple Sequence Alignment (MSA).”

  • The paper relies on this without support.

    No previous studies have investigated the use of GAIL for tackling the MSA problem or for game-related tasks.

    “However, to the best of our knowledge, no previous studies have investigated the use of GAIL for tackling the MSA problem or for game-related tasks.”

  • The paper relies on this without support.

    Humans have natural cognitive abilities, intuition, and common sense reasoning, which allow them to adapt and apply knowledge in various situations.

    “Humans have natural cognitive abilities, intuition, and common sense reasoning, which allow them to adapt and apply knowledge in various situations.”

  • The paper relies on this without support.

    RL algorithms often need large amounts of data and computational resources to achieve similar levels of performance.

    “In contrast, RL algorithms often need large amounts of data and computational resources to achieve similar levels of performance.”

  • The paper relies on this without support.

    BLS offers tens of millions of player solutions.

    “BLS offers the advantage of coming with tens of millions of player solutions, unlocking a unique potential to explore novel strategies not discovered by other algorithms, including AI.”

  • The paper relies on this without support.

    The integration of BLS into the actual game has attracted a significant number of players compared to Phylo.

    “The integration of BLS into the actual game has attracted a significant number of players compared to Phylo.”

  • The paper relies on this without support.

    Game score is a valid measure of AI performance in solving BLS puzzles.

    “In Figure 5 we can compare algorithms performance in terms of game score, a score obtained primarily from correctly matching bricks to guides.”

  • The paper relies on this without support.

    Pareto optimality is a useful proxy for identifying good solutions to the MSA problem.

    “Furthermore, it is important to note that the use of Pareto optimality as a proxy for identifying good solutions to the Multiple Sequence Alignment (MSA) problem does not guarantee the discovery of the absolute best solution for that problem.”

  • The paper relies on this without support.

    Average human performance is an appropriate benchmark for evaluating AI performance.

    “By contrast, the results obtained with the GAIL method are at least close to the average human performance.”

  • The paper relies on this without support.

    The score-gap trade-off is a relevant objective for evaluating MSA solution quality.

    “To evaluate the effectiveness of the proposed GAIL algorithm in generating solutions that are more optimal and display reduced randomness in terms of the score-gap trade-off, we computed the percent of Pareto optimal solutions per original solutions and generated.”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting claims about which model uses the fewest gaps

    Major

    The first statement says DQN uses fewer gaps than the other algorithms, which includes GAIL. The next says GAIL has the lowest gap usage among all algorithms; both cannot be true for the same comparison.

  • Pareto-distance filter excludes the solutions it aims to select

    Minor

    Under the paper's definition, a Pareto-optimal solution has a distance of 1, while the worst human solution has a distance of 0. Excluding distances above 0.7 therefore removes solutions closest to the Pareto front, contrary to the stated selection goal.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. Advances in neural information processing systems 29 (2016).
  2. Roman Joeres. 2021. Multiple sequence alignment using deep reinforcement learning. SKILL 2021 (2021).
  3. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. In Proceedings of the International Conference on Learning Representations.
  4. Ioan-Gabriel Mircea, Maria-Iuliana Bocicor, and Andra. Dîincu. 2014. ON REINFORCEMENT LEARNING BASED MULTIPLE SEQUENCE ALIGNMENT.Studia Universitatis Babes-Bolyai, Informatica 59, 2 (2014).
  5. Chen Wang, Claudia Pérez-D’Arpino, Danfei Xu, Li Fei-Fei, Karen Liu, and Silvio Savarese. 2022. Co-gail: Learning diverse strategies for human-robot collaboration. In Conference on Robot Learning. PMLR, 1279–1290.

Cited by

Sorted by citations

Related

Authors
Mutalova, R., Sarrazin-Gendron, R., Ghasemloo Gheidari, P., Cai, E., Richard, G., Caisse, S., Knight, R., Blanchette, M., Szantner, A., & Waldispühl, J.
Affiliations
  • McGill University (CA)
  • School of Computer Science, McGill University, Canada
  • Gearbox Studio Québec, Canada
  • University of California San Diego (US)
ORCID
Venue
Proceedings of The ACM Collective Intelligence Conference
Publisher
ACM
Published
2023-11-05
Pages
53-62
DOI
10.1145/3582269.3615597
OpenAlex
W4387629015
Semantic Scholar
efff79b55459e94ec44da50020c6fbe32659e53b
Cited by
1
References
27
Citations by year
2024: 1
FWCI
0.18
Type
conference paper
Language
en
Open access
Licence
Funders
  • Genome Quebec
  • Genome Canada
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
multiple sequence alignment, human-in-the-loop learning, collective intelligence, game-based data collection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-26.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-26.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review