CoChat
Sign Up
Cited by 9Genomics and Phylogenetic StudiesEducational Games and GamificationDigital Games and Media

Improving microbial phylogeny with citizen science within a mass-market video game

Open access
Sarrazin-Gendron, Roman, Ghasemloo Gheidari, Parham, Butyaev, Alexander, Keding, Timothy, Cai, Eddie, Zheng, Jiayue, Mutalova, Renata, Mounthanyvong, Julien +28 more·2025·Nature Biotechnology
DOI 10.1038/s41587-024-02175-6PMC PMC11738981
Open this paper in CoChat — free accountOpen PDF

Abstract

Citizen science video games are designed primarily for users already inclined to contribute to science, which severely limits their accessibility for an estimated community of 3 billion gamers worldwide. We created Borderlands Science ( BLS ), a citizen science activity that is seamlessly integrated within a popular commercial video game played by tens of millions of gamers. This integration is facilitated by a novel game-first design of citizen science games, in which the game design aspect has the highest priority, and a suitable task is then mapped to the game design. BLS crowdsources a multiple alignment task of 1 million 16S ribosomal RNA sequences obtained from human microbiome studies. Since its initial release on 7 April 2020, over 4 million players have solved more than 135 million science puzzles, a task unsolvable by a single individual. Leveraging these results, we show that our multiple sequence alignment simultaneously improves microbial phylogeny estimations and UniFrac effect sizes compared to state-of-the-art computational methods. This achievement demonstrates that hyper-gamified scientific tasks attract massive crowds of contributors and offers invaluable resources to the scientific community.

AI snapshot

Written from the paper’s full text.

TL;DR

Borderlands Science (BLS) embedded a 16S rRNA multiple-alignment puzzle inside Borderlands 3 and recruited over 4 million players who solved more than 135 million puzzles. The resulting alignment improved phylogenetic tree metrics and some UniFrac effect sizes relative to state-of-the-art methods, while demonstrating a “game-first” citizen-science design.

Key findings

  • BLS reached 4.45 million players by May 2023; over 4 million completed the tutorial and at least one real task, a 90% engagement rate versus about 10% for Phylo. BLS players completed an average of 35 tasks (median 12), compared with Phylo’s average of 5.7 (median 2).
  • The BLS alignment improved phylogenetic similarity to the Greengenes-SEPP reference: Kendall–Colijn mean 1,115 versus PASTA 2,193, MUSCLE 1,772, MAFFT 1,298, and greedy 1,246; Triplet distance mean 52.2 versus PASTA 86.6, MUSCLE 101.4, MAFFT 80.9, and greedy 80.2; compound distance 48.4 versus PASTA 87.2, MUSCLE 86.1, MAFFT 66.4, and greedy 65.0.
  • BLS improved UniFrac effect sizes over SEPP on many variables; the top five variables by delta were teeth brushing frequency, diabetes, number of types of plants, antibiotic history, and alcohol frequency. BLS achieved a higher significance category than SEPP for 13 variables, while the opposite occurred 10 times.
  • BLS mapped well to the 16S structural model: proportion of unmapped bases was 0.003 for BLS, 0.001 for PASTA, 0.100 for MAFFT, and 0.149 for MUSCLE.

Methods

  • Data: 953,000 150-nucleotide V4 16S rRNA fragments from stool samples in the American Gut Project/TMI. These were clustered with CD-HIT into 10,200 clusters; about 5,000 dissimilar sequences were removed, leaving 9,667 cluster representatives accounting for 946,740 sequences, which were aligned with PASTA 1.9 as a scaffold.
  • Game and puzzle pipeline: BLS players inserted gap tokens to align brick sequences; the first 1.4 million puzzles yielded about 75 million player tasks from April 2020 to July 2021. Each puzzle was released in three versions, aimed at about 45 players total, and solutions were filtered by distance from optimality and player consensus before being converted into alignment votes.
  • Validation: phylogenetic trees were inferred with FastTree and compared to a Greengenes-SEPP reference tree using Kendall–Colijn and Triplet distances; UniFrac effect sizes were computed for 74 non-technical AGP variables with Evident; structural quality was assessed by mapping alignments to the Comparative RNA Web 16S model.

Limitations

  • Stated: there is no universal ground truth for alignment benchmarking; the study uses the Greengenes phylogeny as a proxy, which is state of the art but not perfect.
  • Stated: the analysis covers phase 1 only (first year, over two-thirds of total data); later puzzles with alternative scaffolds and offset puzzles are excluded because they require different analysis, and offset puzzles are noisier.
  • Stated: BLS’s average scientific contribution per task is lower than Phylo’s, and SEPP still outperforms BLS on important variables such as age category; effect sizes were not optimized during alignment production to avoid overfitting. Evident from the text: results are specific to one commercial game, one 16S V4 alignment task, and one player community, so generalizability to other games or scientific tasks is not directly tested.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41587-024-02175-6

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

20 of 31 cited sources checked · 14 supported · 0 flagged · accuracy not yet measured

  • Findingcitation

    The data featured in this article come from a set of 953,000 rRNA fragments sequenced from stool samples submitted by participants in the American Gut Project.

    “The data featured in this article come from a set of 953,000 rRNA fragments sequenced from stool samples submitted by participants in the AGP25, powered by TMI (https://microsetta.ucsd.edu/).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] McDonald, D. et al. American Gut: an open platform for citizen science microbiome research. mSystems 3, e00031–18 (2018). source

      The Methods sections on sample selection and participant recruitment describe fecal samples submitted by American Gut Project participants and sequencing of 16S rRNA gene fragments. The Introduction reports over 467 million fragments across the project as of May 2017, but our copy does not establish that the citing article’s dataset contains 953,000 fragments. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingown data

    The multiple sequence alignment obtained from BLS enables improved UniFrac effect size estimation for many variables known to be important to human digestive health.

    “Our results show that the multiple sequence alignment (MSA) obtained from BLS (Fig. 1b and Methods) improves microbial phylogeny inference compared to other alignment methods and enables improved UniFrac26 effect size estimation for many variables known to be important to human digestive health.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [26] Lozupone, C. & Knight, R. UniFrac: a new phylogenetic method for comparing microbial communities. Appl. Environ. Microbiol. 71, 8228–8235 (2005). source
  • Findingown data

    BLS outperforms SEPP on many variables, including several linked to gut microbe diversity and human health.

    “We observed that BLS outperforms SEPP on many variables, including several that have been linked to gut microbe diversity and human health25,39–42.”
    Quote in the paperSubstantial claim2 of 5 sources checked
    • cites [25] McDonald, D. et al. American Gut: an open platform for citizen science microbiome research. mSystems 3, e00031–18 (2018). source
    • cites [39] Singhal, S. et al. The role of oral hygiene in inflammatory bowel disease. Dig. Dis. Sci. 56, 170–175 (2011). source
    • cites [40] Lau, W. L. et al. Diabetes and the gut microbiome. Semin. Nephrol. 41, 104–113 (2021). source
    • cites [41] Capurso, G. & Lahner, E. The interaction between smoking, alcohol and the gut microbiome. Best Pract. Res. Clin. Gastroenterol. 31, 579–588 (2017). source
    • cites [42] Mutlu, E. A. et al. Colonic microbiome is altered in alcoholism. Am. J. Physiol. Gastrointest. Liver Physiol. 302, G966–G978 (2012). source
  • Findingown data

    The BLS phylogenies are closer to the reference than standard MSA approaches, according to Kendall-Colijn and Triplet distance metrics.

    “Our phylogenetic similarity results, shown in Table 1 for two distance metrics, Kendall–Colijn37 and Triplet38 distance, indicate that the BLS phylogenies are closer to the reference than standard MSA approaches.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [37] Kendall, M. & Colijn, C. Mapping phylogenetic trees to reveal distinct patterns of evolution. Mol. Biol. Evol. 33, 2735–2743 (2016). source
    • cites [38] Dress, A. W. M., Huber, K. T. & Steel, M. ‘Lassoing’ a phylogenetic tree I: basic properties, shellings, and covers. J. Math. Biol. 65, 77–105 (2012). source

      The Introduction (pp. 1–2) addresses reconstructing trees from incomplete leaf-to-leaf distances, not comparing BLS phylogenies with standard MSA approaches. The copy does not address the claimed results or identify Kendall–Colijn or Triplet distance as comparison metrics; searches for those terms found no relevant passage. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
  • Findingcitation

    BLS is designed to improve a reference multiple alignment of 1 million 16S rRNA sequences from The Microsetta Initiative's American Gut Project.

    “BLS is designed to improve a reference multiple alignment of 1 million 16S ribosomal RNA (rRNA) sequences from The Microsetta Initiative’s (TMI) American Gut Project (AGP)25, a citizen science initiative collecting gut microbial genomic data.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] McDonald, D. et al. American Gut: an open platform for citizen science microbiome research. mSystems 3, e00031–18 (2018). source
  • Classified as probably not a substantial claim. 23
  • May combine two claims. 29

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Greengenes has been previously benchmarked for fragment insertion, and SEPP with Greengenes has been shown to outperform de novo phylogeny inference when a high-quality tree and alignment are already available.

    “Greengenes has been previously benchmarked for fragment insertion, and SEPP with Greengenes has been shown to outperform de novo phylogeny inference, when a high-quality tree and alignment are already available36.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [36] Janssen, S. et al. Phylogenetic placement of exact amplicon sequences improves associations with clinical information. mSystems 3, e00021–18 (2018). source

  • May combine two claims.

    Premisecitation

    The target audience of classical citizen science games focuses on users with prior interest in science, which amplifies demographic skew in citizen science projects.

    “Hence, their target audience focuses on users with prior interest in science15, which eventually amplifies the demographic skew observed in citizen science projects19.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [15] Miller, J. A. et al. A survey of citizen science gaming experiences. Citiz. Sci. 7, 34 (2022). source

      The Background section says that previous interest in science is one motivation for citizen science game players. It does not establish that these games target users with that interest. The demographic-skew assertion carries a separate citation (19), so it is not assessed against this source. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [19] Allf, B. C. et al. Citizen science as an ecosystem of engagement: implications for learning and broadening participation. Bioscience 72, 651–663 (2022). source

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The validation process for BLS methods is centered around comparison of phylogenetic trees estimated from the alignment with FastTree 2.1.

    “Thus, the validation process for our methods is centered around the comparison of phylogenetic trees estimated from the alignment with FastTree 2.1 (ref. 34).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [34] Price, M. N., Dehal, P. S. & Arkin, A. P. FastTree 2—approximately maximum-likelihood trees for large alignments. PLoS ONE 5, e9490 (2010). source

  • May combine two claims.

    Premisecitation

    The state-of-the-art method for building phylogenetic trees from alignments has been shown to be outperformed on gut microbiome data by the placement of sequences into an existing tree with SEPP, which, when sufficient data are available, outperforms de novo phylogeny estimation from alignments.

    “The state-of-the-art method for building phylogenetic trees from alignments has been shown to be outperformed on gut microbiome data by the placement of the sequences into an existing tree with SEPP35,36, which, when sufficient data are available, outperforms de novo phylogeny estimation from alignments36.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [35] Mirarab, S., Nguyen, N. & Warnow, T. SEPP: SATé-enabled phylogenetic placement. Pac. Symp. Biocomput. 247–258 (2012). source
    • cites [36] Janssen, S. et al. Phylogenetic placement of exact amplicon sequences improves associations with clinical information. mSystems 3, e00021–18 (2018). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    It is possible for reinforcement learning agents to learn player strategies for solving BLS puzzles.

    “In the first study52, we established that it is possible for reinforcement learning agents to learn player strategies for solving BLS puzzles”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [52] Mutalova, R. et al. Playing the system: can puzzle players teach us how to solve hard problems? In Proc. 2023 CHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/3544548.3581375 (Association for Computing Machinery, 2023). source

  • May combine two claims.

    Findingown data

    Overall improvement over SEPP is limited as SEPP still outperforms BLS on important variables, such as age category.

    “Although overall improvement over SEPP is limited as SEPP still outperforms BLS on important variables, such as age category, BLS does outperform SEPP, and the two approaches lead to distinct solutions that are complementary in improving understanding of gut microbe phylogeny and its impact on human health, thus satisfying criterion d.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There is an enrichment of significant outcomes in variables with high effect sizes; 13 variables have BLS achieving a higher significance category than SEPP, whereas the opposite occurs 10 times, generally on variables with lower effect sizes.

    “We observed an enrichment of significant outcomes in the variables with high effect sizes and, in particular, 13 variables for which BLS achieves a higher significance category than SEPP, whereas the opposite occurs 10 times, generally on variables with lower effect sizes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The rRNA fragments are 150 nucleotides long and come from the V4 region of the 16S rRNA gene.

    “The fragments are 150 nucleotides long and come from the V4 region of the 16S rRNA gene.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Since its initial release on 7 April 2020, over 4 million players have solved more than 135 million science puzzles.

    “Since its initial release on 7 April 2020, over 4 million players have solved more than 135 million science puzzles, a task unsolvable by a single individual.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    During the first year of the initiative, approximately 75 million puzzle solutions were collected, with a mean of 43 per puzzle.

    “During the first year of the initiative, we collected approximately 75 million puzzle solutions (mean, 43 per puzzle).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Of the players who visited the arcade booth, over 4 million completed the 10-min tutorial and at least one real task.

    “Of these players, over 4 million completed the 10-min tutorial and at least one real task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    Since its launch, more than 4 million gamers have played BLS and solved more than 135 million puzzles.

    “Since its launch, more than 4 million gamers have played BLS and solved more than 135 million puzzles.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    BLS achieved a very high level of player engagement, but the average scientific contribution of one task was lower than in Phylo.

    “In the present study, we pushed the envelope on gamification, and our observations match the conundrum: BLS achieved a very high level of player engagement, but the average scientific contribution of one task was lower than in Phylo, where a few hundreds of puzzles are sufficient to improve a small alignment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    BLS alignments built with offset puzzles outperform BLS on some important effect size variables, such as age category.

    “BLS alignments built with offset puzzles show potential, as they outperform BLS on some important effect size variables, such as age category.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Alignments such as BLS and PASTA map rather easily to the 16S structural model, whereas MUSCLE and MAFFT lose substantially more information.

    “It turns out that alignments such as BLS and PASTA map rather easily, whereas MUSCLE and MAFFT lose substantially more information.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    BLS served as a large-scale science outreach initiative.

    “while also serving as a large-scale science outreach initiative”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    BLS achieved player engagement unseen before for a game of its category.

    “The project achieved player engagement unseen before for a game of its category”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    BLS players complete an average of 35 tasks, with a median of 12.

    “On the other hand, BLS players complete an average of 35 tasks, with a median of 12.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The region of the alignment with the most gaps and the most variation between methods is the top of the V4 stem, a region with considerable mapping improvements for BLS.

    “This is important because, as shown in Fig. 2a, the region of the alignment with the most gaps and the most variation between methods is the top of the V4 stem, a region with considerable mapping improvements for BLS in this analysis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The BLS alignment satisfies all criteria for a high-quality alignment and thus constitutes a high-quality MSA.

    “We assert that the BLS alignment satisfies all these criteria and, thus, constitutes a high-quality MSA.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    It had not yet been shown that citizen science can work in any type of video game, specifically gamer communities not familiar with scientific narratives.

    “EVE Online has a user base with a welcoming stance toward scientific narratives, and it had not yet been shown that citizen science can work in any type of video game, specifically gamer communities not familiar with scientific narratives.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The design of classical citizen science games is strongly rooted in the canonical presentation of the scientific task, which may hinder its entertainment value and participation of the general public.

    “The design of these CSGs is strongly rooted in the canonical presentation of the scientific task, which may hinder its entertainment value and, therefore, participation of the general public.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    BLS does outperform SEPP, and the two approaches lead to distinct solutions that are complementary in improving understanding of gut microbe phylogeny and its impact on human health, satisfying criterion d.

    “Although overall improvement over SEPP is limited as SEPP still outperforms BLS on important variables, such as age category, BLS does outperform SEPP, and the two approaches lead to distinct solutions that are complementary in improving understanding of gut microbe phylogeny and its impact on human health, thus satisfying criterion d.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The level of player engagement demonstrates that the ultra-gamification of the task worked and that its integration into the commercial game was seamless.

    “This level of player engagement demonstrates that the ultra-gamification of the task worked and that its integration into the commercial game was seamless.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In 2022, more than 3 billion people played video games worldwide.

    “In 2022, more than 3 billion people played video games worldwide1.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] Newzoo Global Games Market Report 2022 (newzoo, 2022); https://newzoo.com/resources/trend-reports/newzoo-global-games-market-report-2022-free-version

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The evaluation of the BLS alignment output is complicated by the absence of a universal ground truth to benchmark against.

    “The evaluation of the BLS alignment output is complicated by the absence of a universal ground truth to benchmark against.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The BLS alignment outperforms alternatives for the task of inferring a reliable phylogeny, satisfying criterion c.

    “This result shows that the BLS alignment outperforms alternatives for the task of inferring a reliable phylogeny, thus satisfying criterion c.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Solving more than 135 million science puzzles is a task unsolvable by a single individual.

    “Since its initial release on 7 April 2020, over 4 million players have solved more than 135 million science puzzles, a task unsolvable by a single individual.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The BLS alignment shows a gap frequency compatible with state-of-the-art knowledge of the V4 region, satisfying criterion b.

    “Thus, the BLS alignment shows a gap frequency that is compatible with our state-of-the-art knowledge of the V4 region, and criterion b is satisfied.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The V4 region of 16S rRNA is strongly structured and varies in length by only a few base pairs between microbes.

  • May combine two claims.

    Interpretationnone given

    The game-first design approach results in massive gains in public participation without sacrificing the relevance of the collected answers.

    “This approach results in massive gains in public participation without sacrificing the relevance of the collected answers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Participation in the project was fully optional.

    “Participation in the project was done entirely virtually, through the Borderlands 3 game. It was fully optional.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    BLS built on the idea of improving reach through collaboration with non-scientific companies and went to the next level by reaching gamers directly within high-profile video games.

    “BLS built on this idea of improving reach through collaboration with non-scientific companies and went to the next level by reaching gamers directly where they are, within the virtual universe of high-profile video games.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Guide offsets in puzzles allow the user to correct potential errors in the scaffold alignments at a larger scale.

    “these guide offsets allow the user to correct potential errors in the scaffold alignments at a larger scale”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The analysis presented in this study does not apply to puzzles with alternative scaffolds produced after phase 1.

    “After phase 1, we started producing puzzles with alternative scaffolds, and, as such, the analysis presented here does not apply to them”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    No identifying or personal information about participants was collected, and participants were fully anonymous to the research team.

    “No identifying or personal information about the participants was collected, and participants were fully anonymous to the research team.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The exclusion threshold for filtering player solutions was set to exclude about two-thirds of the solutions, to keep a subset most representative of the ensemble of high-quality player solutions.

    “the exclusion threshold was set to exclude about two-thirds of the solutions, to keep a subset that was most representative of the ensemble of high-quality player solutions”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Alignments were optimized on tree metrics, and effect sizes were kept as an independent evaluation.

    “Thus, alignments were optimized on tree metrics, and effect sizes were kept as an independent evaluation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The more gamified the task, the higher the potential for it to play like a real game and stimulate player engagement and retention.

    “However, the more gamified the task, the higher the potential for it to play like a real game and be fun enough to stimulate player engagement and retention.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisecitation

    Participating in a citizen science initiative has inherent benefits, such as improvements in science literacy and increased connectedness of the public to the scientific world.

    “the inherent benefits of participating in a citizen science initiative, such as improvements in science literacy and increased connectedness of the public to the scientific world”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Supplementary Information, section 11.4

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The ultra-gamification of BLS led to a much higher player engagement and retention than in Phylo, which strongly alleviates the downside of lower average scientific contribution per task.

    “Nevertheless, the ultra-gamification of BLS led to a much higher player engagement and retention than in Phylo, which strongly alleviates that downside.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    The aggregation of player solutions and the production of an alignment were not optimized for performance on effect sizes.

    “It should be noted that the aggregation of player solutions and the production of an alignment were not optimized for performance on effect sizes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The modifications to the PASTA scaffold added by BLS led to an improvement of the structural and functional signal of the alignment, satisfying criterion e.

    “This demonstrates that the modifications to the PASTA scaffold added by BLS led to an improvement of the structural and functional signal of the alignment, thus satisfying criterion e.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Player engagement in BLS was driven by a combination of game-first design and genuine interest in science from the public.

    “our experience in this project leads us to think that player engagement was driven by a combination of game-first design and genuine interest in science from the public.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The BLS results unlock the potential for a new generation of gaming-focused CSGs.

    “unlocking the potential for a new generation of gaming-focused CSGs”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The exclusion threshold was selected experimentally based on visual exploration of rejected solutions, such that no obvious outliers were included and all excluded solutions were clear outliers.

    “This threshold was selected experimentally based on visual exploration of rejected solutions: we selected a threshold for which no obvious outliers were included, and all excluded solutions were clear outliers.”
    Quote in the paperSubstantial claim
45 of 45

  • The paper relies on this without support.

    The V4 region of 16S rRNA is strongly structured and varies in length by only a few base pairs between microbes.

    “This low gap frequency is consistent with the strongly structured nature of the V4 region, which varies in length by only a few base pairs between microbes.”

  • The paper relies on this without support.

    Phylo has an engagement rate of around 10%.

    “which has reported an engagement rate of around 10%.”

  • The paper relies on this without support.

    Phylo users complete an average of 5.7 tasks and a median of two.

    “Phylo struggled to retain players for more than a couple puzzles, with an average of 5.7 tasks completed by users, and a median of two.”

  • The paper relies on this without support.

    In Phylo, experts can collectively outperform casual gamers by up to 40% depending on the alignment.

    “Indeed, in Phylo, the experts can collectively outperform casual gamers by up to 40% depending on the alignment, despite being a small group completing a tiny fraction of the tasks solved by casual players.”

  • The authors state this as an assumption.

    The Greengenes phylogeny can serve as a proxy for ground truth for evaluating BLS alignments.

    “in the present study, we relied on the Greengenes phylogeny as a proxy for a ground truth”
    “Although this phylogeny is well understood and state of the art in the field, it is not perfect.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • BLS does not have the highest sum-of-pairs score

    Major

    The text says BLS improves the sum-of-pairs score over all benchmarks, but Table 1 reports 2.17 billion for BLS and a higher 2.18 billion for post-processing PASTA.

  • PASTA scaffold has two reported widths

    Minor

    The methods describe the PASTA scaffold as 183 columns, but Table 1 gives the PASTA alignment a width of 193 columns.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Nebel, S., Schneider, S. & Rey, G. D. Mining learning and crafting scientific experiments: a literature review on the use of Minecraft in education and research. J. Educ. Technol. Soc. 19, 355–366 (2016).

Cited by

Still being checked.

Related

Authors
Sarrazin-Gendron, R., Ghasemloo Gheidari, P., Butyaev, A., Keding, T., Cai, E., Zheng, J., Mutalova, R., Mounthanyvong, J., Zhu, Y., Nazarova, E., Drogaris, C., Erhart, K., Borderlands Science Development Team, Bélanger, D., Bouffard, M., Davidson, J., Falaise, M., Fiset, V., Hebert, S., Hewitt, D., Huot, J., Kim, S., Moreau-Genest, J., Najjab, D., Prince, S., Saintélien, L., players, B. S., Brouillette, A., Richard, G., Pitchford, R., Caisse, S., Blanchette, M., McDonald, D., Knight, R., Szantner, A., & Waldispühl, J.
Affiliations
  • McGill University (CA)
  • Fresno City College (US)
  • University of California San Diego (US)
Corresponding author
Waldispühl, Jérôme
ORCID
Venue
Nature Biotechnology
Publisher
Springer Science and Business Media LLC
Published
2024-04-15
Volume
43
Issue
1
Pages
76-84
ISSN
  • 0733-222X (ISSN-L)
  • 1087-0156
  • 1546-1696
DOI
10.1038/s41587-024-02175-6
PMC
PMC11738981
OpenAlex
W4394821929
Semantic Scholar
6947cd1c60d32e4fddf52dfb432da7f57f80bbaf
Cited by
9
References
59
Citations by year
2026: 1 · 2025: 5 · 2024: 2 · 2014: 1
FWCI
0.91
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Biotechnology (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
Funders
  • Genome Canada
  • Génome Québec
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
  • Social Sciences › Sociology and Political Science › Digital Games and Media
Keywords
citizen science, video games, microbial phylogeny, 16S rRNA sequences, multiple sequence alignment, crowdsourcing, human microbiome, gamification, UniFrac
MeSH terms
Citizen Science, Humans, Phylogeny, RNA, Ribosomal, 16S, Video Games, Crowdsourcing, Microbiota

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-25.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-25.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review