CoChat
Sign Up
Cited by 166Mind wandering and attentionHuman-Automation Interaction and SafetyMobile Crowdsensing and Crowdsourcing

Comparing attentional disengagement between prolific and MTurk samples

Open access
Albert, Derek A., Smilek, Daniel·2023·Scientific Reports
DOI 10.1038/s41598-023-46048-5PMC PMC10667324

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Attention often disengages from primary tasks in favor of secondary tasks (i.e., multitasking) and task-unrelated thoughts (i.e., mind wandering). We assessed whether attentional disengagement, in the context of a cognitive task, can substantially differ between samples from commonly used online participant recruitment platforms, Prolific and Mechanical Turk (MTurk). Initially, eighty participants were recruited through Prolific to perform an attention task in which the risk of losing points for errors was varied (high risk = 80% chance of loss, low risk = 20% chance of loss). Attentional disengagement was measured via task performance along with self-reported mind wandering and multitasking. On Prolific, we observed surprisingly low levels of disengagement. We then conducted the same experiment on MTurk. Strikingly, MTurk participants exhibited more disengagement than Prolific participants. There was also an interaction between risk and platform, with the high-risk group exhibiting less disengagement, in terms of better task performance, than the low-risk group, but only on MTurk. Platform differences in individual traits related to disengagement and relations among study variables were also observed. Platform differences persisted, but were smaller, after increasing MTurk reputation criteria and remuneration in a second experiment. Therefore, recruitment platform and recruitment criteria could impact results related to attentional disengagement.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-023-46048-5

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prolific participants show higher attentiveness, longer and more appropriate completion times, fewer signs of disengagement, and greater response consistency.

    “Prolific participants show higher attentiveness, longer and more appropriate completion times, fewer signs of disengagement, and greater response consistency (Albert and Smilek, 2023; Douglas et al., 2023).”

    From Testing the testing effect on prolific: when retrieval practice fails to boost learning (2026) · cites this paper as Albert and Smilek, 2023

    The Discussion reports greater mind wandering and multitasking and lower attention-task performance on MTurk than on Prolific, supporting the attentiveness and disengagement comparisons. But the Results (Demographics) report significantly longer completion times on MTurk, not Prolific, and do not establish that either platform’s times were more appropriate. The cited passages also do not establish greater response consistency among Prolific participants. (LLM, conf=0.91)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 61

  1. [1]Ralph BCW Wammes JD Barr N Smilek D Wandering minds and wavering goals: Examining the relation between mind wandering and grit in everyday life and the classroom Can. J. Exp. Psychol. Rev. Can. Psychol. Exp. 2017 71 120 132 10.1037/cep0000116 28604049 · DOI
  2. [2]Robison MK Gath KI Unsworth N The neurotic wandering mind: An individual differences investigation of neuroticism, mind-wandering, and executive control Q. J. Exp. Psychol. 2017 70 649 663 10.1080/17470218.2016.1145706 26821933 · DOI
  3. [3]Moran CN Young and restless, old and focused: Age-differences in mind-wandering frequency and phenomenology Psychol. Aging 2021 36 252 10.1037/pag0000526 33539151 · DOI
  4. [4]Seli P Mind-wandering across the age gap: Age-related differences in mind-wandering are partially attributable to age-related differences in motivation J. Gerontol. Ser. B 2021 76 1264 1271 10.1093/geronb/gbaa031 PMC8363030 32107558 · DOI
  5. [5]Brosowsky NP Murray S Schooler JW Seli P Thought dynamics under task demands: Evaluating the influence of task difficulty on unconstrained thought J. Exp. Psychol. Hum. Percept. Perform. 2021 47 1298 1312 10.1037/xhp0000944 34694856 · DOI
  6. [6]Seli P Konishi M Risko EF Smilek D The role of task difficulty in theoretical accounts of mind wandering Conscious. Cogn. 2018 65 255 262 10.1016/j.concog.2018.08.005 30219746 · DOI
  7. [7]Kane MJ For whom the mind wanders, and when, varies across laboratory and daily-life settings Psychol. Sci. 2017 28 1271 1289 10.1177/0956797617706086 28719760 PMC5591044 · DOI
  8. [8]Linz R Pauly R Smallwood J Engert V Mind-wandering content differentially translates from lab to daily life and relates to subjective stress experience Psychol. Res. 2021 85 649 659 10.1007/s00426-019-01275-2 31832761 PMC7900029 · DOI
  9. [9]Drody AC Pereira EJ Smilek D A desire for distraction: Uncovering the rates of media multitasking during online research studies Sci. Rep. 2023 13 781 10.1038/s41598-023-27606-3 36646770 PMC9842732 · DOI
  10. [11]Clifford S Jerit J Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies J. Exp. Political Sci. 2014 1 120 131 10.1017/xps.2014.5 · DOI
  11. [15]Wammes JD Disengagement during lectures: Media multitasking and mind wandering in university classrooms Comput. Educ. 2019 132 76 89 10.1016/j.compedu.2018.12.007 · DOI
  12. [16]Kam JWY Handy TC The neurocognitive consequences of the wandering mind: A mechanistic account of sensory-motor decoupling Front. Psychol. 2013 4 725 10.3389/fpsyg.2013.00725 24133472 PMC3796327 · DOI
  13. [17]Seli P Cheyne JA Smilek D Wandering minds and wavering rhythms: Linking mind wandering and behavioral variability J. Exp. Psychol. Hum. Percept. Perform. 2013 39 1 5 10.1037/a0030954 23244046 · DOI
  14. [18]Jana S Aron AR Mind wandering impedes response inhibition by affecting the triggering of the inhibitory process Psychol. Sci. 2022 33 1068 1085 10.1177/09567976211055371 35699435 PMC9437729 · DOI
  15. [19]Seli P The attention-lapse and motor decoupling accounts of SART performance are not mutually exclusive Conscious. Cogn. 2016 41 189 198 10.1016/j.concog.2016.02.017 26946296 · DOI
  16. [20]Thomson DR Seli P Besner D Smilek D On the link between mind wandering and task performance over time Conscious. Cogn. 2014 27 14 26 10.1016/j.concog.2014.04.001 24780348 · DOI
  17. [23]Wallace JC Chen G Development and validation of a work-specific measure of cognitive failure: Implications for occupational safety J. Occup. Organ. Psychol. 2005 78 615 632 10.1348/096317905X37442 · DOI
  18. [24]Galéra C Mind wandering and driving: Responsibility case-control study BMJ 2012 345 e8105 10.1136/bmj.e8105 23241270 PMC3521876 · DOI
  19. [25]Baird B Inspired by distraction: Mind wandering facilitates creative incubation Psychol. Sci. 2012 23 1117 1122 10.1177/0956797612446024 22941876 · DOI · Paper page
  20. [26]Kapadia C Melwani S More tasks, more ideas: The positive spillover effects of multitasking on subsequent creativity J. Appl. Psychol. 2021 106 542 559 10.1037/apl0000506 32525327 · DOI
  21. [27]Baird B Smallwood J Schooler JW Back to the future: Autobiographical planning and the functionality of mind-wandering Conscious. Cogn. 2011 20 1604 1611 10.1016/j.concog.2011.08.007 21917482 · DOI · Paper page
  22. [28]Kvavilashvili L Rummel J On the nature of everyday prospection: A review and theoretical integration of research on mind-wandering, future thinking, and prospective memory Rev. Gen. Psychol. 2020 24 210 237 10.1177/1089268020918843 · DOI
  23. [29]Ralph BCW Smith AC Seli P Smilek D The relation between task-unrelated media multitasking and task-related motivation Psychol. Res. 2021 85 408 422 10.1007/s00426-019-01246-7 31535206 · DOI
  24. [30]Harbeck EL Glendon AI Driver prototypes and behavioral willingness: Young driver risk perception and reported engagement in risky driving J. Saf. Res. 2018 66 195 204 10.1016/j.jsr.2018.07.009 30121106 · DOI
  25. [31]Zhou R Zhang Y Shi Y Driver’s distracted behavior: The contribution of compensatory beliefs increases with higher perceived risk Int. J. Ind. Ergon. 2020 80 103009 10.1016/j.ergon.2020.103009 · DOI
  26. [33]Cordellieri P Young drivers ‘on the phone: Personality traits, driving attitudes, beliefs, and risk perception on using mobile phone while driving Psychol. Stud. 2022 67 459 467 10.1007/s12646-022-00670-x · DOI
  27. [34]Pope CN Bell TR Stavrinos D Mechanisms behind distracted driving behavior: The role of age and executive function in the engagement of distracted driving Accid. Anal. Prev. 2017 98 123 129 10.1016/j.aap.2016.09.030 27716494 PMC5167635 · DOI
  28. [35]Starkey NJ Isler RB The role of executive function, personality and attitudes to risks in explaining self-reported driving behaviour in adolescent and adult male drivers Transp. Res. Part F Traffic Psychol. Behav. 2016 38 127 136 10.1016/j.trf.2016.01.013 · DOI
  29. [36]Burdett BRD Charlton SG Starkey NJ Mind wandering during everyday driving: An on-road study Accid. Anal. Prev. 2019 122 76 84 10.1016/j.aap.2018.10.001 30315995 · DOI
  30. [37]O’Neill K Smith AP Smilek D Seli P Dissociating the freely-moving thought dimension of mind-wandering from the intentionality and task-unrelated thought dimensions Psychol. Res. 2021 85 2599 2609 10.1007/s00426-020-01419-9 32935185 · DOI
  31. [38]Adams TL Li Y Liu H A replication of beyond the Turk: Alternative platforms for crowdsourcing behavioral research—Sometimes preferable to student groups AIS Trans. Replication Res. 2020 6 15
  32. [39]Chandler J Rosenzweig C Moss AJ Robinson J Litman L Online panels in social science research: Expanding sampling methods beyond Mechanical Turk Behav. Res. Methods 2019 51 2022 2038 10.3758/s13428-019-01273-7 31512174 PMC6797699 · DOI · Paper page
  33. [40]Peer E Brandimarte L Samat S Acquisti A Beyond the Turk: Alternative platforms for crowdsourcing behavioral research J. Exp. Soc. Psychol. 2017 70 153 163 10.1016/j.jesp.2017.01.006 · DOI · Paper page
  34. [41]Peer E Rothschild D Gordon A Evernden Z Damer E Data quality of platforms and panels for online behavioral research Behav. Res. Methods 2021 54 1643 1662 10.3758/s13428-021-01694-3 34590289 PMC8480459 · DOI · Paper page
  35. [42]Litman L Moss A Rosenzweig C Robinson J Reply to MTurk, Prolific or panels? Choosing the right audience for online research SSRN Electron. J. 2021 10.2139/ssrn.3775075 · DOI
  36. [43]Chandler J Mueller P Paolacci G Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers Behav. Res. Methods 2014 46 112 130 10.3758/s13428-013-0365-7 23835650 · DOI · Paper page
  37. [44]Keith MG Tay L Harms PD Systems perspective of Amazon Mechanical Turk for organizational research: Review and recommendations Front. Psychol. 2017 8 1359 10.3389/fpsyg.2017.01359 28848474 PMC5550837 · DOI · Paper page
  38. [46]Robinson J Rosenzweig C Moss AJ Litman L Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool PLoS ONE 2019 14 e0226394 10.1371/journal.pone.0226394 31841534 PMC6913990 · DOI
  39. [47]Zickar MJ Keith MG Innovations in sampling: Improving the appropriateness and quality of samples in organizational research Annu. Rev. Organ. Psychol. Organ. Behav. 2023 10 315 337 10.1146/annurev-orgpsych-120920-052946 · DOI
  40. [48]Calleja MO Willoughby AR The effects of search-irrelevant working memory content on visual search Atten. Percept. Psychophys. 2023 85 293 300 10.3758/s13414-022-02634-9 36596986 · DOI
  41. [49]Kruger TB Marty-Dugas J Ralph BCW Dixon MJ Smilek D The relation between self-reported healthy living and attentional engagement in everyday life Curr. Res. Behav. Sci. 2023 4 100086 10.1016/j.crbeha.2022.100086 · DOI
  42. [50]Kelm MR Diercks CM Dunning ED Lunkenheimer E Parental working memory buffers associations between COVID-19 hardships and child mental health J. Appl. Dev. Psychol. 2023 87 101559 10.1016/j.appdev.2023.101559 37363111 PMC10266502 · DOI
  43. [51]Drody AC Ralph BCW Danckert J Smilek D Boredom and media multitasking Front. Psychol. 2022 13 1214 10.3389/fpsyg.2022.807667 PMC8978561 35386895 · DOI
  44. [52]Chua KQ Relationship between contentment and working memory capacity: Experimental and naturalistic evidence Curr. Psychol. 2022 10.1007/s12144-022-03714-7 · DOI
  45. [53]Smith AP Fixation, flexibility, and creativity: The dynamics of mind wandering J. Exp. Psychol. Hum. Percept. Perform. 2022 48 689 710 10.1037/xhp0001012 35587437 · DOI
  46. [54]Wessel JR Jiang J Stolley JJ Action errors impair active working memory maintenance J. Exp. Psychol. Gen. 2022 151 1325 1340 10.1037/xge0001142 35143251 PMC9237091 · DOI
  47. [55]Rioja K Cekic S Bavelier D Baumgartner SE Unraveling the link between media multitasking and attention across three samples Technol. Mind Behav. 2023 4 tmb0000106 37397029 PMC7614709
  48. [56]Kennedy R The shape of and solutions to the MTurk quality crisis Political Sci. Res. Methods 2020 8 614 629 10.1017/psrm.2020.6 · DOI · Paper page
  49. [58]Vo HL Vo DH The purchasing power parity and exchange-rate economics half a century on J. Econ. Surv. 2023 37 446 479 10.1111/joes.12504 · DOI
  50. [59]Cheyne JA Carriere JSA Smilek D Absent-mindedness: Lapses of conscious awareness and everyday cognitive failures Conscious. Cogn. 2006 15 578 592 10.1016/j.concog.2005.11.009 16427318 · DOI
  51. [62]Hoyle RH Stephenson MT Palmgreen P Lorch EP Donohew RL Reliability and validity of a brief measure of sensation seeking Personal. Individ. Differ. 2002 32 401 414 10.1016/S0191-8869(01)00032-0 · DOI · Paper page
  52. [64]Douglas BD Ewell PJ Brauer M Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA PLoS ONE 2023 18 e0279720 10.1371/journal.pone.0279720 36917576 PMC10013894 · DOI · Paper page
  53. [65]Medina JM Wong W Díaz JA Colonius H Advances in modern mental chronometry Front. Hum. Neurosci. 2015 9 256 10.3389/fnhum.2015.00256 25999843 PMC4422014 · DOI
  54. [68]Seli P How pervasive is mind wandering, really? Conscious. Cogn. 2018 66 74 78 10.1016/j.concog.2018.10.002 30408603 · DOI
  55. [70]Matsuura, T., Hasegawa, A. A., Akiyama, M. & Mori, T. Careless participants are essential for our phishing study: Understanding the impact of screening methods. In European Symposium on Usable Security 2021 36–47 (ACM, 2021). 10.1145/3481357.3481515. · DOI
  56. [71]Oppenlaender, J., Milland, K., Visuri, A., Ipeirotis, P. & Hosio, S. Creativity on paid crowdsourcing platforms. In 1–14 (ACM, 2020). 10.1145/3313831.3376677. · DOI
  57. [72]Garrett EP Impulsivity and loot box engagement Telemat. Inform. 2023 78 101952 10.1016/j.tele.2023.101952 · DOI
  58. [73]Nguyen-Phuoc DQ Oviedo-Trespalacios O Su DN De Gruyter C Nguyen T Mobile phone use among car drivers and motorcycle riders: The effect of problematic mobile phone use, attitudes, beliefs and perceived risk Accid. Anal. Prev. 2020 143 105592 10.1016/j.aap.2020.105592 32485432 · DOI
  59. [74]Chmielewski M Kucker SC An MTurk crisis? Shifts in data quality and the impact on study results Soc. Psychol. Personal. Sci. 2020 11 464 473 10.1177/1948550619875149 · DOI · Paper page
  60. [75]Pyo J Maxfield MG Cognitive effects of inattentive responding in an MTurk sample Soc. Sci. Q. 2021 102 2020 2039 10.1111/ssqu.12954 · DOI
  61. [77]Seli P Schacter DL Risko EF Smilek D Increasing participant motivation reduces rates of intentional and unintentional mind wandering Psychol. Res. 2019 83 1057 1069 10.1007/s00426-017-0914-2 28918525 · DOI

Cited by

Sorted by citations

Authors
Albert, D. A., & Smilek, D.
Affiliations
University of Waterloo (CA)
Corresponding author
Albert, Derek A.
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2023-11-23
Volume
13
Issue
1
Pages
20574-20574
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-023-46048-5
PMC
PMC10667324
OpenAlex
W4388946049
Cited by
166
References
14
Citations by year
2026: 73 · 2025: 66 · 2024: 24 · 2023: 2 · 2019: 1
FWCI
24.71
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Natural Sciences and Engineering Research Council of Canada (RGPIN-2019-04071)
  • Mitacs (IT25494)
  • Lupina Foundation
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Mind wandering and attention
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Keywords
attentional disengagement, Prolific, Mechanical Turk, online participant recruitment, mind wandering, multitasking, task performance, remuneration
MeSH terms
Attention, Humans, Task Performance and Analysis, Self Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review