The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population
Open accessAbstract
Currently, little is known about the association between assessment intensity, burden, data quantity, and data quality in experience sampling method (ESM) studies. Researchers therefore have insufficient information to make informed decisions about the design of their ESM study. Our aim was to investigate the effects of different sampling frequencies and questionnaire lengths on burden, compliance, and careless responding. Students ( n = 163) received either a 30- or 60-item questionnaire three, six, or nine times per day for 14 days. Preregistered multilevel regression analyses and analyses of variance were used to analyze the effect of design condition on momentary outcomes, changes in those outcomes over time, and retrospective outcomes. Our findings offer support for increased burden and compromised data quantity and quality with longer questionnaires, but not with increased sampling frequency. We therefore advise against the use of long ESM questionnaires, while high-sampling frequencies do not seem to be associated with negative consequences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More frequent assessments may not reduce compliance, or may even increase compliance, as long as questionnaires are kept brief.
“It has also previously been reported that more frequent assessments may not reduce compliance (106), or may even increase compliance (107), as long as questionnaires are kept brief (108).”
From Don't Miss the Moment: A Systematic Review of Ecological Momentary Assessment in Suicide Research (2022) · cites this paper as [108]
Overly lengthy measures can induce fatigue and reduce compliance, as well as impact data quality due to increased careless responding or skipping questions.
“However, overly lengthy measures can induce fatigue and reduce compliance, as well as impact data quality due to increased careless responding or skipping questions (108, 110).”
From Don't Miss the Moment: A Systematic Review of Ecological Momentary Assessment in Suicide Research (2022) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Fitting Linear Mixed-Effects Models Using <b>lme4</b>
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · see citing works
null (2016) · Philosophy Study · doi:10.17265/2159-5313/2016.09.003 · Cited by 28,317 · see citing works
Ecological momentary assessment
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008) · Annual Review of Clinical Psychology · doi:10.1146/annurev.clinpsy.3.022806.091415 · Cited by 6,454 · see citing works
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · see citing works
Handbook of research methods for studying daily life
Mehl, M. R., Conner, T. S., & Csíkszentmihályi, M. (2012) · Choice Reviews Online · doi:10.5860/choice.50-1159 · Cited by 2,121 · see citing works
Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey
Galešić, M., & Bošnjak, M. (2009) · Public Opinion Quarterly · doi:10.1093/poq/nfp031 · Cited by 1,324 · see citing works
Detecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L. W., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,315 · see citing works
Response Burden and Questionnaire Length: Is Shorter Better? A Review and Meta-analysis
Rolstad, S., Adler, J., & Rydén, A. (2011) · Value in Health · doi:10.1016/j.jval.2011.06.003 · Cited by 958 · see citing works
Experience sampling methodology in mental health research: new insights and technical developments.
Myin‐Germeys, I., Kasanova, Z., Vaessen, T., Vachon, H., Kirtley, O., Viechtbauer, W., & Reininghaus, U. (2018) · World Psychiatry · doi:10.1002/wps.20513 · Cited by 709 · see citing works
Self-citationCompliance with ecological momentary assessment protocols in substance users: a meta-analysis
Jones, A., Remmerswaal, D., Verveer, I., Robinson, E., Franken, I. H. A., Wen, C. K. F., & Field, M. (2019) · Addiction · doi:10.1111/add.14503 · Cited by 319 · see citing works
Stone, A. A., Broderick, J. E., Schwartz, J. E., Shiffman, S., Litcher-Kelly, L., & Calvanese, P. (2003) · Pain · doi:10.1016/s0304-3959(03)00040-x · Cited by 293 · see citing works
Vachon, H., Viechtbauer, W., Rintala, A., & Myin-Germeys, I. (2019) · Journal of Medical Internet Research · doi:10.2196/14475 · Cited by 252 · see citing works
Self-citationEcological momentary assessment methodology in chronic pain research: a systematic review
May, M., Junghaenel, D. U., Ono, M., Stone, A. A., & Schneider, S. (2018) · The Journal of Pain · doi:10.1016/j.jpain.2018.01.006 · Cited by 247 · see citing works
Piasecki, T. M., Hufford, M. R., Solhan, M. B., & Trull, T. J. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.1.25 · Cited by 196 · see citing works
Compliance with momentary pain measurement using electronic diaries: A systematic review
Morren, M., van Dulmen, S., Ouwerkerk, J. W., & Bensing, J. M. (2008) · European Journal of Pain · doi:10.1016/j.ejpain.2008.05.010 · Cited by 177 · see citing works
Response compliance and predictors thereof in studies using the experience sampling method.
Rintala, A., Wampers, M., Myin‐Germeys, I., & Viechtbauer, W. (2018) · Psychological Assessment · doi:10.1037/pas0000662 · Cited by 176 · see citing works
Self-citationCollins, L. M., & Graham, J. A. (2002) · Drug and Alcohol Dependence · doi:10.1016/s0376-8716(02)00217-x · Cited by 169 · see citing works
Forkmann, T., Spangenberg, L., Rath, D., Hallensleben, N., Hegerl, U., Kersting, A., & Glaesmer, H. (2018) · Journal of Abnormal Psychology · doi:10.1037/abn0000381 · Cited by 165 · cited 2× · see citing works
“This hypothesis was based on previous research where declines in compliance over time have been repeatedly observed (e.g., Courvoisier et al., 2012; Forkmann et al., 2018; Rintala et al., 2018; Silvia et al., 2013).”
MethodResultCourvoisier, D. S., Eid, M., & Lischetzke, T. (2012) · Psychological Assessment · doi:10.1037/a0026733 · Cited by 153 · see citing works
Ambulatory Assessment of Affective Instability in Borderline Personality Disorder
Ebner‐Priemer, U., & Sawitzki, G. (2007) · European Journal of Psychological Assessment · doi:10.1027/1015-5759.23.4.238 · Cited by 125 · see citing works
Ono, M., Schneider, S., Junghaenel, D. U., & Stone, A. A. (2018) · Journal of Medical Internet Research · doi:10.2196/11398 · Cited by 109 · see citing works
A qualitative approach to guide choices for designing a diary study
Janssens, K. A. M., Bos, E. H., Rosmalen, J. G. M., Wichers, M. C., & Riese, H. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0579-6 · Cited by 105 · see citing works
Spook, J. E., Paulussen, T. G., Kok, G., & van Empelen, P. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2617 · Cited by 104 · see citing works
Silvia, P. J., Kwapil, T. R., Eddington, K. M., & Brown, L. H. (2013) · Social Science Computer Review · doi:10.1177/0894439313479902 · Cited by 100 · see citing works
van Berkel, N., Gonçalves, J., Lovén, L., Ferreira, D., Hosio, S. J., & Kostakos, V. (2018) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2018.12.002 · Cited by 96 · see citing works
Factors Predicting Compliance to Ecological Momentary Assessment Among Adolescent Smokers
Sokolovsky, A. W., Mermelstein, R. J., & Hedeker, D. (2013) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntt154 · Cited by 82 · see citing works
Himmelstein, P. H., Woods, W. C., & Wright, A. G. C. (2019) · Psychological Assessment · doi:10.1037/pas0000718 · Cited by 82 · see citing works
mobileQ: A free user-friendly application for collecting experience sampling data
Meers, K., Dejonckheere, E., Kalokerinos, E. K., Rummens, K., & Kuppens, P. (2020) · Behavior Research Methods · doi:10.3758/s13428-019-01330-1 · Cited by 82 · see citing works
Self-citationMcCarthy, D. E., Minami, H., Yeh, V. M., & Bold, K. W. (2015) · Addiction · doi:10.1111/add.12996 · Cited by 76 · see citing works
Effects of Intensive Mobile Happiness Reporting in Daily Life
Conner, T. S., & Reid, K. (2011) · Social Psychological and Personality Science · doi:10.1177/1948550611419677 · Cited by 76 · see citing works
Measurement reactivity and fatigue effects in daily diary research with families.
Reynolds, B. M., Robles, T. F., & Repetti, R. L. (2015) · Developmental Psychology · doi:10.1037/dev0000081 · Cited by 75 · see citing works
Rosenkranz, T., Takano, K., Watkins, E., & Ehring, T. (2019) · PLoS ONE · doi:10.1371/journal.pone.0231783 · Cited by 74 · see citing works
Fuller‐Tyszkiewicz, M., Skouteris, H., Richardson, B., Blore, J. D., Holmes, M., & Mills, J. (2013) · Body Image · doi:10.1016/j.bodyim.2013.06.003 · Cited by 72 · see citing works
Arslan, R. C., Reitz, A. K., Driebe, J. C., Gerlach, T. M., & Penke, L. (2020) · Psychological Methods · doi:10.1037/met0000294 · Cited by 60 · see citing works
The optimisation of experience sampling protocols in people with schizophrenia
Edwards, C., Cella, M., Tarrier, N., & Wykes, T. (2016) · Psychiatry Research · doi:10.1016/j.psychres.2016.07.048 · Cited by 42 · see citing works
Detecting random responders with infrequency scales using an error-balancing threshold
Kim, D. S., McCabe, C. J., Yamasaki, B. L., Louie, K. A., & King, K. M. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0964-9 · Cited by 37 · see citing works
Context-Informed Scheduling and Analysis
van Berkel, N., Gonçalves, J., Koval, P., Hosio, S. J., Dingler, T., Ferreira, D., & Kostakos, V. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300281 · Cited by 34 · see citing works
Soyster, P. D., Bosley, H. G., Reeves, J. W., Altman, A. D., & Fisher, A. J. (2019) · Journal for Person-Oriented Research · doi:10.17505/jpor.2019.06 · Cited by 33 · see citing works
Experience Sampling Methods in Clinical Psychology
Santangelo, P., Ebner‐Priemer, U., & Trull, T. J. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199793549.013.0011 · Cited by 28 · see citing works
Vachon, H., Fortes-Bourbousson, M., Deschamps, T., Doron, J., Bulteau, S., Sauvaget, A., & Thomas-Ollivier, V. (2016) · Psychiatry Research · doi:10.1016/j.psychres.2016.08.034 · Cited by 27 · see citing works
Self-citationTrull, T. J. (2019) · doi:10.31234/osf.io/eakyj · Cited by 20 · see citing works
Temporal Considerations for Self-Report Research Using Short Message Service
Walsh, E., & Brinker, J. K. (2015) · Journal of Media Psychology Theories Methods and Applications · doi:10.1027/1864-1105/a000161 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Arslan, R. C., Reitz, A. K., Driebe, J. C., Gerlach, T. M., & Penke, L. (2019, June 27). Routinely randomise the display and order of items to estimate and adjust for biases in subjective reports. https://doi.org/10.31234/osf.io/va8bx · DOI
- Fox, J., & Weisberg, S. (2011). An R Companion to Applied Regression, Third Edition. Thousand Oaks, CA: Sage. Retrieved from https://socialsciences.mcmaster.ca/jfox/Books/Companion/
- Intille, S. S., Haynes, C., Maniar, D., Ponnada, A., & Manjourides, J. (2016). µEMA : Microinteraction-based Ecological Momentary Assessment (EMA) using a smartwatch. UbiComp '16, 1124-1135. https://doi.org/10.1145/2971648.2971717 · DOI · Paper page
- Larson, R., & Csikszentmihalyi, M. (1983). The Experience Sampling Method. In H. T. Reis (Ed.), New Directions for Methodology of Social and Behavioral Science (pp. 41-56). San Francisco: Jossey-Bass.
- Shrout, P., & Lane, S. (2012). Psychometrics. In Mehl, M., & Conner, T. (Ed.). Handbook of research methods for studying daily life. New York, NY: Guilford.
Cited by
Sorted by citations
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532 · see citing works
Wrzus, C., & Neubauer, A. B. (2023) · Assessment · doi:10.1177/10731911211067538 · Cited by 500 · see citing works
Psy-Flex: A contextually sensitive measure of psychological flexibility
Gloster, A. T., Block, V. J., Klotsche, J., Villanueva, J., Rinner, M. T. B., Benoy, C., Walter, M., Karekla, M., & Bader, K. (2021) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2021.09.001 · Cited by 175 · see citing works
Don't Miss the Moment: A Systematic Review of Ecological Momentary Assessment in Suicide Research
Kivelä, L., van der Does, W. A. J., Riese, H., & Antypa, N. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.876595 · Cited by 163 · see citing works
The accuracy and validity of self-reported social media use measures among adolescents
Verbeij, T., Pouwels, J. L., Beyens, I., & Valkenburg, P. M. (2021) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2021.100090 · Cited by 163 · see citing works
Rauschenberg, C., Boecking, B., Paetzold, I., Schruers, K., Schick, A., van Amelsvoort, T., & Reininghaus, U. (2021) · JMIR Mental Health · doi:10.2196/25650 · Cited by 63 · see citing works
Adolescents’ real-time social and affective experiences of online and face-to-face interactions
Achterhof, R., Kirtley, O. J., Schneider, M., Hagemann, N., Hermans, K. S. F. M., Hiekkaranta, A. P., Lecei, A., Lafit, G., & Myin-Germeys, I. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107159 · Cited by 62 · see citing works
Self-citationOttenstein, C., & Werner, L. (2022) · Assessment · doi:10.1177/10731911211032718 · Cited by 52 · see citing works
Hart, A., Reis, D., Prestele, E., & Jacobson, N. C. (2022) · Journal of Medical Internet Research · doi:10.2196/34015 · Cited by 22 · see citing works
Motives and Consequences of Alcohol Use in People With Social Anxiety Disorder: A Daily Diary Study
Goodman, F. R., Brown, B. A., Silva, G. M., Bradford, D. E., Tennen, H., & Kashdan, T. B. (2022) · Behavior Therapy · doi:10.1016/j.beth.2022.01.005 · Cited by 13 · see citing works
Fang, S., Ding, D., Huang, M., & Zheng, Q. (2024) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2024.100769 · Cited by 8 · see citing works
- Authors
- Eisele, G., Vachon, H., Lafit, G., Kuppens, P., Houben, M., Myin-Germeys, I., & Viechtbauer, W.
- Affiliations
- KU Leuven (BE)
- KU Leuven, Leuven, Belgium
- Maastricht University (NL)
- Corresponding author
- Eisele, Gudrun
- Venue
- Assessment
- Publisher
- SAGE Publications
- Published
- 2020-09-10
- Volume
- 29
- Issue
- 2
- Pages
- 136-151
- ISSN
- 1073-1911 (ISSN-L)
- 1552-3489
- OpenAlex
- W3085273257
- Cited by
- 459
- References
- 42
- Citations by year
- 2026: 87 · 2025: 99 · 2024: 101 · 2023: 66 · 2022: 56 · 2021: 40 · 2020: 8 · 2019: 1
- FWCI
- 41.12
- Type
- journal article
- Language
- en
- Open access
- Locations
- Assessment (publishedVersion)
- PubMed (publishedVersion)
- Research portal (Tilburg University) (acceptedVersion, other-oa)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Lirias (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- Fonds Wetenschappelijk Onderzoek (Odysseus grant awarded to Inez Myin-Germeys (Grant)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- ecological momentary assessment, sampling frequency, questionnaire length, perceived burden, compliance, careless responding, data quality, student population, momentary experience
- MeSH terms
- Ecological Momentary Assessment, Humans, Surveys and Questionnaires, Research Design, Retrospective Studies, Students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
