Internet Versus Mailed Questionnaires: A Randomized Comparison
Open accessAbstract
BACKGROUND: The use of Internet-based questionnaires for collection of data to evaluate patient education and other interventions has increased in recent years. Many self-report instruments have been validated using paper-and-pencil versions, but we cannot assume that the psychometric properties of an Internet-based version will be identical. OBJECTIVES: To look at similarities and differences between the Internet versions and the paper-and-pencil versions of 16 existing self-report instruments useful in evaluation of patient interventions. METHODS: Participants were recruited via the Internet and volunteered to participate (N=397), after which they were randomly assigned to fill out questionnaires online or via mailed paper-and-pencil versions. The self-report instruments measured were overall health, health distress, practice mental stress management, Health Assessment Questionnaire (HAQ) disability, illness intrusiveness, activity limitations, visual numeric for pain, visual numeric for shortness of breath, visual numeric for fatigue, self-efficacy for managing disease, aerobic exercise, stretching and strengthening exercise, visits to MD, hospitalizations, hospital days, and emergency room visits. Means, ranges, and confidence intervals are given for each instrument within each type of questionnaire. The results from the two questionnaires were compared using both parametric and non-parametric tests. Reliability tests were given for multi-item instruments. A separate sample (N=30) filled out identical questionnaires over the Internet within a few days and correlations were used to assess test-retest reliability. RESULTS: Out of 16 instruments, none showed significant differences when the appropriate tests were used. Construct reliability was similar within each type of questionnaire, and Internet test-retest reliability was high. Internet questionnaires required less follow-up to achieve a slightly (non-significant) higher completion rate compared to mailed questionnaires. CONCLUSIONS: Among a convenience sample recruited via the Internet, results from those randomly assigned to Internet participation were at least as good as, if not better than, among those assigned mailed questionnaires, with less recruitment effort required. The instruments administered via the Internet appear to be reliable, and to be answered similarly to the way they are answered when they are administered via traditional mailed paper questionnaires.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
All measures were administered online, a method shown to be reliable and as valid as traditional paper-and-pencil administration.
“All measures were administered online, a method which has been shown to be reliable and as valid as traditional paper-and-pencil administration [40,41].”
From Acceptability, effectiveness, and cost-effectiveness of internet-based exposure treatment for irritable bowel syndrome in a clinical sample: A randomized controlled trial (2011) · cites this paper as [41]
The Abstract Results report similar reliability and no significant differences between the Internet and mailed versions of 16 instruments; the Abstract Conclusions say the online instruments appear reliable and are answered similarly to paper questionnaires. The paper does not establish equal validity for online administration of measures generally, as the citing sentence implies. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Measurement of patient outcome in arthritis
Fries, J. F., Spitz, P. W., Kraines, R. G., & Holman, H. R. (1980) · Arthritis & Rheumatism · doi:10.1002/art.1780230202 · Cited by 4,373 · cited 1× · see citing works
“• Health Assessment Instrument measures disability and is used in the National Health Survey (20 items) (Fries et al, 1980) [15].”
MethodGosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004) · American Psychologist · doi:10.1037/0003-066x.59.2.93 · Cited by 2,932 · cited 1× · see citing works
“However, Gosling et al [19] have argued that Internet samples may actually be more representative than traditional samples.”
BackgroundOutcome Measures for Health Education and Other Health Care Interventions
Lorig, K., Stewart, A., Ritter, P., González, V., Laurent>, D., & Lynch, J. (1996) · doi:10.4135/9781452232966 · Cited by 1,132 · cited 8× · see citing works
“• Self-reported exercise measures minutes/week of aerobic (5 items) and minutes/week stretching and strengthening exercise (1 item) (Lorig et al, 1996) [12].”
BackgroundMethodInfluentialSelf-citationSelf-rated health and mortality in the NHANES I Epidemiologic Follow-up Study
Idler, E. L., & Angel, R. J. (1990) · American Journal of Public Health · doi:10.2105/ajph.80.4.446 · Cited by 864 · cited 1× · see citing works
“This item comes from the National Health Survey and has been found to be predictive of future health status (Idler & Angel, 1990) [14].”
MethodSocial desirability, anonymity, and internet-based questionnaires
Joinson, A. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200723 · Cited by 818 · cited 2× · see citing works
“Joinson (1999) reported that both anonymity and Web usage (compared to paper-and-pencil) resulted in lower scores on a social desirability measure [8].”
BackgroundUsing the Internet for Surveys and Health Research
Eysenbach, G., & Wyatt, J. (2002) · Journal of Medical Internet Research · doi:10.2196/jmir.4.2.e13 · Cited by 790 · cited 2× · see citing works
“A recent overview entitled “Using the Internet for Surveys and Health Research” (Eysenbach & Wyatt 2002, [10]), barely touched on instruments (referring readers to the Quality of Life Instruments Database at Quality of Life Instruments database [11]) and did not discuss validity or reliability of…”
BackgroundUsing the Internet for psychological research: Personality testing on the World Wide Web
Buchanan, T., & Smith, J. L. (1999) · British Journal of Psychology · doi:10.1348/000712699161189 · Cited by 596 · cited 1× · see citing works
“Buchanan and Smith (1999) [5] compared a Web-based personality assessment to a paper-and-pencil version, and using confirmatory factor analyses, found similar psychometric properties in the two tests.”
MethodSelf-reports of health care utilization compared to provider records
Ritter, P. L., Stewart, A. L., Kaymaz, H., Sobel, D. S., Block, D. A., & Lorig, K. R. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00261-4 · Cited by 401 · cited 1× · see citing works
“• Health care utilization (MD visits, hospitalization, hospital days, ER visits) (Lorig et al, 1996; Ritter et al, 2001) [12,18].”
Self-citationThe Use of the Internet in Psychological Research: Comparison of Online and Offline Questionnaires
Riva, G., Teruzzi, T., & Anolli, L. (2003) · CyberPsychology & Behavior · doi:10.1089/109493103321167983 · Cited by 304 · cited 2× · see citing works
“Joinson (1999) reported that both anonymity and Web usage (compared to paper-and-pencil) resulted in lower scores on a social desirability measure [8].”
BackgroundWeb-based administration of a personality questionnaire: Comparison with traditional methods
Davis, R. N. (1999) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03200737 · Cited by 220 · cited 2× · see citing works
“Davis (1999) [6] also compared Web and paper-and-pencil versions of a personality measure (rumination), and concluded that “findings from Web-based questionnaire research are comparable with results obtained using standard procedures.””
ResultLorig, K. R., Laurent, D. D., Deyo, R. A., Marnell, M. E., Minor, M. A., & Ritter, P. L. (2002) · Archives of Internal Medicine · doi:10.1001/archinte.162.7.792 · Cited by 206 · cited 1× · see citing works
“Although we and others have been using these instruments in Internet-based studies for a few years [1], neither these nor most similar instruments had previously been tested for Internet use.”
BackgroundSelf-citationTranslation and validation of arthritis outcome measures into spanish
González, V., Stewart, A., Ritter, P. L., & Lorig, K. R. (1995) · Arthritis & Rheumatism · doi:10.1002/art.1780381010 · Cited by 205 · cited 1× · see citing works
“• Visual numeric instruments for pain, shortness of breath, and fatigue are adaptations of visual analogue instruments that have been found to be easy for subjects to complete (4 items) (González et al, 1995) [17].”
BackgroundSelf-citationDevins, G. M., Mandin, H., Hons, R. B., Burgess, E. D., Klassen, J., Taub, K., Schorr, S., Letourneau, P. K., & Buckle, S. (1990) · Health Psychology · doi:10.1037//0278-6133.9.2.117 · Cited by 183 · cited 1× · see citing works
“Instrument measures how chronic illness affects role function in 5 domains: physical well being and diet, work and finances, marital, sexual and family relations, recreational and social relations, other (13 items) (Devins, 1990) [16].”
BackgroundInternet-based Questionnaire Assessment: Appropriate Use in Clinical Contexts
Buchanan, T. (2003) · Cognitive Behaviour Therapy · doi:10.1080/16506070310000957 · Cited by 168 · cited 1× · see citing works
“And Buchanan (2003) [9] reported that even when Internet-based versions of instruments are reliable and valid, normative data from paper-and-pencil versions may not always compare directly with Internet-mediated psychological testing.”
BackgroundA Web-Based Screening Instrument for Depression and Anxiety Disorders in Primary Care
Farvolden, P., McBride, C., Bagby, R. M., & Ravitz, P. (2003) · Journal of Medical Internet Research · doi:10.2196/jmir.5.3.e23 · Cited by 94 · cited 1× · see citing works
“For example, Lin et al (2003) studied a measure of self-assessment of depression [2], while Farvolden et al (2003) looked at screening for clinical depression [3].”
MethodThe validity of an Internet-based Self-assessment Program for Depression.
Lin, C. C., Li, Y., Bai, Y. M., Tsai, S., Hsiao, M., Wu, C. H., Liu, C.-Y., & Chen, J. Y. (2003) · Europe PMC (PubMed Central) · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Eysenbach, G. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.3.e34 · Cited by 6,480 · see citing works
“In this issue of the Journal of Medical Internet Research we publish two methodological studies exploring the characteristics of Web-based surveys compared to mail-based surveys [1,2].”
BackgroundA guide for the design and conduct of self-administered surveys of clinicians
Burns, K. E. A., Duffett, M., Kho, M. E., Meade, M., Adhikari, N. K. J., Sinuff, T., Cook, D. J., & for the ACCADEMY Group (2008) · Canadian Medical Association Journal · doi:10.1503/cmaj.080372 · Cited by 1,357 · see citing works
Comparing Response Rates from Web and Mail Surveys: A Meta-Analysis
Shih, T.-H., & Fan, X. (2008) · Field Methods · doi:10.1177/1525822x08317085 · Cited by 1,159 · see citing works
The QCAE: a questionnaire of cognitive and affective empathy
Reniers, R. L. E. P., Corcoran, R., Drake, R., Shryane, N. M., & Völlm, B. A. (2011) · Journal of Personality Assessment · doi:10.1080/00223891.2010.528484 · Cited by 961 · see citing works
“However, research has shown that questionnaires administered via the Internet are reliable and are answered in the same way as when they are administered via traditional mailed paper questionnaires ( Ritter, Lorig, Laurent, & Matthews, 2004 ; Riva, Teruzzi, & Anolli, 2003 ).”
BackgroundBiddiss, E. A., & Chau, T. T. (2007) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0b013e3181587f6c · Cited by 646 · see citing works
“) have been demonstrated.(22,25) However, it is important to recognize the possible sampling biases inherent to the results presented herein.”
BackgroundPorst, H., Montorsi, F., Rosen, R. C., Gaynor, L., Grupe, S., & Alexander, J. R. (2006) · European Urology · doi:10.1016/j.eururo.2006.07.004 · Cited by 587 · see citing works
“A recent study compared the response rates of surgeons to a questionnaire sent via e-mail or standard mail and reported a significantly lower response rate with e-mail solicitation [18]; however, the responses to questions obtained using the two survey methods were not different.”
ResultWeb-based Questionnaires: The Future in Epidemiology?
van Gelder, M. M. H. J., Bretveld, R. W., & Roeleveld, N. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq291 · Cited by 581 · see citing works
Consumer design priorities for upper limb prosthetics
Biddiss, E., Beaton, D., & Chau, T. (2007) · Disability and Rehabilitation: Assistive Technology · doi:10.1080/17483100701714733 · Cited by 519 · see citing works
“With some exceptions [44], response rates are generally lower for emailed surveys than for conventional methods [29].”
BackgroundRepresentativeness of the Patient-Reported Outcomes Measurement Information System Internet panel
Liu, H., Cella, D. F., Gershon, R. C., Shen, J., Morales, L. S., Riley, W. T., & Hays, R. D. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.021 · Cited by 447 · see citing works
Kongsved, S. M., Basnov, M., Holm-Christensen, K., & Hjollund, N. H. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.3.e25 · Cited by 395 · see citing works
“Response rates to Internet questionnaires reported in the literature vary a lot between studies [1,6-14].”
BackgroundWilliams, R. J., Volberg, R. A., & Stevens, R. M. G. (2012) · Open ULeth Scholarship (OPUS) (University of Lethbridge) · Cited by 400 · see citing works
van de Poll-Franse, L. V., Horevoorts, N. J. E., van Eenbergen, M. C. H. J., Denollet, J., Roukema, J. A., Aaronson, N. K., Vingerhoets, A. J. J. M., Coebergh, J. W. W., Vries, J. D., Essink‐Bot, M., et al. (2011) · European Journal of Cancer · doi:10.1016/j.ejca.2011.04.034 · Cited by 399 · see citing works
The mental health of university students in the United Kingdom
Macaskill, A. (2013) · British Journal of Guidance and Counselling · doi:10.1080/03069885.2012.743110 · Cited by 335 · see citing works
“…of measures and have found no significant differences in terms of internal consistency, test-retest reliability across the two modalities (Buchanan & Smith, 1999; Denscombe, 2006; Herrero & Menese, 2006; Ritter, Lorig, Laurent, & Mathews, 2004: Vallejo, Jordan, Diaz, Comech, & Ortega, 2007).”
Burnout and Depression: Two Entities or One?
Schonfeld, I. S., & Bianchi, R. (2016) · Journal of Clinical Psychology · doi:10.1002/jclp.22229 · Cited by 345 · see citing works
Fish, R. A., McGuire, B., Hogan, M., Morrison, T. G., & Stewart, I. (2010) · Pain · doi:10.1016/j.pain.2009.12.016 · Cited by 306 · see citing works
“, [8,13,23,43]) and a need to validate measures for use in this context [6,7].”
BackgroundThe Use of the Internet in Psychological Research: Comparison of Online and Offline Questionnaires
Riva, G., Teruzzi, T., & Anolli, L. (2003) · CyberPsychology & Behavior · doi:10.1089/109493103321167983 · Cited by 304 · see citing works
Devins, G. M. (2010) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2009.05.006 · Cited by 294 · see citing works
“, via the internet) without compromising reliability or biasing results [99].”
BackgroundLukens, C. T., & Linscheid, T. R. (2008) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-007-0401-5 · Cited by 278 · see citing works
Ljótsson, B., Falk, L., Vesterlund, A. W., Hedman, E., Lindfors, P., Rück, C., Hursti, T., Andréewitch, S., Jansson, L., Lindefors, N., et al. (2010) · Behaviour Research and Therapy · doi:10.1016/j.brat.2010.03.003 · Cited by 270 · see citing works
Feasibility of Using Web-based Questionnaires in Large Population-based Epidemiological Studies
Ekman, A., Dickman, P. W., Klint, Å., Weiderpass, E., & Litton, J. (2006) · European Journal of Epidemiology · doi:10.1007/s10654-005-6030-4 · Cited by 220 · see citing works
Ljótsson, B., Hedman, E., Andersson, E., Hesser, H., Lindfors, P., Hursti, T., Rydh, S., Rück, C., Lindefors, N., & Andersson, G. (2011) · American Journal of Gastroenterology · doi:10.1038/ajg.2011.139 · Cited by 218 · see citing works
“Data collection All measures were administered online, a method that has been shown to be reliable and produces results very similar to traditional paper-and-pencil administration (”
MethodInternet-based cognitive-behavioural therapy for severe health anxiety: randomised controlled trial
Hedman, E., Andersson, G., Andersson, E., Ljótsson, B., Rück, C., Asmundson, G. J. G., & Lindefors, N. (2011) · British Journal of Psychiatry · doi:10.1192/bjp.bp.110.086843 · Cited by 212 · see citing works
Vallejo, M., Jordán, C., Díaz, M. I., Comeche, M. I., & Ortega, J. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.1.e2 · Cited by 241 · see citing works
“In that sense, a growing number of computerized or online questionnaires related to areas such as panic/agoraphobia [3], youth independence living [7], aggression and impulsivity [8], quality of life in diabetes [9], and a battery of 16 other health-related questionnaires [10] have already been studied.”
BackgroundThe Danish National Health Survey 2010. Study design and respondent characteristics
Christensen, A. I., Ekholm, O., Glümer, C., Andreasen, A. H., Hvidberg, M. F., Kristensen, P. L., Larsen, F. B., Ortiz, B., & Juel, K. (2012) · Scandinavian Journal of Public Health · doi:10.1177/1403494812451412 · Cited by 211 · see citing works
“Despite the methodological concerns associated with it, a mixed-mode approach has many benefits which are largely seen to outweigh its potential challenges [20,22,26].”
BackgroundWhitehead, L. (2007) · Social Science & Medicine · doi:10.1016/j.socscimed.2007.03.005 · Cited by 207 · see citing works
Comparative symptomatology of burnout and depression
Bianchi, R., Boffy, C., Hingray, C., Truchot, D., & Laurent, É. (2013) · Journal of Health Psychology · doi:10.1177/1359105313481079 · Cited by 208 · see citing works
“To form the BOG and the CTR, we emailed the MBI (Dion and Tessier, 1994; Maslach and Jackson, 1986) to several hundred middle and high schools in France by using academic databases—past research has shown that Internet questionnaires are as valid and reliable as traditional, paper-and-pencil questionnaires (Gosling et…”
BackgroundMethodNormative data for the EORTC QLQ-C30 and EORTC-sexuality items in the general Dutch population
van de Poll-Franse, L. V., Mols, F., Gundy, C. M., Creutzberg, C. L., Nout, R. A., Leeuw, I. M. V., TAPHOORN, M. J. B., & Aaronson, N. K. (2010) · European Journal of Cancer · doi:10.1016/j.ejca.2010.11.004 · Cited by 208 · see citing works
Nachtegaal, J., Smit, J. H., Smits, C., Bezemer, P. D., van Beek, J. H. M., Festen, J. Μ., & Kramer, S. E. (2009) · Ear and Hearing · doi:10.1097/aud.0b013e31819c6e01 · Cited by 202 · see citing works
“Overall, it can be stated that there is satisfactory evidence for the reliability, validity, and feasibility of online questionnaires (Hallam et al. 2006; Kongsved et al. 2007; Ritter et al. 2004; Vallejo et al. 2007).”
BackgroundKhazaal, Y., van Singer, M., Chatton, A., Achab, S., Zullino, D., Rothen, S., Khan, R., Billieux, J., & Thorens, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2759 · Cited by 201 · see citing works
“Some studies suggest that the quality of the data provided by Internet-based surveys is at least as good as in those collected by traditional paper-and-pencil methods on self-selected samples [6-8].”
BackgroundRasch model analysis of the Depression, Anxiety and Stress Scales (DASS)
Shea, T. L., Tennant, A., & Pallant, J. (2009) · BMC Psychiatry · doi:10.1186/1471-244x-9-21 · Cited by 184 · see citing works
“Some studies investigating the impact of scale administration methods have reported similar psychometric properties for pencil-and-paper surveys and their web-based counterparts [41-43].”
Is burnout a depressive disorder? A reexamination with special focus on atypical depression.
Bianchi, R., Schonfeld, I. S., & Laurent, É. (2014) · International Journal of Stress Management · doi:10.1037/a0037906 · Cited by 184 · see citing works
Questionnaire Survey Nonresponse: A Comparison of Postal Mail and Internet Surveys
Hoonakker, P., & Carayon, P. (2009) · International Journal of Human-Computer Interaction · doi:10.1080/10447310902864951 · Cited by 173 · see citing works
“Based on their review of the literature Ritter et al. (2004, p. 29) concluded, “although progress is being made, there remains a need to evaluate Internet versions of most of the health-behavior and outcome instruments useful to researchers evaluating patient intervention programs.””
SoftGIS as a Bridge-Builder in Collaborative Urban Planning
Kahila, M., & Kyttä, M. (2009) · The Geojournal library · doi:10.1007/978-1-4020-8952-7_19 · Cited by 168 · see citing works
“On the whole, reviews and comparisons between web and traditional surveys show significant differences in neither the characteristics of the respondents (Gosling et al. 2004) nor in the results of the surveys ( Rittel et al. 2004 ).”
ResultItem non‐response rates: a comparison of online and paper questionnaires
Denscombe, M. (2008) · International Journal of Social Research Methodology · doi:10.1080/13645570802054706 · Cited by 163 · see citing works
The emotion regulation questionnaire: validation of the ERQ-9 in two community samples.
Spaapen, D. L., Waters, F., Brummer, L., Stopa, L. L., & Bucks, R. (2014) · Psychological Assessment · doi:10.1037/a0034474 · Cited by 167 · see citing works
Flanigan, T. S., McFarlane, E., & Cook, S. (2008) · Cited by 160 · see citing works
The Ethics of Internet Research
Eynon, R., Fry, J., & Schroeder, R. (2008) · SAGE Publications Ltd eBooks · doi:10.4135/9780857020055.n2 · Cited by 160 · see citing works
Survey Response Rates and Survey Administration in Counseling and Clinical Psychology
Horn, P. S. V., Green, K. E., & Martinussen, M. (2008) · Educational and Psychological Measurement · doi:10.1177/0013164408324462 · Cited by 156 · see citing works
Heitmann, K., Nordeng, H., Havnen, G., Solheimsnes, A., & Holst, L. (2017) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-017-1249-0 · Cited by 112 · see citing works
Fischer, A., Schröder, J., Vettorazzi, E., Wolf, O. T., Pöttgen, J., Lau, S., Heesen, C., Moritz, S., & Gold, S. M. (2015) · The Lancet Psychiatry · doi:10.1016/s2215-0366(14)00049-2 · Cited by 143 · see citing works
Liang, J., Xian, D., Liu, X., Fu, J., Zhang, X., Tang, B., & Lei, J. (2018) · JMIR mhealth and uhealth · doi:10.2196/11066 · Cited by 145 · see citing works
“They have been widely accepted by clinical medical informatics and consumer health informatics researchers and practitioners, and they are also widely applied in electronic health records, computerized physician order entry, and health applications [36-39].”
BackgroundRelationship between meditative practice and self-reported mindfulness: the MINDSENS composite index
Soler, J., Cebolla, A., Feliu-Soler, A., Demarzo, M. M. P., Pascual, J. C., Baños, R., & García-Campayo, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086622 · Cited by 138 · see citing works
“500) and the existence of studies that confirm the reliability of data obtained from this source [43], these samples are probably more heterogeneous and biased–due to the high non-response rate–than those obtained by other, more traditional methods.”
ResultNew times, new needs; e-epidemiology
Ekman, A., & Litton, J. (2007) · European Journal of Epidemiology · doi:10.1007/s10654-007-9119-0 · Cited by 139 · see citing works
Damholdt, M. F., Mehlsen, M. Y., O’Toole, M. S., Andreasen, R., Pedersen, A., & Zachariae, R. H. (2016) · Psycho-Oncology · doi:10.1002/pon.4058 · Cited by 138 · see citing works
“Generally, previous studies conclude that there are no differences between online and paper-and-pencil questionnaires on psychometric properties [38], particularly when sampling is controlled [39].”
BackgroundLjótsson, B., Andersson, G., Andersson, E., Hedman, E., Lindfors, P., Andréewitch, S., Rück, C., & Lindefors, N. (2011) · BMC Gastroenterology · doi:10.1186/1471-230x-11-110 · Cited by 132 · see citing works
“All measures were administered online, a method which has been shown to be reliable and as valid as traditional paper-and-pencil administration [40,41].”
MethodCrutzen, R., & Göritz, A. S. (2010) · BMC Public Health · doi:10.1186/1471-2458-10-720 · Cited by 129 · see citing works
“, in which web-based assessment is compared with, for example, paper-and-pencil assessment [26,27] or with telephone interviewing [28]) generally have relied on one of three different designs: randomization after recruitment (true experimental design), randomization before recruitment (where there may be differences i…”
MethodLinke, S., Murray, E., Butler, C., & Wallace, P. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e10 · Cited by 127 · see citing works
“It is difficult to know how truthful they were in completing these questionnaires, but there is some evidence to support the suggestion that responses to online questionnaires are comparable to traditional paper-based versions [42,43].”
BackgroundPrevalence of Sexual Dysfunction and Impact of Contraception in Female German Medical Students
Wallwiener, C. W., Wallwiener, L.-M., Seeger, H., Mück, A. O., Bitzer, J., & Wallwiener, M. (2010) · The Journal of Sexual Medicine · doi:10.1111/j.1743-6109.2010.01742.x · Cited by 124 · see citing works
Scott, D. M., Berg, M. J., Tolhurst, B. A., Chauvenet, A. L. M., Smith, G. C., Neaves, K., Lochhead, J., & Baker, P. J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099059 · Cited by 122 · see citing works
“In the modern era, web-based questionnaires potentially represent a more efficient approach for conducting large-scale surveys with significant logistical and cost advantages [45] [46].”
Hunter, L. (2012) · Nurse Researcher · doi:10.7748/nr2012.09.20.1.11.c9303 · Cited by 121 · see citing works
Participant recruitment and data collection through Facebook: the role of personality factors
Rife, S. C., Cate, K. L., Kosiński, M., & Stillwell, D. J. (2014) · International Journal of Social Research Methodology · doi:10.1080/13645579.2014.957069 · Cited by 119 · see citing works
The Role of Online Support Communities
Idriss, S. Z., Kvedar, J. C., & Watson, A. J. (2009) · Archives of Dermatology · doi:10.1001/archdermatol.2008.529 · Cited by 119 · see citing works
Mode of administration does not cause bias in patient-reported outcome results: a meta-analysis
Rutherford, C., Costa, D. S. J., Mercieca‐Bebber, R., Rice, H. E., Gabb, L., & King, M. T. (2015) · Quality of Life Research · doi:10.1007/s11136-015-1110-8 · Cited by 115 · see citing works
Moritz, S., Jahns, A. K., Schröder, J., Berger, T., Lincoln, T. M., Klein, J. P., & Göritz, A. S. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.11.027 · Cited by 115 · see citing works
Hernández, G., Garín, O., Dima, A. L., Pont, À., Marti-Pastor, M., Alonso, J., Ganse, E. H. V., Laforest, L., de Bruin, M., Mayoral, K., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/10178 · Cited by 114 · see citing works
“The internet era has led to implementing Web-based surveys to take advantage of the known benefits such as completeness [41,42], low expenses [43], and better data management.”
Moritz, S., Jelinek, L., Hauschildt, M., & Naber, D. (2010) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2010.12.2/smoritz · Cited by 113 · see citing works
Ljótsson, B., Hedman, E., Lindfors, P., Hursti, T., Lindefors, N., Andersson, G., & Rück, C. (2011) · Behaviour Research and Therapy · doi:10.1016/j.brat.2010.10.006 · Cited by 113 · see citing works
Martins, Y., Lederman, R. I., Lowenstein, C. L., Joffe, S., Neville, B. A., Hastings, B. T., & Abel, G. A. (2012) · British Journal of Cancer · doi:10.1038/bjc.2012.28 · Cited by 112 · see citing works
Evaluating Patients’ Experiences with Individual Physicians
Rodríguez, H. P., von Glahn, T., Rogers, W. H., Chang, H., Fanjiang, G., & Safran, D. G. (2006) · Medical Care · doi:10.1097/01.mlr.0000196961.00933.8e · Cited by 112 · see citing works
Experiences of Fecal Incontinence in People with Inflammatory Bowel Disease
Dibley, L. B., & Norton, C. S. (2013) · Inflammatory Bowel Diseases · doi:10.1097/mib.0b013e318281327f · Cited by 109 · see citing works
Development of a dual school climate and school identification measure–student (SCASIM-St)
Lee, E., Reynolds, K. J., Subašić, E., Bromhead, D., Lin, H., Marinov, V., & Smithson, M. J. (2017) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2017.01.003 · Cited by 113 · see citing works
Raat, H., Mangunkusumo, R. T., Landgraf, J. M., Kloek, G. C., & Brug, J. (2007) · Quality of Life Research · doi:10.1007/s11136-006-9157-1 · Cited by 106 · see citing works
“found no statistically significant score differences between internet and paper modes of administration for 16 health-related measures, but the study was conducted in an opportunity sample retrieved from the internet, which limits its generalizability [23].”
BackgroundGordon, W. V., Shonin, E., Dunn, T. J., García‐Campayo, J., & Griffiths, M. D. (2016) · British Journal of Health Psychology · doi:10.1111/bjhp.12224 · Cited by 103 · see citing works
Zuidgeest, M., Hendriks, M. C. P., Koopman, L., Spreeuwenberg, P. M. M., & Rademakers, J. J. D. J. M. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1241 · Cited by 103 · see citing works
“Other reasons were having no experience with the Internet or not trusting the Internet [31].”
Pregnancy-specific anxiety: which women are highest and what are the alcohol-related risks?
Arch, J. J. (2012) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2012.07.010 · Cited by 105 · see citing works
“[15,32,49]).”
Feasibility of recruiting a birth cohort through the Internet: the experience of the NINFEA cohort
Richiardi, L., Baussano, I., Vizzini, L., Douwes, J. E., Pearce, N. E., & Merletti, F. (2007) · European Journal of Epidemiology · doi:10.1007/s10654-007-9194-2 · Cited by 102 · see citing works
“Recently, some studies have posted surveys on websites, recruiting participants whose contact information was unknown before participation [12–14].”
BackgroundHousing strain, mortgage foreclosure, and health
Cannuscio, C. C., Alley, D. E., Pagán, J. A., Soldo, B. J., Krasny, S., Shardell, M. D., Asch, D. A., & Lipman, T. H. (2011) · Nursing Outlook · doi:10.1016/j.outlook.2011.08.004 · Cited by 98 · see citing works
Impact of the Cost of Cancer Treatment: An Internet-Based Survey
Markman, M., & Luce, R. (2010) · Journal of Oncology Practice · doi:10.1200/jop.091074 · Cited by 97 · see citing works
Horevoorts, N. J. E., Vissers, P. A. J., Mols, F., Thong, M. S. Y., & van de Poll-Franse, L. V. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.3741 · Cited by 94 · see citing works
“Several studies also show that reliability of Web-based questionnaires and paper questionnaires is comparable [8-11].”
Emergency Physicians' Perceptions of Health Information Exchange
Shapiro, J. S., Kannry, J. L., Kushniruk, A. W., & Kuperman, G. J. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2507 · Cited by 93 · see citing works
The Length of Responses to Open-Ended Questions
Denscombe, M. (2007) · Social Science Computer Review · doi:10.1177/0894439307309671 · Cited by 86 · see citing works
Marquet, R. L., Bartelds, A. I. M., van Noort, S. P., Koppeschaar, C. E., Paget, J., Schellevis, F., & van der Zee, J. (2006) · BMC Public Health · doi:10.1186/1471-2458-6-242 · Cited by 85 · see citing works
“It has been argued, however, that internet samples may actually be more representative than traditional samples [3].”
BackgroundMcCambridge, J., Mitcheson, L., WINSTOCK, A. R., & Hunt, N. (2005) · Addiction · doi:10.1111/j.1360-0443.2005.001127.x · Cited by 85 · see citing works
A Web-Based Cross-Sectional Epidemiological Survey of Complex Regional Pain Syndrome
Sharma, A. K., Agarwal, S., Broatch, J., & Raja, S. N. (2009) · Regional Anesthesia & Pain Medicine · doi:10.1097/aap.0b013e3181958f90 · Cited by 81 · see citing works
Inquiry into the correlation between burnout and depression.
Schonfeld, I. S., Verkuilen, J., & Bianchi, R. (2019) · Journal of Occupational Health Psychology · doi:10.1037/ocp0000151 · Cited by 86 · see citing works
Whitehead, L. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1593 · Cited by 78 · see citing works
The Danish health and morbidity surveys: study design and participant characteristics
Jensen, H. A. R., Ekholm, O., Davidsen, M., & Christensen, A. I. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0733-9 · Cited by 77 · see citing works
Long-term effects of an internet-based treatment for posttraumatic stress
Knaevelsrud, C., & Maercker, A. (2010) · Cognitive Behaviour Therapy · doi:10.1080/16506070902999935 · Cited by 76 · see citing works
The Roles of Procedural and Distributive Justice in Logistics Outsourcing Relationships
Hofer, A. R., Knemeyer, A. M., & Murphy, P. R. (2012) · Journal of Business Logistics · doi:10.1111/j.2158-1592.2012.01052.x · Cited by 78 · see citing works
Burnout is associated with a depressive cognitive style
Bianchi, R., & Schonfeld, I. S. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.01.008 · Cited by 78 · see citing works
Varghese, N. E., Sabat, I., Neumann‐Böhme, S., Schreyögg, J., Stargardt, T., Torbica, A., van Exel, J. N. A., Barros, P. P., & Brouwer, W. B. F. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250872 · Cited by 74 · see citing works
“Also, studies show internet based questionnaires to be a “suitable alternative” to more traditional survey methods [27, 28].”
BackgroundThe burden of stroke in the Netherlands: estimating quality of life and costs for 1 year poststroke
van Eeden, M., Heugten, C. M. V., van Mastrigt, G. A. P. G., van Mierlo, M., Visser‐Meily, J. M. A., & Evers, S. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008220 · Cited by 73 · see citing works
“Previous research had found no differences between online questionnaires and questionnaires administered on paper.(19) All questions administered on paper were checked by the research assistant to prevent missing data.”
BackgroundHardré, P. L., Crowson, H. M., Xie, K., & Ly, C. (2006) · British Journal of Educational Technology · doi:10.1111/j.1467-8535.2006.00591.x · Cited by 71 · see citing works
Use of a Web-based Questionnaire in the Black Women's Health Study
Russell, C. W., Boggs, D. A., Palmer, J. R., & Rosenberg, L. A. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq310 · Cited by 71 · see citing works
Web-based survey attracted age-biased sample with more severe illness than paper-based survey
Klovning, A., Sandvik, H., & Hunskaar, S. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.015 · Cited by 70 · see citing works
“The instruments administered through the Internet appear to be reliable, and are to be answered similarly to the way they are answered when administered through traditional postal paper questionnaires [27].”
MethodHughes, C., Davoodi‐Semiromi, Y., Colee, J., Culpepper, T., Dahl, W. J., Mai, V., Christman, M. C., & Langkamp‐Henken, B. (2011) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.111.014126 · Cited by 70 · see citing works
“Ritter and colleagues tested the reliability of 16 different health-related instruments administered online compared to the traditional mailed paper questionnaires (124).”
Moritz, S., Peters, M. J. V., Larøi, F., & Lincoln, T. M. (2010) · Cognitive Neuropsychiatry · doi:10.1080/13546801003783508 · Cited by 69 · see citing works
“In addition, mounting evidence suggests that results obtained over the Internet are comparable in reliability with studies using more conventional measures (Jones, Fernyhough, de-Wit, & Meins, 2008 ; Ritter, Lorig, Laurent, & Matthews, 2004 ; Riva, Teruzzi, & Anolli, 2003 ), even in studies including severely disturbe…”
ResultWestermann, S., Kesting, M.-L., & Lincoln, T. M. (2011) · Behavior Therapy · doi:10.1016/j.beth.2011.07.005 · Cited by 69 · see citing works
Holmes, S. (2008) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2008.08.004 · Cited by 68 · see citing works
“The effect of the Internet on response rates is, however, equivocal and research to date reports inconsistent findings; some have reported moderate to high response rates ranging between 45% and 96% (VanDenKerkhof et al., 2004; Kypri and McAnally, 2004; Leece et al., 2004; McGee and Kypri, 2004) while others have repo…”
ResultInfluentialBloem, B. R., & Stocchi, F. (2011) · European Journal of Neurology · doi:10.1111/j.1468-1331.2011.03532.x · Cited by 68 · see citing works
“In addition, online questionnaires achieve slightly higher completion rates than mailed questionnaires [49].”
Montero-Marín, J., Gaete, J., Demarzo, M., Rodero, B., Lopez, L. C. S., & García-Campayo, J. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01281 · Cited by 66 · see citing works
Characteristics of Gamblers Using a National Online Counselling Service for Problem Gambling
Rodda, S. N., & Lubman, D. I. (2013) · Journal of Gambling Studies · doi:10.1007/s10899-012-9352-7 · Cited by 67 · see citing works
“Self-report instruments developed for paper and pencil administration also appear to maintain their psychometric properties when delivered online (Ritter et al. 2004).”
BackgroundJanda, M., DiSipio, T., Hurst, C. P., Cella, D. F., & Newman, B. M. (2008) · Psycho-Oncology · doi:10.1002/pon.1428 · Cited by 67 · see citing works
Young, N. L., Varni, J. W., Snider, L., McCormick, A., Sawatzky, B., Scott, M., King, G. A., Hetherington, R., Sears, E., & Nicholas, D. B. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.011 · Cited by 65 · see citing works
“found survey response rates on the Internet to be 13% lower than those on paper [16].”
BackgroundPathways to hydrogen as an energy carrier
Sigfusson, T. I. (2007) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2006.1960 · Cited by 65 · see citing works
Deitz, M. F., Williams, S. L., Rife, S. C., & Cantrell, P. J. (2015) · Violence Against Women · doi:10.1177/1077801215573330 · Cited by 64 · see citing works
Burnout and depression: Label-related stigma, help-seeking, and syndrome overlap
Bianchi, R., Verkuilen, J., Brisson, R., Schonfeld, I. S., & Laurent, É. (2016) · Psychiatry Research · doi:10.1016/j.psychres.2016.08.025 · Cited by 63 · see citing works
“Online questionnaires have been shown to be as reliable and valid as traditional, paper-and-pencil questionnaires (Gosling et al., 2004; Jones et al., 2008; Ritter et al., 2004).”
BackgroundInternet- vs. telephone-administered questionnaires in a randomized trial of smoking cessation
Graham, A. L., Papandonatos, G. D., Bock, B. C., Cobb, N. K., Baskin-sommers, A. R., NIAURA, R. S., & Abrams, D. B. (2006) · Nicotine & Tobacco Research · doi:10.1080/14622200601045367 · Cited by 63 · see citing works
“, 2003), personality (Buchanan & Smith, 1999; Davis, 1999), and a variety of measures related to health status and health behaviors (Ritter et al., 2004).”
Backgroundvan Mierlo, M. L., Heugten, C. M. V., Post, M. W. M., Lindeman, E., de Kort, P. L. M., & Visser‐Meily, J. M. A. (2012) · International Journal of Stroke · doi:10.1111/j.1747-4949.2012.00882.x · Cited by 63 · see citing works
Lamberts, M. P., Oudsten, B. L. D., Gerritsen, J. J. G. M., Roukema, J. A., Westert, G. P., Drenth, J. P. H., & van Laarhoven, C. J. H. M. (2015) · British journal of surgery · doi:10.1002/bjs.9887 · Cited by 62 · see citing works
- Authors
- Ritter, P., Lorig, K., Laurent, D., & Matthews, K.
- Affiliations
- Stanford Medicine (US)
- Stanford University (US)
- Corresponding author
- Ritter, Philip
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2004-09-15
- Volume
- 6
- Issue
- 3
- Pages
- e29
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC1550608
- OpenAlex
- W2072139531
- Cited by
- 427
- References
- 16
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 4 · 2023: 11 · 2022: 9 · 2021: 13 · 2020: 10 · 2019: 13 · 2018: 18 · 2017: 18 · 2016: 22 · 2015: 29 · 2014: 25 · 2013: 27 · 2012: 35
- FWCI
- 69.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- TSpace (University of Toronto) (submittedVersion, cc-by)
- Funders
- Robert Wood Johnson Foundation
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Education › Health Education and Validation
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Health Systems, Economic Evaluations, Quality of Life
- Keywords
- internet-based questionnaire, mail surveys, self-report measures, psychometric properties, test-retest reliability, construct reliability, questionnaire completion rates, patient education evaluation, patient-reported outcomes
- MeSH terms
- Adult, Aged, Aged, 80 and over, Female, Humans, Male, Middle Aged, Postal Service, Surveys and Questionnaires, Random Allocation, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
