Internet-based Questionnaire Assessment: Appropriate Use in Clinical Contexts

“An expanding literature concerned with how to collect patient data via the Internet has evolved (17), and it is timely to highlight the pros and cons of online data collection.”

“It is now fairly established that psychometric properties of self-report measures tend to transfer well when the same questionnaires are administered via the Internet, but that each instrument needs to be validated for online use (Buchanan, 2003).”

“And Buchanan (2003) [9] reported that even when Internet-based versions of instruments are reliable and valid, normative data from paper-and-pencil versions may not always compare directly with Internet-mediated psychological testing.”

“As the validity of Internet-administered questionnaires is not yet clear, all outcome measures were administered on paper (cf. Andersson, Kaldo-Sandström, Ström, & Strömgren, 2003; Buchanan, 2003).”

“As the validity of Internet-administered questionnaires is not yet clear, all outcome measures were administered on paper-and-pencil (cf. Andersson, Kaldo-Sandström, Ström, & Strömgren, 2003; Buchanan, 2003).”

“, [8,13,23,43]) and a need to validate measures for use in this context [6,7].”

“Self-report questionnaires generally show the same psychometric properties as paper-and-pencil administration (Buchanan, 2003).”

“However, the psychometric equivalence between paper-pencil and Internet-versions of the same questionnaire cannot be assumed (Buchanan, 2002; 2003)(Buchanan 148-54;Buchanan 100-09).”

“However, when self-report questionnaires are transferred to the Internet, psychometric properties cannot be assumed to be identical to their paper version equivalents (Andersson, Ritterband, & Carlbring, 2008; Buchanan, 2003).”

“It has already been shown how easy it is to create a website containing tools to assess psychological problems or constructs [2].”

“In fact, there has been a call to establish the reliability and validity of each measure used on the Internet (Buchanan, 2003).”

“However, when a questionnaire is placed on the Internet this might subtly affect expected score distributions, perhaps due to increased self-disclosure or increased negative affect induced by the computer situation ( Buchanan 2003; Peterson et al. 1996).”

“However, it has been argued that we cannot assume that the psychometric properties remain the same after such adaptation [11].”

“Buchanan (2003) reported that even when Internet-based versions of instruments are reliable and valid, normative data from paper-and-pencil versions may not always compare directly with Internet-mediated psychological testing.”

“Recent research has found a striking similarity in psychometric properties of scales that have been administrated via the internet compared with those with paper-and-pencil format (Buchanan, 2003; Andersson, Kaldo-Sandström, Ström & Strömgren, in press).”

“These differences may be due to the lessened impact of social desirability, an increase of self-disclosure in online assessment, or as the result of the technology-based presentation itself (Buchanan, 2003).”

“Such a questionnaire would preferably be conducted via the Internet, as it offers quick and easy access to large numbers of users at a low cost [5,6].”

“We have not commented on other applications of the Internet in psychological research such as web-based experiments (Birnbaum, 2000), or studies on the psychometric properties of web-based questionnaires (Buchanan, 2003).”

“Previous research suggests that online administration of questionnaires generally results in excellent psychometric properties (Buchanan, 2003), but web-based survey administration also poses a limitation, as Internet use is not encouraged by some religious groups (Armfield & Holbert, 2003).”

“Previous psychometric research has validated internetadministration of self-rating scales for depression, quality of life and anxiety (Carlbring et al., 2007; Hedman et al., 2010; Holländare et al., 2010; Buchanan, 2003).”

“When we began doing Internet studies we did not consider the implications of using online questionnaires (Buchanan, 2003), and for a while we decided to use paper and pencil tests until we had evaluated the psychometric properties of online questionnaires and tests.”

“This conclusion is supported by Buchanan (2003), who noted demographic, cultural and symptom-based differences between online and offline populations.”

“…diagnostic scales online is now an established practice (Andersson et al., 2008; Nguyen, Klein, Meyer, Austin, & Abbott, 2015), but there are generally less validation data available for measures administered electronically or via the Internet (Buchanan, 2003; van Ballegooijen et al., 2016).”

“Self-report measures were administered via the internet which generally has been found to generate acceptable psychometric properties (Buchanan, 2003; Carlbring et al., 2007).”

“However, when transferring paper-and-pen psychometric questionnaires into electronic forms of administration, it is important to realize that this may influence their outcome (Buchanan, 2003).”

“Such work is required not only for the PMQ, but also for a wide range of other instruments, in order to identify the mechanisms underpinning any possible Internet administration effect and, perhaps, specific psychological constructs likely to be affected in this way (e.g., computer anxiety and, perhaps, negative affec…”

“Although there is evidence to suggest that online measures of psychological traits are valid and reliable, other research does suggest that the psychometric properties of the measure may change when administered via the web (Buchanan, 2000, 2001).”

“…properties in webbased versus paper-pencil questionnaires (Andersson et al., 2003; Carlbring et al., 2007a; Houston et al., 2001; Spek et al., 2008), other studies did not (reviewed by Buchanan (2002), which means that the psychometric equivalence cannot be assumed (Buchanan 2002; 2003).”

“While there is some research to suggest that paper-and-pencil and online measures do not produce significantly different results (Buchanan, 2003), determining the best way to present online measures has yet to be established.”

“None of the measures have been adapted for online use, which limits their comparability to results obtained from paper pencil tests [99].”

“24) Pain duration (n [%]) 3mo-1 y 6 (12) 3 (11) 3 (13) 1 (11) 1-2 y 3 (6) 1 (4) 2 (8) 0 (0) 2-5 y 8 (15) 3 (11) 5 (21) 1 (11) 5-10 y 7 (13) 5 (18) 2 (8) 1 (11) 10-20 y 15 (29) 9 (32) 6 (25) 1 (11) 20-30 y 10 (19) 6 (21) 4 (17) 5 (56) >30 y 3 (6) 1 (4) 2 (8) Sex (n [%]) Men 8 (15) 4 (14) 4 (17) 1 (11) Women 44 (85) 24…”

“Previous research suggests that online administration of questionnaires generally results in excellent psychometric properties, although norms might differ slightly from paper-andpencil administration (Buchanan, 2003).”

“Although it has often been found that well-established scales do well regardless of administration format, online adaptions of validated scales should preferably undergo separate validation [9,10].”

“Some systematic differences have been noted in response to the questionnaire administered online versus on paper-and-pencil [25,26].”

“Self-report measures generally show the same psychometric properties when completed over the Internet as when they are administered in their paper-and-pencil format (Buchanan, 2003).”

“…(for a review, see Wood & Griffiths, 2007), the validity of online diagnoses seems to be uncertain, because systematic investigations evidencing the psychometrical properties of traditional offline instruments used on the Internet have not been carried out (Andersson & Titov, 2014; Buchanan, 2003).”

“This is an important point because the Internet is being increasingly used for data collection and clinical research purposes ( Buchanan, 2003; Scholey et al., 2004).”

“Although web-based administration of questionnaires is widely used, we should remind ourselves that the psychometric properties of internet-administered questionnaires may not be equal to those of paper-and-pencil versions [32].”

“For example, clinicians should be careful when they use Internet-based neuropsychological tools because normative data established for conventional versions of the tests do not necessarily apply, and norms for online versions have not yet been established (Buchanan, 2003).”

“Although most of the research to date has indicated that online questionnaires can be valid, there are reports of instances where factor structures and mean scores have been found to differ (see, e.g., Buchanan, 2001; Buchanan, 2002; Buchanan, 2003; Buchanan et al., 2002a; and other work cited later in this chapter).”

“All assessments will be conducted via online self-report; with online administration of assessments likely to yield equivalent psychometrics as their paper-and-pencil counterparts (Buchanan, 2003).”

“Furthermore, recent evidence indicates that people with mental illness are well equipped and able to use modern communication technologies for outcome assessment [34-36], and that reliable and valid outcome ratings can be obtained via the internet [37].”

“, 2007), which is in line with previous research on internet administration of questionnaires (Buchanan, 2003).”

“…disorders, Internet- and paper-based administration of structured and validated instruments are likely to generate largely equivalent results, although researchers and practitioners need to remain aware of possible exceptions (e.g., Health Anxiety Depression Scale; see Buchanan, 2003).”

“Computer-based screening and assessment has proven to be an empirically sound way to gather information and has demonstrated comparable psychometric properties to other empirically developed methods of administration (e.g., face to face, paper and pencil; Buchanan, 2003).”

“Internet administration of selfreport instruments has been found to have the similar psychometric properties as their paperand-pen equivalents (Buchanan 2003), which also included measures used in this study (Andersson, Kaldo-Sandström, Ström, & Strömgren, 2003).”

“, [85,86]), in this case most studies have only used ad hoc questionnaires or surveys where the validity and reliability of the measuring instruments have not been considered.”

“The development of the Internet and the Web thus provides a new way to collect data (Buchanan, 2003; Reips, 2000; Welch & Krantz, 1996).”

“There is evidence (Buchanan, 2002; 2003) that factor structures of psychological questionnaires may change subtly when they are converted from paper and pencil format to electronic equivalents.”

“It has been suggested that norms derived from computerised methods may differ from those obtained from paper-and-pencil versions of the same instruments (Andersson, Kaldo-Sandström, Ström, & Strömgren, 2003; Buchanan, 2003; Ferrando & Lorenzo-Seva, 2005), and this may be one reason for the relatively inflated EPDS sco…”

“Differences in factor structure imply that instruments do not measure the same construct and cannot be treated as equal or comparable to each other [17].”

“concise ten-item Edinburgh Depression Scale might be a very good internet-administered screening device for depression, although good psychometric properties of a paper and pencil version of a questionnaire do not guarantee good psychometric properties of an internetadministered version (Buchanan, 2003).”

“If Web-based surveys have a substantial direct effect on study outcome, this must lead to the conclusion that results from Web-based questionnaires are not comparable to traditional postal surveys [11].”

“However, it cannot be taken for granted that self-report measures have equivalent properties when administered online as in their paper-and-pencil formats, and separate psychometric evaluations of the administration formats have been recommended (Buchanan, 2003).”

“It has therefore been recommended that questionnaires for Internet use should be validated as results might differ; this topic is covered in detail in this issue (Buchanan, 2003).”

“Buchanan [15] is skeptical of using Web-based questionnaires for normative comparisons, especially in clinical psychology.”

“This finding adds to the evidence showing that outcome assessment via computer is feasible in people with less severe mental disorders (36).”

“In many cases, differences in score distributions can be attributed to differences between samples tested online and offline (Buchanan, 2003); characteristics of samples should, therefore, be examined accurately and this issue should be carefully considered when the collected data are analyzed.”

“These findings further underscore the cautions of researchers such as Buchanan (2003), who has suggested that it may be inappropriate to share normative data across different assessment modalities and Carlbring et al.”

“Some studies found no major differences between the two delivery modes [19], whereas others found variations related to particular scales [20] and still others reported differences between online and offline versions of the same tests in terms of score distributions achieved and psychometric properties of the tests [2…”

“This is a major limitation for all web-based questionnaire assessments (Buchanan, 2003), however our study has additional limitations as a free online screening tool.”

“For example, it has been shown that some tools have different psychometric properties when delivered online, suggesting a need for different cutoff points [44].”

“Early diagnosis and timely intervention of adolescents with mental and behavioral disorders is very important [4], and since 1950 many studies have been carried out on the prevalence of behavior disorders and problems among student adolescents [5, 6].”

“Internet assessment also allows for the provision of immediate feedback to participants, with fewer errors due to automatic scoring and use of item response theory to tailor item difficulty or increase individualization throughout the administration (Buchanan, 2003; Naglieri et al., 2004).”

“Evidence of good psychometric properties in an offline version of the same questionnaire is not enough to assume that they will apply to the online version (Buchanan, 2003).”

“This might be due to higher self-disclosure associated with online administration (Buchanan, 2003), or in the case of Verkuil and Brosschot’s study, the authors suggest that participation could have been more appealing for high worriers, because the study was explicitly advertised as focusing on…”

“However, there are risks in assuming that both formats will produce equivalent results [19,20].”