An Experimental Comparison of Web and Telephone Surveys

Abstract

We carried out an experiment that compared telephone and Web versions of a questionnaire that assessed attitudes toward science and knowledge of basic scientific facts. Members of a random digit dial (RDD) sample were initially contacted by telephone and answered a few screening questions, including one that asked whether they had Internet access. Those with Internet access were randomly assigned to complete either a Web version of the questionnaire or a computer-assisted telephone interview. There were four main findings. First, although we offered cases assigned to the Web survey a larger incentive, fewer of them completed the online questionnaire; almost all those who were assigned to the telephone condition completed the interview. The two samples of Web users nonetheless had similar demographic characteristics. Second, the Web survey produced less item nonresponse than the telephone survey. The Web questionnaire prompted respondents when they left an item blank, whereas the telephone interviewers accepted “no opinion” answers without probing them. Third, Web respondents gave less differentiated answers to batteries of attitude items than their telephone counterparts. The Web questionnaire presented these items in a grid that may have made their similarity more salient. Finally, Web respondents took longer to complete the knowledge items, particularly those requiring open-ended answers, than the telephone respondents, and Web respondents answered a higher percentage of them correctly. These differences between Web and telephone surveys probably reflect both inherent differences between the two modes and incidental features of our implementation of the survey. The mode differences also vary by item type and by respondent age.