CoChat
Sign Up
Cited by 50Survey Methodology and NonresponseInformation and Cyber SecurityPrivacy, Security, and Data Protection

A Summary of Survey Methodology Best Practices for Security and Privacy Researchers

Open access
Redmiles, Elissa M., Acar, Yasemin, Fahl, Sascha, Mazurek, Michelle L.·2017·Digital Repository at the University of Maryland
DOI 10.13016/m22k2w
Open this paper in CoChat — free accountOpen PDF

Abstract

"Given a choice between dancing pigs and security, users will pick dancing pigs every time," warns an oft-cited quote from well-known security researcher Bruce Schneier. This issue of understanding how to make security tools and mechanisms work better for humans (often categorized as usability, broadly construed) has become increasingly important over the past 17 years, as illustrated by the growing body of research. Usable security and privacy research has improved our understanding of how to help users stay safe from phishing attacks, and control access to their accounts, as just three examples. One key technique for understanding and improving how human decision making affects security is the gathering of self-reported data from users. This data is typically gathered via survey and interview studies, and serves to inform the broader security and privacy community about user needs, behaviors, and beliefs. The quality of this data, and the validity of subsequent research results, depends on the choices researchers make when designing their experiments. Contained here is a set of essential guidelines for conducting self-report usability studies distilled from prior work in survey methodology and related fields. Other fields that rely on self-report data, such as the health and social sciences, have established guidelines and recommendations for collecting high quality self-report data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.13016/m22k2w

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The survey relies on self-reported behavior, which is susceptible to social-desirability bias: respondents may over-report diligence and under-report risky practices.

    “Our survey relies on self-reported behavior, which is susceptible to social-desirability bias: respondents may over-report diligence and under-report risky practices [51].”

    From GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models (2026) · cites this paper as [51]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 108 · Cited by 50

Research works printed but not linked: 111

  1. A. Adams and M. A. Sasse. Users are not the enemy. Communications of the ACM, 42(12):40-46, 1999.
  2. I. Adjerid, A. Acquisti, L. Brandimarte, and G. Loewenstein. Sleights of privacy: Framing, disclosures, and the limits of transparency. In Proceedings of the Ninth Symposium on Usable Privacy and Security, page 9. ACM, 2013.
  3. G. Albaum. The likert scale revisited: an alternate version. Journal of the Market Research Society, 39(2):331-332, 1997.
  4. L. Andres. Designing and doing survey research. Sage Publications Ltd, London, 2012.
  5. D. Andrews, B. Nonnecke, and J. Preece. Electronic survey methodol- ogy: A case study in reaching hard-to-involve internet users. Interna- tional journal of human-computer interaction, 16(2):185-210, 2003.
  6. J. S. Armstrong and T. S. Overton. Estimating nonresponse bias in mail surveys. Journal of marketing research, pages 396-402, 1977.
  7. R. Baker, S. J. Blumberg, J. M. Brick, M. P. Couper, M. Courtright, J. M. Dennis, D. Dillman, M. R. Frankel, P. Garland, R. M. Groves, et al. Research synthesis aapor report on online panels. Public Opinion Quarterly, 74(4):711-781, 2010.
  8. S. B. Barnes. A privacy paradox: Social networking in the united states. First Monday, 11(9), 2006.
  9. L. Bauer, L. F. Cranor, R. W. Reeder, M. K. Reiter, and K. Vaniea. A user study of policy creation in a flexible access-control system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 543-552. ACM, 2008.
  10. P. C. Beatty and G. B. Willis. Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2):287-311, 2007.
  11. R. F. Belli, M. W. Traugott, and M. N. Beckmann. What leads to voting overreports? contrasts of overreporters to validated voters and admitted nonvoters in the american national election studies. Journal of Official Statistics, 17(4):479, 2001.
  12. W. A. Belson. Effects of reversing presentation order of verbal rating scales. Journal of Advertising Research, 6(4):30-37, 1966.
  13. L. Betts and J. Hartley. The effects of changes in the order of verbal labels and numerical values on childrens scores on attitude and rating scales. British Educational Research Journal, 38(2):319-331, 2012.
  14. P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, and S. Sudman. Measurement errors in surveys, volume 173. John Wiley & Sons, 2011.
  15. P. Biernacki and D. Waldorf. Snowball sampling: Problems and tech- niques of chain referral sampling. Sociological methods & research, 10(2):141-163, 1981.
  16. G. Blair, K. Imai, and Y.-Y. Zhou. Design and analysis of the randomized response technique. Journal of the American Statistical Association, 110(511):1304-1319, 2015.
  17. J. Blair and F. G. Conrad. Sample size for cognitive interview pretesting. Public Opinion Quarterly, 75(4):636-658, 2011.
  18. N. M. Bradburn and C. Miles. Vague quantifiers. Public Opinion Quarterly, 43(1):92-101, 1979.
  19. J. Brooke et al. Sus-a quick and dirty usability scale. Usability evaluation in industry, 189(194):4-7, 1996.
  20. M. C. Bryson. The literary digest poll: Making of a statistical myth. The American Statistician, 30(4):184-185, 1976.
  21. M. Buhrmester, T. Kwang, and S. D. Gosling. Amazon's mechanical turk a new source of inexpensive, yet high-quality, data? Perspectives on psychological science, 6(1):3-5, 2011.
  22. P. Campanelli. Testing survey questions: New directions in cognitive interviewing. Bulletin de Methodologie Sociologique, 55(1):5-17, 1997.
  23. X. Cao and L. Iverson. Intentional access management: Making access control usable for end-users. In Proceedings of the second symposium on Usable privacy and security, pages 20-31. ACM, 2006.
  24. N. Christin, S. Egelman, T. Vidas, and J. Grossklags. It's All about the Benjamins: An Empirical Study on Incentivizing Users to Ignore Security Advice. In Financial Cryptography, pages 16-30, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.
  25. D. Collins. Pretesting survey instruments: an overview of cognitive methods. Quality of life research, 12(3):229-238, 2003.
  26. D. Colton and R. W. Covert. Designing and constructing instruments for social research and evaluation. John Wiley & Sons, 2007.
  27. D. Corstange. Sensitive questions, truthful answers? modeling the list experiment with listit. Political Analysis, 17(1):45-63, 2009.
  28. M. P. Couper, F. G. Conrad, and R. Tourangeau. Visual context effects in web surveys. Public Opinion Quarterly, 71(4):623-634, 2007.
  29. M. P. Couper and P. V. Miller. Web survey methods introduction. Public Opinion Quarterly, 72(5):831-835, 2008.
  30. M. P. Couper, R. Tourangeau, and K. Kenyon. Picture this! exploring visual effects in web surveys. Public Opinion Quarterly, 68(2):255- 266, 2004.
  31. M. P. Couper, M. W. Traugott, and M. J. Lamias. Web survey design and administration. Public opinion quarterly, 65(2):230-253, 2001.
  32. E. Coutts and B. Jann. Sensitive questions in online surveys: Exper- imental results for the randomized response technique (rrt) and the unmatched count technique (uct). Sociological Methods & Research, 40(1):169-193, 2011.
  33. S. D. Crawford, M. P. Couper, and M. J. Lamias. Web surveys perceptions of burden. Social science computer review, 19(2):146-162, 2001.
  34. R. A. Cummins and E. Gullone. Why we should not use 5-point likert scales: The case for subjective quality of life measurement. In Proceedings, second international conference on quality of life in cities, pages 74-93, 2000.
  35. D. R. Dalton, J. C. Wimbush, and C. M. Daily. Using the unmatched count technique (uct) to estimate base r. Personnel Psychology, 47(4):817, 1994.
  36. E. Deutskens, K. De Ruyter, M. Wetzels, and P. Oosterveld. Response rate and response quality of internet-based surveys: An experimental study. Marketing letters, 15(1):21-36, 2004.
  37. D. A. Dillman. Mail and telephone surveys, volume 3. Wiley Interscience, 1978.
  38. D. A. Dillman. Mail and Internet surveys: The tailored design method- 2007 Update with new Internet, visual, and mixed-mode guide. John Wiley & Sons, 2011.
  39. F. J. Drummond, L. Sharp, A.-E. Carsin, T. Kelleher, and H. Comber. Questionnaire order significantly increased response to a postal survey sent to primary care physicians. Journal of clinical epidemiology, 61(2):177-185, 2008.
  40. J. C. Duffy and J. J. Waterto. Under-reporting of alcohol consumption in sample surveys: The effect of computer interviewing in fieldwork. British journal of addiction, 79(4):303-308, 1984.
  41. S. Egelman and E. Peer. Scaling the security wall: Developing a security behavior intentions scale (sebis). In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 2873-2882. ACM, 2015.
  42. S. Fahl, M. Harbach, T. Muders, M. Smith, and U. Sander. Helping johnny 2.0 to encrypt his facebook conversations. In Proceedings of the Eighth Symposium on Usable Privacy and Security, page 11. ACM, 2012.
  43. M. Fendrich and C. M. Vaughn. Diminished lifetime substance use over time: An inquiry into differential underreporting. Public Opinion Quarterly, 58(1):96-123, 1994.
  44. J. Floyd J. Fowler. Survey Research Methods (4th ed.). SAGE Publications, Inc., 2009.
  45. B. Forsyth, J. M. Rothgeb, and G. B. Willis. Does pretesting make a difference? an experimental test. Methods for testing and evaluating survey questionnaires, pages 525-546, 2004.
  46. F. J. Fowler. Reducing interviewer-related error through interviewer training, supervision, and other means. Measurement errors in surveys, pages 259-278, 1991.
  47. F. J. Fowler. How unclear terms affect survey data. Public Opinion Quarterly, 56(2):218-231, 1992.
  48. A. Frick, M.-T. Bächtiger, and U.-D. Reips. Financial incentives, personal information and drop-out rate in online studies. Dimensions of Internet science, pages 209-219, 1999.
  49. E. T. Fuj, M. Hennessy, and J. Mak. An evaluation of the validity and reliability of survey response data on household electricity conserva- tion. Evaluation Review, 9(1):93-104, 1985.
  50. M. Galesic and R. Tourangeau. What is sexual harassment? it depends on who asks! framing effects on survey responses. Applied cognitive psychology, 21(2):189-202, 2007.
  51. M. Galesic, R. Tourangeau, M. P. Couper, and F. G. Conrad. Eye- tracking data new insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72(5):892- 913, 2008.
  52. R. Garland. The mid-point on a rating scale: Is it desirable. Marketing bulletin, 2(1):66-70, 1991.
  53. A. N. Glynn. What can we learn with statistical truth serum? design and analysis of the list experiment. Public Opinion Quarterly, 77(S1):159- 172, 2013.
  54. L. A. Goodman. Snowball sampling. The annals of mathematical statistics, pages 148-170, 1961.
  55. R. M. Groves, F. J. Fowler Jr, M. P. Couper, J. M. Lepkowski, E. Singer, and R. Tourangeau. Survey methodology, volume 561. John Wiley & Sons, 2009.
  56. E. Hargittai. Survey measures of web-oriented digital literacy. Social science computer review, 23(3):371-379, 2005.
  57. J. Hartley. Some thoughts on likert-type scales. International Journal of Clinical and Health Psychology, 14(1):83-86, 2014.
  58. S. Hatchett and H. Schuman. White respondents and race-of- interviewer effects. The Public Opinion Quarterly, 39(4):523-528, 1975.
  59. A. L. Holbrook, J. A. Krosnick, D. Moore, and R. Tourangeau. Response order effects in dichotomous categorical questions presented orally the impact of question and respondent attributes. Public Opinion Quarterly, 71(3):325-348, 2007.
  60. A. E. Howe, I. Ray, M. Roberts, M. Urbanska, and Z. Byrne. The psychology of security for the home computer user. In 2012 IEEE Symposium on Security and Privacy, pages 209-223. IEEE, 2012.
  61. A. Hughes, J. Chromy, K. Giacoletti, and D. Odom. Impact of inter- viewer experience on respondent reports of substance use. Redesigning an Ongoing National Household Survey, pages 161-84, 2002.
  62. M. Johnson, S. Egelman, and S. M. Bellovin. Facebook and privacy. In the Eighth Symposium, page 1, New York, New York, USA, 2012. ACM Press.
  63. G. Kalton and H. Schuman. The effect of the question on survey responses: A review. Journal of the Royal Statistical Society. Series A (General), pages 42-73, 1982.
  64. E. W. Kane and L. J. Macaulay. Interviewer gender and gender attitudes. Public opinion quarterly, 57(1):1-28, 1993.
  65. R. Kang, S. Brown, L. Dabbish, and S. B. Kiesler. Privacy attitudes of mechanical turk workers and the us public. In SOUPS, pages 37-49, 2014.
  66. K. Kelley. Good practice in the conduct and reporting of survey research. International Journal for Quality in Health Care, 15(3):261- 266, 2003.
  67. P. G. Kelley. Conducting usable privacy & security studies with amazons mechanical turk. In Symposium on Usable Privacy and Security (SOUPS)(Redmond, WA, 2010.
  68. B. Kitchenham and S. L. Pfleeger. Principles of survey research: part 5: populations and samples. ACM SIGSOFT Software Engineering Notes, 27(5):17-20, 2002.
  69. F. Kreuter, S. Presser, and R. Tourangeau. Social desirability bias in cati, ivr, and web surveys the effects of mode and question sensitivity. Public Opinion Quarterly, 72(5):847-865, 2008.
  70. J. A. Krosnick. Survey research. Annual review of psychology, 50(1):537-567, 1999.
  71. M. Krysan. Privacy and the expression of white racial attitudes: A comparison across three contexts. Public Opinion Quarterly, pages 506-544, 1998.
  72. P. Kumaraguru, S. Sheng, A. Acquisti, L. F. Cranor, and J. Hong. Teaching johnny not to fall for phish. ACM Trans. Internet Technol., 2010.
  73. R. N. Landers and T. S. Behrend. An inconvenient truth: Arbitrary distinctions between organizational, mechanical turk, and other conve- nience samples. Industrial and Organizational Psychology, 8(02):142- 164, 2015.
  74. J. Lazar, J. H. Feng, and H. Hochheiser. Research Methods in Human- Computer Interaction. Wiley Publishing, 2010.
  75. B. L. Leech. Asking questions: techniques for semistructured inter- views. Political Science & Politics, 35(04):665-668, 2002.
  76. E. Lin, S. Greenberg, E. Trotter, D. Ma, and J. Aycock. Does domain highlighting help people identify phishing sites? In CHI, 2011.
  77. D. Marques, I. Muslukhov, T. Guerreiro, L. Carric ¸o, and K. Beznosov. Snooping on mobile phones: Prevalence and trends. In Twelfth Symposium on Usable Privacy and Security (SOUPS 2016), 2016.
  78. P. V. Marsden and J. D. Wright, editors. Handbook of survey research. Emerald, Bingley, UK, 2 edition, 2010.
  79. M. N. Marshall. Sampling for qualitative research. Family practice, 13(6):522-526, 1996.
  80. E. Martin, T. J. DeMaio, and P. C. Campanelli. Context effects for census measures of race and hispanic origin. Public Opinion Quarterly, 54(4):551-566, 1990.
  81. N. R. McKenney and C. E. Bennett. Issues regarding data on race and ethnicity: the census bureau experience. Public health reports, 109(1):16, 1994.
  82. B. Mirel. Usability and hardcopy manuals: Evaluating research designs and methods. SIGDOC Asterisk J. Comput. Doc., 14(4):69-77, Sept. 1990.
  83. H. J. Parry and H. M. Crossley. Validity of responses to survey questions. Public Opinion Quarterly, 14(1):61-80, 1950.
  84. M. Q. Patton. Qualitative research. Wiley Online Library, 2005.
  85. S. Preibusch. Guide to measuring privacy concern: Review of survey and observational instruments. International Journal of Human- Computer Studies, 71(12):1133-1143, 2013.
  86. S. Presser and J. Blair. Survey pretesting: Do different methods produce different results. Sociological methodology, 24(1):73-104, 1994.
  87. S. Presser, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, J. M. Rothgeb, and E. Singer. Methods for testing and evaluating survey questions. Public opinion quarterly, 68(1):109-130, 2004.
  88. C. C. Preston and A. M. Colman. Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta psychologica, 104(1):1-15, 2000.
  89. E. M. Redmiles, S. Kross, and M. L. Mazurek. How i learned to be secure: a census-representative survey of security advice sources and behavior. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 666-677. ACM, 2016.
  90. M. T. Roberson and E. Sundstrom. Questionnaire design, return rates, and response favorableness in an employee attitude questionnaire. Journal of Applied Psychology, 75(3):354, 1990.
  91. S. E. Schechter, R. Dhamija, A. Ozment, and I. Fischer. The emperor's new security indicators. In 2007 IEEE Symposium on Security and Privacy (SP'07), pages 51-65. IEEE, 2007.
  92. B. Schneier. Security in the real world: How to evaluate security technology. Computer security journal, 15:1-14, 1999.
  93. E. M. Shaeffer, J. A. Krosnick, G. E. Langer, and D. M. Merkle. Comparing the quality of data obtained by minimally balanced and fully balanced attitude questions. Public Opinion Quarterly, 69(3):417- 428, 2005.
  94. G. M. Shapiro. Sample Size, pages 782-784. Sage Publications, Inc., 0 edition, 2008.
  95. K. B. Sheehan. E-mail survey response rates: A review. Journal of Computer-Mediated Communication, 6(2):0-0, 2001.
  96. S. Sheng, M. Holbrook, P. Kumaraguru, L. F. Cranor, and J. Downs. Who falls for phish?: A demographic analysis of phishing susceptibility and effectiveness of interventions. In CHI, 2010.
  97. D. K. Smetters and N. Good. How users use access control. In Proceedings of the 5th Symposium on Usable Privacy and Security, page 15. ACM, 2009.
  98. A. J. Søgaard, R. Selmer, E. Bjertness, and D. Thelle. The oslo health study: The impact of self-selection in a large, population-based survey. International journal for equity in health, 3(1):1, 2004.
  99. J. Staddon, D. Huffaker, L. Brown, and A. Sedley. Are privacy concerns a turn-off?: engagement and privacy in social networks. In Proceedings of the eighth symposium on usable privacy and security, page 10. ACM, 2012.
  100. J. V. Stulginskas, R. Verreault, and I. B. Pless. A comparison of observed and reported restraint use by children and adults. Accident Analysis & Prevention, 17(5):381-386, 1985.
  101. J. Tarnai and D. L. Moore. Measuring and improving telephone interviewer performance and productivity. De Leeuw, L. Japec, PJ Lavrakas, MW Link, and RL Sangster (Eds.), Advances in Telephone Survey Methodology, pages 359-384, 2008.
  102. R. Teclaw, M. C. Price, and K. Osatuke. Demographic question placement: Effect on item response rates and means of a veterans health administration survey. Journal of Business and Psychology, 27(3):281- 290, 2012.
  103. R. Tourangeau, L. J. Rips, and K. Rasinski. The psychology of survey response. Cambridge University Press, 2000.
  104. R. Tourangeau and T. W. Smith. Asking sensitive questions the impact of data collection mode, question format, and question context. Public opinion quarterly, 60(2):275-304, 1996.
  105. B. Ur, P. G. Kelley, S. Komanduri, J. Lee, M. Maass, M. L. Mazurek, T. Passaro, R. Shay, T. Vidas, L. Bauer, N. Christin, and L. F. Cranor. How does your password measure up? the effect of strength meters on password creation. In USENIX Sec., 2012.
  106. E. R. Van Teijlingen, A.-M. Rennie, V. Hundley, and W. Graham. The importance of conducting and reporting pilot studies: the example of the scottish births survey. Journal of advanced nursing, 34(3):289-295, 2001.
  107. S. L. Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309):63-69, 1965.
  108. R. Wash and E. Rader. Too much knowledge? security beliefs and protective behaviors among united states internet users. In Eleventh Symposium On Usable Privacy and Security (SOUPS 2015), pages 309-325, 2015.
  109. L.-J. Weng and C.-P. Cheng. Effects of response order on likert-type scales. Educational and psychological measurement, 60(6):908-924, 2000.
  110. B. E. Whitley, M. E. Kite, and H. L. Adams. Principles of research in behavioral science. Routledge, 2012.
  111. G. B. Willis. Cognitive interviewing: A tool for improving questionnaire design. Sage Publications, 2004.

Authors
Redmiles, E. M., Acar, Y., Fahl, S., & Mazurek, M. L.
ORCID
Venue
Digital Repository at the University of Maryland
Publisher
Digital Repository at the University of Maryland
Published
2017
DOI
10.13016/m22k2w
OpenAlex
W2621107674
Cited by
50
References
108
Citations by year
2025: 4 · 2024: 1 · 2023: 9 · 2022: 5 · 2021: 11 · 2020: 6 · 2019: 7 · 2018: 7
Type
report
Language
en
Open access
Locations
  • University Libraries (University of Maryland) (submittedVersion)
  • Digital Repository at the University of Maryland (University of Maryland College Park)
Topics
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
  • Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
Keywords
survey methodology, usability testing, self-reported data, human factors in security, user behavior, phishing, access control, research design guidelines, data quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review