Incentive and Reminder Strategies to Improve Response Rate for Internet-Based Physician Surveys: A Randomized Experiment
Open accessAbstract
BACKGROUND: Most research on how to enhance response rates in physician surveys has been done using paper surveys. Uncertainties remain regarding how to enhance response rates in Internet-based surveys. OBJECTIVE: To evaluate the impact of a low-cost nonmonetary incentive and paper mail reminders (formal letter and postcard) on response rates in Internet-based physician surveys. METHODS: We executed a factorial-design randomized experiment while conducting a nationally representative Internet-based physician survey. We invited 3966 physicians (randomly selected from a commercial database of all licensed US physicians) via email to complete an Internet-based survey. We used 2 randomly assigned email messages: one message offered a book upon survey completion, whereas the other did not mention the book but was otherwise identical. All nonrespondents received several email reminders. Some physicians were further assigned at random to receive 1 reminder via paper mail (either a postcard or a letter) or no paper reminder. The primary outcome of this study was the survey response rate. RESULTS: Of the 3966 physicians who were invited, 451 (11.4%) responded to at least one survey question and 336 (8.5%) completed the entire survey. Of those who were offered a book, 345/2973 (11.6%) responded compared with 106/993 (10.7%) who were not offered a book (odds ratio 1.10, 95% CI 0.87-1.38, P=.42). Regarding the paper mail reminder, 168/1572 (10.7%) letter recipients, 148/1561 (9.5%) postcard recipients, and 69/767 (9.0%) email-only recipients responded (P=.35). The response rate for those receiving letters or postcards was similar (odds ratio 1.14, 95% CI 0.91-1.44, P=.26). CONCLUSIONS: Offering a modest nonmonetary incentive and sending a paper reminder did not improve survey response rate. Further research on how to enhance response rates in Internet-based physician surveys is needed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The survey response rate was 29.2%, in line with reported response rates of internet or phone-based surveys.
“However, the survey response rate was 29.2%, which is in line with reported response rates of the internet or phone-based surveys [57].”
From Life Experiences in Neuromuscular Tracheotomized Patients in Times of Covid-19 (2023) · cites this paper as [57]
The Discussion (Limitations; page unavailable) reports that response rates below 20% are not uncommon in Internet surveys of physicians, lending some support to the comparison with Internet surveys. It gives no comparable rate for phone surveys and does not establish that 29.2% is typical of either mode. The 29.2% appears to be the citing authors’ own survey result, not a result attributed to this source. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Response rates to mail surveys published in medical journals
Asch, D. A., Jedrziewski, M. K., & Christakis, N. A. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00126-1 · Cited by 2,110 · see citing works
A guide for the design and conduct of self-administered surveys of clinicians
Burns, K. E. A., Duffett, M., Kho, M. E., Meade, M., Adhikari, N. K. J., Sinuff, T., Cook, D. J., & for the ACCADEMY Group (2008) · Canadian Medical Association Journal · doi:10.1503/cmaj.080372 · Cited by 1,357 · see citing works
Self-citationPhysician response to surveys A review of the literature
Kellerman, S. (2001) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(00)00258-0 · Cited by 956 · see citing works
Response Rates and Nonresponse Errors in Surveys
Johnson, T. P. (2012) · JAMA · doi:10.1001/jama.2012.3532 · Cited by 701 · see citing works
Methodologies for Improving Response Rates in Surveys of Physicians
VanGeest, J. B., Johnson, T. P., & Welch, V. L. L. (2007) · Evaluation & the Health Professions · doi:10.1177/0163278707307899 · Cited by 697 · see citing works
Enhancing Surveys of Health Care Professionals
Cho, Y. I., Johnson, T. P., & VanGeest, J. B. (2013) · Evaluation & the Health Professions · doi:10.1177/0163278713496425 · Cited by 489 · see citing works
Using the Internet to conduct surveys of health professionals: a valid alternative?
Braithwaite, D., Emery, J. D., de Lusignan, S., & Sutton, S. R. (2003) · Family Practice · doi:10.1093/fampra/cmg509 · Cited by 475 · see citing works
Methods for the Design and Administration of Web-based Surveys
Schleyer, T., & Forrest, J. L. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070416 · Cited by 306 · see citing works
Comparison of E-mail, Fax, and Postal Surveys of Pediatricians
McMahon, S. R., Iwamoto, M., Massoudi, M. S., Yusuf, H. R., Stevenson, J. M., David, F., Chu, S. Y., & Pickering, L. K. (2003) · PEDIATRICS · doi:10.1542/peds.111.4.e299 · Cited by 284 · see citing works
Problems in recruiting community-based physicians for health services research
Asch, S. M., Connor, S. E., Hamilton, E. G., & Fox, S. A. (2000) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2000.02329.x · Cited by 234 · see citing works
Dykema, J., Jones, N. R., Piché, T., & Stevenson, J. F. (2013) · Evaluation & the Health Professions · doi:10.1177/0163278713496630 · Cited by 231 · see citing works
Nonresponse Rates are a Problematic Indicator of Nonresponse Bias in Survey Research
Davern, M. J. (2013) · Health Services Research · doi:10.1111/1475-6773.12070 · Cited by 219 · see citing works
Internet Versus Mailed Questionnaires: A Controlled Comparison (2)
Leece, P., Bhandari, M., Sprague, S., Swiontkowski, M. F., Schemitsch, E. H., Tornetta, P., Devereaux, P. J., & Guyatt, G. H. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.4.e39 · Cited by 216 · see citing works
How to obtain excellent response rates when surveying physicians
Thorpe, C. F., Ryan, B. L., McLean, S., et al. (2008) · Family Practice · doi:10.1093/fampra/cmn097 · Cited by 204 · see citing works
Health Care Provider Surveys in the United States, 2000–2010
McLeod, C. C., Klabunde, C. N., Willis, G., & Stark, D. S. (2013) · Evaluation & the Health Professions · doi:10.1177/0163278712474001 · Cited by 190 · see citing works
Field, T. S., Cadoret, C. A., Brown, M. L., Ford, M. E., Greene, S. M., Hill, D. L., Hornbrook, M. C., Meenan, R. T., White, M. J., & Zapka, J. (2002) · Medical Care · doi:10.1097/00005650-200207000-00006 · Cited by 173 · see citing works
Mixing Web and Mail Methods in a Survey of Physicians
Beebe, T. J., Locke, G. R., Barnes, S. A., Davern, M. E., & Anderson, K. J. (2006) · Health Services Research · doi:10.1111/j.1475-6773.2006.00652.x · Cited by 132 · see citing works
Self-citationElectronic mail was not better than postal mail for surveying residents and faculty
Akl, E. A., Maroun, N., Klocke, R. A., et al. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.006 · Cited by 119 · see citing works
E-mail Versus Conventional Postal Mail Survey of Geriatric Chiefs
Raziano, D. B., Jayadevappa, R., Valenzula, D., Weiner, M. G., & Lavizzo‐Mourey, R. J. (2001) · The Gerontologist · doi:10.1093/geront/41.6.799 · Cited by 115 · see citing works
James, K. M., Ziegenfuss, J. Y., Tilburt, J. C., Harris, A. M., & Beebe, T. J. (2010) · Health Services Research · doi:10.1111/j.1475-6773.2010.01181.x · Cited by 107 · see citing works
Self-citationImproving the Quality of Surveys of Physicians and Medical Groups
Klabunde, C. N., Willis, G., McLeod, C. C., Dillman, D. A., Johnson, T. P., Greene, S. M., & Brown, M. L. (2012) · Evaluation & the Health Professions · doi:10.1177/0163278712458283 · Cited by 93 · see citing works
Ensuring Physicians' Competence — Is Maintenance of Certification the Answer?
Iglehart, J. K., & Baron, R. B. (2012) · New England Journal of Medicine · doi:10.1056/nejmhpr1211043 · Cited by 91 · see citing works
E-mail or snail mail? Randomized controlled trial on which works better for surveys.
Seguin, R., Godwin, M. S., Macdonald, S., et al. (2004) · PubMed · Cited by 82 · see citing works
Getting Maintenance of Certification to Work
Cook, D. A., Holmboe, E. S., Sorensen, K. J., et al. (2014) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2014.5437 · Cited by 72 · see citing works
Self-citationDykema, J., Stevenson, J. F., Day, B. F., Sellers, S. L., & Bonham, V. L. (2011) · Evaluation & the Health Professions · doi:10.1177/0163278711406113 · Cited by 70 · see citing works
Enhancing Response Rates in Physician Surveys: The Limited Utility of Electronic Options
Nicholls, K. W., Chapman, K., Shaw, T., Perkins, A., Sullivan, M. M., Crutchfield, S., & Reed, E. (2011) · Health Services Research · doi:10.1111/j.1475-6773.2011.01261.x · Cited by 61 · see citing works
Crouch, S. R., Robinson, P. M., & Pitts, M. K. (2011) · Australian and New Zealand Journal of Public Health · doi:10.1111/j.1753-6405.2011.00687.x · Cited by 60 · see citing works
Reforming the Continuing Medical Education System
Nissen, S. E. (2015) · JAMA · doi:10.1001/jama.2015.4138 · Cited by 58 · see citing works
Facilitators and Barriers to Survey Participation by Physicians
Klabunde, C. N., Willis, G., & Casalino, L. P. (2013) · Evaluation & the Health Professions · doi:10.1177/0163278713496426 · Cited by 56 · see citing works
Kerkhof, E. G. V. D., Parlow, J. L., Goldstein, D. H., & Milne, B. (2004) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/bf03018307 · Cited by 54 · see citing works
Leung, G. M., Johnston, J. M., Saing, H., et al. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.12.021 · Cited by 50 · see citing works
Göritz, A. S. (2010) · American Psychological Association eBooks · doi:10.1037/12076-014 · Cited by 49 · see citing works
Do Additional Recontacts to Increase Response Rate Improve Physician Survey Data Quality?
Willis, G., Smith, T. G., & Lee, H. J. (2013) · Medical Care · doi:10.1097/mlr.0b013e3182a5023d · Cited by 43 · see citing works
Viera, A. J., & Edwards, T. (2012) · BMC Research Notes · doi:10.1186/1756-0500-5-129 · Cited by 25 · see citing works
Differential effects of pre and post-payment on neurologists' response rates to a postal survey
Kanaan, R., Wessely, S. C., & Armstrong, D. (2010) · BMC Neurology · doi:10.1186/1471-2377-10-100 · Cited by 13 · see citing works
Minniear, T. D., McIntosh, E. B., Alexander, N. E., Weidle, P. J., & Fulton, J. P. (2013) · Annals of Epidemiology · doi:10.1016/j.annepidem.2013.05.003 · Cited by 6 · see citing works
Automatic Mass Detection in Ultrasound Breast ImagesUsing Computer Assisted Detection
Khatib, F. (2012) · Medical Informatics · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [23]Thomas M Rogers R Maclean R Collecting data from physicians via web-based surveys: recommendations for improving response rates The Internet Journal of Medical Informatics 2002 1 1 1 6
- [25]VanGeest J Beebe T Johnson T Johnson T Surveys of Physicians Handbook of Survey Methods 2014 Hoboken, NJ John Wiley & Sons, Inc 515 543
- [33]Committee on Planning a Continuing Health Care Professional Education Institute Board on Health Care Services Institute of Medicine Redesigning Continuing Education in the Health Professions 2010 Washington, DC The National Academies Press
- [37]Göritz AS Incentives in Web Studies: Methodological Issues and a Review International Journal of Internet Science 2006 1 58 70
Cited by
Sorted by citations
Pierucci, P., Crimi, C., Carlucci, A., Carpagnano, G. E., Janssens, J.-P., Lujan, M., Noto, A., Wijkstra, P. J., Windisch, W., & Scala, R. (2021) · ERJ Open Research · doi:10.1183/23120541.00911-2020 · Cited by 34 · see citing works
Cross, A. S., Helen Kemp, E., White, A., Walker, L., Meredith, S., Sachdev, P., Krone, N. P., Ross, R. J., Wright, N. P., & Elder, C. J. (2018) · Clinical Endocrinology · doi:10.1111/cen.13559 · Cited by 34 · see citing works
Madhun, N. Z., Galindo, R. J., Donato, J., Hwang, P. R., Shabir, H. F., Fowler, M. J., Molitch-Hou, E., Bena, J. F., Umpierrez, G. E., & Lansang, M. C. (2023) · Diabetes Technology & Therapeutics · doi:10.1089/dia.2022.0226 · Cited by 9 · see citing works
Life Experiences in Neuromuscular Tracheotomized Patients in Times of Covid-19
Volpato, E., Pierucci, P., De Candia, M. L., Casparrini, M., Volpi, V., Pagnini, F., Carpagnano, G. E., & Banfi, P. (2023) · Journal of Neuromuscular Diseases · doi:10.3233/jnd-221597 · Cited by 2 · see citing works
“2%, which is in line with reported response rates of the internet or phone-based surveys [57].”
Result
- Authors
- Cook, D. A., Wittich, C. M., Daniels, W. L., West, C. P., Harris, A. M., & Beebe, T. J.
- Affiliations
- Mayo Clinic (US)
- Corresponding author
- Cook, David A
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2016-09-16
- Volume
- 18
- Issue
- 9
- Pages
- e244
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC5045523
- OpenAlex
- W2519880746
- Cited by
- 203
- References
- 37
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 14 · 2023: 21 · 2022: 31 · 2021: 34 · 2020: 25 · 2019: 20 · 2018: 19 · 2017: 13 · 2016: 1 · 2015: 1
- FWCI
- 50.38
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- survey response rates, internet-based survey, physician survey, non-monetary incentives, email reminders, survey nonresponse
- MeSH terms
- Adult, Age Factors, Aged, Female, Humans, Male, Middle Aged, Motivation, Physicians, Surveys and Questionnaires, Research Design, Reminder Systems, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
