Survey research methods in evaluation and case-control studies
Abstract
Survey research methods are widely used in two types of analytic studies: evaluation studies that measure the effects of interventions; and population‐based case–control studies that investigate the effects of various risk factors on the presence of disease. This paper provides a broad overview of some design and analysis issues related to such studies, illustrated with examples. The lack of random assignment to treatment and control groups in many evaluation studies makes controlling for confounders critically important. Confounder control can be achieved by matching in the design and by various alternative methods in the analysis. One popular analytic method of controlling for confounders is propensity scoring, which bears a close resemblance to survey weighting. The use of population‐based controls has become common in case–control studies. For reasons of cost, population‐based controls are often identified by telephone surveys using random digit dialling (RDD) sampling methods. However, RDD surveys are now experiencing serious problems with response rates. A recent alternative approach is to select controls from frames such as driver license lists that contain valuable demographic information for use in matching. Methods of analysis developed in the survey sampling literature are applicable, at least to some degree, in the analyses of evaluation and population‐based case–control studies. In particular, the effects of complex sample designs can be taken into account using survey sampling variance estimation methods. Several survey analysis software packages are available for carrying out the computations. Copyright © 2007 John Wiley & Sons, Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Survey research methods with inverse participation probability weights were used to fit logistic regression models.
“To ensure that the results represented the study base, we used survey research methods to fi t the logistic regression models, with participants being assigned weights equal to the inverse of the estimated participation probability in strata de fi ned by the entry case group and demographic factors (38).”
From Maternal Intake of Supplemental Iron and Risk of Autism Spectrum Disorder (2014) · cites this paper as [38]
Cites
(2002) · Cited by 13,211 · see citing works
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230 · see citing works
Reducing Bias in Observational Studies Using Subclassification on the Propensity Score
Rosenbaum, P. R., & Rubin, D. B. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478078 · Cited by 3,082 · see citing works
Rosenbaum, P. R. (2002) · Springer series in statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,414 · see citing works
Evaluation: A Systematic Approach
Wallace, D. P. (2005) · Library & Information Science Research · doi:10.1016/j.lisr.2004.09.009 · Cited by 1,938 · see citing works
Sampling Methods for Random Digit Dialing
Waksberg, J. (1978) · Journal of the American Statistical Association · doi:10.1080/01621459.1978.10479995 · Cited by 1,386 · see citing works
Introduction to Variance Estimation.
King, B. F., & Wolter, K. M. (1987) · Journal of the American Statistical Association · doi:10.2307/2289497 · Cited by 1,377 · see citing works
Design and Analysis of Cluster Randomization Trials in Health Research
DONNER, A. P., & Klar, N. S. (2000) · Cited by 1,236 · see citing works
Design and Analysis of Group-Randomized Trials
Sonnad, S. S. (1999) · Medicine & Science in Sports & Exercise · doi:10.1097/00005768-199902000-00025 · Cited by 1,130 · see citing works
Invited commentary: propensity scores.
Joffe, M., & Rosenbaum, P. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a010011 · Cited by 1,117 · see citing works
Korn, E. L., & Graubard, B. I. (1999) · Wiley eBooks · doi:10.1002/9781118032619 · Cited by 1,052 · see citing works
Selection of Controls in Case-Control Studies
Wacholder, S., McLaughlin, J. K., Silverman, D. T., & Mandel, J. S. (1992) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116396 · Cited by 998 · see citing works
Introduction to Variance Estimation
Hidiroglou, M. A., & Wolter, K. M. (1986) · Technometrics · doi:10.2307/1269086 · Cited by 820 · see citing works
(2002) · Routledge eBooks · doi:10.4324/9781410603029 · Cited by 728 · see citing works
Design and analysis of group-randomized trials: a review of recent methodological developments
Murray, D. M., Varnell, S. P., & Blitstein, J. L. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.423 · Cited by 677 · see citing works
Variance estimation for complex surveys using replication techniques
Rust, K., & Rao, J. (1996) · Statistical Methods in Medical Research · doi:10.1177/096228029600500305 · Cited by 662 · see citing works
Selection of Controls in Case-Control Studies: II. Types of Controls
Wacholder, S., Silverman, D. T., McLaughlin, J. K., & Mandel, J. S. (1992) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116397 · Cited by 626 · see citing works
Selection of Controls in Case-Control Studies: III. Design Options
Wacholder, S., Silverman, D. T., McLaughlin, J. K., & Mandel, J. S. (1992) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116398 · Cited by 444 · see citing works
Pitfalls of and controversies in cluster randomization trials
Donner, A., & Klar, N. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.416 · Cited by 442 · see citing works
Mason, S. M., Knowles, E., Colwell, B., Dixon, S., Wardrope, J., Gorringe, R., Snooks, H. A., Perrin, J., & Nicholl, J. P. (2007) · BMJ · doi:10.1136/bmj.39343.649097.55 · Cited by 378 · see citing works
Statistical Design for Research
Spaeth, J. L., & Kish, L. (1990) · Journal of Educational Statistics · doi:10.2307/1165035 · Cited by 272 · see citing works
RANDOM DIGIT DIALING IN SELECTING A POPULATION-BASED CONTROL GROUP
Hartge, P. A., Brinton, L. A., Rosenthal, J. F., CAHILL, J. I., Hoover, R. N., & Waksberg, J. (1984) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a113955 · Cited by 250 · see citing works
Randomized Experiments for Planning and Evaluation
Boruch, R. F. (1997) · Sage eBooks · doi:10.4135/9781412985574 · Cited by 241 · see citing works
Statistical Design for Research
Kish, L. (1987) · Wiley eBooks · doi:10.1002/0471725196 · Cited by 199 · see citing works
Koepsell, T. D., Wagner, E. H., Cheadle, A. D., Patrick, D. L., Martin, D. C., Diehr, P. H., Perrin, E. B., Kristal, A. R., Allan-Andrilla, C. H., & Dey, L. J. (1992) · Annual Review of Public Health · doi:10.1146/annurev.pu.13.050192.000335 · Cited by 193 · see citing works
Oral Contraceptives and Breast Cancer Risk Among Younger Women
Brinton, L. A., Daling, J. R., Liff, J. M., Schoenberg, J. B., Malone, K. E., Stanford, J. L., Coates, R., Gammon, M. D., Hanson, L., & Hoover, R. N. (1995) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/87.11.827 · Cited by 183 · see citing works
Planning and Analysis of Observational Studies
Cochran, W. G. (1983) · Wiley series in probability and statistics · doi:10.1002/9780470316542 · Cited by 168 · see citing works
Randomized Experiments for Planning and Evaluation: A Practical Guide
Kirk, R. E., & Boruch, R. F. (1998) · Journal of the American Statistical Association · doi:10.2307/2670135 · Cited by 147 · see citing works
Bias In List-Assisted Telephone Samples
Brick, J. M., Waksberg, J., Kulp, D., & Starer, A. (1995) · Public Opinion Quarterly · doi:10.1086/269470 · Cited by 137 · see citing works
Community-based Health Intervention Trials: An Overview of Methodological Issues
Atienza, A. A. (2002) · Epidemiologic Reviews · doi:10.1093/epirev/24.1.72 · Cited by 115 · see citing works
Evaluation of the American Stop Smoking Intervention Study (ASSIST): A Report of Outcomes
Stillman, F. A. (2003) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djg098 · Cited by 106 · see citing works
Innovative strategies using SUDAAN for analysis of health surveys with complex samples
LaVange, L. M., Steams, S. C., Lafata, J. E., Koch, G. G., & Shah, B. V. (1996) · Statistical Methods in Medical Research · doi:10.1177/096228029600500306 · Cited by 93 · see citing works
Evaluating AIDS prevention programs
(1991) · Choice Reviews Online · doi:10.5860/choice.29-2140 · Cited by 80 · see citing works
Why have Educational Evaluators Chosen Not to Do Randomized Experiments?
Cook, T. D. (2003) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716203254764 · Cited by 77 · see citing works
(1974) · Elsevier eBooks · doi:10.1016/c2013-0-11369-4 · Cited by 53 · see citing works
Comparison of Telephone Sampling and Area Sampling: Response Rates and Within-Household Coverage
Brogan, D. J. (2001) · American Journal of Epidemiology · doi:10.1093/aje/153.11.1119 · Cited by 51 · see citing works
On the Robustness of Weighted Methods for Fitting Models to Case–Control Edata
Scott, A., & Wild, C. J. (2002) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/1467-9868.00333 · Cited by 45 · see citing works
Evaluation of the DC Opportunity Scholarship Program: First Year Report on Participation
Wolf, P. J., Gutmann, B., Eissa, N., & Puma, M. J. (2005) · Journal of the Arkansas Academy of Science · Cited by 31 · see citing works
National Evaluation of GEAR UP: A Summary of the First Two Years.
Muraskin, L. D. (2003) · Cited by 28 · see citing works
Case–Control Studies with Complex Sampling
Scott, A., & Wild, C. J. (2001) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/1467-9876.00242 · Cited by 24 · see citing works
Commentary: Trade-offs in the Development of a Sample Design for Case-Control Studies
DiGaetano, R., & Waksberg, J. (2002) · American Journal of Epidemiology · doi:10.1093/aje/155.8.771 · Cited by 13 · see citing works
Fitting Logistic Regression Models in Case–Control Studies with Complex Sampling
Scott, A., & Wild, C. J. (2003) · doi:10.1002/0470867205.ch8 · Cited by 5 · see citing works
Flagged first, then by citations
2 of the 44 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Prenatal vitamins, one-carbon metabolism gene variants, and risk for autism
Schmidt, R. J., Hansen, R. L., Hartiala, J., Allayee, H., Schmidt, L. C., Tancredi, D. J., Tassone, F., & Hertz-Picciotto, I. (2011) · Epidemiology · doi:10.1097/ede.0b013e31821d0e30 · Cited by 308 · see citing works
Maternal Intake of Supplemental Iron and Risk of Autism Spectrum Disorder
Schmidt, R. J., Tancredi, D. J., Krakowiak, P., Hansen, R. L., & Ozonoff, S. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu208 · Cited by 184 · see citing works
Bandera, E. V., Chandran, U., Zirpoli, G., McCann, S. E., Ciupak, G. L., & Ambrosone, C. B. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-71 · Cited by 68 · see citing works
App-based rehabilitation program after total knee arthroplasty: a randomized controlled trial
Bäcker, H. C., Wu, C. H., Schulz, M. R. G., Weber-Spickschen, T. S., Perka, C., & Hardt, S. (2021) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-021-03789-0 · Cited by 66 · see citing works
Worksite Study Promoting Activity and Changes in Eating (PACE): Design and Baseline Results
Beresford, S. A. A., Locke, E. R., Bishop, S. K., West, B., McGregor, B. A., Bruemmer, B., Duncan, G. E., & Thompson, B. (2007) · Obesity · doi:10.1038/oby.2007.383 · Cited by 50 · see citing works
Water consumption habits of a south-western Ontario community
Pintar, K. D. M., Waltner‐Toews, D., Charron, D. F., Pollari, F. L., Fazil, A., McEwen, S. A., Nesbitt, A., & Majowicz, S. E. (2009) · Journal of Water and Health · doi:10.2166/wh.2009.038 · Cited by 48 · see citing works
Beyond the question “Does it pay to be green?”: How much green? and when?
Riillo, C. A. F. (2016) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2016.09.039 · Cited by 48 · see citing works
Medicare Spending for Previously Uninsured Adults
McWilliams, J. M., Meara, E., Zaslavsky, A. M., & Ayanian, J. Z. (2009) · Annals of Internal Medicine · doi:10.7326/0000605-200912010-00149 · Cited by 46 · see citing works
Weighting Methods for Population-Based Case–Control Studies with Complex Sampling
Li, Y., Graubard, B. I., & DiGaetano, R. (2010) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/j.1467-9876.2010.00731.x · Cited by 36 · see citing works
Drinking water consumption patterns in Canadian communities (2001–2007)
Roche, S. M., Jones, A. Q., Majowicz, S. E., McEwen, S. A., & Pintar, K. D. M. (2011) · Journal of Water and Health · doi:10.2166/wh.2011.051 · Cited by 34 · see citing works
Validity of Random-Digit-Dialing in Recruiting Controls in a Case-Control Study
Wang, P. (2009) · American Journal of Health Behavior · doi:10.5993/ajhb.33.5.4 · Cited by 32 · see citing works
Clagett, B., Nathanson, K. L., Ciosek, S. L., McDermoth, M., Vaughn, D. J., Mitra, N., Weiss, A., Martonik, R., & Kanetsky, P. A. (2013) · American Journal of Epidemiology · doi:10.1093/aje/kwt164 · Cited by 30 · see citing works
Caffeine-Containing Medicines Increase the Risk of Hemorrhagic Stroke
Lee, S.-M., Choi, N., Lee, B., Cho, K., Yoon, B., & Park, B. (2013) · Stroke · doi:10.1161/strokeaha.111.674077 · Cited by 25 · see citing works
The Impact of Medicaid Peer Support Utilization on Cost
Landers, G., & Zhou, M. (2014) · Medicare & Medicaid Research Review · doi:10.5600/mmrr.004.01.a04 · Cited by 23 · see citing works
for the French PTA Study Group, Lepelletier, D., Pinaud, V., Conte, P. L., Bourigault, C., Asseray, N., Ballereau, F., Caillon, J., Ferron, C., Righini, C. A., et al. (2016) · European Journal of Clinical Microbiology & Infectious Diseases · doi:10.1007/s10096-016-2770-1 · Cited by 16 · see citing works
Walsh, M. C., Trentham‐Dietz, A., Gangnon, R. E., Nieto, F. J., Newcomb, P. A., & Palta, M. (2012) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-11-1066 · Cited by 16 · see citing works
Origins and Development of Health Survey Methods
Johnson, T. P. (2014) · doi:10.1002/9781118594629.ch1 · Cited by 16 · see citing works
An outbreak of hepatitis A in Canada: The use of a control bank to conduct a case-control study
Smith, C. R., Kershaw, T., Johnson, K. O., & Meghnath, K. (2019) · Epidemiology and Infection · doi:10.1017/s0950268819001870 · Cited by 13 · see citing works
Are higher serum IgE concentrations associated with adult cardiovascular disease?
Shiue, I. (2013) · International Journal of Cardiology · doi:10.1016/j.ijcard.2013.01.050 · Cited by 10 · see citing works
Li, Y., & Graubard, B. I. (2012) · Biostatistics · doi:10.1093/biostatistics/kxs008 · Cited by 9 · see citing works
Efficient analysis of case‐control studies with sample weights
Landsman, V., & Graubard, B. I. (2012) · Statistics in Medicine · doi:10.1002/sim.5530 · Cited by 7 · see citing works
Book publishing in Australia: the potential impact of digital technologies on business models
Tian, X. (2011) · RMIT Research Repository (RMIT University Library) · doi:10.25439/rmt.27354594 · Cited by 7 · see citing works
Li, C. X., Matthay, E. C., Rowe, C. L., Bradshaw, P. T., & Ahern, J. (2021) · Annals of Epidemiology · doi:10.1016/j.annepidem.2021.06.019 · Cited by 4 · see citing works
Measuring health in population surveys
Kalton, G. (2012) · Statistical Journal of the IAOS · doi:10.3233/sji-2012-0741 · Cited by 2 · see citing works
Self-citationRiillo, C. A. F., & Jakobs, K. (2022) · The Journal of Technology Transfer · doi:10.1007/s10961-022-09951-z · Cited by 2 · see citing works
Model choices for complex survey analysis
Riyapan, P. (2016) · Research Online (University of Wollongong)
- Authors
- Kalton, G., & Piesse, A.
- Affiliations
- Westat (United States) (US)
- Corresponding author
- Kalton, Graham
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2007-02-05
- Volume
- 26
- Issue
- 8
- Pages
- 1675-1687
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W2113961526
- Cited by
- 27
- References
- 44
- Citations by year
- 2022: 1 · 2021: 2 · 2019: 1 · 2016: 3 · 2014: 4 · 2013: 4 · 2012: 4
- FWCI
- 1.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Statistics in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- survey methodology, evaluation research, case-control study, confounder control, matching, propensity scoring, survey weighting, population-based controls, response rate, complex sampling designs
- MeSH terms
- Adult, Aged, Data Collection, Data Interpretation, Statistical, Evaluation Studies as Topic, Female, Humans, Male, Middle Aged, United States, Bias, Case-Control Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review