Statistical Power in Experimental Audit Studies: Cautions and Calculations for Matched Tests With Nominal Outcomes
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Given their capacity to identify causal relationships, experimental audit studies have grown increasingly popular in the social sciences. Typically, investigators send fictitious auditors who differ by a key factor (e.g., race) to particular experimental units (e.g., employers) and then compare treatment and control groups on a dichotomous outcome (e.g., hiring). In such scenarios, an important design consideration is the power to detect a certain magnitude difference between the groups. But power calculations are not straightforward in standard matched tests for dichotomous outcomes. Given the paired nature of the data, the number of pairs in the concordant cells (when neither or both auditor receives a positive response) contributes to the power, which is lower as the sum of the discordant proportions approaches one. Because these quantities are difficult to determine a priori, researchers must exercise particular care in experimental design. We here present sample size and power calculations for McNemar’s test using empirical data from an audit study on misdemeanor arrest records and employability. We then provide formulas and examples for cases involving more than two treatments (Cochran’s Q test) and nominal outcomes (Stuart–Maxwell test). We conclude with concrete recommendations concerning power and sample size for researchers designing and presenting matched audit studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Pager, D. (2003) · American Journal of Sociology · doi:10.1086/374403 · Cited by 2,840
Getting a Job: Is There a Motherhood Penalty?
Correll, S. J., Benard, S., & Paik, I. (2007) · American Journal of Sociology · doi:10.1086/511799 · Cited by 2,525 · cited 5×
“…in which the experimental unit is contained within a mutually exclusive category of a confounding variable that cannot be randomly assigned (e.g., employers are not contained within race in Pager 2003 or within gender in Correll et al. 2007; cf. employers contained within states in Tilcsik 2011).”
BackgroundMethodInfluentialThe Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
THE COMPARISON OF PERCENTAGES IN MATCHED SAMPLES
COCHRAN, W. G. (1950) · Biometrika · doi:10.1093/biomet/37.3-4.256 · Cited by 1,465
Discrimination in a Low-Wage Labor Market: A Field Experiment
Pager, D., Bonikowski, B., & Western, B. (2009) · American Sociological Review · doi:10.1177/000312240907400505 · Cited by 1,022
Field Experiments of Discrimination in the Market Place
Riach, P. A., & Rich, J. (2002) · The Economic Journal · doi:10.1111/1468-0297.00080 · Cited by 911 · cited 1×
“Unfortunately, many empirical articles do not present results in a paired format such as those in Table 6 (see Riach and Rich 2002 for a summary of economics articles in this format), but rather only present the marginals.”
BackgroundHeckman, J. J. (1998) · Journal of Economic Perspectives · doi:10.1257/jep.12.2.101 · Cited by 824
Pride and Prejudice: Employment Discrimination against Openly Gay Men in the United States
Tilcsik, A. (2011) · American Journal of Sociology · doi:10.1086/661653 · Cited by 687
Walking the Talk? What Employers Say Versus What They Do
Pager, D., & Quillian, L. (2005) · American Sociological Review · doi:10.1177/000312240507000301 · Cited by 640
Pager, D., Western, B., & Sugie, N. (2009) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716208330793 · Cited by 411
THE EDGE OF STIGMA: AN EXPERIMENTAL AUDIT OF THE EFFECTS OF LOW‐LEVEL CRIMINAL RECORDS ON EMPLOYMENT
UGGEN, C., VUOLO, M., LAGESON, S., RUHLAND, E., & K. WHITHAM, H. (2014) · Criminology · doi:10.1111/1745-9125.12051 · Cited by 367 · cited 4×
“We demonstrate this exercise with an empirical example (described in full in Uggen et al. 2014).”
BackgroundMethodResultInfluentialSelf-citationSexual orientation discrimination in the labour market
Drydakis, N. (2009) · Labour Economics · doi:10.1016/j.labeco.2008.12.003 · Cited by 322
Measuring Employment Discrimination through Controlled Experiments
Bendick, M., Jackson, C. W., & Reinoso, V. A. (1994) · The Review of Black Political Economy · doi:10.1007/bf02895739 · Cited by 173
No Foot in the Door: An Experimental Study of Employment Discrimination Against Older Workers
Bendick, M., Brown, L. E., & Wall, K. (1999) · Journal of Aging & Social Policy · doi:10.1300/j031v10n04_02 · Cited by 159
Discrimination against latino job applicants: A controlled experiment
Bendick, M., Jackson, C. W., Reinoso, V. A., & Hodges, L. E. (1991) · Human Resource Management · doi:10.1002/hrm.3930300404 · Cited by 80
On the sample size for studies based upon mcnemar's test
Lachenbruch, P. A. (1992) · Statistics in Medicine · doi:10.1002/sim.4780111110 · Cited by 47
Cited by
THE EDGE OF STIGMA: AN EXPERIMENTAL AUDIT OF THE EFFECTS OF LOW‐LEVEL CRIMINAL RECORDS ON EMPLOYMENT
UGGEN, C., VUOLO, M., LAGESON, S., RUHLAND, E., & K. WHITHAM, H. (2014) · Criminology · doi:10.1111/1745-9125.12051 · Cited by 367
Self-citationHeath, A. F., & Di Stasio, V. (2019) · The British Journal of Sociology · doi:10.1111/1468-4446.12676 · Cited by 167
An Introduction to Audit Studies in the Social Sciences
Gaddis, S. M. (2018) · Audit Studies: Behind the Scenes with Theory, Method, and Nuance · doi:10.1007/978-3-319-71153-9_1 · Cited by 157
Di Stasio, V., Lancee, B., Veit, S., & Yemane, R. (2021) · Journal of Ethnic and Migration Studies · doi:10.1080/1369183x.2019.1622826 · Cited by 148
Lancee, B. (2021) · Journal of Ethnic and Migration Studies · doi:10.1080/1369183x.2019.1622744 · Cited by 90
Metadata
- Authors
- Vuolo, M., Uggen, C., & Lageson, S.
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2015-02-16
- Volume
- 45
- Issue
- 2
- Pages
- 260-303
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W2027200274
- Cited by
- 47
- References
- 18
- FWCI
- 9.97
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.