Statistical Power in Experimental Audit Studies: Cautions and Calculations for Matched Tests With Nominal Outcomes
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Given their capacity to identify causal relationships, experimental audit studies have grown increasingly popular in the social sciences. Typically, investigators send fictitious auditors who differ by a key factor (e.g., race) to particular experimental units (e.g., employers) and then compare treatment and control groups on a dichotomous outcome (e.g., hiring). In such scenarios, an important design consideration is the power to detect a certain magnitude difference between the groups. But power calculations are not straightforward in standard matched tests for dichotomous outcomes. Given the paired nature of the data, the number of pairs in the concordant cells (when neither or both auditor receives a positive response) contributes to the power, which is lower as the sum of the discordant proportions approaches one. Because these quantities are difficult to determine a priori, researchers must exercise particular care in experimental design. We here present sample size and power calculations for McNemar’s test using empirical data from an audit study on misdemeanor arrest records and employability. We then provide formulas and examples for cases involving more than two treatments (Cochran’s Q test) and nominal outcomes (Stuart–Maxwell test). We conclude with concrete recommendations concerning power and sample size for researchers designing and presenting matched audit studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
Pager, D. (2003) · American Journal of Sociology · doi:10.1086/374403 · Cited by 2,840
Getting a Job: Is There a Motherhood Penalty?
Correll, S. J., Benard, S., & Paik, I. (2007) · American Journal of Sociology · doi:10.1086/511799 · Cited by 2,525 · cited 5×
“…in which the experimental unit is contained within a mutually exclusive category of a confounding variable that cannot be randomly assigned (e.g., employers are not contained within race in Pager 2003 or within gender in Correll et al. 2007; cf. employers contained within states in Tilcsik 2011).”
BackgroundMethodInfluentialThe Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis
Hoenig, J. M., & Heisey, D. M. (2001) · The American Statistician · doi:10.1198/000313001300339897 · Cited by 2,046
THE COMPARISON OF PERCENTAGES IN MATCHED SAMPLES
COCHRAN, W. G. (1950) · Biometrika · doi:10.1093/biomet/37.3-4.256 · Cited by 1,465
Discrimination in a Low-Wage Labor Market: A Field Experiment
Pager, D., Bonikowski, B., & Western, B. (2009) · American Sociological Review · doi:10.1177/000312240907400505 · Cited by 1,022
Field Experiments of Discrimination in the Market Place
Riach, P. A., & Rich, J. (2002) · The Economic Journal · doi:10.1111/1468-0297.00080 · Cited by 911 · cited 1×
“Unfortunately, many empirical articles do not present results in a paired format such as those in Table 6 (see Riach and Rich 2002 for a summary of economics articles in this format), but rather only present the marginals.”
BackgroundHeckman, J. J. (1998) · Journal of Economic Perspectives · doi:10.1257/jep.12.2.101 · Cited by 824
Pride and Prejudice: Employment Discrimination against Openly Gay Men in the United States
Tilcsik, A. (2011) · American Journal of Sociology · doi:10.1086/661653 · Cited by 687
Cited by
THE EDGE OF STIGMA: AN EXPERIMENTAL AUDIT OF THE EFFECTS OF LOW‐LEVEL CRIMINAL RECORDS ON EMPLOYMENT
UGGEN, C., VUOLO, M., LAGESON, S., RUHLAND, E., & K. WHITHAM, H. (2014) · Criminology · doi:10.1111/1745-9125.12051 · Cited by 367
Self-citationHeath, A. F., & Di Stasio, V. (2019) · The British Journal of Sociology · doi:10.1111/1468-4446.12676 · Cited by 167
An Introduction to Audit Studies in the Social Sciences
Gaddis, S. M. (2018) · Audit Studies: Behind the Scenes with Theory, Method, and Nuance · doi:10.1007/978-3-319-71153-9_1 · Cited by 157
Di Stasio, V., Lancee, B., Veit, S., & Yemane, R. (2021) · Journal of Ethnic and Migration Studies · doi:10.1080/1369183x.2019.1622826 · Cited by 148
Lancee, B. (2021) · Journal of Ethnic and Migration Studies · doi:10.1080/1369183x.2019.1622744 · Cited by 90
Metadata
- Authors
- Vuolo, M., Uggen, C., & Lageson, S.
- Venue
- Sociological Methods & Research
- Publisher
- SAGE Publications
- Published
- 2015-02-16
- Volume
- 45
- Issue
- 2
- Pages
- 260-303
- ISSN
- 0049-1241 (ISSN-L)
- 1552-8294
- OpenAlex
- W2027200274
- Cited by
- 47
- References
- 18
- FWCI
- 9.97
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistical Power in Experimental Audit Studies: Cautions and Calculations for Matched Tests With Nominal Outcomes. https://app.cochat.ai/papers/10.1177/0049124115570066
- The paper
- Vuolo, M., Uggen, C., & Lageson, S. (2016). Statistical Power in Experimental Audit Studies: Cautions and Calculations for Matched Tests With Nominal Outcomes. Sociological Methods & Research. https://doi.org/10.1177/0049124115570066
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.