Evaluating the Effectiveness of a Police Early Intervention System: From the Predictive Validity of Officer Identification to the Impact of Intervention
The DOI matches its Crossref record; 35 of the 37 references it cites were found in a registry or an index.
Abstract
As a non-punitive approach to addressing and minimizing officer misconduct, early intervention systems (EISs) have become a best practice in policing. These systems focus on identifying officers at risk of future problematic behavior and providing effective early interventions. Despite the widespread use of EISs, limited research has evaluated these systems, and the evaluations that do exist often concentrate solely on either the selection of officers for intervention or the effects of interventions, which restricts our understanding of the overall effectiveness of EISs. Instead of concentrating on evaluating only a part of an EIS, we explore both the implementation and effectiveness of the Phoenix Police Department (PPD) EIS at two different stages. Utilizing data from over 2000 officers employed by the PPD between 2016 and 2020, we use receiver operating characteristic (ROC) curves to analyze the accuracy of officer selection for EIS involvement and a population-averaged interrupted time series model to determine whether EIS implementation is associated with a reduction in problematic officer behavior. Our findings indicated that the indicators and thresholds used by PPD to identify at-risk officers resulted in low predictive validity (none greater than 4%). Furthermore, establishing the EIS was not significantly associated with a decrease in problematic officer behavior. We conclude with a discussion of research and policy implications and future directions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
Generalized Latent Variable Modeling
Skrondal, A., & Rabe-Hesketh, S. (2004) · Chapman & Hall/CRC eBooks · doi:10.1201/9780203489437 · Cited by 1,484
The effectiveness of early childhood development programs
Anderson, L. M., Shinn, C., Fullilove, M. T., Scrimshaw, S. C., Fielding, J. E., Normand, J., & Carande-Kulis, V. G. (2003) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(02)00655-4 · Cited by 614
metandi: Meta-analysis of diagnostic accuracy using hierarchical logistic regression
Harbord, R. M., & Whiting, P. (2009) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x0900900203 · Cited by 453 · cited 1×
“The model also provides the summary parameter estimates for all 37 rolling windows regarding sensitivity and speci fi city, 95% con fi dence intervals (CI), and standard errors (Harbord & Whiting, 2009).”
Organizational Justice and Police Misconduct
Wolfe, S. E., & Piquero, A. R. (2011) · Criminal Justice and Behavior · doi:10.1177/0093854810397739 · Cited by 345
A Survey of Methods for Analyzing Clustered Binary Response Data
Pendergast, J. F., Gange, S. J., Newton, M. A., Lindstrom, M. J., Palta, M., & Fisher, M. R. (1996) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1403425 · Cited by 270
Jackson, D., Law, M., Rücker, G., & Schwarzer, G. (2017) · Statistics in Medicine · doi:10.1002/sim.7411 · Cited by 249
The New World of Police Accountability
Walker, S. E., & Archbold, C. A. (2020) · Sage eBooks · doi:10.4135/9781544339214 · Cited by 174
The Network Structure of Police Misconduct
Wood, G., Roithmayr, D., & Papachristos, A. V. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119879798 · Cited by 74
Early Warning Systems: Responding to the Problem Police Officer
Walker, S., Alpert, G. P., & Kenney, D. J. (2001) · Cited by 72
Research works printed but not linked: 11
- Benchmark Analytics. (2024). Identifying officers at risk of misconduct: How Benchmark analytics supports law enforcement agencies. Presented at the London Data Science Expo. https://www.benchmarkanalytics.com/wp-content/uploads/2024/04/2024-03-19-London-Data-Science-Presentation-Final.pdf
- Bobb M. J., Barge M., Mazar Y., Naguib C., Shugrue T. (2009). Los Angeles county sheriff’s department: 27th semiannual report. Police Assessment Resource Center. https://static1.squarespace.com/static/5498b74ce4b01fe317ef2575/t/54fc7497e4b
- Boston Globe. (1992). Wave of abuse claims laid to a few officers.
- Davis R. C., Henderson N. J., Mandelstam J., Ortiz C. W., Miller J. (2005). Federal intervention in local policing: Pittsburgh’s experience with a consent decree. Office of Community Oriented Policing Services. https://www.hsdl.org/?view&did5463256
- Davis R. C., Ortiz C. W., Henderson N. J., Miller J., Massie M. K. (2002). Turning necessity into virtue: Pittsburgh’s experience with a federal consent decree. Vera Institute of Justice. https://www.vera.org/downloads/Publications/turning-necessities-intovirtue-pittsburghs-experience-with-a-federal-consent-decree/legacy_downloads/Pittsburgh_consent_decree.pdf
- Fyfe J. J., Kane R. (2006). Bad cops: A study of career-ending misconduct among New York city police officers. John Jay College of Criminal Justice.
- Gullion C. L. (2022). Early intervention systems: An exploration of interventions handled by supervisors to address at-risk officer behavior (Publication No. 29605463) [Doctoral dissertation, Sam Huston State University]. ProQuest Dissertations & Theses Global.
- Hickey K. (2009). Report on an inquiry into early intervention systems in the new south wales police force. New South Wales Parliament, Committee on the Office of the Ombudsman and the Police Integrity Commission.
- Katz C. M., Walker S. (2004). An evaluation of Phoenix police department’s personnel assessment system training program. University of Nebraska at Omaha.
- Phoenix Police Department. (n.d.). Early intervention referral template.
- U.S. Census. (2020). Population estimates, July 1, 2019. U.S. Department of Commerce.
Cited by
A Conditional Ordinal Stereotype Model to Estimate Police Officers’ Propensity to Escalate Force
Ridgeway, G. K. (2026) · Journal of the American Statistical Association · doi:10.1080/01621459.2025.2597050 · Cited by 1
The Promises and Pitfalls of Modern Early Intervention Systems for Police Misconduct
Cubitt, T. (2026) · Springer eBooks · doi:10.1007/978-3-032-14070-8_5 · Cited by 1
Emerging Analytical Approaches and New Directions for Police Misconduct Research
Cubitt, T. (2026) · Springer eBooks · doi:10.1007/978-3-032-14070-8_7
What’s in a name? A policy review of the many approaches to police early intervention systems
Carleton, B., & Christoff, T. (2026) · Policing An International Journal · doi:10.1108/pijpsm-03-2026-0081
Metadata
- Authors
- Katz, C. M., Cheon, H., Freemon, K., & Wallace, D.
- Affiliations
- Arizona State University (US)
- The University of Texas at El Paso (US)
- DePaul University (US)
- Corresponding author
- Katz, Charles M.
- Venue
- Police Quarterly
- Publisher
- SAGE Publications
- Published
- 2025-06-22
- Volume
- 29
- Issue
- 1
- Pages
- 6-39
- ISSN
- 1098-6111 (ISSN-L)
- 1552-745X
- OpenAlex
- W4411532461
- Cited by
- 4
- References
- 37
- Citations by year
- 2026: 3 · 2025: 1
- FWCI
- 14.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Police Quarterly (publishedVersion)
- Funders
- The City of Phoenix (153669-0)
- The Watts Family
- Topics
- Social Sciences › Political Science and International Relations › Policing Practices and Perceptions
- Social Sciences › Sociology and Political Science › Crime Patterns and Interventions
- Social Sciences › Education › Education Discipline and Inequality
- Keywords
- officer misconduct, predictive validity, intervention effectiveness, police accountability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating the Effectiveness of a Police Early Intervention System: From the Predictive Validity of Officer Identification to the Impact of Intervention. https://app.cochat.ai/papers/10.1177/10986111251353487
- The paper
- Katz, C. M., Cheon, H., Freemon, K., & Wallace, D. (2026). Evaluating the Effectiveness of a Police Early Intervention System: From the Predictive Validity of Officer Identification to the Impact of Intervention. Police Quarterly. https://doi.org/10.1177/10986111251353487
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.