CoChat
Sign Up
Cited by 4Survey Methodology and NonresponseData Quality and ManagementHuman Mobility and Location-Based Analysis

Expanding the methodological toolkit of criminology and criminal justice with the Total Error Framework

Open access
Snaphaan, Thom, Hardyns, Wim, Pauwels, Lieven J. R.·2024·Journal of Crime and Justice
DOI 10.1080/0735648x.2022.2114099
Open this paper in CoChat — free accountOpen PDF

Abstract

The availability and use of new and emerging data sources has increased exponentially. The variety of these data sources offers opportunities to complement, replace, improve or add to conventional data sources. Survey data is one kind of these conventional data sources. In survey research, a framework to assess the accuracy of survey data already has been around for quite some time, and goes by the name of the Total Survey Error (TSE) framework. The philosophy behind this framework has only recently been universalized to (big) data in general in the form of the Total Error Framework (TEF). The current study introduces the TEF to the methodological toolkit of scholars and practitioners in criminology and criminal justice by outlining this generic framework and applying it to an empirical case study (on calculating spatially-referenced crime rates) utilizing two types of administrative data and mobile phone data. The present study discusses the added value and limitations of adapting the TEF, providing guidance to apply the TEF in research and practice. Finally, we propose promising avenues for future inquiries.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1080/0735648x.2022.2114099

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The logic behind TSE has lately been expanded to various (large) data sources in general, which is why the total error framework (TEF) was established.

    “However, the logic behind TSE has lately been expanded to various (large) data sources in general, which is why the total error framework (TEF) was established (Amaya et al., 2020;Snaphaan et al., 2022).”

    From ‘Led by Intelligence': A Scoping Review on the Experimental Evaluation of Intelligence-Led Policing (2024) · cites this paper as Snaphaan et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 73

  1. Amaya, Ashley, Paul P. Biemer, and David Kinyon. 2020. "Total error in a big data world: Adapting the TSE framework to big data." Journal of Survey Statistics and Methodology 8: 89-119. https://doi.org/10.1093/jssam/smz056 · DOI
  2. Andresen, Martin A. 2006. "Crime measures and the spatial analysis of criminal activity." British Journal of Criminology 46: 258-285.
  3. Andresen, Martin A., and Greg W. Jenion. 2010. "Ambient populations and the calculation of crime rates and risk." Security Journal 23(2): 114-133. https://doi.org/10.1057/sj.2008. · DOI
  4. Andresen, Martin A., and Shannon J. Linning. 2012. "The (in)appropriateness of aggregating across crime types." Applied Geography 35 (1-2): 275-282. https://doi.org/10.1016/j.apgeog.2012.07.007 · DOI
  5. Andresen, Martin A., Nick Malleson, Wouter Steenbeek, Michael Townsley, and Christophe Vandeviver. 2020. "Minimum geocoding match rates: An international study of the impact of data and areal unit sizes." International Journal of Geographical Information Science 34 (7): 1306-1322. https://doi.org/10.1080/13658816.2020.1725015 · DOI
  6. Aneesh, A. 2009. "Global labor: Algocratic modes of organization." Sociological Theory 27(4): 347-370.
  7. Baker, Reginald P. 2017. "Big data: A survey research perspective." In Total Survey Error: Improving Quality in the Era of Big Data, edited by Paul P. Biemer, Edith De Leeuw, Stephanie Eckman, Brad Edwards, Frauke Kreuter, Lars Lyberg, Clyde Tucker, and Brady West, 47-70. Hoboken: Wiley.
  8. Bakker, Bart F. M. 2012. "Estimating the validity of administrative variables." Statistica Neerlandica 66(1): 8-17. https://doi.org/10.1111/j.1467-9574.2011.00504.x · DOI
  9. Biderman, Albert D., and James P. Lynch. 1991. Understanding crime statistics: Why the UCR diverges from the NCS. New York: Springer-Verlag.
  10. Biemer, Paul P. 2010. "Total survey error: Design, implementation, and evaluation." The Public Opinion Quarterly 74 (5): 817-848.
  11. Biemer, Paul P. 2016. "Errors and inference." In Big Data and Social Science: A Practical Guide to Methods and Tools, edited by Ian Foster, Rayid Ghani, Ron S. Jarmin, Frauke Kreuter, and Julia Lane, 265-297. Boca Raton, FL: CRC Press
  12. Biemer, Paul P., and Ashley Amaya. 2021. Total error frameworks for found data. In Big data meets survey science. A collection of innovative methods, edited by Craig A. Hill, Paul P. Biemer, Trent D. Buskirk, Lilli Japec, Antje Kirchner, Stas Kolenikov, and Lars E. Lyberg, 133-161. Hoboken: John Wiley & Sons.
  13. Biemer, Paul P., and Lars E. Lyberg. 2003. Introduction to survey quality. Hoboken: Wiley.
  14. Biemer, Paul P., and Lars E. Lyberg. 2020. Total Survey Error. In SAGE Research Methods Foundations, edited by Paul Atkinson, Sara Delamont, Alexandru Cernat, Joseph W. Sakshaug, and Richard A. Williams. https://www.doi.org/10.4135/9781526421036888444 · DOI
  15. Boggs, Sarah L. 1965. "Urban crime patterns." American Sociological Review 30(6): 899-908.
  16. Bottomley, A. Keith, and Ken Pease. 1986. Crime and punishment: Interpreting the data. Milton Keynes: Open University Press.
  17. Braga, Anthony A., and Ronald V. Clarke. 2014. "Explaining high-risk concentrations of crime in the city: Social disorganization, crime opportunities, and important next steps." Journal of Research in Crime and Delinquency 51: 480-498. https://doi.org/10.1177/0022427814521217 · DOI
  18. Buil-Gil, David, Angelo Moretti, Samuel H. Langton. 2021. "The accuracy of crime statistics: Assessing the impact of police data bias on geographic crime analysis." Journal of Experimental Criminology. https://doi.org/10.1007/s11292-021-09457-y · DOI
  19. Bunge, Mario. 2006. Chasing reality. Strife over realism. Toronto: University of Toronto Press.
  20. Chen, Yifei, Yuenan Li, and Jonathan Li. 2016. "Investigating the influence of tree coverage on property crime: A case study in the city of Vancouver, British Columbia, Canada." The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B2: 695-702. https://doi.org/10.5194/isprs-archives-XLI-B2- 695-2016 · DOI
  21. Cochran, William G. 1953. Sampling techniques. New York: Wiley.
  22. De Kimpe, Lies, Michel Walrave, Thom Snaphaan, Lieven Pauwels, Wim Hardyns, and Koen Ponnet. 2021. "Research Note: An investigation of cybercrime victims' reporting behavior." European Journal of Crime, Criminal Law, and Criminal Justice 29(1): 66- 78. https://doi.org/10.1163/15718174-bja10019 · DOI
  23. De Leeuw, Edith D., and Wim de Heer. 2002. Trends in household survey nonresponse: A longitudinal and international comparison. In Survey nonresponse, edited by Robert M. Groves, Don A. Dillman, John L. Eltinge, and Roderick J. A. Little, 41-54. New York: Wiley.
  24. Deming, William E. 1944. "On errors in surveys." American Sociological Review 9 (4): 359- 369.
  25. Favaretto, Maddalena, Eva De Clercq, and Bernice S. Elger. 2019. "Big Data and discrimination: Perils, promises and solutions. A systematic review." Journal of Big Data 6: Article 12. https://doi.org/10.1186/s40537-019-0177-4 · DOI
  26. Groves, Robert M. 1989. Survey errors and survey costs. New York: Wiley.
  27. Groves, Robert M., and Lars Lyberg. 2010. "Total Survey Error: Past, present, and future." Public Opinion Quarterly 74 (5): 849-879. https://doi.org/10.1093/poq/nfq065 · DOI
  28. Groves, Robert M., Floyd J. Fowler Jr., Mick P. Couper, Eleanor Singer, and Roger Tourangeau. 2004. Survey methodology. Hoboken: Wiley.
  29. Groves, Robert M., Floyd J. Fowler Jr., Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2009. Survey methodology (2nd ed.). Hoboken: Wiley.
  30. Hansen, Morris H., William N. Hurwitz, and William G. Madow. 1953. Sample survey methods and theory. 2 vols. New York: Wiley.
  31. Hardyns, Wim, Thom Snaphaan, and Lieven Pauwels. 2019. "Crime concentrations and micro places: An empirical test of the 'law of crime concentration at places' in Belgium." Australian and New Zealand Journal of Criminology 52(3): 390-410. https://doi.org/10.1177/0004865818807243 · DOI
  32. Harries, Keith D. 1981. "Alternative denominators in conventional crime rates." In Environmental criminology, edited by Paul J. Brantingham, and Patricia L. Brantingham, 147-165. Prospect Heights: Waveland Press.
  33. Hilbert, Martin. 2016. "Big data for development: A review of promises and challenges." Development Policy Review 34(1): 135-174. https://doi.org/10.1111/dpr.12142 · DOI · Paper page
  34. Hox, Joop J., Edith D. de Leeuw, and Don A. Dillman. 2008. The cornerstones of survey research. In International handbook of survey research, edited by Edith D. de Leeuw, Joop J. Hox and Don A. Dillman, 1-17. New York: Taylor & Francis Group.
  35. Hsieh, Y. Patrick, and Joe Murphy. 2017. Total Twitter error: Decomposing public opinion measurement on Twitter from a total survey error perspective. In Total Survey Error in Practice, edited by Paul P. Biemer, Edith D. de Leeuw, Stephanie Eckman, Brad Edwards, Frauke Kreuter, Lars E. Lyberg, N. Clyde Tucker and Brady T. West, 23-46. Hoboken: Wiley.
  36. Japec, Lilli, and Lars Lyberg. 2021. Big data initiatives in official statistics. In Big data meets survey science. A collection of innovative methods, edited by Craig A. Hill, Paul P. Biemer, Trent D. Buskirk, Lilli Japec, Antje Kirchner, Stas Kolenikov, and Lars E. Lyberg, 275-302. Hoboken: John Wiley & Sons.
  37. Japec, Lilli, Frauke Kreuter, Marcus Berg, Paul Biemer, Paul Decker, Cliff Lampe, Julia Lane, Cathy O'Neil, and Abe Usher. 2015. "Big data in survey research: AAPOR task force report." Public Opinion Quarterly 79 (4): 839-880. https://doi.org/10.1093/poq/nfv039 · DOI
  38. Johnson, Timothy P., and Tom W. Smith. 2017. Big data and survey research: Supplement or substitute? In Seeing cities through big data: Research, methods and applications in urban informatics, edited by Piyushimita V. Thakuriah, Nebiyou Tilahun, and Moira Zellner, 113-126. Cham: Springer.
  39. Karr, Alan F., Ashish P. Sanil, and David L. Banks. 2006. "Data quality: A statistical perspective." Statistical Methodology 3(2): 137-173. https://doi.org/10.1016/j.stamet.2005.08.005 · DOI
  40. Keusch, Florian, Sebastian Bähr, Georg-Christoph Haas, Frauke Kreuter, and Mark Trappmann. 2020. "Coverage error in data collection combining surveys with passive measurement using apps: Data from a German national survey." Sociological Methods & Research 1-38. https://doi.org/10.1177/0049124120914924 · DOI
  41. Kirkpatrick, Keith. 2017. "It's not the algorithms, it's the data." Communications of the ACM 60 (2): 21-23. https://doi.org/10.1145/3022181 · DOI
  42. Kish, Leslie. 1965. Survey sampling. New York: Wiley.
  43. Kitchin, Rob, and Gavin McArdle. 2016. "What makes big data, big data? Exploring the ontological characteristics of 26 datasets." Big Data & Society, 3(1): 1-10. https://doi.org/10.1177/2053951716631130 · DOI · Paper page
  44. Kitchin, Rob. 2014. "Big data, new epistemologies and paradigm shifts." Big Data & Society 1(1): 1-12. https://doi.org/10.1177/2053951714528481 · DOI · Paper page
  45. Kitchin, Rob. 2015. "The opportunities, challenges and risks of big data for official statistics." Statistical Journal of the IAOS 31 (3): 471-481.
  46. Klinger, David A., and George S. Bridges. 1997. "Measurement error in calls-for-service as an indicator of crime." Criminology 35 (4): 705-726.
  47. Kreuter, Frauke. 2021. "What surveys really say." Nature 600: 614-615. https://doi.org/10.1038/d41586-021-03604-1 · DOI
  48. Lavrakas, Paul J. 2013. "Applying a total error perspective for improving research quality in the social, behavioral, and marketing sciences." Public Opinion Quarterly 77 (3): 831- 850.
  49. Liao, Dan, Marcus E. Berzofsky, G. Lance Couzens, Ian Thomas, and Alexia Cooper. 2021. Improving quality of administrative data: A case study with FBI's national incident- based reporting system data. In Big data meets survey science. A collection of innovative methods, edited by Craig A. Hill, Paul P. Biemer, Trent D. Buskirk, Lilli Japec, Antje Kirchner, Stas Kolenikov, and Lars E. Lyberg, 217-244. Hoboken: John Wiley & Sons.
  50. Little, Roderick J. A., and Donald B. Rubin. 2002. Statistical analysis with missing data. 2nd ed. Hoboken: Wiley.
  51. Lynch, James P., and John P. Jarvis. 2008. "Missing data and imputation in the Uniform Crime Reports and the effects on national estimates." Journal of Contemporary Criminal Justice 24 (1): 69-85. https://doi.org/10.1177/1043986207313028 · DOI
  52. Lynch, James P., and Lynn A. Addington, eds. 2007. Understanding crime statistics: Revisiting the divergence of the NCVS and UCS. Cambridge: Cambridge University Press.
  53. Malleson, Nick, and Martin A. Andresen. 2015. "The impact of using social media data in crime rate calculations: Shifting hot spots and changing spatial patterns." Cartography and Geographic Information Science 42(2): 112-121. https://doi.org/10.1080/15230406.2014.905756 · DOI
  54. Malleson, Nick, and Martin A. Andresen. 2016. "Exploring the impact of ambient population measures on London crime hotspots." Journal of Criminal Justice 46: 42-63. https://doi.org/10.1016/j.jcrimjus.2016.03.002 · DOI
  55. Neyman, Jerzy. 1934. "On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection." Journal of the Royal Statistical Society 97(4): 558-625.
  56. Peeters, Rik, and Marc Schuilenburg. 2020. The algorithmic society: An introduction. In The algorithmic society, edited by Mark Schuilenburg and Rik Peeters, 1-15. New York: Routledge.
  57. Ratcliffe, Jerry H. 2021. "Policing and public health calls for service in Philadelphia." Crime Science 10: Article 5. https://doi.org/10.1186/s40163-021-00141-0 · DOI
  58. Reid, Gill, Felipa Zabala, and Anders Holmberg. 2017. "Extending TSE to administrative data: A quality framework and case studies from Stats NZ." Journal of Official Statistics 33 (2): 477-511. https://doi.org/10.1515/jos-2017-0023 · DOI
  59. Richardson, Rashida, Jason Schultz, and Kate Crawford. 2019. "Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice." New York University Law Review 94: 192-233.
  60. Robinson, William S. 1950. "Ecological correlations and the behavior of individuals." American Sociological Review 15 (3): 351-357.
  61. Rummens, Anneleen, Thom Snaphaan, Nico Van de Weghe, Dirk Van den Poel, Lieven Pauwels, and Wim Hardyns. 2021. "Do mobile phone data provide a better denominator in crime rates and improve spatiotemporal predictions of crime?" ISPRS International Journal of Geo-Information 10(6): Article 369. https://doi.org/10.3390/ijgi10060369 · DOI
  62. Saris, Willem E. 2014. Total survey error. In Encyclopedia of quality of life and well-being research, edited by Alex C. Michalos. Springer. https://doi.org/10.1007/978-94-007- 0753-5_3034 · DOI
  63. Biemer, Robert M. Groves, Lars E. Lyberg, Nancy A. Mathiowetz, and Seymour Sudman, 575-597. New York: Wiley.
  64. Schober, Michael F., Josh Pasek, Lauren Guggenheim,Cliff Lampe, and Frederick G. Conrad. 2016. "Social media analyses for social measurement." Public Opinion Quarterly 80 (1): 180-211. https://doi.org/10.1093/poq/nfv048 · DOI
  65. Skogan, Wesley G. 1977. "Dimensions of the dark figure of unreported crime." Crime & Delinquency 23 (1): 41-50. https://doi.org/10.1177/001112877702300104 · DOI · Paper page
  66. Snaphaan, Thom, and Wim Hardyns. 2021. "Environmental criminology in the big data era." European Journal of Criminology 18(5): 713-734. https://doi.org/10.1177/1477370819877753 Statistics New Zealand. 2016. "Guide on reporting on administrative data quality." Accessed 26 March 2021. https://www.stats.govt.nz/methods/guide-to-reporting-on- administrative-data-quality · DOI
  67. Thakuriah, Piyushimita V., Nebiyou Tilahun, and Moira Zellner. 2017. Big data and urban informatics: Innovations and challenges to urban planning and knowledge discovery. In Seeing cities through big data: Research, methods and applications in urban informatics, edited by Piyushimita V. Thakuriah, Nebiyou Tilahun, and Moira Zellner, 11-45. Cham: Springer.
  68. Van de Weijer, Steve G. A., Rutger Leukfeldt, and Wim Bernasco. 2019. "Determinants of reporting cybercrime: A comparison between identity theft, consumer fraud, and hacking. European Journal of Criminology 16(4): 486-508. https://doi.org/10.1177/1477370818773610 · DOI
  69. Wallgren, Anders, and Britt Wallgren. 2007. Register-based statistics: Administrative data for statistical purposes. Chichester: Wiley.
  70. Warner, Barbara D., and Glenn L. Pierce. (1993). "Reexamining social disorganization theory using calls to the police as a measure of crime." Criminology 31 (4): 493-517.
  71. Wikström, Per-Olof H. 1991. Urban crime, criminals, and victims. The Swedish experience in an Anglo-American comparative perspective. New York: Springer.
  72. Wikström, Per-Olof H., and Clemens Kroneberg. 2022. "Analytic criminology: Mechanisms and methods in the explanation of crime and its causes." Annual Review of Criminology 5: 179-203. https://doi.org/10.1146/annurev-criminol-030920-091320 · DOI
  73. Zhang, Li-Chun. 2012. "Topics of statistical theory for register-based statistics and data integration." Statistica Neerlandica 66 (1): 41-63. https://doi.org/10.1111/j.1467- 9574.2011.00508. · DOI

Cited by

Sorted by citations

Authors
Snaphaan, T., Hardyns, W., & Pauwels, L. J. R.
Affiliations
  • Ghent University (BE)
  • University of Antwerp (BE)
Corresponding author
Snaphaan, Thom
ORCID
Venue
Journal of Crime and Justice
Publisher
Informa UK Limited
Published
2022-08-23
Volume
47
Issue
1
Pages
112-129
ISSN
  • 0735-648X (ISSN-L)
  • 2158-9119
DOI
10.1080/0735648x.2022.2114099
OpenAlex
W4292723274
Cited by
4
References
0
Citations by year
2025: 1 · 2024: 2 · 2023: 1
FWCI
1.44
Type
journal article
Language
en
Open access
Licence
other-oa
Locations
  • Journal of Crime and Justice (publishedVersion)
  • Ghent University Academic Bibliography (Ghent University) (submittedVersion, other-oa)
Topics
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
  • Social Sciences › Transportation › Human Mobility and Location-Based Analysis
Keywords
total survey error, survey accuracy, big data, administrative data, mobile phone data, crime measurement, data quality assessment, criminological methodology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review