Methods and Metrics for Evaluating Analytic Insider Threat Tools
Open accessAbstract
The insider threat is a prime security concern for government and industry organizations. As insider threat programs come into operational practice, there is a continuing need to assess the effectiveness of tools, methods, and data sources, which enables continual process improvement. This is particularly challenging in operational environments, where the actual number of malicious insiders in a study sample is not known. The present paper addresses the design of evaluation strategies and associated measures of effectiveness; several quantitative/statistical significance test approaches are described with examples, and a new measure, the Enrichment Ratio, is proposed and described as a means of assessing the impact of proposed tools on the organization's operations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1966) · Medical Entomology and Zoology · Cited by 10,293 · see citing works
Signal detection theory and psychophysics
Ingleby, D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,616 · see citing works
On the generalized distance in statistics
Mahalanobis, P. C. (1936) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 6,068 · see citing works
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · see citing works
Elementary Signal Detection Theory
Wickens, T. D. (2001) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195092509.001.0001 · Cited by 1,276 · see citing works
State transition analysis: a rule-based intrusion detection approach
Ilgun, K., Kemmerer, R. A., & Porras, P. A. (1995) · IEEE Transactions on Software Engineering · doi:10.1109/32.372146 · Cited by 753 · see citing works
Some Tests of Significance, Treated by the Theory of Probability
Jeffreys, H. (1935) · Mathematical Proceedings of the Cambridge Philosophical Society · doi:10.1017/s030500410001330x · Cited by 442 · see citing works
A framework for understanding and predicting insider attacks
Schultz, E. (2002) · Computers & Security · doi:10.1016/s0167-4048(02)01009-x · Cited by 325 · see citing works
Lenca, P., Meyer, P. E., Vaillant, B., et al. (2007) · European Journal of Operational Research · doi:10.1016/j.ejor.2006.10.059 · Cited by 206 · see citing works
Greitzer, F. L., & Frincke, D. A. (2010) · Advances in information security · doi:10.1007/978-1-4419-7133-3_5 · Cited by 133 · see citing works
Self-citationConfidence Ratings, Message Reception, and the Receiver Operating Characteristic
Pollack, I. W., & Decker, L. R. (1958) · The Journal of the Acoustical Society of America · doi:10.1121/1.1909571 · Cited by 83 · see citing works
Psychosocial Modeling of Insider Threat Risk Based on Behavioral and Word Use Analysis
Greitzer, Kangas, Noonan, Brown, & Ferryman (2013) · e-Service Journal · doi:10.2979/eservicej.9.1.106 · Cited by 47 · see citing works
Shaw, E. D., & Fischer, L. F. (2005) · doi:10.21236/ada441293 · Cited by 45 · see citing works
Identifying at-risk employees: A behavioral model for predicting potential insider threats
Greitzer, F. L., Kangas, L. J., Noonan, C. F., & Dalton, A. C. (2010) · doi:10.2172/1000159 · Cited by 45 · see citing works
Self-citationPredicting Insider Threat Risks through Linguistic Analysis of Electronic Communication
Brown, C. R., Watkins, A., & Greitzer, F. L. (2013) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2013.453 · Cited by 40 · see citing works
Self-citationSoft computing in intrusion detection: the state of the art
Langin, C., & Rahimi, S. (2010) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-010-0012-4 · Cited by 37 · see citing works
Social/Ethical Issues in Predictive Insider Threat Monitoring
Greitzer, F. L., Frincke, D. A., & Zabriskie, M. (2013) · IGI Global eBooks · doi:10.4018/978-1-61350-323-2.ch506 · Cited by 25 · see citing works
Self-citationInsider Threat Program Best Practices
Guido, M., & Brooks, M. W. (2013) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2013.279 · Cited by 21 · see citing works
Employee Monitoring and Ethics
Kiser, A. I. T., Porter, T., & Vequist, D. G., IV (2010) · International Journal of Digital Literacy and Digital Competence · doi:10.4018/jdldc.2010100104 · Cited by 18 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [1]1. D. Capelli, A. Moore and R. Trzeciak, The CERT Guide to Insider Threats, Upper Saddle River, NJ: Addison-Wesley, 2012.
- [12]12. D. Vysochanskij and Y. Petunin, "Justification of the 3σ rule for unimodal distributions.," Theory of Probability and Mathematical Statistics, vol. 21, pp. 25-36, 1980.
Cited by
Sorted by citations
Alzaabi, F. R., & Mehmood, A. (2024) · IEEE Access · doi:10.1109/access.2024.3369906 · Cited by 204 · see citing works
“TP represents the correct identification of genuine insider threats, while TN corresponds to accurately classifying benign activities as non-threats [89].”
BackgroundClassification of Insider Threat Detection Techniques
Sanzgiri, A., & Dasgupta, D. (2016) · Proceedings of the 11th Annual Cyber and Information Security Research Conference · doi:10.1145/2897795.2897799 · Cited by 68 · see citing works
TWOS: A Dataset of Malicious Insider Threat Behavior Based on a Gamified Competition
Harilal, A., Toffalini, F., Castellanos, J., Guarnizo, J., Homoliak, I., & Ochoa, M. (2017) · Proceedings of the 2017 International Workshop on Managing Insider Security Threats · doi:10.1145/3139923.3139929 · Cited by 53 · see citing works
- Authors
- Greitzer, F. L., & Ferryman, T. A.
- ORCID
- Greitzer, Frank L.
- Venue
- 2013 IEEE Security and Privacy Workshops
- Publisher
- IEEE
- Published
- 2013-05
- Pages
- 90-97
- OpenAlex
- W2034247102
- Cited by
- 22
- References
- 19
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 1 · 2019: 1 · 2018: 3 · 2017: 2 · 2016: 8 · 2015: 1 · 2014: 1 · 2013: 1
- FWCI
- 2.95
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- IEEE Security and Privacy Workshops (SPW) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- insider threats, tool evaluation, evaluation metrics, enrichment ratio, statistical significance testing, malicious insiders, effectiveness measurement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
