An explainable and efficient deep learning framework for video anomaly detection
Open accessAbstract
Deep learning-based video anomaly detection methods have drawn significant attention in the past few years due to their superior performance. However, almost all the leading methods for video anomaly detection rely on large-scale training datasets with long training times. As a result, many real-world video analysis tasks are still not applicable for fast deployment. On the other hand, the leading methods cannot provide interpretability due to the uninterpretable feature representations hiding the decision-making process when anomaly detection models are considered as a black box. However, the interpretability for anomaly detection is crucial since the corresponding response to the anomalies in the video is determined by their severity and nature. To tackle these problems, this paper proposes an efficient deep learning framework for video anomaly detection and provides explanations. The proposed framework uses pre-trained deep models to extract high-level concept and context features for training denoising autoencoder (DAE), requiring little training time (i.e., within 10 s on UCSD Pedestrian datasets) while achieving comparable detection performance to the leading methods. Furthermore, this framework presents the first video anomaly detection use of combing autoencoder and SHapley Additive exPlanations (SHAP) for model interpretability. The framework can explain each anomaly detection result in surveillance videos. In the experiments, we evaluate the proposed framework's effectiveness and efficiency while also explaining anomalies behind the autoencoder's prediction. On the USCD Pedestrian datasets, the DAE achieved 85.9% AUC with a training time of 5 s on the USCD Ped1 and 92.4% AUC with a training time of 2.9 s on the UCSD Ped2.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · see citing works
Smoothing and Differentiation of Data by Simplified Least Squares Procedures.
Savitzky, A., & Golay, M. J. E. (1964) · Analytical Chemistry · doi:10.1021/ac60214a047 · Cited by 21,904 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141 · see citing works
Abnormal crowd behavior detection using social force model
Mehran, R., Oyama, A., & Shah, M. A. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206641 · Cited by 1,698 · see citing works
Anomaly detection in crowded scenes
Mahadevan, V. S., Li, W., Bhalodia, V., & Vasconcelos, N. M. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539872 · Cited by 1,564 · see citing works
Learning Temporal Regularity in Video Sequences
Hasan, M., Choi, J., Neumann, J., Roy–Chowdhury, A. K., & Davis, L. S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.86 · Cited by 1,510 · see citing works
Abnormal Event Detection at 150 FPS in MATLAB
Lu, C., Shi, J., & Jia, J. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.338 · Cited by 1,466 · see citing works
Panoptic Feature Pyramid Networks
Kirillov, A., Girshick, R., He, K., & Dollár, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00656 · Cited by 1,460 · see citing works
A Connection Between Score Matching and Denoising Autoencoders
Vincent, P. (2011) · Neural Computation · doi:10.1162/neco_a_00142 · Cited by 1,011 · see citing works
Robust Real-Time Unusual Event Detection using Multiple Fixed-Location Monitors
Adam, A., Rivlin, E., Shimshoni, I., & Reinitz, D. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.70825 · Cited by 920 · see citing works
End-to-End Learning of Driving Models from Large-Scale Video Datasets
Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.376 · Cited by 884 · see citing works
Speech enhancement based on deep denoising autoencoder
Lu, X., Tsao, Y., Matsuda, S., & Hori, C. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-130 · Cited by 858 · see citing works
Bisong, E. (2019) · Apress eBooks · doi:10.1007/978-1-4842-4470-8_7 · Cited by 804 · see citing works
Kim, J., & Grauman, K. L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206569 · Cited by 797 · see citing works
Madden, H. H. (1978) · Analytical Chemistry · doi:10.1021/ac50031a048 · Cited by 746 · see citing works
Not Just a Black Box: Learning Important Features Through Propagating Activation DifferencesPreprint
Shrikumar, A., Greenside, P., Shcherbina, A., & Kundaje, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01713 · Cited by 704 · see citing works
Remembering history with convolutional LSTM for anomaly detection
Luo, W., Liu, W., & Gao, S. (2017) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2017.8019325 · Cited by 634 · see citing works
Abnormal Event Detection in Videos Using Spatiotemporal Autoencoder
Chong, Y. S., & Tay, Y. H. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-59081-3_23 · Cited by 613 · see citing works
Spatio-Temporal AutoEncoder for Video Anomaly Detection
Zhao, Y., Deng, B., Shen, C., Liu, Y., Lu, H., & Hua, X. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123451 · Cited by 593 · see citing works
Learning Deep Representations of Appearance and Motion for Anomalous Event Detection
Xu, D., Ricci, E., Yan, Y., Song, J., & Sebe, N. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.8 · Cited by 553 · see citing works
Abnormal event detection in videos using generative adversarial nets
Ravanbakhsh, M., Nabi, M., Sangineto, E., Marcenaro, L., Regazzoni, C. S., & Sebe, N. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296547 · Cited by 546 · see citing works
Anomaly Detection in Video Sequence With Appearance-Motion Correspondence
Nguyen, T.-N., & Meunier, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00136 · Cited by 481 · see citing works
Understanding Batch Normalization
Bjorck, N., Gomes, C. P., Selman, B., & Weinberger, K. Q. (2018) · neural information processing systems · Cited by 471 · see citing works
Scale Match for Tiny Person Detection
Yu, X., Gong, Y., Jiang, N., Ye, Q., & Han, Z. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093394 · Cited by 444 · see citing works
Zhou, S., Shen, W., Zeng, D., Fang, M., Wei, Y., & Zhang, Z. (2016) · Signal Processing Image Communication · doi:10.1016/j.image.2016.06.007 · Cited by 258 · see citing works
Video Anomaly Detection with Sparse Coding Inspired Deep Neural Networks
Luo, W., Liu, W., Lian, D., Tang, J., Duan, L., Peng, X., & Gao, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2944377 · Cited by 245 · see citing works
Spatial-Temporal Cascade Autoencoder for Video Anomaly Detection in Crowded Scenes
Li, N., Chang, F., & Liu, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2984093 · Cited by 184 · see citing works
Video Anomaly Detection With Compact Feature Sets for Online Performance
Leyva, R., Sánchez, V., & Li, C. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2695105 · Cited by 166 · see citing works
Context-Aware Activity Recognition and Anomaly Detection in Video
Zhu, Y., Nayak, N. M., & Roy-Chowdhury, A. K. (2013) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2234722 · Cited by 121 · see citing works
Deep Appearance Features for Abnormal Behavior Detection in Video
Smeureanu, S., Ionescu, R. T., Popescu, M., et al. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-68548-9_70 · Cited by 117 · see citing works
Abnormal Event Detection From Videos Using a Two-Stream Recurrent Variational Autoencoder
Yan, S., Smith, J. S., Lu, W., & Zhang, B. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2018.2883368 · Cited by 116 · see citing works
Video anomaly detection based on a hierarchical activity discovery within spatio-temporal contexts
Xu, D., Song, R., Wu, X., et al. (2014) · Neurocomputing · doi:10.1016/j.neucom.2014.06.011 · Cited by 104 · see citing works
STAN: Spatio-Temporal Adversarial Networks for Abnormal Event Detection
Lee, S., Kim, H. G., & Ro, Y. M. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462388 · Cited by 104 · see citing works
Abnormal Event Detection and Localization via Adversarial Event Prediction
Yu, J., Lee, Y., Yow, K., Jeon, M., & Pedrycz, W. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3053563 · Cited by 102 · see citing works
Gao, D., & Vasconcelos, N. M. (2008) · Neural Computation · doi:10.1162/neco.2009.11-06-391 · Cited by 87 · see citing works
Mining Semantic Context Information for Intelligent Video Surveillance of Traffic Scenes
Zhang, T., Liu, S., Xu, C., & Lu, H. (2012) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2012.2218251 · Cited by 86 · see citing works
Explaining Anomalies Detected by Autoencoders Using SHAPPreprint
Antwarg, L., Miller, R. M., Shapira, B., & Rokach, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.02407 · Cited by 83 · see citing works
Unsupervised video summarization framework using keyframe extraction and video skimming
Jadon, S., & Jasim, M. (2020) · IEEE International Conference on Computing, Communication and Automation (ICCCA) · doi:10.1109/iccca49541.2020.9250764 · Cited by 79 · see citing works
Hierarchical Context Modeling for Video Event Recognition
Wang, X., & Ji, Q. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2616308 · Cited by 52 · see citing works
Comparing Factors Affecting Injury Severity of Passenger Car and Truck Drivers
Zhou, B., Wang, X., Zhang, S., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3018183 · Cited by 50 · see citing works
An Efficient Anomaly Detection System for Crowded Scenes Using Variational Autoencoders
Xu, M., Yu, X., Chen, D., et al. (2019) · Applied Sciences · doi:10.3390/app9163337 · Cited by 48 · see citing works
Video Anomaly Detection using Pre-Trained Deep Convolutional Neural Nets and Context Mining
Wu, C., Shao, S., Tunc, C., & Hariri, S. (2020) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa50499.2020.9316538 · Cited by 36 · see citing works
Self-citationTowards Real-Time Multi-Object TrackingPreprint
Wang, Z., Zheng, L., Liu, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12605 · Cited by 32 · see citing works
A causal framework to determine the effectiveness of dynamic quarantine policy to mitigate COVID-19
Kristjanpoller, W., Michell, K., & Minutolo, M. C. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107241 · Cited by 28 · see citing works
Predicting Engagement in Video LecturesPreprint
Bulathwela, S., Pérez‐Ortiz, M., Lipani, A., Yılmaz, E., & Shawe-Taylor, J. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.00592 · Cited by 24 · see citing works
Recursive Hybrid Fusion Pyramid Network for Real-Time Small Object Detection on Embedded Devices
Chen, P.-Y., Hsieh, J.-W., Wang, C.-Y., et al. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00209 · Cited by 21 · see citing works
Convex Polytope Ensembles for Spatio-Temporal Anomaly Detection
Turchini, F., Seidenari, L., & Bimbo, A. D. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-68560-1_16 · Cited by 20 · see citing works
Shao, S., Tunc, C., Alshawi, A., & Hariri, S. (2019) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa47632.2019.9035309 · Cited by 17 · see citing works
Self-citationEnergy-based Models for Video Anomaly DetectionPreprint
Vu, H. T., Phung, D., Nguyen, T. D., Trevors, A., & Venkatesh, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05211 · Cited by 17 · see citing works
Automated Twitter Author Clustering with Unsupervised Learning for Social Media Forensics
Shao, S., Tunc, C., Alshawi, A., & Hariri, S. (2019) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa47632.2019.9035286 · Cited by 16 · see citing works
Self-citationAn Ensemble of Ensembles Approach to Author Attribution for Internet Relay Chat Forensics
Shao, S., Tunc, C., Alshawi, A., & Hariri, S. (2020) · ACM Transactions on Management Information Systems · doi:10.1145/3409455 · Cited by 14 · see citing works
Self-citationSegmentation of COVID-19 pneumonia lesions: A deep learning approach
Ghomi, Z., Mirshahi, R., Bagheri, A. K., Fattahpour, A., Mohammadiun, S., Gharahbagh, A. A., Djavadifar, A., Arabalibeik, H., Sadiq, R., & Hewage, K. (2020) · Medical Journal of the Islamic Republic of Iran · doi:10.47176/mjiri.34.174 · Cited by 14 · see citing works
Gao, X. (2019) · Signal Image and Video Processing · doi:10.1007/s11760-019-01592-4 · Cited by 10 · see citing works
Detecting Anomalies in Image Classification by Means of Semantic Relationships
Pasini, A., & Baralis, E. (2019) · IEEE International Conference on Artificial Intelligence and Knowledge Engineering (AIKE) · doi:10.1109/aike.2019.00047 · Cited by 8 · see citing works
Gao, X. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09201-0 · Cited by 7 · see citing works
Spatio-Temporal Processing for Automatic Vehicle Detection in Wide-Area Aerial Video
Gao, X., Szep, J., Satam, P., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3033466 · Cited by 6 · see citing works
Self-citationSeVA: An AI Solution for Age Friendly Care of Hospitalized Older Adults
Wu, C., Szep, J., Hariri, S., et al. (2021) · International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC) · doi:10.5220/0010313605830591 · Cited by 4 · see citing works
Self-citationAn Intra-Frame Classification Network for Video Anomaly Detection and Localization
Xu, K., Jiang, X., & Sun, T. (2018) · International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) · doi:10.1109/cisp-bmei.2018.8633010 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [24]Chen, N.F., Du, Z., Ng, K. H.: Scene Graphs for Interpretable Video Anomaly Classification. Conference on Neural Information Processing Systems Workshop on Visually Grounded Interaction and Language (2018)
Cited by
Sorted by citations
Deep Learning-Based Anomaly Detection in Video Surveillance: A Survey
Duong, H.-T., Le, V.-T., & Hoang, V. T. (2023) · Sensors · doi:10.3390/s23115024 · Cited by 178 · see citing works
A Survey on Explainable Anomaly Detection
Li, Z., Zhu, Y., & Van Leeuwen, M. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3609333 · Cited by 139 · see citing works
“Ref Spec Pers Tech Data Loc Pros Cons [156] A F Feature selection Static TC G Handles noisy features well Only applicable to categorical data [158] A F Feature selection Static TC G Linear time complexity to data size Only applicable to categorical data [82] A F & S Feature selection + Instance selection Static TC G J…”
BackgroundMethod
- Authors
- Wu, C., Shao, S., Tunc, C., Satam, P., & Hariri, S.
- Affiliations
- University of Arizona (US)
- University of North Texas (US)
- Corresponding author
- Wu, Chongke
- Venue
- Cluster Computing
- Published
- 2021-11-23
- Volume
- 25
- Issue
- 4
- Pages
- 2715-2737
- ISSN
- 1386-7857 (ISSN-L)
- 1573-7543
- PMC
- PMC8609273
- OpenAlex
- W3216015240
- Cited by
- 88
- References
- 63
- Citations by year
- 2026: 13 · 2025: 26 · 2024: 15 · 2023: 21 · 2022: 13
- FWCI
- 4.31
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Cluster Computing (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- UA Campus Repository (The University of Arizona) (acceptedVersion)
- Funders
- Air Force Office of Scientific Research (FA9550-18-1-0427)
- National Science Foundation (NSF-1624668, NSF-1849113, DUE-1303362)
- National Institute of Standards and Technology (70NANB18H263)
- National Nuclear Security Administration (DE-NA0003946)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video anomaly detection, explainable AI, SHAP, denoising autoencoder, model interpretability, pre-trained feature extraction, efficient training, surveillance video analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
