ConvGRU-CNN: Spatiotemporal Deep Learning for Real-World Anomaly Detection in Video Surveillance System.
DOI:
https://doi.org/10.9781/ijimai.2023.05.006Keywords:
Anomaly Detection, Crime Detection, Convolutional Neural Network (CNN), Deep Learning, Video Surveillance, Convolutional Gated Recurrent Unit (Convolutional GRU)Abstract
Video surveillance for real-world anomaly detection and prevention using deep learning is an important and difficult research area. It is imperative to detect and prevent anomalies to develop a nonviolent society. Realworld video surveillance cameras automate the detection of anomaly activities and enable the law enforcement systems for taking steps toward public safety. However, a human-monitored surveillance system is vulnerable to oversight anomaly activity. In this paper, an automated deep learning model is proposed in order to detect and prevent anomaly activities. The real-world video surveillance system is designed by implementing the ResNet-50, a Convolutional Neural Network (CNN) model, to extract the high-level features from input streams whereas temporal features are extracted by the Convolutional GRU (ConvGRU) from the ResNet-50 extracted features in the time-series dataset. The proposed deep learning video surveillance model (named ConvGRUCNN) can efficiently detect anomaly activities. The UCF-Crime dataset is used to evaluate the proposed deep learning model. We classified normal and abnormal activities, thereby showing the ability of ConvGRU-CNN to find a correct category for each abnormal activity. With the UCF-Crime dataset for the video surveillance-based anomaly detection, ConvGRU-CNN achieved 82.22% accuracy. In addition, the proposed model outperformed the related deep learning models.
Downloads
References
T. Hospedales, S. Gong, and T. Xiang, “Video behaviour mining using a dynamic topic model,” International journal of computer vision, vol. 98, no. 3, pp. 303-323, 2012.
M. Cristani, R. Raghavendra, A. Del Bue, and V. Murino, “Human behavior analysis in video surveillance: A social signal processing perspective,” Neurocomputing, vol. 100, pp. 86-97, 2013.
B. Tian, B.T. Morris, M. Tang, Y. Liu, Y. Yao, C. Gou, ... and S. Tang, “Hierarchical and networked vehicle surveillance in ITS: a survey,” IEEE transactions on intelligent transportation systems, vol. 16, no. 2, pp. 557- 580, 2017.
J. Yu, K.C. Yow, and M. Jeon, “Joint representation learning of appearance and motion for abnormal event detection,” Machine Vision and Applications, vol. 29, no. 7, pp. 1157-1170, 2018.
S. R. Bandekar and C. Vijayalakshmi, “Design and analysis of machine learning algorithms for the reduction of crime rates in India,” Procedia Computer Science, vol. 172, no. 122-127, 2020.
J. Varadarajan and J. M. Odobez, “Topic models for scene analysis and abnormality detection,” in 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, IEEE, 2009, pp. 1338-1345.
C. Tabedzki, A. Thirumalaiswamy, P. van Vliet, S. Agarwal and S. Sun, “Yo home to Bel-Air: predicting crime on the streets of Philadelphia,” University of Pennsylvania, CIS, 520, 2018.
K. A. Joshi and D.G. Thakore, “A survey on moving object detection and tracking in video surveillance system,” International Journal of Soft Computing and Engineering, vol. 2, no. 3, pp. 44-48, 2012.
R. Socha and B. Kogut, “Urban video surveillance as a tool to improve security in public spaces,” Sustainability, vol. 12, no. 15, 6210, 2020.
A. Selvaraj, J. Selvaraj, S. Maruthaiappan, G.C. Babu and P.M. Kumar, “L1 norm based pedestrian detection using video analytics technique,” Computational Intelligence, vol. 36, no. 4, pp. 1569-1579, 2020.
D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 4489-4497.
L. Alkanhal, D. Alotaibi, N. Albrahim, S. Alrayes, G. Alshemali and O. Bchir, “Super-resolution using deep learning to support person identification in surveillance video,” International Journal of Advanced Computer Science and Applications, vol. 11, no. 7, 2020.
J. Athanesious, V. Srinivasan, V. Vijayakumar, S. Christobel and S.C. Sethuraman, “Detecting abnormal events in traffic video surveillance using superorientation optical flow feature,” IET Image processing, vol. 14, no. 9, pp. 1881-1891, 2020.
H. Zhang, P. Li, Z. Du and W. Dou, “Risk entropy modeling of surveillance camera for public security application,” IEEE Access, vol. 8, pp. 45343- 45355, 2020.
B.S. Harish and S.A. Kumar, “Anomaly based Intrusion Detection using Modified Fuzzy Clustering,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 4, no. 6, pp. 54-60, 2017.
J. Deng, W. Dong, R. Socher, L.J. Li, K. Li and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, IEEE, 2009, pp. 248-255.
A.A. Sodemann, M.P. Ross and B.J. Borghetti, “A review of anomaly detection in automated surveillance,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp. 1257-1272, 2012.
I.V. Pustokhina, D.A. Pustokhin, T. Vaiyapuri, D. Gupta, S. Kumar and K. Shankar, “An automated deep learning based anomaly detection in pedestrian walkways for vulnerable road users safety,” Safety science, vol. 142, 105356, 2021.
R. Nawaratne, D. Alahakoon, D. De Silva and X. Yu, “Spatiotemporal anomaly detection using deep learning for real-time video surveillance,” IEEE Transactions on Industrial Informatics, vol. 16, no. 1, pp. 393-402, 2019.
M. Á. López, J.M. Lombardo, M. López, C.M. Alba, S. Velasco, M.A. Braojos and M. Fuentes-García, “Intelligent Detection and Recovery from Cyberattacks for Small and Medium-Sized Enterprises,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 6, no. 3, 2020.
K. Rezaee, S.M. Rezakhani, M.R. Khosravi and M.K. Moghimi, “A survey on deep learning-based real-time crowd anomaly detection for secure distributed video surveillance,” Personal and Ubiquitous Computing, pp. 1-17, 2021.
F. Rezaei and M. Yazdi, “A New Semantic and Statistical DistanceBased Anomaly Detection in Crowd Video Surveillance,” Wireless Communications and Mobile Computing, vol. 2021, 5513582, 2021.
W. Ullah, A. Ullah, I.U. Haq, K. Muhammad, M. Sajjad and S.W. Baik, “CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks,” Multimedia Tools and Applications, vol. 80, no. 11, pp. 16979-16995, 2021.
W. Ullah, A. Ullah, T. Hussain, Z.A. Khan and S.W. Baik, “An Efficient Anomaly Recognition Framework Using an Attention Residual LSTM in Surveillance Videos,” Sensors, vol. 21, no. 8, 2811, 2021.
Y. Luo, Y. Xiao, L. Cheng, G. Peng and D. Yao, “Deep learningbased anomaly detection in cyber-physical systems: Progress and opportunities,” ACM Computing Surveys (CSUR), vol. 54, no. 5, pp. 1-36, 2021.
K.K. Santhosh, D.P. Dogra, P.P. Roy and A. Mitra, “Vehicular Trajectory Classification and Traffic Anomaly Detection in Videos Using a Hybrid CNN-VAE Architecture,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 8, pp. 11891-11902, 2021.
H. Fanta, Z. Shao and L. Ma, “SiTGRU: single-tunnelled gated recurrent unit for abnormality detection,” Information Sciences, vol. 524, pp. 15-32, 2020.
W. Shin, S.J. Bu and S.B. Cho, “3D-convolutional neural network with generative adversarial network and autoencoder for robust anomaly detection in video surveillance,” International Journal of Neural Systems, vol. 30, no. 6, 2050034, 2020.
W. Sultani, C. Chen, and M. Shah, “Real-world anomaly detection in surveillance videos,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6479-6488.
A. Rummens, W. Hardyns and L. Pauwels, “The use of predictive analysis in spatiotemporal crime forecasting: Building and testing a model in an urban context,” Applied geography, vol. 86, pp. 255-261, 2017.
S. Kim, P. Joshi, P.S. Kalsi and P. Taheri, “Crime analysis through machine learning,” in 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), IEEE, 2018, pp. 415-420.
V. Tsakanikas and T. Dagiuklas, “Video surveillance systems-current status and future trends,” Computers & Electrical Engineering, vol. 70, pp. 736-753, 2018.
S. Prithi, S. Aravindan, E. Anusuya and A.M. Kumar, “GUI based prediction of crime rate using machine learning approach,” International Journal of Computer Science and Mobile Computing, vol. 9, no. 3, pp. 221-229, 2020.
H.W. Kang and H.B. Kang, “Prediction of crime occurrence from multimodal data using deep learning”, PLOS ONE, vol. 12, no. 4, e0176244, 2017.
S. Hossain, A. Abtahee, I. Kashem, M.M. Hoque and I.H. Sarker, “Crime prediction using spatio-temporal data,” in International Conference on Computing Science, Communication and Security, Springer, Singapore, 2020, pp. 277-289.
G.N. Obuandike, I. Audu and A. John, “Analytical study of some selected classification algorithms in WEKA using real crime data,” International Journal of Advanced Research in Artificial Intelligence, vol. 4, no. 12, 2015.
C. C. Sun, C. Yao, X. Li and K. Lee, “Detecting Crime Types Using Classification Algorithms,” Journal of Digital Information Management, vol. 12, no. 5, pp. 321-327, 2014.
M. Jangra and S. Kalsi, “Crime analysis for multistate network using naive Bayes classifier,” International Journal of Computer Science and Mobile Computing, vol. 8, no. 6, pp. 134-143, 2019.
F. Vanhoenshoven, G. Nápoles, S. Bielen and K. Vanhoof, “Fuzzy cognitive maps employing ARIMA components for time series forecasting,” in International Conference on Intelligent Decision Technologies, Springer, Cham, 2017, pp 255-264.
W. Gorr, A. Olligschlaeger and Y. Thompson, “Assessment of crime forecasting accuracy for deployment of police,” International journal of forecasting, 743-754, 2000.
C.H. Yu, M.W. Ward, M. Morabito and W. Ding, “Crime forecasting using data mining techniques,” in 2011 IEEE 11th international conference on data mining workshops, IEEE, 2011, pp. 779-786.
L.G. Alves, H.V. Ribeiro and F.A. Rodrigues, “Crime prediction through urban metrics and statistical learning,” Physica A: Statistical Mechanics and its Applications, vol. 505, pp. 435-443, 2018.
M.Q. Gandapur, “E2E-VSDL: End-to-end video surveillance-based deep learning model to detect and prevent criminal activities,” Image and Vision Computing, vol.123, 104467, 2022.
M. Adimoolam, S. Mohan, A. John and G. Srivastava, “A Novel Technique to Detect and Track Multiple Objects in Dynamic Video Surveillance Systems,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 7, no. 4, 2022.
P. Christiansen, L.N. Nielsen, K.A. Steen, R.N. Jørgensen and H. Karstoft, “DeepAnomaly: Combining background subtraction and deep learning for detecting obstacles and anomalies in an agricultural field,” Sensors, vol. 16, no. 11, 1904, 2016.
L. Dong, Y. Zhang, C. Wen and H. Wu, “Camera anomaly detection based on morphological analysis and deep learning,” in 2016 IEEE International Conference on Digital Signal Processing (DSP), IEEE, 2016, pp. 266-270.
D. Xu, E. Ricci, Y. Yan, J. Song and N. Sebe, “Learning deep representations of appearance and motion for anomalous event detection,” 2015, arXiv preprint arXiv:1510.01553.
M. Hasan, J. Choi, J. Neumann, A.K. Roy-Chowdhury and L.S. Davis, “Learning temporal regularity in video sequences,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 733-742.
V. Nguyen, D. Phung, D.S. Pham and S. Venkatesh, “Bayesian nonparametric approaches to abnormality detection in video surveillance,” Annals of Data Science, vol. 2, no. 1, pp. 21-41, 2015.
T. Ergen and S.S. Kozat, “Unsupervised anomaly detection with LSTM neural networks,” IEEE transactions on neural networks and learning systems, vol. 31, no. 8, pp. 3127-3141, 2019.
J.X. Zhong, N. Li, W. Kong, S. Liu, T.H. Li and G. Li, “Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1237-1246.
S. Sudhakaran and O. Lanz, “Learning to detect violent videos using convolutional long short-term memory”, in 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), IEEE, 2017, pp. 1-6.
K. Zhang, M. Sun, T.X. Han, X. Yuan, L. Guo and T. Liu, “Residual networks of residual networks: Multilevel residual networks,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 6, pp. 1303-1314, 2017.
Y. Wu, Q. Wu, N. Dey and S. Sherratt, “Learning models for semantic classification of insufficient plantar pressure images,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 6, no. 1, pp. 51-61, 2020.
M.G. Huddar, S.S. Sannakki and V.S. Rajpurohit, “Attention-based Multimodal Sentiment Analysis and Emotion Detection in Conversation using RNN,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 6, no. 6, pp. 112-121, 2021.
S.M. Erfani, S. Rajasegarar, S. Karunasekera and C. Leckie, “Highdimensional and large-scale anomaly detection using a linear one-class SVM with deep learning,” Pattern Recognition, vol. 58, pp. 121-134, 2016.
C. Lu, J. Shi and J. Jia, “Abnormal event detection at 150 fps in matlab,” in Proceedings of the IEEE international conference on computer vision, 2013, pp. 2720-2727.
S. Vosta and K.C. Yow, “A CNN-RNN Combined Structure for Real-World Violence Detection in Surveillance Cameras,” Applied Sciences, vol. 12, no. 3, 1021, 2022.
M.U.K. Khan, H.S. Park and C.M. Kyung, “Rejecting motion outliers for efficient crowd anomaly detection,” IEEE Transactions on Information Forensics and Security, vol. 14, no. 2, pp. 541-556, 2018.
Hosseinzadeh, M., Rahmani, A. M., Vo, B., Bidaki, M., Masdari, M., & Zangakani, M. “Improving security using SVM-based anomaly detection: issues and challenges,” Soft Computing, vol. 25, 3195-3223.
A.A. Alvarez and F. Gómez, “Motivic Pattern Classification of Music Audio Signals Combining Residual and LSTM Networks,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 6, no. 6, 2021.
N. Saleem, J. Gao, M. Irfan, E. Verdu and J.P. Fuente, “E2E-V2SResNet: Deep residual convolutional neural networks for end-to-end video driven speech synthesis,” Image and Vision Computing, vol. 119, 104389, 2022.
Downloads
Published
-
Abstract249
-
PDF47
Submit Article
Information
