Abstract

Anomaly detection is the process of identifying data points, observations or events which deviate from normal behavior. To detect anomalies, we have used the PyCaret machine learning library. PyCaret is a low code, open-source library that automates our machine learning processes and helps us detect outliers/anomalies Its Anomaly detection and Regression modules contain various machine learning algorithms and frameworks such as XGBoost, CatBoost, Isolation Forest, DBSCAN, etc. It is a deployment-ready library that is easy to use and helps users to perform end-to-end experiments efficiently. In this paper, we applied clustering and regression-based methods on the NAB Twitter dataset for time series anomaly detection. In the regression method, we predicted the tweets, calculated the difference by comparing them with actual tweets, and used the thresholding technique for anomaly detection.

References

• Prokhorenkova L, Gusev G, Vorobev A, Dorogush A, Gulin A. CatBoost: unbiased boosting with categorical features.
• Astha G, Wenyu, Jules S, Ramasamy S, Chuan-Sheng F. An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series. IEEE Transactions on Neural Networks and Learning Systems
• Yu Qin, YuanSheng Lou. Hydrological Time Series Anomaly Pattern Detection based on Isolation Forest. 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC 2019)
• Yiyang Niu. Walmart Sales Forecasting using XGBoost algorithm and Feature engineering. 2020 International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE)
• Subutai Ahmad, Alexander Lavin, Scott Purdy, Zuha Agha, “Unsupervised real-time anomaly detection for streaming data”, Neurocomputing, Volume 262, 1 November 2017, Pages 134-147, ISSN 0925-2312.
• S. Zhong, S. Fu, L. Lin, X. Fu, Z. Cui, and R. Wang, "A novel unsupervised anomaly detection for gas turbine using Isolation Forest," 2019 IEEE International
• Conference on Prognostics and Health Management (ICPHM), 2019, pp. 1-6, DOI: 10.1109/ICPHM.2019.8819409.
• Fattah, Jamal & Ezzine, Latifa & Aman, Zineb & Moussami, Haj & Lachhab, Abdeslam. (2018). Forecasting of demand using ARIMA model. International Journal of Engineering Business Management. 10. 184797901880867. 10.1177/1847979018808673.
• Y. Wang, J. Wong and A. Miner, "Anomaly intrusion detection using one-class SVM," Proceedings from the Fifth Annual IEEE SMC Information Assurance Workshop, 2004., 2004, pp. 358-364, DOI: 10.1109/IAW.2004.1437839.
• Chun-Hsiang Lee, Xu Lu, Xiunao Lin, Hongfeng Tao, Yaolei Xue, and Chao Wu. 2020. Anomaly Detection of Storage Battery Based on Isolation Forest and Hyperparameter Tuning. In Proceedings of the 2020 5th International Conference on Mathematics and Artificial Intelligence (ICMAI 2020). Association for Computing Machinery, New York, NY, USA, 229–233. https://doi.org/10.1145/3395260.3395271
• Jebb, A. T., Tay, L., Wang, W., & Huang, Q. (2015). Time series analysis for psychological research: examining and forecasting change. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2015.00727
• Y. Jin, C. Qiu, L. Sun, X. Peng and J. Zhou, "Anomaly detection in time series via robust PCA," 2017 2nd IEEE International Conference on Intelligent Transportation Engineering (ICITE), 2017, pp. 352-355, DOI: 10.1109/ICITE.2017.8056937.
• Wentai Wu, Ligang He, Weiwei Lin, Yi Su, Yuhua Cui, Carsten Maple, Stephen A. Jarvis, "Developing an Unsupervised Real-time Anomaly Detection Scheme for Time Series with Multi-seasonality," in IEEE Transactions on Knowledge and Data Engineering, DOI: 10.1109/TKDE.2020.3035685.