Abstract

Modern network infrastructures face increasing cyber threats including malware attacks, distributed denial-of-service attacks, and unauthorized access attempts. Traditional intrusion detection systems primarily rely on signature-based or rule-based detection mechanisms, which are limited in detecting unknown or evolving attack patterns. While artificial intelligence techniques have been increasingly applied to improve network traffic analysis, many machine learning models lack contextual reasoning and dynamic decision-making capabilities. This paper proposes and evaluates an intelligent intrusion prevention framework that integrates agentic artificial intelligence with retrieval-augmented generation (RAG) for network traffic analysis. The proposed system combines real-time traffic monitoring, anomaly detection, knowledge retrieval, and autonomous response mechanisms. Experimental evaluation using the NSL-KDD, CICIDS2017, and UNSWNB15 datasets demonstrates improved detection accuracy (0.96) and reduced false positive rates (0.05) compared with traditional machine learning models. Ablation studies confirm that the RAG component reduces false positives by 37.5% compared to the anomaly detector alone. The study indicates that combining agentic AI with retrieval-based reasoning provides adaptive and explainable security mechanisms for modern network environments.

References

• M Ahmed, A Mahmood, and J Hu. Network traffic anomaly detection using machine learning. Journal of Network and Computer Applications, 215:103678, 2023.
• Francesco Blefari, Cristian Cosentino, Francesco Aurelio Pironti, Angelo Furfaro, and Fabrizio Marozzo. Cyberrag: An agentic rag cyber attack classification and reporting tool. Future Generation Computer Systems, page 108186, 2025.
• A Buczak and E Guven. A survey of data mining and machine learning methods for cybersecurity intrusion detection. IEEE Communications Surveys & Tutorials, 18(2):1153– 1176, 2016.
• D Denning. An intrusion detection model. IEEE Transactions on Software Engineering, SE-13(2):222–232, 1987.
• P Garcia-Teodoro, J Diaz-Verdejo, G Macia-Fernandez, and E Vazquez. Anomaly-based network intrusion detection: Techniques, systems and challenges. Computers & Security, 28(1-2):18–28, 2009.
• Firuz Kamalov, Sherif Moussa, Rita Zgheib, and Omar Mashaal. Feature selection for intrusion detection systems. In 2020 13th International Symposium on Computational Intelligence and Design (ISCID), pages 265–269, 2020.
• S Kasongo and Y Sun. Performance analysis of intrusion detection systems using feature selection on the unsw-nb15 dataset. Journal of Big Data, 7(1):1–20, 2020.
• G Kim, S Lee, and S Kim. A novel hybrid intrusion detection method integrating anomaly detection with misuse detection. Expert Systems with Applications, 41(4):1690–1700, 2014.
• P Lewis, E Perez, A Piktus, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459– 9474, 2020.
• Hongjuan Li, Hui Kang, Jiahui Li, Geng Sun, Ruichen Zhang, JiachengWang, Dusit Niyato,Wei Ni, and Abbas Jamalipour. Multi-agent collaborative intrusion detection for low-altitude economy iot: An llm-enhanced agentic ai framework. arXiv preprint arXiv:2601.17817, 2026.
• Zong-Xun Li, Yu-Jun Li, Yi-Wei Liu, Cheng Liu, and Nan- Xin Zhou. K-ctiaa: Automatic analysis of cyber threat intelligence based on a knowledge graph. Symmetry, 15(2):337, 2023.
• R Lippmann, J Haines, D Fried, J Korba, and K Das. The 1999 darpa off-line intrusion detection evaluation. Computer Networks, 34(4):579–595, 2000.
• Xiang Luo, Chang Liu, Gang Xiong, Chen Yang, Gaopeng Gou, Yaochen Ren, and Zhen Li. Malrag: A retrievalaugmented llm framework for open-set malicious traffic identification. arXiv preprint arXiv:2511.14129, 2025.
• N Moustafa and Jill Slay. The unsw-nb15 dataset for network intrusion detection systems. In Military Communications and Information Systems Conference, pages 1–6, 2015.
• S Russell and P Norvig. Artificial Intelligence: A Modern Approach. Pearson, 4th edition, 2021.
• I Sharafaldin, A Lashkari, and A Ghorbani. Toward generating a new intrusion detection dataset and intrusion traffic characterization. In International Conference on Information Systems Security and Privacy, pages 108–116, 2018.
• N Shone, T Ngoc, V Phai, and Q Shi. Deep learning approach to network intrusion detection. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(3):234– 245, 2022.
• Marco Simoni, Andrea Saracino, Vinod P, and Mauro Conti. Morse: Bridging the gap in cybersecurity expertise with retrieval augmented generation. In Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing, pages 1213–1222, 2025.
• R Sommer and V Paxson. Outside the closed world: On using machine learning for network intrusion detection. In IEEE Symposium on Security and Privacy, pages 305–316, 2010.
• M Tavallaee, E Bagheri, W Lu, and A Ghorbani. A detailed analysis of the kdd cup 99 data set. In IEEE Symposium on Computational Intelligence for Security and Defense Applications, pages 1–6, 2009.
• W Wang, Y Sheng, J Wang, et al. Hast-ids: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection. IEEE Transactions on Information Forensics and Security, 17:1234–1247, 2022.
• C Yin, Y Zhu, J Fei, and X He. A deep learning approach for intrusion detection using recurrent neural networks. IEEE Access, 5:21954–21961, 2017.