Abstract

Insider threats remain one of the most difficult security challenges because malicious actions often originate from trusted users and evolve over time. Traditional rule-based and static incident response systems struggle to adapt to changing insider behaviours, leading to delayed or suboptimal responses. This study proposes an adaptive incident response framework based on reinforcement learning that dynamically selects response actions according to observed system states and threat severity. The framework models incident response as a sequential decision-making process, where an agent learns optimal response policies through interaction with a simulated enterprise environment. States capture security context and threat indicators, actions represent response options, and rewards are designed to balance rapid containment, operational continuity, and false positive reduction. Experimental evaluation demonstrates that the proposed approach consistently outperforms static and heuristic-based baselines in response effectiveness, convergence stability, and adaptability to evolving attack patterns. Results show improved response accuracy, faster containment times, and stable learning behaviour across training episodes. The Q Learning model performed better than Support Vector Machine and Random Forest models, reaching 96.8 percent accuracy, an F1 score of 0.944, and a Matthews Correlation Coefficient (MCC) of 0.917. When connected to Security Orchestration, Automation and Response (SOAR) platforms, the system can make fast and context aware decisions that reduce the work of analysts and shorten response time. The findings confirm that reinforcement learning offers a practical and scalable solution for adaptive insider threat incident response. This work contributes an automated decision framework that improves resilience, reduces manual intervention, and supports trustworthy security operations in dynamic environments.

References

• U.S. CISA, “Guidance for SIEM and SOAR Implementation,” resource hub, May 27, 2025. https://www.cisa.gov/resources-tools/resources/guidance-siem-and-soar-implementation
• 12Y. Gong, S. Cui, S. Liu, B. Jiang, C. Dong, and Z. Lu, “Graph-based insider threat detection: A survey,” Computer Networks, vol. 254, 2024, Art. 110757. DOI: 10.1016/j.comnet.2024.110757. https://www.sciencedirect.com/science/article/abs/pii/S1389128624005899
• C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, pp. 279–292, 1992. DOI: 10.1007/BF00992698.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA: MIT Press, 2018. https://mitpress.mit.edu/9780262039246/reinforcement-learning/
• A. M. K. Adawadkar and N. Kulkarni, “Cyber-security and reinforcement learning — A brief survey,” Engineering Applications of Artificial Intelligence, vol. 114, 2022, Art. 105116. DOI: 10.1016/j.engappai.2022.105116.
• I. Miles et al., “Reinforcement Learning for Autonomous Resilient Cyber Defence,” Frazer-Nash Consultancy White Paper, 2024.https://www.fnc.co.uk/media/mwcnckij/us-24-milesfarmer-reinforcementlearningforautonomousresilientcyberdefence-wp.pdf
• B. Lindauer, “Insider Threat Test Dataset.” Carnegie Mellon University, Software Engineering Institute, 2020. DOI: 10.1184/R1/12841247.v1.
• J. Glasser and B. Lindauer, “Bridging the Gap: A Pragmatic Approach to Generating Insider Threat Data,” 2013 IEEE Security and Privacy Workshops, pp. 98–104, 2013. DOI: 10.1109/SPW.2013.37.
• H. Han, W.-Y. Wang, and B.-H. Mao, “Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning,” in Advances in Intelligent Computing, 2005, pp. 878–887. DOI: 10.1007/11538059_91.
• Y. Sun, M. Zhang, and C. Li, “Borderline SMOTE algorithm and feature selection-based network anomalies detection strategy,” Energies, vol. 15, no. 13, 2022, Art. 4751. DOI: 10.3390/en15134751.
• D. Chicco and G. Jurman, “The advantages of the MCC over F1 score and accuracy in binary classification evaluation,” BMC Genomics, vol. 21, no. 6, 2020. DOI: 10.1186/s12864-019-6413-7.
• Y. Gong, S. Cui, S. Liu, B. Jiang, C. Dong, and Z. Lu, “Graph-based insider threat detection: A survey,” Computer Networks, vol. 254, 2024, Art. 110757. DOI: 10.1016/j.comnet.2024.110757. https://www.sciencedirect.com/science/article/abs/pii/S1389128624005899
• X. Tao, Z. Cao, and J. Huang, “An insider threat detection method based on improved Test-Time Training,” High-Confidence Computing, 2025. (Online first) https://www.sciencedirect.com/science/article/pii/S2667295224000862.
• T. Tian, X. Luo, and X. Li, “Insider threat detection for specific threat scenarios,” Cybersecurity, 2025. https://cybersecurity.springeropen.com/articles/10.1186/s42400-024-00321-w