Abstract

The research presented in this paper examines how Human in the Loop (HWiL) models can be embedded in contemporary Incident Management (IM) systems, and the potential for achieving the dual objectives of automation efficiency and human intuition .The research in this paper aims at the embedding of HwiL frameworks in modern Incident Management (IM) system, and the potential of achieving the twin goals of automation efficiency and human intuition. The main goal is to create and test user interfaces that enable operators to easily interact with the suggestions generated by the AI model, minimizing false alerts and continuously refining the model by receiving user feedback. The data set we use is a special one, containing 215 incidents of alerts from IT infrastructure, organized by the severity of the alert and by the way they've previously been resolved. The methodology uses a proprietary simulation environment called HITL-Manager 2.0, which is developed for real-time collaborative decision. To ensure that operators are aware of why the AI is recommending a correction, and that the suggested correction is easy to implement, we have found that accuracy of the system increases significantly with time. The results indicate that validation of operators in an iterative process is a key signal in training the underlying machine learning models. This paper discusses the architectural needs for such interfaces, focusing on minimizing the cognitive load and building confidence with the human expert and the automated system. The paper highlights the potential for collaboration between human and algorithmic response capabilities and recommends a more resilient and adaptive approach by emphasizing how both can work together. The paper emphasizes the potential for synergizing human and algorithmic response capabilities and suggests a more resilient and adaptive approach by highlighting the convergence of both.

References

• N. Aoki, “The importance of the assurance that ‘humans are still in the decision loop’ for public trust in artificial intelligence: Evidence from an online experiment,” Computers in Human Behavior, vol. 114, p. 106572, 2020. https://doi.org/10.1016/j.chb.2020.106572
• D. Dellermann, A. Calma, N. Lipusch, T. Weber, S. Weigel, and P. Ebel, “The future of human-AI collaboration: A taxonomy of design knowledge for hybrid intelligence systems,” in Proc. 52nd Hawaii Int. Conf. System Sciences, 2019, pp. 274–283. https://doi.org/10.24251/HICSS.2019.034
• D. E. Ehrmann, S. N. Gallant, S. Nagaraj, S. D. Goodfellow, D. Eytan, A. Goldenberg, and M. L. Mazwi, “Evaluating and reducing cognitive load should be a priority for machine learning in healthcare,” Nature Medicine, vol. 28, no. 7, pp. 1331–1333, 2022. https://doi.org/10.1038/s41591-022-01833-z
• J. E. Fischer, C. Greenhalgh, W. Jiang, S. D. Ramchurn, F. Wu, and T. Rodden, “In-the-loop or on-the-loop? Interactional arrangements to support team coordination with a planning agent,” Concurrency and Computation: Practice and Experience, vol. 33, no. 8, p. e4082, 2017. https://doi.org/10.1002/cpe.4082
• O. O. Garibay et al., “Six human-centered artificial intelligence grand challenges,” International Journal of Human-Computer Interaction, vol. 39, no. 3, pp. 391–437, 2023. https://doi.org/10.1080/10447318.2022.2153320
• P. Hemmer, M. Westphal, M. Schemmer, S. Vetter, M. Vössing, and G. Satzger, “Human-AI collaboration: The effect of AI delegation on human task performance and task satisfaction,” arXiv preprint arXiv:2303.09224, 2023. https://arxiv.org/abs/2303.09224
• J. Jiang, A. J. Karran, C. K. Coursaris, P. Léger, and J. Beringer, “A situation awareness perspective on human-AI interaction: Tensions and opportunities,” International Journal of Human-Computer Interaction, vol. 39, no. 9, pp. 1789–1806, 2022. https://doi.org/10.1080/10447318.2022.2093863
• S. Kumar, S. Datta, V. Singh, D. Datta, S. K. Singh, and R. Sharma, “Applications, challenges, and future directions of human-in-the-loop learning,” IEEE Access, vol. 12, pp. 75735–75760, 2024. https://doi.org/10.1109/ACCESS.2024.3401547
• N. Merat et al., “The ‘Out-of-the-Loop’ concept in automated driving: Proposed definition, measures and implications,” Cognition, Technology & Work, vol. 21, no. 1, pp. 87–98, 2018. https://doi.org/10.1007/s10111-018-0525-8
• S. Middleton, E. Letouzé, A. Hossaini, and A. Chapman, “Trust, regulation, and human-in-the-loop AI,” Communications of the ACM, vol. 65, no. 4, pp. 64–68, 2022. https://doi.org/10.1145/3511597
• E. Mosqueira-Rey, E. Hernández-Pereira, D. Alonso-Ríos, J. Bobes-Bascarán, and Á. Fernández-Leal, “Human-in-the-loop machine learning: A state of the art,” Artificial Intelligence Review, vol. 56, no. 4, pp. 3005–3054, 2022. https://doi.org/10.1007/s10462-022-10246-w
• T. A. Schoonderwoerd, E. M. Van Zoelen, K. Van Den Bosch, and M. A. Neerincx, “Design patterns for human-AI co-learning: A wizard-of-Oz evaluation in an urban-search-and-rescue task,” International Journal of Human-Computer Studies, vol. 164, p. 102831, 2022. https://doi.org/10.1016/j.ijhcs.2022.102831