Abstract

Voice fraud has recently become a critical cybersecurity concern, where fraudsters impersonate trusted individuals over phone calls to deceive victims and obtain confidential information. Conventional fraud detection systems, which rely on predetermined rules or post-call detection, are increasingly becoming ineffective against evolving fraud patterns and do not ensure user safety in real-time. This study proposes the design and implementation of a real-time voice fraud detection system that analyses real-time conversations to identify potential fraudulent activity. The proposed system integrates OpenAI’s Whisper model for speech-to-text transcription with a fine-tuned Phi-3-mini Large Language Model (LLM) for contextual fraud detection. The system architecture uses the Browser MediaStream API for audio capture and WebSocket communication to transmit streaming data to a FastAPI-based backend for real-time processing. The fine-tuned LLM was trained on a dataset comprising 447 labeled conversational samples and subsequently evaluated on 307 unseen test samples, achieving an overall accuracy of 95%. Experimental results demonstrate a precision of 97% and a recall of 98% for normal conversation, and a precision of 75% and a recall of 67% for fraudulent conversation. These findings underscore the effectiveness of leveraging context-aware language modelling and real-time audio transcription in detecting fraudulent speech patterns, thereby surpassing the limitations of traditional rule-based systems. The study contributes to the body of knowledge by demonstrating the feasibility of combining speech processing and large language models to enable proactive, real-time detection of voice fraud during phone calls.

References

• Ifeoma, I.-I. A., and Anulika, C. O. 2022. Conversational analysis of scam calls in Nigeria. Awka Journal of Linguistics and Languages, Special Edition 2, pp. 269–297.
• Behera, S. K., and Nayak, M. M. 2020. Natural language processing for text and speech processing: A review paper. SSRN Scholarly Paper No. 3878634. Social Science Research Network. Available at: https://papers.ssrn.com/abstract=3878634
• Bhargavi, D. K., and Shivani, B. M. 2024. Detection of fraudulent phone calls in mobile applications. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 15(2).
• Detecting fraud calls vis-à-vis natural language processing. n.d. ResearchGate. Available at: https://www.researchgate.net/publication/381320394_Detecting_Fraud_Calls_vis-a-vis_Natural_Language_Processing
• Federal Trade Commission. 2025. New FTC data show a big jump in reported losses to fraud to $12.5 billion in 2024. March 2025. Available at: https://www.ftc.gov/news-events/news/press-releases/2025/03/new-ftc-data-show-big-jump-reported-losses-fraud-125-billion-2024 [Accessed 22 April 2025].
• Hong, B., Connie, T., and Goh, M. K. O. 2023. Scam calls detection using machine learning approaches. In Proceedings of the 11th International Conference on Information and Communication Technology (ICoICT), pp. 442–447.
• Hossain, M. I. n.d. Software development life cycle (SDLC) methodologies for information systems project management.
• IBM Think. 2024. What is NLP (natural language processing)? 11 August 2024. Available at: https://www.ibm.com/think/topics/natural-language-processing [Accessed 9 June 2025].
• Jiang, L. 2024. Detecting scams using large language models. arXiv. DOI: 10.48550/arXiv.2402.03147.
• Macháček, D., Dabre, R., and Bojar, O. 2023. Turning Whisper into a real-time transcription system. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 17–24. DOI: 10.18653/v1/2023.ijcnlp-demo.3.
• Miah, M. S. U., Kabir, M. M., Sarwar, T. B., Safran, M., Alfarhood, S., and Mridha, M. F. 2024. A multimodal approach to cross-lingual sentiment analysis with an ensemble of transformer and LLM. DOI: 10.1038/s41598-024-60210-7.
• Olorunshola, O. E., and Ogwueleka, F. N. 2022. Review of system development life cycle (SDLC) models for effective application delivery. In Joshi, A., Mahmud, M., Ragel, R. G., and Thakur, N. V. (Eds.), Information and Communication Technology for Competitive Strategies (ICTCS 2020), Vol. 191, pp. 281–289.
• RapidBTS. 2024. VoIP growth in Africa: Market size and forecast. Available at: https://rapidbts.ng/voip-growth-africa/ [Accessed 22 February 2025].
• Singh, G., Singh, P., and Singh, M. 2025. Advanced real-time fraud detection using RAG-based LLMs. Version 1. arXiv. DOI: 10.48550/ARXIV.2501.15290.
• Subudhi, S., and Panigrahi, S. 2018. A hybrid mobile call fraud detection model using optimized fuzzy C-means clustering and group method of data handling-based network. Vietnam Journal of Computer Science, 5(3), pp. 205–217. DOI: 10.1007/s40595-018-0116-x.
• Valarmathi, C., and Sharanya, S. 2024. Scam call detection using NLP and Naïve Bayes classifier. International Journal of Scientific Research in Engineering and Management, 8(07), pp. 1–6.
• Xing, J., Yu, M., Wang, S., Zhang, Y., and Ding, Y. 2020. Automated fraudulent phone call recognition through deep learning. Wireless Communications and Mobile Computing, 2020, pp. 1–9.
• Zhan, T., Shi, C., Shi, Y., Li, H., and Lin, Y. 2024. Optimization techniques for sentiment analysis based on LLM (GPT-3). arXiv. DOI: 10.48550/arXiv.2405.09770.
• Zhao, Q., Chen, K., Li, T., Yang, Y., and Wang, X. 2018. Detecting telecommunication fraud by understanding the contents of a call. Cybersecurity, 1(1), p. 8.