Abstract

In modern and future telecommunication systems—particularly within 5G-Advanced and emerging 6G networks—data transmission optimization remains a critical challenge in the context of the terahertz (THz) spectrum, ultra-reliable low-latency communications (URLLC), and massive Internet of Things (mMTC). Analytical models relying on traditional Shannon-Kotelnikov theory fall short in achieving fully efficient resource management and signal recovery under dynamic environments characterized by high fading, multipath propagation, and intense interference. This research provides a comprehensive analysis of the integration of artificial intelligence (AI) and machine learning (ML) algorithms across all stages of data transmission. It proposes a hybrid physical-statistical framework that merges the mathematical models of the physical layer with data-driven ML methods based on Edge AI. As part of this study, a 5G HetNet topology comprising macro-cells, pico-cells, and IoT nodes was simulated using NS-3 (Network Simulator 3). The performance of SVM, Decision Trees, K-means, and PCA was evaluated alongside reinforcement learning models (Q-learning, Deep Q-Networks - DQN). The obtained simulation results demonstrate that the hybrid ML approach, and specifically deep reinforcement learning (RL), enables reducing network latency by up to 22%, increasing overall throughput by 18%, raising the accuracy of anomaly and security attack detection to 94%, and significantly enhancing energy efficiency compared to traditional routing and resource allocation methods.

References

• Y. Wang, M. Liu, and X. Zhang, "Deep Learning-Based Physical Layer Signal Processing and Channel Estimation for 6G Wireless Networks," IEEE Transactions on Wireless Communications, vol. 22, no. 4, pp. 2415–2429, Apr. 2023.
• H. Zhou, J. Chen, and K. Smith, "Hybrid Model-Driven and Data-Driven Architecture for Next-Generation HetNets," IEEE Journal on Selected Areas in Communications, vol. 41, no. 8, pp. 2510–2524, Aug. 2023.
• R. Aliyev, T. Hasanov, and S. Mammadov, "Application of Artificial Intelligence Models and Channel Analysis in 5G and 6G Telecommunication Networks," Scientific Works of AzTU, vol. 34, no. 2, pp. 45–54, 2024. (in Azerbaijani)
• T. O’Shea and J. Hoydis, "An Introduction to Deep Learning at the Physical Layer," IEEE Transactions on Cognitive Communications and Networking, vol. 3, no. 4, pp. 563–575, Dec. 2017.
• L. Dai, R. Jiao, F. Adachi, H. V. Poor, and L. Hanzo, "Deep Learning for Wireless Communications: An Emerging Paradigm," IEEE Wireless Communications, vol. 27, no. 4, pp. 134–141, Aug. 2020.
• X. Zhang, Y. Huang, and L. Wu, "Model-Driven Deep Learning for Physical Layer Autoencoders in Optical and Wireless Communications," IEEE Communications Magazine, vol. 61, no. 3, pp. 88–95, Mar. 2023.
• J. Gao, M. Xu, and W. Wang, "Deep Reinforcement Learning for Dynamic Resource Allocation and Network Slicing in 5G-Advanced and 6G," IEEE Transactions on Network and Service Management, vol. 20, no. 2, pp. 1120–1134, Jun. 2023.
• K. B. Letaief, Y. Shi, J. Lu, and J. Lu, "The Roadmap to 6G: AI Empowered Wireless Networks," IEEE Communications Magazine, vol. 57, no. 8, pp. 84–90, Aug. 2019.
• F. Huseynov and E. Gasimov, "Signal Recovery and BER Performance Optimization Using Neural Networks in Non-Linear Communication Channels," Journal of Communication and Information Technologies, vol. 12, no. 1, pp. 18–27, 2023. (in Azerbaijani)
• S. Cammerer, F. A. Aoudia, J. Hoydis, and S. ten Brink, "Trainable Communication Systems: Concepts and Physical Layer Implementation," IEEE Journal on Selected Areas in Communications, vol. 38, no. 7, pp. 1566–1579, Jul. 2020.
• Z. M. Fadlullah, F. Tang, B. Mao, N. Kato, O. Akashi, T. Inoue, and K. Mizutani, "State-of-the-Art Deep Learning: Evolving Intelligent Applications for Future Communication Networks," IEEE Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2432–2451, 2017.
• C. Zhang, P. Patras, and H. Haddadi, "Deep Learning in Mobile and Wireless Networking: A Survey," IEEE Communications Surveys & Tutorials, vol. 21, no. 3, pp. 2224–2287, 2019.
• Y. Li, M. Sheng, and J. Li, "Resource Allocation and Beamforming Optimization for Non-Terrestrial Networks (NTN) in 6G," IEEE Transactions on Vehicular Technology, vol. 72, no. 9, pp. 11845–11858, Sep. 2023.
• A. Zappone, M. Di Renzo, and M. Debbah, "Wireless Networks Design in the Era of Deep Learning: Model-Based, Data-Driven, and Hybrid Approaches," IEEE Transactions on Communications, vol. 67, no. 8, pp. 5473–5495, Aug. 2019.
• N. Rajabov and H. Ahmadov, "Machine Learning Approaches for Channel State Information (CSI) Compression and Estimation in Massive MIMO and Sub-THz Networks," News of Azerbaijan Engineering Academy, vol. 15, no. 3, pp. 62–71, 2023. (in Azerbaijani)