Abstract

Large groups of autonomous agents, like mobile robots or drones, can work together to accomplish complex tasks in unpredictable and dynamic environments thanks to swarm coordination. The flexibility, scalability, and communication effectiveness of traditional rule-based or reinforcement-learning approaches are frequently hampered. To improve swarm coordination's robustness and scalability, this paper suggests a Deep Adaptive Learning (DAL) framework that combines attention-based communication, multi-agent reinforcement learning, and meta-adaptive learning. Reducing communication overhead and increasing coordination efficiency, each agent uses a deep neural policy network with a dynamic attention mechanism to selectively process pertinent neighbour information. Additionally, quick policy adaptation to environmental changes without complete retraining is made possible by an environment-change detection module in conjunction with meta-learning. In contrast to current methods, DAL offers a scalable solution for intelligent swarm systems by achieving faster convergence, higher cumulative rewards, and superior resilience to agent loss and communication noise, as demonstrated by experimental results from dynamic area coverage, target tracking, and formation-switching tasks.

References

• J. Kennedy and R. Eberhart, “Particle swarm optimization,” Proceedings of IEEE International Conference on Neural Networks, vol. 4, pp. 1942–1948, 1995.
• E. Şahin, “Swarm robotics: From sources of inspiration to domains of application,” in Swarm Robotics, Springer, Berlin, Heidelberg, pp. 10–20, 2005.
• L. Bayındır, “A review of swarm robotics tasks,” Neurocomputing, vol. 172, pp. 292–321, 2016.
• M. Dorigo, M. Birattari, and M. Brambilla, “Swarm robotics,” Scholarpedia, vol. 9, no. 1, pp. 1463, 2014.
• Y. Tan and Z. Zheng, “Research advance in swarm robotics,” Defence Technology, vol. 9, no. 1, pp. 18–39, 2013.
• C. W. Reynolds, “Flocks, herds, and schools: A distributed behavioral model,” ACM SIGGRAPH Computer Graphics, vol. 21, no. 4, pp. 25–34, 1987.
• T. Vicsek and A. Zafeiris, “Collective motion,” Physics Reports, vol. 517, no. 3–4, pp. 71–140, 2012.
• H. Hamann, Swarm Robotics: A Formal Approach, Springer, Cham, 2018.
• Rubenstein, A. Cornejo, and R. Nagpal, “Programmable self-assembly in a thousand-robot swarm,” Science, vol. 345, no. 6198, pp. 795–799, 2014.
• W. Li and C. G. Cassandras, “Distributed cooperative coverage control of sensor networks,” IEEE Transactions on Robotics and Automation, vol. 20, no. 4, pp. 691–695, 2004.
• M. Brambilla, E. Ferrante, M. Birattari, and M. Dorigo, “Swarm robotics: A review from the swarm engineering perspective,” Swarm Intelligence, vol. 7, pp. 1–41, 2013.
• V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
• L. Busoniu, R. Babuska, and B. De Schutter, “A comprehensive survey of multiagent reinforcement learning,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 38, no. 2, pp. 156–172, 2008.
• Y. Zhang, Y. Tian, and Y. Jin, “A survey on multi-agent reinforcement learning methods for cooperative games,” IEEE Transactions on Cybernetics, vol. 50, no. 4, pp. 1666–1681, 2020.
• S. Omidshafiei, J. Pazís, C. Amato, J. P. How, and J. Vian, “Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability,” in Proceedings of the 35th International Conference on Machine Learning (ICML), 2017.
• R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments,” in Advances in Neural Information Processing Systems (NeurIPS), pp. 6379–6390, 2017.
• T. Chu, J. Wang, L. Codeca, and Z. Li, “Multi-agent deep reinforcement learning for large-scale traffic signal control,” IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 3, pp. 1086–1095, 2020.
• J. Jiang, M. Dun, and Z. Lu, “Graph convolutional reinforcement learning,” Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
• R. Das, A. Gervet, and K. Narasimhan, “Tarmac: Targeted multi-agent communication,” Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
• L. Matignon, G. J. Laurent, and N. Le Fort-Piat, “Independent reinforcement learners in cooperative Markov games: A survey regarding coordination problems,” Knowledge Engineering Review, vol. 27, no. 1, pp. 1–31, 2012.
• H. Dong and J. Li, “Adaptive evolutionary reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 5, pp. 2074–2085, 2021.
• D. Ha and J. Schmidhuber, “World models,” arXiv preprint arXiv:1803.10122, 2018.
• C. W. Reynolds, “Flocks, herds, and schools: A distributed behavioral model,” ACM SIGGRAPH Computer Graphics, vol. 21, no. 4, pp. 25–34, 1987.
• M. Dorigo and T. Stützle, Ant Colony Optimization, MIT Press, Cambridge, MA, 2004.
• R. Poli, J. Kennedy, and T. Blackwell, “Particle swarm optimization: An overview,” Swarm Intelligence, vol. 1, no. 1, pp. 33–57, 2007.
• A. Martinoli, “Collective complexity out of individual simplicity,” in Proceedings of the International Conference on Autonomous Agents, pp. 568–573, 1999.
• G. Beni, “From swarm intelligence to swarm robotics,” in Swarm Robotics, Springer, Berlin, Heidelberg, pp. 1–9, 2005.
• M. Gauci, J. Chen, W. Li, T. J. Dodd, and R. Gross, “Self-organized aggregation without computation,” International Journal of Robotics Research, vol. 33, no. 8, pp. 1145–1161, 2014.
• H. Tanner, A. Jadbabaie, and G. Pappas, “Stable flocking of mobile agents, Part I: Fixed topology,” Proceedings of the 42nd IEEE Conference on Decision and Control (CDC), 2003.
• M. Schwager, D. Rus, and J. J. Slotine, “Decentralized, adaptive control for coverage with networked robots,” International Journal of Robotics Research, vol. 28, no. 3, pp. 357–375, 2009.
• R. Olfati-Saber, J. A. Fax, and R. M. Murray, “Consensus and cooperation in networked multi-agent systems,” Proceedings of the IEEE, vol. 95, no. 1, pp. 215–233, 2007.
• F. Bullo, J. Cortés, and S. Martínez, Distributed Control of Robotic Networks: A Mathematical Approach to Motion Coordination Algorithms, Princeton University Press, 2009.
• Z. Lin, B. Francis, and M. Maggiore, “Necessary and sufficient graphical conditions for formation control of unicycles,” IEEE Transactions on Automatic Control, vol. 50, no. 1, pp. 121–127, 2005.
• R. Olfati-Saber, “Flocking for multi-agent dynamic systems: Algorithms and theory,” IEEE Transactions on Automatic Control, vol. 51, no. 3, pp. 401–420, 2006.
• J. Cortés, S. Martínez, and F. Bullo, “Spatially-distributed coverage optimization and control with limited-range interactions,” ESAIM: Control, Optimisation and Calculus of Variations, vol. 11, pp. 691–719, 2005.
• Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, pp. 436–444, 2015.
• D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, pp. 484–489, 2016.
• A. Tampuu, T. Matiisen, D. Kodelja, K. Aru, J. Aru, and R. Vicente, “Multiagent cooperation and competition with deep reinforcement learning,” PLOS ONE, vol. 12, no. 4, e0172395, 2017.