Abstract

This paper presents an adaptive multi-objective reinforcement learning (MORL) framework integrating classical Deep Q-Networks (DQN) and Variational Quantum Circuit DQN (VQC-DQN). A modified CartPole environment provides two reward signals combined through an adaptive weighting mechanism that updates every 10 episodes and converges in 14 of 15 runs; a corrected replay buffer rescalarizes stored transitions with the current weight instead of a stale one. Across five weight configurations, classical DQN achieves the highest whole-training balance reward, while a parameter-matched Tiny-Classical baseline (37 parameters) attains the best normalized hypervolume (0.412 vs. 0.376 for VQC-DQN and 0.262 for classical); VQC-DQN outperforms Tiny-Classical on raw balance in 3 of 5 weight settings. Retraining VQC-DQN for 200 episodes instead of 150 increases near-convergence balance by 32–163%. At 200 episodes, VQC-DQN has higher point-estimate near-convergence balance than classical DQN across the tested weight settings; however, this comparison is suggestive rather than statistically established because of unequal seed counts, a single 20-episode evaluation window, and unavailable per-seed variance for this metric. A qubit-count sweep under a matched gradient-step budget finds no evidence of barren plateaus within the tested 2–10 qubit range (gradient norm increases, not decreases, with qubit count), while a Deep Sea Treasure benchmark shows classical DQN fails under extreme weights whereas VQC-DQN consistently reaches a treasure but converges to the shallowest policy. These results suggest that classical-versus-quantum comparisons are budget-dependent rather than absolute, pending variance-controlled, multi-seed confirmation.

References

• Sutton, R. S. and Barto, A. G. 2018. Reinforcement Learning: An Introduction, 2nd ed. MIT Press, Cambridge, MA.
• Mnih, V. et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (Feb. 2015), 529-533.
• Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. 2017. Deep reinforcement learning: A brief survey. IEEE Signal Process. Mag. 34, 6 (Nov. 2017), 26-38.
• Roijers, D. M., Whiteson, S., and Oliehoek, F. A. 2013. A survey of multi-objective sequential decision-making. J. Artif. Intell. Res. 48 (2013), 67-113.
• Hayes, C. F. et al. 2022. A practical guide to multi-objective reinforcement learning and planning. Auton. Agents Multi-Agent Syst. 36, 1 (Apr. 2022), Article 26.
• Basaklar, T., Gumussoy, S., and Ogras, U. Y. 2023. PD-MORL: Preference-driven multi-objective reinforcement learning algorithm. In Proceedings of the International Conference on Learning Representations (ICLR 2023).
• Chen, S. Y.-C., Yang, C.-H. H., Qi, J., Chen, P.-Y., Ma, X., and Goan, H.-S. 2020. Variational quantum circuits for deep reinforcement learning. IEEE Access 8 (2020), 141007-141024.
• Lockwood, O. and Si, M. 2020. Reinforcement learning with quantum variational circuit. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 16 (2020), 245-251.
• Skolik, A., Jerbi, S., and Dunjko, V. 2022. Quantum agents in the Gym: A variational quantum algorithm for deep Q-learning. Quantum 6 (May 2022), 720.
• Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), 1861-1870.
• Dulac-Arnold, G. et al. 2019. Challenges of real-world reinforcement learning. arXiv:1904.12901.
• Van Moffaert, K. and Nowe, A. 2014. Multi-objective reinforcement learning using sets of Pareto dominating policies. J. Mach. Learn. Res. 15 (2014), 3483-3512.
• Vamplew, P., Dazeley, R., Berry, A., Issabekov, R., and Dekker, E. 2011. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine Learning 84, 1-2 (2011), 51-80.
• Biamonte, J. et al. 2017. Quantum machine learning. Nature 549 (Sep. 2017), 195-202.
• Cerezo, M. et al. 2021. Variational quantum algorithms. Nat. Rev. Phys. 3 (2021), 625-644.
• Jerbi, S. et al. 2021. Parametrized quantum policies for reinforcement learning. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021).
• Abdelkhalek, A. et al. 2023. Quantum deep reinforcement learning. PRX Quantum 4, 2 (2023).
• Wang, S. et al. 2023. Exploring quantum advantage in reinforcement learning. Quantum Inf. Process. (2023).
• Schuld, M. and Petruccione, F. 2018. Supervised Learning with Quantum Computers. Springer, Berlin, Germany.
• Huang, H.-Y. et al. 2022. Quantum advantage in learning from experiments. Science 376, 6598 (2022), 1182-1186.