Abstract

Enterprises increasingly adopt multi cloud architectures to take advantage of diverse database engines, regional availability, and cost models. In these environments, ETL pipelines must process large, distributed datasets while minimizing latency and transfer cost. Push down optimization, which executes transformation logic within database engines rather than within the ETL tool, has proven highly effective in single cloud systems. However, when applied across multiple clouds, it faces challenges related to data movement, heterogeneous SQL engines, orchestration complexity, and fragmented security controls. This paper examines the feasibility of push down optimization in multi cloud ETL pipelines and analyzes its benefits and limitations. It evaluates localized push down, hybrid models, and data federation techniques that reduce cross cloud traffic while improving performance. A case study across Redshift and BigQuery demonstrates measurable gains, including lower end to end runtime, reduced transfer volume, and improved cost efficiency. The study highlights practical strategies that organizations can adopt to improve ETL scalability and reliability in distributed cloud environments.

References

• D. Sitaram, S. Harwalkar, C. Sureka, H. Garg, M. Dinesh, M. Kejriwal, S. Gupta, and V. Kapoor, “Orchestration based hybrid or multi clouds and interoperability standardization,” in Proc. 2018 IEEE Int. Conf. Cloud Comput. Emerg. Markets (CCEM), 2018, pp. 67–71. doi: 10.1109/CCEM.2018.00018.
• W. Gao, Y. Wen and H. Zhang, “An Optimization Method of Federated Database Join Query Based on Computational Push-Down”, 2024 IEEE 2nd International Conference on Control, Electronics and Computer Technology (ICCECT), Jilin, China, 2024, pp. 225-229, doi: 10.1109/ICCECT60629.2024.10545893.
• A. I. Saada, G. A. El Khayat and S. K. Guirguis, “Cloud computing based ETL technique usingWarehouse Intermediate Agents,” The 2011 International Conference on Computer Engineering and Systems, Cairo, Egypt, 2011, pp. 301-306, doi: 10.1109/ICCES.2011.6141060.
• P. K. Veerapaneni, “Real-time data transformation in modern ETL pipelines: A shift towards streaming architectures,” International Journal of Research in Computer Applications and Information Technology (IJRCAIT), vol. 6, no. 1, pp. 121– 132, 2023.
• S. G. Aarella, V. P. Yanambaka, S. P. Mohanty, and E. Kougianos, “Fortified-Edge 2.0: Advanced Machine- Learning-Driven Framework for Secure PUF-Based Authentication in Collaborative Edge Computing,” Future Internet, vol. 17, p. 272, 2025. doi: 10.3390/fi17070272.
• Qamar Nomani; Julie Davila; Rehman Khan, Mastering Cloud Security Posture Management (CSPM): Secure multicloud infrastructure across AWS, Azure, and Google Cloud using proven techniques , Packt Publishing, 2024.
• A. Celesti, F. Tusa, M. Villari, and A. Puliafito, “How to enhance cloud architectures to enable cross-federation,” in Proc. 2010 IEEE 3rd Int. Conf. Cloud Comput., 2010, pp. 337–345. doi: 10.1109/CLOUD.2010.46.
• V. Punniyamoorthy, A. G. Parthi, M. Palanigounder, R. K. Kodali, B. Kumar, and K. Kannan, “A Privacy- Preserving Cloud Architecture for Distributed Machine Learning at Scale,” International Journal of Engineering Research and Technology (IJERT), vol. 14, no. 11, Nov. 2025.
• B. K. Malamuthu, V. S. Pandi, S. D, A. H. Jaber, J. Giri and R. Y. P, ”Examining Multi-Cloud Architectures to Provide Enterprises with Resilient, Scalable, and Economical Cloud Computing Solutions,” 2025 International Conference on Engineering Innovations and Technologies (ICoEIT), Bhopal, India, 2025, pp. 970-975, doi: 10.1109/ICoEIT63558.2025.11211532.
• A. Nagpal, B. Pothineni, A. G. Parthi, D. Maruthavanan, A. R. Banarse, P. K. Veerapaneni, S. R. Sankiti, and V. Jayaram, “Framework for automating compliance verification in CI/CD pipelines,” International Journal of Computer Science and Information Technology Research (IJCSITR), vol. 5, no. 4, pp. 17–27, 2024. doi: 10.5281/zenodo.1425967.
• S. Gupta, M. Sundararamaiah, and G. Geeta, “Leveraging cloud-native data engineering for big data analytics,” in Proc. 2025 3rd Int. Conf. Adv. Comput. Comput. Technol. (InCACCT), 2025, pp. 976–979. doi: 10.1109/In- CACCT65424.2025.11011292.
• J. Levandoski, G. Casto, M. Deng, R. Desai, P. Edara, T. Hottelier, A. Hormati, A. Johnson, J. Johnson, D. Kurzyniec, S. McVeety, P. Ramanathan, G. Saxena, V. Shanmugan, and Y. Volobuev, “BigLake: BigQuery’s evolution toward a multicloud lakehouse,” in Proc. Companion Int. Conf. Management of Data (SIGMOD ’24), Santiago, Chile, 2024, pp. 334– 346, doi: 10.1145/3626246.3653388.
• A. Gupta, D. Agarwal, D. Tan, J. Kulesza, R. Pathak, S. Stefani, and V. Srinivasan, “Amazon Redshift and the case for simpler data warehouses,” in Proc. ACM SIGMOD Int. Conf. Management of Data (SIGMOD ’15), Melbourne, Australia, 2015, pp. 1917–1923, doi: 10.1145/2723372.2742795.
• J. Aguilar-Saborit et al., “POLARIS: The distributed SQL engine in Azure Synapse,” in Proc. Int. Conf. Very Large Data Bases (VLDB), 2020, pp. 3204–3216.