Machine Learning Pipelines: A Survey on Automation, Scalability, and Deployment Strategies

Year : 2025 | Volume : 12 | Issue : 02 | Page : 17 28
    By

    Sandeep Gupta,

  1. Senior Researcher, Department of Computer Science Engineering, Samrat Ashok Technological Institute (SATI), Vidisha, Madhya Pradesh, India

Abstract

Machine learning (ML) has become a critical enabler of intelligent applications across domains, requiring robust, efficient, and scalable deployment workflows. This review paper provides an in-depth overview of machine learning pipelines, emphasizing three key dimensions: automation, scalability, and deployment methodologies. It begins by exploring automation techniques that reduce manual effort in data ingestion, preprocessing, model selection, and hyperparameter tuning. Tools such as AutoML, TFX, and workflow orchestration platforms are examined for their role in streamlining repetitive tasks and improving consistency. The paper then delves into the scalability of ML pipelines, addressing horizontal and vertical scaling, resource optimization using Kubernetes and ML-driven autoscores, and the integration of real-time and streaming architectures. Finally, investigated modern deployment strategies, including model serving frameworks, CI/CD pipelines for continuous updates, and edge-hybrid architectures that ensure low latency, data privacy, and efficient resource use. The paper synthesizes current research trends, highlights challenges in integrating modular, scalable components, and discusses opportunities for enhancing MLOps practices. By surveying recent advancements and limitations, this work provides researchers and practitioners with valuable insights to design resilient and future-ready ML pipeline architectures suited for production environments.

Keywords: Machine learning pipelines, MLOps, automation, scalability, deployment strategies, CI/CD, edge computing

[This article belongs to Journal of Advances in Shell Programming ]

How to cite this article:
Sandeep Gupta. Machine Learning Pipelines: A Survey on Automation, Scalability, and Deployment Strategies. Journal of Advances in Shell Programming. 2025; 12(02):17-28.
How to cite this URL:
Sandeep Gupta. Machine Learning Pipelines: A Survey on Automation, Scalability, and Deployment Strategies. Journal of Advances in Shell Programming. 2025; 12(02):17-28. Available from: https://journals.stmjournals.com/joasp/article=2025/view=225885


References

  1. Liang P, Song B, Zhan X, Chen Z, Yuan J. Automating the training and deployment of models in MLOps by integrating systems with machine learning. arXiv preprint arXiv:2405.09819. 2024 May 16.
  2. Pahune S, Akhtar Z. Transitioning from MLOps to LLMOps: Navigating the unique challenges of large language models. Information. 2025;16(2):87.
  3. Neeli SS. Optimizing Database Management with DevOps: Strategies and Real-World Examples. J. Adv. Dev. Res. 2020;11(1):8.
  4. Singla A. Machine learning operations (MLOps): challenges and strategies. J Knowl Learn Sci Technol. 2023 Aug 14;2(3):333–40.
  5. Peter H. Cloud Native MLOps Automating Machine Learning Pipelines with Continuous Integration Continuous Deployment and Infrastructure as Code. 2024; https://www.researchgate.net/profile/ Harry-Peter/publication/392101895_Cloud_Native_MLOps_Automating_Machine_Learning_Pipe lines_with_Continuous_Integration_Continuous_Deployment_and_Infrastructure_as_Code/links/683500ed026fee1034fc2233/Cloud-Native-MLOps-Automating-Machine-Learning-Pipelines-with-Continuous-Integration-Continuous-Deployment-and-Infrastructure-as-Code.pdf
  6. Garg S. AI/ML Driven Proactive Performance Monitoring, Resource Allocation and Effective Cost Management in SAAS Operations. Int J Core Eng Manag. 2019;6(6):263–73.
  7. Shivashankar K, Hajj GS, Martini A. Scalability and Maintainability Challenges and Solutions in Machine Learning: Systematic Literature Review. arXiv preprint arXiv:2504.11079. 2025 Apr 15.
  8. Arora S, Thota SR, Gupta S. Artificial Intelligence-Driven Big Data Analytics for Business Intelligence in SaaS Products. In2024 First International Conference on Pioneering Developments in Computer Science & Digital Technologies (IC2SDT) 2024 Aug 2 (pp. 164–169). IEEE.
  9. Chitraju Gopal Varma S. Optimizing Machine Learning Pipelines: Best Practices for Scalable and Efficient Model Deployment. Available at SSRN 5226791. 2024 Nov 20.
  10. Malali N. Cloud-Native Security and Compliance in Life and Annuities Insurance: Challenges and Best Practices. International Journal of Interdisciplinary Research Methods (IJIRM). 2025;12(1):50–73.
  11. Pal G, Li G, Atkinson K. Big data real time ingestion and machine learning. In2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP) 2018 Aug 21 (pp. 25–31). IEEE.
  12. Thota SR, Arora S, Gupta S. Al-Driven Automated Software Documentation Generation for Enhanced Development Productivity. In2024 International Conference on Data Science and Network Security (ICDSNS) 2024 Jul 26 (pp. 1–7). IEEE.
  13. a Ilemobayo J, Durodola O, Alade O, Awotunde OJ, Olanrewaju AT, Falana O, Ogungbire A, Osinuga A, Ogunbiyi D, Ifeanyi A, Odezuligbo IE. Hyperparameter tuning in machine learning: a comprehensive review. J Eng Res Reports. 2024 Jun 7;26(6):388–95.
  14. Filippou K, Aifantis G, Papakostas GA, Tsekouras GE. Structure learning and hyperparameter optimization using an automated machine learning (AutoML) pipeline. Information. 2023 Apr 9;14(4):232.
  15. Pradhan AK, Kumar D, Mishra MK, Singh MK. Edge, Fog, and Cloud Computing in Industry 5.0. InIndustry 5.0: Key Technologies and Drivers 2025 Jul 18 (pp. 1–27). Cham: Springer Nature Switzerland.
  16. Berberi L, Kozlov V, Nguyen G, Sáinz-Pardo Díaz J, Calatrava A, Moltó G, Tran V, López García Á. Machine learning operations landscape: platforms and tools. Artif Intell Rev. 2025 Mar 17;58(6):167.
  17. Singh S. Unifying DevOps and MLOps Pipelines Via AI-driven Observability: A Mixed-Methods Study. Asian J Res Comput Sci. 2025 Jun 10;18(6):334–42.
  18. Nielsen MC, Ayvaz S. POE-ML: An Automated Pipeline for Optimization and Evaluation of Machine Learning. In2025 IEEE 22nd International Conference on Software Architecture Companion (ICSA-C) 2025 Mar 31 (pp. 508–515). IEEE.
Regular Issue Subscription Review Article
Volume 12
Issue 02
Received 20/07/2025
Accepted 24/07/2025
Published 08/09/2025
Publication Time 50 Days
334

Login


My IP

PlumX Metrics