Machine Learning Pipelines: A Survey on Automation, Scalability, and Deployment Strategies

[{“box”:0,”content”:”n[if 992 equals=”Open Access”]n

n

n

n

Open Access

nn

n

n[/if 992]n[if 2704 equals=”Yes”]n

n

Notice

nThis is an unedited manuscript accepted for publication and provided as an Article in Press for early access at the author’s request. The article will undergo copyediting, typesetting, and galley proof review before final publication. Please be aware that errors may be identified during production that could affect the content. All legal disclaimers of the journal apply.n

n[/if 2704]n

n

Year : 2025 [if 2224 equals=””]08/09/2025 at 4:28 PM[/if 2224] | [if 1553 equals=””] Volume : 12 [else] Volume : 12[/if 1553] | [if 424 equals=”Regular Issue”]Issue : [/if 424][if 424 equals=”Special Issue”]Special Issue[/if 424] [if 424 equals=”Conference”][/if 424] 02 | Page : 17 28

n

n

nn

n

n

n

    By

    n

    [foreach 286]n

    n

    Sandeep Gupta,

    n t

  • n

    n[/foreach]

    n

n[if 2099 not_equal=”Yes”]n

    [foreach 286] [if 1175 not_equal=””]n t

  1. Senior Researcher, Department of Computer Science Engineering, Samrat Ashok Technological Institute (SATI), Vidisha, Madhya Pradesh, India

    2. n[/if 1175][/foreach]

n[/if 2099][if 2099 equals=”Yes”][/if 2099]n

n

Abstract

n

n

nMachine learning (ML) has become a critical enabler of intelligent applications across domains, requiring robust, efficient, and scalable deployment workflows. This review paper provides an in-depth overview of machine learning pipelines, emphasizing three key dimensions: automation, scalability, and deployment methodologies. It begins by exploring automation techniques that reduce manual effort in data ingestion, preprocessing, model selection, and hyperparameter tuning. Tools such as AutoML, TFX, and workflow orchestration platforms are examined for their role in streamlining repetitive tasks and improving consistency. The paper then delves into the scalability of ML pipelines, addressing horizontal and vertical scaling, resource optimization using Kubernetes and ML-driven autoscores, and the integration of real-time and streaming architectures. Finally, investigated modern deployment strategies, including model serving frameworks, CI/CD pipelines for continuous updates, and edge-hybrid architectures that ensure low latency, data privacy, and efficient resource use. The paper synthesizes current research trends, highlights challenges in integrating modular, scalable components, and discusses opportunities for enhancing MLOps practices. By surveying recent advancements and limitations, this work provides researchers and practitioners with valuable insights to design resilient and future-ready ML pipeline architectures suited for production environments.nn

n

n

n

Keywords: Machine learning pipelines, MLOps, automation, scalability, deployment strategies, CI/CD, edge computing

n[if 424 equals=”Regular Issue”][This article belongs to Journal of Advances in Shell Programming ]

n

[/if 424][if 424 equals=”Special Issue”][This article belongs to Special Issue under section in Journal of Advances in Shell Programming (joasp)][/if 424][if 424 equals=”Conference”]This article belongs to Conference [/if 424]

n

n

n

How to cite this article:
nSandeep Gupta. [if 2584 equals=”][226 wpautop=0 striphtml=1][else]Machine Learning Pipelines: A Survey on Automation, Scalability, and Deployment Strategies[/if 2584]. Journal of Advances in Shell Programming. 08/09/2025; 12(02):17-28.

n

How to cite this URL:
nSandeep Gupta. [if 2584 equals=”][226 striphtml=1][else]Machine Learning Pipelines: A Survey on Automation, Scalability, and Deployment Strategies[/if 2584]. Journal of Advances in Shell Programming. 08/09/2025; 12(02):17-28. Available from: https://journals.stmjournals.com/joasp/article=08/09/2025/view=0

nn

n

n[if 992 equals=”Open Access”]Full Text PDF[/if 992]n

n

n[if 992 not_equal=”Open Access”]n

n

n[/if 992]n

nn

nnn

n[if 379 not_equal=””]nn

Browse Figures

n

n

n[foreach 379]

figures

[/foreach]n

n

n

n[/if 379]

n

n

n

n

n

References n

n[if 1104 equals=””]n

  1. Liang P, Song B, Zhan X, Chen Z, Yuan J. Automating the training and deployment of models in MLOps by integrating systems with machine learning. arXiv preprint arXiv:2405.09819. 2024 May 16.
  2. Pahune S, Akhtar Z. Transitioning from MLOps to LLMOps: Navigating the unique challenges of large language models. Information. 2025;16(2):87.
  3. Neeli SS. Optimizing Database Management with DevOps: Strategies and Real-World Examples. J. Adv. Dev. Res. 2020;11(1):8.
  4. Singla A. Machine learning operations (MLOps): challenges and strategies. J Knowl Learn Sci Technol. 2023 Aug 14;2(3):333–40.
  5. Peter H. Cloud Native MLOps Automating Machine Learning Pipelines with Continuous Integration Continuous Deployment and Infrastructure as Code. 2024; https://www.researchgate.net/profile/ Harry-Peter/publication/392101895_Cloud_Native_MLOps_Automating_Machine_Learning_Pipe lines_with_Continuous_Integration_Continuous_Deployment_and_Infrastructure_as_Code/links/683500ed026fee1034fc2233/Cloud-Native-MLOps-Automating-Machine-Learning-Pipelines-with-Continuous-Integration-Continuous-Deployment-and-Infrastructure-as-Code.pdf
  6. Garg S. AI/ML Driven Proactive Performance Monitoring, Resource Allocation and Effective Cost Management in SAAS Operations. Int J Core Eng Manag. 2019;6(6):263–73.
  7. Shivashankar K, Hajj GS, Martini A. Scalability and Maintainability Challenges and Solutions in Machine Learning: Systematic Literature Review. arXiv preprint arXiv:2504.11079. 2025 Apr 15.
  8. Arora S, Thota SR, Gupta S. Artificial Intelligence-Driven Big Data Analytics for Business Intelligence in SaaS Products. In2024 First International Conference on Pioneering Developments in Computer Science & Digital Technologies (IC2SDT) 2024 Aug 2 (pp. 164–169). IEEE.
  9. Chitraju Gopal Varma S. Optimizing Machine Learning Pipelines: Best Practices for Scalable and Efficient Model Deployment. Available at SSRN 5226791. 2024 Nov 20.
  10. Malali N. Cloud-Native Security and Compliance in Life and Annuities Insurance: Challenges and Best Practices. International Journal of Interdisciplinary Research Methods (IJIRM). 2025;12(1):50–73.
  11. Pal G, Li G, Atkinson K. Big data real time ingestion and machine learning. In2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP) 2018 Aug 21 (pp. 25–31). IEEE.
  12. Thota SR, Arora S, Gupta S. Al-Driven Automated Software Documentation Generation for Enhanced Development Productivity. In2024 International Conference on Data Science and Network Security (ICDSNS) 2024 Jul 26 (pp. 1–7). IEEE.
  13. a Ilemobayo J, Durodola O, Alade O, Awotunde OJ, Olanrewaju AT, Falana O, Ogungbire A, Osinuga A, Ogunbiyi D, Ifeanyi A, Odezuligbo IE. Hyperparameter tuning in machine learning: a comprehensive review. J Eng Res Reports. 2024 Jun 7;26(6):388–95.
  14. Filippou K, Aifantis G, Papakostas GA, Tsekouras GE. Structure learning and hyperparameter optimization using an automated machine learning (AutoML) pipeline. Information. 2023 Apr 9;14(4):232.
  15. Pradhan AK, Kumar D, Mishra MK, Singh MK. Edge, Fog, and Cloud Computing in Industry 5.0. InIndustry 5.0: Key Technologies and Drivers 2025 Jul 18 (pp. 1–27). Cham: Springer Nature Switzerland.
  16. Berberi L, Kozlov V, Nguyen G, Sáinz-Pardo Díaz J, Calatrava A, Moltó G, Tran V, López García Á. Machine learning operations landscape: platforms and tools. Artif Intell Rev. 2025 Mar 17;58(6):167.
  17. Singh S. Unifying DevOps and MLOps Pipelines Via AI-driven Observability: A Mixed-Methods Study. Asian J Res Comput Sci. 2025 Jun 10;18(6):334–42.
  18. Nielsen MC, Ayvaz S. POE-ML: An Automated Pipeline for Optimization and Evaluation of Machine Learning. In2025 IEEE 22nd International Conference on Software Architecture Companion (ICSA-C) 2025 Mar 31 (pp. 508–515). IEEE.

nn[/if 1104][if 1104 not_equal=””]n

    [foreach 1102]n t

  1. [if 1106 equals=””], [/if 1106][if 1106 not_equal=””],[/if 1106]

    2. n[/foreach]

n[/if 1104]

n

nn[if 1114 equals=”Yes”]n

n[/if 1114]

n

n

[if 424 not_equal=””]Regular Issue[else]Published[/if 424] Subscription Review Article

n

n

[if 2146 equals=”Yes”][/if 2146][if 2146 not_equal=”Yes”][/if 2146]n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n

n[if 1748 not_equal=””]

[else]

[/if 1748]n

n[if 1746 equals=”Retracted”]n

n

n

n

[/if 1746]n[if 4734 not_equal=””]

n

n

n

[/if 4734]n

n

Volume 12
[if 424 equals=”Regular Issue”]Issue[/if 424][if 424 equals=”Special Issue”]Special Issue[/if 424] [if 424 equals=”Conference”][/if 424] 02
Received 20/07/2025
Accepted 24/07/2025
Published 08/09/2025
Retracted
Publication Time 50 Days

n

n

239

nn

n

Login

n
My IP
n

PlumX Metrics

nn

n

n

n[if 1746 equals=”Retracted”]n

[/if 1746]nnn

nnn”}]