Automated Machine Learning System for Model Selection and Hyperparameter Optimization

Year : 2026 | Volume : 03 | Issue : 02 | Page : 15 23
By

Ankit Maurya,

Shiraj Ahmad,

  1. Student, Greater Noida Institute of Technology Greater Noida, Uttar Pradesh, India
  2. Assistant Professor, Greater Noida Institute of Technology Greater Noida, Uttar Pradesh, India

Abstract

The proliferation of machine learning applications in various scientific and industrial domains has given rise to an urgent need for developing principled, automated techniques for optimal architecture selection and hyperparameter tuning for machine learning models without human expert intervention. In this paper, we introduce the Automated Machine Learning System for Model selection and hyperparameter Optimization (AMLSMO)—a state-of-the-art, all-encompassing AutoML system that combines the power of meta-learning-based warm-starting, Bayesian Optimization with Hyperband Scheduling, and stochastic ensemble construction to solve the combined algorithm selection and hyperparameter optimization (CASH) problem for diverse search spaces with over 30 different base learner families, including gradient boosting, deep neural networks, support vector machines, and attention-based transformers, among others. The system uses a meta-feature extractor based on various dataset features, such as statistical moments, landmarking metrics, and information-theoretic measures, to warm-start the Bayesian optimizer with an optimal initial solution, thus overcoming the cold-start efficiency problems that were common in all previous AutoML systems. The system uses a multi-fidelity successive halving approach with support for parallel execution to optimize hyperparameters for deep neural architectures without training from scratch with full budget. In this study, we perform an exhaustive evaluation of our system across four different benchmark sets, including CIFAR-10 (computer vision), ImageNet (large-scale vision), UCI Adult (tabular classification), and our custom multi-domain NLP benchmark, achieving state-of-the-art accuracy rates of 96.4%, 83.7%, 93.2%, and 91.8%, respectively, while outperforming state-of-the-art AutoML systems, including Auto-Sklearn 2.0, SMAC3, and DARTS, by margins ranging from 3.3 to 8.1%. In addition, we perform cross-domain transferability tests to demonstrate our system’s robust generalization capabilities across different data modalities. To further demonstrate our system’s efficiency, we perform ablation tests for each architectural module, including our meta-learning initializer, Bayesian optimizer with hyperband scheduling, and ensemble stacking module. The system is reproducible, open-source, and can be run with minimal modifications from any workstation with a single GPU to large-scale Kubernetes clusters.

Keywords: Automated machine learning, hyperparameter optimization, model selection, bayesian optimization, meta-learning, BOHB, neural architecture search, ensemble learning, cash problem, auto ML

[This article belongs to Recent Trends in Mathematics ]

How to cite this article: Ankit Maurya, Shiraj Ahmad. Automated Machine Learning System for Model Selection and Hyperparameter Optimization. Recent Trends in Mathematics. 2026; 03(02):15-23.
How to cite this URL: Ankit Maurya, Shiraj Ahmad. Automated Machine Learning System for Model Selection and Hyperparameter Optimization. Recent Trends in Mathematics. 2026; 03(02):15-23. Available from: https://journals.stmjournals.com/rtm/article=2026/view=252053

References

  1. Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res. 2012;13:281-305.
  2. Bergstra J, Bardenet R, Bengio Y, Kégl B. Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems (NeurIPS). 2011;24.
  3. Brazdil P, Soares C, Da Costa JP. Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Mach Learn. 2003;50(3):251-77.
  4. Caruana R, Niculescu-Mizil A, Crew G, Ksikes A. Ensemble selection from libraries of models. In: Proceedings of the 21st International Conference on Machine Learning (ICML); 2004. p. 18-26.
  5. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2009. p. 248-55.
  6. Falkner S, Klein A, Hutter F. BOHB: Robust and efficient hyperparameter optimization at scale. In: Proceedings of the 35th International Conference on Machine Learning (ICML); 2018. p. 1437-46.
  7. Feurer M, Klein A, Eggensperger K, Springenberg JT, Blum M, Hutter F. Efficient and robust automated machine learning. In: Advances in Neural Information Processing Systems (NeurIPS). 2015;28.
  8. Feurer M, Eggensperger K, Falkner S, Lindauer M, Hutter F. Auto-Sklearn 2.0: Hands-free AutoML via meta-learning. arXiv. 2020. Available from: https://arxiv.org/abs/2007.04074
  9. Hutter F, Hoos HH, Leyton-Brown K. Sequential model-based optimization for general algorithm configuration. In: Proceedings of the 5th International Conference on Learning and Intelligent Optimization (LION); 2011. p. 507-23.
  10. Hutter F, Hoos HH, Leyton-Brown K. An efficient approach for assessing hyperparameter importance. In: Proceedings of the 31st International Conference on Machine Learning (ICML); 2014. p. 754-62.
  11. Krizhevsky A. Learning multiple layers of features from tiny images. Toronto (ON): University of Toronto; 2009. Technical Report.
  12. Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A. Hyperband: A novel bandit-based approach to hyperparameter optimization. J Mach Learn Res. 2018;18(185):1-52.
  13. Liu H, Simonyan K, Yang Y. DARTS: Differentiable architecture search. In: Proceedings of the International Conference on Learning Representations (ICLR); 2019.
  14. Olson RS, Moore JH. TPOT: A tree-based pipeline optimization tool for automating machine learning. In: Proceedings of the AutoML Workshop at the International Conference on Machine Learning (ICML); 2016.
  15. Real E, Moore S, Selle A, Saxena S, Suematsu YL, Tan J, et al. Large-scale evolution of image classifiers. In: Proceedings of the 34th International Conference on Machine Learning (ICML); 2017.
  16. Thornton C, Hutter F, Hoos HH, Leyton-Brown K. Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD); 2013. p. 847-55.
  17. Vanschoren J, van Rijn JN, Bischl B, Torgo L. OpenML: Networked science in machine learning. ACM SIGKDD Explor Newsl. 2013;15(2):49-60.
  18. Wistuba M, Schilling N, Schmidt-Thieme L. Hyperparameter optimization machines. In: Proceedings of the IEEE International Conference on Data Science and Advanced Analytics (DSAA); 2015.
  19. Yao Q, Wang M, Chen Y, Dai W, Li YF, Tu WW, et al. Taking human out of learning applications: A survey on automated machine learning. arXiv. 2018. Available from: https://arxiv.org/abs/1810.13306
  20. Zoph B, Le QV. Neural architecture search with reinforcement learning. In: Proceedings of the International Conference on Learning Representations (ICLR); 2017.
Regular Issue Subscription Original Research
Volume 03
Issue 02
Received 06/07/2026
Accepted 17/07/2026
Published 28/07/2026
Publication Time 22 Days

Login

My IP

PlumX Metrics