Comparison of Models of Machine Learning and Hyperparameter Optimization Methods on Various Datasets

Year : 2026 | Volume : 13 | Issue : 01 | Page : 35 42
By

Naviya Shetty,

Anuja Shinde,

Shiksha Dubey,

  1. Research Scholar, Department of Computer Science and Engineering, Thakur Institute of Management Studies, Career Development and Research, Mumbai, Maharashtra, India
  2. Research Scholar, Department of Computer Science and Engineering, Thakur Institute of Management Studies, Career Development and Research, Mumbai, Maharashtra, India
  3. Research Supervisor, Department of Computer Science and Engineering, Thakur Institute of Management Studies, Career Development and Research, Mumbai, Maharashtra, India

Abstract

The most likely phase in achieving powerful and robust machine learning models is probably the hyperparameter tuning step. The traditional exhaustive methods of search (grid search and others) ensure that the search space is covered, but are computationally inexpensive; random search is less expensive and can still miss good regions; and lastly, the modern model-based and population-based methods (Bayesian optimization, tree-structured Parzen estimator (TPE), genetic algorithms) are thought to provide better trade-offs between performance and resource utilization. This study performs a comparative study of five tuning algorithms (grid search, random search, Bayesian optimization, TPE/Optuna, and genetic algorithm) applied to a wide range of supervised models (Random Forest, XGBoost, support vector machine (SVM), multi-layer perceptron (MLP)) on a number of classification and regression data sets. To record the results, we configured the experiments to measure accuracy (or RMSE), running time, memory consumption, and interpret and visualize the results in one of the analytical dashboards of Streamlit. Based on the results, model-based optimizers (TPE and Bayesian) achieved near-optimal solutions at a fraction of the time and memory that the grid search method needed; thus is a suitable fit in low-resource environments. The paper outlines the replication procedure and stages and technical configurations of the dashboard and export/reporting pipeline (refer to project files in the code modules). The choice of hyperparameters, including learning rates, regularization coefficients, the depth of tree depth, and the number of estimators, etc., greatly affects the model’s generalization and efficiency. It is not always possible to run an exhaustive tuning of a large parameter grid when using complex models and multiple datasets; the importance of using more intelligent searching strategies increases again, particularly when they are limited to running the experiments on small Central Processing Units (CPUs), memory, or edge hardware. Here, we comparatively tune a range of tuning methods on smaller data and track model and wrap experiments, visualization, and statistical testing on a reusable Streamlit dashboard. We will arrive at solutions to both tuning strategies that provide a good trade-off between accuracy and computational cost and provide a well-documented, reproducible workflow to practitioners

Keywords: Streamlit, hyperparameter tuning, grid search, random search, Bayesian optimization, optuna

[This article belongs to Recent Trends in Programming languages ]

How to cite this article: Naviya Shetty, Anuja Shinde, Shiksha Dubey. Comparison of Models of Machine Learning and Hyperparameter Optimization Methods on Various Datasets. Recent Trends in Programming languages. 2026; 13(01):35-42.
How to cite this URL: Naviya Shetty, Anuja Shinde, Shiksha Dubey. Comparison of Models of Machine Learning and Hyperparameter Optimization Methods on Various Datasets. Recent Trends in Programming languages. 2026; 13(01):35-42. Available from: https://journals.stmjournals.com/rtpl/article=2026/view=242305

References

  1. Bergstra J, Yamins D, Cox D. Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. 30th International Conference on Machine Learning (ICML), Atlanta, GA, USA, 2013. p. 115–123.
  2. Snoek J, Larochelle H, Adams RP. Practical Bayesian optimization of machine learning algorithms [preprint]. 2012. arXiv:1206.2944. doi:10.48550/arXiv.1206.2944.
  3. Akiba T, Sano S, Yanase T, Ohta T, Koyama M. Optuna: a next-generation hyperparameter optimization framework. 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Anchorage, AK, USA, 2019. p. 2623–2631. doi:10.1145/3292500.3330701.
  4. Dasgupta S, Sen J. A comparative study of hyperparameter tuning methods [preprint]. 2024. arXiv:2408.16425. doi:10.48550/arXiv.2408.16425.
  5. Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A. Hyperband: a novel bandit-based approach to hyperparameter optimization. J Mach Learn Res. 2018;18:1–52.
  6. Feurer M, Hutter F. Hyperparameter optimization. In: Hutter F, Kotthoff L, Vanschoren J, editors. Automated machine learning. Cham: Springer; 2019. p. 3–33. doi:10.1007/978-3-030-05318-5_1.
  7. Yang L, Shami A. On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing. 2020;415:295–316. doi:10.1016/j.neucom.2020.07.061.
  8. Neutatz F, Lindauer M, Abedjan Z. AutoML in heavily constrained applications. VLDB J. 2024;33:957–979. doi:10.1007/s00778-023-00820-1.
  9. Bischl B, Binder M, Lang M, Pielok T, Richter J, Coors S, et al. Hyperparameter optimization: foundations, algorithms, best practices, and open challenges. WIREs Data Min Knowl Discov. 2023;13:e1484. doi:10.1002/widm.1484.
  10. Franceschi L, Donini M, Perrone V, Klein A, Archambeau C, Seeger M, et al. Hyperparameter optimization in machine learning. Found Trends Mach Learn. 2025;18(6):1054–1201. doi:10.1561/2200000088.
  11. Turner R, Eriksson D, McCourt M, Kiili J, Laaksonen E, Xu Z, et al. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: analysis of the black-box optimization challenge 2020 [preprint]. 2021. arXiv:2104.10201. doi:10.48550/arXiv.2104.10201.
  12. Violos J, Pagoulatou T, Tsanakas S, Tserpes K, Varvarigou T. Predicting resource usage in edge computing infrastructures with CNN and a hybrid Bayesian particle swarm hyper-parameter optimization model. In: Arai K, editor. Intelligent computing. Lecture Notes in Networks and Systems, vol 284. Cham: Springer; 2021. p. 523–534. doi:10.1007/978-3-030-80126-7_40.
  13. Klein A, Falkner S, Bartels S, Hennig P, Hutter F. Fast Bayesian optimization of machine learning hyperparameters on large datasets. 20th International Conference on Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, FL, USA, 2017. p. 528–536.
  14. Jaderberg M, Dalibard V, Osindero S, Czarnecki WM, Donahue J, Razavi A, et al. Population based training of neural networks [preprint]. 2017. arXiv:1711.09846. doi:10.48550/arXiv.1711.09846.
  15. Smith LN. A disciplined approach to neural network hyper-parameters: part 1—learning rate, batch size, momentum, and weight decay [preprint]. 2018. arXiv:1803.09820. doi:10.48550/arXiv.1803.09820.
Regular Issue Subscription Review Article
Volume 13
Issue 01
Received 10/03/2026
Accepted 02/04/2026
Published 30/04/2026
Publication Time 51 Days

Login

My IP

PlumX Metrics

Support