Intelligent Big Data Processing: Optimizing Analytics Pipelines Through Ensemble Machine Learning, Distributed Computing, and Adaptive AI Architectures

Year : 2026 | Volume : 13 | Issue : 02 | Page : 1 11
By

Ashok Kumar Sidar,

Fateh Bahadur Kunwar,

  1. Student, Department of Information Technology Greater Noida Institute of Technology, Uttar Pradesh, India
  2. Associate Professor & Dean, Department of Computer Science and Engineering, Greater Noida Institute of Technology, Uttar Pradesh, India

Abstract

Traditional data analytics methods are facing significant challenges due to the explosive rise of digital data. Global data generation is expected to reach nearly 175 zettabytes by 2025. This puts significant pressure on existing computing infrastructure. Conventional analytics methods often struggle with scalability, processing speed, and smart decision-making. Rule-based systems and standalone machine learning models frequently cannot handle the complexity and scale of modern big data environments effectively. To tackle these challenges, this research introduces the adaptive distributed analytics intelligence (ADAI) framework. This framework is a multi-layer architecture that combines distributed data processing with machine learning techniques. The first layer emphasizes distributed stream and batch processing, using technologies like Apache Spark and Apache Kafka to manage large-scale data pipelines efficiently. The second layer presents an ensemble machine learning system that merges gradient-boosted trees, deep neural networks, and Bayesian model averaging to improve prediction accuracy and reliability. The third part is the adaptive architecture controller (AAC), which dynamically distributes workloads across different processing modes by analyzing real-time system resources and query complexity. The performance of this framework was assessed using four big data benchmarks: the TPC-DS decision support workload, the Higgs physics dataset with 11 million samples, a financial fraud detection dataset containing 284,807 transactions, and an industrial IoT dataset with 2.4 billion sensor readings from 6,400 devices. Experimental results indicate that ADAI significantly improves analytics performance. The framework achieves up to 3.8 times higher throughput compared to standard Spark-based systems, increases classification F1 scores by 4.7 to 9.3 percentage points, and lowers energy consumption per query by about 31.7% through adaptive workload management. Additional fault tolerance tests show that the system maintains 97.4% processing throughput even when a cluster node fails, underscoring its reliability for real-world enterprise big data applications.

Keywords: Big data analytics, ensemble machine learning, Apache Spark, distributed computing, adaptive AI, stream processing, gradient boosting, Bayesian model averaging, IoT analytics, fault tolerance, energy-efficient AI

[This article belongs to Journal of Multimedia Technology & Recent Advancements ]

How to cite this article: Ashok Kumar Sidar, Fateh Bahadur Kunwar. Intelligent Big Data Processing: Optimizing Analytics Pipelines Through Ensemble Machine Learning, Distributed Computing, and Adaptive AI Architectures. Journal of Multimedia Technology & Recent Advancements. 2026; 13(02):1-11.
How to cite this URL: Ashok Kumar Sidar, Fateh Bahadur Kunwar. Intelligent Big Data Processing: Optimizing Analytics Pipelines Through Ensemble Machine Learning, Distributed Computing, and Adaptive AI Architectures. Journal of Multimedia Technology & Recent Advancements. 2026; 13(02):1-11. Available from: https://journals.stmjournals.com/jomtra/article=2026/view=259936

References

  1. Laney D. 3D data management: Controlling data volume, velocity and variety. META Group Research Note. 2001;6:1.
  2. Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters. Commun ACM. 2008;51(1):107-13. doi:10.1145/1327452.1327492.
  3. Zaharia M, Xin RS, Wendell P, Das T, Armbrust M, Dave A, et al. Apache Spark: A unified engine for big data processing. Commun ACM. 2016;59(11):56-65. doi:10.1145/2934664.
  4. Carbone P, Katsifodimos A, Ewen S, Markl V, Haridi S, Tzoumas K. Apache Flink: Stream and batch processing in a single engine. IEEE Data Eng Bull. 2015;38(4):28-38.
  5. Chen T, Guestrin C. XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016. p. 785-94. doi:10.1145/2939672.2939785.
  6. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, et al. LightGBM: A highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst. 2017;30.
  7. Breiman L. Random forests. Mach Learn. 2001;45(1):5-32. doi:10.1023/A:1010933404324.
  8. Hoeting JA, Madigan D, Raftery AE, Volinsky CT. Bayesian model averaging: A tutorial. Stat Sci. 1999;14:382-417.
  9. Avnur R, Hellerstein JM. Eddies: continuously adaptive query processing. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data; 2000; Dallas, TX, USA. New York (NY): Association for Computing Machinery; 2000. p. 261-72. doi:10.1145/342009.335420.
  10. Marcus R, Negi P, Mao H, Tatbul N, Alizadeh M, Kraska T. Bao: Making learned query optimization practical. In: Proceedings of the 2021 International Conference on Management of Data; 2021. p. 1275-88. doi:10.1145/3448016.3452838.
  11. Baldi P, Sadowski P, Whiteson D. Searching for exotic particles in high-energy physics with deep learning. Nat Commun. 2014;5:4308. doi:10.1038/ncomms5308. PMID:24986233.
  12. Gandomi A, Haider M. Beyond the hype: Big data concepts, methods, and analytics. Int J Inf Manag. 2015;35(2):137-44. doi:10.1016/j.ijinfomgt.2014.10.007.
  13. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms [Preprint]. 2017 Jul 20. arXiv:1707.06347.
  14. Bifet A, Holmes G, Pfahringer B, Kranen P, Kremer H, Jansen T, et al. MOA: Massive online analysis, a framework for stream classification and clustering. In: Proceedings of the First Workshop on Applications of Pattern Analysis. PMLR; 2010. p. 44-50.
  15. Wang H, Fan W, Yu PS, Han J. Mining concept-drifting data streams using ensemble classifiers. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2003. p. 226-35. doi:10.1145/956750.956778.
  16. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic minority over-sampling technique. J Artif Intell Res. 2002;16:321-57. doi:10.1613/jair.953.
  17. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, et al. Human-level control through deep reinforcement learning. Nature. 2015;518:529-33. doi:10.1038/nature14236. PMID:25719670.
  18. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:261-72.
  19. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436-44. doi:10.1038/nature14539. PMID:26017442.
Regular Issue Subscription Original Research
Volume 13
Issue 02
Received 24/06/2026
Accepted 30/07/2026
Published 15/08/2026
Publication Time 52 Days

Login

My IP

PlumX Metrics

Support