Manas Kumar Yogi
- Assistant Professor, Department of Computer Science and Engineering, Pragati Engineering College (A), Surampalem, Andhra Pradesh, India
Abstract
The integration of parallel computing into the realm of big data analytics promises accelerated processing speeds and enhanced scalability, but it is not without its formidable challenges. This abstract explores the multifaceted hurdles faced in the pursuit of efficient parallel processing for large-scale data analytics. The intricate task of distributing and partitioning massive datasets across multiple processing units demands adept strategies to ensure equitable workloads. Load balancing emerges as a critical concern to prevent bottlenecks and maximize parallelism, necessitating dynamic mechanisms for workload distribution. Communication overhead, a ubiquitous challenge in distributed systems, requires thoughtful optimization to minimize latency and enhance overall efficiency. Synchronization complexities demand a delicate balance to maintain data consistency without sacrificing performance. Scalability issues arise with the increasing size of datasets or processing units, demanding the adoption of scalable frameworks like Apache Hadoop and Spark. Fault tolerance becomes paramount in the face of hardware or software failures, urging the implementation of robust recovery mechanisms. Algorithm design, heterogeneous architectures, energy efficiency, and data locality further contribute to the intricate tapestry of challenges in parallel computing for big data analytics. A comprehensive understanding of these challenges is essential for researchers and practitioners to devise innovative solutions, paving the way for more effective and sustainable parallel processing in the era of big data.
Keywords: Big Data, Analytics, Parallel Computing, Parallelism, Massive Dataset
[This article belongs to Recent Trends in Parallel Computing(rtpc)]
References
1. Alias N, Suhari NN, Saipol HF, Dahawi AA, Saidi MM, Hamlan HA, Teh CR. Parallel computing of numerical schemes and big data analytic for solving real life applications. Jurnal Teknologi. 2016 Jan 1;78:8–2.
2. Dobre C, Xhafa F. Parallel programming paradigms and frameworks in big data era. International Journal of Parallel Programming. 2014 Oct;42(5):710–38.
3. Zhang Y, Cao T, Li S, Tian X, Yuan L, Jia H, Vasilakos AV. Parallel processing systems for big data: a survey. Proceedings of the IEEE. 2016 Aug 19;104(11):2114–36.
4. Ahmad A, Paul A, Din S, Rathore MM, Choi GS, Jeon G. Multilevel data processing using parallel algorithms for analyzing big data in high-performance computing. International Journal of Parallel Programming. 2018 Jun;46:508–27.
5. Wei CC, Chou TH. Typhoon quantitative rainfall prediction from big data analytics by using the apache hadoop spark parallel computing framework. Atmosphere. 2020 Aug 17;11(8):870.
6. Wu Z, Sun J, Zhang Y, Wei Z, Chanussot J. Recent developments in parallel and distributed computing for remotely sensed big data processing. Proceedings of the IEEE. 2021 Jun 17;109(8):1282–305.
7. Jung G, Gnanasambandam N, Mukherjee T. Synchronous parallel processing of big-data analytics services to optimize performance in federated clouds. In 2012 IEEE Fifth International Conference on Cloud Computing 2012 Jun 24 (pp. 811–818). IEEE.
8. Kambatla K, Kollias G, Kumar V, Grama A. Trends in big data analytics. Journal of parallel and distributed computing. 2014 Jul 1;74(7):2561–73.
9. Naeem M, Jamal T, Diaz-Martinez J, Butt SA, Montesano N, Tariq MI, De-la-Hoz-Franco E, De-La-Hoz-Valdiris E. Trends and future perspective challenges in big data. In Advances in Intelligent Data Analysis and Applications: Proceeding of the Sixth Euro-China Conference on Intelligent Data Analysis and Applications, 15–18 October 2019, Arad, Romania 2022 (pp. 309–325). Springer Singapore.
10. Choudhury T, Chhabra AS, Kumar P, Sharma S. A recent trends on big data analytics. In2016 International Conference System Modeling & Advancement in Research Trends (SMART) 2016 Nov 25 (pp. 225–231). IEEE.
Recent Trends in Parallel Computing
| Volume | 11 |
| Issue | 01 |
| Received | January 9, 2024 |
| Accepted | February 21, 2024 |
| Published | March 28, 2024 |