Vanisha Mavi,
- Research Scholar, Department of CSE, Shobhit Institute of Engineering & Technology, (Deemed to be University), Uttar Pradesh, India
Abstract
In today’s data-driven era, efficiently handling vast amounts of information has become increasingly important. Data compression plays a vital role in this regard—it is essentially a method of encoding information in such a way that significantly reduces the number of bits required to store or transmit a file. By shrinking data to its most compact form, compression techniques help save storage space, reduce bandwidth consumption, and improve the overall speed of data transfer. A wide variety of data compression techniques exist today, operating across both online and offline environments. However, the sheer number of available options makes it genuinely challenging for practitioners and researchers to determine which technique best fits their specific needs. Every method has its own set of trade-offs regarding speed, compression efficiency, and computational complexity. In this paper, we explore and compare several key techniques used to compress and decompress text data, aiming to give readers a clearer understanding of how these methods work in practice. Beyond compression alone, we also take a closer look at Hadoop—one of the most widely adopted frameworks for distributed data processing—covering its design philosophy, development journey, and where it currently stands in real-world deployment. Additionally, we provide a comprehensive overview of YARN (Yet Another Resource Negotiator), the resource management layer within the Hadoop ecosystem. In this paper, we discuss its core concepts, the challenges that arise during its use, and the various tools and techniques that can be leveraged alongside it to build more efficient and scalable data pipelines.
Keywords: Hadoop, MapReduce, YARN, data compression, textual substitution
[This article belongs to International Journal of Algorithms Design and Analysis Review ]
References
- Mo Z, Li Y. Research of Big Data based on the views of technology and application. Am J Ind Bus Manag. 2015;5(4):192-7. doi:10.4236/ajibm.2015.54021.
- Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, et al. Apache Hadoop YARN: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing. New York (NY): Association for Computing Machinery; 2013. p.1–16. doi:10.1145/2523616.2523633.
- The Apache Software Foundation. (2026). Tez Incubation Status. [Online] Apache Incubator. Available from: https://incubator.apache.org/projects/tez.html.
- Mavi V, Tyagi N. Apache Hadoop’s second generation: Performance and resource management through YARN. Int J Eng Sci Humanit. 2023;13:52–60.
- Bali N, Chaudhary D, Singla A. Big data and its applications. J Technol Manag Grow Econ. 2020;11:63–67. doi:10.15415/jtmge.2020.112008.
- Lama P, Zhou X. AROMA: automated resource allocation and configuration of MapReduce environment in the cloud. In: Proceedings of the 9th International Conference on Autonomic Computing. New York (NY): Association for Computing Machinery; 2012. p.63–72. doi:10.1145/2371536.2371547.
- Saha B, Shah H, Seth S, Vijayaraghavan G, Murthy A, Curino C. Apache Tez: A unifying framework for modeling and building data processing applications. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. New York (NY): Association for Computing Machinery; 2015. p. 1357–1369. doi:10.1145/2723372.2742790.
- Kaur R, Goyal M. A survey on the different text data compression techniques. Int J Adv Res Comput Eng Technol. 2013;2:711–714.
- Shanmugasundaram S, Lourdusamy R. A comparative study of text compression algorithms. Int J Wisdom Based Comput. 2011;1:68–76.
- Singh A, Bhatnagar Y. Enhancement of data compression using incremental encoding. Int J Sci Eng Res. 2012;3:1–5.
- Altarawneh H, Altarawneh M. Data compression techniques on text files: A comparison study. Int J Comput Appl. 2011;26:42–54. doi:10.5120/3097-4249.
- Kaur H. A review of data compression techniques and data compression symmetry. Int J Comput Sci Inf Technol. 2013;4:183–186.
- Sharma A, Vyas S. Hadoop2 YARN. IPASJ Int J Comput Sci. 2015;3:30–34.
- Bhosale HS, Gadekar DP. A review paper on big data and Hadoop. Int J Sci Res Publ. 2014;4:1–7.
- Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters. Commun ACM. 2008;51:107–113. doi:10.1145/1327452.1327492.
- Grover M, Malaska T, Seidman J, Shapira G. Hadoop Application Architectures: Designing Real-World Big Data Applications. Sebastopol (CA): O’Reilly Media; 2015.
- Hashem IAT, Anuar NB, Gani A, Yaqoob I, Xia F, Khan SU. MapReduce: Review and open challenges. Scientometrics. 2016;109:389–422. doi:10.1007/s11192-016-1945-y.
- Park K, Peng L. A design of high-speed big data query processing system for social data analysis: Using Spark SQL. Int J Appl Eng Res. 2016;11:8221–8225.
- Karun AK, Chitharanjan K. A review on Hadoop—HDFS infrastructure extensions. In: 2013 IEEE Conference on Information & Communication Technologies (ICT). Thuckalay, India. 2013. p.132–137. doi:10.1109/CICT.2013.6558077.
- Perwej Y, Kerim B, Adrees MS, Sheta OE. An empirical exploration of the YARN in big data. Int J Appl Inf Syst. 2017;12:19–29. doi:10.5120/ijais2017451730.
International Journal of Algorithms Design and Analysis Review
| Volume | 04 | |
| Issue | 02 | |
| Received | 15/04/2026 | |
| Accepted | 29/04/2026 | |
| Published | 02/05/2026 | |
| Publication Time | 17 Days |