Jahanvi Maheshwari,
Naveen Hemrajani,
Seema Sharma,
- Student, Department of Computer Science and Engineering, JECRC University, Jaipur, Rajasthan, India
- Professor and Dean, Department of Computer Science and Engineering, JECRC University, Jaipur, Rajasthan, India
- Assistant Professor, Department of Computer Science and Engineering, JECRC University, Jaipur, Rajasthan, India
Abstract
The fast proliferation of large language models (LLMs) has increased the need to optimize the process of fine-tuning, but the existing workflows that require a graphics processing unit (GPU) are still expensive, intensive, and unavailable to most researchers. This paper is driven by the desire to have a more cost-efficient and democratized version by examining a CPU-efficient implementation of parameter-efficient fine-tuning (PEFT) based on low-rank adaptation (LoRA). The major purpose of the study is to find out whether CPUs with low-rank matrix updates, mixed-precision quantization, SIMD vectorization, operator fusion, and NUMA-sensitive scheduling can be as precise as GPU-based LoRA at a lower computational cost. The suggested methodology presents a full CPU-LoRA pipeline that consists of selective transformation to the layer of transformers, quantizing from 4-bit to 8-bit, implementing matrix operations using high-performance basic linear algebra subprogram (BLAS) libraries, and using dynamic memory efficient batching. Open-source LLMs, including large language model meta AI (LLaMA), Falcon, and Mistral, were experimented on with standard natural language processing (NLP) datasets. The measures of evaluation were accuracy, perplexity, F1/ Recall-oriented understudy for gisting evaluation (ROUGE) measures, throughput, memory consumption, and power. Empirical evidence shows that CPU-optimized LoRA can be trained with close to 92–98% accuracy levels of GPU-LoRA and has considerable practical advantages: up to 70–90% memory reductions and massive operational and energy savings. According to the study, critical CPU bottlenecks are also discovered, and dedicated optimizations are provided to enable CPU-based fine-tuning as a practical alternative to small institutions and researchers who lack access to GPUs. All in all, the results support the idea that CPU-optimized LoRA is a viable, economically effective, and scalable option in terms of customization of LLM, which can further the larger objective of making high-quality model adaptation more accessible and sustainable.
Keywords: AI efficiency, CPU optimization, LLMS, LoRA, low-rank adaptation, model compression, parameter-efficient fine-tuning
[This article belongs to Recent Trends in Parallel Computing ]
References
- Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners [preprint]. 2020. arXiv:2005.14165. doi:10.48550/arXiv.2005.14165.
- Narayanan D, Shoeybi M, Casper J, LeGresley P, Patwary MM, Korthikanti VA, et al. Efficient large-scale language model training on GPU clusters using Megatron-LM [preprint]. 2021. arXiv:2104.04473. doi:10.48550/arXiv.2104.04473.
- Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. LoRA: low-rank adaptation of large language models [preprint]. 2021. arXiv:2106.09685. doi:10.48550/arXiv.2106.09685.
- Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need [preprint]. 2017. arXiv:1706.03762. doi:10.48550/arXiv.1706.03762.
- Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021). Online; Punta Cana, Dominican Republic; 2021:3045–3059. doi:10.18653/v1/
emnlp-main.243. - Han Z, Gao C, Liu J, Zhang J, Zhang SQ. Parameter-efficient fine-tuning for large models: a comprehensive survey [preprint]. 2024. arXiv:2403.14608. doi:10.48550/arXiv.2403.14608.
- Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLoRA: efficient finetuning of quantized large language models. Advances in Neural Information Processing Systems (NeurIPS 36). New Orleans, USA; 2023:10088–10115. doi:10.52202/075280-0441.
- Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, de Laroussilhe Q, Gesmundo A, et al. Parameter-efficient transfer learning for NLP [preprint]. 2019. arXiv:1902.00751. doi:10.48550/arXiv.1902.00751.
- Carneiro AR, Serpa MS, Navaux POA. Lightweight deep learning applications on AVX-512. 2021 IEEE Symposium on Computers and Communications (ISCC), Athens, Greece. 2021:1–6. doi:10.1109/ISCC53001.2021.9631464.
- Hasan MM, Islam MM. High-performance computing architectures for training large-scale transformer models in cyber-resilient applications. ASRC Procedia Glob Perspect Sci Scholarsh. 2022;2:193–226. doi:10.63125/6zt59y89.
- Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, et al. LLaMA: open and efficient foundation language models [preprint]. 2023. arXiv:2302.13971. doi:10.48550/arXiv.2302.13971.
- Gao T, Fisch A, Chen D. Making pre-trained language models better few-shot learners. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021). Online; 2021:3816–3830. doi:10.18653/v1/2021.acl-long.295.
- Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. Transformers: state-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP 2020). Online; 2020:38–45. doi:10.18653/v1/2020.emnlp-demos.6.
Recent Trends in Parallel Computing
| Volume | 13 | |
| Issue | 01 | |
| Received | 11/02/2026 | |
| Accepted | 13/03/2026 | |
| Published | 30/04/2026 | |
| Publication Time | 78 Days |