Log Identification and Monitoring System Using Generative AI

Year : 2026 | Volume : 17 | Issue : 01 | Page : 08 16
    By

    Nikhil Santosh Shinde,

  • Ajay Shiketod,

  • Swati Andhale,

  1. Student, Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune, Maharashtra, India
  2. Professor, Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune, Maharashtra, India
  3. Professor, Head of Department, Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune, Maharashtra, India

Abstract

In contemporary software ecosystems, application and infrastructure logs play a vital role in ensuring system reliability, performance optimization, fault diagnosis, and security compliance. As applications become increasingly distributed and cloud native, the volume, velocity, and variety of generated log data have grown dramatically. This rapid expansion makes traditional manual log inspection inefficient, error-prone, and largely impractical. To address these challenges, this paper proposes an artificial intelligence (AI) driven log monitoring and analysis system designed to automate log processing, anomaly detection, alert generation, and visualization in real time The proposed system is built on a scalable and high-performance architecture that combines a FastAPI backend for asynchronous log ingestion and processing with a Next.js-based frontend that enables interactive dashboards and visual analytics. Logs collected from distributed applications are parsed and structured using advanced pattern-matching techniques, including Grok and Vector.dev, ensuring consistency and accuracy in data extraction. An intelligent anomaly detection engine powered by machine learning is integrated to identify unusual patterns, deviations, and potential system failures. To further enhance detection accuracy and computational efficiency, the system employs a trie-based adaptive caching mechanism that optimizes repeated pattern recognition. Additionally, the platform integrates Grafana and Prometheus to support comprehensive monitoring and time-series visualization, while real-time alerts are delivered through external notification channels such as Slack and email. Experimental results indicate that the system effectively identifies anomalies with minimal latency, provides actionable insights, and significantly reduces the operational burden on engineers. Overall, the proposed solution improves observability, accelerates incident response, and enhances the reliability of modern distributed systems.

Keywords: Anomaly detection, distributed systems, generative AI, log monitoring, real-time alerting

[This article belongs to Journal of Computer Technology & Applications ]

How to cite this article:
Nikhil Santosh Shinde, Ajay Shiketod, Swati Andhale. Log Identification and Monitoring System Using Generative AI. Journal of Computer Technology & Applications. 2026; 17(01):08-16.
How to cite this URL:
Nikhil Santosh Shinde, Ajay Shiketod, Swati Andhale. Log Identification and Monitoring System Using Generative AI. Journal of Computer Technology & Applications. 2026; 17(01):08-16. Available from: https://journals.stmjournals.com/jocta/article=2026/view=237219


References

  1. Xu W, Huang L, Fox A, Patterson D, Jordan MI. Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles (SOSP ’09); 2009 Oct 11–14; Big Sky, MT, USA. New York (NY): Association for Computing Machinery; 2009. p. 117–132. doi:10.1145/1629575.1629587.
  2. He P, Zhu J, Zheng Z, Lyu MR. Drain: An online log parsing approach with fixed depth tree. 2017 IEEE International Conference on Web Services (ICWS), Honolulu, HI, USA. 2017. p. 33–40. doi:10.1109/ICWS.2017.13.
  3. Du M, Li F, Zheng G, Srikumar V. DeepLog: Anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS ’17); 2017 Oct 30–Nov 3; Dallas, TX, USA. New York (NY): Association for Computing Machinery; 2017. p.1285–1298. doi:10.1145/3133956.3134015.
  4. Yadav RB, Kumar PS, Dhavale SV. A survey on log anomaly detection using deep learning. 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India. 2020. p. 1215–1220. doi:10.1109/ICRITO48877.2020.9197818.
  5. Zhou J, Ying S, Wang S, Zhao D, Xiang J, Liang K, et al. LogDLR: Unsupervised cross-system log anomaly detection through domain-invariant latent representation. IEEE Trans Dependable Secure Comput. 2025;22:4456–4471. doi:10.1109/TDSC.2025.3548050.
  6.  Zhang X, Xu Y, Lin Q, Qiao B, Zhang H, Dang Y, et al. Robust log-based anomaly detection on unstable log data. In: Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019); 2019 Aug 26–30; Tallinn, Estonia. New York (NY): Association for Computing Machinery; 2019. p.807–817. doi:10.1145/3338906.3338931.
  7. Pankajashan S, Maragatham G, Kirthiga Devi T. Hybrid approach with deep auto-encoder and optimized LSTM-based deep learning approach to detect anomaly in cloud logs. J Intell Fuzzy Syst. 2022;42(6):6257–6271. doi:10.3233/JIFS-201707.
  8. Guo H, Yang J, Liu J, Bai J, Wang B, Li Z, et al. Logformer: A pre-train and tuning pipeline for log anomaly detection. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38:135–143. doi:10.1609/aaai.v38i1.27764.
  9. Lou JG, Fu Q, Yang S, Xu Y, Li J. Mining invariants from console logs for system problem detection. In: Proceedings of the 2010 USENIX Annual Technical Conference (USENIX ATC ’10); 2010 Jun 23–25; Boston, MA, USA. Berkeley (CA): USENIX Association; 2010. p. 1–14.
  10. lim A, Clegg RG, Mai L, Rupprecht L, Seckler E, Costa P, Pietzuch P, Wolf AL, Sultana N, Crowcroft J, Madhavapeddy A, Moore AW, Mortier R, Koleni M, Oviedo L, McAuley D, Migliavacca M. FLICK: Developing and running application-specific network services. In: Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16); 2016 Jun 22–24; Denver, CO, USA. Berkeley (CA): USENIX Association; 2016. Available from: https://www.usenix.org/conference/atc16/technical-sessions/presentation/alim
  11.  Kou L, Li Y, Zhang F, Gong X, Hu Y, Yuan Q, et al. Review on monitoring, operation and maintenance of smart offshore wind farms. Sensors. 2022;22(8):2822. doi:10.3390/s22082822. 1
  12. Huang S, Liu Y, Fung C, Wang H, Yang H, Luan Z. Improving log-based anomaly detection by pre-training hierarchical transformers. IEEE Trans Comput. 2023;72(9):2656–2667. doi:10.1109/TC.2023.3257518.
  13. OpenAI; Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, et al. GPT-4 technical report. [Preprint]. 2023. arXiv:2303.08774. doi:10.48550/arXiv.2303.08774.
  14. Prometheus.io. (2026). Data model. [Online] Prometheus Documentation. Available from: https://prometheus.io/docs/concepts/data_model/
  15. Grafana Labs. (2025). Grafana OSS and Enterprise: Query, visualize, alert on, and explore your metrics, logs, and traces. [online] Grafana documentation. Available from: https://grafana.com/docs/grafana/latest/
  16. Lubanovic B. FastAPI. Sebastopol (CA): O’Reilly Media Inc.; 2023.
  17. Lazuardy MF, Anggraini D. Modern front-end web architectures with React.js and Next.js. Res J. Adv. Eng Sci. 2022;7(1):132–141.
  18. D’Amore L, Arcucci R, Mele V, Scotti G, Murli A. Technical documentation L-BFGS for GPU-CUDA reference manual and user’s guide. SSRN Electron J. 2013;167. doi:10.2139/ssrn.2332125.
  19. Johnson HA. Slack. J Med Libr Assoc. 2018;106(1):148. doi:10.5195/jmla.2018.315.
  20. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine learning in Python. J Mach Learn Res. 2011;12:2825–2830.
Regular Issue Subscription Original Research
Volume 17
Issue 01
Received 25/07/2025
Accepted 18/12/2025
Published 20/02/2026
Publication Time 210 Days
130

Login


My IP

PlumX Metrics