LOG IDENTIFICATION AND MONITORING SYSTEM USING GEN-AI

Notice

This is an unedited manuscript accepted for publication and provided as an Article in Press for early access at the author’s request. The article will undergo copyediting, typesetting, and galley proof review before final publication. Please be aware that errors may be identified during production that could affect the content. All legal disclaimers of the journal apply.

Year : 2026 | Volume : 17 | 01 | Page :
    By

    Nikhil Santosh Shinde,

  • Ajay Shiketod,

  • Swati Andhale,

  1. Student, Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune, Maharashtra, India
  2. Professor, Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune, Maharashtra, India
  3. HOD, Department of MCA, Parvatibai Genba Moze College of Engineering, Wagholi, Pune, Maharashtra, India

Abstract

In contemporary software ecosystems, application and infrastructure logs play a vital role in ensuring system reliability, performance optimization, fault diagnosis, and security compliance. As applications become increasingly distributed and cloud-native, the volume, velocity, and variety of generated log data have grown dramatically. This rapid expansion makes traditional manual log inspection inefficient, error-prone, and largely impractical. To address these challenges, this paper proposes an AI-driven log monitoring and analysis system designed to automate log processing, anomaly detection, alert generation, and visualization in real time. The proposed system is built on a scalable and high-performance architecture that combines a FastAPI backend for asynchronous log ingestion and processing with a Next.js-based frontend that enables interactive dashboards and visual analytics. Logs collected from distributed applications are parsed and structured using advanced pattern-matching techniques, including Grok and Vector.dev, ensuring consistency and accuracy in data extraction. An intelligent anomaly detection engine powered by machine learning is integrated to identify unusual patterns, deviations, and potential system failures. To further enhance detection accuracy and computational efficiency, the system employs a Trie-based adaptive caching mechanism that optimizes repeated pattern recognition. Additionally, the platform integrates Grafana and Prometheus to support comprehensive monitoring and time-series visualization, while real-time alerts are delivered through external notification channels such as Slack and email. Experimental results indicate that the system effectively identifies anomalies with minimal latency, provides actionable insights, and significantly reduces the operational burden on engineers. Overall, the proposed solution improves observability, accelerates incident response, and enhances the reliability of modern distributed systems.

Keywords: Log Monitoring, Generative AI, Anomaly Detection, Distributed Systems, Real-Time Alerting

How to cite this article:
Nikhil Santosh Shinde, Ajay Shiketod, Swati Andhale. LOG IDENTIFICATION AND MONITORING SYSTEM USING GEN-AI. Journal of Computer Technology & Applications. 2026; 17(01):-.
How to cite this URL:
Nikhil Santosh Shinde, Ajay Shiketod, Swati Andhale. LOG IDENTIFICATION AND MONITORING SYSTEM USING GEN-AI. Journal of Computer Technology & Applications. 2026; 17(01):-. Available from: https://journals.stmjournals.com/jocta/article=2026/view=237219


References

  1. Xu W, Huang L, Fox A, Patterson D, Jordan MI. Detecting large-scale system problems by mining console logs. InProceedings of the ACM SIGOPS 22nd symposium on Operating systems principles 2009 Oct 11 (pp. 117-132).
  2. He P, Zhu J, Zheng Z, Lyu MR. Drain: An online log parsing approach with fixed depth tree. In2017 IEEE international conference on web services (ICWS) 2017 Jun 25 (pp. 33-40). IEEE.
  3. Du M, Li F, Zheng G, Srikumar V. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. InProceedings of the 2017 ACM SIGSAC conference on computer and communications security 2017 Oct 30 (pp. 1285-1298).
  4. Yadav RB, Kumar PS, Dhavale SV. A survey on log anomaly detection using deep learning. In2020 8th international conference on reliability, infocom technologies and optimization (Trends and Future Directions)(ICRITO) 2020 Jun 4 (pp. 1215-1220). IEEE.
  5. Zhou J, Ying S, Wang S, Zhao D, Xiang J, Liang K, Liu P. LogDLR: Unsupervised Cross-System Log Anomaly Detection Through Domain-Invariant Latent Representation. IEEE Transactions on Dependable and Secure Computing. 2025 Mar 5.
  6. Zhang X, Xu Y, Lin Q, Qiao B, Zhang H, Dang Y, Xie C, Yang X, Cheng Q, Li Z, Chen J. Robust log-based anomaly detection on unstable log data. InProceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering 2019 Aug 12 (pp. 807-817).
  7. Pankajashan S, Maragatham G, Kirthiga Devi T. Hybrid approach with Deep Auto-Encoder and optimized LSTM based Deep Learning approach to detect anomaly in cloud logs. Journal of Intelligent & Fuzzy Systems. 2022 Apr 28;42(6):6257-71.
  8. Guo H, Yang J, Liu J, Bai J, Wang B, Li Z, Zheng T, Zhang B, Peng J, Tian Q. Logformer: A pre- train and tuning pipeline for log anomaly detection. InProceedings of the AAAI conference on artificial intelligence 2024 Mar 25 (Vol. 38, No. 1, pp. 135-143).
  9. Lou JG, Fu Q, Yang S, Xu Y, Li J. Mining invariants from console logs for system problem detection. In2010 USENIX Annual Technical Conference (USENIX ATC 10) 2010.
  10. Gulati A, Weatherspoon H. Proceedings of 2016 USENIX Annual Technical Conference. InProceedings of 2016 USENIX Annual Technical Conference 2016 Jan.
  11. Kou L, Li Y, Zhang F, Gong X, Hu Y, Yuan Q, Ke W. Review on monitoring, operation and maintenance of smart offshore wind farms. Sensors. 2022 Apr 7;22(8):2822.
  12. Huang S, Liu Y, Fung C, Wang H, Yang H, Luan Z. Improving log-based anomaly detection by pre-training hierarchical transformers. IEEE Transactions on Computers. 2023 Mar 15;72(9):2656- 67.
  13. Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, Almeida D, Altenschmidt J, Altman S, Anadkat S, Avila R. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023 Mar 15.
  14. Prometheus – Monitoring system & time series database. Prometheus.io. 2025. Available from: https://prometheus.io/
  15. Technical documentation | Grafana Labs. Grafana Labs. 2025. Available from: https://grafana.com/docs/
  16. Lubanovic B. FastAPI. ” O’Reilly Media, Inc.”; 2023 Nov 6.
  17. Lazuardy MF, Anggraini D. Modern front end web architectures with react. js and next. js. Research Journal of Advanced Engineering and Science. 2022;7(1):132-41.
  18. D’Amore L, Arcucci R, Mele V, Scotti G, Murli A. Technical Documentation L-BFGS for GPU- CUDA Reference Manual and User’s Guide. CMCC Research Paper. 2013 Feb 1(167).
  19. Johnson HA. Slack. Journal of the Medical Library Association: JMLA. 2018 Jan 2;106(1):148.
  20. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J. Scikit-learn: Machine learning in Python. the Journal of machine Learning research. 2011 Nov 1;12:2825-30.
Ahead of Print Subscription Original Research
Volume 17
01
Received 25/07/2025
Accepted 18/12/2025
Published 20/02/2026
Publication Time 210 Days
53

Login


My IP

PlumX Metrics