System log detection model based on conformal prediction

Yitong Ren, Zhaojun Gu, Zhi Wang, Zhihong Tian, Chunbo Liu, Hui Lu, Xiaojiang Du, Mohsen Guizani

Research output: Contribution to journalArticlepeer-review

9 Scopus citations

Abstract

With the rapid development of the Internet of Things, the combination of the Internet of Things with machine learning, Hadoop and other fields are current development trends. Hadoop Distributed File System (HDFS) is one of the core components of Hadoop, which is used to process files that are divided into data blocks distributed in the cluster. Once the distributed log data are abnormal, it will cause serious losses. When using machine learning algorithms for system log anomaly detection, the output of threshold‐based classification models are only normal or abnormal simple predictions. This paper used the statistical learning method of conformity measure to calculate the similarity between test data and past experience. Compared with detection methods based on static threshold, the statistical learning method of the conformity measure can dynamically adapt to the changing log data. By adjusting the maximum fault tolerance, a system administrator can better manage and monitor the system logs. In addition, the computational efficiency of the statistical learning method for conformity measurement was improved. This paper implemented an intranet anomaly detection model based on log analysis, and conducted trial detection on HDFS data sets quickly and efficiently.

Original languageEnglish
Article number232
JournalElectronics (Switzerland)
Volume9
Issue number2
DOIs
StatePublished - Feb 2020

Keywords

  • Anomaly detection
  • Conformal prediction
  • Confusion matrix
  • HDFS

Fingerprint

Dive into the research topics of 'System log detection model based on conformal prediction'. Together they form a unique fingerprint.

Cite this