TY - GEN
T1 - Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection
AU - Li, Zhihan
AU - Zhao, Youjian
AU - Liu, Rong
AU - Pei, Dan
N1 - Publisher Copyright:
© 2018 IEEE.
PY - 2019/1/22
Y1 - 2019/1/22
N2 - For large Internet companies, it is very important to monitor a large number of KPIs (Key Performance Indicators) and detect anomalies to ensure the service quality and reliability. However, large-scale anomaly detection on millions of KPIs is very challenging due to the large overhead of model selection, parameter tuning, model training, or labeling. In this paper we argue that KPI clustering can help: we can cluster millions of KPIs into a small number of clusters and then select and train model on a per-cluster basis. However, KPI clustering faces new challenges that are not present in classic time series clustering: KPIs are typically much longer than other time series, and noises, anomalies, phase shifts and amplitude differences often change the shape of KPIs and mislead the clustering algorithm. To tackle the above challenges, in this paper we propose a robust and rapid KPI clustering algorithm, ROCKA. It consists of four steps: preprocessing, baseline extraction, clustering and assignment. These techniques help group KPIs according to their underlying shapes with high accuracy and efficiency. Our evaluation using real-world KPIs shows that ROCKA gets F-score higher than 0.85, and reduces model training time of a state-of-the-art anomaly detection algorithm by 90%, with only 15% performance loss.
AB - For large Internet companies, it is very important to monitor a large number of KPIs (Key Performance Indicators) and detect anomalies to ensure the service quality and reliability. However, large-scale anomaly detection on millions of KPIs is very challenging due to the large overhead of model selection, parameter tuning, model training, or labeling. In this paper we argue that KPI clustering can help: we can cluster millions of KPIs into a small number of clusters and then select and train model on a per-cluster basis. However, KPI clustering faces new challenges that are not present in classic time series clustering: KPIs are typically much longer than other time series, and noises, anomalies, phase shifts and amplitude differences often change the shape of KPIs and mislead the clustering algorithm. To tackle the above challenges, in this paper we propose a robust and rapid KPI clustering algorithm, ROCKA. It consists of four steps: preprocessing, baseline extraction, clustering and assignment. These techniques help group KPIs according to their underlying shapes with high accuracy and efficiency. Our evaluation using real-world KPIs shows that ROCKA gets F-score higher than 0.85, and reduces model training time of a state-of-the-art anomaly detection algorithm by 90%, with only 15% performance loss.
UR - http://www.scopus.com/inward/record.url?scp=85062631336&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85062631336&partnerID=8YFLogxK
U2 - 10.1109/IWQoS.2018.8624168
DO - 10.1109/IWQoS.2018.8624168
M3 - Conference contribution
AN - SCOPUS:85062631336
T3 - 2018 IEEE/ACM 26th International Symposium on Quality of Service, IWQoS 2018
BT - 2018 IEEE/ACM 26th International Symposium on Quality of Service, IWQoS 2018
T2 - 26th IEEE/ACM International Symposium on Quality of Service, IWQoS 2018
Y2 - 4 June 2018 through 6 June 2018
ER -