Master's Defense

Finding Needles in the Haystack: Harnessing Syslogs for Data Center Management

Speaker:Chen Liang
cliang at
Date: Tuesday, April 5, 2016
Time: 3:30pm - 5:30pm
Location: D344 LSRC, Duke


Network syslog data has been widely used to characterize and troubleshoot physical network failures. However syslogs contain a wealth of information that characterize not only link and device failures but also the network's perspective on daily operational activities such as provisioning and infrastructure maintenance. The nature of operational information as reflected by syslogs and their implications on diagnosing management tasks is poorly understood.

Towards improving our understanding, we propose and implement a tool that applies various analysis techniques to analyze syslog data. It performs statistical analysis to infers causal relationships between syslogs, then construct causal graphs to group syslog messages into events. In addition, it detects abnormalities within the syslog data. Then, we perform classification and apply various analysis to explore underlying patterns of templates, abnormalities and causal graphs from different perspectives, such as longitudinal pattern, and how do network layers and types of devices affect patterns.

We validate our findings by examining the implications on traffic as reflected by problem tickets. We also perform classification and analysis on trouble ticket data. Our correlations between problem tickets and syslogs can help operators and system designers in understanding the necessary features to use when developing diagnosis frameworks for troubleshooting managing data centers.

Advisor(s): Theo Benson
Committee: Jeffrey Chase, Bruce Maggs