|Search Duke CS||
To quantify the performance behavior of production supercomputer file systems, we introduce a statistical benchmarking methodology to measure the impact of pa- rameter choices on burst absorption rates. Our approach combines many samples of their impacts over time to filter out interference caused by the transient congestion from competing workloads in the production setting. These samples are also used to characterize the performance of individual stages and components in the multi-stage write pipelines, and their variations over time.
We find that Titan’s I/O system is variable across the machine at fine time scales. This variability has two major implications. First, stragglers lessen the benefit of coupled I/O parallelism (striping). Peak median output bandwidths are obtained with parallel writes to many independent files, with no striping or write-sharing of files across clients (compute nodes). I/O parallelism is most effective when the application—or its I/O middleware system—distributes the I/O load so that each target stores files for multiple clients and each client writes files on multiple targets, in a balanced way with minimal contention. Second, our results suggest that the potential benefit of dynamic adaptation is limited. In particular, it is not fruitful to attempt to identify “good locations” in the machine or in the file system: component performance is driven by transient load conditions, and past performance is not a useful predictor of future performance. For example, we do not observe diurnal load patterns that are predictable.
Beyond the observation of performance variability of Titan, we also notice that: (1) the mean performance is stable and consistent over typical application run times; (2) output performance is non-linearly related to its correlated parameters due to interference and saturation on individual stages on the path. These observations enable us to build a predictive model of expected write times of output patterns and I/O configurations, using feature transformations to capture non-linear relationships. We identify the candidate features based on the structure of the Lustre/Titan write path, and use feature transformation functions to produce a linear model space with 135,000 candidate models. By searching for the minimal mean square error in this space we identify a good model and show that it is effective.