Confidence: Analyzing Performance With Empirical Probabilities
By:
Settlemyer, Bradley W; Hodson, Stephen W; Kuehn, Jeffery A; Poole, Stephen W
Publication Date:
September 2010
Publisher Location:
IEEE Computer Society
Conference Name:
Workshop on Application/Architecture Co-design for Extreme-scale Computing (AACEC) in Conjunction with IEEE Cluster 2010
Conference Location:
Hersonninsos, Greece
Conference Sponsor:
IEEE
Abstract
Variability in the performance of shared system components is a major obstacle in analyzing the effective throughput of leadership class computers. Shared file systems and networks are serious impediments to achieving repeatable application performance on HPC systems. In particular, performance analysts are likely to be interested in quantifying differences between average-case behavior, worst-case behavior, and standard deviation for shared system components. Typical descriptions of these statistics assume a normal distribution; however, in non-linear and multi-modal performance distributions, summary statistics are often misleading. In this paper we describe Confidence, a tool for analyzing the full spectrum of performance for a benchmarking code. By including all of the experimental outcomes in the analysis without discarding any measurements, Confidence enables a novel analysis of benchmark performance. Finally, we use Confidence to characterize the network performance of three high performance computer systems, and we provide recommendations for leveraging network performance in latency sensitive applications.