Reducing Application Runtime Variability on Jaguar XT5

Abstract

Operating system (OS) noise is defined as interference generated by the OS that prevents a compute core from performing ``useful'' work. Compute node kernel daemons, network interfaces, and other OS related services are major sources of such interference. This interference on individual compute cores can vary in duration and frequency, and can cause de-synchronization (jitter) in collective communication tasks and thus results in variable (degraded) overall parallel application performance. This behavior is more observable in large-scale applications using certain types of collective communication primitives, such as MPI_Allreduce. This paper presents our effort towards reducing the overall effect of OS noise on our large-scale parallel applications. Our tests were performed on the quad-core Jaguar, the Cray XT5 at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). At the time of these tests, Jaguar was a 1.4 PFLOPS supercomputer with 149,504 compute cores and 8 cores per node. We aggregated OS noise sources onto a single core for each node. The scientific application was then run on six of the remaining cores in each node. Our results show that we were able to improve the MPI_Allreduce performance by two orders of magnitude. We demonstrated up to a 30% boost in the performance of the Parallel Ocean Program (POP) using this technique.