Performance Engineering of the Community Climate System ModelApril 30, 2013
"Performance Engineering of the Community Climate System Model" (PECCSM) was a project associated with the SciDAC2 Science Application project "A Scalable and Extensible Earth System Model for Climate Change Science" (SEESM) during its final two years. There was a predecessor Science Application Partnership (SAP) project called "Performance Engineering for the Next Generation Community Climate System Model" (PENG) that addressed performance engineering issues during the first three years of SEESM. You may know the history of SAPs in SciDAC2 better than I, but the SAP program wound down for the most part after the first three years of SciDAC2. The mid-project review of SEESM recognized important contributions by PENG and encouraged its continuation, if at all possible, because of the continued importance of performance engineering to SEESM. Based on this, the PECCSM proposal was developed and, subsequently, funded.
Both PENG and PECCSM had well-defined research plans and statements of work, but ultimately both were intended to be flexible enough to address performance issues that arose within SEESM. I believe that we were reasonably successful in this.
The goals of PENG were to identify and address structural impediments to performance and performance scalability in the Community Climate System Model (CCSM), especially with regards to "next generation" problem scenarios and HPC systems. At the time, this led naturally to a focus on the performance of the atmosphere, to helping port the model to and optimize performance on the IBM BG/L and BG/P and the Cray XT4 and XT5, and to helping with the development of a new parallel I/O layer (PIO).
During this period there were a number of new numerical methods being developed for the atmosphere dynamics, because the existing algorithm (finite volume based method, FV, on a longitude-latitude grid) was not viewed as being sufficiently scalable for next generation architectures and problem sizes. We felt that it was important to establish exactly what the performance capabilities of the existing algorithm were, both to provide a realistic baseline for determining whether the new methods were in fact superior and because the existing FV algorithm would be used for both production and other aspects of model development for at least the duration of SEESM. (The latter turned out to be true. The transition to a new atmosphere dynamics algorithm is just now taking place.) The FV performance optimization work and dynamics algorithm performance comparisons began in PENG, and were finished in PECCSM. The attempt to extend the performance and performance scalability of the FV algorithm led to the introduction of a number of scalability enhancements that had general applicability. The work also helped established that at least one of the new algorithms had superior performance at scale.
Building on the PENG work, PECCSM was able to focus more on the performance of the full coupled model, still in the context of large problem sizes and large scale parallelism. Due to changes in the ocean grid and in the target platforms, performance of the ocean component went from scaling very well to scaling relatively poorly in high resolution configurations, and so ocean (and sea ice) components became important targets for performance analysis and optimization. As part of this, significant effort went into node level performance optimization, including introducing OpenMP parallelism where it did not yet exist and further optimizing it where it did. (This also increased exploitable parallelism and/or decreased MPI overhead, so was more than just an optimization of node performance.) With the imminent public release of the new model and its use in AR5, it was also critical to determine how best to configure the model for performance (there being many options) for production scenarios and what performance to expect on target platforms. Finally, work began on further analyzing and optimizing performance of the expected successor to the FV dynamics algorithm, a spectral element method on a cubed sphere grid.