Achievement: A team of researchers from Oak Ridge National Laboratory demonstrated highly scalable performance across thousands of GPUs in a newly released version of the open-source MEUMAPPS phase-field simulation framework. This new version is a C++ rewrite of the original Fortran version of MEUMAPPS (https://doi.org/10.11578/dc.20201001.96, https://www.osti.gov/doecode/biblio/45884, authored by Radhakrishnan Balasubramaniam, Gorti Sarma and Younggil Song), and takes advantage of several software technology investments by the US DOE Exascale Computing Project. For a full-physics test problem, the new version on GPUs is 15x faster than the original on GPUs. MEUMAPPS C++ exhibits a 10x GPU speedup on a Summit node (comparing 6 GPUs to all 42 CPU cores), demonstrating efficient usage of the heterogeneous node architecture. As seen in Fig. 1, for a similar test problem on an 11523 mesh, near-ideal scaling to at least 3072 GPUs (512 Summit nodes) has been achieved, with scaling tests to larger numbers of nodes planned. In addition to the performance improvements, MEUMAPPS C++ has also been refactored to enable greater modularity. It now includes separate applications that share a common a generalized set of functionalities in a core library, streamlining the process of performing phase-field simulations with MEUMAPPS C++ with different governing equations. The release of MEUMAPPS C++ includes the flagship MEUMAPPS-SS application for multi-component, multi-phase solid-solid phase transformation simulations from the original Fortran code, and also includes five simpler applications with other governing equations. One of these applications uses high-order, adaptive time stepping to yield a 10-20x performance improvement over a typical fixed-step first-order scheme (for similar, physically relevant error levels). MEUMAPPS C++ also has an emphasis on performance portability, being able to efficiently use a variety of computing hardware and has been used on NVIDIA GPUs, AMD GPUs, and CPU-only machines.
Significance and Impact: Many materials, and metal alloys in particular, have features or building blocks that range from the micrometer or nanometer scale that have a large impact on the properties of the material. These features are collectively known as the “microstructure” of the material. Understanding why and how the microstructure forms in a material is of fundamental scientific interest as well as of significant technological importance. The capability to predict microstructure evolution in a material allows the intentional design of microstructures and hence the intentional design of material properties. The phase-field method is one of the leading methods for high fidelity predictions of microstructure evolution. One of the most significant problems for phase-field models is their computational expense. Even limited phase-field simulations can easily require thousands of CPU core-hours to complete, which significantly limits their use. This code provides both a general framework for creating scalable, GPU-accelerated phase-field model applications as well as several applications themselves. The code is capable of using hundreds of GPUs efficiently, which greatly reduces the time required to perform simulations.
- MEUMAPPS C++ , like the original Fortran version, solves partial differential equations using a pseudospectral approach with semi-implicit time integration.
- The Fourier transforms required for pseudospectral methods are performed using a more modern library, heFFTe, than the one used in the original Fortran version. The heFFTe library provides GPU-accelerated, distributed, scalable, 3D FFTs.
- The non-FFT portions of MEUMAPPS C++ make heavy use of the Kokkos library for performance portable parallel execution and data structures.
- Most applications use a fixed-step, semi-implicit Euler time integration scheme (as in the original Fortran version), but MEUMAPPS C++ also includes an interface with the Sundials library for high-order, adaptive, semi-implicit Runge-Kutta time integration.
Facility: This work has been performed using several computing systems at the OLCF: Summit, SummitDev, Ascent, Spock, and Crusher.
Sponsor/Funding: DOE ECP (ExaAM)
Primary Contact and Affiliation: Stephen DeWitt, Scalable Algorithms and Coupled Physics Research Group, Computational Sciences and Engineering Division, ORNL
Team: Stephen DeWitt, Philip Fackler, Younggil Song, Balasubramaniam Radhakrishnan and Gorti Sarma
Citation and DOI: DeWitt, Stephen, Fackler, Philip, Song, Younggil, Radhakrishnan, Balasubramaniam, and Gorti, Sarma. MEUMAPPS (C++ Version). Computer Software. https://code.ornl.gov/meumapps/meumapps. USDOE National Nuclear Security Administration (NNSA), USDOE Office of Science (SC). 14 Jan. 2022. Web. doi:10.11578/dc.20220114.1.
Summary: Phase-field simulations enable high-fidelity predictions of microstructure evolution in metal alloys and other materials to better understand the link between the processing (or environmental) conditions for a material and its properties. MEUMAPPS C++ is a newly released reimplementation of the open-source MEUMAPPS phase-field simulation framework, with an emphasis on simulating solid-solid phase transformations in alloys. This new version is ready to take advantage of cutting-edge leadership computing resources with a 10x GPU speedup on Summit (comparing 6 GPUs to 42 CPU cores) and with near-ideal scaling to at least 3072 GPUs. This performance and the ability to efficiently use computing systems with a variety of architectures is enabled through the use of several libraries supported by the Exascale Computing Project: heFFTe (fast Fourier transforms), Kokkos (data structures and execution patterns), and Sundials (time integration). The flexibility in the design of MEUMAPPS C++ allows incorporation of new physics advancements as they evolve from the theory, and experimental communities. With performance improvements from MEUMAPPS C++, the phase-field community can leverage leadership-scale computing investigate new aspects of microstructure evolution in a wide range of materials of interest to the DOE, from materials for energy systems to materials created through additive manufacturing.