SHARE
Research Highlight

MEUMAPPS C++: A Scalable, Performance-Portable, Open-Source Framework for Phase-Field Modeling

Topic:

Achievement: A team of researchers from Oak Ridge National Laboratory demonstrated highly scalable performance across thousands of GPUs in a newly released version of the open-source MEUMAPPS phase-field simulation framework. This new version is a C++ rewrite of the original Fortran version of MEUMAPPS (https://doi.org/10.11578/dc.20201001.96, https://www.osti.gov/doecode/biblio/45884, authored by Radhakrishnan Balasubramaniam, Gorti Sarma and Younggil Song), and takes advantage of several software technology investments by the US DOE Exascale Computing Project. For a full-physics test problem, the new version on GPUs is 15x faster than the original on GPUs. MEUMAPPS C++ exhibits a 10x GPU speedup on a Summit node (comparing 6 GPUs to all 42 CPU cores), demonstrating efficient usage of the heterogeneous node architecture. As seen in Fig. 1, for a similar test problem on an 11523 mesh, near-ideal scaling to at least 3072 GPUs (512 Summit nodes) has been achieved, with scaling tests to larger numbers of nodes planned. In addition to the performance improvements, MEUMAPPS C++ has also been refactored to enable greater modularity. It now includes separate applications that share a common a generalized set of functionalities in a core library, streamlining the process of performing phase-field simulations with MEUMAPPS C++ with different governing equations. The release of MEUMAPPS C++ includes the flagship MEUMAPPS-SS application for multi-component, multi-phase solid-solid phase transformation simulations from the original Fortran code, and also includes five simpler applications with other governing equations. One of these applications uses high-order, adaptive time stepping to yield a 10-20x performance improvement over a typical fixed-step first-order scheme (for similar, physically relevant error levels). MEUMAPPS C++ also has an emphasis on performance portability, being able to efficiently use a variety of computing hardware and has been used on NVIDIA GPUs, AMD GPUs, and CPU-only machines.

Significance and Impact: Many materials, and metal alloys in particular, have features or building blocks that range from the micrometer or nanometer scale that have a large impact on the properties of the material. These features are collectively known as the “microstructure” of the material. Understanding why and how the microstructure forms in a material is of fundamental scientific interest as well as of significant technological importance. The capability to predict microstructure evolution in a material allows the intentional design of microstructures and hence the intentional design of material properties. The phase-field method is one of the leading methods for high fidelity predictions of microstructure evolution. One of the most significant problems for phase-field models is their computational expense. Even limited phase-field simulations can easily require thousands of CPU core-hours to complete, which significantly limits their use. This code provides both a general framework for creating scalable, GPU-accelerated phase-field model applications as well as several applications themselves. The code is capable of using hundreds of GPUs efficiently, which greatly reduces the time required to perform simulations.

Research Details

  • MEUMAPPS C++ , like the original Fortran version, solves partial differential equations using a pseudospectral approach with semi-implicit time integration.
  • The Fourier transforms required for pseudospectral methods are performed using a more modern library, heFFTe, than the one used in the original Fortran version. The heFFTe library provides GPU-accelerated, distributed, scalable, 3D FFTs.
  • The non-FFT portions of MEUMAPPS C++ make heavy use of the Kokkos library for performance portable parallel execution and data structures.
  • Most applications use a fixed-step, semi-implicit Euler time integration scheme (as in the original Fortran version), but MEUMAPPS C++ also includes an interface with the Sundials library for high-order, adaptive, semi-implicit Runge-Kutta time integration.

Facility: This work has been performed using several computing systems at the OLCF: Summit, SummitDev, Ascent, Spock, and Crusher.

Sponsor/Funding: DOE ECP (ExaAM)

Primary Contact and Affiliation: Stephen DeWitt, Scalable Algorithms and Coupled Physics Research Group, Computational Sciences and Engineering Division, ORNL 

Team: Stephen DeWitt, Philip Fackler, Younggil Song, Balasubramaniam Radhakrishnan and Gorti Sarma

Citation and DOI: DeWitt, Stephen, Fackler, Philip, Song, Younggil, Radhakrishnan, Balasubramaniam, and Gorti, Sarma. MEUMAPPS (C++ Version). Computer Software. https://code.ornl.gov/meumapps/meumapps. USDOE National Nuclear Security Administration (NNSA), USDOE Office of Science (SC). 14 Jan. 2022. Web. doi:10.11578/dc.20220114.1.
 

MEUMAPPS C++: A Scalable, Performance-Portable, Open-Source Framework for Phase-Field Modeling CSED Computational Sciences and Engineering ORNL
(a) Example using MEUMAPPS C++ to simulate the evolution of γ'' precipitates in an Ni-Nb-Mo alloy, where the three colors represent the three crystallographically equivalent orientation variants of the precipitates. (b) Combined strong and weak scaling plots showing the wall time required for simulations with three different problem sizes (2883 to 11523) and ten GPU counts (6 to 3,072). The dashed lines for each problem size denote ideal strong scaling – if the data points lie on the dashed line then the run time decreases proportionately as the number of GPUs increases. Initialization times are not included in this plot.

Summary: Phase-field simulations enable high-fidelity predictions of microstructure evolution in metal alloys and other materials to better understand the link between the processing (or environmental) conditions for a material and its properties. MEUMAPPS C++ is a newly released reimplementation of the open-source MEUMAPPS phase-field simulation framework, with an emphasis on simulating solid-solid phase transformations in alloys. This new version is ready to take advantage of cutting-edge leadership computing resources with a 10x GPU speedup on Summit (comparing 6 GPUs to 42 CPU cores) and with near-ideal scaling to at least 3072 GPUs. This performance and the ability to efficiently use computing systems with a variety of architectures is enabled through the use of several libraries supported by the Exascale Computing Project: heFFTe (fast Fourier transforms), Kokkos (data structures and execution patterns), and Sundials (time integration). The flexibility in the design of MEUMAPPS C++ allows incorporation of new physics advancements as they evolve from the theory, and experimental communities. With performance improvements from MEUMAPPS C++, the phase-field community can leverage leadership-scale computing investigate new aspects of microstructure evolution in a wide range of materials of interest to the DOE, from materials for energy systems to materials created through additive manufacturing.