Skip to main content

Large-Scale Compute-Intensive Analysis via a Combined In-situ and Co-scheduling Workflow Approach...

Publication Type
Conference Paper
Publication Date
Page Number
Publisher Location
New York, New Jersey, United States of America
Conference Name
International Conference for High Performance Computing, Networking, Storage and Analysis
Conference Location
Austin, Texas, United States of America
Conference Date

Large-scale simulations can produce tens of terabytes of data per
analysis cycle, complicating and limiting the efficiency of workflows.
Traditionally, outputs are stored on the file system and analyzed
in post-processing. With the rapidly increasing size and
complexity of simulations, this approach faces an uncertain future.
Trending techniques consist of performing the analysis in situ, utilizing
the same resources as the simulation, and/or off-loading subsets
of the data to a compute-intensive analysis system. We introduce
an analysis framework developed for HACC, a cosmological
N-body code, that uses both in situ and co-scheduling approaches
for handling Petabyte-size outputs. An initial in situ step is used to
reduce the amount of data to be analyzed, and to separate out the
data-intensive tasks handled off-line. The analysis routines are implemented
using the PISTON/VTK-m framework, allowing a single
implementation of an algorithm that simultaneously targets a
variety of GPU, multi-core, and many-core architectures.