Skip to main content
Publication

Organizing Large Data Sets for Efficient Analyses on HPC Systems...

Publication Type
Conference Paper
Journal Name
Journal of Physics: Conference Series
Publication Date
Page Number
012042
Volume
2224
Issue
1
Conference Name
2021 2nd International Symposium on Automation, Information and Computing (ISAIC 2021)
Conference Location
Beijing (virtual), China
Conference Sponsor
Beijing Jiaotong University
Conference Date
-

Upcoming exascale applications could introduce significant data management challenges due to their large sizes, dynamic work distribution, and involvement of accelerators such as graphical processing units, GPUs. In this work, we explore the performance of reading and writing operations involving one such scientific application on two different supercomputers. Our tests showed that the Adaptable Input and Output System, ADIOS, was able to achieve speeds over 1TB/s, a significant fraction of the peak I/O performance on Summit. We also demonstrated the querying functionality in ADIOS could effectively support common selective data analysis operations, such as conditional histograms. In tests, this query mechanism was able to reduce the execution time by a factor of five. More importantly, ADIOS data management framework allows us to achieve these performance improvements with only a minimal amount of coding effort.