Skip to main content
SHARE
Publication

Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

Publication Type
Conference Paper
Book Title
ISC High Performance 2024 Research Paper Proceedings (39th International Conference)
Publication Date
Page Numbers
1 to 12
Publisher Location
New Jersey, United States of America
Conference Name
ISC High Performance 2024 (39th International Conference)
Conference Location
Hamburg, Germany
Conference Sponsor
IEEE
Conference Date
-

In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-aligned engine which provides up to 2.3× improved performance than when using the existing DAOS-POSIX interface, without requiring any application modifications.