Skip to main content

Online and Scalable Data Compression Pipeline with Guarantees on Quantities of Interest...

Publication Type
Conference Paper
Book Title
2023 IEEE 19th International Conference on e-Science (e-Science)
Publication Date
Page Numbers
1 to 10
Publisher Location
New Jersey, United States of America
Conference Name
IEEE International Conference on e-Science 2023
Conference Location
Limassol, Cyprus
Conference Sponsor
Conference Date

Data compression is becoming critical for data-intensive scientific applications. Scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). Prior work has shown that a pipeline can be built to guarantee error on the primary data (PD) within user-defined bounds and achieve near-floating point QoI errors. In this paper, we present novel computational approaches for accelerating the pipeline and demonstrate results that enable concurrent execution of compression in parallel with the simulation nodes. This allows compression, including the writing of the required compression data, for the previous time step to be completed while the simulation proceeds with the current time step. Overall, the approach presented in this paper results in a 6–8 times improvement in computational overhead compared to previous work. These results were obtained using data generated by a large-scale fusion code called XGC, which produces hundreds of terabytes of data in a single day.