Skip to main content

"PoliMOR: A Policy Engine \"Made-to-Order\" for Automated and Scalable Data Management in Lustre"

Publication Type
Conference Paper
Book Title
Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis
Publication Date
Page Numbers
1202 to 1208
Publisher Location
New York, New York, United States of America
Conference Name
Conference Location
Denver, Colorado, United States of America
Conference Sponsor
Conference Date

Modern supercomputing systems are increasingly reliant on hierarchical, multi-tiered file and storage system architectures due to cost-performance-capacity trade-offs. Within such multi-tiered systems, data management services are required to maintain healthy utilization, performance, and capacity levels. We present PoliMOR, a pragmatic and reliable policy-driven data management framework. PoliMOR is composed of modular, single-purpose agents that gather file system metadata and enforce policies on storage systems. PoliMOR facilitates automated and scalable data management with customizable agents tailored to HPC facility-specific storage systems and policies. Our evaluations demonstrate the scalability and performance of PoliMOR both by its individual agents and as a collective entity. We believe PoliMOR is widely applicable across HPC facilities with large-scale data management challenges and will garner interest from the HPC community, given its flexible and open-source nature.