Skip to main content

DenKv: Addressing Design Trade-offs of Key-value Stores for Scientific Applications

by Awais Khan, Hakki S Oral
Publication Type
Conference Paper
Book Title
2022 IEEE/ACM International Parallel Data Systems Workshop (PDSW)
Publication Date
Page Numbers
20 to 25
Publisher Location
New Jersey, United States of America
Conference Name
Conference Location
Dallas, TX, Texas, United States of America
Conference Sponsor
IEEE/ACM Supercomputing SC 2022
Conference Date

High-performance computing (HPC) facilities have employed flash-based storage tier near to compute nodes to absorb high I/O demand by HPC applications during periodic system-level checkpoints. To accelerate these checkpoints, proxy-based distributed key-value stores (PD-KVS) gained particular attention for their flexibility to support multiple backends and different network configurations. PD-KVS rely internally on monolithic KVS, such as LevelDB or RocksDB, to exploit the KV interface and query support. However, PD-KVS are unaware of the high redundancy factor in checkpoint data, which can be up to GBs to TBs, and therefore, tend to generate high write and space amplification on these storage layers. In this paper, we propose DenKv which is deduplication-extended node-local LSM-tree-based KVS. DenKv employs asynchronous partially inline dedup (APID) and aims to maintain the performance characteristics of LSM-tree-based KVS while reducing the write and space amplification problems. We implemented DenKv atop BlobDB and showed that our proposed solution maintains performance while reducing write amplification up to 2× and space amplification by 4× on average.