Skip to main content
SHARE
Publication

DenKv: Addressing Design Trade-offs of Key-value Stores for Scientific Applications

by Awais Khan, Hakki S Oral
Publication Type
Conference Paper
Book Title
2022 IEEE/ACM International Parallel Data Systems Workshop (PDSW)
Publication Date
Page Numbers
20 to 25
Issue
978-1-6654
Publisher Location
New Jersey, United States of America
Conference Name
PDSW 2022: 7TH INTERNATIONAL PARALLEL DATA SYSTEMS WORKSHOP
Conference Location
Dallas, TX, Texas, United States of America
Conference Sponsor
IEEE/ACM Supercomputing SC 2022
Conference Date
-

High-performance computing (HPC) facilities have employed flash-based storage tier near to compute nodes to absorb high I/O demand by HPC applications during periodic system-level checkpoints. To accelerate these checkpoints, proxy-based distributed key-value stores (PD-KVS) gained particular attention for their flexibility to support multiple backends and different network configurations. PD-KVS rely internally on monolithic KVS, such as LevelDB or RocksDB, to exploit the KV interface and query support. However, PD-KVS are unaware of the high redundancy factor in checkpoint data, which can be up to GBs to TBs, and therefore, tend to generate high write and space amplification on these storage layers. In this paper, we propose DenKv which is deduplication-extended node-local LSM-tree-based KVS. DenKv employs asynchronous partially inline dedup (APID) and aims to maintain the performance characteristics of LSM-tree-based KVS while reducing the write and space amplification problems. We implemented DenKv atop BlobDB and showed that our proposed solution maintains performance while reducing write amplification up to 2× and space amplification by 4× on average.