CODA: Breaking CPU Bottlenecks for Scalable Ceph with DPU-Based OSD Decoupling

Abstract

Ceph, a widely used distributed storage system, underutilizes NVMe SSD bandwidth due to CPU bottlenecks, even in high-performance environments. This inefficiency is amplified in resource-constrained settings such as edge clouds and small-scale clusters. Prior work offloads the transport layer (e.g., TCP/IP) to DPUs, but fails to address the dominant overheads arising from OSD-level logic, including message handling and replication coordination. We present CODA, a Ceph OSD Decoupling Architecture that separates the OSD stack and offloads CPU-intensive front-end logic to the DPU. It runs Messenger and OSD core on the DPU while keeping BlueStore on the host, enabling a disaggregated OSD across heterogeneous devices. We further design: (i) Flow-Direct for split execution, (ii) Flow-Adapt for batching and pipelining under DMA constraints, and (iii) Flow-Sync for low-overhead synchronization under OSD scaling. We implement CODA on a Ceph cluster with NVIDIA BlueField-3 DPUs. CODA reduces host CPU usage by up to 88% and improves throughput by up to 49% without replication and 94% with 3-way replication.