Skip to main content

Design Considerations and Analysis of Multi-Level Erasure Coding in Large-Scale Data Centers...

Publication Type
Conference Paper
Book Title
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
Publication Date
Page Numbers
1 to 13
Publisher Location
New York, New York, United States of America
Conference Name
International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 23)
Conference Location
Denver, Colorado, United States of America
Conference Sponsor
Conference Date

Multi-level erasure coding (MLEC) has seen large deployments in the field, but there is no in-depth study of design considerations for MLEC at scale. In this paper, we provide comprehensive design considerations and analysis of MLEC at scale. We introduce the design space of MLEC in multiple dimensions, including various code parameter selections, chunk placement schemes, and various repair methods. We quantify their performance and durability, and show which MLEC schemes and repair methods can provide the best tolerance against independent/correlated failures and reduce repair network traffic by orders of magnitude. To achieve this, we use various evaluation strategies including simulation, splitting, dynamic programming, and mathematical modeling. We also compare the performance and durability of MLEC with other EC schemes such as SLEC and LRC and show that MLEC can provide high durability with higher encoding throughput and less repair network traffic over both SLEC and LRC.