Data Analysis on High Performance Computing Platforms

Abstract

Leadership-class computing facilities, with compute and storage in the peta-scale are one of the largest sources of ‘big data’. The compute intensive applications and data analytical tools rely on very different system architectures, which poses a challenge for large-scale computing centers to efficiently use the existing peta-scale infrastructure for data analytics. In this paper, we describe and evaluate the deployment of the Hadoop MapReduce analytical framework, on a shared, high-performance parallel file system. Our file system, Spider, is a Lustre-based parallel file system actively used across several computing clusters, including Titan, one of the world’s fastest supercomputers. Although Hadoop is designed for a very different architecture than ours (i.e., shared-nothing), we demonstrate that Hadoop running over a shared file system is practical and can yield some significant performance benefits. To our knowledge, this paper is the first to provide an exhaustive and rigorous experimental evaluation of an on-demand Hadoop instance running over a shared, Lustre parallel file system. We evaluated our implementation across both Lustre and Hadoop specific parameters using standard benchmarks. Our evaluation was done while Spider was in active use, thus allowing us to capture realistic performance data.