Skip to main content

Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid...

Publication Type
Conference Paper
Book Title
Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI. SMC 2020.
Publication Date
Page Numbers
175 to 186
Conference Name
Smoky Mountains Computational Sciences and Engineering Conference (SMC)
Conference Location
Kingsport, Tennessee, United States of America
Conference Sponsor
Conference Date

Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s world-class high performance computing (HPC) facilities, and (2) our short-term efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows. We propose a more disruptive domain-specific solution: the No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.