Optimizing the end-to-end management of scientific data to enable knowledge discovery
The mission of the Workflow Systems Group is to advance scientific discovery through innovative computational methods that make complex data more accessible, efficient, and meaningful. Group members develop cutting-edge solutions for managing and analyzing massive scientific datasets, with a particular focus on AI, high-performance computing (HPC), and data reduction techniques that reduce energy consumption while maintaining scientific accuracy. Through this work, including the following efforts, the group aims to accelerate breakthroughs across multiple scientific domains while promoting sustainable computing practices that will shape the future of scientific research:
- Work on AI/ML and HPC
- Focus on managing and analyzing large scientific datasets
- Emphasis on energy efficiency and sustainability
- Development of data compression techniques
- Cross-cutting impact across multiple scientific domains
- Balance between computational efficiency and scientific accuracy
Members of the Workflow Systems Group are advancing the frontiers of scientific computing by leveraging innovative approaches to coupled workflow automation, scientific data management, and data reduction. HPC and AI permeate all areas of the group’s research, allowing us to work with hundreds of mission-critical applications, including areas of fusion energy, accelerator science, aerospace, and radio astronomy.
Software tools the group has created include:
The Adaptable Input/Output System (ADIOS) is a high-performance I/O middleware developed to efficiently manage the storage, retrieval, and movement of large-scale scientific data in HPC environments. ADIOS provides a flexible, scalable, and portable framework for reading and writing data from simulations, experiments, or instruments. It supports a variety of data transport methods and file formats, allowing users to optimize I/O performance based on their application’s needs. With its support for in situ data processing and tight integration with analysis and visualization tools, ADIOS plays a critical role in reducing I/O bottlenecks and enabling real-time data insights in exascale and data-intensive computing workflows.
MultiGrid Adaptive Reduction of Data (MGARD) is a high-performance, error-bounded compression and refactoring framework designed for scientific data. Built on multigrid numerical methods, MGARD enables efficient storage and transmission of large, multidimensional floating point datasets by decomposing them hierarchically and preserving user-specified accuracy. It supports in situ workflows, progressive data access, and AI readiness by maintaining critical data features while significantly reducing size. MGARD is widely used in HPC applications to overcome I/O and storage limitations without sacrificing scientific fidelity.
SimGrid is an open-source simulation framework for modeling and evaluating the performance of large-scale distributed systems such as HPC platforms, cloud infrastructures, and peer-to-peer networks. It enables users to simulate system behavior with high scalability and accuracy by using detailed models of computation and communication. With support for Message Passing Interface application simulation and customizable scheduling policies, SimGrid is widely used in research to study resource allocation, fault tolerance, and energy efficiency, thereby allowing reproducible experiments without the need for physical deployments.
Experimental Framework for Integrated Scientific Simulations (EFFIS) is a software ecosystem developed by the U.S. Department of Energy that is designed to support large-scale scientific workflows by enabling the integration of simulation, data management, and in situ analysis. Built to run on HPC systems, EFFIS provides tools for orchestrating complex multiphysics simulations, managing data flow between components, and performing real-time monitoring and visualization. By streamlining these tasks, EFFIS enhances productivity and scientific insight, especially in exascale environments where efficient coordination of computation and data is critical.
Campaign Management of Scientific Data (CMSD) refers to the coordinated planning, collection, processing, storage, and analysis of data generated during large-scale scientific experiments or simulations. It involves organizing data across distributed teams and facilities to ensure consistency, traceability, and compliance with data standards. Effective campaign management enables seamless integration of simulation and experimental results, supports real-time decision-making, and ensures data is accessible, well documented, and AI ready for future analysis. It is critical for maximizing scientific return, particularly in complex, multi-institutional research efforts such as fusion energy, climate modeling, or high energy physics.
The Workflow Systems Group’s work is distinguished by its practical application across multiple scientific domains. The group’s tools and methods are being used in fusion energy research, materials science, and cosmology to help address fundamental questions about the universe while advancing sustainable computing practices. Its innovations in energy-efficient computing and data management are particularly relevant as scientific computing faces an increasing number of sustainability challenges.
