Supercomputing and Computation

SHARE

IO

ADIOS


The Adaptable IO System (ADIOS) is a parallel IO middleware designed for large-scale scientific simulations. The goal of this project is to provide fast, adaptable and scalable IO interfaces so that scientific codes can run highly efficiently across all computing platforms.ADIOS provides a simple, flexible way for scientists to describe the data in their code that may need to be written, read, or processed outside of the running simulation. By providing an external XML file describing the various elements, their types, and how you wish to process them, the routines in the host code can transparently change how they process the data. Based on the settings in this XML file, the data will be processed differently. For example, a user could select file IO, including MPI-IO, POSIX-IO, or staging IO, by which data can be streamed to auxiliary compute nodes through high-speed interconnections for further processing, without having to either change the source code or even recompile. So far, ADIOS has been integrated into many codes that have large allocation at leadership computing facilities, including S3D (combustion), GTC(fusion), GPC-P(fusion), XGC(fusion), SCEC(earthquake), Chimera (Astrophysics) etc, and significant IO improvements have been achieved for these codes. ADIOS is also adopted by DOE SDAV institute as a software framework that will further incorporate compression, indexing and other technologies into it.

The Adaptable IO System (ADIOS) provides a simple, flexible way for scientists to describe the data in their code that may need to be written, read, or processed outside of the running simulation. By providing an external to the code XML file describing the various elements, their types, and how you wish to process them this run, the routines in the host code (either Fortran or C) can transparently change how they process the data.

The in code IO routines were modeled after standard Fortran POSIX IO routines for simplicity and clarity. The additional complexity including organization into hierarchies, data type specifications, process grouping, and how to process the data is stored in an XML file that is read once on code startup. Based on the settings in this XML file, the data will be processed differently. For example, you could select MPI individual IO, MPI colletive IO, POSIX IO, an asynchronous IO technique, visualization engine, or even NULL for no output and cause the code to process the data differently without having to either change the source code or even recompile.

The real goal of this system is to give a level of adaptability such that the scientist can change how the IO in their code works simply by changing a single entry in the XML file and restarting the code. The ability to control at a per element basis and not just a data grouping such as a restart, diagnostic output, or analysis output makes this approach very flexible. Along with this detail level, a user can also just change which transport method is used for a data type such as a restart, analysis, or diagnostic write.

For the transport method implementer, the system provides a series of standard function calls to encode/decode data in the standardized .bp file format as well as “interactive” processing of the data by providing direct downcalls into the implementation for each data item written and also callbacks when processing a data stream once a data item has been identified along with its dimensions and a second callback once the data has been read giving the implementation the option to allocate memory and process the data as close to the data source as is reasonable.

Website: http://www.olcf.ornl.gov/center-projects/adios/

ASK ORNL

We're always happy to get feedback from our users. Please use the Comments form to send us your comments, questions, and observations.