Training large neural network models is one of the most energy-intensive activities in modern computing. This technology introduces phantom parallelism, a novel training strategy that significantly reduces the total energy consumed during large-scale neural network training and inferencing. Experiments on the FRONTIER supercomputer using up to 256 GPUs demonstrated approximately 50% reduction in energy consumption to reach the same training loss of large feed forward networks. Additionally, smaller phantom models on fewer GPUs were shown to achieve the same performance as larger tensor parallel feed forward network models on far greater GPU counts. Together, the new training alternative offers the potential for even greater energy savings at scale and a path for organizations to lower operational costs and improved performance for large AI and machine learning applications.
Description
This technology introduces a distributed training methodology for large neural network models that reduces communication overhead and computational burden during model training and inferencing. The approach uses a modified parallel architecture that strategically compresses and exchanges information between distributed processors to improve scaling and energy-efficiency in multi-GPU environments. By compressing intermediate activations before exchanging them, the approach enables more energy-efficient training across distributed GPU environments without compromising model performance.
Unlike conventional tensor parallel methods, the new approach reduces both data transfer volume and floating-point operations per training iteration. Together with the fact that phantom models are structurally smaller, they require fewer iterations to reach the same training loss while substantially lowering the total energy consumption. The energy advantage of the method has been benchmarked with standalone feed-forward neural network architectures with a clear extension path to transformer models, where feed-forward sub-blocks appear after every attention layer.
Testing on large GPU configurations demonstrated substantial reductions in training energy consumption compared with traditional tensor parallel approaches. The technology also demonstrated the ability to achieve comparable model performance using fewer computing resources which has the clear potential to reduce infrastructure demands and improve scalability for large AI workloads.
Benefits
- Reduces communication overhead in distributed AI training
- Lowers computational and operational resource requirements
- Improves scalability across multi-GPU computing environments
- Maintains model performance while using fewer computing resources
- Compatible with existing high-performance computing infrastructure
Applications and Industries
- Large language model training
- Artificial intelligence and machine learning platforms
- High-performance computing systems
- Data center and cloud computing operations
- Advanced analytics and scientific computing
- Government and defense AI applications
Contact
To learn more about this technology, email [email protected] or call 865-574-1051.
Researchers
Contact
To learn more about this technology, email [email protected] or call 865-574-1051.