Skip to main content
SHARE
Publication

Parallel Hybrid Metaheuristics with Distributed Intensification and Diversification for Large-scale Optimization in Big Data Statistical Analysis

by Wendy Cho, Yan Liu
Publication Type
Conference Paper
Book Title
Proceedings of the 2019 IEEE International Conference on Big Data (BigData 2019)
Publication Date
Page Numbers
1 to 9
Publisher Location
Washington, District of Columbia, United States of America
Conference Name
2019 IEEE International Conference on Big Data
Conference Location
Los Angeles, California, United States of America
Conference Sponsor
IEEE
Conference Date
-

Important insights into many problems that are traditionally analyzed via statistical models can be obtained by re-formulating and evaluating within a large-scale optimization framework. The theoretical underpinnings of the statistical model often shift the goal of the solution space traversal from a traditional search for a single optimal solution to a traversal with the purpose of yielding a set of high quality, independent solutions. We examine statistical frameworks with astronomical solution spaces where the independence requirement constitutes a significant additional challenge for standard optimization methodologies. We design a hybrid metaheuristic with specialized intensification and diversification protocols in the base search algorithm. We extend our algorithm to the high-performance-computing realm using the Stampede2 supercomputer. We experimentally demonstrate the effectiveness of our algorithm to utilize multiple processors to collaboratively hill climb, broadcast messages to one another regarding landscape characteristics, diversify across the solution landscape, and request aid in climbing particularly difficult peaks.