Skip to main content
SHARE
Publication

OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

Publication Type
Conference Paper
Book Title
SIGSPATIAL '24: Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems
Publication Date
Page Numbers
597 to 600
Publisher Location
New York, New York, United States of America
Conference Name
ACM SIGSPATIAL: International Conference on Advances in Geographic Information Systems
Conference Location
Atlanta, Georgia, United States of America
Conference Sponsor
ACM
Conference Date
-

While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.