Skip to main content

An Agenda for Multimodal Foundation Models for Earth Observation...

Publication Type
Conference Paper
Book Title
IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium
Publication Date
Page Numbers
1237 to 1240
Publisher Location
New Jersey, United States of America
Conference Name
International Geoscience and Remote Sensing Symposium 2023 (IGARSS)
Conference Location
Pasadena, California, United States of America
Conference Sponsor
Conference Date

Archives of remote sensing (RS) data are increasing swiftly as new sensing modalities with enhanced spatiotemporal resolution become operational. While promising new breakthroughs, the sheer volume of RS archives stretches the limits of human analysts and existing AI tools, as most models are: i) limited to single data modalities; ii) task-specific; iii) heavily reliant on labeled data. The emerging Foundation Models (FMs) have the potential to address these limitations. Trained on vast unlabeled datasets through self-supervised learning, FMs enable generic feature extraction that facilitate specialization to a wide variety of downstream tasks. This paper describes a vision towards an FM for multimodal Earth Observation data (FM4EO), discussing key building blocks and open challenges. We put particular emphasis on multimodal reasoning, a topic underexplored in EO. Our ultimate goal is a practical path toward FM4EO with capacity to unlock breakthroughs in few-shot learning scenarios, multimodal geographic knowledge integration, synthesis, and hypothesis generation.