EvaMAE: How Helpful Are DEM Data in Enhancing Geo-Foundation Models for Earth Imagery?

Numerous geo-foundation models have been pre-trained recently on plentiful unlabeled Earth imagery datasets by self-supervised learning, and they have been demonstrated to enhance performance in downstream supervised geospatial tasks such as flood extent mapping. However, these approaches generally ignore the terrain data that are readily available in the format of digital elevation model (DEM) from sources such as USGS's 3D Elevation Program (3DEP). On the other hand, a few works have shown that elevation guidance can improve the performance of flood extent mapping on conventional models trained from scratch. This is intuitive since in natural disaster events such as flooding, landslide and avalanche, the floodwater, loose earth or snow moves downhill.In this work, we explore the use of DEM data in geo-foundation models by introducing EvaMAE, a Masked Autoencoder (MAE) architecture that integrates elevation data for pre-training. Different strategies for incorporating DEM data are studied including convolution- and cross-attention-based approaches. We also explore the use of ControlNet to integrate DEM data during fine-tuning. Extensive experiments on downstream tasks such as flood and landslide segmentations demonstrate that (i) incorporating DEM data is helpful in both the pre-training and the fine-tuning stages, that (ii) the best-performing model for pre-training uses cross-attention to combine DEM and RGB features in both the MAE encoder and decoder, and that (iii) the best-performing model for fine-tuning uses ControlNet to incorporate DEM data. We also release a new large-scale annotated flood mapping dataset called EvaFlood used in our model training. All our code, pre-trained models, and the EvaFlood dataset are available at https://github.com/saugatadhikari/EvaMAE.