NeurIPS 2026 · Position Paper Track

No One Knows the State of the Art in Geospatial Foundation Models

  • Isaac Corley1
  • Nils Lehmann2
  • Caleb Robinson3
  • Gabriel Tseng4
  • Anthony Fuller5,6
  • Hamed Alemohammad7
  • Evan Shelhamer5,8
  • Jennifer Marcus1
  • Hannah Kerner1,9

1Taylor Geospatial 2Technical University of Munich 3Microsoft AI for Good Research Lab 4Allen Institute for AI 5Vector Institute 6Carleton University 7Clark University 8University of British Columbia 9Arizona State University

152 papers audited, 2019–2025
47 of 252 repeated results differ by at least 5 points between papers
94/126 papers pretrain on a set of datasets no other paper uses
39% of papers release no model weights

Position

Nobody knows the current state of the art for geospatial foundation models.

Papers in the audited corpus do not share an evaluation framework, they report different numbers for the same model on the same benchmark, and they change architecture and pretraining data together without ablations that separate the two. We think this is a coordination failure and not the fault of any one lab, and the authors of this paper have contributed to it.

Findings

Five trends in the 152-paper corpus

T1 Model weights are not released

Of the 152 papers, 59 (39%) release no model weights. Twenty-six say so explicitly, and we could not find weights for the other 33. Twenty-nine of the 59 provide public code but no weights, so reusing those models requires retraining them.

T2 No shared core of benchmarks

The papers evaluate on 401 distinct benchmarks in 1,046 evaluations. EuroSAT, NWPU-RESISC45, and AID are the most used, yet together they account for 10.6% of evaluations, and 50 of 143 papers (35%) use none of the ten most common benchmarks. The Gini coefficient of benchmark usage has been stable since 2023, while the number of benchmarks used by a single paper grew from 13 in 2022 to 98 in 2025.

Top-10 of 401 benchmarks by paper count: EuroSAT 42, NWPU-RESISC45 36, AID 33, BigEarthNet-S2 32, UCMerced 28, Potsdam 22, OSCD 19, DIOR 17, fMoW 15, LEVIR-CD 13. Gini 0.51.
Ten most-used of 401 benchmarks, by paper count.
Histogram of per-paper overlap with the top-10 benchmarks; mean 0.27, 50 of 143 papers at zero overlap.
Per-paper overlap with the ten most-used benchmarks.
Gini coefficient of benchmark usage by publication year, stable after 2023.
Gini coefficient of benchmark usage by year.

T3 Reported numbers disagree across papers

We grouped the 11,488 reported results by model, benchmark, metric, evaluation protocol, and training regime. After excluding generic baselines and detection benchmarks, 252 groups are reported by at least two papers. The reported values differ by at least 5 points in 47 of them (18.7%), by at least 10 in 18, and by at least 20 in 6. For example, CROMA's mIoU on m-Cashew-Plantation is 30.1 in DOFA and 62.2 in SatDiFuser, and the papers do not give enough detail to tell whether the decoders and evaluation settings were the same. The median spread is 0.1 points and the 90th percentile is 8.3.

Of 252 recipe tuples, 47 spread at least 5 points, 18 at least 10, and 6 at least 20.
Cross-paper spread within the same model, benchmark, metric, and protocol.
Top-10 reported disagreements, led by GPT-4o on UCMerced (45 points) and CROMA on m-Cashew-Plantation (32 points).
Ten largest reported disagreements.
Ten largest disagreements, with the setup detail the papers leave unspecified
ModelBenchmarkMetric / eval#MinMaxMin / max papersLikely missing setup detail
GPT-4oUCMercedAcc. / 0-shot243.588.8RingMo-Agent; EarthDialIdentical prompts, API snapshot, scoring, and samples are not established.
CROMAm-CashewmIoU / linear430.162.2DOFA; SatDiFuserDecoder details are weakly described.
ViT-22kOSCDRecall / FT222.552.4GFM; SpectralGPTGFM describes 192×192 tiles; SpectralGPT describes 128×128 crops with 50% overlap. Unclear why that would produce such a gap.
DOFAm-CashewmIoU / linear327.753.9TerraFM; SatDiFuserDecoder details are weakly described.
Satlasm-CashewmIoU / linear225.149.1TerraFM; SatDiFuserTerraFM reports the same Satlas score as Galileo, but which decoder produced it is unclear.
ViT-22kOSCDF1 / FT330.752.2GFM; SpectralGPTThe papers do not establish that they use the same evaluation recipe.
CaCoNWPU-RESISC45Acc. / FT268.688.3CtxMIM; SelectiveMAECtxMIM's head-fine-tuning description is ambiguous.
Scale-MAENWPU-RESISC45Acc. / FT778.095.7CtxMIM; Scale-MAEFreezing is unclear in CtxMIM. The same score in several papers does not show each reran it.
GPT-4oAIDAcc. / 0-shot258.374.7RingMo-Agent; EarthDialIdentical samples, prompts, answer mapping, and API snapshot are not established.
ResNet-50OSCDRecall / FT325.138.9GFM; SpectralGPTGFM labels it ResNet-50, but its scores match a ResNet-18 run in SeCo.

T4 Dataset bundles without a shared harness

GEO-Bench, PANGAEA, FoMo-Bench, and PhilEO Bench provide curated datasets and splits, but each researcher still writes their own training and evaluation code, so the optimizer, learning rate, evaluation crop, and IoU averaging vary between papers that use the same dataset. The field has no counterpart to lm-evaluation-harness, a versioned tool that every model owner runs to produce a reported number.

T5 Architecture confounded with pretraining data

The 126 papers that name a pretraining dataset use 87 distinct datasets, and the most common, MillionAID, appears in 9 papers. Only 32 papers (25%) pretrain on the same set of datasets as any other paper, and the other 94 each use a unique set. When a paper changes both the model and the data without holding one fixed, readers cannot tell which change produced the gain.

Figure: ten most-used pretraining datasets

Recommendations

Six recommendations for authors, reviewers, and venues

The paper also gives a reviewer checklist (C1–C6) that maps to these.

  1. R1

    Release weights under a named license

    Release them by camera-ready, or state the constraint that prevents release.

  2. R2

    Report on a shared core of benchmarks

    Use core benchmarks that fit the model's modality and claimed tasks, and state the protocol.

  3. R3

    Mark each baseline as copied or rerun

    Copied rows cite the source paper and its protocol, and rerun rows give the configuration.

  4. R4

    Report seed variability

    Give mean ± std over at least three runs where affordable, and otherwise say the result is from one run.

  5. R5

    Build a shared evaluation harness

    A third party maintains one tool with versioned protocols and automated submission checks.

  6. R6

    Separate data from architecture and algorithm

    When more than one changes, include a controlled run on a public pretraining dataset.

Data

Dataset

Two JSON Lines files contain the 152 paper records and the 11,488 normalized benchmark scores used in the paper. The data dictionary describes each key and how to match scores to papers.

Cite

@inproceedings{corley2026noone,
  title     = {No One Knows the State of the Art in
               Geospatial Foundation Models},
  author    = {Corley, Isaac and Lehmann, Nils and
               Robinson, Caleb and Tseng, Gabriel and
               Fuller, Anthony and Alemohammad, Hamed and
               Shelhamer, Evan and Marcus, Jennifer and
               Kerner, Hannah},
  booktitle = {Advances in Neural Information Processing
               Systems (NeurIPS), Position Paper Track},
  year      = {2026},
  eprint    = {2605.12678},
  archivePrefix = {arXiv}
}