[Feature]: Embedded Seamless Data

[adamjstewart]

Problem

I'm interested in using this repo to quickly replace the images in existing benchmark datasets with embeddings for ALL existing precomputed embeddings. Going off the list from https://torchgeo.readthedocs.io/en/stable/api/datasets.html#embeddings-datasets, you already have quite a lot of the pixel embeddings.

Proposed solution

I would be interested in adding Embedded Seamless Data (ESD) (paper). I'm wondering if anyone has looked into this one (I realize it's quite new) and how difficult it would be to connect that API. I'm happy to hack on this if it seems feasible.

Example or context

emb = get_embedding("esd", spatial=..., temporal=..., output=...)

Notes

No response

Checks

• I searched existing issues and pull requests before opening this request.

[Dinghye]

Hi @adamjstewart ! Thanks for your interest in rs-embed!

Extension

Adding ESD as a pre-computed embedding should be quite straightforward. In practice, you would mainly need to register the model in catalog.py and add the implementation in embedders/precomputed_esa.py.

Following the extension guide, the minimum you need to implement is describe() and get_embedding(). If you have a way to accelerate retrieval for multiple images at once, you can also implement get_embeddings_batch(). Otherwise, batch embedding retrieval will automatically fall back to loop calling get_embedding().

Benchmark

Using rs-embed to generate embeddings for all benchmarks may be a bit more challenging. In particular:

• Our current querying pipeline is based on time and space, so if you want to serve your own local data and obtain embeddings from it, you would likely need to extend the backend yourself, for example by replacing the current GEE-based backend with a local one.
• Some models are temporal and require multiple images as input, such as AnySat and AgriFM.

We have thought about supporting a local backend before, but we did not find a clean way to integrate it into the current interface, so we have not implemented it yet. However, we would be very happy to extend the framework further and collaborate on solving these issues together. 🍻

Also, if your goal is to produce reusable embeddings for benchmarks, understanding and adjusting the settings can be very important. In rs-embed, some defaults are chosen to balance computational cost, so they are not always the best possible settings for embedding quality. This guide may be helpful.

[adamjstewart]

Started digging into this. ESDNet appears to be closed source, and I haven't yet found a public API for downloading precomputed embeddings from iEarth DataHub. Contacted both parties to try to make progress on this.