add tmp file write/read step

[j-atkins]

This PR implements a write-to-tmp files step, to improve performance for Parcels v4 fieldset ingestion combined with copernicusmarine streaming (i.e. #357).

UPDATE: As posted in comment below:

This is now ready for review and appears the most performant strategy for VirtualShip for now.

• New test for _via_tmp_ds()

Please note test failures are expected, and will be addressed in root #342 PR

[j-atkins]

@erikvansebille, initial experimentation shows that the tmp file performance fix is indeed a simple implementation and substantially improves actual computation performance, but the tradeoff here is a slow-down by the fact that VirtualShip, in most cases, calls to global datasets from copernicusmarine.

In the case of e.g. a Drifter simulation the ds's (single level) are therefore ~6GB and are not super quick to write/read to/from disk, resulting in a simple single Drifter simulation (63 day lifetime) taking about 3.5 minutes all in, the majority of which time is writing the file to disk. Should note though that this is still a performance increase relative to .to_windowed_array() attempts!

A .sel() to a smaller spatial window (see quick example below) doesn't improve much, presumably something to do with the comms across the streaming.

  def _via_tmp_ds(self, ds) -> xr.Dataset:
      """Create and re-load a temporary local dataset."""
      tmpdir = tempfile.TemporaryDirectory()
      tmp_fpath = Path(tmpdir.name).joinpath("tmp.nc")

      # reduce dataset to smaller spatial window to reduce filesize
      HALO_DEGREES = 20.0  # TODO ...arbitrary but want a healthy buffer for drifters
      ds_minimised = ds.sel(
          lat=slice(self.min_lat - HALO_DEGREES, self.max_lat + HALO_DEGREES),
          lon=slice(self.min_lon - HALO_DEGREES, self.max_lon + HALO_DEGREES),
      )

      ds_minimised.to_netcdf(tmp_fpath)
      del ds, ds_minimised

      return xr.open_dataset(tmp_fpath)

I'll experiment further with file size reduction strategies (e.g. at the copernicusmarine.open_dataset() stage) to improve speed.

Of course the motivation for calling to global datasets in the first place was to avoid any risk of out of bounds errors so that will also be a priority.

[j-atkins]

In the case of e.g. a Drifter simulation the ds's (single level) are therefore ~6GB and are not super quick to write/read >to/from disk, resulting in a simple single Drifter simulation (63 day lifetime) taking about 3.5 minutes all in, the majority of >which time is writing the file to disk. Should note though that this is still a performance increase relative to .to_windowed_array() attempts!

Following up on this, the file writing speeds don't appear to be overwhelmingly tied to the the spatial extent of the requested ds, in fact they appear more sensitive to the temporal extent (perhaps to do with chunking strategy somewhere on the copernicusmarine end).

Performance for other instruments, such as CTDs and Argo Floats, is pretty good relative to using Parcels v3 though. So, overall, it's promising and it appears that it's just Drifters which slow down (a bit but not massively) relative to v3.

[j-atkins]

I think it would probably be a good idea to reduce the file sizes for Drifter simulations, even if just tmp files, because ~6GB global datasets feel a bit unnecessary and potentially problematic. Even adding a generous spatial buffer (+/- 30 degrees) would reduce the size substantially whilst avoiding out of bounds errors in nearly all cases.

This would effectively create a hard limit on lifetimes (velocity dependent) but perhaps a necessary (temporary) trade-off for now. Also, at least this wouldn't crash the simulations, as of #348...

What do you think @erikvansebille?

[erikvansebille]

I think it would probably be a good idea to reduce the file sizes for Drifter simulations, even if just tmp files, because ~6GB global datasets feel a bit unnecessary and potentially problematic. Even adding a generous spatial buffer (+/- 30 degrees) would reduce the size substantially whilst avoiding out of bounds errors in nearly all cases.

This would effectively create a hard limit on lifetimes (velocity dependent) but perhaps a necessary (temporary) trade-off for now. Also, at least this wouldn't crash the simulations, as of #348...

What do you think @erikvansebille?

Yes, I agree that using a buffer for the drifters is the most effective solution for now.

Hopefully, as we improve the streaming performance in Parcels v4 (see Parcels-code/parcels-benchmarks#49), we can then remove the buffer again. But I have no idea if that performance improvement will take one week or half a year...

[j-atkins]

This is now ready for review and appears the most performant strategy for VirtualShip for now.

• New test for _via_tmp_ds()

[erikvansebille]

Looks good, just one comment below

[erikvansebille]

Is it worth checking/making sure what the backend is, here? Either Dask or NumPy? See also the (draft) performance guide at https://parcels--2705.org.readthedocs.build/en/2705/user_guide/examples/explanation_performance.html.

If it turns out to use Dask under the hood, then make sure to also do a to_windowed_arrays call; otherwise performance could still be poor

[j-atkins]

The fieldsets are consistently NumPy backed across different instruments and checked against large/long expeditions. As discussed though, I have added to_windowed_arrays as a safe option, just in case there are rare scenarios where Dask is used.