Introduce xarray-prism #1

mo-dkrz · 2026-01-06T08:56:06Z

In this PR we are going to introduce the xarray-prism as prism engine for opening all available climate data including remote and posix data.

How to install:

pip install xarray-prism

examples for remote data:

import xarray

nscc_data="https://thredds.ucar.edu/thredds/ncss/grid/grib/NCEP/GFS/Global_0p5deg_ana/GFS_Global_0p5deg_ana_20260104_1200.grib2?var=Temperature_altitude_above_msl&north=90.000&west=-180.000&east=180.000&south=-90.000&horizStride=1&time_start=2026-01-04T12:00:00Z&time_end=2026-01-04T12:00:00Z&&&accept=netcdf3"
xarray.open_dataset(nscc_data, engine="prism")

grib_remote="https://thredds.ucar.edu/thredds/fileServer/grib/NCEP/GFS/Global_0p25deg_ana/GFS_Global_0p25deg_ana_20260105_0600.grib2"
xarray.open_dataset(grib_remote, engine="prism")

nc3_remote="https://icdc.cen.uni-hamburg.de/thredds/fileServer/ftpthredds/ar5_sea_level_rise/gia_mean.nc"
xarray.open_dataset(nc3_remote, engine="prism")

opendap_data="https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/ar5_sea_level_rise/gia_mean.nc"
xarray.open_dataset(opendap_data, engine="prism")

tif_remote="https://github.com/mommermi/geotiff_sample/raw/refs/heads/master/sample.tif"
xarray.open_dataset(tif_remote, engine="prism")

…n gdal

…reating issue for unknown data

antarcticrainforest

Looks really good.

A couple for general comments.

There are some formats that aren't covered yet. It's some weird json reference files but I'd have to investigate myself.
I think the name freva-xarray or engine freva is to loaded. How about changing the name of this.

how about xarray-prism / prism, or xarray-switchboard, switchboard. Or xarray-gateway/gateway? Just some thoughts

antarcticrainforest · 2026-01-06T11:02:49Z

README.md

+> If you deal with a data that `freva` engine is not able to open that, please
+> report the data [here](https://github.com/freva-org/freva-xarray/issues/new)
+> to let us improve this engine to be able to be versitile and work with all
+> sort of climate data.


Suggested change

> If you deal with a data that `freva` engine is not able to open that, please

> report the data [here](https://github.com/freva-org/freva-xarray/issues/new)

> to let us improve this engine to be able to be versitile and work with all

> sort of climate data.

> If you encounter with a data formats that `freva` engine is not able to open, please

> files an issue report [here](https://github.com/freva-org/freva-xarray/issues/new).

> This helps us to improve the engine enabling users work with different kinds of climate data.

antarcticrainforest · 2026-01-06T11:10:20Z

src/freva_xarray/backends/cloud.py

+    extra_lines = 0
+    if show_progress:
+        fmt = "GRIB" if engine == "cfgrib" else "NetCDF3"
+        print(f"[warning] Remote {fmt} requires full file download")


warning.warn?

antarcticrainforest · 2026-01-06T11:14:34Z

src/freva_xarray/entrypoint.py

+    ) -> xr.Dataset:
+        """Xarray Generic function: Open dataset with
+        automatic format detection."""
+        if not isinstance(filename_or_obj, (str, Path)):


os.PathLike? This covers also fsspec stuff.

antarcticrainforest · 2026-01-06T11:15:05Z

src/freva_xarray/entrypoint.py

+        lines_printed = 0
+
+        if is_remote:
+            sys.stdout.write("[info] Detecting format...")


antarcticrainforest · 2026-01-06T11:17:02Z

src/freva_xarray/utils.py

+        self._render()
+
+    def _render(self) -> None:
+        mb = self._current / 1024 / 1024


Suggested change

mb = self._current / 1024 / 1024

mb = self._current / 1024**2

antarcticrainforest · 2026-01-06T11:24:42Z

@mannreis could you take a look at this and advertise if needed. This seems to be really nice!!!

mannreis

Really good work and Mo! I left my 2cents. My only concern is regarding caching the remote data which may not be, ever, an issue.

mannreis · 2026-01-06T12:43:47Z

src/freva_xarray/utils.py

+            pct = min(self._current / self._total, 1.0)
+            filled = int(self.width * pct)
+            bar = "█" * filled + "░" * (self.width - filled)
+            total_mb = self._total / 1024 / 1024


based on Martin's suggestion:

Suggested change

total_mb = self._total / 1024 / 1024

total_mb = self._total / 1024**2

mannreis · 2026-01-07T07:45:34Z

src/freva_xarray/_detection.py

+        return None
+
+
+def _detect_from_magic_bytes(header: bytes, lower_path: str) -> str:


Since typing is around is something like this an improvement? Or returning "unknown" defeats the effort?

Suggested change

def _detect_from_magic_bytes(header: bytes, lower_path: str) -> str:

Engine = Literal["cfgrib", "scipy", "h5netcdf", "rasterio", "unknown"]

def _detect_from_magic_bytes(header: bytes, lower_path: str) -> Engine:

I guess it would also have to be propagated up the call stack and for that reason inconvenient

mannreis · 2026-01-07T08:01:14Z

src/xarray_prism/backends/cloud.py

+
+    # GRIB: cache locally
+    if engine == "cfgrib":
+        local_path = _cache_remote_file(


Caching is good but I'm not sure whether is better to enable it explicitly or allow to disable.
Regardless I would not do it on /tmp for these applications which I believe is the default in many systems.

Thanks for the review. Regarding caching, I already though quite a bit about this, and for this purpose, we designed the FREVA_XARRAY_CACHE environment variable in place to user be able to change the cache location. Or user can also adjust it on storage_options argument of open_dataset function of xarray. Also when the user opening the remote files, progress logging appears to tell the user, it's storing data on /tmp and user sees the progress. It means users can cancel the procedure when it's downloading the data by watching the logs.

Also main purpose of designing the adjustable FREVA_XARRAY_CACHE was for our own data-loader. We wanted to adjust the cache on deployment to /scratch to don't run into storage limit.

Apart from those, we really don't have many other options for opening the grib and netcdf3 data. Because of the structure of those formats, we need to download the full data and then open it. So in short, /tmp was the safest way we could come up with. If there is any more secure suggestion, please let us know.

mannreis · 2026-01-07T08:08:57Z

I think the name freva-xarray or engine freva is to loaded. How about changing the name of this.

how about xarray-prism / prism, or xarray-switchboard, switchboard. Or xarray-gateway/gateway? Just some thoughts

I first didn't mind the name but then after seeing the structure of xarray backends, I see Martin's point. Unless I misunderstood something this could even be a contribution upstream to xarray, thus freva is not involved.

mo-dkrz · 2026-01-08T11:01:03Z

@antarcticrainforest thanks for the reviw.

regarding this point:

There are some formats that aren't covered yet. It's some weird json reference files but I'd have to investigate myself.

could you please give a link or example of this data to dig more for making it compatible with this?

mo-dkrz added 20 commits January 5, 2026 09:36

initial commit

22cedd6

setup ci workflow

3688990

use mambe on the ci to use pre-build wheels

1b52dd1

install no deps since conda is install it already to avoid conflict i…

b8000fd

…n gdal

enable site-package in tests of ci

ec2c3f9

add _version for release

e123017

add _version for release

c3d3120

fix relase ci

c4e8d63

import __version__ in __init__

f71f5d5

read from version in the ci

d760a5a

remove the py12 and py13 from black

5425337

apply black and fix flak8 errors

de9bde2

fix version importing

0f007ec

let's test TEST_PYPI_TOKEN if it's still valid

2ec541c

add id-token permission to make the ci work

4b3ba3d

fix github release

fa9a2e1

remove token api

5c04528

increase version to test

696e13e

get version only from _version

2f00905

support NSSC data

2935584

mo-dkrz requested a review from antarcticrainforest January 6, 2026 08:56

mo-dkrz added 3 commits January 6, 2026 11:48

add a lightweight terminal detector to be able to use hyperlink for c…

fbbd1f5

…reating issue for unknown data

increase test version

58be2d4

fix linter

eee8263

antarcticrainforest requested changes Jan 6, 2026

View reviewed changes

mannreis approved these changes Jan 7, 2026

View reviewed changes

mo-dkrz closed this Jan 7, 2026

freva-xarray to xarray-prism

eba4de4

mo-dkrz added 2 commits January 7, 2026 10:04

replace leftover of freva-xarray to xarray-prism

6f44b53

resolve the change requests on the first review round

afb8528

mo-dkrz changed the title ~~Introduce freva-xarray~~ Introduce xarray-prism Jan 8, 2026

mo-dkrz reopened this Jan 8, 2026

mo-dkrz requested a review from antarcticrainforest January 8, 2026 10:59

	mb = self._current / 1024 / 1024
	mb = self._current / 1024**2

	total_mb = self._total / 1024 / 1024
	total_mb = self._total / 1024**2

		return None


		def _detect_from_magic_bytes(header: bytes, lower_path: str) -> str:

-def _detect_from_magic_bytes(header: bytes, lower_path: str) -> str:
+Engine = Literal["cfgrib", "scipy", "h5netcdf", "rasterio", "unknown"]
+def _detect_from_magic_bytes(header: bytes, lower_path: str) -> Engine:

Introduce xarray-prism #1

Are you sure you want to change the base?

Introduce xarray-prism #1

Uh oh!

Conversation

mo-dkrz commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antarcticrainforest left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antarcticrainforest commented Jan 6, 2026

Uh oh!

mannreis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mo-dkrz Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mannreis commented Jan 7, 2026

Uh oh!

mo-dkrz commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mo-dkrz commented Jan 6, 2026 •

edited

Loading

mo-dkrz Jan 7, 2026 •

edited

Loading