Not an issue but more asking on advice/help on data binning (similar to variable time)

[ivicajan]

Thanks for the package you develop as this could be game changer in my case.
I'm using a large datasets and performing analysis that would benefit a lot from DASK/Xarray.
What I am after is related to ocean glider data that is recording ocean temperature, salinity and depth along its path.
All variables and coordinates have the same dimension - that is TIME (as glider is flying along the path - defined with increasing time).
Coordinates are TIME, LON, LAT, DEPTH and are having dimension TIME.

What I would like to do is a binning of say TEMP variable on the standard - predefined depths (bins). That would have the variable time records falling in the specific depth bins. As glider goes in "v" shape pattern (up and down) it is sampling the ocean and recording the data in time.. DEPTH varies from the surface to the bottom during the multiple dives... and I would like to have binned data within depth cells.

Happy to provide data and jupyter notebook if needed.

Thanks in advance,
Ivica

[Descanonge]

Hello,

I didn't understand if you wanted to only separate data by depth or if you wanted to also bin your variable.
This package can only do the latter, using a 2D histogram. Something along the lines of:

from xarray_histogram import histogram2d
hist = histogram2d(ds.temp, ds.depth, ...)

which would be of dimensions (temp_bins, depth_bins).

If you want to separate data by depth I guess groupby_bins would be the way to go, something like ds.groupby_bins("depth", bins=...).

I'd be happy to help further

[ivicajan]

Hi,
thanks for getting back. I need binning of my data into fixed depth cells.Think it is not working that easy as DEPTH coordinate is not monotonically increasing (glider dives and then goes back to the surface and again for many dives)...
This is a snip of the code you can test:

`import xarray as xr

url = 'https://thredds.aodn.org.au/thredds/dodsC/IMOS/ANFOG/slocum_glider/Pilbara20130919/IMOS_ANFOG_BCEOPSTUV_20130919T030752Z_SL287_FV01_timeseries_END-20131009T002816Z.nc'

ds = xr.open_dataset(url)
vars = ('TIME', 'DEPTH', 'TEMP')

drop_vars = []
for v in list(ds):
if (v in vars):
continue
else:
drop_vars.append(v)
ds.close()
ds = xr.open_dataset(url, drop_variables = drop_vars)
`

The problem is now to bin the data into vertical cells of 2m:
i.e. ddepth = np.arange(5,200,2)

I should get TEMP variable binned within ddepth cells and I guess TIME will be binned as well (time records that falls into depth bins). Need stats within each cell, something like mean, max, min, var within each cell, and that is easy to do if I get proper structure.

Cheers,
Ivica

[Descanonge]

Thanks for the snippet this is useful.

I still think the two techniques would work. I tested them on a small subset and the mean seems correct. Xarray deals well with the depth not being monotonic (you can check the indices selected in each depth cell by looking at ds.groupby_bins(...).groups).

import boost_histogram as bh
import xarray as xr
from xarray_histogram import histogram2d
import xarray_histogram.accessor

url = "https://thredds.aodn.org.au/thredds/dodsC/IMOS/ANFOG/slocum_glider/Pilbara20130919/IMOS_ANFOG_BCEOPSTUV_20130919T030752Z_SL287_FV01_timeseries_END-20131009T002816Z.nc"

ds = xr.open_dataset(url)
variables = ("TIME", "DEPTH", "TEMP")

ds = ds.drop_vars([v for v in ds if v not in variables])
ds_cut = ds.isel(TIME=slice(0, 500))

# Using histogarm
depth_bins = bh.axis.Regular(97, 5, 199)
hist = histogram2d(ds_cut.TEMP, ds_cut.DEPTH, bins=[50, depth_bins])
mean_hist = hist.hist.mean("TEMP")

# Using group
grp = ds_cut.groupby_bins("DEPTH", depth_bins.edges)
mean_grp = grp.mean()

# Comparison
hist.plot.imshow(cmap="Blues")
mean_grp.TEMP.plot.line(color="r")
mean_hist.plot.line(color="k", ls="--")

Of course using an histogram creates an intermediate step, but if the temperature bins are correctly choosen (I haven't looked at it) and there is enough data in each depth cell, the histogram should accurately represent it. I used the package new accessor to compute the mean accross all depth cells and you can do other similar statistical computations (https://xarray-histogram.readthedocs.io/en/latest/accessor.html)

Using groupby_bins is more direct but I have no experience in how well it would scale with so much data. You should make sure flox is installed to improve performance (https://docs.xarray.dev/en/stable/user-guide/groupby.html).
I think in both cases data should be chunked along the time dimension ds = ds.chunk(TIME=8192).

[ivicajan]

Thanks for the swift reply,
will do some testing and report back.

Just wondering how well the xarray-histogram is integrated within Dask system?
Using dask inside my analysis makes things go fast and just doing quick test you provided didn't show histogram using workers (or I might done something wrong)..

Cheers,
Ivica

[Descanonge]

Yes the package integrates with Dask. To use it, you should chunk along the time dimension.
I would recommend to compute/load the histogram and then the statistics.

ds = ds.chunk(TIME=8192)  (I choose the number arbitrarily)
hist = histogram2d(...)
hist = hist.compute()

mean = hist.hist.mean("TEMP")
...

[ivicajan]

Thanks for suggestions.
In case I want to bin data but keep TIME dimension (i.e. binning TIME data using depth bins) what would you suggest?
Simple way would be to loop over each dive (glider going down then up) and then compute xhistogram along dive (I can find start and end of dive as change in pressure) and then jump on dive going up and repeat binning. That way I would get "new time coordinate" that is binned too, depth bins that are constant and binned data (i.e. binned temp along each dive that is having dimension the same as "new time"). I could do pcolormesh(new_time, depth_bins, temp_bins)
Hope you got what I mean.

Cheers,
Ivica