to_netcdf: Unexpected drop of "units" attribute of attached "bounds" #8368

leonfoks · 2023-10-24T18:15:05Z

What happened?

When writing a Dataset to netcdf, any DataArrays that are linked as bounds through another variables attrs['bounds'] entry, have their (specifically) 'units' attribute dropped inside the written netcdf file.

See example

What did you expect to happen?

Units attribute to be written to the netcdf file.

Minimal Complete Verifiable Example

import numpy as np
import xarray as xr

# Create a new Dataset
ds = xr.Dataset()

# Add the x variable, Specify 'x_bnds' as bounds, defined later.
ds['x'] = xr.DataArray(np.arange(10), dims='x', attrs={'units':'m', 'bounds':'x_bnds'})

# Bounds require an extra dimension equal to number of vertices.
ds['nv'] = xr.DataArray(np.r_[0, 1], dims='nv')

# Add the actual bounding values for variable x.
ds['x_bnds'] = xr.DataArray(np.squeeze(np.dstack([np.arange(10)-0.5, np.arange(10)+0.5])),  
                            dims=['x', 'nv'],
                            attrs={'test':4, 'units':'m', })

print('Units is attached to the bounds in the dataset before writing', 'units' in ds['x_bnds'].attrs)

# Write to netcdf file
ds.to_netcdf('tmp.nc', format='netcdf4', engine='netcdf4')

# Open the dataset and check x_bnds attrs.  units is dropped.
new = xr.open_dataset('tmp.nc')
print(new['x_bnds'].attrs)

# Confirm that units were never written to the file.
!h5dump -d /x_bnds tmp.nc

MVCE confirmation

Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
Complete example — the example is self-contained, including all data and the text of any traceback.
Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
New issue — a search of GitHub Issues suggests this is not a duplicate.
Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ]
python-bits: 64
OS: Darwin
OS-release: 21.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.9.3-development

xarray: 2023.10.1
pandas: 2.1.1
numpy: 1.26.1
scipy: 1.11.3
netCDF4: 1.6.4
pydap: None
h5netcdf: None
h5py: 3.10.0
Nio: None
zarr: None
cftime: 1.6.3
nc_time_axis: None
PseudoNetCDF: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.8.0
cartopy: None
seaborn: None
numbagg: None
fsspec: None
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 68.0.0
pip: 23.3
conda: None
pytest: None
mypy: None
IPython: 8.16.1
sphinx: 7.2.6

welcome · 2023-10-24T18:15:08Z

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

dcherian · 2023-10-24T18:18:35Z

This behaviour is recommended by CF: http://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html#cell-boundaries

Boundary variable attributes which determine the coordinate type (units, standard_name, axis and positive) or those which affect the interpretation of the array values (units, calendar, leap_month, leap_year and month_lengths) must always agree exactly with the same attributes of its associated coordinate, scalar coordinate or auxiliary coordinate variable. To avoid duplication, however, it is recommended that these are not provided to a boundary variable.

but I agree, we should just write what the user has assigned.

leonfoks · 2023-10-24T18:37:16Z

Okay thank you! I was wondering whether it was in fact expected behaviour.

st-bender · 2024-04-09T11:03:14Z

Hi,
I just observed the same thing and was scratching my head where they got amiss.
It is not even documented.
Apparently the change was introduced here: #2921 (issue) and #2965 (PR).
Please make this dropping somehow explicit, by a keyword or something.

Edit: you can remove lines 797--815 in xarray/conventions.py, which looks
like a whole bunch of attributes that get removed. But sometimes having them
(twice) in "bounds" variables is desired.

Cheers.

Issue pydata#2921 is about mismatching time units between a time variable and its "bounds" companion. However, pydata#2965 does more than fixing pydata#2921, it removes all double attributes from "bounds" variables which has the undesired side effect that there is currently no way to save them to netcdf with xarray. Since the mentioned link is a recommendation and not a hard requirement for CF compliance, these attributes should be left to the caller to prepare the dataset variables appropriately if required. Reduces the amount of surprise that attributes are not written to disk and fixes pydata#8368.

leonfoks added bug needs triage Issue that has not been reviewed by xarray team member labels Oct 24, 2023

dcherian added topic-CF conventions needs triage Issue that has not been reviewed by xarray team member and removed needs triage Issue that has not been reviewed by xarray team member labels Oct 24, 2023

dcherian removed the needs triage Issue that has not been reviewed by xarray team member label Oct 24, 2023

st-bender linked a pull request Apr 10, 2024 that will close this issue

Keep attributes for "bounds" variables #8924

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

to_netcdf: Unexpected drop of "units" attribute of attached "bounds" #8368

to_netcdf: Unexpected drop of "units" attribute of attached "bounds" #8368

leonfoks commented Oct 24, 2023

INSTALLED VERSIONS

welcome bot commented Oct 24, 2023

dcherian commented Oct 24, 2023

leonfoks commented Oct 24, 2023

st-bender commented Apr 9, 2024 •

edited

to_netcdf: Unexpected drop of "units" attribute of attached "bounds" #8368

to_netcdf: Unexpected drop of "units" attribute of attached "bounds" #8368

Comments

leonfoks commented Oct 24, 2023

What happened?

What did you expect to happen?

Minimal Complete Verifiable Example

MVCE confirmation

Relevant log output

Anything else we need to know?

Environment

INSTALLED VERSIONS

welcome bot commented Oct 24, 2023

dcherian commented Oct 24, 2023

leonfoks commented Oct 24, 2023

st-bender commented Apr 9, 2024 • edited

st-bender commented Apr 9, 2024 •

edited