error from netCDF in cf.write

[JonathanGregory]

I am processing some CMIP6 data on Jasmin (making time-means). I get the following error. I realise that you probably need more background to diagnose it, but perhaps it's obvious from this much information? As a result of the error, the file is not properly written, just the start of it.

Thanks. Jonathan

Traceback (most recent call last):
  File "/home/users/jonathan/bin/grouptime", line 97, in <module>
    cf.write(n,ofile)
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/write.py", line 826, in __new__
    netcdf.write(
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5232, in write
    self._file_io_iteration(
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5478, in _file_io_iteration
    self._write_global_attributes(fields)
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4610, in _write_global_attributes
    g["netcdf"].setncattr(
  File "src/netCDF4/_netCDF4.pyx", line 3087, in netCDF4._netCDF4.Dataset.setncattr
  File "src/netCDF4/_netCDF4.pyx", line 1858, in netCDF4._netCDF4._set_att
  File "src/netCDF4/_netCDF4.pyx", line 6733, in netCDF4._netCDF4._strencode
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 61-63: surrogates not allowed

[davidhassell]

Thanks for the report, Jonathan.

Could you let us know which version of cf-python you are using (the output from cf.environment() is ideal), and if you could send the JASMIN path to the file I can investigate further..

David

[JonathanGregory]

Dear David

On jasmin in /gws/ssde/j25a/ncas_climate/vol2/users/jonathan/cmip6 I execute /home/users/jonathan/bin/grouptime -v prthetao/prthetao_Emon_MPI-ESM1-2-HR_faf-heat_r1i1p1f1_gn_191501-191912.nc /tmp, where the first argument is the input file, the second the output directory. The verbose (-v) terminal output is below, starting with the cf environment. Is this what you need?

Best wishes

Jonathan

Platform: Linux-5.14.0-687.26.1.el9_8.x86_64-x86_64-with-glibc2.34 
Python: 3.12.11 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/bin/python
packaging: 25.0 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/packaging/__init__.py
numpy: 2.2.6 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/numpy/__init__.py
cfdm.core: 1.12.2.0 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/core/__init__.py
udunits2 library: /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/libudunits2.so.0 
HDF5 library: 1.14.3 
netcdf library: 4.9.2 
netCDF4: 1.7.2 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/netCDF4/__init__.py
h5netcdf: 1.6.3 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/h5netcdf/__init__.py
h5py: 3.13.0 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/h5py/__init__.py
zarr: 3.0.10 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/zarr/__init__.py
s3fs: 2025.5.1 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/s3fs/__init__.py
scipy: 1.15.2 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/scipy/__init__.py
dask: 2025.5.1 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/dask/__init__.py
cftime: 1.6.4 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cftime/__init__.py
cfunits: 3.3.7 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfunits/__init__.py
cfdm: 1.12.2.0 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/__init__.py
esmpy/ESMF: 8.8.1 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/esmpy/__init__.py
psutil: 7.0.0 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/psutil/__init__.py
matplotlib: 3.10.3 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/matplotlib/__init__.py
cfplot: 3.4.0 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfplot/__init__.py
cf: 3.18.0 /apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cf/__init__.py
output file will be /tmp/prthetao_Eyr_MPI-ESM1-2-HR_faf-heat_r1i1p1f1_gn_191501-191912.nc
input file: ['prthetao/prthetao_Emon_MPI-ESM1-2-HR_faf-heat_r1i1p1f1_gn_191501-191912.nc']
select: ncvar%prthetao
selected input field:
[<CF Field: sea_water_redistributed_potential_temperature(time(60), depth(40), long_name=cell index along second dimension(404), long_name=cell index along first dimension(802)) degC>]
T-axis to collapse: time(60) days since 1850-1-1 00:00:00 proleptic_gregorian
1915 1 1919 12 60 60
output field
Field: sea_water_redistributed_potential_temperature (ncvar%prthetao)
---------------------------------------------------------------------
Data            : sea_water_redistributed_potential_temperature(time(5), depth(40), long_name=cell index along second dimension(404), long_name=cell index along first dimension(802)) degC
Cell methods    : area: mean where sea time(5): mean time(5): mean
Dimension coords: time(5) = [1915-07-02 12:00:00, ..., 1919-07-02 12:00:00] proleptic_gregorian
                : depth(40) = [6.0, ..., 5720.0] m
                : long_name=cell index along second dimension(404) = [0, ..., 403] 1
                : long_name=cell index along first dimension(802) = [0, ..., 801] 1
Auxiliary coords: latitude(long_name=cell index along second dimension(404), long_name=cell index along first dimension(802)) = [[51.14887947045643, ..., -78.67488116867928]] degrees_north
                : longitude(long_name=cell index along second dimension(404), long_name=cell index along first dimension(802)) = [[83.10237982639018, ..., 83.30388029396939]] degrees_east
Cell measures   : measure:area (external variable: ncvar%areacello)
                : measure:volume (external variable: ncvar%volcello)
collapsed T-axis: time(5) days since 1850-1-1 00:00:00 proleptic_gregorian
1915 1919 5 5
Traceback (most recent call last):
  File "/home/users/jonathan/bin/grouptime", line 102, in <module>
    cf.write(n,ofile)
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/write.py", line 826, in __new__
    netcdf.write(
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5232, in write
    self._file_io_iteration(
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5478, in _file_io_iteration
    self._write_global_attributes(fields)
  File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4610, in _write_global_attributes
    g["netcdf"].setncattr(
  File "src/netCDF4/_netCDF4.pyx", line 3087, in netCDF4._netCDF4.Dataset.setncattr
  File "src/netCDF4/_netCDF4.pyx", line 1858, in netCDF4._netCDF4._set_att
  File "src/netCDF4/_netCDF4.pyx", line 6733, in netCDF4._netCDF4._strencode
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 61-63: surrogates not allowed

[davidhassell]

Thanks, Jonathan.

I'm not sure what's going on, but it's surely a bug. I tried with v3.20.0, and it sill failed until I added netcdf_backend='netCDF4' to the cf.write call:

cf.write(n,ofile, netcdf_backend='netCDF4')

Then it worked:

output file will be prthetao_Eyr_MPI-ESM1-2-HR_faf-heat_r1i1p1f1_gn_191501-191912.nc
input file: ['prthetao/prthetao_Emon_MPI-ESM1-2-HR_faf-heat_r1i1p1f1_gn_191501-191912.nc']
select: ncvar%prthetao
selected input field:
[<CF Field: sea_water_redistributed_potential_temperature(time(60), depth(40), long_name=cell index along second dimension(404), long_name=cell index along first dimension(802)) degC>]
T-axis to collapse: time(60) days since 1850-1-1 00:00:00 proleptic_gregorian
1915 1 1919 12 60 60
output field
Field: sea_water_redistributed_potential_temperature (ncvar%prthetao)
---------------------------------------------------------------------
Data            : sea_water_redistributed_potential_temperature(time(5), depth(40), long_name=cell index along second dimension(404), long_name=cell index along first dimension(802)) degC
Cell methods    : area: mean where sea time(5): mean time(5): mean
Dimension coords: time(5) = [1915-07-02 12:00:00, ..., 1919-07-02 12:00:00] proleptic_gregorian
                : depth(40) = [6.0, ..., 5720.0] m
                : long_name=cell index along second dimension(404) = [0, ..., 403] 1
                : long_name=cell index along first dimension(802) = [0, ..., 801] 1
Auxiliary coords: latitude(long_name=cell index along second dimension(404), long_name=cell index along first dimension(802)) = [[51.14887947045643, ..., -78.67488116867928]] degrees_north
                : longitude(long_name=cell index along second dimension(404), long_name=cell index along first dimension(802)) = [[83.10237982639018, ..., 83.30388029396939]] degrees_east
Cell measures   : measure:area (external variable: ncvar%areacello)
                : measure:volume (external variable: ncvar%volcello)
collapsed T-axis: time(5) days since 1850-1-1 00:00:00 proleptic_gregorian
1915 1919 5 5
prthetao_Eyr_MPI-ESM1-2-HR_faf-heat_r1i1p1f1_gn_191501-191912.nc written

We can't fix v3.18.0, so I recommend upgrading to 3.20.0 and using the new cf.write keyword. I can then look into the still-present bug without holding you up any longer.

[JonathanGregory]

Thanks for investigating, @davidhassell. On Jasmin I use module load jaspy, which I presume is the latest they support. How would I combine this with a later cf-python?

[davidhassell]

Dear Jonathan,

The easiest way might be to install a local copy of cf-python at the newer version:

$ pip install --user cf-python==3.20.0

This puts that version of cf in (something like) ~/.local/lib/python3.12/site-packages/, and I think that location will automatically be at the front of your PYTHONPATH:

>>> import cf
>>> cf.__version__
'3.20.0'
>>> cf.__file__
'~/.local/lib/python3.12/site-packages/cf/__init__.py'

If it's still picking up 3.18.0, you can force it to find your newer version with:

$ export PYTHONPATH=~/.local/lib/python3.12/site-packages:$PYTHONPATH # (or whatever your local directory is)

[JonathanGregory]

Dear David

Thanks very much. That works exactly as you described and avoids the problem.

Should I be concerned about these warnings:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
s3fs 2025.5.1 requires fsspec==2025.5.1, but you have fsspec 2026.7.0 which is incompatible.
gcsfs 2025.5.1 requires fsspec==2025.5.1, but you have fsspec 2026.7.0 which is incompatible.

What is p5dump for, which it warns me isn't in my PATH?

Best wishes

Jonathan

[davidhassell]

All red herrings, and safely ignored :) - you are using local data (hence no need to call on fsspec), and p5dump is a command-line tool that ships with pyfive, which is nothing to do with cf-python

[JonathanGregory]

😄

[davidhassell]

This will be fixed in the next release, meaning you shouldn't need to specify the netCDF4 backend: NCAS-CMS/cfdm#423

[JonathanGregory]

Thanks. J