Diskless persisted datasets are no longer persisted

I found that as of version 1.7.5, creating a Dataset with diskless=True and persist=True no longer works. Previously it would write the file only when closing the file. Now, it doesn't write the file at all.

Here's a minimal example:

 from pathlib import Path

 import netCDF4

 print(f"netCDF4 package: {netCDF4.__version__}")
 print(f"netCDF library: {netCDF4.__netcdf4libversion__}")
 print(f"HDF5 library: {netCDF4.__hdf5libversion__}")

 file = Path("persisted-output.nc")
 file.unlink(missing_ok=True)

 with netCDF4.Dataset(filename=file, mode="w", diskless=True, persist=True) as f:
     f.createGroup(groupname="test")

 if file.exists():
     print(f"Successfully wrote: {file}")
 else:
     print(f"File was not written: {file}")

Output for 1.7.4 and 1.7.5:

$ uv run --with netcdf4==1.7.4 test.py
netCDF4 package: 1.7.4
netCDF library: 4.9.3
HDF5 library: 1.14.6
Successfully wrote: persisted-output.nc

$ uv run --with netcdf4==1.7.5 test.py
netCDF4 package: 1.7.5
netCDF library: 4.10.1
HDF5 library: 2.2.0
File was not written: persisted-output.nc

These are using the manylinux_2_28_x86_64 wheels on PyPI.

However, it looks like it's actually an issue with the bundled libraries in the wheels. When I build the package from source, it works:

$ uv run --with netcdf4==1.7.5 --no-binary-package=netcdf4 test.py
netCDF4 package: 1.7.5
netCDF library: 4.9.3
HDF5 library: 1.14.6
Successfully wrote: persisted-output.nc

This is a Fedora 44 environment, so these netCDF/HDF5 library versions are what Fedora is currently packaging. I don't know if it's an issue with the netCDF C library or the HDF5 library, I haven't tried narrowing it down.

Within a Fedora 45 (beta) container:

$ uv run --with netcdf4==1.7.5 --no-binary-package=netcdf4 test.py
netCDF4 package: 1.7.5
netCDF library: 4.10.1
HDF5 library: 2.1.1
File was not written: persisted-output.nc

I reran the test after compiling the hdf5 and netcdf libraries from source.

	netcdf 4.9.3	netcdf 4.10.0
hdf5 1.14.6	works	fails
hdf5 2.0.0	can't compile	fails

So this points to the netcdf library. I'll report the issue there.

Well, I was all set to report it, but I can't seem to create a minimal case in C that fails!

Here's what I have:

test.c

#include <netcdf.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(void) {
  const char *out_file = "persisted-output.nc";

  int stat = unlink(out_file);
  if (stat != 0) {
    // A non-existing file is not a problem
    if (errno != ENOENT) {
      perror("Failed to unlink the output file");
      return EXIT_FAILURE;
    }
  }

  int ncid;
  stat = nc_create(out_file, NC_NETCDF4 | NC_WRITE | NC_CLOBBER | NC_DISKLESS | NC_PERSIST, &ncid);
  if (stat != NC_NOERR) {
    printf("netcdf: %s\n", nc_strerror(stat));
    return EXIT_FAILURE;
  }

  stat = nc_close(ncid);
  if (stat != NC_NOERR) {
    printf("netcdf: %s\n", nc_strerror(stat));
    return EXIT_FAILURE;
  }

  stat = access(out_file, F_OK);
  if (stat == 0) {
    printf("Success: file exists: %s\n", out_file);
  } else {
    if (errno == ENOENT) {
      printf("Failed: file does not exist: %s\n", out_file);
    } else {
      perror("failed to check file access");
      return EXIT_FAILURE;
    }
  }

  return EXIT_SUCCESS;
}

And whether I run it in Fedora 44 (using netcdf 4.9.3) or Fedora 45 (netcdf 4.10.1), it creates the file as expected. I also tried compiling netcdf from source (either version) and it also works as expected. Is there something extra that _netCDF4.pyx is doing when it calls nc_create()?

No, not really. We have test coverage for diskless and persist=True (test/test_diskless.py) - can you try running that test on your system?

I ran it, and it passes for 1.7.4 but is skipped for 1.7.5:

# The DeprecationWarnings here are omitted for brevity
$ uv run --with netcdf4==1.7.4 ./test_diskless.py 
.
----------------------------------------------------------------------
Ran 1 test in 0.019s
OK

$ uv run --with netcdf4==1.7.5 ./test_diskless.py 
s
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK (skipped=1)

It's because of the version comparison check, which #1474 fixed in _netCDF4.pyx but not in this test case. When I comment out the @unittest.skipIf() decorator, then the test fails:

$ uv run --with netcdf4==1.7.5 ./test_diskless.py 
E
======================================================================
ERROR: runTest (__main__.DisklessTestCase.runTest)
testing diskless file capability
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_diskless.py", line 78, in runTest
    f = netCDF4.Dataset(self.file2)
  File "src/netCDF4/_netCDF4.pyx", line 2525, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2162, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -51] NetCDF: Unknown file format: '/tmp/tmposkmfi_e.nc'

----------------------------------------------------------------------
Ran 1 test in 0.012s

FAILED (errors=1)

I was curious why the test fails with the "Unknown file format" error. The file does exist, but it's empty. It's because it is created before the test runs with tempfile.NamedTemporaryFile().

So this assertion always passes:

assert os.path.isfile(self.file2)==True

It's a quick hack, but when I update the test so it deletes the temporary file first:

         self.file2 = FILE_NAME2
+        os.remove(self.file2)
         f2 = netCDF4.Dataset(self.file2,'w',diskless=True, persist=True)

Then the test fails that assertion with the 1.7.5 wheel but not with the 1.7.4 wheel:

$ uv run --with netcdf4==1.7.5 ./test_diskless.py
FE
======================================================================
ERROR: runTest (__main__.DisklessTestCase.runTest)
testing diskless file capability
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_diskless.py", line 61, in tearDown
    os.remove(self.file2)
    ~~~~~~~~~^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpyhlj_c0p.nc'

======================================================================
FAIL: runTest (__main__.DisklessTestCase.runTest)
testing diskless file capability
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_diskless.py", line 78, in runTest
    assert os.path.isfile(self.file2)==True
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

----------------------------------------------------------------------
Ran 1 test in 0.011s

FAILED (failures=1, errors=1)

(The first failure is because the tearDown() can't remove the file which doesn't exist, the second is because that assertion that the file exists fails.)

This is now fixed in PR #1514 - I'll create a new 1.7.5.1 release with this fix and the fix for PR #1513.

Ah, I see, due to the version check the NC_PERSIST value was aliased to NC_DISKLESS. That's why I couldn't reproduce it in C. Thank you!

And I can confirm that the 1.7.5.1 release fixed the problem I was facing. Thanks!

Navigation Menu

Repository navigation

Diskless persisted datasets are no longer persisted #1515

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Diskless persisted datasets are no longer persisted #1515

Description

Activity

richli commented on Oct 21, 2026

richli commented on Oct 22, 2026

richli commented on Oct 22, 2026

jswhit commented on Oct 22, 2026

richli commented on Oct 22, 2026

richli commented on Oct 22, 2026

jswhit commented on Oct 22, 2026

richli commented on Oct 22, 2026

richli commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions