[FabianIsensee]

Hi Francesc,

two commits on main about chunks that use a codec or filter registered from Python (register_codec()/register_filter()). Both need to know whether a chunk uses one, so both read it from the chunk header with one new helper, _chunk_runs_python(): the codec (including the user-defined codec byte of the extended header) and the 6 filter ids, checked against the id arrays from #766. No Python lookups; with nothing registered from Python it returns at once.

1. Crash with more than one thread. decompress2(), vldecompress() and decompress() crash when the chunk uses a Python codec or filter and the call uses more than one thread: the callbacks then run on C-Blosc2's worker threads, which never hold the GIL. decompress2() and vldecompress() use blosc2.nthreads by default, so a plain blosc2.decompress2(chunk) segfaults on any multi-core machine once the chunk has more than one block; decompress() does after blosc2.set_nthreads(n > 1). SChunks refuse such dparams in _check_dparams(), but these functions only get a chunk.

blosc2.register_codec("zlib", 170, encoder, decoder)
cparams = {"typesize": 4, "nthreads": 1, "codec": 170, "blocksize": 1000, "filters": [blosc2.Filter.NOFILTER], "filters_meta": [0]}
chunk = blosc2.SChunk(chunksize=4000, data=np.arange(1000, dtype=np.int32), cparams=cparams, dparams={"nthreads": 1}).get_chunk(0)
blosc2.decompress2(chunk, nthreads=4)  # SIGSEGV; with nthreads=1 the decoder raises "Cannot apply user codec without an SChunk"

Fix: decompress2() and vldecompress() use one thread for such chunks (they now create their context after looking at src, which also stops them from leaking it when src is too short); decompress() decompresses them in a context of one thread of its own instead of the global context (_blosc1_decompress()). vldecompress_block() needs nothing, C-Blosc2 decodes a single VL block on the calling thread. Such calls now end as they do with one thread (the callback's RuntimeError). Chunks whose header names a Python codec but that C-Blosc2 decodes without calling it (memcpyed or special-value chunks) still decompress; that is why this drops to one thread instead of raising up front. Other chunks keep their threads, also while a Python codec is registered; using one thread whenever anything is registered would have been simpler, but would make e.g. a 16 MiB LZ4 chunk take 651 instead of ~355 us on our test node (measured yesterday).

2. GIL kept for calls that do not use Python callbacks. With set_releasegil(True), decompress2() kept the GIL whenever any codec or filter was registered from Python, and SChunk.to_cframe() whenever _can_release_gil() said so, which is also the case whenever anything is registered. Now decompress2() keeps it only for a chunk that uses a Python codec or filter. to_cframe() releases it for super-chunks in memory: a contiguous one is only copied (blosc2_schunk_to_buffer_alloc() copies frame->cframe), and a sparse one without prefilter and postfilter gets its chunks copied as they are, so only its vlmetalayers are decompressed, and their headers get the same check. Everything else keeps the previous check: with a prefilter or postfilter C-Blosc2 decompresses and recompresses every chunk, and a frame on disk can get its vlmetalayers reloaded from the file during the call. The set_releasegil() docstring says that decompress2() keeps the GIL for such chunks. One difference to before: if another thread changes a writable src between the header check and the C call, a Python codec named only by the new header could run without the GIL; I think that data race is the caller's.

Tests:

• tests/test_python_codec_threads.py runs, in a subprocess with 1 and with 4 threads, decompress2() and decompress() of a codec chunk and a filter chunk, vldecompress() of VL-block chunks whose header names the Python codec or filter (vlcompress() cannot run them), and decompress2() of a plain chunk. It checks that the callbacks raise, that the 4-thread outcomes equal the 1-thread ones, and that the plain chunk decompresses. Without the first commit the 4-thread subprocess segfaults (each of the 6 calls does on its own). Skipped on WASM.
• tests/test_releasegil_callbacks.py: test_decompress2_releases_the_gil_next_to_python_codecs and test_to_cframe_releases_the_gil_next_to_python_codecs (contiguous and sparse) check with a ticker thread, as the write test does, that the call lets other threads run while an unused Python codec and filter are registered; without the second commit all three fail (the longest gap is 0.88-0.99 of the call). test_to_cframe_python_codec_filter (codec and filter, contiguous and sparse, GIL held and released) checks that serializing such a super-chunk calls none of the callbacks and that the frame reads back; test_to_cframe_contiguous_callbacks does the same with a Python prefilter or postfilter on a contiguous frame. The retry loop of the write test moved into assert_releases_the_gil().
• pytest tests on main 5ff9823: 11013 passed, main 10997 (+16 new cases); the 2 failures are the b2view http tests that need aiohttp, on both. Fork CI (da5cf56): green on Linux x86 (Python 3.12 default and legacy, 3.14), Linux ARM, macOS, Windows, NumPy 1.26 and WASM.

Perf (EPYC 9554 node, 3 interleaved rounds against main and a second build of main, us per call): decompress2() of 4 KiB and 64 KiB chunks with 1 and 4 threads, with and without an unused Python codec registered, is within ±1.1% of main (the second build of main: ±1.2%); 16 MiB chunks with 4 threads and an unused Python codec registered keep their threads (335 vs 368 us on main, which is noisy there: second build 349). With set_releasegil(True) and an unused Python codec registered, 4 threads calling decompress2() on 256 KiB zstd chunks take 7.1 us per chunk instead of 21.2 (7.3 without the codec).

Not in this PR (I can follow up):

• C-Blosc2 replaces dparams.nthreads with BLOSC_NTHREADS in blosc2_create_dctx(), so with BLOSC_NTHREADS=4 these calls still run the callbacks on worker threads, and so do SChunks with a Python codec (_check_dparams() checks the requested nthreads, not the effective ones). A super-chunk made by schunk_from_cframe() or opened from a file gets C-Blosc2's global number of threads without _check_dparams(), with the same result after blosc2.set_nthreads(n > 1). A flag on blosc2_codec/blosc2_filter registration saying "needs the calling thread and the GIL" would let C-Blosc2 run such chunks serially itself and cover all of these for every binding, and would also let the NDArray slice reads, which still keep the GIL whenever anything is registered from Python, decide per chunk. I can propose that in c-blosc2 if you like the idea.

I used Claude Code (Anthropic) to find the causes, write the fixes and the tests, and run the benchmarks; I reviewed the result.

Best,
Fabian

🤖 Generated with Claude Code

https://claude.ai/code/session_01DiPFztdpnKMJkzrjT36BuP