[FabianIsensee]

Hi Francesc,

a small follow-up to the improved-miniexpr merge (#741). It is based on #772 (it changes lines that #772 touches), so please look at it after #772 (only the last commit, 3c1f4c3, is new here; the first one is #772); I will rebase it once #772 is merged.

NDArray.get_1d_span_numpy() (used by the indexing planner and the storage readers) and the native span readers of graph executions (NativeGraphSchedule.execute_sources(..., native_spans=True)) decompress without the GIL, whatever codec or filter the chunk uses. With a codec or filter registered from Python, its callbacks then run Python without the GIL and the interpreter crashes:

import zlib
import numpy as np, blosc2

def encoder(src, dst, meta, schunk):
    out = zlib.compress(src.tobytes(), 1)
    dst[: len(out)] = np.frombuffer(out, np.uint8)
    return len(out)

def decoder(src, dst, meta, schunk):
    out = zlib.decompress(src.tobytes())
    dst[: len(out)] = np.frombuffer(out, np.uint8)
    return len(out)

blosc2.register_codec("span_zlib", 180, encoder, decoder)
a = blosc2.asarray(np.arange(2**16, dtype=np.int32), chunks=(2**14,), blocks=(2**12,),
                   cparams={"codec": 180, "nthreads": 1, "filters": [], "splitmode": blosc2.SplitMode.NEVER_SPLIT})
out = np.empty(2**13, dtype=np.int32)
a.get_1d_span_numpy(out, 1, 0, out.size)  # SIGSEGV on main; a[2**14:2**14 + 2**13] works

The same happens with a filter registered from Python, with nthreads=4 for the span, and in a graph execution with native span readers over such an array (all SIGSEGV on main 5ff9823 and on #772). A slice read of the same chunk keeps the GIL (_can_release_gil()) and works.

What this PR does (blosc2_ext.pyx): while any codec or filter is registered from Python (the C flag _python_ids_registered, the same check that decompress2() uses), get_1d_span_numpy() decompresses with the GIL and with one thread (C-Blosc2's worker threads never hold the GIL), and graph executions with native span readers keep the GIL. Without registrations nothing changes. I did not use _can_release_gil(): it also keeps the GIL for postfilters, which take the GIL themselves since #741, also on worker threads when a span uses several threads, so those threads would wait for a GIL that the calling thread holds.

Tests (tests/test_span_reads_python_callbacks.py, each case in a child interpreter, skipped on WASM): span reads of a Python codec and of a Python filter with 1 and 4 threads, and graph executions with native span readers over both (skipped where the build has no native graph readers). All 6 fail (crash) on main and on #772 and pass here. Full suite: 11005 passed (main 10997), same 2 failures as on main (b2view http tests that need aiohttp).

How this relates to #774: #774 keeps operands whose cparams use a Python codec or filter away from the paths that decode them on C-Blosc2's worker threads (the miniexpr, matmul and Menudet prefilters and the parallel native-graph route, via _reads_need_python()), where holding the GIL in the caller would not help. This PR covers the two entry points that decode on the calling thread, get_1d_span_numpy() (also used by the storage readers of the remaining graph routes) and execute_sources(..., native_spans=True), by keeping the GIL there. The two merge cleanly and do not overlap: the tests of this PR still crash with #774 alone, and #774's lazy-expression cases pass without this PR. The check here is registry-wide, as in _decompress_ctx(); once #774 (or #769, with its chunk-header test _chunk_runs_python()) is in, it could become per array or per chunk, so that registering a Python codec does not serialize span reads of unrelated arrays.

Perf: no change without registrations. 1024-item span reads, wheels built with GCC 14, 6 interleaved rounds: 1.597 vs 1.628 us (1 thread) and 4.075 vs 4.125 us per read (4 Python threads) on main, within the per-round spread.

I used Claude Code (Anthropic) to find and reproduce the crash, write the change and the tests, and run the benchmarks; I reviewed the result.

Best,
Fabian

🤖 Generated with Claude Code

https://claude.ai/code/session_01DiPFztdpnKMJkzrjT36BuP