[FabianIsensee]

Codecs and filters registered with Python functions (register_codec(), register_filter()) need the GIL, so C-Blosc2 must not run them on its worker threads. The SChunk/NDArray constructors refuse nthreads > 1 for them, but C-Blosc2 picks the thread count itself in two cases, and the interpreter crashes:

• Containers made by schunk_from_cframe()/ndarray_from_cframe(), or opened with SChunk(urlpath=..., mode="r"), get C-Blosc2's global thread count. After blosc2.set_nthreads(4), reading one segfaults. blosc2.open() gives the container blosc2.nthreads (the number of cores by default) and then refuses it with ValueError: Cannot use multi-threading with user defined Python codecs, unless the caller passes cparams and dparams with nthreads=1.
• BLOSC_NTHREADS replaces the nthreads of every context that C-Blosc2 creates, an explicit 1 included. With BLOSC_NTHREADS=4, blosc2.SChunk(data=..., cparams={"codec": <python codec>, "nthreads": 1, ...}) segfaults while compressing.

This PR changes only python-blosc2:

• One thread for existing containers: _one_thread_for_python_callbacks() gives a super-chunk whose cparams name a codec or filter registered from Python one-thread contexts (the constructor's rule). It is called after schunk_from_cframe(), ndarray_from_cframe(), SChunk(mode="r") and open(). open() also uses nthreads=1 in the cparams/dparams it applies to such containers, so it opens them instead of raising; explicit nthreads > 1 passed to open() for such a container is reduced to 1 (the Python-level open() fills in blosc2.nthreads as the default, so the two cannot be told apart). Other containers are not affected. These are the only places where a container gets contexts that python-blosc2 did not ask for. The stores, proxies and from_cframe() go through them; copy(), slices and the setters build their contexts from parameters that _check_cparams()/_check_dparams() check.

• BLOSC_NTHREADS: while it is above 1, the code raises ValueError: BLOSC_NTHREADS=4 is not supported with codecs or filters registered from Python: C-Blosc2 would run them on 4 threads, but they need one (unset BLOSC_NTHREADS or set it to 1) instead of crashing. The effective count comes from context_nthreads(1), which creates a context asking for one thread and reads back what C-Blosc2 gave it. It is checked at three sites:

  • register_codec()/register_filter() with Python functions. This catches the usual case, the variable set before the process starts, for every path. A process that sets BLOSC_NTHREADS above 1 and registers a Python codec now fails at registration instead of crashing at first use.
  • _check_cparams()/_check_dparams() when the cparams name such a codec or filter, which covers the constructors and the cparams/dparams setters.
  • _one_thread_for_python_callbacks(), which covers the containers above, open() included.

  The last two are needed because C-Blosc2 reads the variable again for every context, so it can also be set after registering.

• Not covered when BLOSC_NTHREADS is set only after registering:

  • the standalone chunk functions (compress2()/decompress2()/decompress()/vldecompress(); Python codecs need an SChunk, so these calls fail anyway, and Decide threads and the GIL per chunk for Python codecs and filters #769 gives them one thread);
  • the postfilter setters;
  • span reads (Keep the GIL in span reads of chunks that can run Python codecs or filters #775).

  Python prefilters/postfilters with BLOSC_NTHREADS are not covered at all (same cause, but they are per-context callbacks, not registered ids).

• Cost: a few byte lookups per container made from a frame or file; context_nthreads(1) (one context created and freed) only for data that uses a Python codec or filter.

Tests (tests/test_python_callbacks_nthreads.py, all in child interpreters, skipped on WASM):

• After set_nthreads(4): an SChunk from schunk_from_cframe(), open() and SChunk(mode="r"), and an NDArray from ndarray_from_cframe() and open(), with a Python codec, plus (frame, open) with a Python filter. Each case reads and, where possible, writes.
• BLOSC_NTHREADS=4 at start-up: registering a Python codec or filter must raise.
• BLOSC_NTHREADS=4 set after registering (codec and filter): a new SChunk, a frame copy, open() and the dparams setter must raise. The check is skipped if C-Blosc2 does not see a variable set through os.environ, since it cannot use it either then.
• The callbacks exit the child if they ever run off the main thread.
• All 11 fail on main (8 with SIGSEGV, 3 with the open() ValueError) and pass here.
• Full suite: 11008 passed vs 10997 on main; the same 2 b2view tests that need aiohttp fail locally on both.

Perf (GCC 14 wheels, main built twice, EPYC node, 5 interleaved rounds): schunk_from_cframe() (copy and view), ndarray_from_cframe(), SChunk(mode="r") and open() of an SChunk and an NDArray (8-72 us per call), each with and without a Python codec registered, are within -1.2%..+1.5% of main (main vs main: -0.6%..+1.8%), so there is no measurable change.

Relation to open PRs (git merge-tree: clean with all of them, #769-#775 and #777-#783):

• Decide threads and the GIL per chunk for Python codecs and filters #769 makes decompress2()/decompress()/vldecompress() use one thread for chunks of Python codecs; BLOSC_NTHREADS defeats that. Here the variable is refused at registration (and for containers), not inside those calls.
• Keep the GIL in span reads of chunks that can run Python codecs or filters #775 does the same for span reads, with the same limit.
• Make the matmul, miniexpr and Menudet prefilters unable to need the GIL #774 adds _reads_need_python(sc), the same check as _runs_python_callbacks(sc.storage.cparams) here; the names differ, so both build together, and once both are in, one can go.
• Wait for every C call that runs without the GIL before blosc2_destroy() at exit #772 does not interact.

Question for Francesc: would you want a C-level fix as well? C-Blosc2 could itself run chunks of codecs and filters that need the calling thread one block after another, whatever nthreads or BLOSC_NTHREADS say. This could be done with additive registration functions that take a flags argument (blosc2_register_codec_flags()/blosc2_register_filter_flags(), no change to the structs or the ABI). That would make the one-thread workarounds here, in #769 and in #775 unnecessary, and BLOSC_NTHREADS would keep working with Python codecs. I have a prototype with tests and can send it as a PR if you want it.

I used Claude Code (Anthropic) to trace the crashes, write the change and the tests, and run the test suites and benchmarks; I reviewed the result.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DiPFztdpnKMJkzrjT36BuP