Skip to content

Release the input buffer when SChunk(data=...) fails; test the prefilter writes without the GIL - #777

Open
FabianIsensee wants to merge 1 commit into
Blosc:mainfrom
FabianIsensee:schunk-data-release-buffer
Open

FabianIsensee wants to merge 1 commit into
Blosc:mainfrom
FabianIsensee:schunk-data-release-buffer

Conversation

@FabianIsensee

Copy link
Copy Markdown
Contributor

Hi Francesc,

a follow-up to #765, from the look at the merged version you asked for. The squash resolved the conflicts with #766 as intended (_store_buffer() decides on the GIL with _can_compress_without_gil(), and errors go through _raise_callback_error()), except in SChunk(data=...): there the resolution kept main's loop, which releases the input buffer only when c-blosc2 returns an error. When _store_buffer() raises, for a chunk larger than BLOSC2_MAX_BUFFERSIZE, the buffer stays exported for good:

import blosc2

data = bytearray(2**31 - 1)
try:
    blosc2.SChunk(chunksize=2**31 - 1, data=data)  # ValueError: Cannot compress more than 2147483615 bytes into one chunk
except ValueError:
    pass
data.append(0)  # main: BufferError: Existing exports of data: object cannot be re-sized (and the 2 GiB stay pinned)

The loop is back in a try/finally, as on the PR branch.

test_data_writes_reach_prefilter ran with the GIL held only, because Python prefilters crashed with set_releasegil(True) before #766. Now that #766 keeps the GIL for them, it runs with the GIL released too, which checks that _store_buffer() follows that rule.

Tests: there is no test for the leak itself, as it needs a 2 GiB buffer; the snippet above raises the BufferError on main and resizes the bytearray with this PR. Full suite: 11001 passed; the 2 b2view http tests fail without aiohttp, as on main.

Merge order: none needed. It merges cleanly with each of the open #769-#775, including #772, which changes _store_buffer() itself.

I used Claude Code (Anthropic) to check the merged version, write the change and run the tests; I reviewed the result.

Best,
Fabian

🤖 Generated with Claude Code

The squash of Blosc#765 resolved the conflict in SChunk.__init__() from
main's side, which releases the input buffer only when c-blosc2 returns
an error.  When _store_buffer() raises (a chunk larger than
BLOSC2_MAX_BUFFERSIZE), the buffer stayed exported.  The loop is back in
a try/finally.

test_data_writes_reach_prefilter() also runs with set_releasegil(True)
now: since Blosc#766 the writes keep the GIL for Python prefilters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant