Repository navigation
Check the value's shape in NDArray.__setitem__ and broadcast it as NumPy does - #784
Open
FabianIsensee wants to merge 2 commits into
Open
FabianIsensee wants to merge 2 commits into
FabianIsensee wants to merge 2 commits into
Conversation
_dictionary_eq() and _dictionary_null_pred() assign the predicate over all of the column's codes to a mask with one item per physical row. The codes have one item per allocated slot, which can be more than the rows (4096 vs. 16 in the tests), and this worked only because NDArray.__setitem__ wrote the first items of any value at least as large as the selection. Write min(len(codes), n_phys) items explicitly (the same result, and only those are evaluated), so that __setitem__ can check the value's shape as NumPy does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DiPFzsfE5i74cBSxZN13M8
set_slice() takes the value as one buffer and checks only that it is large enough, so a[0:2, 0:3] = np.arange(6).reshape(3, 2) wrote the items in C order, a too large value had its first items written, and a value NumPy broadcasts, such as a[0:2, 0:3] = np.arange(3), raised "RuntimeError: Error while setting the slice". The paths for steps and fancy keys indexed the value with the selection's coordinates, so there too large values were cut and broadcastable ones raised IndexError. Compute NumPy's shape of the selection on every path (no dimension for an integer index, 1 for each None; get_basic_start_stop() already has it for basic keys and now returns it) and check the value's shape against it before anything is written: leading dimensions of length 1 are ignored and the rest must broadcast, as in NumPy; anything else raises NumPy's "could not broadcast" ValueError. Values that hold the items of the selection in C order are written as before, without a copy. Values that need broadcasting are converted to the array's dtype first, then broadcast and copied once; a size-1 value fills like a scalar. The step and fancy paths get the value broadcast to the selection's shape. np.isscalar() (~250 ns for an array) is now called once per call, and the basic path reuses the shape the extension computes, so same-shape and scalar writes into small arrays got 2-8% faster instead of slower. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DiPFzsfE5i74cBSxZN13M8
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NDArray.__setitem__passes the value toset_slice()as one buffer, andb2nd_set_slice_cbuffer()only checks that the buffer is large enough. The value's shape was never compared with the selection:a[0:2, 0:3] = np.arange(6).reshape(3, 2)succeeds, where NumPy raises "could not broadcast input array from shape (3,2) into shape (2,3)". The same happens with a flat(6,)value,(2, 3, 1),(2, 1, 3), and NDArray values of those shapes.a[0:2, 0:3] = np.arange(7)). On the step path the value is indexed with the selection's coordinates, soa[0:4:2, 0:3] = np.arange(12).reshape(4, 3)writes rows 0 and 1 of the value.RuntimeError: Error while setting the slice. Examples:a[0:2, 0:3] = np.arange(3), a(2, 1)column,np.array([7]), a row of another dtype. On the step and fancy paths these raise IndexError or ValueError instead.a[...] = np.zeros(0)stores an item read from past the end of the empty value.What this PR changes:
__setitem__computes NumPy's shape of the selection on each of its paths, and checks the value's shape against it before anything is written. The paths are basic keys, keys with None, steps, and fancy/boolean keys; the shape has no dimension for an integer index and a 1 for each None. For basic keys,get_basic_start_stop()already computes this shape and now returns it.np.broadcast_to()and copied into the buffer once. A size-1 value fills the region the same way a scalar does, chunk by chunk.ctable.pyonly) is for Francesc to look at: two Column methods relied on silent truncation.Column._dictionary_eq()andColumn._dictionary_null_pred()wrote the predicate over all of a dictionary column's codes (one per allocated slot, 4096 in the tests) into a mask with one item per table row (16). That worked only because__setitem__wrote the first items of any value that was large enough. They now writemin(len(codes), n_phys)items explicitly, with the same result; with the shape check alone, two ctable tests failed.Where it still differs from NumPy. These are deliberate, and every value accepted is written correctly:
a[1, 2] = np.ones(1)) is accepted, as on main. NumPy 1.26 accepts it with a DeprecationWarning; NumPy 2 raises.np.asarray(), as before. NumPy limits a nested list's depth to the target's number of dimensions.Tests in
tests/ndarray/test_setitem.py, all of which fail without the change:test_setitem_value_shape_like_numpycovers 16 keys: slices, Ellipsis, integers (dropped dimensions), None, empty selections, steps (also negative), an integer-array key, a boolean key, a 3-d array with an inner integer, larger arrays that take the chunk-wise fill and copy paths, and a 0-d array. Each key gets 13-14 values as NumPy arrays, lists and NDArrays: right, wrong, too large and broadcastable shapes, and other dtypes. NumPy is the reference. Where it raises, blosc2 must raise the same error and leave the array unchanged; otherwise the data must match.test_setitem_lazyexpr_value_shapechecks a broadcast LazyExpr value, and that a LazyExpr of the wrong shape raises before anything is written.The full suite gives the same results as main apart from the 49 new tests, and fork CI is green (Linux incl. legacy, ARM, Windows, NumPy 1.26, WASM; macOS was green before the rebase onto #741 and got no runner after it). The stricter check also did not break downstream code we maintain: nnU-Net's test suite and an nnInteractive session replay that writes all its interactions through blosc2 give the same results with and without this PR.
Performance: the check costs nothing on µs-scale writes. I measured 1 thread, small arrays (same-shape blocks of 10 to 512 items, and scalars), on an EPYC 9554, with builds interleaved over 8-16 rounds. To leave out code-placement effects, the comparison uses the branch's extension once with main's Python files and once with this PR's: same-shape and scalar writes are 2.6-8.6% faster with this PR (0.4-0.9 µs per call). The saving comes from calling
np.isscalar()once instead of twice (about 250 ns for an array) and from reusing the shape the extension already computes. Main built twice differs by up to 1.6%. The step and fancy paths show no measurable change.This is based on current main (after #741 and #765) and does not depend on other open PRs; it merges cleanly with #769-#775 and #779-#783.
Not in this PR (pre-existing on main; happy to follow up):
__setitem__with list or tuple keys raises AttributeError (NDArray.__setitem__ raises AttributeError for list index keys #764;a[[0, 2]] = 1,a[[0, 2], 0:3] = v): the scalar-bool check readskey.shapebeforehasattr(key, "dtype"). Only bare array keys work today. The fix is not just swapping the order, because that path indexes with integers turned into length-1 slices, which gives other dimensions than NumPy for mixed integer/array keys.memoryviewvalues count as scalars (np.isscalar(memoryview(...))is True), so they go to the scalar fill and raise.b[::-1] = bgives other data than NumPy (NDArray.__setitem__ with steps writes wrong data when the value is the array itself (b[::-1] = b) #776).get_set_slice()ignoresbuffersize, so a short buffer is read past (set) or written past (get). This PR stops__setitem__from reaching it.NDArray.set_slice()does not release the value's buffer when the C call fails.I used Claude Code (Anthropic) to find the affected paths, write the fix and the tests, and measure them; I reviewed the changes and the numbers.
🤖 Generated with Claude Code