Skip to content

Check the value's shape in NDArray.__setitem__ and broadcast it as NumPy does - #784

Open
FabianIsensee wants to merge 2 commits into
Blosc:mainfrom
FabianIsensee:fix-setitem-shape
Open

FabianIsensee wants to merge 2 commits into
Blosc:mainfrom
FabianIsensee:fix-setitem-shape

Conversation

@FabianIsensee

Copy link
Copy Markdown
Contributor

NDArray.__setitem__ passes the value to set_slice() as one buffer, and b2nd_set_slice_cbuffer() only checks that the buffer is large enough. The value's shape was never compared with the selection:

  • A value with the wrong shape but the right number of items is written in C order: a[0:2, 0:3] = np.arange(6).reshape(3, 2) succeeds, where NumPy raises "could not broadcast input array from shape (3,2) into shape (2,3)". The same happens with a flat (6,) value, (2, 3, 1), (2, 1, 3), and NDArray values of those shapes.
  • A value that is too large has its first items written (a[0:2, 0:3] = np.arange(7)). On the step path the value is indexed with the selection's coordinates, so a[0:4:2, 0:3] = np.arange(12).reshape(4, 3) writes rows 0 and 1 of the value.
  • Every value that NumPy would broadcast raises RuntimeError: Error while setting the slice. Examples: a[0:2, 0:3] = np.arange(3), a (2, 1) column, np.array([7]), a row of another dtype. On the step and fancy paths these raise IndexError or ValueError instead.
  • On a 0-d array, a[...] = np.zeros(0) stores an item read from past the end of the empty value.

What this PR changes:

  • __setitem__ computes NumPy's shape of the selection on each of its paths, and checks the value's shape against it before anything is written. The paths are basic keys, keys with None, steps, and fancy/boolean keys; the shape has no dimension for an integer index and a 1 for each None. For basic keys, get_basic_start_stop() already computes this shape and now returns it.
  • The check follows NumPy's rule for assignment: leading dimensions of length 1 in the value are ignored and the rest must broadcast. Anything else raises NumPy's ValueError ("could not broadcast input array from shape ... into shape ...").
  • Values that already hold the selection's items in C order are written as before, without a copy. That covers the same shape and shapes that differ only in dimensions of length 1. NDArray values of the region's shape keep the chunk-wise copy.
  • Values that need broadcasting are converted to the array's dtype first, then broadcast with np.broadcast_to() and copied into the buffer once. A size-1 value fills the region the same way a scalar does, chunk by chunk.
  • On the step and fancy paths, the value is broadcast to the selection's shape before it is indexed.
  • The PR has two commits. The first one (ctable.py only) is for Francesc to look at: two Column methods relied on silent truncation. Column._dictionary_eq() and Column._dictionary_null_pred() wrote the predicate over all of a dictionary column's codes (one per allocated slot, 4096 in the tests) into a mask with one item per table row (16). That worked only because __setitem__ wrote the first items of any value that was large enough. They now write min(len(codes), n_phys) items explicitly, with the same result; with the shape check alone, two ctable tests failed.
  • The docstring says that the value is broadcast. It also drops the outdated sentence that steps are not supported.

Where it still differs from NumPy. These are deliberate, and every value accepted is written correctly:

  • A size-1 array assigned to an all-integer key (a[1, 2] = np.ones(1)) is accepted, as on main. NumPy 1.26 accepts it with a DeprecationWarning; NumPy 2 raises.
  • A value with more than one dimension assigned to an N-d boolean mask is broadcast. NumPy's boolean-assignment path only takes 0-d or 1-d values there.
  • Lists are converted with np.asarray(), as before. NumPy limits a nested list's depth to the target's number of dimensions.

Tests in tests/ndarray/test_setitem.py, all of which fail without the change:

  • test_setitem_value_shape_like_numpy covers 16 keys: slices, Ellipsis, integers (dropped dimensions), None, empty selections, steps (also negative), an integer-array key, a boolean key, a 3-d array with an inner integer, larger arrays that take the chunk-wise fill and copy paths, and a 0-d array. Each key gets 13-14 values as NumPy arrays, lists and NDArrays: right, wrong, too large and broadcastable shapes, and other dtypes. NumPy is the reference. Where it raises, blosc2 must raise the same error and leave the array unchanged; otherwise the data must match.
  • test_setitem_lazyexpr_value_shape checks a broadcast LazyExpr value, and that a LazyExpr of the wrong shape raises before anything is written.

The full suite gives the same results as main apart from the 49 new tests, and fork CI is green (Linux incl. legacy, ARM, Windows, NumPy 1.26, WASM; macOS was green before the rebase onto #741 and got no runner after it). The stricter check also did not break downstream code we maintain: nnU-Net's test suite and an nnInteractive session replay that writes all its interactions through blosc2 give the same results with and without this PR.

Performance: the check costs nothing on µs-scale writes. I measured 1 thread, small arrays (same-shape blocks of 10 to 512 items, and scalars), on an EPYC 9554, with builds interleaved over 8-16 rounds. To leave out code-placement effects, the comparison uses the branch's extension once with main's Python files and once with this PR's: same-shape and scalar writes are 2.6-8.6% faster with this PR (0.4-0.9 µs per call). The saving comes from calling np.isscalar() once instead of twice (about 250 ns for an array) and from reusing the shape the extension already computes. Main built twice differs by up to 1.6%. The step and fancy paths show no measurable change.

This is based on current main (after #741 and #765) and does not depend on other open PRs; it merges cleanly with #769-#775 and #779-#783.

Not in this PR (pre-existing on main; happy to follow up):

  • Fancy __setitem__ with list or tuple keys raises AttributeError (NDArray.__setitem__ raises AttributeError for list index keys #764; a[[0, 2]] = 1, a[[0, 2], 0:3] = v): the scalar-bool check reads key.shape before hasattr(key, "dtype"). Only bare array keys work today. The fix is not just swapping the order, because that path indexes with integers turned into length-1 slices, which gives other dimensions than NumPy for mixed integer/array keys.
  • LazyExpr values with array keys fail, because the fancy path indexes the value with fancy sub-indices (planned as a follow-up).
  • memoryview values count as scalars (np.isscalar(memoryview(...)) is True), so they go to the scalar fill and raise.
  • b[::-1] = b gives other data than NumPy (NDArray.__setitem__ with steps writes wrong data when the value is the array itself (b[::-1] = b) #776).
  • c-blosc2: the 0-d branch of get_set_slice() ignores buffersize, so a short buffer is read past (set) or written past (get). This PR stops __setitem__ from reaching it.
  • NDArray.set_slice() does not release the value's buffer when the C call fails.

I used Claude Code (Anthropic) to find the affected paths, write the fix and the tests, and measure them; I reviewed the changes and the numbers.

🤖 Generated with Claude Code

FabianIsensee and others added 2 commits November 6, 2026 16:16
_dictionary_eq() and _dictionary_null_pred() assign the predicate over
all of the column's codes to a mask with one item per physical row.  The
codes have one item per allocated slot, which can be more than the rows
(4096 vs. 16 in the tests), and this worked only because
NDArray.__setitem__ wrote the first items of any value at least as large
as the selection.  Write min(len(codes), n_phys) items explicitly (the
same result, and only those are evaluated), so that __setitem__ can
check the value's shape as NumPy does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DiPFzsfE5i74cBSxZN13M8
set_slice() takes the value as one buffer and checks only that it is
large enough, so a[0:2, 0:3] = np.arange(6).reshape(3, 2) wrote the
items in C order, a too large value had its first items written, and a
value NumPy broadcasts, such as a[0:2, 0:3] = np.arange(3), raised
"RuntimeError: Error while setting the slice".  The paths for steps and
fancy keys indexed the value with the selection's coordinates, so there
too large values were cut and broadcastable ones raised IndexError.

Compute NumPy's shape of the selection on every path (no dimension for
an integer index, 1 for each None; get_basic_start_stop() already has it
for basic keys and now returns it) and check the value's shape against
it before anything is written: leading dimensions of length 1 are
ignored and the rest must broadcast, as in NumPy; anything else raises
NumPy's "could not broadcast" ValueError.  Values that hold the items of
the selection in C order are written as before, without a copy.  Values
that need broadcasting are converted to the array's dtype first, then
broadcast and copied once; a size-1 value fills like a scalar.  The step
and fancy paths get the value broadcast to the selection's shape.

np.isscalar() (~250 ns for an array) is now called once per call, and
the basic path reuses the shape the extension computes, so same-shape
and scalar writes into small arrays got 2-8% faster instead of slower.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DiPFzsfE5i74cBSxZN13M8

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant