Skip to content

Fix NumPy serialization, Arrow schemas, and publisher isolation - #299

Merged
timkpaine merged 1 commit into
mainfrom
bug/array-and-publisher-fixes
Nov 15, 2026
Merged

timkpaine merged 1 commit into
mainfrom
bug/array-and-publisher-fixes

Conversation

@timkpaine

@timkpaine timkpaine commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Description

Consolidates #296, #297, and #298, including their original fixes and regression tests by @pengpengyi92.

  • Honor declared custom item serializers in typed NumPy arrays without changing NumPy validation or coercion.
  • Copy configured composite-publisher templates before binding field data, preventing shared publishers from overwriting earlier outputs or mutating templates.
  • Preserve metadata and nullability of extra Arrow fields during weak-schema validation.

Adversarial review found that the nested TypeAdapter in #296 discards enclosing-model serialization configuration and the caller-provided fallback. Replace it with a guarded recursive core schema that stays inside the Pydantic serialization pipeline. Additional regression coverage checks model byte encoding, caller fallback, serializers that accept coerced integers, and list-subclass items. The guard works on Pydantic 2.6 as well as current releases.

No dependency changes, version bump, or unrelated registry/validation changes. One commit signed off by Tim Paine; no co-author trailer.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Refactor / code cleanup
  • CI / build configuration
  • Other

Validation

  • Full default/Hydra suite on Pydantic 2.14.3: 1,632 passed, 87 subtests passed, 2 skipped.
  • Final affected slice with Lerna/Pydantic 2.14.3: 93 passed, 10 subtests passed.
  • Final affected slice with Pydantic 2.11.10: 93 passed, 10 subtests passed.
  • Final affected slice with Pydantic 2.6.4: 90 passed, 10 subtests passed, 2 skipped, with the pre-existing test_union_field deselected. The skips cover serialization APIs unavailable in that version.
  • make lint, fresh wheel/sdist builds, check-dist --pre-built, twine check, and git diff --check pass.
  • Independent read-only reviews covered the publisher, Arrow, and revised NumPy changes. Additional Arrow runtime checks covered nested and dictionary-encoded fields.

Two existing failures were reproduced against untouched main at 83f2b0a:

  • The full Lerna run has one failure in TestRegistryLoadingErrors.test_bad_class_error: its assertion requires Hydra-specific error wording. The remaining 1,631 tests and 87 subtests pass, with two skips.
  • On Pydantic 2.6.4, the original test_union_field fails when a dictionary reaches the ndarray serializer. Native array validation is unchanged here.

Neither baseline assertion was weakened or removed.

Checklist

  • Linting passes (make lint)
  • Default test suite passes; compatibility caveats documented above
  • Regression tests added and failures reproduced before fixes
  • Documentation updated (not applicable)
  • Changelog / version bump (not included)

Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.91667% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.89%. Comparing base (83f2b0a) to head (0742720).

Files with missing lines Patch % Lines
ccflow/tests/exttypes/test_pydantic_numpy.py 96.39% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #299      +/-   ##
==========================================
+ Coverage   93.85%   93.89%   +0.04%     
==========================================
  Files         205      205              
  Lines       22944    23126     +182     
  Branches     1511     1517       +6     
==========================================
+ Hits        21533    21715     +182     
+ Misses       1120     1119       -1     
- Partials      291      292       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

    1 files  ± 0      1 suites  ±0   2m 13s ⏱️ -32s
1 634 tests +24  1 632 ✅ +24  2 💤 ±0  0 ❌ ±0 
1 721 runs  +28  1 719 ✅ +28  2 💤 ±0  0 ❌ ±0 

Results for commit 0742720. ± Comparison against base commit 83f2b0a.

♻️ This comment has been updated with latest results.

@timkpaine
timkpaine merged commit 58ae11d into main Nov 15, 2026
20 checks passed
@timkpaine
timkpaine deleted the bug/array-and-publisher-fixes branch November 15, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant