Preserve CBOR container representation through decode

[cffls]

Follow-up to #503. Fixes #507, and closes the remaining case in #402.

Summary

#503 moved PyCardano onto cbor2 6 and restored definite vs indefinite arrays by re-scanning the raw payload after decoding. Two things were left:

• the same representation loss still applied to indefinite-length strings and maps
• the array path had three defects of its own

Indefinite strings and maps

The scanner already recorded that a container was indefinite; the apply step discarded it. That flag is now carried through to four new types (IndefiniteByteString, IndefiniteTextString, IndefiniteDict, IndefiniteFrozenDict). Each subclasses its builtin, so isinstance, equality and hashing are unchanged and only an added attribute carries the chunk lengths.

Chunk boundaries are part of the encoding, so the scanner records each chunk length and the encoder replays exactly those rather than renormalising to 64. cbor2 encodes all four natively as their definite base types, which is why each is claimed explicitly in the encoder registration — the same reason indefinite lists already needed that treatment.

This is what the ledger needs. A Plutus byte string longer than 64 bytes arrives chunked, and such a datum previously came back with a different hash:

in : d8799f5f5840…48…ffff   81 bytes, chunked      hash 7e67a4eda8456f31
out: d8799f5848…ff          78 bytes, definite     hash a8c2a2cf5d5e09da

That is #507: TransactionBuilder deep-copies its outputs and RawPlutusData.__deepcopy__ goes through CBOR, so the builder emitted datums the node rejected with ByteString exceeds 64 bytes.

It is also the remaining half of #402. The Liqwid fixture added in #493 only holds byte strings short enough for one definite-length chunk, so it passed before this branch — but a datum with a longer value still moved the body hash on decode, which is exactly the "cannot recover the signature from unsigned_cbor" symptom reported there:

                   main 877be3a        this branch
body hash before   f2b2d3af52ef3a74    f2b2d3af52ef3a74
body hash after    9597847ad9afd1f6    f2b2d3af52ef3a74
hash stable        False               True

Three defects in the array path

• Duplicate map keys collapse on decode, leaving the shape walk misaligned with the decoded dict and silently producing wrong bytes. Such payloads are now rejected at the source (allow_duplicate_keys=False), and the opt-in path raises DeserializeException rather than guessing — guessing means a wrong transaction hash.
• Bare asserts guarded trailing bytes and the COSE message body, and vanish under python -O. Both now raise DeserializeException. The COSE guard also could never fire as written, since verify() prepends the tag byte itself; it now validates the message body, as pycose's own decoder did before Support cbor2 6 #503.
• A tag consumed by a semantic decoder left its content immutable, so the self-describe tag decoded to a tuple instead of a list. Type guards keep decoders that return a non-container (Fraction, Decimal, bignum, datetime) untouched.

Performance

Scalar shapes are shared rather than allocated per decoded item, which offsets most of the added work. A real 3.6 KB transaction decodes in 0.050 ms against 0.037 ms on the old cbor2pure path. The pathological 190 KB synthetic payload went from 19.4 ms to 15.3 ms.

Poetry pin (second commit)

pipx install poetry resolved to whatever was newest when a job ran, so an unchanged commit could pass one day and fail the next — 2.4.2 was published two days before #502 and broke the integration tests that way. It also matters for the lock file: Poetry stamps poetry.lock with a content hash and versions differ in how they compute it, so a mismatched Poetry rejects the lock and refuses to sync the environment. Pinned to the generator recorded in the lock file's first line, declared once per workflow. publish.yml too, since it runs poetry install and poetry check before pushing to PyPI.

Happy to split this into its own PR if you'd rather keep the two separate.

Testing

• 694 passed, 5 xfailed; make qa clean (flake8, mypy, black)
• 28 new tests: byte-exact round trips for indefinite strings and maps (empty, nested, multibyte, map-key and Plutus-tag forms), the chunked datum, builtin semantics, frozen-map immutability, deep copy and pickle, chunk-length validation, duplicate keys on both paths, trailing bytes, self-describe mutability, consumed-tag decoders, and the COSE guard
• The RawPlutusData loses >64-byte bytestring chunking on deepcopy, so TransactionBuilder emits ledger-invalid datums #507 regression test fails on 877be3a at the deepcopy assertion and passes here
• The Unable to recover signature from unsigned_cbor when RawPlutusData in output #402 test signs the body hash as a wallet would before export, decodes, then verifies that same signature against the re-encoded body; it fails on 877be3a at the body hash assertion
• Real transaction fixtures round trip identically to before this change

Notes

Two pre-existing issues this does not address, both unchanged from before #503:

• Two transaction fixtures in the suite still fail to round trip. These are data-model normalisations, not representation loss: one collapses a Value with an empty multi-asset map to a bare coin, the other reshapes the inputs field.
• A RawPlutusData built directly from raw long bytes (rather than via from_dict or CBOR) still emits a definite byte string, since there is no chunking to preserve. RawPlutusData loses >64-byte bytestring chunking on deepcopy, so TransactionBuilder emits ledger-invalid datums #507's own suggested fix targets this case; worth a follow-up, and note that the snippet as written in the issue does not work on its own — _dfs never recurses into a decoded CBORTag, because its isinstance(obj.value, list) guard is false for IndefiniteList.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 94.11765% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.96%. Comparing base (877be3a) to head (4698edc).

Files with missing lines	Patch %	Lines
pycardano/serialization.py	93.98%	5 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #508      +/-   ##
==========================================
+ Coverage   90.79%   90.96%   +0.17%     
==========================================
  Files          37       37              
  Lines        5422     5537     +115     
  Branches      825      843      +18     
==========================================
+ Hits         4923     5037     +114     
  Misses        309      309              
- Partials      190      191       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nielstron]

I think we can clean this up a bit by making all indefinite-classes inherit from one shared indefinite superclass. it should at least simplify the is instance call, maybe also the shared write/read logic for bytes and strings.

[nielstron]

this test looks weird to me should we not expect that allow duplicate keys returns an object with the right keys? or can we currently not handle that?

[cffls]

Good catch, that test was asserting the wrong thing. Changed in 4698edc.

We genuinely cannot restore the representation of a duplicate-key map: the decoded dict holds one entry per distinct key, so the scanned shapes no longer line up with the values they came from, and re-encoding can't reproduce the input bytes no matter what we do. But you're right that the keys themselves are fine, and raising at a caller who explicitly passed allow_duplicate_keys=True was unhelpful — they asked for lenient decoding and got an exception telling them to pass False.

So the opt-in path now hands back the decoded map and the test asserts that:

decoded = loads(bytes.fromhex("a2018102019f02ff"), allow_duplicate_keys=True)
assert decoded == {1: [2]}
assert dumps(decoded) == bytes.fromhex("a1018102")

The surviving value keeps no array shape, so the bytes differ from the input — that's the documented cost of opting in, and the reason the default still rejects duplicates. I moved that explanation onto loads' docstring.

[cffls]

Thanks — both addressed in 4698edc.

Shared superclass. Indefinite is now the common base of all six types, and IndefiniteString sits between it and the two string variants, holding the chunk bookkeeping they share (chunks validation, default chunking, and slicing the payload into the chunks the encoder writes). That collapsed more than the isinstance call:

• default_encoder's assertion lists Indefinite instead of four concrete classes
• the two chunk-writing branches became one, dispatching on IndefiniteString
• _write_indefinite_chunks(encoder, header, payload, chunks, text=...) became _write_indefinite_string(encoder, value) — the header and the per-chunk form now come off the class, so the text flag is gone
• the long-ByteString fallback reuses that same path rather than repeating the chunking inline

The one thing that can't key off the base is the cbor2 registration, since it looks encoders up by exact type, so each concrete class is still claimed individually. That list is now a module-level _INDEFINITE_TYPES tuple so the loop has a single source, with a comment saying why the base isn't enough. Added a parametrised test asserting all four decoded forms carry the marker, so the encoder's dispatch can't silently miss one.

Duplicate keys. Replied on the thread — short version is you were right, the opt-in path now returns the decoded map with correct keys instead of raising, and the limitation is documented on loads rather than expressed as an exception.

698 passed, 5 xfailed; make qa clean. All 25 byte-exact round trips still hold, the ByteString chunk-reuse and 64-byte fallback both still behave, and decode timing is unchanged (15.7 ms median on the 190 KB synthetic, against 15.3 before).