Skip to content

feat(display-format): decode igbinary and Avro values - #3962

Open
J2TeamNNL wants to merge 11 commits into
TableProApp:mainfrom
J2TeamNNL:feat/3607-igbinary-avro
Open

J2TeamNNL wants to merge 11 commits into
TableProApp:mainfrom
J2TeamNNL:feat/3607-igbinary-avro

Conversation

@J2TeamNNL

@J2TeamNNL J2TeamNNL commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Display As gains igbinary and Avro Schema… on binary columns, under the feat(display-format): decode MessagePack and Protobuf, and swapped-time UUIDs #3761 rules: per column, scoped to connection and table, rendering only. Copy and Find read the decoded text; filters and highlight rules read the stored bytes. A value that does not decode stays hex and the viewer says why.
  • igbinary Text view and copy match serialize() byte for byte: r: for an object seen again, R: for a PHP reference, numbered by PHP's var-hash rules, arrays repeated by value written in full, integer member keys of __serialize objects (ArrayObject), non-UTF-8 strings, property names and Serializable payloads with their bytes and byte length, floats as serialize_precision -1 writes them.
  • igbinary refuses a value that expands past 10M values or 20 MB of string text through repeated arrays or string IDs, or nests past 256 levels through copies. The text writer cuts each string to what the preview shows before copying it.
  • PHP Serialized viewer: r: and R: parse as different values, and an O: with i: member keys parses instead of failing.
  • Avro schema file: checked in the open panel, which stays open on a file that is not a schema. Kept as AvroSchemas/<hash>.avsc in Application Support and removed once no saved format names it. The menu shows the schema's root type name. Choosing a file this Mac had lost keeps it again (a copy that went bad is written over) and an open viewer decodes again; grids show it from their next load.
  • Avro decoding: unions show their branch; decimal, uuid, date, time-*, timestamp-*, local-timestamp-* (millis, micros, nanos) and duration read as values; a named fixed keeps its logical type where referenced by name; a redefined name is refused; size, precision and scale must be integers.
  • Avro framing: single-object encoding (C3 01) is read when its fingerprint is the schema's (CRC-64-AVRO of the Parsing Canonical Form) and refused otherwise; a Confluent prefix is tried once the plain datum fails, and a bytes schema's body, decimal included, is taken as is. The viewer names the schema ID or fingerprint.
  • Avro bounds: the cell's budget stops building, not reading, so the cell picks the frame the viewer picks; at most 10M values are read, built or not, so records that take no bytes cannot nest exponentially, and a grid cell stops at 1M and leaves the value to the viewer; zero-byte items are capped across every block without overflow, and which types take no bytes is worked out once per schema, without recursion; a decimal wider than its precision holds shows as bytes, and conversion runs in base 10^9.
  • Storage: formats 0.83 does not know go in a second key per table (<key>.added), so 0.83.1 still decodes the first. Keys are read entry by entry, an unknown name is kept, and the first key wins where both hold a column.
  • ValueDisplayFormat is no longer a raw-value enum, so .avro(schemaID:) carries its schema through every per-column path. Existing stored names are unchanged; an Avro ID must be 32 hex digits.
  • A missing schema gets its own banner instead of "Not valid Avro".
  • Tests changed on purpose: PhpSerializeParserSpecialTests.referenceLowerR (r: is .objectReference), ValueDisplayFormatTests (Avro IDs are 32 hex digits).

Test plan

  • IgbinaryDecoderTests: 38 fixtures from igbinary 3.2.17RC1 on PHP 8.3, each checked against serialize() text and its parsed tree, plus binary strings, expansion and depth limits, broken values.
  • AvroDecoderTests: fastavro fixtures, Avro Java fingerprint vectors, single-object and Confluent framing, the budgeted framing decision, decimal width, named logical types, nanos, duration, zero-byte caps, the overflow value and nested zero-byte records.
  • ValueDisplayFormatStorageTests: the first key decodes as 0.83.1's enum, a 0.83 write wins, unknown names survive, rename moves both keys, schema files follow the formats that name them.
  • AvroSchemaStoreTests, BinaryDisplayFormatTests, ValueDisplayFormatTests, CellInteractionResolverBinaryTests, FieldEditorResolverTests, PhpSerializeParser*Tests, PhpTreeBuilderTests.

Docs: TableProApp/docs#149 (draft, merges after this one).

Part of #3607

igbinary decodes into the PHP serialized tree, numbered as serialize() numbers it. Avro decodes with a schema file chosen per column, stored by a hash of its text.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants