Skip to content

feat(display-format): decode Protobuf schemas and Schema Registry values - #3979

Open
J2TeamNNL wants to merge 13 commits into
TableProApp:mainfrom
J2TeamNNL:feat/3607-protobuf-schema-registry
Open

J2TeamNNL wants to merge 13 commits into
TableProApp:mainfrom
J2TeamNNL:feat/3607-protobuf-schema-registry

Conversation

@J2TeamNNL

@J2TeamNNL J2TeamNNL commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Builds on #3962 (igbinary and Avro), which is still open: this branch starts from it because it needs that PR's Avro decoder and schema file storage. Until #3962 merges, its commits show in this diff too; the new work is the last commits.

  • Display As > Protobuf Schema… on binary columns takes a .proto file or a descriptor set (protoc --descriptor_set_out), then asks which message when the file has several. The tree names each field and badges it with its declared type. Text matches protoc --decode; fields the schema does not name, or names with another wire type, show by number. A singular message sent twice is merged, as a parser does.
  • Display As > Schema Registry, offered when the connection names a registry. Each value's prefix gives the schema ID; the schema is fetched from GET /schemas/ids/{id}, with its references from /subjects/{subject}/versions/{version} (Avro and Protobuf), and kept for the session. Protobuf picks the message by the frame's message index. JSON Schema values open in the JSON tree.
  • The grid never waits: a miss starts the fetch, the cell shows hex, and the column redraws when the schema lands. The viewer waits for it. A fetch that fails stays hex with the registry's reason, and is retried after 30 seconds.
  • Kafka connection settings gain Schema Registry URL, User and Password (Keychain), in the plugin and the curated snapshot. The app reads them by field id, so any driver can declare them. Only http and https; redirects to another host or scheme are not followed; responses are capped at 4 MB and 60 seconds.
  • Kafka: a value that starts with a zero byte makes the column binary, since a framed value can still be valid UTF-8. The Schema Registry format also works on a text column for an older plugin.
  • Schema files: AvroSchemaStore's file handling moved to a generic SchemaFileCache; ProtobufSchemaStore keeps ProtobufSchemas/<hash>.pbschema the same way, swept with the Avro ones.
  • Defaults chosen: a .proto file is read alone (its imports are not opened), so a field of an imported type shows by number; a descriptor set built with --include_imports covers that case. The registry is reached directly, not through the connection's SSH tunnel.

Test plan

  • ProtobufSchemaDecoderTests: fixtures written by protoc --encode (libprotoc 36.2), compared with protoc --decode output, for a .proto file and the descriptor set protoc built from it: every scalar type, enums, packed and unpacked repeats, maps, oneof, nesting, a proto2 group, unknown fields, merged messages, the packed-field budget, broken values and bad schema files.
  • SchemaRegistryTests: responses recorded from confluentinc/cp-schema-registry 7.7.1, values written by confluent-kafka 2.6.1's AvroSerializer and ProtobufSerializer against it. Avro, Protobuf with a reference and message index [0] and [1], basic auth, the 404 reason, Avro references, broken values, the cache's single fetch, its notification and kept failure.
  • ValueDisplayFormatTests, BinaryDisplayFormatTests, ValueDisplayFormatStorageTests for the new stored formats, the cell and the schema sweep.
  • The decoders were also run outside the app against the same fixtures; output matches protoc --decode line for line. Hosted tests compile here (build-for-testing), and run in CI.

Docs: TableProApp/docs#155 (draft, merges after this one).

Part of #3607

J2TeamNNL and others added 13 commits November 1, 2026 04:28
igbinary decodes into the PHP serialized tree, numbered as serialize() numbers it. Avro decodes with a schema file chosen per column, stored by a hash of its text.
…formats readable by 0.83

Fixes the review of TableProApp#3962: igbinary r:/R: numbering, copies, caps and float text; Avro framing, decimal width, named logical types and zero-byte caps; a second storage key for formats 0.83 lacks; schema files instead of defaults.
…hemas from a Schema Registry

Display As > Protobuf Schema… reads a .proto file or descriptor set and shows field names and declared types. Display As > Schema Registry fetches the Avro, Protobuf or JSON schema each Kafka value names by its wire-format prefix, from the registry set on the connection.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants