Skip to content

*: add text annotations and their go and ruby codecs - #17

Draft
AlexisMontagne wants to merge 21 commits into
masterfrom
am/text-annotations
Draft

AlexisMontagne wants to merge 21 commits into
masterfrom
am/text-annotations

Conversation

@AlexisMontagne

Copy link
Copy Markdown
Member

Typed values like ownerships (company:42) and credential references (shopify:7) are turned into strings and back by code written by hand in each language: baseutil in Go, identity-thrift-utils and base-thrift-utils in Ruby. Those copies have drifted apart. This PR describes the text form once, in the IDL, with structured annotations, and adds Go and Ruby codecs that read them at runtime. Both codecs are checked against the same test cases, so they accept and produce exactly the same texts.

include "types/annotation/text.thrift"

@text.EnumFormat{"trim_suffixes": ["OAuth"], "accept_thrift_names": true}
enum CredentialType {
  @text.Name{"value": "gmail"}
  GoogleOAuth = 1,
  ShopifyOAuth = 4, // "shopify"
}

@text.Format{"pattern": "{type}:{id}"}
struct CredentialReference {
  1: required CredentialType type;
  2: required i64 id;
}

Annotations: types/annotation/text.thrift

  • Format takes a python-like f-string pattern made of {field} placeholders and literal separators, with {{ and }} as escaped braces. Format specs and conversions are not supported.
  • Name overrides the text of a single enum value.
  • EnumFormat sets the defaults for an enum: a Case, trim_prefixes / trim_suffixes (the first match in list order is trimmed), and accept_thrift_names, which also accepts the thrift names such as Type_Company.
  • Words are split on underscores, on case changes and at the end of acronyms, while keeping IDs whole. Camel and Pascal case render the Go initialisms.

Compiler:

  • Enum values accept structured annotations (thrifty.yy).
  • Plugins now receive the structured annotations; they used to get an empty list (t_serializer.h).
  • Go generator:
    • Each enum gets a func (T) EnumDefinition() thrift.EnumDefinition holding the annotations of the enum and of its values.
    • Struct field definitions carry their Requiredness.
  • Ruby generator: enum modules gain NAME, NAMESPACE, THRIFT_NAMES (the names as declared, since VALUE_MAP capitalizes them), the enum annotations, and VALUE_LEGACY_ANNOTATIONS / VALUE_STRUCTURED_ANNOTATIONS.
  • JSON generator: enum values now include their annotations.
  • TEnumIterator gains operator==, which fixes the build with recent libc++ on macOS.

Go: lib/go/thrift/types/annotation/text

  • MarshalStruct, UnmarshalStruct[T], MarshalEnum and UnmarshalEnum[E].
  • Struct[T] and Enum[E] wrappers for JSON, SQL and flags.
  • Validate reports invalid annotations, for use in tests.
  • CompileStruct and CompileEnum work from definitions only, for dynamic callers such as tcurl or an MCP gateway.
  • The pattern and caser code live in internal packages. internal/goinitialism is now the single Go initialisms list, shared with gocodegen, whose output doesn't change.

Ruby: Thrift::Types::Annotation::Text, loaded by require 'thrift'

  • dump / load, dump_enum / load_enum and validate.
  • Errors are subclasses of ArgumentError.

Plugin helpers:

  • thrift.FindAnnotations[T].
  • value.DecodeValue, which decodes a structured annotation value into its generated type.
  • Runtime() on AnnotationDefinition, EnumDefinition and StructDefinition, which convert plugin definitions into the runtime ones. With these, a plugin can validate annotations with text.CompileEnum / text.CompileStruct.

Tests:

  • test/TextTestCases.json holds the cases shared by the Go tests and the Ruby spec.

  • test/TextTest.thrift is generated by the lib/go/test and lib/rb test pipelines. Both use THRIFT_TYPES_PATH, so the generated code imports the annotations the way consumer code does.

  • Did you create an Apache Jira ticket? (not required for trivial changes)

  • If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?

  • Did you squash your changes to a single commit? (not required, but preferred)

  • Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?

  • If your change does not involve any code, add [skip ci] at the end of your pull request to free up build resources.

Notes:

  • The generated code of every program changes, but only by addition: Go enums get EnumDefinition(), Go struct fields get Requiredness, Ruby enums get the new constants, and JSON enum values get annotation keys.
  • One edge case in Ruby: an enum with a value literally named NAME, NAMESPACE or THRIFT_NAMES would collide with the new constants.
  • Committed stubs aren't regenerated. Only the text stubs are new; the other stubs under lib/ (rpc, program_definition…) will gain EnumDefinition() on their next regeneration.
  • Two Ruby specs already fail on master: json_protocol_spec.rb "pretty print union" and "pretty print union with a container field". They fail the same way with code from the released v2.7.14 compiler.
  • Follow-ups once this is released:
    • upfluence-if switches to types/annotation/text.thrift and drops its text-codec plugin.
    • The baseutil, identity-thrift-utils and base-thrift-utils serializers delegate to the codecs.
    • tcurl accepts text forms as input.

Recent libc++ compares iterators with == when building a map from a range,
so the compiler failed to build on macOS with only != defined.
…tations to plugins

Structured annotations were always serialized as an empty list, so plugins
could not read them. Their values are now converted to types.value.Value,
following the type of the annotation.
Enum values only accepted legacy annotations, so plugins could not attach
typed metadata to them, e.g. the text form of a value.
Runtime consumers of the annotations, such as the text codec, need the
annotations of the enum values and whether a struct field is optional.
…initions

Every enum gets an EnumDefinition method holding the annotations of the
enum and of its values, and the field definitions of the structs hold their
requiredness, so libraries can read them without code generation.
…es and annotations

Enum modules now hold their name, namespace, annotations and the
annotations of their values. THRIFT_NAMES keeps the names as declared since
VALUE_MAP capitalizes them, so the ruby and go libraries derive the same
texts from them.
Describes the canonical text form of structs and enum values in the IDL,
e.g. "company:42" for an ownership, so it is defined once instead of being
hand written in every language.
Encodes the structs annotated with Format and the enums as their text
form, with wrappers for JSON, SQL and flags. The test cases are shared with
the ruby library so both accept and produce exactly the same texts.
Mirrors the go text codec: Thrift::Types::Annotation::Text dumps and loads
the structs annotated with Format and the enums, checked against the test
cases shared with the go library.
Generates the test types in the go and ruby test pipelines, resolving the
annotations as standard types like the code generated by consumers does.
…alue annotations

Tools consuming the JSON descriptors, such as tcurl, need the annotations
of the enum values, e.g. text.Name, to encode them like the libraries do.
Every consumer of the structured annotations looped over them with a type
assertion, the text codec and the naming extractor now share the helper.
gocodegen and the text codec each kept a copy of the revive list, in
different shapes. The canonical spelling also removes the IDs special case
of gocodegen.
…tting words

The acronym rule split UserIDs into "User I Ds", so the snake case of an
enum value named after IDs was "user_i_ds". Go and ruby now keep the
initialisms ending in lower case whole, the caser always applying the Go
initialisms.
Plugins receive the structured annotations as values, they can now decode
them into the generated annotation types instead of reading every field by
hand.
Gives plugins the definitions the libraries read at runtime, with the
structured annotations decoded into their generated types, so they can
validate them with the same code, e.g. text.CompileEnum.
@AlexisMontagne AlexisMontagne self-assigned this Nov 13, 2026
Thrift structs render as an object with ActiveSupport and as their
inspection with Oj in compat mode, as used by userializer. Including the
module renders a struct as its text form with both, without changing the
JSON of the structs that did not opt in.
…tional fields

A struct field annotated with Format can be referenced by a pattern, its
pattern being inlined, so composite identifiers such as crm-server's entity
segments ("production:list:42:selected") reuse the text form of the entity.
Optional fields can be left out of the pattern, for structs carrying hints
such as the handle of a media.
…t optional fields

Mirrors the go codec, dump and load building the nested structs, checked
against the shared test cases.
Callers building the field types of CompileStruct, such as the plugins,
only need to resolve the referenced fields, skipping the optional fields
left out of the pattern, which may reference the struct itself.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant