Skip to content

feat(rust): add opt-in enableExtensionData to keep undeclared JSON properties - #318

Open
NopeCharon wants to merge 1 commit into
accordproject:mainfrom
NopeCharon:nopecharon/i206/rust-extension-data
Open

NopeCharon wants to merge 1 commit into
accordproject:mainfrom
NopeCharon:nopecharon/i206/rust-extension-data

Conversation

@NopeCharon

Copy link
Copy Markdown
Contributor

Closes #206

Generated Rust structs only declare their own Concerto properties, so serde drops every other JSON property when it deserializes. This PR adds an opt-in enableExtensionData parameter to RustVisitor, named after the C# visitor option from #275. With it on, every generated struct gets a #[serde(flatten)] catch-all field. Undeclared properties land there when deserializing and are written back when serializing. For example, a ConceptDeclaration read through the metamodel's Declaration struct now keeps isAbstract, superType and properties, which is the case #206 describes.

#[serde(flatten)]
pub extension_data: std::collections::HashMap<String, serde_json::Value>,

Changes

  • RustVisitor.visitClassDeclaration writes the field when parameters.enableExtensionData is set. Rust has no struct inheritance, so every struct gets one; in C# only the root classes do.
  • New getExtensionDataFieldName helper: if a class property already maps to extension_data, it appends underscores until the name is unique.
  • The field uses fully qualified paths, so the generated use lines don't change. When the option is on, the generated crate needs serde_json.
  • The option is documented in the RustVisitor class JSDoc. It defaults to false, so existing output and snapshots are unchanged.
  • Unit tests in test/codegen/fromcto/rust/rustvisitor.js cover the emitted field, the default, subclasses, name clashes and imports.
  • A verification test in test/verification/rust.compile.test.js generates the metamodel plus a small user model and builds and runs them with cargo run. It checks that a ConceptDeclaration read as a Declaration, and a Person with extra top-level and nested fields, round-trip unchanged. It is skipped when cargo is not installed, like the other Rust verification tests.

Flags

Design questions for a maintainer:

  1. Option name. I reused enableExtensionData from the C# visitor so the same parameter works for both targets, and named the Rust field extension_data to match. preserveUnknownFields is an alternative if you'd rather have a Rust-specific name.
  2. Which structs. feat(rust): add serde flatten to Declaration in generated output #206 suggested hard-coding this for Declaration only. I made it an opt-in for every struct so it works for any model, including newer model versions read by older generated code. feat(rust): add serde flatten to Declaration in generated output #206 mentioned a stack overflow with fix(rust): add serde flatten for subclass field preservation #205's approach. I couldn't reproduce one: the whole metamodel generated with the option builds, and deserializes nested Model → Declaration data, with serde 1 / serde_json 1. It also builds together with flattenSubclassesToUnion.

What I ran:

  • npm test: lint, licence check and JSDoc pass. 702 tests pass, 59 pending, 0 snapshot changes. The 8 Avro verification failures are from my machine's Java 8, which is too old for avro-tools; they are unrelated to this change.
  • npx mocha test/codegen/fromcto/rust/rustvisitor.js test/verification/rust.compile.test.js with cargo 1.100 installed: every Rust verification case passes, including the new round-trip test. With the visitor change reverted, the 3 new unit tests that expect the field and the round-trip test fail.

Screenshots or Video

N/A

Related Issues

Author Checklist

  • Ensure you provide a DCO sign-off for your commits using the --signoff option of git commit.
  • Vital features and changes captured in unit and/or integration tests
  • Commits messages follow AP format
  • Extend the documentation, if necessary
  • Merging to main from fork:branchname

…operties

Generated Rust structs only declare the properties of their Concerto class,
so serde silently drops every other JSON property when deserializing. A
ConceptDeclaration read through the metamodel's Declaration struct loses
isAbstract, superType and properties, and data written by a newer version
of a model loses the fields the older generated code does not know about.

Add an enableExtensionData parameter to RustVisitor, mirroring the C#
visitor option of the same name. When it is set, every generated struct
gets a catch-all field:

    #[serde(flatten)]
    pub extension_data: std::collections::HashMap<String, serde_json::Value>,

serde collects the undeclared properties into it when deserializing and
writes them back when serializing, so they survive a round trip. Rust has
no struct inheritance, so the field is added to every struct rather than
only to root classes as in C#. If a class property already maps to
extension_data, underscores are appended until the name is unique. The
field uses fully qualified paths, so the generated imports do not change;
the generated code does need the serde_json crate when the option is on.

The option defaults to false, so existing output and snapshots are
unchanged. It is documented in the RustVisitor class JSDoc.

Tests:
- unit tests for the emitted field, the default (no field), subclasses,
  name clashes and imports
- a verification test that generates the metamodel plus a user model,
  then builds and runs them with cargo to check that a ConceptDeclaration
  read as a Declaration, and a Person with extra nested fields, round-trip
  unchanged (skipped when cargo is not installed)

Closes accordproject#206

Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
@mira-accord-project

Copy link
Copy Markdown

Thanks for the PR. It adds an opt-in enableExtensionData option to the Rust generator. The maintainers requested as reviewers (jamieshorten, sanketshevkar) will look at it and comment here. We don't have a date for that yet.

Drafted by an Accord Project agent, reviewed by a maintainer.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regression test does not exercise the nested deserialization behind #206’s stack overflow and needs Model and typed MapDeclaration round trips that invoke the generated MapKeyType and MapValueType deserializers.

1 open issue
What changed in this PR

Adds opt-in preservation of undeclared JSON properties to generated Rust structs, addressing #206’s data-loss use case while leaving default output unchanged.

Changes:

  • Adds enableExtensionData to emit a flattened catch-all map on every struct.
  • Avoids field-name collisions and documents the required serde_json dependency.
  • Adds unit coverage and Cargo-based round-trip verification for metamodel declarations and nested user data.
File Description
test/​verification/​rust.compile.test.js Adds runtime checks for unknown-property preservation.
test/​codegen/​fromcto/​rust/​rustvisitor.js Tests opt-in output, defaults, subclasses, naming collisions, and imports.
lib/​codegen/​fromcto/​rust/​rustvisitor.js Implements and documents extension-data generation.

Review effort: Balanced

'}',
'',
'fn main() {',
' let declaration: Declaration = round_trip(r#"{"$class":"concerto.metamodel@1.0.0.ConceptDeclaration","name":"Dog","isAbstract":false,"properties":[]}"#);',

@mira-accord-project mira-accord-project Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this change. The opt-in enableExtensionData option is small and well scoped. It defaults to off, so existing output and snapshots don't change. The name-clash helper is sensible, and the PR description follows the repo's guidance: commands run, numbered design questions, and the choices explained. The unit tests pin the exact output and the default. One gap remains in the verification test: it doesn't exercise the nested case that issue #206 warns about. If the round-trip covers a Model with non-empty declarations, including a typed MapDeclaration, this is ready for a maintainer to decide on the two design questions (option name, and every struct versus Declaration only).

Nothing blocking; some suggestions below. Findings: 1 should-fix.


Automated review by Mira, an Accord Project agent, at 1837e66. Findings are labelled blocking, should-fix or nit; a maintainer makes the final call.

'}',
'',
'fn main() {',
' let declaration: Declaration = round_trip(r#"{"$class":"concerto.metamodel@1.0.0.ConceptDeclaration","name":"Dog","isAbstract":false,"properties":[]}"#);',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should-fix · tests

The round-trip test only reads a ConceptDeclaration with properties: [] as a Declaration. Declaration doesn't declare properties or isAbstract, so those land in extension_data as raw serde_json::Value and the generated nested deserializers never run. Issue #206 says flatten on all base types overflowed the stack on MapKeyType/MapValueType, and the PR description says nested Model → Declaration data deserializes fine. The committed test doesn't pin that claim. Please extend the Rust program to round-trip a Model with non-empty declarations, including a MapDeclaration with a typed key and value. Assert the typed fields as well as extension_data, so the MapKeyType and MapValueType deserializers are really exercised. This also answers the open review thread on the same line.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(rust): add serde flatten to Declaration in generated output

2 participants