From b22b828fd659d9ead261686eb53e50d134da38fe Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 19:19:14 +0000 Subject: [PATCH 01/21] feat(rest): store a REST response in a file document (#922) A "Call REST service" activity configured to store its response in a file document had no MDL form at all, and both readers fell through to String. DESCRIBE reported `returns String`, and the describe -> exec round trip then wrote that back. Measured on mxbuild 11.6.6: stored before: ResultHandlingType = FileDocument VariableType = ObjectType(MyModule.MyFile) stored after: ResultHandlingType = String VariableType = StringType mx check: unchanged, only the project's pre-existing baseline error So the activity was silently retyped with a valid model and a green build -- nothing downstream could notice. On the legacy engine the output variable went too, because the unread result handling took the `$var =` fallback with it. The two readers failed differently, and each looked fine from inside itself: the legacy parseResultHandling had no FileDocument case and returned nil, while modelsdk's restResultHandlingFromRaw read VariableType.Entity and then discarded it, keeping only a literal match on System.HttpResponse so that everything else -- FileDocument AND any other object type -- became String. It now reads Mendix's own ResultHandlingType discriminator, falling back to the VariableType because that property is omitempty. Syntax is `returns Module.Entity`, added as the LAST alternative of restCallReturnsClause so the keyword forms keep winning; a test pins that. Mendix rejects the BASE System.FileDocument as a return type (CE0362), so the entity is always a specialization -- which is possible because CE1540 lists FileDocument among the four System entities that may be specialized. MDL064 reports the base type and an unqualified name before the write. The other half of the report needed no change. `returns response` round-trips losslessly, and the suggested "HttpResponse specialization" cannot exist: CE1540 permits only User, FileDocument, Image and Paging, so `response` already names the only type such a result can have. Verified, and pinned by a round-trip test so a future change here has to stay honest. The silent String fallbacks in the describer are removed. A result handling mxcli cannot reconstruct now renders as text the parser REJECTS, naming what was unsupported: a describe that fails is recoverable, one that quietly means something else is not (ADR-0005). A test asserts the refusal does not parse. Verified: round trip on BOTH engines (gateEngines) -- the check that finds this class, since either reader alone looks consistent; the authored activity builds at baseline under mx check; 338 shipped examples scanned with 0 new hits; 19 microflows described on both engines with no spurious refusal; unit tests confirmed to fail with the visitor branch and the renderer case stubbed. Also adds `mxcli syntax rest.call`, which documents the activity and all five RETURNS forms -- there was no topic for it before. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- .claude/skills/fix-issue.md | 1 + .claude/skills/mendix/write-microflows.md | 21 ++ cmd/mxcli/syntax/features_integration.go | 38 +++- docs/01-project/MDL_QUICK_REFERENCE.md | 3 +- .../922-rest-call-base-filedocument.fail.mdl | 41 ++++ .../922-rest-call-returns-file-document.mdl | 71 ++++++ mdl/ast/ast_microflow.go | 6 +- .../modelsdk/microflow_mapping_test.go | 4 +- .../modelsdk/microflow_read_actions.go | 28 ++- mdl/backend/modelsdk/microflow_write.go | 10 + mdl/executor/cmd_microflows_builder_calls.go | 10 + mdl/executor/cmd_microflows_format_action.go | 31 ++- mdl/executor/rest_filedocument_result_test.go | 203 ++++++++++++++++++ .../roundtrip_rest_filedocument_test.go | 84 ++++++++ mdl/executor/validate_microflow.go | 3 + mdl/executor/validate_microflow_rest.go | 56 +++++ mdl/grammar/domains/MDLMicroflow.g4 | 1 + mdl/visitor/visitor_microflow_actions.go | 7 + sdk/microflows/microflows_actions.go | 14 ++ sdk/mpr/parser_microflow_actions.go | 14 ++ sdk/mpr/writer_microflow_actions.go | 19 ++ 21 files changed, 652 insertions(+), 13 deletions(-) create mode 100644 mdl-examples/bug-tests/922-rest-call-base-filedocument.fail.mdl create mode 100644 mdl-examples/bug-tests/922-rest-call-returns-file-document.mdl create mode 100644 mdl/executor/rest_filedocument_result_test.go create mode 100644 mdl/executor/roundtrip_rest_filedocument_test.go create mode 100644 mdl/executor/validate_microflow_rest.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index bc9a1f48e..1e2469243 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -561,3 +561,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli check` reports `✓ Syntax OK`, `exec` writes the microflow, and the defect appears only when a human opens Studio Pro's Errors pane: CE0038 on a value-less `declare`, CE0068 on a `return` inside a loop, CE0111 on `declare $X` followed by `$X = call microflow …` | Nothing in the MDL rule set covered them — each is a Mendix consistency rule with no MDL counterpart. CE0111's real scope is far wider than the reported case: a microflow's variable namespace is **flat**, so parameters, loop iterators and every activity output share it, and neither a branch nor a loop body opens a scope (all seven combinations measured on mxbuild 11.6.6) | `mdl/executor/validate_microflow_ce_gaps.go` (MDL061/062/063), wired from `validate_microflow.go`; fixtures `mdl-examples/bug-tests/893-check-gaps-*.mdl` | Error severity alone closes the gap — `exec` pre-flights the whole script and refuses with nothing written; they are deliberately kept OUT of `execEnforcedMicroflowRules` so `--no-check` still works. **Run any new rule over `mdl-examples/` before wiring it up**: this one hit 4 of 374 files and 3 were FALSE positives, because the rule reads the AST while the outcome depends on what the BUILDER emits — `while true` becomes an ExclusiveMerge back-edge and not a loop object (#350), `returns T as $Var` routes the End event elsewhere, `set $x = contains($str,$str)` parses as a ListOperationStmt that the builder rewrites to a Change Variable (ledger #53/#63), and an `@excluded` document is never checked by mxbuild at all. The 4th was a genuine CE0111 in a shipped example. The shared predicate `stringOverloadedListOp` keeps rule and builder from drifting. Issue #893 items 1/2/6 | | `calculated by Module.Microflow` on an attribute is accepted by `check` and by exec ("Added attribute"), but the stored document holds a plain `DomainModels$StoredValue` with no calculation link — the microflow name appears nowhere in the domain model unit, `mx check` reports **0 errors**, and the attribute is simply empty at runtime. Both the CREATE (inline) and ALTER (`ADD`/`MODIFY ATTRIBUTE`) paths. A microflow whose signature cannot work is accepted too, masked by the same drop | `attributeToGen` in the **modelsdk** writer had arms for OqlViewValue / ODataMappedValue / ODataMappedPrimitiveCollectionValue and a `default:` that emits StoredValue — no `CalculatedValue` arm — so the binding the executor had already resolved fell through and was discarded. The **legacy** writer had the arm all along (`sdk/mpr/writer_domainmodel.go`), which is why the feature read as implemented; modelsdk is the default engine (`--engine`), so everyone hit the broken path. The reader had no `CalculatedValue` case either, so an unrelated ALTER on the same entity destroyed a binding made in Studio Pro | `mdl/backend/modelsdk/domainmodel_write.go` (`attributeToGen`) + `domainmodel.go` (`attributeFromGen`) + `mdl/executor/calculated_attributes.go` (`resolveCalculatedValue`, called from the three sites in `cmd_entities.go`) | Add the write arm (`genDm.NewCalculatedValue`, `SetMicroflowQualifiedName` → the `Microflow` ByNameRef key, `SetPassEntity`) **and** the read arm — a write-only fix leaves the read-modify-write data loss in place, which is the worse half. Derive `PassEntity` from the signature rather than hardcoding it (legacy hardcoded `microflowRef != ""`): measured on 11.13.0, an entity-parameter microflow (`PassEntity=true`) and a parameterless one (`PassEntity=false`) BOTH build at 0 errors, so refusing the parameterless form would have been wrong. Signature rules are refused at exec (the #833 placement), and each was checked against mxbuild rather than assumed — wrong entity parameter and wrong return type are both **CE7247**, but the return-type message is *"should be Integer/Long"*, so **Integer and Long are one family** and a strict equality check refuses valid MDL (caught only by reading the CE text). To ask mxbuild about a binding mxcli now refuses, stub the check and rebuild — `--engine legacy` does NOT bypass it, because the validation lives in the engine-independent executor. Tests `TestAttributeToGen_CalculatedValue`, `TestAttributeFromGen_CalculatedValue`, `TestResolveCalculatedValue_*` (the backend three fail with `value is *domainmodels.StoredValue` when reverted); fixture `mdl-examples/bug-tests/917-calculated-attribute-binding.mdl`. Issue #917 | | `mxcli check -p app.mpr` passes a microflow containing `if $obj/Status = 'Open'` — comparing an enumeration attribute to a string literal, the first example in the type-checking proposal and the shape people actually write. The same mistake written as a create or change member *is* caught, which makes the gap look arbitrary | Two causes, both invisible. (a) **`inferKind` returned `KindUnknown` for every `AttributePathExpr`**, so `$obj/Attr` typed to nothing and every rule downstream of it stayed quiet. `exprcheck.Scope` is `Lookup(name) (TypeKind, bool)` — it can say "$P is an Object" but not *which* entity, so there was nowhere to put the answer; the adapter computed a variable→entity map (`buildVarEntityScope`) and used it only to label a slot path, never passing it to the checker, and it covered body-introduced variables but **not parameters** — the ordinary case. (b) Even resolved, nothing fired: **E001 keys off the SLOT** (`CreateItem.Value:Entity.Attr`), which exists for an assignment and not for a comparison | `mdl/exprcheck/interfaces.go` (`EntityScope`, `Context.Entities`), `mdl/exprcheck/parser.go` (`attributePathKind`, `pathTargetEntity`, `checkEnumComparedToString`), `mdl/exprcheck/adapters/` (`walkFlow`, `addParamEntities`, `entityScope`), `mdl/exprcatalog/` (`AssociationTarget`) | Put the object side in its **own seam** (`EntityScope`: `VariableEntity` + `AssociationTarget`) beside `Scope` rather than widening either it or `CatalogReader` — `CatalogReader`'s shape is what stays re-syncable from the upstream fork. **A multi-hop expression path is not an XPath path**: XPath spells the intermediate entity (`[Assoc/Entity/Attr]`) so a walk can read it off, while an expression does not (`$O/Mod.Assoc/Attr`), so every hop must resolve through the association index. For (b), emit the **same code and message** from the comparison as from the slot — one defect should not have two names depending on where it was spotted. **Generalisable**: when a checker is silent, separate "did the input resolve" from "is there a rule for this shape" before touching either — here resolution and detection were both missing, and fixing only one would have looked like the fix failed. Verify with a **probe corpus that actually exercises the construct**: 21 microflows described back to MDL contained 40 attribute-path lines including an association hop, and were clean before and after, which is what makes "no false positives" mean something. Controls: path→`KindUnknown` restored, `Entities` dropped from the Context, and `addParamEntities` removed each fail a distinct test. **Still open**: a *terminal* association step (`$Order/Mod.Order_Lines`) types to unknown — Object vs List depends on the association's kind and direction, and guessing costs false positives | +| `describe microflow` renders a REST call's result as `returns String` when the activity actually stores the response in a **file document**, and the describe → exec round trip then writes String back — retyping the activity with `mx check` reporting nothing either way. On the legacy engine the `$var =` output variable disappears too | MDL had no syntax for the case, `sdk/microflows` had no FileDocument variant, and both readers fell through to String: the legacy `parseResultHandling` had no `FileDocument` case (so the whole handling read back nil, taking the output-variable fallback with it), while modelsdk's `restResultHandlingFromRaw` read `VariableType.Entity` and then discarded it, matching only the literal `System.HttpResponse`. Mendix stores all five variants as ONE `Microflows$ResultHandling` carrying a `VariableType`, discriminated by the `ResultHandlingType` enum | `mdl/grammar/domains/MDLMicroflow.g4` (`restCallReturnsClause`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `sdk/microflows/microflows_actions.go`, `sdk/mpr/parser_microflow_actions.go` + `writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go` + `microflow_write.go`, `mdl/executor/cmd_microflows_builder_calls.go` + `cmd_microflows_format_action.go`, rule in `validate_microflow_rest.go` (MDL064) | Add `returns Module.Entity` as the LAST alternative so the keyword forms still win, and read Mendix's own `ResultHandlingType` discriminator rather than inferring from `VariableType` (keep the inference as a fallback — the property is `omitempty`). **Delete the silent `String` fallbacks**: an unreconstructable result handling must describe as something the parser REJECTS, or a reader gap becomes a model rewrite (ADR-0005). Mendix constraints worth knowing before designing syntax here: the BASE `System.FileDocument` is rejected as a return type (**CE0362**) so a specialization is mandatory, and HttpResponse **cannot** be specialized (**CE1540** allows only User, FileDocument, Image, Paging) — which is why `returns response` needed no change and the reporter's suggested "HttpResponse specialization" does not exist. Cross-engine round trip (`gateEngines`) is what proves it: the two readers failed differently and each looked fine from inside itself. Issue #922 | diff --git a/.claude/skills/mendix/write-microflows.md b/.claude/skills/mendix/write-microflows.md index 83f95f83b..039b0b809 100644 --- a/.claude/skills/mendix/write-microflows.md +++ b/.claude/skills/mendix/write-microflows.md @@ -1411,6 +1411,27 @@ rest call delete 'https://api.example.com/items/{1}' with ( - `returns response` — returns `System.HttpResponse` object - `returns mapping Module.ImportMapping as Module.Entity` — single object result - `returns mapping Module.ImportMapping as list of Module.Entity` — list result +- `returns Module.MyFile` — store the body in a **file document** + +**The file document form takes a specialization, never `System.FileDocument` +itself.** Mendix rejects the base type as a return type with `CE0362`, and +MDL064 reports it before the write. Create one first: + +```mdl +create persistent entity MyModule.MyFile extends System.FileDocument (); + +create microflow MyModule.ACT_Download ($Location: String) +begin + $file = rest call get '{1}' with ({1} = $Location) + header 'Accept' = 'application/octet-stream' + timeout 300 + returns MyModule.MyFile; +end; +``` + +There is **no** equivalent for an HttpResponse specialization: Mendix allows only +`User`, `FileDocument`, `Image` and `Paging` to be specialized (`CE1540`), so +`returns response` already names the only type that result can have. **Pick `as` vs `as list of` based on the call site, not the mapping shape.** The same import mapping can yield either a single object or a list — Studio Pro stores the cardinality on the microflow's `ImportMappingCall` (`Range.SingleObject` + `ForceSingleOccurrence`). Use `as Module.Entity` when the response is a single object (the mapping may still be list-typed; Studio Pro binds the first item). Use `as list of Module.Entity` when the response should bind a list. Mismatching the cardinality with the surrounding code produces `mx check` `CE0117` at the End event or `CE0013` / `CE0100` on downstream loop / aggregate / list-operation activities. diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index 12783c706..9dad67f93 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -194,7 +194,43 @@ func init() { }, Syntax: "SHOW REST CLIENTS [IN Module];\nSHOW PUBLISHED REST SERVICES [IN Module];\nDESCRIBE REST CLIENT Module.Name;\nDESCRIBE PUBLISHED REST SERVICE Module.Name;", Example: "SHOW REST CLIENTS;\nDESCRIBE REST CLIENT MyModule.PetStoreAPI;\nSHOW PUBLISHED REST SERVICES IN MyModule;", - SeeAlso: []string{"rest.consumed", "rest.published", "integration"}, + SeeAlso: []string{"rest.call", "rest.consumed", "rest.published", "integration"}, + }) + + Register(SyntaxFeature{ + Path: "rest.call", + Summary: "REST CALL activity inside a microflow, and its five RETURNS forms", + Keywords: []string{ + "rest call", "call rest service", "http get", "http post", + "returns response", "returns string", "returns mapping", + "file document", "filedocument", "download", "httpresponse", + }, + Syntax: "[$Var =] REST CALL GET|POST|PUT|PATCH|DELETE '' [WITH ({1} = expr, ...)]\n" + + " [HEADER 'Name' = expr]\n" + + " [AUTH BASIC $user PASSWORD $pass]\n" + + " [BODY ...]\n" + + " [TIMEOUT expr]\n" + + " RETURNS ;\n\n" + + "RETURNS String -- the response body as a string\n" + + "RETURNS response -- the whole System.HttpResponse object\n" + + "RETURNS Module.MyFile -- store the body in a file document\n" + + "RETURNS MAPPING Module.IMM AS Module.E -- apply an import mapping (single object)\n" + + "RETURNS MAPPING Module.IMM AS LIST OF Module.E\n" + + "RETURNS NONE | NOTHING -- ignore the response\n\n" + + "-- The file document form takes a SPECIALIZATION of System.FileDocument.\n" + + "-- Mendix rejects the base type as a return type (CE0362), and MDL064\n" + + "-- reports that before the write. There is no matching form for an\n" + + "-- HttpResponse specialization because Mendix does not allow one\n" + + "-- (CE1540) — `RETURNS response` already names the only type it can be.", + Example: "create persistent entity MyModule.MyFile extends System.FileDocument ();\n\n" + + "create microflow MyModule.ACT_Download ($Location: String)\n" + + "begin\n" + + " $file = rest call get '{1}' with ({1} = $Location)\n" + + " header 'Accept' = 'application/octet-stream'\n" + + " timeout 300\n" + + " returns MyModule.MyFile;\n" + + "end;", + SeeAlso: []string{"rest", "rest.consumed", "microflow"}, }) Register(SyntaxFeature{ diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index b1c2fcb19..b45142500 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -462,7 +462,8 @@ it is for pages. | Call web service | `$Result = call web service Module.Service operation OperationName;` | Legacy SOAP; quoted refs are fallback for dangling raw IDs | | Call web service raw | `$Result = call web service raw 'base64-bson';` | Escape hatch for byte-for-byte legacy SOAP round-trip | | REST call (string) | `$Var = rest call get '' returns string;` | Body as string | -| REST call (response) | `$Var = rest call get '' returns response;` | `System.HttpResponse` object | +| REST call (response) | `$Var = rest call get '' returns response;` | `System.HttpResponse` object. There is no specialization form — Mendix does not allow HttpResponse to be specialized (CE1540) | +| REST call (file document) | `$Var = rest call get '' returns Module.MyFile;` | Stores the body in a file document. Must be a **specialization** of `System.FileDocument` — the base type is rejected as a return type (CE0362 / MDL064) | | REST call (mapping single) | `$Var = rest call get '' returns mapping Module.IMM as Module.Entity;` | Single object — Studio Pro emits `ForceSingleOccurrence=true` | | REST call (mapping list) | `$Var = rest call get '' returns mapping Module.IMM as list of Module.Entity;` | List result | | REST call (none) | `rest call get '' returns nothing;` | Discard response | diff --git a/mdl-examples/bug-tests/922-rest-call-base-filedocument.fail.mdl b/mdl-examples/bug-tests/922-rest-call-base-filedocument.fail.mdl new file mode 100644 index 000000000..014e35a4a --- /dev/null +++ b/mdl-examples/bug-tests/922-rest-call-base-filedocument.fail.mdl @@ -0,0 +1,41 @@ +-- ============================================================================ +-- Bug #922 companion: the two REST file-document result types MDL064 refuses +-- ============================================================================ +-- +-- Both are rejected before `exec` writes anything. +-- +-- 1. The BASE System.FileDocument. Mendix does not allow it as a return type: +-- measured on mxbuild 11.6.6 as +-- CE0362 "System entity 'System.FileDocument' is not allowed as a return +-- type." at Call REST service activity +-- A specialization is required — see 922-rest-call-returns-file-document.mdl. +-- +-- 2. An unqualified entity name, which is indistinguishable from a typo and is +-- MDL008 everywhere else in the language. +-- +-- Usage (this file is expected to FAIL — .fail.mdl): +-- mxcli check mdl-examples/bug-tests/922-rest-call-base-filedocument.fail.mdl +-- Expect two MDL064 errors. +-- ============================================================================ + +create module BugTest922Fail; + +-- (1) the base type +create microflow BugTest922Fail.ACT_BaseType ( + $Location: String +) +begin + $file = rest call get '{1}' with ({1} = $Location) + returns System.FileDocument; +end; +/ + +-- (2) no module prefix +create microflow BugTest922Fail.ACT_Unqualified ( + $Location: String +) +begin + $file = rest call get '{1}' with ({1} = $Location) + returns MyFile; +end; +/ diff --git a/mdl-examples/bug-tests/922-rest-call-returns-file-document.mdl b/mdl-examples/bug-tests/922-rest-call-returns-file-document.mdl new file mode 100644 index 000000000..dffbc06b2 --- /dev/null +++ b/mdl-examples/bug-tests/922-rest-call-returns-file-document.mdl @@ -0,0 +1,71 @@ +-- ============================================================================ +-- Bug #922: REST call storing its response in a file document +-- ============================================================================ +-- +-- Symptom (before fix): a "Call REST service" activity configured to store the +-- response in a file document was described as `returns String`, and the +-- describe → exec round trip then WROTE that back — retyping the activity +-- from FileDocument to String. Measured on mxbuild 11.6.6: +-- +-- stored before: ResultHandlingType = FileDocument +-- VariableType = ObjectType(MyModule.MyFile) +-- stored after: ResultHandlingType = String +-- VariableType = StringType +-- mx check: unchanged — only the project's pre-existing baseline error +-- +-- So the corruption was silent: valid model, green build, wrong behaviour at +-- runtime. On the legacy engine the output variable was lost as well, because +-- the unread result handling took the `$var =` fallback down with it. +-- +-- Root cause: MDL had no syntax for it, the semantic model had no FileDocument +-- variant, and both readers fell through to String — the legacy parser had no +-- `FileDocument` case at all, and the modelsdk reader read the entity out of +-- VariableType and then discarded it, keeping only a literal match on +-- System.HttpResponse. +-- +-- After fix: `returns Module.Entity` is the file document form, round-trips on +-- both engines, and an unreadable result handling now renders as text the +-- parser REJECTS instead of a plausible `returns String`. +-- +-- Note on the other half of the report: `returns response` was NOT a defect. It +-- round-trips losslessly, and Mendix does not allow HttpResponse to be +-- specialized (CE1540 lists only User, FileDocument, Image and Paging), so +-- `response` already names the only type such a result can have. +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/922-rest-call-returns-file-document.mdl -p app.mpr +-- mxcli -p app.mpr -c "describe microflow BugTest922.ACT_DownloadFile" +-- -- the describe must say `returns BugTest922.MyFile`, never `returns String` +-- mx check app.mpr # no new errors +-- ============================================================================ + +create module BugTest922; + +-- Mendix rejects the BASE System.FileDocument as a REST return type (CE0362), +-- so a specialization is mandatory. FileDocument is one of the four System +-- entities that may be specialized (CE1540: User, FileDocument, Image, Paging). +create or modify persistent entity BugTest922.MyFile extends System.FileDocument (); +/ + +create or modify microflow BugTest922.ACT_DownloadFile ( + $Location: String +) +begin + $fileResponseGet = rest call get '{1}' with ({1} = $Location) + header 'Accept' = 'application/octet-stream' + timeout 300 + returns BugTest922.MyFile; +end; +/ + +-- Control: the HttpResponse form, unchanged by this fix and verified to +-- round-trip losslessly both before and after it. +create or modify microflow BugTest922.ACT_FetchResponse ( + $Location: String +) +begin + $httpResponseGet = rest call get '{1}' with ({1} = $Location) + timeout 300 + returns response; +end; +/ diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index a6aa61924..ff9a890d6 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -823,13 +823,17 @@ const ( RestResultResponse // Return HttpResponse object RestResultMapping // Use import mapping RestResultNone // Ignore response + // RestResultFileDocument stores the response in a file document. Mendix + // requires a SPECIALIZATION here: `System.FileDocument` itself is rejected + // as a return type with CE0362, so ResultEntity always names a subclass. + RestResultFileDocument ) // RestResult represents the response handling configuration. type RestResult struct { Type RestResultType // Result type MappingName QualifiedName // Import mapping name (for Mapping type) - ResultEntity QualifiedName // Result entity type (for Mapping type) + ResultEntity QualifiedName // Result entity type (for Mapping and FileDocument types) // IsList distinguishes `as Module.Entity` (single object) from // `as list of Module.Entity` (list). Studio Pro stores this on the // microflow's ImportMappingCall (Range.SingleObject / diff --git a/mdl/backend/modelsdk/microflow_mapping_test.go b/mdl/backend/modelsdk/microflow_mapping_test.go index 18076fb9f..15bd9234c 100644 --- a/mdl/backend/modelsdk/microflow_mapping_test.go +++ b/mdl/backend/modelsdk/microflow_mapping_test.go @@ -108,7 +108,9 @@ func TestRestResultHandling_ObjectTypeIsSingle(t *testing.T) { {Key: "Entity", Value: "Sprintr.UserProfileResponse"}, }}, }) - h, ok := restResultHandlingFromRaw(doc).(*microflows.ResultHandlingMapping) + // "Mapping" is the discriminator Mendix stores for this shape; the + // ImportMappingCall branch is taken before it is consulted either way. + h, ok := restResultHandlingFromRaw(doc, "Mapping").(*microflows.ResultHandlingMapping) if !ok { t.Fatalf("restResultHandlingFromRaw → not a mapping handling") } diff --git a/mdl/backend/modelsdk/microflow_read_actions.go b/mdl/backend/modelsdk/microflow_read_actions.go index aaac40c73..818da2717 100644 --- a/mdl/backend/modelsdk/microflow_read_actions.go +++ b/mdl/backend/modelsdk/microflow_read_actions.go @@ -328,7 +328,7 @@ func actionFromGen(el element.Element) microflows.MicroflowAction { out.RequestHandling = restRequestHandlingFromRaw(rh) } if rh, ok := raw.Lookup("ResultHandling").DocumentOK(); ok { - out.ResultHandling = restResultHandlingFromRaw(rh) + out.ResultHandling = restResultHandlingFromRaw(rh, rawStr(raw, "ResultHandlingType")) } return out @@ -788,10 +788,18 @@ func restRequestHandlingFromRaw(doc bson.Raw) microflows.RequestHandling { } // restResultHandlingFromRaw reconstructs a REST call's result handling. A Mapping -// result carries an ImportMappingCall; the other variants discriminate on the -// VariableType ($Type Void → Nothing, ObjectType System.HttpResponse → response, -// else String). Inverse of restResultHandlingToGen. -func restResultHandlingFromRaw(doc bson.Raw) microflows.ResultHandling { +// result carries an ImportMappingCall; the rest are told apart by Mendix's own +// ResultHandlingType discriminator, falling back to the VariableType when that +// property is absent (it is omitempty, so a Studio Pro document need not carry +// it). Inverse of restResultHandlingToGen. +// +// The fallback must distinguish FileDocument from HttpResponse by entity name, +// because both are stored as DataTypes$ObjectType. Reading it as "anything that +// is not literally System.HttpResponse is a String" was issue #922: a REST call +// storing into a file document described as `returns String`, and a describe → +// exec round trip rewrote the stored type from FileDocument to String — with +// mxbuild still reporting zero errors, so nothing downstream noticed. +func restResultHandlingFromRaw(doc bson.Raw, handlingType string) microflows.ResultHandling { id := model.ID(rawStr(doc, "$ID")) resultVar := rawStr(doc, "ResultVariableName") if imc, ok := doc.Lookup("ImportMappingCall").DocumentOK(); ok { @@ -811,11 +819,17 @@ func restResultHandlingFromRaw(doc bson.Raw) microflows.ResultHandling { entity = rawStr(vt, "Entity") } switch { - case vtType == "DataTypes$VoidType": + case handlingType == "FileDocument" || + (handlingType == "" && vtType == "DataTypes$ObjectType" && entity != "" && entity != "System.HttpResponse"): + h := µflows.ResultHandlingFileDocument{VariableName: resultVar, EntityRef: entity} + h.ID = id + return h + case handlingType == "None" || vtType == "DataTypes$VoidType": h := µflows.ResultHandlingNone{} h.ID = id return h - case vtType == "DataTypes$ObjectType" && entity == "System.HttpResponse": + case handlingType == "HttpResponse" || + (handlingType == "" && vtType == "DataTypes$ObjectType" && entity == "System.HttpResponse"): h := µflows.ResultHandlingHttpResponse{VariableName: resultVar} h.ID = id return h diff --git a/mdl/backend/modelsdk/microflow_write.go b/mdl/backend/modelsdk/microflow_write.go index 611fcc443..88a0686ab 100644 --- a/mdl/backend/modelsdk/microflow_write.go +++ b/mdl/backend/modelsdk/microflow_write.go @@ -1473,6 +1473,8 @@ func restCallActionToGen(a *microflows.RestCallAction) element.Element { resultHandlingType = "HttpResponse" case *microflows.ResultHandlingMapping: resultHandlingType = "Mapping" + case *microflows.ResultHandlingFileDocument: + resultHandlingType = "FileDocument" case *microflows.ResultHandlingNone: resultHandlingType = "None" } @@ -1572,6 +1574,14 @@ func restResultHandlingToGen(rh microflows.ResultHandling, outputVar string) ele addStr(vt, "Entity", "System.HttpResponse") addPart(e, "VariableType", vt) return e + case *microflows.ResultHandlingFileDocument: + e := newElem("Microflows$ResultHandling", string(h.ID)) + addBool(e, "Bind", outputVar != "") + addStr(e, "ResultVariableName", outputVar) + vt := newElem("DataTypes$ObjectType", "") + addStr(vt, "Entity", h.EntityRef) + addPart(e, "VariableType", vt) + return e case *microflows.ResultHandlingNone: e := newElem("Microflows$ResultHandling", string(h.ID)) addBool(e, "Bind", false) diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index 8bbd5b29e..253cfbe8c 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -1208,6 +1208,16 @@ func (fb *flowBuilder) addRestCallAction(s *ast.RestCallStmt) model.ID { BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, VariableName: s.OutputVariable, } + case ast.RestResultFileDocument: + // `returns Module.Entity` — store the response in a file document. The + // entity is always a System.FileDocument specialization; the base type + // is rejected by Mendix as a return type (CE0362), which checkRestCall + // reports before the write. Issue #922. + resultHandling = µflows.ResultHandlingFileDocument{ + BaseElement: model.BaseElement{ID: model.ID(types.GenerateID())}, + VariableName: s.OutputVariable, + EntityRef: s.Result.ResultEntity.String(), + } case ast.RestResultMapping: mappingQN := s.Result.MappingName.Module + "." + s.Result.MappingName.Name entityQN := s.Result.ResultEntity.Module + "." + s.Result.ResultEntity.Name diff --git a/mdl/executor/cmd_microflows_format_action.go b/mdl/executor/cmd_microflows_format_action.go index 4ed5f68d0..4328820fe 100644 --- a/mdl/executor/cmd_microflows_format_action.go +++ b/mdl/executor/cmd_microflows_format_action.go @@ -1242,6 +1242,19 @@ func extractFieldName(attribute, association string) string { return parts[len(parts)-1] } +// unsupportedRestResult renders a result handling mxcli cannot express as MDL. +// +// It deliberately produces something the parser REJECTS. A describe whose output +// silently means something else is worse than one that fails: the previous +// behaviour rendered any unrecognised handling as `returns String`, so a +// describe → exec round trip rewrote a FileDocument result into a String one and +// mxbuild still reported zero errors (#922). Anything that reaches this is a gap +// in the reader, and the text names it so the report says which. +func unsupportedRestResult(what string) string { + return fmt.Sprintf("<>", what) +} + // formatRestCallAction formats a REST call action as MDL. func formatRestCallAction(ctx *ExecContext, a *microflows.RestCallAction) string { var sb strings.Builder @@ -1256,6 +1269,8 @@ func formatRestCallAction(ctx *ExecContext, a *microflows.RestCallAction) string outputVar = rh.VariableName case *microflows.ResultHandlingMapping: outputVar = rh.ResultVariable + case *microflows.ResultHandlingFileDocument: + outputVar = rh.VariableName } } if outputVar != "" { @@ -1384,13 +1399,25 @@ func formatRestCallAction(ctx *ExecContext, a *microflows.RestCallAction) string } sb.WriteString(string(rh.ResultEntityID)) } + case *microflows.ResultHandlingFileDocument: + // A file document result is always bound to a System.FileDocument + // specialization, so the entity is the whole clause. Rendering this + // as "String" was #922: the description round-tripped into a model + // whose activity returned a string instead of a file, and mxbuild + // reported nothing. + sb.WriteString(rh.EntityRef) case *microflows.ResultHandlingNone: sb.WriteString("Nothing") default: - sb.WriteString("String") + // Refuse rather than guess. The previous "String" fallback here and + // below is what turned an unread result handling into a silent + // retyping of the activity (#922); an unknown one is a gap in the + // reader, and saying so is the only outcome that cannot corrupt a + // round trip. + sb.WriteString(unsupportedRestResult(fmt.Sprintf("%T", rh))) } } else { - sb.WriteString("String") + sb.WriteString(unsupportedRestResult("no result handling")) } // Note: Error handling suffix is added at the activity level, not here diff --git a/mdl/executor/rest_filedocument_result_test.go b/mdl/executor/rest_filedocument_result_test.go new file mode 100644 index 000000000..cf1cc90d3 --- /dev/null +++ b/mdl/executor/rest_filedocument_result_test.go @@ -0,0 +1,203 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// upstream #922. A REST call storing its response in a file document had no MDL +// form at all: the grammar offered only String / response / mapping / none, the +// model had no FileDocument variant, and BOTH readers fell back to "String". +// +// Measured on mxbuild 11.6.6 before the fix, on a project whose baseline is one +// error: a describe → exec round trip rewrote the stored ResultHandlingType from +// FileDocument to String and the VariableType from ObjectType(MyModule.MyFile) +// to StringType — and mx check still reported only the baseline, so nothing +// downstream could notice the activity had been retyped. + +func parseRestResult(t *testing.T, returns string) ast.RestResult { + t.Helper() + src := `create microflow Synthetic.MF_Rest (Location: String) +begin + $out = rest call get '{1}' with ({1} = $Location) + timeout 300 + returns ` + returns + `; +end;` + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse error for `returns %s`: %v", returns, errs[0]) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + for _, s := range mf.Body { + if rc, ok := s.(*ast.RestCallStmt); ok { + return rc.Result + } + } + t.Fatalf("no rest call statement parsed from `returns %s`", returns) + return ast.RestResult{} +} + +func TestRestReturnsFileDocumentParses(t *testing.T) { + got := parseRestResult(t, "MyModule.MyFile") + if got.Type != ast.RestResultFileDocument { + t.Fatalf("Type = %v, want RestResultFileDocument", got.Type) + } + if got.ResultEntity.String() != "MyModule.MyFile" { + t.Errorf("ResultEntity = %q, want MyModule.MyFile", got.ResultEntity.String()) + } +} + +// The new alternative sits last in restCallReturnsClause, so it must not shadow +// any keyword form. Each of these is its own token, but a grammar change that +// reordered the alternatives would silently reclassify them — and `String` +// becoming a "file document named String" is exactly the class of bug #922 was. +func TestRestReturnsKeywordFormsStillWin(t *testing.T) { + cases := map[string]ast.RestResultType{ + "String": ast.RestResultString, + "response": ast.RestResultResponse, + "none": ast.RestResultNone, + "nothing": ast.RestResultNone, + } + for returns, want := range cases { + t.Run(returns, func(t *testing.T) { + if got := parseRestResult(t, returns).Type; got != want { + t.Errorf("`returns %s` parsed as %v, want %v", returns, got, want) + } + }) + } +} + +// The mapping form takes qualified names too, so it is the alternative most at +// risk from the new one. +func TestRestReturnsMappingUnaffected(t *testing.T) { + got := parseRestResult(t, "mapping MyModule.IMM_X as list of MyModule.Item") + if got.Type != ast.RestResultMapping { + t.Fatalf("Type = %v, want RestResultMapping", got.Type) + } + if !got.IsList { + t.Error("IsList = false, want true for `as list of`") + } + if got.MappingName.String() != "MyModule.IMM_X" || got.ResultEntity.String() != "MyModule.Item" { + t.Errorf("mapping=%q entity=%q, want MyModule.IMM_X / MyModule.Item", + got.MappingName.String(), got.ResultEntity.String()) + } +} + +// The describer is the half the issue was filed about: it must render the entity +// rather than the word String. +func TestFormatRestCallRendersFileDocumentEntity(t *testing.T) { + a := µflows.RestCallAction{ + OutputVariable: "fileResponseGet", + HttpConfiguration: µflows.HttpConfiguration{ + HttpMethod: microflows.HttpMethodGet, + LocationTemplate: "{1}", + LocationParams: []string{"$Location"}, + }, + ResultHandling: µflows.ResultHandlingFileDocument{ + VariableName: "fileResponseGet", + EntityRef: "MyModule.MyFile", + }, + } + out := formatRestCallAction(nil, a) + if !strings.Contains(out, "returns MyModule.MyFile") { + t.Errorf("describe output does not name the file document type:\n%s", out) + } + if strings.Contains(out, "returns String") { + t.Errorf("describe output still claims String — this is the #922 symptom:\n%s", out) + } + if !strings.Contains(out, "$fileResponseGet = ") { + t.Errorf("describe output lost the output variable, which the legacy reader also used to drop:\n%s", out) + } +} + +// A result handling the reader could not reconstruct must produce something the +// parser REJECTS, not a plausible-looking `returns String`. Rendering an unknown +// handling as String is what let a FileDocument result round-trip into a String +// one with a green build; a describe that fails is recoverable, a describe that +// lies is not. +func TestFormatRestCallRefusesUnknownResultHandling(t *testing.T) { + for name, a := range map[string]*microflows.RestCallAction{ + "nil handling": { + OutputVariable: "x", + HttpConfiguration: µflows.HttpConfiguration{HttpMethod: microflows.HttpMethodGet}, + }, + } { + t.Run(name, func(t *testing.T) { + out := formatRestCallAction(nil, a) + if strings.Contains(out, "returns String;") { + t.Fatalf("an unreadable result handling still renders as a valid `returns String`:\n%s", out) + } + if !strings.Contains(out, "unsupported result handling") { + t.Errorf("output does not say what went wrong:\n%s", out) + } + // It must not parse: that is the whole point. + if _, errs := visitor.Build("create microflow M.F () begin " + out + " end;"); len(errs) == 0 { + t.Errorf("the refusal text parses as valid MDL, so it can still round-trip:\n%s", out) + } + }) + } +} + +// MDL064: the base type is rejected by Mendix itself (CE0362), and an +// unqualified name is a typo waiting to happen. +func TestMDL064_FileDocumentResultType(t *testing.T) { + cases := []struct { + name string + returns string + wantMsg string + }{ + {"base System.FileDocument is refused", "System.FileDocument", "CE0362"}, + {"unqualified name is refused", "MyFile", "module prefix"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + src := `create microflow Synthetic.MF_Rest (Location: String) +begin + $out = rest call get '{1}' with ({1} = $Location) + returns ` + tc.returns + `; +end;` + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse error: %v", errs[0]) + } + vs := ValidateMicroflow(prog.Statements[0].(*ast.CreateMicroflowStmt)) + var found bool + for _, v := range vs { + if v.RuleID == "MDL064" { + found = true + if !strings.Contains(v.Message, tc.wantMsg) { + t.Errorf("MDL064 message = %q, want it to mention %q", v.Message, tc.wantMsg) + } + } + } + if !found { + t.Fatalf("expected MDL064 for `returns %s`, got %#v", tc.returns, vs) + } + }) + } +} + +// The control: a real specialization must pass, or the rule makes the feature +// unusable. Verified against mxbuild — this exact shape builds at baseline. +func TestMDL064_SpecializationIsClean(t *testing.T) { + src := `create microflow Synthetic.MF_Rest (Location: String) +begin + $out = rest call get '{1}' with ({1} = $Location) + returns MyModule.MyFile; +end;` + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse error: %v", errs[0]) + } + for _, v := range ValidateMicroflow(prog.Statements[0].(*ast.CreateMicroflowStmt)) { + if v.RuleID == "MDL064" { + t.Errorf("MDL064 fired on a valid FileDocument specialization: %+v", v) + } + } +} diff --git a/mdl/executor/roundtrip_rest_filedocument_test.go b/mdl/executor/roundtrip_rest_filedocument_test.go new file mode 100644 index 000000000..e6fcf3297 --- /dev/null +++ b/mdl/executor/roundtrip_rest_filedocument_test.go @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package executor + +import ( + "testing" +) + +// upstream #922. A REST call storing its response in a file document round-trips +// on BOTH engines. +// +// Running it on both is the point. The two readers failed differently and would +// each have looked fine from inside itself: the legacy parser had no +// FileDocument case at all and returned nil handling (so the describer printed +// `returns String` AND dropped the output variable), while the modelsdk reader +// read the entity out of VariableType and then threw it away, keeping only a +// literal match on System.HttpResponse and treating everything else as String. +// +// Measured on mxbuild 11.6.6 before the fix: describe → exec rewrote the stored +// ResultHandlingType from FileDocument to String and the VariableType from +// ObjectType(MyFile) to StringType, and mx check still reported only the +// project's pre-existing baseline error — the corruption had no downstream +// signal at all. +func TestRoundtripRestCall_FileDocumentResult(t *testing.T) { + for _, eng := range gateEngines { + t.Run(eng.name, func(t *testing.T) { testRoundtripRestFileDocument(t, eng) }) + } +} + +func testRoundtripRestFileDocument(t *testing.T, eng gateEngine) { + env := setupTestEnvWithBackend(t, eng.factory) + defer env.teardown() + + // Mendix rejects the base System.FileDocument as a return type (CE0362), so + // the result must be a specialization — CE1540 permits FileDocument to be + // specialized, which is what makes this feature expressible at all. + entityMDL := `create or modify persistent entity ` + testModule + `.MyFile extends System.FileDocument ();` + if err := env.executeMDL(entityMDL); err != nil { + t.Fatalf("failed to create the file document specialization: %v", err) + } + + createMDL := `create or modify microflow ` + testModule + `.MF_FetchFile (Location: String) +begin + $fileResponseGet = rest call get '{1}' with ({1} = $Location) + timeout 300 + returns ` + testModule + `.MyFile; +end;` + + env.assertContains(createMDL, []string{ + // The entity, not "String" — the reported symptom. + "returns " + testModule + ".MyFile", + // The output variable, which the legacy engine also used to lose because + // its nil result handling took the variable fallback with it. + "$fileResponseGet = ", + }) +} + +// The neighbouring forms must keep round-tripping: `returns response` was +// reported alongside the file document case but is CORRECT as it stands. +// HttpResponse cannot be specialized (CE1540 permits only User, FileDocument, +// Image and Paging), so `response` already names the only type such a result can +// have and there is nothing for the describer to add. +func TestRoundtripRestCall_ResponseResultUnchanged(t *testing.T) { + for _, eng := range gateEngines { + t.Run(eng.name, func(t *testing.T) { + env := setupTestEnvWithBackend(t, eng.factory) + defer env.teardown() + + createMDL := `create or modify microflow ` + testModule + `.MF_FetchResponse (Location: String) +begin + $httpResponseGet = rest call get '{1}' with ({1} = $Location) + timeout 300 + returns response; +end;` + + env.assertContains(createMDL, []string{ + "returns response", + "$httpResponseGet = ", + }) + }) + } +} diff --git a/mdl/executor/validate_microflow.go b/mdl/executor/validate_microflow.go index c757de93a..d124d842e 100644 --- a/mdl/executor/validate_microflow.go +++ b/mdl/executor/validate_microflow.go @@ -297,6 +297,9 @@ func (v *microflowValidator) walkBody(body []ast.MicroflowStatement) { v.checkAssociationObjectArgs("microflow "+stmt.MicroflowName.String(), stmt.Arguments) case *ast.CallNanoflowStmt: v.checkAssociationObjectArgs("nanoflow "+stmt.NanoflowName.String(), stmt.Arguments) + case *ast.RestCallStmt: + // #922: `returns Module.Entity` must name a FileDocument specialization. + v.checkRestFileDocumentResult(stmt) case *ast.LoopStmt: // Check: @caption on a loop is silently dropped — Mendix for-loops // have no caption (Microflows$LoopedActivity has no Caption diff --git a/mdl/executor/validate_microflow_rest.go b/mdl/executor/validate_microflow_rest.go new file mode 100644 index 000000000..89fe402ed --- /dev/null +++ b/mdl/executor/validate_microflow_rest.go @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// systemFileDocument is the base type a REST file-document result must specialize. +const systemFileDocument = "System.FileDocument" + +// checkRestFileDocumentResult validates `rest call … returns Module.Entity` — MDL064. +// +// Two things are checked, both measured against mxbuild 11.6.6: +// +// 1. The base type is rejected by Mendix itself: storing a response into +// System.FileDocument builds as CE0362 "System entity 'System.FileDocument' +// is not allowed as a return type." A specialization is mandatory, and one +// is always available because CE1540 lists FileDocument among the four +// System entities that MAY be specialized (User, FileDocument, Image, +// Paging) — which is also why there is no equivalent form for an +// HttpResponse result: HttpResponse cannot be specialized at all, so +// `returns response` already names the only type it can have. +// +// 2. The name must be qualified. `returns MyFile` is indistinguishable from a +// typo, and an unqualified entity reference is MDL008 everywhere else. +// +// Whether the named entity really specializes FileDocument needs the project, +// so it is left to `check --references`; this rule is the part that works +// without one. +func (v *microflowValidator) checkRestFileDocumentResult(stmt *ast.RestCallStmt) { + if stmt.Result.Type != ast.RestResultFileDocument { + return + } + entity := stmt.Result.ResultEntity + + if entity.Module == "" { + v.addViolation("MDL064", linter.SeverityError, + fmt.Sprintf("rest call 'returns %s': the file document type needs a module prefix", + entity.Name), + fmt.Sprintf("Write the qualified name, e.g. 'MyModule.%s'", entity.Name)) + return + } + + if entity.String() == systemFileDocument { + v.addViolation("MDL064", linter.SeverityError, + "rest call 'returns System.FileDocument': Mendix does not allow the base "+ + "System.FileDocument as a REST result type — mxbuild rejects it with CE0362 "+ + "\"System entity 'System.FileDocument' is not allowed as a return type.\"", + "Create an entity that specializes it (create persistent entity MyModule.MyFile "+ + "extends System.FileDocument) and return that instead") + } +} diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index 1495e7c57..d3a8f9729 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -611,6 +611,7 @@ restCallReturnsClause | RETURNS MAPPING qualifiedName AS qualifiedName // Import mapping → single object | RETURNS NONE // Ignore response | RETURNS NOTHING // Ignore response (alias) + | RETURNS qualifiedName // Store in file document (a System.FileDocument specialization) ; /** diff --git a/mdl/visitor/visitor_microflow_actions.go b/mdl/visitor/visitor_microflow_actions.go index ffd9d6bd6..08547d8d3 100644 --- a/mdl/visitor/visitor_microflow_actions.go +++ b/mdl/visitor/visitor_microflow_actions.go @@ -1529,6 +1529,13 @@ func buildRestCallStatement(ctx parser.IRestCallStatementContext) *ast.RestCallS } } else if returnsCtx.NONE() != nil || returnsCtx.NOTHING() != nil { result.Type = ast.RestResultNone + } else if qns := returnsCtx.AllQualifiedName(); len(qns) == 1 { + // `returns Module.Entity` — store the response in a file document. + // Reached only after the keyword alternatives, so a bare entity name + // can never shadow `String` / `response` / `none` / `nothing`, each + // of which is its own token. + result.Type = ast.RestResultFileDocument + result.ResultEntity = buildQualifiedName(qns[0]) } stmt.Result = result diff --git a/sdk/microflows/microflows_actions.go b/sdk/microflows/microflows_actions.go index cbaec08f7..5b56bf76c 100644 --- a/sdk/microflows/microflows_actions.go +++ b/sdk/microflows/microflows_actions.go @@ -875,6 +875,20 @@ type ResultHandlingHttpResponse struct { func (ResultHandlingHttpResponse) isResultHandling() {} +// ResultHandlingFileDocument stores the response in a file document. +// +// EntityRef is always a SPECIALIZATION of System.FileDocument, never the base: +// Mendix rejects `System.FileDocument` itself as a return type with CE0362, +// while CE1540 permits FileDocument to be specialized (unlike HttpResponse, +// which cannot be — so there is no matching field on ResultHandlingHttpResponse). +type ResultHandlingFileDocument struct { + model.BaseElement + VariableName string `json:"variableName,omitempty"` + EntityRef string `json:"entityRef,omitempty"` // qualified name, e.g. MyModule.MyFile +} + +func (ResultHandlingFileDocument) isResultHandling() {} + // ResultHandlingMapping uses an import mapping. type ResultHandlingMapping struct { model.BaseElement diff --git a/sdk/mpr/parser_microflow_actions.go b/sdk/mpr/parser_microflow_actions.go index c375c6f68..0f1a89ab4 100644 --- a/sdk/mpr/parser_microflow_actions.go +++ b/sdk/mpr/parser_microflow_actions.go @@ -623,6 +623,20 @@ func parseResultHandling(raw map[string]any, handlingType string) microflows.Res result.ID = model.ID(extractBsonID(raw["$ID"])) result.VariableName = extractString(raw["ResultVariableName"]) return result + case "FileDocument": + // The entity lives in VariableType and is always a specialization of + // System.FileDocument — the base is rejected as a return type (CE0362). + // Without this case the whole handling read back as nil, which the + // describer rendered as `returns String` while also losing the output + // variable, so a describe → exec round trip silently retyped the + // activity and still built clean. Issue #922. + result := µflows.ResultHandlingFileDocument{} + result.ID = model.ID(extractBsonID(raw["$ID"])) + result.VariableName = extractString(raw["ResultVariableName"]) + if varType := toMap(raw["VariableType"]); varType != nil { + result.EntityRef = extractString(varType["Entity"]) + } + return result case "Mapping": result := µflows.ResultHandlingMapping{} result.ID = model.ID(extractBsonID(raw["$ID"])) diff --git a/sdk/mpr/writer_microflow_actions.go b/sdk/mpr/writer_microflow_actions.go index 3af311a9e..89dbb5995 100644 --- a/sdk/mpr/writer_microflow_actions.go +++ b/sdk/mpr/writer_microflow_actions.go @@ -714,6 +714,8 @@ func serializeRestCallAction(a *microflows.RestCallAction) bson.D { resultHandlingType = "HttpResponse" case *microflows.ResultHandlingMapping: resultHandlingType = "Mapping" + case *microflows.ResultHandlingFileDocument: + resultHandlingType = "FileDocument" case *microflows.ResultHandlingNone: resultHandlingType = "None" } @@ -1058,6 +1060,23 @@ func serializeRestResultHandling(rh microflows.ResultHandling, outputVar string) }}, } + case *microflows.ResultHandlingFileDocument: + // Same shape as HttpResponse, but the entity is authored rather than + // fixed: it is always a System.FileDocument specialization (CE0362 + // rejects the base). Issue #922. + return bson.D{ + {Key: "$ID", Value: idToBsonBinary(string(h.ID))}, + {Key: "$Type", Value: "Microflows$ResultHandling"}, + {Key: "Bind", Value: outputVar != ""}, + {Key: "ImportMappingCall", Value: nil}, + {Key: "ResultVariableName", Value: outputVar}, + {Key: "VariableType", Value: bson.D{ + {Key: "$ID", Value: idToBsonBinary(GenerateID())}, + {Key: "$Type", Value: "DataTypes$ObjectType"}, + {Key: "Entity", Value: h.EntityRef}, + }}, + } + case *microflows.ResultHandlingNone: return bson.D{ {Key: "$ID", Value: idToBsonBinary(string(h.ID))}, From 35a495170c2657a5543329187cc26507c33ffe4a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 09:25:05 +0000 Subject: [PATCH 02/21] fix(examples): version-gate DecimalScale, which 10.24 does not have The nightly failed on the Mendix 10.24 matrix entry, on both engines, in TestMxCheck_DoctypeScripts: Execution error: this project does not store the model setting DecimalScale 14-project-settings-examples.mdl set it unconditionally. Measured against a blank project of each version: 10.24 stores 11 model settings, 11.6.6 stores 12, and DecimalScale is the only difference. Executing the other five settings from that statement one at a time on 10.24, each is accepted -- so DecimalScale alone is at fault, and because the refusal covers the WHOLE statement it took five portable settings down with it. mxcli's refusal is correct and is not changed here: Studio Pro will not open a model carrying a property its version does not define, and mxbuild does not catch it. The example was simply not version-gated, despite its own comment three lines above explaining that which settings a project stores depends on its version. Also adds TestDoctypeScriptsParseAfterVersionFiltering, which filters every doctype script for each version in the nightly matrix and asserts the result still parses. It needs no mxbuild, so a mis-gated script fails in seconds on push instead of hours later in one nightly job. Writing that guard found the trap that makes this easy to get wrong, and the first attempt at this fix walked straight into it: a `/** */` block is a DOCUMENTATION comment bound to the statement after it. Gating the statement while leaving its doc comment outside the section orphans the comment, and the script dies with "no viable alternative at input '/**...'" -- reported at the NEXT statement, tens of lines further down, so it reads like an unrelated syntax error in code nobody touched. The comment now sits inside the gated section; `--` line comments are free-standing and safe either side. Verified: the full doctype suite (59 scripts, both engines) passes on 10.24, where it failed before; the changed script passes on 10.24, 11.6.6 and 11.13.0; the control -- the pre-fix file -- still reproduces the reported error on 10.24; and the new guard, with the doc comment moved back outside the gate, fails on exactly the 10.24 entry and passes once restored. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- .claude/skills/fix-issue.md | 1 + CHANGELOG.md | 4 + .../14-project-settings-examples.mdl | 29 ++++++- mdl/executor/doctype_version_gating_test.go | 82 +++++++++++++++++++ 4 files changed, 115 insertions(+), 1 deletion(-) create mode 100644 mdl/executor/doctype_version_gating_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index d64563e93..346206e42 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -563,3 +563,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `DESCRIBE IMPORT MAPPING` output does not reproduce the script that made it: `Total = total` comes back as `Total = Total`, an array binding as `= ItemItem`, `LineId = id` as `LineId = _id` — and the output cannot be re-run at all, failing with "import mapping already exists". Export mappings identically (unreported) | DESCRIBE printed the element's **ExposedName** (Mendix's display name — capitalised initial, `Item` suffix on an array's item object) instead of the raw JSON key from `JsonPath`, and emitted a bare `create` header where every other DESCRIBE emits `create or modify` | `mdl/executor/cmd_import_mappings.go` (`mappingMemberName`, the four print sites, the header) + `cmd_export_mappings.go` (same four + header) | Print the raw key derived from `JsonPath` — strip a trailing `\|(Object)` first, because an array's mapping element sits at the ITEM object while the script addressed the array (that suffix is what produced `ItemItem`). Fall back to ExposedName when there is no JsonPath (XML-schema / message-definition mappings have none). Safe by construction: the raw path is `jsonSchemaIndex.resolve`'s FIRST lookup, so it cannot regress #882. **Do NOT "fix" ExposedName itself** — the capitalisation is Mendix's own, confirmed against a Studio Pro-authored document in the blank app (`ExposedName "Uuid"` vs `Path "(Object)|uuid"`); rewriting it would diverge from Studio Pro. The `Item` suffix could NOT be confirmed the same way (a blank app has no Studio Pro array structure) and was left alone — with a separate `ExposedItemName` property in the BSON, that is worth checking against a marketplace module before anyone touches storage. Note the issue's framing was half wrong: the mapping DID round-trip semantically (re-executing the old output rebuilt byte-identical JsonPaths), so this was a text/diff defect, not a broken mapping — measure before agreeing with a title. Tests `TestMappingMemberName`, `TestDescribe{Import,Export}Mapping_RoundTripsMemberNames` (fail with the reported symptoms when reverted); two existing header assertions needed updating with the intentional change; fixture `mdl-examples/bug-tests/915-mapping-describe-roundtrip.mdl` is a DESCRIBE fixed point. Issue #915 | | `mxcli check -p app.mpr` passes a microflow containing `if $obj/Status = 'Open'` — comparing an enumeration attribute to a string literal, the first example in the type-checking proposal and the shape people actually write. The same mistake written as a create or change member *is* caught, which makes the gap look arbitrary | Two causes, both invisible. (a) **`inferKind` returned `KindUnknown` for every `AttributePathExpr`**, so `$obj/Attr` typed to nothing and every rule downstream of it stayed quiet. `exprcheck.Scope` is `Lookup(name) (TypeKind, bool)` — it can say "$P is an Object" but not *which* entity, so there was nowhere to put the answer; the adapter computed a variable→entity map (`buildVarEntityScope`) and used it only to label a slot path, never passing it to the checker, and it covered body-introduced variables but **not parameters** — the ordinary case. (b) Even resolved, nothing fired: **E001 keys off the SLOT** (`CreateItem.Value:Entity.Attr`), which exists for an assignment and not for a comparison | `mdl/exprcheck/interfaces.go` (`EntityScope`, `Context.Entities`), `mdl/exprcheck/parser.go` (`attributePathKind`, `pathTargetEntity`, `checkEnumComparedToString`), `mdl/exprcheck/adapters/` (`walkFlow`, `addParamEntities`, `entityScope`), `mdl/exprcatalog/` (`AssociationTarget`) | Put the object side in its **own seam** (`EntityScope`: `VariableEntity` + `AssociationTarget`) beside `Scope` rather than widening either it or `CatalogReader` — `CatalogReader`'s shape is what stays re-syncable from the upstream fork. **A multi-hop expression path is not an XPath path**: XPath spells the intermediate entity (`[Assoc/Entity/Attr]`) so a walk can read it off, while an expression does not (`$O/Mod.Assoc/Attr`), so every hop must resolve through the association index. For (b), emit the **same code and message** from the comparison as from the slot — one defect should not have two names depending on where it was spotted. **Generalisable**: when a checker is silent, separate "did the input resolve" from "is there a rule for this shape" before touching either — here resolution and detection were both missing, and fixing only one would have looked like the fix failed. Verify with a **probe corpus that actually exercises the construct**: 21 microflows described back to MDL contained 40 attribute-path lines including an association hop, and were clean before and after, which is what makes "no false positives" mean something. Controls: path→`KindUnknown` restored, `Entities` dropped from the Context, and `addParamEntities` removed each fail a distinct test. **Still open**: a *terminal* association step (`$Order/Mod.Order_Lines`) types to unknown — Object vs List depends on the association's kind and direction, and guessing costs false positives | | `describe microflow` renders a REST call's result as `returns String` when the activity actually stores the response in a **file document**, and the describe → exec round trip then writes String back — retyping the activity with `mx check` reporting nothing either way. On the legacy engine the `$var =` output variable disappears too | MDL had no syntax for the case, `sdk/microflows` had no FileDocument variant, and both readers fell through to String: the legacy `parseResultHandling` had no `FileDocument` case (so the whole handling read back nil, taking the output-variable fallback with it), while modelsdk's `restResultHandlingFromRaw` read `VariableType.Entity` and then discarded it, matching only the literal `System.HttpResponse`. Mendix stores all five variants as ONE `Microflows$ResultHandling` carrying a `VariableType`, discriminated by the `ResultHandlingType` enum | `mdl/grammar/domains/MDLMicroflow.g4` (`restCallReturnsClause`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `sdk/microflows/microflows_actions.go`, `sdk/mpr/parser_microflow_actions.go` + `writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go` + `microflow_write.go`, `mdl/executor/cmd_microflows_builder_calls.go` + `cmd_microflows_format_action.go`, rule in `validate_microflow_rest.go` (MDL064) | Add `returns Module.Entity` as the LAST alternative so the keyword forms still win, and read Mendix's own `ResultHandlingType` discriminator rather than inferring from `VariableType` (keep the inference as a fallback — the property is `omitempty`). **Delete the silent `String` fallbacks**: an unreconstructable result handling must describe as something the parser REJECTS, or a reader gap becomes a model rewrite (ADR-0005). Mendix constraints worth knowing before designing syntax here: the BASE `System.FileDocument` is rejected as a return type (**CE0362**) so a specialization is mandatory, and HttpResponse **cannot** be specialized (**CE1540** allows only User, FileDocument, Image, Paging) — which is why `returns response` needed no change and the reporter's suggested "HttpResponse specialization" does not exist. Cross-engine round trip (`gateEngines`) is what proves it: the two readers failed differently and each looked fine from inside itself. Issue #922 | +| The nightly fails on ONE Mendix matrix version only, in `TestMxCheck_DoctypeScripts`, with `Execution error: this project does not store the model setting ` — while the same script passes on every newer version | The example script set a model setting that version does not have. Measured: a blank 10.24 stores 11 model settings and a blank 11.6.6 stores 12, `DecimalScale` being the only difference. mxcli's refusal is CORRECT — Studio Pro will not open a model carrying a property its version does not define, and mxbuild does not catch it — so the bug is the ungated example, not the guard. The refusal covers the WHOLE statement, so one version-specific setting takes every portable setting in the same `alter` down with it | `mdl-examples/doctype-tests/14-project-settings-examples.mdl`; guard in `mdl/executor/doctype_version_gating_test.go` | Split the version-specific setting into its own statement inside a `-- @version: N.N+` section, closed with `-- @version: any`. **Put the `/** */` doc comment INSIDE the gated section**: a block comment is a documentation comment bound to the statement after it, so gating the statement while leaving the comment outside orphans it and the script dies with `no viable alternative at input '/**...'` — reported at the NEXT statement, tens of lines away, which reads like an unrelated syntax error. `--` line comments are free-standing and safe either side. Isolate which setting is at fault by exec'ing them one at a time against a blank project of that version (`mx create-project` in a SHORT path — a long one dies with PathTooLongException). `TestDoctypeScriptsParseAfterVersionFiltering` now parses every doctype script under each nightly matrix version without needing mxbuild, so a mis-gated script fails in seconds on push instead of hours later in one nightly job | diff --git a/CHANGELOG.md b/CHANGELOG.md index 29d8ffd81..a3156db8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- **The Mendix 10.24 nightly is green again** — `14-project-settings-examples.mdl` set `DecimalScale`, which 10.24 does not have, so `TestMxCheck_DoctypeScripts` failed on that matrix entry on both engines while passing on every 11.x. Measured against blank projects: 10.24 stores 11 model settings, 11.6.6 stores 12, and `DecimalScale` is the only difference — each of the other five settings in that statement is accepted on 10.24 on its own. mxcli's refusal was correct (Studio Pro will not open a model carrying a property its version does not define, and mxbuild does not catch it); the example simply was not version-gated, and because the refusal covers the whole statement, one unsupported setting took five portable ones down with it. It is now its own `-- @version: 11.0+` section. + + A new guard, `TestDoctypeScriptsParseAfterVersionFiltering`, filters every doctype script for each nightly matrix version and asserts the result still parses. It needs no mxbuild, so a mis-gated script fails in seconds on push rather than hours later in a single nightly job. Writing it surfaced the trap that makes this easy to get wrong: a `/** */` block is a *documentation* comment bound to the statement after it, so gating the statement while leaving its comment outside the section orphans the comment and the script stops parsing — reported at the *next* statement, tens of lines away, which reads like an unrelated syntax error. + - **A microflow's StartEvent no longer moves on a describe→exec round-trip** — the start has no MDL statement to annotate and `DESCRIBE` cannot emit its position, so the builder always derived one (first annotated activity minus one spacing unit). A Studio-Pro-authored flow whose start sat at `145;200` came back at `100;200` — the only coordinate in it that did not survive. The position is now carried over from the microflow being replaced, the way the folder and allowed module roles already are; a fresh `CREATE` still derives it. ### Added diff --git a/mdl-examples/doctype-tests/14-project-settings-examples.mdl b/mdl-examples/doctype-tests/14-project-settings-examples.mdl index cea4e711d..fb2fc7aca 100644 --- a/mdl-examples/doctype-tests/14-project-settings-examples.mdl +++ b/mdl-examples/doctype-tests/14-project-settings-examples.mdl @@ -82,15 +82,42 @@ alter settings model EnableDataStorageOptimisticLocking = true; * Which of these a project stores depends on its Mendix version: a blank 9.24 * project stores 12 model settings, a blank 11.13 stores 17. An alter naming one * this project does not store is refused rather than introducing it. + * + * That refusal covers the WHOLE statement, so a version-specific setting has to + * be its own alter -- naming it alongside portable ones takes them down with it. + * These five are stored by every version in the test matrix (measured on a blank + * 10.24: all five present, and each accepted on its own). */ alter settings model FirstDayOfWeek = 'Monday', - DecimalScale = 8, UseDatabaseForeignKeyConstraints = true, UseOQLVersion2 = true, SslCertificateAlgorithm = 'PKIX', DefaultTimeZoneCode = 'Europe/Amsterdam'; +-- The @version directive opens the gated section, and everything up to the next +-- directive is removed on a version that does not match. The `/** */` block +-- below therefore sits INSIDE the section, not above it: a doc comment binds to +-- the statement that follows it, so gating the statement while leaving the +-- comment outside orphans it and the script stops parsing ("no viable +-- alternative at input '/**...'"). Line comments like this one are free-standing +-- and safe either side of the directive. +-- @version: 11.0+ +/** + * Example 1.6: A setting older versions do not store + * + * DecimalScale is the one model setting a blank 11.6.6 project stores that a + * blank 10.24 does not -- measured: 11 settings against 12, and the only + * difference. Ungated, this line failed the 10.24 nightly on both engines. + * + * mxcli refused it correctly: Studio Pro will not open a model carrying a + * property its version does not define, and mxbuild does not catch it. The + * refusal covers the whole statement, which is why this is its own alter rather + * than one more line in the block above. + */ +alter settings model DecimalScale = 8; +-- @version: any + -- Note: UseSystemContextForBackgroundTasks is read and preserved but NOT -- writable -- Mendix withdrew it, and mx check on 11.13 rejects a project -- holding true with CE9436 "not supported anymore". diff --git a/mdl/executor/doctype_version_gating_test.go b/mdl/executor/doctype_version_gating_test.go new file mode 100644 index 000000000..1ccbbda18 --- /dev/null +++ b/mdl/executor/doctype_version_gating_test.go @@ -0,0 +1,82 @@ +// SPDX-License-Identifier: Apache-2.0 + +//go:build integration + +package executor + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/mpr/version" +) + +// nightlyMatrix is the Mendix version set .github/workflows/nightly.yml runs the +// doctype scripts against. A script that only parses on the newest of them is a +// nightly failure on the others, reported hours later against whatever landed in +// between — which is how the DecimalScale gating below was found. +var nightlyMatrix = []*version.ProjectVersion{ + {MajorVersion: 10, MinorVersion: 24, ProductVersion: "10.24.24.119349"}, + {MajorVersion: 11, MinorVersion: 6, ProductVersion: "11.6.8"}, + {MajorVersion: 11, MinorVersion: 12, ProductVersion: "11.12.2"}, + {MajorVersion: 11, MinorVersion: 13, ProductVersion: "11.13.0"}, +} + +// TestDoctypeScriptsParseAfterVersionFiltering guards the whole doctype corpus +// against gating that produces MDL the parser rejects. +// +// A `-- @version:` section is removed by blanking its lines, and what is left +// has to be a valid script on its own. The trap this catches is that a `/** */` +// block is a DOCUMENTATION comment bound to the statement after it: gate the +// statement while leaving its doc comment outside the section and the comment is +// orphaned, so the script dies with "no viable alternative at input '/**...'" — +// at the NEXT statement, tens of lines further down, which reads like an +// unrelated syntax error. Line comments (`--`) are free-standing and safe on +// either side of the directive. +// +// This runs without mxbuild, so a mis-gated script fails in seconds on every +// push rather than in the nightly for one matrix entry. +func TestDoctypeScriptsParseAfterVersionFiltering(t *testing.T) { + all, err := filepath.Glob("../../mdl-examples/doctype-tests/*.mdl") + if err != nil { + t.Fatalf("glob doctype scripts: %v", err) + } + // `.test.mdl` / `.tests.mdl` are microflow test SPECS, not MDL scripts: they + // carry @test/@expect annotations and do not parse with this parser even + // unfiltered. TestMxCheck_DoctypeScripts excludes them for the same reason. + var scripts []string + for _, s := range all { + name := filepath.Base(s) + if strings.HasSuffix(name, ".test.mdl") || strings.HasSuffix(name, ".tests.mdl") { + continue + } + scripts = append(scripts, s) + } + if len(scripts) == 0 { + t.Fatal("no doctype scripts found — the glob is wrong, and a passing run here would prove nothing") + } + + for _, script := range scripts { + content, err := os.ReadFile(script) + if err != nil { + t.Errorf("%s: %v", script, err) + continue + } + name := filepath.Base(script) + + for _, pv := range nightlyMatrix { + t.Run(name+"/"+pv.ProductVersion, func(t *testing.T) { + filtered, _ := filterByVersion(string(content), pv) + if _, errs := visitor.Build(filtered); len(errs) > 0 { + t.Errorf("filtered for Mendix %s, the script no longer parses: %v\n"+ + "A gated section must contain everything that belongs to it — including any "+ + "`/** */` doc comment, which binds to the statement that follows it.", + pv.ProductVersion, errs[0]) + } + }) + } + } +} From fec4333dda3d67c17172d903469a577720e1c26c Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 10:07:22 +0000 Subject: [PATCH 03/21] fix(canon): carry stored element $IDs onto a rewritten document MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Studio Pro's version-control view showed an entire nanoflow as changed after editing one argument of one JavaScript action call, and the same one-literal change to a microflow read as a wholesale replacement. Measured on the reporter's project: 36 of 37 element identities re-minted in the nanoflow, 21 of 22 in the microflow. The damage was cumulative — changing a value and changing it back produced a semantically identical document sharing none of the original's element IDs. Elision (ADR-0008) never covered this. It answers "did anything change?" and keeps a no-op write off disk; when something *did* change the document was still rebuilt from the MDL, and every sub-element of a rebuild gets a freshly random $ID. canon.TransplantIDs matches the incoming document against the stored one element by element and puts the stored $ID back on every element that still corresponds. Alignment is by $Type plus the shape one level down (Action=Microflows$LogMessageAction) plus Name where there is one, LCS-anchored within each list with positional fill in the gaps — the deeper key is what tells two otherwise identical ActionActivity wrappers apart when an activity is inserted, without which the newcomer inherited its neighbour's whole subtree. References are rewritten in the same pass, which is the rule PR #125 broke: a pointer is a primitive binary property that a containment walk never sees. The substitution therefore covers every 16-byte binary in the document rather than a maintained list of pointer property names, on canon's own insight that any occurrence of one of the document's element IDs is a reference by definition. It is applied in place on a copy, so a fixed-width binary keeps the framing intact and nothing is re-marshalled. The correctness bar is lower than it looks and the code says so: a wrong match only makes a diff bigger, because every reference moves with its element and nothing outside the unit remembers an $ID. The one real failure is two elements sharing one, guarded by dropCollisions (run to a fixed point) and by reading the result back and checking the id set. GUIDs are untouched — those are the database's identity and were already preserved. Measured, both engines: 37 of 37 identities kept on the reported nanoflow with the BSON diff down to the one changed argument; a change plus its revert returns to the original bytes exactly; inserting or deleting an activity mints IDs only for the genuinely new elements (22 of 22 kept, 6 new). mx check on the reporter's app is unchanged at its one pre-existing CE0117. TestWriteMicroflowTwice_ControlChurnsWhenElisionOff had to change: it proved elision was doing the work by forcing writes with MXCLI_ALWAYS_WRITE and watching the bytes churn, and identity preservation now makes a forced write byte-stable too. The control moves down a layer to the raw codec output (TestRebuildChurnsSubElementIDs), and the forced-write case becomes a positive assertion about the transplant. The shell equivalent is documented as a control on mtimes rather than hashes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- CLAUDE.md | 27 +- docs-site/src/internals/idempotent-writes.md | 42 +- ...0-rewrite-preserves-element-identities.mdl | 70 +++ .../modelsdk/microflow_idempotence_test.go | 109 ++++- modelsdk/canon/identity.go | 7 + modelsdk/canon/transplant.go | 413 ++++++++++++++++++ modelsdk/canon/transplant_test.go | 334 ++++++++++++++ 7 files changed, 987 insertions(+), 15 deletions(-) create mode 100644 mdl-examples/bug-tests/910-rewrite-preserves-element-identities.mdl create mode 100644 modelsdk/canon/transplant.go create mode 100644 modelsdk/canon/transplant_test.go diff --git a/CLAUDE.md b/CLAUDE.md index 970992194..e0fdbe47f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -334,6 +334,16 @@ so comparing bytes would skip nothing. The policy lives in `modelsdk/canon` `modelsdk/mpr/writer_core.go` (`updateUnit` *and* `WriteTransaction.WriteUnit` — `codec.Store` reaches storage through the latter) and `sdk/mpr/writer_units.go`. +When something *has* changed, `Reconcile` still does not let the rebuild's fresh +`$ID`s reach disk: `canon.TransplantIDs` matches the incoming document against the +stored one element by element (by `$Type` and shape, by `Name` where there is one, +LCS-anchored within each list) and puts the **stored** `$ID` back on every element +that still corresponds. Without it a one-argument edit re-minted 36 of a nanoflow's +37 element identities and Studio Pro painted the whole document as changed (#910). +Its correctness bar is lower than it looks and worth knowing: a *wrong* match only +makes a diff bigger, because every reference is rewritten with the element — the +one real failure is two elements sharing an `$ID`, which `dropCollisions` guards. + Three rules follow, and each has already been violated once: 1. **Never rewrite an element `$ID` without rewriting every reference to it in the @@ -341,7 +351,10 @@ Three rules follow, and each has already been violated once: `ChildProperty`, so a containment walk traverses the whole document and never sees one. PR #125 renumbered IDs this way and made projects unopenable (`KeyNotFoundException` at `ResolvePostponedProperties`). A unit is rewritten - wholesale or not at all. + wholesale or not at all. The transplant obeys this by substituting over *every* + 16-byte binary in the document rather than a maintained list of pointer + properties — any occurrence of one of the document's element IDs is a reference + by definition, the same insight the canonical form rests on. 2. **Adding a write path means wiring it to `canon.Reconcile`.** A new choke point that writes directly will silently churn while everything else is quiet — the worst kind of inconsistency, because the diff blames the wrong change. @@ -360,8 +373,14 @@ qualified name breaks that assumption and invalidates the argument in ADR-0008. `MXCLI_ALWAYS_WRITE=1` forces every write to land, for bisecting. It does not disable identity preservation. **Any test asserting "nothing changed" must include -the control run with it set** — otherwise the test passes against a build that -never had the fix, which is exactly how PR #125 shipped green. +a control** — otherwise the test passes against a build that never had the fix, +which is exactly how PR #125 shipped green. Note what the control can now be: +since identities are carried, a forced write of an in-sync unit produces the +**same bytes**, so "flip `MXCLI_ALWAYS_WRITE` and watch the content change" no +longer distinguishes anything (measured: same sha, mtime moves). Control on the +**rebuild** instead — encode the document twice and show the raw codec output +differs (`TestRebuildChurnsSubElementIDs`) — or, from the shell, on **mtimes** +rather than hashes. ### The Tunnel Is Linux-Only, On Purpose — Do Not "Restore" It @@ -750,7 +769,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati **Implemented:** - Default styling + runtime theme switching (`mxcli theme list/show/apply/remove/switcher`, `mxcli new --theme`): three embedded themes (**signal** light-first, **ledger** light-first, **console** dark-first), each a palette in `theme/web/custom-variables.scss` + a shared Atlas wiring partial + a theme partial imported from `theme/web/main.scss` (which compiles last), plus vendored fonts. **No model changes**, so it hot-applies under `run --local --watch` and cannot affect a build. Generated regions are digest-fenced: a block carrying local edits is refused rather than overwritten. Applying a theme removes the previous one. `--variant auto` (default) ships both palettes — the app follows `prefers-color-scheme` before first paint and honours a `theme-light`/`theme-dark` class on ``; `light`/`dark` bakes one. `theme switcher install` is the only part that writes to the model (JS actions + a nanoflow for a toggle button). Package: `cmd/mxcli/theme/`. See `docs/11-proposals/PROPOSAL_default_styling.md` - MPR v1/v2 reading and writing -- Idempotent writes (ADR-0008): a unit whose new content is semantically equal to what is stored is **not written**, so re-running an MDL script against an in-sync project leaves the `.mpr` and `mprcontents/` byte-identical and Studio Pro shows no version-control changes. Comparison is on a canonical form (element `$ID`s normalised away — a rebuild mints them randomly, so byte comparison would skip nothing); `Microflows$Microflow.StableId` is carried from the stored document rather than re-minted, because the build derives every client-callable microflow's operation id from it. One policy in `modelsdk/canon`, called from both engines' write choke points. `MXCLI_ALWAYS_WRITE=1` disables elision (not preservation) for bisecting. See `docs-site/src/internals/idempotent-writes.md` +- Idempotent writes (ADR-0008): a unit whose new content is semantically equal to what is stored is **not written**, so re-running an MDL script against an in-sync project leaves the `.mpr` and `mprcontents/` byte-identical and Studio Pro shows no version-control changes. Comparison is on a canonical form (element `$ID`s normalised away — a rebuild mints them randomly, so byte comparison would skip nothing); `Microflows$Microflow.StableId` is carried from the stored document rather than re-minted, because the build derives every client-callable microflow's operation id from it. When a write **does** land, `canon.TransplantIDs` matches the rebuild against the stored document and reuses its element `$ID`s (rewriting every pointer in the same pass), so a changed document's diff is the change rather than a wholesale replacement — measured on #910's nanoflow: 1 of 37 identities survived an argument edit before, 37 of 37 after, and a change plus its revert returns to the original bytes. Inserting or deleting an activity mints IDs only for the genuinely new elements. One policy in `modelsdk/canon`, called from both engines' write choke points. `MXCLI_ALWAYS_WRITE=1` disables elision (not preservation) for bisecting — which means it no longer changes the resulting bytes, only the mtimes. See `docs-site/src/internals/idempotent-writes.md` - Domain model (entities, attributes, associations) - ALTER ENTITY (add/rename/modify/drop attributes, indexes, documentation) - Microflows/Nanoflows with 60+ activity types, JavaScript action calls, nanoflow validation parity diff --git a/docs-site/src/internals/idempotent-writes.md b/docs-site/src/internals/idempotent-writes.md index aae624d0b..05755edec 100644 --- a/docs-site/src/internals/idempotent-writes.md +++ b/docs-site/src/internals/idempotent-writes.md @@ -40,6 +40,33 @@ microflow. mxcli carries the stored value onto the rebuilt document instead of minting a new one, so re-running a script does not renumber operations in your deployed model. +## A changed document only shows what changed + +Skipping the write answers "nothing changed". When something *did* change, the +document is still rebuilt from scratch — so without further care, a one-line edit +lands as a wholesale replacement. Editing a single argument of a single +JavaScript action call used to re-mint **36 of a nanoflow's 37** element +identities; the same edit to a microflow re-minted 21 of 22. Studio Pro's changes +view and `git diff` both key on those identities, so a two-line change read as +"the whole document was replaced". It was also cumulative: changing a value and +changing it back produced a semantically identical document that shared none of +its element IDs with the original. + +mxcli now matches the rebuilt document against the stored one element by element +— by shape, and by name where there is one — and puts the stored `$ID` back on +every element that still corresponds, rewriting each pointer to it in the same +pass. On the case above the diff is now the one changed line, all 37 identities +kept, and a change plus its revert returns to the original bytes exactly. + +Inserting or deleting an activity mints IDs only for the elements that are +genuinely new; the rest of the flow keeps its identities rather than shifting +onto its neighbours'. + +Element IDs are *not* the database's identity — a `GUID` is, and that has always +been carried through. See [the note in +CLAUDE.md](https://github.com/ako/mxcli/blob/main/CLAUDE.md) on why the two must +not be confused. + ## Both engines, every write path The policy lives in one place (`modelsdk/canon`) and is applied at the single @@ -75,9 +102,18 @@ Two cautions, both of which produce a meaningless zero: - **Make sure the script is actually re-runnable.** A script containing `create module` or `create enumeration` fails on the second run and writes nothing, so the diff is empty for the wrong reason. Check the run's output. -- **Run the control.** Repeat with `MXCLI_ALWAYS_WRITE=1` and confirm the diff is - *non-empty*. If it is empty too, your measurement cannot detect churn and the - clean result proves nothing. +- **Run the control** — and note that it is a control on *mtimes*, not on + content. Since identities are carried, `MXCLI_ALWAYS_WRITE=1` produces the same + **bytes**; what it changes is that the files are rewritten at all. Compare + `stat -c %y` instead of `sha256sum` and confirm the timestamps *do* move. If + they do not, nothing ran and the clean result proves nothing. + + ```bash + find mprcontents -name '*.mxunit' | sort | xargs stat -c '%Y %n' > t-before.txt + MXCLI_ALWAYS_WRITE=1 mxcli exec script.mdl -p app.mpr + find mprcontents -name '*.mxunit' | sort | xargs stat -c '%Y %n' > t-after.txt + diff t-before.txt t-after.txt # expect output — writes landed + ``` For a per-unit view of what would be skipped, `scripts/mprsnapshot -canon` emits canonical digests keyed by unit id. diff --git a/mdl-examples/bug-tests/910-rewrite-preserves-element-identities.mdl b/mdl-examples/bug-tests/910-rewrite-preserves-element-identities.mdl new file mode 100644 index 000000000..466e49a92 --- /dev/null +++ b/mdl-examples/bug-tests/910-rewrite-preserves-element-identities.mdl @@ -0,0 +1,70 @@ +-- #910 — a one-line change must not re-mint every element identity. +-- +-- Reported as "Studio Pro shows the entire nanoflow as changed": editing one +-- argument of one JavaScript action call rewrote 36 of the nanoflow's 37 element +-- $IDs, so version control painted the whole document as replaced. The same +-- one-literal change to a microflow rewrote 21 of its 22. +-- +-- Elision (ADR-0008) never covered this: it only skips a write when *nothing* +-- changed. What was missing was carrying the stored $IDs onto a write that lands. +-- +-- How to use this file. Run it once to settle the project, then run it again +-- with the marked line changed, and compare the element identities of +-- Bug910.ONL_Advisor before and after: +-- +-- mxcli exec mdl-examples/bug-tests/910-rewrite-preserves-element-identities.mdl -p app.mpr +-- # ... edit the CHANGE ME line below ... +-- mxcli exec mdl-examples/bug-tests/910-rewrite-preserves-element-identities.mdl -p app.mpr +-- +-- Expected: every $ID in the nanoflow's .mxunit survives, and the BSON diff is +-- the changed argument alone. Re-running it unchanged reports +-- "Unchanged nanoflow: Bug910.ONL_Advisor" and does not touch the file. + +CREATE MODULE Bug910; + +@position(100, 100) +CREATE OR MODIFY PERSISTENT ENTITY Bug910."Brand" ( + "Name": String(100) +); + +@position(400, 100) +CREATE OR MODIFY NON-PERSISTENT ENTITY Bug910."Settings" ( + "ApiUrl": String(500), + "AccentColour": String(50) +); + +CREATE OR MODIFY MICROFLOW Bug910."SUB_AdvisorSettings" ( + $Brand: Bug910.Brand +) +RETURNS Bug910.Settings AS $Config +BEGIN + $Config = CREATE Bug910."Settings" ( + "ApiUrl" = 'https://advisor.example.com/api', + "AccentColour" = '#94fc1c' + ); + RETURN $Config; +END; +/ + +CREATE OR MODIFY JAVASCRIPT ACTION Bug910."JS_LoadAdvisor" ( + "apiUrl": String, + "accentColour": String +) +RETURNS Void +PLATFORM All +AS $$ + console.log("advisor", apiUrl, accentColour); +$$; + +CREATE OR MODIFY NANOFLOW Bug910."ONL_Advisor" ( + $Brand: Bug910.Brand +) +BEGIN + $Config = CALL MICROFLOW Bug910."SUB_AdvisorSettings" ("Brand" = $Brand); + $Result = CALL JAVASCRIPT ACTION Bug910."JS_LoadAdvisor" ( + "apiUrl" = $Config/"ApiUrl", + -- CHANGE ME: swap for $Brand/"Name" and re-run. Every other element of the + -- nanoflow must keep its $ID. + "accentColour" = $Config/"AccentColour" + ); +END; diff --git a/mdl/backend/modelsdk/microflow_idempotence_test.go b/mdl/backend/modelsdk/microflow_idempotence_test.go index 9fd6c735a..15aecb778 100644 --- a/mdl/backend/modelsdk/microflow_idempotence_test.go +++ b/mdl/backend/modelsdk/microflow_idempotence_test.go @@ -7,6 +7,9 @@ import ( "testing" "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/canon" + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/modelsdk/element" mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" "github.com/mendixlabs/mxcli/sdk/microflows" "go.mongodb.org/mongo-driver/bson" @@ -127,11 +130,68 @@ func TestWriteMicroflowTwice_SecondWriteIsElided(t *testing.T) { } } -// TestWriteMicroflowTwice_ControlChurnsWhenElisionOff proves the elision is what -// stopped the churn, rather than the rebuild happening to be byte-stable on its -// own. Without this control the test above would pass against a build that never -// had the fix — which is exactly how PR #125 shipped green. -func TestWriteMicroflowTwice_ControlChurnsWhenElisionOff(t *testing.T) { +// rebuildMicroflow returns the bytes the codec produces for mf — the write path +// of UpdateMicroflow up to but not including storage, so the rebuild can be +// observed before Reconcile has had a chance to normalise it. +func rebuildMicroflow(t *testing.T, proj string, mf *microflows.Microflow) []byte { + t.Helper() + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + defer b.Disconnect() + + gm := microflowToGen(mf, b.majorVersion()) + gm.SetID(element.ID(mf.ID)) + assignMicroflowIDs(gm) + contents, err := (&codec.Encoder{}).Encode(gm) + if err != nil { + t.Fatalf("encode: %v", err) + } + return contents +} + +// TestRebuildChurnsSubElementIDs is the control both storage tests rest on: the +// rebuild really is a function of the script *and a random source*, so a test +// showing stable stored bytes is showing the mechanism working rather than a +// rebuild that was byte-stable all along. Without this, the tests here would +// pass against a build that never had either fix — which is how PR #125 shipped +// green. +func TestRebuildChurnsSubElementIDs(t *testing.T) { + proj := copyFixture(t) + mf := aMicroflow(t, proj) + + first := rebuildMicroflow(t, proj, mf) + second := rebuildMicroflow(t, proj, mf) + + if bytes.Equal(first, second) { + t.Fatal("two rebuilds of the same microflow produced identical bytes — the fixture " + + "cannot demonstrate either mechanism, so pick one whose rebuild mints sub-element IDs") + } + // Two things are re-minted per rebuild and they are handled by different + // mechanisms: StableId, a top-level identity field CarryIdentity copies over, + // and every sub-element $ID, which the transplant matches up. Carrying the + // first leaves exactly the second, which is what the storage tests above + // depend on being there. + eq, err := canon.Equal(canon.CarryIdentity(second, first), first) + if err != nil { + t.Fatalf("canon.Equal: %v", err) + } + if !eq { + t.Fatal("with StableId carried, the two rebuilds still differ by more than their " + + "choice of element IDs — this control is measuring some other instability") + } +} + +// TestWriteMicroflowTwice_ForcedWritesStayByteStable is ADR-0008's other half — +// carrying the stored element $IDs onto a write that lands. Elision is turned +// off, so both writes really do reach storage; the bytes are identical anyway +// because the second write inherits the ids the first one left behind. +// +// The same mechanism is what keeps a *changed* document's diff down to what +// changed: without it, editing one argument of one activity re-minted 36 of a +// nanoflow's 37 element identities. +func TestWriteMicroflowTwice_ForcedWritesStayByteStable(t *testing.T) { t.Setenv("MXCLI_ALWAYS_WRITE", "1") proj := copyFixture(t) mf := aMicroflow(t, proj) @@ -142,9 +202,42 @@ func TestWriteMicroflowTwice_ControlChurnsWhenElisionOff(t *testing.T) { writeMicroflow(t, proj, mf) second := storedUnit(t, proj, mf.ID) - if bytes.Equal(first, second) { - t.Fatal("with elision disabled the rebuild was already byte-stable, so the elision test " + - "proves nothing — the rebuild must be re-minting sub-element IDs for this to be a real control") + if !bytes.Equal(first, second) { + t.Errorf("a forced rewrite of an unchanged microflow (%q) changed its stored bytes: "+ + "%d -> %d; stored element identities were not carried onto the rebuild", + mf.Name, len(first), len(second)) + } +} + +// TestWriteStatsSeeTheElision wires the executor's "Modified …" vs "Unchanged …" +// reporting to something real. The counter has to move for the *offer* and stay +// put for the *write*, because that difference is the only evidence the executor +// has that a write was skipped — a counter that merely tracked calls would let +// the console keep claiming writes that never happened (#910). +func TestWriteStatsSeeTheElision(t *testing.T) { + proj := copyFixture(t) + mf := aMicroflow(t, proj) + writeMicroflow(t, proj, mf) // first write lands; second is the one measured + + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + defer b.Disconnect() + + before := b.WriteStats() + if err := b.UpdateMicroflow(mf); err != nil { + t.Fatalf("UpdateMicroflow: %v", err) + } + after := b.WriteStats() + + if after.Offered <= before.Offered { + t.Errorf("Offered %d -> %d: the write never reached storage's elision check", + before.Offered, after.Offered) + } + if after.Written != before.Written { + t.Errorf("Written %d -> %d: an unchanged microflow was counted as written", + before.Written, after.Written) } } diff --git a/modelsdk/canon/identity.go b/modelsdk/canon/identity.go index fabda12a6..910e597e3 100644 --- a/modelsdk/canon/identity.go +++ b/modelsdk/canon/identity.go @@ -40,6 +40,13 @@ func Reconcile(contents, stored []byte) (out []byte, unchanged bool) { // is a change to the app rather than a debugging aid. contents = CarryIdentity(contents, stored) + // The same reasoning one level down, for the writes that do land: a rebuild + // mints a fresh $ID for every sub-element, so a two-line change reads in + // version control as a whole-document replacement. TransplantIDs puts the + // stored ids back on the elements that still correspond, rewriting every + // reference with them. + contents = TransplantIDs(contents, stored) + if alwaysWrite() { return contents, false } diff --git a/modelsdk/canon/transplant.go b/modelsdk/canon/transplant.go new file mode 100644 index 000000000..6c92a1ad1 --- /dev/null +++ b/modelsdk/canon/transplant.go @@ -0,0 +1,413 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + "go.mongodb.org/mongo-driver/v2/bson" + "go.mongodb.org/mongo-driver/v2/x/bsonx/bsoncore" +) + +// TransplantIDs returns contents with every element $ID replaced by the $ID the +// structurally corresponding element in stored already carries — and with every +// reference to that element rewritten in the same pass. +// +// # Why +// +// Elision (Reconcile) answers "did anything change?". When something did, the +// whole document is still rebuilt from the MDL, and every sub-element of a +// rebuild gets a freshly random $ID. Measured on the ticket's own model: +// changing one argument of one JavaScript action call rewrote 36 of a nanoflow's +// 37 element identities, and the same one-line change to a microflow rewrote 21 +// of 22. Studio Pro's version-control view keys on those identities, so a +// two-line semantic change reads as "the entire document was replaced". Worse, +// it is cumulative: making a change and reverting it lands on a document that is +// semantically identical to where it started and shares none of its ids. +// +// This is the other half of ADR-0008. Elision keeps a no-op write off disk; +// carrying identities keeps a *real* write down to what really changed. +// +// # Why this is safe +// +// The correctness bar is lower than it looks, and it is worth being precise +// about where it sits, because the neighbouring operation — renumbering $IDs — +// is what PR #125 did and it made projects unopenable. +// +// - **A wrong match is not a wrong document.** If two elements are paired that +// a person would not have paired, the result is a suboptimal diff, not a +// broken model: every reference to the element is rewritten with it, so +// every pointer still names the same node. Only which UUID labels the node +// changes, and nothing outside the unit remembers it (ADR-0008: of 9,910 +// binary $ID pointers in a real project, 0 cross a unit boundary). +// - **Duplicating an id is the one real failure**, and it is guarded against +// explicitly below: the mapping is injective by construction, and an entry +// whose target is an id some *unmapped* element still holds is dropped. +// - **References are found without knowing which properties hold them**, the +// same insight the canonical form is built on: a pointer is a plain binary +// property, invisible to a containment walk, but any occurrence of one of +// this document's element ids *is* a reference by definition. So the +// substitution is over every 16-byte binary in the document, not over a list +// of pointer property names that would have to be maintained. +// +// The direction also matters. Transplanting moves the written document *towards* +// the ids already on disk, which is exactly what eliding a write does implicitly +// — so it cannot be less safe than the elision that already ships. +// +// # Limits +// +// Only 16-byte binary $IDs participate. The patch is applied in place on a copy, +// which a fixed-width binary makes framing-safe; a string $ID would change the +// document's length and is left alone. Nothing is re-marshalled, so a document +// the codec produced reaches storage exactly as the codec produced it apart from +// these bytes. +// +// Anything that cannot be read — a malformed document on either side — yields no +// mapping and the contents pass through untouched, which is the behaviour that +// existed before this function. +func TransplantIDs(contents, stored []byte) []byte { + m := idMapping(contents, stored) + if len(m) == 0 { + return contents + } + out := append([]byte(nil), contents...) + if !substituteIDs(out, m) { + return contents + } + return out +} + +// idMapping pairs the two documents structurally and returns new id → stored id +// for every pair that is safe to apply. Entries that would be no-ops are kept so +// the collision check can see them, and skipped when applied. +func idMapping(contents, stored []byte) map[string][]byte { + var newDoc, oldDoc bson.D + if err := bson.Unmarshal(contents, &newDoc); err != nil { + return nil + } + if err := bson.Unmarshal(stored, &oldDoc); err != nil { + return nil + } + w := &pairer{pairs: map[string][]byte{}, claimed: map[string]bool{}} + w.pairValue(newDoc, oldDoc) + if len(w.pairs) == 0 { + return nil + } + return dropCollisions(w.pairs, collectElementIDs(newDoc)) +} + +// pairer accumulates the correspondence. claimed enforces injectivity: a stored +// id can be handed out at most once, so two elements can never end up sharing +// one however the alignment behaves. +type pairer struct { + pairs map[string][]byte + claimed map[string]bool +} + +func (p *pairer) record(newID string, oldID string, oldData []byte) { + if _, seen := p.pairs[newID]; seen { + return + } + if p.claimed[oldID] { + return + } + p.claimed[oldID] = true + p.pairs[newID] = oldData +} + +func (p *pairer) pairValue(newV, oldV any) { + if nd, ok := asDoc(newV); ok { + od, ok := asDoc(oldV) + if !ok { + return + } + p.pairDoc(nd, od) + return + } + if news, ok := asSlice(newV); ok { + olds, ok := asSlice(oldV) + if !ok { + return + } + for _, pair := range alignSlices(news, olds) { + p.pairValue(news[pair[0]], olds[pair[1]]) + } + } +} + +// pairDoc corresponds two elements. A differing $Type means these are not the +// same element — inheriting the identity there would claim a replacement was an +// edit — so neither they nor anything under them is paired. +func (p *pairer) pairDoc(nd, od map[string]any) { + if typeName(nd) != typeName(od) { + return + } + if nid, ndata, ok := binaryElementID(nd); ok { + if oid, odata, ok := binaryElementID(od); ok && len(ndata) == len(odata) { + p.record(nid, oid, odata) + } + } + for _, k := range sortedKeys(nd) { + if k == "$ID" { + continue + } + if ov, ok := od[k]; ok { + p.pairValue(nd[k], ov) + } + } +} + +func typeName(d map[string]any) string { + s, _ := d["$Type"].(string) + return s +} + +// binaryElementID reads a 16-byte binary $ID. A string $ID is deliberately not +// accepted: substituting one would change the document's length, and the patch +// is in place. +func binaryElementID(d map[string]any) (uuid string, data []byte, ok bool) { + b, isBin := d["$ID"].(bson.Binary) + if !isBin || len(b.Data) != 16 { + return "", nil, false + } + return blobToUUID(b.Data), b.Data, true +} + +// alignSlices decides which entry of the new list corresponds to which entry of +// the stored one, returning [newIndex, storedIndex] pairs in order. +// +// Positional alignment alone would be enough for the ticket's case (one value +// changed, shape untouched), but it is wrong for the change people actually make +// next: inserting an activity would shift every element after it onto its +// neighbour's identity and churn the rest of the flow — the very symptom being +// fixed. So entries are anchored on a match key first, longest-common- +// subsequence style, and only the gaps between anchors fall back to position. +func alignSlices(news, olds []any) [][2]int { + if len(news) == 0 || len(olds) == 0 { + return nil + } + // A quadratic table on a pathologically long list is not worth it; the + // positional fallback is what alignment degrades to anyway. + const lcsLimit = 1024 + if len(news) > lcsLimit || len(olds) > lcsLimit { + return positionalPairs(0, 0, len(news), len(olds)) + } + + anchors := lcsPairs(matchKeys(news), matchKeys(olds)) + + var out [][2]int + ni, oi := 0, 0 + for _, a := range anchors { + out = append(out, positionalPairs(ni, oi, a[0], a[1])...) + out = append(out, a) + ni, oi = a[0]+1, a[1]+1 + } + return append(out, positionalPairs(ni, oi, len(news), len(olds))...) +} + +// positionalPairs pairs [nStart,nEnd) with [oStart,oEnd) by offset. Used only +// inside a gap between anchors, where the entries did not match by key — a +// renamed element lands here, and pairDoc still refuses if the types differ. +func positionalPairs(nStart, oStart, nEnd, oEnd int) [][2]int { + var out [][2]int + for i := 0; nStart+i < nEnd && oStart+i < oEnd; i++ { + out = append(out, [2]int{nStart + i, oStart + i}) + } + return out +} + +// matchKey identifies a list entry well enough to line two lists up, without +// being so specific that editing the entry stops it matching itself — the whole +// point is that an *edited* element keeps its identity. +// +// $Type plus Name is the "by position, or by name where there is one" the report +// asks for, but on its own it is too blunt for a flow: every activity in a +// microflow is a `Microflows$ActionActivity` with no name, so inserting one at +// the front made LCS pair the *wrapper* correctly and hand the newcomer the old +// activity's whole action subtree. Adding the $Type of each immediate child +// element — the shape one level down, `Action=Microflows$LogMessageAction` — is +// what tells two otherwise identical wrappers apart. Measured on the ticket's +// microflow, inserting a log activity: 17 of 22 identities kept before, 22 of 22 +// after. +// +// Deliberately no other property values. A key built from content would stop an +// element matching itself the moment someone edited it, which is the case this +// whole file exists to preserve. +func matchKey(v any) string { + d, ok := asDoc(v) + if !ok { + return "\x00scalar" + } + key := typeName(d) + if name, ok := d["Name"].(string); ok { + key += "\x00" + name + } + for _, k := range sortedKeys(d) { + child, ok := asDoc(d[k]) + if !ok { + continue + } + if t := typeName(child); t != "" { + key += "\x01" + k + "=" + t + } + } + return key +} + +func matchKeys(vs []any) []string { + out := make([]string, len(vs)) + for i, v := range vs { + out[i] = matchKey(v) + } + return out +} + +// lcsPairs returns a longest common subsequence of the two key sequences as +// index pairs. +func lcsPairs(a, b []string) [][2]int { + n, m := len(a), len(b) + table := make([][]int, n+1) + for i := range table { + table[i] = make([]int, m+1) + } + for i := n - 1; i >= 0; i-- { + for j := m - 1; j >= 0; j-- { + if a[i] == b[j] { + table[i][j] = table[i+1][j+1] + 1 + continue + } + if table[i+1][j] >= table[i][j+1] { + table[i][j] = table[i+1][j] + } else { + table[i][j] = table[i][j+1] + } + } + } + var out [][2]int + for i, j := 0, 0; i < n && j < m; { + switch { + case a[i] == b[j]: + out = append(out, [2]int{i, j}) + i, j = i+1, j+1 + case table[i+1][j] >= table[i][j+1]: + i++ + default: + j++ + } + } + return out +} + +// dropCollisions removes any entry that would leave two elements sharing an id. +// +// Applying the mapping renames the ids in its domain and leaves every other id +// alone, so the only way to collide is to rename x onto an id y that some +// element still holds — that is, y is an id of the new document and is not +// itself being renamed away. Dropping an entry can invalidate another (its +// target stops being renamed away), so this runs to a fixed point. +// +// In practice the domain is either all-fresh random ids, where no target is ever +// present in the new document, or an ALTER-style rewrite where most pairs are +// x → x and trivially safe. The guard is here for the mixed case. +func dropCollisions(pairs map[string][]byte, newIDs map[string]int) map[string][]byte { + for { + var doomed []string + for k, v := range pairs { + target := blobToUUID(v) + if _, held := newIDs[target]; !held { + continue + } + if _, renamedAway := pairs[target]; !renamedAway { + doomed = append(doomed, k) + } + } + if len(doomed) == 0 { + return pairs + } + for _, k := range doomed { + delete(pairs, k) + } + } +} + +// substituteIDs rewrites, in place, every 16-byte binary in raw whose value is a +// key of m. It reports whether the document was walked cleanly. +// +// Every slot is visited exactly once and its replacement chosen from the value +// it held on entry, so a mapping that swaps two ids applies correctly rather +// than renaming one onto the other and back. +// +// The patch relies on bsoncore's looked-up values aliasing raw rather than +// copying out of it — a property of the library, not of this package. Rather +// than assume it, the result is read back and checked against the id set the +// substitution was supposed to produce. +func substituteIDs(raw []byte, m map[string][]byte) bool { + before, ok := elementIDsOf(raw) + if !ok { + return false + } + // What the document's ids must be afterwards: renamed where the mapping says + // so, untouched otherwise. Deriving this up front is what makes the check + // survive a mapping that swaps two ids, where an id being both a source and a + // target is correct rather than evidence of a half-applied patch. + want := make(map[string]bool, len(before)) + for id := range before { + if repl, renamed := m[id]; renamed { + want[blobToUUID(repl)] = true + } else { + want[id] = true + } + } + + if !patchDocument(bsoncore.Document(raw), m) { + return false + } + + after, ok := elementIDsOf(raw) + if !ok || len(after) != len(want) { + return false + } + for id := range after { + if !want[id] { + return false + } + } + return true +} + +func elementIDsOf(raw []byte) (map[string]int, bool) { + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + return nil, false + } + return collectElementIDs(d), true +} + +func patchDocument(doc bsoncore.Document, m map[string][]byte) bool { + elems, err := doc.Elements() + if err != nil { + return false + } + for _, e := range elems { + v := e.Value() + switch v.Type { + case bsoncore.TypeBinary: + _, data, ok := v.BinaryOK() + if !ok || len(data) != 16 { + continue + } + if repl, found := m[blobToUUID(data)]; found { + copy(data, repl) + } + case bsoncore.TypeEmbeddedDocument: + sub, ok := v.DocumentOK() + if !ok || !patchDocument(sub, m) { + return false + } + case bsoncore.TypeArray: + arr, ok := v.ArrayOK() + if !ok || !patchDocument(bsoncore.Document(arr), m) { + return false + } + } + } + return true +} diff --git a/modelsdk/canon/transplant_test.go b/modelsdk/canon/transplant_test.go new file mode 100644 index 000000000..e17392eab --- /dev/null +++ b/modelsdk/canon/transplant_test.go @@ -0,0 +1,334 @@ +// SPDX-License-Identifier: Apache-2.0 + +package canon + +import ( + "testing" + + "go.mongodb.org/mongo-driver/v2/bson" +) + +// flowDoc models the shape the ticket is about: a flow with two activities and a +// sequence flow whose endpoints are plain binary pointers at the activities. +// caption is the one piece of content, so a "same document, one value changed" +// rewrite can be expressed. +func flowDoc(t *testing.T, root, a, b, seq byte, caption string) []byte { + t.Helper() + return marshal(t, bson.D{ + {Key: "$Type", Value: "Microflows$Nanoflow"}, + {Key: "$ID", Value: bin(root)}, + {Key: "Name", Value: "Flow"}, + {Key: "ObjectCollection", Value: bson.D{ + {Key: "$Type", Value: "Microflows$MicroflowObjectCollection"}, + {Key: "$ID", Value: bin(root + 100)}, + {Key: "Objects", Value: bson.A{ + int32(3), // typed-array marker + bson.D{{Key: "$Type", Value: "Microflows$StartEvent"}, {Key: "$ID", Value: bin(a)}}, + bson.D{ + {Key: "$Type", Value: "Microflows$ActionActivity"}, + {Key: "$ID", Value: bin(b)}, + {Key: "Caption", Value: caption}, + }, + }}, + }}, + {Key: "Flows", Value: bson.A{ + int32(3), + bson.D{ + {Key: "$Type", Value: "Microflows$SequenceFlow"}, + {Key: "$ID", Value: bin(seq)}, + {Key: "OriginPointer", Value: bin(a)}, + {Key: "DestinationPointer", Value: bin(b)}, + }, + }}, + }) +} + +// idSet returns every element $ID in a document, as this package's UUID strings. +func idSet(t *testing.T, raw []byte) map[string]bool { + t.Helper() + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatalf("unmarshal: %v", err) + } + out := map[string]bool{} + for id := range collectElementIDs(d) { + out[id] = true + } + return out +} + +func lookupPath(t *testing.T, raw []byte, path ...string) any { + t.Helper() + var cur any + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatalf("unmarshal: %v", err) + } + cur = d + for _, key := range path { + m, ok := asDoc(cur) + if !ok { + t.Fatalf("path %v: not a document at %q", path, key) + } + cur = m[key] + } + return cur +} + +// TestTransplantKeepsStoredIDsAcrossAContentChange is the ticket. Changing one +// value in a nanoflow rewrote 36 of its 37 element identities, because every +// sub-element of a rebuild gets a fresh random $ID. Studio Pro's version-control +// view then paints the whole document as changed. +func TestTransplantKeepsStoredIDsAcrossAContentChange(t *testing.T) { + stored := flowDoc(t, 1, 2, 3, 4, "before") + fresh := flowDoc(t, 9, 8, 7, 6, "after") // same shape, all-new IDs, one value changed + + out := TransplantIDs(fresh, stored) + + got, want := idSet(t, out), idSet(t, stored) + for id := range want { + if !got[id] { + t.Errorf("stored element id %s was not carried over", id) + } + } + if len(got) != len(want) { + t.Errorf("element count changed: %d ids after transplant, %d stored", len(got), len(want)) + } + // The content change must survive: this is a rewrite, not a revert. + coll := lookupPath(t, out, "ObjectCollection", "Objects") + objs, _ := asSlice(coll) + act, _ := asDoc(objs[2]) + if act["Caption"] != "after" { + t.Errorf("Caption = %v, want the new value", act["Caption"]) + } +} + +// TestTransplantRewritesReferences is the invariant that PR #125 broke: a +// pointer is a primitive binary property, not a containment edge, so a +// containment walk never sees it. Renumbering ids without following every +// reference in the same pass is what made projects unopenable. +func TestTransplantRewritesReferences(t *testing.T) { + stored := flowDoc(t, 1, 2, 3, 4, "before") + fresh := flowDoc(t, 9, 8, 7, 6, "after") + + out := TransplantIDs(fresh, stored) + + flows := lookupPath(t, out, "Flows") + fl, _ := asSlice(flows) + seq, _ := asDoc(fl[1]) + origin, ok := seq["OriginPointer"].(bson.Binary) + if !ok { + t.Fatalf("OriginPointer is %T", seq["OriginPointer"]) + } + dest := seq["DestinationPointer"].(bson.Binary) + + objs, _ := asSlice(lookupPath(t, out, "ObjectCollection", "Objects")) + start, _ := asDoc(objs[1]) + activity, _ := asDoc(objs[2]) + startID := start["$ID"].(bson.Binary) + activityID := activity["$ID"].(bson.Binary) + + if blobToUUID(origin.Data) != blobToUUID(startID.Data) { + t.Errorf("OriginPointer %s no longer names the start event %s", + blobToUUID(origin.Data), blobToUUID(startID.Data)) + } + if blobToUUID(dest.Data) != blobToUUID(activityID.Data) { + t.Errorf("DestinationPointer %s no longer names the activity %s", + blobToUUID(dest.Data), blobToUUID(activityID.Data)) + } +} + +// TestTransplantRoundTripIsStable is what makes the churn cumulative: changing a +// value and changing it back produced a semantically identical document with +// another set of fresh ids. With identities carried, the round trip returns to +// exactly the bytes it started from. +func TestTransplantRoundTripIsStable(t *testing.T) { + original := flowDoc(t, 1, 2, 3, 4, "before") + + changed := TransplantIDs(flowDoc(t, 9, 8, 7, 6, "after"), original) + reverted := TransplantIDs(flowDoc(t, 5, 4, 3, 2, "before"), changed) + + if string(reverted) != string(original) { + t.Errorf("a change and its revert did not return to the original bytes\n got %x\nwant %x", + reverted, original) + } +} + +// TestTransplantLeavesInsertedElementsFresh pins that only *corresponding* +// elements are matched. An activity added in the middle must not take over its +// neighbour's identity and push the churn down the rest of the list. +func TestTransplantLeavesInsertedElementsFresh(t *testing.T) { + mk := func(root, a, b, extra byte, withExtra bool) []byte { + objs := bson.A{ + int32(3), + bson.D{{Key: "$Type", Value: "Microflows$StartEvent"}, {Key: "$ID", Value: bin(a)}}, + } + if withExtra { + objs = append(objs, bson.D{ + {Key: "$Type", Value: "Microflows$ActionActivity"}, + {Key: "$ID", Value: bin(extra)}, + {Key: "Caption", Value: "inserted"}, + }) + } + objs = append(objs, bson.D{ + {Key: "$Type", Value: "Microflows$EndEvent"}, + {Key: "$ID", Value: bin(b)}, + }) + return marshal(t, bson.D{ + {Key: "$Type", Value: "Microflows$Nanoflow"}, + {Key: "$ID", Value: bin(root)}, + {Key: "Objects", Value: objs}, + }) + } + + stored := mk(1, 2, 3, 0, false) + out := TransplantIDs(mk(9, 8, 7, 6, true), stored) + + objs, _ := asSlice(lookupPath(t, out, "Objects")) + if len(objs) != 4 { + t.Fatalf("expected marker + 3 objects, got %d entries", len(objs)) + } + start, _ := asDoc(objs[1]) + inserted, _ := asDoc(objs[2]) + end, _ := asDoc(objs[3]) + + if got := blobToUUID(start["$ID"].(bson.Binary).Data); got != blobToUUID(bin(2).Data) { + t.Errorf("start event took id %s, want the stored one", got) + } + // The one the ticket is really about: the end event is *after* the insertion + // point, and must still keep its identity. + if got := blobToUUID(end["$ID"].(bson.Binary).Data); got != blobToUUID(bin(3).Data) { + t.Errorf("end event took id %s, want the stored one — an insertion must not "+ + "shift every following element's identity", got) + } + if got := blobToUUID(inserted["$ID"].(bson.Binary).Data); got != blobToUUID(bin(6).Data) { + t.Errorf("inserted activity took id %s, want its own fresh one", got) + } +} + +// TestTransplantIgnoresMismatchedTypes pins that correspondence is structural. A +// slot whose $Type changed is a different element, and inheriting the old one's +// identity would claim a change is not a change. +func TestTransplantIgnoresMismatchedTypes(t *testing.T) { + mk := func(root, child byte, childType string) []byte { + return marshal(t, bson.D{ + {Key: "$Type", Value: "Microflows$Nanoflow"}, + {Key: "$ID", Value: bin(root)}, + {Key: "Action", Value: bson.D{ + {Key: "$Type", Value: childType}, + {Key: "$ID", Value: bin(child)}, + }}, + }) + } + stored := mk(1, 2, "Microflows$MicroflowCallAction") + out := TransplantIDs(mk(9, 8, "Microflows$JavaScriptActionCallAction"), stored) + + child, _ := asDoc(lookupPath(t, out, "Action")) + if got := blobToUUID(child["$ID"].(bson.Binary).Data); got != blobToUUID(bin(8).Data) { + t.Errorf("an element of a different $Type inherited id %s", got) + } + // The document itself still corresponds, so its own id is carried. + root, _ := asDoc(lookupPath(t, out)) + if got := blobToUUID(root["$ID"].(bson.Binary).Data); got != blobToUUID(bin(1).Data) { + t.Errorf("root id = %s, want the stored one", got) + } +} + +// TestTransplantNeverDuplicatesAnID is the correctness guard. Quality of the +// match only affects how big a diff looks, but two elements sharing an $ID is a +// corrupt document — so a mapping that would rename one element onto an id +// another element still holds must be dropped, not applied. +func TestTransplantNeverDuplicatesAnID(t *testing.T) { + // The mapping is partial: the second child's $Type changed, so it does not + // correspond and keeps its own id — which is 8, the very id the first child + // is about to be renamed onto. Applying that entry would leave two elements + // answering to 8. + mk := func(root, a, b byte, bType string) []byte { + return marshal(t, bson.D{ + {Key: "$Type", Value: "Microflows$Nanoflow"}, + {Key: "$ID", Value: bin(root)}, + {Key: "Objects", Value: bson.A{ + int32(3), + bson.D{{Key: "$Type", Value: "Microflows$StartEvent"}, {Key: "$ID", Value: bin(a)}}, + bson.D{{Key: "$Type", Value: bType}, {Key: "$ID", Value: bin(b)}}, + }}, + }) + } + stored := mk(1, 8, 2, "Microflows$EndEvent") + out := TransplantIDs(mk(1, 3, 8, "Microflows$ActionActivity"), stored) + + if n, distinct := countIDSlots(t, out), len(idSet(t, out)); n != distinct { + t.Errorf("%d $ID slots but only %d distinct ids — the transplant duplicated an identity", + n, distinct) + } +} + +func countIDSlots(t *testing.T, raw []byte) int { + t.Helper() + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatalf("unmarshal: %v", err) + } + n := 0 + var walk func(any) + walk = func(v any) { + if m, ok := asDoc(v); ok { + if _, ok := m["$ID"]; ok { + n++ + } + for _, k := range sortedKeys(m) { + walk(m[k]) + } + return + } + if s, ok := asSlice(v); ok { + for _, e := range s { + walk(e) + } + } + } + walk(d) + return n +} + +// TestTransplantOnANewUnit pins the no-stored-document case: there is nothing to +// carry, and the contents must come through byte-identical. +func TestTransplantOnANewUnit(t *testing.T) { + fresh := flowDoc(t, 9, 8, 7, 6, "after") + if got := TransplantIDs(fresh, nil); string(got) != string(fresh) { + t.Error("a unit with no stored counterpart was rewritten") + } + if got := TransplantIDs(fresh, []byte("not bson")); string(got) != string(fresh) { + t.Error("an unreadable stored document must leave the contents alone") + } +} + +// TestTransplantDoesNotMutateItsInput pins that the caller's buffer is not +// patched behind its back — the same contract CarryIdentity holds to. +func TestTransplantDoesNotMutateItsInput(t *testing.T) { + stored := flowDoc(t, 1, 2, 3, 4, "before") + fresh := flowDoc(t, 9, 8, 7, 6, "after") + before := string(fresh) + + TransplantIDs(fresh, stored) + + if string(fresh) != before { + t.Error("TransplantIDs mutated the contents it was given") + } +} + +// TestReconcileCarriesIDs pins the wiring: both engines reach identity +// preservation through Reconcile, so the transplant has to happen there rather +// than at one engine's call site. +func TestReconcileCarriesIDs(t *testing.T) { + stored := flowDoc(t, 1, 2, 3, 4, "before") + out, unchanged := Reconcile(flowDoc(t, 9, 8, 7, 6, "after"), stored) + if unchanged { + t.Fatal("a changed caption must not be elided") + } + for id := range idSet(t, stored) { + if !idSet(t, out)[id] { + t.Errorf("Reconcile did not carry stored element id %s", id) + } + } +} From f9b816f7aa614854272eec909a2aed1629f6a086 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 10:07:36 +0000 Subject: [PATCH 04/21] fix(executor): stop reporting writes that storage skipped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-running an already-applied script printed Modified javascript action: MxCore.JS_LoadAiAdvisor Replaced nanoflow: HomeScan.ONL_AIAdvisor against a project whose files were not touched — not even their mtimes. The elision was real and correct (ADR-0008); only the reporting was wrong, and it is wrong in the direction that costs most: someone diagnosing version-control churn from console output concludes mxcli rewrites everything on every run, which is exactly how #910 was first mis-diagnosed. The handler cannot tell, because ctx.Backend.UpdateNanoflow returns nil whether or not storage kept the write. Both engines' writers now count unit writes offered versus written at their single choke point, backends expose that through an optional backend.WriteStatsReporter, and ExecContext.ReportMutation downgrades the verb to "Unchanged" when writes were offered since the last report and none of them landed. The sampling is per report rather than per statement, so a handler that rewrites several documents in a loop (constants, module roles) labels each on its own merits. The verb is only downgraded on positive evidence, so a mutation that never reaches unit storage — a theme file, a mock backend, any engine with no notion of units — is reported exactly as it was before. WriteStatsReporter is deliberately not part of FullBackend: it says nothing about the model, only about what a storage engine did, and a backend with no units has no honest answer. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .claude/skills/fix-issue.md | 1 + CLAUDE.md | 8 +- docs-site/src/internals/idempotent-writes.md | 3 + mdl/backend/modelsdk/backend.go | 13 ++ mdl/backend/mpr/backend.go | 13 ++ mdl/backend/writestats.go | 31 +++++ mdl/executor/cmd_agenteditor_models.go | 2 +- mdl/executor/cmd_agenteditor_write.go | 6 +- mdl/executor/cmd_associations.go | 4 +- mdl/executor/cmd_businessevents.go | 2 +- mdl/executor/cmd_constants.go | 2 +- mdl/executor/cmd_datatransformer.go | 2 +- mdl/executor/cmd_dbconnection.go | 2 +- mdl/executor/cmd_entities.go | 6 +- mdl/executor/cmd_enumerations.go | 2 +- mdl/executor/cmd_export_mappings.go | 2 +- mdl/executor/cmd_imagecollections.go | 2 +- mdl/executor/cmd_import_mappings.go | 2 +- mdl/executor/cmd_javaactions.go | 2 +- mdl/executor/cmd_javascript_actions_write.go | 2 +- mdl/executor/cmd_jsonstructures.go | 2 +- mdl/executor/cmd_menus.go | 2 +- mdl/executor/cmd_microflows_create.go | 2 +- mdl/executor/cmd_nanoflows_create.go | 2 +- mdl/executor/cmd_odata.go | 6 +- mdl/executor/cmd_published_rest.go | 2 +- mdl/executor/cmd_queues.go | 2 +- mdl/executor/cmd_regularexpressions.go | 2 +- mdl/executor/cmd_scheduledevents.go | 2 +- mdl/executor/cmd_security_write.go | 6 +- mdl/executor/exec_context.go | 6 + mdl/executor/executor_dispatch.go | 1 + mdl/executor/report_mutation.go | 63 +++++++++ mdl/executor/report_mutation_test.go | 130 +++++++++++++++++++ modelsdk/mpr/writer_core.go | 22 +++- sdk/mpr/writer_core.go | 14 ++ sdk/mpr/writer_units.go | 2 + 37 files changed, 338 insertions(+), 35 deletions(-) create mode 100644 mdl/backend/writestats.go create mode 100644 mdl/executor/report_mutation.go create mode 100644 mdl/executor/report_mutation_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 346206e42..0e9a00a9d 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -564,3 +564,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli check -p app.mpr` passes a microflow containing `if $obj/Status = 'Open'` — comparing an enumeration attribute to a string literal, the first example in the type-checking proposal and the shape people actually write. The same mistake written as a create or change member *is* caught, which makes the gap look arbitrary | Two causes, both invisible. (a) **`inferKind` returned `KindUnknown` for every `AttributePathExpr`**, so `$obj/Attr` typed to nothing and every rule downstream of it stayed quiet. `exprcheck.Scope` is `Lookup(name) (TypeKind, bool)` — it can say "$P is an Object" but not *which* entity, so there was nowhere to put the answer; the adapter computed a variable→entity map (`buildVarEntityScope`) and used it only to label a slot path, never passing it to the checker, and it covered body-introduced variables but **not parameters** — the ordinary case. (b) Even resolved, nothing fired: **E001 keys off the SLOT** (`CreateItem.Value:Entity.Attr`), which exists for an assignment and not for a comparison | `mdl/exprcheck/interfaces.go` (`EntityScope`, `Context.Entities`), `mdl/exprcheck/parser.go` (`attributePathKind`, `pathTargetEntity`, `checkEnumComparedToString`), `mdl/exprcheck/adapters/` (`walkFlow`, `addParamEntities`, `entityScope`), `mdl/exprcatalog/` (`AssociationTarget`) | Put the object side in its **own seam** (`EntityScope`: `VariableEntity` + `AssociationTarget`) beside `Scope` rather than widening either it or `CatalogReader` — `CatalogReader`'s shape is what stays re-syncable from the upstream fork. **A multi-hop expression path is not an XPath path**: XPath spells the intermediate entity (`[Assoc/Entity/Attr]`) so a walk can read it off, while an expression does not (`$O/Mod.Assoc/Attr`), so every hop must resolve through the association index. For (b), emit the **same code and message** from the comparison as from the slot — one defect should not have two names depending on where it was spotted. **Generalisable**: when a checker is silent, separate "did the input resolve" from "is there a rule for this shape" before touching either — here resolution and detection were both missing, and fixing only one would have looked like the fix failed. Verify with a **probe corpus that actually exercises the construct**: 21 microflows described back to MDL contained 40 attribute-path lines including an association hop, and were clean before and after, which is what makes "no false positives" mean something. Controls: path→`KindUnknown` restored, `Entities` dropped from the Context, and `addParamEntities` removed each fail a distinct test. **Still open**: a *terminal* association step (`$Order/Mod.Order_Lines`) types to unknown — Object vs List depends on the association's kind and direction, and guessing costs false positives | | `describe microflow` renders a REST call's result as `returns String` when the activity actually stores the response in a **file document**, and the describe → exec round trip then writes String back — retyping the activity with `mx check` reporting nothing either way. On the legacy engine the `$var =` output variable disappears too | MDL had no syntax for the case, `sdk/microflows` had no FileDocument variant, and both readers fell through to String: the legacy `parseResultHandling` had no `FileDocument` case (so the whole handling read back nil, taking the output-variable fallback with it), while modelsdk's `restResultHandlingFromRaw` read `VariableType.Entity` and then discarded it, matching only the literal `System.HttpResponse`. Mendix stores all five variants as ONE `Microflows$ResultHandling` carrying a `VariableType`, discriminated by the `ResultHandlingType` enum | `mdl/grammar/domains/MDLMicroflow.g4` (`restCallReturnsClause`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `sdk/microflows/microflows_actions.go`, `sdk/mpr/parser_microflow_actions.go` + `writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go` + `microflow_write.go`, `mdl/executor/cmd_microflows_builder_calls.go` + `cmd_microflows_format_action.go`, rule in `validate_microflow_rest.go` (MDL064) | Add `returns Module.Entity` as the LAST alternative so the keyword forms still win, and read Mendix's own `ResultHandlingType` discriminator rather than inferring from `VariableType` (keep the inference as a fallback — the property is `omitempty`). **Delete the silent `String` fallbacks**: an unreconstructable result handling must describe as something the parser REJECTS, or a reader gap becomes a model rewrite (ADR-0005). Mendix constraints worth knowing before designing syntax here: the BASE `System.FileDocument` is rejected as a return type (**CE0362**) so a specialization is mandatory, and HttpResponse **cannot** be specialized (**CE1540** allows only User, FileDocument, Image, Paging) — which is why `returns response` needed no change and the reporter's suggested "HttpResponse specialization" does not exist. Cross-engine round trip (`gateEngines`) is what proves it: the two readers failed differently and each looked fine from inside itself. Issue #922 | | The nightly fails on ONE Mendix matrix version only, in `TestMxCheck_DoctypeScripts`, with `Execution error: this project does not store the model setting ` — while the same script passes on every newer version | The example script set a model setting that version does not have. Measured: a blank 10.24 stores 11 model settings and a blank 11.6.6 stores 12, `DecimalScale` being the only difference. mxcli's refusal is CORRECT — Studio Pro will not open a model carrying a property its version does not define, and mxbuild does not catch it — so the bug is the ungated example, not the guard. The refusal covers the WHOLE statement, so one version-specific setting takes every portable setting in the same `alter` down with it | `mdl-examples/doctype-tests/14-project-settings-examples.mdl`; guard in `mdl/executor/doctype_version_gating_test.go` | Split the version-specific setting into its own statement inside a `-- @version: N.N+` section, closed with `-- @version: any`. **Put the `/** */` doc comment INSIDE the gated section**: a block comment is a documentation comment bound to the statement after it, so gating the statement while leaving the comment outside orphans it and the script dies with `no viable alternative at input '/**...'` — reported at the NEXT statement, tens of lines away, which reads like an unrelated syntax error. `--` line comments are free-standing and safe either side. Isolate which setting is at fault by exec'ing them one at a time against a blank project of that version (`mx create-project` in a SHORT path — a long one dies with PathTooLongException). `TestDoctypeScriptsParseAfterVersionFiltering` now parses every doctype script under each nightly matrix version without needing mxbuild, so a mis-gated script fails in seconds on push instead of hours later in one nightly job | +| Studio Pro's version-control view shows an **entire** nanoflow/microflow as changed after editing one activity argument; `git diff` on `mprcontents/` is unreadable. A change and its revert leave a semantically identical document that shares no element IDs with the original. Separately, re-running an already-applied script prints `Modified …`/`Replaced …` for files it did not touch | Two independent things. (a) `create or replace` rebuilds the document and every sub-element gets a freshly random `$ID`; elision (ADR-0008) only covers the case where *nothing* changed, so a real change wrote a whole new identity set — measured 36 of 37 on a nanoflow, 21 of 22 on a microflow. (b) The `Modified …` lines are printed by the handler right after `ctx.Backend.Update*`, which returns nil whether or not storage elided the write | `modelsdk/canon/transplant.go` (`TransplantIDs`, called from `Reconcile` in `identity.go`); `mdl/executor/report_mutation.go` + `mdl/backend/writestats.go` | Match the incoming document against the stored one and reuse its `$ID`s: `$Type` + shape one level down (`Action=Microflows$LogMessageAction`) + `Name` as the LCS match key, positional fill in the gaps, then substitute **in place over every 16-byte binary** — a pointer is a primitive property a containment walk never sees, and any occurrence of one of the document's element IDs *is* a reference (the same insight `canon` rests on). **The correctness bar is lower than it looks**: a wrong match only makes a diff bigger, since every reference moves with the element; the one real failure is two elements sharing an `$ID`, so guard it explicitly (`dropCollisions`, run to a fixed point) and verify the resulting id set by read-back. **Do not touch `GUID`** — that is the database's identity and was already preserved (measured 8 of 8 through `ALTER ENTITY ADD ATTRIBUTE`). **A key built from content is the trap**: it stops an element matching itself the moment someone edits it, which is the case being fixed — key on shape and name only. **Watch for the control you invalidate**: `MXCLI_ALWAYS_WRITE=1` no longer changes the written bytes (identities are carried), so `TestWriteMicroflowTwice_ControlChurnsWhenElisionOff` became unprovable and had to move down a layer to the raw codec output (`TestRebuildChurnsSubElementIDs`); from the shell, control on mtimes rather than hashes. For the reporting half, downgrade the verb only on **positive** evidence (unit writes offered since the last report, none landed) so a mutation that never reaches unit storage — a theme file, a mock backend — reads exactly as before. Measured on the reporter's own project: 37/37 identities kept, diff down to the one changed line, insert/delete mints only the genuinely new elements, `mx check` unchanged at its 1 pre-existing error, both engines | diff --git a/CLAUDE.md b/CLAUDE.md index e0fdbe47f..d6e902319 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -382,6 +382,12 @@ longer distinguishes anything (measured: same sha, mtime moves). Control on the differs (`TestRebuildChurnsSubElementIDs`) — or, from the shell, on **mtimes** rather than hashes. +The executor reports which of the two happened: a statement whose unit writes were +all elided prints `Unchanged nanoflow: …` instead of `Replaced nanoflow: …` +(`ExecContext.ReportMutation`, fed by each writer's `WriteStats`). The verb is only +downgraded on positive evidence — writes offered, none landed — so a mutation that +never touches unit storage is reported exactly as before. + ### The Tunnel Is Linux-Only, On Purpose — Do Not "Restore" It `mxcli run --hub` and `mxcli tunnel-hub` embed [chisel](https://github.com/jpillora/chisel), @@ -769,7 +775,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati **Implemented:** - Default styling + runtime theme switching (`mxcli theme list/show/apply/remove/switcher`, `mxcli new --theme`): three embedded themes (**signal** light-first, **ledger** light-first, **console** dark-first), each a palette in `theme/web/custom-variables.scss` + a shared Atlas wiring partial + a theme partial imported from `theme/web/main.scss` (which compiles last), plus vendored fonts. **No model changes**, so it hot-applies under `run --local --watch` and cannot affect a build. Generated regions are digest-fenced: a block carrying local edits is refused rather than overwritten. Applying a theme removes the previous one. `--variant auto` (default) ships both palettes — the app follows `prefers-color-scheme` before first paint and honours a `theme-light`/`theme-dark` class on ``; `light`/`dark` bakes one. `theme switcher install` is the only part that writes to the model (JS actions + a nanoflow for a toggle button). Package: `cmd/mxcli/theme/`. See `docs/11-proposals/PROPOSAL_default_styling.md` - MPR v1/v2 reading and writing -- Idempotent writes (ADR-0008): a unit whose new content is semantically equal to what is stored is **not written**, so re-running an MDL script against an in-sync project leaves the `.mpr` and `mprcontents/` byte-identical and Studio Pro shows no version-control changes. Comparison is on a canonical form (element `$ID`s normalised away — a rebuild mints them randomly, so byte comparison would skip nothing); `Microflows$Microflow.StableId` is carried from the stored document rather than re-minted, because the build derives every client-callable microflow's operation id from it. When a write **does** land, `canon.TransplantIDs` matches the rebuild against the stored document and reuses its element `$ID`s (rewriting every pointer in the same pass), so a changed document's diff is the change rather than a wholesale replacement — measured on #910's nanoflow: 1 of 37 identities survived an argument edit before, 37 of 37 after, and a change plus its revert returns to the original bytes. Inserting or deleting an activity mints IDs only for the genuinely new elements. One policy in `modelsdk/canon`, called from both engines' write choke points. `MXCLI_ALWAYS_WRITE=1` disables elision (not preservation) for bisecting — which means it no longer changes the resulting bytes, only the mtimes. See `docs-site/src/internals/idempotent-writes.md` +- Idempotent writes (ADR-0008): a unit whose new content is semantically equal to what is stored is **not written**, so re-running an MDL script against an in-sync project leaves the `.mpr` and `mprcontents/` byte-identical and Studio Pro shows no version-control changes. Comparison is on a canonical form (element `$ID`s normalised away — a rebuild mints them randomly, so byte comparison would skip nothing); `Microflows$Microflow.StableId` is carried from the stored document rather than re-minted, because the build derives every client-callable microflow's operation id from it. When a write **does** land, `canon.TransplantIDs` matches the rebuild against the stored document and reuses its element `$ID`s (rewriting every pointer in the same pass), so a changed document's diff is the change rather than a wholesale replacement — measured on #910's nanoflow: 1 of 37 identities survived an argument edit before, 37 of 37 after, and a change plus its revert returns to the original bytes. Inserting or deleting an activity mints IDs only for the genuinely new elements. One policy in `modelsdk/canon`, called from both engines' write choke points. `MXCLI_ALWAYS_WRITE=1` disables elision (not preservation) for bisecting — which means it no longer changes the resulting bytes, only the mtimes. The executor's output distinguishes the two: `Unchanged nanoflow: …` where the write was skipped. See `docs-site/src/internals/idempotent-writes.md` - Domain model (entities, attributes, associations) - ALTER ENTITY (add/rename/modify/drop attributes, indexes, documentation) - Microflows/Nanoflows with 60+ activity types, JavaScript action calls, nanoflow validation parity diff --git a/docs-site/src/internals/idempotent-writes.md b/docs-site/src/internals/idempotent-writes.md index 05755edec..49a63817a 100644 --- a/docs-site/src/internals/idempotent-writes.md +++ b/docs-site/src/internals/idempotent-writes.md @@ -115,6 +115,9 @@ Two cautions, both of which produce a meaningless zero: diff t-before.txt t-after.txt # expect output — writes landed ``` +The console tells you the same thing, per document: a statement whose write was +skipped reports `Unchanged nanoflow: …` rather than `Replaced nanoflow: …`. + For a per-unit view of what would be skipped, `scripts/mprsnapshot -canon` emits canonical digests keyed by unit id. diff --git a/mdl/backend/modelsdk/backend.go b/mdl/backend/modelsdk/backend.go index d3a183c57..e7c06f747 100644 --- a/mdl/backend/modelsdk/backend.go +++ b/mdl/backend/modelsdk/backend.go @@ -28,6 +28,7 @@ import ( // Compile-time guarantee that the backend satisfies the whole interface (via the // embedded `unimplemented` for every method it doesn't override). var _ backend.FullBackend = (*Backend)(nil) +var _ backend.WriteStatsReporter = (*Backend)(nil) // Backend reads and writes a Mendix project through the modelsdk codec engine. // It embeds `unimplemented` (generated, see gen_unimplemented.go) so any @@ -53,6 +54,18 @@ func errUnimplemented(method string) error { return fmt.Errorf("modelsdk engine: %s not implemented yet — rerun with MXCLI_ENGINE=legacy", method) } +// WriteStats reports how many unit writes reached storage versus how many were +// elided as no-ops (ADR-0008). Zero before Connect, and after Disconnect the +// writer is gone with its counters — a caller sampling across a statement holds +// the connection open for both reads. +func (b *Backend) WriteStats() backend.WriteStats { + if b.writer == nil { + return backend.WriteStats{} + } + offered, written := b.writer.WriteStats() + return backend.WriteStats{Offered: offered, Written: written} +} + // --- ConnectionBackend --- // Connect opens the project read-write through the modelsdk reader/writer diff --git a/mdl/backend/mpr/backend.go b/mdl/backend/mpr/backend.go index 134380d42..85ba986d9 100644 --- a/mdl/backend/mpr/backend.go +++ b/mdl/backend/mpr/backend.go @@ -24,6 +24,7 @@ import ( var _ backend.FullBackend = (*MprBackend)(nil) var _ linter.LintReader = (*MprBackend)(nil) +var _ backend.WriteStatsReporter = (*MprBackend)(nil) // MprBackend implements backend.FullBackend by delegating to mpr.Reader // and mpr.Writer. @@ -55,6 +56,18 @@ func Wrap(writer *mpr.Writer, path string) *MprBackend { } } +// WriteStats reports how many unit writes reached storage versus how many were +// elided as no-ops (ADR-0008). Zero before Connect, and after Disconnect the +// writer is gone with its counters — a caller sampling across a statement holds +// the connection open for both reads. +func (b *MprBackend) WriteStats() backend.WriteStats { + if b.writer == nil { + return backend.WriteStats{} + } + offered, written := b.writer.WriteStats() + return backend.WriteStats{Offered: offered, Written: written} +} + // --------------------------------------------------------------------------- // ConnectionBackend // --------------------------------------------------------------------------- diff --git a/mdl/backend/writestats.go b/mdl/backend/writestats.go new file mode 100644 index 000000000..6912fed02 --- /dev/null +++ b/mdl/backend/writestats.go @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: Apache-2.0 + +package backend + +// WriteStats counts what a session's unit writes actually did to storage. +// +// Storage does not write a unit whose new content is semantically equal to what +// is stored (ADR-0008), so "the handler called UpdateMicroflow and got no error" +// does not mean anything reached disk. Without a way to tell the two apart, a +// re-run of an already-applied script announces "Modified …"/"Replaced …" for +// every statement while changing nothing — which is exactly how the churn in +// #910 was first mis-diagnosed from console output. +// +// Offered counts writes handed to storage; Written counts those that were not +// elided. Both are cumulative for the life of the backend, so a caller measures +// one statement by taking the difference across it. +type WriteStats struct { + Offered int + Written int +} + +// WriteStatsReporter is implemented by backends that can report the above. +// +// Deliberately not part of FullBackend: it says nothing about the model, only +// about what a particular storage engine did, and a backend with no notion of +// units (a mock, a future MCP/PED backend) has no honest answer. Callers +// type-assert and fall back to reporting the mutation unqualified — the +// behaviour that existed before this interface. +type WriteStatsReporter interface { + WriteStats() WriteStats +} diff --git a/mdl/executor/cmd_agenteditor_models.go b/mdl/executor/cmd_agenteditor_models.go index adde30b4e..6e3deab14 100644 --- a/mdl/executor/cmd_agenteditor_models.go +++ b/mdl/executor/cmd_agenteditor_models.go @@ -191,7 +191,7 @@ func execCreateAgentEditorModel(ctx *ExecContext, s *ast.CreateModelStmt) error return mdlerrors.NewBackend("update model", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified model: %s\n", s.Name) + ctx.ReportMutation("Modified", "model: %s", s.Name) return nil } diff --git a/mdl/executor/cmd_agenteditor_write.go b/mdl/executor/cmd_agenteditor_write.go index 8b4a16f6f..bca2b476a 100644 --- a/mdl/executor/cmd_agenteditor_write.go +++ b/mdl/executor/cmd_agenteditor_write.go @@ -59,7 +59,7 @@ func execCreateConsumedMCPService(ctx *ExecContext, s *ast.CreateConsumedMCPServ return mdlerrors.NewBackend("update consumed mcp service", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified consumed mcp service: %s\n", s.Name) + ctx.ReportMutation("Modified", "consumed mcp service: %s", s.Name) return nil } @@ -150,7 +150,7 @@ func execCreateKnowledgeBase(ctx *ExecContext, s *ast.CreateKnowledgeBaseStmt) e return mdlerrors.NewBackend("update knowledge base", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified knowledge base: %s\n", s.Name) + ctx.ReportMutation("Modified", "knowledge base: %s", s.Name) return nil } @@ -301,7 +301,7 @@ func execCreateAgent(ctx *ExecContext, s *ast.CreateAgentStmt) error { return mdlerrors.NewBackend("update agent", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified agent: %s\n", s.Name) + ctx.ReportMutation("Modified", "agent: %s", s.Name) return nil } diff --git a/mdl/executor/cmd_associations.go b/mdl/executor/cmd_associations.go index ebec16d4d..20a83f45d 100644 --- a/mdl/executor/cmd_associations.go +++ b/mdl/executor/cmd_associations.go @@ -114,7 +114,7 @@ func execCreateAssociation(ctx *ExecContext, s *ast.CreateAssociationStmt) error invalidateHierarchy(ctx) invalidateDomainModelsCache(ctx) ctx.trackModifiedDomainModel(module.ID, module.Name) - fmt.Fprintf(ctx.Output, "Modified association: %s\n", s.Name) + ctx.ReportMutation("Modified", "association: %s", s.Name) return nil } } @@ -136,7 +136,7 @@ func execCreateAssociation(ctx *ExecContext, s *ast.CreateAssociationStmt) error invalidateHierarchy(ctx) invalidateDomainModelsCache(ctx) ctx.trackModifiedDomainModel(module.ID, module.Name) - fmt.Fprintf(ctx.Output, "Modified association: %s\n", s.Name) + ctx.ReportMutation("Modified", "association: %s", s.Name) return nil } } diff --git a/mdl/executor/cmd_businessevents.go b/mdl/executor/cmd_businessevents.go index 8cf133bfe..f96cea7e0 100644 --- a/mdl/executor/cmd_businessevents.go +++ b/mdl/executor/cmd_businessevents.go @@ -375,7 +375,7 @@ func createBusinessEventService(ctx *ExecContext, stmt *ast.CreateBusinessEventS if err := ctx.Backend.UpdateBusinessEventService(svc); err != nil { return mdlerrors.NewBackend("update business event service", err) } - fmt.Fprintf(ctx.Output, "Modified business event service: %s.%s\n", moduleName, stmt.Name.Name) + ctx.ReportMutation("Modified", "business event service: %s.%s", moduleName, stmt.Name.Name) } else { if err := ctx.Backend.CreateBusinessEventService(svc); err != nil { return mdlerrors.NewBackend("create business event service", err) diff --git a/mdl/executor/cmd_constants.go b/mdl/executor/cmd_constants.go index a19a98a01..b5e5ca222 100644 --- a/mdl/executor/cmd_constants.go +++ b/mdl/executor/cmd_constants.go @@ -300,7 +300,7 @@ func createConstant(ctx *ExecContext, stmt *ast.CreateConstantStmt) error { return mdlerrors.NewBackend("update constant", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified constant: %s.%s\n", modName, c.Name) + ctx.ReportMutation("Modified", "constant: %s.%s", modName, c.Name) return nil } return mdlerrors.NewAlreadyExistsMsg("constant", modName+"."+c.Name, fmt.Sprintf("constant already exists: %s.%s (use create or modify to update)", modName, c.Name)) diff --git a/mdl/executor/cmd_datatransformer.go b/mdl/executor/cmd_datatransformer.go index 37476d1fc..ea2aefef2 100644 --- a/mdl/executor/cmd_datatransformer.go +++ b/mdl/executor/cmd_datatransformer.go @@ -149,7 +149,7 @@ func execCreateDataTransformer(ctx *ExecContext, s *ast.CreateDataTransformerStm return mdlerrors.NewBackend("update data transformer", err) } if !ctx.Quiet { - fmt.Fprintf(ctx.Output, "Modified data transformer: %s.%s (%d steps)\n", + ctx.ReportMutation("Modified", "data transformer: %s.%s (%d steps)", s.Name.Module, s.Name.Name, len(dt.Steps)) } return nil diff --git a/mdl/executor/cmd_dbconnection.go b/mdl/executor/cmd_dbconnection.go index ad18740d1..6669cb4b4 100644 --- a/mdl/executor/cmd_dbconnection.go +++ b/mdl/executor/cmd_dbconnection.go @@ -135,7 +135,7 @@ func createDatabaseConnection(ctx *ExecContext, stmt *ast.CreateDatabaseConnecti return mdlerrors.NewBackend("update database connection", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified database connection: %s.%s\n", stmt.Name.Module, stmt.Name.Name) + ctx.ReportMutation("Modified", "database connection: %s.%s", stmt.Name.Module, stmt.Name.Name) return nil } diff --git a/mdl/executor/cmd_entities.go b/mdl/executor/cmd_entities.go index 63f5e7d5d..6591d8927 100644 --- a/mdl/executor/cmd_entities.go +++ b/mdl/executor/cmd_entities.go @@ -381,7 +381,7 @@ func execCreateEntity(ctx *ExecContext, s *ast.CreateEntityStmt) error { // Invalidate caches so updated entity is visible invalidateHierarchy(ctx) invalidateDomainModelsCache(ctx) - fmt.Fprintf(ctx.Output, "Modified entity: %s\n", s.Name) + ctx.ReportMutation("Modified", "entity: %s", s.Name) } else { // Create new entity if err := ctx.Backend.CreateEntity(dm.ID, entity); err != nil { @@ -642,7 +642,7 @@ func execCreateViewEntity(ctx *ExecContext, s *ast.CreateViewEntityStmt) error { // Invalidate caches so updated entity is visible invalidateHierarchy(ctx) invalidateDomainModelsCache(ctx) - fmt.Fprintf(ctx.Output, "Modified view entity: %s\n", s.Name) + ctx.ReportMutation("Modified", "view entity: %s", s.Name) } else { // Create new entity if err := ctx.Backend.CreateEntity(dm.ID, entity); err != nil { @@ -950,7 +950,7 @@ func execAlterEntity(ctx *ExecContext, s *ast.AlterEntityStmt) error { } invalidateHierarchy(ctx) invalidateDomainModelsCache(ctx) - fmt.Fprintf(ctx.Output, "Modified attribute '%s' on entity %s\n", s.AttributeName, s.Name) + ctx.ReportMutation("Modified", "attribute '%s' on entity %s", s.AttributeName, s.Name) case ast.AlterEntityDropAttribute: // System attribute pseudo-names: drop by clearing entity flags diff --git a/mdl/executor/cmd_enumerations.go b/mdl/executor/cmd_enumerations.go index a72d1edd8..c2e353396 100644 --- a/mdl/executor/cmd_enumerations.go +++ b/mdl/executor/cmd_enumerations.go @@ -85,7 +85,7 @@ func execCreateEnumeration(ctx *ExecContext, s *ast.CreateEnumerationStmt) error return mdlerrors.NewBackend("update enumeration", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified enumeration: %s\n", s.Name) + ctx.ReportMutation("Modified", "enumeration: %s", s.Name) return nil } diff --git a/mdl/executor/cmd_export_mappings.go b/mdl/executor/cmd_export_mappings.go index 645af902c..87ef64ee0 100644 --- a/mdl/executor/cmd_export_mappings.go +++ b/mdl/executor/cmd_export_mappings.go @@ -238,7 +238,7 @@ func execCreateExportMapping(ctx *ExecContext, s *ast.CreateExportMappingStmt) e return mdlerrors.NewBackend("update export mapping", err) } if !ctx.Quiet { - fmt.Fprintf(ctx.Output, "Modified export mapping %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Modified", "export mapping %s.%s", s.Name.Module, s.Name.Name) } return nil } diff --git a/mdl/executor/cmd_imagecollections.go b/mdl/executor/cmd_imagecollections.go index 74524daab..3d1095966 100644 --- a/mdl/executor/cmd_imagecollections.go +++ b/mdl/executor/cmd_imagecollections.go @@ -76,7 +76,7 @@ func execCreateImageCollection(ctx *ExecContext, s *ast.CreateImageCollectionStm if err := ctx.Backend.UpdateImageCollection(ic); err != nil { return mdlerrors.NewBackend("update image collection", err) } - fmt.Fprintf(ctx.Output, "Modified image collection: %s\n", s.Name) + ctx.ReportMutation("Modified", "image collection: %s", s.Name) } else { if err := ctx.Backend.CreateImageCollection(ic); err != nil { return mdlerrors.NewBackend("create image collection", err) diff --git a/mdl/executor/cmd_import_mappings.go b/mdl/executor/cmd_import_mappings.go index ddf213d05..32d97948e 100644 --- a/mdl/executor/cmd_import_mappings.go +++ b/mdl/executor/cmd_import_mappings.go @@ -281,7 +281,7 @@ func execCreateImportMapping(ctx *ExecContext, s *ast.CreateImportMappingStmt) e return mdlerrors.NewBackend("update import mapping", err) } if !ctx.Quiet { - fmt.Fprintf(ctx.Output, "Modified import mapping %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Modified", "import mapping %s.%s", s.Name.Module, s.Name.Name) } return nil } diff --git a/mdl/executor/cmd_javaactions.go b/mdl/executor/cmd_javaactions.go index 4867a4b60..13eae1083 100644 --- a/mdl/executor/cmd_javaactions.go +++ b/mdl/executor/cmd_javaactions.go @@ -449,7 +449,7 @@ func execCreateJavaAction(ctx *ExecContext, s *ast.CreateJavaActionStmt) error { ctx.InvalidateCache() if existingJAID != "" { - fmt.Fprintf(ctx.Output, "Modified java action: %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Modified", "java action: %s.%s", s.Name.Module, s.Name.Name) } else { fmt.Fprintf(ctx.Output, "Created java action: %s.%s\n", s.Name.Module, s.Name.Name) } diff --git a/mdl/executor/cmd_javascript_actions_write.go b/mdl/executor/cmd_javascript_actions_write.go index 387bddcf6..acba95818 100644 --- a/mdl/executor/cmd_javascript_actions_write.go +++ b/mdl/executor/cmd_javascript_actions_write.go @@ -157,7 +157,7 @@ func execCreateJavaScriptAction(ctx *ExecContext, s *ast.CreateJavaScriptActionS ctx.InvalidateCache() if existingID != "" { - fmt.Fprintf(ctx.Output, "Modified javascript action: %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Modified", "javascript action: %s.%s", s.Name.Module, s.Name.Name) } else { fmt.Fprintf(ctx.Output, "Created javascript action: %s.%s\n", s.Name.Module, s.Name.Name) } diff --git a/mdl/executor/cmd_jsonstructures.go b/mdl/executor/cmd_jsonstructures.go index 6cca8e908..3aa3e323f 100644 --- a/mdl/executor/cmd_jsonstructures.go +++ b/mdl/executor/cmd_jsonstructures.go @@ -227,7 +227,7 @@ func execCreateJsonStructure(ctx *ExecContext, s *ast.CreateJsonStructureStmt) e if err := ctx.Backend.UpdateJsonStructure(js); err != nil { return mdlerrors.NewBackend("update json structure", err) } - fmt.Fprintf(ctx.Output, "Modified json structure: %s\n", s.Name) + ctx.ReportMutation("Modified", "json structure: %s", s.Name) } else { if err := ctx.Backend.CreateJsonStructure(js); err != nil { return mdlerrors.NewBackend("create json structure", err) diff --git a/mdl/executor/cmd_menus.go b/mdl/executor/cmd_menus.go index d4ddecbb8..a0cd06721 100644 --- a/mdl/executor/cmd_menus.go +++ b/mdl/executor/cmd_menus.go @@ -89,7 +89,7 @@ func execCreateMenu(ctx *ExecContext, s *ast.CreateMenuStmt) error { if err := ctx.Backend.UpdateMenuDocument(md); err != nil { return mdlerrors.NewBackend("update menu", err) } - fmt.Fprintf(ctx.Output, "Modified menu %s\n", s.Name.String()) + ctx.ReportMutation("Modified", "menu %s", s.Name.String()) return nil } diff --git a/mdl/executor/cmd_microflows_create.go b/mdl/executor/cmd_microflows_create.go index e7220e24e..87559e06f 100644 --- a/mdl/executor/cmd_microflows_create.go +++ b/mdl/executor/cmd_microflows_create.go @@ -304,7 +304,7 @@ func execCreateMicroflow(ctx *ExecContext, s *ast.CreateMicroflowStmt) error { if err := ctx.Backend.UpdateMicroflow(mf); err != nil { return mdlerrors.NewBackend("update microflow", err) } - fmt.Fprintf(ctx.Output, "Replaced microflow: %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Replaced", "microflow: %s.%s", s.Name.Module, s.Name.Name) } else { if err := ctx.Backend.CreateMicroflow(mf); err != nil { return mdlerrors.NewBackend("create microflow", err) diff --git a/mdl/executor/cmd_nanoflows_create.go b/mdl/executor/cmd_nanoflows_create.go index ad7b4e5d1..55d4a8ed7 100644 --- a/mdl/executor/cmd_nanoflows_create.go +++ b/mdl/executor/cmd_nanoflows_create.go @@ -262,7 +262,7 @@ func execCreateNanoflow(ctx *ExecContext, s *ast.CreateNanoflowStmt) error { if err := ctx.Backend.UpdateNanoflow(nf); err != nil { return mdlerrors.NewBackend("update nanoflow", err) } - fmt.Fprintf(ctx.Output, "Replaced nanoflow: %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Replaced", "nanoflow: %s.%s", s.Name.Module, s.Name.Name) } else { if err := ctx.Backend.CreateNanoflow(nf); err != nil { return mdlerrors.NewBackend("create nanoflow", err) diff --git a/mdl/executor/cmd_odata.go b/mdl/executor/cmd_odata.go index 3121705ac..ce4c6107b 100644 --- a/mdl/executor/cmd_odata.go +++ b/mdl/executor/cmd_odata.go @@ -891,7 +891,7 @@ func execCreateExternalEntity(ctx *ExecContext, s *ast.CreateExternalEntityStmt) if err := ctx.Backend.UpdateEntity(dm.ID, existingEntity); err != nil { return mdlerrors.NewBackend("update external entity", err) } - fmt.Fprintf(ctx.Output, "Modified external entity: %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Modified", "external entity: %s.%s", s.Name.Module, s.Name.Name) return nil } @@ -1052,7 +1052,7 @@ func createODataClient(ctx *ExecContext, stmt *ast.CreateODataClientStmt) error return mdlerrors.NewBackend("update OData client", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified OData client: %s.%s\n", modName, svc.Name) + ctx.ReportMutation("Modified", "OData client: %s.%s", modName, svc.Name) return nil } return mdlerrors.NewAlreadyExistsMsg("OData client", modName+"."+svc.Name, fmt.Sprintf("OData client already exists: %s.%s (use create or modify to update)", modName, svc.Name)) @@ -1466,7 +1466,7 @@ func createODataService(ctx *ExecContext, stmt *ast.CreateODataServiceStmt) erro return mdlerrors.NewBackend("update OData service", err) } invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Modified OData service: %s.%s\n", modName, svc.Name) + ctx.ReportMutation("Modified", "OData service: %s.%s", modName, svc.Name) return nil } return mdlerrors.NewAlreadyExistsMsg("OData service", modName+"."+svc.Name, fmt.Sprintf("OData service already exists: %s.%s (use create or modify to update)", modName, svc.Name)) diff --git a/mdl/executor/cmd_published_rest.go b/mdl/executor/cmd_published_rest.go index efb552fd7..c69396b2b 100644 --- a/mdl/executor/cmd_published_rest.go +++ b/mdl/executor/cmd_published_rest.go @@ -253,7 +253,7 @@ func execCreatePublishedRestService(ctx *ExecContext, s *ast.CreatePublishedRest return mdlerrors.NewBackend("update published rest service", err) } if !ctx.Quiet { - fmt.Fprintf(ctx.Output, "Modified published rest service %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Modified", "published rest service %s.%s", s.Name.Module, s.Name.Name) } } else { if err := ctx.Backend.CreatePublishedRestService(svc); err != nil { diff --git a/mdl/executor/cmd_queues.go b/mdl/executor/cmd_queues.go index d69901413..23abcf4b7 100644 --- a/mdl/executor/cmd_queues.go +++ b/mdl/executor/cmd_queues.go @@ -83,7 +83,7 @@ func execCreateQueue(ctx *ExecContext, s *ast.CreateQueueStmt) error { if err := ctx.Backend.UpdateQueue(q); err != nil { return mdlerrors.NewBackend("update queue", err) } - fmt.Fprintf(ctx.Output, "Modified queue: %s\n", s.Name.String()) + ctx.ReportMutation("Modified", "queue: %s", s.Name.String()) return nil } if err := ctx.Backend.CreateQueue(q); err != nil { diff --git a/mdl/executor/cmd_regularexpressions.go b/mdl/executor/cmd_regularexpressions.go index 55b61c5cf..d82a6b9be 100644 --- a/mdl/executor/cmd_regularexpressions.go +++ b/mdl/executor/cmd_regularexpressions.go @@ -78,7 +78,7 @@ func execCreateRegularExpression(ctx *ExecContext, s *ast.CreateRegularExpressio if err := ctx.Backend.UpdateRegularExpression(re); err != nil { return mdlerrors.NewBackend("update regular expression", err) } - fmt.Fprintf(ctx.Output, "Modified regular expression: %s\n", s.Name.String()) + ctx.ReportMutation("Modified", "regular expression: %s", s.Name.String()) return nil } if err := ctx.Backend.CreateRegularExpression(re); err != nil { diff --git a/mdl/executor/cmd_scheduledevents.go b/mdl/executor/cmd_scheduledevents.go index a89b2c4ac..8895aedb9 100644 --- a/mdl/executor/cmd_scheduledevents.go +++ b/mdl/executor/cmd_scheduledevents.go @@ -122,7 +122,7 @@ func execCreateScheduledEvent(ctx *ExecContext, s *ast.CreateScheduledEventStmt) if err := ctx.Backend.UpdateScheduledEvent(ev); err != nil { return mdlerrors.NewBackend("update scheduled event", err) } - fmt.Fprintf(ctx.Output, "Modified scheduled event: %s\n", s.Name.String()) + ctx.ReportMutation("Modified", "scheduled event: %s", s.Name.String()) return nil } if err := ctx.Backend.CreateScheduledEvent(ev); err != nil { diff --git a/mdl/executor/cmd_security_write.go b/mdl/executor/cmd_security_write.go index 408a768f4..aded3aa38 100644 --- a/mdl/executor/cmd_security_write.go +++ b/mdl/executor/cmd_security_write.go @@ -68,7 +68,7 @@ func execCreateModuleRole(ctx *ExecContext, s *ast.CreateModuleRoleStmt) error { return mdlerrors.NewBackend("modify module role", err) } if !ctx.Quiet { - fmt.Fprintf(ctx.Output, "Modified module role: %s.%s\n", s.Name.Module, s.Name.Name) + ctx.ReportMutation("Modified", "module role: %s.%s", s.Name.Module, s.Name.Name) } return nil } @@ -228,7 +228,7 @@ func execCreateUserRole(ctx *ExecContext, s *ast.CreateUserRoleStmt) error { if err := ctx.Backend.AlterUserRoleModuleRoles(ps.ID, s.Name, true, moduleRoleNames); err != nil { return mdlerrors.NewBackend("update user role", err) } - fmt.Fprintf(ctx.Output, "Modified user role: %s\n", s.Name) + ctx.ReportMutation("Modified", "user role: %s", s.Name) return nil } } @@ -1146,7 +1146,7 @@ func execCreateDemoUser(ctx *ExecContext, s *ast.CreateDemoUserStmt) error { if err := ctx.Backend.AddDemoUser(ps.ID, s.UserName, s.Password, entity, mergedRoles); err != nil { return mdlerrors.NewBackend("update demo user", err) } - fmt.Fprintf(ctx.Output, "Modified demo user: %s\n", s.UserName) + ctx.ReportMutation("Modified", "demo user: %s", s.UserName) return nil } } diff --git a/mdl/executor/exec_context.go b/mdl/executor/exec_context.go index 6ea3c3261..7af2ff8e1 100644 --- a/mdl/executor/exec_context.go +++ b/mdl/executor/exec_context.go @@ -88,6 +88,12 @@ type ExecContext struct { // empty EndEvent in a value-returning microflow, where bare `return;` is invalid. DescribingMicroflowHasReturnValue bool + // lastWriteStats is the storage write watermark as of the previous + // ReportMutation call (or of this context's construction, i.e. the start of + // the statement). Its only use is telling "Modified X" from "X was already + // in sync"; see report_mutation.go. + lastWriteStats backend.WriteStats + // ScriptDepth tracks the current EXECUTE SCRIPT nesting level. // Incremented on each recursive call; execExecuteScript rejects calls // that exceed maxScriptDepth to prevent infinite self-referencing scripts. diff --git a/mdl/executor/executor_dispatch.go b/mdl/executor/executor_dispatch.go index ea2ddf105..00f006064 100644 --- a/mdl/executor/executor_dispatch.go +++ b/mdl/executor/executor_dispatch.go @@ -90,6 +90,7 @@ func (e *Executor) newExecContext(ctx context.Context) *ExecContext { return &ExecContext{ Context: ctx, Backend: e.backend, + lastWriteStats: currentWriteStats(e.backend), Output: e.output, Format: e.format, Quiet: e.quiet, diff --git a/mdl/executor/report_mutation.go b/mdl/executor/report_mutation.go new file mode 100644 index 000000000..793b844c5 --- /dev/null +++ b/mdl/executor/report_mutation.go @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/backend" +) + +// ReportMutation announces that a document was rewritten — but says "Unchanged" +// instead when the rewrite turned out to be a no-op. +// +// Storage skips a unit whose new content is semantically equal to what is +// already there (ADR-0008), so a handler's `ctx.Backend.UpdateNanoflow(nf)` +// returning nil does not mean anything reached disk. Re-running an +// already-applied script used to print +// +// Modified javascript action: MxCore.JS_LoadAiAdvisor +// Replaced nanoflow: HomeScan.ONL_AIAdvisor +// +// against a project whose files were not touched — not even their mtimes. The +// elision was real and correct; only the reporting was wrong, and it is wrong in +// the direction that costs the most: someone diagnosing version-control churn +// from console output concludes mxcli rewrites everything on every run, which is +// what happened in #910. +// +// The verb is downgraded only on positive evidence: unit writes were offered +// since the last report and none of them landed. A mutation that never reaches +// unit storage — a theme file, a backend with no notion of units, a mock — leaves +// no evidence either way and is reported unqualified, exactly as before. +// +// Sampling is per report rather than per statement so a handler that rewrites +// several documents in a loop (constants, module roles) still labels each one on +// its own merits. +func (ctx *ExecContext) ReportMutation(verb, format string, args ...any) { + if ctx.mutationWasElided() { + verb = "Unchanged" + } + fmt.Fprintf(ctx.Output, "%s %s\n", verb, fmt.Sprintf(format, args...)) +} + +// mutationWasElided reports whether every unit write since the previous call was +// skipped as a no-op, and advances the watermark. +func (ctx *ExecContext) mutationWasElided() bool { + now := currentWriteStats(ctx.Backend) + prev := ctx.lastWriteStats + ctx.lastWriteStats = now + // Offered but nothing written is the only case that proves an elision. The + // first call in a statement compares against the watermark newExecContext + // took, so work done by earlier statements is never attributed to this one. + return now.Offered > prev.Offered && now.Written == prev.Written +} + +// currentWriteStats reads a backend's write counters, or the zero value from one +// that does not keep them. +func currentWriteStats(b backend.FullBackend) backend.WriteStats { + reporter, ok := b.(backend.WriteStatsReporter) + if !ok { + return backend.WriteStats{} + } + return reporter.WriteStats() +} diff --git a/mdl/executor/report_mutation_test.go b/mdl/executor/report_mutation_test.go new file mode 100644 index 000000000..bc4033718 --- /dev/null +++ b/mdl/executor/report_mutation_test.go @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mock" +) + +// countingBackend is a mock that also reports write statistics, so a test can +// drive the elision path without a project on disk. +type countingBackend struct { + *mock.MockBackend + stats backend.WriteStats +} + +func (b *countingBackend) WriteStats() backend.WriteStats { return b.stats } + +// offer records n unit writes of which written actually landed. +func (b *countingBackend) offer(offered, written int) { + b.stats.Offered += offered + b.stats.Written += written +} + +func reportCtx(t *testing.T) (*ExecContext, *countingBackend, *bytes.Buffer) { + t.Helper() + mb := &countingBackend{MockBackend: &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + }} + out := &bytes.Buffer{} + return &ExecContext{Backend: mb, Output: out}, mb, out +} + +// TestReportMutationSaysUnchangedWhenNothingWasWritten is the reporting half of +// the ticket: re-running an already-applied script printed "Modified …" and +// "Replaced …" for documents whose files were not touched, not even their +// mtimes. The elision was right; the console was not, and someone diagnosing +// version-control churn from it concludes mxcli rewrites on every run. +func TestReportMutationSaysUnchangedWhenNothingWasWritten(t *testing.T) { + ctx, mb, out := reportCtx(t) + + mb.offer(1, 0) // one unit write offered to storage, elided as a no-op + ctx.ReportMutation("Replaced", "nanoflow: %s", "HomeScan.ONL_AIAdvisor") + + if got := out.String(); got != "Unchanged nanoflow: HomeScan.ONL_AIAdvisor\n" { + t.Errorf("reported %q, want the write to be described as unchanged", got) + } +} + +// TestReportMutationKeepsTheVerbWhenTheWriteLanded is the control. Without it +// the test above would pass against a helper that said "Unchanged" always. +func TestReportMutationKeepsTheVerbWhenTheWriteLanded(t *testing.T) { + ctx, mb, out := reportCtx(t) + + mb.offer(1, 1) + ctx.ReportMutation("Replaced", "nanoflow: %s", "HomeScan.ONL_AIAdvisor") + + if got := out.String(); got != "Replaced nanoflow: HomeScan.ONL_AIAdvisor\n" { + t.Errorf("reported %q, want the original verb", got) + } +} + +// TestReportMutationJudgesEachReportSeparately pins the granularity. A handler +// that rewrites several documents in one statement — constants, module roles — +// must label each on its own merits rather than tarring them all with the +// statement's overall outcome. +func TestReportMutationJudgesEachReportSeparately(t *testing.T) { + ctx, mb, out := reportCtx(t) + + mb.offer(1, 1) + ctx.ReportMutation("Modified", "constant: %s", "App.First") + mb.offer(1, 0) + ctx.ReportMutation("Modified", "constant: %s", "App.Second") + mb.offer(1, 1) + ctx.ReportMutation("Modified", "constant: %s", "App.Third") + + want := strings.Join([]string{ + "Modified constant: App.First", + "Unchanged constant: App.Second", + "Modified constant: App.Third", + "", + }, "\n") + if got := out.String(); got != want { + t.Errorf("got:\n%s\nwant:\n%s", got, want) + } +} + +// TestReportMutationNeedsPositiveEvidence pins the failure direction. A mutation +// that never reaches unit storage — a theme file, a backend with no notion of +// units — leaves no evidence either way, and must be reported as it always was +// rather than guessed at. +func TestReportMutationNeedsPositiveEvidence(t *testing.T) { + ctx, _, out := reportCtx(t) + ctx.ReportMutation("Modified", "theme: %s", "signal") + if got := out.String(); got != "Modified theme: signal\n" { + t.Errorf("with no unit write offered, reported %q", got) + } + + // A backend that does not implement WriteStatsReporter at all is the same + // case, and is what every mock-driven executor test goes through. + plain := &ExecContext{ + Backend: &mock.MockBackend{IsConnectedFunc: func() bool { return true }}, + Output: &bytes.Buffer{}, + } + plain.ReportMutation("Replaced", "microflow: %s", "App.Flow") + if got := plain.Output.(*bytes.Buffer).String(); got != "Replaced microflow: App.Flow\n" { + t.Errorf("a backend with no write stats reported %q", got) + } +} + +// TestReportMutationIgnoresEarlierStatements pins that the watermark starts at +// the statement boundary. newExecContext samples the backend's counters when it +// builds the context, so a write by an earlier statement cannot make this one +// look like it landed. +func TestReportMutationIgnoresEarlierStatements(t *testing.T) { + ctx, mb, out := reportCtx(t) + mb.offer(5, 5) // five earlier statements, all of which really wrote + ctx.lastWriteStats = mb.WriteStats() + + mb.offer(1, 0) + ctx.ReportMutation("Modified", "entity: %s", "App.Thing") + + if got := out.String(); got != "Unchanged entity: App.Thing\n" { + t.Errorf("reported %q — earlier statements' writes were counted against this one", got) + } +} diff --git a/modelsdk/mpr/writer_core.go b/modelsdk/mpr/writer_core.go index 247e16045..099123bde 100644 --- a/modelsdk/mpr/writer_core.go +++ b/modelsdk/mpr/writer_core.go @@ -42,6 +42,20 @@ type Writer struct { // callback instead of going to disk. Used by import-style flows that // want to batch many unit updates into a single transaction. sessionBuf func(unitID string, contents []byte) error + + // unitsOffered / unitsWritten count what reached reconcileWithStored and how + // much of it survived no-op elision (ADR-0008). The executor reads them to + // tell "Modified X" from "Modified nothing, X was already in sync" — without + // them, re-running a script that changes nothing still announces a write for + // every statement, which is how the churn in #910 was misdiagnosed. + unitsOffered int + unitsWritten int +} + +// WriteStats reports how many unit writes this session offered to storage and +// how many were not elided as no-ops. +func (w *Writer) WriteStats() (offered, written int) { + return w.unitsOffered, w.unitsWritten } // SetSessionBuf installs a callback that intercepts every updateUnit call. @@ -529,11 +543,17 @@ func (w *Writer) updateUnit(unitID string, contents []byte) error { // reconcileWithStored applies the shared no-op-elision policy (canon.Reconcile, // ADR-0008 decision 1) to a write against this project. func (w *Writer) reconcileWithStored(unitID string, contents []byte) (out []byte, unchanged bool) { + w.unitsOffered++ stored, err := w.reader.GetRawUnitBytes(unitID) if err != nil { + w.unitsWritten++ return contents, false // new unit, or unreadable — write it } - return canon.Reconcile(contents, stored) + out, unchanged = canon.Reconcile(contents, stored) + if !unchanged { + w.unitsWritten++ + } + return out, unchanged } // UpdateRawUnit saves raw BSON bytes for a unit, bypassing deserialization. diff --git a/sdk/mpr/writer_core.go b/sdk/mpr/writer_core.go index b8ef645e6..b37e38b40 100644 --- a/sdk/mpr/writer_core.go +++ b/sdk/mpr/writer_core.go @@ -33,6 +33,20 @@ func idToBsonBinary(id string) primitive.Binary { // Writer provides methods to write Mendix project files. type Writer struct { reader *Reader + + // unitsOffered / unitsWritten count what reached updateUnit and how much of + // it survived no-op elision (ADR-0008). The executor reads them to tell + // "Modified X" from "Modified nothing, X was already in sync" — without + // them, re-running a script that changes nothing still announces a write for + // every statement, which is how the churn in #910 was misdiagnosed. + unitsOffered int + unitsWritten int +} + +// WriteStats reports how many unit writes this session offered to storage and +// how many were not elided as no-ops. +func (w *Writer) WriteStats() (offered, written int) { + return w.unitsOffered, w.unitsWritten } // NewWriter creates a new writer from a reader opened in read-write mode. diff --git a/sdk/mpr/writer_units.go b/sdk/mpr/writer_units.go index 69c7f3336..322b1f51e 100644 --- a/sdk/mpr/writer_units.go +++ b/sdk/mpr/writer_units.go @@ -141,12 +141,14 @@ func (w *Writer) updateUnit(unitID string, contents []byte) error { // modelsdk engine's policy rather than reimplementing it. The two engines // must agree here: which one ran is an --engine flag, not something a user // should be able to see in their diff. + w.unitsOffered++ if stored, err := w.reader.GetRawUnitBytes(model.ID(unitID)); err == nil { var unchanged bool if contents, unchanged = canon.Reconcile(contents, stored); unchanged { return nil } } + w.unitsWritten++ // Convert UUID string to 16-byte blob unitIDBlob := uuidToBlob(unitID) From 8113e7fdd4c2eb62a39b85472e7bb862aeea542a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 10:36:36 +0000 Subject: [PATCH 05/21] feat(mappings): reach a nested JSON leaf without an entity per level (#927) Studio Pro can bind a leaf several levels below the object element it belongs to: tick a nested leaf without ticking its parents and it is offered on the nearest object element you kept. Mendix stores that as ONE multi-segment JsonPath -- "(Object)|customer|name" on a value element whose parent object element is at "(Object)", with nothing mapped for "customer". One entity, values pulled from several depths. MDL could not express it, so a generated module needed an entity per object level purely to reach a value -- one endpoint added 21 entities, almost all pass-throughs holding nothing but an association. Attr = customer/contact/email `/` is the MDL spelling of Mendix's `|`: it reads better here, and on this side of `=` it cannot collide with the association form on the other side. The member is resolved one segment at a time, so every step keeps the raw-key/exposed-name tolerance from #882. DESCRIBE was mis-reading the shape, which is the half the report did not mention. Value elements were printed as the last segment of their JsonPath alone, so a project holding "(Object)|customer|name" described as `CustomerName = name` -- a description of a model that does not exist -- and re-executing mxcli's own output failed with `"name" is not a member of the JSON structure at (Object)`. Members are now rendered RELATIVE to the enclosing object element, on both engines and for both mapping kinds. Nothing was ever corrupted: the #882 guard refused the bad re-execution. The parent path is used verbatim when computing that relative member. An array's object element already carries the ITEM path, so trimming the "|(Object)" marker off it made a child of that item render as "(Object)/sku" -- caught by the new array cases, and visible in #915's existing test. Two shapes are REFUSED rather than written, each measured on mxbuild 11.13 rather than assumed: * An EXPORT mapping cannot collapse levels. Three-way control: the same member in an import mapping is 0 errors, the same export mapping with only top-level members is 0 errors, and the collapsed export is CE5015 "There is no child mapping matching schema element". An export has to PRODUCE the intermediate node, so something must map it. * An import member cannot cross a 0..* element. Patched into a stored mapping, mxbuild answers CE0256 "Between value mapping 'sku' and parent element '(Object)' is a schema element with wrong occurrence (0..*)" -- the rule is occurrence, not "array" loosely, and the message says so. Both refusals name the build error they prevent, because each would otherwise be valid MDL that passes `mxcli check` and fails only in the build. Verified: the collapsed mapping builds at 0 errors on mxbuild 11.13 and round-trips byte-for-byte through DESCRIBE on BOTH engines; a control with the relative-member fix stubbed fails exactly the collapsed cases while the direct-child cases keep passing; 15 shipped mapping examples still parse and the association forms of both mapping kinds are unaffected; 76 unit packages green. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- .claude/skills/fix-issue.md | 1 + CHANGELOG.md | 11 ++ cmd/mxcli/syntax/features_integration.go | 18 +- docs/01-project/MDL_QUICK_REFERENCE.md | 4 +- .../927-mapping-nested-member-path.fail.mdl | 49 +++++ .../927-mapping-nested-member-path.mdl | 64 +++++++ mdl/executor/cmd_export_mappings.go | 32 +++- mdl/executor/cmd_import_mappings.go | 80 ++++++-- .../mapping_describe_roundtrip_test.go | 27 +-- mdl/executor/mapping_nested_member_test.go | 174 ++++++++++++++++++ mdl/grammar/domains/MDLDomainModel.g4 | 20 +- mdl/visitor/visitor_import_export_mapping.go | 63 ++++--- 12 files changed, 479 insertions(+), 64 deletions(-) create mode 100644 mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl create mode 100644 mdl-examples/bug-tests/927-mapping-nested-member-path.mdl create mode 100644 mdl/executor/mapping_nested_member_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index d64563e93..95db1f149 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -563,3 +563,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `DESCRIBE IMPORT MAPPING` output does not reproduce the script that made it: `Total = total` comes back as `Total = Total`, an array binding as `= ItemItem`, `LineId = id` as `LineId = _id` — and the output cannot be re-run at all, failing with "import mapping already exists". Export mappings identically (unreported) | DESCRIBE printed the element's **ExposedName** (Mendix's display name — capitalised initial, `Item` suffix on an array's item object) instead of the raw JSON key from `JsonPath`, and emitted a bare `create` header where every other DESCRIBE emits `create or modify` | `mdl/executor/cmd_import_mappings.go` (`mappingMemberName`, the four print sites, the header) + `cmd_export_mappings.go` (same four + header) | Print the raw key derived from `JsonPath` — strip a trailing `\|(Object)` first, because an array's mapping element sits at the ITEM object while the script addressed the array (that suffix is what produced `ItemItem`). Fall back to ExposedName when there is no JsonPath (XML-schema / message-definition mappings have none). Safe by construction: the raw path is `jsonSchemaIndex.resolve`'s FIRST lookup, so it cannot regress #882. **Do NOT "fix" ExposedName itself** — the capitalisation is Mendix's own, confirmed against a Studio Pro-authored document in the blank app (`ExposedName "Uuid"` vs `Path "(Object)|uuid"`); rewriting it would diverge from Studio Pro. The `Item` suffix could NOT be confirmed the same way (a blank app has no Studio Pro array structure) and was left alone — with a separate `ExposedItemName` property in the BSON, that is worth checking against a marketplace module before anyone touches storage. Note the issue's framing was half wrong: the mapping DID round-trip semantically (re-executing the old output rebuilt byte-identical JsonPaths), so this was a text/diff defect, not a broken mapping — measure before agreeing with a title. Tests `TestMappingMemberName`, `TestDescribe{Import,Export}Mapping_RoundTripsMemberNames` (fail with the reported symptoms when reverted); two existing header assertions needed updating with the intentional change; fixture `mdl-examples/bug-tests/915-mapping-describe-roundtrip.mdl` is a DESCRIBE fixed point. Issue #915 | | `mxcli check -p app.mpr` passes a microflow containing `if $obj/Status = 'Open'` — comparing an enumeration attribute to a string literal, the first example in the type-checking proposal and the shape people actually write. The same mistake written as a create or change member *is* caught, which makes the gap look arbitrary | Two causes, both invisible. (a) **`inferKind` returned `KindUnknown` for every `AttributePathExpr`**, so `$obj/Attr` typed to nothing and every rule downstream of it stayed quiet. `exprcheck.Scope` is `Lookup(name) (TypeKind, bool)` — it can say "$P is an Object" but not *which* entity, so there was nowhere to put the answer; the adapter computed a variable→entity map (`buildVarEntityScope`) and used it only to label a slot path, never passing it to the checker, and it covered body-introduced variables but **not parameters** — the ordinary case. (b) Even resolved, nothing fired: **E001 keys off the SLOT** (`CreateItem.Value:Entity.Attr`), which exists for an assignment and not for a comparison | `mdl/exprcheck/interfaces.go` (`EntityScope`, `Context.Entities`), `mdl/exprcheck/parser.go` (`attributePathKind`, `pathTargetEntity`, `checkEnumComparedToString`), `mdl/exprcheck/adapters/` (`walkFlow`, `addParamEntities`, `entityScope`), `mdl/exprcatalog/` (`AssociationTarget`) | Put the object side in its **own seam** (`EntityScope`: `VariableEntity` + `AssociationTarget`) beside `Scope` rather than widening either it or `CatalogReader` — `CatalogReader`'s shape is what stays re-syncable from the upstream fork. **A multi-hop expression path is not an XPath path**: XPath spells the intermediate entity (`[Assoc/Entity/Attr]`) so a walk can read it off, while an expression does not (`$O/Mod.Assoc/Attr`), so every hop must resolve through the association index. For (b), emit the **same code and message** from the comparison as from the slot — one defect should not have two names depending on where it was spotted. **Generalisable**: when a checker is silent, separate "did the input resolve" from "is there a rule for this shape" before touching either — here resolution and detection were both missing, and fixing only one would have looked like the fix failed. Verify with a **probe corpus that actually exercises the construct**: 21 microflows described back to MDL contained 40 attribute-path lines including an association hop, and were clean before and after, which is what makes "no false positives" mean something. Controls: path→`KindUnknown` restored, `Entities` dropped from the Context, and `addParamEntities` removed each fail a distinct test. **Still open**: a *terminal* association step (`$Order/Mod.Order_Lines`) types to unknown — Object vs List depends on the association's kind and direction, and guessing costs false positives | | `describe microflow` renders a REST call's result as `returns String` when the activity actually stores the response in a **file document**, and the describe → exec round trip then writes String back — retyping the activity with `mx check` reporting nothing either way. On the legacy engine the `$var =` output variable disappears too | MDL had no syntax for the case, `sdk/microflows` had no FileDocument variant, and both readers fell through to String: the legacy `parseResultHandling` had no `FileDocument` case (so the whole handling read back nil, taking the output-variable fallback with it), while modelsdk's `restResultHandlingFromRaw` read `VariableType.Entity` and then discarded it, matching only the literal `System.HttpResponse`. Mendix stores all five variants as ONE `Microflows$ResultHandling` carrying a `VariableType`, discriminated by the `ResultHandlingType` enum | `mdl/grammar/domains/MDLMicroflow.g4` (`restCallReturnsClause`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `sdk/microflows/microflows_actions.go`, `sdk/mpr/parser_microflow_actions.go` + `writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go` + `microflow_write.go`, `mdl/executor/cmd_microflows_builder_calls.go` + `cmd_microflows_format_action.go`, rule in `validate_microflow_rest.go` (MDL064) | Add `returns Module.Entity` as the LAST alternative so the keyword forms still win, and read Mendix's own `ResultHandlingType` discriminator rather than inferring from `VariableType` (keep the inference as a fallback — the property is `omitempty`). **Delete the silent `String` fallbacks**: an unreconstructable result handling must describe as something the parser REJECTS, or a reader gap becomes a model rewrite (ADR-0005). Mendix constraints worth knowing before designing syntax here: the BASE `System.FileDocument` is rejected as a return type (**CE0362**) so a specialization is mandatory, and HttpResponse **cannot** be specialized (**CE1540** allows only User, FileDocument, Image, Paging) — which is why `returns response` needed no change and the reporter's suggested "HttpResponse specialization" does not exist. Cross-engine round trip (`gateEngines`) is what proves it: the two readers failed differently and each looked fine from inside itself. Issue #922 | +| `describe import mapping` prints only the LAST segment of a value element's JsonPath, so a mapping that binds a nested leaf onto a parent entity describes as `Attr = name` when the model says `(Object)\|customer\|name` — and re-executing mxcli's own output fails with `"name" is not a member of the JSON structure at (Object)`. Authoring that shape was impossible too (`extraneous input '/'`) | Studio Pro can bind a leaf several levels below the object element it belongs to, with no entity for the levels in between, stored as ONE multi-segment JsonPath. `mappingMemberName` took `LastIndex(path, "\|")`, assuming a value element is always a direct child, and the grammar's value alternative took a single `identifierOrKeyword` | grammar `mdl/grammar/domains/MDLDomainModel.g4` (`jsonMemberPath`), `mdl/visitor/visitor_import_export_mapping.go`, `mdl/executor/cmd_import_mappings.go` (`resolvePath`, `mappingMemberName`) + `cmd_export_mappings.go`; fixtures `mdl-examples/bug-tests/927-mapping-nested-member-path*.mdl` | Render the member RELATIVE to the enclosing object element (thread the parent's JsonPath through the printers) and resolve a `/`-separated member one segment at a time so each step keeps the raw-key/exposed-name tolerance from #882. **Use the parent path verbatim** — for an array the object element's own JsonPath is already the ITEM path, and trimming `\|(Object)` off it makes a child of that item render as `(Object)/sku`. Two shapes must be REFUSED, both measured rather than assumed: an EXPORT mapping cannot collapse levels (**CE5015** — it has to produce the intermediate node; three-way control: same member in an import is 0 errors, same export with only top-level members is 0 errors), and an import member cannot cross a `0..*` element (**CE0256** "a schema element with wrong occurrence"). Measure by patching the JsonPath into a stored mapping and running `mx check` — and assert the patch landed before trusting a 0, since an mxcli parse error leaves the baseline project untouched. Issue #927 | diff --git a/CHANGELOG.md b/CHANGELOG.md index 29d8ffd81..e027518da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Added + +- **Import mappings can reach a nested leaf without an entity per level** (#927) — `Attr = customer/contact/email` binds a value several levels below the object element it belongs to, which is the shape Studio Pro produces when you tick a nested leaf without ticking its parents: one entity, values pulled from several depths. Previously MDL had no way to write it, so every object level in a response became an entity whose only content was an association — one generated endpoint added 21 entities, almost all pass-throughs. + + Two shapes are refused rather than written, each measured on mxbuild 11.13 rather than assumed: an **export** mapping cannot collapse levels (**CE5015** — it has to produce the intermediate node; the same member in an import mapping builds at 0 errors, and the same export mapping with only top-level members builds at 0 errors), and an import member cannot cross a `0..*` element (**CE0256** "a schema element with wrong occurrence"). Both refusals name the build error they prevent. + +### Fixed + +- **`DESCRIBE` no longer mis-reads a mapping that binds a nested leaf** (#927) — value elements were printed as the last segment of their JsonPath alone, so a project holding `(Object)|customer|name` described as `CustomerName = name`. That is a description of a model that does not exist, and re-executing mxcli's own output failed with `"name" is not a member of the JSON structure at (Object)`. Members are now rendered relative to the enclosing object element, on both engines and for both mapping kinds. Nothing was ever corrupted — the #882 guard refused the bad re-execution — but the description was wrong. + + ### Fixed - **A microflow's StartEvent no longer moves on a describe→exec round-trip** — the start has no MDL statement to annotate and `DESCRIBE` cannot emit its position, so the builder always derived one (first annotated activity minus one spacing unit). A Studio-Pro-authored flow whose start sat at `145;200` came back at `100;200` — the only coordinate in it that did not survive. The position is now carried over from the microflow being replaced, the way the folder and allowed module roles already are; a fresh `CREATE` still derives it. diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index 9dad67f93..5b2bde606 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -588,13 +588,21 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, Syntax: "SHOW IMPORT MAPPINGS [IN Module];\nDESCRIBE IMPORT MAPPING Module.Name;\n" + "CREATE [OR MODIFY] IMPORT MAPPING Module.Name\n WITH JSON STRUCTURE Module.JsonStruct\n{\n" + " create|find|find or create Module.Entity {\n Attr = jsonField [KEY],\n" + + " Attr = a/b/c,\n" + " Assoc/Module.Child = nestedKey { ... }\n }\n};\nDROP IMPORT MAPPING Module.Name;\n\n" + "OR MODIFY: updates mapping in-place, preserves UUID.\n\n" + + "Nested members (Attr = a/b/c):\n" + + " Reaches a leaf several levels down with NO entity for the levels in\n" + + " between — the shape Studio Pro produces when you tick a nested leaf\n" + + " without ticking its parents. One entity, values from several depths.\n" + + " Use Assoc/Module.Child = key { ... } instead when you WANT an entity\n" + + " per level. The path may not cross a 0..* element: many items cannot\n" + + " collapse into one value, and mxbuild rejects it with CE0256.\n\n" + "Inherited attributes:\n" + " An entity mapped with EXTENDS can map its inherited attributes too —\n" + " name them exactly like its own. mxcli resolves each to the entity that\n" + " declares it, which is what Studio Pro needs to show the field mapped.", - Example: "CREATE IMPORT MAPPING Shop.IMM_Order\n WITH JSON STRUCTURE Shop.JSON_Order\n{\n create Shop.Order {\n OrderId = orderId KEY,\n TotalAmount = total\n }\n};\n\n-- Idempotent update\nCREATE OR MODIFY IMPORT MAPPING Shop.IMM_Order\n WITH JSON STRUCTURE Shop.JSON_Order\n{\n find or create Shop.Order {\n OrderId = orderId KEY,\n TotalAmount = total,\n Status = status\n }\n};", + Example: "CREATE IMPORT MAPPING Shop.IMM_Order\n WITH JSON STRUCTURE Shop.JSON_Order\n{\n create Shop.Order {\n OrderId = orderId KEY,\n TotalAmount = total,\n -- a leaf two levels down, without entities for customer/contact\n Email = customer/contact/email\n }\n};\n\n-- Idempotent update\nCREATE OR MODIFY IMPORT MAPPING Shop.IMM_Order\n WITH JSON STRUCTURE Shop.JSON_Order\n{\n find or create Shop.Order {\n OrderId = orderId KEY,\n TotalAmount = total,\n Status = status\n }\n};", SeeAlso: []string{"export-mapping", "json-structure"}, }) @@ -606,7 +614,13 @@ DESCRIBE DATABASE CONNECTION Ops.Erp;`, "show export mappings", "describe export mapping", "with json structure", "null values", "as jsonKey", }, - Syntax: "SHOW EXPORT MAPPINGS [IN Module];\nDESCRIBE EXPORT MAPPING Module.Name;\nCREATE [OR MODIFY] EXPORT MAPPING Module.Name\n WITH JSON STRUCTURE Module.JsonStruct\n [NULL VALUES LeaveOutElement|SendAsNil]\n{\n Module.Entity {\n jsonField = Attr,\n Assoc/Module.Child AS nestedKey { ... }\n }\n};\nDROP EXPORT MAPPING Module.Name;\n\nOR MODIFY: updates mapping in-place, preserves UUID.", + Syntax: "SHOW EXPORT MAPPINGS [IN Module];\nDESCRIBE EXPORT MAPPING Module.Name;\nCREATE [OR MODIFY] EXPORT MAPPING Module.Name\n WITH JSON STRUCTURE Module.JsonStruct\n [NULL VALUES LeaveOutElement|SendAsNil]\n{\n Module.Entity {\n jsonField = Attr,\n Assoc/Module.Child AS nestedKey { ... }\n }\n};\nDROP EXPORT MAPPING Module.Name;\n\nOR MODIFY: updates mapping in-place, preserves UUID.\n\n" + + "No nested-member form:\n" + + " An import mapping can write `Attr = a/b/c` to reach a leaf without an\n" + + " entity per level. An export mapping cannot: it has to PRODUCE the\n" + + " intermediate node, so Mendix rejects a collapsed member with CE5015\n" + + " (\"no child mapping matching schema element\"). Give the level its own\n" + + " element: Assoc/Module.Child AS key { ... }.", Example: "CREATE EXPORT MAPPING Shop.EMM_Order\n WITH JSON STRUCTURE Shop.JSON_Order\n NULL VALUES LeaveOutElement\n{\n Shop.Order {\n orderId = OrderId,\n total = TotalAmount\n }\n};\n\n-- Idempotent update\nCREATE OR MODIFY EXPORT MAPPING Shop.EMM_Order\n WITH JSON STRUCTURE Shop.JSON_Order\n{\n Shop.Order {\n orderId = OrderId,\n total = TotalAmount,\n status = Status\n }\n};", SeeAlso: []string{"import-mapping", "json-structure"}, }) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index b45142500..d1516a531 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -990,7 +990,7 @@ source json '{"latitude": 51.9, "current": {"temp": 12.8}}' |-----------|--------|-------| | Show mappings | `show import mappings [in module];` | List all or filter by module | | Describe mapping | `describe import mapping Module.Name;` | Re-executable CREATE statement | -| Create mapping | See below | Assignment syntax: `attr = jsonField` | +| Create mapping | See below | Assignment syntax: `attr = jsonField`, or `attr = a/b/c` to reach a nested leaf with **no entity per level** — the shape Studio Pro produces. The path may not cross a `0..*` element (CE0256) | | Create or modify | `create or modify import mapping Module.Name ...;` | Updates existing mapping, preserves UUID | | Drop mapping | `drop import mapping Module.Name;` | | @@ -1022,7 +1022,7 @@ create Module.OrderResponse_CustomerInfo/Module.CustomerInfo = customer { |-----------|--------|-------| | Show mappings | `show export mappings [in module];` | List all or filter by module | | Describe mapping | `describe export mapping Module.Name;` | Re-executable CREATE statement | -| Create mapping | See below | Assignment syntax: `jsonField = attr` | +| Create mapping | See below | Assignment syntax: `jsonField = attr`. **No nested `a/b/c` form**: an export has to produce the intermediate node, so Mendix rejects a collapsed member with CE5015 — give it its own element | | Create or modify | `create or modify export mapping Module.Name ...;` | Updates existing mapping, preserves UUID | | Drop mapping | `drop export mapping Module.Name;` | | diff --git a/mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl b/mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl new file mode 100644 index 000000000..57fce336b --- /dev/null +++ b/mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl @@ -0,0 +1,49 @@ +-- ============================================================================ +-- Issue #927: the two nested-member shapes Mendix rejects +-- ============================================================================ +-- +-- Both are refused by `mxcli exec` rather than written, because each produces a +-- model that is valid MDL, passes `mxcli check`, and fails only in the build. +-- Each refusal below was measured on mxbuild 11.13 by patching the path into a +-- stored mapping and running `mx check`. +-- +-- 1. An EXPORT mapping reaching a nested member. +-- CE5015 "There is no child mapping matching schema element '(Object)'" +-- Three-way control: the same collapsed member in an IMPORT mapping is 0 +-- errors, and the same export mapping with only top-level members is 0 +-- errors. An export has to produce the intermediate node, so something +-- must map it -- give `customer` its own element with an association. +-- +-- 2. An import member whose path crosses a 0..* element. +-- CE0256 "Between value mapping 'sku' and parent element '(Object)' is a +-- schema element with wrong occurrence (0..*)" +-- A value cannot be pulled through many items, so an array level needs its +-- own entity regardless. +-- +-- Usage (this file is expected to FAIL -- .fail.mdl): +-- mxcli exec mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl -p app.mpr +-- Expect a refusal naming CE5015 or CE0256; nothing is written. +-- +-- Requires the module from 927-mapping-nested-member-path.mdl. +-- ============================================================================ + +-- (1) export mapping cannot collapse levels +create or modify export mapping BugTest927.EMM_Collapsed + with json structure BugTest927.JSON_Order +{ + BugTest927.Order { + orderId = OrderId, + customer/name = CustomerName + } +}; +/ + +-- (2) an import member cannot cross an array +create or modify import mapping BugTest927.IMM_AcrossArray + with json structure BugTest927.JSON_Order +{ + create BugTest927.Order { + CustomerName = items/sku + } +}; +/ diff --git a/mdl-examples/bug-tests/927-mapping-nested-member-path.mdl b/mdl-examples/bug-tests/927-mapping-nested-member-path.mdl new file mode 100644 index 000000000..2dac7171c --- /dev/null +++ b/mdl-examples/bug-tests/927-mapping-nested-member-path.mdl @@ -0,0 +1,64 @@ +-- ============================================================================ +-- Issue #927: reaching a nested JSON leaf without an entity per level +-- ============================================================================ +-- +-- Studio Pro can bind a leaf several levels below the object element it belongs +-- to: tick a nested leaf without ticking its parents and it is offered on the +-- nearest object element you kept. Mendix stores that as ONE multi-segment +-- JsonPath -- "(Object)|customer|name" on a value element whose parent object +-- element is at "(Object)", with nothing mapped for "customer". +-- +-- Before the fix, MDL could not express it (`extraneous input '/'`) and, worse, +-- DESCRIBE mis-read it: only the last segment was printed, so a project holding +-- the collapsed shape described as `CustomerName = name` and re-executing that +-- failed with `"name" is not a member of the JSON structure at (Object)`. +-- +-- Measured on mxbuild 11.13: the collapsed IMPORT mapping below builds at 0 +-- errors. See 927-mapping-nested-member-path.fail.mdl for the two shapes Mendix +-- rejects, which mxcli now refuses instead of writing. +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/927-mapping-nested-member-path.mdl -p app.mpr +-- mxcli -p app.mpr -c "describe import mapping BugTest927.IMM_Order" +-- -- must print `customer/name` and `customer/contact/email`, not `name`/`email` +-- mx check app.mpr # no new errors +-- ============================================================================ + +create module BugTest927; + +create json structure BugTest927.JSON_Order +snippet '{"orderId": 100, "customer": {"name": "Alice", "contact": {"email": "a@b.c"}}, "items": [{"sku": "A1"}]}'; +/ + +create non-persistent entity BugTest927.Order ( + OrderId: integer, + CustomerName: string, + Email: string +); +/ + +-- One entity, values pulled from three depths. Without the `/` form this needs +-- an entity for `customer` AND one for `contact`, each holding nothing but an +-- association, purely so the mapping can reach the leaf. +create or modify import mapping BugTest927.IMM_Order + with json structure BugTest927.JSON_Order +{ + create BugTest927.Order { + OrderId = orderId, + CustomerName = customer/name, + Email = customer/contact/email + } +}; +/ + +-- Control: an export mapping over the same structure. Collapsing is an IMPORT +-- capability only -- an export mapping has to PRODUCE the customer node, so +-- Mendix requires something mapping it (CE5015). Top-level members are fine. +create or modify export mapping BugTest927.EMM_Order + with json structure BugTest927.JSON_Order +{ + BugTest927.Order { + orderId = OrderId + } +}; +/ diff --git a/mdl/executor/cmd_export_mappings.go b/mdl/executor/cmd_export_mappings.go index 645af902c..1115b1e95 100644 --- a/mdl/executor/cmd_export_mappings.go +++ b/mdl/executor/cmd_export_mappings.go @@ -118,7 +118,7 @@ func describeExportMapping(ctx *ExecContext, name ast.QualifiedName) error { if len(em.Elements) > 0 { fmt.Fprintln(ctx.Output, "{") for _, elem := range em.Elements { - printExportMappingElement(ctx.Output, elem, 1, true) + printExportMappingElement(ctx.Output, elem, 1, true, "") fmt.Fprintln(ctx.Output) } fmt.Fprintln(ctx.Output, "};") @@ -126,7 +126,7 @@ func describeExportMapping(ctx *ExecContext, name ast.QualifiedName) error { return nil } -func printExportMappingElement(w io.Writer, elem *model.ExportMappingElement, depth int, isRoot bool) { +func printExportMappingElement(w io.Writer, elem *model.ExportMappingElement, depth int, isRoot bool, parentPath string) { indent := strings.Repeat(" ", depth) if elem.Kind == "Object" { if isRoot { @@ -144,11 +144,11 @@ func printExportMappingElement(w io.Writer, elem *model.ExportMappingElement, de assoc := elem.Association entity := elem.Entity if assoc == "" && entity == "" { - fmt.Fprintf(w, "%s. as %s", indent, mappingMemberName(elem.JsonPath, elem.ExposedName)) + fmt.Fprintf(w, "%s. as %s", indent, mappingMemberName(parentPath, elem.JsonPath, elem.ExposedName)) } else if assoc == "" { - fmt.Fprintf(w, "%s./%s as %s", indent, entity, mappingMemberName(elem.JsonPath, elem.ExposedName)) + fmt.Fprintf(w, "%s./%s as %s", indent, entity, mappingMemberName(parentPath, elem.JsonPath, elem.ExposedName)) } else { - fmt.Fprintf(w, "%s%s/%s as %s", indent, assoc, entity, mappingMemberName(elem.JsonPath, elem.ExposedName)) + fmt.Fprintf(w, "%s%s/%s as %s", indent, assoc, entity, mappingMemberName(parentPath, elem.JsonPath, elem.ExposedName)) } if len(elem.Children) > 0 { fmt.Fprintln(w, " {") @@ -156,7 +156,7 @@ func printExportMappingElement(w io.Writer, elem *model.ExportMappingElement, de } if len(elem.Children) > 0 { for i, child := range elem.Children { - printExportMappingElement(w, child, depth+1, false) + printExportMappingElement(w, child, depth+1, false, elem.JsonPath) if i < len(elem.Children)-1 { fmt.Fprintln(w, ",") } else { @@ -172,7 +172,7 @@ func printExportMappingElement(w io.Writer, elem *model.ExportMappingElement, de if parts := strings.Split(attrName, "."); len(parts) == 3 { attrName = parts[2] } - fmt.Fprintf(w, "%s%s = %s", indent, mappingMemberName(elem.JsonPath, elem.ExposedName), attrName) + fmt.Fprintf(w, "%s%s = %s", indent, mappingMemberName(parentPath, elem.JsonPath, elem.ExposedName), attrName) } } @@ -272,6 +272,24 @@ func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingEle lookupPath = "(Object)" jsElem = idx.byPath[lookupPath] } else { + // An EXPORT mapping cannot collapse levels the way an import mapping can. + // Measured on mxbuild 11.13 with a three-way control: an import mapping + // binding "(Object)|customer|name" under an object element at "(Object)" + // builds at 0 errors, an export mapping over the same structure mapping + // only top-level fields builds at 0 errors, and the same export mapping + // with the collapsed member is CE5015 "There is no child mapping matching + // schema element". That follows from what the two do: an export mapping + // has to PRODUCE the customer node, so something must map it. + // + // Refused here rather than written, because the model would be valid MDL, + // pass `mxcli check`, and fail only in the build. (issue #927) + if strings.Contains(def.JsonName, "/") { + return nil, fmt.Errorf("export mapping member %q: an export mapping cannot reach a nested "+ + "member directly — Mendix rejects it with CE5015 because the intermediate object has "+ + "nothing producing it. Give %q its own element: Association/Module.Entity as %s { ... }. "+ + "(Collapsing levels this way works for IMPORT mappings, which only read.)", + def.JsonName, strings.Split(def.JsonName, "/")[0], strings.Split(def.JsonName, "/")[0]) + } jsElem = idx.resolve(parentPath, def.JsonName) } diff --git a/mdl/executor/cmd_import_mappings.go b/mdl/executor/cmd_import_mappings.go index ddf213d05..f18b7672d 100644 --- a/mdl/executor/cmd_import_mappings.go +++ b/mdl/executor/cmd_import_mappings.go @@ -114,7 +114,7 @@ func describeImportMapping(ctx *ExecContext, name ast.QualifiedName) error { if len(im.Elements) > 0 { fmt.Fprintln(ctx.Output, "{") for _, elem := range im.Elements { - printImportMappingElement(ctx.Output, elem, 1, true) + printImportMappingElement(ctx.Output, elem, 1, true, "") fmt.Fprintln(ctx.Output) } fmt.Fprintln(ctx.Output, "};") @@ -134,7 +134,7 @@ func handlingKeyword(handling string) string { } } -func printImportMappingElement(w io.Writer, elem *model.ImportMappingElement, depth int, isRoot bool) { +func printImportMappingElement(w io.Writer, elem *model.ImportMappingElement, depth int, isRoot bool, parentPath string) { indent := strings.Repeat(" ", depth) if elem.Kind == "Object" { handling := handlingKeyword(elem.ObjectHandling) @@ -153,11 +153,11 @@ func printImportMappingElement(w io.Writer, elem *model.ImportMappingElement, de assoc := elem.Association entity := elem.Entity if assoc == "" && entity == "" { - fmt.Fprintf(w, "%s%s . = %s", indent, handling, mappingMemberName(elem.JsonPath, elem.ExposedName)) + fmt.Fprintf(w, "%s%s . = %s", indent, handling, mappingMemberName(parentPath, elem.JsonPath, elem.ExposedName)) } else if assoc == "" { - fmt.Fprintf(w, "%s%s ./%s = %s", indent, handling, entity, mappingMemberName(elem.JsonPath, elem.ExposedName)) + fmt.Fprintf(w, "%s%s ./%s = %s", indent, handling, entity, mappingMemberName(parentPath, elem.JsonPath, elem.ExposedName)) } else { - fmt.Fprintf(w, "%s%s %s/%s = %s", indent, handling, assoc, entity, mappingMemberName(elem.JsonPath, elem.ExposedName)) + fmt.Fprintf(w, "%s%s %s/%s = %s", indent, handling, assoc, entity, mappingMemberName(parentPath, elem.JsonPath, elem.ExposedName)) } if len(elem.Children) > 0 { fmt.Fprintln(w, " {") @@ -165,7 +165,7 @@ func printImportMappingElement(w io.Writer, elem *model.ImportMappingElement, de } if len(elem.Children) > 0 { for i, child := range elem.Children { - printImportMappingElement(w, child, depth+1, false) + printImportMappingElement(w, child, depth+1, false, elem.JsonPath) if i < len(elem.Children)-1 { fmt.Fprintln(w, ",") } else { @@ -185,12 +185,12 @@ func printImportMappingElement(w io.Writer, elem *model.ImportMappingElement, de if elem.IsKey { keyStr = " key" } - fmt.Fprintf(w, "%s%s = %s%s", indent, attrName, mappingMemberName(elem.JsonPath, elem.ExposedName), keyStr) + fmt.Fprintf(w, "%s%s = %s%s", indent, attrName, mappingMemberName(parentPath, elem.JsonPath, elem.ExposedName), keyStr) } } -// mappingMemberName is the JSON member name DESCRIBE should print for a mapping -// element: the raw key taken from its JsonPath, not the derived ExposedName. +// mappingMemberName is the JSON member DESCRIBE should print for a mapping +// element: the raw key(s) taken from its JsonPath, not the derived ExposedName. // // The two differ for any lowercase-initial key, because Mendix derives // ExposedName by capitalising the initial (and suffixing "Item" for an array's @@ -205,13 +205,32 @@ func printImportMappingElement(w io.Writer, elem *model.ImportMappingElement, de // that resolution. Elements with no JsonPath — an XML-schema or // message-definition mapping — keep the exposed name, which is all they have. // (issue #915) -func mappingMemberName(jsonPath, exposedName string) string { +// +// The member is rendered RELATIVE to the enclosing object element rather than as +// its last segment alone. For a direct child the two are the same, but Studio +// Pro can bind a leaf several levels below the object element it belongs to, +// with no entity for the levels in between: a value element at +// "(Object)|customer|name" under an object element at "(Object)". Printing only +// "name" dropped the intermediate levels, so DESCRIBE reported a mapping the +// project did not contain and its own output no longer re-executed — +// `"name" is not a member of the JSON structure at (Object)`. (issue #927) +func mappingMemberName(parentPath, jsonPath, exposedName string) string { if jsonPath == "" { return exposedName } // An array's item object is addressed by the ARRAY's key: the mapping element // sits at "(Object)|item|(Object)" and the script wrote "item". trimmed := strings.TrimSuffix(jsonPath, "|(Object)") + + // The parent path is used verbatim: for an array, the object element's own + // JsonPath is already the ITEM path ("…|items|(Object)"), so trimming the + // marker off it made a child of that item render as "(Object)/sku". + if parentPath != "" { + if rel := strings.TrimPrefix(trimmed, parentPath+"|"); rel != trimmed && rel != "" { + return strings.ReplaceAll(rel, "|", "/") + } + } + i := strings.LastIndex(trimmed, "|") if i < 0 { return exposedName @@ -317,7 +336,15 @@ func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingEle lookupPath = "(Object)" jsElem = idx.byPath[lookupPath] default: - jsElem = idx.resolve(parentPath, def.JsonName) + var arrayLevel *types.JsonElement + jsElem, arrayLevel = idx.resolvePath(parentPath, def.JsonName) + if arrayLevel != nil { + return nil, fmt.Errorf("%q passes through %q, which occurs 0..* — a value cannot be pulled "+ + "through many items, and mxbuild rejects it with CE0256 (\"a schema element with wrong "+ + "occurrence\" between the value mapping and its parent). Give %q its own element with an "+ + "association: create Assoc/Module.Entity = %s { ... }", + def.JsonName, arrayLevel.ExposedName, arrayLevel.ExposedName, arrayLevel.ExposedName) + } } // Clone properties from the matching JSON structure element. A member that @@ -493,6 +520,37 @@ func (i *jsonSchemaIndex) add(parentPath string, elems []*types.JsonElement) { // Returns nil when the name matches nothing — the caller must refuse rather than // invent a path. A fabricated path passes `mxcli check` and only fails later, in // mxbuild as CE5015 or, worse, at runtime. +// resolvePath resolves a `/`-separated member path under parentPath, one segment +// at a time so every step keeps resolve's tolerance for the raw key or the +// exposed name. +// +// A single segment is the ordinary direct-child case. Several segments reach a +// leaf BELOW the enclosing object element with no entity for the levels in +// between — the shape Studio Pro produces when a nested leaf is ticked without +// its parents, stored as one multi-segment JsonPath. Measured on mxbuild 11.13: +// a value element at "(Object)|customer|name" under an object element at +// "(Object)", with nothing mapped for "customer", builds at 0 errors. +// +// An intermediate ARRAY is refused by the caller: measured, mxbuild rejects a +// value element whose path crosses a 0..* element with CE0256, so many items +// genuinely cannot collapse into one value. +func (i *jsonSchemaIndex) resolvePath(parentPath, path string) (*types.JsonElement, *types.JsonElement) { + segments := strings.Split(path, "/") + current := parentPath + var elem *types.JsonElement + for n, seg := range segments { + elem = i.resolve(current, seg) + if elem == nil { + return nil, nil + } + if n < len(segments)-1 && elem.ElementType == "Array" { + return nil, elem // caller reports which level is the array + } + current = elem.Path + } + return elem, nil +} + func (i *jsonSchemaIndex) resolve(parentPath, name string) *types.JsonElement { if e, ok := i.byPath[parentPath+"|"+name]; ok { return e diff --git a/mdl/executor/mapping_describe_roundtrip_test.go b/mdl/executor/mapping_describe_roundtrip_test.go index d3e6506ac..a250da698 100644 --- a/mdl/executor/mapping_describe_roundtrip_test.go +++ b/mdl/executor/mapping_describe_roundtrip_test.go @@ -15,23 +15,26 @@ import ( // The array case is the one that produced "ItemItem": the mapping element sits // at the item object, but the script addressed the ARRAY. func TestMappingMemberName(t *testing.T) { + // parentPath is the enclosing object element's path (#927); these are all + // direct children, where the member is a single segment either way. cases := []struct { - name string - jsonPath string - exposed string - want string + name string + parentPath string + jsonPath string + exposed string + want string }{ - {"value under root", "(Object)|total", "Total", "total"}, - {"camelCase preserved", "(Object)|camelCase", "CamelCase", "camelCase"}, - {"array item object uses the array's key", "(Object)|item|(Object)", "ItemItem", "item"}, - {"value inside an array item", "(Object)|item|(Object)|id", "_id", "id"}, - {"no JsonPath falls back (XML / message mapping)", "", "Total", "Total"}, - {"root has no member name", "(Object)", "JsonObject", "JsonObject"}, + {"value under root", "(Object)", "(Object)|total", "Total", "total"}, + {"camelCase preserved", "(Object)", "(Object)|camelCase", "CamelCase", "camelCase"}, + {"array item object uses the array's key", "(Object)", "(Object)|item|(Object)", "ItemItem", "item"}, + {"value inside an array item", "(Object)|item|(Object)", "(Object)|item|(Object)|id", "_id", "id"}, + {"no JsonPath falls back (XML / message mapping)", "(Object)", "", "Total", "Total"}, + {"root has no member name", "", "(Object)", "JsonObject", "JsonObject"}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { - if got := mappingMemberName(c.jsonPath, c.exposed); got != c.want { - t.Errorf("mappingMemberName(%q, %q) = %q, want %q", c.jsonPath, c.exposed, got, c.want) + if got := mappingMemberName(c.parentPath, c.jsonPath, c.exposed); got != c.want { + t.Errorf("mappingMemberName(%q, %q, %q) = %q, want %q", c.parentPath, c.jsonPath, c.exposed, got, c.want) } }) } diff --git a/mdl/executor/mapping_nested_member_test.go b/mdl/executor/mapping_nested_member_test.go new file mode 100644 index 000000000..29f6a029a --- /dev/null +++ b/mdl/executor/mapping_nested_member_test.go @@ -0,0 +1,174 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// upstream #927. Studio Pro can bind a leaf several levels below the object +// element it belongs to, with no entity for the levels in between — tick a +// nested leaf without ticking its parents and it is offered on the nearest kept +// object element. It is stored as ONE multi-segment JsonPath. +// +// mxcli printed only the last segment, so DESCRIBE reported a mapping the +// project did not contain: a value element at "(Object)|customer|name" under an +// object element at "(Object)" came out as `CustomerName = name`, and +// re-executing that failed with `"name" is not a member of the JSON structure at +// (Object)`. The description of an existing model was simply wrong. +// +// Measured on mxbuild 11.13: the collapsed import mapping builds at 0 errors, so +// this is a real shape mxcli has to read, not a hypothetical. +func TestMappingMemberName_RelativeToParent(t *testing.T) { + cases := []struct { + name string + parentPath string + jsonPath string + exposed string + want string + }{ + { + name: "direct child is unchanged (the overwhelmingly common case)", + parentPath: "(Object)", + jsonPath: "(Object)|orderId", + exposed: "OrderId", + want: "orderId", + }, + { + name: "one level collapsed", + parentPath: "(Object)", + jsonPath: "(Object)|customer|name", + exposed: "Name", + want: "customer/name", + }, + { + name: "two levels collapsed", + parentPath: "(Object)", + jsonPath: "(Object)|customer|contact|email", + exposed: "Email", + want: "customer/contact/email", + }, + { + name: "collapsed under a nested object element", + parentPath: "(Object)|order", + jsonPath: "(Object)|order|customer|name", + exposed: "Name", + want: "customer/name", + }, + { + name: "an array's item object is still addressed by the array's key", + parentPath: "(Object)", + jsonPath: "(Object)|items|(Object)", + exposed: "ItemsItem", + want: "items", + }, + { + name: "children of an array item are relative to the item path", + parentPath: "(Object)|items|(Object)", + jsonPath: "(Object)|items|(Object)|sku", + exposed: "Sku", + want: "sku", + }, + { + name: "no JsonPath at all (XML schema / message definition) keeps the exposed name", + parentPath: "(Object)", + jsonPath: "", + exposed: "Whatever", + want: "Whatever", + }, + { + name: "unknown parent falls back to the last segment rather than printing a path", + parentPath: "(Object)|somewhereElse", + jsonPath: "(Object)|customer|name", + exposed: "Name", + want: "name", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := mappingMemberName(tc.parentPath, tc.jsonPath, tc.exposed); got != tc.want { + t.Errorf("mappingMemberName(%q, %q, %q) = %q, want %q", + tc.parentPath, tc.jsonPath, tc.exposed, got, tc.want) + } + }) + } +} + +// The index resolves a `/`-separated member one segment at a time, so each step +// keeps resolve's tolerance for the raw key or the exposed name (#882). +func TestJSONSchemaIndex_ResolvePath(t *testing.T) { + idx := newJSONSchemaIndex(orderSchemaElements()) + + t.Run("single segment behaves like resolve", func(t *testing.T) { + got, arr := idx.resolvePath("(Object)", "orderId") + if arr != nil { + t.Fatalf("unexpected array level %v", arr) + } + if got == nil || got.Path != "(Object)|orderId" { + t.Fatalf("got %v, want the orderId element", got) + } + }) + + t.Run("multi-segment reaches the nested leaf", func(t *testing.T) { + got, arr := idx.resolvePath("(Object)", "customer/contact/email") + if arr != nil { + t.Fatalf("unexpected array level %v", arr) + } + if got == nil || got.Path != "(Object)|customer|contact|email" { + t.Fatalf("got %v, want the nested email element", got) + } + }) + + t.Run("exposed names work at every step", func(t *testing.T) { + got, _ := idx.resolvePath("(Object)", "Customer/Contact/Email") + if got == nil || got.Path != "(Object)|customer|contact|email" { + t.Fatalf("got %v, want the nested email element via exposed names", got) + } + }) + + t.Run("a missing segment resolves to nothing rather than a fabricated path", func(t *testing.T) { + got, arr := idx.resolvePath("(Object)", "customer/nope") + if got != nil || arr != nil { + t.Fatalf("got (%v, %v), want (nil, nil) — a fabricated path fails only later, in the build", got, arr) + } + }) + + // Measured on mxbuild 11.13 by patching the path into a stored mapping: + // CE0256 "Between value mapping 'sku' and parent element '(Object)' is a + // schema element with wrong occurrence (0..*)". A value genuinely cannot be + // pulled through many items, so the caller refuses instead of writing it. + t.Run("an intermediate 0..* element is reported, not traversed", func(t *testing.T) { + got, arr := idx.resolvePath("(Object)", "items/sku") + if got != nil { + t.Errorf("resolved through an array to %v; mxbuild rejects that with CE0256", got) + } + if arr == nil || arr.Path != "(Object)|items" { + t.Fatalf("array level = %v, want the items element so the error can name it", arr) + } + }) +} + +// orderSchemaElements is the element tree a JSON structure over +// {"orderId":…, "customer":{"name":…, "contact":{"email":…}}, "items":[{"sku":…}]} +// produces: two collapsible object levels and one array, which is the shape both +// halves of #927 turn on. +func orderSchemaElements() []*types.JsonElement { + email := &types.JsonElement{ExposedName: "Email", Path: "(Object)|customer|contact|email", ElementType: "Value", MaxOccurs: 1} + contact := &types.JsonElement{ExposedName: "Contact", Path: "(Object)|customer|contact", ElementType: "Object", MaxOccurs: 1, + Children: []*types.JsonElement{email}} + name := &types.JsonElement{ExposedName: "Name", Path: "(Object)|customer|name", ElementType: "Value", MaxOccurs: 1} + customer := &types.JsonElement{ExposedName: "Customer", Path: "(Object)|customer", ElementType: "Object", MaxOccurs: 1, + Children: []*types.JsonElement{name, contact}} + orderID := &types.JsonElement{ExposedName: "OrderId", Path: "(Object)|orderId", ElementType: "Value", MaxOccurs: 1} + sku := &types.JsonElement{ExposedName: "Sku", Path: "(Object)|items|(Object)|sku", ElementType: "Value", MaxOccurs: 1} + item := &types.JsonElement{ExposedName: "ItemsItem", Path: "(Object)|items|(Object)", ElementType: "Object", MaxOccurs: 1, + Children: []*types.JsonElement{sku}} + items := &types.JsonElement{ExposedName: "Items", Path: "(Object)|items", ElementType: "Array", MaxOccurs: -1, + Children: []*types.JsonElement{item}} + root := &types.JsonElement{ExposedName: "Root", Path: "(Object)", ElementType: "Object", MaxOccurs: 1, + Children: []*types.JsonElement{orderID, customer, items}} + return []*types.JsonElement{root} +} diff --git a/mdl/grammar/domains/MDLDomainModel.g4 b/mdl/grammar/domains/MDLDomainModel.g4 index 0c83076e9..8975903b3 100644 --- a/mdl/grammar/domains/MDLDomainModel.g4 +++ b/mdl/grammar/domains/MDLDomainModel.g4 @@ -469,8 +469,22 @@ importMappingChild : importMappingObjectHandling qualifiedName SLASH qualifiedName EQUALS identifierOrKeyword LBRACE importMappingChild (COMMA importMappingChild)* RBRACE // nested object with children | importMappingObjectHandling qualifiedName SLASH qualifiedName EQUALS identifierOrKeyword // leaf object - | identifierOrKeyword EQUALS qualifiedName LPAREN identifierOrKeyword RPAREN // value transform: Attr = Module.MF(jsonField) - | identifierOrKeyword EQUALS identifierOrKeyword KEY? // value: Attr = jsonField [KEY] + | identifierOrKeyword EQUALS qualifiedName LPAREN jsonMemberPath RPAREN // value transform: Attr = Module.MF(jsonField) + | identifierOrKeyword EQUALS jsonMemberPath KEY? // value: Attr = a/b/c [KEY] + ; + +/** + * A JSON member, addressed from the enclosing object element. A single name is + * a direct child; `a/b/c` reaches a leaf several levels down WITHOUT an entity + * for the levels in between — the shape Studio Pro produces when you tick a + * nested leaf without ticking its parents. + * + * Stored as Mendix's own pipe-separated JsonPath ("(Object)|a|b|c"); `/` is the + * MDL spelling because `|` reads badly here and `/` on this side of `=` cannot + * collide with the association form on the other side. + */ +jsonMemberPath + : identifierOrKeyword (SLASH identifierOrKeyword)* ; importMappingObjectHandling @@ -513,7 +527,7 @@ exportMappingChild : qualifiedName SLASH qualifiedName AS identifierOrKeyword LBRACE exportMappingChild (COMMA exportMappingChild)* RBRACE // nested object with children | qualifiedName SLASH qualifiedName AS identifierOrKeyword // leaf object - | identifierOrKeyword EQUALS identifierOrKeyword // value: jsonField = Attr + | jsonMemberPath EQUALS identifierOrKeyword // value: a/b/c = Attr ; // ============================================================================= diff --git a/mdl/visitor/visitor_import_export_mapping.go b/mdl/visitor/visitor_import_export_mapping.go index 08cabdafc..07b327b7f 100644 --- a/mdl/visitor/visitor_import_export_mapping.go +++ b/mdl/visitor/visitor_import_export_mapping.go @@ -88,9 +88,8 @@ func buildImportChild(ctx *parser.ImportMappingChildContext) *ast.ImportMappingE } // JSON key after EQUALS - allIdent := ctx.AllIdentifierOrKeyword() - if len(allIdent) >= 1 { - elem.JsonName = identifierOrKeywordText(allIdent[0]) + if id := ctx.IdentifierOrKeyword(); id != nil { + elem.JsonName = identifierOrKeywordText(id) } // Nested children @@ -99,27 +98,21 @@ func buildImportChild(ctx *parser.ImportMappingChildContext) *ast.ImportMappingE elem.Children = append(elem.Children, child) } } else if ctx.LPAREN() != nil { - // Value transform: attr = Module.MF(jsonField) - allIdent := ctx.AllIdentifierOrKeyword() - if len(allIdent) >= 1 { - elem.Attribute = identifierOrKeywordText(allIdent[0]) + // Value transform: attr = Module.MF(a/b/c) + if id := ctx.IdentifierOrKeyword(); id != nil { + elem.Attribute = identifierOrKeywordText(id) } allQN := ctx.AllQualifiedName() if len(allQN) >= 1 { elem.Converter = buildQualifiedName(allQN[0]).String() } - if len(allIdent) >= 2 { - elem.ConverterParam = identifierOrKeywordText(allIdent[1]) - } + elem.ConverterParam = jsonMemberPathText(ctx.JsonMemberPath()) } else { - // Value assignment: attr = jsonField KEY? - allIdent := ctx.AllIdentifierOrKeyword() - if len(allIdent) >= 1 { - elem.Attribute = identifierOrKeywordText(allIdent[0]) - } - if len(allIdent) >= 2 { - elem.JsonName = identifierOrKeywordText(allIdent[1]) + // Value assignment: attr = a/b/c KEY? + if id := ctx.IdentifierOrKeyword(); id != nil { + elem.Attribute = identifierOrKeywordText(id) } + elem.JsonName = jsonMemberPathText(ctx.JsonMemberPath()) if ctx.KEY() != nil { elem.IsKey = true } @@ -201,9 +194,8 @@ func buildExportChild(ctx *parser.ExportMappingChildContext) *ast.ExportMappingE elem.Entity = buildQualifiedName(allQN[1]).String() // JSON key after AS - allIdent := ctx.AllIdentifierOrKeyword() - if len(allIdent) >= 1 { - elem.JsonName = identifierOrKeywordText(allIdent[0].(*parser.IdentifierOrKeywordContext)) + if id := ctx.IdentifierOrKeyword(); id != nil { + elem.JsonName = identifierOrKeywordText(id.(*parser.IdentifierOrKeywordContext)) } // Nested children @@ -212,13 +204,10 @@ func buildExportChild(ctx *parser.ExportMappingChildContext) *ast.ExportMappingE elem.Children = append(elem.Children, child) } } else { - // Value mapping: jsonField = Attr - allIdent := ctx.AllIdentifierOrKeyword() - if len(allIdent) >= 1 { - elem.JsonName = identifierOrKeywordText(allIdent[0].(*parser.IdentifierOrKeywordContext)) - } - if len(allIdent) >= 2 { - elem.Attribute = identifierOrKeywordText(allIdent[1].(*parser.IdentifierOrKeywordContext)) + // Value mapping: a/b/c = Attr + elem.JsonName = jsonMemberPathText(ctx.JsonMemberPath()) + if id := ctx.IdentifierOrKeyword(); id != nil { + elem.Attribute = identifierOrKeywordText(id.(*parser.IdentifierOrKeywordContext)) } } @@ -324,3 +313,23 @@ func extractObjectHandling(ctx *parser.ImportMappingObjectHandlingContext) strin } return "Create" } + +// jsonMemberPathText renders a jsonMemberPath as the `/`-separated string the +// AST carries. A single segment is the ordinary direct-child case; several +// segments reach a leaf below the enclosing object element without an entity +// for the levels in between (issue #927). The executor translates `/` to +// Mendix's `|` when resolving against the JSON structure. +func jsonMemberPathText(ctx parser.IJsonMemberPathContext) string { + if ctx == nil { + return "" + } + pathCtx, ok := ctx.(*parser.JsonMemberPathContext) + if !ok { + return "" + } + var segments []string + for _, seg := range pathCtx.AllIdentifierOrKeyword() { + segments = append(segments, identifierOrKeywordText(seg)) + } + return strings.Join(segments, "/") +} From 1492e571234e3d27ca7d51f35c7f87c5c8b43c6d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 11:02:03 +0000 Subject: [PATCH 06/21] fix(check): catch a nested export-mapping member at check time (MDL-MAP01) CI failed on PR #188 with FAIL (negative test unexpectedly passed): 927-mapping-nested-member-path.fail.mdl `make check-mdl` runs plain `mxcli check` and expects every .fail.mdl to fail THERE, but both #927 refusals lived in the executor, so check exited 0. The Makefile's own note above check-mdl describes this trap exactly -- a working rule made to look regressed, from #891 and #892 -- and prescribes the split, which is what this does: * The EXPORT refusal is purely syntactic: whether a member contains "/" is visible in the statement, no project required. It moves into the no-project pass as MDL-MAP01, beside ValidateGrantRoles, which is there for the same reason (#836). So `mxcli check` now tells the author before a script starts writing, and the negative fixture legitimately fails check. * The array-crossing refusal genuinely needs the project's JSON structure to know an intermediate is 0..*, so it cannot fail check. It is out of the .fail.mdl and covered by the array case in TestJSONSchemaIndex_ResolvePath, which is what the Makefile asks for. The fixture says so, rather than leaving the omission to be rediscovered. The executor keeps its refusal for a statement that reaches exec another way; both now raise the same message through nestedExportMemberError, so the author sees one wording wherever the statement is stopped. Tests mirror the grant rule's: fires without a project, ignores an IMPORT mapping (where collapsing is the supported feature this PR adds), and does not mistake an object element's `Assoc/Entity` -- a `/` on the other side of the mapping -- for a nested member. Verified: `make check-mdl` exits 0, 76 unit packages green, gofmt clean. Also merges origin/main, which had moved on and left the PR unmergeable. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- .../927-mapping-nested-member-path.fail.mdl | 50 ++++------ mdl/executor/cmd_export_mappings.go | 17 +++- .../validate_export_mapping_members.go | 71 +++++++++++++++ .../validate_export_mapping_members_test.go | 91 +++++++++++++++++++ mdl/executor/validate_program.go | 6 ++ 5 files changed, 200 insertions(+), 35 deletions(-) create mode 100644 mdl/executor/validate_export_mapping_members.go create mode 100644 mdl/executor/validate_export_mapping_members_test.go diff --git a/mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl b/mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl index 57fce336b..a3dc0dd7d 100644 --- a/mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl +++ b/mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl @@ -1,30 +1,30 @@ -- ============================================================================ --- Issue #927: the two nested-member shapes Mendix rejects +-- Issue #927: an export mapping cannot reach a nested member -- ============================================================================ -- --- Both are refused by `mxcli exec` rather than written, because each produces a --- model that is valid MDL, passes `mxcli check`, and fails only in the build. --- Each refusal below was measured on mxbuild 11.13 by patching the path into a --- stored mapping and running `mx check`. +-- Refused rather than written, because it would otherwise be valid MDL that +-- passes `mxcli check` and fails only in the build: -- --- 1. An EXPORT mapping reaching a nested member. --- CE5015 "There is no child mapping matching schema element '(Object)'" --- Three-way control: the same collapsed member in an IMPORT mapping is 0 --- errors, and the same export mapping with only top-level members is 0 --- errors. An export has to produce the intermediate node, so something --- must map it -- give `customer` its own element with an association. +-- CE5015 "There is no child mapping matching schema element '(Object)'" -- --- 2. An import member whose path crosses a 0..* element. --- CE0256 "Between value mapping 'sku' and parent element '(Object)' is a --- schema element with wrong occurrence (0..*)" --- A value cannot be pulled through many items, so an array level needs its --- own entity regardless. +-- Measured on mxbuild 11.13 with a three-way control: the same collapsed member +-- in an IMPORT mapping is 0 errors, and the same export mapping with only +-- top-level members is 0 errors. An export mapping has to PRODUCE the +-- intermediate node, so something must map it -- give `customer` its own +-- element with an association. -- --- Usage (this file is expected to FAIL -- .fail.mdl): --- mxcli exec mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl -p app.mpr --- Expect a refusal naming CE5015 or CE0256; nothing is written. +-- The OTHER refused shape is not here, deliberately. An import member whose +-- path crosses a 0..* element is CE0256 ("Between value mapping 'sku' and parent +-- element '(Object)' is a schema element with wrong occurrence"), but deciding +-- that needs the project's JSON structure, so `mxcli check` cannot reach it and +-- naming such a repro .fail.mdl reports "negative test unexpectedly passed" -- +-- a working rule made to look regressed. It is covered by the array case in +-- TestJSONSchemaIndex_ResolvePath instead, which is what the Makefile's note +-- above check-mdl asks for. -- --- Requires the module from 927-mapping-nested-member-path.mdl. +-- Usage (this file is expected to FAIL -- .fail.mdl): +-- mxcli check mdl-examples/bug-tests/927-mapping-nested-member-path.fail.mdl +-- Expect the CE5015 refusal; no project needed. -- ============================================================================ -- (1) export mapping cannot collapse levels @@ -37,13 +37,3 @@ create or modify export mapping BugTest927.EMM_Collapsed } }; / - --- (2) an import member cannot cross an array -create or modify import mapping BugTest927.IMM_AcrossArray - with json structure BugTest927.JSON_Order -{ - create BugTest927.Order { - CustomerName = items/sku - } -}; -/ diff --git a/mdl/executor/cmd_export_mappings.go b/mdl/executor/cmd_export_mappings.go index 1115b1e95..321912062 100644 --- a/mdl/executor/cmd_export_mappings.go +++ b/mdl/executor/cmd_export_mappings.go @@ -284,11 +284,7 @@ func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingEle // Refused here rather than written, because the model would be valid MDL, // pass `mxcli check`, and fail only in the build. (issue #927) if strings.Contains(def.JsonName, "/") { - return nil, fmt.Errorf("export mapping member %q: an export mapping cannot reach a nested "+ - "member directly — Mendix rejects it with CE5015 because the intermediate object has "+ - "nothing producing it. Give %q its own element: Association/Module.Entity as %s { ... }. "+ - "(Collapsing levels this way works for IMPORT mappings, which only read.)", - def.JsonName, strings.Split(def.JsonName, "/")[0], strings.Split(def.JsonName, "/")[0]) + return nil, nestedExportMemberError(def.JsonName) } jsElem = idx.resolve(parentPath, def.JsonName) } @@ -459,3 +455,14 @@ func execDropExportMapping(ctx *ExecContext, s *ast.DropExportMappingStmt) error } return nil } + +// nestedExportMemberError is shared by the check-time guard and the executor so +// the author sees one message wherever the statement is stopped. +func nestedExportMemberError(member string) error { + level := strings.Split(member, "/")[0] + return mdlerrors.NewValidationf("export mapping member %q: an export mapping cannot reach a nested "+ + "member directly — Mendix rejects it with CE5015 because the intermediate object has nothing "+ + "producing it. Give %q its own element: Association/Module.Entity as %s { ... }. "+ + "(Collapsing levels this way works for IMPORT mappings, which only read.)", + member, level, level) +} diff --git a/mdl/executor/validate_export_mapping_members.go b/mdl/executor/validate_export_mapping_members.go new file mode 100644 index 000000000..6bfc9d9b6 --- /dev/null +++ b/mdl/executor/validate_export_mapping_members.go @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time (no-project) validation for export mapping value members. +// +// An IMPORT mapping can collapse levels — `Attr = customer/name` reads a leaf +// several levels below the object element it belongs to, with no entity for the +// levels in between. An EXPORT mapping cannot: it has to PRODUCE the +// intermediate node, so something must map it. +// +// Measured on mxbuild 11.13 with a three-way control: the same collapsed member +// in an import mapping is 0 errors, the same export mapping with only top-level +// members is 0 errors, and the collapsed export is CE5015 "There is no child +// mapping matching schema element". See issue #927. +package executor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// ValidateExportMappingMembers reports (MDL-MAP01) an export mapping value whose +// member is a `/`-separated path. +// +// This lives in the no-project pass rather than the --references pass on +// purpose, for the same reason as ValidateGrantRoles: the answer is in the +// statement itself, so requiring -p would withhold something mxcli can always +// tell the author. It also means a plain `mxcli check` catches it — which is +// what lets the negative fixture be a .fail.mdl at all. +func ValidateExportMappingMembers(prog *ast.Program) []linter.Violation { + var out []linter.Violation + for _, stmt := range prog.Statements { + s, ok := stmt.(*ast.CreateExportMappingStmt) + if !ok || s.RootElement == nil { + continue + } + for _, member := range nestedExportMembers(s.RootElement.Children) { + level := strings.Split(member, "/")[0] + out = append(out, linter.Violation{ + RuleID: "MDL-MAP01", + Severity: linter.SeverityError, + Message: nestedExportMemberError(member).Error(), + Location: linter.Location{ + Module: s.Name.Module, + DocumentType: "export mapping", + DocumentName: s.Name.Name, + }, + Suggestion: "Give " + level + " its own element: Association/Module.Entity as " + level + " { ... }", + }) + } + } + return out +} + +// nestedExportMembers collects every `/`-separated VALUE member in the tree. An +// object element's JsonName is the key it maps, which is always a direct child, +// so only value elements (no Entity) are considered. +func nestedExportMembers(elems []*ast.ExportMappingElementDef) []string { + var out []string + for _, e := range elems { + if e == nil { + continue + } + if e.Entity == "" && strings.Contains(e.JsonName, "/") { + out = append(out, e.JsonName) + } + out = append(out, nestedExportMembers(e.Children)...) + } + return out +} diff --git a/mdl/executor/validate_export_mapping_members_test.go b/mdl/executor/validate_export_mapping_members_test.go new file mode 100644 index 000000000..c67ad95fe --- /dev/null +++ b/mdl/executor/validate_export_mapping_members_test.go @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// ValidateExportMappingMembers is what a plain `mxcli check` calls. It has to +// fire WITHOUT a project — the answer is in the statement — which is also what +// lets the negative fixture be a .fail.mdl at all: `make check-mdl` runs check +// with no -p, and a guard that only fires at exec reports "negative test +// unexpectedly passed" instead. That is how this rule came to exist (#927). +func TestValidateExportMappingMembers_ReportsNestedMemberWithoutProject(t *testing.T) { + prog, errs := visitor.Build(`create export mapping M.EMM_Order + with json structure M.JSON_Order +{ + M.Order { + orderId = OrderId, + customer/name = CustomerName + } +};`) + if len(errs) > 0 { + t.Fatalf("parse error: %v", errs[0]) + } + + got := ValidateExportMappingMembers(prog) + if len(got) != 1 { + t.Fatalf("got %d violations, want 1: %#v", len(got), got) + } + if got[0].RuleID != "MDL-MAP01" { + t.Errorf("RuleID = %q, want MDL-MAP01", got[0].RuleID) + } + if !strings.Contains(got[0].Message, "CE5015") { + t.Errorf("message should name the build error it prevents, got %q", got[0].Message) + } + if !strings.Contains(got[0].Message, "customer/name") { + t.Errorf("message should quote the offending member, got %q", got[0].Message) + } + if !strings.Contains(got[0].Suggestion, "customer") { + t.Errorf("suggestion should name the level needing its own element, got %q", got[0].Suggestion) + } +} + +// The controls. A top-level member is the ordinary case; an object element's +// `Assoc/Entity` is a `/` on the OTHER side of the mapping and must not be +// mistaken for a nested member — that pair is what the rule has to tell apart. +func TestValidateExportMappingMembers_CleanCases(t *testing.T) { + cases := []struct { + name string + src string + }{ + { + name: "top-level members only", + src: `create export mapping M.EMM_A with json structure M.JS +{ M.Order { orderId = OrderId, total = Total } };`, + }, + { + name: "an association element is not a nested member", + src: `create export mapping M.EMM_B with json structure M.JS +{ M.Order { orderId = OrderId, M.Order_Line/M.Line as lines { sku = Sku } } };`, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + prog, errs := visitor.Build(tc.src) + if len(errs) > 0 { + t.Fatalf("parse error: %v", errs[0]) + } + if got := ValidateExportMappingMembers(prog); len(got) != 0 { + t.Errorf("MDL-MAP01 fired on a valid export mapping: %#v", got) + } + }) + } +} + +// An IMPORT mapping may collapse levels, so the rule must not touch it. Getting +// this wrong would refuse the very feature #927 adds. +func TestValidateExportMappingMembers_IgnoresImportMappings(t *testing.T) { + prog, errs := visitor.Build(`create import mapping M.IMM_Order with json structure M.JS +{ create M.Order { OrderId = orderId, CustomerName = customer/name } };`) + if len(errs) > 0 { + t.Fatalf("parse error: %v", errs[0]) + } + if got := ValidateExportMappingMembers(prog); len(got) != 0 { + t.Errorf("MDL-MAP01 fired on an IMPORT mapping, where collapsing is supported: %#v", got) + } +} diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index 588c6b540..fd3aa6bbd 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -146,6 +146,12 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { // under --references, where it would only fire with -p (#836). violations = append(violations, ValidateGrantRoles(prog)...) + // Flag an export mapping value whose member is a nested path — an export has + // to produce the intermediate node, so Mendix rejects it with CE5015. The + // answer is in the statement, so it runs here rather than under --references + // (#927). + violations = append(violations, ValidateExportMappingMembers(prog)...) + // Flag a REST client operation whose Body/Response mapping clause has no // `{ ... }` body — Mendix cannot reference a mapping document from an // operation, so the mapping would be dropped in silence (#843). From 55f77c8416e6cdfbe86b2156adf1280a6e4a6765 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 11:11:12 +0000 Subject: [PATCH 07/21] fix(run-local): use an mxbuild this host can execute, and say so when none can MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On macOS and Windows the local loop executed whatever Linux binary the cache held: fork/exec ~/.mxcli/mxbuild/11.12.0/modeler/mxbuild: exec format error MxBuildCDNURL branches on GOARCH and never on GOOS, and both URLs are Linux tarballs — Mendix ships the macOS mxbuild inside Studio Pro, not on the CDN. So an arm64 Mac caches a Linux *aarch64* ELF: the architecture matches, which is why nothing notices until exec. The rule was already in the codebase twice. `setup mxbuild` asks NativeMxBuildForSetup, and `docker build` resolves Studio Pro before the cache — its comment even says "On Windows, CDN downloads are Linux binaries". The local loop did neither: runlocal.go called DownloadMxBuild directly and StartServe looked only at the cache. Three changes: * The local path resolves through NativeMxBuildForSetup, so Studio Pro wins over the cache on any non-Linux host. Linux is unchanged. * LocalRunOptions.MxBuildPath is honoured. It was documented as an override and never read for the serve binary, so a user hitting this had no way out. * A magic-byte check (ELF / Mach-O incl. fat / PE) refuses a foreign binary before exec, naming Studio Pro and --mxbuild-path instead of "exec format error". An unrecognised format is allowed through: a shell wrapper has no magic, and guessing would block a working setup. The download is deliberately NOT blocked on Windows — the cache holds a Linux binary there on purpose, for Docker builds. The fix belongs at resolve and exec time. The host OS is injected into the helpers rather than read from runtime.GOOS at the point of use, because this bug is platform-specific and is otherwise unreachable from a Linux runner. Reverting makes the explicit-path test fail by spending 34 seconds downloading from the CDN, which is the ignored override made visible. Not verified: no macOS host was available, so resolveStudioProDirMacOS itself is still covered only by its existing tests. The failure mode was reproduced on Linux by planting a Mach-O at the cache path, which reproduces the reported message verbatim. --- .claude/skills/fix-issue.md | 1 + .claude/skills/mendix/run-local.md | 19 +++ cmd/mxcli/docker/mxbuild_platform.go | 131 ++++++++++++++++++++ cmd/mxcli/docker/mxbuild_platform_test.go | 144 ++++++++++++++++++++++ cmd/mxcli/docker/mxserve.go | 6 + cmd/mxcli/docker/runlocal.go | 5 +- 6 files changed, 305 insertions(+), 1 deletion(-) create mode 100644 cmd/mxcli/docker/mxbuild_platform.go create mode 100644 cmd/mxcli/docker/mxbuild_platform_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 346206e42..be138156d 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -564,3 +564,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli check -p app.mpr` passes a microflow containing `if $obj/Status = 'Open'` — comparing an enumeration attribute to a string literal, the first example in the type-checking proposal and the shape people actually write. The same mistake written as a create or change member *is* caught, which makes the gap look arbitrary | Two causes, both invisible. (a) **`inferKind` returned `KindUnknown` for every `AttributePathExpr`**, so `$obj/Attr` typed to nothing and every rule downstream of it stayed quiet. `exprcheck.Scope` is `Lookup(name) (TypeKind, bool)` — it can say "$P is an Object" but not *which* entity, so there was nowhere to put the answer; the adapter computed a variable→entity map (`buildVarEntityScope`) and used it only to label a slot path, never passing it to the checker, and it covered body-introduced variables but **not parameters** — the ordinary case. (b) Even resolved, nothing fired: **E001 keys off the SLOT** (`CreateItem.Value:Entity.Attr`), which exists for an assignment and not for a comparison | `mdl/exprcheck/interfaces.go` (`EntityScope`, `Context.Entities`), `mdl/exprcheck/parser.go` (`attributePathKind`, `pathTargetEntity`, `checkEnumComparedToString`), `mdl/exprcheck/adapters/` (`walkFlow`, `addParamEntities`, `entityScope`), `mdl/exprcatalog/` (`AssociationTarget`) | Put the object side in its **own seam** (`EntityScope`: `VariableEntity` + `AssociationTarget`) beside `Scope` rather than widening either it or `CatalogReader` — `CatalogReader`'s shape is what stays re-syncable from the upstream fork. **A multi-hop expression path is not an XPath path**: XPath spells the intermediate entity (`[Assoc/Entity/Attr]`) so a walk can read it off, while an expression does not (`$O/Mod.Assoc/Attr`), so every hop must resolve through the association index. For (b), emit the **same code and message** from the comparison as from the slot — one defect should not have two names depending on where it was spotted. **Generalisable**: when a checker is silent, separate "did the input resolve" from "is there a rule for this shape" before touching either — here resolution and detection were both missing, and fixing only one would have looked like the fix failed. Verify with a **probe corpus that actually exercises the construct**: 21 microflows described back to MDL contained 40 attribute-path lines including an association hop, and were clean before and after, which is what makes "no false positives" mean something. Controls: path→`KindUnknown` restored, `Entities` dropped from the Context, and `addParamEntities` removed each fail a distinct test. **Still open**: a *terminal* association step (`$Order/Mod.Order_Lines`) types to unknown — Object vs List depends on the association's kind and direction, and guessing costs false positives | | `describe microflow` renders a REST call's result as `returns String` when the activity actually stores the response in a **file document**, and the describe → exec round trip then writes String back — retyping the activity with `mx check` reporting nothing either way. On the legacy engine the `$var =` output variable disappears too | MDL had no syntax for the case, `sdk/microflows` had no FileDocument variant, and both readers fell through to String: the legacy `parseResultHandling` had no `FileDocument` case (so the whole handling read back nil, taking the output-variable fallback with it), while modelsdk's `restResultHandlingFromRaw` read `VariableType.Entity` and then discarded it, matching only the literal `System.HttpResponse`. Mendix stores all five variants as ONE `Microflows$ResultHandling` carrying a `VariableType`, discriminated by the `ResultHandlingType` enum | `mdl/grammar/domains/MDLMicroflow.g4` (`restCallReturnsClause`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `sdk/microflows/microflows_actions.go`, `sdk/mpr/parser_microflow_actions.go` + `writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go` + `microflow_write.go`, `mdl/executor/cmd_microflows_builder_calls.go` + `cmd_microflows_format_action.go`, rule in `validate_microflow_rest.go` (MDL064) | Add `returns Module.Entity` as the LAST alternative so the keyword forms still win, and read Mendix's own `ResultHandlingType` discriminator rather than inferring from `VariableType` (keep the inference as a fallback — the property is `omitempty`). **Delete the silent `String` fallbacks**: an unreconstructable result handling must describe as something the parser REJECTS, or a reader gap becomes a model rewrite (ADR-0005). Mendix constraints worth knowing before designing syntax here: the BASE `System.FileDocument` is rejected as a return type (**CE0362**) so a specialization is mandatory, and HttpResponse **cannot** be specialized (**CE1540** allows only User, FileDocument, Image, Paging) — which is why `returns response` needed no change and the reporter's suggested "HttpResponse specialization" does not exist. Cross-engine round trip (`gateEngines`) is what proves it: the two readers failed differently and each looked fine from inside itself. Issue #922 | | The nightly fails on ONE Mendix matrix version only, in `TestMxCheck_DoctypeScripts`, with `Execution error: this project does not store the model setting ` — while the same script passes on every newer version | The example script set a model setting that version does not have. Measured: a blank 10.24 stores 11 model settings and a blank 11.6.6 stores 12, `DecimalScale` being the only difference. mxcli's refusal is CORRECT — Studio Pro will not open a model carrying a property its version does not define, and mxbuild does not catch it — so the bug is the ungated example, not the guard. The refusal covers the WHOLE statement, so one version-specific setting takes every portable setting in the same `alter` down with it | `mdl-examples/doctype-tests/14-project-settings-examples.mdl`; guard in `mdl/executor/doctype_version_gating_test.go` | Split the version-specific setting into its own statement inside a `-- @version: N.N+` section, closed with `-- @version: any`. **Put the `/** */` doc comment INSIDE the gated section**: a block comment is a documentation comment bound to the statement after it, so gating the statement while leaving the comment outside orphans it and the script dies with `no viable alternative at input '/**...'` — reported at the NEXT statement, tens of lines away, which reads like an unrelated syntax error. `--` line comments are free-standing and safe either side. Isolate which setting is at fault by exec'ing them one at a time against a blank project of that version (`mx create-project` in a SHORT path — a long one dies with PathTooLongException). `TestDoctypeScriptsParseAfterVersionFiltering` now parses every doctype script under each nightly matrix version without needing mxbuild, so a mis-gated script fails in seconds on push instead of hours later in one nightly job | +| `run --local` / `test --local` on macOS (or Windows) die with a raw `fork/exec ~/.mxcli/mxbuild//modeler/mxbuild: exec format error`, while `setup mxbuild` and `docker build` correctly use Studio Pro's binary. `--mxbuild-path` does not help | `MxBuildCDNURL` branches on **GOARCH only, never GOOS** — both URLs are Linux tarballs (Mendix ships macOS mxbuild inside Studio Pro, not on the CDN), so an arm64 Mac caches a Linux *aarch64* ELF: the arch matches, so nothing notices until exec. `docker build` resolves via `resolveMxBuild` (PATH → Studio Pro → known locations → cache) and `setup mxbuild` via `NativeMxBuildForSetup`, but the local loop called `DownloadMxBuild` directly at `runlocal.go` and `StartServe` looked only at the cache. `LocalRunOptions.MxBuildPath` was documented as an override and never read for the serve binary, so there was no workaround either | `cmd/mxcli/docker/mxbuild_platform.go` (new: `ResolveMxBuildForLocal`, `binaryOS`, `verifyRunsHere`) + `runlocal.go` (the `DownloadMxBuild` call) + `mxserve.go` (`StartServe`) | Reuse the rule the codebase already had rather than inventing one: `NativeMxBuildForSetup(goos, version)` returns Studio Pro's path, or "" meaning "Linux, download is fine", or an error plus guidance. Order: explicit path → (non-Linux) Studio Pro → cache/CDN. **Do not stop the download on Windows** — the cache holds a Linux binary there deliberately, for Docker builds; the fix belongs at resolve/exec time. Guard exec with a magic-byte check (ELF / Mach-O incl. fat / PE) and let an unrecognised format through, since a shell wrapper has no magic and refusing on a guess would block a working setup. **Inject `goos` into the helpers** (`resolveMxBuildForLocalOn`, `verifyRunsOn`): the whole bug is platform-specific and is otherwise untestable from a Linux runner. Repro without a Mac: plant a Mach-O magic at the cache path and call `StartServe` — reproduces the reporter's message verbatim. Tests `TestBinaryOS`, `TestVerifyRunsOn_LinuxBinaryOnMac`, `TestResolveMxBuildForLocal_*`; reverting makes `ExplicitPathWins` fail **by downloading from the CDN for 34s**, which is the ignored override made visible. **Unverified:** no macOS host was available, so the Studio Pro discovery path itself (`resolveStudioProDirMacOS`) is exercised only by its own existing tests. Issue #916 | diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index 60ea7830a..fed17bca7 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -58,6 +58,25 @@ association catalog only at startup; behavioural changes are hot-reloaded. createdb -h 127.0.0.1 -U mendix "$(basename app.mpr .mpr | tr '[:upper:]' '[:lower:]')" ``` +### Which mxbuild the loop uses + +- **Linux** — the CDN download cached at `~/.mxcli/mxbuild//`, as before. +- **macOS / Windows** — **Studio Pro's bundled mxbuild**, resolved before the cache. + The Mendix CDN publishes **Linux archives only** (the URL varies by architecture, + not by OS), so a cached download on a Mac is a Linux `aarch64` ELF — the arch + matches, which is why it looks fine until exec. +- `--mxbuild-path` overrides both, and is now honoured by the local loop (it used + to be documented and ignored — #916). + +If nothing runnable is found, the command says so up front instead of failing with +`fork/exec …: exec format error`: + +``` +mxbuild from the Mendix CDN is a Linux binary and cannot run natively on darwin + Install Mendix Studio Pro 11.12.0 and use its bundled mx … + Or point mxcli at it explicitly with --mxbuild-path. +``` + ## The intended loop ```bash diff --git a/cmd/mxcli/docker/mxbuild_platform.go b/cmd/mxcli/docker/mxbuild_platform.go new file mode 100644 index 000000000..80bc14768 --- /dev/null +++ b/cmd/mxcli/docker/mxbuild_platform.go @@ -0,0 +1,131 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "io" + "os" + "runtime" +) + +// The Mendix CDN publishes Linux mxbuild archives only — MxBuildCDNURL branches +// on GOARCH, never GOOS, so on an arm64 Mac it fetches a Linux *aarch64* ELF. +// The architecture matches, which is why nothing notices until exec: +// +// fork/exec ~/.mxcli/mxbuild/11.12.0/modeler/mxbuild: exec format error +// +// `setup mxbuild` has always known this (NativeMxBuildForSetup) and `docker +// build` resolves Studio Pro before the cache (resolveMxBuild). The local loop +// did neither: it called DownloadMxBuild directly, so on macOS and Windows it +// executed whatever Linux binary the cache held. (issue #916) + +// ResolveMxBuildForLocal picks the mxbuild that `run --local` / `test --local` +// can actually execute on this host, and reports why when none can. +// +// Order, and why: +// +// 1. An explicit --mxbuild-path wins. It was documented as an override and was +// silently ignored by the local path, so a user hitting the platform +// mismatch had no way out. +// 2. On a non-Linux host, Studio Pro BEFORE the cache. The cache may legitimately +// hold a Linux binary — Windows keeps one for Docker builds — and on macOS it +// is exactly what a previous `run --local` downloaded. Preferring it is the bug. +// 3. Otherwise (Linux) the cache or a CDN download, unchanged. +// +// Whatever is chosen is checked for executability on this host before it is +// handed back, so a stale or hand-placed foreign binary fails with an +// explanation rather than a raw exec error. +func ResolveMxBuildForLocal(explicitPath, version string, w io.Writer) (string, error) { + return resolveMxBuildForLocalOn(runtime.GOOS, explicitPath, version, w) +} + +// resolveMxBuildForLocalOn is ResolveMxBuildForLocal with the host OS injected, +// so the macOS and Windows branches are testable from any host — the platform +// mismatch this fixes cannot otherwise be exercised in CI. +func resolveMxBuildForLocalOn(goos, explicitPath, version string, w io.Writer) (string, error) { + if explicitPath != "" { + resolved, err := resolveMxBuild(explicitPath, version) + if err != nil { + return "", err + } + if err := verifyRunsOn(resolved, goos); err != nil { + return "", err + } + return resolved, nil + } + + if native, guidance, err := NativeMxBuildForSetup(goos, version); err != nil { + // Non-Linux host with no Studio Pro: downloading would cache something + // that cannot run. Say so, with the same guidance `setup mxbuild` gives. + return "", fmt.Errorf("%w\n %s", err, guidance) + } else if native != "" { + if err := verifyRunsOn(native, goos); err != nil { + return "", err + } + return native, nil + } + + downloaded, err := DownloadMxBuild(version, w) + if err != nil { + return "", err + } + if err := verifyRunsOn(downloaded, goos); err != nil { + return "", err + } + return downloaded, nil +} + +// binaryOS reports the OS an executable is built for, from its magic bytes, or +// "" when the format is not recognised (a shell-script wrapper, or anything +// else this does not need to be clever about). +func binaryOS(path string) string { + f, err := os.Open(path) + if err != nil { + return "" + } + defer f.Close() + + var magic [4]byte + if n, err := f.Read(magic[:]); err != nil || n < 4 { + return "" + } + switch { + case magic[0] == 0x7F && magic[1] == 'E' && magic[2] == 'L' && magic[3] == 'F': + return "linux" + case magic[0] == 'M' && magic[1] == 'Z': + return "windows" + } + // Mach-O, thin (feedface/feedfacf) or fat (cafebabe), either endianness. + switch be := uint32(magic[0])<<24 | uint32(magic[1])<<16 | uint32(magic[2])<<8 | uint32(magic[3]); be { + case 0xFEEDFACE, 0xFEEDFACF, 0xCEFAEDFE, 0xCFFAEDFE, 0xCAFEBABE, 0xBEBAFECA: + return "darwin" + } + return "" +} + +// verifyRunsHere refuses a binary built for another operating system. +// +// The raw failure is `fork/exec …: exec format error`, which names neither the +// cause nor a remedy — the reporter of #916 had to run `file` on the cached +// binary to find out. An unrecognised format is allowed through: a script +// wrapper has no magic to read, and guessing wrong would block a working setup. +func verifyRunsHere(path string) error { return verifyRunsOn(path, runtime.GOOS) } + +// verifyRunsOn is verifyRunsHere with the host OS injected, so the macOS case +// can be asserted from a Linux CI runner. +func verifyRunsOn(path, goos string) error { + got := binaryOS(path) + if got == "" || got == goos { + return nil + } + msg := fmt.Sprintf("mxbuild at %s is a %s binary and cannot run on %s", path, got, goos) + if got == "linux" && goos != "linux" { + msg += "\n The Mendix CDN only publishes Linux mxbuild, so a cached download cannot run here." + + "\n Install Mendix Studio Pro for this project's Mendix version, or pass --mxbuild-path" + + "\n pointing at its bundled mxbuild." + } else { + msg += "\n Pass --mxbuild-path pointing at an mxbuild built for " + goos + "." + } + return fmt.Errorf("%s", msg) +} diff --git a/cmd/mxcli/docker/mxbuild_platform_test.go b/cmd/mxcli/docker/mxbuild_platform_test.go new file mode 100644 index 000000000..e0bc6dc18 --- /dev/null +++ b/cmd/mxcli/docker/mxbuild_platform_test.go @@ -0,0 +1,144 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" +) + +// writeBinary drops a file whose first bytes are magic, so the format checks can +// be exercised without shipping real executables. +func writeBinary(t *testing.T, dir, name string, magic []byte) string { + t.Helper() + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + p := filepath.Join(dir, name) + if err := os.WriteFile(p, append(magic, []byte("padding-padding")...), 0o755); err != nil { + t.Fatal(err) + } + return p +} + +var ( + elfMagic = []byte{0x7F, 'E', 'L', 'F'} + machoMagic = []byte{0xCF, 0xFA, 0xED, 0xFE} + peMagic = []byte{'M', 'Z', 0x90, 0x00} +) + +func TestBinaryOS(t *testing.T) { + dir := t.TempDir() + cases := []struct { + name string + magic []byte + want string + }{ + {"elf", elfMagic, "linux"}, + {"macho", machoMagic, "darwin"}, + {"macho-fat", []byte{0xCA, 0xFE, 0xBA, 0xBE}, "darwin"}, + {"pe", peMagic, "windows"}, + {"shell wrapper is not classified", []byte("#!/b"), ""}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := binaryOS(writeBinary(t, dir, c.name, c.magic)); got != c.want { + t.Errorf("binaryOS = %q, want %q", got, c.want) + } + }) + } + + t.Run("too short to classify", func(t *testing.T) { + p := filepath.Join(dir, "tiny") + if err := os.WriteFile(p, []byte{0x7F}, 0o755); err != nil { + t.Fatal(err) + } + if got := binaryOS(p); got != "" { + t.Errorf("binaryOS = %q, want empty for a 1-byte file", got) + } + }) +} + +// TestVerifyRunsOn_LinuxBinaryOnMac is the #916 failure itself: the cached CDN +// download is a Linux ELF, and macOS produced only `exec format error`. The +// message has to name the cause and a way out. +func TestVerifyRunsOn_LinuxBinaryOnMac(t *testing.T) { + p := writeBinary(t, t.TempDir(), "mxbuild", elfMagic) + + err := verifyRunsOn(p, "darwin") + if err == nil { + t.Fatal("a Linux binary must be refused on darwin") + } + for _, want := range []string{"linux binary", "cannot run on darwin", "Studio Pro", "--mxbuild-path"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got:\n%v", want, err) + } + } +} + +func TestVerifyRunsOn_MatchingAndUnknown(t *testing.T) { + dir := t.TempDir() + if err := verifyRunsOn(writeBinary(t, dir, "native", elfMagic), "linux"); err != nil { + t.Errorf("a matching binary must be accepted: %v", err) + } + // A wrapper script has no magic to read; refusing it would block a working + // setup on a guess. + if err := verifyRunsOn(writeBinary(t, dir, "wrapper", []byte("#!/b")), "darwin"); err != nil { + t.Errorf("an unclassifiable file must be allowed through: %v", err) + } +} + +// TestResolveMxBuildForLocal_ExplicitPathWins covers the half that left the +// reporter with no workaround: --mxbuild-path was documented as an override and +// the local path ignored it, calling DownloadMxBuild unconditionally. +func TestResolveMxBuildForLocal_ExplicitPathWins(t *testing.T) { + dir := t.TempDir() + explicit := writeBinary(t, dir, "mxbuild", elfMagic) + + got, err := resolveMxBuildForLocalOn("linux", explicit, "11.12.0", &bytes.Buffer{}) + if err != nil { + t.Fatalf("explicit path rejected: %v", err) + } + if got != explicit { + t.Errorf("resolved %q, want the explicit path %q", got, explicit) + } +} + +// TestResolveMxBuildForLocal_ExplicitPathMustRunHere — an override pointing at a +// foreign binary is refused rather than exec'd. +func TestResolveMxBuildForLocal_ExplicitPathMustRunHere(t *testing.T) { + explicit := writeBinary(t, t.TempDir(), "mxbuild", machoMagic) + + if _, err := resolveMxBuildForLocalOn("linux", explicit, "11.12.0", &bytes.Buffer{}); err == nil { + t.Fatal("a darwin binary passed via --mxbuild-path must be refused on linux") + } +} + +// TestResolveMxBuildForLocal_NonLinuxWithoutStudioProRefuses pins the behaviour +// that could not be reached from CI before: on a host the CDN has no build for, +// resolution must fail with guidance instead of downloading a Linux binary and +// exec'ing it. No network is touched — NativeMxBuildForSetup returns the error +// before any download is attempted. +func TestResolveMxBuildForLocal_NonLinuxWithoutStudioProRefuses(t *testing.T) { + // A version no Studio Pro install will match, so the darwin branch reaches + // its "no native mxbuild" outcome on any machine. + var out bytes.Buffer + _, err := resolveMxBuildForLocalOn("darwin", "", "99.99.99", &out) + if err == nil { + t.Fatal("darwin without Studio Pro must refuse, not download a Linux binary") + } + for _, want := range []string{"Linux binary", "darwin"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got:\n%v", want, err) + } + } + if !strings.Contains(err.Error(), "--mxbuild-path") && !strings.Contains(err.Error(), "Studio Pro") { + t.Errorf("error should offer a way forward, got:\n%v", err) + } + if out.Len() > 0 { + t.Errorf("nothing should have been downloaded, but progress was written:\n%s", out.String()) + } +} diff --git a/cmd/mxcli/docker/mxserve.go b/cmd/mxcli/docker/mxserve.go index af78aeabb..dacf63da5 100644 --- a/cmd/mxcli/docker/mxserve.go +++ b/cmd/mxcli/docker/mxserve.go @@ -112,6 +112,12 @@ func StartServe(opts ServeOptions) (*ServeServer, error) { if err := verifyMxBuildCache(mxbuildPath); err != nil { return nil, err } + // The cache can hold a binary for another OS — the CDN only ships Linux, and + // Windows keeps one deliberately for Docker builds. Exec'ing it produces a + // bare "exec format error" naming neither cause nor remedy (#916). + if err := verifyRunsHere(mxbuildPath); err != nil { + return nil, err + } javaHome := opts.JavaHome if javaHome == "" { diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index 63dfd8f84..cff1e44bc 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -575,7 +575,10 @@ func RunLocal(opts LocalRunOptions) error { // 2. Ensure mxbuild + runtime are cached, and linked for the serve javac step. fmt.Fprintln(w, "Ensuring MxBuild and runtime are available...") - mxbuildPath, err := DownloadMxBuild(version, w) + // Resolve what this host can actually execute: Studio Pro before the cache + // on macOS/Windows, and honour --mxbuild-path, which the local path used to + // ignore (#916). + mxbuildPath, err := ResolveMxBuildForLocal(opts.MxBuildPath, version, w) if err != nil { return fmt.Errorf("setting up mxbuild: %w", err) } From f78d9f21e5f1dd2d8c8cfe63db510b6b9fe87d3a Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 11:24:54 +0000 Subject: [PATCH 08/21] fix(windows): append .exe to the java path, and find a per-user JDK MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Detection appended the .exe Windows needs; every consumer built the path by hand and did not. The path passed to mxbuild as --java-exe-path, and the one exec.Command runs to boot the runtime, were both "\bin\java" — so a correctly-detected JDK was handed on in a form that need not resolve, which from outside reads as "mxcli does not detect Java". Five sites built that join; they now share JavaExePath. The sixth would have repeated it. On the JDK search: "add Studio Pro's JDK" turned out to be a non-task. Mendix's install guide lists Eclipse Temurin JDK 21 as the prerequisite and installs it when absent — Studio Pro bundles no JDK of its own, so the existing Adoptium glob already IS Studio Pro's JDK, and inventing a Mendix\\jdk path would have been dead code. What was genuinely missing is the per-user install location (%LOCALAPPDATA%\Programs\…) that winget and the Temurin MSI can produce, which no Program Files glob reaches. The not-found error now lists every location searched, and says which JDK Mendix itself uses. "JDK 21 not found" alone sent a user reading mxcli's source to find out what it had looked at. Gradle needs no change and is worth recording: it ships inside the mxbuild bundle (modeler/tools/gradle) and mxbuild invokes it. mxcli never calls gradle, so a "Gradle missing" from a local run points at a foreign or incomplete mxbuild bundle rather than a missing system Gradle — the same root as the platform mismatch in #916. goos is injected into jdkSearchPathsFor and javaExeName because the only Windows CI job is the tunnel seam, scoped with -run: it compiles Windows code and executes almost none of it, which is how a missing .exe survives. Reverting the suffix fails TestJavaExeName. Not verified: no Windows host was available. This is a code-level fix with OS-injected tests, and the report came via a user relay rather than a reproducible case. --- .claude/skills/fix-issue.md | 1 + .claude/skills/mendix/run-local.md | 13 +++++ cmd/mxcli/docker/build.go | 2 +- cmd/mxcli/docker/detect.go | 45 ++++++++++++++--- cmd/mxcli/docker/javaexe.go | 36 ++++++++++++++ cmd/mxcli/docker/javaexe_test.go | 79 ++++++++++++++++++++++++++++++ cmd/mxcli/docker/localboot.go | 2 +- cmd/mxcli/docker/mxserve.go | 2 +- cmd/mxcli/docker/settle.go | 2 +- 9 files changed, 170 insertions(+), 12 deletions(-) create mode 100644 cmd/mxcli/docker/javaexe.go create mode 100644 cmd/mxcli/docker/javaexe_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index be138156d..eab338de9 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -565,3 +565,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `describe microflow` renders a REST call's result as `returns String` when the activity actually stores the response in a **file document**, and the describe → exec round trip then writes String back — retyping the activity with `mx check` reporting nothing either way. On the legacy engine the `$var =` output variable disappears too | MDL had no syntax for the case, `sdk/microflows` had no FileDocument variant, and both readers fell through to String: the legacy `parseResultHandling` had no `FileDocument` case (so the whole handling read back nil, taking the output-variable fallback with it), while modelsdk's `restResultHandlingFromRaw` read `VariableType.Entity` and then discarded it, matching only the literal `System.HttpResponse`. Mendix stores all five variants as ONE `Microflows$ResultHandling` carrying a `VariableType`, discriminated by the `ResultHandlingType` enum | `mdl/grammar/domains/MDLMicroflow.g4` (`restCallReturnsClause`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `sdk/microflows/microflows_actions.go`, `sdk/mpr/parser_microflow_actions.go` + `writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go` + `microflow_write.go`, `mdl/executor/cmd_microflows_builder_calls.go` + `cmd_microflows_format_action.go`, rule in `validate_microflow_rest.go` (MDL064) | Add `returns Module.Entity` as the LAST alternative so the keyword forms still win, and read Mendix's own `ResultHandlingType` discriminator rather than inferring from `VariableType` (keep the inference as a fallback — the property is `omitempty`). **Delete the silent `String` fallbacks**: an unreconstructable result handling must describe as something the parser REJECTS, or a reader gap becomes a model rewrite (ADR-0005). Mendix constraints worth knowing before designing syntax here: the BASE `System.FileDocument` is rejected as a return type (**CE0362**) so a specialization is mandatory, and HttpResponse **cannot** be specialized (**CE1540** allows only User, FileDocument, Image, Paging) — which is why `returns response` needed no change and the reporter's suggested "HttpResponse specialization" does not exist. Cross-engine round trip (`gateEngines`) is what proves it: the two readers failed differently and each looked fine from inside itself. Issue #922 | | The nightly fails on ONE Mendix matrix version only, in `TestMxCheck_DoctypeScripts`, with `Execution error: this project does not store the model setting ` — while the same script passes on every newer version | The example script set a model setting that version does not have. Measured: a blank 10.24 stores 11 model settings and a blank 11.6.6 stores 12, `DecimalScale` being the only difference. mxcli's refusal is CORRECT — Studio Pro will not open a model carrying a property its version does not define, and mxbuild does not catch it — so the bug is the ungated example, not the guard. The refusal covers the WHOLE statement, so one version-specific setting takes every portable setting in the same `alter` down with it | `mdl-examples/doctype-tests/14-project-settings-examples.mdl`; guard in `mdl/executor/doctype_version_gating_test.go` | Split the version-specific setting into its own statement inside a `-- @version: N.N+` section, closed with `-- @version: any`. **Put the `/** */` doc comment INSIDE the gated section**: a block comment is a documentation comment bound to the statement after it, so gating the statement while leaving the comment outside orphans it and the script dies with `no viable alternative at input '/**...'` — reported at the NEXT statement, tens of lines away, which reads like an unrelated syntax error. `--` line comments are free-standing and safe either side. Isolate which setting is at fault by exec'ing them one at a time against a blank project of that version (`mx create-project` in a SHORT path — a long one dies with PathTooLongException). `TestDoctypeScriptsParseAfterVersionFiltering` now parses every doctype script under each nightly matrix version without needing mxbuild, so a mis-gated script fails in seconds on push instead of hours later in one nightly job | | `run --local` / `test --local` on macOS (or Windows) die with a raw `fork/exec ~/.mxcli/mxbuild//modeler/mxbuild: exec format error`, while `setup mxbuild` and `docker build` correctly use Studio Pro's binary. `--mxbuild-path` does not help | `MxBuildCDNURL` branches on **GOARCH only, never GOOS** — both URLs are Linux tarballs (Mendix ships macOS mxbuild inside Studio Pro, not on the CDN), so an arm64 Mac caches a Linux *aarch64* ELF: the arch matches, so nothing notices until exec. `docker build` resolves via `resolveMxBuild` (PATH → Studio Pro → known locations → cache) and `setup mxbuild` via `NativeMxBuildForSetup`, but the local loop called `DownloadMxBuild` directly at `runlocal.go` and `StartServe` looked only at the cache. `LocalRunOptions.MxBuildPath` was documented as an override and never read for the serve binary, so there was no workaround either | `cmd/mxcli/docker/mxbuild_platform.go` (new: `ResolveMxBuildForLocal`, `binaryOS`, `verifyRunsHere`) + `runlocal.go` (the `DownloadMxBuild` call) + `mxserve.go` (`StartServe`) | Reuse the rule the codebase already had rather than inventing one: `NativeMxBuildForSetup(goos, version)` returns Studio Pro's path, or "" meaning "Linux, download is fine", or an error plus guidance. Order: explicit path → (non-Linux) Studio Pro → cache/CDN. **Do not stop the download on Windows** — the cache holds a Linux binary there deliberately, for Docker builds; the fix belongs at resolve/exec time. Guard exec with a magic-byte check (ELF / Mach-O incl. fat / PE) and let an unrecognised format through, since a shell wrapper has no magic and refusing on a guess would block a working setup. **Inject `goos` into the helpers** (`resolveMxBuildForLocalOn`, `verifyRunsOn`): the whole bug is platform-specific and is otherwise untestable from a Linux runner. Repro without a Mac: plant a Mach-O magic at the cache path and call `StartServe` — reproduces the reporter's message verbatim. Tests `TestBinaryOS`, `TestVerifyRunsOn_LinuxBinaryOnMac`, `TestResolveMxBuildForLocal_*`; reverting makes `ExplicitPathWins` fail **by downloading from the CDN for 34s**, which is the ignored override made visible. **Unverified:** no macOS host was available, so the Studio Pro discovery path itself (`resolveStudioProDirMacOS`) is exercised only by its own existing tests. Issue #916 | +| Windows: `run --local` behaves as though Java were undetected, and a local run can fail looking for Gradle | Two separate things. (a) **The `.exe` suffix**: `isJDK21` appended it, every CONSUMER did not — `--java-exe-path` handed to mxbuild and the path `exec.Command` runs to boot the runtime were both `\bin\java`, so a correctly-detected JDK was passed on in a form that need not resolve. Five sites built it by hand. (b) **Gradle is not mxcli's**: it ships inside the mxbuild bundle (`modeler/tools/gradle`, 8.5) and mxbuild invokes it — mxcli never calls gradle, so "Gradle missing" indicates a foreign/incomplete mxbuild bundle, usually #916 | `cmd/mxcli/docker/javaexe.go` (new `JavaExePath`) used from `mxserve.go`, `build.go`, `settle.go`, `localboot.go`, `detect.go`; `jdkSearchPathsFor` in `detect.go` | One helper, not five hand-built joins — the sixth call site would have repeated the bug. **Check the platform's own docs before adding a search path**: "add Studio Pro's JDK" turned out to be a non-task, because Mendix's install guide says Studio Pro installs **Eclipse Temurin 21** rather than bundling a JDK, so the existing Adoptium glob already IS Studio Pro's JDK; what was genuinely missing was the per-user `%LOCALAPPDATA%\Programs` install location. Inventing a `Mendix\\jdk` path would have been dead code. Make the not-found error list what was searched — "JDK 21 not found" alone sends a user reading mxcli's source. Inject `goos` (`jdkSearchPathsFor`, `javaExeName`): the only Windows CI job is the tunnel seam, scoped with `-run`, so it compiles Windows code and executes almost none of it — which is exactly how a missing `.exe` survives. **Unverified**: no Windows host; the fix is code-level with OS-injected tests. Reported via a user relay, not an issue | diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index fed17bca7..0194eaca1 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -77,6 +77,19 @@ mxbuild from the Mendix CDN is a Linux binary and cannot run natively on darwin Or point mxcli at it explicitly with --mxbuild-path. ``` +### Windows and macOS toolchain + +- **JDK 21** — Mendix Studio Pro does not bundle one; its installer puts **Eclipse + Temurin JDK 21** in the usual place, which is where mxcli looks (`Eclipse + Adoptium` / `Java` / `Microsoft` under both Program Files, plus the per-user + `%LOCALAPPDATA%\Programs\…` installs winget produces). `JAVA_HOME` wins over all + of them. When nothing is found the error now lists every location it searched. +- **Gradle** — bundled inside mxbuild (`modeler/tools/gradle`) and invoked by + mxbuild, not by mxcli. Studio Pro extracts its own copy to the parent of its + install directory (usually `C:\Program Files\Mendix`). A "Gradle not found" + from a local run therefore points at an incomplete or foreign mxbuild bundle — + check which mxbuild was resolved before looking for a system Gradle. + ## The intended loop ```bash diff --git a/cmd/mxcli/docker/build.go b/cmd/mxcli/docker/build.go index df63a5158..0d761089c 100644 --- a/cmd/mxcli/docker/build.go +++ b/cmd/mxcli/docker/build.go @@ -161,7 +161,7 @@ func Build(opts BuildOptions) error { fmt.Fprintf(w, "Running MxBuild (target=portable-app-package)...\n") fmt.Fprintf(w, " Output: %s\n", outputDir) - javaExePath := filepath.Join(javaHome, "bin", "java") + javaExePath := JavaExePath(javaHome) cmd := exec.Command(mxbuildPath, "--target=portable-app-package", diff --git a/cmd/mxcli/docker/detect.go b/cmd/mxcli/docker/detect.go index 9e84e095c..9a0cd6b68 100644 --- a/cmd/mxcli/docker/detect.go +++ b/cmd/mxcli/docker/detect.go @@ -324,7 +324,19 @@ func resolveJDK21() (string, error) { } } - return "", fmt.Errorf("JDK 21 not found; set JAVA_HOME or install JDK 21") + // Name what was searched. "JDK 21 not found" alone sent a Windows user + // hunting through mxcli's source for the detection logic; the list makes it + // obvious whether their JDK simply sits somewhere unlisted. + searched := jdkSearchPaths() + msg := "JDK 21 not found" + if jh := os.Getenv("JAVA_HOME"); jh != "" { + msg += fmt.Sprintf("\n JAVA_HOME is set to %s but is not a JDK 21 (java -version reports %s)", jh, javaVersionString(jh)) + } + if len(searched) > 0 { + msg += "\n Searched: " + strings.Join(searched, ", ") + } + msg += "\n Install Eclipse Temurin JDK 21 (what Mendix Studio Pro itself uses), or set JAVA_HOME to one." + return "", fmt.Errorf("%s", msg) } // resolveMacOSJavaHome uses /usr/libexec/java_home to find a JDK 21 on macOS. @@ -337,8 +349,20 @@ func resolveMacOSJavaHome() (string, error) { } // jdkSearchPaths returns OS-specific glob patterns for JDK installations. -func jdkSearchPaths() []string { - switch runtime.GOOS { +func jdkSearchPaths() []string { return jdkSearchPathsFor(runtime.GOOS) } + +// jdkSearchPathsFor is jdkSearchPaths with the OS injected, so the Windows list +// is assertable from a Linux runner — nothing in CI executes Windows code, which +// is how the java.exe suffix stayed broken. +// +// Studio Pro does not ship a JDK of its own to point at: Mendix's install guide +// lists "Eclipse Temurin JDK 21 (x64 or ARM64)" as the prerequisite and installs +// it if absent, so the Temurin location below IS Studio Pro's JDK. (Its Gradle +// 8.5 does live with Studio Pro — "extracted to the parent directory of the +// folder where Studio Pro is installed (usually C:\Program Files\Mendix)" — +// but mxbuild invokes that itself; mxcli never calls gradle.) +func jdkSearchPathsFor(goos string) []string { + switch goos { case "windows": var paths []string for _, dir := range windowsProgramDirs() { @@ -348,6 +372,14 @@ func jdkSearchPaths() []string { filepath.Join(dir, "Microsoft", "jdk-21*"), ) } + // Per-user installs (winget and the Temurin MSI both offer one) land + // outside Program Files, where nothing above would find them. + if local := os.Getenv("LOCALAPPDATA"); local != "" { + paths = append(paths, + filepath.Join(local, "Programs", "Eclipse Adoptium", "jdk-21*"), + filepath.Join(local, "Programs", "Microsoft", "jdk-21*"), + ) + } return paths case "darwin": return []string{ @@ -364,10 +396,7 @@ func jdkSearchPaths() []string { // isJDK21 checks if the given JAVA_HOME points to a JDK 21 installation. func isJDK21(javaHome string) bool { - javaBin := filepath.Join(javaHome, "bin", "java") - if runtime.GOOS == "windows" { - javaBin += ".exe" - } + javaBin := JavaExePath(javaHome) if _, err := os.Stat(javaBin); err != nil { return false } @@ -384,7 +413,7 @@ var jdk21VersionRegex = regexp.MustCompile(`version "21[\.\s"]`) // javaVersionString runs java -version and returns the output for diagnostics. func javaVersionString(javaHome string) string { - javaBin := filepath.Join(javaHome, "bin", "java") + javaBin := JavaExePath(javaHome) out, err := exec.Command(javaBin, "-version").CombinedOutput() if err != nil { return "(unknown)" diff --git a/cmd/mxcli/docker/javaexe.go b/cmd/mxcli/docker/javaexe.go new file mode 100644 index 000000000..aacc39e3b --- /dev/null +++ b/cmd/mxcli/docker/javaexe.go @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "path/filepath" + "runtime" +) + +// JavaExePath is /bin/java, with the .exe Windows needs. +// +// Detection got this right (isJDK21 appends it) and every consumer got it +// wrong: the path handed to mxbuild as --java-exe-path, and the one +// exec.Command runs to boot the runtime, were both built as a bare +// "…\bin\java". So on Windows a correctly-detected JDK was passed on in a form +// that need not resolve — which reads, from the outside, as "mxcli does not +// detect Java". +// +// One helper rather than five call sites, because the next one added would have +// made the same mistake. +func JavaExePath(javaHome string) string { + exe := "java" + if runtime.GOOS == "windows" { + exe += ".exe" + } + return filepath.Join(javaHome, "bin", exe) +} + +// javaExeName is the java binary's file name for a given OS, so the Windows +// form is assertable from a Linux runner (nothing in CI executes Windows code). +func javaExeName(goos string) string { + if goos == "windows" { + return "java.exe" + } + return "java" +} diff --git a/cmd/mxcli/docker/javaexe_test.go b/cmd/mxcli/docker/javaexe_test.go new file mode 100644 index 000000000..79ca8f918 --- /dev/null +++ b/cmd/mxcli/docker/javaexe_test.go @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "path/filepath" + "runtime" + "strings" + "testing" +) + +// TestJavaExePath_UsesHostSuffix pins the bug this helper exists for: every +// consumer built "/bin/java" without the .exe Windows needs, while detection +// appended it — so a correctly-detected JDK was passed to mxbuild, and exec'd for +// the runtime, in a form that need not resolve. +func TestJavaExePath_UsesHostSuffix(t *testing.T) { + got := JavaExePath(filepath.Join("C:", "jdk-21")) + want := "java" + if runtime.GOOS == "windows" { + want = "java.exe" + } + if filepath.Base(got) != want { + t.Errorf("JavaExePath = %q, want it to end in %q", got, want) + } +} + +// TestJavaExeName covers the Windows form from any host. The suffix cannot be +// exercised on a Linux runner otherwise, which is why it stayed wrong. +func TestJavaExeName(t *testing.T) { + if got := javaExeName("windows"); got != "java.exe" { + t.Errorf("javaExeName(windows) = %q, want java.exe", got) + } + for _, goos := range []string{"linux", "darwin"} { + if got := javaExeName(goos); got != "java" { + t.Errorf("javaExeName(%s) = %q, want java", goos, got) + } + } +} + +// TestJdkSearchPathsFor_Windows asserts the list a Windows host searches, +// including the per-user install locations that Program Files globs miss. +// Studio Pro contributes no path of its own: it installs Eclipse Temurin, which +// the Adoptium entries already cover. +func TestJdkSearchPathsFor_Windows(t *testing.T) { + t.Setenv("PROGRAMFILES", `C:\Program Files`) + t.Setenv("LOCALAPPDATA", `C:\Users\dev\AppData\Local`) + + paths := jdkSearchPathsFor("windows") + if len(paths) == 0 { + t.Fatal("no JDK search paths for windows") + } + joined := strings.Join(paths, "|") + // Separator-agnostic: filepath.Join uses "/" on the Linux runner this test + // normally executes on. + for _, want := range []string{"Eclipse Adoptium", "Microsoft", "AppData", "Programs"} { + if !strings.Contains(joined, want) { + t.Errorf("windows search paths should include %q, got:\n%s", want, strings.Join(paths, "\n")) + } + } + for _, p := range paths { + if !strings.Contains(p, "jdk-21") { + t.Errorf("every pattern should pin JDK 21, got %q", p) + } + } +} + +func TestJdkSearchPathsFor_UnixHosts(t *testing.T) { + for _, goos := range []string{"linux", "darwin"} { + paths := jdkSearchPathsFor(goos) + if len(paths) == 0 { + t.Fatalf("no JDK search paths for %s", goos) + } + for _, p := range paths { + if !strings.Contains(p, "21") { + t.Errorf("%s: every pattern should pin JDK 21, got %q", goos, p) + } + } + } +} diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index 85d922b54..f01aa816a 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -459,7 +459,7 @@ func StartLocalRuntime(opts LocalRuntimeOptions) (*LocalRuntime, error) { // configuration up to but not including start. It is used both for the initial // boot and for a restart (config is per-process and must be re-applied). func (rt *LocalRuntime) spawnAndConfigure() error { - javaExe := filepath.Join(rt.opts.JavaHome, "bin", "java") + javaExe := JavaExePath(rt.opts.JavaHome) cmd := exec.Command(javaExe, rt.opts.jvmArgs()...) cmd.Dir = rt.opts.runtimeDir() cmd.Env = localRuntimeEnv(rt.opts) diff --git a/cmd/mxcli/docker/mxserve.go b/cmd/mxcli/docker/mxserve.go index dacf63da5..191e3098a 100644 --- a/cmd/mxcli/docker/mxserve.go +++ b/cmd/mxcli/docker/mxserve.go @@ -127,7 +127,7 @@ func StartServe(opts ServeOptions) (*ServeServer, error) { } javaHome = jh } - javaExe := filepath.Join(javaHome, "bin", "java") + javaExe := JavaExePath(javaHome) host := opts.Host if host == "" { diff --git a/cmd/mxcli/docker/settle.go b/cmd/mxcli/docker/settle.go index 60331528c..ec75ed096 100644 --- a/cmd/mxcli/docker/settle.go +++ b/cmd/mxcli/docker/settle.go @@ -42,7 +42,7 @@ func SettleGeneratedSources(projectPath, mxPath, version string, w io.Writer) er cmd := exec.Command(mxbuildPath, "--target=deploy", fmt.Sprintf("--java-home=%s", javaHome), - fmt.Sprintf("--java-exe-path=%s", filepath.Join(javaHome, "bin", "java")), + fmt.Sprintf("--java-exe-path=%s", JavaExePath(javaHome)), projectPath, ) cmd.Dir = filepath.Dir(projectPath) From c4055a4f8893d7fead9a90b9ec3baa58fe274626 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 11:42:19 +0000 Subject: [PATCH 09/21] fix(executor): count writes that land outside unit storage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The write-stats reporting judged a statement on unit writes alone, which was wrong for code actions: a JavaScript or Java action's *body* does not live in its unit — the unit carries the signature, the source lives in javascriptsource//actions/.js. Editing only the body elided the unit write, so the statement reported Unchanged javascript action: DT.JS_Ping while the user's edit had just landed in a file. That is a worse lie than the one the reporting was introduced to fix, and it was found by measuring the fix across document types rather than only on the nanoflow from the report. The generated source file now goes through javaactions.WriteSourceIfChanged, which reports whether it differed, and both engines fold that into WriteStats. Skipping an identical file is worth having on its own: an unconditional rewrite moved the file's mtime on every run, which git does not notice but an incremental build's caching does. Also fixes the page path, which printed "Created page X" on the replace path too, so re-running a script against an unchanged page claimed to create it every time. It now reports Replaced/Unchanged like every other document type. Deleting duplicate same-named pages is a real change that unit-write counting cannot see, so only a one-for-one replacement is eligible for the downgrade. Measured across document types on one project, changing one thing in each: enumeration 10/10 element identities kept, microflow 22/22, nanoflow 16/16, page 65/65, and the domain model 29 of 30 — the single new element being the attribute's type node, correctly unmatched because Integer and Long are different $Types. All 8 GUIDs survived, including the retyped attribute's. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- mdl/backend/modelsdk/backend.go | 21 ++++++++++++-- mdl/backend/modelsdk/java_write.go | 4 ++- mdl/backend/modelsdk/javascript_write.go | 4 ++- mdl/executor/cmd_pages_create_v3.go | 14 +++++++++- mdl/executor/report_mutation_test.go | 21 ++++++++++++++ modelsdk/mpr/writer_core.go | 21 +++++++------- sdk/javaactions/writefile.go | 35 ++++++++++++++++++++++++ sdk/mpr/writer_core.go | 27 ++++++++++-------- sdk/mpr/writer_javaactions.go | 12 ++++++-- sdk/mpr/writer_javascriptactions.go | 7 ++++- sdk/mpr/writer_units.go | 4 +-- 11 files changed, 138 insertions(+), 32 deletions(-) create mode 100644 sdk/javaactions/writefile.go diff --git a/mdl/backend/modelsdk/backend.go b/mdl/backend/modelsdk/backend.go index e7c06f747..a95145699 100644 --- a/mdl/backend/modelsdk/backend.go +++ b/mdl/backend/modelsdk/backend.go @@ -40,6 +40,12 @@ type Backend struct { reader *mmpr.Reader writer *mmpr.Writer path string + + // fileWrites counts writes that do not go through unit storage — the + // generated .java/.js source of a code action, whose body lives in + // javasource/ or javascriptsource/ rather than in its unit. They are folded + // into WriteStats so a body-only edit is not reported as "Unchanged". + fileWrites backend.WriteStats } // New constructs a modelsdk backend. @@ -60,10 +66,21 @@ func errUnimplemented(method string) error { // the connection open for both reads. func (b *Backend) WriteStats() backend.WriteStats { if b.writer == nil { - return backend.WriteStats{} + return b.fileWrites } offered, written := b.writer.WriteStats() - return backend.WriteStats{Offered: offered, Written: written} + return backend.WriteStats{ + Offered: offered + b.fileWrites.Offered, + Written: written + b.fileWrites.Written, + } +} + +// noteFileWrite records a non-unit write and whether it changed anything. +func (b *Backend) noteFileWrite(changed bool) { + b.fileWrites.Offered++ + if changed { + b.fileWrites.Written++ + } } // --- ConnectionBackend --- diff --git a/mdl/backend/modelsdk/java_write.go b/mdl/backend/modelsdk/java_write.go index 76b08248a..e22b3faea 100644 --- a/mdl/backend/modelsdk/java_write.go +++ b/mdl/backend/modelsdk/java_write.go @@ -101,7 +101,9 @@ func (b *Backend) WriteJavaSourceFile(moduleName, actionName string, javaCode st return fmt.Errorf("WriteJavaSourceFile: create dir: %w", err) } source := javaactions.GenerateSource(moduleName, actionName, javaCode, params, returnType, extraImports, extraCode) - if err := os.WriteFile(filepath.Join(javaDir, actionName+".java"), []byte(source), 0o644); err != nil { + changed, err := javaactions.WriteSourceIfChanged(filepath.Join(javaDir, actionName+".java"), source) + b.noteFileWrite(changed) + if err != nil { return fmt.Errorf("WriteJavaSourceFile: write: %w", err) } return nil diff --git a/mdl/backend/modelsdk/javascript_write.go b/mdl/backend/modelsdk/javascript_write.go index d093a3b61..5d6b0b220 100644 --- a/mdl/backend/modelsdk/javascript_write.go +++ b/mdl/backend/modelsdk/javascript_write.go @@ -172,7 +172,9 @@ func (b *Backend) WriteJavaScriptSourceFile(moduleName, actionName string, jsCod return fmt.Errorf("WriteJavaScriptSourceFile: create dir: %w", err) } source := javaactions.GenerateJavaScriptSource(actionName, jsCode, params, returnType) - if err := os.WriteFile(filepath.Join(dir, actionName+".js"), []byte(source), 0o644); err != nil { + changed, err := javaactions.WriteSourceIfChanged(filepath.Join(dir, actionName+".js"), source) + b.noteFileWrite(changed) + if err != nil { return fmt.Errorf("WriteJavaScriptSourceFile: write: %w", err) } return nil diff --git a/mdl/executor/cmd_pages_create_v3.go b/mdl/executor/cmd_pages_create_v3.go index 7427a0ffd..bb3d2fb18 100644 --- a/mdl/executor/cmd_pages_create_v3.go +++ b/mdl/executor/cmd_pages_create_v3.go @@ -132,7 +132,19 @@ func execCreatePageV3(ctx *ExecContext, s *ast.CreatePageStmtV3) error { // Invalidate hierarchy cache so the new page's container is visible invalidateHierarchy(ctx) - fmt.Fprintf(ctx.Output, "Created page %s\n", s.Name.String()) + // "Created" was printed for the replace path too, so re-running a script + // against an unchanged page claimed to have created it every time. Duplicate + // pages of the same name are deleted above, and a delete is a real change + // that unit-write counting does not see — so only a plain one-for-one + // replacement is eligible to be reported as unchanged. + switch { + case len(pagesToDelete) == 1: + ctx.ReportMutation("Replaced", "page %s", s.Name.String()) + case len(pagesToDelete) > 1: + fmt.Fprintf(ctx.Output, "Replaced page %s\n", s.Name.String()) + default: + fmt.Fprintf(ctx.Output, "Created page %s\n", s.Name.String()) + } return nil } diff --git a/mdl/executor/report_mutation_test.go b/mdl/executor/report_mutation_test.go index bc4033718..0d1b74503 100644 --- a/mdl/executor/report_mutation_test.go +++ b/mdl/executor/report_mutation_test.go @@ -128,3 +128,24 @@ func TestReportMutationIgnoresEarlierStatements(t *testing.T) { t.Errorf("reported %q — earlier statements' writes were counted against this one", got) } } + +// TestReportMutationCountsWritesOutsideUnitStorage pins the case that made this +// helper wrong on its first outing. A code action's body does not live in its +// unit — the unit carries the signature, the source lives in +// javascriptsource//actions/.js — so editing only the body elides +// the unit write. Judging on unit writes alone called that "Unchanged" while the +// user's edit had just landed in a file, which is a worse lie than the one this +// helper was written to fix. +func TestReportMutationCountsWritesOutsideUnitStorage(t *testing.T) { + ctx, mb, out := reportCtx(t) + + // Unit write offered and elided (the signature did not change), source file + // written (the body did). + mb.offer(1, 0) + mb.offer(1, 1) + ctx.ReportMutation("Modified", "javascript action: %s", "MxCore.JS_LoadAiAdvisor") + + if got := out.String(); got != "Modified javascript action: MxCore.JS_LoadAiAdvisor\n" { + t.Errorf("reported %q — a body-only edit must not read as unchanged", got) + } +} diff --git a/modelsdk/mpr/writer_core.go b/modelsdk/mpr/writer_core.go index 099123bde..a71e7bfef 100644 --- a/modelsdk/mpr/writer_core.go +++ b/modelsdk/mpr/writer_core.go @@ -43,19 +43,20 @@ type Writer struct { // want to batch many unit updates into a single transaction. sessionBuf func(unitID string, contents []byte) error - // unitsOffered / unitsWritten count what reached reconcileWithStored and how + // writesOffered / writesLanded count what reached reconcileWithStored and how // much of it survived no-op elision (ADR-0008). The executor reads them to - // tell "Modified X" from "Modified nothing, X was already in sync" — without - // them, re-running a script that changes nothing still announces a write for - // every statement, which is how the churn in #910 was misdiagnosed. - unitsOffered int - unitsWritten int + // tell "Modified X" from "X was already in sync" — without them, re-running a + // script that changes nothing still announces a write for every statement, + // which is how the churn in #910 was misdiagnosed. The backend adds its own + // non-unit writes (generated .java/.js source) to this total. + writesOffered int + writesLanded int } // WriteStats reports how many unit writes this session offered to storage and // how many were not elided as no-ops. func (w *Writer) WriteStats() (offered, written int) { - return w.unitsOffered, w.unitsWritten + return w.writesOffered, w.writesLanded } // SetSessionBuf installs a callback that intercepts every updateUnit call. @@ -543,15 +544,15 @@ func (w *Writer) updateUnit(unitID string, contents []byte) error { // reconcileWithStored applies the shared no-op-elision policy (canon.Reconcile, // ADR-0008 decision 1) to a write against this project. func (w *Writer) reconcileWithStored(unitID string, contents []byte) (out []byte, unchanged bool) { - w.unitsOffered++ + w.writesOffered++ stored, err := w.reader.GetRawUnitBytes(unitID) if err != nil { - w.unitsWritten++ + w.writesLanded++ return contents, false // new unit, or unreadable — write it } out, unchanged = canon.Reconcile(contents, stored) if !unchanged { - w.unitsWritten++ + w.writesLanded++ } return out, unchanged } diff --git a/sdk/javaactions/writefile.go b/sdk/javaactions/writefile.go new file mode 100644 index 000000000..5f801b6e9 --- /dev/null +++ b/sdk/javaactions/writefile.go @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: Apache-2.0 + +package javaactions + +import "os" + +// WriteSourceIfChanged writes a generated .java/.js stub, and reports whether it +// actually differed from what was already there. +// +// Both halves matter, for different reasons. +// +// Not rewriting an identical file is ADR-0008's rule one level out from unit +// storage: a `create or modify` that changes nothing should leave the working +// tree alone, and an unconditional rewrite moves the file's mtime on every run — +// enough to defeat an incremental build's caching even though git sees no +// change. +// +// Reporting *whether* it changed is what lets the executor say "Unchanged +// javascript action" honestly. A code action's body does not live in its unit — +// the unit carries the signature, the source lives in +// `javascriptsource//actions/.js` — so a body-only edit elides the +// unit write entirely. Judging the statement on unit writes alone would call +// that "Unchanged" while the user's edit had just landed in a file. +// +// A file that cannot be read is treated as different, so the write still +// happens: the failure direction is a redundant write, never a lost one. +func WriteSourceIfChanged(path string, source string) (changed bool, err error) { + if existing, readErr := os.ReadFile(path); readErr == nil && string(existing) == source { + return false, nil + } + if err := os.WriteFile(path, []byte(source), 0o644); err != nil { + return false, err + } + return true, nil +} diff --git a/sdk/mpr/writer_core.go b/sdk/mpr/writer_core.go index b37e38b40..386d3cea2 100644 --- a/sdk/mpr/writer_core.go +++ b/sdk/mpr/writer_core.go @@ -34,19 +34,22 @@ func idToBsonBinary(id string) primitive.Binary { type Writer struct { reader *Reader - // unitsOffered / unitsWritten count what reached updateUnit and how much of - // it survived no-op elision (ADR-0008). The executor reads them to tell - // "Modified X" from "Modified nothing, X was already in sync" — without - // them, re-running a script that changes nothing still announces a write for - // every statement, which is how the churn in #910 was misdiagnosed. - unitsOffered int - unitsWritten int -} - -// WriteStats reports how many unit writes this session offered to storage and -// how many were not elided as no-ops. + // writesOffered / writesLanded count what this session tried to persist and + // how much of it was not skipped as a no-op (ADR-0008). Both unit writes and + // generated source files count: a code action's body lives in + // javascriptsource/ rather than in its unit, so counting units alone would + // call a body-only edit unchanged. The executor reads these to tell + // "Modified X" from "X was already in sync" — without them, re-running a + // script that changes nothing still announces a write for every statement, + // which is how the churn in #910 was misdiagnosed. + writesOffered int + writesLanded int +} + +// WriteStats reports how many writes this session offered to storage and how +// many of them actually changed something. func (w *Writer) WriteStats() (offered, written int) { - return w.unitsOffered, w.unitsWritten + return w.writesOffered, w.writesLanded } // NewWriter creates a new writer from a reader opened in read-write mode. diff --git a/sdk/mpr/writer_javaactions.go b/sdk/mpr/writer_javaactions.go index 43a8c6d18..98d5c9a14 100644 --- a/sdk/mpr/writer_javaactions.go +++ b/sdk/mpr/writer_javaactions.go @@ -99,11 +99,19 @@ func (w *Writer) WriteJavaSourceFile(moduleName, actionName string, javaCode str // Generate Java source (shared with the modelsdk engine) source := javaactions.GenerateSource(moduleName, actionName, javaCode, params, returnType, extraImports, extraCode) - // Write the file + // Write the file, unless it already says exactly this. The counters feed the + // executor's "Modified" vs "Unchanged" reporting: a code action's body lives + // here rather than in its unit, so judging the statement on unit writes alone + // would call a body-only edit unchanged. filePath := filepath.Join(javaDir, actionName+".java") - if err := os.WriteFile(filePath, []byte(source), 0644); err != nil { + w.writesOffered++ + changed, err := javaactions.WriteSourceIfChanged(filePath, source) + if err != nil { return fmt.Errorf("failed to write Java source file: %w", err) } + if changed { + w.writesLanded++ + } return nil } diff --git a/sdk/mpr/writer_javascriptactions.go b/sdk/mpr/writer_javascriptactions.go index 6ff951abc..52c23b640 100644 --- a/sdk/mpr/writer_javascriptactions.go +++ b/sdk/mpr/writer_javascriptactions.go @@ -142,9 +142,14 @@ func (w *Writer) WriteJavaScriptSourceFile(moduleName, actionName string, jsCode return fmt.Errorf("failed to create javascriptsource directory: %w", err) } source := javaactions.GenerateJavaScriptSource(actionName, jsCode, params, returnType) - if err := os.WriteFile(filepath.Join(dir, actionName+".js"), []byte(source), 0o644); err != nil { + w.writesOffered++ + changed, err := javaactions.WriteSourceIfChanged(filepath.Join(dir, actionName+".js"), source) + if err != nil { return fmt.Errorf("failed to write JavaScript source file: %w", err) } + if changed { + w.writesLanded++ + } return nil } diff --git a/sdk/mpr/writer_units.go b/sdk/mpr/writer_units.go index 322b1f51e..ad17da5ee 100644 --- a/sdk/mpr/writer_units.go +++ b/sdk/mpr/writer_units.go @@ -141,14 +141,14 @@ func (w *Writer) updateUnit(unitID string, contents []byte) error { // modelsdk engine's policy rather than reimplementing it. The two engines // must agree here: which one ran is an --engine flag, not something a user // should be able to see in their diff. - w.unitsOffered++ + w.writesOffered++ if stored, err := w.reader.GetRawUnitBytes(model.ID(unitID)); err == nil { var unchanged bool if contents, unchanged = canon.Reconcile(contents, stored); unchanged { return nil } } - w.unitsWritten++ + w.writesLanded++ // Convert UUID string to 16-byte blob unitIDBlob := uuidToBlob(unitID) From de7ea7137602d6041d13ed6d5ebc82555353e3db Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 11:52:16 +0000 Subject: [PATCH 10/21] fix(widgets): bind a pluggable text template's contentparams, warn on editable (#928) Two reported bindings that passed `mxcli check`, were written by `exec`, and did nothing. They share one cause: the allow-lists behind MDL-WIDGET01 and MDL-WIDGET07 are widget-type AGNOSTIC. isBuiltinPropName is a single flat list holding both ContentParams and Editable; it answers "is this a real MDL property name anywhere", and both validators read it as "is this valid on THIS widget". The engine then acts on each only for the widget kinds that support it, so the rest is dropped in silence. Bug 1 -- FIXED. A pluggable widget's text-template property imageUrl: '{1}', contentparams: [{1} = PictureUrl] stored a template with an EMPTY parameter list, and mxbuild answered CE0720 "Place holder index 1 is greater than 0, the number of parameter(s)" on the FIRST write -- no describe round-trip needed, contrary to the report. A dynamictext with identical syntax stored the parameter correctly, which is the control that localised it to the pluggable path. The engine took the parameters path only for mxcli's `{AttrName}` spelling, so Mendix's own numeric `{1}` had no route. Both spellings now reach the same stored shape via SetTextTemplateWithClientParams, added to the builder interface and to both implementations. Verified: the reported script is 0 errors and the Image's template holds one parameter bound to Product.PictureUrl, byte-comparable with the dynamictext control. Bug 2 -- NOT FIXABLE AS ASKED, so reported instead (MDL-WIDGET20). Mendix models editability on INPUT widgets only: measured against generated/metamodel, exactly eleven Pages types carry Editability / ConditionalEditabilitySettings -- ten inputs plus DataView -- and not one of the fourteen button types does. There is no field to write, so `editable:` -> Editability on a button cannot be implemented; the issue's own second option is the right one. The warning names conditional visibility, which buttons do support. Both MDL spellings are caught. `editable: 'x'` lowers to `Editable`, the bracket form `editable: [expr]` to `EditableIf` -- and the bracket form is the one that genuinely works on inputs, so leaving it unflagged on a button would have been the more surprising silent drop. The type list is a hand-maintained bridge between MDL and Mendix names, so a test parses the metamodel and fails if that set of eleven changes. Also MDL-WIDGET21, the residue of fixing bug 1: contentparams with no `{N}` placeholder to consume them still had nothing to attach to and were dropped without a word. Neither rule is a .fail.mdl. Both are warnings, so `check` exits 0 and such a fixture would report "negative test unexpectedly passed" -- the trap the Makefile documents above check-mdl, and the one that broke #927's CI. They are demonstrated by a plain .mdl and pinned by unit tests. Verified: reported script 0 errors under mx check (was CE0720); controls with each fix removed reproduce CE0720 and drop all three MDL-WIDGET20 cases; input widgets and 339 shipped examples produce no new warning; `make check-mdl` exits 0; 76 unit packages green; gofmt clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- .claude/skills/fix-issue.md | 1 + .claude/skills/mendix/create-page.md | 39 ++++ CHANGELOG.md | 13 ++ .../bug-tests/928-widget-binding-gaps.mdl | 69 +++++++ .../bug-tests/928-widget-binding-warnings.mdl | 50 +++++ mdl/backend/mcp/widget.go | 39 ++++ mdl/backend/mutation.go | 7 + mdl/backend/widgetobj/builder.go | 26 +++ mdl/executor/validate_widget_contentparams.go | 46 +++++ mdl/executor/validate_widget_editability.go | 99 ++++++++++ mdl/executor/validate_widgets.go | 8 + mdl/executor/widget_binding_gaps_928_test.go | 184 ++++++++++++++++++ mdl/executor/widget_engine.go | 15 ++ 13 files changed, 596 insertions(+) create mode 100644 mdl-examples/bug-tests/928-widget-binding-gaps.mdl create mode 100644 mdl-examples/bug-tests/928-widget-binding-warnings.mdl create mode 100644 mdl/executor/validate_widget_contentparams.go create mode 100644 mdl/executor/validate_widget_editability.go create mode 100644 mdl/executor/widget_binding_gaps_928_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 346206e42..3ae6c7ba5 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -564,3 +564,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli check -p app.mpr` passes a microflow containing `if $obj/Status = 'Open'` — comparing an enumeration attribute to a string literal, the first example in the type-checking proposal and the shape people actually write. The same mistake written as a create or change member *is* caught, which makes the gap look arbitrary | Two causes, both invisible. (a) **`inferKind` returned `KindUnknown` for every `AttributePathExpr`**, so `$obj/Attr` typed to nothing and every rule downstream of it stayed quiet. `exprcheck.Scope` is `Lookup(name) (TypeKind, bool)` — it can say "$P is an Object" but not *which* entity, so there was nowhere to put the answer; the adapter computed a variable→entity map (`buildVarEntityScope`) and used it only to label a slot path, never passing it to the checker, and it covered body-introduced variables but **not parameters** — the ordinary case. (b) Even resolved, nothing fired: **E001 keys off the SLOT** (`CreateItem.Value:Entity.Attr`), which exists for an assignment and not for a comparison | `mdl/exprcheck/interfaces.go` (`EntityScope`, `Context.Entities`), `mdl/exprcheck/parser.go` (`attributePathKind`, `pathTargetEntity`, `checkEnumComparedToString`), `mdl/exprcheck/adapters/` (`walkFlow`, `addParamEntities`, `entityScope`), `mdl/exprcatalog/` (`AssociationTarget`) | Put the object side in its **own seam** (`EntityScope`: `VariableEntity` + `AssociationTarget`) beside `Scope` rather than widening either it or `CatalogReader` — `CatalogReader`'s shape is what stays re-syncable from the upstream fork. **A multi-hop expression path is not an XPath path**: XPath spells the intermediate entity (`[Assoc/Entity/Attr]`) so a walk can read it off, while an expression does not (`$O/Mod.Assoc/Attr`), so every hop must resolve through the association index. For (b), emit the **same code and message** from the comparison as from the slot — one defect should not have two names depending on where it was spotted. **Generalisable**: when a checker is silent, separate "did the input resolve" from "is there a rule for this shape" before touching either — here resolution and detection were both missing, and fixing only one would have looked like the fix failed. Verify with a **probe corpus that actually exercises the construct**: 21 microflows described back to MDL contained 40 attribute-path lines including an association hop, and were clean before and after, which is what makes "no false positives" mean something. Controls: path→`KindUnknown` restored, `Entities` dropped from the Context, and `addParamEntities` removed each fail a distinct test. **Still open**: a *terminal* association step (`$Order/Mod.Order_Lines`) types to unknown — Object vs List depends on the association's kind and direction, and guessing costs false positives | | `describe microflow` renders a REST call's result as `returns String` when the activity actually stores the response in a **file document**, and the describe → exec round trip then writes String back — retyping the activity with `mx check` reporting nothing either way. On the legacy engine the `$var =` output variable disappears too | MDL had no syntax for the case, `sdk/microflows` had no FileDocument variant, and both readers fell through to String: the legacy `parseResultHandling` had no `FileDocument` case (so the whole handling read back nil, taking the output-variable fallback with it), while modelsdk's `restResultHandlingFromRaw` read `VariableType.Entity` and then discarded it, matching only the literal `System.HttpResponse`. Mendix stores all five variants as ONE `Microflows$ResultHandling` carrying a `VariableType`, discriminated by the `ResultHandlingType` enum | `mdl/grammar/domains/MDLMicroflow.g4` (`restCallReturnsClause`), `mdl/ast/ast_microflow.go`, `mdl/visitor/visitor_microflow_actions.go`, `sdk/microflows/microflows_actions.go`, `sdk/mpr/parser_microflow_actions.go` + `writer_microflow_actions.go`, `mdl/backend/modelsdk/microflow_read_actions.go` + `microflow_write.go`, `mdl/executor/cmd_microflows_builder_calls.go` + `cmd_microflows_format_action.go`, rule in `validate_microflow_rest.go` (MDL064) | Add `returns Module.Entity` as the LAST alternative so the keyword forms still win, and read Mendix's own `ResultHandlingType` discriminator rather than inferring from `VariableType` (keep the inference as a fallback — the property is `omitempty`). **Delete the silent `String` fallbacks**: an unreconstructable result handling must describe as something the parser REJECTS, or a reader gap becomes a model rewrite (ADR-0005). Mendix constraints worth knowing before designing syntax here: the BASE `System.FileDocument` is rejected as a return type (**CE0362**) so a specialization is mandatory, and HttpResponse **cannot** be specialized (**CE1540** allows only User, FileDocument, Image, Paging) — which is why `returns response` needed no change and the reporter's suggested "HttpResponse specialization" does not exist. Cross-engine round trip (`gateEngines`) is what proves it: the two readers failed differently and each looked fine from inside itself. Issue #922 | | The nightly fails on ONE Mendix matrix version only, in `TestMxCheck_DoctypeScripts`, with `Execution error: this project does not store the model setting ` — while the same script passes on every newer version | The example script set a model setting that version does not have. Measured: a blank 10.24 stores 11 model settings and a blank 11.6.6 stores 12, `DecimalScale` being the only difference. mxcli's refusal is CORRECT — Studio Pro will not open a model carrying a property its version does not define, and mxbuild does not catch it — so the bug is the ungated example, not the guard. The refusal covers the WHOLE statement, so one version-specific setting takes every portable setting in the same `alter` down with it | `mdl-examples/doctype-tests/14-project-settings-examples.mdl`; guard in `mdl/executor/doctype_version_gating_test.go` | Split the version-specific setting into its own statement inside a `-- @version: N.N+` section, closed with `-- @version: any`. **Put the `/** */` doc comment INSIDE the gated section**: a block comment is a documentation comment bound to the statement after it, so gating the statement while leaving the comment outside orphans it and the script dies with `no viable alternative at input '/**...'` — reported at the NEXT statement, tens of lines away, which reads like an unrelated syntax error. `--` line comments are free-standing and safe either side. Isolate which setting is at fault by exec'ing them one at a time against a blank project of that version (`mx create-project` in a SHORT path — a long one dies with PathTooLongException). `TestDoctypeScriptsParseAfterVersionFiltering` now parses every doctype script under each nightly matrix version without needing mxbuild, so a mis-gated script fails in seconds on push instead of hours later in one nightly job | +| A widget property is accepted by `mxcli check`, written by `exec`, and does nothing: a pluggable widget's `imageUrl: '{1}', contentparams: [...]` builds as **CE0720** ("place holder index greater than parameter count"), and `editable:` on a button silently leaves it enabled with no warning at all | Both allow-lists are widget-type AGNOSTIC. `isBuiltinPropName` (`widget_engine.go`) is ONE flat list holding both `ContentParams` and `Editable`, and it backs both MDL-WIDGET01 (pluggable) and MDL-WIDGET07 (static): it answers "is this a real MDL property name anywhere", and both validators read it as "is this valid on THIS widget". The engine then only acts on each for the widget kinds that support it | `mdl/executor/widget_engine.go` (`numericTemplatePlaceholderRe`, the TextTemplate case), `mdl/backend/mutation.go` + `widgetobj/builder.go` + `mcp/widget.go` (`SetTextTemplateWithClientParams`), rules in `validate_widget_editability.go` (MDL-WIDGET20) and `validate_widget_contentparams.go` (MDL-WIDGET21) | For the template: the engine took the parameters path only for mxcli's `{AttrName}` spelling, so Mendix's own numeric `{1}` had no route and the template was written with `Parameters=[2]` (empty). A `dynamictext` with identical syntax is the control that localises it. For editability: **check the metamodel before implementing** — `PagesActionButton` has `ConditionalVisibilitySettings` but no `Editability`, and exactly 11 Pages types have editability (10 inputs + DataView) against 14 button types with none, so the request was impossible and the fix is to WARN. Both MDL spellings must be caught: `editable:` lowers to `Editable`, the bracket form `editable: [expr]` to **`EditableIf`** — dump the parsed `w.Properties` rather than assuming the key. Neither rule can be a `.fail.mdl`: they are warnings, `check` exits 0, and the fixture would report "negative test unexpectedly passed". Issue #928 | diff --git a/.claude/skills/mendix/create-page.md b/.claude/skills/mendix/create-page.md index 651e59b41..720422c50 100644 --- a/.claude/skills/mendix/create-page.md +++ b/.claude/skills/mendix/create-page.md @@ -846,6 +846,45 @@ alter page Mod.Home { For theme images, use paths relative to `theme/web/` (e.g., `img/logo.svg` → `theme/web/img/logo.svg`). +**A per-row image URL comes from the entity, two ways.** `imageUrl` is a text +template, so it takes either spelling: + +```sql +-- named placeholder: shortest form for a single attribute +pluggablewidget 'com.mendix.widget.web.image.Image' cardImage ( + datasource: imageUrl, imageUrl: '{PictureUrl}' +) + +-- numbered placeholders + contentparams: needed for several values, or a format block +pluggablewidget 'com.mendix.widget.web.image.Image' cardImage ( + datasource: imageUrl, + imageUrl: '{1}/{2}', contentparams: [{1} = BaseUrl, {2} = PictureUrl] +) +``` + +Every `{N}` must have a matching parameter — Mendix rejects a shortfall with +`CE0720` ("place holder index N is greater than …, the number of parameter(s)"). +Parameters with no `{N}` to fill are reported by MDL-WIDGET21 rather than +dropped in silence. + +### Buttons Have Visibility, Not Editability + +`editable:` only exists on **input** widgets — Mendix gives exactly eleven page +widgets an editability setting (textbox, textarea, checkbox, datepicker, +dropdown, radiobuttons, referenceselector, inputreferencesetselector, +filemanager, imageuploader, and dataview). No button of any kind has one, so +`editable:` on a button is reported by MDL-WIDGET20 and does nothing. + +To disable a button conditionally, hide it instead — buttons do support +conditional visibility — or put the condition in the microflow it calls: + +```sql +actionbutton btnSubmit ( + caption: 'Submit', action: microflow Mod.ACT_Submit, + visible: [$currentObject/Status = Mod.Status.Draft] +) +``` + ### CONTAINER / CUSTOMCONTAINER Widgets Generic container for grouping widgets. `customcontainer` is an alias for `container` (both map to `Forms$DivContainer`): diff --git a/CHANGELOG.md b/CHANGELOG.md index a3156db8f..fdc089110 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Fixed + +- **A pluggable widget's text template no longer drops its `contentparams`** (#928) — `imageUrl: '{1}', contentparams: [{1} = PictureUrl]` on an Image widget stored a template with an **empty** parameter list, and mxbuild rejected it with `CE0720` ("place holder index 1 is greater than 0, the number of parameter(s)") on the **first write** — no describe round-trip needed. The engine took the parameters path only for mxcli's `{AttrName}` convenience spelling, so Mendix's own numeric `{1}` form had no route. Both spellings now reach the same stored shape; a `dynamictext` with identical syntax was the control that localised it to the pluggable path. + +### Added + +- **`MDL-WIDGET20` — `editable:` on a widget that has no editability** (#928) — accepted on any widget and silently dropped, so a button bound this way passed check, passed the build, and stayed enabled: a silent functional failure rather than a caught error. It cannot be implemented as asked. Measured against `generated/metamodel`, exactly **11** Pages types carry `Editability`/`ConditionalEditabilitySettings` — ten input widgets plus DataView — and **none** of the fourteen button types does; a button has conditional *visibility*, not editability. So mxcli reports it instead, naming visibility as the alternative. Both spellings are caught: `editable: 'x'` and the bracket form `editable: [expr]`, which is the one that genuinely works on inputs and would be the more surprising silent drop. A test pins the list against the metamodel so it cannot drift. + +- **`MDL-WIDGET21` — `contentparams` with no placeholder to consume them** — the residue of the fix above: parameters supplied where no property text carries a `{1}`-style placeholder have nothing to attach to and are dropped on write. Previously silent. + + Root cause of both reports is one thing: the allow-lists behind MDL-WIDGET01 and MDL-WIDGET07 are widget-type **agnostic**. `isBuiltinPropName` is a single flat list holding both `ContentParams` and `Editable`; it answers "is this a real MDL property name anywhere", and both validators read it as "is this valid on this widget". + + ### Fixed - **The Mendix 10.24 nightly is green again** — `14-project-settings-examples.mdl` set `DecimalScale`, which 10.24 does not have, so `TestMxCheck_DoctypeScripts` failed on that matrix entry on both engines while passing on every 11.x. Measured against blank projects: 10.24 stores 11 model settings, 11.6.6 stores 12, and `DecimalScale` is the only difference — each of the other five settings in that statement is accepted on 10.24 on its own. mxcli's refusal was correct (Studio Pro will not open a model carrying a property its version does not define, and mxbuild does not catch it); the example simply was not version-gated, and because the refusal covers the whole statement, one unsupported setting took five portable ones down with it. It is now its own `-- @version: 11.0+` section. diff --git a/mdl-examples/bug-tests/928-widget-binding-gaps.mdl b/mdl-examples/bug-tests/928-widget-binding-gaps.mdl new file mode 100644 index 000000000..ba2ce8051 --- /dev/null +++ b/mdl-examples/bug-tests/928-widget-binding-gaps.mdl @@ -0,0 +1,69 @@ +-- ============================================================================ +-- Issue #928: two widget property bindings that passed check and wrote nothing +-- ============================================================================ +-- +-- Bug 1 -- a pluggable widget's text-template property dropped its parameters. +-- +-- `imageUrl: '{1}', contentparams: [{1} = PictureUrl]` stored a template with +-- an EMPTY parameter list, and mxbuild answered +-- CE0720 "Place holder index 1 is greater than 0, the number of +-- parameter(s)." at Image 'cardImage' +-- on the FIRST write -- no describe round-trip needed. A `dynamictext` with +-- the same syntax stored Parameters=[2, {ClientTemplateParameter}], which is +-- the control that located it: only the pluggable path was affected. +-- +-- Cause: the engine took the parameters path only for mxcli's `{AttrName}` +-- convenience spelling, so Mendix's own numeric `{1}` form had no route. +-- Both spellings now reach the same stored shape; this file proves it by +-- using them side by side over the same attribute. +-- +-- Bug 2 -- `editable:` on a button did nothing, and nothing said so. +-- +-- See 928-widget-binding-warnings.mdl. Mendix models editability on INPUT +-- widgets only: exactly eleven Pages types carry Editability / +-- ConditionalEditabilitySettings, and not one of the fourteen button types +-- does. So it cannot be implemented -- it is reported instead (MDL-WIDGET20). +-- +-- Usage: +-- mxcli exec mdl-examples/bug-tests/928-widget-binding-gaps.mdl -p app.mpr +-- mx check app.mpr # no new errors -- CE0720 is gone +-- ============================================================================ + +create module BugTest928; + +create persistent entity BugTest928.Product ( + Name: string(200), + PictureUrl: string(500) +); +/ + +create or modify page BugTest928.Catalogue ( + Title: 'Catalogue', + Layout: Atlas_Core.Atlas_Default +) +{ + LISTVIEW lv (DataSource: DATABASE BugTest928.Product) { + + -- The control: a built-in widget with the same syntax. This always worked, + -- and is what showed the defect was specific to the pluggable path. + DYNAMICTEXT dt ( Content: '{1}', ContentParams: [{1} = PictureUrl] ) + + -- Bug 1, the reported form: numeric placeholder + explicit contentparams. + PLUGGABLEWIDGET 'com.mendix.widget.web.image.Image' cardImage ( + datasource: imageUrl, + imageUrl: '{1}', contentparams: [{1} = PictureUrl] + ) + + -- The same binding in the `{AttrName}` spelling, which already worked and + -- keeps working. Both store one parameter bound to Product.PictureUrl. + PLUGGABLEWIDGET 'com.mendix.widget.web.image.Image' cardImage2 ( + datasource: imageUrl, + imageUrl: '{PictureUrl}' + ) + + -- Editability on a widget that HAS it stays clean (MDL-WIDGET20 must not + -- fire here, or the documented idiom would be flagged). + TEXTBOX tbName ( Attribute: Name, editable: [false] ) + } +} +/ diff --git a/mdl-examples/bug-tests/928-widget-binding-warnings.mdl b/mdl-examples/bug-tests/928-widget-binding-warnings.mdl new file mode 100644 index 000000000..f1792c110 --- /dev/null +++ b/mdl-examples/bug-tests/928-widget-binding-warnings.mdl @@ -0,0 +1,50 @@ +-- ============================================================================ +-- Issue #928: the two bindings mxcli now WARNS about instead of dropping +-- ============================================================================ +-- +-- This is a plain .mdl, not a .fail.mdl, on purpose. Both rules are WARNINGS, +-- so `mxcli check` still exits 0 — naming it .fail.mdl would report "negative +-- test unexpectedly passed" and make a working rule look regressed (the trap +-- the Makefile documents above check-mdl). The rules themselves are covered by +-- unit tests in mdl/executor/widget_binding_gaps_928_test.go. +-- +-- Running `mxcli check` on this file should report: +-- +-- MDL-WIDGET20 editable on a button — Mendix models editability on INPUT +-- widgets only. Measured against generated/metamodel: exactly +-- eleven Pages types carry Editability / +-- ConditionalEditabilitySettings (ten input widgets plus +-- DataView) and none of the fourteen button types does. So this +-- cannot be implemented; a button gets conditional VISIBILITY. +-- Both spellings are caught — `editable: 'x'` and the bracket +-- form `editable: [expr]`, which is the one that works on inputs +-- and so would be the more surprising silent drop. +-- +-- MDL-WIDGET21 contentparams with no `{1}`-style placeholder to consume them. +-- The residue of fixing bug 1: the parameters have nothing to +-- attach to and are dropped on write. +-- +-- Requires the module from 928-widget-binding-gaps.mdl. +-- ============================================================================ + +create or modify page BugTest928.Warnings ( + Title: 'Warnings', + Layout: Atlas_Core.Atlas_Default +) +{ + LISTVIEW lv (DataSource: DATABASE BugTest928.Product) { + + -- MDL-WIDGET20: the reported form. + ACTIONBUTTON btnPlain ( Caption: 'Go', Action: NOTHING, editable: 'false' ) + + -- MDL-WIDGET20: the bracket form, which DOES work on an input widget. + ACTIONBUTTON btnBracket ( Caption: 'Go2', Action: NOTHING, editable: [false] ) + + -- MDL-WIDGET21: parameters with no placeholder to fill. + PLUGGABLEWIDGET 'com.mendix.widget.web.image.Image' imgNoPlaceholder ( + datasource: imageUrl, + imageUrl: 'static.png', contentparams: [{1} = PictureUrl] + ) + } +} +/ diff --git a/mdl/backend/mcp/widget.go b/mdl/backend/mcp/widget.go index 556d1761e..b24b9ab21 100644 --- a/mdl/backend/mcp/widget.go +++ b/mdl/backend/mcp/widget.go @@ -297,6 +297,45 @@ var templateAttrPlaceholderRe = regexp.MustCompile(`\{([A-Za-z][A-Za-z0-9_]*)\}` // (widgetobj.createClientTemplateBSONWithParams). The parameterised // Pages$ClientTemplate shape was verified live on 11.12 (attributeRef // persisted, ped_check_errors clean). +// SetTextTemplateWithClientParams stores a text template whose `{1}`-style +// placeholders are backed by author-supplied parameters (MDL `contentparams:`). +// Same stored shape as SetTextTemplateWithParams; only the source of the +// parameters differs — there they are derived from `{AttrName}` placeholders, +// here they are given. (#928) +func (w *mcpWidgetBuilder) SetTextTemplateWithClientParams(propertyKey, text string, params []*pages.ClientTemplateParameter) { + if text == "" { + return + } + if len(params) == 0 { + w.object["ct:"+propertyKey] = text + return + } + out := make([]any, 0, len(params)) + for _, p := range params { + if p == nil { + continue + } + out = append(out, map[string]any{ + "$Type": "Pages$ClientTemplateParameter", + "attributeRef": map[string]any{"$Type": "DomainModels$AttributeRef", "attribute": p.AttributeRef}, + "formattingInfo": map[string]any{ + "$Type": "Pages$FormattingInfo", + "decimalPrecision": 2, + "groupDigits": false, + "enumFormat": "Text", + "dateFormat": "Date", + "customDateFormat": "", + }, + }) + } + w.object["ct:"+propertyKey] = map[string]any{ + "$Type": "Pages$ClientTemplate", + "t:template": text, + "parameters": out, + "t:fallback": "", + } +} + func (w *mcpWidgetBuilder) SetTextTemplateWithParams(propertyKey, text, entityContext string) { if text == "" { return diff --git a/mdl/backend/mutation.go b/mdl/backend/mutation.go index 875a3e0a8..932da1df6 100644 --- a/mdl/backend/mutation.go +++ b/mdl/backend/mutation.go @@ -317,6 +317,13 @@ type WidgetObjectBuilder interface { SetChildWidgets(propertyKey string, children []pages.Widget) SetTextTemplate(propertyKey string, text string) SetTextTemplateWithParams(propertyKey string, text string, entityContext string) + // SetTextTemplateWithClientParams sets a text template whose `{1}`-style + // placeholders are backed by parameters the AUTHOR supplied (MDL + // `contentparams:`), rather than derived from `{AttrName}` placeholders + // against the entity context. Both spellings reach the same stored shape; + // without this one a pluggable widget's `{1}` template was written with an + // empty parameter list, which mxbuild rejects with CE0720 (#928). + SetTextTemplateWithClientParams(propertyKey string, text string, params []*pages.ClientTemplateParameter) SetAction(propertyKey string, action pages.ClientAction) SetAttributeObjects(propertyKey string, attributePaths []string) diff --git a/mdl/backend/widgetobj/builder.go b/mdl/backend/widgetobj/builder.go index e4a40eb26..a50b12899 100644 --- a/mdl/backend/widgetobj/builder.go +++ b/mdl/backend/widgetobj/builder.go @@ -193,6 +193,32 @@ func (ob *Builder) SetTextTemplateWithParams(propertyKey string, text string, en }) } +// SetTextTemplateWithClientParams sets a text template from author-supplied +// parameters (MDL `contentparams:`), for a `{1}`-style template. +// +// SetTextTemplateWithParams derives parameters by matching `{AttrName}` against +// the entity context; that covers the named spelling but leaves the numeric one +// with no route, so a pluggable widget's `imageUrl: '{1}', contentparams: [...]` +// was stored with Parameters=[2] (empty) and mxbuild answered CE0720 "Place +// holder index 1 is greater than 0, the number of parameter(s)". (#928) +func (ob *Builder) SetTextTemplateWithClientParams(propertyKey string, text string, params []*pages.ClientTemplateParameter) { + if text == "" { + return + } + tmpl := BuildClientTemplateWithTextAndParams(text, params) + ob.object = updateWidgetPropertyValue(ob.object, ob.propertyTypeIDs, propertyKey, func(val bson.D) bson.D { + result := make(bson.D, 0, len(val)) + for _, elem := range val { + if elem.Key == "TextTemplate" { + result = append(result, bson.E{Key: "TextTemplate", Value: tmpl}) + } else { + result = append(result, elem) + } + } + return result + }) +} + func (ob *Builder) SetAction(propertyKey string, action pages.ClientAction) { if action == nil { return diff --git a/mdl/executor/validate_widget_contentparams.go b/mdl/executor/validate_widget_contentparams.go new file mode 100644 index 000000000..44f30ef83 --- /dev/null +++ b/mdl/executor/validate_widget_contentparams.go @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time validation for `contentparams:` on a pluggable widget. +// +// A pluggable widget's text-template property takes `{1}`-style placeholders +// backed by `contentparams:`, or mxcli's `{AttrName}` convenience spelling which +// is resolved against the entity context. Parameters with no numeric placeholder +// to fill have nothing to attach to and are dropped on write — the same silent +// class as the bug #928 was filed for, and the residue left by fixing it. +package executor + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// validatePluggableContentParams reports (MDL-WIDGET21) `contentparams:` on a +// pluggable widget where no property text carries a `{N}` placeholder to consume +// them. +// +// A warning, not an error: the widget's property vocabulary comes from its own +// definition, and a text-template property whose value arrives by some route +// this check cannot see would make a hard reject a false positive. +func validatePluggableContentParams(w *ast.WidgetV3, locationPrefix string) []linter.Violation { + if w == nil || len(w.GetContentParams()) == 0 { + return nil + } + for _, v := range w.Properties { + if s, ok := v.(string); ok && numericTemplatePlaceholderRe.MatchString(s) { + return nil // something can consume them + } + } + return []linter.Violation{{ + RuleID: "MDL-WIDGET21", + Severity: linter.SeverityWarning, + Message: fmt.Sprintf( + "%s: widget `%s` (%s) has `contentparams` but no property text contains a `{1}`-style "+ + "placeholder to use them, so they are dropped on write", + locationPrefix, w.Name, w.Type, + ), + Suggestion: "Put a numbered placeholder in the text property (e.g. `imageUrl: '{1}'`), or drop " + + "the contentparams — a single attribute can also be written inline as `'{AttrName}'`", + }} +} diff --git a/mdl/executor/validate_widget_editability.go b/mdl/executor/validate_widget_editability.go new file mode 100644 index 000000000..4e2a05cea --- /dev/null +++ b/mdl/executor/validate_widget_editability.go @@ -0,0 +1,99 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time validation for the `editable:` widget property. +// +// Mendix models editability on INPUT widgets only. Measured against +// generated/metamodel (the arbiter per CLAUDE.md), exactly eleven Pages types +// carry Editability / ConditionalEditabilitySettings — ten input widgets plus +// DataView — and not one of the fourteen button types does. A button has +// visibility, not editability. +// +// mxcli accepted `editable:` on any widget because the allow-lists behind +// MDL-WIDGET01 and MDL-WIDGET07 are widget-type-AGNOSTIC: isBuiltinPropName +// answers "is this a real MDL property name anywhere", and both validators read +// it as "is this valid on THIS widget". So `editable:` on a button passed check, +// passed exec, wrote nothing, and the button stayed enabled — a silent +// functional failure rather than a caught error. (issue #928) +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// editableWidgetTypes are the MDL widget types whose Mendix counterpart carries +// Editability / ConditionalEditabilitySettings. +// +// Kept in sync with generated/metamodel by +// TestEditableWidgetTypesMatchMetamodel, which fails if Mendix's set changes — +// the list cannot be derived at runtime because MDL type names are not the +// metamodel's type names. +var editableWidgetTypes = map[string]bool{ + "checkbox": true, // Pages$CheckBox + "dataview": true, // Pages$DataView + "datepicker": true, // Pages$DatePicker + "dropdown": true, // Pages$DropDown + "filemanager": true, // Pages$FileManager + "imageuploader": true, // Pages$ImageUploader + "inputreferencesetselector": true, // Pages$InputReferenceSetSelector + "radiobuttons": true, // Pages$RadioButtonGroup + "radiobuttongroup": true, // Pages$RadioButtonGroup (alternate spelling) + "referenceselector": true, // Pages$ReferenceSelector + "textarea": true, // Pages$TextArea + "textbox": true, // Pages$TextBox +} + +// ValidateWidgetEditability reports (MDL-WIDGET20) an `editable:` property on a +// widget type that has no editability in the Mendix model. +// +// It runs in the no-project pass: the answer is the widget's own type, already +// in the statement. A warning rather than an error, matching MDL-WIDGET07 — the +// same "silently dropped on write" family, and the same reason (a hard reject on +// a vocabulary that cannot be proven complete risks false positives). +// +// Pluggable widgets are excluded. Their property vocabulary comes from the +// widget's own definition and is checked by MDL-WIDGET01; `editable` on one is +// either a real property of that widget or already reported there. +func validateWidgetEditability(w *ast.WidgetV3, locationPrefix string) []linter.Violation { + if w == nil { + return nil + } + key, ok := widgetEditableKey(w) + if !ok { + return nil + } + if editableWidgetTypes[strings.ToLower(w.Type)] { + return nil + } + return []linter.Violation{{ + RuleID: "MDL-WIDGET20", + Severity: linter.SeverityWarning, + Message: fmt.Sprintf( + "%s: widget `%s` (%s) has an `%s` property, but Mendix models editability on input "+ + "widgets only — %s has no Editability, so this is silently dropped on write and the "+ + "widget stays enabled", + locationPrefix, w.Name, w.Type, key, w.Type, + ), + Suggestion: "Use `visible: [ ... ]` to hide it conditionally (buttons do support conditional " + + "visibility), or move the condition into the microflow the button calls", + }} +} + +// widgetEditableKey finds an editability property however the author spelled it. +// Both MDL forms have to be caught: `editable: 'false'` stays as `Editable`, +// while the bracket form `editable: [expr]` is lowered by the visitor to +// `EditableIf`. Missing the second would leave the shape the docs +// actually recommend silently dropped, which is the bug. The key is returned as +// spelled so the message quotes the author's casing. +func widgetEditableKey(w *ast.WidgetV3) (string, bool) { + for key := range w.Properties { + l := strings.ToLower(key) + if l == "editable" || l == "editableif" { + return key, true + } + } + return "", false +} diff --git a/mdl/executor/validate_widgets.go b/mdl/executor/validate_widgets.go index 75133c677..496b59b55 100644 --- a/mdl/executor/validate_widgets.go +++ b/mdl/executor/validate_widgets.go @@ -117,6 +117,10 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc mapping := parentObjectLists[strings.ToUpper(w.Type)] isObjectListItem := mapping != nil || isUniversalObjectListKeyword(w.Type) out = append(out, validatePluggableWidgetProperties(w, registry, locationPrefix)...) + // #928: contentparams with no `{N}` placeholder to consume them. + if lookupWidgetDef(w, registry) != nil { + out = append(out, validatePluggableContentParams(w, locationPrefix)...) + } out = append(out, validateWidgetVisibility(w, registry, locationPrefix)...) out = append(out, validateStaticWidget(w, locationPrefix)...) out = append(out, validateDynamicTextFormatting(w, locationPrefix)...) @@ -128,6 +132,10 @@ func validateWidgetTreeIn(widgets []*ast.WidgetV3, registry *WidgetRegistry, loc def := lookupWidgetDef(w, registry) if def == nil && !isObjectListItem { out = append(out, validateStaticWidgetUnknownProps(w, locationPrefix)...) + // #928: `editable:` on a widget Mendix gives no editability — same + // "silently dropped on write" family, but the flat property + // allow-list cannot see it because it is type-agnostic. + out = append(out, validateWidgetEditability(w, locationPrefix)...) } if mapping != nil { out = append(out, validateObjectListItemEnums(w, mapping, locationPrefix)...) diff --git a/mdl/executor/widget_binding_gaps_928_test.go b/mdl/executor/widget_binding_gaps_928_test.go new file mode 100644 index 000000000..b6c73c585 --- /dev/null +++ b/mdl/executor/widget_binding_gaps_928_test.go @@ -0,0 +1,184 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// upstream #928, bug 2. `editable:` was accepted on any widget and silently +// dropped, because the allow-lists behind MDL-WIDGET01/07 are widget-type +// AGNOSTIC — isBuiltinPropName answers "is this a real MDL property name +// anywhere", and both validators read it as "is this valid on THIS widget". +// +// A button that stays enabled is the dangerous shape: check passes, the build +// passes, and only the running app is wrong. +func TestMDLWIDGET20_EditableOnAWidgetWithoutEditability(t *testing.T) { + cases := []struct { + name string + src string + want string // the property key the message should quote + }{ + { + name: "plain form on a button (the reported case)", + src: buttonPage(`editable: 'false'`), + want: "Editable", + }, + { + // The bracket form is the one that WORKS on inputs, so leaving it + // unflagged on a button would silently drop the shape the docs + // recommend — the worse half of the bug. + name: "bracket form on a button", + src: buttonPage(`editable: [false]`), + want: "EditableIf", + }, + { + name: "bracket form with a real expression", + src: buttonPage(`editable: [$currentObject/Name != '']`), + want: "EditableIf", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := widgetViolations(t, tc.src, "MDL-WIDGET20") + if len(got) != 1 { + t.Fatalf("got %d MDL-WIDGET20 violations, want 1", len(got)) + } + if got[0].Severity != linter.SeverityWarning { + t.Errorf("severity = %v, want Warning (matching MDL-WIDGET07's family)", got[0].Severity) + } + if !strings.Contains(got[0].Message, tc.want) { + t.Errorf("message should quote the property as spelled (%s), got %q", tc.want, got[0].Message) + } + if !strings.Contains(got[0].Suggestion, "visible") { + t.Errorf("suggestion should point at conditional visibility, which buttons DO support: %q", got[0].Suggestion) + } + }) + } +} + +// The control. An input widget genuinely has editability — flagging it would +// refuse the documented, working idiom, and 8 shipped examples use it. +func TestMDLWIDGET20_InputWidgetsAreClean(t *testing.T) { + for _, w := range []string{ + `TEXTBOX tb ( Attribute: Name, editable: [false] )`, + `TEXTAREA ta ( Attribute: Name, editable: [false] )`, + `CHECKBOX cb ( Attribute: Name, editable: [false] )`, + `DATEPICKER dp ( Attribute: Name, editable: [false] )`, + } { + t.Run(strings.Fields(w)[0], func(t *testing.T) { + if got := widgetViolations(t, listviewPage(w), "MDL-WIDGET20"); len(got) != 0 { + t.Errorf("MDL-WIDGET20 fired on an editable-capable widget: %#v", got) + } + }) + } +} + +// A widget with no editability property at all must stay silent — the rule keys +// on the property being present, not on the widget type. +func TestMDLWIDGET20_SilentWithoutTheProperty(t *testing.T) { + if got := widgetViolations(t, buttonPage(`Class: 'btn'`), "MDL-WIDGET20"); len(got) != 0 { + t.Errorf("MDL-WIDGET20 fired on a button with no editable property: %#v", got) + } +} + +// editableWidgetTypes is a hand-maintained bridge between MDL type names and +// Mendix's, so it can drift from the metamodel it claims to mirror. This reads +// generated/metamodel — the arbiter per CLAUDE.md — and fails if the set of +// Pages types carrying editability changes, which is the event that would make +// the rule wrong in either direction. +func TestEditableWidgetTypesMatchMetamodel(t *testing.T) { + fset := token.NewFileSet() + path := filepath.Join("..", "..", "generated", "metamodel", "types.go") + f, err := parser.ParseFile(fset, path, nil, 0) + if err != nil { + t.Fatalf("parse metamodel: %v", err) + } + + withEditability := map[string]bool{} + ast.Inspect(f, func(n ast.Node) bool { + ts, ok := n.(*ast.TypeSpec) + if !ok || !strings.HasPrefix(ts.Name.Name, "Pages") { + return true + } + st, ok := ts.Type.(*ast.StructType) + if !ok { + return true + } + for _, fld := range st.Fields.List { + for _, nm := range fld.Names { + if nm.Name == "Editability" || nm.Name == "ConditionalEditabilitySettings" { + withEditability[ts.Name.Name] = true + } + } + } + return true + }) + + if len(withEditability) == 0 { + t.Fatal("found no Pages type with editability — the parse is wrong, and a passing run would prove nothing") + } + + // Every metamodel type must be reachable from the rule's list, via the + // comment that names it. A new editable widget in Mendix therefore fails + // here rather than silently becoming a false positive. + src, err := parseRuleComments() + if err != nil { + t.Fatalf("read the rule's list: %v", err) + } + for typeName := range withEditability { + mendixName := "Pages$" + strings.TrimPrefix(typeName, "Pages") + if !strings.Contains(src, mendixName) { + t.Errorf("%s carries editability in the metamodel but is not named in editableWidgetTypes — "+ + "a widget mxcli would now wrongly warn about", mendixName) + } + } + if len(withEditability) != 11 { + t.Errorf("the metamodel now has %d editable Pages types, not the 11 measured for #928 (%v) — "+ + "re-check editableWidgetTypes against it", len(withEditability), withEditability) + } +} + +func parseRuleComments() (string, error) { + b, err := os.ReadFile("validate_widget_editability.go") + return string(b), err +} + +func buttonPage(prop string) string { + return listviewPage(`ACTIONBUTTON btn ( Caption: 'Go', Action: NOTHING, ` + prop + ` )`) +} + +func listviewPage(widget string) string { + return `create page W.P ( Title: 'P' ) +{ + LISTVIEW lv (DataSource: DATABASE W.Product) { + ` + widget + ` + } +}` +} + +// widgetViolations parses a page and returns the violations of one rule. +func widgetViolations(t *testing.T, src, ruleID string) []linter.Violation { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse error: %v", errs[0]) + } + var out []linter.Violation + for _, v := range ValidateWidgetProperties(prog, "") { + if v.RuleID == ruleID { + out = append(out, v) + } + } + return out +} diff --git a/mdl/executor/widget_engine.go b/mdl/executor/widget_engine.go index 6eb1fa64e..3a59ac4c8 100644 --- a/mdl/executor/widget_engine.go +++ b/mdl/executor/widget_engine.go @@ -455,6 +455,16 @@ func (e *PluggableWidgetEngine) Build(def *WidgetDefinition, w *ast.WidgetV3) (* case "Expression": builder.SetExpression(propName, strVal) case "TextTemplate": + // `contentparams:` supplies the parameters for a `{1}`-style template. + // Without this branch the numeric spelling had no route on a pluggable + // widget — the template was written with an empty parameter list and + // mxbuild answered CE0720. The named `{AttrName}` spelling keeps its + // own path, which derives parameters from the entity context. (#928) + if params := e.pageBuilder.buildClientTemplateParams(w.GetContentParams()); len(params) > 0 && + numericTemplatePlaceholderRe.MatchString(strVal) { + builder.SetTextTemplateWithClientParams(propName, strVal, params) + break + } entityCtx := e.pageBuilder.entityContext builder.SetTextTemplateWithParams(propName, strVal, entityCtx) case "Attribute": @@ -1471,6 +1481,11 @@ func (e *PluggableWidgetEngine) applyChildSlots(builder backend.WidgetObjectBuil // isBuiltinPropName returns true for property names that are handled by // dedicated MDL keywords (DataSource, Attribute, etc.) rather than by // the explicit property pass. +// numericTemplatePlaceholderRe matches Mendix's own `{1}` placeholder form, the +// one that needs explicit ClientTemplate parameters. `{AttrName}` is mxcli's +// convenience spelling and is resolved separately against the entity context. +var numericTemplatePlaceholderRe = regexp.MustCompile(`\{[0-9]+\}`) + func isBuiltinPropName(name string) bool { switch name { case "DataSource", "Attribute", "Label", "Caption", "Action", From 2a9c69f5a41499e528c8b30867672744957677a0 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 18:11:11 +0000 Subject: [PATCH 11/21] fix(mappings): an unauthored import range is All, not First MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `import from mapping M.IMM($json)` — no range keyword — built a document the Mendix runtime refuses: MicroflowException: key not found: Path(QName(None,),None,) at ...integration.importer.mapping.MappingCache.storeValueMappingElement The model is valid, so nothing static caught it: mxcli check, mx check (0 errors) and mxbuild all pass. It fails only when the activity runs, and the repo had no runtime coverage of import mappings — every existing test stopped at mx check. The Range and the result variable's cardinality are separate axes (#881), but an unauthored range set neither pointer, so ForceSingleOccurrence and ConstantRange.SingleObject both fell back to SingleObject — true for an object-rooted mapping. That is Studio Pro's First ("take one of a list"), not a single-object import. Studio Pro writes both flags false and expresses "one object" solely through VariableType=ObjectType. `all`, `first` and limit/offset each set the pointers explicitly, so only the bare form was affected — the form the shipped examples use (06-rest-client-examples.mdl:1506, :1541). `first` is unchanged. Measured against a Studio Pro-authored app on 11.13.0: in one boot, over the same mapping and the same JSON, `... all` imported and the bare form threw; with the fix both import. The cross-test is what located it — an mxcli microflow calling Studio Pro's own mapping fails too, which rules out the mapping document, the JSON structure and the entity. Also updates TestImportRange_UnauthoredKeepsTheOldInference, which asserted the faulty fallback. It passed only because it exercised the list-rooted case, where the fallback is false either way. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Dp1syGhH8yr7Hve2wwjzqg --- .claude/skills/fix-issue.md | 1 + .../import-mapping-single-object-runtime.mdl | 90 +++++++++++++++++++ mdl/executor/cmd_microflows_builder_calls.go | 15 ++++ .../cmd_microflows_import_range_test.go | 54 +++++++++-- 4 files changed, 152 insertions(+), 8 deletions(-) create mode 100644 mdl-examples/bug-tests/import-mapping-single-object-runtime.mdl diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index d213be3e6..bb986e7df 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -569,3 +569,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `run --local` / `test --local` on macOS (or Windows) die with a raw `fork/exec ~/.mxcli/mxbuild//modeler/mxbuild: exec format error`, while `setup mxbuild` and `docker build` correctly use Studio Pro's binary. `--mxbuild-path` does not help | `MxBuildCDNURL` branches on **GOARCH only, never GOOS** — both URLs are Linux tarballs (Mendix ships macOS mxbuild inside Studio Pro, not on the CDN), so an arm64 Mac caches a Linux *aarch64* ELF: the arch matches, so nothing notices until exec. `docker build` resolves via `resolveMxBuild` (PATH → Studio Pro → known locations → cache) and `setup mxbuild` via `NativeMxBuildForSetup`, but the local loop called `DownloadMxBuild` directly at `runlocal.go` and `StartServe` looked only at the cache. `LocalRunOptions.MxBuildPath` was documented as an override and never read for the serve binary, so there was no workaround either | `cmd/mxcli/docker/mxbuild_platform.go` (new: `ResolveMxBuildForLocal`, `binaryOS`, `verifyRunsHere`) + `runlocal.go` (the `DownloadMxBuild` call) + `mxserve.go` (`StartServe`) | Reuse the rule the codebase already had rather than inventing one: `NativeMxBuildForSetup(goos, version)` returns Studio Pro's path, or "" meaning "Linux, download is fine", or an error plus guidance. Order: explicit path → (non-Linux) Studio Pro → cache/CDN. **Do not stop the download on Windows** — the cache holds a Linux binary there deliberately, for Docker builds; the fix belongs at resolve/exec time. Guard exec with a magic-byte check (ELF / Mach-O incl. fat / PE) and let an unrecognised format through, since a shell wrapper has no magic and refusing on a guess would block a working setup. **Inject `goos` into the helpers** (`resolveMxBuildForLocalOn`, `verifyRunsOn`): the whole bug is platform-specific and is otherwise untestable from a Linux runner. Repro without a Mac: plant a Mach-O magic at the cache path and call `StartServe` — reproduces the reporter's message verbatim. Tests `TestBinaryOS`, `TestVerifyRunsOn_LinuxBinaryOnMac`, `TestResolveMxBuildForLocal_*`; reverting makes `ExplicitPathWins` fail **by downloading from the CDN for 34s**, which is the ignored override made visible. **Unverified:** no macOS host was available, so the Studio Pro discovery path itself (`resolveStudioProDirMacOS`) is exercised only by its own existing tests. Issue #916 | | Windows: `run --local` behaves as though Java were undetected, and a local run can fail looking for Gradle | Two separate things. (a) **The `.exe` suffix**: `isJDK21` appended it, every CONSUMER did not — `--java-exe-path` handed to mxbuild and the path `exec.Command` runs to boot the runtime were both `\bin\java`, so a correctly-detected JDK was passed on in a form that need not resolve. Five sites built it by hand. (b) **Gradle is not mxcli's**: it ships inside the mxbuild bundle (`modeler/tools/gradle`, 8.5) and mxbuild invokes it — mxcli never calls gradle, so "Gradle missing" indicates a foreign/incomplete mxbuild bundle, usually #916 | `cmd/mxcli/docker/javaexe.go` (new `JavaExePath`) used from `mxserve.go`, `build.go`, `settle.go`, `localboot.go`, `detect.go`; `jdkSearchPathsFor` in `detect.go` | One helper, not five hand-built joins — the sixth call site would have repeated the bug. **Check the platform's own docs before adding a search path**: "add Studio Pro's JDK" turned out to be a non-task, because Mendix's install guide says Studio Pro installs **Eclipse Temurin 21** rather than bundling a JDK, so the existing Adoptium glob already IS Studio Pro's JDK; what was genuinely missing was the per-user `%LOCALAPPDATA%\Programs` install location. Inventing a `Mendix\\jdk` path would have been dead code. Make the not-found error list what was searched — "JDK 21 not found" alone sends a user reading mxcli's source. Inject `goos` (`jdkSearchPathsFor`, `javaExeName`): the only Windows CI job is the tunnel seam, scoped with `-run`, so it compiles Windows code and executes almost none of it — which is exactly how a missing `.exe` survives. **Unverified**: no Windows host; the fix is code-level with OS-injected tests. Reported via a user relay, not an issue | | Studio Pro's version-control view shows an **entire** nanoflow/microflow as changed after editing one activity argument; `git diff` on `mprcontents/` is unreadable. A change and its revert leave a semantically identical document that shares no element IDs with the original. Separately, re-running an already-applied script prints `Modified …`/`Replaced …` for files it did not touch | Two independent things. (a) `create or replace` rebuilds the document and every sub-element gets a freshly random `$ID`; elision (ADR-0008) only covers the case where *nothing* changed, so a real change wrote a whole new identity set — measured 36 of 37 on a nanoflow, 21 of 22 on a microflow. (b) The `Modified …` lines are printed by the handler right after `ctx.Backend.Update*`, which returns nil whether or not storage elided the write | `modelsdk/canon/transplant.go` (`TransplantIDs`, called from `Reconcile` in `identity.go`); `mdl/executor/report_mutation.go` + `mdl/backend/writestats.go` | Match the incoming document against the stored one and reuse its `$ID`s: `$Type` + shape one level down (`Action=Microflows$LogMessageAction`) + `Name` as the LCS match key, positional fill in the gaps, then substitute **in place over every 16-byte binary** — a pointer is a primitive property a containment walk never sees, and any occurrence of one of the document's element IDs *is* a reference (the same insight `canon` rests on). **The correctness bar is lower than it looks**: a wrong match only makes a diff bigger, since every reference moves with the element; the one real failure is two elements sharing an `$ID`, so guard it explicitly (`dropCollisions`, run to a fixed point) and verify the resulting id set by read-back. **Do not touch `GUID`** — that is the database's identity and was already preserved (measured 8 of 8 through `ALTER ENTITY ADD ATTRIBUTE`). **A key built from content is the trap**: it stops an element matching itself the moment someone edits it, which is the case being fixed — key on shape and name only. **Watch for the control you invalidate**: `MXCLI_ALWAYS_WRITE=1` no longer changes the written bytes (identities are carried), so `TestWriteMicroflowTwice_ControlChurnsWhenElisionOff` became unprovable and had to move down a layer to the raw codec output (`TestRebuildChurnsSubElementIDs`); from the shell, control on mtimes rather than hashes. For the reporting half, downgrade the verb only on **positive** evidence (unit writes offered since the last report, none landed) so a mutation that never reaches unit storage — a theme file, a mock backend — reads exactly as before. Measured on the reporter's own project: 37/37 identities kept, diff down to the one changed line, insert/delete mints only the genuinely new elements, `mx check` unchanged at its 1 pre-existing error, both engines | +| `import from mapping M.IMM($json)` builds cleanly and then throws at runtime: `MicroflowException: key not found: Path(QName(None,),None,)` at `com.mendix.integration.importer.mapping.MappingCache.storeValueMappingElement`. Reported as "import mapping documents are broken / JSON path resolution is broken in mxcli's serialization" | **Not the mapping document — the ACTIVITY.** The Range and the result variable's cardinality are separate axes (#881), but an **unauthored** range set neither pointer, so `ForceSingleOccurrence` and `ConstantRange.SingleObject` both fell back to `SingleObject` — true for an object-rooted mapping. That is Studio Pro's **First** ("take one of a list"), a different activity. Studio Pro writes **both flags false** for a plain single-object import and expresses "one object" solely through `VariableType=ObjectType`. `all` and `first` and limit/offset all set the pointers explicitly, so only the **bare** form — the one every doc and example uses — was broken | `mdl/executor/cmd_microflows_builder_calls.go` (`addImportFromMappingAction`: add the `else` branch writing both pointers false); tests `mdl/executor/cmd_microflows_import_range_test.go`; fixture `mdl-examples/bug-tests/import-mapping-single-object-runtime.mdl` | **A valid model that fails only at runtime**: `mxcli check`, `mx check` (0 errors) and mxbuild all pass — the document is well-formed, it just means something else. The repo had **no runtime coverage of import mappings at all**; every existing test stopped at `mx check`, which is exactly the gap `.claude/skills/verify-in-runtime.md` exists for. **Get a Studio Pro-authored app and cross the variables** — `ako/TestApp` settled this in minutes after days of BSON diffing: run the SP microflow and an mxcli one *in the same boot*, then cross them (mxcli microflow → SP mapping, and mxcli mapping → SP structure+entity). The cross-test is what separated the activity from the document; an "isolation" where BOTH artifacts are mxcli's isolates nothing, and I asserted the wrong culprit twice before running it. **A control is only evidence if its harness is known good** — an earlier run had Studio Pro's own mapping failing too, which "exonerated" mxcli, but the probe app was independently broken. **Diff the ENCODING, not just the values**: `bson.M` loses key order and non-canonical extended JSON hides `int32` vs `int64`; dump with `MarshalExtJSONIndent(doc, true, …)` and compare raw key order via file order. Four differences found that way were all real and none causal (int32-vs-int64 on 14 numerics, root `MinOccurs` 0 vs 1, missing `MessageDefinition2`, blanked `OriginalValue`) — chasing document differences was the wrong tree entirely. Also: #882's "Studio Pro leaves `OriginalValue` empty" is contradicted by a second app, so it was generalised from too small a sample | diff --git a/mdl-examples/bug-tests/import-mapping-single-object-runtime.mdl b/mdl-examples/bug-tests/import-mapping-single-object-runtime.mdl new file mode 100644 index 000000000..feb0c9e1a --- /dev/null +++ b/mdl-examples/bug-tests/import-mapping-single-object-runtime.mdl @@ -0,0 +1,90 @@ +-- Bug test: `import from mapping` with NO range keyword built a document the +-- Mendix RUNTIME refuses, while every static check passed. +-- +-- com.mendix.modules.microflowengine.MicroflowException: +-- key not found: Path(QName(None,),None,) +-- at com.mendix.integration.importer.mapping.MappingCache.storeValueMappingElement +-- +-- WHY NO CHECK CAUGHT IT: the model is valid. `mxcli check` passes, `mx check` +-- reports 0 errors, mxbuild builds. The failure happens only when the activity +-- actually runs, and the repo had no runtime coverage of import mappings at all +-- — every existing mapping test stops at `mx check`. +-- +-- ROOT CAUSE: the Range and the result variable's cardinality are two axes +-- (issue #881). An unauthored range set NEITHER pointer, so both +-- ForceSingleOccurrence and ConstantRange.SingleObject fell back to +-- SingleObject — true for an object-rooted mapping — which is Studio Pro's +-- FIRST ("take one of a list"), not a single-object import. Studio Pro writes +-- both flags FALSE and expresses "one object" solely through +-- VariableType=ObjectType. +-- +-- MEASURED: against a Studio Pro-authored app on 11.13.0, in a single boot over +-- the same mapping and the same JSON, `... all` imported and the bare form threw. +-- Studio Pro's own microflow (ako/TestApp) stores ForceSingleOccurrence=false, +-- ConstantRange{SingleObject:false}, VariableType=ObjectType. +-- +-- To verify at runtime (not just check): +-- mxcli run --local -p app.mpr # wire ACT_ImportFlat to after-startup +-- and confirm the log shows the imported value instead of the exception above. + +create module ImportSingle; + +create non-persistent entity ImportSingle.Item ( + Title: string, + Qty: string +); + +create json structure ImportSingle.JSON_Item + snippet '{"title": "hello", "qty": "3"}'; + +create import mapping ImportSingle.IMM_Item + with json structure ImportSingle.JSON_Item +{ + create ImportSingle.Item { + Title = title, + Qty = qty + } +}; + +-- The regression: no range keyword. Must store ConstantRange{SingleObject:false} +-- and ForceSingleOccurrence=false against an ObjectType variable. +create microflow ImportSingle.ACT_ImportFlat () +returns string as $out +begin + declare $out string = 'EMPTY'; + declare $Json string = '{"title": "hello", "qty": "3"}'; + $Item = import from mapping ImportSingle.IMM_Item($Json); + if $Item != empty then + set $out = $Item/Title; + end if; + return $out; +end; +/ + +-- The control that always worked: an explicit range. If this one passes at +-- runtime and the bare form above does not, the regression is back. +create microflow ImportSingle.ACT_ImportAll () +returns string as $out +begin + declare $out string = 'EMPTY'; + declare $Json string = '{"title": "hello", "qty": "3"}'; + $Items = import from mapping ImportSingle.IMM_Item($Json) all; + set $out = 'ok'; + return $out; +end; +/ + +-- `first` keeps Studio Pro's First semantics — both flags true — and is +-- deliberately NOT changed by the fix. +create microflow ImportSingle.ACT_ImportFirst () +returns string as $out +begin + declare $out string = 'EMPTY'; + declare $Json string = '{"title": "hello", "qty": "3"}'; + $One = import from mapping ImportSingle.IMM_Item($Json) first; + set $out = 'ok'; + return $out; +end; +/ + +describe microflow ImportSingle.ACT_ImportFlat; diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index 253cfbe8c..8464c42ef 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -1562,6 +1562,21 @@ func (fb *flowBuilder) addImportFromMappingAction(s *ast.ImportFromMappingStmt) if s.OffsetExpr != nil { resultHandling.OffsetExpression = fb.exprToString(s.OffsetExpr) } + } else { + // No range keyword: the range is ALL, written explicitly. Leaving both + // pointers nil let them fall back to SingleObject — true for an + // object-rooted mapping — which stores Studio Pro's FIRST. That builds + // cleanly (mx check: 0 errors) and then throws at import time: + // + // MicroflowException: key not found: Path(QName(None,),None,) + // at ...importer.mapping.MappingCache.storeValueMappingElement + // + // Studio Pro writes both flags false for a plain single-object import and + // expresses "one object" solely through VariableType=ObjectType. Only the + // RANGE is set here; the cardinality inference below is untouched. + f := false + resultHandling.RangeSingleObject = &f + resultHandling.ForceSingleOccurrence = &f } // Only FIRST overrides the mapping's cardinality; saying nothing leaves both // axes to the inference, which is what keeps existing scripts writing what diff --git a/mdl/executor/cmd_microflows_import_range_test.go b/mdl/executor/cmd_microflows_import_range_test.go index 9b9387537..9f1a396cd 100644 --- a/mdl/executor/cmd_microflows_import_range_test.go +++ b/mdl/executor/cmd_microflows_import_range_test.go @@ -115,19 +115,23 @@ func TestImportRange_LimitIsARangeNotACardinality(t *testing.T) { } } -// Saying nothing must keep writing what mxcli always wrote: cardinality inferred -// from the mapping's root, range mirroring it. Every hand-written script that -// predates the syntax depends on this. -func TestImportRange_UnauthoredKeepsTheOldInference(t *testing.T) { +// Saying nothing must keep the CARDINALITY inference: inferred from the +// mapping's root. Every hand-written script that predates the syntax depends on +// this, and it is unchanged. +// +// What did change: the range is now written explicitly as All rather than left +// nil to fall back to the cardinality. This test previously asserted that +// fallback (RangeSingleObject == nil), which was the defect — for an +// object-rooted mapping it stored First and the runtime threw `key not found: +// Path(QName(None,),None,)`. It only passed because it exercised the +// list-rooted case, where the fallback is false either way and so harmless. +func TestImportRange_UnauthoredKeepsTheCardinalityInference(t *testing.T) { h := buildImportRange(t, true, &ast.ImportFromMappingStmt{}) - if h.RangeSingleObject != nil { - t.Errorf("RangeSingleObject = %v, want nil (unauthored)", *h.RangeSingleObject) - } if h.SingleObject { t.Error("SingleObject = true, want false — a list-rooted mapping still infers a list") } if microflows.RangeSingleObjectOf(h) { - t.Error("the stored range must fall back to the inferred cardinality when unauthored") + t.Error("range SingleObject = true, want false — an unauthored range is All") } } @@ -169,3 +173,37 @@ func TestFormatImportMappingRange(t *testing.T) { t.Errorf("nil handling = %q, want empty", got) } } + +// upstream: `import from mapping M.IMM($json)` with no range keyword produced a +// document the RUNTIME refuses, while mx check reported 0 errors: +// +// MicroflowException: key not found: Path(QName(None,),None,) +// at com.mendix.integration.importer.mapping.MappingCache.storeValueMappingElement +// +// Unauthored set NEITHER range pointer, so both ForceSingleOccurrence and +// ConstantRange.SingleObject fell back to h.SingleObject — true for an +// object-rooted mapping. Studio Pro writes BOTH false there and expresses "one +// object" purely through VariableType=ObjectType; SingleObject=true means +// "First" (take one of a list), which is a different activity. +// +// Measured against a Studio Pro-authored app on 11.13.0: in one boot, `all` +// imported and the bare form threw, over the same mapping and the same JSON. +func TestImportRange_UnauthoredObjectRootedWritesAllNotFirst(t *testing.T) { + h := buildImportRange(t, false, &ast.ImportFromMappingStmt{}) + + if microflows.RangeSingleObjectOf(h) { + t.Error("range SingleObject = true, want false — an unauthored range is All, " + + "not First; First makes the runtime resolve an occurrence path and fail " + + "with `key not found: Path(QName(None,),None,)`") + } + if h.ForceSingleOccurrence == nil || *h.ForceSingleOccurrence { + t.Errorf("ForceSingleOccurrence = %v, want explicit false (Studio Pro's shape)", + h.ForceSingleOccurrence) + } + // The cardinality axis is untouched: an object-rooted mapping still binds an + // OBJECT, or mxbuild rejects the list with CE0243. + if !h.SingleObject { + t.Error("SingleObject = false, want true — the object-rooted mapping still " + + "binds an object; only the RANGE changed") + } +} From cbed3b08a8d62da02fe661a728c2f439accfd1cc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 20:09:51 +0000 Subject: [PATCH 12/21] fix(rest): refuse `body: file`, which sent the expression text (MDL-REST02) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Body: file from $Doc` on a consumed REST operation sent the four bytes `$Doc` instead of the document. Measured against httpbingo with an 8090-byte PNG: Content-Length: 4 data:application/octet-stream;base64,JERvYw== -> b'$Doc' mxcli check passed, mx check reported 0 errors and the call returned HTTP 200. Every signal a user or an agent checks said success. Both engines folded FILE into the TEMPLATE branch and wrote a Rest$StringBody whose ValueTemplate is the expression text — consumed_rest_write.go:218 (the default modelsdk engine) and writer_rest.go:250 (legacy). `describe` renders it back as `Body: template '$Doc'`, so the round trip looked self-consistent. There is no better type to write. Mendix's 11.13 metamodel has exactly three request-body types — Rest$JsonBody, Rest$StringBody, Rest$ImplicitMappingBody — and none is binary, so binary upload is not expressible in MDL at all. The fix is therefore to refuse the clause, as MDL-REST01 refuses a mapping document in an inline mapping, rather than degrade it into something that looks like it works. A Java action is the route. One function guards both the check pass and exec, so `mxcli check` and `mxcli exec` cannot disagree — and because it sits in buildRestClientOperation, `--no-check` does not reopen the silent write either. The refusal is narrow: `Response: file as $Doc` downloads correctly and is untouched. Two shipped examples advertised this as a working "File Upload" feature and never worked; 06-rest-client-examples.mdl is corrected — one is rewritten as the download that does work, the other replaced by a note. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Dp1syGhH8yr7Hve2wwjzqg --- .claude/skills/fix-issue.md | 1 + cmd/mxcli/syntax/features_integration.go | 2 +- .../bug-tests/rest-file-request-body.fail.mdl | 39 +++++++++ .../doctype-tests/06-rest-client-examples.mdl | 38 +++++---- mdl/executor/cmd_rest_clients.go | 44 ++++++++++ .../cmd_rest_clients_file_body_test.go | 84 +++++++++++++++++++ mdl/executor/validate_rest_mapping.go | 30 ++++--- 7 files changed, 209 insertions(+), 29 deletions(-) create mode 100644 mdl-examples/bug-tests/rest-file-request-body.fail.mdl create mode 100644 mdl/executor/cmd_rest_clients_file_body_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index bb986e7df..682f0133d 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -570,3 +570,4 @@ extracting `OffsetExpression`/`LimitExpression`. | Windows: `run --local` behaves as though Java were undetected, and a local run can fail looking for Gradle | Two separate things. (a) **The `.exe` suffix**: `isJDK21` appended it, every CONSUMER did not — `--java-exe-path` handed to mxbuild and the path `exec.Command` runs to boot the runtime were both `\bin\java`, so a correctly-detected JDK was passed on in a form that need not resolve. Five sites built it by hand. (b) **Gradle is not mxcli's**: it ships inside the mxbuild bundle (`modeler/tools/gradle`, 8.5) and mxbuild invokes it — mxcli never calls gradle, so "Gradle missing" indicates a foreign/incomplete mxbuild bundle, usually #916 | `cmd/mxcli/docker/javaexe.go` (new `JavaExePath`) used from `mxserve.go`, `build.go`, `settle.go`, `localboot.go`, `detect.go`; `jdkSearchPathsFor` in `detect.go` | One helper, not five hand-built joins — the sixth call site would have repeated the bug. **Check the platform's own docs before adding a search path**: "add Studio Pro's JDK" turned out to be a non-task, because Mendix's install guide says Studio Pro installs **Eclipse Temurin 21** rather than bundling a JDK, so the existing Adoptium glob already IS Studio Pro's JDK; what was genuinely missing was the per-user `%LOCALAPPDATA%\Programs` install location. Inventing a `Mendix\\jdk` path would have been dead code. Make the not-found error list what was searched — "JDK 21 not found" alone sends a user reading mxcli's source. Inject `goos` (`jdkSearchPathsFor`, `javaExeName`): the only Windows CI job is the tunnel seam, scoped with `-run`, so it compiles Windows code and executes almost none of it — which is exactly how a missing `.exe` survives. **Unverified**: no Windows host; the fix is code-level with OS-injected tests. Reported via a user relay, not an issue | | Studio Pro's version-control view shows an **entire** nanoflow/microflow as changed after editing one activity argument; `git diff` on `mprcontents/` is unreadable. A change and its revert leave a semantically identical document that shares no element IDs with the original. Separately, re-running an already-applied script prints `Modified …`/`Replaced …` for files it did not touch | Two independent things. (a) `create or replace` rebuilds the document and every sub-element gets a freshly random `$ID`; elision (ADR-0008) only covers the case where *nothing* changed, so a real change wrote a whole new identity set — measured 36 of 37 on a nanoflow, 21 of 22 on a microflow. (b) The `Modified …` lines are printed by the handler right after `ctx.Backend.Update*`, which returns nil whether or not storage elided the write | `modelsdk/canon/transplant.go` (`TransplantIDs`, called from `Reconcile` in `identity.go`); `mdl/executor/report_mutation.go` + `mdl/backend/writestats.go` | Match the incoming document against the stored one and reuse its `$ID`s: `$Type` + shape one level down (`Action=Microflows$LogMessageAction`) + `Name` as the LCS match key, positional fill in the gaps, then substitute **in place over every 16-byte binary** — a pointer is a primitive property a containment walk never sees, and any occurrence of one of the document's element IDs *is* a reference (the same insight `canon` rests on). **The correctness bar is lower than it looks**: a wrong match only makes a diff bigger, since every reference moves with the element; the one real failure is two elements sharing an `$ID`, so guard it explicitly (`dropCollisions`, run to a fixed point) and verify the resulting id set by read-back. **Do not touch `GUID`** — that is the database's identity and was already preserved (measured 8 of 8 through `ALTER ENTITY ADD ATTRIBUTE`). **A key built from content is the trap**: it stops an element matching itself the moment someone edits it, which is the case being fixed — key on shape and name only. **Watch for the control you invalidate**: `MXCLI_ALWAYS_WRITE=1` no longer changes the written bytes (identities are carried), so `TestWriteMicroflowTwice_ControlChurnsWhenElisionOff` became unprovable and had to move down a layer to the raw codec output (`TestRebuildChurnsSubElementIDs`); from the shell, control on mtimes rather than hashes. For the reporting half, downgrade the verb only on **positive** evidence (unit writes offered since the last report, none landed) so a mutation that never reaches unit storage — a theme file, a mock backend — reads exactly as before. Measured on the reporter's own project: 37/37 identities kept, diff down to the one changed line, insert/delete mints only the genuinely new elements, `mx check` unchanged at its 1 pre-existing error, both engines | | `import from mapping M.IMM($json)` builds cleanly and then throws at runtime: `MicroflowException: key not found: Path(QName(None,),None,)` at `com.mendix.integration.importer.mapping.MappingCache.storeValueMappingElement`. Reported as "import mapping documents are broken / JSON path resolution is broken in mxcli's serialization" | **Not the mapping document — the ACTIVITY.** The Range and the result variable's cardinality are separate axes (#881), but an **unauthored** range set neither pointer, so `ForceSingleOccurrence` and `ConstantRange.SingleObject` both fell back to `SingleObject` — true for an object-rooted mapping. That is Studio Pro's **First** ("take one of a list"), a different activity. Studio Pro writes **both flags false** for a plain single-object import and expresses "one object" solely through `VariableType=ObjectType`. `all` and `first` and limit/offset all set the pointers explicitly, so only the **bare** form — the one every doc and example uses — was broken | `mdl/executor/cmd_microflows_builder_calls.go` (`addImportFromMappingAction`: add the `else` branch writing both pointers false); tests `mdl/executor/cmd_microflows_import_range_test.go`; fixture `mdl-examples/bug-tests/import-mapping-single-object-runtime.mdl` | **A valid model that fails only at runtime**: `mxcli check`, `mx check` (0 errors) and mxbuild all pass — the document is well-formed, it just means something else. The repo had **no runtime coverage of import mappings at all**; every existing test stopped at `mx check`, which is exactly the gap `.claude/skills/verify-in-runtime.md` exists for. **Get a Studio Pro-authored app and cross the variables** — `ako/TestApp` settled this in minutes after days of BSON diffing: run the SP microflow and an mxcli one *in the same boot*, then cross them (mxcli microflow → SP mapping, and mxcli mapping → SP structure+entity). The cross-test is what separated the activity from the document; an "isolation" where BOTH artifacts are mxcli's isolates nothing, and I asserted the wrong culprit twice before running it. **A control is only evidence if its harness is known good** — an earlier run had Studio Pro's own mapping failing too, which "exonerated" mxcli, but the probe app was independently broken. **Diff the ENCODING, not just the values**: `bson.M` loses key order and non-canonical extended JSON hides `int32` vs `int64`; dump with `MarshalExtJSONIndent(doc, true, …)` and compare raw key order via file order. Four differences found that way were all real and none causal (int32-vs-int64 on 14 numerics, root `MinOccurs` 0 vs 1, missing `MessageDefinition2`, blanked `OriginalValue`) — chasing document differences was the wrong tree entirely. Also: #882's "Studio Pro leaves `OriginalValue` empty" is contradicted by a second app, so it was generalised from too small a sample | +| `Body: file from $Doc` on a consumed REST operation sends the literal text `$Doc` instead of the file. `mxcli check` passes, `mx check` reports 0 errors, the request returns **HTTP 200** — and the payload is 4 bytes where the document held 8090 | Mendix has **no binary request body**. `generated/metamodel` has exactly three: `Rest$JsonBody`, `Rest$StringBody`, `Rest$ImplicitMappingBody`. Both engines folded `FILE` into the `TEMPLATE` branch and wrote a `Rest$StringBody` whose `ValueTemplate` is the **expression text** — `mdl/backend/modelsdk/consumed_rest_write.go:218` (default engine) and `sdk/mpr/writer_rest.go:250` (legacy). `describe` shows it straight back as `Body: template '$Doc'`, so the round trip looks consistent | `mdl/executor/cmd_rest_clients.go` (`checkFileRequestBody`, called from `buildRestClientOperation`); `mdl/executor/validate_rest_mapping.go` (MDL-REST02); tests `mdl/executor/cmd_rest_clients_file_body_test.go`; fixture `mdl-examples/bug-tests/rest-file-request-body.fail.mdl` | **Refuse; do not write a different type** — there is no correct type, so binary upload is not expressible in MDL and a Java action is the route. A silent downgrade that returns 200 is the worst outcome available: every signal a user or an agent checks says success. Same shape as MDL-REST01, and one function is called from **both** the check pass and exec so `mxcli check` and `mxcli exec` cannot disagree. **Fix both engines, not just the one the reporter cited** — the report named only `sdk/mpr`, but the default engine is `modelsdk` and had the identical branch; a legacy-only fix would have left the default path broken. Keep the refusal **narrow**: `Response: file as $Doc` downloads correctly and is untouched (its own defect is a `CHANGE` on the result, which is separate). Judge an upload by what the server **echoes** (httpbingo `/post` returns `Content-Length` and the body), never by "the call did not throw" — a 200 proved nothing here | diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index 5b2bde606..dbc4322fb 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -242,7 +242,7 @@ func init() { "body", "response", "mapping", "authentication", "json structure", "import mapping", "export mapping", }, - Syntax: "CREATE [OR MODIFY] REST CLIENT Module.Name (\n BaseUrl: 'https://...',\n Authentication: NONE | BASIC (...)\n)\n{\n OPERATION Name {\n Method: GET|POST|PUT|DELETE|PATCH,\n Path: '/path/{param}',\n Parameters: ($param: Type),\n Query: ($param: Type),\n Headers: ('Key' = 'Value'),\n Timeout: 30,\n Body: JSON FROM $var | MAPPING Entity { jsonField = Attribute, ... },\n Response: JSON AS $var | MAPPING Entity { Attribute = jsonField, ... }\n }\n};\n\n-- MAPPING takes a target ENTITY plus a body listing the JSON fields; Mendix\n-- stores it inline on the operation. An existing import/export mapping\n-- document cannot be referenced here (rejected as MDL-REST01).", + Syntax: "CREATE [OR MODIFY] REST CLIENT Module.Name (\n BaseUrl: 'https://...',\n Authentication: NONE | BASIC (...)\n)\n{\n OPERATION Name {\n Method: GET|POST|PUT|DELETE|PATCH,\n Path: '/path/{param}',\n Parameters: ($param: Type),\n Query: ($param: Type),\n Headers: ('Key' = 'Value'),\n Timeout: 30,\n Body: JSON FROM $var | MAPPING Entity { jsonField = Attribute, ... },\n Response: JSON AS $var | MAPPING Entity { Attribute = jsonField, ... }\n }\n};\n\n-- MAPPING takes a target ENTITY plus a body listing the JSON fields; Mendix\n-- stores it inline on the operation. An existing import/export mapping\n-- document cannot be referenced here (rejected as MDL-REST01).\n-- There is no FILE request body: Mendix's consumed operation stores one of\n-- Rest$JsonBody, Rest$StringBody or Rest$ImplicitMappingBody, so a file\n-- document has nowhere to go. `Body: FILE FROM $Doc` is rejected as\n-- MDL-REST02 rather than sent as the literal text \"$Doc\" (it used to be,\n-- returning 200 with a 4-byte payload). Upload binary from a Java action.\n-- `Response: FILE AS $Doc` is unaffected — downloads work.", Example: "CREATE REST CLIENT Module.PetStore (\n BaseUrl: 'https://petstore.example.com/api',\n Authentication: NONE\n)\n{\n OPERATION GetPet {\n Method: GET,\n Path: '/pets/{id}',\n Parameters: ($id: String),\n Query: ($verbose: String),\n Response: MAPPING Module.Pet {\n Name = name,\n Status = status\n }\n }\n};", SeeAlso: []string{"rest", "rest.published"}, }) diff --git a/mdl-examples/bug-tests/rest-file-request-body.fail.mdl b/mdl-examples/bug-tests/rest-file-request-body.fail.mdl new file mode 100644 index 000000000..d4e61dbd3 --- /dev/null +++ b/mdl-examples/bug-tests/rest-file-request-body.fail.mdl @@ -0,0 +1,39 @@ +-- Negative test: `Body: file from $Doc` must be REFUSED (MDL-REST02). +-- +-- It used to be accepted and silently degraded. Both engines folded FILE into +-- the TEMPLATE branch and wrote a Rest$StringBody holding the EXPRESSION TEXT, +-- so the request sent the four bytes `$Doc` instead of the document: +-- +-- Content-Length: 4 +-- data:application/octet-stream;base64,JERvYw== -> b'$Doc' +-- +-- measured against httpbingo with an 8090-byte PNG. `mxcli check` passed, +-- `mx check` reported 0 errors, and the call returned HTTP 200 — every signal +-- said success while the payload was wrong. +-- +-- There is no correct type to write instead: Mendix's 11.13 metamodel has +-- exactly three request-body types (Rest$JsonBody, Rest$StringBody, +-- Rest$ImplicitMappingBody) and none is binary, so binary upload is not +-- expressible in MDL. Refusing is the fix; a Java action is the route. +-- +-- Writer sites that produced the bad document: +-- mdl/backend/modelsdk/consumed_rest_write.go (default engine) +-- sdk/mpr/writer_rest.go (legacy engine) + +create module FileBody; + +create entity FileBody.Upload extends System.FileDocument ( + Caption: string +); + +create rest client FileBody.RC_Upload ( + BaseUrl: 'https://httpbingo.org', + Authentication: NONE +) +{ + OPERATION PostFile { + Method: POST, + Path: '/post', + Body: FILE FROM $Doc + } +}; diff --git a/mdl-examples/doctype-tests/06-rest-client-examples.mdl b/mdl-examples/doctype-tests/06-rest-client-examples.mdl index 479c0528e..eecfdd6d5 100644 --- a/mdl-examples/doctype-tests/06-rest-client-examples.mdl +++ b/mdl-examples/doctype-tests/06-rest-client-examples.mdl @@ -372,19 +372,29 @@ create rest client RestTest.RC014_JsonBodyAPI ( }; -- ============================================================================ --- Level 6.2: File Upload +-- Level 6.2: File Upload — NOT SUPPORTED, and this example used to lie -- ============================================================================ +-- +-- `body: file from $Doc` was accepted here and sent the literal text `$Doc`. +-- Measured against httpbingo: Content-Length 4 for an 8090-byte PNG, HTTP 200, +-- `mx check` 0 errors. It is now refused as MDL-REST02. +-- +-- Mendix has no binary request body: a consumed operation stores one of +-- Rest$JsonBody, Rest$StringBody or Rest$ImplicitMappingBody. Upload binary +-- from a Java action, or send the contents as a string/JSON body if the +-- endpoint accepts that. +-- +-- Downloading is unaffected — `response: file as $Doc` works. -create rest client RestTest.RC015_FileUploadAPI ( +create rest client RestTest.RC015_FileDownloadAPI ( BaseUrl: 'https://httpbin.org', authentication: none ) { - operation UploadFile { - method: post, - path: '/post', - body: file from $FileDocument, - response: json as $UploadResult + operation DownloadFile { + method: get, + path: '/image/png', + response: file as $Downloaded } }; @@ -560,17 +570,9 @@ create rest client RestTest.RC018_PetStoreAPI ( response: none } - /** - * Upload pet image - * @param petId The pet to associate the image with - */ - operation UploadImage { - method: post, - path: '/pet/{petId}/uploadImage', - parameters: ($petId: integer), - body: file from $ImageFile, - response: json as $UploadResult - } + -- Upload pet image: NOT expressible. `body: file from $ImageFile` is refused + -- as MDL-REST02 — Mendix has no binary request body, and mxcli used to send + -- the expression text instead of the file. Use a Java action. }; -- ============================================================================ diff --git a/mdl/executor/cmd_rest_clients.go b/mdl/executor/cmd_rest_clients.go index ebc867eb3..502239520 100644 --- a/mdl/executor/cmd_rest_clients.go +++ b/mdl/executor/cmd_rest_clients.go @@ -442,6 +442,9 @@ func buildRestClientOperation(opDef *ast.RestOperationDef) (*model.RestClientOpe if err := checkInlineMappingBody(opDef); err != nil { return nil, err } + if err := checkFileRequestBody(opDef); err != nil { + return nil, err + } // model.RestClientOperation documents BodyType/ResponseType as upper-case // tokens ("JSON", "EXPORT_MAPPING", "MAPPING", ...) and every consumer // compares against that spelling — the serializers in both engines, and the @@ -552,6 +555,47 @@ func checkInlineMappingBody(opDef *ast.RestOperationDef) error { return nil } +// checkFileRequestBody rejects `Body: file from $Doc`. +// +// Mendix has no binary request body. The 11.13 metamodel offers exactly three +// implementations of a consumed operation's request body — Rest$JsonBody, +// Rest$StringBody and Rest$ImplicitMappingBody — so there is nowhere for a file +// document to go. Both serializers folded FILE into the TEMPLATE branch and +// wrote a Rest$StringBody holding the *expression text*, which meant the +// request sent the four bytes `$Doc` in place of the document: +// +// Content-Length: 4 +// data:application/octet-stream;base64,JERvYw== -> b'$Doc' +// +// measured against httpbingo with an 8090-byte PNG. `mxcli check` passed, +// `mx check` reported 0 errors and the call returned 200 — nothing anywhere +// said the payload was wrong. +// +// Refusing is the fix, not writing a different type: there is no correct type to +// write, and a silent downgrade that returns 200 is worse than an error. Binary +// upload needs a Java action. +// +// The RESPONSE side is deliberately untouched: `Response: file as $Doc` +// downloads correctly. +func checkFileRequestBody(opDef *ast.RestOperationDef) error { + if !strings.EqualFold(opDef.BodyType, "FILE") { + return nil + } + target := opDef.BodyVariable + if target == "" { + target = "$Doc" + } + return fmt.Errorf( + "Body: file from %s cannot be stored — Mendix has no binary request body.\n"+ + " A consumed REST operation's body is one of Rest$JsonBody, Rest$StringBody or\n"+ + " Rest$ImplicitMappingBody, so a file document has nowhere to go. mxcli used to\n"+ + " write a string body holding the literal text %q, which sends %d bytes and\n"+ + " still returns 200.\n"+ + " Send binary with a Java action instead, or post the file's contents as a\n"+ + " string/JSON body if the endpoint accepts that.", + target, target, len(target)) +} + // convertMappingEntries converts AST RestMappingEntry slices to model RestResponseMapping slices. // importDirection=true: Left=entityAttr, Right=jsonField (import/response) // importDirection=false: Left=jsonField, Right=entityAttr (export/body) diff --git a/mdl/executor/cmd_rest_clients_file_body_test.go b/mdl/executor/cmd_rest_clients_file_body_test.go new file mode 100644 index 000000000..f8b25a03d --- /dev/null +++ b/mdl/executor/cmd_rest_clients_file_body_test.go @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// `Body: file from $Doc` was written as a Rest$StringBody whose value is the +// EXPRESSION TEXT, so the request sent the four bytes `$Doc` instead of the +// document. Measured against httpbingo: `Content-Length: 4` for an 8090-byte +// PNG, HTTP 200, `mxcli check` clean and `mx check` 0 errors — the worst shape +// a defect can take. +// +// There is no better type to write: Mendix's 11.13 metamodel has exactly three +// request-body types (Rest$JsonBody, Rest$StringBody, Rest$ImplicitMappingBody) +// and none of them is binary. So the fix is to REFUSE the clause, the way +// MDL-REST01 refuses a mapping document in an inline mapping, rather than +// degrade it to something that looks like it works. +func TestBuildRestClientOperation_RefusesFileRequestBody(t *testing.T) { + _, err := buildRestClientOperation(&ast.RestOperationDef{ + Name: "Upload", + BodyType: "file", + BodyVariable: "$Doc", + }) + if err == nil { + t.Fatal("a file request body must be refused: writing it sends the expression text, not the file") + } + for _, want := range []string{"binary", "$Doc", "Java action"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should mention %q, got:\n%v", want, err) + } + } +} + +// The check-time pass must catch it too, so `mxcli check` refuses the script +// before anything is written — same function behind both, as with MDL-REST01. +func TestValidateRestClientMappings_ReportsFileRequestBody(t *testing.T) { + prog := &ast.Program{Statements: []ast.Statement{ + &ast.CreateRestClientStmt{ + Name: ast.QualifiedName{Module: "M", Name: "RC"}, + Operations: []*ast.RestOperationDef{ + {Name: "Upload", BodyType: "file", BodyVariable: "$Doc"}, + }, + }, + }} + violations := ValidateRestClientMappings(prog) + if len(violations) == 0 { + t.Fatal("check must report a file request body") + } + v := violations[0] + if v.RuleID != "MDL-REST02" { + t.Errorf("RuleID = %q, want MDL-REST02", v.RuleID) + } + if !strings.Contains(v.Message, "Upload") { + t.Errorf("message should name the operation, got %q", v.Message) + } +} + +// The controls. Refusing too much would break every working script, and the +// RESPONSE side is a separate matter — downloading a file works, and only the +// CHANGE on the result is broken (a different defect, not this one). +func TestBuildRestClientOperation_FileBodyRefusalIsNarrow(t *testing.T) { + for _, tc := range []struct { + name string + def *ast.RestOperationDef + }{ + {"json body", &ast.RestOperationDef{Name: "A", BodyType: "json", BodyVariable: "$x"}}, + {"template body", &ast.RestOperationDef{Name: "B", BodyType: "template", BodyVariable: "'hi'"}}, + {"file RESPONSE is not a request body", &ast.RestOperationDef{ + Name: "C", ResponseType: "file", ResponseVariable: "$Doc", + }}, + {"no body at all", &ast.RestOperationDef{Name: "D"}}, + } { + t.Run(tc.name, func(t *testing.T) { + if _, err := buildRestClientOperation(tc.def); err != nil { + t.Errorf("must be accepted, got: %v", err) + } + }) + } +} diff --git a/mdl/executor/validate_rest_mapping.go b/mdl/executor/validate_rest_mapping.go index c28d633a8..186b777be 100644 --- a/mdl/executor/validate_rest_mapping.go +++ b/mdl/executor/validate_rest_mapping.go @@ -29,17 +29,27 @@ func ValidateRestClientMappings(prog *ast.Program) []linter.Violation { continue } for _, opDef := range createStmt.Operations { - err := checkInlineMappingBody(opDef) - if err == nil { - continue + if err := checkInlineMappingBody(opDef); err != nil { + out = append(out, linter.Violation{ + RuleID: "MDL-REST01", + Severity: linter.SeverityError, + Message: "operation \"" + opDef.Name + "\": " + err.Error(), + Suggestion: "List the JSON fields inline. A consumed REST operation stores its mapping on the " + + "operation itself, so an existing import/export mapping document cannot be referenced here.", + }) + } + // MDL-REST02: a file request body has no representation in Mendix and + // used to be downgraded to a string body holding the expression text, + // which returns 200 with the wrong payload. + if err := checkFileRequestBody(opDef); err != nil { + out = append(out, linter.Violation{ + RuleID: "MDL-REST02", + Severity: linter.SeverityError, + Message: "operation \"" + opDef.Name + "\": " + err.Error(), + Suggestion: "Upload binary from a Java action. Mendix's consumed REST operation has no " + + "binary request body, so there is no syntax here that would send the file.", + }) } - out = append(out, linter.Violation{ - RuleID: "MDL-REST01", - Severity: linter.SeverityError, - Message: "operation \"" + opDef.Name + "\": " + err.Error(), - Suggestion: "List the JSON fields inline. A consumed REST operation stores its mapping on the " + - "operation itself, so an existing import/export mapping document cannot be referenced here.", - }) } } return out From 79c473f5d5bc1a086775a845a1f5656dd316edd1 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 21:10:36 +0000 Subject: [PATCH 13/21] feat(rest): binary request body on REST CALL (`body binary $Doc/Contents`) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Binary POST was reported as impossible in MDL — a consumed REST operation has only Rest$JsonBody, Rest$StringBody and Rest$ImplicitMappingBody, none binary, so the previous commit refused `Body: file` and pointed at a Java action. That was the right conclusion about the wrong document. Mendix models a binary request body on the microflow REST CALL activity: "RequestHandling": {"$Type": "Microflows$BinaryRequestHandling", "Expression": "$FirstFileDoc/Contents"}, "RequestHandlingType": "Binary" It is a Microflows$ type, which is why grepping the metamodel for Rest$*Body finds only the three non-binary ones and appears to prove it impossible. A Studio Pro-authored example (ako/TestApp, 11.13.0) settled it. mxcli could PARSE that shape and could neither write, read on the modelsdk engine, nor describe it. So a Studio Pro binary POST described as a REST call with no body at all, and re-executing that DESCRIBE produced a request that sent nothing. Wired full-stack: grammar (`BODY BINARY expression`), AST, visitor, builder, both writers, the modelsdk reader and the DESCRIBE formatter. The expression is the FileDocument's Contents MEMBER and is carried as source text — quoting it would send the path as a string literal. RequestHandlingType was hardcoded "Custom" in both engines regardless of the handler. Only the Binary case is derived; the others are left alone, having no measured Studio Pro reference and working today. Verified by re-executing mxcli's DESCRIBE of the Studio Pro microflow and diffing the BSON: same $Type, same discriminator, same expression, mxbuild 0 errors. `go test ./...` green, `make check-mdl` 354 PASS / 0 FAIL. MDL-REST02's message, the syntax help, the shipped examples and the symptom table now point at this route instead of a Java action. Adds cmd/bsondump, a dev helper that prints a .mxunit as canonical extended JSON — the technique the symptom table prescribes for this class of bug (plain bson.M loses key order and hides int32 vs int64). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Dp1syGhH8yr7Hve2wwjzqg --- .claude/skills/fix-issue.md | 3 +- cmd/bsondump/main.go | 34 ++++++++ cmd/mxcli/syntax/features_integration.go | 6 +- docs/01-project/MDL_QUICK_REFERENCE.md | 1 + mdl-examples/bug-tests/rest-binary-post.mdl | 45 +++++++++++ .../bug-tests/rest-file-request-body.fail.mdl | 9 ++- .../doctype-tests/06-rest-client-examples.mdl | 13 +-- mdl/ast/ast_microflow.go | 6 +- .../modelsdk/microflow_read_actions.go | 8 ++ mdl/backend/modelsdk/microflow_write.go | 15 +++- .../cmd_microflows_binary_body_test.go | 79 +++++++++++++++++++ mdl/executor/cmd_microflows_builder_calls.go | 8 ++ mdl/executor/cmd_microflows_format_action.go | 8 ++ mdl/executor/cmd_rest_clients.go | 7 +- .../cmd_rest_clients_file_body_test.go | 8 +- mdl/grammar/domains/MDLMicroflow.g4 | 3 +- mdl/visitor/visitor_microflow_actions.go | 10 ++- sdk/mpr/writer_microflow_actions.go | 20 ++++- 18 files changed, 261 insertions(+), 22 deletions(-) create mode 100644 cmd/bsondump/main.go create mode 100644 mdl-examples/bug-tests/rest-binary-post.mdl create mode 100644 mdl/executor/cmd_microflows_binary_body_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 682f0133d..9e38fd4d6 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -570,4 +570,5 @@ extracting `OffsetExpression`/`LimitExpression`. | Windows: `run --local` behaves as though Java were undetected, and a local run can fail looking for Gradle | Two separate things. (a) **The `.exe` suffix**: `isJDK21` appended it, every CONSUMER did not — `--java-exe-path` handed to mxbuild and the path `exec.Command` runs to boot the runtime were both `\bin\java`, so a correctly-detected JDK was passed on in a form that need not resolve. Five sites built it by hand. (b) **Gradle is not mxcli's**: it ships inside the mxbuild bundle (`modeler/tools/gradle`, 8.5) and mxbuild invokes it — mxcli never calls gradle, so "Gradle missing" indicates a foreign/incomplete mxbuild bundle, usually #916 | `cmd/mxcli/docker/javaexe.go` (new `JavaExePath`) used from `mxserve.go`, `build.go`, `settle.go`, `localboot.go`, `detect.go`; `jdkSearchPathsFor` in `detect.go` | One helper, not five hand-built joins — the sixth call site would have repeated the bug. **Check the platform's own docs before adding a search path**: "add Studio Pro's JDK" turned out to be a non-task, because Mendix's install guide says Studio Pro installs **Eclipse Temurin 21** rather than bundling a JDK, so the existing Adoptium glob already IS Studio Pro's JDK; what was genuinely missing was the per-user `%LOCALAPPDATA%\Programs` install location. Inventing a `Mendix\\jdk` path would have been dead code. Make the not-found error list what was searched — "JDK 21 not found" alone sends a user reading mxcli's source. Inject `goos` (`jdkSearchPathsFor`, `javaExeName`): the only Windows CI job is the tunnel seam, scoped with `-run`, so it compiles Windows code and executes almost none of it — which is exactly how a missing `.exe` survives. **Unverified**: no Windows host; the fix is code-level with OS-injected tests. Reported via a user relay, not an issue | | Studio Pro's version-control view shows an **entire** nanoflow/microflow as changed after editing one activity argument; `git diff` on `mprcontents/` is unreadable. A change and its revert leave a semantically identical document that shares no element IDs with the original. Separately, re-running an already-applied script prints `Modified …`/`Replaced …` for files it did not touch | Two independent things. (a) `create or replace` rebuilds the document and every sub-element gets a freshly random `$ID`; elision (ADR-0008) only covers the case where *nothing* changed, so a real change wrote a whole new identity set — measured 36 of 37 on a nanoflow, 21 of 22 on a microflow. (b) The `Modified …` lines are printed by the handler right after `ctx.Backend.Update*`, which returns nil whether or not storage elided the write | `modelsdk/canon/transplant.go` (`TransplantIDs`, called from `Reconcile` in `identity.go`); `mdl/executor/report_mutation.go` + `mdl/backend/writestats.go` | Match the incoming document against the stored one and reuse its `$ID`s: `$Type` + shape one level down (`Action=Microflows$LogMessageAction`) + `Name` as the LCS match key, positional fill in the gaps, then substitute **in place over every 16-byte binary** — a pointer is a primitive property a containment walk never sees, and any occurrence of one of the document's element IDs *is* a reference (the same insight `canon` rests on). **The correctness bar is lower than it looks**: a wrong match only makes a diff bigger, since every reference moves with the element; the one real failure is two elements sharing an `$ID`, so guard it explicitly (`dropCollisions`, run to a fixed point) and verify the resulting id set by read-back. **Do not touch `GUID`** — that is the database's identity and was already preserved (measured 8 of 8 through `ALTER ENTITY ADD ATTRIBUTE`). **A key built from content is the trap**: it stops an element matching itself the moment someone edits it, which is the case being fixed — key on shape and name only. **Watch for the control you invalidate**: `MXCLI_ALWAYS_WRITE=1` no longer changes the written bytes (identities are carried), so `TestWriteMicroflowTwice_ControlChurnsWhenElisionOff` became unprovable and had to move down a layer to the raw codec output (`TestRebuildChurnsSubElementIDs`); from the shell, control on mtimes rather than hashes. For the reporting half, downgrade the verb only on **positive** evidence (unit writes offered since the last report, none landed) so a mutation that never reaches unit storage — a theme file, a mock backend — reads exactly as before. Measured on the reporter's own project: 37/37 identities kept, diff down to the one changed line, insert/delete mints only the genuinely new elements, `mx check` unchanged at its 1 pre-existing error, both engines | | `import from mapping M.IMM($json)` builds cleanly and then throws at runtime: `MicroflowException: key not found: Path(QName(None,),None,)` at `com.mendix.integration.importer.mapping.MappingCache.storeValueMappingElement`. Reported as "import mapping documents are broken / JSON path resolution is broken in mxcli's serialization" | **Not the mapping document — the ACTIVITY.** The Range and the result variable's cardinality are separate axes (#881), but an **unauthored** range set neither pointer, so `ForceSingleOccurrence` and `ConstantRange.SingleObject` both fell back to `SingleObject` — true for an object-rooted mapping. That is Studio Pro's **First** ("take one of a list"), a different activity. Studio Pro writes **both flags false** for a plain single-object import and expresses "one object" solely through `VariableType=ObjectType`. `all` and `first` and limit/offset all set the pointers explicitly, so only the **bare** form — the one every doc and example uses — was broken | `mdl/executor/cmd_microflows_builder_calls.go` (`addImportFromMappingAction`: add the `else` branch writing both pointers false); tests `mdl/executor/cmd_microflows_import_range_test.go`; fixture `mdl-examples/bug-tests/import-mapping-single-object-runtime.mdl` | **A valid model that fails only at runtime**: `mxcli check`, `mx check` (0 errors) and mxbuild all pass — the document is well-formed, it just means something else. The repo had **no runtime coverage of import mappings at all**; every existing test stopped at `mx check`, which is exactly the gap `.claude/skills/verify-in-runtime.md` exists for. **Get a Studio Pro-authored app and cross the variables** — `ako/TestApp` settled this in minutes after days of BSON diffing: run the SP microflow and an mxcli one *in the same boot*, then cross them (mxcli microflow → SP mapping, and mxcli mapping → SP structure+entity). The cross-test is what separated the activity from the document; an "isolation" where BOTH artifacts are mxcli's isolates nothing, and I asserted the wrong culprit twice before running it. **A control is only evidence if its harness is known good** — an earlier run had Studio Pro's own mapping failing too, which "exonerated" mxcli, but the probe app was independently broken. **Diff the ENCODING, not just the values**: `bson.M` loses key order and non-canonical extended JSON hides `int32` vs `int64`; dump with `MarshalExtJSONIndent(doc, true, …)` and compare raw key order via file order. Four differences found that way were all real and none causal (int32-vs-int64 on 14 numerics, root `MinOccurs` 0 vs 1, missing `MessageDefinition2`, blanked `OriginalValue`) — chasing document differences was the wrong tree entirely. Also: #882's "Studio Pro leaves `OriginalValue` empty" is contradicted by a second app, so it was generalised from too small a sample | -| `Body: file from $Doc` on a consumed REST operation sends the literal text `$Doc` instead of the file. `mxcli check` passes, `mx check` reports 0 errors, the request returns **HTTP 200** — and the payload is 4 bytes where the document held 8090 | Mendix has **no binary request body**. `generated/metamodel` has exactly three: `Rest$JsonBody`, `Rest$StringBody`, `Rest$ImplicitMappingBody`. Both engines folded `FILE` into the `TEMPLATE` branch and wrote a `Rest$StringBody` whose `ValueTemplate` is the **expression text** — `mdl/backend/modelsdk/consumed_rest_write.go:218` (default engine) and `sdk/mpr/writer_rest.go:250` (legacy). `describe` shows it straight back as `Body: template '$Doc'`, so the round trip looks consistent | `mdl/executor/cmd_rest_clients.go` (`checkFileRequestBody`, called from `buildRestClientOperation`); `mdl/executor/validate_rest_mapping.go` (MDL-REST02); tests `mdl/executor/cmd_rest_clients_file_body_test.go`; fixture `mdl-examples/bug-tests/rest-file-request-body.fail.mdl` | **Refuse; do not write a different type** — there is no correct type, so binary upload is not expressible in MDL and a Java action is the route. A silent downgrade that returns 200 is the worst outcome available: every signal a user or an agent checks says success. Same shape as MDL-REST01, and one function is called from **both** the check pass and exec so `mxcli check` and `mxcli exec` cannot disagree. **Fix both engines, not just the one the reporter cited** — the report named only `sdk/mpr`, but the default engine is `modelsdk` and had the identical branch; a legacy-only fix would have left the default path broken. Keep the refusal **narrow**: `Response: file as $Doc` downloads correctly and is untouched (its own defect is a `CHANGE` on the result, which is separate). Judge an upload by what the server **echoes** (httpbingo `/post` returns `Content-Length` and the body), never by "the call did not throw" — a 200 proved nothing here | +| `Body: file from $Doc` on a consumed REST operation sends the literal text `$Doc` instead of the file. `mxcli check` passes, `mx check` reports 0 errors, the request returns **HTTP 200** — and the payload is 4 bytes where the document held 8090 | Mendix has **no binary request body**. `generated/metamodel` has exactly three: `Rest$JsonBody`, `Rest$StringBody`, `Rest$ImplicitMappingBody`. Both engines folded `FILE` into the `TEMPLATE` branch and wrote a `Rest$StringBody` whose `ValueTemplate` is the **expression text** — `mdl/backend/modelsdk/consumed_rest_write.go:218` (default engine) and `sdk/mpr/writer_rest.go:250` (legacy). `describe` shows it straight back as `Body: template '$Doc'`, so the round trip looks consistent | `mdl/executor/cmd_rest_clients.go` (`checkFileRequestBody`, called from `buildRestClientOperation`); `mdl/executor/validate_rest_mapping.go` (MDL-REST02); tests `mdl/executor/cmd_rest_clients_file_body_test.go`; fixture `mdl-examples/bug-tests/rest-file-request-body.fail.mdl` | **Refuse; do not write a different type** — there is no correct type for a CONSUMED OPERATION body. Binary POST is expressible, on the microflow REST CALL activity (`Microflows$BinaryRequestHandling`), which is where the refusal now points. A silent downgrade that returns 200 is the worst outcome available: every signal a user or an agent checks says success. Same shape as MDL-REST01, and one function is called from **both** the check pass and exec so `mxcli check` and `mxcli exec` cannot disagree. **Fix both engines, not just the one the reporter cited** — the report named only `sdk/mpr`, but the default engine is `modelsdk` and had the identical branch; a legacy-only fix would have left the default path broken. Keep the refusal **narrow**: `Response: file as $Doc` downloads correctly and is untouched (its own defect is a `CHANGE` on the result, which is separate). Judge an upload by what the server **echoes** (httpbingo `/post` returns `Content-Length` and the body), never by "the call did not throw" — a 200 proved nothing here | +| Binary upload is reported as impossible in MDL — "a consumed REST operation has no binary body, so use a Java action". A Studio Pro-authored binary POST also **describes with no body at all**, and re-executing that DESCRIBE produces a request that sends nothing | Right conclusion about the **wrong document**. Mendix models a binary request body on the microflow **REST CALL activity** as `Microflows$BinaryRequestHandling` + an action-level `RequestHandlingType: "Binary"` — a `Microflows$` type, which is why a metamodel grep for `Rest$*Body` finds only the three non-binary ones and "proves" it impossible. mxcli could **parse** it (`sdk/mpr/parser_microflow_actions.go`) and could neither write, read (modelsdk) nor describe it, so it survived a legacy read and vanished everywhere else | Grammar `mdl/grammar/domains/MDLMicroflow.g4` (`restCallBodyClause`); `mdl/ast/ast_microflow.go` (`RestBodyBinary`); `mdl/visitor/visitor_microflow_actions.go`; `mdl/executor/cmd_microflows_builder_calls.go`; writers `mdl/backend/modelsdk/microflow_write.go` + `sdk/mpr/writer_microflow_actions.go`; reader `mdl/backend/modelsdk/microflow_read_actions.go` (`restRequestHandlingFromRaw`); formatter `mdl/executor/cmd_microflows_format_action.go`; tests `mdl/executor/cmd_microflows_binary_body_test.go`; fixture `mdl-examples/bug-tests/rest-binary-post.mdl` | **"Not in the metamodel" needs the right namespace before it is a conclusion** — the search was for `Rest$…Body` and the answer lives under `Microflows$…RequestHandling`. Ask for a Studio Pro example instead of reasoning from an absence: one 4-KB unit settled in minutes what a metamodel grep had "disproved". **The discriminator and the sub-element must agree** — `RequestHandlingType` was hardcoded `"Custom"` in BOTH engines regardless of the handler; only the Binary case is derived here, because the others have no measured reference and work today (a latent mismatch worth a separate look). **The expression is the `Contents` MEMBER** (`$Doc/Contents`), not the document, and is stored as source text — quoting it sends the path as a string literal. **Verify a round trip by re-executing DESCRIBE and diffing the BSON against Studio Pro's**: mxcli's output reproduced the Studio Pro action exactly (same type, same discriminator, same expression) with mxbuild 0 errors, which is stronger than any assertion about what "should" be written. Beware the sibling trap: read support in one engine and not the other looks like a describe bug | diff --git a/cmd/bsondump/main.go b/cmd/bsondump/main.go new file mode 100644 index 000000000..f2dd2ae5a --- /dev/null +++ b/cmd/bsondump/main.go @@ -0,0 +1,34 @@ +// Command bsondump prints an MPR v2 .mxunit as indented canonical extended +// JSON, so numeric/boolean properties and their BSON types are readable. +// Development helper; not part of the shipped CLI surface. +package main + +import ( + "fmt" + "os" + + "go.mongodb.org/mongo-driver/bson" +) + +func main() { + if len(os.Args) != 2 { + fmt.Fprintln(os.Stderr, "usage: bsondump ") + os.Exit(2) + } + data, err := os.ReadFile(os.Args[1]) + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + var doc bson.M + if err := bson.Unmarshal(data, &doc); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + out, err := bson.MarshalExtJSONIndent(doc, true, false, "", " ") + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + fmt.Println(string(out)) +} diff --git a/cmd/mxcli/syntax/features_integration.go b/cmd/mxcli/syntax/features_integration.go index dbc4322fb..b4583a8ff 100644 --- a/cmd/mxcli/syntax/features_integration.go +++ b/cmd/mxcli/syntax/features_integration.go @@ -204,11 +204,13 @@ func init() { "rest call", "call rest service", "http get", "http post", "returns response", "returns string", "returns mapping", "file document", "filedocument", "download", "httpresponse", + "body binary", "binary", "upload", "post binary", }, Syntax: "[$Var =] REST CALL GET|POST|PUT|PATCH|DELETE '' [WITH ({1} = expr, ...)]\n" + " [HEADER 'Name' = expr]\n" + " [AUTH BASIC $user PASSWORD $pass]\n" + - " [BODY ...]\n" + + " [BODY '