From 4895932c1630489523a6f4565908758d9bd4c453 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 20:32:55 +0000 Subject: [PATCH 01/20] Stop a module's bundled widgets rolling back newer ones MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A module .mpk carries a copy of every widget its pages use, pinned at its author's release time, and different modules pin different versions of the same widget. InstallPackageFiles copied everything the package shipped, so the last module installed decided the project's widget versions — and nothing reported it, because an out-of-date widget is not a check error. Measured on the published packages: Atlas_Web_Content 4.3.0 ships five Data Widgets at 3.4.0 that DataWidgets 3.11.3 ships at 3.11.3, so updating Atlas after DataWidgets rolled all five back. A bundled widget now never replaces a newer copy, and the skip is reported with both versions. Verified against the real packages: installing DataWidgets 3.11.3 then Atlas_Web_Content 4.3.0 keeps exactly those five at 3.11.3. Reading the right version is the whole trick. package.xml carries a version on (the manifest schema, 1.0 on every widget ever published) and one on (the widget's own). Comparing the first makes every widget look equal, which is as broken as not comparing. An unparseable version is never treated as older: the default is to install, and a wrong "older" verdict withholds a file the package shipped. Same pass handles a package that ships a widget twice — FeedbackModule 5.0.0 carries SprintrFeedbackWidget as both a .mpk and an unpacked tree of the same version, and installing both left a duplicate. The .mpk wins; an unpacked widget with no packaged twin still installs. Separately: `marketplace update --no-baseline`. Both update and diff download the *installed* version to establish the local-edit baseline, so both fail when that version has been unpublished — a blank 11.13 app ships NanoflowCommons 6.0.0 and the 6.x line now starts at 6.1.1, so the module most in need of updating is the one whose baseline cannot be built. --force was never going to help: it overrides a finding, and the comparison never ran. The refusal now names --no-baseline, and that flag says plainly that local edits go without being named. Both reported from a real 11.13.0 provisioning run (mxcli-chat FINDINGS §14, §15, §18). Each new test verified to fail with the reported symptom when the fix is stubbed out. --- .claude/skills/fix-issue.md | 2 + .../mendix/download-marketplace-content.md | 55 +++++ cmd/mxcli/cmd_marketplace_install.go | 1 + cmd/mxcli/cmd_marketplace_update.go | 97 +++++--- cmd/mxcli/marketplace/update.go | 99 +++++++-- cmd/mxcli/marketplace/widgetversion.go | 148 +++++++++++++ cmd/mxcli/marketplace/widgetversion_test.go | 209 ++++++++++++++++++ docs-site/src/guides/marketplace.md | 17 ++ 8 files changed, 583 insertions(+), 45 deletions(-) create mode 100644 cmd/mxcli/marketplace/widgetversion.go create mode 100644 cmd/mxcli/marketplace/widgetversion_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index dcf2816cc..f9849be03 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -490,3 +490,5 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli new` into a deep output directory dies with `System.IO.PathTooLongException` and leaves the directory holding ~259 files and no `.mpr` — output that looks like a project until you try to open it | MxToolset refuses any full destination path over **259 characters** (its own Windows-compatibility limit, not the filesystem's) and aborts extraction PART WAY THROUGH. mxcli pointed `mx create-project` straight at the user's output directory, so the abort happened there | `cmd/mxcli/newproject_paths.go` (new: `stagedProjectDirs`, `longestRelativePath`, `warnIfPathTooLongForStudioPro`, `moveProject`), wired into `cmd/mxcli/cmd_new.go` step 2 | **Stage the work somewhere safe and move it in, rather than validating a path you could just avoid** — creating in a short temp dir and renaming makes ANY depth work (POSIX allows 4096) instead of merely failing politely, and the destination is never partially populated because nothing is written there until creation succeeded. Check the relocation is safe first: `grep -rl ` returned **0 files**, so a fresh Mendix project embeds no absolute paths. **Bisect for the real threshold instead of trusting arithmetic** — 77 characters creates a project on 11.13.0, 78 fails leaving 259 files, which pins `len(dest) + 1 + longest ≤ 259` exactly. **Measure the template, don't hardcode it**: the longest relative path is 181 on 11.13.0 and 182 on 11.12.0, so walk the staged tree and use the real number, or the reported budget drifts a character per release. **Warn, don't refuse, when the finished path is over budget** — the project works on POSIX, so refusing would block a machine where it is fine; but Studio Pro on Windows would not open it, so silence would be worse. Cross-device staging needs an `os.CopyFS` fallback: `os.Rename` cannot cross filesystems. Tests `cmd/mxcli/newproject_paths_test.go`. upstream #825 | | `show callers of ` and `show references to ` report "(no callers found)" for a document reached only from a page action button — a false negative that reads as "safe to delete" | TWO independent defects behind one symptom. (1) `scanWidgetOwnRefs` collected `Entity`/`Microflow`/`Nanoflow` from a widget's raw BSON but not **`Form`**, the key a PAGE reference uses, so `widgets_data` had no page column and the refs projection had no page row. (2) `execShowCallers` filtered `RefKind = 'call'` — the kind a microflow CALL ACTIVITY produces — so the button→microflow row, which was already in the refs table, was hidden by the query | `mdl/catalog/builder_pages.go` (`scanWidgetOwnRefs` + `rawWidgetInfo.PageRef`), `mdl/catalog/tables.go` (`widgets_data.PageRef`), `mdl/catalog/builder_references.go` (projection row), `mdl/executor/cmd_search.go` (`callerRefKinds`) | **Find the live code path before fixing anything** — `builder_references.go` has an inviting `extractWidgetRefs` with a per-widget-type switch, and it is DEAD: its only caller is its own recursion. Extending it changes nothing. The live path is a SQL projection out of `widgets_data`, and the standing `NOTE: widget-level datasource/action refs still require a parsed widget tree` comment beside it is the tell. **Separate "the reference is missing" from "the query hides it"** by reading the refs table directly: the button→microflow row was present all along, so fixing only the scanner would have closed half the issue and left the reporter's second scenario broken. **`Form` is `Page`** — the same rename behind `ShowFormAction`/`CloseFormAction` (CLAUDE.md's storage-name table); grepping for `Page` in a BSON scanner finds nothing. **One action can carry two references**: `create object … then open page` holds an entity AND a page, so collecting the entity alone still leaves the page unreferenced. **Do not widen `callers` into `references`** — `datasource`/`parameter`/`generalize` are uses of a TYPE, not invocations, and including them makes the two commands synonyms; the test pins both the included and the excluded set. Tests `builder_pages_test.go` (`TestScanWidgetOwnRefs_PageReference`), `cmd_search_callers_test.go`. upstream #773 | | `ALTER PAGE` over `--mcp` fails against Studio Pro **11.13** with `pg_patch_page: … PROP_NOT_PRIMITIVE: Property 'widgets' is not a primitive property`. `CREATE PAGE` is fine; the page itself is left intact | 11.13 gave `pg_read_page` a **`depth` argument defaulting to 4**, replacing anything deeper with the literal string `"..."`. ALTER PAGE is read-modify-**replace-whole-page**, so the truncated read went straight back as the new page body. Measured live: `Administration.Account_Overview` read 32,594 bytes at full depth but **1,052 bytes** at the default, its entire tree reduced to `{"widgets":["...","..."]}`. Every ordinary page truncates — three of three PgTest pages did | `mdl/backend/mcp/page.go` (`pgReadPage`, `pgReadFullDepth`, `hasTruncationSentinel`), `mdl/backend/mcp/client.go` (`SupportsToolArg`) | Request the full depth, and **guard rather than trust it**: refuse a read still carrying the sentinel instead of letting a partial page reach a write (ADR-0005 guard-don't-drop). Two traps. (1) **Do not send `depth` unconditionally** — 11.11/11.12 declare `pg_read_page` `additionalProperties:false` without it, so the whole call fails; gate on a live `tools/list` probe of the tool's input schema, because `serverInfo.version` is frozen at `1.0.0` across 11.11/11.12/11.13 and cannot discriminate releases. (2) **Match the sentinel only as an array element** — a caption or title legitimately reading `"..."` is real content, and a naive substring scan rejects valid pages. The release notes announced none of this, exactly as 11.12 silently removed `pg_write_page` (#697): on any Studio Pro upgrade, re-probe `tools/list` and diff the input schemas, not just the tool names. Tests `mdl/backend/mcp/page_depth_test.go`; controls: stub the depth arg (full-depth test fails) and stub the guard (truncation test fails) | +| Widgets silently go backwards: a project on DataWidgets 3.11.3 ends up running 3.4.0 widget code after updating an unrelated module (Atlas_Web_Content, FeedbackModule). `mx check` stays clean, `show modules` shows the new module version, and nothing mentions widgets | A module `.mpk` bundles a copy of every widget its pages use, pinned at its author's release time — and different modules pin different versions of the **same** widget. `InstallPackageFiles` copied everything the package shipped, so the last module installed decided the widget versions. Measured on the published packages: Atlas_Web_Content 4.3.0 ships 5 Data Widgets at 3.4.0 against DataWidgets 3.11.3's 3.11.3 | `cmd/mxcli/marketplace/widgetversion.go` (`widgetVersionInMpk`, `versionLess`, `unpackedTwinOf`), `update.go` (`InstallPackageFiles` returns `[]SkippedFile`), `cmd_marketplace_update.go` (`reportSkippedFiles`) | **Read ``, not ``** — the root attribute is the manifest schema and is `1.0` on every widget ever published, so comparing it makes every version look equal, which is exactly as broken as not comparing. **An unparseable version must never count as "older"** — the default is to install, and a wrong "older" verdict silently withholds a file the package shipped. **Skipping must be reported**: a silent skip is the same class of bug as the silent overwrite it replaces. **Test the negative half** — a guard that skips every widget also passes "the newer one survived"; assert that a newer *and* an equal bundled widget still install. Same file handles FINDINGS §18: a package that ships a widget as both `.mpk` and unpacked tree (FeedbackModule 5.0.0, both 12.0.4) installs only the `.mpk`, but an unpacked widget with **no** packaged twin must still install. Verified against the real packages: 5 kept, 13 twin entries skipped. Reported in mxcli-chat FINDINGS §14/§18 | +| `mxcli marketplace update`/`diff` fail with `version "6.0.0" not found` on a module the project actually has, and `--force` does not bypass it | Both download the **installed** version to build the local-edit baseline, and that version had been unpublished. A blank 11.13 app ships NanoflowCommons 6.0.0 while the 6.x line now starts at 6.1.1 — so the module most in need of updating is the one whose baseline cannot be built | `cmd/mxcli/cmd_marketplace_update.go` (`--no-baseline`, and the refusal now names it) | **`--force` was never going to work**: it overrides a *finding*, and here the comparison never ran, so there is no finding. Needing a second flag is the signal that these are two different decisions — "I accept losing edits I have been shown" vs "I accept not being shown". **Say what the flag costs in the flag's own message** — with no baseline, local edits go without being named, so the honest instruction is to commit first. Reported in mxcli-chat FINDINGS §15 | diff --git a/.claude/skills/mendix/download-marketplace-content.md b/.claude/skills/mendix/download-marketplace-content.md index 64700722b..1d93e0730 100644 --- a/.claude/skills/mendix/download-marketplace-content.md +++ b/.claude/skills/mendix/download-marketplace-content.md @@ -139,6 +139,61 @@ Dependencies include **widget content**, not only modules — `ConversationalUI` `Markdown viewer` (230248) and `Events` (224259) widget packages, which surface as `CE0462 "Could not find widget ... in the 'widgets' directory"`. +### Module packages bundle their own widgets — install order used to matter + +A module's `.mpk` carries a copy of every widget its pages use, pinned to +whatever its author had at release time, and different modules pin different +versions of the **same** widget. Measured on the published packages: +Atlas_Web_Content 4.3.0 ships five Data Widgets at **3.4.0** that DataWidgets +3.11.3 ships at **3.11.3**. + +`install` and `update` never roll a widget back: a bundled copy older than the +one in the project is kept out and reported. + +```text + Kept 5 newer widget(s) the package would have rolled back: + widgets/com.mendix.widget.web.Datagrid.mpk — kept 3.11.3, package ships 3.4.0 + ... +``` + +Before this, updating modules in one order and then another silently downgraded +widgets, and nothing surfaced it — an older widget is not a `mx check` error, so +the app just ran old widget code. If you are on an older mxcli, check the +versions by hand: + +```bash +for f in widgets/*.mpk; do + printf "%-50s %s\n" "$(basename $f)" \ + "$(unzip -p "$f" package.xml | grep -oP ']*version="\K[^"]+')" +done +``` + +Read ``, not the `` on the root element — +that one is the manifest schema and is `1.0` for every widget ever published. + +A package that ships a widget **twice** (as a `.mpk` and as an unpacked tree — +FeedbackModule 5.0.0 does) installs only the `.mpk`; the unpacked twin is skipped +and reported. + +### When the installed version has been unpublished + +`update` and `diff` download the *installed* version to establish the "has anyone +edited this?" baseline, so both fail when that version is gone from the +marketplace. A blank 11.13 app ships NanoflowCommons 6.0.0 and the 6.x line now +starts at 6.1.1, so the module most in need of updating is exactly the one whose +baseline cannot be built. + +```text +version "6.0.0" not found; run 'mxcli marketplace versions ' to list available versions + The installed version is the baseline for "has anyone edited this?", so it has to be + downloadable. It is not, and --force does not help: there is nothing to compare against. + hint: re-run with --no-baseline to update without that check (local edits are lost silently) +``` + +`--force` does not help — it overrides a *finding*, and here there is no finding. +`--no-baseline` accepts that the question cannot be answered and updates anyway. +Commit first: local edits to that module go without being named. + ## Step 4 — Repair the model after the install (required, headless) ```bash diff --git a/cmd/mxcli/cmd_marketplace_install.go b/cmd/mxcli/cmd_marketplace_install.go index 3836eb363..34e80ad99 100644 --- a/cmd/mxcli/cmd_marketplace_install.go +++ b/cmd/mxcli/cmd_marketplace_install.go @@ -192,6 +192,7 @@ func installModule(ctx context.Context, client *marketplace.Client, v *marketpla moduleName, v.VersionNumber, filepath.Base(mprPath)) fmt.Fprintf(out, " %d units copied, %d bundled file(s) installed.\n", res.UnitsCopied, len(res.FilesInstalled)) + reportSkippedFiles(out, res.FilesSkipped) // A headless install leaves the model needing two repairs only Mendix's own // tools can make (CE0463, CE6087). 'mxcli fix' runs them without the v2 -> // v1 conversion the bare mx commands perform. diff --git a/cmd/mxcli/cmd_marketplace_update.go b/cmd/mxcli/cmd_marketplace_update.go index b66da731c..193d934e2 100644 --- a/cmd/mxcli/cmd_marketplace_update.go +++ b/cmd/mxcli/cmd_marketplace_update.go @@ -69,6 +69,7 @@ func runMarketplaceUpdate(cmd *cobra.Command, args []string) error { moduleName, _ := cmd.Flags().GetString("module") saveEdits, _ := cmd.Flags().GetString("save-edits") force, _ := cmd.Flags().GetBool("force") + noBaseline, _ := cmd.Flags().GetBool("no-baseline") ctx := cmd.Context() out := cmd.OutOrStdout() @@ -112,35 +113,46 @@ func runMarketplaceUpdate(cmd *cobra.Command, args []string) error { } // Has anyone edited this module? Answering needs the version it was installed - // from, built as a reference exactly as `marketplace diff` does. - base, err := pickVersion(versions.Items, installedVersionID, installedVersion) - if err != nil { - return err - } - baseRef, basePkgModule, err := referenceFor(ctx, client, base, mendixVersion, work, "base") - if err != nil { - return err - } - installed, err := marketplace.SnapshotModule(mprPath, moduleName, newBackendFactory()) - if err != nil { - return fmt.Errorf("read %s from the project: %w", moduleName, err) - } - published, err := marketplace.SnapshotModule(baseRef, basePkgModule, newBackendFactory()) - if err != nil { - return fmt.Errorf("read %s from its published package: %w", basePkgModule, err) - } - drift := marketplace.Compare(installed, published) + // from, built as a reference exactly as `marketplace diff` does — which is + // impossible when that version has been unpublished. A blank 11.13 app ships + // NanoflowCommons 6.0.0 and the 6.x line now starts at 6.1.1, so the module + // that most needs updating is the one whose baseline cannot be built. + if noBaseline { + fmt.Fprintf(out, "\n--no-baseline: skipping the local-edit check for %s.\n", moduleName) + fmt.Fprintln(out, " Any local edits to this module will be discarded without being named.") + } else { + base, berr := pickVersion(versions.Items, installedVersionID, installedVersion) + if berr != nil { + return fmt.Errorf("%w\n"+ + " The installed version is the baseline for \"has anyone edited this?\", so it has to be\n"+ + " downloadable. It is not, and --force does not help: there is nothing to compare against.\n"+ + " hint: re-run with --no-baseline to update without that check (local edits are lost silently)", berr) + } + baseRef, basePkgModule, rerr := referenceFor(ctx, client, base, mendixVersion, work, "base") + if rerr != nil { + return rerr + } + installed, ierr := marketplace.SnapshotModule(mprPath, moduleName, newBackendFactory()) + if ierr != nil { + return fmt.Errorf("read %s from the project: %w", moduleName, ierr) + } + published, perr := marketplace.SnapshotModule(baseRef, basePkgModule, newBackendFactory()) + if perr != nil { + return fmt.Errorf("read %s from its published package: %w", basePkgModule, perr) + } + drift := marketplace.Compare(installed, published) - if saveEdits != "" { - written, unsaved, serr := marketplace.SaveEdits(saveEdits, drift) - if serr != nil { - return serr + if saveEdits != "" { + written, unsaved, serr := marketplace.SaveEdits(saveEdits, drift) + if serr != nil { + return serr + } + reportSavedEdits(out, saveEdits, written, unsaved) } - reportSavedEdits(out, saveEdits, written, unsaved) - } - if err := gateOnLocalEdits(out, drift, force, saveEdits); err != nil { - return err + if gerr := gateOnLocalEdits(out, drift, force, saveEdits); gerr != nil { + return gerr + } } // Build the version being moved to, and replace. @@ -233,6 +245,7 @@ func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { if len(r.FilesInstalled) > 0 { fmt.Fprintf(out, " %d bundled file(s) replaced (widgets, themesource, ...).\n", len(r.FilesInstalled)) } + reportSkippedFiles(out, r.FilesSkipped) if len(r.IdentitiesLost) > 0 { fmt.Fprintf(out, "\n Removed in %s (%d) — their database columns or tables will go on the next deploy:\n", @@ -272,7 +285,39 @@ func init() { marketplaceUpdateCmd.Flags().String("module", "", "module name in the project, when it cannot be identified automatically") marketplaceUpdateCmd.Flags().String("save-edits", "", "write locally changed elements to this directory as re-executable MDL") marketplaceUpdateCmd.Flags().Bool("force", false, "update even though local edits will be discarded") + marketplaceUpdateCmd.Flags().Bool("no-baseline", false, + "update without checking for local edits, when the installed version is no longer published") marketplaceUpdateCmd.Flags().String("profile", auth.ProfileDefault, "credential profile") marketplaceCmd.AddCommand(marketplaceUpdateCmd) } + +// reportSkippedFiles names the bundled files that were deliberately not written. +// +// A module package pins its own copy of every widget its pages use, and +// different modules pin different versions of the same widget — so installing +// modules in one order and then another used to roll widgets backwards with no +// sign that anything had happened (an older widget is not a check error). These +// lines are what makes that visible. +func reportSkippedFiles(out io.Writer, skipped []marketplace.SkippedFile) { + if len(skipped) == 0 { + return + } + var kept, dupes []marketplace.SkippedFile + for _, s := range skipped { + if s.Kept != "" { + kept = append(kept, s) + } else { + dupes = append(dupes, s) + } + } + if len(kept) > 0 { + fmt.Fprintf(out, "\n Kept %d newer widget(s) the package would have rolled back:\n", len(kept)) + for _, s := range kept { + fmt.Fprintf(out, " %s — kept %s, package ships %s\n", s.Path, s.Kept, s.Offered) + } + } + for _, s := range dupes { + fmt.Fprintf(out, " Skipped %s (%s).\n", s.Path, s.Reason) + } +} diff --git a/cmd/mxcli/marketplace/update.go b/cmd/mxcli/marketplace/update.go index 4fad86c31..cdf6ac380 100644 --- a/cmd/mxcli/marketplace/update.go +++ b/cmd/mxcli/marketplace/update.go @@ -95,18 +95,31 @@ func safeFileName(k ElementKey) string { // UpdateResult is what an update did. type UpdateResult struct { - Module string - FromVersion string - ToVersion string - UnitsCopied int - IdentitiesKept int - IdentitiesLost []string - GrantsRestored int - GrantsDropped []string - FilesInstalled []string + Module string + FromVersion string + ToVersion string + UnitsCopied int + IdentitiesKept int + IdentitiesLost []string + GrantsRestored int + GrantsDropped []string + FilesInstalled []string + // FilesSkipped records bundled files that were deliberately not installed: + // a widget older than the copy the project already has, or the unpacked twin + // of a widget the package also ships as a .mpk. Reported rather than silent — + // "the package wanted a different version" is the fact that was missing. + FilesSkipped []SkippedFile ForcedOverEdits []string } +// SkippedFile is one bundled file InstallPackageFiles chose not to write. +type SkippedFile struct { + Path string + Reason string + Kept string // version left in place, for a version skip + Offered string // version the package carried +} + // PerformUpdate replaces an installed module with the copy in referenceMpr, // preserving the two things that do not survive a plain replace: the `GUID`s the // database keys on (§8) and the user-role grants of the module's roles. @@ -152,7 +165,7 @@ func PerformUpdate(mprPath, referenceMpr, targetMpk, moduleName, fromVersion, to if err := StampMarketplaceVersion(mprPath, moduleName, toVersion, toVersionID); err != nil { return nil, fmt.Errorf("record the installed version: %w", err) } - files, err := InstallPackageFiles(targetMpk, filepath.Dir(mprPath)) + files, skippedFiles, err := InstallPackageFiles(targetMpk, filepath.Dir(mprPath)) if err != nil { return nil, fmt.Errorf("install the new version's bundled files: %w", err) } @@ -167,6 +180,7 @@ func PerformUpdate(mprPath, referenceMpr, targetMpk, moduleName, fromVersion, to GrantsRestored: restored, GrantsDropped: dropped, FilesInstalled: files, + FilesSkipped: skippedFiles, }, nil } @@ -273,13 +287,27 @@ func setBoolField(doc bson.D, key string, value bool) { // The second looked like a cross-module dependency on a newer Atlas. It was not. // Copying everything the package ships, rather than enumerating the directories // that seem to matter, is what stops there being a third instance. -func InstallPackageFiles(mpkPath, projectDir string) (written []string, err error) { +// Two entries are deliberately not installed, both measured on real packages — +// see widgetversion.go: a bundled widget older than the copy the project already +// has (it would silently roll the project back), and the unpacked twin of a +// widget the same package also ships as a .mpk. Both are returned in skipped +// rather than dropped quietly, because "the package wanted a different version" +// is exactly the thing that was invisible before. +func InstallPackageFiles(mpkPath, projectDir string) (written []string, skipped []SkippedFile, err error) { zr, err := zip.OpenReader(mpkPath) if err != nil { - return nil, fmt.Errorf("open package %s: %w", filepath.Base(mpkPath), err) + return nil, nil, fmt.Errorf("open package %s: %w", filepath.Base(mpkPath), err) } defer zr.Close() + // Which unpacked trees duplicate a .mpk in this same package. + twins := map[string]string{} + for _, f := range zr.File { + if prefix := unpackedTwinOf(f.Name); prefix != "" { + twins[prefix] = f.Name + } + } + for _, f := range zr.File { if f.FileInfo().IsDir() { continue @@ -288,34 +316,66 @@ func InstallPackageFiles(mpkPath, projectDir string) (written []string, err erro case packageProjectEntry, "package.xml": continue // the model and its manifest, handled by the transplant } + if twin, dup := duplicateOfPackagedWidget(f.Name, twins); dup { + skipped = append(skipped, SkippedFile{ + Path: f.Name, + Reason: "unpacked copy of " + twin + ", which this package also ships", + }) + continue + } // Refuse a path that escapes the project. Nothing in a Mendix package // should contain "..", and honouring one would let a package write // anywhere on disk. clean := filepath.Clean(f.Name) if strings.HasPrefix(clean, "..") || filepath.IsAbs(clean) { - return written, fmt.Errorf("package entry %q would write outside the project", f.Name) + return written, skipped, fmt.Errorf("package entry %q would write outside the project", f.Name) } dst := filepath.Join(projectDir, clean) if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { - return written, fmt.Errorf("create %s: %w", filepath.Dir(dst), err) + return written, skipped, fmt.Errorf("create %s: %w", filepath.Dir(dst), err) } rc, oerr := f.Open() if oerr != nil { - return written, fmt.Errorf("read %s from the package: %w", f.Name, oerr) + return written, skipped, fmt.Errorf("read %s from the package: %w", f.Name, oerr) } body, rerr := io.ReadAll(rc) _ = rc.Close() if rerr != nil { - return written, fmt.Errorf("read %s: %w", f.Name, rerr) + return written, skipped, fmt.Errorf("read %s: %w", f.Name, rerr) + } + // A bundled widget never rolls back a newer copy already in the project. + if isWidgetPackage(f.Name) { + have, want := widgetVersionOnDisk(dst), widgetVersionInMpk(body) + if have != "" && want != "" && versionLess(want, have) { + skipped = append(skipped, SkippedFile{ + Path: clean, + Reason: fmt.Sprintf("package ships %s, project already has %s", want, have), + Kept: have, + Offered: want, + }) + continue + } } if err := os.WriteFile(dst, body, 0o644); err != nil { - return written, fmt.Errorf("write %s: %w", dst, err) + return written, skipped, fmt.Errorf("write %s: %w", dst, err) } written = append(written, clean) } sort.Strings(written) - return written, nil + sort.Slice(skipped, func(i, j int) bool { return skipped[i].Path < skipped[j].Path }) + return written, skipped, nil +} + +// duplicateOfPackagedWidget reports whether an entry sits inside an unpacked +// widget tree that the same package also ships as a .mpk. +func duplicateOfPackagedWidget(name string, twins map[string]string) (string, bool) { + for prefix, mpk := range twins { + if strings.HasPrefix(name, prefix) { + return mpk, true + } + } + return "", false } // PerformInstall adds a module that is not yet in the project, copying it from @@ -336,7 +396,7 @@ func PerformInstall(mprPath, referenceMpr, packageMpk, moduleName, version, vers if err := StampMarketplaceVersion(mprPath, moduleName, version, versionID); err != nil { return nil, fmt.Errorf("record the installed version: %w", err) } - files, err := InstallPackageFiles(packageMpk, filepath.Dir(mprPath)) + files, skippedFiles, err := InstallPackageFiles(packageMpk, filepath.Dir(mprPath)) if err != nil { return nil, fmt.Errorf("install the package's bundled files: %w", err) } @@ -345,5 +405,6 @@ func PerformInstall(mprPath, referenceMpr, packageMpk, moduleName, version, vers ToVersion: version, UnitsCopied: copied, FilesInstalled: files, + FilesSkipped: skippedFiles, }, nil } diff --git a/cmd/mxcli/marketplace/widgetversion.go b/cmd/mxcli/marketplace/widgetversion.go new file mode 100644 index 000000000..8273e8716 --- /dev/null +++ b/cmd/mxcli/marketplace/widgetversion.go @@ -0,0 +1,148 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "archive/zip" + "bytes" + "encoding/xml" + "io" + "os" + "path" + "strconv" + "strings" +) + +// A module package is not only its model: it bundles copies of every widget its +// pages use. Those copies are pinned to whatever the module's author had at +// release time, and different modules pin different versions of the *same* +// widget. Measured on the 11.13 marketplace: Atlas_Web_Content 4.3.0 ships five +// Data Widgets at 3.4.0, which DataWidgets 3.11.3 ships at 3.11.3. +// +// So installing modules in one order and then another silently downgrades +// widgets — the project keeps whichever module was installed last. Nothing +// reports it: an older widget is not a `mx check` error, so the app just runs +// yesterday's widget code (mxcli-chat FINDINGS §14, where five Data Widgets and +// Charts were rolled back by a later module update). +// +// The rule below is therefore: a bundled widget never replaces a newer copy the +// project already has. Skipped replacements are reported, not silent — the +// caller has to be able to see that the package wanted a different version. + +// clientModuleVersion is the version a widget package declares for itself, in +// the `` element of its package.xml. The `` root attribute is the *manifest schema* version and is 1.0 for +// every widget ever published — reading that one instead makes every comparison +// come out equal, which is exactly as broken as not comparing at all. +func clientModuleVersion(packageXML []byte) string { + dec := xml.NewDecoder(bytes.NewReader(packageXML)) + for { + tok, err := dec.Token() + if err != nil { + return "" + } + start, ok := tok.(xml.StartElement) + if !ok || start.Name.Local != "clientModule" { + continue + } + for _, a := range start.Attr { + if a.Name.Local == "version" { + return a.Value + } + } + return "" + } +} + +// widgetVersionInMpk reads the declared version out of a widget .mpk given its +// bytes. Returns "" when the file is not a readable widget package, which the +// caller treats as "cannot compare" rather than as "older". +func widgetVersionInMpk(mpk []byte) string { + zr, err := zip.NewReader(bytes.NewReader(mpk), int64(len(mpk))) + if err != nil { + return "" + } + for _, f := range zr.File { + if f.Name != "package.xml" { + continue + } + rc, oerr := f.Open() + if oerr != nil { + return "" + } + body, rerr := io.ReadAll(rc) + _ = rc.Close() + if rerr != nil { + return "" + } + return clientModuleVersion(body) + } + return "" +} + +// widgetVersionOnDisk is widgetVersionInMpk for a path, returning "" when the +// file is absent or unreadable. +func widgetVersionOnDisk(path string) string { + body, err := os.ReadFile(path) + if err != nil { + return "" + } + return widgetVersionInMpk(body) +} + +// versionLess reports whether a is an earlier version than b, comparing dotted +// numeric components. It returns false when either side cannot be parsed, so an +// unparseable version never causes a skip: the caller's default is to install, +// and a wrong "older" verdict would silently withhold the file the package +// shipped. +func versionLess(a, b string) bool { + as, bs := strings.Split(a, "."), strings.Split(b, ".") + if len(as) == 0 || len(bs) == 0 { + return false + } + n := max(len(as), len(bs)) + for i := range n { + ai, bi := 0, 0 + if i < len(as) { + v, err := strconv.Atoi(strings.TrimSpace(as[i])) + if err != nil { + return false + } + ai = v + } + if i < len(bs) { + v, err := strconv.Atoi(strings.TrimSpace(bs[i])) + if err != nil { + return false + } + bi = v + } + if ai != bi { + return ai < bi + } + } + return false +} + +// isWidgetPackage reports whether a package entry is a widget .mpk under +// widgets/. +func isWidgetPackage(name string) bool { + return strings.HasPrefix(name, "widgets/") && + strings.EqualFold(path.Ext(name), ".mpk") && + !strings.Contains(strings.TrimPrefix(name, "widgets/"), "/") +} + +// unpackedTwinOf returns the widgets// prefix that duplicates a +// widgets/.mpk entry, or "" when name is not such an entry. +// +// Some packages ship a widget both ways. FeedbackModule 5.0.0 carries +// `widgets/SprintrFeedbackWidget.mpk` *and* an unpacked +// `widgets/SprintrFeedbackWidget/` tree of the same widget at the same version +// (12.0.4, measured). Installing both leaves a duplicate in the project that +// `mx check` tolerates and nobody asked for (FINDINGS §18). +func unpackedTwinOf(name string) string { + if !isWidgetPackage(name) { + return "" + } + return strings.TrimSuffix(name, path.Ext(name)) + "/" +} diff --git a/cmd/mxcli/marketplace/widgetversion_test.go b/cmd/mxcli/marketplace/widgetversion_test.go new file mode 100644 index 000000000..8399326df --- /dev/null +++ b/cmd/mxcli/marketplace/widgetversion_test.go @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: Apache-2.0 + +// mxcli-chat FINDINGS §14 and §18, both reproduced from the published packages +// themselves before this was written: +// +// - Atlas_Web_Content 4.3.0 bundles five Data Widgets at 3.4.0 that DataWidgets +// 3.11.3 ships at 3.11.3, so installing the modules in that order rolled the +// project's widgets back with nothing reported. An older widget is not a +// `mx check` error, so the app simply ran old widget code. +// - FeedbackModule 5.0.0 ships SprintrFeedbackWidget twice — as a .mpk and as +// an unpacked tree of the same version — and installing both left a duplicate +// nobody asked for. +package marketplace + +import ( + "archive/zip" + "bytes" + "os" + "path/filepath" + "testing" +) + +// widgetMpk builds a minimal widget package declaring the given version, shaped +// like a real one: the manifest schema version on is 1.0 for every +// widget ever published, and the widget's own version is on . +func widgetMpk(t *testing.T, version string) []byte { + t.Helper() + var buf bytes.Buffer + zw := zip.NewWriter(&buf) + w, err := zw.Create("package.xml") + if err != nil { + t.Fatal(err) + } + xml := ` + + + + +` + if _, err := w.Write([]byte(xml)); err != nil { + t.Fatal(err) + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + return buf.Bytes() +} + +// modulePackage builds a module .mpk carrying the given entries. +func modulePackage(t *testing.T, entries map[string][]byte) string { + t.Helper() + path := filepath.Join(t.TempDir(), "Module.mpk") + f, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + zw := zip.NewWriter(f) + for name, body := range entries { + w, cerr := zw.Create(name) + if cerr != nil { + t.Fatal(cerr) + } + if _, werr := w.Write(body); werr != nil { + t.Fatal(werr) + } + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + return path +} + +// The version on is the widget's; the one on is the +// manifest schema and is 1.0 everywhere. Reading the wrong one makes every +// comparison come out equal, which is as broken as not comparing. +func TestClientModuleVersion_IgnoresTheManifestSchemaVersion(t *testing.T) { + if got := widgetVersionInMpk(widgetMpk(t, "3.11.3")); got != "3.11.3" { + t.Errorf("widget version = %q, want 3.11.3 (1.0 means the attribute was read)", got) + } +} + +// The core fix: a module's bundled widget must not replace a newer copy. +func TestInstallPackageFiles_KeepsANewerWidgetTheProjectAlreadyHas(t *testing.T) { + proj := t.TempDir() + widget := filepath.Join(proj, "widgets", "com.mendix.widget.web.Datagrid.mpk") + if err := os.MkdirAll(filepath.Dir(widget), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(widget, widgetMpk(t, "3.11.3"), 0o644); err != nil { + t.Fatal(err) + } + + pkg := modulePackage(t, map[string][]byte{ + "widgets/com.mendix.widget.web.Datagrid.mpk": widgetMpk(t, "3.4.0"), + "themesource/x/web/design-properties.json": []byte("{}"), + }) + + written, skipped, err := InstallPackageFiles(pkg, proj) + if err != nil { + t.Fatalf("InstallPackageFiles: %v", err) + } + if got := widgetVersionOnDisk(widget); got != "3.11.3" { + t.Errorf("widget on disk is %s; the package's older 3.4.0 overwrote it — this is the silent rollback", got) + } + if len(skipped) != 1 || skipped[0].Kept != "3.11.3" || skipped[0].Offered != "3.4.0" { + t.Errorf("skip not reported usefully: %+v — a silent skip is as bad as a silent overwrite", skipped) + } + // The negative half: everything else the package ships still lands. + if len(written) != 1 || written[0] != filepath.Join("themesource", "x", "web", "design-properties.json") { + t.Errorf("written = %v, want only the themesource file", written) + } +} + +// A guard that skips every widget would also pass the test above. A newer +// bundled widget, and an equal one, must still be installed. +func TestInstallPackageFiles_InstallsNewerAndEqualWidgets(t *testing.T) { + for _, tc := range []struct{ have, ships string }{ + {"3.4.0", "3.11.3"}, + {"3.11.3", "3.11.3"}, + } { + proj := t.TempDir() + widget := filepath.Join(proj, "widgets", "com.mendix.widget.web.Datagrid.mpk") + if err := os.MkdirAll(filepath.Dir(widget), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(widget, widgetMpk(t, tc.have), 0o644); err != nil { + t.Fatal(err) + } + pkg := modulePackage(t, map[string][]byte{ + "widgets/com.mendix.widget.web.Datagrid.mpk": widgetMpk(t, tc.ships), + }) + if _, skipped, err := InstallPackageFiles(pkg, proj); err != nil { + t.Fatal(err) + } else if len(skipped) != 0 { + t.Errorf("have %s, package ships %s: skipped %+v, want it installed", tc.have, tc.ships, skipped) + } + if got := widgetVersionOnDisk(widget); got != tc.ships { + t.Errorf("have %s, package ships %s: on disk %s", tc.have, tc.ships, got) + } + } +} + +// A widget the project does not have yet is always installed — there is nothing +// to compare against, and "cannot compare" must not mean "skip". +func TestInstallPackageFiles_InstallsAWidgetTheProjectLacks(t *testing.T) { + proj := t.TempDir() + pkg := modulePackage(t, map[string][]byte{ + "widgets/com.mendix.widget.web.Datagrid.mpk": widgetMpk(t, "3.4.0"), + }) + written, skipped, err := InstallPackageFiles(pkg, proj) + if err != nil { + t.Fatal(err) + } + if len(skipped) != 0 || len(written) != 1 { + t.Errorf("written=%v skipped=%+v, want the widget installed", written, skipped) + } +} + +// FINDINGS §18: the same widget shipped twice, once packed and once not. +func TestInstallPackageFiles_SkipsTheUnpackedTwinOfAPackagedWidget(t *testing.T) { + proj := t.TempDir() + pkg := modulePackage(t, map[string][]byte{ + "widgets/SprintrFeedbackWidget.mpk": widgetMpk(t, "12.0.4"), + "widgets/SprintrFeedbackWidget/package.xml": []byte(""), + "widgets/SprintrFeedbackWidget/SprintrFeedback.xml": []byte(""), + "widgets/SprintrFeedbackWidget/SprintrFeedback.js": []byte("//"), + "widgets/OtherUnpackedWidget/OtherUnpacked.xml": []byte(""), + }) + written, skipped, err := InstallPackageFiles(pkg, proj) + if err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(proj, "widgets", "SprintrFeedbackWidget")); !os.IsNotExist(err) { + t.Errorf("the unpacked twin was installed alongside the .mpk (err=%v)", err) + } + if len(skipped) != 3 { + t.Errorf("skipped %d entries, want the 3 files of the unpacked twin: %+v", len(skipped), skipped) + } + // An unpacked widget with NO packaged twin is a different thing and must be + // installed — skipping it would drop a widget the module needs. + if _, err := os.Stat(filepath.Join(proj, "widgets", "OtherUnpackedWidget", "OtherUnpacked.xml")); err != nil { + t.Errorf("an unpacked widget with no .mpk twin was skipped: %v", err) + } + if len(written) != 2 { + t.Errorf("written = %v, want the .mpk and the unrelated unpacked widget", written) + } +} + +func TestVersionLess(t *testing.T) { + cases := []struct { + a, b string + want bool + }{ + {"3.4.0", "3.11.3", true}, // 4 < 11: string compare would say otherwise + {"3.11.3", "3.4.0", false}, + {"3.11.3", "3.11.3", false}, + {"1.2", "1.2.1", true}, + {"6.3.0", "6.3.2", true}, + // Unparseable never counts as older: the default must be to install. + {"1.0.0-beta", "1.0.1", false}, + {"", "1.0.0", false}, + } + for _, c := range cases { + if got := versionLess(c.a, c.b); got != c.want { + t.Errorf("versionLess(%q, %q) = %v, want %v", c.a, c.b, got, c.want) + } + } +} diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index fb5628ace..1f86c1e75 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -178,6 +178,23 @@ A headless install or update leaves two repairs for Mendix's own tools: **CE0463 Measured: Administration 4.3.2 → 4.5.0 and DataWidgets 3.5.0 → 3.11.3 both reach **0 errors** afterwards. +## Bundled widgets and install order + +A module package carries a copy of every widget its pages use, pinned at the module author's release time — and different modules pin different versions of the same widget. Atlas_Web_Content 4.3.0 ships five Data Widgets at 3.4.0 that DataWidgets 3.11.3 ships at 3.11.3. + +`install` and `update` keep the newer copy and say so: + +```text + Kept 5 newer widget(s) the package would have rolled back: + widgets/com.mendix.widget.web.Datagrid.mpk — kept 3.11.3, package ships 3.4.0 +``` + +Without this, module install order silently decided which widget versions the project ended up with, and nothing reported it — an out-of-date widget is not a check error. A package that ships the same widget both as a `.mpk` and as an unpacked tree (FeedbackModule 5.0.0) installs only the `.mpk`. + +## When the installed version is no longer published + +`diff` and `update` both download the installed version to establish the local-edit baseline, so both fail when it has been unpublished — as NanoflowCommons 6.0.0 has, while a blank 11.13 app still ships it. `--force` does not help: it overrides a finding, and there is no finding to override. `mxcli marketplace update … --no-baseline` accepts that the question cannot be answered and updates anyway, discarding any local edits to that module without naming them. + ## Repairing the model (`mxcli fix`) `mx update-widgets` and `mx rename-design-properties` each fix something only Mendix can fix, and each rewrites an MPR v2 project into the single-file v1 format while doing it. Measured on 11.12.1: `update-widgets` took 369 `.mxunit` files to 0 and a 69,632-byte index to 14,405,632 bytes; `rename-design-properties` took 1,865 files to 0 and a 249,856-byte index to 39,895,040 bytes, having renamed 149 design properties across 41 documents. The conversion is one-way. From fb4e40b0d5bc73493aca48d46152e1c6940baa67 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 20:54:34 +0000 Subject: [PATCH 02/20] Apply a configuration's constants at boot; stop diff inventing edits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §33. `alter settings constant 'Encryption.EncryptionKey' value '…' in configuration 'Default'` executed, reported success, round-tripped through `describe settings` — and never reached the app. mxbuild writes each constant's *default* into deployment/model/config.json, and that map is what `run --local` hands the runtime as MicroflowConstants; the configuration's values were not in it and nothing read them. An app ran for hours with an empty encryption key while the model said otherwise. Studio Pro runs a configuration; so does this now. The run resolves one (--configuration, else the only one, else "Default") and merges its shared constant values over the defaults at boot. Verified on a real project: the run prints `Applying 1 constant value(s) from configuration "Default": MyFirstModule.ApiKey`. Three judgements worth stating. It merges rather than replaces, because config.json carries the defaults for constants a configuration is silent about — replacing drops them and the app 530s on the first microflow that reads one, which is exactly the shape `--runtime-setting MicroflowConstants={…}` has. A private override has no value in the model at all, so applying it would blank the constant; it is skipped and named. And with several configurations and no "Default" it applies none and says why, rather than guessing which environment a local run means. It also prints when it applies nothing. The bug was invisible because the run said nothing about constants either way, so silence had to stop meaning "your override is in effect". §16. `marketplace diff` accused an untouched blank app of editing an Atlas_Core snippet and an Atlas_Web_Content building block, and --save-edits wrote the snippet as an empty `{ }` body — a file offered as a rescue that would have emptied it on replay. The comparison only asked whether DESCRIBE *errored*; output that succeeds while saying nothing about the element still counted as evidence, and two such renderings can differ over one unresolved name. A difference now has to come from output that could carry an edit: non-empty, and not the informational text a read-only handler emits. Equality is still checked first, so identical renderings stay Unchanged whatever the type — inverting that would mark every Atlas building block unknown and drain `verified` of its meaning. The same rule gates --save-edits, including OnlyInstalled findings, which never pass through classify. Each new test verified to fail with the reported symptom when the fix is stubbed out. --- .claude/skills/fix-issue.md | 2 + .../mendix/download-marketplace-content.md | 8 + .claude/skills/mendix/run-local.md | 28 ++++ cmd/mxcli/cmd_run.go | 11 ++ cmd/mxcli/docker/localboot.go | 28 ++++ cmd/mxcli/docker/localboot_constants_test.go | 45 +++++ cmd/mxcli/docker/runlocal.go | 6 + cmd/mxcli/marketplace/compare.go | 11 ++ cmd/mxcli/marketplace/inconclusive_test.go | 99 +++++++++++ cmd/mxcli/marketplace/snapshot.go | 54 ++++++ cmd/mxcli/marketplace/update.go | 10 ++ cmd/mxcli/runconstants.go | 154 ++++++++++++++++++ cmd/mxcli/runconstants_test.go | 123 ++++++++++++++ 13 files changed, 579 insertions(+) create mode 100644 cmd/mxcli/docker/localboot_constants_test.go create mode 100644 cmd/mxcli/marketplace/inconclusive_test.go create mode 100644 cmd/mxcli/runconstants.go create mode 100644 cmd/mxcli/runconstants_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index f9849be03..d8b2979a7 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -492,3 +492,5 @@ extracting `OffsetExpression`/`LimitExpression`. | `ALTER PAGE` over `--mcp` fails against Studio Pro **11.13** with `pg_patch_page: … PROP_NOT_PRIMITIVE: Property 'widgets' is not a primitive property`. `CREATE PAGE` is fine; the page itself is left intact | 11.13 gave `pg_read_page` a **`depth` argument defaulting to 4**, replacing anything deeper with the literal string `"..."`. ALTER PAGE is read-modify-**replace-whole-page**, so the truncated read went straight back as the new page body. Measured live: `Administration.Account_Overview` read 32,594 bytes at full depth but **1,052 bytes** at the default, its entire tree reduced to `{"widgets":["...","..."]}`. Every ordinary page truncates — three of three PgTest pages did | `mdl/backend/mcp/page.go` (`pgReadPage`, `pgReadFullDepth`, `hasTruncationSentinel`), `mdl/backend/mcp/client.go` (`SupportsToolArg`) | Request the full depth, and **guard rather than trust it**: refuse a read still carrying the sentinel instead of letting a partial page reach a write (ADR-0005 guard-don't-drop). Two traps. (1) **Do not send `depth` unconditionally** — 11.11/11.12 declare `pg_read_page` `additionalProperties:false` without it, so the whole call fails; gate on a live `tools/list` probe of the tool's input schema, because `serverInfo.version` is frozen at `1.0.0` across 11.11/11.12/11.13 and cannot discriminate releases. (2) **Match the sentinel only as an array element** — a caption or title legitimately reading `"..."` is real content, and a naive substring scan rejects valid pages. The release notes announced none of this, exactly as 11.12 silently removed `pg_write_page` (#697): on any Studio Pro upgrade, re-probe `tools/list` and diff the input schemas, not just the tool names. Tests `mdl/backend/mcp/page_depth_test.go`; controls: stub the depth arg (full-depth test fails) and stub the guard (truncation test fails) | | Widgets silently go backwards: a project on DataWidgets 3.11.3 ends up running 3.4.0 widget code after updating an unrelated module (Atlas_Web_Content, FeedbackModule). `mx check` stays clean, `show modules` shows the new module version, and nothing mentions widgets | A module `.mpk` bundles a copy of every widget its pages use, pinned at its author's release time — and different modules pin different versions of the **same** widget. `InstallPackageFiles` copied everything the package shipped, so the last module installed decided the widget versions. Measured on the published packages: Atlas_Web_Content 4.3.0 ships 5 Data Widgets at 3.4.0 against DataWidgets 3.11.3's 3.11.3 | `cmd/mxcli/marketplace/widgetversion.go` (`widgetVersionInMpk`, `versionLess`, `unpackedTwinOf`), `update.go` (`InstallPackageFiles` returns `[]SkippedFile`), `cmd_marketplace_update.go` (`reportSkippedFiles`) | **Read ``, not ``** — the root attribute is the manifest schema and is `1.0` on every widget ever published, so comparing it makes every version look equal, which is exactly as broken as not comparing. **An unparseable version must never count as "older"** — the default is to install, and a wrong "older" verdict silently withholds a file the package shipped. **Skipping must be reported**: a silent skip is the same class of bug as the silent overwrite it replaces. **Test the negative half** — a guard that skips every widget also passes "the newer one survived"; assert that a newer *and* an equal bundled widget still install. Same file handles FINDINGS §18: a package that ships a widget as both `.mpk` and unpacked tree (FeedbackModule 5.0.0, both 12.0.4) installs only the `.mpk`, but an unpacked widget with **no** packaged twin must still install. Verified against the real packages: 5 kept, 13 twin entries skipped. Reported in mxcli-chat FINDINGS §14/§18 | | `mxcli marketplace update`/`diff` fail with `version "6.0.0" not found` on a module the project actually has, and `--force` does not bypass it | Both download the **installed** version to build the local-edit baseline, and that version had been unpublished. A blank 11.13 app ships NanoflowCommons 6.0.0 while the 6.x line now starts at 6.1.1 — so the module most in need of updating is the one whose baseline cannot be built | `cmd/mxcli/cmd_marketplace_update.go` (`--no-baseline`, and the refusal now names it) | **`--force` was never going to work**: it overrides a *finding*, and here the comparison never ran, so there is no finding. Needing a second flag is the signal that these are two different decisions — "I accept losing edits I have been shown" vs "I accept not being shown". **Say what the flag costs in the flag's own message** — with no baseline, local edits go without being named, so the honest instruction is to commit first. Reported in mxcli-chat FINDINGS §15 | +| A constant set with `alter settings constant 'M.C' value '…' in configuration 'Default'` has no effect on the running app. The statement succeeds, `describe settings` round-trips it, `mx check` is clean, and the app behaves as if the constant were still at its default | mxbuild writes `deployment/model/config.json` with each constant's **default** value, and that map is what `run --local` hands the standalone runtime as `MicroflowConstants` — the configuration's overrides are not in it, and nothing read them. An app ran for hours with an empty encryption key while the model said otherwise | `cmd/mxcli/runconstants.go` (`resolveConstantOverrides`, `reportConstantOverrides`), `cmd_run.go` (`--configuration`), `cmd/mxcli/docker/localboot.go` (`mergeConstantOverrides`) | **A silent no-op is the worst shape a bug can take** — every layer reported success. The fix therefore prints in *every* case, including "no overrides applied", so silence stops meaning "your value is in effect". **Merge, never replace**: `config.json` carries the defaults for constants the configuration is silent about, and replacing the map drops them (the app 530s on the first microflow that reads one) — which is exactly the shape `--runtime-setting MicroflowConstants={…}` has. **A private override has no value in the model at all** (a `Settings$PrivateValue` marker; the value lives on the workstation), so applying it would blank the constant — skip and name it. **Do not guess between configurations**: with several and no `Default`, applying one silently could push production's API key into a local run. Tests `cmd/mxcli/runconstants_test.go`, `cmd/mxcli/docker/localboot_constants_test.go`; verified live — the run now prints `Applying 1 constant value(s) from configuration "Default"`. Reported in mxcli-chat FINDINGS §33 | +| `mxcli marketplace diff` reports local edits on a project nobody has edited — typically an Atlas snippet and a building block. `--save-edits` then writes MDL that would *destroy* the element if replayed (an empty `create or modify snippet X (Folder: 'Web') { }`) | The comparison is on DESCRIBE output, and `Describable()` only asked whether describe *errored*. Some types describe successfully into output that says nothing: an empty `{ }` body, or a building block under "Building blocks are read-only; they cannot be created via MDL". Two such renderings can still differ — one unresolved name is enough — and the difference was reported as a user edit | `cmd/mxcli/marketplace/snapshot.go` (`Element.Conclusive`, `declaredReadOnly`, `emptyBody`), `compare.go` (`classify`), `update.go` (`SaveEdits`) | **Check equality BEFORE conclusiveness** — identical text is solid evidence of "unchanged" whatever the type, and inverting the order marks every Atlas building block unknown and drains `verified` of meaning. **Judge the output, not the type**: a type list goes stale the moment a describe handler improves, while "this text contains nothing to compare" stays true by construction — match the handler's own read-only wording so the two stay in step. **A rescue file must never be a deletion**: the same rule gates `--save-edits`, including `OnlyInstalled` findings, which reach it without passing through `classify`. **Test that a real edit is still Modified** — a guard that returns unknown for everything passes every "not evidence" test. Tests `cmd/mxcli/marketplace/inconclusive_test.go`. Reported in mxcli-chat FINDINGS §16 | diff --git a/.claude/skills/mendix/download-marketplace-content.md b/.claude/skills/mendix/download-marketplace-content.md index 1d93e0730..0dc5a7a0f 100644 --- a/.claude/skills/mendix/download-marketplace-content.md +++ b/.claude/skills/mendix/download-marketplace-content.md @@ -266,6 +266,14 @@ built **at the project's own Mendix version** (a mismatch is refused, not warned Mendix's own conversions would otherwise read as your edits), and compares `DESCRIBE` output on both sides. +**A "modified" verdict now means the difference is real.** Some element types +DESCRIBE renders imperfectly — a snippet whose body comes out `{ }`, a building +block under "Building blocks are read-only; they cannot be created via MDL" — and +two imperfect renderings can differ for reasons that have nothing to do with you. +Those are reported `unknown`, never `changed`, and `--save-edits` refuses to write +them: replaying `create or modify snippet X (Folder: 'Web') { }` would **empty** +the snippet. + **Read `verified`, not just `locallyModified`.** An element that cannot be described is reported as `unknown`, never as unchanged, and `verified: false` means "no modifications found" is not a conclusion: diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index c9513672a..8dad49671 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -400,3 +400,31 @@ secret registers it owner-less. and `.../dist/index.js` also returns `200` (client bundle served). - [ ] With `--watch`, editing a microflow logs `applied via reload`; adding an entity logs `applied via restart` and creates the table in Postgres. + +## Constant values come from a configuration + +`mxcli run --local` applies the constant values of the project configuration it +is running, merged over each constant's default: + +```text +Applying 1 constant value(s) from configuration "Default": Encryption.EncryptionKey +``` + +Before this they were ignored: mxbuild writes each constant's **default** into +`deployment/model/config.json`, and that map is what the runtime is handed — so +`alter settings constant … in configuration 'Default'` executed, round-tripped +through `describe settings`, and did nothing. An app ran for hours with an empty +encryption key while the model said otherwise. + +- `--configuration ` picks one. With several configurations and none named + `Default`, mxcli applies **none** and says so rather than guessing which + environment this run means. +- A **private** override has no value in the model at all (the value lives on the + developer's workstation), so the default is used and the constant is named. +- The line prints in every case, including "no overrides" — silence used to mean + "your override is in effect" when it was not. + +Setting a constant on a *running* app is a different mechanism: `MicroflowConstants` +over the M2EE admin port, which is how Mendix Cloud injects per-environment values. +Note that `--runtime-setting 'MicroflowConstants={…}'` **replaces** the whole map +rather than merging into it, so it drops every constant it does not mention. diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index 6f28d7418..77102d0b8 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -152,8 +152,17 @@ Examples: trace = true // --trace-otlp implies --trace } + // Constant values set per configuration are not in the deployment's + // config.json (mxbuild writes each constant's default there), so without + // this the app runs with defaults while the model says otherwise — + // silently. See runconstants.go. + configuration, _ := cmd.Flags().GetString("configuration") + overrides := constantOverridesFor(projectPath, configuration) + reportConstantOverrides(os.Stdout, overrides) + opts := docker.LocalRunOptions{ ProjectPath: projectPath, + ConstantOverrides: overrides.Values, Hub: hub, HubSecret: hubSecret, HubKey: hubKey, @@ -232,6 +241,8 @@ Examples: } func init() { + runCmd.Flags().String("configuration", "", + "Which project configuration's constant values to run with (default: the only one, or \"Default\")") runCmd.Flags().Bool("local", false, "Run locally without Docker (warm serve + standalone runtime)") runCmd.Flags().String("hub", "", "Expose the running app in a browser via your own mxcli tunnel-hub URL (e.g. https://hub.example.com). Implies --local; the app stays local and is reverse-tunnelled out") runCmd.Flags().String("hub-secret", "", "Shared auth secret for --hub (\"user:pass\"), matching the hub's --secret") diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index ded08b3a2..ab583a97a 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -71,6 +71,10 @@ type LocalRuntimeOptions struct { // payload (e.g. "Metrics.Registries", "OpenTelemetry._RuntimeSpanFilters"). // Merged here because the admin action replaces rather than merges. RuntimeSettings map[string]any + // ConstantOverrides are the running configuration's constant values, + // merged over the defaults mxbuild wrote into the deployment. See + // mergeConstantOverrides. + ConstantOverrides map[string]string // Trace attaches the bundled OpenTelemetry Java agent to the runtime JVM // (traces via the console exporter → the tee'd runtime log). The caller should // also set OpenTelemetry._RuntimeSpanFilters via RuntimeSettings — unfiltered @@ -325,6 +329,29 @@ func readDeploymentConstants(deployDir string) (map[string]string, error) { return cfg.Constants, nil } +// mergeConstantOverrides layers a configuration's constant values over the +// defaults mxbuild resolved into the deployment. +// +// The merge direction is the whole point. config.json carries every constant's +// *default*, which is what the runtime needs for the ones a configuration does +// not override; the configuration's values win where both exist. Replacing the +// map instead — the shape `--runtime-setting MicroflowConstants=…` has — drops +// every constant the configuration is silent about, and the app 530s on the +// first microflow that reads one. +func mergeConstantOverrides(defaults, overrides map[string]string) map[string]string { + if len(overrides) == 0 { + return defaults + } + merged := make(map[string]string, len(defaults)+len(overrides)) + for k, v := range defaults { + merged[k] = v + } + for k, v := range overrides { + merged[k] = v + } + return merged +} + // ensureDataDirs creates the data/{files,tmp,model-upload} directories the // runtime expects under the deployment dir. m2ee normally creates these; a bare // serve Deploy / unzipped .mda does not. @@ -450,6 +477,7 @@ func (rt *LocalRuntime) spawnAndConfigure() error { if err != nil { return err } + constants = mergeConstantOverrides(constants, rt.opts.ConstantOverrides) if _, err := CallM2EE(rt.m2ee, "update_configuration", runtimeConfigParams(rt.opts, constants)); err != nil { return fmt.Errorf("update_configuration: %w", err) } diff --git a/cmd/mxcli/docker/localboot_constants_test.go b/cmd/mxcli/docker/localboot_constants_test.go new file mode 100644 index 000000000..e92f0f6b5 --- /dev/null +++ b/cmd/mxcli/docker/localboot_constants_test.go @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import "testing" + +// mxcli-chat FINDINGS §33: the configuration's constant values have to reach the +// runtime, and §33's caveat: they must be MERGED over mxbuild's resolved +// defaults, never replace them. `--runtime-setting MicroflowConstants={…}` has +// the replacing shape, which drops every constant the configuration is silent +// about — and the app 530s on the first microflow that reads one. +func TestMergeConstantOverrides(t *testing.T) { + defaults := map[string]string{ + "Encryption.EncryptionKey": "", + "App.Timeout": "30", + "App.BaseUrl": "http://localhost", + } + merged := mergeConstantOverrides(defaults, map[string]string{ + "Encryption.EncryptionKey": "95d6", + }) + + if merged["Encryption.EncryptionKey"] != "95d6" { + t.Errorf("the override did not win: %q", merged["Encryption.EncryptionKey"]) + } + if merged["App.Timeout"] != "30" || merged["App.BaseUrl"] != "http://localhost" { + t.Errorf("defaults the configuration is silent about were dropped: %v — this is the replace bug", merged) + } + if len(merged) != 3 { + t.Errorf("merged has %d entries, want 3: %v", len(merged), merged) + } + // The caller's map must not be mutated: the same defaults are read once per + // boot and a restart re-uses them. + if defaults["Encryption.EncryptionKey"] != "" { + t.Error("mergeConstantOverrides mutated the defaults map it was given") + } +} + +// No overrides is the pre-existing behaviour and must stay allocation-free of +// surprises: the defaults go through untouched. +func TestMergeConstantOverrides_NoOverrides(t *testing.T) { + defaults := map[string]string{"A.B": "v"} + if got := mergeConstantOverrides(defaults, nil); got["A.B"] != "v" || len(got) != 1 { + t.Errorf("got %v, want the defaults unchanged", got) + } +} diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index 8911a41b0..fe77b7bd8 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -44,6 +44,11 @@ type LocalAppInfo struct { type LocalRunOptions struct { // ProjectPath is the .mpr file. ProjectPath string + // ConstantOverrides are the constant values of the configuration this run + // represents (qualified name -> value), merged over the defaults mxbuild + // wrote into the deployment. Empty means "every constant keeps its default", + // which is what a local run always did before — see runconstants.go. + ConstantOverrides map[string]string // DeployDir is where the serve Deploy target writes (default /deployment). DeployDir string // MxBuildPath overrides mxbuild resolution (optional). @@ -728,6 +733,7 @@ func RunLocal(opts LocalRunOptions) error { Trace: opts.Trace, TraceServiceName: traceService, TraceOTLPEndpoint: opts.TraceOTLP, + ConstantOverrides: opts.ConstantOverrides, Env: opts.Env, Stdout: w, Stderr: stderr, diff --git a/cmd/mxcli/marketplace/compare.go b/cmd/mxcli/marketplace/compare.go index e108a7e9a..a1b68aba0 100644 --- a/cmd/mxcli/marketplace/compare.go +++ b/cmd/mxcli/marketplace/compare.go @@ -128,9 +128,20 @@ func classify(k ElementKey, inst, pkg Element, hasInst, hasPkg bool) Finding { if !inst.Describable() || !pkg.Describable() { return Finding{Key: k, Verdict: Unknown, Reason: unknownReason(inst, pkg)} } + // Identical text is solid evidence of "unchanged" whatever the type, so this + // is checked before asking whether the output is conclusive — otherwise every + // building block in Atlas would be reported as unknown for no reason. if inst.MDL == pkg.MDL { return Finding{Key: k, Verdict: Unchanged} } + // They differ — but a difference is only evidence of an edit if the output + // could carry one. See Element.Conclusive. + if ok, why := inst.Conclusive(); !ok { + return Finding{Key: k, Verdict: Unknown, Reason: why} + } + if ok, why := pkg.Conclusive(); !ok { + return Finding{Key: k, Verdict: Unknown, Reason: why} + } return Finding{Key: k, Verdict: Modified, InstalledMDL: inst.MDL, PackageMDL: pkg.MDL} } diff --git a/cmd/mxcli/marketplace/inconclusive_test.go b/cmd/mxcli/marketplace/inconclusive_test.go new file mode 100644 index 000000000..8c128f24e --- /dev/null +++ b/cmd/mxcli/marketplace/inconclusive_test.go @@ -0,0 +1,99 @@ +// SPDX-License-Identifier: Apache-2.0 + +// mxcli-chat FINDINGS §16: on an untouched blank app, `marketplace diff` accused +// the user of editing an Atlas_Core snippet, an Atlas_Web_Content building block +// and four FeedbackModule elements. Nobody had touched them — DESCRIBE renders +// those types imperfectly, and two imperfect renderings can differ. Worse, +// `--save-edits` wrote the snippet out as an empty `{ }` body, so the file +// offered as a rescue would have emptied it on replay. +package marketplace + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// The real rendering from the finding. +const emptySnippetMDL = `create or modify snippet Atlas_Core.FeedbackWidget (Folder: 'Web') { }` + +const readOnlyBuildingBlockMDL = `-- Building blocks are read-only; they cannot be created via MDL. +create building block Atlas_Web_Content.Master_Detail ( + DataSource: database from , +)` + +func TestClassify_AnEmptyBodyIsNotEvidenceOfAnEdit(t *testing.T) { + f := classify(ElementKey{Type: "SNIPPET", Name: "Atlas_Core.FeedbackWidget"}, + Element{MDL: emptySnippetMDL}, + Element{MDL: `create or modify snippet Atlas_Core.FeedbackWidget (Folder: 'Web', Caption: 'x') { }`}, + true, true) + if f.Verdict != Unknown { + t.Errorf("verdict = %s, want unknown — an empty body cannot show that the user edited anything", f.Verdict) + } + if f.Reason == "" { + t.Error("an unknown verdict without a reason tells the user nothing") + } +} + +func TestClassify_AReadOnlyDescribeIsNotEvidenceOfAnEdit(t *testing.T) { + f := classify(ElementKey{Type: "BUILDING_BLOCK", Name: "Atlas_Web_Content.Master_Detail"}, + Element{MDL: readOnlyBuildingBlockMDL}, + Element{MDL: strings.Replace(readOnlyBuildingBlockMDL, "database from ,", "database from Module.Entity,", 1)}, + true, true) + if f.Verdict != Unknown { + t.Errorf("verdict = %s, want unknown — this output is informational, not re-executable MDL", f.Verdict) + } +} + +// The negative half, and the reason equality is checked first: an element whose +// two renderings are identical is unchanged, whatever its type. Marking every +// building block unknown would make `verified` false on every project and drain +// the signal from it. +func TestClassify_IdenticalRenderingsAreUnchangedEvenWhenInconclusive(t *testing.T) { + for _, mdl := range []string{emptySnippetMDL, readOnlyBuildingBlockMDL} { + f := classify(ElementKey{Type: "X", Name: "M.N"}, Element{MDL: mdl}, Element{MDL: mdl}, true, true) + if f.Verdict != Unchanged { + t.Errorf("verdict = %s for identical output, want unchanged", f.Verdict) + } + } +} + +// And a real edit to a normally-describable element must still be Modified — +// a guard that made everything unknown would pass the tests above. +func TestClassify_ARealEditIsStillReported(t *testing.T) { + f := classify(ElementKey{Type: "ENTITY", Name: "Administration.Account"}, + Element{MDL: "create or modify persistent entity Administration.Account (\n Name: String,\n Mine: String,\n)"}, + Element{MDL: "create or modify persistent entity Administration.Account (\n Name: String,\n)"}, + true, true) + if f.Verdict != Modified { + t.Errorf("verdict = %s, want modified — this is a genuine local edit", f.Verdict) + } +} + +// The saved file is offered as a rescue; it must never be a deletion. +func TestSaveEdits_RefusesToWriteAnEmptyBodyAsAnEdit(t *testing.T) { + dir := t.TempDir() + rep := &Report{Findings: []Finding{ + {Key: ElementKey{Type: "SNIPPET", Name: "Atlas_Core.FeedbackWidget"}, + Verdict: OnlyInstalled, InstalledMDL: emptySnippetMDL}, + {Key: ElementKey{Type: "ENTITY", Name: "M.Real"}, + Verdict: Modified, InstalledMDL: "create or modify persistent entity M.Real (\n Name: String,\n)"}, + }} + written, unsaved, err := SaveEdits(dir, rep) + if err != nil { + t.Fatal(err) + } + for _, w := range written { + if strings.Contains(w, "snippet") { + body, _ := os.ReadFile(w) + t.Errorf("wrote %s, which replays as an empty snippet:\n%s", filepath.Base(w), body) + } + } + if len(unsaved) != 1 || !strings.Contains(unsaved[0], "FeedbackWidget") { + t.Errorf("unsaved = %v, want the snippet reported rather than silently dropped", unsaved) + } + if len(written) != 1 { + t.Errorf("written = %v, want the real entity still saved", written) + } +} diff --git a/cmd/mxcli/marketplace/snapshot.go b/cmd/mxcli/marketplace/snapshot.go index f58025994..fcc185a07 100644 --- a/cmd/mxcli/marketplace/snapshot.go +++ b/cmd/mxcli/marketplace/snapshot.go @@ -53,6 +53,60 @@ type Element struct { // Describable reports whether the element could be read at all. func (e Element) Describable() bool { return e.Err == "" } +// Conclusive reports whether this element's DESCRIBE output is good enough to +// carry a "you edited this" verdict. +// +// Describable() is not enough. Some types describe *successfully* into output +// that says nothing about the element: a snippet whose body comes out as `{ }`, +// a building block that renders under "-- Building blocks are read-only; they +// cannot be created via MDL." Two such renderings can still differ — a resolved +// name that resolves in one project and not the other is enough — and the +// difference is then an artefact of DESCRIBE, not an edit. +// +// Reporting those as Modified is worse than reporting nothing: on an untouched +// blank app, `diff` accused the user of editing an Atlas_Core snippet and an +// Atlas_Web_Content building block, and `--save-edits` wrote MDL that would have +// *emptied* the snippet if replayed (mxcli-chat FINDINGS §16). Unknown is the +// honest verdict — `verified:false` already tells the caller that "no +// modifications found" is not a conclusion. +// +// The two signals are deliberately about the *output*, not a list of types: a +// type list goes stale the moment a describe handler improves, while "this text +// contains nothing to compare" stays true by construction. +func (e Element) Conclusive() (bool, string) { + if !e.Describable() { + return false, e.Err + } + if declaredReadOnly(e.MDL) { + return false, "DESCRIBE output is informational for this type, not re-executable MDL" + } + if emptyBody(e.MDL) { + return false, "DESCRIBE produced no body, so a difference here is not evidence of an edit" + } + return true, "" +} + +// declaredReadOnly spots the comment a describe handler emits when its output +// cannot be replayed. Matching the handler's own words keeps the two in step: +// a handler that stops saying it is read-only has become authorable. +func declaredReadOnly(mdl string) bool { + return strings.Contains(mdl, "read-only; they cannot be created via MDL") +} + +// emptyBody reports whether a describe rendered a statement with nothing in it — +// `… { }` or `… ( … );` with no inner lines. Such output is identical for an +// element with content and one without, so it cannot distinguish them. +func emptyBody(mdl string) bool { + trimmed := strings.TrimSpace(mdl) + if trimmed == "" { + return true + } + // A body was rendered but holds nothing: "{ }" or "{" immediately followed + // by "}". + compact := strings.Join(strings.Fields(trimmed), " ") + return strings.HasSuffix(compact, "{ }") || strings.HasSuffix(compact, "{}") +} + // Snapshot is the describable content of one module at one point in time. type Snapshot struct { Module string diff --git a/cmd/mxcli/marketplace/update.go b/cmd/mxcli/marketplace/update.go index cdf6ac380..b4bb8b795 100644 --- a/cmd/mxcli/marketplace/update.go +++ b/cmd/mxcli/marketplace/update.go @@ -40,6 +40,16 @@ func SaveEdits(dir string, rep *Report) (written []string, unsaved []string, err for _, f := range rep.Findings { switch f.Verdict { case Modified, OnlyInstalled: + // An element whose DESCRIBE carries no replayable content must not be + // written out as something to replay. Saving `create or modify snippet + // X (Folder: 'Web') { }` and replaying it EMPTIES the snippet — the + // file reads as a rescue and is a deletion (FINDINGS §16). Modified + // findings are already filtered by Compare; OnlyInstalled ones reach + // here unchecked, so the same rule is applied to both. + if ok, why := (Element{MDL: f.InstalledMDL}).Conclusive(); !ok { + unsaved = append(unsaved, fmt.Sprintf("%s (%s)", f.Key, why)) + continue + } wanted = append(wanted, f) case Unknown: unsaved = append(unsaved, fmt.Sprintf("%s (%s)", f.Key, f.Reason)) diff --git a/cmd/mxcli/runconstants.go b/cmd/mxcli/runconstants.go new file mode 100644 index 000000000..42abb7784 --- /dev/null +++ b/cmd/mxcli/runconstants.go @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "io" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/model" +) + +// constantOverridesFor reads the running configuration's constant values from a +// project. A project that cannot be read yields no overrides and a note — a run +// must not be blocked by this, since every run before it applied none. +func constantOverridesFor(projectPath, configuration string) constantOverrides { + b := newBackendFactory()() + if err := b.Connect(projectPath); err != nil { + return constantOverrides{Values: map[string]string{}, Note: "could not read the project's settings: " + err.Error()} + } + defer func() { _ = b.Disconnect() }() + + ps, err := b.GetProjectSettings() + if err != nil { + return constantOverrides{Values: map[string]string{}, Note: "could not read the project's settings: " + err.Error()} + } + return resolveConstantOverrides(ps, configuration) +} + +// Constant values set per *configuration* never reached a locally-run app. +// +// mxbuild writes /model/config.json with each constant's **default** +// value, and that map is what the standalone runtime is handed as +// MicroflowConstants — the configuration's overrides are not in it. So +// +// alter settings constant 'Encryption.EncryptionKey' value '…' in configuration 'Default'; +// +// executed, reported success, survived a round-trip through `describe settings`, +// and then did nothing: the app ran with the constant's default. Measured in +// mxcli-chat FINDINGS §33, where an app ran for hours with an empty encryption +// key while the model said otherwise. Nothing failed — a wrong constant is not a +// build error — which is what makes a silent no-op the worst shape for this bug. +// +// Studio Pro runs a *configuration*; so does this now. resolveConstantOverrides +// picks one and returns its shared constant values, for the caller to merge over +// the defaults at boot. +// +// Two things it deliberately does not do: +// +// - A **private** override carries no value in the model at all (the stored +// node is a Settings$PrivateValue marker — the value lives on the developer's +// workstation). Applying it would blank the constant, so it is skipped and +// named. "" here means "not in the model", never "overridden with empty". +// - It does not guess when the project has several configurations and none is +// obviously the one to run. Applying the wrong environment's database URL or +// API key silently is worse than applying none. +type constantOverrides struct { + Configuration string // the configuration whose values these are + Values map[string]string // constant qualified name -> value + Private []string // overrides whose value is not in the model + Note string // why nothing was applied, when Values is empty +} + +// resolveConstantOverrides chooses a configuration and reads its shared constant +// values. want names one explicitly; empty means "pick the obvious one". +func resolveConstantOverrides(ps *model.ProjectSettings, want string) constantOverrides { + out := constantOverrides{Values: map[string]string{}} + if ps == nil || ps.Configuration == nil || len(ps.Configuration.Configurations) == 0 { + out.Note = "the project has no configurations" + return out + } + cfgs := ps.Configuration.Configurations + + var chosen *model.ServerConfiguration + switch { + case want != "": + for _, c := range cfgs { + if strings.EqualFold(c.Name, want) { + chosen = c + break + } + } + if chosen == nil { + out.Note = fmt.Sprintf("no configuration named %q (have: %s)", want, configurationNames(cfgs)) + return out + } + case len(cfgs) == 1: + chosen = cfgs[0] + default: + for _, c := range cfgs { + if strings.EqualFold(c.Name, "Default") { + chosen = c + break + } + } + if chosen == nil { + // Several configurations and no "Default": which one this run means is + // the user's call, not a guess worth making silently. + out.Note = fmt.Sprintf("the project has %d configurations and none is named \"Default\" (%s); "+ + "pass --configuration to choose one", len(cfgs), configurationNames(cfgs)) + return out + } + } + + out.Configuration = chosen.Name + for _, cv := range chosen.ConstantValues { + if cv == nil { + continue + } + if cv.IsPrivate { + out.Private = append(out.Private, cv.ConstantId) + continue + } + out.Values[cv.ConstantId] = cv.Value + } + sort.Strings(out.Private) + return out +} + +func configurationNames(cfgs []*model.ServerConfiguration) string { + names := make([]string, 0, len(cfgs)) + for _, c := range cfgs { + names = append(names, c.Name) + } + sort.Strings(names) + return strings.Join(names, ", ") +} + +// reportConstantOverrides says what will be applied before the app boots. +// +// It prints even when nothing is applied. The failure this fixes was invisible +// precisely because the run said nothing about constants either way, so silence +// has to stop meaning "your override is in effect". +func reportConstantOverrides(w io.Writer, o constantOverrides) { + switch { + case len(o.Values) > 0: + names := make([]string, 0, len(o.Values)) + for k := range o.Values { + names = append(names, k) + } + sort.Strings(names) + fmt.Fprintf(w, "Applying %d constant value(s) from configuration %q: %s\n", + len(names), o.Configuration, strings.Join(names, ", ")) + case o.Configuration != "": + fmt.Fprintf(w, "Configuration %q sets no constant values; using each constant's default.\n", o.Configuration) + case o.Note != "": + fmt.Fprintf(w, "Using each constant's default value (%s).\n", o.Note) + } + if len(o.Private) > 0 { + fmt.Fprintf(w, " %d override(s) are private, so their value is not in the model and the default is used:\n %s\n", + len(o.Private), strings.Join(o.Private, "\n ")) + } +} diff --git a/cmd/mxcli/runconstants_test.go b/cmd/mxcli/runconstants_test.go new file mode 100644 index 000000000..1b3c1d0bf --- /dev/null +++ b/cmd/mxcli/runconstants_test.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +// mxcli-chat FINDINGS §33: `alter settings constant … in configuration 'Default'` +// executed, reported success, round-tripped through `describe settings` — and +// never reached the running app, because mxbuild writes each constant's +// *default* into deployment/model/config.json and that map is what the runtime +// is handed. The app ran for hours with an empty encryption key. +package main + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/model" +) + +func settingsWith(cfgs ...*model.ServerConfiguration) *model.ProjectSettings { + return &model.ProjectSettings{ + Configuration: &model.ConfigurationSettings{Configurations: cfgs}, + } +} + +func cfg(name string, values ...*model.ConstantValue) *model.ServerConfiguration { + return &model.ServerConfiguration{Name: name, ConstantValues: values} +} + +func shared(id, value string) *model.ConstantValue { + return &model.ConstantValue{ConstantId: id, Value: value} +} + +func private(id string) *model.ConstantValue { + return &model.ConstantValue{ConstantId: id, IsPrivate: true} +} + +func TestResolveConstantOverrides_ReadsTheConfigurationsValues(t *testing.T) { + ps := settingsWith(cfg("Default", shared("Encryption.EncryptionKey", "95d6"))) + got := resolveConstantOverrides(ps, "") + if got.Configuration != "Default" { + t.Errorf("configuration = %q, want Default", got.Configuration) + } + if got.Values["Encryption.EncryptionKey"] != "95d6" { + t.Errorf("values = %v, want the override — this is the whole bug", got.Values) + } +} + +// The only configuration is the one this run means, whatever it is called. +func TestResolveConstantOverrides_UsesTheOnlyConfiguration(t *testing.T) { + ps := settingsWith(cfg("Whatever", shared("A.B", "v"))) + if got := resolveConstantOverrides(ps, ""); got.Configuration != "Whatever" || got.Values["A.B"] != "v" { + t.Errorf("got %+v, want the sole configuration applied", got) + } +} + +// With several configurations and no "Default", picking one silently could apply +// production's database URL or API key to a local run. Refuse and say so. +func TestResolveConstantOverrides_DoesNotGuessBetweenConfigurations(t *testing.T) { + ps := settingsWith(cfg("Acceptance", shared("A.B", "acc")), cfg("Production", shared("A.B", "prod"))) + got := resolveConstantOverrides(ps, "") + if len(got.Values) != 0 { + t.Errorf("applied %v without being told which configuration to run", got.Values) + } + if !strings.Contains(got.Note, "--configuration") { + t.Errorf("note does not say how to resolve it: %q", got.Note) + } + // ...and naming one resolves it. + if got := resolveConstantOverrides(ps, "Production"); got.Values["A.B"] != "prod" { + t.Errorf("--configuration Production gave %+v", got) + } +} + +func TestResolveConstantOverrides_NamesAnUnknownConfiguration(t *testing.T) { + ps := settingsWith(cfg("Default"), cfg("Production")) + got := resolveConstantOverrides(ps, "Staging") + if len(got.Values) != 0 { + t.Errorf("applied values for a configuration that does not exist: %v", got.Values) + } + for _, want := range []string{"Staging", "Default", "Production"} { + if !strings.Contains(got.Note, want) { + t.Errorf("note %q should name %q", got.Note, want) + } + } +} + +// A private override's value is not in the model — the stored node is a marker. +// Applying it would blank the constant, which is worse than leaving the default. +func TestResolveConstantOverrides_SkipsPrivateOverridesAndNamesThem(t *testing.T) { + ps := settingsWith(cfg("Default", shared("A.Shared", "v"), private("A.Private"))) + got := resolveConstantOverrides(ps, "") + if _, applied := got.Values["A.Private"]; applied { + t.Error("a private override was applied; its value is not in the model, so this blanks the constant") + } + if len(got.Private) != 1 || got.Private[0] != "A.Private" { + t.Errorf("private overrides not reported: %v", got.Private) + } + if got.Values["A.Shared"] != "v" { + t.Error("the shared override in the same configuration was dropped too") + } +} + +func TestResolveConstantOverrides_ProjectWithNoConfigurations(t *testing.T) { + if got := resolveConstantOverrides(&model.ProjectSettings{}, ""); len(got.Values) != 0 || got.Note == "" { + t.Errorf("got %+v, want no values and a reason", got) + } +} + +// Silence used to mean "your override is in effect" when it was not. Every +// outcome has to print something. +func TestReportConstantOverrides_SaysSomethingInEveryCase(t *testing.T) { + cases := []constantOverrides{ + {Configuration: "Default", Values: map[string]string{"A.B": "v"}}, + {Configuration: "Default", Values: map[string]string{}}, + {Values: map[string]string{}, Note: "the project has no configurations"}, + {Configuration: "Default", Values: map[string]string{}, Private: []string{"A.P"}}, + } + for i, c := range cases { + var buf bytes.Buffer + reportConstantOverrides(&buf, c) + if strings.TrimSpace(buf.String()) == "" { + t.Errorf("case %d printed nothing: %+v", i, c) + } + } +} From 44cab2914da3afa0c4a2093cc2bf8422b7a5a1c4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 21:21:34 +0000 Subject: [PATCH 03/20] Resolve associations inherited from a generalization in GRANT MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §26. `GRANT … ON Module.Specialization (READ (…, SomeAssociation))` was refused with "entity X has no member(s) SomeAssociation; grant only names members of the entity or of an entity it inherits from" — for an association declared on exactly such an entity. The member walk resolved inherited attributes through the generalization chain (#758) but collected associations only from the entity's own domain model where ParentID == entity.ID, so anything declared on an ancestor was invisible. That made the rule OpenAIConnector ships impossible to express in MDL: OpenAIDeployedModel extends GenAICommons.DeployedModel, and DeployedModel_InputModality is declared on the parent. The walk now follows the same chain the attribute walk does, qualifying each reference against the module that declares the association. It finds the declaring domain model by looking for the ancestor entity rather than by matching module names — the name lookup goes through the hierarchy cache and returns "" often enough that filtering on it collected nothing, which is how the first cut of this looked right and did nothing. Verified on a real project: the grant that was refused is accepted. Not fixed, and now localised. The emitted entry does not reach storage. On a two-entity fixture built for this — Derived EXTENDS Base, the association declared on Base — the executor passes three MemberAccess entries and the stored rule holds two, so `GRANT … ON Derived (READ *, WRITE *)` still reports CE0066 while the same rule on Base checks clean. That reproduces FINDINGS §25 with no marketplace module involved, and places the loss between EntityAccessRuleParams.MemberAccesses and the persisted DomainModels$MemberAccess list. Recorded in the symptom table as the place to pick it up. Tests verified to fail with the reported symptom when the walk is stubbed out, including the negative case: an unrelated entity must not receive the entry, since a walk collecting every association in the module passes the positive tests and puts entries exactly where Mendix reports CE0066 for having them. --- .claude/skills/fix-issue.md | 1 + .../cmd_security_inherited_assoc_test.go | 152 ++++++++++++++++++ mdl/executor/cmd_security_write.go | 117 ++++++++++++++ 3 files changed, 270 insertions(+) create mode 100644 mdl/executor/cmd_security_inherited_assoc_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index d8b2979a7..d8fdbfb8c 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -494,3 +494,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli marketplace update`/`diff` fail with `version "6.0.0" not found` on a module the project actually has, and `--force` does not bypass it | Both download the **installed** version to build the local-edit baseline, and that version had been unpublished. A blank 11.13 app ships NanoflowCommons 6.0.0 while the 6.x line now starts at 6.1.1 — so the module most in need of updating is the one whose baseline cannot be built | `cmd/mxcli/cmd_marketplace_update.go` (`--no-baseline`, and the refusal now names it) | **`--force` was never going to work**: it overrides a *finding*, and here the comparison never ran, so there is no finding. Needing a second flag is the signal that these are two different decisions — "I accept losing edits I have been shown" vs "I accept not being shown". **Say what the flag costs in the flag's own message** — with no baseline, local edits go without being named, so the honest instruction is to commit first. Reported in mxcli-chat FINDINGS §15 | | A constant set with `alter settings constant 'M.C' value '…' in configuration 'Default'` has no effect on the running app. The statement succeeds, `describe settings` round-trips it, `mx check` is clean, and the app behaves as if the constant were still at its default | mxbuild writes `deployment/model/config.json` with each constant's **default** value, and that map is what `run --local` hands the standalone runtime as `MicroflowConstants` — the configuration's overrides are not in it, and nothing read them. An app ran for hours with an empty encryption key while the model said otherwise | `cmd/mxcli/runconstants.go` (`resolveConstantOverrides`, `reportConstantOverrides`), `cmd_run.go` (`--configuration`), `cmd/mxcli/docker/localboot.go` (`mergeConstantOverrides`) | **A silent no-op is the worst shape a bug can take** — every layer reported success. The fix therefore prints in *every* case, including "no overrides applied", so silence stops meaning "your value is in effect". **Merge, never replace**: `config.json` carries the defaults for constants the configuration is silent about, and replacing the map drops them (the app 530s on the first microflow that reads one) — which is exactly the shape `--runtime-setting MicroflowConstants={…}` has. **A private override has no value in the model at all** (a `Settings$PrivateValue` marker; the value lives on the workstation), so applying it would blank the constant — skip and name it. **Do not guess between configurations**: with several and no `Default`, applying one silently could push production's API key into a local run. Tests `cmd/mxcli/runconstants_test.go`, `cmd/mxcli/docker/localboot_constants_test.go`; verified live — the run now prints `Applying 1 constant value(s) from configuration "Default"`. Reported in mxcli-chat FINDINGS §33 | | `mxcli marketplace diff` reports local edits on a project nobody has edited — typically an Atlas snippet and a building block. `--save-edits` then writes MDL that would *destroy* the element if replayed (an empty `create or modify snippet X (Folder: 'Web') { }`) | The comparison is on DESCRIBE output, and `Describable()` only asked whether describe *errored*. Some types describe successfully into output that says nothing: an empty `{ }` body, or a building block under "Building blocks are read-only; they cannot be created via MDL". Two such renderings can still differ — one unresolved name is enough — and the difference was reported as a user edit | `cmd/mxcli/marketplace/snapshot.go` (`Element.Conclusive`, `declaredReadOnly`, `emptyBody`), `compare.go` (`classify`), `update.go` (`SaveEdits`) | **Check equality BEFORE conclusiveness** — identical text is solid evidence of "unchanged" whatever the type, and inverting the order marks every Atlas building block unknown and drains `verified` of meaning. **Judge the output, not the type**: a type list goes stale the moment a describe handler improves, while "this text contains nothing to compare" stays true by construction — match the handler's own read-only wording so the two stay in step. **A rescue file must never be a deletion**: the same rule gates `--save-edits`, including `OnlyInstalled` findings, which reach it without passing through `classify`. **Test that a real edit is still Modified** — a guard that returns unknown for everything passes every "not evidence" test. Tests `cmd/mxcli/marketplace/inconclusive_test.go`. Reported in mxcli-chat FINDINGS §16 | +| `GRANT ON Module.Specialization (READ (…, SomeAssociation))` is refused with "entity X has no member(s) SomeAssociation; grant only names members of the entity or of an entity it inherits from" — when the association *is* declared on an entity it inherits from | The member walk resolved inherited **attributes** through the generalization chain (#758) but collected associations only from the entity's own domain model where `ParentID == entity.ID`. An association declared on an ancestor was therefore invisible, which made the rule OpenAIConnector ships impossible to express in MDL (`OpenAIDeployedModel extends GenAICommons.DeployedModel`) | `mdl/executor/cmd_security_write.go` (`inheritedAssociations`, `domainModelHasEntity`) | **Find the declaring domain model by looking for the entity, not by matching module names** — the module-name lookup goes through the hierarchy cache and returns "" often enough that filtering on it silently collected nothing; the first cut of this fix looked right and did nothing. **Qualify the ref against the module that DECLARES the association**, as the attribute walk does. **Test that an unrelated entity does not get the entry** — a walk collecting every association in the module passes the positive tests and puts entries exactly where Mendix reports CE0066 for having them. Tests `mdl/executor/cmd_security_inherited_assoc_test.go`; the mock needs `GetModuleByNameFunc`, without which even own attributes fail to resolve. Reported in mxcli-chat FINDINGS §26. **STILL OPEN — the emitted entry does not reach storage.** Measured on a two-entity fixture (`Derived EXTENDS Base`, association declared on `Base`): the executor now passes 3 MemberAccess entries, the stored rule holds 2, and `GRANT … ON Derived (READ *, WRITE *)` still gives CE0066 while the same rule on `Base` checks clean. The loss is between `EntityAccessRuleParams.MemberAccesses` and the persisted `DomainModels$MemberAccess` list — that is where FINDINGS §25 should be picked up | diff --git a/mdl/executor/cmd_security_inherited_assoc_test.go b/mdl/executor/cmd_security_inherited_assoc_test.go new file mode 100644 index 000000000..a689271d4 --- /dev/null +++ b/mdl/executor/cmd_security_inherited_assoc_test.go @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/security" +) + +// mxcli-chat FINDINGS §26: a GRANT naming an association declared on the +// generalization was refused — +// +// entity OpenAIConnector.OpenAIDeployedModel has no member(s) +// DeployedModel_InputModality; grant only names members of the entity or of an +// entity it inherits from +// +// — which contradicts its own rule: OpenAIDeployedModel extends +// GenAICommons.DeployedModel, and that association is declared on the parent. +// Inherited *attributes* resolved; inherited *associations* did not, which made +// the rule the module itself ships impossible to express in MDL. +func inheritedAssocFixture(t *testing.T) (*ExecContext, **backend.EntityAccessRuleParams) { + t.Helper() + const ( + baseID = model.ID("e-base") + derivedID = model.ID("e-derived") + otherID = model.ID("e-other") + ) + mod := mkModule("IT") + h := mkHierarchy(mod) + + base := &domainmodel.Entity{ + BaseElement: model.BaseElement{ID: baseID}, + Name: "Base", + Attributes: []*domainmodel.Attribute{{Name: "Code"}}, + } + derived := &domainmodel.Entity{ + BaseElement: model.BaseElement{ID: derivedID}, + Name: "Derived", + Attributes: []*domainmodel.Attribute{{Name: "Extra"}}, + GeneralizationRef: "IT.Base", + } + other := &domainmodel.Entity{ + BaseElement: model.BaseElement{ID: otherID}, + Name: "Other", + Attributes: []*domainmodel.Attribute{{Name: "Label"}}, + } + dm := &domainmodel.DomainModel{ + BaseElement: model.BaseElement{ID: "dm-it"}, + ContainerID: mod.ID, + Entities: []*domainmodel.Entity{base, derived, other}, + Associations: []*domainmodel.Association{{ + Name: "Base_Other", + ParentID: baseID, // declared on the GENERALIZATION + ChildID: otherID, + Owner: domainmodel.AssociationOwnerDefault, + }}, + } + + var captured *backend.EntityAccessRuleParams + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleByNameFunc: func(string) (*model.Module, error) { return mod, nil }, + ListDomainModelsFunc: func() ([]*domainmodel.DomainModel, error) { return []*domainmodel.DomainModel{dm}, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + GetModuleSecurityFunc: func(model.ID) (*security.ModuleSecurity, error) { + return &security.ModuleSecurity{ModuleRoles: []*security.ModuleRole{{Name: "Admin"}}}, nil + }, + AddEntityAccessRuleFunc: func(p backend.EntityAccessRuleParams) error { + cp := p + captured = &cp + return nil + }, + ReconcileMemberAccessesFunc: func(model.ID, string) (int, error) { return 0, nil }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + return ctx, &captured +} + +func TestGrantEntityAccess_NamingAnInheritedAssociationIsAccepted(t *testing.T) { + ctx, captured := inheritedAssocFixture(t) + + err := execGrantEntityAccess(ctx, &ast.GrantEntityAccessStmt{ + Entity: ast.QualifiedName{Module: "IT", Name: "Derived"}, + Roles: []ast.QualifiedName{{Module: "IT", Name: "Admin"}}, + Rights: []ast.EntityAccessRight{{ + Type: ast.EntityAccessReadMembers, + Members: []string{"Extra", "Code", "Base_Other"}, + }}, + }) + if err != nil { + t.Fatalf("grant naming an inherited association was refused: %v", err) + } + if *captured == nil { + t.Fatal("no rule was written") + } + var found bool + for _, ma := range (*captured).MemberAccesses { + if ma.AssociationRef == "IT.Base_Other" { + found = true + } + } + if !found { + t.Errorf("no MemberAccess for the inherited association: %+v", (*captured).MemberAccesses) + } +} + +// READ * covers inherited members too, associations included. +func TestGrantEntityAccess_ReadAllCoversAnInheritedAssociation(t *testing.T) { + ctx, captured := inheritedAssocFixture(t) + + if err := execGrantEntityAccess(ctx, &ast.GrantEntityAccessStmt{ + Entity: ast.QualifiedName{Module: "IT", Name: "Derived"}, + Roles: []ast.QualifiedName{{Module: "IT", Name: "Admin"}}, + Rights: []ast.EntityAccessRight{{Type: ast.EntityAccessReadAll}}, + }); err != nil { + t.Fatalf("grant failed: %v", err) + } + for _, ma := range (*captured).MemberAccesses { + if ma.AssociationRef == "IT.Base_Other" { + return + } + } + t.Errorf("READ * left the inherited association out: %+v", (*captured).MemberAccesses) +} + +// The negative half: the association must NOT appear on an unrelated entity's +// rule. A walk that collected every association in the module would pass the +// tests above and put entries where Mendix reports CE0066 for having them. +func TestGrantEntityAccess_InheritedAssociationOnlyOnSpecializations(t *testing.T) { + ctx, captured := inheritedAssocFixture(t) + + if err := execGrantEntityAccess(ctx, &ast.GrantEntityAccessStmt{ + Entity: ast.QualifiedName{Module: "IT", Name: "Other"}, + Roles: []ast.QualifiedName{{Module: "IT", Name: "Admin"}}, + Rights: []ast.EntityAccessRight{{Type: ast.EntityAccessReadAll}}, + }); err != nil { + t.Fatalf("grant failed: %v", err) + } + for _, ma := range (*captured).MemberAccesses { + if ma.AssociationRef == "IT.Base_Other" { + t.Errorf("the TO entity of a Default-owner association got a MemberAccess for it (CE0066): %+v", + (*captured).MemberAccesses) + } + } +} diff --git a/mdl/executor/cmd_security_write.go b/mdl/executor/cmd_security_write.go index 52aa3bb4d..408a768f4 100644 --- a/mdl/executor/cmd_security_write.go +++ b/mdl/executor/cmd_security_write.go @@ -486,6 +486,28 @@ func execGrantEntityAccess(ctx *ExecContext, s *ast.GrantEntityAccessStmt) error for _, other := range otherModuleBothOwnerAssociations(ctx, module.Name, entityQN) { addAssociationAccess(other.Name, other.Ref) } + // Associations declared on an ancestor. Mendix inheritance is multi-table: + // a specialization has ALL of its generalization's members, associations + // included, and its access rule needs an entry for each — exactly as it does + // for inherited attributes (#758). + // + // Leaving them out was CE0066 "Entity access is out of date" on the + // specialization's own module, and it made the rule OpenAIConnector ships + // impossible to express in MDL: `OpenAIDeployedModel extends + // GenAICommons.DeployedModel`, and `DeployedModel_InputModality` is declared + // on the parent, so the grant naming it was refused as "no such member" + // (mxcli-chat FINDINGS §26). Reproduced on a two-entity fixture with no + // marketplace module in sight: `GRANT … ON Derived (READ *, WRITE *)` gave + // CE0066 while the same rule on the base entity checked clean. + // + // The reference is qualified against the module that DECLARES the + // association, not this entity's — the same rule the attribute walk follows. + for _, inh := range inheritedAssociations(ctx, entityQN) { + if grantedMembers[inh.Name] { + continue // an association of this entity's own shadows it + } + addAssociationAccess(inh.Name, inh.Ref) + } // A member named in the GRANT that matched nothing used to be dropped in // silence — the command reported success and the access simply was not there, @@ -1594,3 +1616,98 @@ func execUpdateSecurity(ctx *ExecContext, s *ast.UpdateSecurityStmt) error { // Executor method wrappers — delegate to free functions for callers that // still use the Executor receiver (e.g. executor_query.go). + +// inheritedAssociations returns the associations declared on entityQN's +// ancestors, qualified against the module that declares each. +// +// It walks the generalization chain the same way EntityMembersFor does, and +// stops at the same place: System.User's own members are Mendix's, and a user +// entity must not carry access entries for them. +func inheritedAssociations(ctx *ExecContext, entityQN string) []namedAssociation { + if ctx == nil || ctx.Backend == nil { + return nil + } + dms, err := ctx.Backend.ListDomainModels() + if err != nil { + return nil + } + var out []namedAssociation + seen := map[string]bool{} + claimed := map[string]bool{} + + current := entityQN + for depth := 0; current != ""; depth++ { + if seen[current] { + break // cycle guard, as in EntityMembersFor + } + seen[current] = true + + ent, ok := findEntityByQN(ctx.Backend, current) + if !ok { + break + } + parent := ent.GeneralizationRef + if parent == "" || strings.EqualFold(parent, userEntityBase) { + break + } + ancestor, ok := findEntityByQN(ctx.Backend, parent) + if !ok { + break + } + ancestorModule := qualifiedModuleOf(parent) + for _, dm := range dms { + // Find the domain model that DECLARES the ancestor by looking for the + // entity itself, rather than by matching module names: the module-name + // lookup goes through the hierarchy cache and returns "" often enough + // that filtering on it silently collected nothing (which is how the + // first cut of this still produced CE0066). + if !domainModelHasEntity(dm, ancestor.ID) { + continue + } + collect := func(name string, parentID, childID model.ID, owner domainmodel.AssociationOwner) { + ownedThere := parentID == ancestor.ID || + (owner == domainmodel.AssociationOwnerBoth && childID == ancestor.ID) + if !ownedThere || claimed[name] { + return + } + claimed[name] = true + out = append(out, namedAssociation{Name: name, Ref: ancestorModule + "." + name}) + } + for _, a := range dm.Associations { + collect(a.Name, a.ParentID, a.ChildID, a.Owner) + } + for _, ca := range dm.CrossAssociations { + // A cross-module association names its remote end by qualified name, + // so the Both-owner case is matched on ChildRef rather than an ID. + childID := model.ID("") + if ca.Owner == domainmodel.AssociationOwnerBoth && ca.ChildRef == parent { + childID = ancestor.ID + } + collect(ca.Name, ca.ParentID, childID, ca.Owner) + } + } + current = parent + } + return out +} + +// qualifiedModuleOf is the module part of "Module.Entity". +func qualifiedModuleOf(qn string) string { + if i := strings.Index(qn, "."); i > 0 { + return qn[:i] + } + return "" +} + +// domainModelHasEntity reports whether a domain model declares the given entity. +func domainModelHasEntity(dm *domainmodel.DomainModel, id model.ID) bool { + if dm == nil { + return false + } + for _, e := range dm.Entities { + if e != nil && e.ID == id { + return true + } + } + return false +} From 961a3493dedb49dbb2368cd24d764c43cf218bdc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 22:56:41 +0000 Subject: [PATCH 04/20] Read a java action's microflow parameter as a microflow, not a String MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A java action parameter typed Microflow — MCPServer.AddTool's ExecutingMicroflow, and every other "register a callback" action — read back from the model as a String, because the modelsdk engine's gen->semantic converter had no case for it and its default returned StringType. The microflow builder's microflowTypeParams branch was therefore never taken: it authored the callback as Microflows$BasicCodeActionParameterValue{Argument: "'M.MyFlow'"} instead of Microflows$MicroflowParameterValue{Microflow: "M.MyFlow"}, and Mendix reported CE0115. The legacy parser and the executor both handled this correctly; only the converter degraded silently, so every other layer looked right in isolation. Fix read and write together. Without the write case an update of such an action would rewrite the parameter as a String, which is worse than the read bug. The stored shape is a direct JavaActions$MicroflowJavaActionParameterType (not wrapped in a BasicParameterType), measured against MCP Server 5.1.0. DESCRIBE now prints the honest `Microflow` for these parameters, which round-tripped into an entity type with an empty module (`.Microflow`) until astDataTypeToJavaActionParamType learned the bare word; only the unqualified name is treated this way, so a real Module.Microflow entity is unaffected. Verified end to end: the call now serializes as MicroflowParameterValue and `mx check` on 11.12.1 goes from CE0115 to 0 errors. Reported in mxcli-chat FINDINGS §36. --- .claude/skills/mendix/write-microflows.md | 22 ++++++++ .../javaaction-microflow-parameter.mdl | 55 +++++++++++++++++++ mdl/backend/modelsdk/java_read.go | 14 +++++ mdl/backend/modelsdk/java_read_test.go | 47 ++++++++++++++++ mdl/backend/modelsdk/java_write.go | 11 ++++ mdl/executor/cmd_javaactions.go | 25 +++++++++ mdl/executor/cmd_javaactions_test.go | 35 ++++++++++++ 7 files changed, 209 insertions(+) create mode 100644 mdl-examples/bug-tests/javaaction-microflow-parameter.mdl diff --git a/.claude/skills/mendix/write-microflows.md b/.claude/skills/mendix/write-microflows.md index ef50abce4..8005d9a47 100644 --- a/.claude/skills/mendix/write-microflows.md +++ b/.claude/skills/mendix/write-microflows.md @@ -1312,6 +1312,28 @@ New scripts should bind every parameter to a real expression. Use `empty` for a Java-action argument only when regenerating MDL from an existing project that already had an unbound parameter. +## Microflow-Typed Java-Action Parameters + +Some Java actions take a **microflow** — a callback the action invokes later. +`MCPServer.AddTool` (`ExecutingMicroflow`) and `MCPServer.CreateMCPServer` +(`AuthenticationMicroflow`) are the ones you meet first. Pass the microflow's +qualified name as a quoted string; mxcli resolves the parameter's declared type +from the Java action and stores a microflow reference, not a string literal. + +```mdl +$Tool = call java action MCPServer.AddTool( + McpServer = $Server, + Name = 'memory_add', + Description = 'Stores a memory', + ExecutingMicroflow = 'MyModule.MF_MemoryAdd', + Schema = '' +); +``` + +`DESCRIBE JAVA ACTION` prints such a parameter's type as the bare word +`Microflow` (`Nanoflow` for JavaScript actions), and that spelling is what +`CREATE JAVA ACTION` accepts, so the round-trip is stable. + ## Error Handling MDL supports error handling for activities that may fail (microflow calls, commits, external service calls, etc.). diff --git a/mdl-examples/bug-tests/javaaction-microflow-parameter.mdl b/mdl-examples/bug-tests/javaaction-microflow-parameter.mdl new file mode 100644 index 000000000..11cbf9038 --- /dev/null +++ b/mdl-examples/bug-tests/javaaction-microflow-parameter.mdl @@ -0,0 +1,55 @@ +-- ============================================================================ +-- mxcli-chat FINDINGS §36 / §37 — java actions with a microflow parameter +-- +-- §36 A java action parameter of type Microflow (MCPServer.AddTool's +-- ExecutingMicroflow, and every other "register a callback" action) read +-- back from the model as a String, so the microflow builder authored the +-- callback as Microflows$BasicCodeActionParameterValue holding +-- 'Module.MyFlow' as a literal instead of Microflows$MicroflowParameterValue. +-- Mendix reported CE0115. +-- +-- §37 `mxcli check --references` reported "java action not found" for an action +-- created earlier in the SAME script. Entities, microflows, pages and +-- nanoflows were exempt; java actions were not. +-- +-- Verify: +-- mxcli check mdl-examples/bug-tests/javaaction-microflow-parameter.mdl \ +-- -p app.mpr --references # §37 — must pass, no "not found" +-- mxcli exec ... -p app.mpr +-- mx check app.mpr # §36 — 0 errors, no CE0115 +-- mxcli -p app.mpr -c "describe java action Callbacks.RegisterTool" +-- # -> "ExecutingMicroflow: Microflow" +-- ============================================================================ + +create module Callbacks; +/ + +-- The parameter type is the bare word `Microflow`; that is also what DESCRIBE +-- prints, so this file is what a describe/re-execute round-trip produces. +create java action Callbacks.RegisterTool( + ToolName: String not null, + ExecutingMicroflow: Microflow not null +) returns Boolean +as $$ + return true; +$$; +/ + +create microflow Callbacks.MF_Tool (Input: String) +returns string +begin + return $Input; +end; +/ + +-- §37: RegisterTool is two statements old and not in the project yet. +-- §36: ExecutingMicroflow must serialize as MicroflowParameterValue. +create microflow Callbacks.MF_Register () +returns boolean +begin + $Registered = call java action Callbacks.RegisterTool( + ToolName = 'echo', + ExecutingMicroflow = 'Callbacks.MF_Tool'); + return $Registered; +end; +/ diff --git a/mdl/backend/modelsdk/java_read.go b/mdl/backend/modelsdk/java_read.go index 0a28dfa9d..45e4a111e 100644 --- a/mdl/backend/modelsdk/java_read.go +++ b/mdl/backend/modelsdk/java_read.go @@ -132,6 +132,20 @@ func codeActionParamTypeFromGen(el element.Element) javaactions.CodeActionParame // directly-typed element) to the semantic parameter type. func codeActionBasicFromGen(el element.Element) javaactions.CodeActionParameterType { switch t := el.(type) { + // A microflow-typed parameter (a java action that takes a microflow to call + // back into — MCPServer.AddTool's ExecutingMicroflow, and every "register a + // handler" action). Missing here, it fell through to the default and read + // back as a String, so the microflow builder authored a + // BasicCodeActionParameterValue holding the microflow's name as a literal + // and `mx check` reported CE0115. mxcli-chat FINDINGS §36. + case *genJa.MicroflowJavaActionParameterType: + m := &javaactions.MicroflowType{} + m.ID = model.ID(t.ID()) + return m + case *genJa.MicroflowParameterType: + m := &javaactions.MicroflowType{} + m.ID = model.ID(t.ID()) + return m case *genCa.EnumerationType: return &javaactions.EnumerationType{Enumeration: t.EnumerationQualifiedName()} case *genCa.ConcreteEntityType: diff --git a/mdl/backend/modelsdk/java_read_test.go b/mdl/backend/modelsdk/java_read_test.go index d5b1ef205..ad94050b7 100644 --- a/mdl/backend/modelsdk/java_read_test.go +++ b/mdl/backend/modelsdk/java_read_test.go @@ -57,3 +57,50 @@ func TestReadJavaActionByName_RoundTrip(t *testing.T) { t.Errorf("return type = %T, want *BooleanType", got.ReturnType) } } + +// A microflow-typed parameter — the shape MCPServer.AddTool's ExecutingMicroflow +// and every other "register a callback" java action uses — read back as a String, +// because codeActionBasicFromGen had no case for it and fell through to the +// default. The microflow builder then authored the callback as a +// BasicCodeActionParameterValue holding 'Module.MyFlow' as a literal instead of a +// MicroflowParameterValue, and `mx check` reported CE0115. mxcli-chat FINDINGS §36. +func TestReadJavaActionByName_MicroflowParameterType(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + ja := &javaactions.JavaAction{ + ContainerID: mod.ID, + Name: "ZzRegisterTool", + Parameters: []*javaactions.JavaActionParameter{ + {Name: "Name", IsRequired: true, ParameterType: &javaactions.StringType{}}, + {Name: "ExecutingMicroflow", IsRequired: true, ParameterType: &javaactions.MicroflowType{}}, + }, + } + if err := b.CreateJavaAction(ja); err != nil { + t.Fatalf("CreateJavaAction: %v", err) + } + + got, err := b.ReadJavaActionByName("MyFirstModule.ZzRegisterTool") + if err != nil { + t.Fatalf("ReadJavaActionByName: %v", err) + } + if len(got.Parameters) != 2 { + t.Fatalf("params = %d, want 2", len(got.Parameters)) + } + if _, ok := got.Parameters[0].ParameterType.(*javaactions.StringType); !ok { + t.Errorf("param 0 type = %T, want *StringType", got.Parameters[0].ParameterType) + } + if _, ok := got.Parameters[1].ParameterType.(*javaactions.MicroflowType); !ok { + t.Fatalf("param 1 type = %T, want *MicroflowType — a microflow callback read back as this "+ + "is what makes the caller author a literal string and fail CE0115", + got.Parameters[1].ParameterType) + } +} diff --git a/mdl/backend/modelsdk/java_write.go b/mdl/backend/modelsdk/java_write.go index a7e69f952..76b08248a 100644 --- a/mdl/backend/modelsdk/java_write.go +++ b/mdl/backend/modelsdk/java_write.go @@ -268,6 +268,17 @@ func codeActionParamTypeToGen(t javaactions.CodeActionParameterType) element.Ele assignID(e) e.SetTypeParameterID(element.ID(v.TypeParameterID)) return e + case *javaactions.MicroflowType: + // Direct, not wrapped in a BasicParameterType — the shape Studio Pro + // stores (measured on MCP Server 5.1.0's AddTool.ExecutingMicroflow). + // Without this an update of a java action that has a microflow-typed + // parameter would rewrite it as a String. See codeActionBasicFromGen. + m := genJa.NewMicroflowJavaActionParameterType() + if v.ID != "" { + m.SetID(element.ID(v.ID)) + } + assignID(m) + return m default: b := genCa.NewBasicParameterType() assignID(b) diff --git a/mdl/executor/cmd_javaactions.go b/mdl/executor/cmd_javaactions.go index c9b7d9d9b..8a5564370 100644 --- a/mdl/executor/cmd_javaactions.go +++ b/mdl/executor/cmd_javaactions.go @@ -512,6 +512,19 @@ func astDataTypeToJavaActionParamType(dt ast.DataType) javaactions.CodeActionPar Enumeration: dt.EnumRef.Module + "." + dt.EnumRef.Name, } } + // `Microflow` / `Nanoflow` are what DESCRIBE prints for a callback + // parameter (MCPServer.AddTool's ExecutingMicroflow, and every other + // "register a handler" action). The parser cannot tell a bare name from + // an entity, so without this the DESCRIBE output round-tripped into an + // entity type with an empty module — `.Microflow`. Only the unqualified + // name is treated this way; a real `Module.Microflow` entity is not. + if bare := bareDataTypeName(dt); bare == "Microflow" || bare == "Nanoflow" { + id := model.ID(types.GenerateID()) + if bare == "Nanoflow" { + return &javaactions.NanoflowType{BaseElement: model.BaseElement{ID: id}} + } + return &javaactions.MicroflowType{BaseElement: model.BaseElement{ID: id}} + } entityName := "" if dt.EntityRef != nil { entityName = dt.EntityRef.Module + "." + dt.EntityRef.Name @@ -548,6 +561,18 @@ func astDataTypeToJavaActionParamType(dt ast.DataType) javaactions.CodeActionPar } } +// bareDataTypeName returns the name of an unqualified entity/enumeration data +// type (no module part), or "" when the type is qualified or absent. +func bareDataTypeName(dt ast.DataType) string { + switch { + case dt.EntityRef != nil && dt.EntityRef.Module == "": + return dt.EntityRef.Name + case dt.EnumRef != nil && dt.EnumRef.Module == "": + return dt.EnumRef.Name + } + return "" +} + // astDataTypeToJavaActionReturnType converts an AST DataType to a Java action return type. func astDataTypeToJavaActionReturnType(dt ast.DataType) javaactions.CodeActionReturnType { switch dt.Kind { diff --git a/mdl/executor/cmd_javaactions_test.go b/mdl/executor/cmd_javaactions_test.go index 7139788d1..86ad4eab0 100644 --- a/mdl/executor/cmd_javaactions_test.go +++ b/mdl/executor/cmd_javaactions_test.go @@ -360,3 +360,38 @@ func TestEntityTypeCodeActionParameterValue_Fields(t *testing.T) { t.Errorf("got %q", v.Entity) } } + +// A callback parameter is printed by DESCRIBE as the bare word `Microflow` +// (`Nanoflow` for JavaScript actions). The parser cannot tell a bare name from an +// entity, so re-executing that output used to create an entity-typed parameter +// with an empty module — DESCRIBE then printed `.Microflow` and the round-trip +// was lost. mxcli-chat FINDINGS §36. +func TestAstDataTypeToJavaActionParamType_BareMicroflowAndNanoflow(t *testing.T) { + bare := func(name string) ast.DataType { + return ast.DataType{Kind: ast.TypeEnumeration, EnumRef: &ast.QualifiedName{Name: name}} + } + + if got := astDataTypeToJavaActionParamType(bare("Microflow")); !isType[*javaactions.MicroflowType](got) { + t.Errorf("Microflow -> %T, want *MicroflowType", got) + } + if got := astDataTypeToJavaActionParamType(bare("Nanoflow")); !isType[*javaactions.NanoflowType](got) { + t.Errorf("Nanoflow -> %T, want *NanoflowType", got) + } + + // A qualified entity that happens to be named Microflow is still an entity — + // the special case is only for the unqualified word. + qualified := ast.DataType{ + Kind: ast.TypeEntity, + EntityRef: &ast.QualifiedName{Module: "MyModule", Name: "Microflow"}, + } + ent, ok := astDataTypeToJavaActionParamType(qualified).(*javaactions.EntityType) + if !ok || ent.Entity != "MyModule.Microflow" { + t.Errorf("MyModule.Microflow -> %#v, want EntityType{MyModule.Microflow}", + astDataTypeToJavaActionParamType(qualified)) + } +} + +func isType[T any](v any) bool { + _, ok := v.(T) + return ok +} From 568726e752104718cf19eefc2df15a7272e285c4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 22:56:48 +0000 Subject: [PATCH 05/20] Skip the project lookup for code actions a script itself creates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mxcli check --references` reported "java action not found" for an action the same script created a few statements earlier. scriptContext — the set of objects a script defines, consulted exactly so a script can be checked against its own output — had categories for modules, entities, enumerations, microflows, nanoflows, pages and snippets, but not for java or JavaScript actions, so their branch went straight to the project. Store the declared parameter names rather than a bool: exempting the action from "not found" must not also exempt it from the parameter-name check, which the script has everything it needs to run. allNames() and has() gain the categories together — annotateForwardRef reads both, and updating one alone would make a created action look "defined later in this script" forever. Reported in mxcli-chat FINDINGS §37. --- .claude/skills/fix-issue.md | 2 + mdl/executor/validate.go | 61 ++++++++++++ .../validate_script_javaactions_test.go | 93 +++++++++++++++++++ 3 files changed, 156 insertions(+) create mode 100644 mdl/executor/validate_script_javaactions_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index d8fdbfb8c..550d4526f 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -495,3 +495,5 @@ extracting `OffsetExpression`/`LimitExpression`. | A constant set with `alter settings constant 'M.C' value '…' in configuration 'Default'` has no effect on the running app. The statement succeeds, `describe settings` round-trips it, `mx check` is clean, and the app behaves as if the constant were still at its default | mxbuild writes `deployment/model/config.json` with each constant's **default** value, and that map is what `run --local` hands the standalone runtime as `MicroflowConstants` — the configuration's overrides are not in it, and nothing read them. An app ran for hours with an empty encryption key while the model said otherwise | `cmd/mxcli/runconstants.go` (`resolveConstantOverrides`, `reportConstantOverrides`), `cmd_run.go` (`--configuration`), `cmd/mxcli/docker/localboot.go` (`mergeConstantOverrides`) | **A silent no-op is the worst shape a bug can take** — every layer reported success. The fix therefore prints in *every* case, including "no overrides applied", so silence stops meaning "your value is in effect". **Merge, never replace**: `config.json` carries the defaults for constants the configuration is silent about, and replacing the map drops them (the app 530s on the first microflow that reads one) — which is exactly the shape `--runtime-setting MicroflowConstants={…}` has. **A private override has no value in the model at all** (a `Settings$PrivateValue` marker; the value lives on the workstation), so applying it would blank the constant — skip and name it. **Do not guess between configurations**: with several and no `Default`, applying one silently could push production's API key into a local run. Tests `cmd/mxcli/runconstants_test.go`, `cmd/mxcli/docker/localboot_constants_test.go`; verified live — the run now prints `Applying 1 constant value(s) from configuration "Default"`. Reported in mxcli-chat FINDINGS §33 | | `mxcli marketplace diff` reports local edits on a project nobody has edited — typically an Atlas snippet and a building block. `--save-edits` then writes MDL that would *destroy* the element if replayed (an empty `create or modify snippet X (Folder: 'Web') { }`) | The comparison is on DESCRIBE output, and `Describable()` only asked whether describe *errored*. Some types describe successfully into output that says nothing: an empty `{ }` body, or a building block under "Building blocks are read-only; they cannot be created via MDL". Two such renderings can still differ — one unresolved name is enough — and the difference was reported as a user edit | `cmd/mxcli/marketplace/snapshot.go` (`Element.Conclusive`, `declaredReadOnly`, `emptyBody`), `compare.go` (`classify`), `update.go` (`SaveEdits`) | **Check equality BEFORE conclusiveness** — identical text is solid evidence of "unchanged" whatever the type, and inverting the order marks every Atlas building block unknown and drains `verified` of meaning. **Judge the output, not the type**: a type list goes stale the moment a describe handler improves, while "this text contains nothing to compare" stays true by construction — match the handler's own read-only wording so the two stay in step. **A rescue file must never be a deletion**: the same rule gates `--save-edits`, including `OnlyInstalled` findings, which reach it without passing through `classify`. **Test that a real edit is still Modified** — a guard that returns unknown for everything passes every "not evidence" test. Tests `cmd/mxcli/marketplace/inconclusive_test.go`. Reported in mxcli-chat FINDINGS §16 | | `GRANT ON Module.Specialization (READ (…, SomeAssociation))` is refused with "entity X has no member(s) SomeAssociation; grant only names members of the entity or of an entity it inherits from" — when the association *is* declared on an entity it inherits from | The member walk resolved inherited **attributes** through the generalization chain (#758) but collected associations only from the entity's own domain model where `ParentID == entity.ID`. An association declared on an ancestor was therefore invisible, which made the rule OpenAIConnector ships impossible to express in MDL (`OpenAIDeployedModel extends GenAICommons.DeployedModel`) | `mdl/executor/cmd_security_write.go` (`inheritedAssociations`, `domainModelHasEntity`) | **Find the declaring domain model by looking for the entity, not by matching module names** — the module-name lookup goes through the hierarchy cache and returns "" often enough that filtering on it silently collected nothing; the first cut of this fix looked right and did nothing. **Qualify the ref against the module that DECLARES the association**, as the attribute walk does. **Test that an unrelated entity does not get the entry** — a walk collecting every association in the module passes the positive tests and puts entries exactly where Mendix reports CE0066 for having them. Tests `mdl/executor/cmd_security_inherited_assoc_test.go`; the mock needs `GetModuleByNameFunc`, without which even own attributes fail to resolve. Reported in mxcli-chat FINDINGS §26. **STILL OPEN — the emitted entry does not reach storage.** Measured on a two-entity fixture (`Derived EXTENDS Base`, association declared on `Base`): the executor now passes 3 MemberAccess entries, the stored rule holds 2, and `GRANT … ON Derived (READ *, WRITE *)` still gives CE0066 while the same rule on `Base` checks clean. The loss is between `EntityAccessRuleParams.MemberAccesses` and the persisted `DomainModels$MemberAccess` list — that is where FINDINGS §25 should be picked up | +| A `call java action` whose parameter is a **microflow** (`MCPServer.AddTool`'s `ExecutingMicroflow`, and every other "register a callback" action) fails `mx check` with **CE0115** — the microflow's name is stored as a literal string | The modelsdk engine's java-action read had no case for the parameter type, so `codeActionBasicFromGen` fell through to its `default` and returned a `StringType`. The microflow builder's `microflowTypeParams` branch was therefore never taken and it authored `Microflows$BasicCodeActionParameterValue{Argument: "'M.MyFlow'"}` instead of `Microflows$MicroflowParameterValue{Microflow: "M.MyFlow"}` | `mdl/backend/modelsdk/java_read.go` (`codeActionBasicFromGen`), `java_write.go` (`codeActionParamTypeToGen`), `mdl/executor/cmd_javaactions.go` (`astDataTypeToJavaActionParamType`, `bareDataTypeName`) | **A `default:` that returns a plausible type is how a read path lies.** The legacy parser and the executor both handled this correctly for years; only the gen→semantic converter did not, and it degraded to String rather than erroring — so every layer downstream looked correct in isolation. **Fix read and write together**: without the write case an *update* of such an action rewrites the parameter as a String, which is worse than the read bug. **Watch what DESCRIBE starts printing** — the fix made it emit the honest `Microflow`, which then round-tripped into an entity type with an empty module (`.Microflow`) until `astDataTypeToJavaActionParamType` learned the bare word. Probe the stored `$Type` from the marketplace `.mpk` before theorising; MCP Server 5.1.0 stores `JavaActions$MicroflowJavaActionParameterType`. Tests `mdl/backend/modelsdk/java_read_test.go`, `mdl/executor/cmd_javaactions_test.go`, example `mdl-examples/bug-tests/javaaction-microflow-parameter.mdl`; verified 0 errors under `mx check` 11.12.1. Reported in mxcli-chat FINDINGS §36 | +| `mxcli check --references` reports "java action not found: M.X (referenced by call java action)" for an action the **same script** creates a few statements earlier. Entities, microflows, pages and nanoflows in the same position are accepted | `scriptContext` — the set of objects a script defines, consulted so a script can be checked against its own output — had categories for modules, entities, enumerations, microflows, nanoflows, pages and snippets, but not for java or JavaScript actions. The java-action branch of `validateFlowBodyReferences` went straight to the project lookup | `mdl/executor/validate.go` (`scriptContext.javaActions`/`javaScriptActions`, `collectDefinitions`, `collectSingle`, `allNames`, `has`, `validateFlowBodyReferences`) | **Store the declared parameter names, not a bool.** Exempting the action from "not found" must not also exempt it from the parameter-name check — the script declares the parameters, so a typo is still catchable, and a bool would have silently dropped that. **`allNames()` and `has()` move together**: `annotateForwardRef` reads both, so adding a category to one and not the other makes a *created* object look "defined later in this script" forever. Tests `mdl/executor/validate_script_javaactions_test.go` (including the control that a misspelled parameter still errors). Reported in mxcli-chat FINDINGS §37 | diff --git a/mdl/executor/validate.go b/mdl/executor/validate.go index 09b030932..6a72e0737 100644 --- a/mdl/executor/validate.go +++ b/mdl/executor/validate.go @@ -26,6 +26,13 @@ type scriptContext struct { nanoflows map[string]bool // Nanoflows created (Module.Nanoflow) pages map[string]bool // Pages created (Module.Page) snippets map[string]bool // Snippets created (Module.Snippet) + + // Java/JavaScript actions created in the script, mapped to their declared + // parameter names. A bool would be enough to stop the false "not found", + // but keeping the names means a call to a script-defined action still gets + // its parameters checked, exactly as a call to a stored one does. + javaActions map[string][]string // Module.Action -> parameter names + javaScriptActions map[string][]string // Module.Action -> parameter names } // newScriptContext creates a new script context. @@ -38,9 +45,22 @@ func newScriptContext() *scriptContext { nanoflows: make(map[string]bool), pages: make(map[string]bool), snippets: make(map[string]bool), + + javaActions: make(map[string][]string), + javaScriptActions: make(map[string][]string), } } +// codeActionParamNames returns the declared parameter names of a CREATE JAVA +// ACTION / CREATE JAVASCRIPT ACTION statement, in declaration order. +func codeActionParamNames(params []ast.JavaActionParam) []string { + names := make([]string, 0, len(params)) + for _, p := range params { + names = append(names, p.Name) + } + return names +} + // collectDefinitions scans a program and collects all objects that will be created. func (sc *scriptContext) collectDefinitions(prog *ast.Program) { for _, stmt := range prog.Statements { @@ -79,6 +99,14 @@ func (sc *scriptContext) collectDefinitions(prog *ast.Program) { if s.Name.Module != "" { sc.snippets[s.Name.String()] = true } + case *ast.CreateJavaActionStmt: + if s.Name.Module != "" { + sc.javaActions[s.Name.String()] = codeActionParamNames(s.Parameters) + } + case *ast.CreateJavaScriptActionStmt: + if s.Name.Module != "" { + sc.javaScriptActions[s.Name.String()] = codeActionParamNames(s.Parameters) + } } } } @@ -120,6 +148,14 @@ func (sc *scriptContext) collectSingle(stmt ast.Statement) { if s.Name.Module != "" { sc.snippets[s.Name.String()] = true } + case *ast.CreateJavaActionStmt: + if s.Name.Module != "" { + sc.javaActions[s.Name.String()] = codeActionParamNames(s.Parameters) + } + case *ast.CreateJavaScriptActionStmt: + if s.Name.Module != "" { + sc.javaScriptActions[s.Name.String()] = codeActionParamNames(s.Parameters) + } } } @@ -144,6 +180,12 @@ func (sc *scriptContext) allNames() []string { for n := range sc.snippets { names = append(names, n) } + for n := range sc.javaActions { + names = append(names, n) + } + for n := range sc.javaScriptActions { + names = append(names, n) + } return names } @@ -172,6 +214,12 @@ func annotateForwardRef(err error, stmt ast.Statement, created, allDefined *scri // has returns true if the name exists in any category. func (sc *scriptContext) has(name string) bool { + if _, ok := sc.javaActions[name]; ok { + return true + } + if _, ok := sc.javaScriptActions[name]; ok { + return true + } return sc.modules[name] || sc.entities[name] || sc.enumerations[name] || sc.microflows[name] || sc.nanoflows[name] || sc.pages[name] || sc.snippets[name] } @@ -607,6 +655,15 @@ func validateFlowBodyReferences(ctx *ExecContext, body []ast.MicroflowStatement, if isBuiltinModuleEntity(qualifiedNameModule(ref.name)) { continue } + // An action created earlier in the same script is not in the + // project yet. Entities, microflows, pages and nanoflows were + // already exempt; java actions were not, so a script that created + // one and called it failed reference checking against its own + // output. mxcli-chat FINDINGS §37. + if declared, inScript := sc.javaActions[ref.name]; inScript { + errors = append(errors, validateCodeActionParams("java action", ref, declared)...) + continue + } if !known[ref.name] { errors = append(errors, fmt.Sprintf("java action not found: %s (referenced by call java action)", ref.name)) continue @@ -627,6 +684,10 @@ func validateFlowBodyReferences(ctx *ExecContext, body []ast.MicroflowStatement, if isBuiltinModuleEntity(qualifiedNameModule(ref.name)) { continue } + if declared, inScript := sc.javaScriptActions[ref.name]; inScript { + errors = append(errors, validateCodeActionParams("javascript action", ref, declared)...) + continue + } if !known[ref.name] { errors = append(errors, fmt.Sprintf("javascript action not found: %s (referenced by call javascript action)", ref.name)) continue diff --git a/mdl/executor/validate_script_javaactions_test.go b/mdl/executor/validate_script_javaactions_test.go new file mode 100644 index 000000000..75296efee --- /dev/null +++ b/mdl/executor/validate_script_javaactions_test.go @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: Apache-2.0 + +// mxcli-chat FINDINGS §37: `mxcli check --references` reported "java action not +// found" for an action the same script had just created. Entities, microflows, +// pages, snippets and nanoflows were all exempted from the project lookup when +// defined in the script; java and JavaScript actions were not, so a script that +// created an action and then called it failed reference checking against its own +// output — a false negative with no way to silence it short of splitting the +// script in two and running it twice. +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// javaActionScript is a two-statement program: create an action, then call it. +func javaActionScript(argName string) *ast.Program { + return &ast.Program{Statements: []ast.Statement{ + &ast.CreateJavaActionStmt{ + Name: ast.QualifiedName{Module: "MyFirstModule", Name: "ZzHelper"}, + Parameters: []ast.JavaActionParam{{Name: "Input"}}, + }, + &ast.CreateMicroflowStmt{ + Name: ast.QualifiedName{Module: "MyFirstModule", Name: "MF_UseHelper"}, + Body: []ast.MicroflowStatement{ + &ast.CallJavaActionStmt{ + ActionName: ast.QualifiedName{Module: "MyFirstModule", Name: "ZzHelper"}, + Arguments: []ast.CallArgument{ + {Name: argName, Value: &ast.LiteralExpr{Kind: ast.LiteralString, Value: "x"}}, + }, + }, + }, + }, + }} +} + +func TestValidate_JavaActionCreatedInScriptIsNotReportedMissing(t *testing.T) { + ctx, _ := newMockCtx(t) // an empty project: no java actions stored + + prog := javaActionScript("Input") + sc := newScriptContext() + sc.collectDefinitions(prog) + + mf := prog.Statements[1].(*ast.CreateMicroflowStmt) + if errs := validateMicroflowReferences(ctx, mf, sc); len(errs) != 0 { + t.Fatalf("reference errors for an action created in the same script: %v", errs) + } +} + +// Exempting the action from the "not found" check must not also exempt it from +// the parameter check — the script declares the parameters, so a typo is still +// catchable and still worth catching (Mendix reports it as CE1613). +func TestValidate_JavaActionCreatedInScriptStillChecksParameterNames(t *testing.T) { + ctx, _ := newMockCtx(t) + + prog := javaActionScript("Inputt") + sc := newScriptContext() + sc.collectDefinitions(prog) + + mf := prog.Statements[1].(*ast.CreateMicroflowStmt) + errs := validateMicroflowReferences(ctx, mf, sc) + if len(errs) != 1 || !strings.Contains(errs[0], `has no parameter "Inputt"`) { + t.Fatalf("errors = %v, want one complaint about the misspelled parameter", errs) + } +} + +// The forward-reference hint reads allNames()/has(); both have to know about the +// new categories or a created action looks "defined later" forever. +func TestScriptContext_KnowsCodeActionsByName(t *testing.T) { + sc := newScriptContext() + sc.collectDefinitions(&ast.Program{Statements: []ast.Statement{ + &ast.CreateJavaActionStmt{Name: ast.QualifiedName{Module: "M", Name: "Ja"}}, + &ast.CreateJavaScriptActionStmt{Name: ast.QualifiedName{Module: "M", Name: "Jsa"}}, + }}) + + for _, name := range []string{"M.Ja", "M.Jsa"} { + if !sc.has(name) { + t.Errorf("has(%q) = false", name) + } + found := false + for _, n := range sc.allNames() { + if n == name { + found = true + } + } + if !found { + t.Errorf("allNames() omits %q", name) + } + } +} From 844ec19d87347e086561b573083d3e79f9fd04da Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 05:25:41 +0000 Subject: [PATCH 06/20] Keep a specialization's inherited associations when reconciling access rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GRANT on a specialization wrote a model Mendix rejects with CE0066 "Entity access is out of date", while the same grant on the generalization checked clean. A specialization has every member of its generalization, associations included, so its access rule needs an entry for each of them; the grant walk collected them, but the reconcile that runs next deleted them again — the executor passed 3 MemberAccess entries and storage held 2. ReconcileMemberAccesses recomputed each rule's expected member set from the entity's own attributes and the module's FROM-side associations, so an inherited association matched neither and was classed stale. It now walks the generalization chain within the module, and preserves a reference qualified with another module the way the attribute branch has since #758 — an association is qualified by the module that declares it, so an ancestor elsewhere (the reported OpenAIDeployedModel extends GenAICommons.DeployedModel) names a domain model that is not loaded here and cannot be validated at all. Reconcile also has to ADD such an association, or a rule written before the ancestor gained one never catches up. Stale detection is unchanged for everything it could already judge, and the TO side of an OWNER Default association still gets no entry — having one is itself CE0066, so a walk that simply collected every association in the module would pass the positive cases and fail there. Measured on a fixture with no marketplace module involved (Base <- Derived, association FROM Base): 1 error -> 0 on mxbuild 11.12.1. Reported in mxcli-chat FINDINGS §25. --- .claude/skills/fix-issue.md | 3 +- .../bug-tests/grant-on-specialization.mdl | 49 ++++ .../modelsdk/domainmodel_security_write.go | 82 +++++- .../reconcile_inherited_assoc_test.go | 238 ++++++++++++++++++ 4 files changed, 368 insertions(+), 4 deletions(-) create mode 100644 mdl-examples/bug-tests/grant-on-specialization.mdl create mode 100644 mdl/backend/modelsdk/reconcile_inherited_assoc_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 550d4526f..9ba0d7af6 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -494,6 +494,7 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli marketplace update`/`diff` fail with `version "6.0.0" not found` on a module the project actually has, and `--force` does not bypass it | Both download the **installed** version to build the local-edit baseline, and that version had been unpublished. A blank 11.13 app ships NanoflowCommons 6.0.0 while the 6.x line now starts at 6.1.1 — so the module most in need of updating is the one whose baseline cannot be built | `cmd/mxcli/cmd_marketplace_update.go` (`--no-baseline`, and the refusal now names it) | **`--force` was never going to work**: it overrides a *finding*, and here the comparison never ran, so there is no finding. Needing a second flag is the signal that these are two different decisions — "I accept losing edits I have been shown" vs "I accept not being shown". **Say what the flag costs in the flag's own message** — with no baseline, local edits go without being named, so the honest instruction is to commit first. Reported in mxcli-chat FINDINGS §15 | | A constant set with `alter settings constant 'M.C' value '…' in configuration 'Default'` has no effect on the running app. The statement succeeds, `describe settings` round-trips it, `mx check` is clean, and the app behaves as if the constant were still at its default | mxbuild writes `deployment/model/config.json` with each constant's **default** value, and that map is what `run --local` hands the standalone runtime as `MicroflowConstants` — the configuration's overrides are not in it, and nothing read them. An app ran for hours with an empty encryption key while the model said otherwise | `cmd/mxcli/runconstants.go` (`resolveConstantOverrides`, `reportConstantOverrides`), `cmd_run.go` (`--configuration`), `cmd/mxcli/docker/localboot.go` (`mergeConstantOverrides`) | **A silent no-op is the worst shape a bug can take** — every layer reported success. The fix therefore prints in *every* case, including "no overrides applied", so silence stops meaning "your value is in effect". **Merge, never replace**: `config.json` carries the defaults for constants the configuration is silent about, and replacing the map drops them (the app 530s on the first microflow that reads one) — which is exactly the shape `--runtime-setting MicroflowConstants={…}` has. **A private override has no value in the model at all** (a `Settings$PrivateValue` marker; the value lives on the workstation), so applying it would blank the constant — skip and name it. **Do not guess between configurations**: with several and no `Default`, applying one silently could push production's API key into a local run. Tests `cmd/mxcli/runconstants_test.go`, `cmd/mxcli/docker/localboot_constants_test.go`; verified live — the run now prints `Applying 1 constant value(s) from configuration "Default"`. Reported in mxcli-chat FINDINGS §33 | | `mxcli marketplace diff` reports local edits on a project nobody has edited — typically an Atlas snippet and a building block. `--save-edits` then writes MDL that would *destroy* the element if replayed (an empty `create or modify snippet X (Folder: 'Web') { }`) | The comparison is on DESCRIBE output, and `Describable()` only asked whether describe *errored*. Some types describe successfully into output that says nothing: an empty `{ }` body, or a building block under "Building blocks are read-only; they cannot be created via MDL". Two such renderings can still differ — one unresolved name is enough — and the difference was reported as a user edit | `cmd/mxcli/marketplace/snapshot.go` (`Element.Conclusive`, `declaredReadOnly`, `emptyBody`), `compare.go` (`classify`), `update.go` (`SaveEdits`) | **Check equality BEFORE conclusiveness** — identical text is solid evidence of "unchanged" whatever the type, and inverting the order marks every Atlas building block unknown and drains `verified` of meaning. **Judge the output, not the type**: a type list goes stale the moment a describe handler improves, while "this text contains nothing to compare" stays true by construction — match the handler's own read-only wording so the two stay in step. **A rescue file must never be a deletion**: the same rule gates `--save-edits`, including `OnlyInstalled` findings, which reach it without passing through `classify`. **Test that a real edit is still Modified** — a guard that returns unknown for everything passes every "not evidence" test. Tests `cmd/mxcli/marketplace/inconclusive_test.go`. Reported in mxcli-chat FINDINGS §16 | -| `GRANT ON Module.Specialization (READ (…, SomeAssociation))` is refused with "entity X has no member(s) SomeAssociation; grant only names members of the entity or of an entity it inherits from" — when the association *is* declared on an entity it inherits from | The member walk resolved inherited **attributes** through the generalization chain (#758) but collected associations only from the entity's own domain model where `ParentID == entity.ID`. An association declared on an ancestor was therefore invisible, which made the rule OpenAIConnector ships impossible to express in MDL (`OpenAIDeployedModel extends GenAICommons.DeployedModel`) | `mdl/executor/cmd_security_write.go` (`inheritedAssociations`, `domainModelHasEntity`) | **Find the declaring domain model by looking for the entity, not by matching module names** — the module-name lookup goes through the hierarchy cache and returns "" often enough that filtering on it silently collected nothing; the first cut of this fix looked right and did nothing. **Qualify the ref against the module that DECLARES the association**, as the attribute walk does. **Test that an unrelated entity does not get the entry** — a walk collecting every association in the module passes the positive tests and puts entries exactly where Mendix reports CE0066 for having them. Tests `mdl/executor/cmd_security_inherited_assoc_test.go`; the mock needs `GetModuleByNameFunc`, without which even own attributes fail to resolve. Reported in mxcli-chat FINDINGS §26. **STILL OPEN — the emitted entry does not reach storage.** Measured on a two-entity fixture (`Derived EXTENDS Base`, association declared on `Base`): the executor now passes 3 MemberAccess entries, the stored rule holds 2, and `GRANT … ON Derived (READ *, WRITE *)` still gives CE0066 while the same rule on `Base` checks clean. The loss is between `EntityAccessRuleParams.MemberAccesses` and the persisted `DomainModels$MemberAccess` list — that is where FINDINGS §25 should be picked up | +| `GRANT ON Module.Specialization (READ (…, SomeAssociation))` is refused with "entity X has no member(s) SomeAssociation; grant only names members of the entity or of an entity it inherits from" — when the association *is* declared on an entity it inherits from | The member walk resolved inherited **attributes** through the generalization chain (#758) but collected associations only from the entity's own domain model where `ParentID == entity.ID`. An association declared on an ancestor was therefore invisible, which made the rule OpenAIConnector ships impossible to express in MDL (`OpenAIDeployedModel extends GenAICommons.DeployedModel`) | `mdl/executor/cmd_security_write.go` (`inheritedAssociations`, `domainModelHasEntity`) | **Find the declaring domain model by looking for the entity, not by matching module names** — the module-name lookup goes through the hierarchy cache and returns "" often enough that filtering on it silently collected nothing; the first cut of this fix looked right and did nothing. **Qualify the ref against the module that DECLARES the association**, as the attribute walk does. **Test that an unrelated entity does not get the entry** — a walk collecting every association in the module passes the positive tests and puts entries exactly where Mendix reports CE0066 for having them. Tests `mdl/executor/cmd_security_inherited_assoc_test.go`; the mock needs `GetModuleByNameFunc`, without which even own attributes fail to resolve. Reported in mxcli-chat FINDINGS §26. (The emitted entry then did not reach storage; that half is the FINDINGS §25 row below.) | | A `call java action` whose parameter is a **microflow** (`MCPServer.AddTool`'s `ExecutingMicroflow`, and every other "register a callback" action) fails `mx check` with **CE0115** — the microflow's name is stored as a literal string | The modelsdk engine's java-action read had no case for the parameter type, so `codeActionBasicFromGen` fell through to its `default` and returned a `StringType`. The microflow builder's `microflowTypeParams` branch was therefore never taken and it authored `Microflows$BasicCodeActionParameterValue{Argument: "'M.MyFlow'"}` instead of `Microflows$MicroflowParameterValue{Microflow: "M.MyFlow"}` | `mdl/backend/modelsdk/java_read.go` (`codeActionBasicFromGen`), `java_write.go` (`codeActionParamTypeToGen`), `mdl/executor/cmd_javaactions.go` (`astDataTypeToJavaActionParamType`, `bareDataTypeName`) | **A `default:` that returns a plausible type is how a read path lies.** The legacy parser and the executor both handled this correctly for years; only the gen→semantic converter did not, and it degraded to String rather than erroring — so every layer downstream looked correct in isolation. **Fix read and write together**: without the write case an *update* of such an action rewrites the parameter as a String, which is worse than the read bug. **Watch what DESCRIBE starts printing** — the fix made it emit the honest `Microflow`, which then round-tripped into an entity type with an empty module (`.Microflow`) until `astDataTypeToJavaActionParamType` learned the bare word. Probe the stored `$Type` from the marketplace `.mpk` before theorising; MCP Server 5.1.0 stores `JavaActions$MicroflowJavaActionParameterType`. Tests `mdl/backend/modelsdk/java_read_test.go`, `mdl/executor/cmd_javaactions_test.go`, example `mdl-examples/bug-tests/javaaction-microflow-parameter.mdl`; verified 0 errors under `mx check` 11.12.1. Reported in mxcli-chat FINDINGS §36 | | `mxcli check --references` reports "java action not found: M.X (referenced by call java action)" for an action the **same script** creates a few statements earlier. Entities, microflows, pages and nanoflows in the same position are accepted | `scriptContext` — the set of objects a script defines, consulted so a script can be checked against its own output — had categories for modules, entities, enumerations, microflows, nanoflows, pages and snippets, but not for java or JavaScript actions. The java-action branch of `validateFlowBodyReferences` went straight to the project lookup | `mdl/executor/validate.go` (`scriptContext.javaActions`/`javaScriptActions`, `collectDefinitions`, `collectSingle`, `allNames`, `has`, `validateFlowBodyReferences`) | **Store the declared parameter names, not a bool.** Exempting the action from "not found" must not also exempt it from the parameter-name check — the script declares the parameters, so a typo is still catchable, and a bool would have silently dropped that. **`allNames()` and `has()` move together**: `annotateForwardRef` reads both, so adding a category to one and not the other makes a *created* object look "defined later in this script" forever. Tests `mdl/executor/validate_script_javaactions_test.go` (including the control that a misspelled parameter still errors). Reported in mxcli-chat FINDINGS §37 | +| `GRANT ON Module.Specialization (READ *, WRITE *)` writes a model Mendix rejects with **CE0066** "Entity access is out of date", while the same grant on the generalization checks clean. The executor passes 3 MemberAccess entries and storage holds 2 | `ReconcileMemberAccesses` — which runs after every program and after every `create association` — recomputes each rule's expected member set from the entity's **own** attributes and the module's **FROM-side** associations. An inherited association matched neither, so it was classed stale and deleted, one step after the GRANT that had just written it correctly | `mdl/backend/modelsdk/domainmodel_security_write.go` (`sameModuleAncestors`, `entitiesByName`, `assocRefBelongsTo`, `ownerIDs` in `ReconcileMemberAccesses`), `mdl/executor/cmd_associations.go` (reconcile after DROP) | **The write path was innocent** — `AddEntityAccessRule` stored all 3; the deletion happened in the reconcile that ran next. When a value is written and then absent, instrument the *later* pass before the writer. **Preserve what cannot be checked**, as the attribute branch already did (#758): an association is qualified by the module that DECLARES it, so an ancestor in another module (the reported `OpenAIDeployedModel extends GenAICommons.DeployedModel`) names a domain model that is not loaded here. **The counter-control is the whole test**: a walk that collected every association in the module passes both positive cases and puts an entry on the TO side of an `OWNER Default` association, which is *itself* CE0066 — so assert that the unrelated entity does **not** get one. **Reconcile must add as well as keep**, or a rule written before the ancestor gained the association never catches up. Verifying this surfaced a neighbouring bug: `drop association` never reconciled at all, leaving entries Mendix rejects with CE1613 — on the declaring entity too, so it predates inheritance support. Tests `mdl/backend/modelsdk/reconcile_inherited_assoc_test.go` (5 cases incl. both controls), `mdl/executor/cmd_associations_mock_test.go`; example `mdl-examples/bug-tests/grant-on-specialization.mdl`, measured 1 error → 0 on mxbuild 11.12.1. Reported in mxcli-chat FINDINGS §25 | diff --git a/mdl-examples/bug-tests/grant-on-specialization.mdl b/mdl-examples/bug-tests/grant-on-specialization.mdl new file mode 100644 index 000000000..628788c2e --- /dev/null +++ b/mdl-examples/bug-tests/grant-on-specialization.mdl @@ -0,0 +1,49 @@ +-- ============================================================================ +-- mxcli-chat FINDINGS §25 — GRANT on a specialization produced CE0066 +-- +-- A specialization has every member of its generalization, associations +-- included, so its access rule needs an entry for each of them. The executor's +-- grant walk collects them, but ReconcileMemberAccesses — which runs after +-- every program and after every new association — recomputed the expected set +-- from the entity's OWN attributes and the module's FROM-side associations +-- only. The inherited association was therefore "stale" and deleted, one step +-- after the GRANT that had just written it, and Mendix reported +-- +-- CE0066 "Entity access is out of date." at Domain model of module 'Spec25' +-- +-- The reported case was OpenAIConnector's `OpenAIDeployedModel extends +-- GenAICommons.DeployedModel`; this fixture reproduces it with no marketplace +-- module at all, which is what showed the rule was wrong for ANY specialization. +-- +-- Verify: +-- mxcli exec mdl-examples/bug-tests/grant-on-specialization.mdl -p app.mpr +-- mx check app.mpr # 0 errors +-- mxcli -p app.mpr -c "describe entity Spec25.Derived" +-- +-- The three GRANTs together are the measurement: +-- Base — the control that checked clean before the fix +-- Derived — the failing case: must carry Base.Code AND Spec25.Base_Other +-- Other — the counter-control: the TO side of an OWNER Default association +-- must NOT gain an entry (Mendix reports CE0066 for having one), +-- so a walk that simply collected every association in the module +-- would pass the first two and fail here +-- ============================================================================ + +create module Spec25; +/ + +create persistent entity Spec25.Base ( Code: String ); +create persistent entity Spec25.Other ( Label: String ); +create persistent entity Spec25.Derived extends Spec25.Base ( Extra: String ); +/ + +create association Spec25.Base_Other from Spec25.Base to Spec25.Other; +/ + +create module role Spec25.User; +/ + +grant Spec25.User on Spec25.Base (read *, write *); +grant Spec25.User on Spec25.Derived (read *, write *); +grant Spec25.User on Spec25.Other (read *, write *); +/ diff --git a/mdl/backend/modelsdk/domainmodel_security_write.go b/mdl/backend/modelsdk/domainmodel_security_write.go index 790aff92a..c5bbee57b 100644 --- a/mdl/backend/modelsdk/domainmodel_security_write.go +++ b/mdl/backend/modelsdk/domainmodel_security_write.go @@ -313,6 +313,8 @@ func (b *Backend) ReconcileMemberAccesses(unitID model.ID, moduleName string) (i return 0, err } + byName := entitiesByName(dm) + modified := 0 for _, el := range dm.EntitiesItems() { ent, ok := el.(*genDm.Entity) @@ -325,6 +327,16 @@ func (b *Backend) ReconcileMemberAccesses(unitID model.ID, moduleName string) (i continue } + // A specialization has every member of its generalization, associations + // included, so its access rule needs an entry for each of them. Only the + // part of the chain that lives in THIS module can be walked; an ancestor + // in another module is handled by preserving its references below rather + // than by resolving them. + ownerIDs := map[string]bool{entityID: true} + for _, anc := range sameModuleAncestors(ent, byName, moduleName) { + ownerIDs[string(anc.ID())] = true + } + // Attributes (in order) with calculated flags. type attrInfo struct { qn string @@ -372,13 +384,13 @@ func (b *Backend) ReconcileMemberAccesses(unitID model.ID, moduleName string) (i // Mendix reports as CE0066 "Entity access is out of date" // (issuetracker #20). Verified on mxbuild 11.12.1: the same model with // `OWNER Default` checks clean, so the owner mode is the trigger. - if string(a.ParentRefID()) == entityID || - (a.Owner() == "Both" && string(a.ChildRefID()) == entityID) { + if ownerIDs[string(a.ParentRefID())] || + (a.Owner() == "Both" && ownerIDs[string(a.ChildRefID())]) { addAssoc(a.Name()) } } for _, ce := range dm.CrossAssociationsItems() { - if ca, ok := ce.(*genDm.CrossAssociation); ok && string(ca.ParentRefID()) == entityID { + if ca, ok := ce.(*genDm.CrossAssociation); ok && ownerIDs[string(ca.ParentRefID())] { addAssoc(ca.Name()) } } @@ -464,7 +476,18 @@ func (b *Backend) ReconcileMemberAccesses(unitID model.ID, moduleName string) (i covSys[assocRef] = true case assocSet[assocRef]: covAssoc[assocRef] = true + case !assocRefBelongsTo(assocRef, moduleName): + // An association is qualified by the module that DECLARES it, so + // one inherited from a generalization in another module names + // that module. Its domain model is not loaded here, so the + // reference cannot be validated at all — preserve it, as the + // attribute branch above does, rather than delete what cannot be + // checked. Without this, every rule on a specialization of a + // marketplace entity lost its inherited associations. + covAssoc[assocRef] = true default: + // Genuinely stale: an association of this module that no longer + // has this entity (or an ancestor of it) on its FROM side. rule.RemoveMemberAccesses(i) changed = true } @@ -524,6 +547,59 @@ func newMemberAccess(rights, qualifiedName string, isAttr bool) *genDm.MemberAcc return ma } +// entitiesByName indexes a domain model's entities by name, for resolving a +// generalization's qualified name within the same module. +func entitiesByName(dm *genDm.DomainModel) map[string]*genDm.Entity { + out := map[string]*genDm.Entity{} + for _, el := range dm.EntitiesItems() { + if e, ok := el.(*genDm.Entity); ok && e.Name() != "" { + out[e.Name()] = e + } + } + return out +} + +// sameModuleAncestors walks an entity's generalization chain and returns the +// ancestors that live in this module, nearest first. +// +// The walk stops at the first ancestor it cannot resolve — one in another module +// (or in System). That is not a failure: an association declared by such an +// ancestor is qualified with the ancestor's module, so it is recognised by +// assocRefBelongsTo rather than by being found here. +func sameModuleAncestors(ent *genDm.Entity, byName map[string]*genDm.Entity, moduleName string) []*genDm.Entity { + var out []*genDm.Entity + seen := map[string]bool{string(ent.ID()): true} + for cur := ent; ; { + gen, ok := cur.Generalization().(*genDm.Generalization) + if !ok { + return out + } + qn := gen.GeneralizationQualifiedName() + idx := strings.LastIndex(qn, ".") + if idx < 0 || !strings.EqualFold(qn[:idx], moduleName) { + return out + } + anc, found := byName[qn[idx+1:]] + if !found || seen[string(anc.ID())] { + return out // unresolvable, or a cycle a corrupt model could contain + } + seen[string(anc.ID())] = true + out = append(out, anc) + cur = anc + } +} + +// assocRefBelongsTo reports whether a MemberAccess association reference +// ("Module.Association") is declared in the given module, and so can be checked +// against the domain model at hand. +func assocRefBelongsTo(assocRef, moduleName string) bool { + idx := strings.LastIndex(assocRef, ".") + if idx < 0 { + return false + } + return strings.EqualFold(assocRef[:idx], moduleName) +} + // attrRefBelongsTo reports whether a MemberAccess attribute reference // ("Module.Entity.Attribute") names one of the given entity's OWN attributes, // rather than one inherited from an ancestor. diff --git a/mdl/backend/modelsdk/reconcile_inherited_assoc_test.go b/mdl/backend/modelsdk/reconcile_inherited_assoc_test.go new file mode 100644 index 000000000..5f2ac8312 --- /dev/null +++ b/mdl/backend/modelsdk/reconcile_inherited_assoc_test.go @@ -0,0 +1,238 @@ +// SPDX-License-Identifier: Apache-2.0 + +// mxcli-chat FINDINGS §25: `GRANT ON Module.Specialization (READ *, WRITE *)` +// produced a model Mendix rejects with CE0066 "Entity access is out of date", +// while the same grant on the generalization checked clean. +// +// A specialization has every member of its generalization, associations +// included. The executor's grant walk learned to collect them (#26), but +// ReconcileMemberAccesses — which runs after every program and after every new +// association — recomputed the expected member set from the entity's OWN +// attributes and the module's FROM-side associations only. An inherited +// association was therefore "stale" and deleted, immediately after the grant +// that had just written it. Measured: the executor passed 3 entries, storage +// held 2. +package modelsdkbackend + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/domainmodel" +) + +// inheritanceFixture builds Base <- Derived plus an unrelated Other, with +// Base_Other declared FROM Base, and returns the connected backend + module. +func inheritanceFixture(t *testing.T) (*Backend, *model.Module, *domainmodel.DomainModel) { + t.Helper() + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + dm, err := b.GetDomainModel(mod.ID) + if err != nil { + t.Fatalf("GetDomainModel: %v", err) + } + + base := &domainmodel.Entity{Name: "ZzBase", Persistable: true, + Attributes: []*domainmodel.Attribute{{Name: "Code", Type: &domainmodel.StringAttributeType{}}}} + other := &domainmodel.Entity{Name: "ZzOther", Persistable: true, + Attributes: []*domainmodel.Attribute{{Name: "Label", Type: &domainmodel.StringAttributeType{}}}} + for _, e := range []*domainmodel.Entity{base, other} { + if err := b.CreateEntity(dm.ID, e); err != nil { + t.Fatalf("CreateEntity %s: %v", e.Name, err) + } + } + derived := &domainmodel.Entity{Name: "ZzDerived", Persistable: true, + GeneralizationRef: "MyFirstModule.ZzBase", + Attributes: []*domainmodel.Attribute{{Name: "Extra", Type: &domainmodel.StringAttributeType{}}}} + if err := b.CreateEntity(dm.ID, derived); err != nil { + t.Fatalf("CreateEntity ZzDerived: %v", err) + } + // CreateEntity does not write the minted ID back onto the struct, so the + // association's endpoints have to be read out of the stored model. + ids := entityIDs(t, b, mod.ID) + if err := b.CreateAssociation(dm.ID, &domainmodel.Association{ + Name: "ZzBase_ZzOther", ParentID: ids["ZzBase"], ChildID: ids["ZzOther"], + Type: "Reference", Owner: "Default", + }); err != nil { + t.Fatalf("CreateAssociation: %v", err) + } + return b, mod, dm +} + +// entityIDs maps entity name -> stored ID for a module's domain model. +func entityIDs(t *testing.T, b *Backend, modID model.ID) map[string]model.ID { + t.Helper() + dm, err := b.GetDomainModel(modID) + if err != nil { + t.Fatalf("GetDomainModel: %v", err) + } + out := map[string]model.ID{} + for _, e := range dm.Entities { + out[e.Name] = e.ID + } + return out +} + +// memberRefs returns an entity's first access rule's member references, in +// storage order, as "attr:X" / "assoc:X". +func memberRefs(t *testing.T, b *Backend, modID model.ID, entityName string) []string { + t.Helper() + dm, err := b.GetDomainModel(modID) + if err != nil { + t.Fatalf("GetDomainModel: %v", err) + } + for _, e := range dm.Entities { + if e.Name != entityName { + continue + } + if len(e.AccessRules) == 0 { + return nil + } + var out []string + for _, ma := range e.AccessRules[0].MemberAccesses { + switch { + case ma.AttributeName != "": + out = append(out, "attr:"+ma.AttributeName) + case ma.AssociationName != "": + out = append(out, "assoc:"+ma.AssociationName) + } + } + return out + } + t.Fatalf("entity %s not found", entityName) + return nil +} + +func hasRef(refs []string, want string) bool { + for _, r := range refs { + if r == want { + return true + } + } + return false +} + +func grantAll(t *testing.T, b *Backend, dmID model.ID, entityName string, members []types.EntityMemberAccess) { + t.Helper() + if err := b.AddEntityAccessRule(backend.EntityAccessRuleParams{ + UnitID: dmID, EntityName: entityName, + RoleNames: []string{"MyFirstModule.User"}, + DefaultMemberAccess: "ReadWrite", + MemberAccesses: members, + }); err != nil { + t.Fatalf("AddEntityAccessRule %s: %v", entityName, err) + } +} + +func TestReconcile_KeepsAssociationInheritedFromAGeneralization(t *testing.T) { + b, mod, dm := inheritanceFixture(t) + + // What the executor's grant walk writes for `READ *, WRITE *` on the + // specialization: own attribute, inherited attribute, inherited association. + grantAll(t, b, dm.ID, "ZzDerived", []types.EntityMemberAccess{ + {AttributeRef: "MyFirstModule.ZzDerived.Extra", AccessRights: "ReadWrite"}, + {AttributeRef: "MyFirstModule.ZzBase.Code", AccessRights: "ReadWrite"}, + {AssociationRef: "MyFirstModule.ZzBase_ZzOther", AccessRights: "ReadWrite"}, + }) + + if _, err := b.ReconcileMemberAccesses(dm.ID, mod.Name); err != nil { + t.Fatalf("ReconcileMemberAccesses: %v", err) + } + + refs := memberRefs(t, b, mod.ID, "ZzDerived") + if !hasRef(refs, "assoc:MyFirstModule.ZzBase_ZzOther") { + t.Fatalf("reconcile dropped the inherited association: %v — this is the CE0066", refs) + } + if !hasRef(refs, "attr:MyFirstModule.ZzBase.Code") { + t.Errorf("reconcile dropped the inherited attribute (#758 regression): %v", refs) + } +} + +// Reconcile also has to ADD it: a rule written before the ancestor gained the +// association must pick it up, which is what Studio Pro's "Update security" +// button does and what `create association` triggers. +func TestReconcile_AddsAncestorAssociationToASpecializationsRule(t *testing.T) { + b, mod, dm := inheritanceFixture(t) + + grantAll(t, b, dm.ID, "ZzDerived", []types.EntityMemberAccess{ + {AttributeRef: "MyFirstModule.ZzDerived.Extra", AccessRights: "ReadWrite"}, + }) + + if _, err := b.ReconcileMemberAccesses(dm.ID, mod.Name); err != nil { + t.Fatalf("ReconcileMemberAccesses: %v", err) + } + + if refs := memberRefs(t, b, mod.ID, "ZzDerived"); !hasRef(refs, "assoc:MyFirstModule.ZzBase_ZzOther") { + t.Fatalf("reconcile did not add the ancestor's association: %v", refs) + } +} + +// The control that a walk collecting every association in the module would fail: +// ZzOther is the TO side of an `OWNER Default` association and is not related to +// ZzBase, so it must NOT gain an entry. Mendix reports CE0066 for having one. +func TestReconcile_DoesNotGiveAnUnrelatedEntityTheAssociation(t *testing.T) { + b, mod, dm := inheritanceFixture(t) + + grantAll(t, b, dm.ID, "ZzOther", []types.EntityMemberAccess{ + {AttributeRef: "MyFirstModule.ZzOther.Label", AccessRights: "ReadWrite"}, + }) + + if _, err := b.ReconcileMemberAccesses(dm.ID, mod.Name); err != nil { + t.Fatalf("ReconcileMemberAccesses: %v", err) + } + + if refs := memberRefs(t, b, mod.ID, "ZzOther"); hasRef(refs, "assoc:MyFirstModule.ZzBase_ZzOther") { + t.Fatalf("the TO side of an OWNER Default association got a member entry: %v", refs) + } +} + +// Preserving inherited references must not blunt stale detection: an association +// of THIS module that no longer exists is still removed, on the specialization +// as well as on the entity that declared it (Mendix: CE1613). +func TestReconcile_StillDropsAnAssociationThatNoLongerExists(t *testing.T) { + b, mod, dm := inheritanceFixture(t) + + grantAll(t, b, dm.ID, "ZzDerived", []types.EntityMemberAccess{ + {AttributeRef: "MyFirstModule.ZzDerived.Extra", AccessRights: "ReadWrite"}, + {AssociationRef: "MyFirstModule.ZzBase_ZzGone", AccessRights: "ReadWrite"}, + }) + + if _, err := b.ReconcileMemberAccesses(dm.ID, mod.Name); err != nil { + t.Fatalf("ReconcileMemberAccesses: %v", err) + } + + if refs := memberRefs(t, b, mod.ID, "ZzDerived"); hasRef(refs, "assoc:MyFirstModule.ZzBase_ZzGone") { + t.Fatalf("a deleted association was preserved: %v", refs) + } +} + +// An ancestor in another module cannot be resolved from this domain model, so +// its associations are preserved rather than validated — the same "preserve what +// cannot be checked" rule the attribute branch uses (#758). +func TestReconcile_PreservesAnAssociationFromAnotherModule(t *testing.T) { + b, mod, dm := inheritanceFixture(t) + + grantAll(t, b, dm.ID, "ZzDerived", []types.EntityMemberAccess{ + {AttributeRef: "MyFirstModule.ZzDerived.Extra", AccessRights: "ReadWrite"}, + {AssociationRef: "GenAICommons.DeployedModel_Provider", AccessRights: "ReadWrite"}, + }) + + if _, err := b.ReconcileMemberAccesses(dm.ID, mod.Name); err != nil { + t.Fatalf("ReconcileMemberAccesses: %v", err) + } + + if refs := memberRefs(t, b, mod.ID, "ZzDerived"); !hasRef(refs, "assoc:GenAICommons.DeployedModel_Provider") { + t.Fatalf("a reference this domain model cannot validate was deleted: %v", refs) + } +} From 1d2d7f8094c102606ae38eb0352aa44e5923cf9f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 05:25:41 +0000 Subject: [PATCH 07/20] Reconcile access rules after dropping an association MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Creating an association reconciles every access rule in the module so the new member gets an entry; dropping one did nothing, so every MemberAccess that named it stayed behind and Mendix rejected the model with CE1613 "The selected association 'X' no longer exists". This is independent of inheritance — it hit the entity that declared the association just as hard — but it only became visible while verifying the specialization fix, because until then a specialization never carried the entry to begin with. DROP now runs the same reconcile CREATE does, and tracks the domain model as modified so the finalize step sees it. Measured: drop-then-check went from 2 errors to 0 on mxbuild 11.12.1. --- mdl/executor/cmd_associations.go | 17 +++++++++ mdl/executor/cmd_associations_mock_test.go | 40 ++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/mdl/executor/cmd_associations.go b/mdl/executor/cmd_associations.go index 55e7b2834..ebec16d4d 100644 --- a/mdl/executor/cmd_associations.go +++ b/mdl/executor/cmd_associations.go @@ -315,11 +315,27 @@ func execDropAssociation(ctx *ExecContext, s *ast.DropAssociationStmt) error { return mdlerrors.NewBackend("get domain model", err) } + // Dropping an association leaves a MemberAccess entry behind on every access + // rule that named it, which Mendix rejects with CE1613 "The selected + // association … no longer exists". Creating one already reconciles; dropping + // one has to as well, or the drop produces a model that will not build. + afterDrop := func() { + invalidateHierarchy(ctx) + invalidateDomainModelsCache(ctx) + if freshDM, err := ctx.Backend.GetDomainModel(module.ID); err == nil { + if count, err := ctx.Backend.ReconcileMemberAccesses(freshDM.ID, module.Name); err == nil && count > 0 { + fmt.Fprintf(ctx.Output, "Reconciled %d access rule(s) after dropping the association\n", count) + } + } + ctx.trackModifiedDomainModel(module.ID, module.Name) + } + for _, assoc := range dm.Associations { if assoc.Name == s.Name.Name { if err := ctx.Backend.DeleteAssociation(dm.ID, assoc.ID); err != nil { return mdlerrors.NewBackend("delete association", err) } + afterDrop() fmt.Fprintf(ctx.Output, "Dropped association: %s\n", s.Name) return nil } @@ -329,6 +345,7 @@ func execDropAssociation(ctx *ExecContext, s *ast.DropAssociationStmt) error { if err := ctx.Backend.DeleteCrossAssociation(dm.ID, ca.ID); err != nil { return mdlerrors.NewBackend("delete cross-module association", err) } + afterDrop() fmt.Fprintf(ctx.Output, "Dropped cross-module association: %s\n", s.Name) return nil } diff --git a/mdl/executor/cmd_associations_mock_test.go b/mdl/executor/cmd_associations_mock_test.go index 78919ce3f..d20af8b53 100644 --- a/mdl/executor/cmd_associations_mock_test.go +++ b/mdl/executor/cmd_associations_mock_test.go @@ -4,6 +4,7 @@ package executor import ( "fmt" + "strings" "testing" "github.com/mendixlabs/mxcli/mdl/ast" @@ -254,3 +255,42 @@ func TestCreateAssociation_AlreadyExists_NoOrModify(t *testing.T) { }) assertError(t, err) } + +// Creating an association reconciles every access rule in the module; dropping +// one did not, so the MemberAccess entries that named it stayed behind and +// Mendix rejected the model with CE1613 "The selected association 'X' no longer +// exists" — on the entity that declared it and on every specialization of it. +// Found while verifying mxcli-chat FINDINGS §25. +func TestExecDropAssociation_ReconcilesAccessRules(t *testing.T) { + mod := mkModule("MyModule") + ent1 := mkEntity(mod.ID, "Order") + ent2 := mkEntity(mod.ID, "Customer") + assoc := mkAssociation(mod.ID, "Order_Customer", ent1.ID, ent2.ID) + dm := mkDomainModel(mod.ID, ent1, ent2) + dm.Associations = []*domainmodel.Association{assoc} + + reconciled := "" + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetDomainModelFunc: func(model.ID) (*domainmodel.DomainModel, error) { return dm, nil }, + DeleteAssociationFunc: func(model.ID, model.ID) error { return nil }, + ReconcileMemberAccessesFunc: func(_ model.ID, moduleName string) (int, error) { + reconciled = moduleName + return 1, nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb)) + if err := execDropAssociation(ctx, &ast.DropAssociationStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Order_Customer"}, + }); err != nil { + t.Fatalf("execDropAssociation: %v", err) + } + if reconciled != "MyModule" { + t.Fatalf("ReconcileMemberAccesses was not called for the module (got %q) — stale entries stay and the app fails CE1613", reconciled) + } + if !strings.Contains(buf.String(), "Reconciled 1 access rule(s)") { + t.Errorf("output does not report the reconcile: %q", buf.String()) + } +} From c766a76c6fbdaa63daf17d51ad74f4362752bcb8 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 06:36:34 +0000 Subject: [PATCH 08/20] Correct what update_configuration does, measured against a live runtime MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The comment claimed the admin update_configuration action "REPLACES rather than merges". Measured on 11.12.1 against a running standalone runtime, that is not what the runtime does with MicroflowConstants: a payload carrying one constant left every omitted constant resolving to its deployment default, and the database kept working. What is true is narrower, and is the actual reason to fold settings into one boot call: the Go map below overwrites by key, so --runtime-setting MicroflowConstants=... replaces the map mxcli just built, and at boot there is no prior configuration to fall back on for BasePath/DatabaseName (neither is in config.json). Also records two findings for anyone tempted to drive this API live: the call is staged, not applied — the running app keeps the old value until the next reload_model while still answering result:0 — and there is no read-back to merge against (get_configuration, get_current_configuration, runtime_config and get_current_runtime_status are all "Action not found"). Comment only; no behaviour change. --- cmd/mxcli/docker/localboot.go | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index ab583a97a..2b160ddd2 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -290,10 +290,25 @@ func runtimeConfigParams(o LocalRuntimeOptions, constants map[string]string) map params["ApplicationRootUrl"] = o.ApplicationRootUrl } // Overlay extra runtime settings (e.g. Metrics.Registries, - // OpenTelemetry._RuntimeSpanFilters) into this SAME payload. The admin - // update_configuration action REPLACES rather than merges and has no - // read-back, so merging here — into mxcli's single boot call — is the only - // safe way to add settings without clobbering the DB/BasePath config. + // OpenTelemetry._RuntimeSpanFilters) into this SAME payload, because the + // map below overwrites by key: a caller passing + // `--runtime-setting MicroflowConstants=…` replaces the constants map built + // above rather than adding to it, and at boot there is nothing to fall back + // on for BasePath/DatabaseName (they are not in config.json). Folding + // everything into mxcli's single boot call is what keeps that safe. + // + // The admin action also has no read-back — get_configuration, + // get_current_configuration, runtime_config and get_current_runtime_status + // are all "Action not found" on 11.12.1 — so a caller cannot merge by + // reading first. + // + // Measured on 11.12.1, for anyone tempted to drive this API live: the + // runtime treats an update_configuration MicroflowConstants map as an + // overlay on the deployment defaults (constants omitted from the map still + // resolve), and the call is staged rather than applied — the running app + // keeps the old value until the next reload_model, while still answering + // result:0. A "set a constant on a running app" feature is therefore + // update_configuration + reload_model, verified by observation. for k, v := range o.RuntimeSettings { params[k] = v } From 5fd3b4352b3efb1fb0f9a340b6740200afa48b7f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 06:45:54 +0000 Subject: [PATCH 09/20] Propose a precedence chain for constant values A Mendix constant has a value in four possible places, mxcli can write two of them, and only one of those two reaches a running app. The proposal defines one chain -- --constant, a per-machine gitignored store, the configuration's shared overrides, the default -- applied identically by run --local, test --local and test --attach. It carries three findings measured against a live 11.12.1 runtime, in an appendix so the design can be checked against them: update_configuration is staged rather than applied (the app keeps the old value until the next reload_model, while answering result:0), the runtime treats the payload's constants as an overlay on the deployment defaults rather than a replacement, and the admin API has no read-back to merge against. It also records why Mendix's own private-constant slot cannot serve a headless agent -- per-user encrypted, off-model, Studio Pro only -- which is the reason mxcli needs a store of its own rather than a way to write that one. Four slices, smallest first: the test --local divergence is a bug and ships alone. --- docs/11-proposals/PROPOSAL_constant_values.md | 330 ++++++++++++++++++ 1 file changed, 330 insertions(+) create mode 100644 docs/11-proposals/PROPOSAL_constant_values.md diff --git a/docs/11-proposals/PROPOSAL_constant_values.md b/docs/11-proposals/PROPOSAL_constant_values.md new file mode 100644 index 000000000..265bb9e7c --- /dev/null +++ b/docs/11-proposals/PROPOSAL_constant_values.md @@ -0,0 +1,330 @@ +--- +title: Constant values — one precedence chain, and a slot for secrets +status: proposed +date: 2026-08-13 +--- + +# Proposal: Constant values — one precedence chain, and a slot for secrets + +**Status:** Proposed +**Date:** 2026-08-13 + +A Mendix constant has a value in four possible places, mxcli can write two of +them, and only one of those two reaches a running app. This proposal defines a +single precedence chain across every way mxcli boots an app, adds the missing +slot for a value that must not reach git, and closes a divergence where +`mxcli test --local` runs against different constant values than +`mxcli run --local`. + +Everything measured below was measured on **Mendix 11.12.1**, against a live +standalone runtime. The measurements are in [Appendix A](#appendix-a--measurements). + +## Problem Statement + +### 1. A configuration override used to reach nothing (fixed, and the shape to avoid) + +`alter settings constant 'M.C' value 'x' in configuration 'Default'` executed, +reported success, and round-tripped through `describe settings` — and the app ran +with the constant's *default*, because mxbuild writes each constant's default +into `deployment/model/config.json` and that map is what the runtime is handed. +An app ran for hours with an empty encryption key while the model said otherwise +(mxcli-chat FINDINGS §33). + +That is fixed: `run --local` now resolves a configuration's shared constant +values and merges them over the defaults at boot, and prints what it applied in +every case including "nothing". The failure is worth restating because **every +remaining gap below has the same shape**: a layer reports success and the value +silently does not arrive. + +### 2. `mxcli test --local` and `mxcli test --attach` disagree + +`LocalAppOptions` — the headless boot used by both test runners — has no +`ConstantOverrides` field, so `test --local` boots with defaults only. +`test --attach` runs against an app booted by `run --local`, which *does* apply +the configuration's values. The same suite therefore sees different constants +depending on a flag that is documented as an optimisation ("no boot needed"), +not as a semantic change. Nothing errors; a test that depends on a constant just +quietly asserts against a different value. + +### 3. There is no way to set a value for one run + +Every route mxcli offers writes to the model, and therefore to git. For an API +key that is backwards. The only workaround is `alter settings constant …`, run, +then revert — and a forgotten revert commits the key. + +`--runtime-setting MicroflowConstants={…}` looks like an escape hatch and is +not one: `RuntimeSettings` is applied *after* the constants map in +`runtimeConfigParams`, so it replaces the map mxcli built rather than adding to +it, and at boot there is nothing to fall back on for `BasePath`/`DatabaseName` +(neither is in `config.json`). + +### 4. Mendix's own secret slot is unreachable headlessly + +Mendix 10.9+ stores a **private** configuration value encrypted on the local +machine, readable only by that user account +([Configurations Tab](https://docs.mendix.com/refguide/configurations-tab/)). +The docs are explicit that a *shared* value "is stored as part of the app", so +committing it shares it with everyone who can read the repository, and that +relying on a constant's default or on shared configuration settings is unsafe +for exactly that reason +([App Setup Best Practices](https://docs.mendix.com/refguide/app-setup-best-practices/)). + +So the right home for a secret exists — **but only where Studio Pro runs.** It +is per-user encrypted, off-model, and Windows/Mac. In a Linux devcontainer or a +Claude Code session there is no Studio Pro and no such store: nothing can write +it and nothing can read it. The only headless reader is +`mxbuild --export-secrets`, whose own help scopes it to +`target=portable-app-package` — i.e. reading secrets a Studio Pro user already +wrote on that machine. + +**For a headless run the sensitive-settings slot is simply empty.** Defaults and +shared values both go to git; the one safe slot is off-platform. mxcli's current +refusal to write a private override is therefore correct, and for a stronger +reason than the one in the code comment ("the value is not in the model"). + +## Non-goals + +- **Writing Mendix private values.** Out of reach (§4 above), and attempting it + would mean reimplementing a per-user encryption scheme mxcli cannot verify. + mxcli mirrors the *concept* with its own store instead, and says so. +- **Changing what `alter settings constant` does.** Shared per-configuration + overrides stay exactly as they are; they remain the right place for a value + the team *should* share. +- **A secrets manager.** No Vault/KMS integration, no encryption at rest beyond + file permissions. The local store is a gitignored 0600 file — the same bar as + `~/.mxcli/auth.json`, and honestly labelled. + +## Proposed precedence chain + +Highest first. Each layer is one sentence you can hold in your head. + +| # | Layer | Set with | Lives in | In git? | +|---|-------|----------|----------|---------| +| 1 | **This run** | `--constant Module.Name=value` | nothing — the process only | no | +| 2 | **This machine** | `mxcli constant set Module.Name 'value' --local` | `/.mxcli/constants.json` (0600) | **no** (already gitignored) | +| 3 | **This configuration** | `alter settings constant 'Module.Name' value '…' in configuration 'X'` | the model | yes | +| 4 | **Default** | `create [or modify] constant Module.Name … default '…'` | the model | yes | + +Layer 2 is mxcli's answer to §4: same semantics as a Mendix private value +(per-machine, not shared, for secrets), different mechanism, and named as +mxcli's own rather than pretending to be Mendix's. `mxcli init` already writes +`.mxcli/` into the project `.gitignore`, so the home exists and is out of version +control by construction. + +The chain applies **identically** to `run --local`, `test --local` and +`test --attach`, which is what fixes §2. + +### Reporting + +`run --local` already prints what it applied. It gains the layer each value came +from, and a `mxcli constant list` shows the resolved view: + +``` +$ mxcli constant list -p app.mpr --configuration Default +CONSTANT VALUE FROM +MyModule.ApiKey **** machine (.mxcli/constants.json) +MyModule.ServiceUrl https://… configuration "Default" +MyModule.Retries 3 default +Encryption.EncryptionKey (private) Studio Pro — not in the model, default used +``` + +A layer-2 value is masked by default (`--show-values` to print it), because the +whole point of the layer is that it holds things you would not paste into a +terminal transcript. + +## The live path — `update_configuration` + `reload_model` + +Injecting a constant into an *already running* app is buildable, and is two +calls, not one. Measured (Appendix A): + +- `update_configuration` is **staged, not applied**: the running app keeps the + old value until the next `reload_model`, while answering `result:0`. +- The runtime treats the payload's `MicroflowConstants` as an **overlay on the + deployment defaults** — constants omitted from the map still resolve. +- There is **no read-back**: `get_configuration`, `get_current_configuration`, + `runtime_config` and `get_current_runtime_status` are all *"Action not found"*. + +So: + +``` +mxcli constant set MyModule.ApiKey 'sk-…' --local --apply +``` + +writes layer 2 **and** applies it to a running dev loop, as +`update_configuration` (full payload, constants overlaid) → `reload_model` → +**verify by observation**. Verification is not optional: the admin API returned +success for the call that changed nothing, which is precisely the §33 shape. + +This only works where mxcli owns the boot payload (`run --local`), because the +configuration cannot be read back to merge against. Against an app mxcli did not +boot, `--apply` refuses rather than sending a partial payload. + +## BSON / storage + +No new document type, and no new BSON. The model side is already implemented: + +| Element | `$Type` | mxcli | +|---------|---------|-------| +| Constant | `Constants$Constant` | read + write (`create constant`, default value) | +| Per-configuration value | `Settings$ConstantValue` | read + write | +| Shared value | `Settings$SharedValue` (nested, carries `Value`) | read + write | +| Private value | `Settings$PrivateValue` (nested, **no properties at all**) | read (as a marker); **write refused** | + +The `SharedValue` / `PrivateValue` distinction is the polymorphic-child trap +already recorded in CLAUDE.md: the variants differ in *arity*, not just field +values, so a write must dispatch on `$Type` before assigning `Value` — assigning +it to the marker corrupts the document into something `mx check` accepts and +Studio Pro cannot open. `mdl/settingsoverlay` already does this and this proposal +does not change it. + +Layer 2's own file is not BSON. Proposed shape: + +```json +{ + "version": 1, + "constants": { + "MyModule.ApiKey": "sk-…" + } +} +``` + +Flat, per-project, no configuration dimension: this layer means "on this +machine", and a machine runs one thing at a time. If that proves wrong, a +`configurations` key can be added without breaking `version: 1` readers. + +## Implementation plan + +Four slices, each independently shippable and independently verifiable. + +### Slice 1 — close the `test --local` gap (bug fix) + +The smallest correct change, and the one with a user-visible bug behind it. + +| File | Change | +|------|--------| +| `cmd/mxcli/docker/localapp.go` | `LocalAppOptions.ConstantOverrides`; pass to `StartLocalRuntime` | +| `cmd/mxcli/testrunner/runner_local.go`, `runner_endpoint.go` | resolve and pass the overrides | +| `cmd/mxcli/cmd_test_run.go` | `--configuration` flag, mirroring `run` | + +Test: a `.test.mdl` asserting a constant, run under `--local` and under +`--attach` against the same project, must agree. That test fails today. + +### Slice 2 — layer 1, `--constant Module.Name=value` + +| File | Change | +|------|--------| +| `cmd/mxcli/constants_resolve.go` (new) | the chain: layers 4→1, returning value + provenance | +| `cmd/mxcli/runconstants.go` | fold `resolveConstantOverrides` into the chain | +| `cmd/mxcli/cmd_run.go`, `cmd_test_run.go` | `--constant` (repeatable) | + +Refuses an unknown constant name rather than passing it through: a typo'd +override is silently ignored by the runtime, which is the §33 shape again. + +### Slice 3 — layer 2, the machine store + +| File | Change | +|------|--------| +| `cmd/mxcli/constantstore/` (new) | load/save `/.mxcli/constants.json`, 0600, atomic rename | +| `cmd/mxcli/cmd_constant.go` (new) | `mxcli constant set/unset/list` | +| `cmd/mxcli/init.go` | assert `.mxcli/` is in `.gitignore` (it is; make it a checked invariant) | + +`constant set` refuses to write a name the project does not declare, and warns +when the same constant also has a shared override, naming which one wins. + +### Slice 4 — `--apply` (the live path) + +| File | Change | +|------|--------| +| `cmd/mxcli/docker/localboot.go` | export a `SetConstants(...)` that re-sends the full payload with constants overlaid, then reloads | +| `cmd/mxcli/cmd_constant.go` | `--apply`: locate the dev loop via `.mxcli/test-endpoint.json`-style handshake, apply, verify | + +Verification is by observation, not by return code. The handshake file that +`run --local --test-endpoint` already publishes is the model for locating the +running loop; a dev loop without it can still be reached by admin port, but +`--apply` must then be given `--admin-port` explicitly rather than guessing. + +## Version compatibility + +Not version-gated. `deployment/model/config.json`, the M2EE +`update_configuration` action and `reload_model` are stable across the 10.x/11.x +range mxcli supports, and layers 1–2 are mxcli's own. The one version-specific +fact — Mendix private values existing from 10.9 — is a *non*-goal, so it gates +nothing; it only appears in the explanatory text of `constant list`. + +`sdk/versions/mendix-*.yaml` needs no new entry. + +## Test plan + +| Layer the symptom lives in | Test | +|---|---| +| Precedence resolution | unit tests in `cmd/mxcli/` — each layer wins over the one below; provenance is reported; an unknown name is refused | +| The machine store | unit tests — 0600, atomic write, absent file is not an error, malformed file is a named error not a panic | +| Files on disk | `.mxcli/constants.json` is matched by the generated `.gitignore` (assert, don't assume) | +| **The value a running app actually uses** | **`.claude/skills/verify-in-runtime.md`** — boot `run --local --test-endpoint`, read the constant through a microflow over the test endpoint. This is the only layer that can prove the value arrived, and every bug in this area has been invisible at every layer above it | +| The live path | the same, with the control that matters: change the value, read **without** a reload, assert it is *unchanged*; then reload and assert it changed | + +MDL examples: `mdl-examples/doctype-tests/09-constant-examples.mdl` already +covers layers 3–4 and needs no change. Layers 1–2 are CLI, not MDL, so they get +CLI tests rather than doctype tests. + +The runtime test must include the negative control. A test that only asserts +"after set + apply the value is X" passes against an implementation that applies +the value at boot and ignores `--apply` entirely. + +## Open questions + +1. **Should `--constant` exist at all, given layer 2?** A flag puts the secret in + shell history and in `ps` output. The argument for keeping it is CI, where the + value comes from the runner's secret store and there is no persistent machine. + An env route (`MXCLI_CONSTANT_.`) avoids `ps` but not the + environment. **Recommendation:** ship `--constant` in slice 2 for the + ephemeral/CI case, document the exposure in its own flag help, and let layer 2 + be the recommended path for a developer machine. +2. **Per-configuration layer 2?** Deliberately flat above. Someone running two + configurations against one checkout would need it; nobody has asked. +3. **Does `mxcli docker run` get the same chain?** It should, for consistency, but + the container boots through a different path and this proposal does not cover + it. Worth a follow-up rather than a rushed slice. +4. **Masking in `constant list`.** Masking every layer-2 value is safe but + annoying for non-secrets. An explicit `--secret` marker on `constant set` + would let non-secrets print — at the cost of a decision on every set. + +## Appendix A — measurements + +Mendix 11.12.1, `mxcli run --local --test-endpoint`, values read by invoking a +microflow that returns `@MyFirstModule.ApiKey` over the test endpoint by raw +HTTP. Reading this way is load-bearing: `mxcli test --attach` rebuilds and +hot-reloads on every run, which would have made the reload the confound. + +| # | Action | Value the running microflow returned | +|---|--------|--------------------------------------| +| 0 | boot with a configuration `Default` override | `RUNTIME-KEY` | +| 1 | `update_configuration` → INJECTED, then `reload_model` | `INJECTED-KEY` | +| **2** | **`update_configuration` → LIVE, no reload** | **`INJECTED-KEY` — unchanged** | +| 3 | `reload_model`, no further config call | `LIVE-KEY` | +| 4 | payload carrying **one** constant, then reload | `PARTIAL-KEY`; the omitted constant still resolved to its default; the database still worked | + +Row 0 is a live re-confirmation that the §33 configuration-override fix works +end to end. Row 2 is the control that proves the call is staged. Row 4 corrected +a comment in `localboot.go` that claimed the admin action "REPLACES rather than +merges" — at the runtime level, for constants, it does not. + +`get_configuration`, `get_current_configuration`, `runtime_config` and +`get_current_runtime_status` all returned `{"result":-5,"message":"Action not +found."}`. + +**Not measured:** a runtime *restart* with a partial configuration still in +place — the `shutdown` used to test it ended the JVM. In practice mxcli always +sends the full payload at boot, so a partial configuration never survives a +restart. + +## References + +- mxcli-chat FINDINGS §33 — the silent no-op this builds on +- `cmd/mxcli/runconstants.go` — layer 3 resolution as it exists today +- `cmd/mxcli/docker/localboot.go` — the boot payload and what the admin action does +- `mdl/settingsoverlay/settingsoverlay.go` — shared/private guard on the model side +- [Configurations](https://docs.mendix.com/refguide/configuration/) · + [Configurations Tab](https://docs.mendix.com/refguide/configurations-tab/) · + [Constants](https://docs.mendix.com/refguide/constants/) · + [App Setup Best Practices](https://docs.mendix.com/refguide/app-setup-best-practices/) From 89af9a26800841e8583e32e5fe20505d89043324 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 07:34:48 +0000 Subject: [PATCH 10/20] Run --local tests with the same constants as run --local MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `.test.mdl` asserting on something a constant feeds could pass under `mxcli test --attach` and fail under `mxcli test --local`, with nothing in either output to explain it. `--local` boots an app of its own through StartLocalApp, whose options had no ConstantOverrides field, so it ran with each constant's default out of deployment/model/config.json; `--attach` runs against an app `run --local` booted, which applies the configuration's shared overrides. A constant resolving to the wrong value is not an error, so both runs reported success and only the assertion differed. The two --local runners each built their own LocalAppOptions literal, so a field added for one would not have reached the other. Both now go through localAppOptions, and the LocalAppOptions -> LocalRuntimeOptions step is a runtimeOptions() method, so the forwarding is assertable without booting anything — a field dropped there is otherwise invisible until an app runs with the wrong configuration. `mxcli test` gains --configuration, resolved by the same code `run --local` uses and reported the same way. Only a --local run resolves it: --attach inherits the constants of the app it attached to, and printing a resolution it does not use would be a confident lie. Verified at the layer the symptom lives in, on 11.12.1: one project with a constant defaulting to DEFAULT-KEY and a configuration override of RUNTIME-KEY, the same suite run both ways. With the wiring the --local run passes on RUNTIME-KEY; with the wiring reverted it fails with exactly the reported symptom. docs/11-proposals/PROPOSAL_constant_values.md slice 1. --- .claude/skills/fix-issue.md | 1 + .claude/skills/mendix/test-microflows.md | 26 ++++++++ cmd/mxcli/cmd_test_run.go | 11 ++++ cmd/mxcli/docker/localapp.go | 44 +++++++++---- cmd/mxcli/docker/localboot_constants_test.go | 33 ++++++++++ cmd/mxcli/main.go | 1 + cmd/mxcli/testrunner/localapp_options.go | 41 ++++++++++++ cmd/mxcli/testrunner/localapp_options_test.go | 65 +++++++++++++++++++ cmd/mxcli/testrunner/runner.go | 10 +++ cmd/mxcli/testrunner/runner_endpoint.go | 21 ++---- cmd/mxcli/testrunner/runner_local.go | 29 +++------ docs-site/src/tools/running-tests.md | 7 ++ 12 files changed, 239 insertions(+), 50 deletions(-) create mode 100644 cmd/mxcli/testrunner/localapp_options.go create mode 100644 cmd/mxcli/testrunner/localapp_options_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 9ba0d7af6..397cf5d7d 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -498,3 +498,4 @@ extracting `OffsetExpression`/`LimitExpression`. | A `call java action` whose parameter is a **microflow** (`MCPServer.AddTool`'s `ExecutingMicroflow`, and every other "register a callback" action) fails `mx check` with **CE0115** — the microflow's name is stored as a literal string | The modelsdk engine's java-action read had no case for the parameter type, so `codeActionBasicFromGen` fell through to its `default` and returned a `StringType`. The microflow builder's `microflowTypeParams` branch was therefore never taken and it authored `Microflows$BasicCodeActionParameterValue{Argument: "'M.MyFlow'"}` instead of `Microflows$MicroflowParameterValue{Microflow: "M.MyFlow"}` | `mdl/backend/modelsdk/java_read.go` (`codeActionBasicFromGen`), `java_write.go` (`codeActionParamTypeToGen`), `mdl/executor/cmd_javaactions.go` (`astDataTypeToJavaActionParamType`, `bareDataTypeName`) | **A `default:` that returns a plausible type is how a read path lies.** The legacy parser and the executor both handled this correctly for years; only the gen→semantic converter did not, and it degraded to String rather than erroring — so every layer downstream looked correct in isolation. **Fix read and write together**: without the write case an *update* of such an action rewrites the parameter as a String, which is worse than the read bug. **Watch what DESCRIBE starts printing** — the fix made it emit the honest `Microflow`, which then round-tripped into an entity type with an empty module (`.Microflow`) until `astDataTypeToJavaActionParamType` learned the bare word. Probe the stored `$Type` from the marketplace `.mpk` before theorising; MCP Server 5.1.0 stores `JavaActions$MicroflowJavaActionParameterType`. Tests `mdl/backend/modelsdk/java_read_test.go`, `mdl/executor/cmd_javaactions_test.go`, example `mdl-examples/bug-tests/javaaction-microflow-parameter.mdl`; verified 0 errors under `mx check` 11.12.1. Reported in mxcli-chat FINDINGS §36 | | `mxcli check --references` reports "java action not found: M.X (referenced by call java action)" for an action the **same script** creates a few statements earlier. Entities, microflows, pages and nanoflows in the same position are accepted | `scriptContext` — the set of objects a script defines, consulted so a script can be checked against its own output — had categories for modules, entities, enumerations, microflows, nanoflows, pages and snippets, but not for java or JavaScript actions. The java-action branch of `validateFlowBodyReferences` went straight to the project lookup | `mdl/executor/validate.go` (`scriptContext.javaActions`/`javaScriptActions`, `collectDefinitions`, `collectSingle`, `allNames`, `has`, `validateFlowBodyReferences`) | **Store the declared parameter names, not a bool.** Exempting the action from "not found" must not also exempt it from the parameter-name check — the script declares the parameters, so a typo is still catchable, and a bool would have silently dropped that. **`allNames()` and `has()` move together**: `annotateForwardRef` reads both, so adding a category to one and not the other makes a *created* object look "defined later in this script" forever. Tests `mdl/executor/validate_script_javaactions_test.go` (including the control that a misspelled parameter still errors). Reported in mxcli-chat FINDINGS §37 | | `GRANT ON Module.Specialization (READ *, WRITE *)` writes a model Mendix rejects with **CE0066** "Entity access is out of date", while the same grant on the generalization checks clean. The executor passes 3 MemberAccess entries and storage holds 2 | `ReconcileMemberAccesses` — which runs after every program and after every `create association` — recomputes each rule's expected member set from the entity's **own** attributes and the module's **FROM-side** associations. An inherited association matched neither, so it was classed stale and deleted, one step after the GRANT that had just written it correctly | `mdl/backend/modelsdk/domainmodel_security_write.go` (`sameModuleAncestors`, `entitiesByName`, `assocRefBelongsTo`, `ownerIDs` in `ReconcileMemberAccesses`), `mdl/executor/cmd_associations.go` (reconcile after DROP) | **The write path was innocent** — `AddEntityAccessRule` stored all 3; the deletion happened in the reconcile that ran next. When a value is written and then absent, instrument the *later* pass before the writer. **Preserve what cannot be checked**, as the attribute branch already did (#758): an association is qualified by the module that DECLARES it, so an ancestor in another module (the reported `OpenAIDeployedModel extends GenAICommons.DeployedModel`) names a domain model that is not loaded here. **The counter-control is the whole test**: a walk that collected every association in the module passes both positive cases and puts an entry on the TO side of an `OWNER Default` association, which is *itself* CE0066 — so assert that the unrelated entity does **not** get one. **Reconcile must add as well as keep**, or a rule written before the ancestor gained the association never catches up. Verifying this surfaced a neighbouring bug: `drop association` never reconciled at all, leaving entries Mendix rejects with CE1613 — on the declaring entity too, so it predates inheritance support. Tests `mdl/backend/modelsdk/reconcile_inherited_assoc_test.go` (5 cases incl. both controls), `mdl/executor/cmd_associations_mock_test.go`; example `mdl-examples/bug-tests/grant-on-specialization.mdl`, measured 1 error → 0 on mxbuild 11.12.1. Reported in mxcli-chat FINDINGS §25 | +| A `.test.mdl` asserting on something a **constant** feeds passes under `mxcli test --attach` and fails under `mxcli test --local` (or the reverse), with nothing in either output to explain the difference | `--local` boots an app of its own through `StartLocalApp`, whose options had no `ConstantOverrides` field — so it ran with each constant's **default** from `deployment/model/config.json`, while `--attach` runs against an app `run --local` booted, which applies the configuration's shared overrides. A constant resolving to the wrong value is not an error, so both runs reported success and only the assertion differed | `cmd/mxcli/docker/localapp.go` (`LocalAppOptions.ConstantOverrides`, `runtimeOptions`), `cmd/mxcli/testrunner/localapp_options.go` (new, shared by both `--local` runners), `runner.go` (`RunOptions.ConstantOverrides`), `cmd/mxcli/cmd_test_run.go` + `main.go` (`--configuration`) | **A boot path that takes options needs one mapping function, not an inline struct literal per caller** — the two `--local` runners each built their own `LocalAppOptions`, so a field added for one would not reach the other, and neither had the constants. Both now go through `localAppOptions`, and the `LocalAppOptions`→`LocalRuntimeOptions` step is `runtimeOptions()` so the forwarding is assertable without booting anything: a dropped field there is otherwise invisible until an app runs with the wrong configuration. **Resolve in one place**: `cmd/` decides precedence and the runner only carries the map, or "which configuration wins" gets two answers. **Only report what this run actually uses** — `--attach` inherits the constants of the app it attached to, so resolving and printing them there would be a confident lie. Verified at the layer the symptom lives in (`.claude/skills/verify-in-runtime.md`): the same suite, one project, `--local` and `--attach` must agree, with the reverted-wiring control run showing the constant's default. Unit tests `cmd/mxcli/testrunner/localapp_options_test.go`, `cmd/mxcli/docker/localboot_constants_test.go`. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 1 | diff --git a/.claude/skills/mendix/test-microflows.md b/.claude/skills/mendix/test-microflows.md index 2a2f90ff8..994edb762 100644 --- a/.claude/skills/mendix/test-microflows.md +++ b/.claude/skills/mendix/test-microflows.md @@ -33,6 +33,32 @@ mxcli test tests/ -p app.mpr # Docker same project while the tests run — the tests never write into the database you are looking at in the browser. The database is created on first use. +### Constants + +A `--local` run boots the app with the **same constant values `mxcli run --local` +uses**: the project configuration's shared overrides, layered over each +constant's default. It prints what it applied before the run: + +``` +Applying 1 constant value(s) from configuration "Default": MyModule.ApiKey +``` + +Pass `--configuration ` to pick one when the project has several and none +is called `Default` (it refuses to guess rather than run production's values by +accident). `--attach` takes no `--configuration`: it runs against an app someone +else booted and inherits **that app's** constants. + +This is worth knowing when a test asserts on something a constant feeds. Before +this was wired up, `--local` ran with each constant's *default* while `--attach` +ran with the configuration's, so the same suite could pass one way and fail the +other with nothing in the output to explain it. + +A value that must not reach version control has nowhere safe to live yet — a +constant's default and a shared configuration override are both committed, and +Mendix's own private values are encrypted per user account by Studio Pro and +unreachable headlessly. See +`docs/11-proposals/PROPOSAL_constant_values.md`. + --- ## Test File Formats diff --git a/cmd/mxcli/cmd_test_run.go b/cmd/mxcli/cmd_test_run.go index 46b7abcdd..103093d49 100644 --- a/cmd/mxcli/cmd_test_run.go +++ b/cmd/mxcli/cmd_test_run.go @@ -134,6 +134,7 @@ Examples: watch, _ := cmd.Flags().GetBool("watch") attach, _ := cmd.Flags().GetBool("attach") skipAppStartup, _ := cmd.Flags().GetBool("skip-app-startup") + configuration, _ := cmd.Flags().GetString("configuration") verbose, _ := cmd.Flags().GetBool("verbose") color, _ := cmd.Flags().GetBool("color") timeoutStr, _ := cmd.Flags().GetString("timeout") @@ -178,6 +179,16 @@ Examples: Stderr: os.Stderr, } + // Only a --local run boots an app of its own, so only it decides which + // constants that app runs with. --attach inherits the constants of the app + // it attaches to, and the Docker path configures the container — reporting + // a resolution neither of them uses would be a lie in the output. + if local && !attach { + overrides := constantOverridesFor(projectPath, configuration) + reportConstantOverrides(os.Stdout, overrides) + opts.ConstantOverrides = overrides.Values + } + result, err := testrunner.Run(opts) if err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) diff --git a/cmd/mxcli/docker/localapp.go b/cmd/mxcli/docker/localapp.go index 2c4e3e029..056c10510 100644 --- a/cmd/mxcli/docker/localapp.go +++ b/cmd/mxcli/docker/localapp.go @@ -40,11 +40,42 @@ type LocalAppOptions struct { // LocalRuntimeOptions.Env) — how a secret reaches the runtime without being // written to disk. Env []string + // ConstantOverrides are the constant values this run should use, layered over + // the defaults mxbuild wrote into deployment/model/config.json. + // + // A headless boot needs these for the same reason `run --local` does: the + // deployment carries only each constant's DEFAULT, so an app booted without + // them runs as if no configuration existed. Missing here, `mxcli test --local` + // ran a suite against different values than the same suite under `--attach`, + // which runs against an app `run --local` booted — silently, since a constant + // resolving to the wrong value is not an error. See + // docs/11-proposals/PROPOSAL_constant_values.md and mxcli-chat FINDINGS §33. + ConstantOverrides map[string]string // Stdout/Stderr receive progress messages. Stdout io.Writer Stderr io.Writer } +// runtimeOptions is the LocalAppOptions -> LocalRuntimeOptions mapping, split +// out so the forwarding is assertable without booting anything. Every field the +// runtime needs has to appear here; one omitted is invisible until an app runs +// with the wrong configuration. +func (o LocalAppOptions) runtimeOptions(installPath string) LocalRuntimeOptions { + return LocalRuntimeOptions{ + DeployDir: o.DeployDir, + InstallPath: installPath, + AppPort: o.AppPort, + AdminPort: o.AdminPort, + AdminPass: o.AdminPass, + DB: o.DB, + RuntimeLogPath: o.RuntimeLogPath, + Env: o.Env, + ConstantOverrides: o.ConstantOverrides, + Stdout: o.Stdout, + Stderr: o.Stderr, + } +} + // LocalApp is a booted local app: an mxbuild serve server plus the standalone // runtime it deployed to. It is the Docker-free equivalent of `docker compose // up` for callers that need an app running and then stopped again. @@ -177,18 +208,7 @@ func StartLocalApp(opts LocalAppOptions) (*LocalApp, error) { } // 5. Boot the runtime against the deployment. - rt, err := StartLocalRuntime(LocalRuntimeOptions{ - DeployDir: opts.DeployDir, - InstallPath: installPath, - AppPort: opts.AppPort, - AdminPort: opts.AdminPort, - AdminPass: opts.AdminPass, - DB: opts.DB, - RuntimeLogPath: opts.RuntimeLogPath, - Env: opts.Env, - Stdout: opts.Stdout, - Stderr: opts.Stderr, - }) + rt, err := StartLocalRuntime(opts.runtimeOptions(installPath)) if err != nil { app.Stop() return nil, err diff --git a/cmd/mxcli/docker/localboot_constants_test.go b/cmd/mxcli/docker/localboot_constants_test.go index e92f0f6b5..e00e97db5 100644 --- a/cmd/mxcli/docker/localboot_constants_test.go +++ b/cmd/mxcli/docker/localboot_constants_test.go @@ -43,3 +43,36 @@ func TestMergeConstantOverrides_NoOverrides(t *testing.T) { t.Errorf("got %v, want the defaults unchanged", got) } } + +// StartLocalApp is the headless boot behind `mxcli test --local`. Its options +// have to reach LocalRuntimeOptions or the app runs with each constant's +// default while `--attach` runs with the configuration's — the divergence in +// docs/11-proposals/PROPOSAL_constant_values.md slice 1. +func TestLocalAppOptions_ForwardsToTheRuntime(t *testing.T) { + opts := LocalAppOptions{ + DeployDir: "/tmp/app/deployment", + AppPort: 8081, + AdminPort: 8091, + AdminPass: "pass", + DB: DBConfig{Name: "app_test"}, + RuntimeLogPath: "/tmp/app/.mxcli/test-runtime.log", + Env: []string{"MXCLI_TEST_TOKEN=tok"}, + ConstantOverrides: map[string]string{"MyModule.ApiKey": "v"}, + } + + rt := opts.runtimeOptions("/install/path") + + if rt.ConstantOverrides["MyModule.ApiKey"] != "v" { + t.Fatalf("ConstantOverrides = %v, want the value to reach the runtime", rt.ConstantOverrides) + } + // The fields that were already forwarded stay forwarded: this mapping is the + // single place a runtime option can be dropped without anything failing. + if rt.DeployDir != opts.DeployDir || rt.AppPort != opts.AppPort || rt.AdminPort != opts.AdminPort || + rt.AdminPass != opts.AdminPass || rt.DB.Name != opts.DB.Name || + rt.RuntimeLogPath != opts.RuntimeLogPath || len(rt.Env) != 1 { + t.Errorf("a field was dropped in the mapping: %+v", rt) + } + if rt.InstallPath != "/install/path" { + t.Errorf("InstallPath = %q", rt.InstallPath) + } +} diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index c6fc84421..330f39ceb 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -375,6 +375,7 @@ func init() { testRunCmd.Flags().BoolP("watch", "w", false, "With --local, keep the runtime warm and re-run the suite on every test or model change (Ctrl-C to stop)") testRunCmd.Flags().Bool("skip-app-startup", false, "With --local, do not run the project's own after-startup microflow during the test run (it runs by default, so tests see the app as it really boots)") testRunCmd.Flags().Bool("attach", false, "Run against an app already started with 'mxcli run --local --test-endpoint' instead of booting one (tests hit that app's database)") + testRunCmd.Flags().String("configuration", "", "With --local, which project configuration's constant values to run the tests with (default: the only one, or \"Default\") — the same resolution 'mxcli run --local' uses, so a suite sees the same constants either way") testRunCmd.Flags().BoolP("verbose", "v", false, "Show all runtime log output") testRunCmd.Flags().BoolP("color", "", false, "Use colored output") testRunCmd.Flags().StringP("timeout", "t", "5m", "Timeout for runtime startup and test execution") diff --git a/cmd/mxcli/testrunner/localapp_options.go b/cmd/mxcli/testrunner/localapp_options.go new file mode 100644 index 000000000..a9f64e1dc --- /dev/null +++ b/cmd/mxcli/testrunner/localapp_options.go @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: Apache-2.0 + +package testrunner + +import ( + "io" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// localAppOptions builds the headless boot for a `--local` test run, shared by +// the endpoint runner and the legacy after-startup runner. +// +// The two runners differ only in what reaches the runtime's environment and in +// which log they read, so everything else — ports, the scratch database, and +// the constant values the app runs with — is decided in one place. It was the +// constants that made this worth sharing: they were absent from both runners, +// so a suite saw a different value under `--local` than the same suite saw +// under `--attach` (which runs against an app `run --local` booted, with the +// configuration's values applied). Nothing errored; the assertion just ran +// against the wrong constant. See docs/11-proposals/PROPOSAL_constant_values.md. +func localAppOptions(opts RunOptions, logPath string, env []string, w io.Writer) docker.LocalAppOptions { + return docker.LocalAppOptions{ + ProjectPath: opts.ProjectPath, + AppPort: localTestAppPort, + AdminPort: localTestAdminPort, + ServePort: localTestServePort, + DB: docker.DBConfig{ + // A scratch database, so a `run --local` dev loop can keep serving the + // same project while the tests run. + Name: docker.DeriveDBName(opts.ProjectPath) + localTestDBSuffix, + }, + EnsureDB: true, + SkipBuild: opts.SkipBuild, + Env: env, + ConstantOverrides: opts.ConstantOverrides, + RuntimeLogPath: logPath, + Stdout: w, + Stderr: w, + } +} diff --git a/cmd/mxcli/testrunner/localapp_options_test.go b/cmd/mxcli/testrunner/localapp_options_test.go new file mode 100644 index 000000000..fea8117ac --- /dev/null +++ b/cmd/mxcli/testrunner/localapp_options_test.go @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: Apache-2.0 + +// A `--local` test run booted with each constant's DEFAULT, because the headless +// boot never carried the configuration's values. The same suite under `--attach` +// runs against an app `run --local` booted, which does apply them — so the two +// modes silently disagreed on every constant a test touched. Nothing errored: a +// constant resolving to the wrong value is not an error, it is just a different +// assertion. docs/11-proposals/PROPOSAL_constant_values.md slice 1. +package testrunner + +import ( + "io" + "testing" +) + +func TestLocalAppOptions_CarriesTheConstantOverrides(t *testing.T) { + opts := RunOptions{ + ProjectPath: "/tmp/app/App.mpr", + ConstantOverrides: map[string]string{"MyModule.ApiKey": "from-the-configuration"}, + } + + got := localAppOptions(opts, "/tmp/app/.mxcli/test-runtime.log", nil, io.Discard) + + if got.ConstantOverrides["MyModule.ApiKey"] != "from-the-configuration" { + t.Fatalf("ConstantOverrides = %v, want the configuration's value — without it a "+ + "--local run asserts against the constant's default while --attach asserts "+ + "against the configuration's", got.ConstantOverrides) + } +} + +// Both --local runners boot through this, so both have to carry the values. The +// endpoint runner adds the token to the environment and the legacy runner does +// not; that is the only difference between them. +func TestLocalAppOptions_SameForBothRunners(t *testing.T) { + opts := RunOptions{ + ProjectPath: "/tmp/app/App.mpr", + ConstantOverrides: map[string]string{"A.B": "v"}, + } + + endpoint := localAppOptions(opts, "log", []string{endpointTokenEnv + "=tok"}, io.Discard) + legacy := localAppOptions(opts, "log", nil, io.Discard) + + if endpoint.ConstantOverrides["A.B"] != "v" || legacy.ConstantOverrides["A.B"] != "v" { + t.Errorf("endpoint=%v legacy=%v, want both to carry the value", + endpoint.ConstantOverrides, legacy.ConstantOverrides) + } + if endpoint.AppPort != legacy.AppPort || endpoint.DB.Name != legacy.DB.Name { + t.Errorf("the runners disagree on ports/database: %+v vs %+v", endpoint, legacy) + } + if len(endpoint.Env) != 1 || len(legacy.Env) != 0 { + t.Errorf("env differs from expectation: endpoint=%v legacy=%v", endpoint.Env, legacy.Env) + } +} + +// The scratch database is what lets a `run --local` dev loop keep serving the +// same project while tests run. Sharing the option builder must not lose it. +func TestLocalAppOptions_UsesAScratchDatabase(t *testing.T) { + got := localAppOptions(RunOptions{ProjectPath: "/tmp/app/App.mpr"}, "log", nil, io.Discard) + if want := "app" + localTestDBSuffix; got.DB.Name != want { + t.Errorf("DB.Name = %q, want %q", got.DB.Name, want) + } + if !got.EnsureDB { + t.Error("EnsureDB is false; the scratch database would have to exist already") + } +} diff --git a/cmd/mxcli/testrunner/runner.go b/cmd/mxcli/testrunner/runner.go index 33e7e903e..0113fae72 100644 --- a/cmd/mxcli/testrunner/runner.go +++ b/cmd/mxcli/testrunner/runner.go @@ -68,6 +68,16 @@ type RunOptions struct { // counts. SkipAppStartup bool + // ConstantOverrides are the constant values the app should run with, layered + // over the defaults in the deployment. Resolved by the caller from the + // project's configuration (and, in future, the higher layers of + // docs/11-proposals/PROPOSAL_constant_values.md) so the runner stays a + // carrier rather than a second place that decides precedence. + // + // Only --local uses these: --attach runs against an app someone else booted + // and inherits ITS constants, and the Docker path configures the container. + ConstantOverrides map[string]string + // Timeout for runtime startup and test execution. Timeout time.Duration diff --git a/cmd/mxcli/testrunner/runner_endpoint.go b/cmd/mxcli/testrunner/runner_endpoint.go index 969c33d56..e2df68e4b 100644 --- a/cmd/mxcli/testrunner/runner_endpoint.go +++ b/cmd/mxcli/testrunner/runner_endpoint.go @@ -25,23 +25,10 @@ func bootForTests(opts RunOptions, token string, timeout time.Duration, w io.Wri logPath := filepath.Join(filepath.Dir(opts.ProjectPath), ".mxcli", "test-runtime.log") fmt.Fprintln(w, "Starting local runtime (no Docker)...") - app, err := docker.StartLocalApp(docker.LocalAppOptions{ - ProjectPath: opts.ProjectPath, - AppPort: localTestAppPort, - AdminPort: localTestAdminPort, - ServePort: localTestServePort, - DB: docker.DBConfig{ - Name: docker.DeriveDBName(opts.ProjectPath) + localTestDBSuffix, - }, - EnsureDB: true, - SkipBuild: opts.SkipBuild, - // The token reaches the runtime through its environment and is never - // written to the project. See endpointTokenEnv. - Env: []string{endpointTokenEnv + "=" + token}, - RuntimeLogPath: logPath, - Stdout: w, - Stderr: w, - }) + // The token reaches the runtime through its environment and is never written + // to the project. See endpointTokenEnv. + app, err := docker.StartLocalApp( + localAppOptions(opts, logPath, []string{endpointTokenEnv + "=" + token}, w)) if err != nil { // Unlike the after-startup path, a boot failure here is never a test // result — no test has run yet. It is always a real error. diff --git a/cmd/mxcli/testrunner/runner_local.go b/cmd/mxcli/testrunner/runner_local.go index 9092c6d62..114545876 100644 --- a/cmd/mxcli/testrunner/runner_local.go +++ b/cmd/mxcli/testrunner/runner_local.go @@ -34,27 +34,14 @@ func runLocalAndCapture(opts RunOptions, timeout time.Duration, w io.Writer) (st offset := fileSize(logPath) fmt.Fprintln(w, "Starting local runtime (no Docker)...") - app, err := docker.StartLocalApp(docker.LocalAppOptions{ - ProjectPath: opts.ProjectPath, - AppPort: localTestAppPort, - AdminPort: localTestAdminPort, - ServePort: localTestServePort, - DB: docker.DBConfig{ - Name: docker.DeriveDBName(opts.ProjectPath) + localTestDBSuffix, - }, - EnsureDB: true, - SkipBuild: opts.SkipBuild, - // The runner reports through an after-startup microflow, so its LOG output - // is produced DURING the start action — before the runtime's own log - // subscriber is attached. What carries it is the JVM console tee, which is - // live from spawn. Verified on 11.12.1; registering the subscriber early - // instead is not an option, the runtime rejects it pre-start with a - // LoggingException. If a future runtime stops echoing to the console the - // failure is loud, not silent: unseen tests are reported as errors. - RuntimeLogPath: logPath, - Stdout: w, - Stderr: w, - }) + // The runner reports through an after-startup microflow, so its LOG output is + // produced DURING the start action — before the runtime's own log subscriber + // is attached. What carries it is the JVM console tee, which is live from + // spawn. Verified on 11.12.1; registering the subscriber early instead is not + // an option, the runtime rejects it pre-start with a LoggingException. If a + // future runtime stops echoing to the console the failure is loud, not + // silent: unseen tests are reported as errors. + app, err := docker.StartLocalApp(localAppOptions(opts, logPath, nil, w)) if err != nil { // A failing test IS a failed boot: the generated runner returns false, so // the runtime's after-startup action fails and `start` reports an error. diff --git a/docs-site/src/tools/running-tests.md b/docs-site/src/tools/running-tests.md index e79a9d837..5c331c318 100644 --- a/docs-site/src/tools/running-tests.md +++ b/docs-site/src/tools/running-tests.md @@ -12,6 +12,13 @@ and **Docker is only needed for the first**: (8081/8091) and its own `_test` database, so a `mxcli run --local` dev loop can keep serving the same project while tests run. +A `--local` run boots the app with the same **constant values** `mxcli run --local` +uses — the project configuration's shared overrides layered over each constant's +default — and prints what it applied. Use `--configuration ` to choose +between several; `--attach` takes none, since it inherits the constants of the app +it attaches to. This matters whenever a test asserts on something a constant +feeds: the two modes used to disagree silently. + `--local` also downloads what it needs on first use. To pre-cache it: ```bash From 353c5cb9645d5f529ecec1fd2edde29f6419ee42 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 08:44:17 +0000 Subject: [PATCH 11/20] Add --constant, for a value that reaches one run and nothing else MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every route mxcli offered for setting a constant wrote to the model, and therefore to git, so running once with a different API key meant committing it or remembering to revert. `--constant Module.Name=value` (repeatable, on `run --local` and `test --local`) wins over the configuration and is never written anywhere. A name the project does not declare is refused before anything boots. The runtime silently ignores a MicroflowConstants entry matching no constant, so a typo would otherwise be accepted, reported as applied, and do nothing — the mxcli-chat §33 failure shape, reintroduced by the flag meant to help with it. A missing "=" is refused for the same reason: `--constant M.C` almost certainly meant the value to be the next argument, and quietly setting the constant to "" is the same class of silent wrong value. The report now names the layer each value came from rather than assuming one source, and a flag that covers a private override stops that constant being reported as private-and-defaulted — which would contradict what the app is about to do. --attach refuses --constant rather than ignoring it: it runs against an app someone else booted and inherits that app's constants. Verified on 11.12.1 against a real runtime: with the project's Default configuration setting RUNTIME-KEY, a suite asserting FLAG-KEY passes under --constant MyFirstModule.ApiKey=FLAG-KEY, so the flag reached the app and beat the configuration. The three refusals were checked to fire before any boot. docs/11-proposals/PROPOSAL_constant_values.md slice 2. --- .claude/skills/mendix/run-local.md | 24 +- .claude/skills/mendix/test-microflows.md | 22 +- cmd/mxcli/cmd_run.go | 16 +- cmd/mxcli/cmd_test_run.go | 21 +- cmd/mxcli/constants_resolve.go | 215 ++++++++++++++++++ cmd/mxcli/constants_resolve_test.go | 170 ++++++++++++++ cmd/mxcli/main.go | 1 + cmd/mxcli/runconstants.go | 45 +--- cmd/mxcli/runconstants_test.go | 19 -- docs-site/src/tools/running-tests.md | 5 +- docs/11-proposals/PROPOSAL_constant_values.md | 8 +- 11 files changed, 464 insertions(+), 82 deletions(-) create mode 100644 cmd/mxcli/constants_resolve.go create mode 100644 cmd/mxcli/constants_resolve_test.go diff --git a/.claude/skills/mendix/run-local.md b/.claude/skills/mendix/run-local.md index 8dad49671..0ba6dbdcb 100644 --- a/.claude/skills/mendix/run-local.md +++ b/.claude/skills/mendix/run-local.md @@ -407,7 +407,8 @@ secret registers it owner-less. is running, merged over each constant's default: ```text -Applying 1 constant value(s) from configuration "Default": Encryption.EncryptionKey +Applying 1 constant value(s): + Encryption.EncryptionKey configuration "Default" ``` Before this they were ignored: mxbuild writes each constant's **default** into @@ -423,8 +424,19 @@ encryption key while the model said otherwise. developer's workstation), so the default is used and the constant is named. - The line prints in every case, including "no overrides" — silence used to mean "your override is in effect" when it was not. - -Setting a constant on a *running* app is a different mechanism: `MicroflowConstants` -over the M2EE admin port, which is how Mendix Cloud injects per-environment values. -Note that `--runtime-setting 'MicroflowConstants={…}'` **replaces** the whole map -rather than merging into it, so it drops every constant it does not mention. +- `--constant Module.Name=value` (repeatable) sets a value for **this run only**. + It wins over the configuration, is never written to the project, and is + reported as coming from `--constant` so the output says which layer won. A + constant the project does not declare is refused before the app boots: the + runtime ignores a value for a constant that does not exist, so a typo would + otherwise be reported as applied and do nothing. + +Setting a constant on an app that is *already running* is a different mechanism +again: `MicroflowConstants` over the M2EE admin port, which is how Mendix Cloud +injects per-environment values. Measured on 11.12.1, `update_configuration` is +**staged rather than applied** — the running app keeps the old value until the +next `reload_model`, while the call answers `result:0` — and the admin API has no +read-back to check against. Do not reach for +`--runtime-setting 'MicroflowConstants={…}'`: it replaces the map mxcli built +rather than adding to it, and at boot there is nothing to fall back on for +`BasePath`/`DatabaseName`. Use `--constant`. diff --git a/.claude/skills/mendix/test-microflows.md b/.claude/skills/mendix/test-microflows.md index 994edb762..1e2c74730 100644 --- a/.claude/skills/mendix/test-microflows.md +++ b/.claude/skills/mendix/test-microflows.md @@ -45,18 +45,32 @@ Applying 1 constant value(s) from configuration "Default": MyModule.ApiKey Pass `--configuration ` to pick one when the project has several and none is called `Default` (it refuses to guess rather than run production's values by -accident). `--attach` takes no `--configuration`: it runs against an app someone -else booted and inherits **that app's** constants. +accident). `--attach` takes neither flag: it runs against an app someone else +booted and inherits **that app's** constants. + +To set a value for one run without touching the project, use `--constant` +(repeatable). It wins over the configuration and is never written to the model: + +```bash +mxcli test tests/ -p app.mpr --local --constant MyModule.ApiKey=sk-test-123 +``` + +A name the project does not declare is **refused**, before anything boots — the +runtime silently ignores a value for a constant that does not exist, so a typo +would otherwise be reported as applied and do nothing. + +The value is visible in shell history and in `ps`. That is fine for a throwaway +test value and wrong for a real secret. This is worth knowing when a test asserts on something a constant feeds. Before this was wired up, `--local` ran with each constant's *default* while `--attach` ran with the configuration's, so the same suite could pass one way and fail the other with nothing in the output to explain it. -A value that must not reach version control has nowhere safe to live yet — a +For a secret that has to **persist** across runs there is still nowhere safe: a constant's default and a shared configuration override are both committed, and Mendix's own private values are encrypted per user account by Studio Pro and -unreachable headlessly. See +unreachable headlessly. `--constant` covers the one-run case only. See `docs/11-proposals/PROPOSAL_constant_values.md`. --- diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index 77102d0b8..d444d84d3 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -157,8 +157,18 @@ Examples: // this the app runs with defaults while the model says otherwise — // silently. See runconstants.go. configuration, _ := cmd.Flags().GetString("configuration") - overrides := constantOverridesFor(projectPath, configuration) - reportConstantOverrides(os.Stdout, overrides) + constantArgs, _ := cmd.Flags().GetStringArray("constant") + constantFlags, err := parseConstantFlags(constantArgs) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + overrides, err := constantChainFor(projectPath, configuration, constantFlags) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + reportConstantChain(os.Stdout, overrides) opts := docker.LocalRunOptions{ ProjectPath: projectPath, @@ -243,6 +253,8 @@ Examples: func init() { runCmd.Flags().String("configuration", "", "Which project configuration's constant values to run with (default: the only one, or \"Default\")") + runCmd.Flags().StringArray("constant", nil, + "Set a constant for THIS RUN only: Module.Name=value (repeatable). Wins over the configuration and is never written to the project. The value is visible in shell history and in `ps` — see docs/11-proposals/PROPOSAL_constant_values.md") runCmd.Flags().Bool("local", false, "Run locally without Docker (warm serve + standalone runtime)") runCmd.Flags().String("hub", "", "Expose the running app in a browser via your own mxcli tunnel-hub URL (e.g. https://hub.example.com). Implies --local; the app stays local and is reverse-tunnelled out") runCmd.Flags().String("hub-secret", "", "Shared auth secret for --hub (\"user:pass\"), matching the hub's --secret") diff --git a/cmd/mxcli/cmd_test_run.go b/cmd/mxcli/cmd_test_run.go index 103093d49..2f7f0fc79 100644 --- a/cmd/mxcli/cmd_test_run.go +++ b/cmd/mxcli/cmd_test_run.go @@ -135,6 +135,7 @@ Examples: attach, _ := cmd.Flags().GetBool("attach") skipAppStartup, _ := cmd.Flags().GetBool("skip-app-startup") configuration, _ := cmd.Flags().GetString("configuration") + constantArgs, _ := cmd.Flags().GetStringArray("constant") verbose, _ := cmd.Flags().GetBool("verbose") color, _ := cmd.Flags().GetBool("color") timeoutStr, _ := cmd.Flags().GetString("timeout") @@ -184,9 +185,25 @@ Examples: // it attaches to, and the Docker path configures the container — reporting // a resolution neither of them uses would be a lie in the output. if local && !attach { - overrides := constantOverridesFor(projectPath, configuration) - reportConstantOverrides(os.Stdout, overrides) + constantFlags, err := parseConstantFlags(constantArgs) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + overrides, err := constantChainFor(projectPath, configuration, constantFlags) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + reportConstantChain(os.Stdout, overrides) opts.ConstantOverrides = overrides.Values + } else if len(constantArgs) > 0 { + // --attach runs against an app someone else booted, and the Docker path + // configures the container. Accepting --constant there and doing nothing + // with it is the failure this feature exists to stop. + fmt.Fprintln(os.Stderr, "Error: --constant applies only to a --local test run; "+ + "--attach uses the constants of the app it attaches to") + os.Exit(1) } result, err := testrunner.Run(opts) diff --git a/cmd/mxcli/constants_resolve.go b/cmd/mxcli/constants_resolve.go new file mode 100644 index 000000000..d8e651569 --- /dev/null +++ b/cmd/mxcli/constants_resolve.go @@ -0,0 +1,215 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "io" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/model" +) + +// constantLayer names where a resolved constant value came from. It exists so a +// run can SAY which layer won: with more than one place to set a value, "the +// value is X" stops being useful on its own, and the whole family of bugs this +// resolves (mxcli-chat FINDINGS §33) is values arriving from somewhere other +// than where the author looked. +type constantLayer string + +const ( + layerFlag constantLayer = "--constant" + layerConfiguration constantLayer = "configuration" +) + +// constantChain is the resolved set of values to hand a booting app, plus the +// layer each one came from. +// +// A constant absent from Values keeps its default, which the deployment already +// carries — mxbuild writes every constant's default into +// deployment/model/config.json, and mxcli layers over that rather than +// replacing it. See docs/11-proposals/PROPOSAL_constant_values.md. +type constantChain struct { + Configuration string // the configuration whose values these are + Values map[string]string // constant qualified name -> value + From map[string]constantLayer // and which layer set it + Private []string // overrides whose value is not in the model + Note string // why no configuration contributed +} + +// parseConstantFlags turns repeated `--constant Module.Name=value` into a map. +// +// A missing "=" is an error rather than a value of "": `--constant M.C` almost +// certainly means the author expected the next argument to be the value, and +// silently setting the constant to empty is exactly the kind of quiet wrong +// value this whole feature exists to stop. +func parseConstantFlags(flags []string) (map[string]string, error) { + out := make(map[string]string, len(flags)) + for _, f := range flags { + name, value, found := strings.Cut(f, "=") + name = strings.TrimSpace(name) + if !found { + return nil, fmt.Errorf("--constant %q: expected Module.Name=value", f) + } + if strings.Count(name, ".") != 1 || strings.HasPrefix(name, ".") || strings.HasSuffix(name, ".") { + return nil, fmt.Errorf("--constant %q: %q is not a qualified constant name (Module.Name)", f, name) + } + if _, dup := out[name]; dup { + return nil, fmt.Errorf("--constant %s given more than once", name) + } + out[name] = value + } + return out, nil +} + +// resolveConstantChain layers `--constant` values over a configuration's shared +// values. known is the set of constants the project declares; a flag naming +// anything else is refused. +// +// Refusing an unknown name is the point of passing `known` at all. The runtime +// ignores a MicroflowConstants entry that matches no constant, so a typo would +// otherwise be accepted, reported as applied, and do nothing — the §33 failure +// shape, reintroduced by the very flag meant to fix it. +func resolveConstantChain(ps *model.ProjectSettings, want string, flags map[string]string, known map[string]bool) (constantChain, error) { + base := resolveConstantOverrides(ps, want) + chain := constantChain{ + Configuration: base.Configuration, + Values: map[string]string{}, + From: map[string]constantLayer{}, + Private: base.Private, + Note: base.Note, + } + for name, value := range base.Values { + chain.Values[name] = value + chain.From[name] = layerConfiguration + } + + var unknown []string + for name := range flags { + if known != nil && !known[name] { + unknown = append(unknown, name) + } + } + if len(unknown) > 0 { + sort.Strings(unknown) + return constantChain{}, fmt.Errorf("no constant named %s in this project\n"+ + " hint: 'mxcli -p -c \"show constant values\"' lists them; the runtime ignores "+ + "a value for a constant that does not exist, so this would have been applied to nothing", + strings.Join(unknown, ", ")) + } + + for name, value := range flags { + chain.Values[name] = value + chain.From[name] = layerFlag + } + // A flag overrides a private configuration value too, and then the default is + // NOT what runs — so the constant must stop being reported as private-and- + // defaulted, or the report contradicts what the app will do. + if len(flags) > 0 && len(chain.Private) > 0 { + kept := chain.Private[:0] + for _, name := range chain.Private { + if _, overridden := flags[name]; !overridden { + kept = append(kept, name) + } + } + chain.Private = kept + } + return chain, nil +} + +// constantChainFor reads a project and resolves the values a run should use. +// +// A project that cannot be read yields no values and a note — a run must not be +// blocked by this, since every run before configurations were applied used the +// defaults anyway. The exception is when `--constant` was passed: the author +// named something specific, and applying it unvalidated (or dropping it +// silently) are both worse than saying the project could not be read. +func constantChainFor(projectPath, configuration string, flags map[string]string) (constantChain, error) { + b := newBackendFactory()() + if err := b.Connect(projectPath); err != nil { + return unreadableProject(err, flags) + } + defer func() { _ = b.Disconnect() }() + + ps, err := b.GetProjectSettings() + if err != nil { + return unreadableProject(err, flags) + } + known, err := knownConstantNames(b) + if err != nil { + return unreadableProject(err, flags) + } + return resolveConstantChain(ps, configuration, flags, known) +} + +func unreadableProject(err error, flags map[string]string) (constantChain, error) { + if len(flags) > 0 { + return constantChain{}, fmt.Errorf("could not read the project's constants to apply --constant: %w", err) + } + return constantChain{ + Values: map[string]string{}, + From: map[string]constantLayer{}, + Note: "could not read the project's settings: " + err.Error(), + }, nil +} + +// knownConstantNames returns the qualified names of every constant the project +// declares. Folders do not appear in a qualified name, so the hierarchy is used +// only to find each constant's module. +func knownConstantNames(b backend.FullBackend) (map[string]bool, error) { + constants, err := b.ListConstants() + if err != nil { + return nil, err + } + h, err := executor.NewContainerHierarchyFromBackend(b) + if err != nil { + return nil, err + } + out := make(map[string]bool, len(constants)) + for _, c := range constants { + if module := h.GetModuleName(h.FindModuleID(c.ContainerID)); module != "" { + out[module+"."+c.Name] = true + } + } + return out, nil +} + +// reportConstantChain says what will be applied before the app boots, and from +// where. +// +// It prints even when nothing is applied. The failure this fixes was invisible +// precisely because the run said nothing about constants either way, so silence +// has to stop meaning "your override is in effect". +func reportConstantChain(w io.Writer, c constantChain) { + switch { + case len(c.Values) > 0: + names := make([]string, 0, len(c.Values)) + width := 0 + for k := range c.Values { + names = append(names, k) + if len(k) > width { + width = len(k) + } + } + sort.Strings(names) + fmt.Fprintf(w, "Applying %d constant value(s):\n", len(names)) + for _, n := range names { + from := string(c.From[n]) + if c.From[n] == layerConfiguration { + from = fmt.Sprintf("configuration %q", c.Configuration) + } + fmt.Fprintf(w, " %-*s %s\n", width, n, from) + } + case c.Configuration != "": + fmt.Fprintf(w, "Configuration %q sets no constant values; using each constant's default.\n", c.Configuration) + case c.Note != "": + fmt.Fprintf(w, "Using each constant's default value (%s).\n", c.Note) + } + if len(c.Private) > 0 { + fmt.Fprintf(w, " %d override(s) are private, so their value is not in the model and the default is used:\n %s\n", + len(c.Private), strings.Join(c.Private, "\n ")) + } +} diff --git a/cmd/mxcli/constants_resolve_test.go b/cmd/mxcli/constants_resolve_test.go new file mode 100644 index 000000000..cb9d3865e --- /dev/null +++ b/cmd/mxcli/constants_resolve_test.go @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Layer 1 of docs/11-proposals/PROPOSAL_constant_values.md: `--constant +// Module.Name=value`, for a value that should reach one run and nothing else. +// Every route mxcli offered before this wrote to the model, and therefore to +// git, so setting an API key for a single run meant committing it (or +// remembering to revert). +package main + +import ( + "bytes" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/model" +) + +var knownTwo = map[string]bool{"A.Key": true, "A.Url": true} + +func TestParseConstantFlags(t *testing.T) { + got, err := parseConstantFlags([]string{"A.Key=sk-123", "A.Url=https://x/y=z"}) + if err != nil { + t.Fatalf("parseConstantFlags: %v", err) + } + if got["A.Key"] != "sk-123" { + t.Errorf("A.Key = %q", got["A.Key"]) + } + // Only the FIRST "=" separates; a value may contain more. + if got["A.Url"] != "https://x/y=z" { + t.Errorf("A.Url = %q, want the whole value after the first '='", got["A.Url"]) + } +} + +// An empty value is legitimate ("run with this constant blank"); a missing "=" +// is not — it almost certainly means the value was meant to be the next +// argument, and quietly setting the constant to "" is the exact class of silent +// wrong value this feature exists to prevent. +func TestParseConstantFlags_RejectsMalformed(t *testing.T) { + if got, err := parseConstantFlags([]string{"A.Key="}); err != nil || got["A.Key"] != "" { + t.Errorf("A.Key= should set an empty value, got %v / %v", got, err) + } + for _, bad := range []string{"A.Key", "NoDots=v", "A.B.C=v", ".Key=v", "A.=v"} { + if _, err := parseConstantFlags([]string{bad}); err == nil { + t.Errorf("parseConstantFlags(%q) accepted it", bad) + } + } + if _, err := parseConstantFlags([]string{"A.Key=1", "A.Key=2"}); err == nil { + t.Error("the same constant given twice was accepted; which one wins is a coin flip") + } +} + +func TestResolveConstantChain_FlagWinsOverTheConfiguration(t *testing.T) { + ps := settingsWith(cfg("Default", shared("A.Key", "from-configuration"), shared("A.Url", "u"))) + + got, err := resolveConstantChain(ps, "", map[string]string{"A.Key": "from-the-flag"}, knownTwo) + if err != nil { + t.Fatalf("resolveConstantChain: %v", err) + } + if got.Values["A.Key"] != "from-the-flag" { + t.Errorf("A.Key = %q, want the flag to win", got.Values["A.Key"]) + } + if got.From["A.Key"] != layerFlag { + t.Errorf("A.Key came from %q, want %q", got.From["A.Key"], layerFlag) + } + // A constant the flag does not name keeps the configuration's value. + if got.Values["A.Url"] != "u" || got.From["A.Url"] != layerConfiguration { + t.Errorf("A.Url = %q from %q, want the configuration's", got.Values["A.Url"], got.From["A.Url"]) + } +} + +// The runtime ignores a MicroflowConstants entry naming no constant, so a typo +// would be accepted, reported as applied, and do nothing — the mxcli-chat §33 +// shape, reintroduced by the flag meant to fix it. +func TestResolveConstantChain_RefusesAnUnknownConstant(t *testing.T) { + ps := settingsWith(cfg("Default")) + + _, err := resolveConstantChain(ps, "", map[string]string{"A.Keyy": "v"}, knownTwo) + if err == nil { + t.Fatal("a constant that does not exist was accepted") + } + if !strings.Contains(err.Error(), "A.Keyy") { + t.Errorf("the error does not name the constant: %v", err) + } +} + +// A flag can be used precisely BECAUSE a private override has no value in the +// model. Reporting the constant as "private, default used" afterwards would +// contradict what the app is about to do. +func TestResolveConstantChain_FlagOverridesAPrivateValueAndStopsReportingIt(t *testing.T) { + ps := settingsWith(cfg("Default", private("A.Key"), private("A.Url"))) + + got, err := resolveConstantChain(ps, "", map[string]string{"A.Key": "v"}, knownTwo) + if err != nil { + t.Fatalf("resolveConstantChain: %v", err) + } + if got.Values["A.Key"] != "v" { + t.Errorf("the flag did not override the private value: %v", got.Values) + } + if len(got.Private) != 1 || got.Private[0] != "A.Url" { + t.Errorf("Private = %v, want only the one the flag did not cover", got.Private) + } +} + +// With no flags the chain has to behave exactly as the configuration-only +// resolution did — including refusing to guess between configurations. +func TestResolveConstantChain_WithoutFlagsMatchesTheConfigurationResolution(t *testing.T) { + ps := settingsWith(cfg("Acceptance", shared("A.Key", "acc")), cfg("Production", shared("A.Key", "prod"))) + + got, err := resolveConstantChain(ps, "", nil, knownTwo) + if err != nil { + t.Fatalf("resolveConstantChain: %v", err) + } + if len(got.Values) != 0 || !strings.Contains(got.Note, "--configuration") { + t.Errorf("got %+v, want no values and the hint", got) + } +} + +// An unknown name is refused even when the project has no configurations at +// all: the check is against what the project DECLARES, not against what some +// configuration happens to override. +func TestResolveConstantChain_ValidatesAgainstDeclaredConstantsNotOverrides(t *testing.T) { + if _, err := resolveConstantChain(&model.ProjectSettings{}, "", map[string]string{"A.Nope": "v"}, knownTwo); err == nil { + t.Error("an unknown constant was accepted because no configuration was present") + } + if _, err := resolveConstantChain(&model.ProjectSettings{}, "", map[string]string{"A.Key": "v"}, knownTwo); err != nil { + t.Errorf("a declared constant was refused: %v", err) + } +} + +// Silence used to mean "your override is in effect" when it was not. Every +// outcome has to print something, and now also say which layer won. +func TestReportConstantChain_SaysSomethingInEveryCase(t *testing.T) { + cases := []constantChain{ + {Configuration: "Default", Values: map[string]string{"A.B": "v"}, From: map[string]constantLayer{"A.B": layerConfiguration}}, + {Values: map[string]string{"A.B": "v"}, From: map[string]constantLayer{"A.B": layerFlag}}, + {Configuration: "Default", Values: map[string]string{}}, + {Values: map[string]string{}, Note: "the project has no configurations"}, + {Configuration: "Default", Values: map[string]string{}, Private: []string{"A.P"}}, + } + for i, c := range cases { + var buf bytes.Buffer + reportConstantChain(&buf, c) + if strings.TrimSpace(buf.String()) == "" { + t.Errorf("case %d printed nothing: %+v", i, c) + } + } +} + +func TestReportConstantChain_NamesTheLayer(t *testing.T) { + var buf bytes.Buffer + reportConstantChain(&buf, constantChain{ + Configuration: "Default", + Values: map[string]string{"A.Key": "sk-SECRETVALUE", "A.Url": "https://SECRETHOST"}, + From: map[string]constantLayer{"A.Key": layerFlag, "A.Url": layerConfiguration}, + }) + out := buf.String() + if !strings.Contains(out, "A.Key") || !strings.Contains(out, "--constant") { + t.Errorf("the flag-set constant is not attributed to --constant:\n%s", out) + } + if !strings.Contains(out, `configuration "Default"`) { + t.Errorf("the configuration-set constant is not attributed:\n%s", out) + } + // A constant can hold an API key, so the report names constants and layers + // and never prints a value. + for _, secret := range []string{"sk-SECRETVALUE", "SECRETHOST"} { + if strings.Contains(out, secret) { + t.Errorf("the report printed the value %q; only names and layers belong here:\n%s", secret, out) + } + } +} diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index 330f39ceb..57529af05 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -376,6 +376,7 @@ func init() { testRunCmd.Flags().Bool("skip-app-startup", false, "With --local, do not run the project's own after-startup microflow during the test run (it runs by default, so tests see the app as it really boots)") testRunCmd.Flags().Bool("attach", false, "Run against an app already started with 'mxcli run --local --test-endpoint' instead of booting one (tests hit that app's database)") testRunCmd.Flags().String("configuration", "", "With --local, which project configuration's constant values to run the tests with (default: the only one, or \"Default\") — the same resolution 'mxcli run --local' uses, so a suite sees the same constants either way") + testRunCmd.Flags().StringArray("constant", nil, "With --local, set a constant for THIS RUN only: Module.Name=value (repeatable). Never written to the project. The value is visible in shell history and in `ps` — for a value that must not be, see docs/11-proposals/PROPOSAL_constant_values.md") testRunCmd.Flags().BoolP("verbose", "v", false, "Show all runtime log output") testRunCmd.Flags().BoolP("color", "", false, "Use colored output") testRunCmd.Flags().StringP("timeout", "t", "5m", "Timeout for runtime startup and test execution") diff --git a/cmd/mxcli/runconstants.go b/cmd/mxcli/runconstants.go index 42abb7784..05649256c 100644 --- a/cmd/mxcli/runconstants.go +++ b/cmd/mxcli/runconstants.go @@ -4,30 +4,13 @@ package main import ( "fmt" - "io" + "sort" "strings" "github.com/mendixlabs/mxcli/model" ) -// constantOverridesFor reads the running configuration's constant values from a -// project. A project that cannot be read yields no overrides and a note — a run -// must not be blocked by this, since every run before it applied none. -func constantOverridesFor(projectPath, configuration string) constantOverrides { - b := newBackendFactory()() - if err := b.Connect(projectPath); err != nil { - return constantOverrides{Values: map[string]string{}, Note: "could not read the project's settings: " + err.Error()} - } - defer func() { _ = b.Disconnect() }() - - ps, err := b.GetProjectSettings() - if err != nil { - return constantOverrides{Values: map[string]string{}, Note: "could not read the project's settings: " + err.Error()} - } - return resolveConstantOverrides(ps, configuration) -} - // Constant values set per *configuration* never reached a locally-run app. // // mxbuild writes /model/config.json with each constant's **default** @@ -126,29 +109,3 @@ func configurationNames(cfgs []*model.ServerConfiguration) string { sort.Strings(names) return strings.Join(names, ", ") } - -// reportConstantOverrides says what will be applied before the app boots. -// -// It prints even when nothing is applied. The failure this fixes was invisible -// precisely because the run said nothing about constants either way, so silence -// has to stop meaning "your override is in effect". -func reportConstantOverrides(w io.Writer, o constantOverrides) { - switch { - case len(o.Values) > 0: - names := make([]string, 0, len(o.Values)) - for k := range o.Values { - names = append(names, k) - } - sort.Strings(names) - fmt.Fprintf(w, "Applying %d constant value(s) from configuration %q: %s\n", - len(names), o.Configuration, strings.Join(names, ", ")) - case o.Configuration != "": - fmt.Fprintf(w, "Configuration %q sets no constant values; using each constant's default.\n", o.Configuration) - case o.Note != "": - fmt.Fprintf(w, "Using each constant's default value (%s).\n", o.Note) - } - if len(o.Private) > 0 { - fmt.Fprintf(w, " %d override(s) are private, so their value is not in the model and the default is used:\n %s\n", - len(o.Private), strings.Join(o.Private, "\n ")) - } -} diff --git a/cmd/mxcli/runconstants_test.go b/cmd/mxcli/runconstants_test.go index 1b3c1d0bf..337a6f397 100644 --- a/cmd/mxcli/runconstants_test.go +++ b/cmd/mxcli/runconstants_test.go @@ -8,7 +8,6 @@ package main import ( - "bytes" "strings" "testing" @@ -103,21 +102,3 @@ func TestResolveConstantOverrides_ProjectWithNoConfigurations(t *testing.T) { t.Errorf("got %+v, want no values and a reason", got) } } - -// Silence used to mean "your override is in effect" when it was not. Every -// outcome has to print something. -func TestReportConstantOverrides_SaysSomethingInEveryCase(t *testing.T) { - cases := []constantOverrides{ - {Configuration: "Default", Values: map[string]string{"A.B": "v"}}, - {Configuration: "Default", Values: map[string]string{}}, - {Values: map[string]string{}, Note: "the project has no configurations"}, - {Configuration: "Default", Values: map[string]string{}, Private: []string{"A.P"}}, - } - for i, c := range cases { - var buf bytes.Buffer - reportConstantOverrides(&buf, c) - if strings.TrimSpace(buf.String()) == "" { - t.Errorf("case %d printed nothing: %+v", i, c) - } - } -} diff --git a/docs-site/src/tools/running-tests.md b/docs-site/src/tools/running-tests.md index 5c331c318..f55c13d52 100644 --- a/docs-site/src/tools/running-tests.md +++ b/docs-site/src/tools/running-tests.md @@ -15,7 +15,10 @@ and **Docker is only needed for the first**: A `--local` run boots the app with the same **constant values** `mxcli run --local` uses — the project configuration's shared overrides layered over each constant's default — and prints what it applied. Use `--configuration ` to choose -between several; `--attach` takes none, since it inherits the constants of the app +between several, and `--constant Module.Name=value` (repeatable) to set one for +this run only — it wins over the configuration and is never written to the +project. A constant the project does not declare is refused before anything +boots. `--attach` takes none of these, since it inherits the constants of the app it attaches to. This matters whenever a test asserts on something a constant feeds: the two modes used to disagree silently. diff --git a/docs/11-proposals/PROPOSAL_constant_values.md b/docs/11-proposals/PROPOSAL_constant_values.md index 265bb9e7c..95fe18b22 100644 --- a/docs/11-proposals/PROPOSAL_constant_values.md +++ b/docs/11-proposals/PROPOSAL_constant_values.md @@ -1,12 +1,12 @@ --- title: Constant values — one precedence chain, and a slot for secrets -status: proposed +status: accepted date: 2026-08-13 --- # Proposal: Constant values — one precedence chain, and a slot for secrets -**Status:** Proposed +**Status:** Accepted — slices 1 and 2 shipped; 3 and 4 open **Date:** 2026-08-13 A Mendix constant has a value in four possible places, mxcli can write two of @@ -196,7 +196,7 @@ machine", and a machine runs one thing at a time. If that proves wrong, a Four slices, each independently shippable and independently verifiable. -### Slice 1 — close the `test --local` gap (bug fix) +### Slice 1 — close the `test --local` gap (bug fix) — **shipped** The smallest correct change, and the one with a user-visible bug behind it. @@ -209,7 +209,7 @@ The smallest correct change, and the one with a user-visible bug behind it. Test: a `.test.mdl` asserting a constant, run under `--local` and under `--attach` against the same project, must agree. That test fails today. -### Slice 2 — layer 1, `--constant Module.Name=value` +### Slice 2 — layer 1, `--constant Module.Name=value` — **shipped** | File | Change | |------|--------| From 68505eef9ab2d46ad47343a0665d1903bfd73c03 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 12:22:40 +0000 Subject: [PATCH 12/20] Add a machine-local constant store, for a secret that must not be committed A constant's default and a shared configuration override both go to git. Mendix's own private configuration value is the correct slot and is unreachable: from 10.9 it is encrypted per user account by Studio Pro, so nothing headless can read or write it. Until now a value that had to persist without being committed had nowhere to live. mxcli constant set/unset/list writes /.mxcli/constants.json, mode 0600, sitting between --constant and the configuration. It is labelled as mxcli's own store rather than pretending to be Mendix's, and its security is file permissions, not encryption. The promise is made true and then checked. `mxcli init` writes a .gitignore only when the project has none, and a Mendix project usually already has one, so the entry the whole layer rests on could simply be absent: `constant set` appends it and then asks git whether the path is really ignored, refusing to write the value if it is not. Which rule defeats an entry is not guessable and was measured -- `!.mxcli/**` does not re-include anything, because git cannot re-include a file whose parent directory is excluded, while `!.mxcli` does. Two asymmetries worth knowing. A corrupt store is fatal, because it means values the author deliberately set are about to be silently absent; a stale entry naming a constant the project no longer declares is skipped and named, because refusing would fail every run until the user's own file was hand-edited. And an empty store is deleted rather than written as {} -- a file that configures nothing should not exist. Verified on 11.12.1 against a real runtime: with the project's Default configuration setting one value and the store another, a suite asserting the store's value passes; adding --constant makes the same suite pass on the flag's value instead. git status never sees the file. docs/11-proposals/PROPOSAL_constant_values.md slice 3. --- .claude/skills/fix-issue.md | 1 + .claude/skills/mendix/test-microflows.md | 24 +- cmd/mxcli/cmd_constant.go | 235 ++++++++++++++++++ cmd/mxcli/constant_gitignore.go | 97 ++++++++ cmd/mxcli/constant_gitignore_test.go | 135 ++++++++++ cmd/mxcli/constants_resolve.go | 84 +++++-- cmd/mxcli/constants_resolve_test.go | 87 ++++++- cmd/mxcli/constantstore/store.go | 117 +++++++++ cmd/mxcli/constantstore/store_test.go | 121 +++++++++ cmd/mxcli/init.go | 8 + cmd/mxcli/main.go | 1 + docs-site/src/tools/running-tests.md | 26 +- docs/11-proposals/PROPOSAL_constant_values.md | 4 +- 13 files changed, 905 insertions(+), 35 deletions(-) create mode 100644 cmd/mxcli/cmd_constant.go create mode 100644 cmd/mxcli/constant_gitignore.go create mode 100644 cmd/mxcli/constant_gitignore_test.go create mode 100644 cmd/mxcli/constantstore/store.go create mode 100644 cmd/mxcli/constantstore/store_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 397cf5d7d..a5e0203a5 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -499,3 +499,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli check --references` reports "java action not found: M.X (referenced by call java action)" for an action the **same script** creates a few statements earlier. Entities, microflows, pages and nanoflows in the same position are accepted | `scriptContext` — the set of objects a script defines, consulted so a script can be checked against its own output — had categories for modules, entities, enumerations, microflows, nanoflows, pages and snippets, but not for java or JavaScript actions. The java-action branch of `validateFlowBodyReferences` went straight to the project lookup | `mdl/executor/validate.go` (`scriptContext.javaActions`/`javaScriptActions`, `collectDefinitions`, `collectSingle`, `allNames`, `has`, `validateFlowBodyReferences`) | **Store the declared parameter names, not a bool.** Exempting the action from "not found" must not also exempt it from the parameter-name check — the script declares the parameters, so a typo is still catchable, and a bool would have silently dropped that. **`allNames()` and `has()` move together**: `annotateForwardRef` reads both, so adding a category to one and not the other makes a *created* object look "defined later in this script" forever. Tests `mdl/executor/validate_script_javaactions_test.go` (including the control that a misspelled parameter still errors). Reported in mxcli-chat FINDINGS §37 | | `GRANT ON Module.Specialization (READ *, WRITE *)` writes a model Mendix rejects with **CE0066** "Entity access is out of date", while the same grant on the generalization checks clean. The executor passes 3 MemberAccess entries and storage holds 2 | `ReconcileMemberAccesses` — which runs after every program and after every `create association` — recomputes each rule's expected member set from the entity's **own** attributes and the module's **FROM-side** associations. An inherited association matched neither, so it was classed stale and deleted, one step after the GRANT that had just written it correctly | `mdl/backend/modelsdk/domainmodel_security_write.go` (`sameModuleAncestors`, `entitiesByName`, `assocRefBelongsTo`, `ownerIDs` in `ReconcileMemberAccesses`), `mdl/executor/cmd_associations.go` (reconcile after DROP) | **The write path was innocent** — `AddEntityAccessRule` stored all 3; the deletion happened in the reconcile that ran next. When a value is written and then absent, instrument the *later* pass before the writer. **Preserve what cannot be checked**, as the attribute branch already did (#758): an association is qualified by the module that DECLARES it, so an ancestor in another module (the reported `OpenAIDeployedModel extends GenAICommons.DeployedModel`) names a domain model that is not loaded here. **The counter-control is the whole test**: a walk that collected every association in the module passes both positive cases and puts an entry on the TO side of an `OWNER Default` association, which is *itself* CE0066 — so assert that the unrelated entity does **not** get one. **Reconcile must add as well as keep**, or a rule written before the ancestor gained the association never catches up. Verifying this surfaced a neighbouring bug: `drop association` never reconciled at all, leaving entries Mendix rejects with CE1613 — on the declaring entity too, so it predates inheritance support. Tests `mdl/backend/modelsdk/reconcile_inherited_assoc_test.go` (5 cases incl. both controls), `mdl/executor/cmd_associations_mock_test.go`; example `mdl-examples/bug-tests/grant-on-specialization.mdl`, measured 1 error → 0 on mxbuild 11.12.1. Reported in mxcli-chat FINDINGS §25 | | A `.test.mdl` asserting on something a **constant** feeds passes under `mxcli test --attach` and fails under `mxcli test --local` (or the reverse), with nothing in either output to explain the difference | `--local` boots an app of its own through `StartLocalApp`, whose options had no `ConstantOverrides` field — so it ran with each constant's **default** from `deployment/model/config.json`, while `--attach` runs against an app `run --local` booted, which applies the configuration's shared overrides. A constant resolving to the wrong value is not an error, so both runs reported success and only the assertion differed | `cmd/mxcli/docker/localapp.go` (`LocalAppOptions.ConstantOverrides`, `runtimeOptions`), `cmd/mxcli/testrunner/localapp_options.go` (new, shared by both `--local` runners), `runner.go` (`RunOptions.ConstantOverrides`), `cmd/mxcli/cmd_test_run.go` + `main.go` (`--configuration`) | **A boot path that takes options needs one mapping function, not an inline struct literal per caller** — the two `--local` runners each built their own `LocalAppOptions`, so a field added for one would not reach the other, and neither had the constants. Both now go through `localAppOptions`, and the `LocalAppOptions`→`LocalRuntimeOptions` step is `runtimeOptions()` so the forwarding is assertable without booting anything: a dropped field there is otherwise invisible until an app runs with the wrong configuration. **Resolve in one place**: `cmd/` decides precedence and the runner only carries the map, or "which configuration wins" gets two answers. **Only report what this run actually uses** — `--attach` inherits the constants of the app it attached to, so resolving and printing them there would be a confident lie. Verified at the layer the symptom lives in (`.claude/skills/verify-in-runtime.md`): the same suite, one project, `--local` and `--attach` must agree, with the reverted-wiring control run showing the constant's default. Unit tests `cmd/mxcli/testrunner/localapp_options_test.go`, `cmd/mxcli/docker/localboot_constants_test.go`. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 1 | +| A constant value that must not be committed has nowhere to live: a constant's default and a shared configuration override both go to git, and Mendix's own **private** configuration value — the correct slot — is encrypted per user account by Studio Pro from 10.9, so nothing headless can read or write it | Not a defect so much as an absent layer. mxcli mirrors the concept with a store it can actually reach: `/.mxcli/constants.json`, mode 0600, gitignored, sitting between `--constant` and the configuration | `cmd/mxcli/constantstore/` (load/save), `cmd/mxcli/cmd_constant.go` (`constant set/unset/list`), `cmd/mxcli/constant_gitignore.go` (`ensureStoreIgnored`), `cmd/mxcli/constants_resolve.go` (`layerMachine`) | **The promise has to be made true, then checked — asserting it is not enough.** `mxcli init` writes a `.gitignore` only when the project has none, and a Mendix project usually already has one, so the entry the whole layer rests on could simply be absent. `constant set` appends it *and then asks git*, refusing to write the value on "not ignored": a store that leaks is worse than no store. **Which rule defeats an ignore entry is not guessable** — `!.mxcli/**` does NOT re-include anything (git cannot re-include a file whose parent directory is excluded), while `!.mxcli` does; the test was corrected against measured git behaviour rather than the assumed case. **A corrupt store is an error, a stale entry is not**: an unparseable file means values the author set are about to be silently absent (fatal), while an entry naming a constant the project dropped is the user's own file and is skipped-and-named, or every run fails until it is hand-edited. **An empty store is deleted, not written as `{}`** — a file that configures nothing should not exist. Tests `cmd/mxcli/constantstore/store_test.go`, `cmd/mxcli/constant_gitignore_test.go`; verified live that the value reaches a running app and that `git status` never sees the file. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 3 | diff --git a/.claude/skills/mendix/test-microflows.md b/.claude/skills/mendix/test-microflows.md index 1e2c74730..150e8aae0 100644 --- a/.claude/skills/mendix/test-microflows.md +++ b/.claude/skills/mendix/test-microflows.md @@ -40,7 +40,8 @@ uses**: the project configuration's shared overrides, layered over each constant's default. It prints what it applied before the run: ``` -Applying 1 constant value(s) from configuration "Default": MyModule.ApiKey +Applying 1 constant value(s): + MyModule.ApiKey configuration "Default" ``` Pass `--configuration ` to pick one when the project has several and none @@ -67,11 +68,22 @@ this was wired up, `--local` ran with each constant's *default* while `--attach` ran with the configuration's, so the same suite could pass one way and fail the other with nothing in the output to explain it. -For a secret that has to **persist** across runs there is still nowhere safe: a -constant's default and a shared configuration override are both committed, and -Mendix's own private values are encrypted per user account by Studio Pro and -unreachable headlessly. `--constant` covers the one-run case only. See -`docs/11-proposals/PROPOSAL_constant_values.md`. +For a secret that has to **persist** across runs, use the machine store: + +```bash +mxcli constant set MyModule.ApiKey 'sk-live-...' -p app.mpr +mxcli constant list -p app.mpr # values from the store are masked +mxcli constant unset MyModule.ApiKey -p app.mpr +``` + +It writes `/.mxcli/constants.json` (mode 0600), adds `.mxcli/` to the +project's `.gitignore` if missing, and then **asks git whether the path is +really ignored** — refusing to write the value if it is not. It beats the +configuration and loses to `--constant`. + +This is mxcli's own store, not Mendix's. Mendix's private configuration values +are encrypted per user account by Studio Pro from 10.9, so nothing headless can +read or write them. See `docs/11-proposals/PROPOSAL_constant_values.md`. --- diff --git a/cmd/mxcli/cmd_constant.go b/cmd/mxcli/cmd_constant.go new file mode 100644 index 000000000..e3736e4c8 --- /dev/null +++ b/cmd/mxcli/cmd_constant.go @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/cmd/mxcli/constantstore" + "github.com/spf13/cobra" +) + +var constantCmd = &cobra.Command{ + Use: "constant", + Short: "Set constant values that stay on this machine, and see which value a run will use", + Long: `Manage the constant values that belong to THIS MACHINE. + +They live in /.mxcli/constants.json, mode 0600. 'constant set' adds +.mxcli/ to the project's .gitignore if it is missing and then asks git whether +the path is really ignored — refusing to write the value if it is not, because +a store that leaks is worse than none. It is the slot for an API key or a +connection string that must not reach version control. + +This is mxcli's own store, not Mendix's. Mendix has the same concept — a +configuration value marked private — but from 10.9 it is encrypted per user +account by Studio Pro, so nothing headless can read or write it. mxcli mirrors +the idea with a file it can actually reach, at the security of file permissions +rather than encryption. + +Where a value comes from, highest first: + + 1. --constant Module.Name=value this run only, written nowhere + 2. this store this machine, gitignored + 3. the project configuration shared, in git ('alter settings constant') + 4. the constant's default shared, in git + +'mxcli constant list' shows the winner for every constant and which layer set +it. Values from this store are masked unless you pass --show-values.`, +} + +var constantSetCmd = &cobra.Command{ + Use: "set ", + Short: "Set a constant's value on this machine (gitignored, never committed)", + Args: cobra.ExactArgs(2), + Example: ` mxcli constant set MyModule.ApiKey 'sk-live-...' -p app.mpr + mxcli constant list -p app.mpr`, + Run: func(cmd *cobra.Command, args []string) { + projectPath := requireProjectPath(cmd) + name, value := args[0], args[1] + + read, err := projectConstantDefaults(projectPath) + if err != nil { + exitf("could not read the project's constants: %v", err) + } + if _, ok := read.defaults[name]; !ok { + exitf("no constant named %s in this project\n"+ + " hint: 'mxcli constant list -p %s' shows the ones there are; a value for a "+ + "constant that does not exist is ignored by the runtime, so this would apply to nothing", + name, projectPath) + } + + // Establish the promise BEFORE writing the value. A secret written into a + // path git would commit is worse than no store at all. + switch status, err := ensureStoreIgnored(projectPath); { + case err != nil: + exitf("could not make %s git-ignored, so the value was not written: %v", + constantstore.Path(projectPath), err) + case status == ignoreNotIgnored: + exitf("git says %s would still be committed even after adding .mxcli/ to .gitignore\n"+ + " (something later in the ignore rules re-includes it, or the path is already tracked)\n"+ + " the value was NOT written — this store exists to keep values out of version control", + constantstore.Path(projectPath)) + case status == ignoreUnverified: + fmt.Printf("note: could not ask git whether %s is ignored (no git, or not a repository)\n", + filepath.Dir(constantstore.Path(projectPath))) + } + + store, err := constantstore.Load(projectPath) + if err != nil { + exitf("%v", err) + } + store.Constants[name] = value + if err := constantstore.Save(projectPath, store); err != nil { + exitf("%v", err) + } + fmt.Printf("Set %s in %s (this machine only; not in version control)\n", + name, constantstore.Path(projectPath)) + + // Naming the shadowed layer matters: the author may have just set a value + // here that the team's configuration also sets, and silently winning is + // how "but the configuration says X" starts. + if chain, err := resolveConstantChain(read.settings, "", nil, nil, nil); err == nil { + if _, alsoShared := chain.Values[name]; alsoShared { + fmt.Printf(" note: configuration %q also sets %s; this machine's value wins\n", + chain.Configuration, name) + } + } + }, +} + +var constantUnsetCmd = &cobra.Command{ + Use: "unset ", + Short: "Remove a constant's machine-local value", + Args: cobra.ExactArgs(1), + Run: func(cmd *cobra.Command, args []string) { + projectPath := requireProjectPath(cmd) + name := args[0] + + store, err := constantstore.Load(projectPath) + if err != nil { + exitf("%v", err) + } + if _, ok := store.Constants[name]; !ok { + fmt.Printf("%s has no machine-local value; nothing to remove.\n", name) + return + } + delete(store.Constants, name) + if err := constantstore.Save(projectPath, store); err != nil { + exitf("%v", err) + } + fmt.Printf("Removed the machine-local value for %s; runs now use the configuration or the default.\n", name) + }, +} + +var constantListCmd = &cobra.Command{ + Use: "list", + Short: "Show the value each constant resolves to, and which layer set it", + Run: func(cmd *cobra.Command, args []string) { + projectPath := requireProjectPath(cmd) + configuration, _ := cmd.Flags().GetString("configuration") + showValues, _ := cmd.Flags().GetBool("show-values") + + read, err := projectConstantDefaults(projectPath) + if err != nil { + exitf("could not read the project's constants: %v", err) + } + store, err := constantstore.Load(projectPath) + if err != nil { + exitf("%v", err) + } + known := make(map[string]bool, len(read.defaults)) + for n := range read.defaults { + known[n] = true + } + chain, err := resolveConstantChain(read.settings, configuration, nil, store.Constants, known) + if err != nil { + exitf("%v", err) + } + + names := make([]string, 0, len(read.defaults)) + for n := range read.defaults { + names = append(names, n) + } + sort.Strings(names) + private := map[string]bool{} + for _, n := range chain.Private { + private[n] = true + } + + nameW, valueW := len("CONSTANT"), len("VALUE") + type row struct{ name, value, from string } + rows := make([]row, 0, len(names)) + for _, n := range names { + value, from := read.defaults[n], "default" + if v, ok := chain.Values[n]; ok { + value = v + from = string(chain.From[n]) + if chain.From[n] == layerConfiguration { + from = fmt.Sprintf("configuration %q", chain.Configuration) + } + // A value from this store is the one that exists BECAUSE it should not + // be seen; printing it into a terminal transcript by default would + // undo the point of the layer. + if chain.From[n] == layerMachine && !showValues { + value = "****" + } + } + if private[n] { + from = "default (a private override exists; its value is not in the model)" + } + rows = append(rows, row{n, value, from}) + if len(n) > nameW { + nameW = len(n) + } + if len(value) > valueW { + valueW = len(value) + } + } + + fmt.Printf("%-*s %-*s %s\n", nameW, "CONSTANT", valueW, "VALUE", "FROM") + for _, r := range rows { + fmt.Printf("%-*s %-*s %s\n", nameW, r.name, valueW, r.value, r.from) + } + if len(rows) == 0 { + fmt.Println("(this project declares no constants)") + } + if chain.Note != "" { + fmt.Printf("\nNote: %s\n", chain.Note) + } + if len(chain.Stale) > 0 { + fmt.Printf("\n%d value(s) in %s name no constant of this project and are skipped:\n %s\n", + len(chain.Stale), constantstore.FileName, strings.Join(chain.Stale, "\n ")) + } + if !showValues && len(store.Constants) > 0 { + fmt.Println("\nMachine-local values are masked; pass --show-values to print them.") + } + }, +} + +// requireProjectPath resolves -p, exiting with the same message every other +// project-scoped command uses. +func requireProjectPath(cmd *cobra.Command) string { + projectPath, _ := cmd.Flags().GetString("project") + if projectPath == "" { + fmt.Fprintln(os.Stderr, "Error: --project (-p) is required") + os.Exit(1) + } + return projectPath +} + +func exitf(format string, a ...any) { + fmt.Fprintf(os.Stderr, "Error: "+format+"\n", a...) + os.Exit(1) +} + +func init() { + constantListCmd.Flags().String("configuration", "", + "Which configuration's values to resolve against (default: the only one, or \"Default\")") + constantListCmd.Flags().Bool("show-values", false, + "Print machine-local values instead of masking them") + constantCmd.AddCommand(constantSetCmd, constantUnsetCmd, constantListCmd) +} diff --git a/cmd/mxcli/constant_gitignore.go b/cmd/mxcli/constant_gitignore.go new file mode 100644 index 000000000..aa3b3ff79 --- /dev/null +++ b/cmd/mxcli/constant_gitignore.go @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" +) + +// ignoreStatus is what we could establish about the machine store's path. +type ignoreStatus int + +const ( + ignoreConfirmed ignoreStatus = iota // git says the path is ignored + ignoreUnverified // no git, or not a repository — cannot tell + ignoreNotIgnored // git says the path WOULD be committed +) + +// gitignoreEntry is what mxcli adds, with the reason attached. `.mxcli/` also +// holds the run/test handshake and logs, none of which belong in a repository. +const gitignoreEntry = "\n# mxcli working directory: machine-local constant values (may contain\n" + + "# secrets), the run/test handshake, and runtime logs. Never commit these.\n.mxcli/\n" + +// ensureStoreIgnored makes the machine store's directory git-ignored, then +// checks that it actually is. +// +// Both halves matter. `mxcli init` writes a .gitignore only when the project has +// none, and a Mendix project usually already has one — so the entry this layer's +// whole promise rests on ("never committed, never shared") could simply be +// absent. Adding it is not enough either: a later negation rule, or a path +// already tracked, can defeat it, and the only authority on that is git itself. +// +// A caller writing a secret must refuse on ignoreNotIgnored. Unverified is not +// the same thing — a project outside version control has nothing to leak into. +func ensureStoreIgnored(projectPath string) (ignoreStatus, error) { + dir := filepath.Dir(projectPath) + path := filepath.Join(dir, ".gitignore") + + body, err := os.ReadFile(path) + switch { + case os.IsNotExist(err): + if err := os.WriteFile(path, []byte(strings.TrimPrefix(gitignoreEntry, "\n")), 0o644); err != nil { + return ignoreUnverified, fmt.Errorf("creating %s: %w", path, err) + } + case err != nil: + return ignoreUnverified, fmt.Errorf("reading %s: %w", path, err) + case !mentionsMxcliDir(string(body)): + content := string(body) + if !strings.HasSuffix(content, "\n") { + content += "\n" + } + if err := os.WriteFile(path, []byte(content+gitignoreEntry), 0o644); err != nil { + return ignoreUnverified, fmt.Errorf("appending to %s: %w", path, err) + } + } + return checkIgnored(dir, filepath.Join(dir, ".mxcli", "x")), nil +} + +// mentionsMxcliDir reports whether a .gitignore already covers `.mxcli/`. It is +// deliberately a cheap check for "did we already add our line" — whether the +// path is REALLY ignored is git's answer, not this one's. +func mentionsMxcliDir(body string) bool { + for _, line := range strings.Split(body, "\n") { + switch strings.TrimSpace(line) { + case ".mxcli/", ".mxcli", "/.mxcli/", "/.mxcli": + return true + } + } + return false +} + +// checkIgnored asks git whether a path is ignored. Exit 0 means ignored, 1 means +// not, anything else (no git, not a repository) means we cannot tell. +func checkIgnored(dir, path string) ignoreStatus { + cmd := exec.Command("git", "check-ignore", "-q", path) + cmd.Dir = dir + err := cmd.Run() + if err == nil { + return ignoreConfirmed + } + var exitErr *exec.ExitError + if ok := asExitError(err, &exitErr); ok && exitErr.ExitCode() == 1 { + return ignoreNotIgnored + } + return ignoreUnverified +} + +func asExitError(err error, target **exec.ExitError) bool { + e, ok := err.(*exec.ExitError) + if ok { + *target = e + } + return ok +} diff --git a/cmd/mxcli/constant_gitignore_test.go b/cmd/mxcli/constant_gitignore_test.go new file mode 100644 index 000000000..daa1b1d3c --- /dev/null +++ b/cmd/mxcli/constant_gitignore_test.go @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: Apache-2.0 + +// The machine store's promise — "never committed, never shared" — rests +// entirely on .mxcli/ being git-ignored. `mxcli init` writes a .gitignore only +// when the project has none, and a Mendix project usually already has one, so +// the entry could simply be absent. These tests cover making it true and then +// checking it, because a store that leaks is worse than no store at all. +package main + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +func gitProject(t *testing.T, gitignore string) string { + t.Helper() + dir := t.TempDir() + if gitignore != "" { + if err := os.WriteFile(filepath.Join(dir, ".gitignore"), []byte(gitignore), 0o644); err != nil { + t.Fatal(err) + } + } + for _, args := range [][]string{{"init", "-q"}, {"config", "user.email", "t@t"}, {"config", "user.name", "t"}} { + cmd := exec.Command("git", args...) + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil { + t.Skipf("git unavailable: %v (%s)", err, out) + } + } + return filepath.Join(dir, "App.mpr") +} + +func readIgnore(t *testing.T, projectPath string) string { + t.Helper() + body, err := os.ReadFile(filepath.Join(filepath.Dir(projectPath), ".gitignore")) + if err != nil { + t.Fatalf("reading .gitignore: %v", err) + } + return string(body) +} + +func TestEnsureStoreIgnored_AppendsToAnExistingGitignore(t *testing.T) { + p := gitProject(t, "deployment/\n*.mpr.bak\n") + + got, err := ensureStoreIgnored(p) + if err != nil { + t.Fatalf("ensureStoreIgnored: %v", err) + } + if got != ignoreConfirmed { + t.Errorf("status = %v, want confirmed", got) + } + body := readIgnore(t, p) + if !strings.Contains(body, ".mxcli/") { + t.Errorf(".mxcli/ was not added:\n%s", body) + } + // The lines that were already there stay there. + if !strings.Contains(body, "deployment/") || !strings.Contains(body, "*.mpr.bak") { + t.Errorf("existing entries were lost:\n%s", body) + } +} + +func TestEnsureStoreIgnored_CreatesAGitignoreWhenThereIsNone(t *testing.T) { + p := gitProject(t, "") + + if got, err := ensureStoreIgnored(p); err != nil || got != ignoreConfirmed { + t.Fatalf("got %v / %v, want confirmed", got, err) + } + if !strings.Contains(readIgnore(t, p), ".mxcli/") { + t.Error(".mxcli/ missing from the created .gitignore") + } +} + +// Running twice must not append the block twice. +func TestEnsureStoreIgnored_IsIdempotent(t *testing.T) { + p := gitProject(t, "deployment/\n") + + for range 3 { + if _, err := ensureStoreIgnored(p); err != nil { + t.Fatalf("ensureStoreIgnored: %v", err) + } + } + if n := strings.Count(readIgnore(t, p), ".mxcli/"); n != 1 { + t.Errorf(".mxcli/ appears %d times, want 1", n) + } +} + +// The check that matters: adding the line is not proof, so git is asked. +// +// Which rule defeats it is not obvious and was measured rather than assumed — +// `!.mxcli/**` does NOT re-include anything, because git cannot re-include a +// file whose parent directory is excluded. `!.mxcli` unexcludes the directory +// itself, and then the file inside is not ignored. That is the case a caller +// writing a secret has to refuse on. +func TestEnsureStoreIgnored_ReportsNotIgnoredWhenTheDirectoryIsReIncluded(t *testing.T) { + p := gitProject(t, ".mxcli/\n!.mxcli\n") + + got, err := ensureStoreIgnored(p) + if err != nil { + t.Fatalf("ensureStoreIgnored: %v", err) + } + if got != ignoreNotIgnored { + t.Fatalf("status = %v, want notIgnored — a negation rule defeats the entry, and "+ + "writing a secret there would leak it", got) + } +} + +// Outside a repository there is nothing to leak into, so this is "cannot tell", +// not "unsafe" — the caller proceeds and says so. +func TestEnsureStoreIgnored_UnverifiedOutsideARepository(t *testing.T) { + p := filepath.Join(t.TempDir(), "App.mpr") + + got, err := ensureStoreIgnored(p) + if err != nil { + t.Fatalf("ensureStoreIgnored: %v", err) + } + if got != ignoreUnverified { + t.Errorf("status = %v, want unverified outside a git repository", got) + } +} + +func TestMentionsMxcliDir(t *testing.T) { + for _, yes := range []string{".mxcli/", ".mxcli", "/.mxcli/", " .mxcli/ "} { + if !mentionsMxcliDir("a\n" + yes + "\nb") { + t.Errorf("mentionsMxcliDir(%q) = false", yes) + } + } + for _, no := range []string{".mxclifoo/", "x.mxcli/", "# .mxcli/"} { + if mentionsMxcliDir(no) { + t.Errorf("mentionsMxcliDir(%q) = true", no) + } + } +} diff --git a/cmd/mxcli/constants_resolve.go b/cmd/mxcli/constants_resolve.go index d8e651569..cb3633a18 100644 --- a/cmd/mxcli/constants_resolve.go +++ b/cmd/mxcli/constants_resolve.go @@ -8,6 +8,7 @@ import ( "sort" "strings" + "github.com/mendixlabs/mxcli/cmd/mxcli/constantstore" "github.com/mendixlabs/mxcli/mdl/backend" "github.com/mendixlabs/mxcli/mdl/executor" "github.com/mendixlabs/mxcli/model" @@ -22,6 +23,7 @@ type constantLayer string const ( layerFlag constantLayer = "--constant" + layerMachine constantLayer = "this machine" layerConfiguration constantLayer = "configuration" ) @@ -37,6 +39,7 @@ type constantChain struct { Values map[string]string // constant qualified name -> value From map[string]constantLayer // and which layer set it Private []string // overrides whose value is not in the model + Stale []string // machine-store entries naming no constant of this project Note string // why no configuration contributed } @@ -73,7 +76,7 @@ func parseConstantFlags(flags []string) (map[string]string, error) { // ignores a MicroflowConstants entry that matches no constant, so a typo would // otherwise be accepted, reported as applied, and do nothing — the §33 failure // shape, reintroduced by the very flag meant to fix it. -func resolveConstantChain(ps *model.ProjectSettings, want string, flags map[string]string, known map[string]bool) (constantChain, error) { +func resolveConstantChain(ps *model.ProjectSettings, want string, flags, machine map[string]string, known map[string]bool) (constantChain, error) { base := resolveConstantOverrides(ps, want) chain := constantChain{ Configuration: base.Configuration, @@ -101,17 +104,34 @@ func resolveConstantChain(ps *model.ProjectSettings, want string, flags map[stri strings.Join(unknown, ", ")) } + // A constant the project no longer declares must not be applied from the + // machine store either — but a stale entry there is the user's own file, not + // a typo they can fix by rerunning, so it is skipped and named rather than + // refused. Refusing would make every run fail until the file was edited by + // hand, over a value the project stopped caring about. + var stale []string + for name, value := range machine { + if known != nil && !known[name] { + stale = append(stale, name) + continue + } + chain.Values[name] = value + chain.From[name] = layerMachine + } + sort.Strings(stale) + chain.Stale = stale + for name, value := range flags { chain.Values[name] = value chain.From[name] = layerFlag } - // A flag overrides a private configuration value too, and then the default is - // NOT what runs — so the constant must stop being reported as private-and- - // defaulted, or the report contradicts what the app will do. - if len(flags) > 0 && len(chain.Private) > 0 { + // A value set above the configuration means the default is NOT what runs, so + // the constant must stop being reported as private-and-defaulted, or the + // report contradicts what the app is about to do. + if len(chain.Private) > 0 { kept := chain.Private[:0] for _, name := range chain.Private { - if _, overridden := flags[name]; !overridden { + if _, overridden := chain.Values[name]; !overridden { kept = append(kept, name) } } @@ -128,21 +148,49 @@ func resolveConstantChain(ps *model.ProjectSettings, want string, flags map[stri // named something specific, and applying it unvalidated (or dropping it // silently) are both worse than saying the project could not be read. func constantChainFor(projectPath, configuration string, flags map[string]string) (constantChain, error) { + // The machine store is read FIRST and its failure is always fatal. Unlike a + // project that cannot be read — where falling back to the defaults is what + // every earlier run did anyway — a store that exists and cannot be parsed + // means values the author deliberately set are about to be silently absent. + store, err := constantstore.Load(projectPath) + if err != nil { + return constantChain{}, err + } + + defaults, err := projectConstantDefaults(projectPath) + if err != nil { + return unreadableProject(err, flags) + } + known := make(map[string]bool, len(defaults.defaults)) + for name := range defaults.defaults { + known[name] = true + } + return resolveConstantChain(defaults.settings, configuration, flags, store.Constants, known) +} + +// projectRead is what one open of the project yields: its settings and every +// constant it declares, with each default value. +type projectRead struct { + settings *model.ProjectSettings + defaults map[string]string // constant qualified name -> default value +} + +func projectConstantDefaults(projectPath string) (projectRead, error) { b := newBackendFactory()() if err := b.Connect(projectPath); err != nil { - return unreadableProject(err, flags) + return projectRead{}, err } defer func() { _ = b.Disconnect() }() ps, err := b.GetProjectSettings() if err != nil { - return unreadableProject(err, flags) + return projectRead{}, err } - known, err := knownConstantNames(b) + defaults, err := constantDefaults(b) if err != nil { - return unreadableProject(err, flags) + return projectRead{}, err } - return resolveConstantChain(ps, configuration, flags, known) + return projectRead{settings: ps, defaults: defaults}, nil } func unreadableProject(err error, flags map[string]string) (constantChain, error) { @@ -156,10 +204,10 @@ func unreadableProject(err error, flags map[string]string) (constantChain, error }, nil } -// knownConstantNames returns the qualified names of every constant the project -// declares. Folders do not appear in a qualified name, so the hierarchy is used +// constantDefaults maps every constant the project declares to its default +// value. Folders do not appear in a qualified name, so the hierarchy is used // only to find each constant's module. -func knownConstantNames(b backend.FullBackend) (map[string]bool, error) { +func constantDefaults(b backend.FullBackend) (map[string]string, error) { constants, err := b.ListConstants() if err != nil { return nil, err @@ -168,10 +216,10 @@ func knownConstantNames(b backend.FullBackend) (map[string]bool, error) { if err != nil { return nil, err } - out := make(map[string]bool, len(constants)) + out := make(map[string]string, len(constants)) for _, c := range constants { if module := h.GetModuleName(h.FindModuleID(c.ContainerID)); module != "" { - out[module+"."+c.Name] = true + out[module+"."+c.Name] = c.DefaultValue } } return out, nil @@ -212,4 +260,8 @@ func reportConstantChain(w io.Writer, c constantChain) { fmt.Fprintf(w, " %d override(s) are private, so their value is not in the model and the default is used:\n %s\n", len(c.Private), strings.Join(c.Private, "\n ")) } + if len(c.Stale) > 0 { + fmt.Fprintf(w, " %d value(s) in %s name no constant of this project and were skipped:\n %s\n", + len(c.Stale), constantstore.FileName, strings.Join(c.Stale, "\n ")) + } } diff --git a/cmd/mxcli/constants_resolve_test.go b/cmd/mxcli/constants_resolve_test.go index cb9d3865e..2c42f03ff 100644 --- a/cmd/mxcli/constants_resolve_test.go +++ b/cmd/mxcli/constants_resolve_test.go @@ -52,7 +52,7 @@ func TestParseConstantFlags_RejectsMalformed(t *testing.T) { func TestResolveConstantChain_FlagWinsOverTheConfiguration(t *testing.T) { ps := settingsWith(cfg("Default", shared("A.Key", "from-configuration"), shared("A.Url", "u"))) - got, err := resolveConstantChain(ps, "", map[string]string{"A.Key": "from-the-flag"}, knownTwo) + got, err := resolveConstantChain(ps, "", map[string]string{"A.Key": "from-the-flag"}, nil, knownTwo) if err != nil { t.Fatalf("resolveConstantChain: %v", err) } @@ -74,7 +74,7 @@ func TestResolveConstantChain_FlagWinsOverTheConfiguration(t *testing.T) { func TestResolveConstantChain_RefusesAnUnknownConstant(t *testing.T) { ps := settingsWith(cfg("Default")) - _, err := resolveConstantChain(ps, "", map[string]string{"A.Keyy": "v"}, knownTwo) + _, err := resolveConstantChain(ps, "", map[string]string{"A.Keyy": "v"}, nil, knownTwo) if err == nil { t.Fatal("a constant that does not exist was accepted") } @@ -89,7 +89,7 @@ func TestResolveConstantChain_RefusesAnUnknownConstant(t *testing.T) { func TestResolveConstantChain_FlagOverridesAPrivateValueAndStopsReportingIt(t *testing.T) { ps := settingsWith(cfg("Default", private("A.Key"), private("A.Url"))) - got, err := resolveConstantChain(ps, "", map[string]string{"A.Key": "v"}, knownTwo) + got, err := resolveConstantChain(ps, "", map[string]string{"A.Key": "v"}, nil, knownTwo) if err != nil { t.Fatalf("resolveConstantChain: %v", err) } @@ -106,7 +106,7 @@ func TestResolveConstantChain_FlagOverridesAPrivateValueAndStopsReportingIt(t *t func TestResolveConstantChain_WithoutFlagsMatchesTheConfigurationResolution(t *testing.T) { ps := settingsWith(cfg("Acceptance", shared("A.Key", "acc")), cfg("Production", shared("A.Key", "prod"))) - got, err := resolveConstantChain(ps, "", nil, knownTwo) + got, err := resolveConstantChain(ps, "", nil, nil, knownTwo) if err != nil { t.Fatalf("resolveConstantChain: %v", err) } @@ -119,10 +119,10 @@ func TestResolveConstantChain_WithoutFlagsMatchesTheConfigurationResolution(t *t // all: the check is against what the project DECLARES, not against what some // configuration happens to override. func TestResolveConstantChain_ValidatesAgainstDeclaredConstantsNotOverrides(t *testing.T) { - if _, err := resolveConstantChain(&model.ProjectSettings{}, "", map[string]string{"A.Nope": "v"}, knownTwo); err == nil { + if _, err := resolveConstantChain(&model.ProjectSettings{}, "", map[string]string{"A.Nope": "v"}, nil, knownTwo); err == nil { t.Error("an unknown constant was accepted because no configuration was present") } - if _, err := resolveConstantChain(&model.ProjectSettings{}, "", map[string]string{"A.Key": "v"}, knownTwo); err != nil { + if _, err := resolveConstantChain(&model.ProjectSettings{}, "", map[string]string{"A.Key": "v"}, nil, knownTwo); err != nil { t.Errorf("a declared constant was refused: %v", err) } } @@ -168,3 +168,78 @@ func TestReportConstantChain_NamesTheLayer(t *testing.T) { } } } + +// Layer 2 — the machine store. It sits between --constant and the +// configuration: higher than the shared value the team committed, lower than +// what this invocation asked for. +func TestResolveConstantChain_MachineStoreBeatsConfigurationAndLosesToFlag(t *testing.T) { + ps := settingsWith(cfg("Default", shared("A.Key", "shared"), shared("A.Url", "shared-url"))) + machine := map[string]string{"A.Key": "machine", "A.Url": "machine-url"} + + got, err := resolveConstantChain(ps, "", map[string]string{"A.Key": "flag"}, machine, knownTwo) + if err != nil { + t.Fatalf("resolveConstantChain: %v", err) + } + if got.Values["A.Key"] != "flag" || got.From["A.Key"] != layerFlag { + t.Errorf("A.Key = %q from %q, want the flag to win", got.Values["A.Key"], got.From["A.Key"]) + } + if got.Values["A.Url"] != "machine-url" || got.From["A.Url"] != layerMachine { + t.Errorf("A.Url = %q from %q, want the machine store to beat the configuration", + got.Values["A.Url"], got.From["A.Url"]) + } +} + +// A stale machine entry is the user's own file, not a typo they can fix by +// rerunning. Refusing would make every run fail until the file was hand-edited, +// over a value the project stopped declaring — so it is skipped and named, +// unlike a --constant flag, which IS refused. +func TestResolveConstantChain_SkipsAndNamesAStaleMachineEntry(t *testing.T) { + machine := map[string]string{"A.Key": "v", "A.Removed": "old"} + + got, err := resolveConstantChain(settingsWith(cfg("Default")), "", nil, machine, knownTwo) + if err != nil { + t.Fatalf("a stale machine entry should not fail the run: %v", err) + } + if got.Values["A.Key"] != "v" { + t.Errorf("the still-valid entry was dropped: %v", got.Values) + } + if _, applied := got.Values["A.Removed"]; applied { + t.Error("a value for a constant the project no longer declares was applied") + } + if len(got.Stale) != 1 || got.Stale[0] != "A.Removed" { + t.Errorf("Stale = %v, want it named so the user can clean it up", got.Stale) + } +} + +// The private-override note must not survive ANY layer above it, not just a +// flag: a machine value means the default is not what runs either. +func TestResolveConstantChain_MachineValueAlsoClearsThePrivateNote(t *testing.T) { + ps := settingsWith(cfg("Default", private("A.Key"), private("A.Url"))) + + got, err := resolveConstantChain(ps, "", nil, map[string]string{"A.Key": "v"}, knownTwo) + if err != nil { + t.Fatalf("resolveConstantChain: %v", err) + } + if len(got.Private) != 1 || got.Private[0] != "A.Url" { + t.Errorf("Private = %v, want only the one no layer covered", got.Private) + } +} + +func TestReportConstantChain_NamesTheMachineLayerAndStaleEntries(t *testing.T) { + var buf bytes.Buffer + reportConstantChain(&buf, constantChain{ + Values: map[string]string{"A.Key": "SECRETVALUE"}, + From: map[string]constantLayer{"A.Key": layerMachine}, + Stale: []string{"A.Removed"}, + }) + out := buf.String() + if !strings.Contains(out, string(layerMachine)) { + t.Errorf("the machine layer is not named:\n%s", out) + } + if !strings.Contains(out, "A.Removed") { + t.Errorf("the stale entry is not reported:\n%s", out) + } + if strings.Contains(out, "SECRETVALUE") { + t.Errorf("the report printed a machine-local value:\n%s", out) + } +} diff --git a/cmd/mxcli/constantstore/store.go b/cmd/mxcli/constantstore/store.go new file mode 100644 index 000000000..a267f8653 --- /dev/null +++ b/cmd/mxcli/constantstore/store.go @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package constantstore holds constant values that belong to THIS MACHINE and +// must not reach version control. +// +// Mendix has this concept — a configuration value marked private — but from +// 10.9 it is encrypted per user account by Studio Pro, which is Windows/Mac +// only. In a Linux devcontainer or an agent session there is no Studio Pro and +// no such store: nothing can write it and nothing can read it. So for a +// headless run the one safe slot is unreachable, and a constant's default and a +// shared configuration override both go to git. +// +// This is mxcli's own equivalent, and is labelled as mxcli's rather than +// pretending to be Mendix's: a gitignored file next to the project, mode 0600. +// That is the same bar as ~/.mxcli/auth.json — file permissions, not +// encryption. See docs/11-proposals/PROPOSAL_constant_values.md. +package constantstore + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" +) + +// FileName is the store's name inside the project's .mxcli directory. +const FileName = "constants.json" + +// currentVersion is the on-disk schema version. It exists so a later shape +// (per-configuration values, say) can be added without a reader of this one +// silently misreading it. +const currentVersion = 1 + +// Store is a project's machine-local constant values. +type Store struct { + Version int `json:"version"` + Constants map[string]string `json:"constants"` +} + +// Path is the store's location for a project. +func Path(projectPath string) string { + return filepath.Join(filepath.Dir(projectPath), ".mxcli", FileName) +} + +// Load reads the store for a project. A missing file is not an error — it is +// the normal case, and means "this machine sets no constants". +func Load(projectPath string) (*Store, error) { + body, err := os.ReadFile(Path(projectPath)) + if os.IsNotExist(err) { + return &Store{Version: currentVersion, Constants: map[string]string{}}, nil + } + if err != nil { + return nil, fmt.Errorf("reading %s: %w", Path(projectPath), err) + } + var s Store + if err := json.Unmarshal(body, &s); err != nil { + // Named, not swallowed: silently treating a corrupt store as empty would + // boot the app with different values than the author set, which is the + // whole failure class this feature exists to remove. + return nil, fmt.Errorf("%s is not valid JSON: %w", Path(projectPath), err) + } + if s.Version > currentVersion { + return nil, fmt.Errorf("%s was written by a newer mxcli (version %d, this build understands %d)", + Path(projectPath), s.Version, currentVersion) + } + if s.Constants == nil { + s.Constants = map[string]string{} + } + return &s, nil +} + +// Save writes the store back, atomically and 0600. +// +// A store with no constants left is REMOVED rather than written empty, so +// unsetting the last value leaves no file behind claiming to configure +// something. +func Save(projectPath string, s *Store) error { + path := Path(projectPath) + if len(s.Constants) == 0 { + if err := os.Remove(path); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("removing %s: %w", path, err) + } + return nil + } + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return fmt.Errorf("creating %s: %w", filepath.Dir(path), err) + } + s.Version = currentVersion + body, err := json.MarshalIndent(s, "", " ") + if err != nil { + return err + } + body = append(body, '\n') + // Temp file + rename so a reader can never see a half-written store, and + // 0600 from the moment it exists rather than after a chmod — the window + // matters when the value is an API key. + tmp := path + ".tmp" + if err := os.WriteFile(tmp, body, 0o600); err != nil { + return fmt.Errorf("writing %s: %w", tmp, err) + } + if err := os.Rename(tmp, path); err != nil { + _ = os.Remove(tmp) + return fmt.Errorf("publishing %s: %w", path, err) + } + return nil +} + +// Names returns the constants this store sets, sorted. +func (s *Store) Names() []string { + names := make([]string, 0, len(s.Constants)) + for k := range s.Constants { + names = append(names, k) + } + sort.Strings(names) + return names +} diff --git a/cmd/mxcli/constantstore/store_test.go b/cmd/mxcli/constantstore/store_test.go new file mode 100644 index 000000000..140d42dc8 --- /dev/null +++ b/cmd/mxcli/constantstore/store_test.go @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: Apache-2.0 + +package constantstore + +import ( + "os" + "path/filepath" + "testing" +) + +func project(t *testing.T) string { + t.Helper() + return filepath.Join(t.TempDir(), "App.mpr") +} + +// A missing store is the normal case: most projects set nothing on the machine. +func TestLoad_MissingFileIsNotAnError(t *testing.T) { + s, err := Load(project(t)) + if err != nil { + t.Fatalf("Load: %v", err) + } + if len(s.Constants) != 0 { + t.Errorf("Constants = %v, want empty", s.Constants) + } +} + +func TestSaveLoad_RoundTrip(t *testing.T) { + p := project(t) + if err := Save(p, &Store{Constants: map[string]string{"A.Key": "sk-123"}}); err != nil { + t.Fatalf("Save: %v", err) + } + got, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if got.Constants["A.Key"] != "sk-123" { + t.Errorf("Constants = %v", got.Constants) + } + if got.Version != currentVersion { + t.Errorf("Version = %d, want %d", got.Version, currentVersion) + } +} + +// The store's whole reason for existing is holding values that must not be +// readable by anyone else on the machine. +func TestSave_Mode0600(t *testing.T) { + p := project(t) + if err := Save(p, &Store{Constants: map[string]string{"A.Key": "v"}}); err != nil { + t.Fatalf("Save: %v", err) + } + info, err := os.Stat(Path(p)) + if err != nil { + t.Fatalf("stat: %v", err) + } + if perm := info.Mode().Perm(); perm != 0o600 { + t.Errorf("mode = %o, want 600 — the file can hold an API key", perm) + } + // And no temp file left behind carrying the same secret at another mode. + if _, err := os.Stat(Path(p) + ".tmp"); !os.IsNotExist(err) { + t.Errorf("the temp file survived the write: %v", err) + } +} + +// Unsetting the last value should leave no file claiming to configure +// something. A store of {} is not the same as no store when someone reads the +// directory to see whether this machine overrides anything. +func TestSave_RemovesAnEmptyStore(t *testing.T) { + p := project(t) + if err := Save(p, &Store{Constants: map[string]string{"A.Key": "v"}}); err != nil { + t.Fatalf("Save: %v", err) + } + if err := Save(p, &Store{Constants: map[string]string{}}); err != nil { + t.Fatalf("Save empty: %v", err) + } + if _, err := os.Stat(Path(p)); !os.IsNotExist(err) { + t.Errorf("an empty store was left on disk: %v", err) + } + // ...and removing it again is not an error. + if err := Save(p, &Store{Constants: map[string]string{}}); err != nil { + t.Errorf("Save on an already-absent store: %v", err) + } +} + +// A corrupt store must be NAMED, not treated as empty: silently booting with +// different values than the author set is the failure this whole feature +// exists to remove. +func TestLoad_CorruptFileIsAnError(t *testing.T) { + p := project(t) + if err := os.MkdirAll(filepath.Dir(Path(p)), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(Path(p), []byte("{not json"), 0o600); err != nil { + t.Fatal(err) + } + if _, err := Load(p); err == nil { + t.Fatal("a corrupt store loaded as if it were empty") + } +} + +// A store written by a newer mxcli may mean something this build would +// misread — refuse rather than apply half of it. +func TestLoad_RefusesANewerVersion(t *testing.T) { + p := project(t) + if err := os.MkdirAll(filepath.Dir(Path(p)), 0o755); err != nil { + t.Fatal(err) + } + body := []byte(`{"version": 99, "constants": {"A.Key": "v"}}`) + if err := os.WriteFile(Path(p), body, 0o600); err != nil { + t.Fatal(err) + } + if _, err := Load(p); err == nil { + t.Fatal("a store from a newer schema was accepted") + } +} + +func TestPath_LivesBesideTheProject(t *testing.T) { + p := "/some/where/App.mpr" + if want := "/some/where/.mxcli/" + FileName; Path(p) != want { + t.Errorf("Path = %q, want %q", Path(p), want) + } +} diff --git a/cmd/mxcli/init.go b/cmd/mxcli/init.go index 1ed8d93cf..9afcdbbce 100644 --- a/cmd/mxcli/init.go +++ b/cmd/mxcli/init.go @@ -572,6 +572,14 @@ Container Runtime: } else { fmt.Println("\nCreated .gitignore") } + } else if mprFile != "" { + // A project that already has a .gitignore keeps it — but .mxcli/ has to + // be in there regardless, because it holds machine-local constant values + // (which may be secrets), the run/test handshake and its token, and + // runtime logs. Only the create-if-missing branch above covered it. + if _, err := ensureStoreIgnored(filepath.Join(absDir, mprFile)); err != nil { + fmt.Fprintf(os.Stderr, " Warning: could not ensure .mxcli/ is git-ignored: %v\n", err) + } } // Create .playwright/cli.config.json for playwright-cli diff --git a/cmd/mxcli/main.go b/cmd/mxcli/main.go index 57529af05..b1320bd9e 100644 --- a/cmd/mxcli/main.go +++ b/cmd/mxcli/main.go @@ -416,4 +416,5 @@ func init() { rootCmd.AddCommand(evalCmd) rootCmd.AddCommand(tuiCmd) rootCmd.AddCommand(fmtCmd) + rootCmd.AddCommand(constantCmd) } diff --git a/docs-site/src/tools/running-tests.md b/docs-site/src/tools/running-tests.md index f55c13d52..4fe304bc9 100644 --- a/docs-site/src/tools/running-tests.md +++ b/docs-site/src/tools/running-tests.md @@ -16,11 +16,13 @@ A `--local` run boots the app with the same **constant values** `mxcli run --loc uses — the project configuration's shared overrides layered over each constant's default — and prints what it applied. Use `--configuration ` to choose between several, and `--constant Module.Name=value` (repeatable) to set one for -this run only — it wins over the configuration and is never written to the -project. A constant the project does not declare is refused before anything -boots. `--attach` takes none of these, since it inherits the constants of the app -it attaches to. This matters whenever a test asserts on something a constant -feeds: the two modes used to disagree silently. +this run only — it wins over everything and is never written anywhere. For a +value that should persist on this machine without being committed, use +`mxcli constant set` (see [Constant values](#constant-values) below). A constant +the project does not declare is refused before anything boots. `--attach` takes +none of these, since it inherits the constants of the app it attaches to. This +matters whenever a test asserts on something a constant feeds: the two modes +used to disagree silently. `--local` also downloads what it needs on first use. To pre-cache it: @@ -35,6 +37,20 @@ The `mx` binary, when you need it directly: | Dev container | `~/.mxcli/mxbuild/{version}/modeler/mx` | | Repository | `reference/mxbuild/modeler/mx` | +## Constant values + +Highest layer wins: + +| Layer | Set with | In git? | +|---|---|---| +| this run | `--constant Module.Name=value` | no | +| this machine | `mxcli constant set Module.Name value` | no — gitignored, 0600 | +| this configuration | `alter settings constant … in configuration 'X'` | yes | +| default | `create constant … default '…'` | yes | + +`mxcli constant list` shows the winner for each constant and which layer set it, +masking machine-local values unless `--show-values` is passed. + ## Basic Usage ```bash diff --git a/docs/11-proposals/PROPOSAL_constant_values.md b/docs/11-proposals/PROPOSAL_constant_values.md index 95fe18b22..050900e0f 100644 --- a/docs/11-proposals/PROPOSAL_constant_values.md +++ b/docs/11-proposals/PROPOSAL_constant_values.md @@ -6,7 +6,7 @@ date: 2026-08-13 # Proposal: Constant values — one precedence chain, and a slot for secrets -**Status:** Accepted — slices 1 and 2 shipped; 3 and 4 open +**Status:** Accepted — slices 1, 2 and 3 shipped; 4 open **Date:** 2026-08-13 A Mendix constant has a value in four possible places, mxcli can write two of @@ -220,7 +220,7 @@ Test: a `.test.mdl` asserting a constant, run under `--local` and under Refuses an unknown constant name rather than passing it through: a typo'd override is silently ignored by the runtime, which is the §33 shape again. -### Slice 3 — layer 2, the machine store +### Slice 3 — layer 2, the machine store — **shipped** | File | Change | |------|--------| From 6f9dd5c4ad4abcbf6e00d159d348511c2a899d08 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 13:06:43 +0000 Subject: [PATCH 13/20] Apply a constant change to a running app without restarting it Constants are a boot-time layer: an app already up keeps serving the old value until someone restarts it. `mxcli constant set/unset --apply` pushes the change into a `mxcli run --local` that is already running. It is two admin calls, not one. Measured on 11.12.1, update_configuration is STAGED -- the running app keeps its old values and the call still answers result:0 -- and only the following reload_model applies them, so a version sending just the first would report success and change nothing. Both live in ApplyConstants rather than at the call site for that reason. The whole boot payload is re-sent, not the constants alone, because the admin API has no read-back: whatever is not sent is simply gone from the configuration afterwards. A second process cannot ask what the runtime was booted with, so `run --local` now publishes it -- with the ports and admin credential -- in a 0600 handshake beside the project, removed when the loop exits. A handshake whose process is gone is refused rather than used: its ports may since have been taken by something else. One correction to the proposal, found while building it. "Verify by observation" is not achievable from outside the app: no admin action exposes a constant's value. So --apply performs both calls, reports what it did, and names what would actually confirm it, rather than claiming a success it cannot check. Failure to apply is not failure to set -- the value is on disk and the next boot uses it, so --apply reports and returns instead of exiting non-zero over a half-succeeded command. Verified on 11.12.1 against a warm dev loop, reading the constant through a microflow over the test endpoint so no model change is involved: boot RUNTIME-KEY; set without --apply still RUNTIME-KEY (the control); set --apply STORE-APPLIED with no restart; unset --apply back to RUNTIME-KEY. docs/11-proposals/PROPOSAL_constant_values.md slice 4. --- .claude/skills/fix-issue.md | 1 + .claude/skills/mendix/test-microflows.md | 13 ++ cmd/mxcli/cmd_constant.go | 66 ++++++++++ cmd/mxcli/cmd_run.go | 25 ++++ cmd/mxcli/devloop_handshake.go | 116 +++++++++++++++++ cmd/mxcli/devloop_handshake_test.go | 122 ++++++++++++++++++ cmd/mxcli/docker/localboot.go | 54 +++++++- cmd/mxcli/docker/localboot_constants_test.go | 87 ++++++++++++- cmd/mxcli/docker/runlocal.go | 14 +- docs-site/src/tools/running-tests.md | 5 + docs/11-proposals/PROPOSAL_constant_values.md | 13 +- 11 files changed, 506 insertions(+), 10 deletions(-) create mode 100644 cmd/mxcli/devloop_handshake.go create mode 100644 cmd/mxcli/devloop_handshake_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index a5e0203a5..3e1b5ab67 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -500,3 +500,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `GRANT ON Module.Specialization (READ *, WRITE *)` writes a model Mendix rejects with **CE0066** "Entity access is out of date", while the same grant on the generalization checks clean. The executor passes 3 MemberAccess entries and storage holds 2 | `ReconcileMemberAccesses` — which runs after every program and after every `create association` — recomputes each rule's expected member set from the entity's **own** attributes and the module's **FROM-side** associations. An inherited association matched neither, so it was classed stale and deleted, one step after the GRANT that had just written it correctly | `mdl/backend/modelsdk/domainmodel_security_write.go` (`sameModuleAncestors`, `entitiesByName`, `assocRefBelongsTo`, `ownerIDs` in `ReconcileMemberAccesses`), `mdl/executor/cmd_associations.go` (reconcile after DROP) | **The write path was innocent** — `AddEntityAccessRule` stored all 3; the deletion happened in the reconcile that ran next. When a value is written and then absent, instrument the *later* pass before the writer. **Preserve what cannot be checked**, as the attribute branch already did (#758): an association is qualified by the module that DECLARES it, so an ancestor in another module (the reported `OpenAIDeployedModel extends GenAICommons.DeployedModel`) names a domain model that is not loaded here. **The counter-control is the whole test**: a walk that collected every association in the module passes both positive cases and puts an entry on the TO side of an `OWNER Default` association, which is *itself* CE0066 — so assert that the unrelated entity does **not** get one. **Reconcile must add as well as keep**, or a rule written before the ancestor gained the association never catches up. Verifying this surfaced a neighbouring bug: `drop association` never reconciled at all, leaving entries Mendix rejects with CE1613 — on the declaring entity too, so it predates inheritance support. Tests `mdl/backend/modelsdk/reconcile_inherited_assoc_test.go` (5 cases incl. both controls), `mdl/executor/cmd_associations_mock_test.go`; example `mdl-examples/bug-tests/grant-on-specialization.mdl`, measured 1 error → 0 on mxbuild 11.12.1. Reported in mxcli-chat FINDINGS §25 | | A `.test.mdl` asserting on something a **constant** feeds passes under `mxcli test --attach` and fails under `mxcli test --local` (or the reverse), with nothing in either output to explain the difference | `--local` boots an app of its own through `StartLocalApp`, whose options had no `ConstantOverrides` field — so it ran with each constant's **default** from `deployment/model/config.json`, while `--attach` runs against an app `run --local` booted, which applies the configuration's shared overrides. A constant resolving to the wrong value is not an error, so both runs reported success and only the assertion differed | `cmd/mxcli/docker/localapp.go` (`LocalAppOptions.ConstantOverrides`, `runtimeOptions`), `cmd/mxcli/testrunner/localapp_options.go` (new, shared by both `--local` runners), `runner.go` (`RunOptions.ConstantOverrides`), `cmd/mxcli/cmd_test_run.go` + `main.go` (`--configuration`) | **A boot path that takes options needs one mapping function, not an inline struct literal per caller** — the two `--local` runners each built their own `LocalAppOptions`, so a field added for one would not reach the other, and neither had the constants. Both now go through `localAppOptions`, and the `LocalAppOptions`→`LocalRuntimeOptions` step is `runtimeOptions()` so the forwarding is assertable without booting anything: a dropped field there is otherwise invisible until an app runs with the wrong configuration. **Resolve in one place**: `cmd/` decides precedence and the runner only carries the map, or "which configuration wins" gets two answers. **Only report what this run actually uses** — `--attach` inherits the constants of the app it attached to, so resolving and printing them there would be a confident lie. Verified at the layer the symptom lives in (`.claude/skills/verify-in-runtime.md`): the same suite, one project, `--local` and `--attach` must agree, with the reverted-wiring control run showing the constant's default. Unit tests `cmd/mxcli/testrunner/localapp_options_test.go`, `cmd/mxcli/docker/localboot_constants_test.go`. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 1 | | A constant value that must not be committed has nowhere to live: a constant's default and a shared configuration override both go to git, and Mendix's own **private** configuration value — the correct slot — is encrypted per user account by Studio Pro from 10.9, so nothing headless can read or write it | Not a defect so much as an absent layer. mxcli mirrors the concept with a store it can actually reach: `/.mxcli/constants.json`, mode 0600, gitignored, sitting between `--constant` and the configuration | `cmd/mxcli/constantstore/` (load/save), `cmd/mxcli/cmd_constant.go` (`constant set/unset/list`), `cmd/mxcli/constant_gitignore.go` (`ensureStoreIgnored`), `cmd/mxcli/constants_resolve.go` (`layerMachine`) | **The promise has to be made true, then checked — asserting it is not enough.** `mxcli init` writes a `.gitignore` only when the project has none, and a Mendix project usually already has one, so the entry the whole layer rests on could simply be absent. `constant set` appends it *and then asks git*, refusing to write the value on "not ignored": a store that leaks is worse than no store. **Which rule defeats an ignore entry is not guessable** — `!.mxcli/**` does NOT re-include anything (git cannot re-include a file whose parent directory is excluded), while `!.mxcli` does; the test was corrected against measured git behaviour rather than the assumed case. **A corrupt store is an error, a stale entry is not**: an unparseable file means values the author set are about to be silently absent (fatal), while an entry naming a constant the project dropped is the user's own file and is skipped-and-named, or every run fails until it is hand-edited. **An empty store is deleted, not written as `{}`** — a file that configures nothing should not exist. Tests `cmd/mxcli/constantstore/store_test.go`, `cmd/mxcli/constant_gitignore_test.go`; verified live that the value reaches a running app and that `git status` never sees the file. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 3 | +| A constant value changed on disk (machine store, or the project configuration) does not reach an app that is **already running** — a `mxcli run --local` keeps serving the old value until it is restarted | Constants are a boot-time layer: mxcli sends them once, in the `update_configuration` call at start. Changing them live needs the M2EE admin API, and needs BOTH of its calls | `cmd/mxcli/docker/localboot.go` (`ApplyConstants`, `LocalRuntime.BootConfig`), `cmd/mxcli/devloop_handshake.go`, `cmd/mxcli/cmd_constant.go` (`--apply`) | **`update_configuration` is STAGED, not applied** — measured on 11.12.1: the running app keeps its old values, and the call still answers `result:0`. Only the next `reload_model` applies them, so a version sending just the first call reports success and changes nothing. That is why the two calls live in one function rather than at the call site. **Re-send the WHOLE payload**: the admin action has no read-back (`get_configuration`, `get_current_configuration`, `runtime_config`, `get_current_runtime_status` are all "Action not found"), so anything not sent is simply gone from the configuration afterwards — which is why the dev loop publishes what it booted with rather than a second process guessing. **A handshake from a dead process is refused, not used**: its ports may since have been taken by something else, and pushing a configuration change into the wrong process is worse than reporting no dev loop. **Say plainly what cannot be confirmed** — with no read-back, two successful calls are the strongest evidence available, so the command names what would actually confirm it (a microflow that returns the constant) instead of claiming success. **Failure to apply is not failure to set**: the value is already on disk and the next boot uses it, so `--apply` reports and returns rather than exiting non-zero over a half-succeeded command. Tests `cmd/mxcli/docker/localboot_constants_test.go`, `cmd/mxcli/devloop_handshake_test.go`. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 4 | diff --git a/.claude/skills/mendix/test-microflows.md b/.claude/skills/mendix/test-microflows.md index 150e8aae0..069c7eaea 100644 --- a/.claude/skills/mendix/test-microflows.md +++ b/.claude/skills/mendix/test-microflows.md @@ -81,6 +81,19 @@ project's `.gitignore` if missing, and then **asks git whether the path is really ignored** — refusing to write the value if it is not. It beats the configuration and loses to `--constant`. +By default the new value takes effect at the next boot. Add `--apply` to push it +into a `mxcli run --local` that is already up, without restarting it: + +```bash +mxcli constant set MyModule.ApiKey 'sk-live-...' -p app.mpr --apply +``` + +That is two admin calls, not one: `update_configuration` is *staged* — the +running app keeps its old values and the call still answers success — and only +the following `reload_model` applies them. mxcli does both. It cannot confirm +the result, because the admin API has no way to read a constant back, so it says +so and points you at the app itself. + This is mxcli's own store, not Mendix's. Mendix's private configuration values are encrypted per user account by Studio Pro from 10.9, so nothing headless can read or write them. See `docs/11-proposals/PROPOSAL_constant_values.md`. diff --git a/cmd/mxcli/cmd_constant.go b/cmd/mxcli/cmd_constant.go index e3736e4c8..ce3496958 100644 --- a/cmd/mxcli/cmd_constant.go +++ b/cmd/mxcli/cmd_constant.go @@ -10,6 +10,7 @@ import ( "strings" "github.com/mendixlabs/mxcli/cmd/mxcli/constantstore" + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" "github.com/spf13/cobra" ) @@ -98,9 +99,63 @@ var constantSetCmd = &cobra.Command{ chain.Configuration, name) } } + + applyIfAsked(cmd, projectPath, read, store.Constants) }, } +// applyIfAsked pushes the resolved values into a running `mxcli run --local` +// when --apply is set. +// +// Without it, the store is a boot-time layer only: an app already up keeps +// serving the old value until someone restarts it. With it, the change lands in +// about a second. +// +// Failure to apply is NOT failure to set. The value is already on disk and the +// next boot will use it, so this reports and returns rather than exiting +// non-zero — the alternative is a command that half-succeeded and looks like it +// did nothing. +func applyIfAsked(cmd *cobra.Command, projectPath string, read projectRead, machine map[string]string) { + apply, _ := cmd.Flags().GetBool("apply") + if !apply { + return + } + configuration, _ := cmd.Flags().GetString("configuration") + + hs, err := readDevLoopHandshake(projectPath) + if err != nil { + fmt.Fprintf(os.Stderr, "\nThe value is saved, but not applied to a running app:\n %v\n", err) + return + } + known := make(map[string]bool, len(read.defaults)) + for n := range read.defaults { + known[n] = true + } + chain, err := resolveConstantChain(read.settings, configuration, nil, machine, known) + if err != nil { + fmt.Fprintf(os.Stderr, "\nThe value is saved, but not applied: %v\n", err) + return + } + + fmt.Printf("\nApplying to the app on port %d (pid %d)...\n", hs.AppPort, hs.PID) + if err := docker.ApplyConstants( + docker.M2EEOptions{Host: "127.0.0.1", Port: hs.AdminPort, Token: hs.AdminPass, Direct: true}, + hs.BootConfig, chain.Values, + ); err != nil { + fmt.Fprintf(os.Stderr, " the value is saved, but the running app still has the old one: %v\n", err) + return + } + fmt.Println(" configuration updated and model reloaded; the app is now serving the new value.") + // Said plainly because it cannot be checked from here. The M2EE admin API has + // no read-back — get_configuration, get_current_configuration, runtime_config + // and get_current_runtime_status are all "Action not found" on 11.12.1 — and + // update_configuration answers result:0 even when it changes nothing. So the + // two calls succeeding is the strongest evidence available, and the honest + // framing is to name what would actually confirm it. + fmt.Println(" (the admin API offers no way to read a constant back, so confirm from the app " + + "itself — a microflow that returns it, or the behaviour that depends on it.)") +} + var constantUnsetCmd = &cobra.Command{ Use: "unset ", Short: "Remove a constant's machine-local value", @@ -122,6 +177,11 @@ var constantUnsetCmd = &cobra.Command{ exitf("%v", err) } fmt.Printf("Removed the machine-local value for %s; runs now use the configuration or the default.\n", name) + + read, err := projectConstantDefaults(projectPath) + if err == nil { + applyIfAsked(cmd, projectPath, read, store.Constants) + } }, } @@ -227,6 +287,12 @@ func exitf(format string, a ...any) { } func init() { + for _, c := range []*cobra.Command{constantSetCmd, constantUnsetCmd} { + c.Flags().Bool("apply", false, + "Also apply the change to a running 'mxcli run --local' (update_configuration + reload_model). Without this the value takes effect at the next boot") + c.Flags().String("configuration", "", + "With --apply, which configuration's values to resolve the rest of the chain against") + } constantListCmd.Flags().String("configuration", "", "Which configuration's values to resolve against (default: the only one, or \"Default\")") constantListCmd.Flags().Bool("show-values", false, diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index d444d84d3..b3c3a96d4 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -6,6 +6,7 @@ import ( "fmt" "os" "path/filepath" + "time" "github.com/mendixlabs/mxcli/cmd/mxcli/docker" "github.com/mendixlabs/mxcli/cmd/mxcli/hubauth" @@ -241,10 +242,34 @@ Examples: } } + // Publish the dev-loop handshake so another mxcli process can reach this + // app — `mxcli constant set --apply` needs the admin port and the payload + // the runtime was booted with. Chained rather than assigned, because + // --test-endpoint may already have installed an OnReady of its own. + previousOnReady := opts.OnReady + opts.OnReady = func(info docker.LocalAppInfo) { + if previousOnReady != nil { + previousOnReady(info) + } + if err := writeDevLoopHandshake(projectPath, devLoopHandshake{ + Project: projectPath, + PID: os.Getpid(), + AppPort: info.AppPort, + AdminPort: info.AdminPort, + AdminPass: info.AdminPass, + BootConfig: info.BootConfig, + Started: time.Now(), + }); err != nil { + fmt.Fprintf(os.Stderr, "Warning: could not publish the dev-loop handshake: %v\n", err) + } + } + defer removeDevLoopHandshake(projectPath) + if err := docker.RunLocal(opts); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) // os.Exit skips deferred calls, so remove explicitly here. hosted.Remove() + removeDevLoopHandshake(projectPath) os.Exit(1) } }, diff --git a/cmd/mxcli/devloop_handshake.go b/cmd/mxcli/devloop_handshake.go new file mode 100644 index 000000000..3a7e40718 --- /dev/null +++ b/cmd/mxcli/devloop_handshake.go @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "syscall" + "time" +) + +// devLoopHandshakeName is what `mxcli run --local` publishes so another mxcli +// process can reach the app it is serving. +// +// It is the sibling of testrunner's test-endpoint.json, and exists for the same +// reason: a second process cannot otherwise know which ports this dev loop chose +// or what configuration its runtime was booted with. The admin API has no +// read-back, so "what was it booted with" cannot be asked — only remembered. +const devLoopHandshakeName = "run-local.json" + +// devLoopHandshake is the contract between a running `mxcli run --local` and a +// command that wants to change something about the app it is serving. +// +// It carries a live credential and the runtime's database password, so it is +// written 0600 into the gitignored .mxcli directory and removed when the loop +// exits. It is not a secret store: the values only work against a loopback +// admin port on this machine, and only while that runtime is up. +type devLoopHandshake struct { + // Project is the .mpr this loop is serving, so a caller can refuse a + // handshake left behind by a different project. + Project string `json:"project"` + // PID of the hosting process, used to detect a stale file. + PID int `json:"pid"` + // AppPort is where the app itself is reachable. + AppPort int `json:"appPort"` + // AdminPort/AdminPass reach the M2EE admin API. + AdminPort int `json:"adminPort"` + AdminPass string `json:"adminPass"` + // BootConfig is the update_configuration payload the runtime was started + // with. A caller changing one setting re-sends this with that key replaced — + // the only way to avoid guessing at everything it does not want to change. + BootConfig map[string]any `json:"bootConfig"` + // Started is when this was published, for a clearer stale message. + Started time.Time `json:"started"` +} + +func devLoopHandshakePath(projectPath string) string { + return filepath.Join(filepath.Dir(projectPath), ".mxcli", devLoopHandshakeName) +} + +// writeDevLoopHandshake publishes the handshake, replacing any existing one. +func writeDevLoopHandshake(projectPath string, h devLoopHandshake) error { + path := devLoopHandshakePath(projectPath) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return fmt.Errorf("creating %s: %w", filepath.Dir(path), err) + } + body, err := json.MarshalIndent(h, "", " ") + if err != nil { + return err + } + tmp := path + ".tmp" + if err := os.WriteFile(tmp, body, 0o600); err != nil { + return fmt.Errorf("writing %s: %w", tmp, err) + } + if err := os.Rename(tmp, path); err != nil { + os.Remove(tmp) + return fmt.Errorf("publishing %s: %w", path, err) + } + return nil +} + +func removeDevLoopHandshake(projectPath string) { + _ = os.Remove(devLoopHandshakePath(projectPath)) +} + +// readDevLoopHandshake returns the handshake for a project, or an error a user +// can act on. +// +// A handshake whose process is gone is refused rather than used: the ports in it +// may since have been taken by something else, and sending a configuration +// change to the wrong process is worse than saying no dev loop is running. +func readDevLoopHandshake(projectPath string) (devLoopHandshake, error) { + var h devLoopHandshake + path := devLoopHandshakePath(projectPath) + body, err := os.ReadFile(path) + if os.IsNotExist(err) { + return h, fmt.Errorf("no 'mxcli run --local' is serving this project\n" + + " start one in another terminal, or drop --apply to only record the value") + } + if err != nil { + return h, fmt.Errorf("reading %s: %w", path, err) + } + if err := json.Unmarshal(body, &h); err != nil { + return h, fmt.Errorf("%s is not valid JSON: %w", path, err) + } + if !processAlive(h.PID) { + return h, fmt.Errorf("%s refers to process %d, which is no longer running\n"+ + " (the dev loop was stopped without cleaning up; start a new one)", path, h.PID) + } + return h, nil +} + +// processAlive reports whether a pid exists. Signal 0 is delivered to no one but +// still performs the existence and permission checks. +func processAlive(pid int) bool { + if pid <= 0 { + return false + } + p, err := os.FindProcess(pid) + if err != nil { + return false + } + return p.Signal(syscall.Signal(0)) == nil +} diff --git a/cmd/mxcli/devloop_handshake_test.go b/cmd/mxcli/devloop_handshake_test.go new file mode 100644 index 000000000..161e09831 --- /dev/null +++ b/cmd/mxcli/devloop_handshake_test.go @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: Apache-2.0 + +// The dev-loop handshake is how `mxcli constant set --apply` finds the app a +// `mxcli run --local` is serving, and — crucially — the configuration payload +// that runtime was booted with. The admin API has no read-back, so a second +// process cannot ask what the configuration is; it can only be told. +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func handshakeProject(t *testing.T) string { + t.Helper() + return filepath.Join(t.TempDir(), "App.mpr") +} + +func TestDevLoopHandshake_RoundTrip(t *testing.T) { + p := handshakeProject(t) + want := devLoopHandshake{ + Project: p, + PID: os.Getpid(), + AppPort: 8080, + AdminPort: 8090, + AdminPass: "mxcli-local-dev", + BootConfig: map[string]any{ + "BasePath": "/tmp/app/deployment", + "DatabaseName": "app", + "MicroflowConstants": map[string]any{"A.B": "v"}, + }, + Started: time.Now(), + } + if err := writeDevLoopHandshake(p, want); err != nil { + t.Fatalf("write: %v", err) + } + + got, err := readDevLoopHandshake(p) + if err != nil { + t.Fatalf("read: %v", err) + } + if got.AdminPort != 8090 || got.AdminPass != "mxcli-local-dev" { + t.Errorf("admin details lost: %+v", got) + } + if got.BootConfig["BasePath"] != "/tmp/app/deployment" { + t.Fatalf("BootConfig lost: %v — without it a caller has to guess at every "+ + "setting it is not changing", got.BootConfig) + } +} + +// It carries a live admin credential and the runtime's database password. +func TestDevLoopHandshake_Mode0600(t *testing.T) { + p := handshakeProject(t) + if err := writeDevLoopHandshake(p, devLoopHandshake{PID: os.Getpid()}); err != nil { + t.Fatalf("write: %v", err) + } + info, err := os.Stat(devLoopHandshakePath(p)) + if err != nil { + t.Fatalf("stat: %v", err) + } + if perm := info.Mode().Perm(); perm != 0o600 { + t.Errorf("mode = %o, want 600", perm) + } +} + +// No dev loop is a normal state, and the message has to say what to do about it +// rather than name a missing file. +func TestReadDevLoopHandshake_MissingSaysWhatToDo(t *testing.T) { + _, err := readDevLoopHandshake(handshakeProject(t)) + if err == nil { + t.Fatal("a missing handshake read as success") + } + if !strings.Contains(err.Error(), "run --local") { + t.Errorf("the error does not say how to fix it: %v", err) + } +} + +// A handshake left behind by a dead process must be refused, not used: its ports +// may since have been taken by something else, and sending a configuration +// change to the wrong process is worse than reporting no dev loop. +func TestReadDevLoopHandshake_RefusesADeadProcess(t *testing.T) { + p := handshakeProject(t) + if err := writeDevLoopHandshake(p, devLoopHandshake{Project: p, PID: 999999, AdminPort: 8090}); err != nil { + t.Fatalf("write: %v", err) + } + + _, err := readDevLoopHandshake(p) + if err == nil { + t.Fatal("a handshake from a dead process was accepted") + } + if !strings.Contains(err.Error(), "999999") { + t.Errorf("the error does not name the stale pid: %v", err) + } +} + +func TestReadDevLoopHandshake_CorruptFileIsNamed(t *testing.T) { + p := handshakeProject(t) + if err := os.MkdirAll(filepath.Dir(devLoopHandshakePath(p)), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(devLoopHandshakePath(p), []byte("{nope"), 0o600); err != nil { + t.Fatal(err) + } + if _, err := readDevLoopHandshake(p); err == nil { + t.Fatal("a corrupt handshake read as success") + } +} + +func TestRemoveDevLoopHandshake(t *testing.T) { + p := handshakeProject(t) + if err := writeDevLoopHandshake(p, devLoopHandshake{PID: os.Getpid()}); err != nil { + t.Fatal(err) + } + removeDevLoopHandshake(p) + if _, err := os.Stat(devLoopHandshakePath(p)); !os.IsNotExist(err) { + t.Errorf("the handshake survived removal: %v", err) + } + removeDevLoopHandshake(p) // idempotent +} diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index 2b160ddd2..17e55ee42 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -117,6 +117,8 @@ type LocalRuntime struct { logFile *os.File // open when RuntimeLogPath is set; runtime stdout/stderr tee m2ee M2EEOptions ctrl *RuntimeController + // bootConfig is the update_configuration payload sent at start; see BootConfig. + bootConfig map[string]any } func (o *LocalRuntimeOptions) applyDefaults() { @@ -493,12 +495,62 @@ func (rt *LocalRuntime) spawnAndConfigure() error { return err } constants = mergeConstantOverrides(constants, rt.opts.ConstantOverrides) - if _, err := CallM2EE(rt.m2ee, "update_configuration", runtimeConfigParams(rt.opts, constants)); err != nil { + // Kept so a LATER caller can re-send this exact payload with a different + // constants map. The admin action has no read-back, so the only way to change + // one setting without guessing at the rest is to have kept the rest. + rt.bootConfig = runtimeConfigParams(rt.opts, constants) + if _, err := CallM2EE(rt.m2ee, "update_configuration", rt.bootConfig); err != nil { return fmt.Errorf("update_configuration: %w", err) } return nil } +// BootConfig is the update_configuration payload this runtime was started with. +// +// It is what makes a live constant change possible from ANOTHER process: that +// process cannot read the configuration back (the admin API has no such action), +// so it re-sends this with MicroflowConstants replaced. See ApplyConstants. +func (rt *LocalRuntime) BootConfig() map[string]any { return rt.bootConfig } + +// AdminOptions is how to reach this runtime's admin API. +func (rt *LocalRuntime) AdminOptions() M2EEOptions { return rt.m2ee } + +// ApplyConstants changes a running app's constant values: it re-sends a boot +// payload with MicroflowConstants replaced, then reloads the model. +// +// BOTH calls are required, and that is the whole reason this is a function +// rather than a one-liner at the call site. Measured on 11.12.1: +// update_configuration is STAGED — the running app keeps its old values and +// still answers result:0 — and only the next reload_model applies them. Shipping +// just the first call would produce a command that reports success and changes +// nothing, which is the exact failure this feature exists to remove +// (mxcli-chat FINDINGS §33). +// +// It re-sends the whole payload rather than MicroflowConstants alone because +// the admin action offers no read-back to merge against: whatever is not sent is +// simply not in the configuration afterwards. +func ApplyConstants(m2ee M2EEOptions, bootConfig map[string]any, constants map[string]string) error { + if len(bootConfig) == 0 { + return fmt.Errorf("no boot configuration to re-send") + } + payload := make(map[string]any, len(bootConfig)) + for k, v := range bootConfig { + payload[k] = v + } + if constants == nil { + constants = map[string]string{} + } + payload["MicroflowConstants"] = constants + + if _, err := CallM2EE(m2ee, "update_configuration", payload); err != nil { + return fmt.Errorf("update_configuration: %w", err) + } + if _, err := CallM2EE(m2ee, "reload_model", nil); err != nil { + return fmt.Errorf("reload_model (the new values are staged but NOT in effect): %w", err) + } + return nil +} + // openRuntimeLog opens (creating the parent dir) the runtime log for appending // and writes a start marker. A prior handle (from an earlier spawn/restart) is // closed first so the file is reused across restarts rather than leaked. diff --git a/cmd/mxcli/docker/localboot_constants_test.go b/cmd/mxcli/docker/localboot_constants_test.go index e00e97db5..724404a8d 100644 --- a/cmd/mxcli/docker/localboot_constants_test.go +++ b/cmd/mxcli/docker/localboot_constants_test.go @@ -2,7 +2,14 @@ package docker -import "testing" +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "testing" +) // mxcli-chat FINDINGS §33: the configuration's constant values have to reach the // runtime, and §33's caveat: they must be MERGED over mxbuild's resolved @@ -76,3 +83,81 @@ func TestLocalAppOptions_ForwardsToTheRuntime(t *testing.T) { t.Errorf("InstallPath = %q", rt.InstallPath) } } + +// ApplyConstants is the live path: change a running app's constants without a +// restart. Both calls are load-bearing — update_configuration is STAGED (the app +// keeps its old values and still answers result:0) and only reload_model applies +// them, measured on 11.12.1. A version that sent only the first call would report +// success and change nothing. +func TestApplyConstants_SendsTheConfigurationAndThenReloads(t *testing.T) { + var actions []string + var sent map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var body struct { + Action string `json:"action"` + Params map[string]any `json:"params"` + } + _ = json.NewDecoder(r.Body).Decode(&body) + actions = append(actions, body.Action) + if body.Action == "update_configuration" { + sent = body.Params + } + _, _ = w.Write([]byte(`{"result":0,"feedback":{}}`)) + })) + defer srv.Close() + + boot := map[string]any{ + "BasePath": "/deploy", + "DatabaseName": "app", + "MicroflowConstants": map[string]string{"A.Old": "stale"}, + } + if err := ApplyConstants(m2eeFor(srv), boot, map[string]string{"A.New": "fresh"}); err != nil { + t.Fatalf("ApplyConstants: %v", err) + } + + if len(actions) != 2 || actions[0] != "update_configuration" || actions[1] != "reload_model" { + t.Fatalf("actions = %v, want update_configuration then reload_model — without the "+ + "reload the call is staged and the app keeps the old value", actions) + } + // The rest of the boot payload is re-sent, because the admin API has no + // read-back: anything not sent is simply gone from the configuration. + if sent["BasePath"] != "/deploy" || sent["DatabaseName"] != "app" { + t.Errorf("the boot payload was not carried through: %v", sent) + } + if got, ok := sent["MicroflowConstants"].(map[string]any); !ok || got["A.New"] != "fresh" { + t.Errorf("MicroflowConstants = %v, want the new map", sent["MicroflowConstants"]) + } + if got := sent["MicroflowConstants"].(map[string]any); got["A.Old"] != nil { + t.Errorf("the old constants map survived: %v", got) + } +} + +// The caller's map must not be mutated, and neither must the stored boot config — +// a dev loop re-reads its own BootConfig on every apply. +func TestApplyConstants_DoesNotMutateTheBootConfig(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`{"result":0,"feedback":{}}`)) + })) + defer srv.Close() + + boot := map[string]any{"BasePath": "/deploy", "MicroflowConstants": map[string]string{"A.Old": "stale"}} + if err := ApplyConstants(m2eeFor(srv), boot, map[string]string{"A.New": "fresh"}); err != nil { + t.Fatalf("ApplyConstants: %v", err) + } + got, _ := boot["MicroflowConstants"].(map[string]string) + if got["A.Old"] != "stale" { + t.Errorf("the caller's boot config was mutated: %v", boot) + } +} + +func TestApplyConstants_RefusesWithoutABootConfig(t *testing.T) { + if err := ApplyConstants(M2EEOptions{}, nil, map[string]string{"A.B": "v"}); err == nil { + t.Error("applying with no boot payload was accepted; it would blank the configuration") + } +} + +func m2eeFor(srv *httptest.Server) M2EEOptions { + u, _ := url.Parse(srv.URL) + port, _ := strconv.Atoi(u.Port()) + return M2EEOptions{Host: u.Hostname(), Port: port, Token: "pass", Direct: true} +} diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index fe77b7bd8..455db38e7 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -38,6 +38,11 @@ type LocalAppInfo struct { AdminPort int ServePort int AdminPass string + // BootConfig is the update_configuration payload the runtime was started + // with. A caller wanting to change ONE setting on the running app re-sends + // this with that key replaced — the admin API has no read-back, so anything + // not re-sent is simply gone from the configuration. + BootConfig map[string]any } // LocalRunOptions configures RunLocal. @@ -757,10 +762,11 @@ func RunLocal(opts LocalRunOptions) error { if opts.OnReady != nil { opts.OnReady(LocalAppInfo{ - AppPort: opts.AppPort, - AdminPort: opts.AdminPort, - ServePort: opts.ServePort, - AdminPass: opts.AdminPass, + AppPort: opts.AppPort, + AdminPort: opts.AdminPort, + ServePort: opts.ServePort, + AdminPass: opts.AdminPass, + BootConfig: rt.BootConfig(), }) } diff --git a/docs-site/src/tools/running-tests.md b/docs-site/src/tools/running-tests.md index 4fe304bc9..1c91bb007 100644 --- a/docs-site/src/tools/running-tests.md +++ b/docs-site/src/tools/running-tests.md @@ -51,6 +51,11 @@ Highest layer wins: `mxcli constant list` shows the winner for each constant and which layer set it, masking machine-local values unless `--show-values` is passed. +A machine-local value normally takes effect at the next boot. `mxcli constant +set … --apply` pushes it into a `mxcli run --local` that is already up, as +`update_configuration` followed by `reload_model` — both are needed, since the +first call only stages the change. + ## Basic Usage ```bash diff --git a/docs/11-proposals/PROPOSAL_constant_values.md b/docs/11-proposals/PROPOSAL_constant_values.md index 050900e0f..6b907135c 100644 --- a/docs/11-proposals/PROPOSAL_constant_values.md +++ b/docs/11-proposals/PROPOSAL_constant_values.md @@ -6,7 +6,7 @@ date: 2026-08-13 # Proposal: Constant values — one precedence chain, and a slot for secrets -**Status:** Accepted — slices 1, 2 and 3 shipped; 4 open +**Status:** Accepted — all four slices shipped **Date:** 2026-08-13 A Mendix constant has a value in four possible places, mxcli can write two of @@ -151,8 +151,13 @@ mxcli constant set MyModule.ApiKey 'sk-…' --local --apply ``` writes layer 2 **and** applies it to a running dev loop, as -`update_configuration` (full payload, constants overlaid) → `reload_model` → -**verify by observation**. Verification is not optional: the admin API returned +`update_configuration` (full payload, constants overlaid) → `reload_model`. +**Correction, found while building it:** "verify by observation" is not +achievable from outside the app. A constant's value is not exposed by any admin +action, so mxcli cannot confirm the change — it performs both calls, reports +exactly what it did, and names what *would* confirm it (a microflow that returns +the constant). The original wording promised more than the API allows. The admin +API returned success for the call that changed nothing, which is precisely the §33 shape. This only works where mxcli owns the boot payload (`run --local`), because the @@ -231,7 +236,7 @@ override is silently ignored by the runtime, which is the §33 shape again. `constant set` refuses to write a name the project does not declare, and warns when the same constant also has a shared override, naming which one wins. -### Slice 4 — `--apply` (the live path) +### Slice 4 — `--apply` (the live path) — **shipped** | File | Change | |------|--------| From 92e1df019ad4bd839c8ff10fc71cd2b5659b6695 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 13:30:27 +0000 Subject: [PATCH 14/20] fix(microflow): author and round-trip the import-mapping Range Mendix stores the import activity's Range on the ImportMappingCall in two variants: Microflows$ConstantRange{SingleObject} All / First Microflows$CustomRange{LimitExpression, OffsetExpression} Custom mxcli only ever wrote the first and read only SingleObject, so "Custom" was not merely undescribed but unrepresentable: a bounded import became unbounded the moment anything rewrote the activity. DESCRIBE emitted no range at all, so all three settings round-tripped identically and describe -> edit -> exec silently changed the activity's meaning. Adds the trailing clause: import from mapping M.IMM($src) [all | first | limit [offset ]] Omitting it keeps the pre-existing inference from the mapping's root shape, so scripts that predate the syntax write exactly what they always did. DESCRIBE always emits one of the three, because silence re-enters that inference and an object-rooted mapping set to All -- Studio Pro's own default, shipped in the blank app -- comes back as First. The Range and the result variable's cardinality are separate axes. Folding them writes a ListType against an object-rooted mapping, which mxbuild rejects with CE0243; Mendix's own FeedbackModule.SUB_Feedback_PostToAppInsights pairs ConstantRange{SingleObject:false} with an ObjectType variable. The stored VariableType is the authority on cardinality, the range is its own flag (RangeSingleObject, nil = follow the inference), and only `first` pins both. The ImportMappingCall is built at three sites -- the import statement, REST result handling, and the legacy writer -- so the range selection is one shared helper per engine; fixing two of the three let a limit reach the model while a ConstantRange was still written. Both engines are fixed: they share the semantic model, and a fix in one is invisible to a user on the other. Verified end-to-end on mxbuild 11.6.6 (mx check: 0 errors on both engines, all three ranges re-describing verbatim). `offset` is rejected by Mendix with CE6100 unless the mapping's root is a list, while `limit` is accepted either way; that is documented rather than validated, because mxcli cannot currently author a list-rooted import mapping to test the positive case against. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- .claude/skills/fix-issue.md | 1 + .../mendix/json-structures-and-mappings.md | 26 +++ cmd/mxcli/syntax/features_microflow.go | 24 +++ docs/01-project/MDL_QUICK_REFERENCE.md | 2 +- .../bug-tests/881-import-mapping-range.mdl | 90 +++++++++ mdl/ast/ast_microflow.go | 15 ++ .../modelsdk/microflow_import_range_test.go | 191 ++++++++++++++++++ .../modelsdk/microflow_read_actions.go | 33 ++- mdl/backend/modelsdk/microflow_write.go | 36 +++- mdl/executor/cmd_microflows_builder_calls.go | 41 +++- mdl/executor/cmd_microflows_format_action.go | 40 +++- .../cmd_microflows_import_range_test.go | 171 ++++++++++++++++ mdl/grammar/MDLLexer.g4 | 1 + mdl/grammar/domains/MDLMicroflow.g4 | 27 +++ mdl/grammar/domains/MDLSettings.g4 | 2 +- mdl/visitor/visitor_import_export_mapping.go | 22 +- sdk/microflows/microflows_actions.go | 31 +++ sdk/mpr/parser_microflow_actions.go | 59 +++++- sdk/mpr/parser_microflow_import_range_test.go | 165 +++++++++++++++ sdk/mpr/writer_microflow_actions.go | 38 +++- 20 files changed, 981 insertions(+), 34 deletions(-) create mode 100644 mdl-examples/bug-tests/881-import-mapping-range.mdl create mode 100644 mdl/backend/modelsdk/microflow_import_range_test.go create mode 100644 mdl/executor/cmd_microflows_import_range_test.go create mode 100644 sdk/mpr/parser_microflow_import_range_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index dcf2816cc..791cf8a20 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -490,3 +490,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `mxcli new` into a deep output directory dies with `System.IO.PathTooLongException` and leaves the directory holding ~259 files and no `.mpr` — output that looks like a project until you try to open it | MxToolset refuses any full destination path over **259 characters** (its own Windows-compatibility limit, not the filesystem's) and aborts extraction PART WAY THROUGH. mxcli pointed `mx create-project` straight at the user's output directory, so the abort happened there | `cmd/mxcli/newproject_paths.go` (new: `stagedProjectDirs`, `longestRelativePath`, `warnIfPathTooLongForStudioPro`, `moveProject`), wired into `cmd/mxcli/cmd_new.go` step 2 | **Stage the work somewhere safe and move it in, rather than validating a path you could just avoid** — creating in a short temp dir and renaming makes ANY depth work (POSIX allows 4096) instead of merely failing politely, and the destination is never partially populated because nothing is written there until creation succeeded. Check the relocation is safe first: `grep -rl ` returned **0 files**, so a fresh Mendix project embeds no absolute paths. **Bisect for the real threshold instead of trusting arithmetic** — 77 characters creates a project on 11.13.0, 78 fails leaving 259 files, which pins `len(dest) + 1 + longest ≤ 259` exactly. **Measure the template, don't hardcode it**: the longest relative path is 181 on 11.13.0 and 182 on 11.12.0, so walk the staged tree and use the real number, or the reported budget drifts a character per release. **Warn, don't refuse, when the finished path is over budget** — the project works on POSIX, so refusing would block a machine where it is fine; but Studio Pro on Windows would not open it, so silence would be worse. Cross-device staging needs an `os.CopyFS` fallback: `os.Rename` cannot cross filesystems. Tests `cmd/mxcli/newproject_paths_test.go`. upstream #825 | | `show callers of ` and `show references to ` report "(no callers found)" for a document reached only from a page action button — a false negative that reads as "safe to delete" | TWO independent defects behind one symptom. (1) `scanWidgetOwnRefs` collected `Entity`/`Microflow`/`Nanoflow` from a widget's raw BSON but not **`Form`**, the key a PAGE reference uses, so `widgets_data` had no page column and the refs projection had no page row. (2) `execShowCallers` filtered `RefKind = 'call'` — the kind a microflow CALL ACTIVITY produces — so the button→microflow row, which was already in the refs table, was hidden by the query | `mdl/catalog/builder_pages.go` (`scanWidgetOwnRefs` + `rawWidgetInfo.PageRef`), `mdl/catalog/tables.go` (`widgets_data.PageRef`), `mdl/catalog/builder_references.go` (projection row), `mdl/executor/cmd_search.go` (`callerRefKinds`) | **Find the live code path before fixing anything** — `builder_references.go` has an inviting `extractWidgetRefs` with a per-widget-type switch, and it is DEAD: its only caller is its own recursion. Extending it changes nothing. The live path is a SQL projection out of `widgets_data`, and the standing `NOTE: widget-level datasource/action refs still require a parsed widget tree` comment beside it is the tell. **Separate "the reference is missing" from "the query hides it"** by reading the refs table directly: the button→microflow row was present all along, so fixing only the scanner would have closed half the issue and left the reporter's second scenario broken. **`Form` is `Page`** — the same rename behind `ShowFormAction`/`CloseFormAction` (CLAUDE.md's storage-name table); grepping for `Page` in a BSON scanner finds nothing. **One action can carry two references**: `create object … then open page` holds an entity AND a page, so collecting the entity alone still leaves the page unreferenced. **Do not widen `callers` into `references`** — `datasource`/`parameter`/`generalize` are uses of a TYPE, not invocations, and including them makes the two commands synonyms; the test pins both the included and the excluded set. Tests `builder_pages_test.go` (`TestScanWidgetOwnRefs_PageReference`), `cmd_search_callers_test.go`. upstream #773 | | `ALTER PAGE` over `--mcp` fails against Studio Pro **11.13** with `pg_patch_page: … PROP_NOT_PRIMITIVE: Property 'widgets' is not a primitive property`. `CREATE PAGE` is fine; the page itself is left intact | 11.13 gave `pg_read_page` a **`depth` argument defaulting to 4**, replacing anything deeper with the literal string `"..."`. ALTER PAGE is read-modify-**replace-whole-page**, so the truncated read went straight back as the new page body. Measured live: `Administration.Account_Overview` read 32,594 bytes at full depth but **1,052 bytes** at the default, its entire tree reduced to `{"widgets":["...","..."]}`. Every ordinary page truncates — three of three PgTest pages did | `mdl/backend/mcp/page.go` (`pgReadPage`, `pgReadFullDepth`, `hasTruncationSentinel`), `mdl/backend/mcp/client.go` (`SupportsToolArg`) | Request the full depth, and **guard rather than trust it**: refuse a read still carrying the sentinel instead of letting a partial page reach a write (ADR-0005 guard-don't-drop). Two traps. (1) **Do not send `depth` unconditionally** — 11.11/11.12 declare `pg_read_page` `additionalProperties:false` without it, so the whole call fails; gate on a live `tools/list` probe of the tool's input schema, because `serverInfo.version` is frozen at `1.0.0` across 11.11/11.12/11.13 and cannot discriminate releases. (2) **Match the sentinel only as an array element** — a caption or title legitimately reading `"..."` is real content, and a naive substring scan rejects valid pages. The release notes announced none of this, exactly as 11.12 silently removed `pg_write_page` (#697): on any Studio Pro upgrade, re-probe `tools/list` and diff the input schemas, not just the tool names. Tests `mdl/backend/mcp/page_depth_test.go`; controls: stub the depth arg (full-depth test fails) and stub the guard (truncation test fails) | +| An import activity's Range — Studio Pro's **All / First / Custom** — is absent from `DESCRIBE MICROFLOW`, and a `limit`/`offset` set in Studio Pro does not survive an mxcli round trip | Worse than "undescribed". `Microflows$ImportMappingCall.Range` is polymorphic — `ConstantRange{SingleObject}` (All/First) or `CustomRange{LimitExpression, OffsetExpression}` (Custom) — and mxcli wrote only the first and read only `SingleObject`. So **Custom was unrepresentable**, a bounded import became unbounded on any rewrite, and all three settings described identically, so describe→edit→exec silently changed the activity | `mdl/grammar/domains/MDLMicroflow.g4` (`importMappingRange`) + `MDLLexer.g4` (`FIRST`) + `MDLSettings.g4` (keyword list), `mdl/visitor/visitor_import_export_mapping.go`, `mdl/ast/ast_microflow.go`, `sdk/microflows/microflows_actions.go` (`RangeSingleObject`, `RangeSingleObjectOf`), `mdl/executor/cmd_microflows_builder_calls.go` (`addImportFromMappingAction`), `mdl/executor/cmd_microflows_format_action.go` (`formatImportMappingRange`), `mdl/backend/modelsdk/microflow_write.go` (`importMappingRangeToGen`) + `microflow_read_actions.go`, `sdk/mpr/writer_microflow_actions.go` (`importMappingRange`) + `parser_microflow_actions.go` | **The Range and the result variable's CARDINALITY are separate axes** — the first fix folded them and every `all` wrote a `ListType` against an object-rooted mapping, which mxbuild rejects with **CE0243** ("the mapping used to return 'List of X' but now returns 'X'"). Mendix's own `FeedbackModule.SUB_Feedback_PostToAppInsights` settles it: `ConstantRange{SingleObject:false}` against an **ObjectType** variable. The stored `VariableType` is the authority on cardinality; the range is its own flag (hence `RangeSingleObject *bool`, nil = "not authored, fall back"). **Count the write sites before declaring a serialization fixed** — the `ImportMappingCall` is built at THREE places (`importXmlActionToGen`, the REST `ResultHandlingMapping` case, and the legacy writer), and patching two let a limit reach the model while a `ConstantRange` was still written; extract one helper. **DESCRIBE must emit one of the three, never nothing** — silence re-enters the builder's inference, and an object-rooted mapping set to All (Studio Pro's default) comes back as First. **`first` is not `limit 1`**: one binds an OBJECT, the other a one-element LIST, so they cannot share syntax. **A platform rule you cannot author a positive case for is documentation, not validation** — Mendix rejects `offset` on a non-list mapping with **CE6100** while accepting `limit`, and mxcli cannot currently author a list-rooted import mapping (array-root mappings emit CE5015), so the constraint is documented rather than guessed at in a checker. Tests `mdl/executor/cmd_microflows_import_range_test.go`, `mdl/backend/modelsdk/microflow_import_range_test.go`, `sdk/mpr/parser_microflow_import_range_test.go`, example `mdl-examples/bug-tests/881-import-mapping-range.mdl`. upstream #881 | diff --git a/.claude/skills/mendix/json-structures-and-mappings.md b/.claude/skills/mendix/json-structures-and-mappings.md index d709bf41d..765658c53 100644 --- a/.claude/skills/mendix/json-structures-and-mappings.md +++ b/.claude/skills/mendix/json-structures-and-mappings.md @@ -304,6 +304,32 @@ $PetResponse = import from mapping Module.IMM_Pet($JsonContent); import from mapping Module.IMM_Pet($JsonContent); ``` +#### Range — how much of the result to bind + +Optional trailing clause, matching Studio Pro's **All / First / Custom** setting +on the activity. Omit it and mxcli infers from the mapping's own root shape, as +it always has. + +```sql +$Pets = import from mapping Module.IMM_Pets($Json) all; -- All (the default) +$Pet = import from mapping Module.IMM_Pets($Json) first; -- First: ONE object +$Page = import from mapping Module.IMM_Pets($Json) limit 10; -- Custom +$Page = import from mapping Module.IMM_Pets($Json) limit 10 offset 5; +``` + +`first` is a separate word from `limit 1` on purpose: `limit 1` is a *list* of +one, `first` binds a single *object*, so the result variable's type differs. + +Two things the range does **not** do: + +- **It does not change what the mapping returns.** An object-rooted mapping + binds an object under every range — `all` on one is Studio Pro's own default, + and the blank app ships one (`FeedbackModule.SUB_Feedback_PostToAppInsights`). + Only `first` narrows a list mapping to a single object. +- **`offset` is not accepted everywhere.** Mendix rejects it with + **CE6100** ("This entity does not support offset") unless the mapping's root + is a list; `limit` alone is fine either way. Verified on mxbuild 11.6.6. + ### Export to Mapping (entity → JSON) ```sql diff --git a/cmd/mxcli/syntax/features_microflow.go b/cmd/mxcli/syntax/features_microflow.go index c151f94a7..8bc7a4f37 100644 --- a/cmd/mxcli/syntax/features_microflow.go +++ b/cmd/mxcli/syntax/features_microflow.go @@ -179,4 +179,28 @@ func init() { Example: "VALIDATION FEEDBACK $Order/Quantity MESSAGE 'Quantity must be positive';\nVALIDATION FEEDBACK $Customer/Email MESSAGE '{1} is not valid'\n OBJECTS [$Customer/Email];", SeeAlso: []string{"microflow.error-handling"}, }) + + Register(SyntaxFeature{ + Path: "microflow.mapping", + Summary: "IMPORT FROM MAPPING / EXPORT TO MAPPING, and the import Range (All/First/Custom)", + Keywords: []string{ + "import from mapping", "export to mapping", "import mapping activity", + "range", "all", "first", "limit", "offset", "single object", + }, + Syntax: "[$Var =] IMPORT FROM MAPPING Module.IMM ($SourceVar) [];\n" + + "$Var = EXPORT TO MAPPING Module.EMM ($EntityVar);\n\n" + + " — Studio Pro's Range on the import activity:\n" + + " ALL bind the whole result\n" + + " FIRST bind ONE object (not a one-element list)\n" + + " LIMIT [OFFSET ] a bounded list\n\n" + + "Omit it and the cardinality is inferred from the mapping's root shape.\n" + + "The range does not change WHAT the mapping returns: an object-rooted\n" + + "mapping binds an object under ALL too (Studio Pro's own default).\n" + + "Mendix rejects OFFSET on a non-list mapping with CE6100.", + Example: "$Pets = import from mapping Shop.IMM_Pets($Json) all;\n" + + "$Pet = import from mapping Shop.IMM_Pets($Json) first;\n" + + "$Page = import from mapping Shop.IMM_Pets($Json) limit 10 offset 5;\n" + + "$Json2 = export to mapping Shop.EMM_Pet($Pet);", + SeeAlso: []string{"import-mapping", "export-mapping", "json-structure"}, + }) } diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index fa8ab3e1b..057810ebf 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -408,7 +408,7 @@ it is for pages. | WHILE | `while condition begin ... end while;` | Condition-based loop | | Return | `return $value;` | Required at end of every flow path | | Execute DB query | `$Result = execute database query Module.Conn.Query;` | 3-part name; supports DYNAMIC, params, CONNECTION override | -| Import mapping | `[$Var =] import from mapping Module.IMM($SourceVar);` | Apply import mapping to string variable | +| Import mapping | `[$Var =] import from mapping Module.IMM($SourceVar) [all\|first\|limit [offset ]];` | Apply import mapping to string variable. Trailing clause is Studio Pro's Range; omitted = infer from the mapping's root. `first` binds one OBJECT (`limit 1` is a one-element LIST). Mendix rejects `offset` on a non-list mapping (CE6100) | | Export mapping | `$Var = export to mapping Module.EMM($EntityVar);` | Apply export mapping to entity, returns string | | Error handling | `... on error continue\|rollback\|{ handler };` | Not supported on EXECUTE DATABASE QUERY | diff --git a/mdl-examples/bug-tests/881-import-mapping-range.mdl b/mdl-examples/bug-tests/881-import-mapping-range.mdl new file mode 100644 index 000000000..ca4a631d7 --- /dev/null +++ b/mdl-examples/bug-tests/881-import-mapping-range.mdl @@ -0,0 +1,90 @@ +-- Bug test for upstream issue #881: the import activity's Range. +-- +-- Mendix stores the Range on the ImportMappingCall in two variants: +-- +-- Microflows$ConstantRange{SingleObject} All / First +-- Microflows$CustomRange{LimitExpression, OffsetExpression} Custom +-- +-- mxcli only ever wrote the first, and read only SingleObject, so: +-- * "Custom" was not merely undescribed but UNREPRESENTABLE — a bounded +-- import became unbounded the moment anything rewrote the activity; +-- * DESCRIBE emitted no range at all, so all three settings round-tripped +-- identically and describe -> edit -> exec silently changed the activity. +-- +-- The trap the fix had to avoid: the Range and the RESULT VARIABLE's +-- cardinality are SEPARATE axes. Conflating them writes a ListType against an +-- object-rooted mapping, which mxbuild rejects with CE0243 ("the mapping used +-- to return 'List of X' but now returns 'X'"). Mendix's own +-- FeedbackModule.SUB_Feedback_PostToAppInsights is the proof: it stores +-- ConstantRange{SingleObject:false} against an ObjectType variable. +-- +-- Expected: `mx check` reports 0 errors, and re-describing each microflow +-- reproduces the range verbatim. Verified on mxbuild 11.6.6, both engines. +-- +-- NOTE on `offset`: Mendix rejects it with CE6100 ("This entity does not +-- support offset") unless the mapping's root is a list — `limit` alone is +-- accepted either way. The mapping below is object-rooted, so this file uses +-- `limit` without `offset`; adding one is the way to reproduce CE6100. + +create json structure MyFirstModule.JSON_881 +snippet '{"items": [{"total": 5, "name": "a"}]}'; + +create non-persistent entity MyFirstModule.Item881 ( Total: integer, Name: string ); +/ +create non-persistent entity MyFirstModule.Root881 ( Dummy: string ); +/ +create association MyFirstModule.Item881_Root881 + from MyFirstModule.Item881 to MyFirstModule.Root881; +/ + +-- Object-rooted: the mapping returns ONE Root881 under every range. +create import mapping MyFirstModule.IMM_881 + with json structure MyFirstModule.JSON_881 +{ + create MyFirstModule.Root881 { + create MyFirstModule.Item881_Root881/MyFirstModule.Item881 = items { + Total = total, + Name = name + } + } +}; +/ + +-- ALL: Studio Pro's default. The variable stays an OBJECT because the mapping +-- is object-rooted — writing a list here is the CE0243 case. +create microflow MyFirstModule.MF881_All ( $P: string ) +returns MyFirstModule.Root881 as $R +begin + $R = import from mapping MyFirstModule.IMM_881($P) all; + return $R; +end; +/ + +-- FIRST: the one form that also pins the variable to a single object. +create microflow MyFirstModule.MF881_First ( $P: string ) +returns MyFirstModule.Root881 as $R +begin + $R = import from mapping MyFirstModule.IMM_881($P) first; + return $R; +end; +/ + +-- CUSTOM: selects Microflows$CustomRange. A range setting only — it does not +-- change what the mapping returns. +create microflow MyFirstModule.MF881_Custom ( $P: string ) +returns MyFirstModule.Root881 as $R +begin + $R = import from mapping MyFirstModule.IMM_881($P) limit 10; + return $R; +end; +/ + +-- Unauthored: keeps the pre-#881 inference, so every hand-written script that +-- predates the syntax writes exactly what it always did. +create microflow MyFirstModule.MF881_Inferred ( $P: string ) +returns MyFirstModule.Root881 as $R +begin + $R = import from mapping MyFirstModule.IMM_881($P); + return $R; +end; +/ diff --git a/mdl/ast/ast_microflow.go b/mdl/ast/ast_microflow.go index 8c72e331e..efafa833c 100644 --- a/mdl/ast/ast_microflow.go +++ b/mdl/ast/ast_microflow.go @@ -838,6 +838,21 @@ type ImportFromMappingStmt struct { SourceVariable string // Input string variable (without $) ErrorHandling *ErrorHandlingClause // Optional ON ERROR clause Annotations *ActivityAnnotations // Optional @position, @caption, @color, @annotation + + // Range — how much of the mapping's result to bind. Mendix stores this on + // the ImportMappingCall as ConstantRange{SingleObject} or + // CustomRange{LimitExpression, OffsetExpression}; before #881 MDL could say + // none of it, so all three settings described identically and a + // describe→edit→exec cycle silently changed the activity's meaning. + // + // All fields unset = the range was not authored, and the builder keeps + // inferring cardinality from the mapping's own root shape, as it always has. + // DESCRIBE always emits one of All/First/Limit so a round trip cannot fall + // back on that inference and change the activity's meaning. + All bool // ALL — bind the whole list, explicitly + First bool // FIRST — bind ONE object rather than a list + LimitExpr Expression // LIMIT — Custom range + OffsetExpr Expression // OFFSET — Custom range } func (s *ImportFromMappingStmt) isMicroflowStatement() {} diff --git a/mdl/backend/modelsdk/microflow_import_range_test.go b/mdl/backend/modelsdk/microflow_import_range_test.go new file mode 100644 index 000000000..b44cac29c --- /dev/null +++ b/mdl/backend/modelsdk/microflow_import_range_test.go @@ -0,0 +1,191 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/sdk/microflows" + "go.mongodb.org/mongo-driver/v2/bson" +) + +// upstream #881: the import activity's Range round trip. +// +// Mendix stores two variants and mxcli only ever wrote the first, so the +// "Custom" setting was not merely undescribed but UNREPRESENTABLE: +// +// Microflows$ConstantRange{SingleObject} All / First +// Microflows$CustomRange{LimitExpression, OffsetExpression} Custom +// +// The reader has to dispatch on $Type for the same reason: without it a Custom +// range read back as All and the limit was lost on the next write. + +func importRangeBSON(rng bson.D, variableType string) bson.Raw { + return mustMarshalFlow(bson.D{ + {Key: "$ID", Value: "a-1"}, + {Key: "$Type", Value: "Microflows$ImportXmlAction"}, + {Key: "ErrorHandlingType", Value: "Rollback"}, + {Key: "XmlDocumentVariableName", Value: "resp"}, + {Key: "ResultHandling", Value: bson.D{ + {Key: "$ID", Value: "rh-1"}, + {Key: "$Type", Value: "Microflows$ResultHandling"}, + {Key: "ResultVariableName", Value: "out"}, + {Key: "ImportMappingCall", Value: bson.D{ + {Key: "$ID", Value: "imc-1"}, + {Key: "$Type", Value: "Microflows$ImportMappingCall"}, + {Key: "ReturnValueMapping", Value: "M.IMM"}, + {Key: "ForceSingleOccurrence", Value: false}, + {Key: "Range", Value: rng}, + }}, + {Key: "VariableType", Value: bson.D{ + {Key: "$Type", Value: variableType}, + {Key: "Entity", Value: "M.Root"}, + }}, + }}, + }) +} + +func readImportRange(t *testing.T, raw bson.Raw) *microflows.ResultHandlingMapping { + t.Helper() + var d bson.D + if err := bson.Unmarshal(raw, &d); err != nil { + t.Fatal(err) + } + act := decodeAction(t, d) + im, ok := act.(*microflows.ImportXmlAction) + if !ok { + t.Fatalf("actionFromGen → %T, want *microflows.ImportXmlAction", act) + } + if im.ResultHandling == nil { + t.Fatal("ResultHandling nil") + } + return im.ResultHandling +} + +// A CustomRange must survive the read. Before #881 the reader looked only at +// SingleObject, so a bounded import described as `all` and the next exec wrote +// it back unbounded. +func TestReadImportRange_Custom(t *testing.T) { + h := readImportRange(t, importRangeBSON(bson.D{ + {Key: "$ID", Value: "r-1"}, + {Key: "$Type", Value: "Microflows$CustomRange"}, + {Key: "LimitExpression", Value: "10"}, + {Key: "OffsetExpression", Value: "5"}, + }, "DataTypes$ListType")) + + if h.LimitExpression != "10" || h.OffsetExpression != "5" { + t.Errorf("limit/offset = %q/%q, want 10/5", h.LimitExpression, h.OffsetExpression) + } + if h.SingleObject { + t.Error("SingleObject = true, want false (ListType variable)") + } +} + +// The shape Mendix itself ships in the blank app +// (FeedbackModule.SUB_Feedback_PostToAppInsights): range All against an OBJECT +// variable. The two axes disagree here, so a reader that folds one into the +// other necessarily loses a setting — this one described as `first` and rewrote +// the activity on the next exec. +func TestReadImportRange_AllAgainstAnObjectVariable(t *testing.T) { + h := readImportRange(t, importRangeBSON(bson.D{ + {Key: "$ID", Value: "r-1"}, + {Key: "$Type", Value: "Microflows$ConstantRange"}, + {Key: "SingleObject", Value: false}, + }, "DataTypes$ObjectType")) + + if h.RangeSingleObject == nil || *h.RangeSingleObject { + t.Errorf("RangeSingleObject = %v, want explicit false — the range is All", h.RangeSingleObject) + } + if !h.SingleObject { + t.Error("SingleObject = false, want true — the stored ObjectType is the authority " + + "on the variable's cardinality, not the range") + } +} + +func TestReadImportRange_First(t *testing.T) { + h := readImportRange(t, importRangeBSON(bson.D{ + {Key: "$ID", Value: "r-1"}, + {Key: "$Type", Value: "Microflows$ConstantRange"}, + {Key: "SingleObject", Value: true}, + }, "DataTypes$ObjectType")) + + if h.RangeSingleObject == nil || !*h.RangeSingleObject { + t.Errorf("RangeSingleObject = %v, want explicit true", h.RangeSingleObject) + } + if !h.SingleObject { + t.Error("SingleObject = false, want true") + } +} + +// The write side of the same separation: a limit selects CustomRange, and the +// VariableType follows SingleObject rather than the range. Emitting a ListType +// for an object-rooted mapping is mxbuild's CE0243. +func TestWriteImportRange(t *testing.T) { + single := true + for _, tc := range []struct { + name string + h *microflows.ResultHandlingMapping + wantRange string + wantVarTyp string + }{ + { + "custom range against an object variable", + µflows.ResultHandlingMapping{SingleObject: true, LimitExpression: "10", OffsetExpression: "5"}, + "Microflows$CustomRange", "DataTypes$ObjectType", + }, + { + "all against an object variable", + µflows.ResultHandlingMapping{SingleObject: true, RangeSingleObject: new(bool)}, + "Microflows$ConstantRange", "DataTypes$ObjectType", + }, + { + "first against a list mapping", + µflows.ResultHandlingMapping{SingleObject: true, RangeSingleObject: &single}, + "Microflows$ConstantRange", "DataTypes$ObjectType", + }, + } { + g := importXmlActionToGen(µflows.ImportXmlAction{ResultHandling: tc.h}) + raw, err := (&codec.Encoder{}).Encode(g) + if err != nil { + t.Fatalf("%s: encode: %v", tc.name, err) + } + var doc bson.Raw = raw + rh, ok := doc.Lookup("ResultHandling").DocumentOK() + if !ok { + t.Fatalf("%s: no ResultHandling", tc.name) + } + imc, _ := rh.Lookup("ImportMappingCall").DocumentOK() + rng, _ := imc.Lookup("Range").DocumentOK() + if got := rawStr(rng, "$Type"); got != tc.wantRange { + t.Errorf("%s: Range $Type = %q, want %q", tc.name, got, tc.wantRange) + } + vt, _ := rh.Lookup("VariableType").DocumentOK() + if got := rawStr(vt, "$Type"); got != tc.wantVarTyp { + t.Errorf("%s: VariableType = %q, want %q", tc.name, got, tc.wantVarTyp) + } + } + + // A CustomRange carries the expressions, not SingleObject: a bounded range is + // always bounded, and SingleObject has no meaning there. + g := importXmlActionToGen(µflows.ImportXmlAction{ + ResultHandling: µflows.ResultHandlingMapping{LimitExpression: "$Size", OffsetExpression: "$Skip"}, + }) + raw, err := (&codec.Encoder{}).Encode(g) + if err != nil { + t.Fatal(err) + } + var doc bson.Raw = raw + rh, _ := doc.Lookup("ResultHandling").DocumentOK() + imc, _ := rh.Lookup("ImportMappingCall").DocumentOK() + rng, _ := imc.Lookup("Range").DocumentOK() + if got := rawStr(rng, "LimitExpression"); got != "$Size" { + t.Errorf("LimitExpression = %q, want $Size", got) + } + if got := rawStr(rng, "OffsetExpression"); got != "$Skip" { + t.Errorf("OffsetExpression = %q, want $Skip", got) + } + if _, ok := rng.Lookup("SingleObject").BooleanOK(); ok { + t.Error("a CustomRange must not carry SingleObject") + } +} diff --git a/mdl/backend/modelsdk/microflow_read_actions.go b/mdl/backend/modelsdk/microflow_read_actions.go index af13ca73d..019bc1f8a 100644 --- a/mdl/backend/modelsdk/microflow_read_actions.go +++ b/mdl/backend/modelsdk/microflow_read_actions.go @@ -489,9 +489,24 @@ func actionFromGen(el element.Element) microflows.MicroflowAction { out.ID = model.ID(a.ID()) if rh, ok := a.Raw().Lookup("ResultHandling").DocumentOK(); ok { if imc, ok := rh.Lookup("ImportMappingCall").DocumentOK(); ok { - h, force, _ := readMappingCall(rh, imc) - if !h.SingleObject && force { + h, force, vtType := readMappingCall(rh, imc) + // The stored VariableType is the authority on the result + // variable's cardinality, and it does NOT track the range: + // Mendix's own SUB_Feedback_PostToAppInsights stores + // ConstantRange{SingleObject:false} against an ObjectType. Only + // where no VariableType is stored does ForceSingleOccurrence + // stand in — and never for a bounded range, which is always a + // list, or a Custom range would read back as `first` and lose + // the limit. (issue #881) + switch vtType { + case "DataTypes$ObjectType": h.SingleObject = true + case "DataTypes$ListType": + h.SingleObject = false + default: + if !h.SingleObject && force && h.LimitExpression == "" && h.OffsetExpression == "" { + h.SingleObject = true + } } out.ResultHandling = h } @@ -826,8 +841,20 @@ func readMappingCall(doc, imc bson.Raw) (h *microflows.ResultHandlingMapping, fo h.MappingID = model.ID(mapping) force, _ = imc.Lookup("ForceSingleOccurrence").BooleanOK() h.ForceSingleOccurrence = &force + // The Range is polymorphic: a ConstantRange carries SingleObject (All/First) + // while a CustomRange carries the limit and offset expressions. Reading only + // SingleObject dropped the Custom setting entirely, so a describe→edit→exec + // cycle turned a bounded import into an unbounded one (issue #881). if rng, ok := imc.Lookup("Range").DocumentOK(); ok { - if b, ok := rng.Lookup("SingleObject").BooleanOK(); ok { + if rawStr(rng, "$Type") == "Microflows$CustomRange" { + h.LimitExpression = rawStr(rng, "LimitExpression") + h.OffsetExpression = rawStr(rng, "OffsetExpression") + } else if b, ok := rng.Lookup("SingleObject").BooleanOK(); ok { + // SingleObject is the RANGE's own flag (All/First). It is only a + // fallback for the result variable's cardinality — where a + // VariableType is stored, that wins, because the two disagree in + // Mendix's own models (see RangeSingleObject). + h.RangeSingleObject = &b h.SingleObject = b } } diff --git a/mdl/backend/modelsdk/microflow_write.go b/mdl/backend/modelsdk/microflow_write.go index 03c866ef3..f981248ae 100644 --- a/mdl/backend/modelsdk/microflow_write.go +++ b/mdl/backend/modelsdk/microflow_write.go @@ -814,6 +814,34 @@ func microflowActionToGen(action microflows.MicroflowAction) element.Element { } } +// importMappingRangeToGen builds the Range child of a Microflows$ImportMappingCall. +// +// Mendix has two variants and mxcli only ever wrote the first, so Studio Pro's +// "Custom" setting was unrepresentable rather than merely undescribed: +// +// Microflows$ConstantRange{SingleObject} All (false) / First (true) +// Microflows$CustomRange{LimitExpression, OffsetExpression} Custom +// +// A limit or an offset selects CustomRange; SingleObject has no meaning there, +// because a bounded range is always a list. +// +// Shared because the ImportMappingCall is built at THREE sites in this engine — +// importXmlActionToGen (the `import from mapping` statement), the +// ResultHandlingMapping case (REST result handling), and the legacy writer's own +// copy. Fixing one and not the others is how a limit reached the model and was +// still written as a ConstantRange. (issue #881) +func importMappingRangeToGen(h *microflows.ResultHandlingMapping) *element.Base { + if h.LimitExpression != "" || h.OffsetExpression != "" { + rng := newElem("Microflows$CustomRange", "") + addStr(rng, "LimitExpression", h.LimitExpression) + addStr(rng, "OffsetExpression", h.OffsetExpression) + return rng + } + rng := newElem("Microflows$ConstantRange", "") + addBool(rng, "SingleObject", microflows.RangeSingleObjectOf(h)) + return rng +} + // importXmlActionToGen builds a Microflows$ImportXmlAction ("import from mapping"). // Mirrors serializeImportXmlAction field-for-field, including the ImportMappingCall // sub-element (ReturnValueMapping key) and the Object/List VariableType. @@ -837,9 +865,7 @@ func importXmlActionToGen(a *microflows.ImportXmlAction) element.Element { addBool(imc, "ForceSingleOccurrence", forceSingle) addStr(imc, "ObjectHandlingBackup", "Create") addStr(imc, "ParameterVariableName", "") - rng := newElem("Microflows$ConstantRange", "") - addBool(rng, "SingleObject", rh.SingleObject) - addPart(imc, "Range", rng) + addPart(imc, "Range", importMappingRangeToGen(rh)) addStr(imc, "ReturnValueMapping", string(rh.MappingID)) var vt *element.Base @@ -1533,9 +1559,7 @@ func restResultHandlingToGen(rh microflows.ResultHandling, outputVar string) ele addBool(imc, "ForceSingleOccurrence", forceSingle) addStr(imc, "ObjectHandlingBackup", "Create") addStr(imc, "ParameterVariableName", "") - rng := newElem("Microflows$ConstantRange", "") - addBool(rng, "SingleObject", h.SingleObject) - addPart(imc, "Range", rng) + addPart(imc, "Range", importMappingRangeToGen(h)) addStr(imc, "ReturnValueMapping", string(h.MappingID)) addPart(e, "ImportMappingCall", imc) var vt *element.Base diff --git a/mdl/executor/cmd_microflows_builder_calls.go b/mdl/executor/cmd_microflows_builder_calls.go index 0837eb842..35d35d40c 100644 --- a/mdl/executor/cmd_microflows_builder_calls.go +++ b/mdl/executor/cmd_microflows_builder_calls.go @@ -1471,6 +1471,43 @@ func (fb *flowBuilder) addImportFromMappingAction(s *ast.ImportFromMappingStmt) SingleObject: true, } + // The RANGE and the RESULT VARIABLE's type are two independent axes, and + // conflating them is what made the first attempt at this emit CE0243 ("the + // mapping used to return 'List of X' but now returns 'X'"): + // + // Range which part of the source is imported — the authored keyword + // SingleObject whether the bound variable is an object or a list — the + // MAPPING's own root cardinality, inferred below + // + // Mendix's own SUB_Feedback_PostToAppInsights proves they are separate: it + // stores ConstantRange{SingleObject:false} (i.e. "all") against an ObjectType + // variable, because its mapping is object-rooted. So ALL and LIMIT set only + // the range and leave the inference alone; only FIRST also pins the variable + // to a single object, which is exactly what "first" means. (issue #881) + if s.All { + f := false + resultHandling.RangeSingleObject = &f + resultHandling.ForceSingleOccurrence = &f + } else if s.First { + t := true + resultHandling.SingleObject = true + resultHandling.RangeSingleObject = &t + resultHandling.ForceSingleOccurrence = &t + } else if s.LimitExpr != nil || s.OffsetExpr != nil { + f := false + resultHandling.ForceSingleOccurrence = &f + if s.LimitExpr != nil { + resultHandling.LimitExpression = fb.exprToString(s.LimitExpr) + } + if s.OffsetExpr != nil { + resultHandling.OffsetExpression = fb.exprToString(s.OffsetExpr) + } + } + // Only FIRST overrides the mapping's cardinality; saying nothing leaves both + // axes to the inference, which is what keeps existing scripts writing what + // they always did. + rangeAuthored := s.First + // Determine single vs list and result entity from the import mapping. // JSON structure check covers JSON-backed mappings; for XML schema or // message-definition mappings JsonStructure is empty and the root @@ -1483,7 +1520,7 @@ func (fb *flowBuilder) addImportFromMappingAction(s *ast.ImportFromMappingStmt) parts := strings.SplitN(im.JsonStructure, ".", 2) if len(parts) == 2 { if js, err := fb.backend.GetJsonStructureByQualifiedName(parts[0], parts[1]); err == nil && len(js.Elements) > 0 { - if js.Elements[0].ElementType == "Array" { + if js.Elements[0].ElementType == "Array" && !rangeAuthored { resultHandling.SingleObject = false } resolved = true @@ -1494,7 +1531,7 @@ func (fb *flowBuilder) addImportFromMappingAction(s *ast.ImportFromMappingStmt) // MaxOccurs > 1 or unbounded (-1) signals a list even when // the kind is Object. root := im.Elements[0] - if root.MaxOccurs == -1 || root.MaxOccurs > 1 { + if (root.MaxOccurs == -1 || root.MaxOccurs > 1) && !rangeAuthored { resultHandling.SingleObject = false } } diff --git a/mdl/executor/cmd_microflows_format_action.go b/mdl/executor/cmd_microflows_format_action.go index 0232d614d..168a812a0 100644 --- a/mdl/executor/cmd_microflows_format_action.go +++ b/mdl/executor/cmd_microflows_format_action.go @@ -1560,11 +1560,49 @@ func formatImportXmlAction(ctx *ExecContext, a *microflows.ImportXmlAction, enti sb.WriteString(mappingName) sb.WriteString("($") sb.WriteString(a.XmlDocumentVariable) - sb.WriteString(");") + sb.WriteString(")") + sb.WriteString(formatImportMappingRange(a.ResultHandling)) + sb.WriteString(";") return sb.String() } +// formatImportMappingRange renders the activity's Range — Studio Pro's +// All / First / Custom setting. +// +// ALWAYS emits one of the three, never nothing. Omitting it would leave the +// builder inferring cardinality from the mapping's root shape, and an +// object-rooted mapping set to All is a real state that inference turns into +// First — Studio Pro's own default, shipped in the blank app's +// FeedbackModule.IMM_PostResponse. Before this, all three settings described +// identically, so the describe→edit→exec cycle silently rewrote the activity. +// (issue #881) +func formatImportMappingRange(h *microflows.ResultHandlingMapping) string { + if h == nil { + return "" + } + if h.LimitExpression != "" || h.OffsetExpression != "" { + var sb strings.Builder + if h.LimitExpression != "" { + sb.WriteString(" limit ") + sb.WriteString(h.LimitExpression) + } + if h.OffsetExpression != "" { + sb.WriteString(" offset ") + sb.WriteString(h.OffsetExpression) + } + return sb.String() + } + // The RANGE's own flag, not the result variable's cardinality: the two + // disagree in Mendix's own models (ConstantRange{SingleObject:false} against + // an ObjectType variable), and printing the variable's would describe + // Studio Pro's "All" as `first`. + if microflows.RangeSingleObjectOf(h) { + return " first" + } + return " all" +} + // formatExportXmlAction formats an export mapping action as MDL. // Syntax: $Var = EXPORT TO MAPPING Module.EMM($SourceVar); func formatExportXmlAction(ctx *ExecContext, a *microflows.ExportXmlAction) string { diff --git a/mdl/executor/cmd_microflows_import_range_test.go b/mdl/executor/cmd_microflows_import_range_test.go new file mode 100644 index 000000000..9b9387537 --- /dev/null +++ b/mdl/executor/cmd_microflows_import_range_test.go @@ -0,0 +1,171 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + mdltypes "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// upstream #881. An import activity's Range — Studio Pro's All / First / Custom +// — was neither authorable nor described, so all three settings round-tripped +// identically and describe→edit→exec silently rewrote the activity. +// +// The trap this file guards is the one the first fix fell into: the Range and +// the RESULT VARIABLE's cardinality are SEPARATE axes. Conflating them made +// `all` write a ListType against an object-rooted mapping, which mxbuild rejects +// with CE0243 ("the mapping used to return 'List of X' but now returns 'X'"). +// Mendix's own FeedbackModule.SUB_Feedback_PostToAppInsights is the proof: it +// stores ConstantRange{SingleObject:false} against an ObjectType variable. + +func importRangeBuilder(rootIsArray bool) *flowBuilder { + return &flowBuilder{ + posX: 100, + posY: 100, + spacing: HorizontalSpacing, + varTypes: map[string]string{}, + declaredVars: map[string]string{}, + measurer: &layoutMeasurer{}, + backend: &mock.MockBackend{ + GetImportMappingByQualifiedNameFunc: func(string, string) (*model.ImportMapping, error) { + return &model.ImportMapping{JsonStructure: "M.Payload"}, nil + }, + GetJsonStructureByQualifiedNameFunc: func(string, string) (*mdltypes.JsonStructure, error) { + kind := "Object" + if rootIsArray { + kind = "Array" + } + return &mdltypes.JsonStructure{Elements: []*mdltypes.JsonElement{{ElementType: kind}}}, nil + }, + }, + } +} + +func buildImportRange(t *testing.T, rootIsArray bool, stmt *ast.ImportFromMappingStmt) *microflows.ResultHandlingMapping { + t.Helper() + fb := importRangeBuilder(rootIsArray) + stmt.Mapping = ast.QualifiedName{Module: "M", Name: "IMM"} + stmt.SourceVariable = "P" + stmt.OutputVariable = "R" + fb.addImportFromMappingAction(stmt) + action, ok := fb.objects[0].(*microflows.ActionActivity).Action.(*microflows.ImportXmlAction) + if !ok { + t.Fatalf("built %T, want *microflows.ImportXmlAction", fb.objects[0].(*microflows.ActionActivity).Action) + } + if action.ResultHandling == nil { + t.Fatal("ResultHandling nil") + } + return action.ResultHandling +} + +// `all` sets the RANGE and nothing else. Against an object-rooted mapping the +// bound variable stays a single object — writing a list there is exactly the +// CE0243 the axis separation exists to prevent. +func TestImportRange_AllLeavesVariableCardinalityToTheMapping(t *testing.T) { + h := buildImportRange(t, false, &ast.ImportFromMappingStmt{All: true}) + if microflows.RangeSingleObjectOf(h) { + t.Error("range SingleObject = true, want false — `all` is Studio Pro's All") + } + if !h.SingleObject { + t.Error("SingleObject = false, want true: an object-rooted mapping binds an " + + "OBJECT even when the range is All (mxbuild rejects the list with CE0243)") + } + + // The same statement against a list-rooted mapping binds a list. + h = buildImportRange(t, true, &ast.ImportFromMappingStmt{All: true}) + if h.SingleObject { + t.Error("SingleObject = true, want false for a list-rooted mapping") + } +} + +// `first` is the one form that DOES pin the variable: "first" means one object, +// not a one-element list, so it overrides the mapping's own shape. +func TestImportRange_FirstBindsASingleObjectEvenForAListMapping(t *testing.T) { + h := buildImportRange(t, true, &ast.ImportFromMappingStmt{First: true}) + if !microflows.RangeSingleObjectOf(h) { + t.Error("range SingleObject = false, want true — `first` is Studio Pro's First") + } + if !h.SingleObject { + t.Error("SingleObject = false, want true — `first` binds one object") + } + if h.ForceSingleOccurrence == nil || !*h.ForceSingleOccurrence { + t.Errorf("ForceSingleOccurrence = %v, want explicit true", h.ForceSingleOccurrence) + } +} + +// A limit or an offset selects Mendix's CustomRange. It is a range setting only: +// it must not drag the variable's cardinality with it, or an object-rooted +// mapping gets a ListType and CE0243 again. +func TestImportRange_LimitIsARangeNotACardinality(t *testing.T) { + h := buildImportRange(t, false, &ast.ImportFromMappingStmt{ + LimitExpr: &ast.LiteralExpr{Kind: ast.LiteralInteger, Value: "10"}, + OffsetExpr: &ast.LiteralExpr{Kind: ast.LiteralInteger, Value: "5"}, + }) + if h.LimitExpression != "10" || h.OffsetExpression != "5" { + t.Errorf("limit/offset = %q/%q, want 10/5", h.LimitExpression, h.OffsetExpression) + } + if !h.SingleObject { + t.Error("SingleObject = false, want true — the mapping is object-rooted, and the " + + "range does not change what the mapping returns") + } +} + +// Saying nothing must keep writing what mxcli always wrote: cardinality inferred +// from the mapping's root, range mirroring it. Every hand-written script that +// predates the syntax depends on this. +func TestImportRange_UnauthoredKeepsTheOldInference(t *testing.T) { + h := buildImportRange(t, true, &ast.ImportFromMappingStmt{}) + if h.RangeSingleObject != nil { + t.Errorf("RangeSingleObject = %v, want nil (unauthored)", *h.RangeSingleObject) + } + if h.SingleObject { + t.Error("SingleObject = true, want false — a list-rooted mapping still infers a list") + } + if microflows.RangeSingleObjectOf(h) { + t.Error("the stored range must fall back to the inferred cardinality when unauthored") + } +} + +// DESCRIBE always emits one of the three forms — never nothing. Omitting it +// would leave the builder inferring on re-exec, and an object-rooted mapping set +// to All (Studio Pro's default, shipped in the blank app) would come back as +// First. That silent rewrite is what #881 reported. +func TestFormatImportMappingRange(t *testing.T) { + first, no := true, false + for _, tc := range []struct { + name string + h *microflows.ResultHandlingMapping + want string + }{ + {"first", µflows.ResultHandlingMapping{SingleObject: true, RangeSingleObject: &first}, " first"}, + {"all", µflows.ResultHandlingMapping{SingleObject: false, RangeSingleObject: &no}, " all"}, + { + // Mendix's own SUB_Feedback_PostToAppInsights: range All, variable an + // object. Describing this as `first` — which reading SingleObject does — + // changes the activity on re-exec. + "all against an object variable", + µflows.ResultHandlingMapping{SingleObject: true, RangeSingleObject: &no}, + " all", + }, + {"limit", µflows.ResultHandlingMapping{LimitExpression: "10"}, " limit 10"}, + {"limit+offset", µflows.ResultHandlingMapping{LimitExpression: "10", OffsetExpression: "5"}, " limit 10 offset 5"}, + {"offset only", µflows.ResultHandlingMapping{OffsetExpression: "5"}, " offset 5"}, + { + // No explicit range recorded (a document written before #881): fall back + // to the variable's cardinality rather than emitting nothing. + "legacy single object", µflows.ResultHandlingMapping{SingleObject: true}, " first", + }, + } { + if got := formatImportMappingRange(tc.h); got != tc.want { + t.Errorf("%s: formatImportMappingRange = %q, want %q", tc.name, got, tc.want) + } + } + if got := formatImportMappingRange(nil); got != "" { + t.Errorf("nil handling = %q, want empty", got) + } +} diff --git a/mdl/grammar/MDLLexer.g4 b/mdl/grammar/MDLLexer.g4 index ea8995d3c..c4e089692 100644 --- a/mdl/grammar/MDLLexer.g4 +++ b/mdl/grammar/MDLLexer.g4 @@ -126,6 +126,7 @@ SELECT: S E L E C T; FROM: F R O M; WHERE: W H E R E; HAVING: H A V I N G; +FIRST: F I R S T; OFFSET: O F F S E T; LIMIT: L I M I T; AS: A S; diff --git a/mdl/grammar/domains/MDLMicroflow.g4 b/mdl/grammar/domains/MDLMicroflow.g4 index 0062b527d..e204bf9b3 100644 --- a/mdl/grammar/domains/MDLMicroflow.g4 +++ b/mdl/grammar/domains/MDLMicroflow.g4 @@ -633,9 +633,36 @@ sendRestRequestBodyClause */ importFromMappingStatement : (VARIABLE EQUALS)? IMPORT FROM MAPPING qualifiedName LPAREN VARIABLE RPAREN + importMappingRange? onErrorClause? ; +/** + * How much of the mapping's result to bind — Mendix's "Range" on the import + * activity, stored as Microflows$ConstantRange{SingleObject} or + * Microflows$CustomRange{LimitExpression, OffsetExpression}. + * + * (omitted) infer from the mapping's own root shape, as mxcli + * always has — hand-written MDL keeps working unchanged + * ALL All — bind the whole list, explicitly + * FIRST First — bind ONE object, not a list + * LIMIT e [OFFSET e] Custom — a bounded list + * + * DESCRIBE always emits one of the three, never nothing: an object-rooted + * mapping set to All is a real state (Studio Pro's own default — the blank + * app ships one), and the inference would turn it into First on re-exec. + * + * LIMIT/OFFSET mirror the RETRIEVE clause rather than inventing a second + * spelling for the same idea. FIRST is a separate word on purpose: `limit 1` + * is a LIST of one, while First binds a single OBJECT — different result + * variable types, so they cannot share syntax. (issue #881) + */ +importMappingRange + : ALL + | FIRST + | LIMIT limitExpr=expression (OFFSET offsetExpr=expression)? + ; + /** * Export to mapping: $Var = EXPORT TO MAPPING Module.EMM($SourceVar); */ diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index b12db73a8..3119737df 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -525,7 +525,7 @@ keyword // Query / SQL | SELECT | FROM | WHERE | JOIN | LEFT | RIGHT | INNER | OUTER | FULL | CROSS - | ORDER_BY | GROUP_BY | SORT_BY | HAVING | LIMIT | OFFSET | AS | ON + | ORDER_BY | GROUP_BY | SORT_BY | HAVING | LIMIT | OFFSET | FIRST | AS | ON | AND | OR | NOT | NULL | IN | LIKE | BETWEEN | TRUE | FALSE | COUNT | SUM | AVG | MIN | MAX | DISTINCT | ALL | ASC | DESC | UNION | INTERSECT | SUBTRACT | EXISTS diff --git a/mdl/visitor/visitor_import_export_mapping.go b/mdl/visitor/visitor_import_export_mapping.go index 54d9c1648..08cabdafc 100644 --- a/mdl/visitor/visitor_import_export_mapping.go +++ b/mdl/visitor/visitor_import_export_mapping.go @@ -226,7 +226,9 @@ func buildExportChild(ctx *parser.ExportMappingChildContext) *ast.ExportMappingE } // buildImportFromMappingStatement builds an ImportFromMappingStmt from the grammar context. -// Grammar: (VARIABLE EQUALS)? IMPORT FROM MAPPING qualifiedName LPAREN VARIABLE RPAREN onErrorClause? +// Grammar: (VARIABLE EQUALS)? IMPORT FROM MAPPING qualifiedName LPAREN VARIABLE RPAREN +// +// importMappingRange? onErrorClause? func buildImportFromMappingStatement(ctx antlr.ParserRuleContext) ast.MicroflowStatement { c := ctx.(*parser.ImportFromMappingStatementContext) stmt := &ast.ImportFromMappingStmt{ @@ -241,6 +243,24 @@ func buildImportFromMappingStatement(ctx antlr.ParserRuleContext) ast.MicroflowS stmt.SourceVariable = strings.TrimPrefix(vars[0].GetText(), "$") } + // Range: FIRST, or LIMIT/OFFSET. Absent means "All", which leaves the builder + // inferring cardinality from the mapping's root shape as it always has. (#881) + if r := c.ImportMappingRange(); r != nil { + rc := r.(*parser.ImportMappingRangeContext) + if rc.ALL() != nil { + stmt.All = true + } + if rc.FIRST() != nil { + stmt.First = true + } + if e := rc.GetLimitExpr(); e != nil { + stmt.LimitExpr = buildExpression(e) + } + if e := rc.GetOffsetExpr(); e != nil { + stmt.OffsetExpr = buildExpression(e) + } + } + if ec := c.OnErrorClause(); ec != nil { stmt.ErrorHandling = buildOnErrorClause(ec) } diff --git a/sdk/microflows/microflows_actions.go b/sdk/microflows/microflows_actions.go index 0408be08f..b51799ec5 100644 --- a/sdk/microflows/microflows_actions.go +++ b/sdk/microflows/microflows_actions.go @@ -864,6 +864,24 @@ type ResultHandlingMapping struct { ResultVariable string `json:"resultVariable,omitempty"` SingleObject bool `json:"singleObject,omitempty"` // true when mapping returns a single object (not a list) ForceSingleOccurrence *bool `json:"forceSingleOccurrence,omitempty"` + + // RangeSingleObject is the RANGE's own First flag, independent of + // SingleObject (which decides the result VARIABLE's type). The two are + // separate axes in Mendix: the blank app's FeedbackModule ships an activity + // with Range=ConstantRange{SingleObject:false} — "All" — bound to an + // ObjectType variable, because the mapping itself returns one object. + // Conflating them makes an explicit All on such a mapping write a ListType + // variable, which mxbuild rejects with CE0243. nil = follow SingleObject, + // which is what every caller did before #881. + RangeSingleObject *bool `json:"rangeSingleObject,omitempty"` + + // Custom range. When either is set the activity stores a + // Microflows$CustomRange instead of a ConstantRange — the "Custom" setting in + // Studio Pro's Range dropdown, which mxcli could not represent at all before + // #881 (it always wrote a ConstantRange, so a limit was unauthorable rather + // than merely undescribed). + LimitExpression string `json:"limitExpression,omitempty"` + OffsetExpression string `json:"offsetExpression,omitempty"` } func (ResultHandlingMapping) isResultHandling() {} @@ -1215,3 +1233,16 @@ type UnsupportedAction struct { } func (UnsupportedAction) isMicroflowAction() {} + +// RangeSingleObjectOf returns the ConstantRange.SingleObject value to store: +// the explicit range flag when set, otherwise the result-variable flag, which is +// what every caller relied on before the two axes were separated. (issue #881) +func RangeSingleObjectOf(h *ResultHandlingMapping) bool { + if h == nil { + return false + } + if h.RangeSingleObject != nil { + return *h.RangeSingleObject + } + return h.SingleObject +} diff --git a/sdk/mpr/parser_microflow_actions.go b/sdk/mpr/parser_microflow_actions.go index 9b64ad794..c740ac7ec 100644 --- a/sdk/mpr/parser_microflow_actions.go +++ b/sdk/mpr/parser_microflow_actions.go @@ -634,13 +634,27 @@ func parseResultHandling(raw map[string]any, handlingType string) microflows.Res result.MappingID = model.ID(mappingRef) forceSingleOccurrence := extractBool(call["ForceSingleOccurrence"], false) result.ForceSingleOccurrence = &forceSingleOccurrence + // The Range is polymorphic: a ConstantRange carries SingleObject + // (All/First) while a CustomRange carries the limit and offset + // expressions. Reading only SingleObject dropped the Custom setting + // entirely, so a describe→edit→exec cycle turned a bounded import + // into an unbounded one (issue #881). if rangeMap := toMap(call["Range"]); rangeMap != nil { - result.SingleObject = extractBool(rangeMap["SingleObject"], false) + switch extractString(rangeMap["$Type"]) { + case "Microflows$CustomRange": + result.LimitExpression = extractString(rangeMap["LimitExpression"]) + result.OffsetExpression = extractString(rangeMap["OffsetExpression"]) + default: + result.SingleObject = extractBool(rangeMap["SingleObject"], false) + } } } if varType := toMap(raw["VariableType"]); varType != nil { result.ResultEntityID = model.ID(extractString(varType["Entity"])) - if extractString(varType["$Type"]) == "DataTypes$ObjectType" { + // A bounded range is a LIST, so an ObjectType variable cannot make it + // single — without this guard a CustomRange read back as First. + if extractString(varType["$Type"]) == "DataTypes$ObjectType" && + result.LimitExpression == "" && result.OffsetExpression == "" { result.SingleObject = true } } @@ -746,17 +760,44 @@ func parseImportXmlAction(raw map[string]any) *microflows.ImportXmlAction { } forceSingleOccurrence := extractBool(call["ForceSingleOccurrence"], false) handling.ForceSingleOccurrence = &forceSingleOccurrence + // The Range is polymorphic — a ConstantRange carries SingleObject + // (Studio Pro's All/First), a CustomRange the limit and offset + // expressions. Reading only SingleObject dropped Custom entirely, so + // describe→edit→exec turned a bounded import unbounded. (issue #881) if rangeMap := toMap(call["Range"]); rangeMap != nil { - handling.SingleObject = extractBool(rangeMap["SingleObject"], false) + switch extractString(rangeMap["$Type"]) { + case "Microflows$CustomRange": + handling.LimitExpression = extractString(rangeMap["LimitExpression"]) + handling.OffsetExpression = extractString(rangeMap["OffsetExpression"]) + default: + single := extractBool(rangeMap["SingleObject"], false) + handling.RangeSingleObject = &single + handling.SingleObject = single + } } - // Older XML import mappings may omit Range and encode single-object - // handling only through ForceSingleOccurrence. REST result handling - // stores Range consistently, so this compatibility fallback stays - // XML-specific. - if !handling.SingleObject { - handling.SingleObject = forceSingleOccurrence + // The result variable's cardinality is the stored VariableType where + // there is one; it does NOT track the range (Mendix's own + // SUB_Feedback_PostToAppInsights pairs ConstantRange{SingleObject:false} + // with an ObjectType). Only otherwise does ForceSingleOccurrence stand + // in — and never for a bounded range, which is always a list, or a + // Custom range reads back as First and loses the limit. + switch extractString(toMap(rh["VariableType"])["$Type"]) { + case "DataTypes$ObjectType": + handling.SingleObject = true + case "DataTypes$ListType": + handling.SingleObject = false + default: + if !handling.SingleObject && handling.LimitExpression == "" && handling.OffsetExpression == "" { + handling.SingleObject = forceSingleOccurrence + } } } + // The writer stores VariableType on the ResultHandling, not on the + // ImportMappingCall, so the lookup above finds nothing on anything mxcli or + // Studio Pro writes — leaving the result entity empty. + if varType := toMap(rh["VariableType"]); varType != nil && handling.ResultEntityID == "" { + handling.ResultEntityID = model.ID(extractString(varType["Entity"])) + } action.ResultHandling = handling } diff --git a/sdk/mpr/parser_microflow_import_range_test.go b/sdk/mpr/parser_microflow_import_range_test.go new file mode 100644 index 000000000..4bcd28d93 --- /dev/null +++ b/sdk/mpr/parser_microflow_import_range_test.go @@ -0,0 +1,165 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" + "go.mongodb.org/mongo-driver/bson/primitive" +) + +// upstream #881, legacy engine. Both engines share the semantic model, so a fix +// in one is invisible to anyone running the other (MXCLI_ENGINE=legacy). These +// mirror the modelsdk tests against the legacy parser/serializer. + +// A CustomRange must survive the round trip: the reader dispatches on $Type, and +// the writer selects the variant. Before this, "Custom" was unrepresentable and +// a bounded import silently became unbounded on the next exec. +func TestLegacyImportXmlActionRoundTripsCustomRange(t *testing.T) { + doc := serializeImportXmlAction(µflows.ImportXmlAction{ + BaseElement: model.BaseElement{ID: model.ID("a-1")}, + ResultHandling: µflows.ResultHandlingMapping{ + BaseElement: model.BaseElement{ID: model.ID("rh-1")}, + MappingID: model.ID("M.IMM"), + ResultEntityID: model.ID("M.Root"), + ResultVariable: "Out", + LimitExpression: "10", + OffsetExpression: "5", + }, + XmlDocumentVariable: "Resp", + }) + + rhFields := bsonDMap(asD(t, bsonDMap(doc)["ResultHandling"])) + call := bsonDMap(asD(t, rhFields["ImportMappingCall"])) + rangeDoc := bsonDMap(asD(t, call["Range"])) + + if got := rangeDoc["$Type"]; got != "Microflows$CustomRange" { + t.Fatalf("Range $Type = %v, want Microflows$CustomRange", got) + } + if got := rangeDoc["LimitExpression"]; got != "10" { + t.Errorf("LimitExpression = %v, want 10", got) + } + if got := rangeDoc["OffsetExpression"]; got != "5" { + t.Errorf("OffsetExpression = %v, want 5", got) + } + if _, ok := rangeDoc["SingleObject"]; ok { + t.Error("a CustomRange must not carry SingleObject — a bounded range is always bounded") + } +} + +// The read side of the same. The result entity also lives on the ResultHandling, +// not on the ImportMappingCall, which is where the legacy parser looked — so it +// came back empty for everything mxcli or Studio Pro writes. +func TestLegacyParseImportXmlActionReadsCustomRange(t *testing.T) { + got := parseImportXmlAction(map[string]any{ + "$ID": "a-1", + "XmlDocumentVariableName": "Resp", + "ResultHandling": map[string]any{ + "$ID": "rh-1", + "ResultVariableName": "Out", + "ImportMappingCall": map[string]any{ + "ReturnValueMapping": "M.IMM", + "ForceSingleOccurrence": false, + "Range": map[string]any{ + "$Type": "Microflows$CustomRange", + "LimitExpression": "10", + "OffsetExpression": "5", + }, + }, + "VariableType": map[string]any{ + "$Type": "DataTypes$ListType", + "Entity": "M.Root", + }, + }, + }) + + if got.ResultHandling == nil { + t.Fatal("ResultHandling missing") + } + h := got.ResultHandling + if h.LimitExpression != "10" || h.OffsetExpression != "5" { + t.Errorf("limit/offset = %q/%q, want 10/5", h.LimitExpression, h.OffsetExpression) + } + if h.SingleObject { + t.Error("SingleObject = true, want false (ListType variable)") + } + if string(h.ResultEntityID) != "M.Root" { + t.Errorf("ResultEntityID = %q, want M.Root — VariableType is stored on the "+ + "ResultHandling, not on the ImportMappingCall", h.ResultEntityID) + } +} + +// The shape Mendix ships in the blank app: range All against an OBJECT variable. +// The range and the variable's cardinality are separate axes, and folding one +// into the other describes this as `first` — rewriting the activity on re-exec. +func TestLegacyParseImportXmlActionSeparatesRangeFromCardinality(t *testing.T) { + got := parseImportXmlAction(map[string]any{ + "$ID": "a-1", + "XmlDocumentVariableName": "Resp", + "ResultHandling": map[string]any{ + "$ID": "rh-1", + "ResultVariableName": "Out", + "ImportMappingCall": map[string]any{ + "ReturnValueMapping": "M.IMM", + "ForceSingleOccurrence": false, + "Range": map[string]any{"$Type": "Microflows$ConstantRange", "SingleObject": false}, + }, + "VariableType": map[string]any{"$Type": "DataTypes$ObjectType", "Entity": "M.Root"}, + }, + }) + + h := got.ResultHandling + if h == nil { + t.Fatal("ResultHandling missing") + } + if h.RangeSingleObject == nil || *h.RangeSingleObject { + t.Errorf("RangeSingleObject = %v, want explicit false — the range is All", h.RangeSingleObject) + } + if !h.SingleObject { + t.Error("SingleObject = false, want true — the stored ObjectType is the authority " + + "on the variable's cardinality, and mxbuild rejects the mismatch with CE0243") + } +} + +// The writer's variant choice must read the RANGE's flag, not the variable's: +// serializing Mendix's own All-against-an-object shape as First changes it. +func TestLegacySerializeImportXmlActionWritesRangeFlagNotCardinality(t *testing.T) { + no := false + doc := serializeImportXmlAction(µflows.ImportXmlAction{ + BaseElement: model.BaseElement{ID: model.ID("a-1")}, + ResultHandling: µflows.ResultHandlingMapping{ + BaseElement: model.BaseElement{ID: model.ID("rh-1")}, + MappingID: model.ID("M.IMM"), + ResultEntityID: model.ID("M.Root"), + ResultVariable: "Out", + SingleObject: true, // an object-rooted mapping + RangeSingleObject: &no, // …with the range left at All + }, + XmlDocumentVariable: "Resp", + }) + + rhFields := bsonDMap(asD(t, bsonDMap(doc)["ResultHandling"])) + call := bsonDMap(asD(t, rhFields["ImportMappingCall"])) + rangeDoc := bsonDMap(asD(t, call["Range"])) + if got := rangeDoc["SingleObject"]; got != false { + t.Errorf("Range.SingleObject = %v, want false — the range is All", got) + } + varType := bsonDMap(asD(t, rhFields["VariableType"])) + if got := varType["$Type"]; got != "DataTypes$ObjectType" { + t.Errorf("VariableType = %v, want DataTypes$ObjectType — the variable follows the "+ + "mapping, not the range", got) + } +} + +// asD narrows a nested BSON value so a missing sub-document fails at the field +// it is missing from rather than as a bare type-assertion panic. +func asD(t *testing.T, v any) primitive.D { + t.Helper() + d, ok := v.(primitive.D) + if !ok { + t.Fatalf("expected a BSON document, got %T", v) + } + return d +} diff --git a/sdk/mpr/writer_microflow_actions.go b/sdk/mpr/writer_microflow_actions.go index 70de49860..4ad8c2767 100644 --- a/sdk/mpr/writer_microflow_actions.go +++ b/sdk/mpr/writer_microflow_actions.go @@ -1017,11 +1017,7 @@ func serializeRestResultHandling(rh microflows.ResultHandling, outputVar string) {Key: "ForceSingleOccurrence", Value: forceSingleOccurrence}, {Key: "ObjectHandlingBackup", Value: "Create"}, {Key: "ParameterVariableName", Value: ""}, - {Key: "Range", Value: bson.D{ - {Key: "$ID", Value: idToBsonBinary(GenerateID())}, - {Key: "$Type", Value: "Microflows$ConstantRange"}, - {Key: "SingleObject", Value: h.SingleObject}, - }}, + {Key: "Range", Value: importMappingRange(h)}, {Key: "ReturnValueMapping", Value: string(h.MappingID)}, } doc = append(doc, bson.E{Key: "ImportMappingCall", Value: importCall}) @@ -1469,11 +1465,7 @@ func serializeImportXmlAction(a *microflows.ImportXmlAction) bson.D { {Key: "ForceSingleOccurrence", Value: forceSingleOccurrence}, {Key: "ObjectHandlingBackup", Value: "Create"}, {Key: "ParameterVariableName", Value: ""}, - {Key: "Range", Value: bson.D{ - {Key: "$ID", Value: idToBsonBinary(GenerateID())}, - {Key: "$Type", Value: "Microflows$ConstantRange"}, - {Key: "SingleObject", Value: a.ResultHandling.SingleObject}, - }}, + {Key: "Range", Value: importMappingRange(a.ResultHandling)}, {Key: "ReturnValueMapping", Value: string(a.ResultHandling.MappingID)}, } @@ -1588,3 +1580,29 @@ func serializeExternalActionReturnType(kind string) bson.D { {Key: "$Type", Value: bsonType}, } } + +// importMappingRange builds the Range child of a Microflows$ImportMappingCall. +// +// Mendix has two variants and mxcli only ever wrote the first, so the "Custom" +// setting — a bounded list — was not merely undescribed but unrepresentable: +// +// Microflows$ConstantRange{SingleObject} All (false) / First (true) +// Microflows$CustomRange{LimitExpression, OffsetExpression} Custom +// +// A limit or an offset selects CustomRange; SingleObject has no meaning there, +// because a bounded range is always a list. (issue #881) +func importMappingRange(h *microflows.ResultHandlingMapping) bson.D { + if h.LimitExpression != "" || h.OffsetExpression != "" { + return bson.D{ + {Key: "$ID", Value: idToBsonBinary(GenerateID())}, + {Key: "$Type", Value: "Microflows$CustomRange"}, + {Key: "LimitExpression", Value: h.LimitExpression}, + {Key: "OffsetExpression", Value: h.OffsetExpression}, + } + } + return bson.D{ + {Key: "$ID", Value: idToBsonBinary(GenerateID())}, + {Key: "$Type", Value: "Microflows$ConstantRange"}, + {Key: "SingleObject", Value: microflows.RangeSingleObjectOf(h)}, + } +} From 4fc5251906fa41a8bef1b06cdd405189b4aec634 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 15:29:35 +0000 Subject: [PATCH 15/20] Stop an empty `with ()` from crashing every command that parses the file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `log 'msg' with ()` panicked mxcli with a nil dereference in buildTemplateParams, so `check` produced a stack trace instead of a diagnostic and every other command that parses the script died the same way. The grammar requires at least one templateParam, which is a statement about valid input, not about what reaches the walker: ANTLR error-recovers by handing it a TemplateParamContext with no index token rather than by skipping the rule. The builder called NUMBER_LITERAL().GetText() on it. Guarded at the dereference. The syntax error is already reported, and `check` now says what it always should have: line 3:20 mismatched input ')' expecting '{' The other 11 NUMBER_LITERAL().GetText() sites under mdl/visitor/ were checked and every one already nil-guards, so this was the only bare one and no sweep is needed. Two tests, because the obvious guard is wrong in a way the crash test cannot see: skipping on nil also skips well-formed parameters, so the control asserts `with ({1} = …, {2} = …)` still yields both. Reported in mxcli-chat FINDINGS §55. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/fix-issue.md | 1 + mdl/visitor/parser_batch_findings_test.go | 48 +++++++++++++++++++++++ mdl/visitor/visitor_microflow_actions.go | 12 +++++- 3 files changed, 59 insertions(+), 2 deletions(-) diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 3e1b5ab67..853b1f22b 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -501,3 +501,4 @@ extracting `OffsetExpression`/`LimitExpression`. | A `.test.mdl` asserting on something a **constant** feeds passes under `mxcli test --attach` and fails under `mxcli test --local` (or the reverse), with nothing in either output to explain the difference | `--local` boots an app of its own through `StartLocalApp`, whose options had no `ConstantOverrides` field — so it ran with each constant's **default** from `deployment/model/config.json`, while `--attach` runs against an app `run --local` booted, which applies the configuration's shared overrides. A constant resolving to the wrong value is not an error, so both runs reported success and only the assertion differed | `cmd/mxcli/docker/localapp.go` (`LocalAppOptions.ConstantOverrides`, `runtimeOptions`), `cmd/mxcli/testrunner/localapp_options.go` (new, shared by both `--local` runners), `runner.go` (`RunOptions.ConstantOverrides`), `cmd/mxcli/cmd_test_run.go` + `main.go` (`--configuration`) | **A boot path that takes options needs one mapping function, not an inline struct literal per caller** — the two `--local` runners each built their own `LocalAppOptions`, so a field added for one would not reach the other, and neither had the constants. Both now go through `localAppOptions`, and the `LocalAppOptions`→`LocalRuntimeOptions` step is `runtimeOptions()` so the forwarding is assertable without booting anything: a dropped field there is otherwise invisible until an app runs with the wrong configuration. **Resolve in one place**: `cmd/` decides precedence and the runner only carries the map, or "which configuration wins" gets two answers. **Only report what this run actually uses** — `--attach` inherits the constants of the app it attached to, so resolving and printing them there would be a confident lie. Verified at the layer the symptom lives in (`.claude/skills/verify-in-runtime.md`): the same suite, one project, `--local` and `--attach` must agree, with the reverted-wiring control run showing the constant's default. Unit tests `cmd/mxcli/testrunner/localapp_options_test.go`, `cmd/mxcli/docker/localboot_constants_test.go`. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 1 | | A constant value that must not be committed has nowhere to live: a constant's default and a shared configuration override both go to git, and Mendix's own **private** configuration value — the correct slot — is encrypted per user account by Studio Pro from 10.9, so nothing headless can read or write it | Not a defect so much as an absent layer. mxcli mirrors the concept with a store it can actually reach: `/.mxcli/constants.json`, mode 0600, gitignored, sitting between `--constant` and the configuration | `cmd/mxcli/constantstore/` (load/save), `cmd/mxcli/cmd_constant.go` (`constant set/unset/list`), `cmd/mxcli/constant_gitignore.go` (`ensureStoreIgnored`), `cmd/mxcli/constants_resolve.go` (`layerMachine`) | **The promise has to be made true, then checked — asserting it is not enough.** `mxcli init` writes a `.gitignore` only when the project has none, and a Mendix project usually already has one, so the entry the whole layer rests on could simply be absent. `constant set` appends it *and then asks git*, refusing to write the value on "not ignored": a store that leaks is worse than no store. **Which rule defeats an ignore entry is not guessable** — `!.mxcli/**` does NOT re-include anything (git cannot re-include a file whose parent directory is excluded), while `!.mxcli` does; the test was corrected against measured git behaviour rather than the assumed case. **A corrupt store is an error, a stale entry is not**: an unparseable file means values the author set are about to be silently absent (fatal), while an entry naming a constant the project dropped is the user's own file and is skipped-and-named, or every run fails until it is hand-edited. **An empty store is deleted, not written as `{}`** — a file that configures nothing should not exist. Tests `cmd/mxcli/constantstore/store_test.go`, `cmd/mxcli/constant_gitignore_test.go`; verified live that the value reaches a running app and that `git status` never sees the file. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 3 | | A constant value changed on disk (machine store, or the project configuration) does not reach an app that is **already running** — a `mxcli run --local` keeps serving the old value until it is restarted | Constants are a boot-time layer: mxcli sends them once, in the `update_configuration` call at start. Changing them live needs the M2EE admin API, and needs BOTH of its calls | `cmd/mxcli/docker/localboot.go` (`ApplyConstants`, `LocalRuntime.BootConfig`), `cmd/mxcli/devloop_handshake.go`, `cmd/mxcli/cmd_constant.go` (`--apply`) | **`update_configuration` is STAGED, not applied** — measured on 11.12.1: the running app keeps its old values, and the call still answers `result:0`. Only the next `reload_model` applies them, so a version sending just the first call reports success and changes nothing. That is why the two calls live in one function rather than at the call site. **Re-send the WHOLE payload**: the admin action has no read-back (`get_configuration`, `get_current_configuration`, `runtime_config`, `get_current_runtime_status` are all "Action not found"), so anything not sent is simply gone from the configuration afterwards — which is why the dev loop publishes what it booted with rather than a second process guessing. **A handshake from a dead process is refused, not used**: its ports may since have been taken by something else, and pushing a configuration change into the wrong process is worse than reporting no dev loop. **Say plainly what cannot be confirmed** — with no read-back, two successful calls are the strongest evidence available, so the command names what would actually confirm it (a microflow that returns the constant) instead of claiming success. **Failure to apply is not failure to set**: the value is already on disk and the next boot uses it, so `--apply` reports and returns rather than exiting non-zero over a half-succeeded command. Tests `cmd/mxcli/docker/localboot_constants_test.go`, `cmd/mxcli/devloop_handshake_test.go`. See `docs/11-proposals/PROPOSAL_constant_values.md` slice 4 | +| `mxcli check` on a script containing `log 'msg' with ()` **panics** — `runtime error: invalid memory address or nil pointer dereference` at `mdl/visitor.buildTemplateParams`. Any command that parses the file dies, so there is no diagnostic at all, only a stack trace | The grammar requires at least one `templateParam`, so `with ()` is a syntax error — but ANTLR error-recovers by handing the walker a `TemplateParamContext` with **no index token** rather than by skipping the rule. `buildTemplateParams` called `paramCtx.NUMBER_LITERAL().GetText()` on it | `mdl/visitor/visitor_microflow_actions.go` (`buildTemplateParams`) | **A visitor walks error-recovered trees, so grammar-guaranteed tokens are not guaranteed at the walker.** "The rule requires it" is a statement about valid input; recovery synthesises contexts that satisfy no rule. Guard at the dereference, not by tightening the grammar. The other 11 `NUMBER_LITERAL().GetText()` sites in `mdl/visitor/` were checked and all already nil-guard — this was the only bare one, so no sweep was needed. **The control test is the point**: a guard that `continue`s on nil passes the crash test while silently dropping every parameter, so assert a well-formed `with ({1} = …, {2} = …)` still yields both. Tests `mdl/visitor/parser_batch_findings_test.go` (`TestLogEmptyTemplateParamsIsAnErrorNotAPanic`, `TestLogTemplateParamsStillBuild`); the crash becomes `line 3:20 mismatched input ')' expecting '{'`. Reported in mxcli-chat FINDINGS §55 | diff --git a/mdl/visitor/parser_batch_findings_test.go b/mdl/visitor/parser_batch_findings_test.go index 0dd0431ec..d55456b8a 100644 --- a/mdl/visitor/parser_batch_findings_test.go +++ b/mdl/visitor/parser_batch_findings_test.go @@ -66,3 +66,51 @@ func TestUserRoleQuotingConsistency(t *testing.T) { } } } + +// TestLogEmptyTemplateParamsIsAnErrorNotAPanic guards FINDINGS §55: `log … with ()` +// crashed mxcli with a nil dereference in buildTemplateParams. +// +// The grammar requires at least one templateParam, so ANTLR error-recovers by +// producing a TemplateParamContext with no NUMBER_LITERAL — which the builder +// dereferenced anyway. Every visitor that walks an error-recovered tree has this +// shape available to it, so the guard belongs at the dereference, not in the +// grammar: a malformed statement must come back as a parse error. +func TestLogEmptyTemplateParamsIsAnErrorNotAPanic(t *testing.T) { + input := `create microflow M.LogEmpty () +begin + log 'hello' with (); +end;` + + _, errs := Build(input) // panicked before the fix + if len(errs) == 0 { + t.Fatal("expected a parse error for an empty `with ()` list, got none") + } +} + +// TestLogTemplateParamsStillBuild is the control for the guard above: a well-formed +// `with` list must still produce its parameters. A guard that skipped every param +// would pass the panic test and silently drop the message's arguments. +func TestLogTemplateParamsStillBuild(t *testing.T) { + input := `create microflow M.LogOne () +begin + log 'hello {1} and {2}' with ({1} = 'world', {2} = 'again'); +end;` + + prog, errs := Build(input) + if len(errs) > 0 { + t.Fatalf("unexpected parse errors: %v", errs) + } + mf := prog.Statements[0].(*ast.CreateMicroflowStmt) + log, ok := mf.Body[0].(*ast.LogStmt) + if !ok { + t.Fatalf("first statement is %T, want *ast.LogStmt", mf.Body[0]) + } + if len(log.Template) != 2 { + t.Fatalf("template params = %d, want 2", len(log.Template)) + } + for i, want := range []int{1, 2} { + if log.Template[i].Index != want { + t.Errorf("param %d index = %d, want %d", i, log.Template[i].Index, want) + } + } +} diff --git a/mdl/visitor/visitor_microflow_actions.go b/mdl/visitor/visitor_microflow_actions.go index 2f36cd2ba..a9c487e77 100644 --- a/mdl/visitor/visitor_microflow_actions.go +++ b/mdl/visitor/visitor_microflow_actions.go @@ -83,8 +83,16 @@ func buildTemplateParams(ctx parser.ITemplateParamsContext) []ast.TemplateParam allParams := paramsCtx.AllTemplateParam() for i, param := range allParams { paramCtx := param.(*parser.TemplateParamContext) - indexStr := paramCtx.NUMBER_LITERAL().GetText() - index, _ := strconv.Atoi(indexStr) + // The grammar requires at least one parameter, so `with ()` does not + // parse — but ANTLR error-recovers by handing the walker a templateParam + // with no index token rather than by skipping the rule. The syntax error + // is already reported; dereferencing here killed the whole process + // instead of failing the one statement (FINDINGS §55). + numTok := paramCtx.NUMBER_LITERAL() + if numTok == nil { + continue + } + index, _ := strconv.Atoi(numTok.GetText()) var tp ast.TemplateParam tp.Index = index From db8449e56a11e0e73ab95ed93523a3c812417e16 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 15:29:48 +0000 Subject: [PATCH 16/20] Record what a partial constants payload actually does, against the dispute MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mxcli-chat FINDINGS §57 reports that an update_configuration carrying one constant blanks every constant omitted from the map, and that no reload is needed. Both halves contradict the comment this file gained in c766a76c, and the constants feature rests on that reading, so it was re-measured. Measured on 11.12.1, reading the values back through a microflow that returns two constants over the test endpoint — the app's own view rather than the API's, which is what the earlier measurement lacked: - Staged, confirmed. Without reload_model the app keeps the old value while update_configuration still answers result:0. - Merge, not replace. A payload carrying only ApiKey left ClientIdentifier at the value an EARLIER update_configuration had given it — not at its deployment default, and not blank. So it overlays the running configuration, which is stronger than the "overlay on the deployment defaults" the comment claimed. - Payload shape is not the difference: params carrying only MicroflowConstants, exactly as reported, behaved the same as the full boot config. §57's blanking did not reproduce here in either shape. It is recorded as disputed rather than as wrong: it was inferred from a downstream symptom on 11.13.0, and this is one version and one runtime. The comment now says why the disagreement costs nothing either way: ApplyConstants re-sends the whole resolved chain, which is correct under both readings. Nobody should have to re-run this to find that out. Comment only; no behaviour change. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/docker/localboot.go | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/cmd/mxcli/docker/localboot.go b/cmd/mxcli/docker/localboot.go index 17e55ee42..85d922b54 100644 --- a/cmd/mxcli/docker/localboot.go +++ b/cmd/mxcli/docker/localboot.go @@ -304,13 +304,25 @@ func runtimeConfigParams(o LocalRuntimeOptions, constants map[string]string) map // are all "Action not found" on 11.12.1 — so a caller cannot merge by // reading first. // - // Measured on 11.12.1, for anyone tempted to drive this API live: the - // runtime treats an update_configuration MicroflowConstants map as an - // overlay on the deployment defaults (constants omitted from the map still - // resolve), and the call is staged rather than applied — the running app - // keeps the old value until the next reload_model, while still answering - // result:0. A "set a constant on a running app" feature is therefore - // update_configuration + reload_model, verified by observation. + // Measured on 11.12.1, for anyone tempted to drive this API live. Both + // readings are from a microflow returning two constants over the test + // endpoint, so they are the app's own view rather than the API's: + // + // - The call is STAGED, not applied. The running app keeps the old value + // until the next reload_model, while update_configuration still answers + // result:0. "Set a constant on a running app" is therefore the pair. + // - MicroflowConstants MERGES onto the running configuration. A payload + // carrying one constant left the other at the value an EARLIER + // update_configuration had given it — not at its deployment default, + // and not blank. Payload shape does not change this: params carrying + // only MicroflowConstants behaved the same as the full boot config. + // + // The second point is disputed. mxcli-chat FINDINGS §57 reports the opposite + // on 11.13.0 — a partial map blanking every omitted constant — inferred from + // a downstream symptom rather than read back. It did not reproduce here on + // 11.12.1 in either payload shape. Do not rely on either behaviour: ApplyConstants + // sends the whole resolved chain, which is correct under both readings, and + // that is the reason this disagreement costs mxcli nothing. for k, v := range o.RuntimeSettings { params[k] = v } From 7e35905a62c63ec7579086481650b7861b3883f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 15:34:32 +0000 Subject: [PATCH 17/20] fix(mappings): resolve JSON members by either name, and stop inventing paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A JsonStructures$JsonElement carries TWO names, and for any lowercase-initial key they differ: Path "(Object)|uuid" the raw JSON key — what the RUNTIME resolves by ExposedName "Uuid" Mendix's derived name — what Studio Pro DISPLAYS Mendix derives the exposed name by capitalising the initial, and by suffixing "Item" on an array's item object. Both are Mendix's own: the blank app's Studio-Pro-authored FeedbackModule.JSON_AppInsightsResponse stores ExposedName "Uuid" against Path "(Object)|uuid", and its IMM_PostResponse binds JsonPath "(Object)|uuid". So the capitalisation DESCRIBE shows is faithful rendering. The defect is that mxcli's DESCRIBE emits the exposed name while its builder resolved only raw keys, so mxcli's own output did not round-trip. Re-executing a DESCRIBE fabricated a path from the exposed name: (Object)|total -> (Object)|Total (Object)|entityInstances|__Value|(Object) -> (Object)|EntityInstances|__ValueItem The array's "|(Object)" item marker vanished entirely and MaxOccurs went to 0. `mxcli check` passed; mxbuild reported CE5015. Export mappings had the identical bug. Members now resolve by raw key or exposed name — including an array addressed by its item's exposed name, which resolves to the array so the "|(Object)" step is still taken. A member matching neither is REFUSED, listing the spellings that would have worked, instead of being written with an invented path: that path passed `mxcli check` and surfaced only later, which is the worst failure mode because the tool that wrote it reported success. Separately, both engines wrote IsDefaultType on every ValueMappingElement. The generated metamodel declares it on Import/ExportObjectMappingElement and on neither value type, and Studio Pro's own mappings carry it on the object element alone. Per the overlay-writes rule in CLAUDE.md, a property the type does not own is the shape mxbuild tolerates and Studio Pro refuses to open, so a green build is not evidence — it is dropped from the value writers in both engines. Verified on mxbuild 11.6.6, both engines: the raw-key and exposed-name spellings now produce byte-identical stored paths, and `mx check` reports 0 errors where it previously reported CE5015. Each fix was reverted in turn to confirm the symptom returns. Refs upstream #882. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- .claude/skills/fix-issue.md | 1 + .../mendix/json-structures-and-mappings.md | 31 +++ .../bug-tests/882-mapping-member-names.mdl | 112 +++++++++++ .../modelsdk/mapping_isdefaulttype_test.go | 84 ++++++++ mdl/backend/modelsdk/mapping_write.go | 12 +- mdl/executor/cmd_export_mappings.go | 61 ++++-- mdl/executor/cmd_import_mappings.go | 154 +++++++++++++-- .../cmd_mappings_member_resolution_test.go | 179 ++++++++++++++++++ sdk/mpr/writer_export_mapping.go | 6 +- sdk/mpr/writer_import_mapping.go | 6 +- 10 files changed, 607 insertions(+), 39 deletions(-) create mode 100644 mdl-examples/bug-tests/882-mapping-member-names.mdl create mode 100644 mdl/backend/modelsdk/mapping_isdefaulttype_test.go create mode 100644 mdl/executor/cmd_mappings_member_resolution_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 791cf8a20..8eff1d7c5 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -491,3 +491,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `show callers of ` and `show references to ` report "(no callers found)" for a document reached only from a page action button — a false negative that reads as "safe to delete" | TWO independent defects behind one symptom. (1) `scanWidgetOwnRefs` collected `Entity`/`Microflow`/`Nanoflow` from a widget's raw BSON but not **`Form`**, the key a PAGE reference uses, so `widgets_data` had no page column and the refs projection had no page row. (2) `execShowCallers` filtered `RefKind = 'call'` — the kind a microflow CALL ACTIVITY produces — so the button→microflow row, which was already in the refs table, was hidden by the query | `mdl/catalog/builder_pages.go` (`scanWidgetOwnRefs` + `rawWidgetInfo.PageRef`), `mdl/catalog/tables.go` (`widgets_data.PageRef`), `mdl/catalog/builder_references.go` (projection row), `mdl/executor/cmd_search.go` (`callerRefKinds`) | **Find the live code path before fixing anything** — `builder_references.go` has an inviting `extractWidgetRefs` with a per-widget-type switch, and it is DEAD: its only caller is its own recursion. Extending it changes nothing. The live path is a SQL projection out of `widgets_data`, and the standing `NOTE: widget-level datasource/action refs still require a parsed widget tree` comment beside it is the tell. **Separate "the reference is missing" from "the query hides it"** by reading the refs table directly: the button→microflow row was present all along, so fixing only the scanner would have closed half the issue and left the reporter's second scenario broken. **`Form` is `Page`** — the same rename behind `ShowFormAction`/`CloseFormAction` (CLAUDE.md's storage-name table); grepping for `Page` in a BSON scanner finds nothing. **One action can carry two references**: `create object … then open page` holds an entity AND a page, so collecting the entity alone still leaves the page unreferenced. **Do not widen `callers` into `references`** — `datasource`/`parameter`/`generalize` are uses of a TYPE, not invocations, and including them makes the two commands synonyms; the test pins both the included and the excluded set. Tests `builder_pages_test.go` (`TestScanWidgetOwnRefs_PageReference`), `cmd_search_callers_test.go`. upstream #773 | | `ALTER PAGE` over `--mcp` fails against Studio Pro **11.13** with `pg_patch_page: … PROP_NOT_PRIMITIVE: Property 'widgets' is not a primitive property`. `CREATE PAGE` is fine; the page itself is left intact | 11.13 gave `pg_read_page` a **`depth` argument defaulting to 4**, replacing anything deeper with the literal string `"..."`. ALTER PAGE is read-modify-**replace-whole-page**, so the truncated read went straight back as the new page body. Measured live: `Administration.Account_Overview` read 32,594 bytes at full depth but **1,052 bytes** at the default, its entire tree reduced to `{"widgets":["...","..."]}`. Every ordinary page truncates — three of three PgTest pages did | `mdl/backend/mcp/page.go` (`pgReadPage`, `pgReadFullDepth`, `hasTruncationSentinel`), `mdl/backend/mcp/client.go` (`SupportsToolArg`) | Request the full depth, and **guard rather than trust it**: refuse a read still carrying the sentinel instead of letting a partial page reach a write (ADR-0005 guard-don't-drop). Two traps. (1) **Do not send `depth` unconditionally** — 11.11/11.12 declare `pg_read_page` `additionalProperties:false` without it, so the whole call fails; gate on a live `tools/list` probe of the tool's input schema, because `serverInfo.version` is frozen at `1.0.0` across 11.11/11.12/11.13 and cannot discriminate releases. (2) **Match the sentinel only as an array element** — a caption or title legitimately reading `"..."` is real content, and a naive substring scan rejects valid pages. The release notes announced none of this, exactly as 11.12 silently removed `pg_write_page` (#697): on any Studio Pro upgrade, re-probe `tools/list` and diff the input schemas, not just the tool names. Tests `mdl/backend/mcp/page_depth_test.go`; controls: stub the depth arg (full-depth test fails) and stub the guard (truncation test fails) | | An import activity's Range — Studio Pro's **All / First / Custom** — is absent from `DESCRIBE MICROFLOW`, and a `limit`/`offset` set in Studio Pro does not survive an mxcli round trip | Worse than "undescribed". `Microflows$ImportMappingCall.Range` is polymorphic — `ConstantRange{SingleObject}` (All/First) or `CustomRange{LimitExpression, OffsetExpression}` (Custom) — and mxcli wrote only the first and read only `SingleObject`. So **Custom was unrepresentable**, a bounded import became unbounded on any rewrite, and all three settings described identically, so describe→edit→exec silently changed the activity | `mdl/grammar/domains/MDLMicroflow.g4` (`importMappingRange`) + `MDLLexer.g4` (`FIRST`) + `MDLSettings.g4` (keyword list), `mdl/visitor/visitor_import_export_mapping.go`, `mdl/ast/ast_microflow.go`, `sdk/microflows/microflows_actions.go` (`RangeSingleObject`, `RangeSingleObjectOf`), `mdl/executor/cmd_microflows_builder_calls.go` (`addImportFromMappingAction`), `mdl/executor/cmd_microflows_format_action.go` (`formatImportMappingRange`), `mdl/backend/modelsdk/microflow_write.go` (`importMappingRangeToGen`) + `microflow_read_actions.go`, `sdk/mpr/writer_microflow_actions.go` (`importMappingRange`) + `parser_microflow_actions.go` | **The Range and the result variable's CARDINALITY are separate axes** — the first fix folded them and every `all` wrote a `ListType` against an object-rooted mapping, which mxbuild rejects with **CE0243** ("the mapping used to return 'List of X' but now returns 'X'"). Mendix's own `FeedbackModule.SUB_Feedback_PostToAppInsights` settles it: `ConstantRange{SingleObject:false}` against an **ObjectType** variable. The stored `VariableType` is the authority on cardinality; the range is its own flag (hence `RangeSingleObject *bool`, nil = "not authored, fall back"). **Count the write sites before declaring a serialization fixed** — the `ImportMappingCall` is built at THREE places (`importXmlActionToGen`, the REST `ResultHandlingMapping` case, and the legacy writer), and patching two let a limit reach the model while a `ConstantRange` was still written; extract one helper. **DESCRIBE must emit one of the three, never nothing** — silence re-enters the builder's inference, and an object-rooted mapping set to All (Studio Pro's default) comes back as First. **`first` is not `limit 1`**: one binds an OBJECT, the other a one-element LIST, so they cannot share syntax. **A platform rule you cannot author a positive case for is documentation, not validation** — Mendix rejects `offset` on a non-list mapping with **CE6100** while accepting `limit`, and mxcli cannot currently author a list-rooted import mapping (array-root mappings emit CE5015), so the constraint is documented rather than guessed at in a checker. Tests `mdl/executor/cmd_microflows_import_range_test.go`, `mdl/backend/modelsdk/microflow_import_range_test.go`, `sdk/mpr/parser_microflow_import_range_test.go`, example `mdl-examples/bug-tests/881-import-mapping-range.mdl`. upstream #881 | +| `CREATE IMPORT MAPPING` is reported to "not preserve JSON member names": `total` becomes `Total`, `camelCase` becomes `CamelCase`, `__Value` becomes `__ValueItem`. The import then fails at runtime with `key not found: Path(QName(None,),None,)` at `MappingCache.storeValueMappingElement` | The capitalisation is **Mendix's own** and not the defect. A `JsonStructures$JsonElement` stores TWO names — `Path` (the raw JSON key, `(Object)\|uuid`, which the RUNTIME resolves by) and `ExposedName` (derived by capitalising the initial, and suffixing `Item` on an array's item object, which Studio Pro DISPLAYS). The real defect: mxcli's DESCRIBE emits `ExposedName` while its builder resolved only raw keys, so mxcli's own output did not round-trip — re-executing a DESCRIBE FABRICATED a path (`(Object)\|Total`; for arrays the `\|(Object)` item marker vanished entirely, leaving `(Object)\|EntityInstances\|__ValueItem`) and zeroed MaxOccurs. Export mappings had the identical bug. Separately, mxcli wrote `IsDefaultType` on every `ValueMappingElement`, a property only the OBJECT element types own | `mdl/executor/cmd_import_mappings.go` (`jsonSchemaIndex`, `newJSONSchemaIndex`, `resolve`, `memberNames`; `buildImportMappingElementModel` now returns an error), `mdl/executor/cmd_export_mappings.go` (same), `mdl/backend/modelsdk/mapping_write.go` + `sdk/mpr/writer_import_mapping.go` + `sdk/mpr/writer_export_mapping.go` (drop `IsDefaultType` from the value writers) | **A blank Mendix app is a free Studio-Pro-authored fixture** — `FeedbackModule.IMM_PostResponse` + `JSON_AppInsightsResponse` settle "what does Studio Pro actually write?" with no Studio Pro and no marketplace download: they store `ExposedName "Uuid"` against `Path "(Object)|uuid"`. Author the SAME mapping over the SAME structure in the SAME module with mxcli and diff the two units — the only variable left is which tool wrote it. **Run the reporter's repro as written before believing the diagnosis**: theirs produced paths byte-identical to the structure's, so their MDL was not the failing input; the corruption needed a DESCRIBE in the middle. **DESCRIBE emitting a different name than the parser accepts is a round-trip bug even when both are "correct"** — the fix is to accept BOTH spellings, not to change what DESCRIBE prints, because the exposed name is the one Studio Pro shows. **Never fabricate a path for an unresolved member**: the invented path passed `mxcli check` and surfaced only in mxbuild (CE5015) or at runtime, so the tool that wrote it reported success — refuse, and list the spellings that would have worked. **`generated/metamodel` decides property ownership**: `isDefaultType` is declared on `Import/ExportObjectMappingElement` and on neither `ValueMappingElement`, and per CLAUDE.md's overlay rule an extra property is the shape mxbuild tolerates and Studio Pro will not open — so a green build proves nothing here. Tests `mdl/executor/cmd_mappings_member_resolution_test.go`, `mdl/backend/modelsdk/mapping_isdefaulttype_test.go`, example `mdl-examples/bug-tests/882-mapping-member-names.mdl`. upstream #882 | diff --git a/.claude/skills/mendix/json-structures-and-mappings.md b/.claude/skills/mendix/json-structures-and-mappings.md index 765658c53..119b12541 100644 --- a/.claude/skills/mendix/json-structures-and-mappings.md +++ b/.claude/skills/mendix/json-structures-and-mappings.md @@ -10,6 +10,37 @@ A JSON structure defines the schema of a JSON payload. It stores a JSON snippet ### Import Mappings An import mapping converts a JSON string into Mendix entity objects. It maps JSON fields to entity attributes. +#### Two names per member: the raw key and the exposed name + +Every JSON structure element stores **both**, and for any lowercase-initial key +they differ: + +| | Example | Used for | +|---|---|---| +| **Path** (raw JSON key) | `(Object)\|uuid` | what the **runtime** resolves by | +| **ExposedName** (derived) | `Uuid` | what **Studio Pro displays** | + +Mendix derives the exposed name by capitalising the initial, and for an array's +item object by suffixing `Item` — so `total` → `Total`, `camelCase` → `CamelCase`, +`__Value` (array) → `__ValueItem` (its item). Keys already starting with an +underscore are left alone: `__returnedCount` stays `__returnedCount`. + +This is **Mendix's own convention, not something mxcli does**. A blank app's +Studio-Pro-authored `FeedbackModule.JSON_AppInsightsResponse` stores +`ExposedName: "Uuid"` against `Path: "(Object)|uuid"`, and its `IMM_PostResponse` +binds `JsonPath: "(Object)|uuid"`. + +Consequences worth knowing: + +- **Either spelling works in MDL.** `Total = total` and `Total = Total` produce the + same stored mapping. Write whichever you have. +- **`DESCRIBE` emits the exposed name**, because that is the name Studio Pro shows. + A describe → edit → exec cycle is therefore lossless, but the text you get back + will not match the raw JSON keys you wrote. +- **A member matching neither spelling is refused**, listing what would have + worked. It is never written with a guessed path: such a mapping passed + `mxcli check` and failed later in mxbuild (CE5015) or at runtime. + #### Inherited attributes Mendix inheritance is multi-table: all of a parent's attributes are members of the diff --git a/mdl-examples/bug-tests/882-mapping-member-names.mdl b/mdl-examples/bug-tests/882-mapping-member-names.mdl new file mode 100644 index 000000000..196eb03e8 --- /dev/null +++ b/mdl-examples/bug-tests/882-mapping-member-names.mdl @@ -0,0 +1,112 @@ +-- Bug test for upstream issue #882: JSON member names in import/export mappings. +-- +-- A JSON structure element carries TWO names, and for any lowercase-initial key +-- they differ: +-- +-- Path "(Object)|total" the raw JSON key — what the RUNTIME resolves by +-- ExposedName "Total" Mendix's derived name — what Studio Pro DISPLAYS +-- +-- Both are Mendix's own. A blank app's Studio-Pro-authored +-- FeedbackModule.JSON_AppInsightsResponse stores ExposedName "Uuid" against Path +-- "(Object)|uuid", and its IMM_PostResponse binds JsonPath "(Object)|uuid". So +-- the capitalisation DESCRIBE shows is faithful rendering, not corruption — the +-- part of #882 reported as the bug is Mendix's own convention. +-- +-- The real defect: mxcli's DESCRIBE prints the exposed name, but its builder +-- resolved only raw JSON keys. Its own output therefore did not round-trip — +-- re-executing a DESCRIBE FABRICATED a path from the exposed name: +-- +-- (Object)|total -> (Object)|Total +-- (Object)|entityInstances|__Value|(Object) -> (Object)|EntityInstances|__ValueItem +-- +-- The array's "|(Object)" item marker vanished entirely, and MaxOccurs went to 0. +-- `mxcli check` passed; mxbuild reported CE5015. Export mappings had the same bug. +-- +-- Expected now: both spellings resolve to the SAME stored path, `mx check` +-- reports 0 errors, and re-executing DESCRIBE output is a no-op. A member that +-- matches neither spelling is REFUSED at exec rather than written with an +-- invented path. +-- +-- Verified on mxbuild 11.6.6, both engines. + +create module B882; +/ +create non-persistent entity B882.Root ( Total: integer, Camel: string(200) ); +/ +create non-persistent entity B882.Inner ( Cnt: integer ); +/ +create non-persistent entity B882.Item ( Nm: string(200) ); +/ +create association B882.Root_Inner from B882.Root to B882.Inner; +/ +create association B882.Inner_Item from B882.Item to B882.Inner; +/ + +create json structure B882.JS + snippet $${ "total": 1, "camelCase": "x", "entityInstances": { "__returnedCount": 2, "__Value": [ { "name": "a" } ] } }$$; +/ + +-- Authored with the RAW JSON keys, as hand-written MDL spells them. +create import mapping B882.IM_RawKeys with json structure B882.JS +{ + create B882.Root { + Total = total, + Camel = camelCase, + create B882.Root_Inner/B882.Inner = entityInstances { + Cnt = __returnedCount, + create B882.Inner_Item/B882.Item = __Value { + Nm = name + } + } + } +}; +/ + +-- Authored with the EXPOSED names, as `describe import mapping` emits them. +-- Must produce byte-identical JsonPaths to IM_RawKeys — this is the statement +-- that used to fabricate paths and fail CE5015. +create import mapping B882.IM_ExposedNames with json structure B882.JS +{ + create B882.Root { + Total = Total, + Camel = CamelCase, + create B882.Root_Inner/B882.Inner = EntityInstances { + Cnt = __returnedCount, + create B882.Inner_Item/B882.Item = __ValueItem { + Nm = Name + } + } + } +}; +/ + +-- Export mappings carry the same two names and had the same defect. +create non-persistent entity B882.ExRoot ( Total: integer, Camel: string(200) ); +/ +create json structure B882.JSX snippet '{ "total": 1, "camelCase": "x" }'; +/ +create export mapping B882.EM_RawKeys with json structure B882.JSX +{ + B882.ExRoot { + total = Total, + camelCase = Camel + } +}; +/ +create export mapping B882.EM_ExposedNames with json structure B882.JSX +{ + B882.ExRoot { + Total = Total, + CamelCase = Camel + } +}; +/ + +-- A member matching neither spelling is refused at exec, naming both forms: +-- +-- create import mapping B882.IM_Typo with json structure B882.JS +-- { create B882.Root { Camel = camelCse } }; +-- +-- Error: import mapping B882.IM_Typo: "camelCse" is not a member of the JSON +-- structure at (Object); available: total (or Total), camelCase (or CamelCase), +-- entityInstances (or EntityInstances) diff --git a/mdl/backend/modelsdk/mapping_isdefaulttype_test.go b/mdl/backend/modelsdk/mapping_isdefaulttype_test.go new file mode 100644 index 000000000..76f9fe01c --- /dev/null +++ b/mdl/backend/modelsdk/mapping_isdefaulttype_test.go @@ -0,0 +1,84 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/modelsdk/element" + "go.mongodb.org/mongo-driver/v2/bson" +) + +// upstream #882. IsDefaultType belongs to the OBJECT mapping element type and to +// neither VALUE element type. mxcli wrote it on every value element. +// +// Authority, in order: `generated/metamodel` (built from Mendix's reflection +// data) declares isDefaultType on Import/ExportObjectMappingElement only, and +// modelsdk/gen exposes the accessor on those two types alone; and Studio Pro's +// own mappings in a blank app — FeedbackModule.IMM_PostResponse and +// EMM_PostFeedback — carry it on the object element and not on the value ones. +// +// This matters beyond tidiness: a property the type does not own is exactly the +// shape mxbuild's deserializer tolerates and Studio Pro refuses to open +// (System.InvalidOperationException at MprProperty.cs), so a green build is not +// evidence either way. See the overlay-writes rule in CLAUDE.md. +func TestMappingValueElementsOmitIsDefaultType(t *testing.T) { + imp := importMappingElementToGen(&model.ImportMappingElement{ + Kind: "Object", Entity: "M.Root", ExposedName: "Root", JsonPath: "(Object)", + Children: []*model.ImportMappingElement{ + {Kind: "Value", Attribute: "M.Root.Name", ExposedName: "Name", JsonPath: "(Object)|name", DataType: "String"}, + }, + }, "") + exp := exportMappingElementToGen(&model.ExportMappingElement{ + Kind: "Object", Entity: "M.Root", ExposedName: "Root", JsonPath: "(Object)", + Children: []*model.ExportMappingElement{ + {Kind: "Value", Attribute: "M.Root.Name", ExposedName: "Name", JsonPath: "(Object)|name", DataType: "String"}, + }, + }, "") + + for _, tc := range []struct { + name string + g element.Element + }{ + {"import", imp}, + {"export", exp}, + } { + raw, err := (&codec.Encoder{}).Encode(tc.g) + if err != nil { + t.Fatalf("%s: encode: %v", tc.name, err) + } + var doc bson.Raw = raw + + if _, ok := doc.Lookup("IsDefaultType").BooleanOK(); !ok { + t.Errorf("%s: the OBJECT element must keep IsDefaultType — it owns the property", tc.name) + } + children, ok := doc.Lookup("Children").ArrayOK() + if !ok { + t.Fatalf("%s: no Children array", tc.name) + } + vals, err := children.Values() + if err != nil { + t.Fatalf("%s: read Children: %v", tc.name, err) + } + var seen int + for _, v := range vals { + child, ok := v.DocumentOK() + if !ok { + continue + } + if rawStr(child, "ElementType") != "Value" { + continue + } + seen++ + if _, present := child.Lookup("IsDefaultType").BooleanOK(); present { + t.Errorf("%s: the VALUE element carries IsDefaultType, which its type does not own — "+ + "mxbuild tolerates the unknown property, Studio Pro does not", tc.name) + } + } + if seen == 0 { + t.Errorf("%s: no value element in the encoded document — the assertion proved nothing", tc.name) + } + } +} diff --git a/mdl/backend/modelsdk/mapping_write.go b/mdl/backend/modelsdk/mapping_write.go index 0e053a9d3..daa5b5aa1 100644 --- a/mdl/backend/modelsdk/mapping_write.go +++ b/mdl/backend/modelsdk/mapping_write.go @@ -201,7 +201,11 @@ func importValueElementToGen(id string, elem *model.ImportMappingElement, parent addInt32(g, "MinOccurs", int32(elem.MinOccurs)) addInt32(g, "MaxOccurs", int32(elem.MaxOccurs)) addBool(g, "Nillable", elem.Nillable) - addBool(g, "IsDefaultType", false) + // IsDefaultType is NOT written here: it belongs to the OBJECT element type + // only. The generated metamodel declares it on Import/ExportObjectMappingElement + // and on neither ValueMappingElement, and Studio Pro's own mappings in a blank + // app carry it on the object element alone. A property the type does not own is + // the shape mxbuild accepts and Studio Pro refuses to open. (issue #882) addStr(g, "ElementType", "Value") addStr(g, "Documentation", "") addStr(g, "Converter", "") @@ -366,7 +370,11 @@ func exportValueElementToGen(id string, elem *model.ExportMappingElement, parent // structure also wrote 0 for every element (#841). addInt32(g, "MaxOccurs", int32(elem.MaxOccurs)) addBool(g, "Nillable", true) - addBool(g, "IsDefaultType", false) + // IsDefaultType is NOT written here: it belongs to the OBJECT element type + // only. The generated metamodel declares it on Import/ExportObjectMappingElement + // and on neither ValueMappingElement, and Studio Pro's own mappings in a blank + // app carry it on the object element alone. A property the type does not own is + // the shape mxbuild accepts and Studio Pro refuses to open. (issue #882) addStr(g, "ElementType", "Value") addStr(g, "Documentation", "") addStr(g, "Converter", "") diff --git a/mdl/executor/cmd_export_mappings.go b/mdl/executor/cmd_export_mappings.go index 5cbb2aaf4..04be8c5a2 100644 --- a/mdl/executor/cmd_export_mappings.go +++ b/mdl/executor/cmd_export_mappings.go @@ -211,17 +211,20 @@ func execCreateExportMapping(ctx *ExecContext, s *ast.CreateExportMappingStmt) e em.XmlSchema = s.SchemaRef.String() } - // Build a path→element info map from the JSON structure for schema alignment. - jsElems := map[string]*types.JsonElement{} + // Index the JSON structure for schema alignment. + idx := newJSONSchemaIndex(nil) if s.SchemaKind == "JSON_STRUCTURE" && s.SchemaRef.Module != "" { if js, err2 := ctx.Backend.GetJsonStructureByQualifiedName(s.SchemaRef.Module, s.SchemaRef.Name); err2 == nil && js != nil { - buildJsonElementPathMap(js.Elements, jsElems) + idx = newJSONSchemaIndex(js.Elements) } } // Build element tree from the AST definition, cloning JSON structure properties if s.RootElement != nil { - root := buildExportMappingElementModel(s.Name.Module, s.RootElement, "", "(Object)", jsElems, ctx.Backend, true) + root, err := buildExportMappingElementModel(s.Name.Module, s.RootElement, "", "(Object)", idx, ctx.Backend, true) + if err != nil { + return mdlerrors.NewValidation(fmt.Sprintf("export mapping %s: %v", s.Name.String(), err)) + } em.Elements = append(em.Elements, root) } @@ -248,26 +251,40 @@ func execCreateExportMapping(ctx *ExecContext, s *ast.CreateExportMappingStmt) e // buildExportMappingElementModel converts an AST element definition to a model element. // It clones properties from the matching JSON structure element and adds mapping bindings. -func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingElementDef, parentEntity, parentPath string, jsElems map[string]*types.JsonElement, b backend.FullBackend, isRoot bool) *model.ExportMappingElement { +func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingElementDef, parentEntity, parentPath string, idx *jsonSchemaIndex, b backend.FullBackend, isRoot bool) (*model.ExportMappingElement, error) { elem := &model.ExportMappingElement{ BaseElement: model.BaseElement{ ID: model.ID(types.GenerateID()), }, } - // Determine lookup path - var lookupPath string + // Resolve the member against the JSON structure, accepting the raw JSON key + // or the exposed name. DESCRIBE emits the latter, so both have to work or + // mxcli's own output does not round-trip — re-executing it rewrote + // "(Object)|total" as "(Object)|Total". (issue #882) + var jsElem *types.JsonElement + lookupPath := parentPath + "|" + def.JsonName if isRoot { lookupPath = "(Object)" + jsElem = idx.byPath[lookupPath] } else { - lookupPath = parentPath + "|" + def.JsonName + jsElem = idx.resolve(parentPath, def.JsonName) } - // Clone properties from the matching JSON structure element - if jsElem, ok := jsElems[lookupPath]; ok { + if jsElem == nil && !isRoot { + known := idx.memberNames(parentPath) + if len(known) == 0 { + return nil, fmt.Errorf("%q is not a member of the JSON structure at %s, which has no members there", + def.JsonName, parentPath) + } + return nil, fmt.Errorf("%q is not a member of the JSON structure at %s; available: %s", + def.JsonName, parentPath, strings.Join(known, ", ")) + } + if jsElem != nil { elem.ExposedName = jsElem.ExposedName elem.JsonPath = jsElem.Path elem.MaxOccurs = jsElem.MaxOccurs + lookupPath = jsElem.Path } else { elem.ExposedName = def.JsonName elem.JsonPath = lookupPath @@ -294,7 +311,7 @@ func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingEle } // Check if this is an array element in the JSON structure - if jsElem, ok := jsElems[lookupPath]; ok && jsElem.ElementType == "Array" { + if jsElem != nil && jsElem.ElementType == "Array" { // Export arrays have two levels: // 1. Array container: Kind=Array, entity=container entity, assoc to parent // 2. Item object: Kind=Object, entity=item entity, assoc to container @@ -331,7 +348,7 @@ func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingEle Association: itemAssoc, ObjectHandling: "Find", } - if jsItem, ok2 := jsElems[itemPath]; ok2 { + if jsItem, ok2 := idx.byPath[itemPath]; ok2 { itemElem.ExposedName = jsItem.ExposedName itemElem.JsonPath = jsItem.Path itemElem.MaxOccurs = jsItem.MaxOccurs @@ -342,13 +359,21 @@ func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingEle } // Item's children are the value elements for _, valChild := range itemDef.Children { - itemElem.Children = append(itemElem.Children, buildExportMappingElementModel(moduleName, valChild, itemEntity, itemPath, jsElems, b, false)) + c, err := buildExportMappingElementModel(moduleName, valChild, itemEntity, itemPath, idx, b, false) + if err != nil { + return nil, err + } + itemElem.Children = append(itemElem.Children, c) } elem.Children = append(elem.Children, itemElem) } else { // Fallback: treat children as direct item children (no intermediate entity) for _, child := range def.Children { - elem.Children = append(elem.Children, buildExportMappingElementModel(moduleName, child, entity, itemPath, jsElems, b, false)) + c, err := buildExportMappingElementModel(moduleName, child, entity, itemPath, idx, b, false) + if err != nil { + return nil, err + } + elem.Children = append(elem.Children, c) } } } else { @@ -357,7 +382,11 @@ func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingEle elem.Association = assoc elem.ObjectHandling = handling for _, child := range def.Children { - elem.Children = append(elem.Children, buildExportMappingElementModel(moduleName, child, entity, lookupPath, jsElems, b, false)) + c, err := buildExportMappingElementModel(moduleName, child, entity, lookupPath, idx, b, false) + if err != nil { + return nil, err + } + elem.Children = append(elem.Children, c) } } } else { @@ -382,7 +411,7 @@ func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingEle // JsonPath already set from JSON structure clone above } - return elem + return elem, nil } // execDropExportMapping deletes an export mapping. diff --git a/mdl/executor/cmd_import_mappings.go b/mdl/executor/cmd_import_mappings.go index 3499a1f33..12fe1f885 100644 --- a/mdl/executor/cmd_import_mappings.go +++ b/mdl/executor/cmd_import_mappings.go @@ -220,17 +220,21 @@ func execCreateImportMapping(ctx *ExecContext, s *ast.CreateImportMappingStmt) e im.XmlSchema = s.SchemaRef.String() } - // Build path→JsonElement map from JSON structure — mapping elements clone from this - jsElementsByPath := map[string]*types.JsonElement{} + // Index the JSON structure — mapping elements clone their names, path and + // occurrence bounds from it. + idx := newJSONSchemaIndex(nil) if s.SchemaKind == "JSON_STRUCTURE" && s.SchemaRef.Module != "" { if js, err2 := ctx.Backend.GetJsonStructureByQualifiedName(s.SchemaRef.Module, s.SchemaRef.Name); err2 == nil && js != nil { - buildJsonElementPathMap(js.Elements, jsElementsByPath) + idx = newJSONSchemaIndex(js.Elements) } } // Build element tree from the AST definition, cloning JSON structure properties if s.RootElement != nil { - root := buildImportMappingElementModel(s.Name.Module, s.RootElement, "", "(Object)", ctx.Backend, jsElementsByPath, true) + root, err := buildImportMappingElementModel(s.Name.Module, s.RootElement, "", "(Object)", ctx.Backend, idx, true) + if err != nil { + return mdlerrors.NewValidation(fmt.Sprintf("import mapping %s: %v", s.Name.String(), err)) + } im.Elements = append(im.Elements, root) } @@ -259,23 +263,40 @@ func execCreateImportMapping(ctx *ExecContext, s *ast.CreateImportMappingStmt) e // It clones properties from the matching JSON structure element (ExposedName, JsonPath, // MaxOccurs, ElementType, etc.) and adds mapping-specific bindings (Entity, Attribute, // Association, ObjectHandling). -func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingElementDef, parentEntity, parentPath string, b backend.FullBackend, jsElems map[string]*types.JsonElement, isRoot bool) *model.ImportMappingElement { +func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingElementDef, parentEntity, parentPath string, b backend.FullBackend, idx *jsonSchemaIndex, isRoot bool) (*model.ImportMappingElement, error) { elem := &model.ImportMappingElement{ BaseElement: model.BaseElement{ ID: model.ID(types.GenerateID()), }, } - // Determine lookup path in JSON structure - var lookupPath string - if isRoot { + // Resolve the member against the JSON structure. The authored name may be + // the raw JSON key or the exposed name — DESCRIBE emits the latter, so both + // have to work or mxcli's own output does not round-trip. (#882) + var jsElem *types.JsonElement + lookupPath := parentPath + "|" + def.JsonName + switch { + case isRoot: lookupPath = "(Object)" - } else { - lookupPath = parentPath + "|" + def.JsonName + jsElem = idx.byPath[lookupPath] + default: + jsElem = idx.resolve(parentPath, def.JsonName) + } + + // Clone properties from the matching JSON structure element. A member that + // resolves to nothing is REFUSED, never given a made-up path: the fabricated + // path passed `mxcli check` and surfaced only later — in mxbuild as CE5015, + // or at runtime as an unresolvable mapping. (#882) + if jsElem == nil && !isRoot { + known := idx.memberNames(parentPath) + if len(known) == 0 { + return nil, fmt.Errorf("%q is not a member of the JSON structure at %s, which has no members there", + def.JsonName, parentPath) + } + return nil, fmt.Errorf("%q is not a member of the JSON structure at %s; available: %s", + def.JsonName, parentPath, strings.Join(known, ", ")) } - - // Clone properties from the matching JSON structure element - if jsElem, ok := jsElems[lookupPath]; ok { + if jsElem != nil { elem.ExposedName = jsElem.ExposedName elem.JsonPath = jsElem.Path elem.MinOccurs = jsElem.MinOccurs @@ -286,6 +307,7 @@ func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingEle elem.TotalDigits = jsElem.TotalDigits elem.MaxLength = jsElem.MaxLength } else { + // Root only, and only when the structure could not be read at all. elem.ExposedName = def.JsonName elem.JsonPath = lookupPath elem.Nillable = true @@ -319,10 +341,10 @@ func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingEle // For arrays: skip the container, use the item path directly. // Studio Pro represents arrays as a single ObjectMappingElement at the |(Object) item path. - childPath := lookupPath - if jsElem, ok := jsElems[lookupPath]; ok && jsElem.ElementType == "Array" { - itemPath := lookupPath + "|(Object)" - if jsItem, ok2 := jsElems[itemPath]; ok2 { + childPath := elem.JsonPath + if jsElem != nil && jsElem.ElementType == "Array" { + itemPath := jsElem.Path + "|(Object)" + if jsItem, ok2 := idx.byPath[itemPath]; ok2 { elem.ExposedName = jsItem.ExposedName elem.JsonPath = jsItem.Path elem.MinOccurs = jsItem.MinOccurs @@ -333,7 +355,11 @@ func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingEle } for _, child := range def.Children { - elem.Children = append(elem.Children, buildImportMappingElementModel(moduleName, child, entity, childPath, b, jsElems, false)) + c, err := buildImportMappingElementModel(moduleName, child, entity, childPath, b, idx, false) + if err != nil { + return nil, err + } + elem.Children = append(elem.Children, c) } } else { // Value mapping — bind to attribute @@ -357,7 +383,7 @@ func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingEle elem.Attribute = attr } - return elem + return elem, nil } // buildJsonElementPathMap recursively builds a map from JSON path → JsonElement. @@ -371,6 +397,96 @@ func buildJsonElementPathMap(elems []*types.JsonElement, m map[string]*types.Jso } } +// jsonSchemaIndex resolves a member named in MDL to its JSON structure element. +// +// A JSON structure element carries TWO names, and they routinely differ: +// +// Path "(Object)|uuid" the raw JSON key — what the RUNTIME resolves by +// ExposedName "Uuid" Mendix's derived name — what Studio Pro DISPLAYS +// +// Mendix derives ExposedName by capitalising the initial (and, for an array's +// item object, by suffixing "Item"), so the two diverge for any lowercase-initial +// key. Both are stored, by Studio Pro too — the blank app's own +// FeedbackModule.JSON_AppInsightsResponse holds ExposedName "Uuid" against Path +// "(Object)|uuid". +// +// DESCRIBE prints ExposedName (it is the name Studio Pro shows), so mxcli's own +// output names members the raw-key lookup could not find. Re-executing a DESCRIBE +// therefore FABRICATED a path from the exposed name — "(Object)|Uuid", and for an +// array the "|(Object)" item marker vanished entirely — producing a mapping that +// resolves against nothing at runtime. Accepting both spellings is what makes +// DESCRIBE round-trip. (issue #882) +type jsonSchemaIndex struct { + byPath map[string]*types.JsonElement + children map[string][]*types.JsonElement // parent path → children, in order +} + +func newJSONSchemaIndex(elems []*types.JsonElement) *jsonSchemaIndex { + idx := &jsonSchemaIndex{ + byPath: map[string]*types.JsonElement{}, + children: map[string][]*types.JsonElement{}, + } + idx.add("", elems) + return idx +} + +func (i *jsonSchemaIndex) add(parentPath string, elems []*types.JsonElement) { + for _, e := range elems { + if e == nil { + continue + } + i.byPath[e.Path] = e + i.children[parentPath] = append(i.children[parentPath], e) + i.add(e.Path, e.Children) + } +} + +// resolve finds the element a member name refers to under parentPath, accepting +// the raw JSON key or the exposed name. For an array addressed by its ITEM's +// exposed name ("__ValueItem"), it returns the ARRAY element, so the caller's +// array branch still takes the "|(Object)" step to the item. +// +// Returns nil when the name matches nothing — the caller must refuse rather than +// invent a path. A fabricated path passes `mxcli check` and only fails later, in +// mxbuild as CE5015 or, worse, at runtime. +func (i *jsonSchemaIndex) resolve(parentPath, name string) *types.JsonElement { + if e, ok := i.byPath[parentPath+"|"+name]; ok { + return e + } + for _, c := range i.children[parentPath] { + if c.ExposedName == name { + return c + } + } + for _, c := range i.children[parentPath] { + if c.ElementType != "Array" { + continue + } + if item, ok := i.byPath[c.Path+"|(Object)"]; ok && item.ExposedName == name { + return c + } + } + return nil +} + +// memberNames lists the spellings that would have resolved under parentPath, so a +// rejection can name them instead of leaving the author to guess. +func (i *jsonSchemaIndex) memberNames(parentPath string) []string { + var out []string + for _, c := range i.children[parentPath] { + raw := c.Path + if idx := strings.LastIndex(raw, "|"); idx >= 0 { + raw = raw[idx+1:] + } + if c.ExposedName != "" && c.ExposedName != raw { + out = append(out, fmt.Sprintf("%s (or %s)", raw, c.ExposedName)) + continue + } + out = append(out, raw) + } + return out +} + // resolveAttributeType looks up the data type of an entity attribute from the project. // Returns "String" as default if the attribute cannot be found. func resolveAttributeType(entityQN, attrName string, b backend.DomainModelBackend) string { diff --git a/mdl/executor/cmd_mappings_member_resolution_test.go b/mdl/executor/cmd_mappings_member_resolution_test.go new file mode 100644 index 000000000..14205b21d --- /dev/null +++ b/mdl/executor/cmd_mappings_member_resolution_test.go @@ -0,0 +1,179 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" +) + +// upstream #882. A JSON structure element carries TWO names and they routinely +// differ — Mendix derives the exposed name by capitalising the initial, and for +// an array's item object by suffixing "Item": +// +// Path "(Object)|uuid" the raw JSON key — what the RUNTIME resolves by +// ExposedName "Uuid" Mendix's derived name — what Studio Pro DISPLAYS +// +// Both are Mendix's own: the blank app's Studio-Pro-authored +// FeedbackModule.JSON_AppInsightsResponse stores ExposedName "Uuid" against Path +// "(Object)|uuid", and its IMM_PostResponse binds JsonPath "(Object)|uuid". +// +// mxcli's DESCRIBE prints the exposed name, but its builder resolved only raw +// keys — so mxcli's own output did not round-trip. Re-executing a DESCRIBE +// FABRICATED a path from the exposed name and mxbuild rejected it with CE5015; +// for an array the "|(Object)" item marker vanished entirely. + +// jsonStructureFixture mirrors the shape in the issue: a lowercase-initial +// member, a camelCase one, a nested object, and an object array under an +// underscore-prefixed key. +func jsonStructureFixture() *jsonSchemaIndex { + return newJSONSchemaIndex([]*types.JsonElement{{ + ExposedName: "Root", Path: "(Object)", ElementType: "Object", MinOccurs: 1, MaxOccurs: 1, + Children: []*types.JsonElement{ + {ExposedName: "Total", Path: "(Object)|total", ElementType: "Value", MaxOccurs: 1}, + {ExposedName: "CamelCase", Path: "(Object)|camelCase", ElementType: "Value", MaxOccurs: 1}, + { + ExposedName: "EntityInstances", Path: "(Object)|entityInstances", ElementType: "Object", MaxOccurs: 1, + Children: []*types.JsonElement{ + {ExposedName: "__returnedCount", Path: "(Object)|entityInstances|__returnedCount", ElementType: "Value", MaxOccurs: 1}, + { + ExposedName: "__Value", Path: "(Object)|entityInstances|__Value", ElementType: "Array", MaxOccurs: 1, + Children: []*types.JsonElement{{ + ExposedName: "__ValueItem", Path: "(Object)|entityInstances|__Value|(Object)", ElementType: "Object", MaxOccurs: -1, + Children: []*types.JsonElement{ + {ExposedName: "Name", Path: "(Object)|entityInstances|__Value|(Object)|name", ElementType: "Value", MaxOccurs: 1}, + }, + }}, + }, + }, + }, + }, + }}) +} + +func TestJSONSchemaIndexResolvesEitherSpelling(t *testing.T) { + idx := jsonStructureFixture() + + for _, tc := range []struct { + parent, name, wantPath string + why string + }{ + {"(Object)", "total", "(Object)|total", "the raw JSON key, as hand-written MDL spells it"}, + {"(Object)", "Total", "(Object)|total", "the exposed name, as DESCRIBE emits it"}, + {"(Object)", "camelCase", "(Object)|camelCase", "raw key with interior capitals"}, + {"(Object)", "CamelCase", "(Object)|camelCase", "exposed name of the same member"}, + {"(Object)", "entityInstances", "(Object)|entityInstances", "nested object, raw"}, + {"(Object)", "EntityInstances", "(Object)|entityInstances", "nested object, exposed"}, + {"(Object)|entityInstances", "__returnedCount", "(Object)|entityInstances|__returnedCount", + "an underscore-prefixed key is not capitalised, so both spellings coincide"}, + {"(Object)|entityInstances", "__Value", "(Object)|entityInstances|__Value", "the array itself"}, + {"(Object)|entityInstances", "__ValueItem", "(Object)|entityInstances|__Value", + "the array addressed by its ITEM's exposed name must resolve to the ARRAY, so the " + + "caller still takes the |(Object) step to the item"}, + } { + got := idx.resolve(tc.parent, tc.name) + if got == nil { + t.Errorf("resolve(%q, %q) = nil, want %s (%s)", tc.parent, tc.name, tc.wantPath, tc.why) + continue + } + if got.Path != tc.wantPath { + t.Errorf("resolve(%q, %q).Path = %q, want %q (%s)", tc.parent, tc.name, got.Path, tc.wantPath, tc.why) + } + } + + if got := idx.resolve("(Object)", "nope"); got != nil { + t.Errorf("resolve of an absent member = %q, want nil — the caller must refuse, not invent a path", got.Path) + } +} + +// buildImportMappingElementModel must clone the structure's Path verbatim +// whichever spelling was authored. Fabricating one from the exposed name is what +// #882 was: `mxcli check` passed and mxbuild reported CE5015. +func TestImportMappingClonesTheStructurePathForBothSpellings(t *testing.T) { + // The exposed-name spelling is exactly what `describe import mapping` emits. + def := &ast.ImportMappingElementDef{ + Entity: "B.Root", + Children: []*ast.ImportMappingElementDef{ + {Attribute: "Total", JsonName: "Total"}, + {Attribute: "Camel", JsonName: "CamelCase"}, + { + Entity: "B.Inner", Association: "B.Root_Inner", JsonName: "EntityInstances", + Children: []*ast.ImportMappingElementDef{ + {Attribute: "Cnt", JsonName: "__returnedCount"}, + { + Entity: "B.Item", Association: "B.Inner_Item", JsonName: "__ValueItem", + Children: []*ast.ImportMappingElementDef{{Attribute: "Nm", JsonName: "Name"}}, + }, + }, + }, + }, + } + + root, err := buildImportMappingElementModel("B", def, "", "(Object)", nil, jsonStructureFixture(), true) + if err != nil { + t.Fatalf("build: %v", err) + } + + got := map[string]string{} + var collect func(e *model.ImportMappingElement) + collect = func(e *model.ImportMappingElement) { + if e == nil { + return + } + got[e.ExposedName] = e.JsonPath + for _, c := range e.Children { + collect(c) + } + } + collect(root) + + want := map[string]string{ + "Root": "(Object)", + "Total": "(Object)|total", + "CamelCase": "(Object)|camelCase", + "EntityInstances": "(Object)|entityInstances", + "__returnedCount": "(Object)|entityInstances|__returnedCount", + // The array binds at the ITEM path — the "|(Object)" marker is the whole + // point, and re-executing DESCRIBE used to drop it and leave + // "(Object)|EntityInstances|__ValueItem". + "__ValueItem": "(Object)|entityInstances|__Value|(Object)", + "Name": "(Object)|entityInstances|__Value|(Object)|name", + } + for name, wantPath := range want { + if got[name] != wantPath { + t.Errorf("%s: JsonPath = %q, want %q", name, got[name], wantPath) + } + } +} + +// An unresolvable member is REFUSED. Before this it got a fabricated path that +// passed `mxcli check` and surfaced only in mxbuild (CE5015) or at runtime — the +// worst failure mode, because the tool that wrote it reported success. +func TestImportMappingRefusesAnUnknownMember(t *testing.T) { + def := &ast.ImportMappingElementDef{ + Entity: "B.Root", + Children: []*ast.ImportMappingElementDef{{Attribute: "Camel", JsonName: "camelCse"}}, + } + + _, err := buildImportMappingElementModel("B", def, "", "(Object)", nil, jsonStructureFixture(), true) + if err == nil { + t.Fatal("a member that is in no JSON structure must be refused, not written with an invented path") + } + msg := err.Error() + for _, want := range []string{"camelCse", "(Object)"} { + if !strings.Contains(msg, want) { + t.Errorf("error should name %q, got: %s", want, msg) + } + } + // Both spellings are offered, because either is accepted and the author has + // no way to know which one they mistyped. + for _, want := range []string{"camelCase", "CamelCase"} { + if !strings.Contains(msg, want) { + t.Errorf("error should suggest %q, got: %s", want, msg) + } + } +} diff --git a/sdk/mpr/writer_export_mapping.go b/sdk/mpr/writer_export_mapping.go index 4d2a779ee..4a5f98cc2 100644 --- a/sdk/mpr/writer_export_mapping.go +++ b/sdk/mpr/writer_export_mapping.go @@ -170,7 +170,11 @@ func serializeExportValueElement(id string, elem *model.ExportMappingElement, pa // JSON structure also wrote 0 for every element (#841). {Key: "MaxOccurs", Value: int32(elem.MaxOccurs)}, {Key: "Nillable", Value: true}, - {Key: "IsDefaultType", Value: false}, + // IsDefaultType is NOT written here: it belongs to the OBJECT element type + // only. The generated metamodel declares it on Import/ExportObjectMappingElement + // and on neither ValueMappingElement, and Studio Pro's own mappings in a blank + // app carry it on the object element alone. A property the type does not own is + // the shape mxbuild accepts and Studio Pro refuses to open. (issue #882) {Key: "ElementType", Value: "Value"}, {Key: "Documentation", Value: ""}, {Key: "Converter", Value: ""}, diff --git a/sdk/mpr/writer_import_mapping.go b/sdk/mpr/writer_import_mapping.go index 776f2410a..4c32a5009 100644 --- a/sdk/mpr/writer_import_mapping.go +++ b/sdk/mpr/writer_import_mapping.go @@ -170,7 +170,11 @@ func serializeImportValueElement(id string, elem *model.ImportMappingElement, pa {Key: "MinOccurs", Value: int32(elem.MinOccurs)}, {Key: "MaxOccurs", Value: int32(elem.MaxOccurs)}, {Key: "Nillable", Value: elem.Nillable}, - {Key: "IsDefaultType", Value: false}, + // IsDefaultType is NOT written here: it belongs to the OBJECT element type + // only. The generated metamodel declares it on Import/ExportObjectMappingElement + // and on neither ValueMappingElement, and Studio Pro's own mappings in a blank + // app carry it on the object element alone. A property the type does not own is + // the shape mxbuild accepts and Studio Pro refuses to open. (issue #882) {Key: "ElementType", Value: "Value"}, {Key: "Documentation", Value: ""}, {Key: "Converter", Value: ""}, From 0ed74ab2da4cc6a63c97b1118da9cf83523572df Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 15:50:11 +0000 Subject: [PATCH 18/20] fix(mappings): stop copying the JSON snippet's sample value onto mapping elements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A JSON structure element's OriginalValue is the SAMPLE parsed out of the snippet ("42", "\"Widget\""). It describes the structure, not the mapping, and Studio Pro leaves it empty on every mapping element. Measured across the two Studio-Pro-authored mappings a blank app ships — FeedbackModule.IMM_PostResponse and EMM_PostFeedback, ~15 value elements between them — all write OriginalValue "", while their JSON structures carry 17 non-empty samples. mxcli cloned the sample in, so an mxcli-written mapping differed from a Studio-Pro-written one over the same structure by the snippet's example data, which is the first thing a side-by-side comparison of the two shows. Refs upstream #882. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- .claude/skills/fix-issue.md | 2 +- .../bug-tests/882-mapping-member-names.mdl | 9 +++++ mdl/executor/cmd_import_mappings.go | 9 ++++- .../cmd_mappings_member_resolution_test.go | 40 +++++++++++++++++++ 4 files changed, 58 insertions(+), 2 deletions(-) diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 8eff1d7c5..0c7430715 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -491,4 +491,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `show callers of ` and `show references to ` report "(no callers found)" for a document reached only from a page action button — a false negative that reads as "safe to delete" | TWO independent defects behind one symptom. (1) `scanWidgetOwnRefs` collected `Entity`/`Microflow`/`Nanoflow` from a widget's raw BSON but not **`Form`**, the key a PAGE reference uses, so `widgets_data` had no page column and the refs projection had no page row. (2) `execShowCallers` filtered `RefKind = 'call'` — the kind a microflow CALL ACTIVITY produces — so the button→microflow row, which was already in the refs table, was hidden by the query | `mdl/catalog/builder_pages.go` (`scanWidgetOwnRefs` + `rawWidgetInfo.PageRef`), `mdl/catalog/tables.go` (`widgets_data.PageRef`), `mdl/catalog/builder_references.go` (projection row), `mdl/executor/cmd_search.go` (`callerRefKinds`) | **Find the live code path before fixing anything** — `builder_references.go` has an inviting `extractWidgetRefs` with a per-widget-type switch, and it is DEAD: its only caller is its own recursion. Extending it changes nothing. The live path is a SQL projection out of `widgets_data`, and the standing `NOTE: widget-level datasource/action refs still require a parsed widget tree` comment beside it is the tell. **Separate "the reference is missing" from "the query hides it"** by reading the refs table directly: the button→microflow row was present all along, so fixing only the scanner would have closed half the issue and left the reporter's second scenario broken. **`Form` is `Page`** — the same rename behind `ShowFormAction`/`CloseFormAction` (CLAUDE.md's storage-name table); grepping for `Page` in a BSON scanner finds nothing. **One action can carry two references**: `create object … then open page` holds an entity AND a page, so collecting the entity alone still leaves the page unreferenced. **Do not widen `callers` into `references`** — `datasource`/`parameter`/`generalize` are uses of a TYPE, not invocations, and including them makes the two commands synonyms; the test pins both the included and the excluded set. Tests `builder_pages_test.go` (`TestScanWidgetOwnRefs_PageReference`), `cmd_search_callers_test.go`. upstream #773 | | `ALTER PAGE` over `--mcp` fails against Studio Pro **11.13** with `pg_patch_page: … PROP_NOT_PRIMITIVE: Property 'widgets' is not a primitive property`. `CREATE PAGE` is fine; the page itself is left intact | 11.13 gave `pg_read_page` a **`depth` argument defaulting to 4**, replacing anything deeper with the literal string `"..."`. ALTER PAGE is read-modify-**replace-whole-page**, so the truncated read went straight back as the new page body. Measured live: `Administration.Account_Overview` read 32,594 bytes at full depth but **1,052 bytes** at the default, its entire tree reduced to `{"widgets":["...","..."]}`. Every ordinary page truncates — three of three PgTest pages did | `mdl/backend/mcp/page.go` (`pgReadPage`, `pgReadFullDepth`, `hasTruncationSentinel`), `mdl/backend/mcp/client.go` (`SupportsToolArg`) | Request the full depth, and **guard rather than trust it**: refuse a read still carrying the sentinel instead of letting a partial page reach a write (ADR-0005 guard-don't-drop). Two traps. (1) **Do not send `depth` unconditionally** — 11.11/11.12 declare `pg_read_page` `additionalProperties:false` without it, so the whole call fails; gate on a live `tools/list` probe of the tool's input schema, because `serverInfo.version` is frozen at `1.0.0` across 11.11/11.12/11.13 and cannot discriminate releases. (2) **Match the sentinel only as an array element** — a caption or title legitimately reading `"..."` is real content, and a naive substring scan rejects valid pages. The release notes announced none of this, exactly as 11.12 silently removed `pg_write_page` (#697): on any Studio Pro upgrade, re-probe `tools/list` and diff the input schemas, not just the tool names. Tests `mdl/backend/mcp/page_depth_test.go`; controls: stub the depth arg (full-depth test fails) and stub the guard (truncation test fails) | | An import activity's Range — Studio Pro's **All / First / Custom** — is absent from `DESCRIBE MICROFLOW`, and a `limit`/`offset` set in Studio Pro does not survive an mxcli round trip | Worse than "undescribed". `Microflows$ImportMappingCall.Range` is polymorphic — `ConstantRange{SingleObject}` (All/First) or `CustomRange{LimitExpression, OffsetExpression}` (Custom) — and mxcli wrote only the first and read only `SingleObject`. So **Custom was unrepresentable**, a bounded import became unbounded on any rewrite, and all three settings described identically, so describe→edit→exec silently changed the activity | `mdl/grammar/domains/MDLMicroflow.g4` (`importMappingRange`) + `MDLLexer.g4` (`FIRST`) + `MDLSettings.g4` (keyword list), `mdl/visitor/visitor_import_export_mapping.go`, `mdl/ast/ast_microflow.go`, `sdk/microflows/microflows_actions.go` (`RangeSingleObject`, `RangeSingleObjectOf`), `mdl/executor/cmd_microflows_builder_calls.go` (`addImportFromMappingAction`), `mdl/executor/cmd_microflows_format_action.go` (`formatImportMappingRange`), `mdl/backend/modelsdk/microflow_write.go` (`importMappingRangeToGen`) + `microflow_read_actions.go`, `sdk/mpr/writer_microflow_actions.go` (`importMappingRange`) + `parser_microflow_actions.go` | **The Range and the result variable's CARDINALITY are separate axes** — the first fix folded them and every `all` wrote a `ListType` against an object-rooted mapping, which mxbuild rejects with **CE0243** ("the mapping used to return 'List of X' but now returns 'X'"). Mendix's own `FeedbackModule.SUB_Feedback_PostToAppInsights` settles it: `ConstantRange{SingleObject:false}` against an **ObjectType** variable. The stored `VariableType` is the authority on cardinality; the range is its own flag (hence `RangeSingleObject *bool`, nil = "not authored, fall back"). **Count the write sites before declaring a serialization fixed** — the `ImportMappingCall` is built at THREE places (`importXmlActionToGen`, the REST `ResultHandlingMapping` case, and the legacy writer), and patching two let a limit reach the model while a `ConstantRange` was still written; extract one helper. **DESCRIBE must emit one of the three, never nothing** — silence re-enters the builder's inference, and an object-rooted mapping set to All (Studio Pro's default) comes back as First. **`first` is not `limit 1`**: one binds an OBJECT, the other a one-element LIST, so they cannot share syntax. **A platform rule you cannot author a positive case for is documentation, not validation** — Mendix rejects `offset` on a non-list mapping with **CE6100** while accepting `limit`, and mxcli cannot currently author a list-rooted import mapping (array-root mappings emit CE5015), so the constraint is documented rather than guessed at in a checker. Tests `mdl/executor/cmd_microflows_import_range_test.go`, `mdl/backend/modelsdk/microflow_import_range_test.go`, `sdk/mpr/parser_microflow_import_range_test.go`, example `mdl-examples/bug-tests/881-import-mapping-range.mdl`. upstream #881 | -| `CREATE IMPORT MAPPING` is reported to "not preserve JSON member names": `total` becomes `Total`, `camelCase` becomes `CamelCase`, `__Value` becomes `__ValueItem`. The import then fails at runtime with `key not found: Path(QName(None,),None,)` at `MappingCache.storeValueMappingElement` | The capitalisation is **Mendix's own** and not the defect. A `JsonStructures$JsonElement` stores TWO names — `Path` (the raw JSON key, `(Object)\|uuid`, which the RUNTIME resolves by) and `ExposedName` (derived by capitalising the initial, and suffixing `Item` on an array's item object, which Studio Pro DISPLAYS). The real defect: mxcli's DESCRIBE emits `ExposedName` while its builder resolved only raw keys, so mxcli's own output did not round-trip — re-executing a DESCRIBE FABRICATED a path (`(Object)\|Total`; for arrays the `\|(Object)` item marker vanished entirely, leaving `(Object)\|EntityInstances\|__ValueItem`) and zeroed MaxOccurs. Export mappings had the identical bug. Separately, mxcli wrote `IsDefaultType` on every `ValueMappingElement`, a property only the OBJECT element types own | `mdl/executor/cmd_import_mappings.go` (`jsonSchemaIndex`, `newJSONSchemaIndex`, `resolve`, `memberNames`; `buildImportMappingElementModel` now returns an error), `mdl/executor/cmd_export_mappings.go` (same), `mdl/backend/modelsdk/mapping_write.go` + `sdk/mpr/writer_import_mapping.go` + `sdk/mpr/writer_export_mapping.go` (drop `IsDefaultType` from the value writers) | **A blank Mendix app is a free Studio-Pro-authored fixture** — `FeedbackModule.IMM_PostResponse` + `JSON_AppInsightsResponse` settle "what does Studio Pro actually write?" with no Studio Pro and no marketplace download: they store `ExposedName "Uuid"` against `Path "(Object)|uuid"`. Author the SAME mapping over the SAME structure in the SAME module with mxcli and diff the two units — the only variable left is which tool wrote it. **Run the reporter's repro as written before believing the diagnosis**: theirs produced paths byte-identical to the structure's, so their MDL was not the failing input; the corruption needed a DESCRIBE in the middle. **DESCRIBE emitting a different name than the parser accepts is a round-trip bug even when both are "correct"** — the fix is to accept BOTH spellings, not to change what DESCRIBE prints, because the exposed name is the one Studio Pro shows. **Never fabricate a path for an unresolved member**: the invented path passed `mxcli check` and surfaced only in mxbuild (CE5015) or at runtime, so the tool that wrote it reported success — refuse, and list the spellings that would have worked. **`generated/metamodel` decides property ownership**: `isDefaultType` is declared on `Import/ExportObjectMappingElement` and on neither `ValueMappingElement`, and per CLAUDE.md's overlay rule an extra property is the shape mxbuild tolerates and Studio Pro will not open — so a green build proves nothing here. Tests `mdl/executor/cmd_mappings_member_resolution_test.go`, `mdl/backend/modelsdk/mapping_isdefaulttype_test.go`, example `mdl-examples/bug-tests/882-mapping-member-names.mdl`. upstream #882 | +| `CREATE IMPORT MAPPING` is reported to "not preserve JSON member names": `total` becomes `Total`, `camelCase` becomes `CamelCase`, `__Value` becomes `__ValueItem`. The import then fails at runtime with `key not found: Path(QName(None,),None,)` at `MappingCache.storeValueMappingElement` | The capitalisation is **Mendix's own** and not the defect. A `JsonStructures$JsonElement` stores TWO names — `Path` (the raw JSON key, `(Object)\|uuid`, which the RUNTIME resolves by) and `ExposedName` (derived by capitalising the initial, and suffixing `Item` on an array's item object, which Studio Pro DISPLAYS). The real defect: mxcli's DESCRIBE emits `ExposedName` while its builder resolved only raw keys, so mxcli's own output did not round-trip — re-executing a DESCRIBE FABRICATED a path (`(Object)\|Total`; for arrays the `\|(Object)` item marker vanished entirely, leaving `(Object)\|EntityInstances\|__ValueItem`) and zeroed MaxOccurs. Export mappings had the identical bug. Separately, mxcli wrote `IsDefaultType` on every `ValueMappingElement` (a property only the OBJECT element types own) and cloned the JSON structure's `OriginalValue` — the SAMPLE value from the snippet — onto every mapping element, where Studio Pro leaves it empty | `mdl/executor/cmd_import_mappings.go` (`jsonSchemaIndex`, `newJSONSchemaIndex`, `resolve`, `memberNames`; `buildImportMappingElementModel` now returns an error), `mdl/executor/cmd_export_mappings.go` (same), `mdl/backend/modelsdk/mapping_write.go` + `sdk/mpr/writer_import_mapping.go` + `sdk/mpr/writer_export_mapping.go` (drop `IsDefaultType` from the value writers); `cmd_import_mappings.go` again for the `OriginalValue` clone | **A blank Mendix app is a free Studio-Pro-authored fixture** — `FeedbackModule.IMM_PostResponse` + `JSON_AppInsightsResponse` settle "what does Studio Pro actually write?" with no Studio Pro and no marketplace download: they store `ExposedName "Uuid"` against `Path "(Object)|uuid"`. Author the SAME mapping over the SAME structure in the SAME module with mxcli and diff the two units — the only variable left is which tool wrote it. **Run the reporter's repro as written before believing the diagnosis**: theirs produced paths byte-identical to the structure's, so their MDL was not the failing input; the corruption needed a DESCRIBE in the middle. **DESCRIBE emitting a different name than the parser accepts is a round-trip bug even when both are "correct"** — the fix is to accept BOTH spellings, not to change what DESCRIBE prints, because the exposed name is the one Studio Pro shows. **Never fabricate a path for an unresolved member**: the invented path passed `mxcli check` and surfaced only in mxbuild (CE5015) or at runtime, so the tool that wrote it reported success — refuse, and list the spellings that would have worked. **`generated/metamodel` decides property ownership**: `isDefaultType` is declared on `Import/ExportObjectMappingElement` and on neither `ValueMappingElement`, and per CLAUDE.md's overlay rule an extra property is the shape mxbuild tolerates and Studio Pro will not open — so a green build proves nothing here. **Count the samples before calling a difference a defect**: two Studio-Pro-authored mappings in a blank app (~15 value elements) all write `OriginalValue: ""` while their structures carry 17 non-empty samples, which is what makes "the sample belongs to the structure" a measurement rather than an opinion. **Running the reporter's repro is how you find out it is not a reproduction**: theirs PASSED at runtime on 11.6.6 and 11.13.0, so the runtime error they see needs something their standalone file does not carry — say so instead of claiming the fix. And **control the harness before believing its failure**: an early runtime run failed until the control (Studio Pro's own mapping, same harness) failed identically, which located the fault in the test's missing module-role grants, not in the mapping. Tests `mdl/executor/cmd_mappings_member_resolution_test.go`, `mdl/backend/modelsdk/mapping_isdefaulttype_test.go`, example `mdl-examples/bug-tests/882-mapping-member-names.mdl`. upstream #882 | diff --git a/mdl-examples/bug-tests/882-mapping-member-names.mdl b/mdl-examples/bug-tests/882-mapping-member-names.mdl index 196eb03e8..1bfbe301d 100644 --- a/mdl-examples/bug-tests/882-mapping-member-names.mdl +++ b/mdl-examples/bug-tests/882-mapping-member-names.mdl @@ -102,6 +102,15 @@ create export mapping B882.EM_ExposedNames with json structure B882.JSX }; / +-- Third divergence from Studio Pro, and the one a side-by-side comparison shows +-- first: mxcli copied the JSON structure's OriginalValue — the SAMPLE value +-- parsed out of the snippet ("42", "\"Widget\"") — onto every mapping element. +-- The sample belongs to the STRUCTURE. Studio Pro leaves it empty on the mapping: +-- across the two Studio-Pro-authored mappings a blank app ships +-- (FeedbackModule.IMM_PostResponse and EMM_PostFeedback, ~15 value elements) all +-- are "", while their structures carry 17 non-empty samples. mxcli no longer +-- clones it. +-- -- A member matching neither spelling is refused at exec, naming both forms: -- -- create import mapping B882.IM_Typo with json structure B882.JS diff --git a/mdl/executor/cmd_import_mappings.go b/mdl/executor/cmd_import_mappings.go index 12fe1f885..b966e59ab 100644 --- a/mdl/executor/cmd_import_mappings.go +++ b/mdl/executor/cmd_import_mappings.go @@ -302,7 +302,14 @@ func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingEle elem.MinOccurs = jsElem.MinOccurs elem.MaxOccurs = jsElem.MaxOccurs elem.Nillable = jsElem.Nillable - elem.OriginalValue = jsElem.OriginalValue + // OriginalValue is deliberately NOT cloned. It is the sample value parsed + // out of the JSON structure's snippet ("42", "\"Widget\""), and it belongs + // to the STRUCTURE — Studio Pro leaves it empty on every mapping element. + // Measured across the two Studio-Pro-authored mappings a blank app ships + // (FeedbackModule's IMM_PostResponse and EMM_PostFeedback, ~15 value + // elements between them): all "", while their structures carry 17 non-empty + // samples. Copying the sample in makes an mxcli-written mapping differ from + // a Studio-Pro-written one over the same structure. (issue #882) elem.FractionDigits = jsElem.FractionDigits elem.TotalDigits = jsElem.TotalDigits elem.MaxLength = jsElem.MaxLength diff --git a/mdl/executor/cmd_mappings_member_resolution_test.go b/mdl/executor/cmd_mappings_member_resolution_test.go index 14205b21d..864180fc7 100644 --- a/mdl/executor/cmd_mappings_member_resolution_test.go +++ b/mdl/executor/cmd_mappings_member_resolution_test.go @@ -177,3 +177,43 @@ func TestImportMappingRefusesAnUnknownMember(t *testing.T) { } } } + +// The JSON structure's OriginalValue is the SAMPLE parsed out of the snippet +// ("42", "\"Widget\""). It belongs to the structure, and Studio Pro leaves it +// empty on every mapping element: measured across the two Studio-Pro-authored +// mappings a blank app ships (FeedbackModule's IMM_PostResponse and +// EMM_PostFeedback, ~15 value elements), all "" — while their structures carry +// 17 non-empty samples. +// +// mxcli cloned it in, so an mxcli-written mapping differed from a +// Studio-Pro-written one over the same structure by the snippet's example data. +// That difference is exactly what a reporter comparing the two would see. +func TestImportMappingDoesNotCloneTheSnippetSampleValue(t *testing.T) { + idx := newJSONSchemaIndex([]*types.JsonElement{{ + ExposedName: "Root", Path: "(Object)", ElementType: "Object", MinOccurs: 1, MaxOccurs: 1, + Children: []*types.JsonElement{{ + ExposedName: "Total", Path: "(Object)|total", ElementType: "Value", + MaxOccurs: 1, OriginalValue: "42", + }}, + }}) + + root, err := buildImportMappingElementModel("B", &ast.ImportMappingElementDef{ + Entity: "B.Root", + Children: []*ast.ImportMappingElementDef{{Attribute: "Total", JsonName: "total"}}, + }, "", "(Object)", nil, idx, true) + if err != nil { + t.Fatalf("build: %v", err) + } + + if len(root.Children) != 1 { + t.Fatalf("expected one child, got %d", len(root.Children)) + } + if got := root.Children[0].OriginalValue; got != "" { + t.Errorf("OriginalValue = %q, want empty — the snippet's sample value belongs to the "+ + "JSON structure, and Studio Pro leaves it off the mapping element", got) + } + // The clone itself must still happen: the path is the whole point. + if got := root.Children[0].JsonPath; got != "(Object)|total" { + t.Errorf("JsonPath = %q, want (Object)|total", got) + } +} From 8289656e76da773bba8e716fd3b013c9aa6b8e7a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 15:55:19 +0000 Subject: [PATCH 19/20] Stop rebuilding the same marketplace reference project every time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `marketplace diff` and `marketplace update` answer their question by building a reference project: a blank Mendix app of the consuming project's version with the published module imported into it. That costs a download plus two `mx` invocations, and `update` builds two — the installed version, to establish the local-edit baseline, and the target. Neither result depends on anything but its inputs, and both were thrown away at the end of every run. Two caches under ~/.mxcli/marketplace-refs/, because they miss in different places. blank/ is keyed by Mendix version and takes the `mx create-project` out of EVERY reference build — a run updating six modules built twelve identical blank apps. ref/ is keyed by (published version UUID, Mendix version) and holds the finished reference, so a `diff` followed by an `update` does not build the same base twice. Measured on Administration (23513, 4.3.2 → 4.5.0, Mendix 11.12.1), with byte-identical output across all four runs — 21 of 21 elements unchanged, 6 elements an upgrade would touch: diff, no cache 66s diff, cold cache 49s diff, warm 13s update 4.3.2→4.4.0 24s (base cached, target built; 9 identities preserved, 2 role grants restored) The version UUID is the key, never the version number: numbers collide across content, so a blank 11.12.1 app has Atlas_Web_Content 4.1.0 while Administration has also published a 4.1.0. The Mendix version is in both keys because a reference built at another version reports Mendix's own conversions as user edits, and the project's version stamp is re-verified on the way out of the cache as well as before it goes in — an entry written by an older mxcli is dropped rather than trusted. An entry is a directory tree, so its presence proves nothing: the completion marker is written last, after an atomic rename, and its absence means rebuild. Anything unreadable is deleted and rebuilt rather than diagnosed, because a partial reference does not fail loudly — it reads as local edits. ref/ is bounded to the 6 most recently used entries (34 MB each, and twelve of them is ~400 MB of a container that may have 2 GB free). Running out of disk part way through an update is worse than rebuilding a reference, because `marketplace update` does not roll back. blank/ is unbounded — one entry per Mendix version, and it is the one that pays on every build. MXCLI_REF_CACHE_MAX changes the bound; MXCLI_NO_REF_CACHE=1 builds everything from scratch, so a suspected stale-cache problem can be ruled out without destroying the evidence. Not done, and deliberately: FINDINGS §59 also asks for the baseline to be skipped under --force, on the grounds that it is computed and thrown away. It is not — gateOnLocalEdits PRINTS the locally changed elements it is about to replace, and that list is the point of the check. --no-baseline already exists for "I accept that you cannot tell", which is the case §15 needs. Caching makes the baseline cheap instead of removing the warning. Reported in mxcli-chat FINDINGS §59. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../mendix/download-marketplace-content.md | 10 + cmd/mxcli/cmd_marketplace_diff.go | 31 +- cmd/mxcli/marketplace/refcache.go | 316 ++++++++++++++++++ cmd/mxcli/marketplace/refcache_test.go | 275 +++++++++++++++ cmd/mxcli/marketplace/scratch.go | 179 +++++++++- docs-site/src/guides/marketplace.md | 50 +++ 6 files changed, 848 insertions(+), 13 deletions(-) create mode 100644 cmd/mxcli/marketplace/refcache.go create mode 100644 cmd/mxcli/marketplace/refcache_test.go diff --git a/.claude/skills/mendix/download-marketplace-content.md b/.claude/skills/mendix/download-marketplace-content.md index 0dc5a7a0f..a64ebc666 100644 --- a/.claude/skills/mendix/download-marketplace-content.md +++ b/.claude/skills/mendix/download-marketplace-content.md @@ -261,6 +261,16 @@ Administration — installed 4.3.2 (Mendix 11.12.1) CONFLICT ENTITY Account ``` +**Tell the user the first one is slow.** Answering needs a reference project — +a blank app with the published module imported — and `--to` needs two. Measured +on Administration at 11.12.1: **~50s** the first time, **~13s** afterwards, once +`~/.mxcli/marketplace-refs/` holds the blank app and the built references. Run +`diff` before `update` rather than instead of it: the `update` reuses the base +reference the `diff` just built, so the pair costs little more than the `diff`. + +Set `MXCLI_NO_REF_CACHE=1` if a result looks stale and you want to rule the +cache out — it rebuilds everything without deleting the evidence. + It downloads the installed version's `.mpk`, imports it into a throwaway reference project built **at the project's own Mendix version** (a mismatch is refused, not warned about — Mendix's own conversions would otherwise read as your edits), and compares `DESCRIBE` diff --git a/cmd/mxcli/cmd_marketplace_diff.go b/cmd/mxcli/cmd_marketplace_diff.go index 8578fe215..ff5022aa4 100644 --- a/cmd/mxcli/cmd_marketplace_diff.go +++ b/cmd/mxcli/cmd_marketplace_diff.go @@ -205,20 +205,39 @@ func downloadVersion(ctx context.Context, client *mp.Client, v *mp.Version, work func referenceFor(ctx context.Context, client *mp.Client, v *mp.Version, mendixVersion, work, slot string) (mprPath, pkgModule string, err error) { - mpkPath, err := downloadVersion(ctx, client, v, work, slot) - if err != nil { + refDir := filepath.Join(work, slot+"-ref") + if err := os.MkdirAll(refDir, 0o755); err != nil { + return "", "", err + } + mpkPath := filepath.Join(work, slot+".mpk") + + // A reference is expensive (a download plus two ~10s mx invocations) and + // immutable, so the same published version is never built twice: `diff` + // followed by `update` needs the same base, and so does re-running either. + if cached := marketplace.CachedReference(v.VersionID, mendixVersion, refDir, mpkPath); cached != "" { + if pkgModule, err = marketplace.ModuleNameInPackage(mpkPath); err == nil { + return cached, pkgModule, nil + } + // The package came back unreadable, so the entry is not trustworthy. + // Fall through and rebuild rather than failing the command. + if err := os.RemoveAll(refDir); err == nil { + _ = os.MkdirAll(refDir, 0o755) + } + } + + if _, err := downloadVersion(ctx, client, v, work, slot); err != nil { return "", "", err } pkgModule, err = marketplace.ModuleNameInPackage(mpkPath) if err != nil { return "", "", err } - refDir := filepath.Join(work, slot+"-ref") - if err := os.MkdirAll(refDir, 0o755); err != nil { + mprPath, err = marketplace.PackageProject(ctx, mpkPath, mendixVersion, refDir, newBackendFactory()) + if err != nil { return "", "", err } - mprPath, err = marketplace.PackageProject(ctx, mpkPath, mendixVersion, refDir, newBackendFactory()) - return mprPath, pkgModule, err + marketplace.CacheReference(v.VersionID, mendixVersion, refDir, mpkPath) + return mprPath, pkgModule, nil } // versionIDs projects the marketplace's version list to bare version UUIDs. diff --git a/cmd/mxcli/marketplace/refcache.go b/cmd/mxcli/marketplace/refcache.go new file mode 100644 index 000000000..c5b1099ed --- /dev/null +++ b/cmd/mxcli/marketplace/refcache.go @@ -0,0 +1,316 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "fmt" + "io" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "time" +) + +// Reference projects are expensive and immutable, which is the whole argument +// for caching them. +// +// Building one costs a `mx create-project` (~12s) plus a `mx module-import` +// (~7s) plus the package download (~5s), and `marketplace update` builds TWO — +// the version installed, to answer "has anyone edited this?", and the version +// being moved to. Measured on Administration (21 elements, the smallest module +// in a blank app), `diff` and `update` each took ~67s, almost none of it the +// download. +// +// Two caches, because they miss in different places: +// +// - blankCache, keyed by Mendix version. Every reference build starts from the +// same blank app, so a run updating six modules ran `mx create-project` +// twelve times for twelve identical results. This one hits on the FIRST +// build of every module after the first. +// - refCache, keyed by (marketplace version UUID, Mendix version). The whole +// finished reference. `diff` followed by `update` builds the same base +// reference twice; so does re-running either. This one hits on repeats. +// +// What makes them safe to cache is that a reference is read-only once built: +// SnapshotModule reads it, PerformUpdate copies units out of it, and neither +// shells out to `mx` against it. The cached tree is never handed out directly +// for the same reason a template is not — callers get a copy. + +// refCacheDisabled reports whether the caches are switched off. Set +// MXCLI_NO_REF_CACHE=1 to make every reference build from scratch, which is how +// you bisect a suspected stale-cache problem without deleting anything. +func refCacheDisabled() bool { + v := os.Getenv("MXCLI_NO_REF_CACHE") + return v != "" && v != "0" && !strings.EqualFold(v, "false") +} + +// refCacheRoot returns ~/.mxcli/marketplace-refs, creating nothing. +func refCacheRoot() (string, error) { + home, err := os.UserHomeDir() + if err != nil { + return "", fmt.Errorf("determining home directory: %w", err) + } + return filepath.Join(home, ".mxcli", "marketplace-refs"), nil +} + +// completeMarker names the file that makes an entry usable. An entry is a +// directory tree written by a process that may be killed halfway through, so +// presence of the directory proves nothing — the marker is written last, after +// the atomic rename, and its absence means "rebuild", never "use what is there". +const completeMarker = ".mxcli-complete" + +// blankCacheDir is where a pristine `mx create-project` result for one Mendix +// version lives. +func blankCacheDir(mendixVersion string) (string, error) { + root, err := refCacheRoot() + if err != nil { + return "", err + } + return filepath.Join(root, "blank", safeKey(mendixVersion)), nil +} + +// refCacheDir is where a finished reference project for one published version +// lives. The key is the marketplace version UUID, not the version NUMBER: +// numbers collide across content (a blank 11.12.1 app has Atlas_Web_Content +// 4.1.0, and Administration's content has also published a 4.1.0), so keying on +// the number would serve one module's reference for another's. +// +// The Mendix version is part of the key because a reference built at a +// different version reports Mendix's own conversions as user edits — the same +// reason PackageProject refuses a mismatch outright. +func refCacheDir(versionID, mendixVersion string) (string, error) { + root, err := refCacheRoot() + if err != nil { + return "", err + } + return filepath.Join(root, "ref", safeKey(mendixVersion)+"_"+safeKey(versionID)), nil +} + +// safeKey makes a path component out of a version string or UUID. Anything that +// is not plainly alphanumeric becomes '-', so a malformed version from the API +// cannot walk out of the cache directory. +// +// '.' has to survive (11.12.1 is a version), which is what makes ".." the case +// worth naming: sanitising character by character leaves it untouched, and +// filepath.Join then walks a level up. A key that is nothing but dots is not a +// key. +func safeKey(s string) string { + if s == "" { + return "unknown" + } + var b strings.Builder + for _, r := range s { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '.', r == '-': + b.WriteRune(r) + default: + b.WriteByte('-') + } + } + if strings.Trim(b.String(), ".") == "" { + return "unknown" + } + return b.String() +} + +// defaultRefCacheEntries bounds the finished-reference cache. +// +// An entry is a whole Mendix project: 34 MB measured for Administration, the +// smallest module in a blank app. Updating six modules builds twelve references, +// so an unbounded cache is ~400 MB of a container that may have 2 GB free — and +// running out of disk mid-update is a far worse outcome than rebuilding a +// reference, because `marketplace update` does not roll back. +// +// The blank-project cache is deliberately NOT bounded: it holds one entry per +// Mendix version, and it is the one that pays off on every single build. +const defaultRefCacheEntries = 6 + +// refCacheMaxEntries reads the bound, honouring MXCLI_REF_CACHE_MAX. 0 disables +// pruning for anyone with disk to spare. +func refCacheMaxEntries() int { + if v := os.Getenv("MXCLI_REF_CACHE_MAX"); v != "" { + if n, err := strconv.Atoi(v); err == nil && n >= 0 { + return n + } + } + return defaultRefCacheEntries +} + +// pruneRefCache keeps the newest max entries and removes the rest. +// +// Recency is taken from the entry's marker file, which is written when the entry +// is published and touched when it is served, so "newest" means "most recently +// useful" rather than "most recently built". Ties are broken by name so the +// result does not depend on directory order. +func pruneRefCache(max int) { + if max <= 0 { + return + } + root, err := refCacheRoot() + if err != nil { + return + } + refRoot := filepath.Join(root, "ref") + entries, err := os.ReadDir(refRoot) + if err != nil { + return + } + + type aged struct { + name string + mod time.Time + } + var complete []aged + for _, e := range entries { + if !e.IsDir() { + continue + } + dir := filepath.Join(refRoot, e.Name()) + info, serr := os.Stat(filepath.Join(dir, completeMarker)) + if serr != nil { + // An incomplete entry is never served, so it is pure waste: a build that + // was killed, or a staging directory whose publish failed. Remove it + // regardless of the bound. + if strings.HasPrefix(e.Name(), "building-") { + _ = os.RemoveAll(dir) + } + continue + } + complete = append(complete, aged{e.Name(), info.ModTime()}) + } + if len(complete) <= max { + return + } + sort.Slice(complete, func(i, j int) bool { + if complete[i].mod.Equal(complete[j].mod) { + return complete[i].name < complete[j].name + } + return complete[i].mod.After(complete[j].mod) + }) + for _, e := range complete[max:] { + _ = os.RemoveAll(filepath.Join(refRoot, e.name)) + } +} + +// touchEntry records that an entry was used, so pruning evicts what nobody is +// asking for rather than what was simply built first. +func touchEntry(dir string) { + now := time.Now() + _ = os.Chtimes(filepath.Join(dir, completeMarker), now, now) +} + +// cacheReady reports whether dir holds a complete cache entry. +func cacheReady(dir string) bool { + if refCacheDisabled() { + return false + } + _, err := os.Stat(filepath.Join(dir, completeMarker)) + return err == nil +} + +// publishToCache moves a freshly built tree into the cache and marks it +// complete. +// +// The build happens elsewhere and is renamed in, so a killed process leaves a +// stray temp directory rather than a half-populated entry that the next run +// would trust. A rename across filesystems is not possible, so the caller must +// build under the same root; buildDir is removed on success either way. +// +// Losing a race is not an error. Two mxcli processes may build the same +// reference at once; whoever renames second wins and the other's work is +// discarded, which costs time and never correctness. +func publishToCache(buildDir, cacheDir string) error { + if refCacheDisabled() { + return nil + } + if err := os.MkdirAll(filepath.Dir(cacheDir), 0o755); err != nil { + return err + } + // A previous complete entry is replaced rather than merged: a merge would mix + // two builds' files, and the entry is cheap to rebuild. + staging := cacheDir + ".old" + _ = os.RemoveAll(staging) + if _, err := os.Stat(cacheDir); err == nil { + _ = os.Rename(cacheDir, staging) + } + if err := os.Rename(buildDir, cacheDir); err != nil { + // Put back whatever was there; better a stale-but-complete entry than none. + if _, serr := os.Stat(staging); serr == nil { + _ = os.Rename(staging, cacheDir) + } + return fmt.Errorf("publish cache entry: %w", err) + } + _ = os.RemoveAll(staging) + + f, err := os.Create(filepath.Join(cacheDir, completeMarker)) + if err != nil { + // Without the marker the entry is simply never used, so this is not fatal. + return nil + } + _ = f.Close() + return nil +} + +// copyTree copies a directory tree. Used both to seed a build from the cache and +// to hand a caller its own copy, so the cached tree is never the one written to. +// +// Symlinks are copied as symlinks; a Mendix project has none, and following them +// would let a crafted package escape the destination. +func copyTree(src, dst string) error { + return filepath.Walk(src, func(path string, info os.FileInfo, err error) error { + if err != nil { + return err + } + rel, err := filepath.Rel(src, path) + if err != nil { + return err + } + if rel == "." { + return os.MkdirAll(dst, 0o755) + } + // The marker is cache bookkeeping and has no business in a project tree. + if rel == completeMarker { + return nil + } + target := filepath.Join(dst, rel) + + switch { + case info.IsDir(): + return os.MkdirAll(target, info.Mode().Perm()|0o700) + case info.Mode()&os.ModeSymlink != 0: + link, lerr := os.Readlink(path) + if lerr != nil { + return lerr + } + return os.Symlink(link, target) + case !info.Mode().IsRegular(): + // Sockets, devices and friends are not part of a project; skipping them + // is better than failing the whole copy over one. + return nil + } + return copyFile(path, target, info.Mode().Perm()) + }) +} + +func copyFile(src, dst string, perm os.FileMode) error { + in, err := os.Open(src) + if err != nil { + return err + } + defer in.Close() + + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + return err + } + out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm) + if err != nil { + return err + } + if _, err := io.Copy(out, in); err != nil { + out.Close() + return err + } + return out.Close() +} diff --git a/cmd/mxcli/marketplace/refcache_test.go b/cmd/mxcli/marketplace/refcache_test.go new file mode 100644 index 000000000..2111c4252 --- /dev/null +++ b/cmd/mxcli/marketplace/refcache_test.go @@ -0,0 +1,275 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// TestCacheReadyNeedsTheMarker guards the invariant the whole cache rests on: a +// directory is not an entry. A process killed while writing leaves a populated +// tree, and serving that would surface as invented diff findings — a wrong +// answer, not a slow one. +func TestCacheReadyNeedsTheMarker(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "PackageRef.mpr"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + + if cacheReady(dir) { + t.Error("a directory with content but no marker was treated as a complete entry") + } + if err := os.WriteFile(filepath.Join(dir, completeMarker), nil, 0o644); err != nil { + t.Fatal(err) + } + if !cacheReady(dir) { + t.Error("a marked entry was not treated as complete") + } +} + +// TestCacheDisabledIsNeverReady checks the bypass actually bypasses. Without it, +// "is this a stale-cache problem?" can only be answered by deleting the cache, +// which destroys the evidence. +func TestCacheDisabledIsNeverReady(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, completeMarker), nil, 0o644); err != nil { + t.Fatal(err) + } + if !cacheReady(dir) { + t.Fatal("precondition: entry should be ready before disabling") + } + + t.Setenv("MXCLI_NO_REF_CACHE", "1") + if cacheReady(dir) { + t.Error("MXCLI_NO_REF_CACHE=1 did not disable the cache") + } + // "0" and "false" are the off switches for the off switch; a user who exports + // MXCLI_NO_REF_CACHE=0 means "leave it on". + for _, off := range []string{"0", "false", "FALSE"} { + t.Setenv("MXCLI_NO_REF_CACHE", off) + if !cacheReady(dir) { + t.Errorf("MXCLI_NO_REF_CACHE=%q disabled the cache; it should not", off) + } + } +} + +// TestSafeKeyCannotEscapeTheCache checks that a version string from the API +// cannot become a path. Nothing today produces one, which is exactly why it +// would go unnoticed. +func TestSafeKeyCannotEscapeTheCache(t *testing.T) { + for _, in := range []string{"../../etc/passwd", "a/b", `c\d`, "..", "x\x00y"} { + got := safeKey(in) + if strings.ContainsAny(got, `/\`) || got == ".." || strings.Contains(got, "\x00") { + t.Errorf("safeKey(%q) = %q, which is still a path", in, got) + } + } + if safeKey("") != "unknown" { + t.Errorf("safeKey(\"\") = %q, want %q", safeKey(""), "unknown") + } + // Ordinary keys must survive intact, or every lookup misses and the cache is + // silently useless. + if got := safeKey("11.12.1"); got != "11.12.1" { + t.Errorf("safeKey(%q) = %q, want it unchanged", "11.12.1", got) + } + if got := safeKey("2059615c-c6f1-4103-aedb-14820c077a1c"); got != "2059615c-c6f1-4103-aedb-14820c077a1c" { + t.Errorf("a version UUID was mangled: %q", got) + } +} + +// TestRefCacheKeyIncludesMendixVersion is the correctness guard, not a +// housekeeping one. A reference built at a different Mendix version reports the +// platform's own conversions as user edits, so serving one project's entry to +// another version would produce confident, wrong findings. +func TestRefCacheKeyIncludesMendixVersion(t *testing.T) { + const versionID = "2059615c-c6f1-4103-aedb-14820c077a1c" + + a, err := refCacheDir(versionID, "11.12.1") + if err != nil { + t.Fatal(err) + } + b, err := refCacheDir(versionID, "11.13.0") + if err != nil { + t.Fatal(err) + } + if a == b { + t.Errorf("the same published version shares a cache entry across Mendix versions: %s", a) + } + + // And two different published versions must not collide at one Mendix version. + c, err := refCacheDir("11111111-2222-3333-4444-555555555555", "11.12.1") + if err != nil { + t.Fatal(err) + } + if a == c { + t.Errorf("two published versions share a cache entry: %s", a) + } +} + +// TestCopyTreeRoundTrip covers what a served entry depends on: nested files +// arrive with their contents, and the cache's own bookkeeping does not leak into +// the project tree handed to mx. +func TestCopyTreeRoundTrip(t *testing.T) { + src, dst := t.TempDir(), filepath.Join(t.TempDir(), "out") + + mustWrite(t, filepath.Join(src, "PackageRef.mpr"), "mpr") + mustWrite(t, filepath.Join(src, "mprcontents", "unit.mxunit"), "unit") + mustWrite(t, filepath.Join(src, completeMarker), "") + + if err := copyTree(src, dst); err != nil { + t.Fatalf("copyTree: %v", err) + } + if got := readFile(t, filepath.Join(dst, "PackageRef.mpr")); got != "mpr" { + t.Errorf("top-level file = %q, want %q", got, "mpr") + } + if got := readFile(t, filepath.Join(dst, "mprcontents", "unit.mxunit")); got != "unit" { + t.Errorf("nested file = %q, want %q", got, "unit") + } + if _, err := os.Stat(filepath.Join(dst, completeMarker)); !os.IsNotExist(err) { + t.Error("the cache marker was copied into the project tree") + } +} + +// TestPublishToCacheReplaces checks that a rebuilt entry replaces the old one +// rather than merging into it. A merge would mix two builds' files, and the +// result would be a project that never existed. +func TestPublishToCacheReplaces(t *testing.T) { + root := t.TempDir() + cacheDir := filepath.Join(root, "entry") + + first := filepath.Join(root, "build-1") + mustWrite(t, filepath.Join(first, "keep.txt"), "old") + mustWrite(t, filepath.Join(first, "only-in-old.txt"), "gone") + if err := publishToCache(first, cacheDir); err != nil { + t.Fatalf("first publish: %v", err) + } + if !cacheReady(cacheDir) { + t.Fatal("entry is not ready after publishing") + } + + second := filepath.Join(root, "build-2") + mustWrite(t, filepath.Join(second, "keep.txt"), "new") + if err := publishToCache(second, cacheDir); err != nil { + t.Fatalf("second publish: %v", err) + } + + if got := readFile(t, filepath.Join(cacheDir, "keep.txt")); got != "new" { + t.Errorf("file content = %q, want the rebuilt %q", got, "new") + } + if _, err := os.Stat(filepath.Join(cacheDir, "only-in-old.txt")); !os.IsNotExist(err) { + t.Error("a file from the previous entry survived the replacement") + } + if !cacheReady(cacheDir) { + t.Error("the replaced entry lost its marker") + } +} + +// TestCachedReferenceMissIsEmpty covers the paths that must degrade to "build +// it", not to an error: no entry, and no version ID to key on. +func TestCachedReferenceMissIsEmpty(t *testing.T) { + dest, mpk := t.TempDir(), filepath.Join(t.TempDir(), "pkg.mpk") + + if got := CachedReference("", "11.12.1", dest, mpk); got != "" { + t.Errorf("an empty version ID returned %q, want a miss", got) + } + if got := CachedReference("no-such-version-id", "11.12.1", dest, mpk); got != "" { + t.Errorf("an absent entry returned %q, want a miss", got) + } +} + +func mustWrite(t *testing.T, path, content string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } +} + +func readFile(t *testing.T, path string) string { + t.Helper() + b, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + return string(b) +} + +// TestPruneKeepsNewestAndBoundOfOne covers the bound that stops the cache +// filling a small container's disk. A 34 MB entry per reference and twelve +// references in a provisioning run is ~400 MB, and running out of disk part way +// through an update is worse than a slow update: `marketplace update` does not +// roll back. +// +// The bound-of-1 case is the one worth asserting: pruning happens after +// publishing, so an entry must survive its own prune, or the cache would never +// hold anything and every run would silently rebuild. +func TestPruneKeepsNewestAndBoundOfOne(t *testing.T) { + root := t.TempDir() + t.Setenv("HOME", root) + refRoot := filepath.Join(root, ".mxcli", "marketplace-refs", "ref") + + // Three entries, oldest first, with distinct marker times. + names := []string{"oldest", "middle", "newest"} + for i, n := range names { + dir := filepath.Join(refRoot, n) + mustWrite(t, filepath.Join(dir, "project", "PackageRef.mpr"), n) + mustWrite(t, filepath.Join(dir, completeMarker), "") + when := time.Now().Add(time.Duration(i-len(names)) * time.Hour) + if err := os.Chtimes(filepath.Join(dir, completeMarker), when, when); err != nil { + t.Fatal(err) + } + } + // An abandoned staging directory: never servable, so it goes regardless. + mustWrite(t, filepath.Join(refRoot, "building-123", "junk"), "x") + + pruneRefCache(2) + + for _, keep := range []string{"newest", "middle"} { + if _, err := os.Stat(filepath.Join(refRoot, keep)); err != nil { + t.Errorf("%s was evicted; the newest entries must survive", keep) + } + } + if _, err := os.Stat(filepath.Join(refRoot, "oldest")); !os.IsNotExist(err) { + t.Error("oldest survived a bound of 2") + } + if _, err := os.Stat(filepath.Join(refRoot, "building-123")); !os.IsNotExist(err) { + t.Error("an abandoned staging directory was left behind") + } + + pruneRefCache(1) + if _, err := os.Stat(filepath.Join(refRoot, "newest")); err != nil { + t.Error("a bound of 1 evicted the newest entry, so nothing would ever be cached") + } +} + +// TestPruneDisabledByZero checks the escape hatch for anyone with disk to spare. +func TestPruneDisabledByZero(t *testing.T) { + root := t.TempDir() + t.Setenv("HOME", root) + refRoot := filepath.Join(root, ".mxcli", "marketplace-refs", "ref") + for _, n := range []string{"a", "b", "c"} { + mustWrite(t, filepath.Join(refRoot, n, completeMarker), "") + } + + pruneRefCache(0) + + for _, n := range []string{"a", "b", "c"} { + if _, err := os.Stat(filepath.Join(refRoot, n)); err != nil { + t.Errorf("entry %s was evicted with pruning disabled", n) + } + } + + t.Setenv("MXCLI_REF_CACHE_MAX", "0") + if got := refCacheMaxEntries(); got != 0 { + t.Errorf("MXCLI_REF_CACHE_MAX=0 gave a bound of %d, want 0 (disabled)", got) + } + t.Setenv("MXCLI_REF_CACHE_MAX", "not-a-number") + if got := refCacheMaxEntries(); got != defaultRefCacheEntries { + t.Errorf("a malformed bound gave %d, want the default %d", got, defaultRefCacheEntries) + } +} diff --git a/cmd/mxcli/marketplace/scratch.go b/cmd/mxcli/marketplace/scratch.go index e285b51b5..5d9677e7c 100644 --- a/cmd/mxcli/marketplace/scratch.go +++ b/cmd/mxcli/marketplace/scratch.go @@ -142,15 +142,11 @@ func PackageProject(ctx context.Context, mpkPath, mendixVersion, workDir string, "hint: run 'mxcli setup mxbuild --version %s'", mendixVersion, err, mendixVersion) } - const appName = "PackageRef" - create := exec.CommandContext(ctx, mxPath, "create-project", "--app-name", appName) - create.Dir = workDir - docker.PrepareMxCommand(create) - if out, err := create.CombinedOutput(); err != nil { - return "", fmt.Errorf("mx create-project failed: %w\n%s", err, strings.TrimSpace(string(out))) + if err := blankProjectInto(ctx, mxPath, mendixVersion, workDir); err != nil { + return "", err } - mprPath, err := findScratchMpr(workDir, appName) + mprPath, err := findScratchMpr(workDir, blankAppName) if err != nil { return "", err } @@ -207,6 +203,175 @@ func PackageProject(ctx context.Context, mpkPath, mendixVersion, workDir string, return mprPath, nil } +// blankAppName is what every reference project is created as. It is fixed +// rather than per-run so a cached blank tree can be reused verbatim. +const blankAppName = "PackageRef" + +// blankProjectInto puts a pristine blank Mendix project into workDir, from the +// cache when one is there and from `mx create-project` when not. +// +// `mx create-project` is the single most expensive step in building a reference +// (~12s of ~25s) and its result depends on nothing but the Mendix version, so a +// run updating six modules paid for twelve identical blank apps. The version +// stamp is verified BEFORE anything is cached, so a cache entry can never carry +// the wrong version — which matters more than the time, because comparing across +// versions reports Mendix's own conversions as user edits. +func blankProjectInto(ctx context.Context, mxPath, mendixVersion, workDir string) error { + cacheDir, cerr := blankCacheDir(mendixVersion) + if cerr == nil && cacheReady(cacheDir) { + if err := copyTree(cacheDir, workDir); err == nil { + return nil + } + // A cache entry that cannot be copied is not worth diagnosing: drop it and + // build. Serving a partial project would surface as bogus diff findings. + _ = os.RemoveAll(cacheDir) + } + + create := exec.CommandContext(ctx, mxPath, "create-project", "--app-name", blankAppName) + create.Dir = workDir + docker.PrepareMxCommand(create) + if out, err := create.CombinedOutput(); err != nil { + return fmt.Errorf("mx create-project failed: %w\n%s", err, strings.TrimSpace(string(out))) + } + if cerr != nil { + return nil + } + + // Verify before caching, not after serving. A blank app stamped with the + // wrong version (ResolveMxForVersion falls back to any cached mxbuild) must + // not become the entry every later run trusts. + mprPath, err := findScratchMpr(workDir, blankAppName) + if err != nil { + return err + } + if err := verifyProjectVersion(mprPath, mendixVersion); err != nil { + return err + } + cacheBlankProject(workDir, cacheDir) + return nil +} + +// cacheBlankProject copies a just-built blank project into the cache. Failure is +// silent by design: the caller already has a working project in workDir, and a +// cache that cannot be written is a slow tool, not a broken one. +// +// It copies rather than moves because workDir is what the caller goes on to use. +// The copy lands beside the cache entry and is renamed in, so a process killed +// mid-copy leaves a stray directory rather than a half-entry. +func cacheBlankProject(workDir, cacheDir string) { + if refCacheDisabled() { + return + } + staging, err := os.MkdirTemp(filepath.Dir(cacheDir), "building-") + if err != nil { + if err := os.MkdirAll(filepath.Dir(cacheDir), 0o755); err != nil { + return + } + if staging, err = os.MkdirTemp(filepath.Dir(cacheDir), "building-"); err != nil { + return + } + } + if err := copyTree(workDir, staging); err != nil { + _ = os.RemoveAll(staging) + return + } + if err := publishToCache(staging, cacheDir); err != nil { + _ = os.RemoveAll(staging) + } +} + +// A cache entry holds both halves of a reference, because both are needed and +// re-downloading one to serve the other would give back most of the saving: +// +// /.mxcli-complete written last; its absence means "rebuild" +// /package.mpk the published package +// /project/ the reference project built from it +// +// The package is kept because `marketplace update` takes the module's bundled +// widgets from the .mpk rather than from the reference project, whose widgets/ +// also holds the blank template's. +const ( + entryProject = "project" + entryPackage = "package.mpk" +) + +// CachedReference restores a previously built reference for this published +// version: the project tree into destDir, the package to mpkDest. It returns the +// path to the reference .mpr, or "" when there is no usable entry. +// +// Callers get a copy. PerformUpdate and SnapshotModule only read, but handing +// out the cached tree itself would let one careless writer poison every later +// run — the cost of being wrong here is silent, wrong diff findings. +func CachedReference(versionID, mendixVersion, destDir, mpkDest string) string { + if versionID == "" { + return "" + } + cacheDir, err := refCacheDir(versionID, mendixVersion) + if err != nil || !cacheReady(cacheDir) { + return "" + } + drop := func() string { + // An entry that cannot be served is dropped rather than diagnosed: it will + // be rebuilt on this very run, and a partial reference reads as local edits. + _ = os.RemoveAll(cacheDir) + return "" + } + + if err := copyTree(filepath.Join(cacheDir, entryProject), destDir); err != nil { + return drop() + } + if err := copyFile(filepath.Join(cacheDir, entryPackage), mpkDest, 0o644); err != nil { + return drop() + } + mprPath, err := findScratchMpr(destDir, blankAppName) + if err != nil { + return drop() + } + // The stamp is re-checked on the way out, not only on the way in. An entry + // written by an older mxcli, or before this guard was tightened, is dropped + // rather than trusted — the comparison's whole validity rests on it. + if err := verifyProjectVersion(mprPath, mendixVersion); err != nil { + return drop() + } + touchEntry(cacheDir) + return mprPath +} + +// CacheReference stores a finished reference so the next `diff` or `update` of +// the same published version skips the download and both mx invocations. +// +// Failures are silent for the same reason as cacheBlankProject: the caller +// already has what it needs, and a cache that cannot be written makes the tool +// slow, not wrong. +func CacheReference(versionID, mendixVersion, refDir, mpkPath string) { + if versionID == "" || refCacheDisabled() { + return + } + cacheDir, err := refCacheDir(versionID, mendixVersion) + if err != nil || os.MkdirAll(filepath.Dir(cacheDir), 0o755) != nil { + return + } + staging, err := os.MkdirTemp(filepath.Dir(cacheDir), "building-") + if err != nil { + return + } + if err := copyTree(refDir, filepath.Join(staging, entryProject)); err != nil { + _ = os.RemoveAll(staging) + return + } + if err := copyFile(mpkPath, filepath.Join(staging, entryPackage), 0o644); err != nil { + _ = os.RemoveAll(staging) + return + } + if err := publishToCache(staging, cacheDir); err != nil { + _ = os.RemoveAll(staging) + return + } + // Prune after publishing, not before: the entry just written is the newest, + // so it survives its own prune, and a bound of 1 still works. + pruneRefCache(refCacheMaxEntries()) +} + // findScratchMpr locates the project mx just created. The name follows // --app-name, but that is not contractual, so fall back to whatever .mpr exists. func findScratchMpr(workDir, appName string) (string, error) { diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index 1f86c1e75..4f30c79a5 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -195,6 +195,56 @@ Without this, module install order silently decided which widget versions the pr `diff` and `update` both download the installed version to establish the local-edit baseline, so both fail when it has been unpublished — as NanoflowCommons 6.0.0 has, while a blank 11.13 app still ships it. `--force` does not help: it overrides a finding, and there is no finding to override. `mxcli marketplace update … --no-baseline` accepts that the question cannot be answered and updates anyway, discarding any local edits to that module without naming them. +## Why `diff` and `update` are slow, and what is cached + +Both commands answer their question by building a **reference project**: a blank +Mendix app of the consuming project's version with the published module imported +into it, so the comparison runs against a real project rather than against a +`.mpk`. That costs a download plus two `mx` invocations, and `update` builds +two references — the installed version, to answer "has anyone edited this?", and +the version being moved to. + +Two caches under `~/.mxcli/marketplace-refs/` keep that off the clock: + +| Cache | Keyed by | Saves | +|---|---|---| +| `blank/` | Mendix version | the `mx create-project` in **every** reference build | +| `ref/` | published version UUID + Mendix version | the whole reference, on a repeat | + +Measured on Administration (content 23513, 4.3.2 → 4.5.0, Mendix 11.12.1): + +```text +mxcli marketplace diff 23513 -p app.mpr --to 4.5.0 + + no cache 66s + cold cache 49s (blank app built once, reused by the second reference) + warm 13s +``` + +The Mendix version is part of both keys, because a reference built at a +different version reports Mendix's own conversions as local edits. An entry is +only used once its completion marker is present and the project's version stamp +has been re-checked on the way out, so a half-written or mislabelled entry is +rebuilt rather than trusted. + +A reference is about 34 MB, so `ref/` keeps the 6 most recently used entries and +evicts the rest — running out of disk part way through an update is worse than +rebuilding one, because `update` does not roll back. `blank/` is not bounded: it +holds one entry per Mendix version. + +```bash +MXCLI_REF_CACHE_MAX=20 mxcli marketplace diff … # keep more (0 = keep everything) +MXCLI_NO_REF_CACHE=1 mxcli marketplace diff … # build everything from scratch +``` + +Reach for `MXCLI_NO_REF_CACHE=1` before deleting anything: it answers "is this a +stale-cache problem?" while leaving the evidence in place. + +Note that `--force` still builds the baseline. It is not wasted work — under +`--force` the command prints the locally changed elements it is about to +replace, and that list is the whole point of the check. Use `--no-baseline` when +you genuinely want to skip it. + ## Repairing the model (`mxcli fix`) `mx update-widgets` and `mx rename-design-properties` each fix something only Mendix can fix, and each rewrites an MPR v2 project into the single-file v1 format while doing it. Measured on 11.12.1: `update-widgets` took 369 `.mxunit` files to 0 and a 69,632-byte index to 14,405,632 bytes; `rename-design-properties` took 1,865 files to 0 and a 249,856-byte index to 39,895,040 bytes, having renamed 149 design properties across 41 documents. The conversion is one-way. From 8071df6d17f1ec5ed057b041cca2bcc096897278 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 13 Aug 2026 16:29:35 +0000 Subject: [PATCH 20/20] fix(mappings): only refuse an unknown member when a schema says so MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The refusal added for #882 fired whenever a member failed to resolve, including when there was nothing to resolve against. `create import mapping X { ... }` with no `with json structure` clause is legal MDL, and an XML-schema or message-definition mapping resolves no JSON elements either — so every schema-less mapping was rejected: "id" is not a member of the JSON structure at (Object), which has no members there That broke eight round-trip integration tests, which unit tests did not cover because they all build against a populated index. The refusal now applies only where a schema exists to contradict the name. With no schema loaded, the authored name is taken at face value and becomes both the exposed name and the path segment — the pre-#882 behaviour, unchanged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013uQvFDd5R4eNqqita59jM8 --- mdl/executor/cmd_export_mappings.go | 2 +- mdl/executor/cmd_import_mappings.go | 12 +++++- .../cmd_mappings_member_resolution_test.go | 42 +++++++++++++++++++ 3 files changed, 54 insertions(+), 2 deletions(-) diff --git a/mdl/executor/cmd_export_mappings.go b/mdl/executor/cmd_export_mappings.go index 04be8c5a2..477660464 100644 --- a/mdl/executor/cmd_export_mappings.go +++ b/mdl/executor/cmd_export_mappings.go @@ -271,7 +271,7 @@ func buildExportMappingElementModel(moduleName string, def *ast.ExportMappingEle jsElem = idx.resolve(parentPath, def.JsonName) } - if jsElem == nil && !isRoot { + if jsElem == nil && !isRoot && idx.resolvable() { known := idx.memberNames(parentPath) if len(known) == 0 { return nil, fmt.Errorf("%q is not a member of the JSON structure at %s, which has no members there", diff --git a/mdl/executor/cmd_import_mappings.go b/mdl/executor/cmd_import_mappings.go index b966e59ab..468a97930 100644 --- a/mdl/executor/cmd_import_mappings.go +++ b/mdl/executor/cmd_import_mappings.go @@ -287,7 +287,7 @@ func buildImportMappingElementModel(moduleName string, def *ast.ImportMappingEle // resolves to nothing is REFUSED, never given a made-up path: the fabricated // path passed `mxcli check` and surfaced only later — in mxbuild as CE5015, // or at runtime as an unresolvable mapping. (#882) - if jsElem == nil && !isRoot { + if jsElem == nil && !isRoot && idx.resolvable() { known := idx.memberNames(parentPath) if len(known) == 0 { return nil, fmt.Errorf("%q is not a member of the JSON structure at %s, which has no members there", @@ -476,6 +476,16 @@ func (i *jsonSchemaIndex) resolve(parentPath, name string) *types.JsonElement { return nil } +// resolvable reports whether a JSON structure was actually loaded. +// +// `create import mapping X { ... }` with no `with json structure` clause is +// legal MDL, and an XML-schema or message-definition mapping resolves no JSON +// elements either. There is nothing to validate a member against in those cases, +// so names must be taken at face value — refusing them broke eight round-trip +// tests that create schema-less mappings. The refusal only applies where a +// schema exists to contradict the name. (issue #882) +func (i *jsonSchemaIndex) resolvable() bool { return len(i.byPath) > 0 } + // memberNames lists the spellings that would have resolved under parentPath, so a // rejection can name them instead of leaving the author to guess. func (i *jsonSchemaIndex) memberNames(parentPath string) []string { diff --git a/mdl/executor/cmd_mappings_member_resolution_test.go b/mdl/executor/cmd_mappings_member_resolution_test.go index 864180fc7..26fed211a 100644 --- a/mdl/executor/cmd_mappings_member_resolution_test.go +++ b/mdl/executor/cmd_mappings_member_resolution_test.go @@ -217,3 +217,45 @@ func TestImportMappingDoesNotCloneTheSnippetSampleValue(t *testing.T) { t.Errorf("JsonPath = %q, want (Object)|total", got) } } + +// `create import mapping X { ... }` with no `with json structure` clause is legal +// MDL, and an XML-schema or message-definition mapping resolves no JSON elements +// either. There is nothing to contradict a member name in those cases, so it +// must be taken at face value. +// +// The first cut of the refusal did not check this and broke eight schema-less +// round-trip tests — the refusal has to be scoped to "a schema exists and does +// not contain this name", not "I could not resolve this name". +func TestMappingsWithoutASchemaAcceptAnyMemberName(t *testing.T) { + empty := newJSONSchemaIndex(nil) + if empty.resolvable() { + t.Fatal("an index built from no elements must not be treated as a schema to validate against") + } + + root, err := buildImportMappingElementModel("B", &ast.ImportMappingElementDef{ + Entity: "B.Pet", + Children: []*ast.ImportMappingElementDef{ + {Attribute: "PetId", JsonName: "id", IsKey: true}, + {Attribute: "Name", JsonName: "name"}, + }, + }, "", "(Object)", nil, empty, true) + if err != nil { + t.Fatalf("a schema-less mapping must build, got: %v", err) + } + if len(root.Children) != 2 { + t.Fatalf("expected 2 children, got %d", len(root.Children)) + } + // With nothing to clone from, the authored name is both the exposed name and + // the path segment — the pre-#882 behaviour, unchanged. + if got := root.Children[0].JsonPath; got != "(Object)|id" { + t.Errorf("JsonPath = %q, want (Object)|id", got) + } + if got := root.Children[0].ExposedName; got != "id" { + t.Errorf("ExposedName = %q, want id", got) + } + + // A loaded schema is still validated. + if !jsonStructureFixture().resolvable() { + t.Error("an index built from real elements must be treated as a schema") + } +}