Skip to content

.test.mdl parser: (1) a leading javadoc-style comment fuses into the first @test block and silently drops it; (2) a bare retrieve/count() test block with no RETURN reports PASS unconditionally regardless of the actual value #927

Description

@TOFBuck

Summary

Found live while writing tests/catalogue-seed.test.mdl for a phase-1 catalogue seed plan. Two
separate defects in mxcli test's .test.mdl parsing/execution, both confirmed with a minimal
repro against a genuinely empty database table.

mxcli v0.18.0 (2026-08-14T20:29:13Z), darwin/arm64 host, devcontainer (linux/arm64), Mendix
11.12.2 project, PostgreSQL 15.19.

Bug 1 — a leading standalone comment block fuses into the first @test block

A .test.mdl file with a file-level javadoc-style comment (no statement of its own) placed
before the first real /** @test ... */ block, with no / separator between them, causes the
first test to vanish from the results entirely — every following test's reported number then
shifts down by one, silently.

Repro

/**
 * File-level description with no statement of its own.
 */

/**
 * @test exactly 5 brands are seeded
 * @expect count($Brands) = 5
 * @cleanup none
 */
retrieve $Brands from eShop.CatalogBrand;
/

/**
 * @test exactly 4 types are seeded
 * @expect count($Types) = 4
 * @cleanup none
 */
retrieve $Types from eShop.CatalogType;
/
$ mxcli test tests/repro.test.mdl -p app.mpr --list
Found 1 test(s):
  test_1: exactly 4 types are seeded

"exactly 5 brands are seeded" never appears — not as a pass, not as a fail, not as an error. The
second block silently becomes test_1.

Workaround used: switch the file-level header to -- line comments instead of a /** */
block. That alone was not sufficient — see Bug 1b.

Bug 1b — the same fusion re-triggers if a -- comment's text spells out both javadoc

tokens

Even after switching to -- line comments, the same fusion recurred when one line of that
---commented header described the bug using the literal token sequences that open and close
a javadoc comment (spelled out back-to-back in the comment's prose, e.g. documenting "a leading
[open-token] [close-token] docblock..."). Removing the literal token spelling from the comment
text (describing it in words instead) fixed it. This suggests the tokenizer scans for those two
token sequences by raw substring match, not scoped to actual comment-open state, so mentioning the
sequence inside an unrelated line comment still triggers block-fusion.

Bug 2 — a bare retrieve + count() test block with no RETURN reports PASS regardless of

the actual value

A test block consisting of only a retrieve statement (no CALL MICROFLOW, no explicit
RETURN), asserting @expect count($Var) = N, was found to report PASS unconditionally, even
when the real count is provably different from N and even when N is deliberately absurd.

Repro

Confirmed against a table truncated to 0 rows immediately beforehand (verified via a direct
psql count against the same database mxcli test --local used):

/**
 * @test bogus brand count that must fail on empty db
 * @expect count($Brands) = 999
 * @cleanup none
 */
retrieve $Brands from eShop.CatalogBrand;
/
$ mxcli test tests/probe.test.mdl -p app.mpr --local
Test Results: probe
============================================================
  PASS  bogus brand count that must fail on empty db (4ms)
------------------------------------------------------------
Total: 1  Passed: 1  Failed: 0  Skipped: 0  Time: 4ms
All tests passed.

count($Brands) = 999 against a table with 0 rows should fail every time. It never does, for any
@expect count($Var) = N test block shaped this way (confirmed across several N values in the
same session).

By contrast, a test block that calls a real microflow with an explicit RETURN and asserts on its
return value evaluates correctly in both directions (confirmed PASS and FAIL cases, both matching
independently-verified psql ground truth):

/**
 * @test exactly 5 brands are seeded
 * @expect $BrandCount = 5
 * @cleanup none
 */
$BrandCount = call microflow eShop.QRY_CatalogBrandCount();
/

Workaround used: route every count/aggregate assertion through a small helper microflow with
an explicit RETURN, then assert on the call's own output variable rather than on a bare
retrieve + inline count() in the test block itself. This matches the guidance already present
in test-microflows.md's "If count() inside a test block will not compile, add a small helper
microflow" — except here the direct form does not fail to compile, it silently reports a false
pass, which is a more dangerous failure mode than a compile error because nothing in the console
output flags it.

Impact

Both bugs produce false positives — a test suite that looks fully green while asserting
nothing real for one or more of its blocks. Bug 1 additionally makes a test vanish with no error
of any kind, so a typo'd or removed assertion is indistinguishable from "everything passed."

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions