[Feature]: Replace hand-written workflow expression parser with a Lark LALR grammar #5060

Problem Statement

The workflow expression evaluator (expressions.py) is a hand-written parser. Of the 22 fixes it received between April 2026 and the package split, 11 were structural parser bugs (quoting, whitespace, parentheses, precedence, list literals, filter chains, trailing tokens, multi-argument calls). Its hardest function scores 70 cognitive / 38 cyclomatic (rank E). Some invalid input is silently accepted ([1,2,3][1] returns None; an unclosed quote in a condition evaluates to False).

Background, prototypes and measurements: #4917

Proposed Solution

Replace the hand-written parser with a Lark LALR grammar, without changing the workflow expression language. Only the parser is replaced. The template scanner, namespace resolution, filters, operator semantics and diagnostics stay in our code.

Prototype: chore/workflow-expression-lark-a2 (diff against base).

Delivery as three PRs, in this order:

1. Package split: expressions.py becomes a package and engine.py is the single place where the engine is selected. No behavior change.
2. Gap tests: tests for behavior that is implemented but never asserted (>=/<=, literals, map('a.b'), item/fan_in, …). They pass on main.
3. Lark engine: the LALR grammar, the lark dependency and the 10 documented test changes.

Results against the 96-case behavior corpus and the full suite:

Alternatives Considered

Each was prototyped on its own branch against the same base and corpus (details in the discussion):

• Lark Earley: 94/96, 26 suite failures, 25–30× slower. It missed the precedence guards. Superseded by LALR.
• pyparsing: 94/96, 25 suite failures, 9–10× slower, larger grammar. It is the fallback if a Lark dependency is not acceptable.
• Jinja2: 89/96, 48 suite failures, 5 valid expressions change result. steps.run-tests.output parses as a subtraction, and '10' > '9' compares as text. These cannot be fixed through an extension point.
• python-liquid: 69/96, 93 suite failures, 27 valid expressions change result. Different filter syntax, no in, no list literals.
• simpleeval, cel-python: both fail on hyphenated paths and our filter syntax. They would mean moving to a new language.

Component

Other

Use Cases

1. Maintainers get a grammar that rules out the structural parser bug class by construction.
2. Parse errors include the column, and condition diagnostics use it.
3. Workflow authors keep the existing expression language; nothing in the shipped workflows changes.

Acceptance Criteria

Maintainer decisions needed before work starts:

• Approve a new runtime dependency: lark 1.3.1.
• Accept the 10 test changes. Invalid input is rejected with ValueError instead of silently passing, and some error messages come from the parser (English, with the column).
• Decide whether to keep the precedence guards (a filter after a comparison or after not is a parse error). The prototype keeps them; dropping them would mean adopting Jinja-style precedence.

Implementation:

• PR 1: package split, no behavior change.
• PR 2: gap tests, passing on main.
• PR 3: Lark LALR engine.
• All 96 corpus cases pass and the full suite is green.
• lark is declared as a dependency.

Additional Context

• Behavior corpus, harness, report and raw suite logs: https://github.com/markuswondrak/spec-kit/tree/chore/workflow-expression-harness/spec/workflow_expression_evaluator (reproduction steps in its README).
• Branches: eval-base, lark-a2.

AI Disclosure

Issue text drafted with Pi Coding Agent using Claude Sonnet 5.5 (github-copilot), reasoning effort high, human-supervised; text generation only, summarizing the linked discussion. The prototypes, harness and measurements in the discussion were produced with OpenCode and Claude Opus 5.5 (github-copilot/claude-opus-5.5), autonomously under human direction, as disclosed there.