Skip to content

docs(llm): document executor validation responsibility + processor-level LLM tests - #220

Open
aryyann011 wants to merge 1 commit into
accordproject:mainfrom
aryyann011:fix/llm-executor-concerto-validation
Open

aryyann011 wants to merge 1 commit into
accordproject:mainfrom
aryyann011:fix/llm-executor-concerto-validation

Conversation

@aryyann011

@aryyann011 aryyann011 commented Nov 3, 2026 •

Copy link
Copy Markdown

Closes #219

Summary

#219 documented that LLMExecutor responses were never validated against the template's Concerto model. This reworked PR resolves the issue in the direction set in the review of the previous version: TemplateArchiveProcessor remains the validation boundary, the executor's contract is now documented, the misleading stale TODO is removed, and the LLM lane is covered by processor-level tests.

Changes Made

  • src/llm/LLMExecutor.ts
    • The class doc comment now states that LLM responses are only checked for overall shape (assertInitShape/assertTriggerShape), that they are not validated against the template's Concerto model, and that callers requiring model validation should route execution through TemplateArchiveProcessor (which validates data, request, priorState, state, result and events) or validate the returned pieces themselves.
    • Removed the stale TODO ("replace these with Concerto based deserialization"), which no longer described the intended behavior.
  • Executor-level validation was intentionally reverted per review (double validation, half-boundary coverage, diverging rule-sets, and the Obligation back-reference ordering issue discussed in the review).

Tests in this PR

New test/TemplateArchiveProcessor.llm.test.ts — 8 cases driving TemplateArchiveProcessor in LLM force mode with a canned (stubbed) reasoner, so they need no API key and no network:

  • a conforming init → trigger response accepted (stateful late-delivery template);
  • rejections at the processor's output validation: penalty as a string (Double declared), an invented undeclared field, a state violating its model on the init path (count as a string where Integer is declared), and a state $class the model does not declare;
  • stateless templates keep the empty {} state placeholder, while a non-empty state that is not a runtime State is rejected by the hierarchy check;
  • a FulfillmentObligationEvent (extends runtime Obligation) emitted without its required contract back-reference passes, because populateObligationBackReferences fills it in before validation — the previously uncovered case noted in the review. A new test template fixture (test/archives/llm-obligation-event) declares the concrete Obligation event.

Notes for reviewers

  • If there is demand for validated direct-executor use, the processor's output checks (including the Obligation back-reference filling) could be extracted into an exported helper shared with direct callers, as discussed — happy to take that as a follow-up.
  • Locally: new suite 8/8, processor/noLogic/StateHierarchy suites 31/31, build and lint clean.

@aryyann011
aryyann011 requested a review from a team November 3, 2026 10:24
@mira-accord-project

Copy link
Copy Markdown
Contributor

Welcome to Accord Project, and thank you for the PR, @aryyann011. Validating init and trigger responses against the template's Concerto model is a useful guard, and it comes with tests. The CI checks currently show a failure on this commit. Could you take a look at the logs when you have a moment? A maintainer has been requested to review, and we aim to respond within a few working days.

Drafted by an Accord Project agent, reviewed by a maintainer.

@mttrbrts-agent mttrbrts-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the careful write-up and the tests, @aryyann011. The stubbed-reasoner approach is a nice way to test the LLM path without network access.

I'd rather not add validation at the executor level, though. Callers who use LLMExecutor directly take responsibility for validation, and the normal route through TemplateArchiveProcessor already validates every output. That's fromJSON plus assertRuntimeHierarchy on state, result and each event, for both the TypeScript and LLM paths. My specific concerns:

  1. It runs the same checks twice. With this PR, every LLM response that goes through the processor gets checked by both layers, and the processor's checks already cover it.

  2. It only covers half the boundary. The executor would check its outputs but not its inputs (data, request, priorState), so direct callers still need their own validation. The TypeScript path would have no matching check, which makes the two execution paths behave differently.

  3. The two layers would apply different rules. The executor only accepts the template's own declared types for each slot. The processor accepts any concrete subtype of the runtime State, Response or Event. An output could pass one check and fail the other.

  4. Obligation events. The runtime Obligation has a required --> Contract contract relationship. The processor fills it in with populateObligationBackReferences before it validates events. This PR validates events inside the executor, which runs before that step, so I think an LLM-emitted Obligation event without contract would now be rejected when it succeeds today. None of the current tests cover an Obligation event, so this needs a test whichever way we go.

The issue you raised in #219 is fair, and these two smaller changes would address it:

  • Add a doc comment on LLMExecutor stating that its outputs are only checked for overall shape, and that callers should validate them against the template model or use TemplateArchiveProcessor.
  • Remove the stale TODO in assertInitShape, which you rightly noticed is misleading.

If there's demand for validation without the processor, we could later move the processor's output checks (including filling in the Obligation back-reference) into one exported helper. The processor and direct callers would then share the same code instead of keeping two sets of rules.

Would you be up for reworking this PR into that smaller change? Your stubbed-reasoner tests would also be useful as processor-level LLM tests.

@aryyann011

Copy link
Copy Markdown
Author

Fair review, thanks! all four points land, and the Obligation back-reference catch is a real regression I'd have shipped, so this is appreciated.

Yes, up for the rework. Plan on my side: revert the executor-level checks, keep the TODO removal, add the doc comment stating outputs are only shape-checked and that callers wanting model validation should route through TemplateArchiveProcessor. I'll move the stubbed-reasoner tests to cover the processor's LLM path — and add an Obligation-event case there so the back-reference filling is covered whichever way we go.

One question: when you say "processor-level LLM tests" — should they live in TemplateArchiveProcessor.test.ts or as a separate suite (LLM lane via the force mode)? I'll follow whatever fits the repo.

@github-actions github-actions Bot added the maintainer-engaged A maintainer has commented or reviewed this item label Nov 4, 2026
…vel LLM tests

Document that LLMExecutor responses are only shape-checked and that callers
requiring model validation should route execution through
TemplateArchiveProcessor (or validate the outputs themselves), removing the
stale TODO that claimed executor responses would get Concerto-based
deserialization. See accordproject#219 for the finding and the agreed direction.

Move the LLM-lane coverage to the processor level: the new
test/TemplateArchiveProcessor.llm.test.ts drives TemplateArchiveProcessor in
LLM force mode with a canned (stubbed) reasoner, so no network or API key is
needed. Covers:

- a conforming init/trigger response accepted through output validation;
- rejection of a result primitive of the wrong type (Double declared), an
  invented undeclared field, a state violating its model on the init path
  (Integer declared), and a state $class the model does not declare;
- retention of the empty state placeholder on stateless templates, versus
  rejection of a non-empty state that is not a runtime State;
- an Obligation event emitted without its required contract back-reference,
  which populateObligationBackReferences fills in before validation - the
  previously uncovered case, plus a new test template fixture
  (test/archives/llm-obligation-event) declaring a concrete Obligation event.

Signed-off-by: Aryan Mishra <aryan456.rehla@gmail.com>
@aryyann011
aryyann011 force-pushed the fix/llm-executor-concerto-validation branch from e382dbd to a26b0bc Compare November 4, 2026 17:50
@aryyann011 aryyann011 changed the title feat(llm): validate init/trigger responses against the template's Concerto model docs(llm): document executor validation responsibility + processor-level LLM tests Nov 4, 2026
@github-actions github-actions Bot added maintainer-engaged A maintainer has commented or reviewed this item and removed maintainer-engaged A maintainer has commented or reviewed this item labels Nov 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai/kind:feature maintainer-engaged A maintainer has commented or reviewed this item

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLMExecutor responses are validated with shape checks only; Concerto validation lives only in TemplateArchiveProcessor

2 participants