Repository navigation
docs(llm): document executor validation responsibility + processor-level LLM tests - #220
aryyann011 wants to merge 1 commit into
Conversation
|
Welcome to Accord Project, and thank you for the PR, @aryyann011. Validating Drafted by an Accord Project agent, reviewed by a maintainer. |
mttrbrts-agent
left a comment
There was a problem hiding this comment.
Thanks for the careful write-up and the tests, @aryyann011. The stubbed-reasoner approach is a nice way to test the LLM path without network access.
I'd rather not add validation at the executor level, though. Callers who use LLMExecutor directly take responsibility for validation, and the normal route through TemplateArchiveProcessor already validates every output. That's fromJSON plus assertRuntimeHierarchy on state, result and each event, for both the TypeScript and LLM paths. My specific concerns:
-
It runs the same checks twice. With this PR, every LLM response that goes through the processor gets checked by both layers, and the processor's checks already cover it.
-
It only covers half the boundary. The executor would check its outputs but not its inputs (
data,request,priorState), so direct callers still need their own validation. The TypeScript path would have no matching check, which makes the two execution paths behave differently. -
The two layers would apply different rules. The executor only accepts the template's own declared types for each slot. The processor accepts any concrete subtype of the runtime
State,ResponseorEvent. An output could pass one check and fail the other. -
Obligation events. The runtime
Obligationhas a required--> Contract contractrelationship. The processor fills it in withpopulateObligationBackReferencesbefore it validates events. This PR validates events inside the executor, which runs before that step, so I think an LLM-emitted Obligation event withoutcontractwould now be rejected when it succeeds today. None of the current tests cover an Obligation event, so this needs a test whichever way we go.
The issue you raised in #219 is fair, and these two smaller changes would address it:
- Add a doc comment on
LLMExecutorstating that its outputs are only checked for overall shape, and that callers should validate them against the template model or useTemplateArchiveProcessor. - Remove the stale
TODOinassertInitShape, which you rightly noticed is misleading.
If there's demand for validation without the processor, we could later move the processor's output checks (including filling in the Obligation back-reference) into one exported helper. The processor and direct callers would then share the same code instead of keeping two sets of rules.
Would you be up for reworking this PR into that smaller change? Your stubbed-reasoner tests would also be useful as processor-level LLM tests.
|
Fair review, thanks! all four points land, and the Obligation back-reference catch is a real regression I'd have shipped, so this is appreciated. Yes, up for the rework. Plan on my side: revert the executor-level checks, keep the TODO removal, add the doc comment stating outputs are only shape-checked and that callers wanting model validation should route through TemplateArchiveProcessor. I'll move the stubbed-reasoner tests to cover the processor's LLM path — and add an Obligation-event case there so the back-reference filling is covered whichever way we go. One question: when you say "processor-level LLM tests" — should they live in TemplateArchiveProcessor.test.ts or as a separate suite (LLM lane via the force mode)? I'll follow whatever fits the repo. |
…vel LLM tests Document that LLMExecutor responses are only shape-checked and that callers requiring model validation should route execution through TemplateArchiveProcessor (or validate the outputs themselves), removing the stale TODO that claimed executor responses would get Concerto-based deserialization. See accordproject#219 for the finding and the agreed direction. Move the LLM-lane coverage to the processor level: the new test/TemplateArchiveProcessor.llm.test.ts drives TemplateArchiveProcessor in LLM force mode with a canned (stubbed) reasoner, so no network or API key is needed. Covers: - a conforming init/trigger response accepted through output validation; - rejection of a result primitive of the wrong type (Double declared), an invented undeclared field, a state violating its model on the init path (Integer declared), and a state $class the model does not declare; - retention of the empty state placeholder on stateless templates, versus rejection of a non-empty state that is not a runtime State; - an Obligation event emitted without its required contract back-reference, which populateObligationBackReferences fills in before validation - the previously uncovered case, plus a new test template fixture (test/archives/llm-obligation-event) declaring a concrete Obligation event. Signed-off-by: Aryan Mishra <aryan456.rehla@gmail.com>
e382dbd to
a26b0bc
Compare
Closes #219
Summary
#219 documented that
LLMExecutorresponses were never validated against the template's Concerto model. This reworked PR resolves the issue in the direction set in the review of the previous version:TemplateArchiveProcessorremains the validation boundary, the executor's contract is now documented, the misleading staleTODOis removed, and the LLM lane is covered by processor-level tests.Changes Made
src/llm/LLMExecutor.tsassertInitShape/assertTriggerShape), that they are not validated against the template's Concerto model, and that callers requiring model validation should route execution throughTemplateArchiveProcessor(which validates data, request, priorState, state, result and events) or validate the returned pieces themselves.TODO("replace these with Concerto based deserialization"), which no longer described the intended behavior.Tests in this PR
New
test/TemplateArchiveProcessor.llm.test.ts— 8 cases drivingTemplateArchiveProcessorin LLM force mode with a canned (stubbed) reasoner, so they need no API key and no network:penaltyas a string (Doubledeclared), an invented undeclared field, a state violating its model on the init path (countas a string whereIntegeris declared), and a state$classthe model does not declare;{}state placeholder, while a non-empty state that is not a runtimeStateis rejected by the hierarchy check;FulfillmentObligationEvent(extends runtimeObligation) emitted without its requiredcontractback-reference passes, becausepopulateObligationBackReferencesfills it in before validation — the previously uncovered case noted in the review. A new test template fixture (test/archives/llm-obligation-event) declares the concrete Obligation event.Notes for reviewers