feat(cli): fail commands on a declared terminal failure state (output.outcome) by AshGodfrey · Pull Request #134 · speakeasy-api/openapi-generation

AshGodfrey · 2026-10-22T13:02:12Z

Why

Some APIs report that a request failed inside a successful HTTP response: a 200 whose body carries a terminal status such as failed or cancelled, often with an errors[] array describing why. Interactions, jobs and agent runs commonly work this way, in both the discrete JSON response and the completion event of an SSE stream.

A generated CLI only knows the HTTP outcome, so it exits 0 and prints the payload. Scripts and agents that rely on the exit code treat the failure as success. The async recipe already lets a manifest classify poll states (states: {failed: failure, ...}), and the runtime exits 1 with CLI_ASYNC_FAILED on a failure state, but there was no way to declare the same table for a command that returns the terminal state directly or streams it.

What changed

x-speakeasy-cli-commands gains output.outcome on commands and on operation entries:

output:
  outcome:
    statePointer: $.status                  # state in the JSON response
    streamStatePointer: $.data.interaction.status   # state inside streamed events
    states:
      queued: pending
      in_progress: pending
      completed: success
      incomplete: success
      failed: failure
      cancelled: failure
    errorPointer: $.errors                  # optional failure detail (object or array)
    streamErrorPointer: $.data.interaction.errors
    errorMessageField: message              # default

Extension decoder and linker: at least one state pointer is required; states uses the async classifications and must name at least one failure. statePointer must select a closed string enum in every application/json 2xx response and the table must classify exactly that enum, so no API state can reach the runtime unclassified. streamStatePointer is checked like a stream projection (a string in at least one event shape); when that leaf is a closed enum the table is held to it too (exact match for a stream-only table, every member classified when statePointer already pinned the table), a leaf without an enum warns that the table cannot be verified, and an open enum is rejected. Error pointers must address an object or an array wherever they resolve; errorPointer may exist in only some response variants (a failed union arm) and is rejected only when it resolves nowhere. async and output.outcome on the same command is an error, since async.states already owns the terminal classification. A single-route intent that declares no table of its own inherits its operation entry's, so a state classified for the operation command also fails the intent; async intents never inherit.
Runtime: the table travels on a speakeasy_outcome annotation. For discrete responses the body is peeked before rendering; the payload is rendered as before (JSON, pretty, table, --jq) and then, if the state classifies as failure, the command returns a CLI_OUTCOME_FAILED error (runtime_error, exit 1) whose message names the state and the first failure-detail message. Artifact commands skip the media write on failure. For streams the completion event is classified as it arrives; every event is still rendered and stream metadata still reported, then the same error is returned. The agent-mode envelope carries error_reason: CLI_OUTCOME_FAILED, state, and error (the detail at the error pointer).
Only failure changes behaviour. pending, success and handoff states, unknown states from a newer API, events without the state, and streams that end without a completion event all keep today's exit 0. Undeclared commands are byte-for-byte unchanged.
The response-path walker now treats a null union arm or a null entry in a type list as absence. This also applies to async state and handle pointers: a nullable leaf generates, and the async runtime reports a protocol error only if a null value actually arrives.
CLI_OUTCOME_FAILED joins the runtime reason list (hint validation, normalizeCLIReason, agent README).
Review fixture: render declares a discrete outcome table and say a stream one, so the review CLI exercises both annotations.
Streams decode each event's wire frame only when something consumes it (an outcome table, or the wire re-encode for JSON output); projected and table streams keep their previous per-event cost.

Testing

go test ./internal/extensions/: new decode/link tests cover both pointers, operation-level declarations, and every rejection (missing pointer, no failure class, bad classification, unknown key, state not in the enum, enum member unclassified, non-enum pointer, stream pointer on a discrete operation, stream pointer that resolves nowhere, error pointer that is not an object/array, error bindings without their state pointer, async conflict, planned command).
Inheritance: a bare intent picks up its operation entry's table; an async intent does not. In the primary fixture outcome-check now declares no table and still fails on failed/cancelled.
Pagination: without --all, a first page in a failure state exits 1 and a pending first page exits 0 (the server now serves a configurable first-page status).
Artifacts: the generated artifact harness gains a case where the artifact command's response carries its declared failure state, asserting exit 1, empty stdout, and no file written (runs against the review render command).
Stream states: a misspelled stream state and an unclassified stream enum member are rejected; a stream leaf without an enum produces the unverifiable warning. errorPointer resolving only on one oneOf arm is accepted, resolving on no arm is rejected.
Primary CLI (TARGET=primary make build-cli, new outcome_additional_test.go): a CLI-only overlay adds a discrete interaction lookup and an SSE interaction stream with a status enum. Tests run the generated operation command and the intent command through completed, incomplete, in_progress, failed (with and without detail) and cancelled in JSON, pretty and agent mode, asserting exit code, rendered payload, stderr message and envelope fields; stream tests assert the projected text, NDJSON events, exit code, and that a delta-only stream with no completion event still exits 0.
TARGET=review make build-cli regenerated zSDKs/sdk-cli.
make lint, CLI template type-check, go generate ./....

Public-safety check

This change contains no credentials, customer documents, private repository URLs, private filesystem paths, or unredacted private logs.
Title, body, comments, and commit messages name no customers or customer-derived identifiers, private paths or trackers, or workflow provenance, and are understandable without private context (.claude/skills/public-repo-communication/SKILL.md).
Generated fixtures and review SDK changes are public-safe.
I reviewed git diff --check.

Summary by cubic

Lets generated CLIs exit 1 when a successful HTTP response declares a terminal failure state, for discrete JSON responses, streamed SSE events, and paginated pages.

Previously the CLI only knew the HTTP outcome, so it printed the payload and exited 0 even when the body declared failed or cancelled. Commands can now declare output.outcome: a states table using the async classifications, a pointer to the state, and optional failure detail. When a declared failure state appears, the payload is still rendered, then the command exits 1 with reason CLI_OUTCOME_FAILED, naming the state and the first detail message. Only failure changes behavior; pending, success, handoff, unknown states, and undeclared commands keep exit 0.

statePointer addresses the JSON state and streamStatePointer the state in streamed events, with optional error pointers for detail; a detail array of plain strings contributes its first entry to the message.
The linker pins the JSON state pointer to a closed string enum the table classifies completely, intersecting allOf and ref-sibling enums when verifying; a stream state enum is classified exactly when stream-only, and an enum-less leaf warns while an open enum is rejected.
output.outcome beside async on a command is an error; single-route intents without their own table inherit the operation entry's (async intents do not).
Artifact commands skip the media write on failure; the review CLI declares a discrete table on render and a stream table on say.

^{Written for commit acbb4b5. Summary will update on new commits.}

Some APIs report that a request failed inside a successful HTTP response: a 200 whose body carries a terminal status such as "failed" or "cancelled", often with an errors[] array, in both the discrete JSON response and the completion event of an SSE stream. A generated CLI only knew the HTTP outcome, so it printed the payload and exited 0. x-speakeasy-cli-commands gains output.outcome on commands and operation entries: statePointer (state in the JSON response), streamStatePointer (state inside streamed events), a states table using the async classifications, and optional errorPointer / streamErrorPointer / errorMessageField for the failure detail. The decoder requires a state pointer and at least one failure classification; the linker requires the JSON state pointer to select a closed string enum that the table classifies completely, checks the stream pointer like a stream projection, and rejects an outcome table beside an async recipe. At runtime the table travels on a speakeasy_outcome annotation. Discrete responses are rendered as before and then, when the state classifies as failure, the command returns CLI_OUTCOME_FAILED (runtime_error, exit 1) naming the state and the first failure-detail message; artifact commands skip the media write. Streams render every event and report metadata before returning the same error. Only "failure" changes behaviour; other classifications, unknown states, events without the state, and streams with no completion event keep exit 0. Undeclared commands are unchanged. The review CLI declares a discrete table on render and a stream table on say; the primary CLI overlay adds a CLI-only interaction lookup and stream with tests across JSON, pretty and agent mode.

cubic-dev-ai

4 issues found across 28 files

Confidence score: 3/5

In streaming.go, failure completion events carried as error unions can bypass outcomeStreamFailure, so the CLI may not classify the stream as failed. Classify the raw event before returning the in-band error.
In output.go.stmpl, a response-body read failure can be discarded and the partial body reported as successful output. Propagate the read error.
In output.go, failed artifact responses return before renderResult, leaving the CLI with no response payload. Continue to render the payload before reporting the artifact failure.
In cli_commands_link.go, a misspelled or non-string errorMessageField can silently remove failure details from the CLI message. Validate the configured field.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="templates/templates/cli/auxiliary/internal/output/output.go.stmpl">

<violation number="1" location="templates/templates/cli/auxiliary/internal/output/output.go.stmpl:391">
P2: This pre-render classification can hide response-body read failures: `peekRawBody` discards the read error and leaves a partial body for `renderResult`, which then reports truncated output as successful. Propagate the peek error through outcome classification so rendering retains its existing read-error behavior.</violation>
</file>

<file name="zSDKs/sdk-cli/internal/output/streaming.go">

<violation number="1" location="zSDKs/sdk-cli/internal/output/streaming.go:152">
P2: The outcome classifier runs after the in-band error-event early return, so failure completion events represented by an error union bypass `outcomeStreamFailure`. Classify the raw event before returning `StreamEventError`, while preserving the existing protocol-error fallback when no declared failure state is present.</violation>
</file>

<file name="zSDKs/sdk-cli/internal/output/output.go">

<violation number="1" location="zSDKs/sdk-cli/internal/output/output.go:396">
P2: Failed artifact responses return before `renderResult`, so the CLI emits no response payload at all. Disable artifact handling and continue to `renderResult`, then report the failure afterward.</violation>
</file>

<file name="internal/extensions/cli_commands_link.go">

<violation number="1" location="internal/extensions/cli_commands_link.go:1103">
P2: Discrete outcome linking never validates the declared `errorMessageField`, so a misspelled or non-string message member silently removes the failure detail from the CLI message. Validate the configured message field on each traversable error object, matching async error binding validation.</violation>
</file>

_{Shadow auto-approve: would not auto-approve because issues were found.
Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack

View guided diff | Turn on auto-fix | Re-trigger cubic}

…rrorPointer Paginated commands rendered pages through PaginatedResult without ever consulting the declared statePointer, so --all exited 0 on a page whose body carried a failure state. Each page is now classified after it is rendered and the command fails on the first page in a failure state. The linker also walks streamErrorPointer through the streaming event shapes, requiring an object or array in at least one of them, so a path that resolves nowhere is rejected at generation time instead of silently dropping the failure detail.

cubic-dev-ai

0 issues found across 11 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 7 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

…s once The JSON state and error pointers now go through one response-path walker and accept nullable leaves: a null union arm or a "null" entry in a type list is treated as absence rather than rejected. Each streamed event is decoded once and the frame is shared between outcome classification and the wire rendering. Narrative comments are dropped from the outcome templates so they no longer land in generated CLIs.

cubic-dev-ai

All reported issues were addressed across 9 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

View guided diff | Turn on auto-fix | Re-trigger cubic}

…ept failure detail in any response variant The linker now collects the closed enum a streamStatePointer resolves to and holds the states table to it: a stream-only table must name exactly those members, and when statePointer already pinned the table to the JSON enum every stream member must still be classified. A leaf without a closed enum warns that the table cannot be verified; an open enum is rejected. errorPointer no longer has to resolve in every response variant. Failure detail that only exists on the failed arm of a union is accepted, and the pointer is rejected only when it resolves in no variant of any JSON 2xx response, with a did-you-mean from the properties it passed. The review fixture's streamed status gains an enum so the regenerated review CLI exercises the stream-states check. Co-Authored-By: Claude <noreply@anthropic.com>

cubic-dev-ai

0 issues found across 7 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 9 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic}

…stream frames on demand, and cover the artifact and single-page outcome paths A single-route intent without its own output.outcome now inherits the table its operation entry declared, so a state classified for the operation command also fails the intent; async intents keep async.states as their only classification. The primary fixture's outcome-check command drops its duplicate table and relies on the inheritance. Streams decode an event's wire frame only when an outcome table needs it or the wire re-encode will use it, restoring the previous per-event cost for projected and table output. The unused StateSegments field leaves the outcome model and the unreachable string arm leaves the failure-detail reader. The generated artifact harness gains a case where the response carries the command's declared failure state and asserts nothing is written, and the paginated test now serves a configurable first-page status so a failed first page is shown to exit 1 without --all. Co-Authored-By: Claude <noreply@anthropic.com>

cubic-dev-ai

All reported issues were addressed across 16 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

…t outcome fixture, and document the outcome envelope fields Failure detail arrays of plain strings now contribute their first entry to the CLI_OUTCOME_FAILED message. The generated artifact failure fixture builds array elements for numeric pointer segments and Go-quotes the asserted state. The agent README names the state and error fields the outcome envelope adds. The states map is built on a null prototype so any enum member name survives serialization. perms.go and the review CLI are regenerated.

cubic-dev-ai

0 issues found across 9 files (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. New opt-in CLI feature makes generated commands exit 1 on declared failure states in HTTP-success responses and extends the agent error envelope (CLI_OUTCOME_FAILED, state/error); these contract and artifact choices need human sign-off.

View guided diff | Turn on auto-fix | Re-trigger cubic}

…aragraph

cubic-dev-ai

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Adds opt-in output.outcome so generated CLI commands exit 1 on declared failure states in HTTP-2xx responses, extends the agent error envelope (CLI_OUTCOME_FAILED/state/error), and suppresses artifact payloads on failure; these contract and UX choices need human sign-off.

View guided diff | Turn on auto-fix | Re-trigger cubic}

2ynn · 2026-10-22T18:39:16Z

+	if allOf, ok := schemaMap["allOf"].([]any); ok {
+		branches = append(branches, allOf...)
+	}


allOf goes into the same branch list as oneOf/anyOf here, so it's handled like a union where every arm has to be closed. That breaks for intersections:

allOf: [{type: string}, {enum: [done, cancelled]}]: the type-only arm counts as "not closed", so we only get the "states cannot be verified" warning. A typo like canceled: failure gets through, and a real cancelled event exits 0.

allOf: [{enum: [done, cancelled, extra]}, {enum: [done, cancelled]}]: the values get concatenated, so a correct table is rejected for not classifying extra.

An enum next to allOf or $ref is also dropped, for the same reason.

I think allOf should intersect the arms that have an enum and ignore the arms that only set a type. A regression test for each of the two cases above would help.

Fixed in acbb4b5. leafEnum now treats every keyword conjunctively: a const, a local enum, the referenced component, each allOf arm, and a oneOf/anyOf (whose non-null arms must all be closed) each contribute a member set, and the leaf is closed over their intersection. Arms that only set a type contribute nothing. The walker also stops resolving a $ref that carries sibling constraints before the leaf check, so an enum beside a $ref narrows the target.

TestCLICommands_OutcomeStreamStateAllOfIntersectsEnums covers all three: allOf: [{type: string}, {enum}], two enum arms with an extra member in one, and an enum beside a $ref. Each asserts the valid table decodes with no warning and that canceled is rejected with the did-you-mean.

…e states

cubic-dev-ai

1 issue found across 2 files (changes from recent commits).

Confidence score: 3/5

In internal/extensions/cli_commands_link.go, object-level $ref siblings such as properties are dropped while resolving nested paths, so narrowed state schemas may be treated as the referenced component schema. Preserve those siblings during traversal.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="internal/extensions/cli_commands_link.go">

<violation number="1" location="internal/extensions/cli_commands_link.go:2192">
P2: This only applies `$ref` siblings at the selected leaf. Object-level siblings such as `properties` are dropped while walking remaining segments, so narrowed nested state schemas can be treated as the referenced component's full enum; merge the reference and sibling schemas before continuing the path.</violation>
</file>

_{Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai · 2026-10-22T19:08:56Z

 			w.err = fmt.Errorf("%s %s %s: referenced schema %q does not exist in components.schemas%s", w.owner, w.field, w.selectPath, name, cliDidYouMean(name, cliComponentNames(w.d.schemaIndex.components)))
 			return
 		}
+		if len(segments) == 0 && cliRefHasSiblingConstraints(schemaMap) {


P2: This only applies $ref siblings at the selected leaf. Object-level siblings such as properties are dropped while walking remaining segments, so narrowed nested state schemas can be treated as the referenced component's full enum; merge the reference and sibling schemas before continuing the path.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/extensions/cli_commands_link.go, line 2192: <comment>This only applies `$ref` siblings at the selected leaf. Object-level siblings such as `properties` are dropped while walking remaining segments, so narrowed nested state schemas can be treated as the referenced component's full enum; merge the reference and sibling schemas before continuing the path.</comment> <file context> @@ -2189,6 +2189,10 @@ func (w *cliStreamPathWalker) walk(schema any, segments []cliPathSegment, seen [ w.err = fmt.Errorf("%s %s %s: referenced schema %q does not exist in components.schemas%s", w.owner, w.field, w.selectPath, name, cliDidYouMean(name, cliComponentNames(w.d.schemaIndex.components))) return } + if len(segments) == 0 && cliRefHasSiblingConstraints(schemaMap) { + w.checkLeaf(schemaMap, seen) + return </file context>

AshGodfrey requested a review from a team as a code owner October 22, 2026 13:02

AshGodfrey requested a review from speakeasybot October 22, 2026 13:02