Add progress-event callbacks for evaluation requests - #28
Open
m-messer wants to merge 18 commits into
Open
Conversation
Implement a unified progress reporting framework that supports optional HTTP callbacks: - Define `Event` and `Reporter` abstractions in `internal/progress`. - Add `HTTPFactory` for building per-request HTTP callback reporters. - Introduce `Emit` convenience method to attach/report progress events via context. - Update supervisor and handler logic to emit lifecycle events. - Include comprehensive unit tests for reliability and correctness.
- Integrate callbackUrl field from µEd spec for progress reporting. - Replace progress callback headers with callbackUrl and request ID. - Include evaluation feedback payload in StageCompleted events. - Update tests to reflect callbackUrl usage and validation.
…ck delivery timeout
…k feature - Introduce `--progress-allowed-hosts` flag to restrict allowed callback hostnames. - Add `--progress-allow-private-networks` flag for optional private network access. - Implement automatic request ID generation for traceability and progress correlation. - Expand documentation with guidance on callback URL safety and SSRF prevention. - Update tests and internal logic for new configuration options and request IDs.
- Introduce IP filtering to block private, loopback, and link-local addresses. - Add hostname wildcards for fine-grained allowed host configuration. - Implement custom HTTP transport with DNS-based IP validation. - Add comprehensive unit tests to cover SSRF scenarios and configuration options.
- Introduce IP filtering to block private, loopback, and link-local addresses. - Add hostname wildcards for fine-grained allowed host configuration. - Implement custom HTTP transport with DNS-based IP validation. - Add comprehensive unit tests to cover SSRF scenarios and configuration options.
- Introduce `--progress-sidecar-unbind-grace-period` flag with default value of 250ms. - Add `UnbindAfterGrace` method to allow delayed unbinding with generation-safe logic. - Update supervisor adapter to utilize `UnbindAfterGrace` for improved POST handling.
MinEventInterval's default (200ms) rate-limited a fast evaluation function reporting two checkpoints from compareSets' evaluation function to at most one event per span: even with delivery now serialized on the client side, two closely-spaced report_progress() calls could still both arrive well under any single fixed interval, since arrival timing is governed by local HTTP round-trip cost, not real application-level delay. Add BurstSize (default 5): the first N events in a span bypass MinEventInterval spacing entirely (still bounded by MaxEventsPerSpan), so a handful of legitimate back-to-back checkpoints go through, while MinEventInterval keeps guarding against sustained event spam once the burst is used up. Also lower the MinEventInterval default itself from 200ms to 10ms, since 200ms had no real abuse-prevention basis and was overly aggressive for normal use.
- Add test coverage for new SSEReporter behavior and intermediate step deduplication. - Include tests for dependency graph validation in Lambda and standalone runtime modules. - Add multi-reporter tests to verify fan-out behavior and isolated child panics. - Improve overall test reliability with enhanced mocks and structured assertions.
- Implement SSE-based streaming for progress updates on `/evaluate` responses. - Add tests to validate SSE behavior, event streaming, and live frame correctness. - Collapse repeated lifecycle stages (`preparing`, `evaluating`) into single events per request. - Update README with detailed documentation on SSE usage, configuration, and behavior. - Introduce `--progress-stream-enabled` and `--progress-stream-heartbeat-seconds` flags for configuration. - Ensure terminal frames include accumulated steps and align with live frame data.
- Extract shared SSE streaming logic into `streamProgress` for reuse across `/evaluate` and `/chat`. - Add per-command terminal frame shapes (`sseEnvelope` for `/evaluate`, `sseChatEnvelope` for `/chat`). - Extend progress stages with `starting` and `thinking` for unified lifecycle reporting. - Refactor `/chat` to support streaming progress updates with callback compatibility.
- Extend progress lifecycle to include `starting` stage for improved stage granularity. - Add `/chat` endpoint compatibility with SSE progress streaming and event validation. - Refactor evaluation tests to use `starting` in place of `evaluating` at appropriate stages. - Update README to document new stages and `/chat` progress models. - Add corresponding unit tests for new lifecycle reporting and endpoint behavior.
- Introduce comprehensive test coverage for `/chat` endpoint's Server-Sent Events (SSE) streaming behavior. - Validate `thinking` and `starting` progress stages, terminal frame shapes, and fallback to JSON output. - Add tests for capability-disabled scenarios, callback handling, authentication errors, and heartbeat events.
- Extend `--progress-stream-enabled` flag and behavior to `/chat` responses. - Update README and CLI usage text to reflect new `/chat` SSE streaming support.
- Validate terminal frame payloads against OpenAPI schemas for `/chat` and `/evaluate` SSE responses. - Refactor middleware to rely on runtime response sniffing instead of preflight checks. - Update tests for new validation logic and streaming behavior. - Document terminal frame structure in the OpenAPI schema.
- Add tests to enforce parity between SSE frame structs and OpenAPI schema definitions. - Introduce `ValidateComponentSchema` for validating payloads against specific OpenAPI component schemas. - Refactor `validate_body.go` to centralize schema validation logic and improve testability.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /evaluateendpoint: shimmy POSTs a small JSON event (preparing→evaluating→completed/failed) to a caller-suppliedcallbackUrlat each stage of processing, in addition to the normal synchronous HTTP response.callbackUrlbody field andX-Request-Idheader (now echoed on every response, generated if the caller doesn't supply one, and reused as the progress correlation key).completedevent carries the actual feedback payload indata.feedback, so a caller usingcallbackUrlgets the final result delivered there too, converging with the spec's "deliver feedback results to this URL" wording even though shimmy always takes the synchronous 200 path rather than the spec's 202-Accepted deferred flow.--progress-allowed-hosts) and an explicit opt-out (--progress-allow-private-networks) for trusted private deployments.internal/progresspackage: a generic, reusable event/progress-reporter abstraction threaded through the dispatcher/supervisor/handler layers viacontext.Context, so future custom events (e.g. emitted by the evaluation function itself) are a natural extension rather than a rework./evaluatecallers that don't supplycallbackUrlsee no behavior change.🤖 Generated with Claude Code
https://claude.ai/code/session_0187xesBCDmvKLY1xYbSVfhA