Repository navigation
docs(examples): show how to use DecisionModel - #6339
Open
jpshackelford wants to merge 5 commits into
Open
jpshackelford wants to merge 5 commits into
jpshackelford wants to merge 5 commits into
Conversation
Contributor
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
jpshackelford
marked this pull request as ready for review
October 30, 2026 06:39
1 of 5 tasks
jpshackelford
force-pushed
the
feat/decision-model-examples
branch
2 times, most recently
from
October 30, 2026 07:25
3da2d5c to
8c45c48
Compare
Member
Author
How these examples were testedCI does not run the three examples, because it has no Live runs
Structured options vs plain stringsFor the structured OpenTelemetry spansI ran a small local OTLP/HTTP collector, a stdlib OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces \
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \
uv run python examples/01_standalone_sdk/62_decision_model_custom_provider.py
The SDK assumes gRPC unless Repository checks
Re-check on
|
| Check | Result |
|---|---|
| Example 60, live | Same route and labels. One span with questions and answers, and no state. |
| Example 61, live, plus OTLP | Same four routes. Five decision spans; each of the four triage spans was a child of its triage_issue span. No span recorded the state. This time the agent's reply was hedged (left_work_undone=0.50), so it went to a human. |
| Example 62, live, plus OTLP | Direct sync, direct async and LiteLLM all gave is_bug 0.96 and area import_export. Three spans, and only the LiteLLM one has a cost. |
| New optional fields, both routes | A Choice whose options have no description, and a Noul with only criteria, were accepted by LiteLLM and by TypeSafe's API directly. |
| LiteLLM limits | 129 questions, and 256 options in one Choice, each raised LLMBadRequestError before any request. |
| Bad API key, direct route | Raised the original httpx.HTTPStatusError (401) after one request. No retry, and no wrapper. |
| Bad API key, LiteLLM route | Raised LLMAuthenticationError, with LiteLLM's AuthenticationError as its cause. |
| Subclass returns an extra answer, or a wrong type | LLMResponseError: "answers nobody asked for: …" and "wrong answer type for: is_bug (ScoreAnswer, expected NoulAnswer)". |
| Docs snippets (taken from the docs PR pages) | Both run live. |
| Unit tests | test_decision.py 35 passed; decision.py coverage 98%. The LLM, observability and examples suites: 1,473 passed. |
To repeat the span checks, start a fresh collector process for each run. If you empty its log file while it is still running, the next span is written after a run of NUL bytes on the first line. A line-based grep then misses that span. That is how I first counted one decision span too few, and why I re-ran with a new collector each time.
This comment was drafted by an AI agent on behalf of the user.
2 tasks
jpshackelford
force-pushed
the
feat/decision-model-examples
branch
3 times, most recently
from
October 30, 2026 13:17
04defae to
aa06c83
Compare
Add 60_decision_model.py (one call, many typed questions, routing in code) and 61_decision_model_agent_gate.py (concurrent adecide() triage in front of an agent, plus a second request that reviews the agent's reply). Point the Laminar example at decision-model spans. Exclude both from the CI example run, which has no JEV_API_KEY. Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
…example Describe each kind of issue with what it covers, what it does not, and examples, so bug reports and feature requests are easier to tell apart. Co-authored-by: openhands <openhands@all-hands.dev>
Example 62 lists the decision models LiteLLM supports, then subclasses DecisionModel and overrides send() and asend() to call TypeSafe's HTTP API directly. It asks both routes the same questions with the same reading code, and recommends contributing generally available providers to LiteLLM. Example 60 now prints result.model instead of reading LiteLLM's response. Co-authored-by: openhands <openhands@all-hands.dev>
Example 60 says that going over LiteLLM's limits (128 questions, 255 options per Choice) raises LLMBadRequestError before anything is sent. Example 62's send() raises LLMResponseError for an answer type it cannot read, as DecisionModel does for answers of the wrong type, and its docstring notes that decide() checks the answers. Co-authored-by: openhands <openhands@all-hands.dev>
jpshackelford
force-pushed
the
feat/decision-model-examples
branch
from
October 30, 2026 14:48
aa06c83 to
5baa6ce
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HUMAN:
Examples to demonstrate the new DecisionModel API.
AGENT:
Why
#6068 adds
DecisionModel, but no example shows how to call it or how to design questions for it. A decision model is used differently from an LLM. It works best when you ask many narrow questions about one state in a single call, and when your code acts on the answers and their confidence. The TypeSafe docs for Jev say this, and the examples follow that guidance. Both examples triage GitHub issues, the same kind of task the other SDK examples give to an agent.Summary
60_decision_model.pytriages one issue in a singledecide()call that asks sixNoul,ChoiceandScorequestions. Some of those questions only matter for some kinds of issue. ThekindChoice uses structured options: each says what it covers, what it does not, and gives examples, so bug reports and feature requests are easier to tell apart. The code then reads the typed answers, routes with Noul thresholds and confidence gates, and logs the model version, tokens and cost.61_decision_model_agent_gate.pyputs a decision model in front of an agent. It triages a queue concurrently withadecide(), one request per issue, and sends only the small, clear issues to the agent. Then it makes a second request about the agent's reply. That request has to be separate because the reply did not exist when the first request ran.62_decision_model_custom_provider.pylists the decision models LiteLLM supports. Then it subclassesDecisionModeland overridessend()andasend()to call TypeSafe's HTTP API directly. Both routes answer the same questions, and the code that reads the answers is the same for both. Itssend()raisesLLMResponseErrorfor a reply it cannot read, asDecisionModeldoes for answers of the wrong type. For generally available models, it recommends contributing a provider to LiteLLM instead, so SDK users get it when the SDK takes that LiteLLM release.27_observability_laminar.pynow says that decision-model calls emitdecision.<model>spans. Example 61 wraps each triage in an@observespan, so the decision span nests under it.Issue Number
Refs #6015
Docs: OpenHands/docs#1054 documents all three examples, on a docs branch with the same name as this one.
How to Test
Set
JEV_API_KEY, then runuv run python examples/01_standalone_sdk/60_decision_model.py. Example 61 also needsLLM_API_KEY(andLLM_MODEL/LLM_BASE_URLif you use a proxy).jev-1.14.0) — One call answered all six questions with 788 input tokens, for $0.000033. TheseverityScore came back at 1.48 with confidence 0.28, so the confidence gate did not addpriority:high.kindoptions — On four short issues, both versions picked the same kind. Confidence rose on the two ambiguous ones: "should support paths with spaces" went from 0.44 to 0.72, and "Why does the agent ignore my LLM_MODEL setting?" from 0.64 to 0.72. The two clear issues stayed at 0.97 and 1.00.claude-haiku-4-5— The four issues went to the agent, needs more information, security team and human planning. The agent createdslug.py. The review request returnedclaims_done=0.97andleft_work_undone=0.38, which is in the unsure band, so the code sent the issue to a human. Decision calls cost $0.00012 and the agent cost $0.038. A re-run on2eb3abbc3routed the issues the same way. That time the agent's reply was clear (left_work_undone=0.05), so the issue went toready for review. Between the two runs, both outcomes of the review step have run live.jev-1.14.0):is_bug0.96, areaimport_export, severity 1.20 to 1.25 across runs. Jev's numbers vary by about 0.02 to 0.03 between identical requests. With a local OTLP collector, each of the three calls exported adecision.<model>span with request model, response model and token counts. Only the LiteLLM span has a cost, because TypeSafe's own API reports no price. The spans were checked again on2eb3abbc3.decision.typesafe/jev-latestspans was a child of itstriage_issuespan and had the question, answer, token and cost attributes.triage_issuehad no input attribute, so the issue text stayed off the trace.70245fdc8(review fixes) — I rebased onto it and ran all three examples live with a fresh local collector per run. Routes and answers matched the earlier runs. Example 61 exported five decision spans, and each of the four triage spans was a child of itstriage_issuespan. No span recorded the state. Through both LiteLLM and example 62's subclass, aChoicewhose options have no description and aNoulwith only criteria both worked. LiteLLM's limits (129 questions, 256 options) raisedLLMBadRequestErrorbefore sending. A bad API key on the direct route raised the originalHTTPStatusErrorafter one request, with no retry. On the LiteLLM route it raisedLLMAuthenticationError, with LiteLLM's error as its cause. When the subclass returned an answer nobody asked for, or one of the wrong type,decide()raisedLLMResponseError. Intest_decision.py, 35 tests pass and coverage ofdecision.pyis 98%. The 1,473 tests in the LLM, observability and examples suites pass.uv run pytest tests/examples/test_examples.pypasses. The three new examples are in_EXCLUDED_EXAMPLESbecause CI does not setJEV_API_KEY.Video/Screenshots
None.
Design Doc
None.
Type
Notes
To run these examples in the
Run Examples Scriptsworkflow, add aJEV_API_KEYrepository secret, pass it inrun-examples.yml, and remove the three lines from_EXCLUDED_EXAMPLES. The optional "Docs example" check passes because OpenHands/docs#1054 documents all three examples.This PR was drafted by an AI agent on behalf of the user.
@jpshackelford can click here to continue refining the PR
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:5baa6ce-pythonRun
All tags pushed for this build
About Multi-Architecture Support
5baa6ce-python) is a multi-arch manifest supporting both amd64 and arm645baa6ce-python-amd64) are also available if needed