Route judgment stages to a System One decision model by audreyt · Pull Request #120 · Jigsaw-Code/sensemaking-tools

audreyt · 2026-10-03T17:53:31Z

Summary

System One decision models (Clef, Jev, Mercury Decide and others) answer typed questions with probabilities in one pass. They cannot chat or generate, so MODEL_ENDPOINT_TYPE=openai_api_compatible cannot point at them. This adds a System One client and routes the judgment stages to it when DECISION_ENDPOINT_TYPE=systemone.

Judgment stages that move:

topic assignment and opinion assignment (one yes/no question per label)
moderation and bridging scores (probability per attribute)
opinion autorater verdicts and other eval verdicts (a 0-4 score, no written explanation)
jury approval votes
proposition equivalence (pairwise, then union-find) and the dedup winner / topic-home pick

Generative stages stay on Gemini or the OpenAI-compatible model: topic and opinion names, quote extraction, report text, proposition writing, translation and jury ranking.

Off by default. Enable with:

export DECISION_ENDPOINT_TYPE=systemone
export DECISION_MODEL=clef

With only these set, requests go to POST /v1/systemone on a local Ollama 0.35.1+ server. Optional settings, all read in model_factory.get_decision_model():

DECISION_MODEL_HOST: a bare host such as gpu-box is read the way the Ollama CLI reads OLLAMA_HOST. A URL without a path gets /v1/systemone. A URL with a path is used as given.
DECISION_MODEL_API_KEY: bearer key for a hosted endpoint.
DECISION_MODEL_KEEP_ALIVE: defaults to 30m for the default local Ollama. With a host set it is sent only when given, as other servers reject the field.
DECISION_MAX_QUESTIONS (default 64), DECISION_THRESHOLD (default 0.5), DECISION_MAX_CONCURRENT (default 1, as a local runner has one slot).

Behavior notes:

Other is never asked as a label. It is assigned only when no other label clears the threshold, and a quote that is already Other is not sent to the autorater.
The autorater asks a score question and reads its most likely level. A level below autorater_min_score finalizes the quote as Other without retry, as the decision is deterministic.
Equivalence pairs are asked four per request. Clef's probabilities drift upward when many pair questions share one request.
Rate limits, server errors and dropped connections are retried twice with backoff. A 404 from an Ollama older than 0.35.1 says so.

Test plan

full pytest after merging upstream main (320 tests)
Local: Clef and pplx-decider:27b-q4_K_M on Ollama 0.40.11, and pplx-decider-flash on its own server, for assignment, autorater, approval votes, equivalence and dedup, bridging CLI and moderation scoring
Full local categorization run on a 24-response transit survey: gemma4 for generation, Clef for judgments
Hosted, same request shape: Clef on Workers AI, TypeSafe Jev and Inception Mercury Decide
Error paths: a 404 from an older Ollama names the fix; an HTTP 502 and a dropped connection are retried

Clef cannot chat, so scoring, assignment, approval votes, dedup, and autorater verdicts call /v1/systemone when DECISION_ENDPOINT_TYPE is systemone. Generative stages stay on the existing model.

google-cla · 2026-10-03T17:53:50Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

…-judgments Resolve conflicts with upstream's bridging-score rework: - get_bridging_scores.py: keep upstream's _score_texts/get_bridging_scores split (dedup, reuse, skip reasons, --force_rerun, --min_text_length) and carry over the System One log line and the API-key check that allows a missing Gemini key when the decision path is enabled. - get_gemini_scores_lib.py: keep upstream's parse_score_response, retry cap and missing-pair warning, and re-apply only the System One branch (decision client in __init__, _score_with_decision). The PR's earlier pure-reformatting hunks are dropped where upstream rewrote those lines.

Per the Google Python style guide (2.2 Imports), reference System One types as systemone.SystemOneClient, systemone.NoulQuestion, etc., and decision.AttributeSpec, instead of importing the classes directly. The same applies to dataclasses, contextlib and pydantic in the new modules and to the new decision_test.py.

decision no longer re-exports SystemOneClient now that it imports the systemone module, so reference the class from systemone directly.

_choose_pack formatted its error with answer.choice, but no answer is in scope there, so an out-of-range choice raised NameError instead of a PARSE SystemOneError. Use the parsed choice and add a test.

Move every DECISION_* environment read into model_factory, next to the existing MODEL_ENDPOINT_TYPE handling: get_decision_model() builds the SystemOneClient (or returns None when DECISION_ENDPOINT_TYPE is unset) and get_decision_threshold() reads the cutoff. decision.py no longer calls getenv; decision_client() caches the factory's client and decision_enabled() is derived from it. Callers log client.model instead of reading DECISION_MODEL themselves. An unknown DECISION_ENDPOINT_TYPE is now a config error rather than silently disabling the path. Drop the hard-coded OLLAMA_HOST fallback in decision.py. When OLLAMA_HOST is unset, SystemOneClient's single OLLAMA_DEFAULT_BASE_URL applies; when set, it is parsed the way the Ollama CLI parses it (no scheme means http, no port means 11434), so values shared with the server such as 0.0.0.0:11434 work.

The example exported OLLAMA_HOST, DECISION_THRESHOLD and DECISION_MAX_CONCURRENT at their default values. Export only the two required variables and describe the optional ones and their defaults in prose.

The key is no longer always required for --scorer_type GEMINI: the System One path scores without it. Say so in the error printed by prepare_for_moderation and get_bridging_scores.

metasoarous

Thanks for the updates @audreyt! We're very excited to test this out :-)

I left some additional comments, most of which we'd just like your feedback on, or are optional to address (in particular regarding further decision model abstraction). A few things it would be helpful to address before merging:

Remove "Other" from list of explicit categorization options
Where possible, frame documentation and configuration option names primarily in more general terms, and explicitly reference ollama and clef only as specific examples which have been tested (see also here)
If you agree it makes sense, switch to score for the autorater tasks

Thanks again!

metasoarous · 2026-10-21T22:32:42Z

    raise ValueError(f"Unknown MODEL_ENDPOINT_TYPE: {endpoint_type}")
+
+
+def get_decision_model() -> systemone.SystemOneClient | None:


I think ideally get_decision_model would always return something of abstract type DecisionModel, which could be either a SystemOneDecisionModel or GenerativeDecisionModel. There may be some challenges in fully abstracting over those for the generative case, especially as relates to prompt structures, but probably worth doing long term.

This shouldn't be an obstacle to merging (we're happy to do the additional work to realize this, as it may be a bit involved), but wanted to at least make you aware and get your thoughts on it.

metasoarous · 2026-10-21T22:34:36Z

+          decision_model = decision.decision_client()
+          if decision_model is not None:
+            score = float(
+                await decision.rubric_level(decision_model, str(prompt or ""))
            )
+            explanation = ""
+          else:
+            # Call GenaiModel
+            response = await self.model.call_gemini(
+                prompt=prompt,
+                run_name=f"eval_job_{job_id}",
+                response_mime_type="application/json",
+            )
+
+            result_text = response.get("text", "")
+            score = 0.0
+            explanation = ""
+            try:
+              # The model is instructed to return JSON.
+              json_result = json.loads(result_text)
+              score = float(json_result.get("score", 0))
+              explanation = json_result.get("explanation", "")
+            except (json.JSONDecodeError, ValueError) as e:
+              raise ValueError(
+                  f"Failed to parse JSON response: {result_text}. Error: {e}"
+              )


Again, not a required changed, but pointing out a case where ideally we'd just have one abstract decision model to instantiate, and call in a uniform manner rather than having to bake in a bunch of conditional logic like this.

In reference to comment here: https://github.com/Jigsaw-Code/sensemaking-tools/pull/120/changes#r4342652420

metasoarous · 2026-10-21T22:46:16Z

+    raise SystemOneError(FailureKind.HTTP, str(exc.reason)) from exc
+
+
+def ollama_base_url(host: str) -> str:


Is any of this actually specific to ollama, or could it be generalized to other system one decision model hosts (e.g. Jev)?

Only keep_alive was Ollama-specific. With a configurable endpoint URL, a bearer key and the Workers AI result envelope handled, the same client now works with Jev, Mercury Decide and Clef on Workers AI (c06d9b8, 177a1ec).

metasoarous · 2026-10-21T22:52:33Z

+  if voting_mode == VotingMode.APPROVAL and decision_model is not None:
+    logging.info(
+        "Approval votes will use System One model %s. Ranking stays on the"
+        " generative model.",
+        decision_model.model,
+    )


I think this is fine for now, but thinking ahead, we probably could implement this as a recursive series of system one decision model calls that pick "the next highest ranked item" until all items are ranked.

metasoarous · 2026-10-21T22:56:54Z

+    instructions: str | None = None,
+) -> int:
+  """Picks the 0-4 autorater level. Clef does not write the explanation."""
+  question = systemone.ChoiceQuestion(


Should this be a ScoreQuestion instead? If I'm not mistaken, we're not actually using it anywhere yet, but this is the one part of the codebase where I think it makes sense.

Switched in fa7f842. 5badc0c then reads the score at its most likely level, as rounding the weighted score accepted only 7 of 16 correct assignments against 14 of 16 for the most likely level.

metasoarous · 2026-10-22T00:44:15Z

+##### Running judgments on Ollama Clef
+
+Clef is a System One decision model. It does not implement chat, so `MODEL_ENDPOINT_TYPE=openai_api_compatible` cannot point at it. When the variables below are set, judgment stages call `POST /v1/systemone` on a local Ollama 0.35.1+ server. Generative stages stay on Gemini or the OpenAI-compatible model: topic and opinion names, quote extraction, report text, proposition writing, translation, and jury ranking.


I think it's fine to mention Ollama and Clef as specific examples, but ideally the opening framing would lead with more general descriptions around system one decision models.

Reframed in c06d9b8. The section now leads with System One decision models, with Ollama Clef as the default and the tested hosts as examples.

metasoarous · 2026-10-22T01:00:53Z

+      for quote in statement.quotes or []:
+        if quote.topic.name != topic.name:
+          continue
+        chosen = await decision.assign_labels(


Let's explicitly omit "Other" as an option, and only assign if no other options match.

Done in fa7f842. Other is no longer asked, and is assigned only when no other option clears the threshold.

metasoarous · 2026-10-22T01:10:44Z

+
+import pydantic
+
+MAX_QUESTIONS: Final = 64


This is fine for now, but I think most of the other system one decision models support up to 256 max questions. May eventually want to consider how we would configure this.

DECISION_MAX_QUESTIONS now sets this (default 64, Ollama's limit).

metasoarous · 2026-10-22T01:16:17Z

+            f"{statement}"
+        ),
+        criteria={
+            f"o{opt_index}": option for opt_index, option in enumerate(options)


I think it may be more idiomatic to explicitly use {Agree, Disagree, Pass} as options (as well as {"Strongly Agree", "Strongly Disagree"} for the Likert options). I don't know if it actually affects model performance with Clef, but I imagine that it might impact some system one decision model implementations.

Left as o0, o1 keys for now, as scale labels such as "Strongly Agree" contain spaces and choice keys cannot.

…-judgments

Other is no longer asked as a yes/no label. It is assigned only when no other topic or opinion clears the threshold. The 0-4 autorater rubric is now a score question (levels lowest first), rounded half up to a whole level. On the System One opinion path a level below the upstream autorater_min_score, not a hardcoded 4, finalizes the quote as Other.

Hosts share the state-plus-questions request shape and differ in path and auth, so the client now takes an endpoint, an optional bearer key, and a question limit: - DECISION_MODEL_HOST replaces OLLAMA_HOST. A bare host is still read like the Ollama CLI reads it; a URL without a path gets /v1/systemone; a URL with a path is used as given. - DECISION_MODEL_API_KEY is sent as a bearer token. - keep_alive is Ollama-only (Jev returns 400 and Workers AI 422 for it), so it is sent only when no key is set. - Workers AI's result envelope is unwrapped. - DECISION_MAX_QUESTIONS sets the per-request limit (default 64). The README now leads with System One decision models in general, with local Ollama Clef as the default and the hosted endpoints tested live: Clef on Workers AI, TypeSafe Jev, and Inception Mercury Decide.

A local Ollama older than 0.35.1 answers /v1/systemone with a bare '404 page not found'. The error now names the URL, the Ollama version that serves decisions, and DECISION_MODEL_HOST for other hosts.

When no learned opinion clears the threshold the quote is already Other, and a failing rubric would only finalize it as Other again. A local run (gemma4 generation, Clef judgments, 24 responses) rated 20 such quotes and logged each as 'scored N; using Other'. With the skip the same run gives identical assignments and no wasted ratings.

Running the pipeline against local Clef and pplx-decider on Ollama 0.40.11, pplx-decider-flash, and the hosted endpoints surfaced: - Equivalence asked every pair in one request. With a short fixed state and many pair questions, Clef's yes-probabilities drift upward: 16 or more pairs per request linked unrelated propositions (0.52-0.57) that it rejects when asked four at a time (at most 0.26), and union-find chained them into one cluster. Pairs are now asked four per request, concurrently. Jev, Mercury, and both pplx deciders do not drift; label assignment with 16 labels does not either. - keep_alive was sent whenever no key was set, but pplx-decider-flash's keyless server rejects it ("Unknown field 'keep_alive'"). It now defaults to 30m only for the default local Ollama; DECISION_MODEL_KEEP_ALIVE sets it for other Ollama hosts. - One upstream 502 from a hosted endpoint aborted the stage. Rate limits, server errors and dropped connections are now retried twice with backoff.

Rounding the probability-weighted score made the System One autorater reject correct opinion assignments: probability on the lower levels drags the mean below a clear top answer. On 16 correct and 16 wrong assignments (local and Workers AI Clef), the rounded mean accepted 7 correct, the most likely level 13, and the original choice question 14; none accepted a wrong one. Keeping the score question, as the review asked, and reading its most likely level recovers nearly all of the choice question's recall.

urlopen lets a connection that the host drops before answering escape as http.client.RemoteDisconnected rather than URLError, so it skipped the retry. Connection failures now become a status-less SystemOneError, which is retried. Adds docstrings to the helpers and tests introduced in this upgrade, checks that keep_alive is absent rather than null, and stops pinning rubric wording in a test.

Adds one-line docstrings to the PR's private wire models, parse helpers, test fixtures, nested fake transports and responses, and the older test methods, so every function and class in the PR's files carries one, as AGENTS.md asks. Documentation only.

An AST scan of the PR's model files found five private helpers with multi-line signatures that a line-based check missed: _body and _nbytes in systemone.py, _choice_label, _choose_pack and _clusters in decision.py. Documentation only.

audreyt · 2026-10-26T19:55:10Z

Thanks @metasoarous! Pushed, with upstream main merged in. Point by point:

Other: no longer an explicit option. It is never asked as a yes/no label, and is assigned only when no other label clears the threshold (fa7f842). A quote that is already Other also skips the autorater (4fd4ea5).
General framing: DECISION_MODEL_HOST replaces OLLAMA_HOST, with DECISION_MODEL_API_KEY for hosted endpoints (c06d9b8). The README now leads with System One decision models in general, with local Ollama Clef as the default and the tested hosts listed as examples. The only Ollama-specific piece turned out to be keep_alive, which Jev and Workers AI reject, so it is now sent only to the default local Ollama or when DECISION_MODEL_KEEP_ALIVE is set (177a1ec).
Score for the autorater: switched (fa7f842), with one correction from testing (5badc0c). Rounding the probability-weighted score rejected too many correct assignments, as probability on the lower levels drags the mean below a clear top answer. On 16 correct and 16 wrong opinion assignments, run on both local and Workers AI Clef, the rounded mean accepted 7 correct ones, the most likely level 14 and the earlier choice question 14. None accepted a wrong one. The score question is now read at its most likely level.

Running the whole pipeline locally also surfaced two more fixes. Equivalence now asks four pairs per request, as Clef linked unrelated propositions once 16 or more pairs shared one request; Jev, Mercury and the pplx deciders did not show this (177a1ec). Transient failures are retried, and a 404 from an Ollama older than 0.35.1 now says what to change (5a79e2c, 2ec265a). The System One opinion path also honors the new upstream autorater_min_score.

The DecisionModel abstraction and the recursive ranking idea both sound right to me as follow-ups, and I am happy to help with either.

metasoarous

Great! Thanks for these changes!

Route judgment stages to a local System One model.

47c91d6

Clef cannot chat, so scoring, assignment, approval votes, dedup, and autorater verdicts call /v1/systemone when DECISION_ENDPOINT_TYPE is systemone. Generative stages stay on the existing model.

sorensenjs reviewed Oct 7, 2026

View reviewed changes

audreyt added 8 commits October 10, 2026 13:25

Annotate ContentScorer with systemone.SystemOneClient

32188e0

decision no longer re-exports SystemOneClient now that it imports the systemone module, so reference the class from systemone directly.

Use itertools.combinations for equivalence pairs

9064e57

Fix NameError in choose_one out-of-range error

f6e3993

_choose_pack formatted its error with answer.choice, but no answer is in scope there, so an out-of-range choice raised NameError instead of a PARSE SystemOneError. Use the parsed choice and add a test.

Show only required System One settings in the README example

13609d9

The example exported OLLAMA_HOST, DECISION_THRESHOLD and DECISION_MAX_CONCURRENT at their default values. Export only the two required variables and describe the optional ones and their defaults in prose.

Correct the missing Gemini key error message

a001300

The key is no longer always required for --scorer_type GEMINI: the System One path scores without it. Say so in the error printed by prepare_for_moderation and get_bridging_scores.

This was referenced Oct 16, 2026

Add OpenRouter multi-model and multi-language support (initial scaffolding with GPT-OSS-120B) #38

Closed

[WIP] Add local LM Studio runner, one-shot report script, and i18n output #54

Closed

metasoarous reviewed Oct 22, 2026

View reviewed changes

audreyt added 10 commits October 26, 2026 10:05

Merge upstream Jigsaw-Code/sensemaking-tools main into clef-systemone…

5a2e927

…-judgments

Say what to do when a host does not serve System One decisions

5a79e2c

A local Ollama older than 0.35.1 answers /v1/systemone with a bare '404 page not found'. The error now names the URL, the Ollama version that serves decisions, and DECISION_MODEL_HOST for other hosts.

Document the last undocumented System One helpers

7415d5d

An AST scan of the PR's model files found five private helpers with multi-line signatures that a line-based check missed: _body and _nbytes in systemone.py, _choice_label, _choose_pack and _clusters in decision.py. Documentation only.

audreyt changed the title ~~Route judgment stages to a local System One model~~ Route judgment stages to a System One decision model Oct 26, 2026

This was referenced Oct 26, 2026

Extract quotes as exact spans with Mercury or Clairic (stacked on #120) #124

Draft

Describe participants from what they say about themselves (stacked on #124) #125

Draft

metasoarous approved these changes Oct 27, 2026

View reviewed changes

peterwgnd merged commit c713f86 into Jigsaw-Code:main Oct 27, 2026
6 checks passed

audreyt mentioned this pull request Oct 28, 2026

Support SubQ Snap and Liquid d1-ultra, and long states for hosted models #126

Open

		raise ValueError(f"Unknown MODEL_ENDPOINT_TYPE: {endpoint_type}")


		def get_decision_model() -> systemone.SystemOneClient \| None:

		raise SystemOneError(FailureKind.HTTP, str(exc.reason)) from exc


		def ollama_base_url(host: str) -> str:

		##### Running judgments on Ollama Clef

		Clef is a System One decision model. It does not implement chat, so `MODEL_ENDPOINT_TYPE=openai_api_compatible` cannot point at it. When the variables below are set, judgment stages call `POST /v1/systemone` on a local Ollama 0.35.1+ server. Generative stages stay on Gemini or the OpenAI-compatible model: topic and opinion names, quote extraction, report text, proposition writing, translation, and jury ranking.


		import pydantic

		MAX_QUESTIONS: Final = 64

Conversation

audreyt commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

google-cla Bot commented Oct 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

metasoarous left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

audreyt commented Oct 26, 2026

Uh oh!

metasoarous left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

audreyt commented Oct 3, 2026 •

edited

Loading