Skip to content

RSPEED: workflow provider catalog and secret registry (#51 Phase 1) #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jameswnl wants to merge 1 commit into
base: harness
Choose a base branch
from
Open

RSPEED: workflow provider catalog and secret registry (#51 Phase 1) #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
237 changes: 230 additions & 7 deletions docs/devel_doc/openapi.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11497,7 +11497,7 @@
"workflows"
],
"summary": "Start Workflow Handler",
"description": "Start a new workflow execution.\n\nParameters:\n request: FastAPI request (used by middleware).\n body: Workflow definition and configuration.\n auth: Authentication tuple (used by middleware).\n\nReturns:\n Workflow ID and initial status.",
"description": "Start a new workflow execution.\n\nParameters:\n request: FastAPI request (used by middleware).\n body: Workflow definition and configuration.\n auth: Authentication tuple (used by middleware).\n\nReturns:\n Workflow ID and initial status. A response carrying ``Deprecation`` and\n ``Sunset`` headers when the caller used the deprecated\n ``credentials_secret`` field.",
"operationId": "start_workflow_handler_v1_workflows_run_post",
"requestBody": {
"content": {
Expand All @@ -11514,11 +11514,7 @@
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"additionalProperties": true,
"type": "object",
"title": "Response Start Workflow Handler V1 Workflows Run Post"
}
"schema": {}
}
}
},
Expand Down Expand Up @@ -21760,6 +21756,123 @@
"title": "SearchRankingOptions",
"description": "Options for ranking and filtering search results.\n\nThis class configures how search results are ranked and filtered. You can use algorithm-based\nrerankers (weighted, RRF) or neural rerankers. Defaults from VectorStoresConfig are\nused when parameters are not provided.\n\nExamples:\n # Weighted ranker with custom alpha\n SearchRankingOptions(ranker=\"weighted\", alpha=0.7)\n\n # RRF ranker with custom impact factor\n SearchRankingOptions(ranker=\"rrf\", impact_factor=50.0)\n\n # Use config defaults (just specify ranker type)\n SearchRankingOptions(ranker=\"weighted\") # Uses alpha from VectorStoresConfig\n\n # Score threshold filtering\n SearchRankingOptions(ranker=\"weighted\", score_threshold=0.5)\n\n:param ranker: (Optional) Name of the ranking algorithm to use. Supported values:\n - \"weighted\": Weighted combination of vector and keyword scores\n - \"rrf\": Reciprocal Rank Fusion algorithm\n - \"neural\": Neural reranking model (requires model parameter)\n Note: For OpenAI API compatibility, any string value is accepted, but only the above values are supported.\n:param score_threshold: (Optional) Minimum relevance score threshold for results. Default: 0.0\n:param alpha: (Optional) Weight factor for weighted ranker (0-1).\n - 0.0 = keyword only\n - 0.5 = equal weight (default)\n - 1.0 = vector only\n Only used when ranker=\"weighted\" and weights is not provided.\n Falls back to VectorStoresConfig.chunk_retrieval_params.weighted_search_alpha if not provided.\n:param impact_factor: (Optional) Impact factor (k) for RRF algorithm.\n Lower values emphasize higher-ranked results. Default: 60.0 (optimal from research).\n Only used when ranker=\"rrf\".\n Falls back to VectorStoresConfig.chunk_retrieval_params.rrf_impact_factor if not provided.\n:param weights: (Optional) Dictionary of weights for combining different signal types.\n Keys can be \"vector\", \"keyword\", \"neural\". Values should sum to 1.0.\n Used when combining algorithm-based reranking with neural reranking.\n Example: {\"vector\": 0.3, \"keyword\": 0.3, \"neural\": 0.4}\n:param model: (Optional) Model identifier for neural reranker (e.g., \"transformers/Qwen/Qwen3-Reranker-0.6B\").\n Required when ranker=\"neural\" or when weights contains \"neural\"."
},
"SecretBinding": {
"properties": {
"name": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9/_.-]{0,127}$",
"title": "Logical secret name",
"description": "The SecretRef.name this binding resolves."
},
"backend": {
"type": "string",
"enum": [
"env",
"k8s",
"file"
],
"title": "Backend",
"description": "Where the secret value lives."
},
"env": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Environment variable",
"description": "Used by backend env."
},
"secret_name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Kubernetes Secret name",
"description": "Used by backend k8s."
},
"key": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Secret key",
"description": "Key within the Secret (backend k8s)."
},
"path": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "File path",
"description": "Used by backend file."
}
},
"additionalProperties": false,
"type": "object",
"required": [
"name",
"backend"
],
"title": "SecretBinding",
"description": "Operator-only mapping of a logical secret name to a backend location.\n\nNever accepted from a request. Exactly the fields of the chosen backend\nmay be set.\n\nAttributes:\n name: Logical name this binding resolves (matches ``SecretRef.name``).\n backend: Where the value lives: ``env``, ``k8s`` or ``file``.\n env: Environment variable name (backend ``env``).\n secret_name: Kubernetes Secret name (backend ``k8s``).\n key: Key within the Kubernetes Secret (backend ``k8s``).\n path: File path (backend ``file``)."
},
"SecretRef": {
"properties": {
"name": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9/_.-]{0,127}$",
"title": "Secret name",
"description": "Logical secret name resolved by the operator's registry."
},
"key": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Key",
"description": "Optional key within a multi-value secret."
},
"version": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Version",
"description": "Optional secret version."
}
},
"additionalProperties": false,
"type": "object",
"required": [
"name"
],
"title": "SecretRef",
"description": "Reference to an operator-registered secret by logical name.\n\nAttributes:\n name: Logical name, e.g. ``inference/anthropic-prod``.\n key: Optional key within a multi-value secret.\n version: Optional secret version."
},
"SecurityScheme": {
"anyOf": [
{
Expand Down Expand Up @@ -23680,12 +23793,122 @@
"title": "Transcript retention days",
"description": "Days to retain step execution transcripts.",
"default": 30
},
"providers": {
"items": {
"$ref": "#/components/schemas/WorkflowInferenceProvider"
},
"type": "array",
"title": "Workflow provider catalog",
"description": "Operator-approved inference providers for workflow runs. Empty keeps legacy mode; non-empty switches to governed mode."
},
"secrets": {
"items": {
"$ref": "#/components/schemas/SecretBinding"
},
"type": "array",
"title": "Secret registry",
"description": "Maps logical secret names to backend bindings. Operator-only; never accepted from a request."
},
"default_provider": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Default provider",
"description": "Catalog entry used when a request names no provider (governed mode)."
},
"default_model": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Default model",
"description": "Model used when a request names none (governed mode)."
}
},
"additionalProperties": false,
"type": "object",
"title": "WorkflowEngineConfiguration",
"description": "Workflow engine configuration.\n\nAttributes:\n enabled: Whether the workflow engine is active.\n max_concurrent_workflows: Maximum simultaneous workflow executions.\n transcript_retention_days: Days to retain step transcripts."
"description": "Workflow engine configuration.\n\nAttributes:\n enabled: Whether the workflow engine is active.\n max_concurrent_workflows: Maximum simultaneous workflow executions.\n transcript_retention_days: Days to retain step transcripts.\n providers: Provider catalog. Empty means legacy mode; non-empty means\n governed mode, where unknown names deny.\n secrets: Registry mapping logical secret names to backend bindings.\n default_provider: Default catalog entry on workflow paths (governed mode).\n default_model: Default model on workflow paths (governed mode)."
},
"WorkflowInferenceProvider": {
"properties": {
"name": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9/_.-]{0,127}$",
"title": "Logical provider name",
"description": "Name callers send in the request."
},
"executor_type": {
"type": "string",
"enum": [
"openai",
"anthropic",
"claude",
"gemini",
"azure",
"bedrock"
],
"title": "Executor provider type",
"description": "cloud-agents provider type; must be one the executor supports."
},
"credential": {
"anyOf": [
{
"$ref": "#/components/schemas/SecretRef"
},
{
"type": "null"
}
],
"title": "Credential",
"description": "Logical ref of the one credential this entry uses."
},
"allowed_models": {
"anyOf": [
{
"items": {
"type": "string"
},
"type": "array"
},
{
"type": "null"
}
],
"title": "Allowed models",
"description": "Models callers may select. Omit for any model; an empty list is rejected (remove the provider to disable it)."
},
"base_url": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Base URL",
"description": "Operator-set endpoint for Azure and self-hosted models."
}
},
"additionalProperties": false,
"type": "object",
"required": [
"name",
"executor_type"
],
"title": "WorkflowInferenceProvider",
"description": "One entry in the workflow provider catalog.\n\nAttributes:\n name: Logical name callers send.\n executor_type: cloud-agents provider type this entry runs as.\n credential: The single credential of this entry (a logical ref).\n allowed_models: Models callers may use; None means any model.\n base_url: Operator-set endpoint (Azure and self-hosted); never\n settable by a caller."
}
}
},
Expand Down
29 changes: 18 additions & 11 deletions examples/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,17 @@ This folder is a self-contained, end-to-end worked example of the design in
lightspeed-stack as the sole policy/secret boundary in front of cloud-agents.

> **Status: target state, being built (feature-level TDD).** This folder
> defines where issue #51 and cloud-agents#269 end up. Phase 0a is in
> review (PR #57): caller `credentials_secret` -> 400, custom images,
> `advisory` and `spawn: none/local` need admin -> 403, size and count
> caps. The `lightspeed-stack.yaml` blocks marked `PLAN(stack#51)` do not
> load yet, and the catalog/policy behavior in the workflows does not exist
> yet. `verify.py` pins the contract (see "Verification"); each phase should
> make more of it real until the whole folder runs against a deployment
> (Phases 0b–6).
> defines where issue #51 and cloud-agents#269 end up. Landed so far:
> Phase 0a (caller `credentials_secret` -> 400; custom images, `advisory`
> and `spawn: none/local` need admin -> 403; size and count caps) and
> Phase 1 (the provider catalog, secret registry and defaults load, and
> `/v1/workflows/run` resolves providers through them: the stack builds the
> credential reference, the same-entry rule holds, overrides are rewritten to
> executor types). Blocks marked `PLAN(stack#51)` that belong to later
> phases (`policy`, MCP `secret_headers` / `workflow_enabled`) do not load
> yet. `tests/unit/cloud_agents/test_workflow_contract.py` runs `cases.yaml`
> against the real handler and config for the implemented phases and skips
> the rest; `verify.py` runs all of it against the reference model.

> **Auth prerequisite.** Role-based policy and the admin gates need an auth
> module that resolves roles: `jwk-token` with `role_rules` (used here) or
Expand Down Expand Up @@ -458,9 +461,13 @@ What it checks (~164 assertions):
registry, RBAC `resourceNames` match the K8s-bound Secrets, pod hardening,
digest-pinned images, and the dev and External Secrets sources agree.

`reference_gate.py` is a model of the design, not the implementation. As each
phase lands, run the same `cases.yaml` against the real endpoint and delete the
matching part of the reference.
`reference_gate.py` is a model of the design, not the implementation. Each case
in `cases.yaml` carries a `phase`. The unit test
`tests/unit/cloud_agents/test_workflow_contract.py` runs the cases up to
`IMPLEMENTED_PHASE` against the real handler and `WorkflowEngineConfiguration`
(status only; `reason` strings belong to the reference) and skips later ones.
Finishing a phase means raising `IMPLEMENTED_PHASE`, dropping the matching
`PENDING_ENGINE_KEYS`, and deleting that part of the reference.

## Production checklist

Expand Down
Loading
Loading