Skip to content

feat: usage instructions and architecture drivers, with plugin eval tests - #14

Merged
HasanAlpCaferoglu merged 6 commits into
mainfrom
feat/usage_instructions
Nov 12, 2026
Merged

HasanAlpCaferoglu merged 6 commits into
mainfrom
feat/usage_instructions

Conversation

@mbroecheler

@mbroecheler mbroecheler commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this change?

Add usage instructions to start, and architecture drivers to requirements. Add the first behavior tests for the skills.

start

  • New section "Using DataSQRL": what DataSQRL fits, and what it does not fit (custom or ad hoc queries, strong transactional consistency, zero data copy).
  • New skill description and a routing table with three request kinds: questions about DataSQRL, building, and deploying.
  • A user who already has a requirements document also goes through the requirements check (section 4) and hand-off (section 5) before planning.

requirements

  • New "Architecture drivers" section with 7 drivers: sources, results, freshness, workload/budget/reliability, API latency SLA, authorization model, and "what, not how".
  • The section 4 checklist checks each driver. Each gap goes into ## Open Questions in the file.
  • The hand-off lists each open question on one line, with the selected default. The driver questions come first. The user can answer in the file or in the chat.
  • After a file edit, the skill checks the file again and asks about the gaps that remain. After a chat answer, it writes the answer into the file before it plans.

requirements-template.md

  • New: source ownership, a ## Results section with authorization, and a ## Hard constraints section. The API surface that the user defines counts as a hard constraint.
  • The non-functional requirements now have freshness, API latency SLA, workload, budget and reliability.

deploy and promote

  • deploy tells the user that a new deployment does not stop or pause existing ones.
  • promote says what promotion changes for APIs and data lake views, and asks for a clear confirmation.

Tests

  • New tests/claude-plugin-evals/: 8 claude plugin eval cases for requirements, start and patch. See tests/claude-plugin-evals/README.md.
  • A stub of datasqrl-agent.sh lets the patch case run without Docker. The stub is used only in a temporary copy of the plugin.
  • The cases cost tokens, so they are not in CI.

Version: 0.3.8. Main already has 0.3.7 from #18.

Area

  • Skills
  • Launcher (scripts/datasqrl-agent.sh)
  • Cloud client (scripts/datasqrl-cloud.sh)
  • Manifests
  • CI
  • Docs

How did you test it?

Test Result
.github/check-plugin.sh All checks passed (version 0.3.8, 9 skills)
python -m pytest tests 156 passed
claude plugin eval on skill-specific-tests/: 7 cases, 3 runs each, --threshold 0.8 All 7 cases pass. One grader (postgres-is-open-question) failed in 1 of 3 runs
run-stub-launcher-tests.sh on stub-launcher-tests/: 1 case, 3 runs Score 1.00

Commands:

claude plugin eval . --eval-dir tests/claude-plugin-evals/skill-specific-tests --scaffold --ablation none --allow-tools Write --threshold 0.8
tests/claude-plugin-evals/run-stub-launcher-tests.sh --threshold 0.8

Checklist

  • The PR title starts with a type such as feat:, fix: or docs:
  • .github/check-plugin.sh passes
  • Skills contain no SQRL and no HTML comments
  • A launcher change that needs a new agent image option waits until that image is published
  • README or DATA_HANDLING.md is updated if behavior or data handling changed

Comment thread skills/plan/SKILL.md Resolved
Comment thread skills/start/SKILL.md Resolved
Comment thread skills/plan/SKILL.md Resolved
Comment thread skills/plan/SKILL.md Resolved
- **Freshness**: the acceptable staleness of results, from a record arriving at the source to it showing in the result (seconds, minutes, hours, daily). This drives the architecture. If it is "realtime", say why minutes would not do.
- **API latency SLA**: per-query latency target (for example p99 < 200 ms) and request rate, average and peak.
- **Workload**: data volume and event rates, sustained and peak, expected growth, and what happens above peak.
- **Budget**: what running the pipeline may cost (per month), or the infrastructure it must fit.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the agent side, we probably need to update the need a cost-calculation skill and

- **Access**: public, authenticated, per-tenant? Which field carries the tenant.
- **Access**: which claim scopes each query, mutation and subscription (see Results).

## Test data and acceptance criteria

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In agent side, I guess we need to have some kind of mapping to meet the requirements set in the ## Test data and acceptance criteria. What I mean by mapping, which config we need to use while building the SQRL so that agent can apply those ones. We have a PR that focus on the cost optimization for some configs but it doesn't cover completely. For example, what should agent do if API latecy SLA is set as (p99 < 200 ms)?

@HasanAlpCaferoglu
HasanAlpCaferoglu marked this pull request as ready for review November 12, 2026 21:00
@HasanAlpCaferoglu HasanAlpCaferoglu changed the title feat: usage instructions and architecture drivers feat: usage instructions and architecture drivers, with plugin eval tests Nov 12, 2026
@HasanAlpCaferoglu
HasanAlpCaferoglu merged commit 2b73bd9 into main Nov 12, 2026
4 checks passed
@HasanAlpCaferoglu
HasanAlpCaferoglu deleted the feat/usage_instructions branch November 12, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants