Skip to content

docs: add an agent workflow laboratory and experiment playbook - #7

Merged
R0SEWT merged 2 commits into
mainfrom
docs/agent-workflow-lab
Sep 27, 2026
Merged

R0SEWT merged 2 commits into
mainfrom
docs/agent-workflow-lab

Conversation

@R0SEWT

@R0SEWT R0SEWT commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Why

Project-kit needs an explicit place to experiment with agent workflows and a repeatable route to promote validated changes.

Changes

  • docs/agent-workflow-lab.md: laboratory playbook. Covers isolated worktrees, verifying the skill source actually loaded, promotion through small PRs, failed experiments and a proposed notebook pilot.
  • docs/experiments/TEMPLATE.md: experiment record (evidence) template.
  • README.md: short pointer section.

Refs #6 (the proposed pilot). It does not close it: the pilot has not run.

Validation

  • git diff --check passes.
  • Applies cleanly on main (b9818bf) and merges with docs: define capability versions, consumer pins and rollback #8 without conflicts.
  • Local Markdown links resolve.
  • CI steps run locally: shell syntax, JSON manifests and scaffold dry-run pass. ShellCheck is left to CI; no scripts change.
  • Documentation-only: no runtime or agent-host execution claimed.

Limits

The pilot has not run. This documents the first workflow; it adds no CLI command, and scaffolding does not create branches.

🤖 Generated with Claude Code

Summary by Sourcery

Establish a documented workflow laboratory for safely evaluating and promoting agent capabilities.

New Features:

  • Add a documented, opt-in laboratory workflow for experimenting with agent skills, playbooks, and runbooks before promoting validated changes.
  • Add a reusable experiment record template for capturing hypotheses, evidence, decisions, and rollback plans.

Enhancements:

  • Document verification of the loaded skill source, failure handling, promotion practices, and a proposed notebook-environment pilot.

Documentation:

  • Link the new laboratory playbook and experiment template from the README.

Refs #6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @R0SEWT, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 days and 17 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@sourcery-ai

sourcery-ai Bot commented Sep 26, 2026

Copy link
Copy Markdown

Reviewer's Guide

This documentation-only PR establishes an opt-in agent workflow laboratory: experiments run in isolated worktrees with verified loaded sources and recorded evidence, while validated capabilities are promoted via focused PRs with compatibility and rollback guidance. It also proposes a notebook pilot without claiming that the pilot has been run or adding runtime behavior.

Flow diagram for the agent workflow laboratory

flowchart LR
    Start["Start from clean checkout"] --> Lab["Create or reuse isolated lab worktree"]
    Lab --> Verify["Verify loaded skill source and commit"]
    Verify --> Record["Record hypothesis, baseline, evidence, and rollback"]
    Record --> Evaluate["Run experiment and document failures or recovery"]
    Evaluate --> Decision{"Validated?"}
    Decision -->|No| Reject["Record rejection or next hypothesis"]
    Decision -->|Yes| Promote["Create focused promotion branch or PR"]
    Promote --> Checks["Revalidate checks, behavior, compatibility, and rollback"]
    Checks --> Merge["Review and merge through normal protection"]
Loading

File-Level Changes

Change Details Files
Adds a documented, opt-in workflow for experimenting with agent capabilities and promoting validated results.
  • Defines worktree and lab-branch setup, source-loading verification, experiment isolation, and failure handling.
  • Documents validation, promotion through small PRs, compatibility communication, and rollback expectations.
  • Proposes—but does not execute—a notebook-environment pilot with explicit success criteria.
docs/agent-workflow-lab.md
Introduces a structured evidence record for repeatable experiments and integration decisions.
  • Captures ownership, commits, host and loaded skill source, consumer environment, scope, baseline, acceptance criteria, evidence, decisions, and rollback.
  • Requires observed results, recovery cases, redacted logs, and links to existing tracking issues rather than duplicating task backlogs.
docs/experiments/TEMPLATE.md
Makes the laboratory workflow discoverable while clarifying its manual and non-invasive behavior.
  • Adds links to the laboratory playbook and experiment template.
  • States that the workflow is opt-in and that scaffolding neither creates branches nor installs capabilities.
README.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Comment thread docs/agent-workflow-lab.md Outdated
```bash
git status --short
git fetch origin
git worktree add -b lab/agent-workflows ../project-kit-agent-lab origin/main

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

git worktree add -b lab/agent-workflows ../project-kit-agent-lab origin/main makes the lab branch track origin/main (branch.autoSetupMerge default; verified: @{upstream} = origin/main).

In the lab worktree, the repo's mandatory session-close steps (git pull --rebase then git push, CLAUDE.md) rebase the lab branch onto main, which rewrites the history this guide says not to rewrite. git push then fails under the default push.default=simple because the upstream name doesn't match. With push.default=upstream it would push to main. Fix: add --no-track, and publish with git push -u origin lab/agent-workflows.

Automated review (Claude Code, AI-generated). Validate before acting.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1643e95: added --no-track and git push -u origin lab/agent-workflows, with the reason. (Claude Opus 5.5, on behalf of Rody)

Comment thread docs/agent-workflow-lab.md Outdated
Run experiments against disposable fixtures or a dedicated consumer worktree.
An agent may read the checked-out documentation immediately, while an installed
plugin may still be loading its cached release. Record the actual source path
and commit used. Follow the host's reload procedure in a fresh session and

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Follow the host's reload procedure" can't load the experimental skill. The installed plugin comes from the project-kit-local marketplace (the main checkout), cached under .../project-kit/0.1.0/, so the ../project-kit-agent-lab worktree is never read.

A fresh session keeps loading the cached 0.1.0 skills (install record: gitCommitSha b9818bf), so the "verify the experimental skill is loaded" check fails, or the release gets evaluated by mistake. The guide should name the real mechanism: start the session with claude --plugin-dir ../project-kit-agent-lab, or bump a prerelease version in the worktree and point a separate marketplace at it.

Automated review (Claude Code, AI-generated). Validate before acting.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1643e95: the guide explains that the plugin is served from its cache and loads the lab copy with claude --plugin-dir ../project-kit-agent-lab. (Claude Opus 5.5, on behalf of Rody)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@R0SEWT
R0SEWT merged commit 8f9c613 into main Sep 27, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant