Skip to content

ci: add workflow validating plugins and skills - #8

Merged
ardalis merged 4 commits into
mainfrom
feature/validate-skills-workflow
Aug 30, 2026
Merged

ardalis merged 4 commits into
mainfrom
feature/validate-skills-workflow

Conversation

@ardalis

@ardalis ardalis commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #6

Adds .github/workflows/validate.yml, running on push to main, every PR, and workflow_dispatch. Four layered checks:

Step Covers
claude plugin validate . --strict marketplace catalog schema
claude plugin validate plugins/<p> --strict (loop) each plugin manifest plus the frontmatter of its skills, agents, and commands
python scripts/validate_skills.py every SKILL.md against the Agent Skills spec
dotnet run scripts/ValidateMarketplace.cs catalog ↔ manifests ↔ on-disk layout agreement

The Claude Code CLI tracks latest, with a one-line pin available via CLAUDE_CODE_VERSION if an upstream change ever breaks CI. skills-ref is not published to PyPI, so it installs from a pinned commit.

Why scripts/validate_skills.py wraps skills-ref

skills-ref validate takes one skill path at a time, so something has to walk the tree regardless. The wrapper also handles two deliberate deviations:

  • Allowlists the Claude Code frontmatter extensions. The portable spec permits six fields; Claude Code documents another fourteen (user-invocable, disable-model-invocation, argument-hint, context, hooks, …). The wrapper adds exactly the documented set, so every other skills-ref check — required fields, name/directory match, description length — still gates CI, while a misspelled or invented key (which Claude Code silently ignores at load time) fails the build.
  • Reads SKILL.md as UTF-8. skills_ref.validator.validate() uses the platform default encoding, which crashes on non-ASCII skills under Windows cp1252. Reading explicitly means the script gives the same answer locally as in CI.

Output is GitHub ::error annotations plus a step summary, exiting non-zero on any failure.

Pre-existing failures fixed to get the workflow green

The new checks surfaced four real defects:

  • marketplace.json catalogued a testing plugin at ./plugins/testing, a directory that no longer exists, while plugins/playwright-testing/ was catalogued nowhere — a half-finished rename from cfaac80. Repointed the entry and added "testing": "playwright-testing" to renames so existing installs migrate.
  • plugins/architecture/commands/adr.md had argument-hint: "<decision-description>" [--issue <number>] — a quoted scalar followed by trailing content, which is invalid YAML. The whole command's frontmatter was silently loading as empty metadata at runtime.
  • plugins/dotnet/skills/opentelemetry-dotnet-instrumentation/SKILL.md carried stray version and tags keys, in neither schema. Moved under the spec's free-form metadata map rather than deleted.
  • Ten skills used invocable: true|false, which is not a Claude Code field. The runtime only knows user-invocable and disable-model-invocation, so the key was ignored and the six skills meant as background knowledge (adr-writing, csharp-coding-standards, microsoft-extensions-*, package-management, aspire-integration-testing) were still offered in the / menu. The first cut of this PR allowlisted invocable, hiding the defect. Now invocable: false → user-invocable: false, invocable: true dropped (it is the default), and the allowlist holds only documented fields.

README's contributing section documents all four commands and the skills-ref install. The plan file is at .claude/plans/2026-08-28-issue-6-validate-skills-workflow.md.

Verification

All four steps run clean locally and in CI:

### step: marketplace     exit=0
### step: each plugin     exit=0   (5 plugins, --strict)
### step: skills          exit=0   19/19 skills valid.
### step: structure       exit=0   Checked 5 plugins, 19 skills, 11 agents, 11 commands.

The failure path is exercised too — a skill with a bare invocable key produces the annotation, the summary table, and exit 1, while one using user-invocable, disable-model-invocation, and argument-hint passes.

🤖 Generated with Claude Code

ardalis and others added 4 commits August 26, 2026 16:28
Runs claude plugin validate (marketplace and each plugin, --strict) and the
skills-ref Agent Skills validator on every push and pull request, alongside the
existing structural check in ValidateMarketplace.cs.

skills-ref validate takes one skill at a time and rejects `invocable`, a real
Claude Code frontmatter field the portable spec does not define, so
scripts/validate_skills.py walks every SKILL.md, allowlists that field, and
reads as UTF-8 so it gives the same answer on Windows as in CI.

Closes #6

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- marketplace.json catalogued a `testing` plugin at ./plugins/testing, a
  directory that no longer exists, while plugins/playwright-testing was
  catalogued nowhere. Repoint the entry and record the rename so existing
  installs migrate.
- adr.md's argument-hint was a quoted scalar followed by trailing content,
  which is invalid YAML — the command was loading with empty metadata.
- The opentelemetry skill carried `version` and `tags`, which are in neither
  schema. Move them under the spec's free-form `metadata` map.

Document all four validators in the README contributing section.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`invocable` is not a Claude Code frontmatter field; the runtime only knows
`user-invocable` and `disable-model-invocation`, so the key was silently
ignored and the six skills meant as background knowledge were still listed
in the `/` menu. The first cut of scripts/validate_skills.py allowlisted
`invocable`, which hid the defect from CI.

- `invocable: false` -> `user-invocable: false` (6 skills)
- `invocable: true` dropped (4 skills; true is the default)
- validate_skills.py now allowlists only the extension fields the Claude
  Code docs define, so a misspelled or invented key fails the build
- add the issue 6 plan file per the implement-feature convention

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ardalis
ardalis merged commit 4df81fd into main Aug 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a GitHub Workflow that validates skills

1 participant