Repository navigation
ci: add workflow validating plugins and skills - #8
Merged
Merged
Conversation
Runs claude plugin validate (marketplace and each plugin, --strict) and the skills-ref Agent Skills validator on every push and pull request, alongside the existing structural check in ValidateMarketplace.cs. skills-ref validate takes one skill at a time and rejects `invocable`, a real Claude Code frontmatter field the portable spec does not define, so scripts/validate_skills.py walks every SKILL.md, allowlists that field, and reads as UTF-8 so it gives the same answer on Windows as in CI. Closes #6 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- marketplace.json catalogued a `testing` plugin at ./plugins/testing, a directory that no longer exists, while plugins/playwright-testing was catalogued nowhere. Repoint the entry and record the rename so existing installs migrate. - adr.md's argument-hint was a quoted scalar followed by trailing content, which is invalid YAML — the command was loading with empty metadata. - The opentelemetry skill carried `version` and `tags`, which are in neither schema. Move them under the spec's free-form `metadata` map. Document all four validators in the README contributing section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`invocable` is not a Claude Code frontmatter field; the runtime only knows `user-invocable` and `disable-model-invocation`, so the key was silently ignored and the six skills meant as background knowledge were still listed in the `/` menu. The first cut of scripts/validate_skills.py allowlisted `invocable`, which hid the defect from CI. - `invocable: false` -> `user-invocable: false` (6 skills) - `invocable: true` dropped (4 skills; true is the default) - validate_skills.py now allowlists only the extension fields the Claude Code docs define, so a misspelled or invented key fails the build - add the issue 6 plan file per the implement-feature convention Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6
Adds
.github/workflows/validate.yml, running on push tomain, every PR, andworkflow_dispatch. Four layered checks:claude plugin validate . --strictclaude plugin validate plugins/<p> --strict(loop)python scripts/validate_skills.pySKILL.mdagainst the Agent Skills specdotnet run scripts/ValidateMarketplace.csThe Claude Code CLI tracks
latest, with a one-line pin available viaCLAUDE_CODE_VERSIONif an upstream change ever breaks CI.skills-refis not published to PyPI, so it installs from a pinned commit.Why
scripts/validate_skills.pywraps skills-refskills-ref validatetakes one skill path at a time, so something has to walk the tree regardless. The wrapper also handles two deliberate deviations:user-invocable,disable-model-invocation,argument-hint,context,hooks, …). The wrapper adds exactly the documented set, so every other skills-ref check — required fields, name/directory match, description length — still gates CI, while a misspelled or invented key (which Claude Code silently ignores at load time) fails the build.SKILL.mdas UTF-8.skills_ref.validator.validate()uses the platform default encoding, which crashes on non-ASCII skills under Windows cp1252. Reading explicitly means the script gives the same answer locally as in CI.Output is GitHub
::errorannotations plus a step summary, exiting non-zero on any failure.Pre-existing failures fixed to get the workflow green
The new checks surfaced four real defects:
marketplace.jsoncatalogued atestingplugin at./plugins/testing, a directory that no longer exists, whileplugins/playwright-testing/was catalogued nowhere — a half-finished rename from cfaac80. Repointed the entry and added"testing": "playwright-testing"torenamesso existing installs migrate.plugins/architecture/commands/adr.mdhadargument-hint: "<decision-description>" [--issue <number>]— a quoted scalar followed by trailing content, which is invalid YAML. The whole command's frontmatter was silently loading as empty metadata at runtime.plugins/dotnet/skills/opentelemetry-dotnet-instrumentation/SKILL.mdcarried strayversionandtagskeys, in neither schema. Moved under the spec's free-formmetadatamap rather than deleted.invocable: true|false, which is not a Claude Code field. The runtime only knowsuser-invocableanddisable-model-invocation, so the key was ignored and the six skills meant as background knowledge (adr-writing,csharp-coding-standards,microsoft-extensions-*,package-management,aspire-integration-testing) were still offered in the/menu. The first cut of this PR allowlistedinvocable, hiding the defect. Nowinvocable: false→user-invocable: false,invocable: truedropped (it is the default), and the allowlist holds only documented fields.README's contributing section documents all four commands and the
skills-refinstall. The plan file is at.claude/plans/2026-08-28-issue-6-validate-skills-workflow.md.Verification
All four steps run clean locally and in CI:
The failure path is exercised too — a skill with a bare
invocablekey produces the annotation, the summary table, and exit 1, while one usinguser-invocable,disable-model-invocation, andargument-hintpasses.🤖 Generated with Claude Code