Repository navigation
fix: silence a shellcheck false positive in a proof script - #77
Draft
runewright[bot] wants to merge 7 commits into
Draft
runewright[bot] wants to merge 7 commits into
runewright[bot] wants to merge 7 commits into
Conversation
FullSend takes one task to a merge-ready pull request while the owner reads nothing, and asks once, through the question tool, before any merge. PruneTests deletes tests that no plausible bug can fail, repairs weak ones with a recorded mutation, and hands its pull request to FullSend. Co-Authored-By: Martin Zeman <N4M3Z@users.noreply.github.com>
PreferExistingTools, ProofsShowTheRealApp, and AskBeforeMovingData replace three workshop drafts that carried machine-made names. AnswerBeforeEditing is new. The WhenTheUserSignals draft stays out, because PlainSpokenLanguage already says it. Co-Authored-By: Martin Zeman <N4M3Z@users.noreply.github.com>
The skill told the model to set a directory with gh -R. gh and glab take a remote OWNER/REPO there, so the model wrote gh pr create -R with a local path. The skill now keeps -R for jj and sends a local directory through its own cd call. An eval case found the defect. Co-Authored-By: Martin Zeman <N4M3Z@users.noreply.github.com>
scripts/eval-rune assembles a rune's Claude Code instance, wraps it in a throwaway plugin, and runs its cases through Claude Code's plugin eval with the rune and without it. A rule loads through a session-start hook. Every clean run writes a record under evals/<Rune>/records/, and --record adds a GIF that shows both answers of every case. --canon leaves out a skill's variants. Suites for fourteen runes, each tuned until its cases can fail, with records on claude-haiku-4-5. Graders are deterministic. Vale skips the case files, which quote the prose a rune forbids. Co-Authored-By: Martin Zeman <N4M3Z@users.noreply.github.com>
The variant ran the guard probe as a load-time shell injection. Where Bash was not granted, Claude Code refused to load the skill, so even its fallback for a denied probe never reached the model. On the same cases and six turns, the skill with the variant scores 0 and the canon alone scores 0.44 against a baseline of 0. The canon runs the same probe as an ordinary step and records unknown when it is denied. Co-Authored-By: Martin Zeman <N4M3Z@users.noreply.github.com>
The change's proposal, design, tasks, and three specifications, the recorded eval runs with their transcript, and the changelog entries. Co-Authored-By: Martin Zeman <N4M3Z@users.noreply.github.com>
The trap string assigns status before it reads it, and shellcheck reports SC2154 anyway. On main this fails the commit stage for every file, which a new branch runs on its first push. Co-Authored-By: Martin Zeman <N4M3Z@users.noreply.github.com>
github-actions
Bot
added
size:xlarge
|
review/correctness stood down on |
|
review/correctness stood down on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for
change/runes-carry-eval-records: the session agent fills this body beforerune sign open.Plan
Why this shape, one or two sentences.
Changes
Testing
Release Notes
Note
Add eval suites, FullSend and PruneTests skills, and silence a shellcheck false positive
scripts/eval-rune) that validates a rune suite, assembles a private instance, and packages it with eval cases in a temporary plugin, plus--showoutput and usage handlingevals/, comparing runs with and without the runeFullSendandPruneTestsskills, Claude-specific FullSend instructions, and four core rules (AnswerBeforeEditing,AskBeforeMovingData,PreferExistingTools,ProofsShowTheRealApp)SafetyFirst/claude/Workflow.mdvariantrecord.shfor the trap status-variable false positiveMacroscope summarized 2d4b82f.