Repository navigation
feat(eval): repo-shaped workflow tasks with CI-checked reference solutions - #2642
Merged
Merged
Conversation
…eps mode - ln: accept -n/-T/-v and long options; a directory destination (following a symlink unless -n) receives DIR/basename(TARGET); -sfn replaces a link to a directory; a real directory is never replaced (issue #1577 kept) - InMemoryFs: mkdir (non-recursive) and symlink require a directory parent, so the follow layer resolves a symlinked parent instead of creating a literal link/name entry; OverlayFs copies the parent up first - OverlayFs: file rename keeps the source mode, so sed -i on a mount_text script no longer drops its exec bit
- add -A/--all/-u stage the whole tree (they were taken as paths and staged nothing); other flags are ignored instead of treated as paths - commit parses -a/--all and short bundles like -am MSG, restaging every tracked file before committing - clone PATH|file://PATH [DIR] copies an in-VFS repo with the budgeted cp -R walker and sets origin to the source; refuses missing repos, non-empty destinations and destinations inside the source
New bashkit_repo eval (just eval-repo): 8 multi-turn repo tasks where a setup script builds a fixture git repo and the model runs make test, fixes the bug, and commits. Tasks combine local clone, symlinks (ln -sfn), PATH tweaks, background jobs with per-PID wait, jq/awk/sed, and git. - EvalTask gains optional setup (subshell, set -e, unscored); the eval Bash enables sandboxed git; agent and tests share build_task_bash - new checks: file_not_contains, symlink (relative or absolute target) - reference.rs replays one reference solution per task through the real checks; cargo test requires each to pass and the untouched fixture to fail
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
bashkit | 9adbce0 | Commit Preview URL Branch Preview URL |
Oct 08 2026, 01:31 PM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
bashkit_repo(just eval-repo,data/repo-workflow.jsonl, categoryrepo_workflow): 8 multi-step, repo-shaped tasks. Clone a fixture repo, runmake test, fix the failing code, commit; switch acurrent -> releases/vNsymlink; fix a tool onPATH=$PWD/bin:$PATH; shard work with&+wait $pid; jq config validation; release-branch version bump; a runner whose barewaithides failures. Kept separate frombashkit_bashso existing scores stay comparable.setupscript (builds the fixture git repo, symlinks, exec bits); new checksfile_not_containsandsymlink. Each task also checks the model didn't edit tests/golden files instead of code.reference_solutions_pass_all_checksruns a reference solution per task through bashkit and every check;untouched_fixture_does_not_passproves doing nothing scores zero.ln -sfn/-T/-v/ long options and GNU directory-destination rules (ln -sf f dirlinks insidedir).mkdir via/sublisted nothing before).sed -i(overlay rename) keeps the file mode, so an executable stays executable.git add -A/-u,git commit -am,git clone /local/path(orfile://) inside the sandbox.Why
Level-up roadmap, Step 5 "Harder evals": repo-shaped tasks that exercise symlinks,
PATH, background jobs and git together. (Python is out of scope for this roadmap, so no python tasks.)Before / After (verified)
releases/v2ln: invalid option -- 'n'releases/v2chmod +x bin/t; sed -i s/a/b/ bin/t; bin/t: main gavePermission denied, this branch runs it.git commit -am msg: main failed with "switch 'm' requires a value", this branch commits.Reference solutions: 8/8 tasks pass all checks; untouched fixtures: 0/8.
Risk
bashkit-evalnow enables thegitfeature (bashkit-cli and bashkit-capi already did).miratoolchain); git gaps that remain are listed inknowledge/integrations/git-support.md(no content tracking instatus, timestamp hashes,log --onelineignored).Checklist
cargo test -p bashkit-eval(42), lib, integration (+git), check-okfGenerated by Claude Code