Skip to content

feat(spec-loop): add Grok headless harness profile - #1593

Merged
potiuk merged 2 commits into
apache:mainfrom
KatalKavya96:feat-grok-spec-loop
Oct 11, 2026
Merged

potiuk merged 2 commits into
apache:mainfrom
KatalKavya96:feat-grok-spec-loop

Conversation

@KatalKavya96

Copy link
Copy Markdown
Contributor

Summary

  • Add xAI Grok as a first-class headless harness for tools/spec-loop.
  • Map the existing spec-loop contract onto Grok Build 1.0.50 using --prompt-file, --cwd, --always-approve, output/model/effort flags, and explicit deny rules for remote mutation.
  • Add fixture coverage and update the spec-loop, Grok adapter, registry, and vendor-neutrality documentation so the implementation and declared harness support stay in sync.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other: spec-loop shell runner / headless harness support

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other:
    • bash tools/spec-loop/tests/test_runner_fixtures.sh
    • bash -n tools/spec-loop/loop.sh
    • bash -n tools/spec-loop/lib.sh
    • bash -n tools/spec-loop/tests/test_runner_fixtures.sh
    • live-tested with Grok Build 1.0.50 in a disposable repository
    • verified the headless profile can create a local file while leaving unrelated tracked files unchanged
    • live-verified the exact Bash(git push:*) and Bash(gh:*) deny rules block matching commands even with --always-approve

RFC-AI-0004 compliance

  • HITL — any new mutation is gated on explicit user confirmation
  • Sandbox — no new unrestricted host access; Grok spec-loop continues to rely on the external OS sandbox, with per-invocation deny rules as defense in depth
  • Vendor neutrality — Grok is added through the existing harness-neutral spec-loop contract and vendor-neutrality metadata is regenerated
  • Conversational + correctable — agentic-override path documented if behaviour is adopter-tunable
  • Write-access discipline — the runner explicitly blocks git push and gh from the Grok invocation; the spec-loop remains local-only and does not autonomously open or push PRs
  • Privacy LLM — private content does not reach a non-approved LLM; redactor invoked where needed

Linked issues

Refs #1416

Notes for reviewers (optional)

The Grok profile was validated against Grok Build 1.0.50 rather than inferred from another harness.

The runner maps:

  • assembled prompt -> --prompt-file
  • repository root -> --cwd
  • unattended local execution -> --always-approve
  • SPEC_LOOP_OUTPUT_FORMAT=text -> plain
  • SPEC_LOOP_OUTPUT_FORMAT=stream-json -> streaming-json
  • SPEC_LOOP_MODEL -> --model
  • SPEC_LOOP_EFFORT=low|medium|high -> --reasoning-effort low|medium|high

The exact deny rules used in the runner were also exercised against the installed CLI:

  • Bash(git push:*)
  • Bash(gh:*)

Both were denied before execution under --always-approve.

A disposable-repository probe was used instead of running a destructive full spec-loop build against the Magpie checkout. The profile successfully made the requested local edit and did not modify unrelated tracked files.

@github-actions github-actions Bot added family:tools tools/* family:docs Docs, MISSION.md, READMEs substrate:framework-dev Tool substrate: build / validate / eval the framework itself labels Oct 10, 2026
@KatalKavya96

Copy link
Copy Markdown
Contributor Author

@potiuk You can post this as the maintainer-facing PR comment:
This adds the Grok spec-loop harness slice from #1416.
I verified the profile live against Grok Build 1.0.50, including headless local edits, output/model/effort mapping, and the exact git push / gh deny rules under --always-approve.
Full prek run --all-files passes locally. The change stays within the existing spec-loop harness contract and does not introduce a Grok-specific sandbox profile; the external OS sandbox remains the primary isolation layer.

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — Grok slots into the existing harness switch with a --prompt-file / --cwd / --always-approve profile, and its deny rules are exactly Claude's (Bash(git push:*), Bash(gh:*)), live-verified on 1.0.50 to block under --always-approve. The fixture test pins every flag plus the text-default path without effort or model (the fake agent truncates its log per run, so the "absent" assertions are real), and the README, spec matrix, known gaps, registry and vendor-neutrality index all agree. Good call keeping the docs explicit that the denies are defence in depth on top of the OS sandbox. CI is green.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@potiuk
potiuk merged commit d916316 into apache:main Oct 11, 2026
10 checks passed
@KatalKavya96
KatalKavya96 deleted the feat-grok-spec-loop branch October 11, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

family:docs Docs, MISSION.md, READMEs family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants