[TheSaiEaranti]

On macOS, ssm-exec.sh works once per $TMPDIR. Every call after that fails before it reaches AWS:

mktemp: mkstemp failed on /var/folders/.../T//ssm-cmd-XXXXXXXXXX.json: File exists

The other HyperPod skills send their ad hoc node commands through hyperpod-ssm (for example the ssm-exec.sh calls in hyperpod-node-debugger's references), so those fail after the first call too. Their bundled diagnostic scripts make their own SSM calls and aren't affected.

Related

No existing issue. I found this while reading the hyperpod-ssm skill.

Changes

Two things combine here:

• BSD/macOS mktemp only replaces trailing X's, so the ssm-cmd-XXXXXXXXXX.json template creates a file with that literal name. The template is now ssm-cmd.XXXXXXXXXX. The AWS CLI reads file:// parameters without needing a .json extension.
• Both aws ssm start-session calls ran through exec, which replaces the shell, so the trap 'rm -f "$TMPFILE"' EXIT cleanup never ran and the payload file stayed behind. On Linux GNU mktemp randomizes the name, but every call still left a file in /tmp holding the remote command or the base64 of an uploaded file. exec came in with the unbuffer change in feat(sagemaker-ai): add HyperPod debugging skills #169, and unbuffer attaches its PTY either way, so both calls now run as child processes, the way the script invoked aws before feat(sagemaker-ai): add HyperPod debugging skills #169, and the trap fires. With set -e, a failing aws call still exits the script with the same status.

One behavior change from dropping exec: a signal sent only to the script's own PID (a plain kill <pid>, or a SIGKILL from a caller's timeout) now stops bash and leaves aws running until the remote command ends. Before, that signal went straight to unbuffer or aws. Ctrl-C and GNU timeout signal the whole process group, so they behave as before, and the trap now removes the file.

I also bumped sagemaker-ai to 1.3.1 in the marketplace and both plugin manifests, following the patch-bump rule in the maintainers guide.

Added tools/test_ssm_exec_sh.py. It runs the script with a stub aws that records each --parameters payload. PATH holds only the stubs plus links to the tools the script needs, so a real aws or unbuffer on the machine is never picked up, and each case checks which branch ran:

• with and without unbuffer, three calls in a row each exit 0, send {"command": ["hostname"]}, and leave $TMPDIR empty
• with and without unbuffer, an aws exit status of 254 is returned, and the temp file is still removed
• with unbuffer, a call still works next to a leftover ssm-cmd-XXXXXXXXXX.json from an older version

All 5 cases fail on main. With only the template changed they still fail (the file leaks). With only exec removed, the leftover-file case fails on macOS.

Verification:

• uv run --with pytest pytest tools plugins: 137 passed
• bandit -r . -x ./.tmp,./node_modules: no issues
• semgrep scan --oss-only --config=tools/semgrep/rules-vendored.yaml: no new findings. The test's subprocess import pairs nosec B404 with nosemgrep: gitlab.bandit.B404, as hyperpod_issue_report.py in this plugin does.
• ruff check and ruff format --check (v0.15.0, as in pre-commit) on the new test, dprint check, the lint:manifests ajv checks on the three manifests, and node tools/validate-cross-refs.cjs: clean (the one cross-refs warning, about migration-to-aws, is on main too)

nccl-diagnose.sh uses the same kind of suffixed template (kubectl-check-XXXXXX.err, nccl-ssm-XXXXXX.json). A normal run removes those files, but _ssm_run is called inside $(...), so a run interrupted during an SSM call can leave nccl-ssm-XXXXXX.json behind, and on macOS later runs would then fail at mktemp. I left it alone to keep this PR to one script, and can send a follow-up.

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.