Consider: Encrypt secrets in pillar files at rest

[jpmckinney]

[Drafted by Claude]

Production credentials (database passwords, API keys, etc.) are stored in plaintext in the private pillar repository. This means:

• Anyone with read access to the repository has every production credential.
• A leaked personal access token, or a compromised laptop with a clone, exposes all secrets at once.
• Secrets persist in git history, so removing one requires rotation, not just deletion.
• GitHub itself is in the trust boundary: a GitHub account or infrastructure compromise is a credential compromise.

Since we deploy with salt-ssh (no master), pillar rendering happens on the machine running run.py, so encryption at rest in git protects the repository without changing the deployment trust model — whoever can run salt-ssh already has root SSH to all targets.

Proposal

Encrypt secret values with sops using age keys.

sops encrypts at the value level, so pillar files keep readable keys (db_password: ENC[...]) — diffs and PR review stay meaningful. age keypairs are single short strings (no GPG keyring), and multi-recipient encryption means each deployer's key can decrypt; offboarding is re-encrypting without that key (plus rotating what they could read).

• Add .sops.yaml listing deployers' age public keys and a path regex matching secret pillar files.
• Each deployer keeps their age private key in ~/.config/sops/age/keys.txt (outside the repo).
• Editing workflow: sops edit pillar/private/foo.sls.

Integration with run.py

Options:

1. Decrypt in the wrapper. Before invoking salt-ssh, decrypt matching pillar files into a gitignored directory; move the subprocess.run + sys.exit(returncode) into a try/finally that deletes the decrypted files.
2. sops exec-file. Let sops own the temp-file lifecycle and keep execvp untouched. Awkward with multiple secret pillar files.
3. Salt's GPG renderer instead of sops. Zero wrapper changes (#!yaml|gpg header, gpg_keydir in Saltfile), but whole-value GPG blobs, no meaningful diffs, and GPG key management for every deployer.

Option 1 is preferred.

Migration

1. Generate age keys for each deployer; commit .sops.yaml.
2. Encrypt existing secret pillar files with sops encrypt --in-place.
3. Update run.py per option 1; document setup in the deploy docs.
4. (TBD) Rotate all credentials currently in git history — they remain exposed to anyone who has ever cloned the repository, and encryption going forward doesn't retroactively protect them.

Non-goals

• Protecting the deployment path itself: salt-ssh copies rendered pillar data to targets over SSH regardless. The threat addressed here is repository/GitHub exposure.
• Vault or cloud secret managers: dynamic credentials and audit logs aren't worth running another critical service at our scale.

[RobHooper]

A potential solution is to use Hashicorp Vault to store sensitive pillar data.
A vault file would add AES-256 encryption limiting the risk exposure if anyone gained access to the data through GitHub or our physical devices.

The security consideration moves to how we store and share the decryption key / passphrase, it would potentially be needed on all deployments and queries with Salt (./run.py TARGET pillar.items etc).

I have used this module before with Salt, I remember it feeling clunky but working, however this was a few years ago.

The below extension enables this:

https://github.com/salt-extensions/saltext-vault

[jpmckinney]

Wouldn't Hashicorp Vault involve a central server? sops works offline.

[RobHooper]

Apologies I have mixed up software. I previously used Ansible Vault to store and share pillar private data.
Looking again there is not a native Salt integration that I can find for this.
I must have integrated this through procedure when I set this up before, and that may be why I remember it feeling clunky.

Ansible Vault creates an encrypted offline data blob to store data and secrets.
But compared to the Sops solution we would loose the data structure so I think Sops is still preferred.