[Marketen]

Why

We want to publish web3signer.dnp.dappnode.eth (mainnet) from this repo instead of the archived DAppNodePackage-web3signer. The last attempt, in Nov 2024, broke existing users and was rolled back. I checked the published IPFS images for each release in that window:

On-chain version	Published	Source	What broke
2.0.9	2024‑11‑27 15:41Z	generic	The web3signer image was built before the "set envs to signer" commit landed. It shipped PGPASSWORD=password, POSTGRES_DB=web3signer, so web3signer couldn't reach the slashing DB.
2.0.10	2024‑11‑27 17:55Z	generic	Credentials were fixed, but postgres switched to Alpine (uid 70). Existing volumes are owned by the Debian postgres user (uid 999), so postgres crash-loops and flyway/web3signer never start.
2.0.11	2024‑11‑27 18:26Z	legacy repo	Rollback

main still builds the mainnet postgres from Dockerfile.alpine, so publishing mainnet from here today would break every existing user again.

Changes

• mainnet: postgres is built from Dockerfile.debian, the same postgres:14.1-bullseye base, uid 999 and PG 14.1-1.pgdg110+1 as the legacy package. This is the critical fix.
• flyway: migrates ${POSTGRES_DB} instead of the hardcoded web3signer database, and connects as ${POSTGRES_USER}. This path has been dormant because the schema has been at V12 for years. On the next slashing-protection migration it would fail with database "web3signer" does not exist for mainnet and gnosis, and web3signer would not start.
• web3signer: --http-host-allowlist=*, matching the legacy package since its 2.0.12. This affects all variants.
• postgres: adds the pg_isready healthcheck the legacy package had. This affects all variants.

With these changes, the release docker-compose.yml the SDK generates for mainnet matches the on-chain 2.0.17 compose except for image tags. I diffed both with docker compose config: volumes, user, healthchecks, depends_on and restart policies are all the same.

Testing

I built the mainnet variant with dappnodesdk build --variant mainnet (arm64) and tested it against the real on-chain 2.0.17 images (/ipfs/QmYRtGpa…L9My) in a harness that mimics the DAppManager:

• same compose project, so the same volume names;
• a dncore network with *.web3signer.dappnode aliases, plus the signer.mainnet.dncore.dappnode alias the stakers module adds;
• the global envs injected.

Upgrade 2.0.17 → this build → rollback to 2.0.17 → upgrade again: 35/35 checks passed

• On 2.0.17, imported a keystore and an EIP-3076 history through the keymanager API, then signed an attestation.
• After the upgrade:
  • postgres runs as uid 999 and is healthy.
  • flyway logs Database version: 12 … Exiting and exits 0.
  • web3signer is healthy, and /upcheck and /healthcheck respond.
  • The slashing DB is identical (version, validators, blocks, attestations, GVR), and the keyfiles and brain-db.json are preserved.
• Slashing protection uses the migrated history: a double vote on an epoch signed under 2.0.17, a surround vote, and a vote at the imported low watermark all return 412. A new valid attestation returns 200 and is persisted.
• The brain reaches web3signer through signer.mainnet.dncore.dappnode with no signer or DB errors.
• Rolling back to 2.0.17 on the same volumes works, and 2.0.17 refuses an epoch signed by the new build. Upgrading again also works.

Fresh install: 6/6 checks passed. Schema at V12, import, signing and slashing protection all work.

Flyway migration path: I started a DB at schema V11.

• The old main entrypoint fails with FATAL: database "web3signer" does not exist, and the DB stays at V11.
• The fixed entrypoint baselines at 11, applies V12 to web3signer-mainnet, and web3signer comes up healthy and signs.

Negative control: the same upgrade to the main build (Alpine postgres) reproduces the 2024 failure: initdb: could not access directory "/var/lib/postgresql/data": Permission denied, postgres and flyway crash-loop, and web3signer stays Created.

dappnodesdk build --all-variants passes. Gnosis, lukso and hoodi were only build-tested. Their runtime changes are the relaxed host allowlist, the healthcheck, and a flyway path that doesn't run while the schema is at V12.

Not covered: the tracker, which needs a live beacon node and crash-loops identically on 2.0.17 in the harness, and a real DAppNode with a running consensus client.

Before publishing mainnet

• Release only the mainnet variant (release_on_demand with variants: mainnet) from a single commit.
• Before signing, inspect the built hash and check that the postgres image is Debian/uid 999 and that the web3signer, flyway and postgres images all have PGPASSWORD=mainnet and POSTGRES_DB=web3signer-mainnet.
• Smoke-test the hash on a real DAppNode that is running 2.0.17 with keys.
• The mainnet manifest is at 2.0.19 while on-chain latest is 2.0.17, so check the version publish patch produces. Skipping a patch number is harmless.

Added since opening

• flyway waits for Postgres (up to 10 min) instead of exiting 1 while it starts (527a114):
  • web3signer depends on flyway completing, and the DAppManager rolls an update back when docker compose up errors.
  • On the real node, the published generic flyway exited 1 three times on first start. Compose v5 waited through it; older Compose versions abort up on that.
  • Every test hop now asserts flyway never restarts.

Validation of every release order (#70, #71, #72)

I tested every state main can end up in:

State	Contains
S1	#70
S2	#70 + #71
S3	#70 + #72
S4	all three

Local matrix (arm64), for mainnet, gnosis, lukso and hoodi:

• Each variant starts from its latest on-chain release (2.0.17 / 1.0.15 / 0.1.5 / 0.1.3), seeded with a key and an EIP-3076 slashing history.
• 6 upgrade chains per variant cover every merge order. Each ends with a rollback and a re-upgrade.
• Extras per state: fresh install, flyway V11→V12 migration, and an injected failure during the Postgres upgrade.
• At every hop I check: slashing DB identical, keyfiles and brain-db preserved, double/surround votes refused (412), new attestation signed and saved, flyway never restarted, allowlist accepts/rejects as expected.
• 41 runs, 2,075 checks, 0 failures.

Real DAppNode (amd64, DAppManager 0.2.129, Docker 29.9, Compose v5.6), installed through the DAppManager installer:

Test	Result
Live web3signer-hoodi 0.1.5 (20 remote keys in Lighthouse) → S4	✅ 31/31: all 14 container DNS names accepted (0.1.5 rejected 9), keys/brain-db/slashing DB unchanged, Lighthouse still sees its 20 remote keys, 0 rejections
lukso 0.1.5 (published bytes) → S4	✅ 17/17: Postgres 14→18 upgrade, data identical, slashing protection kept
lukso S4 → published 0.1.5 (downgrade)	✅ fails closed: old Postgres refuses the disabled cluster, web3signer never starts, nothing signs. DAppManager times out (15 min) and restores S4 by itself; 5/5 after
mainnet legacy 2.0.17 (on-chain registry install) → S1	✅ migration: same Postgres 14 cluster (uid 999), slashing DB identical, legacy keystore loaded, double/surround refused, flyway 0 restarts
mainnet S1 → S4	✅ 17/17: Postgres 14→18 of the cluster legacy 2.0.17 created, strict allowlist, all 13 container DNS names accepted

Release risk outside these PRs: DAppNode's IPFS gateways sometimes cut the trustless ?format=car download of large images partway (terminated). On the test node, 3 registry installs failed this way. I reproduced it once from another machine, and later attempts succeeded. It fails safe (nothing is installed), but expect some "download failed" reports right after publishing; the infra team should look at it.

🤖 Generated with Claude Code