Skip to content

Drain the spool on a graceful stop, and persist it - #31

Merged
stephdl merged 2 commits into
mainfrom
drain-on-stop
Aug 25, 2026
Merged

stephdl merged 2 commits into
mainfrom
drain-on-stop

Conversation

@stephdl

@stephdl stephdl commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #30 (supervise-piler-daemons), which puts both piler daemons under supervisord. Retarget to main once that merges.

Problem

Two ways mail accepted by piler-smtp could sit unarchived, or disappear:

  • On stop, nothing waited for piler to finish /var/piler/tmp. The spool was left full, and mail arriving after the next start queued behind that backlog.
  • /var/piler/tmp was not a volume. A container restart kept it, but a recreation — every image update — discarded it, losing mail already accepted with a 250.

Upstream jsuto/piler#493 addresses the first half for the reference image; this is the same idea adapted to supervisord.

Change

  • config/piler-run.sh traps TERM/INT and waits for the spool to empty before stopping piler. piler-smtp has a higher supervisord priority so it is already down, meaning nothing refills the spool while we wait. After PILER_STOP_DRAIN_TIMEOUT it stops piler anyway, then SIGKILLs it if it hasn't exited within 15s.
  • The spool path comes from pilerconf -q workdir, not a hardcoded path.
  • PILER_STOP_DRAIN / PILER_STOP_DRAIN_TIMEOUT / PILER_STOP_DRAIN_INTERVAL (1 / 300 / 2) are ENV in the image. PILER_STOP_DRAIN=0 keeps the old immediate stop. Names match upstream #493 so switching to rc.piler drain-stop later is a rename.
  • [program:piler] gets stopwaitsecs=3600: the wrapper bounds the wait itself, supervisord must not SIGKILL mid-spool.
  • /var/piler/tmp is added to VOLUME, with piler_spool wired in docker-compose.yml.
  • stop_grace_period: 360s in compose. The engine default is 10s, far below the 300s drain default — the effective window is min(grace, PILER_STOP_DRAIN_TIMEOUT).

ns8-piler needs the matching changes: declare the /var/piler/tmp volume (otherwise podman creates a fresh anonymous one per container, defeating the point) and set TimeoutStopSec on the unit so the drain has time to run.

Validation

Built with podman, run through docker-compose.yml:

Scenario Result
2000 mails sent in 5s, 1175 in the spool, stop -t 300 piler-run: spool drained, spool empty, all 2000 rows in metadata
Archiver SIGSTOPed, 50 mails queued, PILER_STOP_DRAIN_TIMEOUT=10 5 × draining spool, 50 file(s) left, then giving up with 50 file(s) left after 10s, then the 15s wait and SIGKILL
Container recreated (rm -f + up) with those 50 still spooled all 50 archived after start, spool empty — they were lost before this change
PILER_STOP_DRAIN=0 immediate stop in 3s, no drain

validate.yml gains a step sending 100 mails, stopping with -t 120, asserting the drain ran in the logs, then restarting and asserting all 100 are archived.

On SIGTERM the wrapper now waits for piler to empty its workdir before
stopping it, bounded by PILER_STOP_DRAIN_TIMEOUT. piler-smtp is already
down at that point, so nothing refills the spool. The spool also becomes
a volume: it held mail accepted but not yet archived, and a container
recreation used to drop it.
@stephdl
stephdl changed the base branch from supervise-piler-daemons to main August 25, 2026 13:19
@stephdl
stephdl merged commit 6148f84 into main Aug 25, 2026
2 checks passed
@stephdl
stephdl deleted the drain-on-stop branch August 25, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant