Drain the spool on a graceful stop, and persist it - #31
Merged
Merged
Conversation
On SIGTERM the wrapper now waits for piler to empty its workdir before stopping it, bounded by PILER_STOP_DRAIN_TIMEOUT. piler-smtp is already down at that point, so nothing refills the spool. The spool also becomes a volume: it held mail accepted but not yet archived, and a container recreation used to drop it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #30 (
supervise-piler-daemons), which puts both piler daemons under supervisord. Retarget tomainonce that merges.Problem
Two ways mail accepted by
piler-smtpcould sit unarchived, or disappear:pilerto finish/var/piler/tmp. The spool was left full, and mail arriving after the next start queued behind that backlog./var/piler/tmpwas not a volume. A container restart kept it, but a recreation — every image update — discarded it, losing mail already accepted with a 250.Upstream jsuto/piler#493 addresses the first half for the reference image; this is the same idea adapted to supervisord.
Change
config/piler-run.shtrapsTERM/INTand waits for the spool to empty before stoppingpiler.piler-smtphas a higher supervisord priority so it is already down, meaning nothing refills the spool while we wait. AfterPILER_STOP_DRAIN_TIMEOUTit stops piler anyway, thenSIGKILLs it if it hasn't exited within 15s.pilerconf -q workdir, not a hardcoded path.PILER_STOP_DRAIN/PILER_STOP_DRAIN_TIMEOUT/PILER_STOP_DRAIN_INTERVAL(1/300/2) areENVin the image.PILER_STOP_DRAIN=0keeps the old immediate stop. Names match upstream #493 so switching torc.piler drain-stoplater is a rename.[program:piler]getsstopwaitsecs=3600: the wrapper bounds the wait itself, supervisord must notSIGKILLmid-spool./var/piler/tmpis added toVOLUME, withpiler_spoolwired indocker-compose.yml.stop_grace_period: 360sin compose. The engine default is 10s, far below the 300s drain default — the effective window ismin(grace, PILER_STOP_DRAIN_TIMEOUT).ns8-piler needs the matching changes: declare the
/var/piler/tmpvolume (otherwise podman creates a fresh anonymous one per container, defeating the point) and setTimeoutStopSecon the unit so the drain has time to run.Validation
Built with podman, run through
docker-compose.yml:stop -t 300piler-run: spool drained, spool empty, all 2000 rows inmetadataSIGSTOPed, 50 mails queued,PILER_STOP_DRAIN_TIMEOUT=10draining spool, 50 file(s) left, thengiving up with 50 file(s) left after 10s, then the 15s wait andSIGKILLrm -f+up) with those 50 still spooledPILER_STOP_DRAIN=0validate.ymlgains a step sending 100 mails, stopping with-t 120, asserting the drain ran in the logs, then restarting and asserting all 100 are archived.