Drain Docker backlog on graceful stop

[iafred-bot]

Problem

When the Docker container is stopped while /var/piler/tmp still contains queued messages, piler-smtp and the worker process are stopped together. On the next startup Piler correctly replays that existing backlog, but new messages accepted just after startup can sit behind the old spool. In a large backlog this makes newly received mail look slow to appear even though startup recovery itself is working.

Change

This adds a drained stop path for intentional Docker stops:

• docker/start.sh traps SIGTERM and SIGINT instead of ending at sleep infinity.
• The trap calls /etc/init.d/rc.piler drain-stop by default.
• drain-stop stops piler-smtp first so no new mail is accepted, lets piler workers drain /var/piler/tmp, then stops the workers.
• PILER_STOP_DRAIN_TIMEOUT controls the wait timeout, defaulting to 300 seconds.
• PILER_STOP_DRAIN=0 and force-stop preserve the old immediate stop path.

Drain logs now expose stable start/end counters (initial_files, remaining_files, and elapsed) so operators can verify how much work was pending and whether the spool reached zero before shutdown.

Crash/forced-kill recovery remains unchanged: if files are still in /var/piler/tmp, the next startup still processes them.

For context, the GitLab fork/project where this was validated is https://gitlab.com/fredbcode-images/piler.

Before / after measurement

Validation used a test Docker image with this patch, an 8000-message backlog, and a live probe sent immediately after startup.

Check	Before: restart with backlog	After: drained stop, then restart
Backlog at startup	/var/piler/tmp still contained mail left by the interrupted run.	docker stop -t 600 drained 5330 remaining files in 221s before the container exited.
First backlog message	Reached both status=stored and the Manticore realtime index in 3.264s after restart.	No backlog replay was needed after restart because /var/piler/tmp was empty.
New mail sent just after startup	A live probe waited behind the backlog and reached status=stored plus Manticore in 89.533s from restart, 86.275s after the probe send finished.	A 20-message live probe reached the Manticore realtime index in 0.843s from send start and 0.006s after SMTP send completed.
State after capture	/var/piler/tmp still had 5864 files and Manticore had 2157 documents.	/var/piler/tmp was empty, Manticore had 8060 documents.

Validation

• bash -n docker/start.sh
• sh -n init.d/rc.piler.in
• git diff --check origin/master...HEAD
• isolated drain-log test: initial_files=2 to remaining_files=0
• isolated timeout-log test: initial_files=1 to remaining_files=1

Prepared by iafred-bot

GitLab: @fredbcode
GitHub: @fredbcode

[iafred-bot]

One additional problem addressed by this PR is mail safety during Docker image upgrades that recreate the container.

After docker compose pull, a subsequent docker compose up -d stops and recreates the service when the image changed. The pending workdir backlog is normally under /var/piler/tmp. In the default Compose layout it is part of the container writable layer, because only /etc/piler and /var/piler/store are persistent volumes. Therefore, accepted but not yet processed messages remaining in /var/piler/tmp could be removed together with the old container.

With this change, SIGTERM stops piler-smtp first and lets the Piler workers empty the backlog before the container exits. A normal Compose recreation can therefore start from an empty spool instead of discarding pending files.

If an administrator already mounts /var/piler/tmp (or the complete Piler workdir) on a persistent volume, the backlog already survives container replacement; the drain still avoids replaying it after startup and prevents new messages from waiting behind it.

Two limits remain:

• If the drain timeout is reached with files still pending, a subsequent recreation can still discard them unless the workdir is persistent.
• The first upgrade from an older, unpatched image cannot use this shutdown path, because the stop is handled by the currently running container. Its backlog should be emptied or persisted before that first recreation.

Prepared by iafred-bot

GitLab: @fredbcode
GitHub: @fredbcode

[iafred-bot]

Added start/end drain counters in 30cf3639: piler backlog drain started: initial_files=2, timeout=5s → piler backlog drain finished: initial_files=2, remaining_files=0, elapsed=1s.

Prepared by iafred-bot

GitLab: @fredbcode
GitHub: @fredbcode