Add tests and fixes for worker restarts, webhook batch limits, idempotency keys, and content-type parsing

[willy45590]

Summary

Add tests and fixes for worker restarts, webhook batch limits, idempotency keys, and content-type parsing

What was solved

#1527 — Add test for email queue worker crash-loop alert threshold (MAX_EMAIL_WORKER_RESTARTS)

Add unit tests proving the email queue worker's monitored restart loop behaves as documented: exactly MAX_EMAIL_WORKER_RESTARTS (5) restart attempts, correct exponential backoff durations, and worker_crash_total incrementing once per crash. The restart-loop logic currently lives inline in the binary's main.rs, so it must be extracted into a testable function/struct (e.g. in shutdown.rs or a new worker_supervisor.rs) without changing runtime behavior.

Addressed:

• Changed: src/shutdown.rs, src/worker_supervisor.rs
• Extract the restart-loop logic (backoff sequence, restart cap, FATAL logging, worker_crash_total increment) from main.rs into a testable function/struct, e.g. in shutdown.rs or a new worker_supervisor.rs.
• Unit test simulates repeated panics and asserts exactly 5 restart attempts (MAX_EMAIL_WORKER_RESTARTS cap).
• Unit test asserts correct exponential backoff durations for each restart attempt.

#1528 — Document and enforce a maximum SendGrid webhook payload event batch size

Add an explicit maximum event-count-per-webhook-request check to the SendGrid webhook handler in services/api/src/email/webhook.rs, rejecting oversized batches, emitting a metric for events-per-request distribution, and adding a test covering a payload at/over the new limit.

Addressed:

• Changed: services/api/src/validation.rs
• Add an explicit maximum event-count-per-webhook-request check in services/api/src/email/webhook.rs, rejecting (or chunking) batches that exceed the limit — distinct from the existing byte-size cap enforced by validation::request_size_validation_middleware.
• Emit a metric tracking the events-per-webhook-request distribution (e.g., histogram/counter of batch size).
• Add a test covering a payload at and over the new event-count limit.

#1529 — Verify idempotency.rs cache key derivation includes the idempotency-key header owner scope

Audit and, if needed, fix the idempotency cache key derivation in services/api/src/idempotency.rs so cached responses are scoped by route path and caller identity/API-key in addition to the raw Idempotency-Key header value. Add a test proving two distinct actors reusing the same idempotency key on the same route receive independent results, and document the key derivation scheme in the module doc comment.

Addressed:

• Changed: services/api/src/idempotency.rs
• Confirm or fix that the idempotency cache key includes route path and caller identity/API-key, not just the raw Idempotency-Key value
• Add a test with two distinct actors reusing the same idempotency key on the same route asserting independent results
• Document the key derivation scheme in the module doc comment

#1530 — Add explicit test for content_type.rs charset parameter handling edge cases

Add a table-driven test suite for services/api/src/content_type.rs that enumerates valid and invalid Content-Type header variants (including charset parameter edge cases and case-insensitivity) and asserts the expected accept/reject outcome, fixing any bug uncovered with a regression test.

Addressed:

• Changed: services/api/src/content_type.rs
• Add a table-driven test enumerating valid/invalid Content-Type header variants with expected accept/reject outcomes.
• Explicitly cover case-insensitivity (e.g. Application/JSON) and charset-parameter edge cases (empty charset value, trailing semicolons, unknown charsets).
• Fix any bug found in content_type.rs and add a regression test for it.

Changes

• src/shutdown.rs (create)
• src/worker_supervisor.rs (create)
• services/api/src/validation.rs (modify)
• services/api/src/idempotency.rs (modify)
• services/api/src/content_type.rs (modify)

Approach

1. Add test for email queue worker crash-loop alert threshold (MAX_EMAIL_WORKER_RESTARTS) #1527 — Add test for email queue worker crash-loop alert threshold (MAX_EMAIL_WORKER_RESTARTS) (Changed: src/shutdown.rs, src/worker_supervisor.rs)
2. Document and enforce a maximum SendGrid webhook payload event batch size #1528 — Document and enforce a maximum SendGrid webhook payload event batch size (Changed: services/api/src/validation.rs)
3. Verify idempotency.rs cache key derivation includes the idempotency-key header owner scope #1529 — Verify idempotency.rs cache key derivation includes the idempotency-key header owner scope (Changed: services/api/src/idempotency.rs)
4. Add explicit test for content_type.rs charset parameter handling edge cases #1530 — Add explicit test for content_type.rs charset parameter handling edge cases (Changed: services/api/src/content_type.rs)

Issues

Closes #1527
Closes #1528
Closes #1529
Closes #1530

[drips-wave]

@willy45590 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits