Saturation: slow down, queue, alert

Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.

Under saturation the system slows down, queues and alerts. It never loses data, never crashes the process and never starves visitor requests. Each case has its own UT, simulated with an injected fault that asserts the degraded behaviour. The saturation suite as built covers: pool exhausted, disk full with relay backoff, endpoint 429 cooldown, Meili down then drained, backlog surge alert, waiter cap, queue starvation and job timeout.

Saturation states

stateDiagram-v2
  [*] --> normal
  normal --> saturated : pool / workers / rate limit / disk / downstream full
  state saturated {
    [*] --> slow_down
    slow_down --> queue : new jobs deferred, never dropped
    queue --> alert : backlog or oldest age over threshold
  }
  saturated --> normal : pressure eases, backlog drains by itself
  saturated --> writes_refused : disk full, transaction fails
  writes_refused --> normal : space recovered
  note right of writes_refused : business write and event fail together, readable error, never a half-write

Cases, behaviour, UT assertions

Saturation	Expected behaviour	UT asserts
Connection pool exhausted	A worker's wait for a connection times out and the job counts as retryable; the reserve for visitor requests is always kept	With the pool full, a job does not crash and turns retryable; the reserved request connections stay available
All workers busy	New jobs queue; no new goroutines; queues do not starve each other	With the webhook queue fully blocked, index jobs still finish on time
Downstream rate limit hit (endpoint 429)	Snooze until Retry-After without consuming an attempt; the endpoint's failing_since is set, so new deliveries to it wait out the 5-minute cooldown	Repeated 429s never exhaust a job into discarded; no extra requests go out during cooldown
Our own mail throttle hit	Snooze until the window ends; never drop	Mails over the limit all go out in the next window, same count
Disk full / Postgres refuses writes	The business write and its event fail together with a readable error; the relay backs off (2 s, doubling, capped at 1 min) instead of spinning	No "changed but no event"; relay retry interval grows
Meili fully down	Index jobs back off and retry (the Meili client's own retries are off); writes are unaffected; the backlog drains after recovery	After recovery every backlogged job completes, none discarded (within the retry window)
Backlog surge (bulk import)	Same-subject index jobs coalesce within a batch (webhooks do not); events_backlog fires above 1,000 unfanned or 5 min	Job count within the bound; the alert flag is set
In-request waiters full	Requests over the cap return the receipt at once instead of queueing	The 65th waiter immediately gets indexed: false
CPU saturation / job timeout	The hard timeout cancels the job, which turns retryable; other queues are unaffected	The timed-out job is cancelled and can be rescued

The mechanisms behind each row live in sibling nodes: snooze, cooldown and backoff in retry-has-one-owner; queue limits, pool budget, timeouts and the waiter cap in concurrency-control; coalescing and alert thresholds in storage-bounds; the business write and its event sharing one transaction in two-sources-of-events. The suite is counted in events-test-plan.