Storage bounds: every runaway has a hard limit

Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.

The queue lives in the database, so the main risk is unbounded storage. Every way storage can run away has a hard bound and is visible. None of them rely on "it should get cleaned up".

When each row is deleted

stateDiagram-v2
  state events_row {
    [*] --> unfanned : trigger / Record
    unfanned --> fanned : relay claims it and sets fanned_out_at
    unfanned --> poisoned : relay failed on it 5 times
    poisoned --> unfanned : events.requeue (Tasks panel)
    fanned --> [*] : fanned out and older than 7 days → hourly events retention deletes it
  }
  state river_job_row {
    [*] --> in_progress : relay enqueues (index jobs coalesce per subject within a batch)
    in_progress --> completed : success
    in_progress --> discarded : permanent failure or attempts exhausted
    completed --> [*] : River cleaner deletes after 24 h
    discarded --> [*] : River cleaner deletes after 7 days
  }
  state endpoint {
    [*] --> enabled
    enabled --> failing : a delivery fails (failing_since set)
    failing --> enabled : a delivery succeeds
    failing --> disabled : 5 days of continuous failure
    disabled --> enabled : owner re-enables
    note right of disabled : once disabled the fan-out enqueues no new deliveries to it
  }

Runaways and their controls

Runaway	How it happens	Control	How it is enforced and seen
events grows forever	Nothing deletes rows, or the relay is stuck and un-fanned rows pile up	The periodic job events retention runs hourly and deletes rows that are fanned out and at least 7 days old. Unfanned and poisoned rows stay: they are the backlog.	The overview shows the backlog, the poisoned count, the age of the oldest un-fanned event and the table size. Alert events_backlog above 1,000 unfanned or oldest over 5 min; alert events_poisoned for any poisoned row. The alert is on the panel, not an email: mail may be the very thing failing.
Finished river_job rows pile up	Completed jobs stay in the table	River's job cleaner on the elected leader, at River's defaults: completed rows after 24 h, discarded rows after 7 days	The overview shows the table size. Alert jobs_discarded while discarded jobs exist.
Retries to a dead endpoint pile up	The receiving site is down for a long time	MaxAttempts 18. While failing_since is set, new deliveries are scheduled 5 min out. The endpoint is disabled after 5 days of continuous failure, and a disabled endpoint gets no new deliveries.	Endpoint state and disable reason are shown in admin.
A bulk operation explodes fan-out	One Obsidian import of 2,000 notes	The relay takes at most 200 rows per pass. corpus.index coalesces per subject within a batch. Webhooks do not coalesce or debounce: each event is a fact, delivered once per subscribing endpoint.	e2e events-bulk-import-bound.
No-op updates emit events	Only updated_at was touched	The trigger has a WHEN clause: it writes only when a watched column really changed	A spec asserts the event sequence before a sentinel event, a positive assertion.
One bad event stalls the pipeline	The relay fails on the same event repeatedly	A failing batch is retried row by row, so one failing row does not block the others. A row that fails 5 times is poisoned and raises an alert; delivery failures are retried by their own jobs.	Same as above: the backlog is visible.
Dead-tuple bloat	High-churn updates on queue tables leave MVCC row versions until vacuum	events has lower autovacuum thresholds (scale factor 0.02). river_job keeps River's settings, because River owns that DDL. The volume is a few thousand rows a day.	The overview shows the table sizes.

Backlog, oldest-event age, table sizes and alerts surface in the tasks-panel overview.

Precedent

The visitor-traffic table would also grow without bound. It is held by a 24 h retention job (visitor traffic retention). The event and job tables follow the same pattern.

Other options do not avoid it

• A Redis queue grows the same way, and our Redis evicts silently when full.
• Kafka has built-in retention by time or size, at the cost of running another service.

See why-not-a-broker.

Acceptance

Backlog and table size are visible in admin (e2e tasks-panel, tasks-panel-more); the bulk-import bound has its own e2e (events-bulk-import-bound); retention and the backlog alert have UTs. See events-roadmap · events-test-plan.