The relay claims rows, never a cursor

Updated · View the entry on sijie.xyz ↗

Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.

The relay claims un-fanned rows with SELECT … WHERE fanned_out_at IS NULL AND poisoned_at IS NULL ORDER BY seq FOR UPDATE SKIP LOCKED, enqueues their jobs and sets fanned_out_at in one transaction. It never reads by a sequence cursor, because a cursor loses events.

How the relay runs

  • The relay is a loop in every process (internal/infra/events/run.go), not a River job. SKIP LOCKED already prevents a double fan-out, so no leader is needed. A frequent periodic job would only fill river_job with rows.
  • NOTIFY standmeet_events from the trigger and from Record wakes it. A 1-minute periodic job, events relay sweep, only pokes it; it covers a NOTIFY lost while the listener reconnects.
  • It claims at most 200 rows per pass. A 2,000-note import never becomes one huge transaction.
  • For each (event, subscriber) match it inserts a River job. It sets fanned_out_at and fanout (which job each subscriber got) in the same transaction.
  • Result: fan-out happens exactly once; handling is at least once.
  • A failing pass backs off: 2 s, doubling, capped at 1 minute. The failing batch is then retried row by row, so one bad row does not block the others. A row that fails 5 times is poisoned (poisoned_at), set aside and raises an alert; the owner puts it back in line from the event's detail in the Tasks panel (events.requeue). Delivery failures are retried by their own jobs (retry-has-one-owner).
  • After the fan-out commits, the relay sends NOTIFY standmeet_events_fanned with the event ids. A write receipt waits on it (AwaitFanout, async-response-contract).

The same SKIP LOCKED claim already runs the microsite build queue: builder-claim-skip-locked.

Coalescing is opt-in per subscription

Within one claimed batch, a subscription with Coalesce: true gets one job per subject, for the latest event. Only corpus.index sets it: it re-reads the note's current state, so the latest change covers the earlier ones. Webhooks and mail never coalesce: two events about one subject are two facts.

  • River unique jobs were rejected for this. River requires running in a unique job's ByState, so a change that arrived while its index job ran would merge into the running job and be lost.
  • An early build coalesced for every subscriber. It dropped supplier.connected when supplier.activated followed in the same batch. The UT TestEverySubjectEventReachesANonCoalescingSubscriber holds the fix.

The loss bug a cursor causes

The first relay design read seq > cursor. Sequence numbers are assigned when a transaction starts, but transactions commit in any order. Design review found the loss:

sequenceDiagram
  participant A as Transaction A
  participant B as Transaction B
  participant O as events
  participant R as relay (cursor version)
  A->>O: INSERT, gets seq 101 (uncommitted)
  B->>O: INSERT, gets seq 102
  B->>O: COMMIT
  R->>O: read seq > 100, sees only 102
  R->>R: cursor moves to 102
  A->>O: COMMIT, 101 only now visible
  Note over R,O: 101 < cursor, never read → lost

Fix: no cursor. Each row carries fanned_out_at. An unmarked row is always claimed eventually, so nothing can be skipped. A UT runs two interleaved transactions, and both events end up fanned out.

End to end: one corpus write

sequenceDiagram
  autonumber
  participant UC as Use case (any write path)
  participant DB as Postgres
  participant T as corpus_notes trigger
  participant R as Relay loop (every process)
  participant Q as River
  participant IX as Subscriber corpus.index
  participant WF as webhook.fanout
  participant WD as webhook.deliver
  UC->>DB: BEGIN, UPDATE corpus_notes …
  DB->>T: AFTER UPDATE (watched column changed)
  T->>DB: INSERT events(corpus.note.changed), pg_notify
  UC->>DB: COMMIT (change and event land or vanish together)
  DB-->>R: NOTIFY wake-up (fallback: 1-minute sweep)
  R->>DB: BEGIN, SELECT events … FOR UPDATE SKIP LOCKED
  R->>Q: InsertTx (one job per matching subscriber, coalesced where opted in)
  R->>DB: UPDATE events SET fanned_out_at, fanout, COMMIT (fan-out exactly once)
  R-->>DB: NOTIFY standmeet_events_fanned
  par in process
    Q->>IX: Handle(event)
    IX-->>Q: ok / error → River backoff retry
  and leaves the instance
    Q->>WF: Handle(event)
    WF->>WF: type globs and scope per endpoint
    WF->>Q: InsertTx webhook.deliver per endpoint (unique by args)
    Q->>WD: Work(endpoint, event)
    WD->>WD: take lease → sign → POST
    WD-->>Q: 2xx / non-2xx → retry on schedule
  end

If the relay crashes

Claim, enqueue and mark share one transaction. A crash mid-batch rolls all three back, and a restart claims the rows again. See message-loss-guarantees.

Tests

The events UTs (about 26 across relay, trigger and recorder): row claiming; interleaved commits lose nothing; batch size; poison marking; concurrent relays never fan out twice; a crash rolls back and re-claims; every event of a subject reaches a non-coalescing subscriber. See events-test-plan.