Concurrency: bounded, time-limited, stoppable workers

Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.

All background work runs in workers that are bounded, time-limited and stoppable. Bare go func in application code no longer exists.

Queues and limits

flowchart LR
  R[relay loop · every process · ≤ 200 rows per pass] --> QI
  R --> QN
  R --> QW
  MCP[jobs.fetch_new] --> QF
  subgraph pool["worker pool (each queue has its own limit, none starves another)"]
    QI["index queue<br/>MaxWorkers 4 · timeout 30s"]
    QN["notify queue<br/>MaxWorkers 2 · timeout 30s"]
    QW["webhook queue<br/>MaxWorkers 8 · timeout 15s<br/>≤ 1 in flight per endpoint"]
    QM["maintenance queue<br/>MaxWorkers 1<br/>periodic jobs"]
    QF["fetch queue<br/>MaxWorkers 3<br/>one job per job source"]
  end
  QI --> MS[(Meili)]
  QN --> SM[SMTP · suppliers]
  QW --> EP[external endpoints]
  QF --> JB[job boards]

Queue	MaxWorkers	Job timeout	What runs there
`index`	4	30 s (reindex 10 min)	`corpus.index`, `corpus.reindex`, the build-settled subscribers
`notify`	2	30 s	`owner.notify`, the approval and confirmation mails, `supplier.invoke`
`webhook`	8	15 s (fan-out 30 s)	`webhook.fanout`, `webhook.deliver`; at most 1 in flight per endpoint
`maintenance`	1	10 min	periodic jobs
`fetch`	3	per kind	`jobs.fetch_source` (added in P4)

The numbers are starting values, tuned by measurement.

Separate queues, separate limits. A slow external endpoint can only fill the webhook queue. It cannot slow indexing or mail. A UT blocks the webhook queue and asserts that index jobs still finish.
At most one in flight per endpoint: a lease row. webhook.deliver runs UPDATE webhook_endpoints SET busy_until = now() + 20 s WHERE id = $1 AND enabled AND (busy_until IS NULL OR busy_until < now()) RETURNING …. If no row comes back, it snoozes 2 s (no attempt spent). The lease outlives the job timeout, so a crashed worker's lease expires on its own. It holds across processes.
- pg_try_advisory_xact_lock was rejected: a transaction-scoped lock would hold a transaction, and its connection, across the POST.
The relay takes at most 200 rows per pass, in every process, with no leader. A 2,000-note import never becomes one huge transaction. See relay-claims-rows-not-cursor.
Queues a request waits on poll every 100 ms (jobs.QueueAwaited: index, notify). The other queues keep River's 1 s. River sends one insert notification per queue per FetchCooldown (100 ms). A write's second index job (the wiki note, ~5 ms after the raw one) sent none, so after the worker's fetch it waited for the 1 s poll: ~0.5 s per write. Now a receipt costs the poll (≤ 100 ms) plus River's batch completer, which records completions every 250 ms and is not configurable.

Connection budget: no network call inside a transaction

sequenceDiagram
  participant W as webhook worker
  participant DB as Postgres
  participant E as external endpoint
  rect rgba(200,80,60,0.08)
  note over W,E: Wrong — HTTP inside a transaction
  W->>DB: BEGIN (holds a connection + locks)
  W->>E: POST (slow, 15 s)
  E-->>W: 200
  W->>DB: COMMIT
  note over W,DB: 8 workers × 15 s = pool drained, visitor requests queue
  end
  rect rgba(80,140,90,0.08)
  note over W,E: Right — lease, read, release the connection, then HTTP
  W->>DB: take lease (one UPDATE, autocommit)
  W->>DB: read event + secret (short reads, connection returned at once)
  W->>E: POST (holds no connection)
  E-->>W: 200
  W->>DB: settle (release lease, clear failing_since)
  end

pgxpool MaxConns is 40. Half is reserved for requests. Boot check: if the sum of MaxWorkers plus the relay (today 4 + 2 + 8 + 1 + 3 + 1 = 19) exceeds half of MaxConns (20), the server refuses to start. A misconfiguration surfaces at boot, not as production queueing.

Timeouts and cancellation

Every job kind has a hard timeout (above). On expiry its context is cancelled and the job counts as retryable.
The webhook HTTP timeout (10 s) is shorter than the job timeout (15 s), so a clear network error arrives first instead of the whole job being killed.
A panic in a worker is caught by the job layer and counted as a failure. It cannot crash the process.

Graceful shutdown (upgrades recreate containers)

sequenceDiagram
  participant D as docker / updater
  participant S as backend
  participant R as relay loop
  participant Q as job workers
  participant W as running job
  D->>S: SIGTERM
  S->>R: stop the relay first
  S->>Q: stop claiming new jobs
  Q->>W: wait for running jobs, up to 20 s
  alt finished within 20 s
    W-->>Q: completed / retryable
  else timeout
    Q->>W: cancel ctx
    Note over Q,W: job stays running, rescued and retried after restart (at least once)
  end
  S-->>D: exit

Backpressure for in-request waits

"Wait up to 2 s for the index" is served by shared LISTEN connections (pgstore.Listener: one per channel per process) that fan out notices, not one polling connection per request. At most 64 waiters per channel. Beyond that the request returns indexed: false at once instead of queueing. See async-response-contract and completion-hooks.

No bare goroutine

The gate check-no-bare-goroutine.sh allows a go statement only in internal/infra/** and cmd/server/** (non-test). Domain code and internal/routes may not use one; to do work in the background, enqueue a job or declare a jobs.Periodic. No exclusion list. See no-bypass-by-structure.
Process plumbing that must own a goroutine lives in infra: detach.Go owns a goroutine and absorbs its panic (the block mount warm uses it); hostsocket.ListenWith starts its own accept loop.
Violations cleared before the gate landed: routes/admin/obsidian.go (goroutine deleted), plugin/adapters/invoke_background.go (deleted), routes/hostdesk/hostdesk.go and agentcore/hostops.go (accept loop moved into hostsocket), plugin/mount/mounted_warm.go (detach.Go).
The gate was shown red once on a planted go func in a scratch copy before it landed.