Retry has one owner: the job layer

Status: released in v0.1.76 (2026-09-27) — design and as-built record in docs/design/event-bus-outbox-webhooks.md in the StandMeet repo.

Only the job layer retries. A handler only classifies a failure; the policy declared on the job kind decides when to try again.

Before vs after

flowchart LR
  subgraph today["Before: three layers each retried, failures multiplied"]
    direction TB
    g[invoke_background · retry.Do] --> m[mail_retry · notifyPolicy] --> h[httpx · auto-retries 5xx/429]
  end
  subgraph after["After: only the job layer retries"]
    direction TB
    j[job layer · backoff per policy + jitter] --> k[handler · only classifies the failure] --> n[httpx · NoRetry]
  end
  today ==>|consolidate| after

3 × 3 × 3 = one failure became up to 27 requests. Now only the job layer retries:

internal/infra/retry and notifyPolicy are deleted. The gate check-retry-only-in-jobs.sh forbids retry.Do outside internal/infra/jobs (see no-bypass-by-structure).
Webhook delivery runs httpx with NoRetry.
The Meili client's own retries are disabled (meilisearch.DisableRetries()); corpus.index jobs retry instead.

Synchronous calls are sent once

A call a visitor waits on (calendar free/busy, insert and delete through the openapi adapter) no longer runs retry.Do inside the request. A transient failure maps to ErrCalendarUnavailable at once. Reason: one retry owner, and a visitor-facing call answers fast. A retry that matters, such as the compensating calendar delete, is a durable supplier.invoke job.

Failure classification: what a handler returns

stateDiagram-v2
  [*] --> running
  running --> completed : nil
  running --> retryable : any other error (timeout · connection failure · 5xx · 408 · 429 without Retry-After)
  running --> snoozed : jobs.Snooze (429 / 503 + Retry-After · endpoint busy · mail cap spent)
  running --> discarded : jobs.Discard (other 4xx · SSRF-blocked URL · endpoint disabled or gone · event pruned)
  retryable --> running : when due (backoff + jitter)
  snoozed --> running : at the time named, no attempt spent
  retryable --> discarded : attempts exhausted
  discarded --> running : owner clicks retry in the panel
  completed --> [*]
  discarded --> [*]

Every 4xx except 408 and 429 is permanent. Retrying would not help and would only hammer the receiver.
A snooze does not consume an attempt. A webhook snooze from Retry-After is capped at 10 h.
On River, jobs.Discard(err) is a cancel-with-error and shows as discarded; a cancel from the panel shows as cancelled.

Policy (declared on the job kind, as data)

Kind	Max attempts	Backoff	When exhausted
`corpus.index`, `corpus.reindex`	10	`DefaultBackoff`: from 1 s, doubling, capped at 15 min	`discarded` + alert; the panel can retry
`owner.notify`, `access_request.approval_mail`, `owner.email_confirmation`	8	20 s, then ×3 per attempt: about 6 h	`discarded` + alert (on the panel, not another email)
`webhook.fanout`	10	`DefaultBackoff`	`discarded` + alert
`webhook.deliver`	18	Svix: 5 s · 5 min · 30 min · 2 h · 5 h · 10 h · 10 h… (about 5.3 days)	`discarded`; the endpoint is disabled after 5 days of continuous failure
periodic jobs	no retry	the next period	the panel shows the last failure

Every backoff adds ±10% jitter, so a batch of jobs that failed together does not retry at the same instant.

Do not hammer a dead endpoint

sequenceDiagram
  participant F as webhook.fanout
  participant Q as queue
  participant WD as webhook.deliver
  participant E as endpoint (down)
  WD->>E: deliver event 1
  E--xWD: 503
  WD->>Q: event 1 retryable, endpoint failing_since set
  F->>Q: events 2, 3… enqueued 5 min out (cooldown)
  Q->>WD: event 1 due on its backoff
  WD->>E: deliver event 1
  E-->>WD: 200
  WD->>Q: failing_since cleared, new deliveries run at once again

Any non-success sets failing_since (410 and 429 too); a success clears it.
At most one delivery per endpoint is in flight, through a lease row (busy_until); see concurrency-control.

Retries with side effects: can it send twice?

Webhook: every retry carries the same webhook-id (the event id). Consumers dedupe on it. See webhooks.
Mail: SMTP has no idempotency key. The order is "send, then mark" (notified_at, replied, or the deleted booking notice); a crash in between sends a duplicate. That is the price of at-least-once, far better than "lost on failure". The message carries Message-ID <event id@standmeet>, and most mailboxes merge duplicates by it.
Mail throttle: the per-recipient cap (30 per hour) snoozes to the next window instead of dropping. The owner-notify burst cap (5 per owner per hour) stays a deliberate drop; see message-loss-guarantees.
Search index: an upsert, naturally idempotent.

Manual retry

The tasks-panel retries a single job; webhooks.redeliver re-delivers every discarded job of one endpoint after the receiver is fixed. Covered: the Phase 2 "sink returns 500 twice" case, 429 with Retry-After → retried later without consuming an attempt, and 410 → discarded immediately.