Global asset pool: one store, reference accounting

Updated · View the entry on sijie.xyz ↗

Verdict (2026-09-05): media is one owner-scoped pool (assets) plus one edge table (asset_references) that says who uses what. References are recomputed from content on every save, never hand-maintained per action. Delete is guarded: a referenced asset refuses to go, and the refusal names the referrers. The dead media_assets table (three genre foreign keys, zero writers) is gone.

Motivating problem. Before 2026-09-05 an asset belonged to its holder — storage_key = <holder_id>/<asset_id>, "delete holder → delete its assets in the same transaction" (the old comment still sits above the table, backend/db/schema.sql:560-578). That shape made reuse impossible (a second entry wanting the same image uploaded a second copy) and made delete destructive once sharing existed: deleting a writing deleted its blobs outright (corpus/usecase/writings_delete.go:1-8 records the old behaviour). The owner flagged reference bookkeeping as the part that is easy to get wrong. The dead table media_assets — three genre foreign keys and no writer — is recorded as removed at schema.sql:304; no code, migration or test names it any more.

The design

  • One pool, owner-scoped. assets(owner_id NOT NULL, holder_id nullable, kind image|attachment, storage_key, content_type, sha256, …) — schema.sql:585-600. holder_id survives only as a migration breadcrumb (which note an old image came from); it is never consulted for "in use" (schema.sql:582-584).
  • asset_references(asset_id, referrer_kind, referrer_id) — schema.sql:608-615, primary key over all three columns, ON DELETE CASCADE as belt-and-braces (the guard means a delete never reaches a referenced row). Two referrer kinds: corpus and microsite (corpus/entity/asset.go:37-38).
  • Recompute on save. The scanner is ScanAssetReferences (corpus/usecase/asset_refs.go:36-50): one regex over the body for standmeet-asset:<uuid> tokens (a UUID — universally unique identifier — or a pending-* placeholder during multipart save). Every note write funnels through RebuildNoteRefs (corpus/usecase/wiki_crosslink.go:109-121), which rebuilds crosslinks and asset references in one hook, so a new write site cannot forget one of them. The recompute (note_asset_refs.go:47-67) sets the note's references to exactly body tokens + cover. Writings: RebuildWritingAssetRefs after the save transaction commits (writings_save.go:142). Microsites: RebuildMicrositeAssetRefs after every build, scanning all source files for the same token (note_asset_refs.go:138-151, wired at cmd/server/boot_wireup_microsites.go:76).
  • Diff, not wipe, for corpus. Attachments (kind=attachment) are attached/detached explicitly and never cited in the body; an asset whose holder is this note stays attached whether or not the body cites it. So the corpus recompute drops only a reused image reference the body no longer cites (staleImageRef, note_asset_refs.go:95-101). A blanket delete-and-reinsert would drop the entry's attachments on the next unrelated save. A microsite has no attachment concept, so its recompute is a plain replace.
  • Delete guard. DeletePoolAsset (corpus/usecase/asset_pool.go:67-82) counts references and returns ErrAssetReferenced; the op layer turns that into 409 Conflict with the message still used by N corpus entries and M microsites — remove those first (corpus/ops/asset_pool.go:93-126). Only an unreferenced asset is removed — blob first, row second. A deleted note or writing frees its references (note_assets.go:279, writings_delete.go:21); the asset stays in the pool.
  • Storage. Blobs live in object storage speaking S3 (the Amazon object-storage protocol) — the minio service in infra/deploy/docker-compose.yml, which since 2026-09-26 runs Silo (pgsty/silo), the maintained community fork of MinIO, after MinIO deleted its own images from Docker Hub — behind infra/storage/storage.go. The body never stores a URL — only the stable token. Every read goes through the backend: GET /api/v1/assets/{id} reads the bytes over the internal network and streams them on the instance's own origin, so storage is never exposed to the browser (routes/public/microsite_assets.go). Two ways in: an asset a microsite references is public and served on a bare id; any other asset needs the signature (?e=&s=) that only an authorized render mints (corpus/usecase/asset_url_token.go), else 404 — so the public endpoint cannot enumerate the pool, and a leaked id never bypasses an entry's ACL. (This replaced a 302 to a 1-hour presigned storage URL, which needed storage to be publicly reachable.)
flowchart LR
  note["corpus note / writing
body: standmeet-asset:id + cover"] -- "save: RebuildNoteRefs" --> refs
  site["microsite source
AssetWidget asset=standmeet-asset:id"] -- "build: RebuildMicrositeAssetRefs" --> refs
  refs["asset_references
(asset_id, referrer_kind, referrer_id)"] --> pool[("assets (owner pool)")]
  del["assets.pool_delete"] -- "refs > 0: 409, names referrers" --> pool
  pool -- "backend reads + streams (signed id)" --> minio[("object storage (Silo)")]

Faces

  • Editor — cite for reuse. PoolReuse in app/src/components/admin/sections/corpus/CorpusAssetsPanel.tsx:88-116: open the pool, pick an asset uploaded elsewhere, its token lands in the body, and the next save references this entry too (4f2634619; corpus-asset-pool-reuse-ui.spec.ts). The insert path is the same whether the asset is on this entry or reused from the pool.
  • Admin — Resources → Assets. /admin/assets (AssetsSection.tsx) is the pool viewer + the guarded delete; the 409 message is what the error toast shows. The sibling /admin/data (DataSection.tsx) is the per-microsite NoSQL store manager, not assets — the resources nav group is microsites · assets · data (app/src/lib/admin/nav.ts:60-63).
  • MCP ops. Per-entry attach/detach: assets.upload / assets.delete (corpus/ops/assets.go). The pool: assets.list / assets.references / assets.pool_delete (corpus/ops/asset_pool.go). The admin routes dispatch the same ops (routes/admin/corpus_crud.go:39-46), and the owner-toolset norm lists all five (e2e/test/norm-outward-toolset.spec.ts:107, :151) — service-handle parity.
  • SDK. AssetWidget (sdk/packages/react/src/widgets/AssetWidget.tsx) takes the full standmeet-asset:<uuid> token on purpose: the same scanner reads it out of built source, so a live page's asset is guarded like a note's (microsites).

The honest ceiling

  • The guard is read-then-delete, not one atomic statement (asset_pool.go:64-66 says so) — fine for a single-owner instance where the only writer of references is the owner; make the delete conditional on no references existing if real concurrency arrives.
  • asset_references is a derived index outside the write transaction (like note_refs): a crash between the row write and the recompute leaves references one save stale, corrected at the next save — stale, not corrupt.
  • The migration leaves orphan blobs in MinIO for assets whose holder no longer resolved (2026-09-05-global-assets.sql:17-19) — invisible, never reclaimed; there is no sweeper.
  • An attachment is invisible to the content scanner by design; its reference lives and dies by explicit attach/detach only.

Standing points

  • asset_references is the only authority for "in use"; holder_id never is.
  • Referrer first, then the asset — the delete order is a product rule, not a cascade.
  • One token grammar for every referrer (note body, writing body, microsite source): standmeet-asset:<uuid>. Same shape as backlinks-as-rebuilt-edge-tables — the edge table is rebuilt from the body, never hand-maintained — and the same taste as ephemeral-over-stateful: derive, don't maintain.

Built 2026-09-05. d81d2782a (pool + asset_references + guard; migration 2026-09-05-global-assets.sql backfills owner_id from the holder note and seeds one corpus reference per existing image), 79136cac5 (recompute-on-save for notes and writings; Assets + Data admin sections), 836af0fea (AssetWidget, microsite recompute, public asset route), 4f2634619 (cite-for-reuse in the editor). Guards: global-assets-guard.spec.ts (referenced → 409 naming corpus; de-reference → 204 and gone), asset-reference-recompute.spec.ts (cite/uncite, cover set/clear, swap A→B moves the reference — not both, not neither; note delete frees; a shared asset refuses until every referrer is gone), assets-manager-ui.spec.ts, corpus-asset-pool-reuse-ui.spec.ts, microsite-asset-widget.spec.ts, plus the older genre-assets*.spec.ts family (per-genre attach, inherit, reader, admin). Writings' zip import/export still carries assets its own way — writings-import-export. Design seed: docs/design/global-assets.md.

Written 2026-09-07 against standmeet-new main 36789537d (v0.1.31).

Related notes