Storage

What holds what, and the retention that actually binds — STORAGE.md

Where the big things go, and whether the console needs object storage

Asked directly: "note on the console architecture, evaluate if we want or need to add object storage to store the bigger blob or text data (or maybe it's already there via Loki and Tempo)."

Short answer: it is already there, the console needs none of its own, and the thing that will actually bite is retention rather than size. The numbers below are measured on 2026-09-07, not estimated.

What holds what, today

The console's own database, broken down:

history/rendering   145 rows   521 kB   largest row 5.6 kB
history/deploy       91 rows    40 kB   largest row  515 B
database total                8.6 MB

The console stores no files and mounts no volumes. Grepped: no open(…,"w"), no write_text, no tempfile, no /tmp/; deploy/console/workload.yaml declares no volumes. Everything it keeps is a jsonb row.

The blob paths, and what each is really costing

• Rendered manifests. The one thing the console is the source of truth for, because a RenderedRelease holds only what its binding serves now and a superseded one exists nowhere on the platform. 145 archived renderings come to 521 kB, largest 5.6 kB — against a MAX_RENDERING_BYTES ceiling of 1 MB, so the largest row on this platform is 0.5% of the cap. This is the path that looks like it wants object storage and does not.
• Build logs and trace payloads. The console never stores them. It re-fetches from Loki and Tempo on every render, caching in derived for 30 days and in a 512-entry per-process dict.
• Commit patches are truncated at 600 lines and say so; /cache reads at most 30 builds; Loki reads are clamped at 5000 lines.

The real constraint is retention, and it is asymmetric

So the console can tell you a build happened eighteen months ago, what it produced, which commit it ran and what its manifests rendered as — and cannot show you a single line it logged or a single span it emitted. The chain timeline on /pipelines/{run} is built entirely around this: it places a chain from recorded times when the spans have aged out, and says which it used.

That asymmetry is the finding. It is not a size problem. Adding object storage to hold 521 kB of manifests would solve nothing; what would change the answer is keeping logs and traces longer, and neither of those is the console's to keep.

Recommendation

Do not add object storage to the console. Three reasons, in order:

1. Nothing is near a limit. The largest blob it stores is 5.6 kB against a 1 MB cap, and the whole database is 8.6 MB. A store added now would be solving a problem that does not exist yet and would have to be maintained until it did.
2. The data that is large is not the console's. Logs belong to Loki and traces to Tempo. A console-side copy would be a second retention policy to reason about, a second thing to back up, and a second answer to "how long do we keep this" that could disagree with the first.
3. Retention belongs under the store that owns the data. If build logs should outlive seven days, the change is one line in platform/bootstrap/values/lgtm-loki.yaml — storage.type: s3 against the Garage that already runs here, which that file's own comment anticipates ("Filesystem rather than object storage is a deliberate simplification… Swap storage.type to s3 when that stops being true"). The same is true of Tempo.

The threshold to act on

Revisit when any one of these becomes true:

• a rendering is refused for size, or routinely exceeds ~100 kB. The push path already records {"too_large": n} so the pair stays known, and the release page now says "archived but exceeded the size the store accepts" rather than "every object renders identically" — so this failure is visible when it arrives rather than silent.
• somebody needs a build's log or trace older than its store keeps. That is the Loki/Tempo storage.type change, not a console change.
• history passes a few hundred megabytes. At the current rate — 561 kB for a platform that has been running for weeks — that is years away, and derived is swept at 30 days.

What is genuinely scarce here, and it is not object storage

/dev/nvme0n1p1   938G   814G used   77G free   92%

The host disk is at 92%, and the three observability PVCs (4 GB between them) are not what filled it. The Dagger layer cache alone has a 60Gi PV. Any proposal to keep more data on this machine competes with that, and object storage on the same disk buys nothing — Garage stores its objects on it too.

The constraint any design here has to respect

project-flow cannot reach the console. Measured, not assumed: it gets a 403 at the gateway the same way grafana.jung.town does, and the console has no in-cluster address the control plane can name. Both a push endpoint (POST /internal/renderings, HMAC-signed) and a pull loop (app/archive.py) exist; only the pull loop runs.

Anything that has the control plane write blobs to the console hits this wall first, whatever the store behind it.

One thing that is worth doing, and is not storage

A rendering the archive never saw is not recoverable, and there is no backfill — a superseded RenderedRelease exists nowhere to backfill from. Coverage therefore starts at switch-on and grows. The console now distinguishes four answers where it used to give two, and that is worth more than a bigger store:

compared            both renderings archived; the rows are exact
no-previous         genuinely the first release recorded
not-archived        the pair predates the archive — not recoverable
too-large           archived without its manifests
store-unavailable   the archive could not be READ, which is not the same
                    as it not holding this

The last one existed as a bug rather than an answer: store returns its empty default on any failure, so a dropped connection reported every pair on the page as unarchived — a store outage stated as a fact about this platform's history.