Credentials on this page are redacted. This is the public export; the real values live in docs/ACCESS.md in the private repository.

The standing brief

What is built, and what is not — NEXT-SESSION.md

Next session: the console, and what is still missing

A standing brief for picking this up with no memory of the sessions that produced it. Everything below is either verified (stated with the evidence) or unknown (said to be). Nothing here is a guess presented as a fact.

The one-line framing, in the user's words: combine the best of CI, with the best of GitLab, with the best of Datadog and New Relic. The console is the place that happens. It exists, it is deployed at https://console.jung.town, and the list below is what it is not yet.

What changed on 2026-09-08

Four things were reported and all four were acted on; one was measured and refused, with the arithmetic written down rather than the conclusion.

The delivery table read the pipeline backwards. It had no sort at all, so rows arrived in Kubernetes name order — a review environment above staging above production, which is inkwell's promotion path in reverse. Four surfaces answered "where does this environment sit" and three answered wrongly; model.environment_ranks() is the single answer now, and it says whether it read a DeploymentPipeline or fell back to convention.

The commit rail got a window, a cap and a real sort. Two buttons labelled Order were switching between two different orderings, so "always in time order" was not askable. Three controls now, and the window is honest: Prometheus keeps seven days here (storage.tsdb.retention.time: 1w, confirmed against the running server) and four templates stated fourteen as a fact. prom.METRIC_RETENTION_HOURS is the one number.

A build page explains its own outputs. One build cuts one release per component; the extras are siblings or re-cuts, and they are grouped and labelled with the construct that moved. Every card carries the commit and when it was cut.

The wedged objects are gone and cannot recur. See FINDINGS.md — the guard was written for objects inside a review environment and the ones it had to collect are outside one by construction. The sweep judges by shape and found a third orphan on its first run.

A development tier does not fit. Asked for, measured, refused: 4.70 cores of CPU requests against 0.585 free. root_env_for() derives the root per project the way OpenChoreo does, which fixed a real mismatch at zero cost and will return development on the day the pipeline gains one.

The documentation is now a set

Twenty-one walkthroughs in apps/console/app/guides/, served at /guides and published as site/ by platform/dev/build-docs-site.py, which imports the console's own renderer so the two cannot drift. Fifty-three screenshots, 7.5 MB of WebP inside the console image via platform/dev/shots-to-console.py. check-links.py reads markdown and YAML now and knows three dead hostnames — every finding of the sweep lived in a file it could not previously see.

The stuck renderings, and what was behind them — closed

The two renderings Terminating since 31 August were the visible end of a chain worth writing down, because every link looked like something it was not.

slo-alerts attached two alert rules to every component. The log-based one could never work: log alerting through the Observer is deliberately not implemented, so all 29 of them sat in phase: Error, reached no backend and produced no PrometheusRule. A critical alert on every service that could never tell anybody anything, while the trait reported success.

It could not be deleted either, and that is what wedged things. The Loki adapter had no do_DELETE, so BaseHTTPRequestHandler answered every deletion with its own built-in 501 and an HTML error page. The Observer reported that as "adapter returned 501" — indistinguishable from the deliberate refusal on create, and only the accident was load-bearing. The rule's finalizer never cleared, the rendering above it waited for ever, and 13 objects plus four review-environment namespaces were held that way.

Fixed in three ordered steps, and the order mattered:

1. do_DELETE on the adapter — deleting a rule that was never created succeeds, because there is nothing to delete and saying so is true. All 13 drained.
2. The four orphaned observability-plane namespaces, which nothing owned, gone.
3. slo-alerts 1.1.0 stops rendering the log rule at all. errorRateThreshold stays, marked INERT: nine manifests set it.

Then six components were promoted so production lost the rule too — chosen because their staging and production releases carry the same image bytes, so the promotion shipped the trait change and nothing else.

All eight were promoted in the end, inkwell-web and ledger last and deliberately, because those two shipped application code as well as the trait: inkwell-web the profiling experiment's second arm, ledger the merged PR #1.

Both were checked before they moved rather than after. The only DECLARED field that differed between each staging release and its production one was the image; everything else was the frozen construct bodies, and of those only slo-alerts had changed — shrinking by exactly the removed rule. The inkwell code change is backwards-compatible by construction: PYROSCOPE_SPAN_PROFILES defaults to true when unset, so a component that does not set it keeps the behaviour it had, and production now sets it explicitly to the value it was already getting.

Zero log-based alert rules remain on the platform. 18 metric rules, all Ready. 38 bindings Ready across three environments, every pod Running or Completed, and inkwell production answering /health/ready with 200.

One thing observed and not explained: the new inkwell-web pod logged Dropped 132/263 stack traces because of errors once, at startup, from the Pyroscope agent. It has not recurred, the pod has no restarts and sits at 144Mi. It looks like agent warmup rather than anything the promotion caused, but it was not there to be compared against on the old pod, so it is written down rather than waved through.

Still open

• The fault-injection runbook now names PodNotReady (for: 20m) as what actually fires, rather than the log-based SLO alert, which cannot. If log alerting is ever implemented, slo-alerts should regain its error-rate rule — errorRateThreshold is still in the contract, marked INERT, for that day.
• platform/project-flow/ still has no CI job. test_controller.py runs by hand and covers the sweep's predicate only; the plumbing around it is verified manually.
• inkwell-worker production still runs with profiling off by hand.

Later the same day: three fixes, one of them in the controller

The buildkit DNS failure was never a flake. 0 of 25 lookups for host.k3d.internal inside the engine against 25 of 25 for a cluster service from the same process. It is an /etc/hosts entry k3d writes on the node, not a record any nameserver serves, so no amount of CoreDNS replicas could have helped. A hostAliases entry on the engine StatefulSet takes DNS out of the path; 30-dagger-engine.sh applies it after the Helm release. The earlier ndots:1 fix verified 480/480 and changed nothing because [dns] in engine.toml shapes exec containers and publish runs in the engine.

The duplicate-build fix ate a build. Gitea sends a push and a pull-request webhook when a PR opens; deduping on (repo, ref, sha) let whichever arrived first win, and the push arm legitimately affects zero components on a fork branch. forward_to_autobuild now takes authoritative, the PR path passes it, and two tests pin both directions.

A promotion rendered components before their database resolved. Found by running the rails walkthrough and looking underneath a green result: both Rails components crashlooped four times in production while every condition read correctly, because the resource-dependency guard ran after the render. platform/openchoreo-patches/resource-dependency-gate.patch moves it before. The controller is now ghcr.io/openchoreo/controller:v1.2.3-resource-dep-gate — three patches applied, not two. The rollback image v1.2.3-construct-versions lives in the host's Docker and not in the node's containerd, so putting it back is a k3d image import first and then set image; see the patches README.

Both walkthroughs pass end to end: static in 537s, rails in 592s.

The platform is STOPPED

It was shut down at the end of 2026-09-08 to give the host back — the four k3d clusters were 24.3GB of the 31GB. Nothing was deleted; every Docker volume, including gitea-data, is intact.

./platform/capacity/power.sh start      # NOT four k3d cluster starts

Use the script rather than starting the clusters by hand. Two things do not survive a container restart and neither says so: host.k3d.internal, which presents as no agents found for plane dataplane/default plus ImagePullBackOff on every node, and the in-memory OpenBao store, which presents as a CreateContainerConfigError the next time anything NEW needs a secret. start runs both repairs. infra/compose.yaml was left running, so Gitea is still the git remote and OpenBao still holds its store.

Expect a few minutes of not-Ready bindings while the cluster agents re-register. openchoreo-dp2 stays stopped; it is the failover demo.

Start here

cd ~/projects/openchoreo-test
export KUBECONFIG=~/.kube/openchoreo.yaml      # zsh: write kubectl out in full

# The CI gate, exactly as the pipeline runs it. Run this before every push.
platform/dev/uv.sh apps/console sync --locked
platform/dev/uv.sh apps/console run --no-sync ruff format --check .
platform/dev/uv.sh apps/console run --no-sync ruff check .
platform/dev/uv.sh apps/console run --no-sync python -m pytest -q     # 188 tests

apps/console is a git submodule with its own remote. Push it first, then bump the pointer in the parent. A push triggers a build; a build takes ~2 minutes and the deployment rolls a minute or so after that.

Traps that have each cost an hour, all of them recorded in FINDINGS.md:

Since then: profiling, 2026-09-07

Round 10's phase six landed. Four things a next session should know before touching any of it — the full write-up is at the end of docs/FEEDBACK.md.

The profiler is the Rails leak, and the experiment is finished. Switching profiling.enabled off took inkwell-worker from +47.0 MiB/h to +1.9 MiB/h with its traffic unchanged, while the untouched control kept leaking and was OOMKilled four times in a day (docs/FINDINGS.md). Which half — the Pyroscope agent or the span processor — is the open question; they are gated by one variable only because the initializer was written that way.

inkwell-worker production is still running with profiling OFF. That is the experiment's patch, applied by hand, held by nothing in the repository. Decide whether to restore it or record it; it will not undo itself.

inkwell-worker-profiled is in staging AND IN PRODUCTION — this paragraph said "deliberately not promoted ... waiting for a promote.sh that has not been run", and that stopped being true at 18:41:46 on 2026-09-07, when cli:thomas promoted it in the same run that moved inkwell-worker. Corrected 2026-09-08 after finding the pod. It is a fourth inkwell component off the same image, polling the same queues, wrapping each job in Vernier and StackProf, and it is the instrument for the question above.

What it is doing in production, measured over eight hours — working set in MiB, every thirty minutes:

410 450 476 496 519 543 | 302 352 379 406 | 295 … 409 429 459

A sawtooth: it climbs 26–39 MiB/h and sheds most of it, twice in eight hours, against a 768Mi limit it peaks at about 70% of. Zero restarts in nine hours — it sheds before the kernel gets to it. In the same window and cell, inkwell-worker with profiling off went 277 → 307 MiB, flat within noise. So the prediction that it would leak was right and the prediction that it would fall over has not happened; the margin is one peak forty percent higher than any observed. Named in bindings-production.yaml as the first thing to unpromote if the cell gets tighter.

Every language now has an oracle, and the oracles earn their keep. Two tests per language that assert a number somebody already knows — 64 MiB held, ~1s of CPU in one frame, floors and never ceilings. The Ruby pair caught, on its first build, that Process.getrusage does not exist in Ruby: the CI plugin had the same line inside a rescue, so the new per-test memory-ceiling column had been reporting exactly 0 for every Ruby example while looking entirely healthy. If you add a profiler or a memory counter here, add an oracle in the same change.

Re-planned 2026-09-04 — where everything actually stands

The numbered sections below are the original brief and are still the detail. This section is the plan over the top of them, because re-checking every item against the code found six claims that were no longer true and one that was never true.

Struck: things the brief still asks for that are done

Done since the brief was written

platform/upgrade-watch closed the third version axis: 43 upstream pins watched, 23 actionable, 3 held on purpose, and the 42 constructs that declare no openchoreo.dev/version are now counted rather than merely absent. The console has an Upgrades page, a panel on every construct, and a button that opens the pull request through project-flow. A project can opt in to having them opened for it with project-flow.openchoreo.dev/upgrades: pr.

Done later on 2026-09-04 — the console walkthrough

Seventeen items came in as one message. The full ledger, grouped by cause, is FEEDBACK.md Round 5; six are done and ten are open. Three things from it belong here because they change how you read the rest of this file:

Anonymous is a different console. Bindings, the platform catalogue and the resource list are all token-scoped platform API reads, so signed out the flow views have no bindings, /platform/{slug}/{name} finds no construct, and a panel built with soft(..., default=[]) reports all of that as nothing exists. When you verify a page with curl or urllib you are verifying the anonymous rendering, which is a real rendering and not the one you were probably testing. Two bugs this session were only visible that way, and one was invisible because of it.

Every source link was a 404, and no URL was wrong. The twelve apps/* Gitea repositories are private, Gitea answers 404 rather than 403 for a private repo, and git.jung.town is genuinely public. Commits and ranges now render in the console with its own token (/commits/<org>~<repo>/<sha>), so do not "fix" this by publishing the repositories — that publishes twelve applications' full history to the internet.

The three views of the estate are /projects (list), /overview (by project) and /environments (by environment), sharing a sub-nav. Every cell drills to the Kubernetes objects, or to the component's deploy history when the binding name cannot be read.

Done 2026-09-05 — every stage that can be profiled, is

Pyroscope had zero values for ci_stage against 36 for ci_build_run: only the rspec and pytest plugins ever pushed CI profiles, so lint, audit, build and publish had no CPU samples at all, and "which stage took the most CPU" was unanswerable from the store that holds the answer.

ci_stage_profile.{rb,py} now profile the non-test stages, arriving the same way the test plugins do (RUBYOPT, sitecustomize) so no application changes anything. Two things from it are worth carrying forward:

• The console pivots on span_name, not ci_stage. A profile tagged only with the stage is correct, queryable by hand, and invisible in the build viewer. Both labels are now pushed; they are different strings on purpose. FINDINGS.md, "One label, three stores, and the console pivots on a fourth".
• build and publish cannot be profiled this way at all — BuildKit runs the application's own Dockerfile, so there is no process a language profiler can sit in. The stage instead offers the engine's own profile over its window, which was already being collected and rendered nowhere. It is approximate by nature: max-parallelism = 4, so the window can hold other builds' work, and the panel says so every time. If a better attribution is ever wanted, it has to come from the engine's own span ids, not from a wider window.

Done 2026-09-05, later: the profiles are now ranked rather than only browsable. "Where the CPU went" on the build page ranks stages and the spans inside them, and both Rails apps were rebuilt to confirm they profile (inkwell-web-run-971e3463, orchard-web-run-5d6ec333). The finding that came out of it: suite boot dominates, 44% and 60% of the suites' CPU respectively, which is the sort of thing no per-test view could ever surface.

Done 2026-09-05, later still. The cost panel was used to find something and then acted on:

• bootsnap's cache was cold on every Rails build, so it was pure overhead. Persisted; boot CPU down ~23% and the bootsnap part of it by ~84%, with to_binary reaching zero as the proof it is now a hit.
• Go builds are profiled by go test -cpuprofile rather than an agent, and rideshare has tests — it had none, and its test stage passed anyway.
• Memory profiling is opt-in per component (profileRich: true), Python only, via memray → folded stacks → Pyroscope. On the console it says the suite's memory goes to pytest's assertion rewriting and AST parsing rather than to the code under test.

Ruby memory is now done too, with stackprof: profileRich: true on inkwell gives a per-build allocation profile, verified on inkwell-web-run-0b5b0045. Its top allocator is InstructionSequence.load_from_binary — bootsnap loading the cache added earlier the same day, which is the honest trade for the CPU it saved.

Two things stay open and both are worth doing:

• ~~vernier's trace_retained~~ Done. The Gecko folder is forty lines in ruby/ci_rich_profile.rb, validated against an oracle, and Ruby now reports retained BYTES like Python. It immediately said that 81% of the inkwell suite's retained memory is bootsnap's loaded instruction sequences — the honest cost of the boot-CPU cache added the same day.
• ~~Real pprof with a declared inuse_space sample type~~ Done. Both languages emit pprof now, validated against Go's own parser. The axis is bytes that mean bytes, coverage went from 79% to 96-100%, and the service_name!~".*-ci-(mem|alloc)" matcher every CPU query used to carry is deleted — the profiles land under a memory type a CPU query cannot see.

Still open from the same investigation, both untouched: the console has a single pyro.CPU constant and queries one of the twelve profile types Pyroscope holds, and ruff is the one stage that can never be profiled, which is now stated rather than rendered as a missing profiler.

The plan, in order

1. Source and timeline (§5). Started 2026-09-04: ordering and navigation are done; the timeline view is not.

app/timeline.py orders builds by COMMIT ANCESTRY, and the axis turned out to be cheap rather than expensive: GET /repos/{repo}/commits?sha=main returns the branch in ancestry order in one call, so a rank per commit is one request per repo and no compare-per-pair is needed. Older/newer navigation is live on the build page and walks the real sequence ("1 of 30"). A commit with no rank — a PR branch, or older than the 400-commit depth — is kept and marked off_branch; with Gitea unreachable the axis falls back to time and the page says "by time only".

Found on the way: prom.builds() was returning 743 rows for 176 builds, because the metric carries the pod that served it and last_over_time keeps a series per pod, so every project-flow rollout duplicated every build it had reported. Fixed; the API now returns one row per build.

Still to do here, in order — re-checked 2026-09-08, and two of the four were already done:

• ~~A timeline view.~~ Done. /components/{c}/timeline exists and is in nav.py; timeline.deployed_where() maps a commit to the environments it reached and when. The project-wide equivalent is /projects/{p}/delivery.
• ~~Older/newer on the deploy view.~~ Done 2026-09-08. deploy_nav in _macros.html. Both neighbours were already computed AND already rendered as "Replaced"/"Replaced by" rows; what was missing was the sequence — navigation at the top and a position, "2 of 4 recorded into staging". A different axis from the build page's seq_nav, which walks commit ancestry: a promotion puts an old commit live today, so the deploy after one can carry its ancestor.
• Releases and tags, not only commits. Still open.
• _find() in app/builds.py is still an O(n) scan over prom.builds(); the rank index is the natural thing to replace it with. Still open.

Also re-checked on 2026-09-08, and stale

• §2 Log viewers — done. main.py passes component/search/minutes through to loki.component(); the brief's "cheapest real win left" is spent.
• §1 Project 360 — largely done. app/three_sixty.py exists, prom.py has a "project 360, natively" section, and project.html renders Apdex with its own T. What the brief listed as absent is mostly present.
• Item 5, versioning the remaining constructs — the producer half is done. The 129 findings upgrade-watch reports as gaps are consumers: components that reference a construct without pinning a version, so they re-render from whatever it says on the day (inkwell 36, chatops 29, showcase 23, chess 19). The constructs themselves publish versions. Pinning all 129 is a real decision with a real cost — every construct bump then needs 129 edits — and is not the mechanical follow-through the brief implies.

2. Log viewers (§2). The cheapest real win left. loki.component() already takes search and minutes, GET /api/logs/{project}/{environment} already takes component/search/minutes, and the only page that renders it (environment.html:86-110) passes none of them. Project-wide and component-wide views plus a search box are wiring a finished API to a template.

3. openchoreo-config as a real submodule. This is the single change that converts the most already-built capability into usable capability: 15 of the 23 actionable upgrades cannot be opened as a pull request purely because their repo is a write-only mirror. platform/dagger-modules is already a real submodule and is deliberately absent from sync-to-gitea.sh's targets, so the pattern exists. Doing this also finally delivers what that script's own header says it wanted — review and history for platform changes.

4. Project 360 on the project page (§1). prom.component_health() already returns rate, error_ratio, p95 and two values nothing renders (memory_ratio, cpu_cores); build_success_ratio is computed in project_index but not project_page. Apdex, transactions, slow spans and log volume exist nowhere. Render natively — Grafana still has no anonymous access.

5. Version the remaining constructs. 42 gaps, and openchoreo_upgrade_gap is the number to watch fall. Nine component types are generated, so the annotation belongs in component-types/generate.sh, not in its output. Two annotations per construct, following resource-types/postgres.yaml.

6. Build network and file IO in the console (§4). The data exists and nothing renders it. Keep the caveat that makes it honest: this measures the POD's IO — the checkout, and the pod talking to the engine — and not the shared dagger-engine's own image pulls and layer writes. Do not label it "build IO" without that distinction; it under-reports the expensive half.

7. Markdoc for every construct. Unblocked. Extend platform/techdocs/generate.py, which already builds from live CRs.

~~8. App-level pins for upgrade-watch.~~ Done 2026-09-04. 17 app pins added across five real projects (devtools, inkwell, chatops, chess, rideshare), so the per-project auto-PR path now has work it can actually land — apps/* are real submodules, not mirrors. 60 pins total, 39 actionable. Every project still defaults to report, so nothing is opened until one opts in.

Two things it immediately found: uv is pinned at two different versions across the apps (0.9.6 in ledger/chess-bot/zulip-bots, 0.5.29 in the other five and in the Dagger python module), and apps/ledger/Dockerfile pins it twice in one file, which is why Pin(all=True) and the repeated-splice path exist.

Lockfiles are deliberately still out of scope — uv.lock, package-lock.json and Gemfile.lock are resolved trees, not pins, and bumping one means re-resolving with the ecosystem's own tool. That is the one thing Renovate would genuinely add.

9. Cache time-saved (§3), only with the argument made. app/cache.py:198-201 deliberately refuses the label and is right to; a baseline-median figure is compatible with that refusal only if it ships as a separate, explicitly estimated number carrying its sample count, and never renames cached_ms.

Decided 2026-09-07 — object storage for the console: no

Measured rather than argued: the largest blob the console stores is 5.6 kB against a 1 MB cap, its whole database is 8.6 MB, and the data that is actually large belongs to Loki and Tempo rather than to it. The real finding is an asymmetry in RETENTION, not in size — the console remembers a build's row and its manifests for ever and can show neither its logs (7 days) nor its spans (48 hours). Fixing that is storage.type: s3 under the store that owns the data, which lgtm-loki.yaml already anticipates in a comment.

The full evaluation, with the thresholds that would change the answer, is STORAGE.md.

Needs a decision, not more work

• ~~The DR data plane and Outline.~~ Decided and done 2026-09-04: both dropped. The five *-dr bindings and all of Outline are gone, and wiki.jung.town no longer appears in README.md or docs/ACCESS.md. There are now zero genuinely-failing bindings on this platform — the only three that are not Ready are orchard-*-production at state: Undeploy, which the console correctly renders as hibernated rather than broken. production-dr the Environment and platform/failover/ are kept, so DR is restorable.
• ~~Self-hosted Renovate for the app ecosystems.~~ Decided 2026-09-04: no. Extending upgrade-watch/pins.py with the app-level pins covers the same ground for no new image and no memory, and — unlike the platform's own pins — those repos are real submodules, so the per-project auto-PR path that is already wired and idle starts working. Renovate stays the answer if transitive lockfile resolution is ever wanted; pins are exact versions in named files, and that is the limit of this approach.
• Anonymous read of the console now that the app repos are private, and Grafana anonymous access for embedded panels. Both are policy, not code.

1. The project page needs the rest of Project 360

The project page has environments, drift, the promotion pipeline, builds, and a link to the dependency graph. platform/grafana-dashboards/openchoreo/project-360.json has considerably more, and none of the following is on the page:

• Requests/s, 5xx errors, p95 latency, Apdex (T = 250 ms) — four stats
• Builds green in range; components running
• Per component: throughput, errors, latency, memory (one table)
• Traffic and latency over time, by component (three timeseries)
• Transactions: every operation, by type
• Slowest spans that have a flame graph
• Log volume by component

component-360.json additionally has memory-against-limit, network in, and a per-component operations table, which the component pages should pick up.

All of it comes from Prometheus and Tempo, which the console already reads without a token — so unlike the platform-API panels, these will work for an anonymous viewer. Two choices to make: render natively (full control, matches the existing SVG islands) or embed (already implemented as a macro, but Grafana has no anonymous access so an embed only renders for someone signed in to it). Recommendation: render the numbers natively, keep the embeds as the "go deeper" link.

2. Log viewers

There is a full build-log viewer with per-stage tabs on the build page, and pod logs on the deploy detail page. Missing:

• A project-wide and component-wide log view (Project 360 has one; the console does not)
• Search and filter within a log, and by level — Loki supports it; the console sends fixed queries
• Live tail for a running build or a running pod
• Log volume over time as context above the lines

app/sources/loki.py already has component(project, environment, name, search, minutes) with search support, and it is exposed as GET /api/logs/{project}/{environment} — but no page in the console renders it. The API is there and the UI is not, which makes this mostly a template job rather than a data one. (build_all was in exactly this state and had zero callers until this session; check for the same pattern before writing anything new.)

3. Cache: measure what caching actually saved

Today the build page reports how many operations came from cache and shows the log lines that prove it. What is wanted is time: for comparable steps, keep a baseline of how long each takes when it is not cached, then attribute the saving.

The method is proven and the data supports it. Prototyped over 14 real builds: 625 operations, 144 cached (23%), and 65 distinct operations had a measured uncached duration. Sample of the baselines it produced:

n=2  median 52.6s  Ruby.test source: Address.directory ...
n=2  median 47.2s  Ruby.test ... withCapabilities ...
n=1  median 37.5s  Container.withExec sh -c 'bundle exec rspec spec ...'

Implementation: app/cache.py already parses every operation out of the build log with its verdict and duration, and _norm() already gives cross-build identity. Keep the DONE durations per normalised operation, take a median, and for each build sum that median across the operations that were CACHED.

Three honesty constraints, all learned the hard way on this feature:

• The prototype above sampled only the slowest 8 operations per build, so those medians are indicative, not final. Retain all DONE rows.
• Some operations have n=1. A median of one sample is a data point, not a baseline; show the sample count or suppress the estimate.
• Present it as estimated time saved. A cached operation returns in microseconds and what it would have cost is genuinely unknowable for that run; the estimate is "what this same operation cost when it last ran".

4. Network and file IO from the build — yes, this exists today

Asked as an open question; it is answered. cAdvisor already scrapes build pods: 190 distinct build pods carry these series over the last 7 days. Real figures for one console build (console-run-5ac1f146):

network receive   24.5 MB          fs reads    560.8 MB
network transmit   2.0 MB          fs writes   408.0 MB

and it is attributable per Argo step:

console-run-5ac1f146-checkout-...              0.0 MB
console-run-5ac1f146-build-and-publish-...    24.5 MB
console-run-5ac1f146-generate-workload-cr-...  0.0 MB

Query shape — note max_over_time, because the pods are deleted by podGC and an instant query returns nothing between builds:

sum(max_over_time(container_network_receive_bytes_total{
  namespace="workflows-default", pod=~"<run>.*"}[2h]))

The caveat that matters: the Dagger work happens in the shared, long-lived dagger-engine, not in the build pod. So this measures the pod's IO — the checkout, and the pod talking to the engine — and not the engine's own image pulls or layer writes. Attributing those per build needs engine-side instrumentation and is genuinely unsolved. Do not label pod IO as "build IO" without that distinction; it under-reports the expensive half.

5. Source and timeline — the largest gap

"I'm used to being able to go forward and backwards in Jenkins builds and each one generally corresponds to a commit." Nothing in the console does this. The build page is an island: no previous, no next, no sense of sequence.

What is wanted:

• Previous / next build on the build page, and on the deploy view
• Each build tied to its commit, and commits shown in order, so a range reads as a range rather than two hashes
• Attachment to releases and tags, not just commits
• Timestamps everywhere, and a timeline view: what went live when
• Connect a deploy backwards to the build, the commit, and the change

Most of the raw material exists: openchoreo_build_completed_timestamp_seconds and openchoreo_build_duration_seconds carry build_run, component, commit and result; prom.deployment_history() gives when each commit went live per environment (fixed this session — it was silently returning nothing); the head span carries the commit message, pusher, branch and Gitea compare URL; and model.deploy_stream() already computes the commit range between two deploys.

What does not exist: ordering by commit ancestry. Builds are ordered by time, and that is not the same as commit order once branches and PR builds are involved. Gitea's compare endpoint gives ancestry between two SHAs, and sources/gitea.py already calls it. Decide early whether the timeline is "builds in time order" (easy, sometimes wrong) or "commits in graph order" (right, needs Gitea per pair).

6. Local inference on the GPU

Deferred twice; the groundwork is verified and unchanged:

• Radeon RX 9060 XT, gfx1200, 16GB. ROCm 7.2.2 has native gfx1200 support — no HSA_OVERRIDE. Ollama runs on the host already using the card.
• The k3d data-plane node is privileged and already sees /dev/kfd and /dev/dri.
• Weights are the problem, not devices. Privileged mode exposes those devices, not arbitrary host paths: ls /home/USER/models inside the node container is "No such file or directory". A hostPath PV would silently mount an empty auto-created directory, so the component type must use type: Directory and fail loudly, never DirectoryOrCreate.
• Interim, no rebuild: docker cp the model blobs into /var/lib/openchoreo-models in the node (6.6GB for qwen3.5). Does not survive k3d cluster delete.
• Durable: a volumes: entry in platform/bootstrap/k3d/config-dp.yaml mounting the host's Ollama store read-only at the same in-node path. Commit it now, inert until the next dp rebuild — choosing the in-node path as the stable one is what makes the switch a no-op.
• ollama/ollama:0.22.0-rocm is 1.44GB compressed against 7.7GB for llama.cpp:server-rocm; on a disk at 89% that decides it.
• Cross-cell reach needs no new NetworkPolicy: declare the endpoint visibility: [namespace] and every project in the environment can reach it.

Then the payoff: an agent that takes a failed build, pulls its logs, trace and diff, and writes an analysis into Zulip. The OpenChoreo API speaks MCP (openchoreo-api-config has mcp: enabled: true) for platform state, and the console's own JSON gives what the build did.

Blocking constraint: memory. ~3GB available with five clusters running, and the build wave in this session OOM-killed two inkwell workers and beyla. Shed load before starting — platform/capacity/hibernate.sh sleep staging is ~2.9GB and the largest single lever.

Carried over, still open

• Exec console into a pod, for those with permission. ocapi.may() exists for the permission check. Deliberately not built: a web terminal into production pods deserves its own security design, not an afterthought.

• Native Pyroscope UI is not publicly routed (only pyroscope.lgtm.openchoreo.localhost). Needs a gateway route and an auth decision. Grafana's Pyroscope datasource is the cheaper path to the same flame graphs and is already embeddable.

• Grafana has no anonymous access, so embedded panels only render for a viewer already signed in to Grafana. Either enable anonymous viewer access or render natively (see §1).

• ~~The console is readable anonymously.~~ **Decided and done 2026-09-04: sign-in is required for every HTML route and every /api/*; only /health/live and /health/ready stay open, for the kubelet. One rendering instead of two, which removes the bug class that produced four confidently-wrong panels in a single session. /internal/render/* is the one addition and it is HMAC-signed, not session-authenticated, because its caller is a bot.

• Four platform repos are still public: platform-eng/dagger-modules, platform, openchoreo-config, templates. dagger-modules cannot be private as things stand — tried it, and the build fails with curl: (22) ... 404 fetching the CI module, because that fetch is dagger call -m <url> and never sees the git secret. The application code is private; the platform's own modules are not.

• Markdoc documentation for every platform construct, generated from the CR and extended by hand. No longer blocked — the blocker was retired on 2026-09-04 without any work, because it was never true.

  The brief said npm ci had never run against a real registry inside the Dagger engine and that a Verdaccio proxy gated everything else. It has, and it does not. Frontend.Lint → WithSource → WithDeps runs npm ci --no-audit --fund=false (platform/dagger-modules/frontend/main.go:56-75), apps/storefront locks 194 packages including react, vite and eslint, and its builds are green — most recently storefront-run-e2050078. apps/brochure proves nothing either way; it locks zero packages.

  So Markdoc is plannable now, and its real cost is what it always was: writing the generator and the prose. platform/techdocs/generate.py already builds from live CRs and is the thing to extend rather than replace.

• Thunder auth flows as a platform resource type, so a project can declare an IdP integration and be provisioned by the standard flow. Registering an OIDC app is six near-identical shell scripts plus a seventh step with a hardcoded application list — which is exactly how the console shipped unable to sign in. zulip-bot.yaml is the pattern to copy. Ranked next after it: webhook registration, and Gitea repo plus deploy-token creation.

• Platform construct versioning is one type deep. postgres declares openchoreo.dev/version and stamps it into what it renders; the other nine resource types and all fourteen component types do not. Until they do, the graph reports unknown for them rather than current-or-behind.

  Now measured, still not fixed. platform/upgrade-watch counts it: 42 of the 43 live constructs (9 resource types, 14 component types, 10 traits, 8 workflows, 1 project type) declare no version, and each is reported as a gap — a question the platform cannot answer — rather than as a failure. openchoreo_upgrade_gap is the number to watch go down. Note that nine component types are GENERATED by component-types/generate.sh, so their annotation belongs in the generator, not in its output.

• Upstream versions are now watched, and the platform is behind on 22 of 43 pins. platform/upgrade-watch is the third axis that nothing observed before: versions.env, the image tags inside resource types and workflows, and the Dagger toolchain constants. Worth acting on soon, in rough order of risk: openchoreo v1.2.3 → v1.2.4; ci-uv 0.5.29 → 0.9.28, which is ~97 releases behind and changes lockfile resolution for every Python app; garage v1.0.1 → v2.3.0 and adminer 4.8.1 → 6.0.1, both major; and the kubectl / k3s-node-image pair, which must move together or the API skews. Three pins are held on purpose and say why. See platform/upgrade-watch/README.md.

• Gitea Actions evaluated and deferred; the reasoning is in FINDINGS.md. GitLab does not fit — documented floor 8GB against ~3GB available.

• The DR data plane (dp2) is stopped, and as of 2026-09-04 nothing pretends otherwise: the bindings that could not apply were removed. Restoring DR means 36-dr-data-plane.sh plus writing the bindings again — see platform/failover/.

Where the code is

platform/upgrade-watch/    what the platform pins, and what has moved on
  versions.py       the ONLY judgement here: is B newer than A, and comparable?
  pins.py           all 43 pins, declared by hand, with their deliberate ceilings
  upstream.py       registries and release feeds; failure is data, never silence
  watcher.py        the scan, /upgrades, /metrics, the Zulip digest, --self-test

apps/console/app/
  main.py            routes; the canonical-host redirect; page()
  depgraph.py        the dependency DAG's DATA (nodes, edges, versions, drift)
  graph.py           the DAG's LAYOUT — server-side, deterministic, no dagre
  cache.py           Dagger cache verdicts, from the build log and the trace
  filters.py         shared filter state, kept in the URL
  model.py           deployments, project pages, promotion order, deploy streams,
                     binding_progress -- the conditions that say why a deploy is slow
  releases.py        release diffs, the RENDERED manifests and their applied
                     state, k8s tree, pod logs, events, describe
  render_diff.py     env-vs-env on the rendering (the default) or on the live
                     objects; and desired-vs-live drift, one-sided on purpose
  projection.py      what a component WOULD render as where it is not bound.
                     A derivation, labelled per field  (READ THE HEADER FIRST)
  permissions.py     the seven gates; the platform decides, this remembers
  access.py          /access — the platform's own answer about the viewer
  nav.py             the rail and the tabs. Both directions are tested now
  builds.py          the build page, the pipeline, /cache
  sources/
    _cache.py        settled-vs-fresh caching  (READ THE HEADER FIRST)
    _http.py         soft() and Unavailable    (READ THE HEADER FIRST)
    ocapi.py         the OpenChoreo API — every call is user-token scoped
    prom.py loki.py tempo.py pyro.py gitea.py alerts.py apis.py catalog.py

The READ THE HEADER FIRST files encode rules that have been broken more than once each. docs/FINDINGS.md is the long form of why. projection.py joins them for a different reason: it is the one module that produces something the platform did not say, and everything in its header is about not letting that be mistaken for something the platform did say.