Credentials on this page are redacted. This is the public export; the real values live in docs/ACCESS.md in the private repository.

Feedback, round by round

What was asked for, and what it turned out to be — FEEDBACK.md

Feedback, and what happened to it

Four rounds of notes used to sit loose in the repository root — fixes.md, more-work.md, grafana-fix.md, more-fixes2.md — two of them not even committed. They are the record of what was actually asked for, so they are kept verbatim below rather than summarised, each with its current status.

The HAR captures that came with rounds 3 and 4 have been deleted. They were full browser session captures (55MB, Grafana / Zulip / Thunder cookies in the clear) and the issues they documented are resolved or recorded here.

Status vocabulary: done · partly done · open · answered (a question, not a defect) · won't (decided against, with the reason).

Round 1 — fixes.md

logging into portal get this error: Origin 'http://portal.TAILNET.ts.net:8080' is not allowed

done. CORS origin allowlist; see platform/bootstrap/values/obs-observer-cors.yaml and the Thunder app scripts under platform/bootstrap/thunder-scripts/.

I want to see the best observability story I can. I'm interested in the grafana lgtm stack, especially pyroscope. I want to see as much as possible, ebpf auto instrumentation is a plus, as well as instrumenting the actual app to get better metrics, access to stack, queries. Open to other observability stacks. I want to trace from commit push to post-deploy as richly as possible. There are notes about network enrichment via cillium too.

done, and still the main line of work. Loki / Tempo / Pyroscope / Prometheus behind Grafana (26-lgtm.sh), Alloy on all four planes (27-alloy.sh), Beyla eBPF on the data plane (28-beyla.sh), Cilium with Hubble (34-cilium.sh), and explicit OTel instrumentation in the apps plus platform/gems/platform_telemetry and platform/pylib/platform_telemetry. Commit-to-deploy is one trace: the push is the root span and the build's stages hang under it — docs/OBSERVABILITY.md.

I would be interested to see additional projects added, a note taking app, based on the outline project, but using cloudnativepg via our platform components. And some kind of chat app, I like zulip, but if that doesn't work another could be acceptable

partly done, and the Outline half was dropped. Zulip runs and is the whole chatops story (five bots, one of them a conductor that scripts conversations so the traces are never empty). Outline was declared under apps/outline/ and never ran: its binding was hibernated and its outline-db / outline-redis / outline-files Resources were never bound.

Removed 2026-09-04 rather than provisioned. It had been a documented-as-reachable URL that could not start for a week, and README.md and docs/ACCESS.md both listed wiki.jung.town — so the cost of keeping it was a lie in the two documents people read first. The note-taking app the round asked for exists and works: inkwell, on CloudNativePG through the platform's own postgres resource type, which is nearer the point than a second vendored wiki.

Round 2 — more-work.md

Can we enforce projects being able to register under a project specific subdomain wildcard as default, preventing them from taking a top level subdomain without a certain level of permission setting it somewhere. We should make it the default even if we can't enforce it easily.

done. Default is <environment>-<project>.jung.town; the apex is refused by a ValidatingAdmissionPolicy, platform/openchoreo-config/policies/restrict-zone-apex.yaml.

Also sign in is painful, if we integrate with oauth/oidc provider like GitHub (open to others) can we simplify account login for others to test. [...] if everyone with access can use their existing account to login to Zulip, test it and the chat bots, and then already be logged in for the portal and other observability services that would be great. Creds in infra folder .github-oauth

done. One GitHub sign-in at the gateway covers portal, Grafana, Gitea, Zulip and Argo; access is a GitHub allowlist (platform/bootstrap/thunder-scripts/66-allowlist.sh). docs/ACCESS.md has the full model, including which three services are exempt and why.

Make sure the docs include walkthroughs for creating a new rails project through deploy, making a change including a rollback. Adding trait [...] that modifies the project to include a solid queue worker, that runs a job to store something in object storage, with the buckets being provisioned, wired up automatically and development environment resources are cleaned up. This should also get a subdomain declared and provisioned automatically as a step. This should also demonstrate adding auth that automatically generates a client to interact with our idp and configures it on the app side.

done. docs/WALKTHROUGH.md and docs/WORKING-ON-IT.md, and the @walkthrough bot drives the whole thing end to end in chat.

Round 3 — grafana-fix.md

How do we improve visibility into deployments, can we add capacitor for flux or integrate more tightly into portal. I assume there are logs, metadata, if not more.

partly done. Flux has its own dashboard (platform/grafana-dashboards/platform/infra-flux.json); Capacitor was not added.

If it isn't a heavy lift to expose the other platform level projects/services in the portal, that would be good. For example gitea, grafana, etc. Anything that has a hostname should exist and have links to their hostname

done. platform/portal-catalog/platform-services.yaml.

(I still do not reliably see dashboard tab links or links on entities by the way)

done. platform/portal-image/entity-links.patch and grafana-plugin.patch.

Also the chess board images do not load/work correctly.

done. chess-render serves both board.png and board.svg; the traffic generator exercises both every pass.

More importantly, I still can't verify any of the grafana information because dashboards fail to load with a lot of 403 on websockets.

done. Gateway/Thunder configuration; the HAR that documented it is deleted.

Round 4 — more-fixes2.md

Localblocks processor not found error accessing traces drilldown.

done. local-blocks is enabled in the metrics generator's overrides.defaults.metrics_generator.processors, and it is in the RUNNING config, not only in the values file — /conf/tempo.yaml:49 inside tempo-0 lists it. platform/bootstrap/values/lgtm-tempo.yaml carries a long note on why it is required: it is the only processor that keeps raw spans queryable, so every Traces Drilldown view fails without it, and span-metrics/service-graphs remote-write aggregates and cannot answer {...} | rate().

Also I'm expecting to see more of the graph/timeline/gantt style view of spans when they are available instead of just lists of traces only.

open. The build waterfall exists (platform/grafana-dashboards/ci/build-pipeline.json) but most trace surfaces still render as lists.

Metrics drill down says I don't have permission.

open. Grafana role/permission for the Metrics Drilldown app.

On the build pipeline view, especially when I get there clicking from a specific build from a message or wherever, it would be good if the charts were scoped for the actual relevant time so they maximize the use of space.

done. links.build_pipeline() brackets the build's own window, and the console's grafana() embed macro does the same — pinned by test_an_embedded_dashboard_is_scoped_to_the_build_not_the_last_hour. The now-1h still in the dashboard JSON is each dashboard's DEFAULT range, which is correct for opening one cold.

I still want to see trace/span views integrated with logs and profiles everywhere possible.

partly done, and the two things that were silently blocking it are fixed: demo-conductor and orchard were both hibernated, so the bot traces and the CPU profiles they feed had no data at all — an empty panel is indistinguishable from a broken one. Both are awake.

Example links from chess bot that redirected and didn't show what I expected: Trace / Profile / Logs [three Grafana explore URLs]

done. There is now one Grafana link builder, platform/pylib/platform_telemetry/platform_telemetry/links.py, and every function in it is window-aware: trace() uses a DAY (a trace id needs no window but Grafana insists on one), and build_pipeline(), transaction() and span_profiles() bracket the caller's own window and pad either side. Its header records the bug by name. alert-relay/relay.py:106 and builds.py:174 carry the same rule for their own links.

In portal UI when viewing builds, it usually says No logs available for this step and it isn't easy to find a link to the argo workflow page from there. Is there a fix?

done, and the cause was the opposite of the obvious one. OpenChoreo decides whether to offer live logs by asking whether the Argo Workflow object still exists. podGC deleted the pods at 2h while deliberately leaving that object, so the portal answered "yes, live logs", asked for pod logs, found no pods, and returned an empty array with HTTP 200 — rendered as No logs available for this step for every step of every build older than two hours. The Loki fallback works but is only consulted when the Workflow object is gone. ttlStrategy now matches podGC's 2h, so pods and object go together and the portal falls through to Loki. See platform/bootstrap/values/wp-argo.yaml.

In grafana the deployed at and other urls for example staing-rideshare.jung.town/rideshare-http which works, not all of them do, even for environments that have https serving the same endpoint, which is the endpoint I generally expect to see

open. Endpoint URLs should prefer the https external URL; some panels use the http one and some components publish neither.

Generally evaluate if we are using dagger according to best practices. I expected the trace/span correlation to be easier since that is most of what dagger cloud appears to do. If we are being limited in our ability to report profiles etc, we do not have to use dagger for the entire pipeline, or can use a different path for rails apps. I hoped it would be possible to obtain that in a standardized way with dagger modules

answered, and acted on. Dagger was not the limit; two things were.

  1. Correlation was never wired up. dagger.Tracer() ships in the generated SDK and the module runtime already stands up an OTel SDK before dispatch — it was simply never called, so every span in a build came from the engine's own instrumentation and was named after API calls (Container.withExec) rather than after anything a person cares about. stage() in platform/dagger-modules/platform/main.go fixes that, and force-flushes per stage because the tail of a pipeline was being lost at module exit.

  2. The pipeline was not shaped like a pipeline. Every stage was awaited before the next began, so lint, audit and the two RSpec suites ran end to end despite sharing nothing. They now run concurrently (runStages), which is what finally makes the waterfall show overlap instead of a staircase: four stages that took 40.2s in sequence take 13.6s.

    Worse, one push ran the pipeline once per component — three times for a Rails app, five for the bots — publishing byte-identical images under different names. One build now serves all of them.

So: no, there is no need to leave Dagger or special-case Rails.

Is there a reason we stood up a new minio vs continuing to use garage by the way? It seems cleaner and has continued support and is already in cluster.

answered, and the honest answer was "we did not, but staging still was". The objectstore resource type was switched from MinIO to Garage and production was promoted onto the new release — but staging stayed pinned to inkwell-files-5d96489b4f from 2026-08-29 and went on running quay.io/minio/minio for five days. One resource type, two backends, and nothing reported it: both bindings were Ready, because each was correctly serving the release it was pinned to.

Fixed 2026-09-03 with occ resource promote inkwell-files --env staging in the cluster only — production was promoted and the repository was not updated, so staging stayed on MinIO. Corrected 2026-09-04; see below. inkwell-enrich needed no change at all — it consumes the objectstore output contract as S3_ENDPOINT / S3_BUCKET / S3_* and never knew which implementation was behind it, which is exactly what that contract is for. Verified by writing an attachment through it after the swap.

The dead resource-types/objectstore-minio.yaml.replaced has been deleted; the live type is resource-types/objectstore-garage.yaml.

The general lesson is worth more than the fix: a resource type can be changed underneath everyone, and environments pinned to older releases keep serving the old one indefinitely without any binding going un-Ready. Promotion is per environment and nothing nags. Worth a drift check — see the open items below.

Tech docs are either missing for most projects or broken.

open. platform/techdocs/generate.py builds from live CRs but coverage is thin.

I'm leaving another har file related to a 404 in tulip for the board, clicking on it loads it, but it spins in the chat view.

open. Zulip's own preview fetch for the board image.

Round 5 — the console walkthrough, 2026-09-04 (newest)

Seventeen items, given as one message. Grouped here by what they turned out to be, because three of them had the same cause and one of them was not a console bug at all.

Done

The deploy page shows all components individually, but I'm generally concerned first with seeing the projects [...] a dag style view showing each project broken down, their environments, the pipeline between each environment, and THEN inside of that broken down by component.

done. graph.pipeline_layout() plus app/pipeline.py. Environments are columns in the order the DeploymentPipeline promotes through, the arrows run between the bands rather than between cells (promotion is a property of the environments, not of any one component), and each column holds that environment's components and its resources — a backing service is part of what an environment is, and leaving databases out is how chat came to look empty. On the project page as a "Flow" section placed first, and fleet-wide at /overview.

Rows are deliberately not aligned across columns: a component missing from one environment leaves no gap, because an empty slot in a grid reads as a fault rather than as an absence.

pipeline_layout is not layout(). A layered DAG assigns layers from the edge graph, which would put the promotion edges between nodes in the same column and have nowhere to draw them.

It should always be possible to drill down to the actual Kubernetes manifest and set of objects eventually.

done, three ways, because "always" means three different things here.

Still not seeing a route to see the actual kube objects deployed in an environment. Environment possibly needs its own dag diagram or there should be an environment filter.

done. /environments and /environments/<env> transpose the flow view: columns are projects, the environment is fixed. No edges — promotion runs between environments and this view has fixed one, so an arrow between two project columns would invent a relationship the pipeline does not describe.

The transpose exposed a latent bug: pipeline_layout stamped the column key onto every cell as environment, harmless while columns are environments and wrong the moment they are projects, where every tooltip would have read "web in inkwell".

The three views of the same estate (list, by project, by environment) share a sub-nav rather than taking a fourth rail entry; the rail is at ten and wraps.

I still get 404 for most links like git.jung.town/apps/bakery/commit/4e24446...

done, and the URL was never wrong. All twelve apps/* repositories are private: true; Gitea answers an unauthenticated request for a private repo with 404 rather than 403, so as not to confirm the repo exists; and git.jung.town is genuinely public — confirmed by fetching it from outside this LAN, which returns the Gitea home page. So the link was correct, the commit was there, and the reader had no Gitea session. Three rounds of "the links are broken" were a missing session reported as a bad URL.

Publishing the repositories would fix every link with no code, and publish twelve applications' full source and history to the internet. Not a side effect worth taking on to make a hyperlink work — so /commits/<org>~<repo>/<sha> and /commits/<org>~<repo>/<base>...<head> render the commit and the range with the console's own token, and source_ref points there, which is every commit reference in the console at once. The Gitea link stays as a labelled secondary, saying it answers 404 rather than 403, because for someone who has a session it is still the fuller tool.

The range route must be declared before the single-commit route: {sha} is [^/]+, which matches aaa...bbb too, so the other order renders every range as a commit page for a sha that does not exist. Pinned by a test, not a comment.

we should display changes in-line on build page when reasonable

done in the previous round; the diff block is now a shared macro so the build page and the commit page cannot drift into rendering the same diff differently.

I think I need time stamps in general on the commit names to know what order they are in.

done. source_ref(..., at=) renders an age beside the sha with the absolute time in the tooltip. Eight hex characters carry no order, so a column of them cannot be read as a sequence — and "3h ago" answers the ordering question at a glance where an ISO timestamp does not.

Open

Can we also do junit test output as a platform option/standard and incorporate it into the console with a richer UI.

done, both halves. TESTING.md is the contract: a JUnit XML file at one path in every language module, plus one OTLP log record per test case carrying the same fields. A new language joins by producing both and the console changes nothing. /builds/{run}/tests is the report — totals, failures first with their type, message and captured output, the ten slowest, every case filterable by outcome and suite, and a per-suite breakdown. The build page previously showed four cases out of 220 inside a stage box.

The platform writes the XML rather than requiring a reporter package: pytest has --junitxml built in, and rspec_junit_formatter would have to go into every application's Gemfile — against this instrumentation's own rule that the application changes nothing, which is why the plugin arrives by --require in the first place. So the rspec plugin writes it, in about fifty lines, and the escaping is tested against an independent parser rather than reasoned about: &<>"' escaped, and the control characters XML 1.0 forbids stripped, because they cannot be escaped and Ruby test output contains them. That input arrives on the day the suite is already broken.

The console reads the log records, not the file — the file never leaves the build container, because build-artifacts is ClusterIP on the workflow plane and the console runs on the data plane. Writing it anyway is what makes this a standard rather than a private schema.

Three things found by verifying rather than by reasoning:

A missing duration is left missing throughout. Zero would make an untimed test look instant, sort it below every real one, and report a 220-case suite as 0.00s — a fast suite rather than absent timings — so the page reports how many cases were timed whenever that is not all of them.

For the git stuff I was kind of hoping to see something more like git-bahn/git-graph, with the git commits visualized as annotated nodes that you can move between, show more information progressively, see when you are on a PR environment that you are branched off a commit.

done. /components/{component}/graph. Gitea's commit payload carries parents, so this is a real DAG: branches are lanes that persist down the page and curve back into the trunk at their merge base. Checked against chess, whose demo branch holds lane 1 for three rows and rejoins lane 0 at the shared ancestor.

Each row draws its own rail. One tall SVG beside the rows breaks the instant a row expands — the rows grow, the SVG does not, and every dot below the open one points at the wrong commit, worst exactly when someone is reading a commit closely. A fixed-height SVG per row makes the alignment structural rather than something to keep in sync; the rail continues behind an expanded row via preserveAspectRatio="none", which stretches the lanes vertically and leaves their x alone.

The annotations are the reason not to use a general git viewer: anyone can draw a commit graph, but "live in production, built twice, and this is what PR 7's review environment forked from" is knowledge only the platform has. The base commit of a pull request is marked forks here and names the review-pr-<n> environment and the -pr-<n> component suffix — which is what the request was actually about.

Progressive disclosure is a <details> per row, no JavaScript: sha, subject, chips and author collapsed; message body, builds (each linking to its test report), deploys, pull requests and three range links expanded.

Two lane-algorithm details, both tested rather than commented:

Found while wiring it: deployed_where returns one entry per deploy event, so a commit deployed to production twice rendered "production, staging, production, staging" — a rendering fault to read, a redeploy in fact. Grouped per environment now, with a count and whether that environment is serving it.

Zulip project still does not display expected information.

part of it found and fixed; the rest is open. chat rendered empty across every telemetry panel because its spans report service_namespace as the Kubernetes namespace — dp-default-chat-development-03220a9b — with no project and no deployment_environment label at all. Every query filtered on service_namespace="chat" therefore matched nothing, in the console and in Grafana alike. A second scope matches the namespace pattern and finds 182 operations there, and the page says how many rows were found only that way, because everything else keyed on those attributes is still empty for them.

The pattern is assembled from names the platform already knows and never parsed back out of a namespace: dp-default-my-app-staging-abc12345 is my-app/staging or my/app-staging with equal justification, and only the platform knows which. The environment filter is deliberately not applied to that scope — the environment is already fixed by the namespace, and those series carry no environment label, so filtering on one makes them vanish again.

Still open: the real fix is upstream, getting Zulip's instrumentation to send service.namespace as the project and to send an environment at all. Until then every other project-keyed panel is empty for it.

Build log still empty, still haven't seen profiles anywhere on console.

Build log: already fixed, and checked across every build type rather than one. Every component's most recent build has a populated log — 288 to 892 lines, with real content in each stage panel: console 892/6 tabs, brochure 288/2, rideshare 389/5, openchoreo-bot 590/6, storefront 304/2, inkwell-enrich 505/6, ledger 473/6, buildviewer 504/6.

Asking why the two static builds have 2 tabs where everyone else has 5 or 6 turned up a real gap: StaticCi never audited, and its own doc comment said it did. It has claimed "lints, audits and builds" since it was written and only ever linted, so storefront and brochure are the only components on this platform that have never had a dependency audit. Frontend.Audit already existed, with its own reasoned --audit-level=high threshold, and nothing called it. Now wired through runStages in parallel with lint, matching RailsCi and FastapiCi.

Still open for static components: they run no tests at all. The frontend module has no Test function, so run-tests is accepted by the workflow and has nothing to call. That is the remaining reason a static build's pipeline is shorter than everyone else's, and it is where the JUnit standard should reach next.

Profiles: done, and the reason they were nowhere was concrete. pyro.service_profile() had zero callers. CI profiles were wired to the build page; the profiles of the applications actually serving traffic were fetched by nobody — while eight services have one, inkwell-web 283 stack levels deep across 1,099 frames and chess-render burning 172 seconds of CPU an hour.

/components/{component}/profile, filterable by operation and window, linked from Project 360's per-component table (beside the CPU column — that column states the quantity, this answers where it went), the commit graph, the timeline and the build page. A page nothing links to is the state this started in.

The numbers come before the picture, which is the answer to "I'm not sure the flame graph is rendered correctly". A flame graph cannot settle that question: a broken renderer and a genuinely flat profile look identical. Hot frames are computed server-side from the same flamebearer and shown as percentages, so a bar that disagrees with a number is checkable — and a failed canvas costs the picture, not the profile.

Three bugs found against live data:

That makes three functions this session that were written, documented and uncalled: loki.build_all (fixed earlier), pyro.service_profile, and Frontend.Audit. Worth a grep for the pattern rather than waiting to trip over the fourth.

The comparison feature is not exactly what I expected [...] I want the field level diff but again I want to see how that actually renders differently between environments with a rich diff.

done. The compare page diffed release pins; it now also diffs the rendered Kubernetes objects — paired by kind, field by field, with the YAML in context.

A raw diff is the wrong answer, and that was measured before deciding. Diffing inkwell-web's staging and production Deployments as text gives 40 changed lines. Substituting the environment name and release hash brings it to 8, and all 8 are environment-uid and dp-resource-hash — identifiers that differ by construction. The real answer is: same image, same replicas, same limits, same env keys. A page showing 40 red and green lines to say "these are the same" buries the case that matters.

So every difference is classified substantive / environment / identity, and the page leads with the substantive count. inkwell-web reports 0 substantive, 36 environment-scoped, 11 identifiers, and says "these two environments render the same".

The classifier took three passes, all against live cluster data:

  1. Exact substitution only. A generated name differs by the environment and the content hash at once, so inkwell-web-staging-379f55e8 vs inkwell-web-production-da55eda4 fell through — 22 substantive differences for two identical renderings, which is the raw-diff problem with new labels.
  2. Tolerate the hash, but only on name-like fields. Safe and too narrow: DB_HOST and DATABASE_URL point at the environment's own database and still read as substantive. Down to 5.
  3. An exclusion list instead — version-like leaves (image, tag, digest, sha, revision) never get hash tolerance. That is the guard everything rests on: an image tag is v1-45db1889 against v1-e8881d0b, exactly the shape of a generated name suffix, and normalising it would classify a genuinely different build as an environment difference and hide the most important thing the page can show.

Verified in both directions: inkwell-web 0 substantive; inkwell-enrich — which really is on different images in the two environments — reports the image and nothing else. (ledger, rideshare and storefront are also on different images across environments, which this page now makes visible.)

identical means identical after classification, not byte-identical: two renders differing only in a UID are the same rendering, and calling that "changed" is what made the old comparison useless. A component whose objects cannot be read is reported as unreadable and left out of the counts — it is not a component whose environments agree.

Verifying it live found two false claims on the same page, both from token-scoped reads falling through to "equal", and both in the one place someone comes to ask whether production matches staging:

That is the fourth time this session the signed-out rendering differed from the authenticated one in a way that mattered, and the second where the wrong answer was a confident sentence rather than a blank panel. The rule in NEXT-SESSION.md is not optional advice.

The project 360 include from grafana is working, but I would like to remove it and natively expose all of the graphs there. I do want a full featured transactions table, filterable by component/env.

done, and rebuilding it found four wrong numbers in the dashboard. The embed is gone; the project page computes the six headline stats, three server-drawn charts (requests/s, 5xx share, p95, each by component) and the per-component table itself, filterable by environment and window from the page, with the filter in the URL. /projects/{name}/transactions is the full table, filterable by environment, component, producer and window.

The four numbers, because each is a different way to be confidently wrong and none of them is visible from inside an iframe:

  1. Apdex read 0.00 for a project answering in 26 ms. The buckets were spelled le="0.25" and le="1.0"; Prometheus labels that second boundary "1". A selector matching no series is an empty result, not an error, so tolerating was zero and every project scored the worst Apdex there is. Boundaries are now read off the histogram, and apdex_t reports which was used — an Apdex quoted against an unstated threshold compares to nothing.
  2. float("+Inf") succeeds in Python, so +Inf became the "nearest boundary at or above T" for any T, and a histogram without sub-second buckets would have scored a flawless 1.00 forever. Found by writing the test for (1).
  3. p95 for one operation read 16384 ms against a true 1.9 ms. traces_spanmetrics_* is written by two independent systems — Tempo's metrics generator and Beyla's eBPF instrumentation — with different bucket boundaries, powers of two against round numbers. Summing those by le makes the histogram non-monotonic and histogram_quantile returns the top bucket. Wrong by a factor of eight thousand.
  4. The two producers were being added. They disagree legitimately and by a lot: 110 calls against 1147 for the same operation, because Beyla sees every call at the network and Tempo only sees spans that were sampled and exported. source is a column now, nothing merges across it, and the page says why one operation has two rows. The dashboard grouped on a label Beyla does not set, which avoided mixing the buckets by accident and collapsed every Beyla row into one blank-named row.

The other integrated dashboards are also too awkward on build page and we should bring the information that isn't already there in-line, and if we can only view something in grafana, create a more targeted dashboard to integrate.

done. No page embeds a Grafana dashboard any more. Comparing the two embeds panel by panel gave a sharper answer than expected.

The build pipeline dashboard was almost entirely duplicative. Build picker, result, duration, commit, pusher, ordered stages, waterfall, critical path, slowest tests, flame graph, per-stage output, full log — all already inline, the waterfall included, and the console's is filterable by any attribute in the trace where the embedded one is not. One panel was missing: Argo's step durations. prom.build_steps(run) was already fetched, already passed to the template, and dropped — the fifth written-and-uncalled thing this session. It matters because it covers the phases around the container: a build spending four minutes in checkout-source looks instant in the stage view, since none of that time is a Dagger stage.

The Component 360 embed showed three things with no inline equivalent, two of them from recording rules the console referenced nowhere — component:network_bytes_in:rate5m and component:container_memory_rss:sum are computed on every scrape and existed only inside that iframe. Memory as a number was already on the project page; only a series answers "is it climbing since I deployed". Three server-rendered charts now.

The third was traces, and tempo.search() had zero callers — TraceQL across the estate, implemented and documented, while the only trace view in the console was one build's waterfall. There is a recent-traces table now. Tempo returns the literal <root span not yet received> for a trace still being written, which becomes an explicit partial flag rather than printing Tempo's internal apology in a service-name column.

The environment page's embed was the worst of the three: Component 360 for e.components[0] — one component — with a note telling the reader to open Grafana to switch. Every component in the environment is a row now, each linking to its own profile.

What still points at Grafana is one link per trace id, because this page renders a waterfall for the build's trace, not an arbitrary application trace. That is the "can only view it in Grafana" case, and it is a link to one trace rather than an iframe.

A test asserts no template embeds a dashboard at all: that regression is silent, because an embed looks fine until you notice it needs a Grafana session the console's readers lack.

On the build page clicking a test span works, but I'm not sure the flame graph is rendered correctly, it's totally even and flat stacked bars. I don't see the test results logs in that box correctly either.

both addressed. The logs: the span-id decode was returning base64 where Loki wants hex, so the query matched every time and found nothing.

The flame graph was not rendering wrongly. A millisecond-long test span is one sample: every level holds exactly one frame at full width with self=0, so the drawing is N identical bars stacked on each other — which is precisely "totally even and flat stacked bars". spans.js now detects that shape (no branching anywhere and no self time anywhere) and says so instead of drawing it: "one stack, N frames deep, and no self time anywhere — too short to sample more than once, so there is no distribution to draw. Profile the whole stage instead."

Not yet verified against a real build, because it needs a signed-in session on a build that has profiles. If a span with some branching still draws flat, the detection is too narrow and the threshold is the thing to revisit.

I also don't see any better docs with the new markdoc features, I would like to see some interactive stripe style examples demonstrating platform features (how to setup a new project via cli or otherwise for example).

done. /guides — six documents, one per question someone arrives with: setting up a new project (the parenthetical above, and the first in the sequence), finding your way, down to the manifest, what changed, is it healthy, every page has a JSON twin.

Tags, not HTML, and that constraint shaped the design. docs.py renders untrusted prose from application repositories with html=False, and that flag is the only thing stopping a <script> in someone's README executing on a page carrying a session cookie. So interactive examples cannot be built by letting authors write HTML. Markdoc's model is the right one: `

{% tag attr="value" %}

annotations a platform-controlled renderer turns into markup. Five tags —tabs, steps, callout, try, api` — because each is a promise about how it renders.

Attribute values cannot break out of their quotes at all: the pattern is "[^"]*", so an injection attempt truncates into a harmless label rather than closing the attribute. An unknown tag renders visibly rather than vanishing, because a typo an author cannot see is worse than one they can.

try and api are what make these interactive rather than illustrated — every example points at a route on this instance. A test resolves all 25 targets against the app's own router, so it asks "does this route exist in this build" rather than "is upstream reachable". Documentation linking to a route someone renamed is worse than none, and that test is the difference between saying so and meaning it.

Tabs are a radio group plus :checked ~, so they work with JavaScript off — the same reason the DAG and the charts are server-rendered SVG. The guides ship with the console rather than being read from a repository: they describe this build's own routes, version with the code providing them, and render with no upstream call and no token — which matters most for the one document whose job is explaining why other pages look empty.

The omission worth recording: the first five guides covered reading the platform and not creating on it. The parenthetical in the request named exactly that, and it took writing this ledger entry to notice.

Found while acting on the above

Round 6 — the console, again, 2026-09-04 (newest)

Nine requests as one message. They were four complaints with nine symptoms: nothing had an identity, nothing was reachable, nothing was refused politely, and nothing was predicted.

Verifying the plan before building it deleted two of its pieces. Four facts in the first draft were wrong, all four checked against the live cluster:

Done

Found while building, each caught by a test refusing to pass for the right reason

Open

Round 7 — Zulip's telemetry, and Resources as a kind of thing

I thought we were also wrapping zulip spans. [...] when I click on a component in an environment, not a drill down

Two questions, one cause. The platform has two kinds of inhabitant and both the telemetry pipeline and the console modelled one of them.

Zulip's attribution: the fix was pod labels, and it was not Zulip-specific

Zulip reported service_namespace="dp-default-chat-development-03220a9b" with no deployment_environment at all, so every panel keyed on project or environment was empty for it.

Beyla reads POD LABELS, not the process environment. lgtm-beyla.yaml:63-66 maps openchoreo.dev/{component,project,environment} onto service.{name,namespace} and deployment.environment, and the file already names the exact symptom of a pod without them. The first hypothesis here — inject OTEL_RESOURCE_ATTRIBUTES — was wrong; Beyla never reads it.

And Zulip was not special. Checked across the estate: Component pods carry the labels, garage's pods carry {project, environment, resource}, and every CNPG database pod carried none at all — same as Zulip's chart pods. A class, not an incident.

So the fix is at the source, where four consumers read the same labels — Beyla, Alloy's k8sattributes, openchoreo:pod_identity and Loki's stream labels. A transform further down would have fixed one of them.

Verified end to end rather than inspected: zero Beyla series remain under a dp-* namespace, and chat now reports 98 series with deployment_environment="development".

Wrapping Zulip as a Component: evaluated and rejected

It would have made the telemetry worse. The otel-instrumentation trait stamps otel-sdk: "true", which is how Alloy decides to drop Beyla's duplicate spans — so the eBPF signal would be filtered and nothing would replace it, since Zulip carries no SDK. Beyond that ComponentType has no readyWhen, so a HelmRelease reports Healthy the instant it is created — which is why helmchart is a ResourceType, argued in its own header — and vendored-service renders one single-container Deployment with fixed probes against Zulip's StatefulSet, three PVCs and a 300s boot.

Django SDK instrumentation: declined, not forgotten. No OTel Operator is installed, so injection needs one added or a forked zulip-server image; Zulip runs supervisord with uwsgi and ~20 queue workers, the hard case for Python auto-instrumentation; and a fork is against this platform's rule that the application changes nothing. The gain would be real — Django and ORM spans instead of socket-level ones — so it is recorded as considered.

The delivery index only counted components

deployment_index() iterated ReleaseBindings and skipped anything without a component, so a project made only of Resources did not exist as far as the platform's own delivery metric was concerned. chat was absent from openchoreo_deployment_info, and therefore from known_projects(), /deploys, the fleet transaction search and everything downstream. Six running pods the platform could not name. It now emits resource rows carrying kind, with no image, commit or build_run — a Resource is deployed, never built, and inventing those fields is how a Commit column starts showing something that is not one.

The existing database pods — done, and the risk was not real

Deferred once on measured conditions (2GB of 31GB free, swap exhausted, load average 42 climbing, one database pod at 36 restarts) because rolling five databases into that is the documented precondition for the CNPG cascade. Done once load had fallen to 6.93 against a 15-minute average of 32.9.

And the caution turned out to be unnecessary, which is the part worth keeping. inheritedMetadata is reconciled onto RUNNING pods: advancing each binding relabelled the existing pod in place with no restart at all — the 29-hour-old r-bakery-db-staging pod simply gained the labels. There was never a rollout to be afraid of. inkwell-db-production, the cluster that cascaded before, held timeline 39 throughout and stayed 3/3.

The order still stands as the right way to do it — hibernated first, the three-instance production cluster last — but the expensive part of the wait was avoidable, and anyone repeating this should know that CNPG does not need a rollout to apply inherited labels.

Result, checked across the whole data plane: every workload pod in every dp-* namespace now carries openchoreo.dev/project. openchoreo:pod_identity went 63 → 78 pods (inkwell 22 → 27, showcase 8 → 10), and Beyla now names the resource pods properly — inkwell-files rather than r-inkwell-files-staging-acc0f746, which is the service.name fallback doing its job. No r--prefixed service names and no dp- namespaces remain in the span metrics.

Found on the way: chat had the inkwell-files fuse

The cluster ran zulip-65cf9f69b8; apps/zulip/openchoreo/bindings.yaml said zulip-7f6ff95767, five days older. development is the PLATFORM_ENV, which manifest sync does apply — so the next push would have rolled Zulip back to a release nobody chose, silently, with both objects Ready throughout because each was correctly serving what it was pinned to. Exactly the fuse recorded for inkwell-files, on a different project, found only because this change had to touch the pin. Both now name zulip-8fdcf7bcb.

Also still open and now visible: 16 ResourceReleases for zulip alone, which is the unbounded release history already recorded below.

Rotated: the webhook secret

infra/.webhook-secret was committed before .gitignore covered it, and .gitignore never untracks — the value is in this repository's history and stays there (git log -S still finds it in two commits). Untracking made the working tree clean; only rotation made the history harmless.

Rotated 2026-09-03 with infra/rotate-webhook-secret.sh, which moves all four places it is trusted together: the file, git-webhook-secrets on the control plane, project-flow-secrets, and every repository webhook.

Two things it teaches, both learned the hard way on the first run:

Correction: the objectstore fix never reached the repository

Found 2026-09-04 by platform/upgrade-watch, confirmed against platform/capacity/drift.sh --inverted, which reported the inversion as still present:

Resource/inkwell-files  (inkwell)
  staging      inkwell-files-5d96489b4f  2026-08-29T09:49:33Z   <-- MinIO
  production   inkwell-files-5786fd4d99  2026-09-02T22:36:12Z   <-- Garage, 4.5d newer

Verified by reading both releases rather than by inference: 5d96489b4f's spec mentions minio, 5786fd4d99's mentions garage.

The promotion advanced production in the cluster and nothing wrote it down. All three of inkwell's binding files still pinned the MinIO-era release.

Fixed 2026-09-04 by pinning all three to inkwell-files-5786fd4d99.

Which environment the stale pin actually endangered

The first version of this entry said "the next push reverts production to MinIO". That was wrong, and the way it was wrong is worth keeping. sync_manifests skips any *ReleaseBinding whose environment is not the ROOT or PLATFORM environment (controller.py:2568, ROOT_ENV=staging, PLATFORM_ENV=development), so bindings-production.yaml is never applied by a push and production was never at risk from one.

The fuse was on staging, which is synced:

So the mechanism is not "the repo overwrites production" but "the repo owns staging, and a promotion of staging that is not written down is undone." Because the repo is now correct, the next push to apps/inkwell moves staging onto Garage through the ordinary path; no out-of-band command is needed.

inkwell-db was deliberately left alone: it is pinned to inkwell-db-66995c964b in every environment and in the repo, so it is merely behind latest — which production is supposed to be, and which drift.sh correctly refuses to alarm on.

Two lessons:

Open, found while acting on the above

These were not reported; they were in the way.

Round 8 — a Resource's objects are not on the API, 2026-09-04 (newest)

Reported: "Could not read rendered releases. HTTPStatusError: Client error '404 Not Found' for url .../api/v1/namespaces/default/renderedreleases" on /resources/orchard-db-staging/detail, and, from the Now page, "when I click on it orchard-db is red which I assume is the failure".

/renderedreleases is not a route, and never has been

The console read it because a RenderedRelease is genuinely where a binding's rendered Kubernetes objects and their per-object healthStatus are recorded — it is what /releasebindings/{name}/k8sresources/tree reads on a component's behalf. But the OpenChoreo API registers no such path. The string rendered does not occur anywhere in its generated router (internal/openchoreo-api/api/gen/server.gen.go), and the live API discriminates it plainly: every real collection answers 401 without a token and this one answers 404.

releasebindings          401
resourcereleasebindings  401
resourcereleases         401
renderedreleases         404      <- no such route

So /resources/{binding}/detail had never rendered for anyone, on any resource, since the day it was added. The unit tests passed because they stubbed ocapi.rendered_releases — the function existed and was correct about the CR; only the route was imaginary.

There is no substitute. k8sresources/tree fetches a typed ReleaseBinding by name (services/k8sresources/service.go:261) and 404s on anything else; ResourceRelease is an immutable snapshot of the declaration, not the rendering; and ResourceReleaseBindingStatus carries conditions and outputs but no object list. A Resource's concrete Kubernetes objects are unreachable from the platform API. That is the finding. Adding /resourcereleasebindings/{name}/k8sresources/tree, or serving RenderedRelease read-only, would close it — either is a patch to the vendored API, like resource-hibernation.patch.

What the console does instead

It stopped pretending. /resources/{binding}/detail has its own template now and answers the questions the API can answer, which turn out to be the ones people bring to a database anyway: its state, its four conditions with the platform's own message, its outputs — host, port, and which Secret under which key holds the password, never the password — the parameters it was cut with, and the object kinds its type declares. The one table it will not draw is the one whose names would have to be guessed from CEL templates.

Hibernated is not broken — the same mistake, two more places

orchard-db-staging is state: Undeploy. Its rendered objects were deleted on purpose, so Synced=False reason=ResourcesUndeployed, no objects, no outputs. Through any generic health rendering that is a database that fell over.

Found while fixing it: the compare page agreed with itself

render_diff guarded unreadable manifests with isinstance(detail, dict) — but deploy_detail never raises. It catches the upstream failure and returns {unavailable: ..., releases: []}, which is a dict. So the guard was dead: a component whose tree could not be read was compared empty-against-empty, scored zero differences, and rendered as "renders the same", counting toward the page's headline claim that the two environments agree. On the page someone reads before promoting to production. Two fixes: the guard reads unavailable, and "These two environments render the same" is now conditional on at least one object actually having been compared.

The same class of error was in the resource path — both sides came back empty from the 404 route, so inkwell-db was reported as agreeing across staging and production having been compared against nothing. Resources are now named as not compared, with a link to each side's own page.

Open

Round 5 — permissions, releases, and the nav

Confirm my permissions I am seeing this error in the console The platform does not permit releasebinding:update for REDACTED@example.com on environment=production, namespace=default, project=inkwell. That is the role binding, not the console. I should have full admin access to everything on the platform with this account, and everyone on the allowlist should be able to fully use the console minus exec.

done, and the message was wrong. The role binding was never the problem. admins → ClusterAuthzRole/admin → actions: ["*"], cluster-wide, no scope, no conditions; the bootstrap log says REDACTED@example.com already in admins.

The console's ocapi.may() did not speak POST /api/v1/authz/evaluates. Four independent defects, each alone enough to refuse every control for every user, checked against the live spec at ${OCAPI}/openapi.json:

the contract what was sent
1 body is an array of EvaluateRequest a bare object
2 resource requires type + hierarchy a flat dict, neither key
3 environment is at context.resource.environment, namespace-prefixed bare, inside resource
4 response is an array of Decision isinstance(body, dict) — False for a list

The request 4xx'd, a blanket except Exception swallowed it, why_not() printed the swallow as a policy decision. Every gated control in the console was dead, for admins and developers alike, and the sentence sent the reader to look at a role binding that granted them everything.

Nothing had ever exercised the wire: every authorization test monkeypatched ocapi.may itself. There is a test for the request body now, and evaluate() returns allowed / asked / reason, so an unreachable platform is a different sentence and a 503 rather than a 403. /access renders authz/profile — which had zero callers — so "confirm my permissions" is a page.

The deny binding that could never fire

developers-nonprod-only denied releasebinding:create/update/delete on

resource.environment in ["production", "production-dr"]

The platform populates that attribute namespace-prefixed: services/utils.go:17 FormatDualScopedResourceName(namespace, name, false) returns namespace + "/" + name, and releasebinding/service_authz.go calls it at lines 53, 79, 105, 131 and 158 for exactly this attribute. The value compared is default/production, never production. The rule was inert for its whole life, and nobody noticed because the console was refusing everything for an unrelated reason. The working form, recorded in platform/openchoreo-config/authz/scoped-bindings.yaml where the binding was:

expression: 'resource.environment in ["default/production", "default/production-dr"]'

It is not reinstated — the standing decision is that everyone on the allowlist can fully use the console, minus a shell. Reinstating it is a deliberate act.

developers-inkwell-scoped went too. A narrower allow beside the shipped cluster-wide developer-binding restricts nothing — bindings are additive — so it granted a subset of what was already granted while reading, in its own comment, as "a developer cannot create components in someone else's project".

Developers gained project:create (as ClusterAuthzRole/project-author, rather than editing the vendored developer role): the console's scaffold control gates on it and the shipped role lacks it.

Exec

Unchanged, and worth stating: it is not an authorization decision. The OpenChoreo API serves no exec endpoint, so the console prints a kubectl exec line and the reader's own kubeconfig is the control. permissions.py used to claim project-flow re-checked the rule at an /exec handler; project-flow has no such handler and never did.

The releases and deploys are still confusing. They say they take time but there are no logs, no way to tell what will be rendered, what was rendered until it is live. Somehow we are running the diff check.

done. GET …/releasebindings/{b}/k8sresources/tree returns renderedReleases[].renderedRelease — the full RenderedRelease CR — and deploy_detail read nodes out of that response and threw the rest away. So the desired manifests were fetched on every visit to the deploy page and discarded at parse time. spec.resources[].object is what the control plane rendered; status.resources[] is what the data plane made of each one.

Three comparisons become possible and none existed:

  1. Desired against live — drift. One-sided on purpose: only fields the rendering asserts are checked, because Kubernetes defaults terminationMessagePath, dnsPolicy, schedulerName and a long tail more, and reporting those would bury the real thing. Verified against inkwell-enrich-production: zero drift, five objects.
  2. Desired against desired across environments — now the default on the compare page. It used to diff the live objects, which picks up whatever other controllers did after apply and presents it as a rendering difference. Both are available and the page says which it drew.
  3. A projection for an environment nothing is bound in yet — below.

"No logs" was also already answered and unread: ReleaseBindingStatus carries five condition types and ~30 reasons, several of which exist to say why a deploy is slow (ResourcesProgressing, ConnectionsPending, RenderingFailed, ResourceApplyFailed), plus observedGeneration and lastSpecUpdateTime — a deploy start time. The console collapsed all of it to one word. And project-flow has served GET /evidence/binding/{name} since it was written, returning those conditions plus 30 control-plane log lines naming that binding; the console had the client and called it only for builds. A promote now lands on the deploy rather than back where the button was.

if it is possible I would really like the template tool … can generate what manifests a person will see across environments without applying

partly, and the honest part is the interesting one. There is no dry-run render on this platform. Checked rather than assumed: 105 paths on the live OpenAPI and none renders without applying; generate-release calls s.k8sClient.Create and produces a frozen declaration, not manifests; no dryRun parameter anywhere; and RenderResources has exactly one caller, the in-process ReleaseBinding controller. occ has no render command.

So: exact for any environment where the component is bound — that is the real RenderedRelease, and it is what the deploy page and the compare page now draw. Projected for one where it is not: derived from an environment that does exist, with the target namespace, environment name and environment-uid substituted and the target binding's workloadOverrides applied. Labelled a projection in a banner and per field, because two things cannot be derived:

Validated by projecting inkwell-enrich's real staging render forward and diffing it against what production actually rendered. That found two bugs a fixture would not have: the environment-uid appears in five places on a real Deployment (spec.selector.matchLabels, spec.template.metadata.labels, spec.selector, spec.podSelector.matchLabels) and the first version substituted only the metadata label — which would have produced a selector matching nothing.

The template tool itself cannot project: a template describes a project that has never existed, so there is no real rendering anywhere to derive from. The page now says exactly that, and says where the answer does live.

the nav is a mess, I have yet to see evidence of diffing, I don't even know how to find it

done. The diffing was /projects/{name}/compare, reachable from one link at the bottom of the project page — and the compare route never passed subtabs, so arriving there dropped the reader out of the navigation entirely. Nine pages were in that condition:

page why it was unreachable
/transactions in no tab list; its one href appended a required ?q=
/projects/{n}/builds, /deploys, /logs, /resources listed in project_tabs(), but no page a reader could arrive on rendered the strip — the project overview did not pass subtabs. No inbound link from any template at all.
/projects/{n}/compare the diffing. Same cause
/projects/{n}/graph same
/projects/{n} listed as "Overview", rendered no strip: a one-way exit
/projects/{n}/environments/{env} in no tab list at all

The template tool was never removed — it was demoted from the rail to a tab under Platform, and its own detail page rendered no strip, so once inside one there was nothing on screen saying where you were.

test_every_navigation_target_is_a_real_route checked nav → route. Nothing checked route → nav, which is exactly how these drifted and stayed drifted. test_every_page_route_is_reachable_from_the_nav does now, with an explicit allowlist for detail pages so exempting one is deliberate. It found a tenth page on its first run.

Found while fixing it: a resource's release history rendered empty, always

/resources/{r}/releases passed releases= and the template read history. Jinja's Undefined is falsy, so the page printed "No ComponentReleases recorded" for every resource that has ever existed — on the only tab a Resource has. And for_resource emitted changes where the template read rows, so the field table would have been empty even once the list arrived. Neither name was checked anywhere: the tests called for_resource directly, and the nav test asserted the link resolved. A route can exist, resolve, return 200, and render the empty state for data it was handed.

Found by reviewing the fix: nine defects, two of them this round's own kind

A /code-review pass over the whole change set found nine. Two were the exact failure this round exists to remove:

And: the projection left the source environment's uid in the selector, the pod template and the podSelector whenever the target uid was unknown (the normal case) — the five-places bug in a second disguise, and the re-walk written to detect it silently found nothing because a label key contains dots; holds_everything ignored denials on the page whose purpose is that deny wins; /access reported an outage to anyone signed out and forwarded component without project; drift paired across kinds; and apply_overrides was dead until wired to the promoted-but-not-yet-reconciled case.

The lesson worth keeping is that a change set built specifically to stop confident wrong answers produced three of them, in the code written to stop them. Reviewing your own fix is not optional.

Open

Round 9 — nine asks at once, 2026-09-06

Why can't I trace more via network headers between things? I kind of expected we would be able to easily trace traffic that passes over the internal network, in and out of the ingress by trace too.

done, and the answer was one policy object and one line of chart values. kgateway/Envoy had no tracer, so the first span in every trace was the application's own server span — Envoy with no tracer emits none and does not synthesise a traceparent; it only generates x-request-id, which nothing collected. Beyla could not cover it either: its discovery is k8s_namespace: dp-* and the proxy is in openchoreo-data-plane.

platform/openchoreo-config/gateway-tracing.yaml adds a second HTTPListenerPolicy (the chart's own is Helm-owned) with Envoy's OTel tracer and a JSON access log carrying the traceparent.

The part that cost the time: Envoy's OTel tracer speaks gRPC only, so it dials Alloy's 4317 — a port nothing had ever used, because every application here speaks HTTP to 4318. kgateway built the upstream cluster from the Service, defaulted to HTTP/1.1, and Alloy rejected all 24 exports. And the symptom says the opposite of what is happening:

tracing.opentelemetry.spans_sent: 37
tracing.opentelemetry.spans_dropped: 0     … with nothing in Tempo

spans_sent counts spans handed to the exporter, not delivered. One level down: upstream_cx_http1_total: 24, upstream_cx_protocol_error: 24. The fix is appProtocol: "kubernetes.io/h2c" on Alloy's otlp-grpc port.

Verified: {resource.service.name="gateway"} returns traces whose services are console,gateway, and the access lines carry a joinable trace_id in Loki.

Not closed: the browser sends no traceparent (no OTel JS in the SPAs), so the trace begins at this network's edge rather than at the user's; and Garage still starts a fresh trace rather than continuing its caller's.

The build page should be focused on the part(s) that failed automatically… clicking a span should show its attributes and events in line in a more tightly formatted box… every major section should be collapsible and there should be a mini nav… buttons to download logs and profiles and traces in a standard format.

done. A failed build opens on the first FAILED stage in pipeline order — cause before consequence — and on the first failed span inside it, which is the exec carrying the command and the exit code rather than the stage span saying "stage failed". Only when the reader asked for nothing: an explicit ?stage= is a link somebody sent. This also made ?span= do something; it had been accepted, documented and read by nothing.

The span detail panel moved into the waterfall. Same element, same renderer — it simply sat a screen and a half below the row that selected it. The SVG leaves a gap under the selected row the way a tree table expands a node.

Sections collapse and there is a mini nav, both read off the <h2>s already on the page rather than written into nine templates — so it cannot desynchronise from the markup, and it applies to the project and compare pages for free.

Downloads: logs as text or NDJSON, the trace as OTLP or as a file Jaeger's UI imports with no server, profiles as real gzipped pprof, speedscope JSON or folded stacks. No new upstream call — every one is a conversion of a response the page already fetches (app/exportfmt.py), and the pprof encoder was validated against github.com/google/pprof/profile, which is the only way to catch leaf-first ordering.

I got a timeout trying to view the delivery cache page. Evaluate if everything is efficient, I'm open to adding a database to the console when it is required.

done. /cache read a trace and up to 4000 log lines per build, serially, for up to sixty builds, on one thread at a 20s timeout each. 121 requests. It is arithmetic, not a mystery.

Same reads, run concurrently, with soft applied per call so one build aged out of Tempo does not blank the other fourteen. The ceiling drops to thirty. Three more of the same shape fixed: project_resources (one deploy_detail round trip per binding), the transaction page's N+1 analysis, and prom.builds — the most-called source in the console — which had no cache at all.

And a database, console-db, on the platform's own postgres resource type. It holds facts derived from the stores the console already reads, about builds that have finished. A miss re-derives; the store being down is a slower console and never a broken one. It is also devtools' first Resource: the console rendered everybody else's databases and had none of its own.

I can't test the rollback/releases history, I get Nothing to roll back to… suggesting maybe we are not recording and linking history of releases correctly.

done, and there were three separate faults.

  1. The console asked a different question from the button. It decided from the history ring; project-flow decides from the previous-release annotation. The ring is newer than most of this platform, so eight production bindings carried the annotation, no entries, and rendered as refused while POST /rollback would have succeeded.
  2. Nothing recorded autoDeploy. Only promote() and rollback() wrote the annotation, and neither is the writer for the root environment — so staging, the environment every change lands in first, could never be rolled back. observe_bindings() now watches every binding and records a move it did not make, with verb: observed and the mechanism rather than an invented actor.
  3. The rollback button ignored the environment. flow.rollback sent "to": "" and project-flow reads body.get("to") or PROD_ENV, so every rollback this console ever issued moved production — including from ?environment=staging, which rendered staging's plan quite correctly above it.

Plus a release picker: rollback() has accepted an explicit release since it was written and no page ever offered one, so the only way to use it was curl.

Proven end to end on bakery-docs-production, which had no history at all that morning:

14:22:10 promote   85fdc5cc57 -> 65fbff7d6   by auto
14:21:53 rollback  65fbff7d6  -> 85fdc5cc57  by session-verification
13:04:06 promote   85fdc5cc57 -> 65fbff7d6   by auto
13:01:51 observed  (unknown)  -> 85fdc5cc57  by autoDeploy or an out-of-band patch

On /projects/rideshare I see binding not readable

answered. It is the deliberate degraded path, not a rideshare fault: the node has a component and no binding name, so the object-graph and log links are suppressed rather than rendered as 404s. Bindings are token-scoped platform API reads, so signed out — or with a token that cannot see the binding — every flow view renders this.

the small medium large labels should link to the platform docs on what that means wherever it appears

done. There was no such page: the prose lived in a bash comment in generate.sh, in each generated type's description, and in one binding's YAML. /guides/size-classes says the thing that most needs saying — a class is relative to the COMPONENT TYPE, so a Rails large and a Go large are different absolute numbers — plus the two pieces of arithmetic that bite (max(sum(containers), max(initContainers)), and maxSurge rounding up while maxUnavailable rounds down). One sizechip macro replaces three hand-rolled copies.

Let's remove the orchard project, document the problem it was meant to show… and then remove it from the platform.

done. Documented as /guides/three-kinds-of-slow — written with the query or download that answers each case rather than screenshots, because a screenshot of this platform is out of date by the next deploy and go tool pprof -top <(curl …) is not. The endpoints moved into inkwell-web, which is the component the Ruby profiling panels already name and the one built with profileRich: true.

Found in the move: db:seed does not run in the cluster. db:prepare seeds only when it CREATES the database; against an existing one it runs pending migrations and stops. /widgets returned {"widgets":[]} with a 200, the table present and empty, nothing reporting a fault anywhere. Orchard got away with it because its database was created by the rollout that first ran its seed. The walkthrough data is a migration now.

Then removed in dependency order; the ownerRef cascade took 69 ComponentReleases and 33 WorkflowRuns with it. The Gitea repository is archived rather than deleted, and the test fixtures named orchard stay — each names the object a real bug was found on, and renaming them would turn a measurement into a story.

We need to get the staging environments working so we can see promotions work their way through and exercise manual promotion and rollback.

done, and it was one field. All twenty staging bindings were Undeploy, and promote() requires is_ready(source) — so no promotion could succeed anywhere on this platform. showcase and chess are awake (inkwell stays asleep; host memory is the binding constraint). Waking showcase immediately triggered a real auto-promotion, because it carries promotion: auto with a 2m soak — which is the platform working, and it moved production forward by three days.

Also: sync_manifests now refuses a pin that would move a binding backwards, comparing the two releases by creation time. Root resource pins are applied on purpose — occ resource promote cannot create the first binding, so the repo genuinely is the source — but a stale file and a deliberate rollback are the same diff, and only one of them should happen by push.

there are too many links that are just the project or component name when it actually links to some sub resource… Survey the console for other instances.

done. Eighteen places, always the same shape: the page wrote down the identity of the row's OWNER instead of the row's. /builds listed the component three times over, each a link to a different build, so the only thing telling two rows apart was the timestamp. There is a Build column now reading console@a1b2c3d4, and a Deploy column on /deploys going to /deploys/{component}/{environment}/{commit} — a route that existed the whole time with one inbound link.

The enabling change: /components/{name} now exists. nav.component_tabs documented its absence since it was written, and that absence is why component names linked to /timeline or a filtered list — there was nowhere honest to send them. A template lint keeps the rule.

We need to figure out exactly how rolling out changes to ComponentTypes will work… consuming projects pin the component type version… upgrading would happen via the same git path.

partly done — the code is written and not yet applied. See platform/openchoreo-patches/construct-versioning.patch and the rewritten §"Pinning a version" in VERSIONING.md.

The blocker worth knowing: Component.spec.componentType was self == oldSelf immutable, which freezes the version along with the identity. Every pinning scheme needs that relaxed — including the "version in the object name" fallback, which would otherwise mean deleting the Component and its releases with it. So the choice between schemes was about ergonomics, not about whether to patch.

Resolution is by label, not by a name suffix: metadata.name has to be unique and the logical name has to be stable, and one string cannot be both. A name-suffix scheme makes the FIRST version's name wrong and makes adopting v2 a rename rather than an addition.

Every platform-authored construct now carries openchoreo.dev/{name,version}, so kubectl get clustercomponenttypes -l openchoreo.dev/name=ruby-service already lists the versions that exist.

Open

Round 10 — the pipeline you can see, 2026-09-07 (newest)

Nine asks at once. Underneath them is one, in the user's own words: "the links between commits and builds and releases of projects to environments and between releases in time are either not there, too hard to reach, visually hidden. In my mind navigating that chain is the 'pipeline'."

Planned as six phases in this order — queue and gate truth first, then the Now page, then build outputs and the release chain, then the delivery graph, then profiling. What follows is phases one to three.

Done — the build queue tells the truth

Builds listed in queue in UI even while I can see they have already run quite a bit of the build. We need to track time waiting in queue on each build. I was imagining something more like a table with running, in queue, with timers showing how long each had.

done, and it was a real wrong answer rather than a display preference.

run_summary() decided queued from the WorkflowRunning condition, which is OpenChoreo's observation of the Argo workflow leaving Pending — so while that observation was outstanding, a build with pods and scrolling logs reported itself queued. The honest discriminant was already used twelve hundred lines away in the same file: build_index() takes the queue's end from min(status.tasks[].startedAt), because Argo does not schedule a step for a Pending workflow. Both signals are kept; either one means running.

The second half is that nothing carried the wait. status.startedAt is ADMISSION, not first work, so every elapsed time the console drew for a build in flight was wait-plus-work labelled as the work — "Started 3m ago" on a build forty seconds into its first step. run_summary() now serves running_since and queued_seconds, measured from the same instant as the _queued metric step so a live figure and a recorded one cannot disagree.

Three places show it: the front page's new build panel, /builds (a Waited column beside Took, with Took now excluding the queue), and the build page's banner and header. Verified end to end on two real builds — chess-opening-run-qcheck01 waited 42s behind brochure-run-qcheck01 and records 42s waited, 53s of work.

Two things found on the way, both fixed:

Done — a gate says why, and what to do about it

Also the gates section should be very prominent about the reason and what you need to do to fix it, fix whatever we need to do for this message "project-flow sends only the keys of this list…"

done, and the one-line change was necessary and nowhere near sufficient.

/state did serialise pending_promotions with list(...), which sends the keys — that is now dict(...), along with pending_reviews, which had the identical defect. But neither writer set a reason, so the em dash would have survived the fix. Both writers record a reason and a remedy now, written at the branch that made the decision, because the numbers that make a reason useful ("74s of a 120s soak") exist only at the point of the refusal.

The larger finding: pending_promotions is a work queue, not a gate list. auto_promotions() has six ways to not promote and parked exactly one of them. A soak still running, a binding that is not Ready, a hibernated source, a failed smoke check, a full quota and a retry backoff were each a bare continue; two of the six reached Zulip and nowhere else. So "why has nothing promoted" had no answer outside that process.

held_promotions is a second, separate key — separate because reconcile() DRAINS the queue, and an entry it cannot complete would break that loop. It is a report rather than a queue: rebuilt whole every pass and swapped in, so a gate that has cleared is gone because it was not written again, and nothing needs a TTL. Six gates, each with its reason, its remedy and a clock where the gate is a clock. Eight self-test cases, and install.sh refuses to install past them.

On the console the section is cards rather than table rows — the useful content is two sentences of prose, which in a table was a wrapped paragraph beside three near-empty columns — ordered worst-first, because sorting alphabetically buries a failed smoke check under three soak timers. The badge counts blocked separately from parked: parked clears on its own, blocked needs somebody, and a soak timer lighting the badge on every ordinary promotion is how a badge stops being read.

Found while fixing it: the promotion grid's gate chip had never rendered once. It tested f"{project}/{component}" in waiting against project-flow's keys, which are auto:<component> — neither spelling can equal either key. The match is on the values now, and the chip says which gate rather than the word "waiting".

Done — the front page shows what is building

I expect the build queue and actively building to be larger and more prominent and quickly updated on the now page with a mini streaming logs panel for the running builds. Also to display builds in cards.

done. The front page showed no in-flight build at all; the only thing rendering any of it was the badge in the rail, three at a time, inside a title= attribute. There is a Building now section with a card per running build — component, commit, project, the step it is on, work and wait as two numbers, and the last eight lines from Loki — above an In queue table carrying position and wait. Cards for the running ones because a running build has things a finished one does not; a table for the queue because position and wait are exactly what compares down a column.

Its own poller at /api/now/builds, and it never self-terminates, which is the one way it differs from the build log fragment. That one ends its loop by omitting its trigger when the build finishes, because a finished build never starts again; this panel's whole job is to notice the NEXT build, so it slows from 3s to 10s instead.

loki.build_tail is not build_all with a small limit: Loki applies limit in the direction it reads, so a forward read of eight lines returns the checkout banner for the whole life of the build.

Found while building, each caught by a test refusing to pass

The queue reading is as fresh as OpenChoreo's mirror, and that is a real limit

Watched live across two pushes. console-run-6ff8e55c queued behind openchoreo-bot-run-7f977efc and reported its wait climbing 8s, 29s, 49s, 69s — then flipped to running and froze at 45s, which is the measured first_task - admitted. Both numbers are correct for what they are, and the gap between them is the point: while a build is queued the figure is now - admitted, and it keeps counting for as long as it takes OpenChoreo to mirror Argo's task list into the WorkflowRun. Measured at roughly 20-30 seconds here.

So a queued build's live wait can over-read by about half a minute, and the recorded one never does. Reading Argo directly would close it, but that is a workflow-plane read from a control-plane process for a number that only matters while a queue is minutes long. Stated rather than fixed.

The same lag is why queued cannot simply be "no tasks yet": for about half a minute after work begins, the WorkflowRun genuinely says there are none. That is what the WorkflowRunning condition is kept for — either signal being true means running, and between them the window where a build looks held while it is not is as small as this data source allows.

Done — the chain from a commit to a manifest

Starting at the build page, there should be a section listing the outputs of the build… be very explicit about what component releases are contained… For each individual release I want to see at a glance what changed in the CR, but also a diff of the Kubernetes layer changes.

done, in four places.

A build page says what the build produced. Two halves, because they are two kinds of output: the ARTIFACT (the image and its digest, what BuildKit made) and the PLATFORM OUTPUTS (the ComponentReleases OpenChoreo cut from it, which are what actually gets deployed), drawn as the same card the object graph and the estate views draw and coloured by whose change each was.

Joined on the digest, and that is the whole of why it works: the tag is v1-<sha>, so a rebuild of the same commit publishes the same tag over different bytes. prom.builds was dropping the digest label in its own by() clause. Several matches are the ANSWER here — one build with siblingWorkloads produces three releases — which is the opposite of by_image, whose job is to name one release and which refuses to guess.

A release page says how it got here. commit → build → release → binding → rendered release, as the same DAG three other views already emit. The commit comes off the IMAGE TAG rather than off the build, which makes it exact for a release of any age while the build ages out of Prometheus in a week; when it has, the build node is still drawn and says "not recorded", because leaving it out shifts every column and implies there never was one. The build edge is dashed: nothing records "that build cut this release", so it is a match on content, and every solid edge is the platform saying so itself.

And a diff of the Kubernetes layer, which could not be answered at all — releases.html had said so in prose for months. A RenderedRelease is stored per BINDING and holds only what that binding serves now (36 here against 720 ComponentReleases) and there is no dry-run render, so the manifests of a superseded release are gone the instant the binding moves. Not hard to find: absent.

So the console keeps them. A new history kind, rendering — and the only one in that table whose upstream has no retention window at all; every other kind forgets in days. The console PULLS: project-flow posting them was the obvious arrangement and it cannot, because from inside the cluster it gets a 403 for console.jung.town the same way grafana.jung.town does, and the console has no in-cluster address the control plane can name. Measured, not assumed. project-flow serves a cheap index instead — one row per (binding, release), no objects — and app/archive.py asks for bodies only for pairs it has not seen.

Verified on inkwell-web-production's real manifests: nine objects compared, the two mutated fields found, nothing misclassified as environment or identity noise.

The per-component release list gained the same vocabulary: whose change each release was, where it is serving (or that it is serving nowhere, which is normal and reads as a fault when blank), and a link to its manifest diff.

Found while building it

Done — delivery is a rail, and a pipeline run has an address

For the delivery timeline view I was still hoping for something a little bit more like git log --all --decorate --oneline --graph (but not ascii, more like SourceTree visually but matching our style) where timestamps and commit cards are on the left and you see the linked events that spawn off them.

done. The events were chips crammed into the right-hand side of a commit card, so a commit that built three components and deployed to two environments was five facts on one line with no order between them. Each is a row now, hanging off its commit by a curve, in the order it happened — and a build sorts before a deploy at equal times, because that is the causal order and a coarse timestamp must not print a deploy above the build that made it.

A lane is a REPOSITORY, not a branch, which is the one thing the picture must not be read as. A project is not a repository — inkwell is six components across four — and ancestry is only defined inside one, so a line here means "the same repository's history, in this order" and nothing stronger. /components/{c}/graph is the drawing that does know about branches. More repositories than lanes says so rather than folding two into one lane, which would read as one history.

A gate is a row, and it is the valuable one: a commit deployed to staging and held out of production looked identical to one that had finished.

The segment vocabulary and the per-row SVG are gitgraph's, deliberately — one tall rail beside the rows points at the wrong commit the moment a row expands.

If we could identify and formalize what an individual instance of a "pipeline run" and give it a dedicated page you can reach that could be good but strongly linking all the concepts here must come first.

done, and the linking did come first — Round 10's build outputs and release chain are what made this answerable. A pipeline run is defined here:

commit -> WorkflowRun -> image+digest -> ComponentRelease(s)
       -> ReleaseBinding per environment -> deploy
       -> [gate] -> promotion -> ReleaseBinding -> deploy

Environments are not nodes on it; they are what a binding IS, and adding them would draw the same fact twice. A gate is a node.

/pipelines/{run} is not the build page and not a copy of it. That page is about the BUILD — its stages, logs, profiles, what it cost, why it failed. This is about the DELIVERY of what it produced, which is a different set of objects and the question a reader actually has when the build is green and the change is not in production yet. Not in the nav, for the same reason a release is not: "a pipeline run" is not a place you navigate to without having one in mind.

Found by the template lint while writing it: a component name linking to a build, which is the exact defect eighteen places were fixed for in Round 9.

Partly done — the review-environment flow

If we could optionally show the nested review environment flow of the same (toggleable) so you would see the delivery pipeline of a PR before it was merged too in line before the squash.

partly. A commit that is not on the default branch is marked on the rail now, and three-valued rather than two: off main, ancestry unknown (Gitea could not be read, which says nothing either way), or neither. delivery.for_project keeps those rows rather than dropping them, so a pull-request commit that built is already in the picture.

What is not done is drawing the PR branch in its own lane with its review environments, rejoining the trunk at the squash commit. It needs a Gitea walk and a merge base per repository — gitgraph._gather does exactly that for one — and it is left undone for a reason worth stating: there are no open review environments on this platform right now, so it could be built and not verified. A view of a flow nobody can look at is how a confidently-wrong drawing ships.

Open

Phase six — profiling, and what the oracles caught

The profiling work landed. What is worth recording is not that it works but what it found, because two of the three findings were things that had been running, green, for some time.

The Python memory switch was wired to nothing on six of seven services. memoryProfiling had been on the continuous-profiling trait since it was written; the profiling dashboard offered it and docs/OBSERVABILITY.md documented it. Exactly one service read PYROSCOPE_MEM_ENABLED. The other six carried a near-identical profiling.py, and the console had an eighth copy under another name. Deleting the duplication into platform_telemetry 0.6.0 fixed it everywhere at once, and the trait's default flip to true is a MAJOR by docs/VERSIONING.md -- so continuous-profiling-v2 at 2.0.0, with the default label moved off v1, and 2.1.0 adding the two heap knobs the file's own prose had been promising before anything could read them.

Ruby has no Process.getrusage, and a rescue turned that into a column of zeroes. The new "ceiling" column -- ru_maxrss delta, the one per-test memory number that cannot be negative -- was written with the Python and Go spelling of a method Ruby does not have. rescue StandardError catches NameError like anything else, so every example of every Ruby build reported a ceiling of exactly 0, the attribute was emitted, and the panel would have ranked a Rails suite entirely by zeroes while looking completely normal. Nothing reported it.

What reported it, on the first build after it was written, was the oracle spec landing in the same change: an example that asserts a number it already knows (64 MiB held, ~1s of CPU in one frame) and has no rescue to hide behind. It failed with the NameError in plain sight. That is the entire argument for oracle tests and it paid for itself immediately. All three languages now have a pair, and all three check out against the panels: Ruby reports 64.1 MiB for the 64 MiB example, Go reports 64.0 MiB of churn and 1.0000s of wall for its two.

Go had no per-test records at all, so its oracle was invisible when written. platform/gomod/citest is the third platform-owned library beside gems/ and pylib/. The interesting part is that Go's testing package has no per-test hook and never has -- testing.M keeps its test list in an unexported field and MainStart's deps argument is an unexported interface, so a wrapper cannot rebuild an M from outside either. Of the three ways round it, running m.Run() per test would have broken the merged CPU profile the CPU oracle depends on, and parsing go test -json cannot see memory at all. So it reaches the field by type through reflect and unsafe, degrades to today's behaviour if the layout ever changes, and buys the risk back with a test that asserts the wrap took against the toolchain in use.

And the leak was the profiler. The controlled experiment started on 2026-09-06 has an answer: switching profiling off took inkwell-worker from +47.0 MiB/h to +1.9 MiB/h with its traffic unchanged, while the untouched control kept leaking and was OOMKilled four times in a day. Written up in docs/FINDINGS.md. It has a direct consequence: inkwell-worker-profiled was built to profile live traffic harder, so on this evidence it will leak at least as fast as the thing it diagnoses. It is bound and running in staging, and its production binding is written, sized and deliberately not promoted.

Round 11 — the corrections

Nine asks arrived as one message. Three of them turned out to be bugs with a single root cause each, and those are done; the rest are planned and open. What follows is the ledger for the three, including what was found to be already true while looking, because two of the items in this file were stale.

Done — the delivery timeline's git rails were sticky

base.html declared .rail unqualified, carrying position:sticky; top:0; z-index:10 and six more properties. The delivery timeline and the commit graph each draw one <svg class="rail"> per row — 112 of them on /projects/inkwell/delivery — so every git rail was pinned to the top of the viewport at the header's own z-index and rode over it on scroll. svg.rail further down reset display and flex; the rest leaked.

Renamed to .toprail rather than reset. A reset has to null every property the header rule will ever grow, and the next one added would quietly reach the git rails again.

A test holds it: any bare, positioning class selector in base.html must match one kind of element. Scoped to positioning deliberately — .mono and .faint are utility classes meant to work on anything, and a check that flagged those is one nobody reads. Verified by reintroducing the bug: it reports rail positions <div>, <svg>.

Done — the timeline clipped where it should have scrolled

Rows are one flex line at a fixed --rowh, and they have to be: each draws its own rail SVG exactly that tall, and the lanes only join up because every row agrees on the height. So a row cannot wrap, and overflow:hidden on it meant the chips, the environment pill and the timestamp were cut off on a phone with no way to reach them. The container scrolls now and rows carry a min-width.

Found beside it: table { display:block; overflow-x:auto } existed only below 640px, so between 640 and 900 a wide table pushed the whole page sideways — the exact symptom that rule exists to prevent, on a tablet.

Done — every bot link lands in the console

for_transaction is appended to every reply from every bot on this platform, and two of its four links went to Grafana. The argument against that was already written down when [transaction] moved: Explore shows one signal at a time, needs a query, and does not open at all for a Grafana Viewer — it silently redirects to the home dashboard. Nothing in that argument was specific to [transaction]. [profiles] and [component] were an unfinished change.

Also rewired: project-flow's four hand-built GRAFANA_URL announcements (the constant is deleted, not left unused), the Gitea commit statuses — Argo while pending, the Grafana build dashboard when finished, both now the console build page with a failed stage deep-linking to ?stage=&span= — the walkthrough and conductor bots, TechDocs, and five hand-encoded Explore URLs in the console's own templates using Explore's superseded ?left={json} form.

Four of those five were trace ids, and /traces/{id} had zero inbound links anywhere in the console. It was built precisely because every bot link went to Grafana, and then the console's own links were never rewired.

The alert relay's [logs] link has been a 404 since it was written. {CONSOLE}/logs?project=… is not a route — logs are /projects/{project}/logs — so every alert this platform has posted carried a dead link, and the query parameters it assembled so carefully were read by nothing. Its [component] link pointed at the project.

platform/dev/check-links.py now checks three failure modes: a Grafana dashboard URL built outside links.py, a console URL naming a route that does not exist, and a query key FastAPI would silently discard. The third found a bug written minutes earlier — ?project= on a route that reads no such parameter — which is the argument for having it. # links-ok marks a deliberate escape hatch per statement; four survive and each says why.

It is a script and not a test because it must see the whole repository at once and nothing here can: each test container mounts only its own directory, and two of the three offending builders live outside all of them.

Done — the stage tiles lagged the log, and it was an absence not a lag

A Dagger stage span reaches Tempo only when the stage ends — the module force-flushes per stage — so a running stage has no span, no tile, and the panel showed the pipeline as it was one stage ago while the log beside it streamed that stage's output.

Three sources know different halves, and each is now used for what only it knows:

source knows freshness
Tempo which stages have finished accurate, late
Loki ci_stage stream label which stage is writing now live
Argo task phases, via project-flow which workflow step is running live, coarse

Loki's stage counts only when Tempo has not already closed it. Measured on console-run-cca9f975: test wrote 584 lines over 45s, so its label tracks in real time — but lint, audit, image and publish wrote one to three lines each, all at the instant the stage returned, because what carries ci_stage for those is the stage function's return value. Reading that as "still running" would be a worse lie than the staleness it replaced.

Verified live on console-run-ae795ab1: the running tile read lint 2.3s — a Dagger stage name, which Argo never uses, so it can only have come from the log — while Tempo still had no span for it. The finished tiles then filled in behind it: [] → [lint] → [lint, audit] → [lint, audit, test].

The poll is 2s and the live cache TTL is 2s because they are one decision. A 3s poll against a 5s cache re-rendered identical bytes two times in three: the page worked harder, the upstreams saw nothing extra, the reader gained nothing. builds.FAST_POLL_SECONDS is now derived from _cache.LIVE_SECONDS. The cost is bounded by the build semaphore — one build at a time, one shared cache — so this is one trace read and one log read every two seconds however many tabs are open on it.

No progress fraction, still — and now by choice rather than by absence. The strip first drew only the tasks Argo had CREATED, which grow as a build runs, so boxes appeared one at a time and it could not say what was still to come. project-flow now serves the WorkflowRun's spec.workflow.name — which was on the object all along and served to nobody — and the console reads that ClusterWorkflow's runTemplate, so the whole pipeline is drawn from the first tick with the steps ahead dashed and dim.

The denominator therefore exists, and a bar is still the wrong drawing: "2 of 3" says less than three named boxes with one lit, and it implies the steps are equal when build-and-publish is most of a build and generate-workload-cr is seconds.

Two traps in resolving it. A WorkflowRun references fastapi-builder while the object is fastapi-builder-v1, because constructs here are versioned and carry their unversioned identity in the openchoreo.dev/name label — matching on metadata.name finds nothing, silently, and an outline that never matches renders exactly like a console that was never given one. And the outline is optional throughout: the rail's live badge has no token and must not acquire a platform-API read to draw a count, so without one argo_steps degrades to the behaviour it had before.

Watched on a real build:

t+12s  checkout-source:running    build-and-publish:pending    generate-workload-cr:pending
t+36s  checkout-source:succeeded  build-and-publish:running    generate-workload-cr:pending
t+96s  checkout-source:succeeded  build-and-publish:succeeded  generate-workload-cr:pending

with the Dagger stage inside build-and-publish moving independently across the same window — audit, then test, then publish — which is the two layers doing what each is for.

Two items in this file were stale, and are struck

Found while building, and worth keeping

Done — a trace reads as a network path, and its id survives being pasted

/traces/{id} was serving BUILD pages for real traces. Tempo's search API returns trace ids with leading zeros stripped, and on this platform that is not a curiosity: across 120 traces from the three producers, not one came back at 32 characters — 28 to 31, with the gateway's clustering at 31 and the SDKs' at 29. Verified against the bytes, since a count like that needs proof rather than a theory: search returns d5f6a58d34279732276bed266c29 and the span inside that trace carries AADV9qWNNCeXMidr7SZsKQ==, which is 16 bytes hexing to 0000d5f6a58d34279732276bed266c29.

looks_like_trace_id anchored on {32}, and the route redirects what it does not recognise to /builds/{id} — reasonably, since a build run name is the thing most often pasted in its place. So nearly every real trace id rendered a build page under a trace's heading, reporting "0 spans" and "Outside retention". Found by opening a gateway trace, not by a test.

Length is no longer what separates the two; a run name carries a dash and hex does not. Ids canonicalise to the padded form and the page 301s onto it.

tempo._fetch discarded what makes a span a network hop. It kept three resource attributes — service.name, service.namespace, deployment.environment — and dropped the rest including telemetry.sdk.name, along with span.kind. So a Beyla span and an application span were indistinguishable in the console, a gateway span was identifiable only because "gateway" happens to be its service name, and SERVER-vs-CLIENT was not in the span dict at all. Gateway tracing, Beyla and Hubble were all configured, working, and undisplayable.

Both are kept now, offered as facets even when unanimous — a trace where every span is beyla is telling you something — and the waterfall is coloured by hop. Verified on a real 30-span trace: amber Envoy ingress and router, green GET /projects/{name}/delivery from the console's SDK, blue for the 21 outbound calls it made. inferred is drawn quietly and separately on purpose: Beyla sees sockets and not intent, so drawing it like an SDK span would promise something it cannot deliver.

What the trace cannot show is written under it rather than left to be rediscovered: the browser, an object-store hop, and context through an uninstrumented process.

Done — network tracing verified as a table

platform/dev/check-network-tracing.py, over four hours on this platform:

envoy          200 traces   gateway only
opentelemetry  200 traces   bakery-worker, chess-*, console, inkwell-*, ledger
beyla          200 traces   brochure, storefront, zulip, zulip-redis, bakery-docs
hubble_drop_total present   2699 dropped flows in 4h

No service appears under more than one producer, which is the result that matters — the SDK and Beyla covering one process would double-count it, and three separate mechanisms exist to stop that. Beyla covers exactly the services with no SDK, including the two SPAs that carry no OTel JS.

A service with no coverage is printed and is not a fault: an idle service looks identical from here and this cannot tell them apart, so it says so.

Done — the release chain, in time

Most of it already existed and was invisible. One trace id already spans the push span, the build root, every stage and project-flow's deploy children. Two things stop it being one trace and both are real: a promotion made later is a different causal chain, and Tempo keeps forty-eight hours while the deploys are kept fourteen days and the release for ever.

So /pipelines/{run} gets a timeline of segments, each naming the trace it belongs to. inkwell-enrich-run-d7480a7d reads: pushed, six minutes queued, one minute built, staging, production — 21m end to end. The waits are drawn to the same scale as the work, because on that chain the waiting is the story.

Its times come from Prometheus rather than the WorkflowRun, which Argo prunes after a day — so for six days in seven the durable set is the only set, and this page lost its whole middle without it. seconds is measured from admission and already contains the queue, so queued_seconds is subtracted: 512s elapsed is 415s waiting and 97s of work.

Done — pull requests get a page, and review environments stop being estate

/pull-requests and /pull-requests/{slug}~{n}, under Delivery rather than beside it: a pull request is the part of delivery that is somebody's own work.

A review environment holds more than the pull request changed, which is the finding that made the page honest. Verified on a real pull request rather than a fixture — apps/ledger #1 produced an environment with nine components. One, ledger-pr-1, was forked from the branch; the other eight are inkwell's, bound from their own staging releases, because ledger is a component of the inkwell project and project-flow forks only components in the same repository. A fork from another repository would have no Workload and no push to build it. Listed flat, that reads as a pull request that changed eight things it never touched.

The count needed its cap. /host has review_envs_open and /state has config.max_review_envs; "0 open" says opposite things at a cap of 1 and a cap of 8. This host allows two, not the documented default of one.

Round 10 recorded the PR lane as designed-but-unverified because there were no open review environments to draw against. There is no longer an excuse: opening one is four API calls and it tears itself down on merge.

Verified on 2026-09-08 against a real pull request — apps/brochure #4, opened, watched, and closed. The page did its job. What it found was four things in the platform underneath it, none of them in the console:

what where
two repositories declaring one Project, last push wins — five components stuck DeploymentPipelineNotFound for fifteen hours while the Project reported Ready apps/bakery/openchoreo/project.yaml; FINDINGS.md
the capacity line said "4.6 GB free" while the node was at 99% of CPU requests and nothing in the review environment could schedule project-flow /host, pull_requests.html
the filtered delivery link filtered the timeline and not the current-state table, and the Order/Show controls dropped the filter on the first click delivery.py, delivery.html
a binding created 52 seconds after its own teardown, orphaned for four hours, from a build in flight when the pull request closed teardown_pr(); FINDINGS.md

The parts that were right were right: the forked/inherited split reads exactly as designed ("Built from this branch 1" against "Around it 4"), the pre-claimed Undeploy binding kept the fork out of staging and rendered as hibernated, the build was granted its semaphore at priority 0 and finished in 63s, and teardown on close removed the environment, the fork, the cell and every binding.

Done — the rest of Round 11

Build QoS: the ceiling stays at one, and priority does the work. Measured before deciding, because per-project lanes do nothing while the global semaphore is 1:

dagger engine working set, over 6h   2.53 -> 6.43 GiB
node memory available, same 6h       6.13 -> 2.28 GiB
build POD peak, over 7d              0.29 GiB

The ranges are anti-correlated — the engine's peak is the host's trough. One build already takes this host from 6.1 GiB free to 2.28, and the pod is not where the memory goes. A second would not buy throughput either: max-parallelism = 4 is the real limit and one Rails build fills it.

So lanes are inert here and deliberately not implemented, and priority is the mechanism — the one that works at a ceiling of one. A main build renders at 100 and a pull-request fork at 0, decided by CEL on the component name. runTemplate turns out to be CEL rather than string interpolation, and when the expression is the whole value the engine returns its native type, so priority stays an integer. It must be quoted in YAML: the ternary contains ": ".

Verified with a discriminating test — the PR build was created first, so creation-time ordering would have granted it the lock:

console-run-2c91dada       p=100  Succeeded
console-run-abc37722       p=100  Running     <- granted
ledger-pr-2-run-c50f81ea   p=0    Pending

Both console call sites carried a comment asserting "every build here has the default priority", which had become false, and both sorted the queue by creation time and called the result the position.

Trace chains are searchable. /traces — nine named interactions, each one run against this Tempo before it was written down and each recording what came back, so a chip that stops matching reads as a change rather than as a quiet afternoon. Plus a TraceQL box for the question the list does not cover. Every operation row on /transactions now links to the chains behind it.

A hyphen in an environment name was 400ing every estate-wide query. namespace_pattern used Python's re.escape, which escapes - as \-; Prometheus matches with RE2, which rejects that as an invalid escape. development, production and staging have no hyphens, so this was invisible until the first review-pr-<n> existed — and then outlasted it, because Prometheus keeps the label for the window's length. The test covering that function asserted the bug: it required my\-app in the output.

A runtime span gets a flame graph, and the note that said it could not was half right. It has no build run; it does have a span name, and Pyroscope's labels here are service_name and span_name. The same note claimed span_id was "reachable" as a per-sample label — it is not among this Pyroscope's label names, so the page now says what is actually drawn: the service's profile over the span's window, not the span's own samples.

The manifest diff came to the release list, lazily, and three of its answers stopped lying — see the entry above.

Object storage: no, and STORAGE.md is the evaluation. The largest blob stored is 5.6 kB against a 1 MB cap and the whole database is 8.6 MB; what is actually scarce is the host disk at 92%, which object storage on the same disk would not help. The real finding is an asymmetry in retention rather than size.