Credentials on this page are redacted. This is the public export; the real values live in docs/ACCESS.md in the private repository.

Findings

Eighty-odd of them, each with the evidence — FINDINGS.md

OpenChoreo v1.2.3 — findings from building a real platform on it

Everything below was hit while standing up the demo in this repo, not read off a page. Each item says what happened, why, and what the workaround was.

Verdict in one paragraph

The abstractions are genuinely good. ClusterComponentType / ClusterTrait / ClusterResourceType / ClusterWorkflow are a real platform-engineering substrate: a platform team can define golden paths as versioned, validated, templated objects, and application teams consume them in a dozen lines of YAML. The templating engine (CEL + JSON Patch + per-environment configs) is more capable than most in-house equivalents. What is rough is the edges: the API is v1alpha1 and behaves like it, the docs describe fields the CRDs reject, and several constraints only surface at deploy time. Nothing here was a dead end — but budget time for the discovery.

Documentation is behind the code

The public docs and every blog post describe a pre-1.0 model that no longer exists: Organization, Service, WebApplication, Endpoint, Build, BuildPlane, ServiceClass, and a choreoctl CLI. In v1.2.3 all of those are gone — Build→Workflow, BuildPlane→WorkflowPlane, the fixed kinds became ClusterComponentType instances, and the CLI is occ. Treat anything written before ~April 2026 as architecturally stale.

Even inside the repo, the multi-cluster README lags the maintained install script. It pins observability-logs-opensearch 0.5.1, which emits TLSRoute at gateway.networking.k8s.io/v1alpha2 — a version that Gateway API v1.5.1 (which that same README installs) no longer serves. The install fails outright. k3d-install.sh pins 0.5.3, which emits v1. Use the script's versions, not the README's.

Things the API rejects that the docs suggest

What the docs imply What the API does
spec.patches[].where on a Trait strict decoding error: unknown field "spec.patches[0].where" — where exists, but nested under target, and filters on resource properties
ClusterResourceType validations unknown field "spec.validations" — ComponentType and Trait have them; ResourceType does not
applied.<id>.spec in readyWhen undefined field 'spec' — only .status is in scope

That middle one is the most annoying in practice. Resource types are exactly where a platform team wants cross-field rules ("a multi-instance production database must have backups"), and it is the one place they cannot be expressed. Single-field constraints can go in the OpenAPI schema; anything relational has to move to admission control.

The template context is narrower than it looks

ComponentType validations can see metadata (platform-generated labels), workload, environment, gateway and trait. There is no component.* scope, so a Component's own labels and annotations are invisible. A rule like:

"app.openchoreo.dev/owner" in metadata.labels

evaluates false for every component and blocks all rendering — it is reading the openchoreo.dev/* labels OpenChoreo generates, not the ones the author wrote.

Ownership is a property of the submitted manifest, so it belongs in admission: see platform/openchoreo-config/policies/require-owner-label.yaml, a ValidatingAdmissionPolicy that rejects at kubectl apply with a clear message instead of failing later during render.

Two more CEL specifics worth knowing:

The workload descriptor is not the Workload CR

occ workload create reads a workload.yaml from the repo, and its schema differs from the CRD in ways that fail confusingly:

That last one is arguably correct — "a worker runs bin/jobs" is a property of the role — so in this repo the command lives in the ComponentType and is exposed as a per-environment config. But it is not documented, and the generated Workload is always named <component>-workload, so a hand-written Workload beside it silently becomes a second, conflicting object.

Also: one workload.yaml per appPath, while appPath is simultaneously the build source and what auto-build path-matches on. Components that share an image must share an appPath but still need distinct descriptors. This repo adds a workloadPath parameter to the workflow to separate the two.

Extending the resource catalog needs RBAC you have to know about

A ClusterResourceType that emits third-party CRDs fails because the data-plane agent has no rights over them:

clusters.postgresql.cnpg.io "..." is forbidden: User
"system:serviceaccount:openchoreo-data-plane:cluster-agent-dataplane"
cannot patch resource "clusters" in API group "postgresql.cnpg.io"

The misleading part: Synced=True and OutputsResolved=True throughout. Only ResourcesReady=False tells you. Grant is in platform/openchoreo-config/resource-types/cnpg-agent-rbac.yaml.

Deploying a third-party Helm chart

There is no first-party Helm support at all — no helm workloadType (the enum is deployment|statefulset|cronjob|job|proxy, and immutable), no HelmRelease anywhere in the CRDs, and the gitops-flux-cd community module is control-plane CR sync that explicitly excludes helm-controller.

What does work: a ClusterResourceType whose resources[].template accepts any object with no GVK allowlist, emitting a Flux HelmRepository + HelmRelease. Working example in resource-types/objectstore-garage.yaml (Garage; the MinIO variant was reverted).

It must be a ResourceType, not a ComponentType. ComponentType has no readyWhen, and OpenChoreo's health checker only understands Deployment, StatefulSet, Pod and CronJob — every other kind falls through to a branch that returns Healthy unconditionally. A HelmRelease under a ComponentType would be reported ready the instant the object was created, before Helm had even pulled the chart, and dependents would start against a bucket that did not exist. readyWhen on the ResourceType is the only mechanism that waits for the real install.

Two things to get right:

A Resource's type is immutable, and changing it destroys the data

Resource.spec.type carries x-kubernetes-validations: self == oldSelf, so swapping a resource from one ClusterResourceType to another is rejected:

The Resource "inkwell-db" is invalid: spec.type:
Invalid value: spec.type cannot be changed after creation

The only migration path is delete-and-recreate, and with the default retainPolicy: Delete that takes the database with it. Worth knowing before you name a resource type, because renaming one later is a data-loss event for every consumer.

The generated resource name (r-<resource>-<env>-<hash>) is derived from the resource identity, not the type — so the replacement lands on the same name and blocks behind the old object's teardown.

CloudNativePG's 30-minute shutdown will block you

CNPG defaults stopDelay to 1800s and copies it onto the pod's terminationGracePeriodSeconds. Delete a cluster and the pod sits Terminating for thirty minutes, holding its PVC — and a replacement cluster with the same name waits behind it. Correct for a production database draining connections; unusable for anything you recreate. The postgres resource type here exposes stopDelaySeconds per environment, defaulting to 60.

Status convergence is a 5-minute poll

OpenChoreo polls applied resources rather than watching them — requeueAfter: ~5m in the renderedrelease controller. A database that is healthy and serving reads as ResourcesProgressing for up to five minutes, and the status snapshot the readyWhen CEL evaluates against is whatever was captured at the last poll. Two consequences:

The latency compounds in a way worth planning around. Changing a database's spec (even something as small as stopDelay) makes CloudNativePG restart the primary. If the 5-minute poll happens to sample during that restart it records Ready=False, and every component that depends on the resource then blocks on that stale snapshot — their dependencies.resources stay unresolved, so they re-render with the database env vars missing entirely:

Ready=False ResourceDependenciesPending: 1 resource dependencies pending, 0 resolved

New pods come up without DATABASE_URL and their init containers fail against a local unix socket. The previous pods keep serving throughout, so there is no outage — but a trivial resource-spec change can take ten minutes to converge, and looks broken for most of it.

The agent connection status field is cosmetic

ClusterDataPlane.status.agentConnection reported {"connected":false,"connectedAgents":0,"message":"No agents connected"} for the entire session, while the gateway logs showed the agent happily proxying requests and every deployment worked. Don't gate anything on it.

Outside OpenChoreo, but hit on the way

fs.inotify.max_user_instances defaults to 128 on Ubuntu. Four k3s nodes need ~1024 and die with a misleading "too many open files".

The tracing collector silently discards most of your spans. OpenChoreo's observability-tracing-opensearch chart enables a tail_sampling processor whose only policy is a rate limiter set to 10 spans per second, with a 100-trace decision buffer. That budget is smaller than one idle Rails pod: Solid Queue's polling alone emits 15-25 spans/sec, so background noise consumes the entire allowance and request-carrying traces are dropped.

The failure mode is what makes this expensive. The limiter drops spans, not traces, and every service exports to the collector independently — so a two-service trace routinely arrives with the caller's spans kept and the callee's discarded. In the portal you get a plausible-looking trace that simply stops at the service boundary, which is indistinguishable from a context propagation bug. In a multi-cluster install the processor runs on both the data-plane exporter and the observability-plane receiver, so the limit applies twice.

It cost me hours, and the wrong diagnosis was seductive: Solid Queue forks its workers, BatchSpanProcessor exports from a background thread, and threads do not survive fork(2) — a real, well-documented trap that fits the symptom exactly. I wrote the Process._fork fix. It was wrong: opentelemetry-sdk already restarts its exporter thread on pid change, and the only effect of the fix was exporting every span twice.

What finally settled it was refusing to reason and instead capturing the wire: pointing one worker at a local echo server proved Ruby was sending traceparent: 00-<trace>-<span>-01, and replaying that same header by hand produced the FastAPI span correctly. Application-side propagation was never broken. The tell I had been misreading all along was intermittency — the same call traced sometimes and not others, which no code path explains but a rate limiter does.

Two lessons worth more than the fix. Query your telemetry store carefully: in OpenSearch name is a keyword field, so my match_phrase filters returned zero hits for spans that were sitting right there, and I twice concluded "not exported" when I meant "not matched". Scanning client-side and counting is slower and honest. And a sampler that drops data is a load-bearing part of an observability platform — this default is defensible as a safety valve on a shared cluster, but it is silent, it is applied per hop, and it undermines the one thing distributed tracing is for. It was overridden in an obs-traces-sampling.yaml values file for as long as that chart was installed; the chart went with the OpenSearch stack (see "Adding a second observability stack"), and the file went with it. Alloy, which replaced it, has no sampler -- by design, and if one is ever added it belongs in lgtm-alloy.yaml with a comment pointing here.

.localhost makes the shipped install unreachable from anywhere else, and the auth chain is pinned to it. Every URL in OpenChoreo's k3d values — Backstage's baseUrl, Thunder's CORS list, the OAuth application's redirect URIs, the sign-in gate's hostname — is *.openchoreo.localhost. That is not a DNS choice you can route around: browsers and resolvers hardcode .localhost to the loopback address, so no split-DNS, VPN or proxy makes it resolve on another device. Opening the portal over a tailnet gets you as far as the login button and then:

Origin 'http://portal.TAILNET.ts.net:8080' is not allowed

Fixing it means four coordinated changes, none of which is discoverable from the error: the origin on Thunder's CORS list, the redirect URI on the OAuth application, backstage.baseUrl (single-valued, so one hostname has to win), and gateClient.hostname — that last one because the authorize endpoint happily accepts the new redirect URI and then sends the browser to a sign-in page on the old hostname, which looks like the fix did not work.

The trap underneath it: Thunder registers its OAuth applications from a helm.sh/hook: pre-install ConfigMap and Job. Helm renders pre-install hooks on install and never again, so helm upgrade with corrected values reports success and changes nothing. The registration keeps its original redirect URIs indefinitely. platform/bootstrap/rerun-thunder-bootstrap.sh re-renders those two objects with the hook annotations stripped; the scripts are idempotent and PUT the application when it already exists.

Worth stating plainly for anyone evaluating: a .localhost default is fine for a laptop demo and actively hostile to the "show your team" use case, because the failure is an authorization error rather than a name-resolution one, and it points at the wrong layer.

Instrumentation options are validated silently. The OpenTelemetry Ruby instrumentation gems validate each option against an allowed set and, on a mismatch, discard the value and keep the default without raising or warning. propagation_style: :parent is not a valid ActiveJob value — the set is %i[link child none] — so background jobs kept the :link default and every job stayed in its own trace, exactly as if the option had never been written.

The boot log is the only feedback, and it reports the option the instrumentation ended up with rather than the one you passed, so it reads as confirmation:

Instrumentation: OpenTelemetry::Instrumentation::ActiveJob was successfully
installed with the following options {propagation_style: :link, ...}

Read that line as an assertion about your config, not a receipt.

Dagger injects its own OTEL_EXPORTER_OTLP_* into every build container so build steps feed its trace UI. An application whose tracing initializer keys off endpoint-presence will silently switch itself on during CI and export app spans into the build's trace stream. Gate on an explicit variable.

Dagger cannot reference a module on a self-hosted git host with a non-standard port. It resolves refs Go-vanity style, strips the :3001, and fails with unrecognized import path. git://, http://…#ref:subpath and host:port/org/repo@ref all fail. Fetching the module (clone or tarball) and calling dagger -m <path> works and is more auditable anyway.

CREATE EXTENSION timescaledb takes the backend down if it is not in shared_preload_libraries — it does not error cleanly, the connection dies (server closed the connection unexpectedly). Platform-owned knowledge; the resource type sets it automatically whenever timescaledb is requested.

timescaledb-oss is the Apache build and has no continuous aggregates. The CloudNativePG community extension catalog ships that build, so hypertables work but CREATE MATERIALIZED VIEW ... WITH (timescaledb.continuous) fails with "functionality not supported under the current apache license". A local timescale/timescaledb-ha image is the TSL build and does not reproduce it — so this only appears once deployed. Capability detection here is three-state as a result.

TimescaleDB ≥ 2.13 defaults continuous aggregates to materialized_only = true, so fresh rows are invisible until the refresh policy runs and a rollup reads as empty. Set materialized_only = false for real-time aggregation.

Rails 8 specifics that cost time: allow_browser versions: :modern returns a bare 422 to curl and anything else it does not recognise; db/schema.rb cannot represent an environment-dependent schema (it dumps whichever database it last saw, then db:prepare fails elsewhere); ENV.fetch("DATABASE_URL") raises during assets:precompile; and ENV["SOLID_QUEUE_IN_PUMA"] is truthy when set to the string "false", which starts the queue supervisor inside the web pod.

The shipped component types default to 256Mi, which OOMKills a Rails 8 boot (exit 137, and the only clue is the restart count). A platform offering a Ruby component type should ship a default that runs Ruby.

Setting a command replaces the image ENTRYPOINT, which for Rails means bin/docker-entrypoint — and its db:prepare — never runs. The symptom is an app that passes health checks, serves 200s, and has no tables. Worse, that entrypoint only migrates when the last two argv entries are exactly ./bin/rails server, so appending -p 8080 defeats it even if you do route through it. Schema preparation belongs in an explicit initContainer the component type owns, not in argv sniffing.

Adding a second observability stack

A new trait is rejected until the ComponentType allows it, and the rejection is easy to miss. ClusterComponentType.spec.allowedTraits is an allowlist, so attaching continuous-profiling to a component failed with:

Ready=False InvalidConfiguration: ComponentType "python-service":
traits [ClusterTrait:continuous-profiling] are not in the allowed list [...]

This is the guardrail working, and it is the right design — a platform team decides which capabilities exist before a team can switch one on. What makes it cost time is where the failure surfaces: the build succeeds, the image is published, the Workload is updated, and then nothing deploys. Neither the WorkflowRun nor the ReleaseBinding says why; only Component.status does. Check the Component when a successful build does not produce a new pod.

Beyla's namespace selector is a glob, not a regular expression. k8s_namespace: "dp-.*" matches nothing; dp-* matches everything. The field reads like a regex, and the only evidence is a DEBUG-level line that reports your pattern failing against a value it visibly matches:

msg="metadata does not match" attr=k8s_namespace value=dp-default-inkwell-development-3b5625d7

At the default log level the symptom is simply an eBPF agent that starts cleanly, reports healthy, emits network flows, and never produces a single application span.

Loki omits data entirely when a query matches nothing. An empty label query returns {"status":"success"} with no data key, which reads as a broken endpoint rather than an empty result. I concluded twice that a write path was failing when the reads were simply early.

Grafana's OpenSearch datasource wants a date-templated index pattern, not a glob. container-logs-* fails its health check with "Index not found" against indices plainly named container-logs-2026-08-29; the working form is [container-logs-]YYYY-MM-DD plus interval: Daily.

Tempo's monolithic chart serves its entire query API on the port named tempo-prom-metrics (3200). The name reads like a scrape endpoint, so the obvious guess — 3100, Loki's port — produces a datasource that connects to nothing.

A single-replica PVC-backed Deployment cannot roll. Grafana's default strategy is RollingUpdate; with a ReadWriteOnce volume the new pod cannot mount until the old one exits, and the chart's init-chown-data then fails on directories it does not own. Recreate plus initChownData.enabled: false is the fix, and applies to any chart in this shape.

Search your telemetry store carefully. In OpenSearch, span name is a keyword field, so match and match_phrase return nothing for spans that are sitting right there. Twice I read "no results" as "not exported" and went looking for a bug in the application. Scanning client-side and counting is slower and does not lie.

Vendoring third-party software

A Resource cannot consume another Resource's outputs. Only a component can, via dependencies.resources[].envBindings. There is no equivalent for a resource, so anything installed as a ClusterResourceType — which is where Helm charts have to go, because ComponentType has no readyWhen — cannot use the catalog's own Postgres. The credential lives in a Secret whose name contains a generated hash that the chart's values cannot know.

The result is a split with no user-facing logic to it:

you have door can use the catalog's Postgres?
a prebuilt image vendored-service ComponentType yes
a Helm chart helmchart ResourceType no

Outline gets CloudNativePG; Zulip has to bring its own Postgres, purely because one arrived as an image and the other as a chart.

Four steps in bringing up a project have no CLI affordance and fail against the wrong object. In order:

  1. A Project reconciles to a ProjectRelease, but nothing binds that release to an environment. Without a ProjectReleaseBinding the data-plane namespace is never created, and every component in the project fails with namespaces "dp-default-wiki-development-..." not found reported against its RenderedRelease. It looks like a component problem.
  2. The same is true per resource. occ resource promote moves an existing binding to the latest release and cannot create the first one — it says no ResourceReleaseBinding found, which reads like a bug rather than a missing prerequisite.
  3. occ secret create fails with Secret API is disabled on this server until features.secretManagement.enabled is set in the control-plane chart. Nothing suggests the toggle is server-side.
  4. Authenticated is not authorized. Every shipped ClusterAuthzRoleBinding matches the groups claim, which only user tokens carry; a client_credentials token has sub, aud and client_id. So occ logged in as a service client gets You do not have permission to perform this operation until you add a binding that matches sub.

An output that renders to an empty string breaks the whole binding. The postgres resource type exposes capabilities as parameters.extensions.join(','). Ask for no extensions and it renders "", which fails the ResourceReleaseBinding status schema:

status.outputs[7]: Invalid value: exactly one of value, secretKeyRef,
or configMapKeyRef must be set

An empty string is indistinguishable from unset. The binding then never reconciles, carries no status at all, and every consumer waits on "resource dependencies pending" with nothing to explain it. It only bites the first consumer that takes the default — Inkwell had passed for days.

A parameters schema needs an explicit type: object at its root, or the CEL type checker cannot resolve anything beneath it: type check error: undefined field 'probes' against a schema that visibly declares probes.

Charts that generate their own subchart passwords may never wire them through. The Zulip chart renders SECRETS_postgres_password, SECRETS_rabbitmq_password and SECRETS_memcached_password as env vars with neither a value nor a valueFrom — literally null — when the Bitnami subcharts generate the passwords themselves. A parent chart cannot read a subchart's generated Secret at render time. The failures surface one layer away from the cause:

django.db.utils.OperationalError: fe_sendauth: no password supplied
bmemcached.exceptions.MemcachedException: Auth failure.

Supplying the passwords yourself is the fix, and Flux's valuesFrom with a targetPath is what keeps them out of git: it writes a value from a Secret into the values tree at install time.

A crash-looping StatefulSet pod blocks its own rolling update. Fixing the Zulip config produced a correct new pod template that never took effect, because the StatefulSet controller will not replace a pod that has never been Ready. kubectl get sts -o jsonpath='{.status.updateRevision}' differing from currentRevision is the tell; deleting the pod is the fix.

Flux's install remediation destroys the evidence. When an install fails, remediation uninstalls the release — including the pods whose logs say why. The HelmRelease reports only failed early due to stalled resources: [StatefulSet/.../zulip status: 'Failed']. Read the logs while the install is still running, or lose them.

Putting Cilium under the data plane

The CNI is chosen at cluster creation and cannot be changed. There is no in-place migration from flannel; k3s has to start with --flannel-backend=none --disable-network-policy and Cilium must be installed before any workload pod can get an address. Adding Cilium therefore meant destroying and rebuilding the data plane.

Do not bind-mount the host's /sys/fs/cgroup into a k3d node. Every Cilium-on-Docker guide shows it, and on k3d it breaks the node's cgroup namespace so thoroughly that no pod sandbox can start at all:

runc create failed: unable to apply cgroup configuration: failed to write 1027:
/sys/fs/cgroup/kubepods/burstable/pod<uid>/<id>/cgroup.procs: no such file or directory

The message names runc and a cgroup path, so it reads as a runtime or kernel problem. It is neither: kubelet is computing paths inside the node's own cgroup namespace that do not exist in the host hierarchy you mounted over it. Mount /sys/fs/bpf only, and let Cilium's mount-cgroup init container do the rest (cgroup.autoMount.enabled: true).

The rebuild was an accidental test of whether the platform is declarative, and it passed. With an empty, re-registered cluster the control plane re-created all three projects, their namespaces, workloads, CloudNativePG clusters, object stores, Redis instances and gateways with no manual step. Two things did not come back:

External Secrets caches a failed lookup. After the value was restored in OpenBao, the ExternalSecret stayed SecretSyncedError — refreshInterval: "0" means sync once, and once includes once-and-failed. Annotating the object to force a sync fixed it instantly.

Rebuilding a cluster orphans its Tailscale devices and silently renames the new ones. The proxies from the destroyed cluster keep their hostnames on the tailnet, so the replacements register as outline-1, zulip-1, development-default-1 — and every documented URL quietly points at a node that is offline. Worse, deleting the stale devices is not enough: each proxy persists its node identity in a per-proxy state Secret in the tailscale namespace, so it re-registers under the same -1 name until that Secret is deleted too.

Helm merges values by key, so a duplicated top-level key silently wins. Adding a second operator: block to the Cilium values dropped the replicas: 1 from the first one — no warning, and the symptom would have been a permanently Pending second operator pod on a single-node cluster.

A freshly created k3d node has no /etc/machine-id. OpenChoreo's fluent-bit DaemonSet mounts it as a hostPath of type File, so on a new cluster the pod sits in Init:0/1 indefinitely:

MountVolume.SetUp failed for volume "etcmachineid":
hostPath type check failed: /etc/machine-id is not a file

No logs ship from that plane, and nothing else complains. Long-lived nodes have the file only because something happened to create it, which is why this never shows up until you rebuild.

Promotion and alerting

Promotion moves a pointer, and that is the whole point. occ component deploy X --to staging copies the immutable ComponentRelease name into the target environment's ReleaseBinding. All three environments here run byte-identical releases — inkwell-web-7f494f68b4 in development, staging and production — while differing in size (1, 2 and 3 database instances) through resourceTypeEnvironmentConfigs on the binding. One rendering of what this is, several statements of how big it is here.

Three bindings are needed per environment and only one of them has a CLI. occ component deploy --to creates the component's binding. The project's ProjectReleaseBinding and each resource's ResourceReleaseBinding do not exist until you write them, and without the project binding no namespace is created — the component then reports ConnectionsPending and the resources sit at Ready=False, neither of which mentions the project. occ resource promote only moves an existing binding to a newer release; it cannot create the first one.

An alert rule with no notification channel is rejected, and nothing tells you. ObservabilityAlertRule requires at least one channel. A trait whose channels default is [] renders fine, the component reports Ready, and the rule is silently never created — the rejection appears only in the controller-manager log against the RenderedRelease:

ObservabilityAlertRule ... is invalid: spec.actions.notifications.channels:
Invalid value: 0: ... should have at least 1 items

The shipped log-alert query assumes structured logs that fluent-bit does not produce. slo-alerts queries level:ERROR OR level:FATAL, but the container-logs-* index has no level field at all — fluent-bit ships the whole line as an unparsed log string. An application that logs plain text can never trigger a log-based alert, however loudly it fails. Either the application emits JSON logs with a level key, or the query has to match the raw text.

log is a keyword field, so match and match_phrase need the exact whole line. Searching for PG::ConnectionBad returned zero while fourteen such lines sat in the index; wildcard found them immediately. This is the second time the same trap cost real time — span name behaves identically. When a search of your telemetry returns nothing, check the field type before concluding the data is missing.

Injecting a fault with Cilium

A CiliumNetworkPolicy is the least invasive way to cause a real incident: nothing is deleted or reconfigured and no pod restarts, the packets simply stop arriving. Deleting the object ends it. deny-db-egress.yaml cuts inkwell-web off from Postgres; requests turn 503, Rails logs PG::ConnectionBad, and Hubble shows the drops — telemetry the application cannot produce about itself, because from inside the pod the request just hangs.

Attaching any egress rule switches the endpoint to default-deny egress. A policy containing only egressDeny for Postgres also denies DNS, and Hubble fills with drops to CoreDNS while the intended target is a footnote. The fix is an explicit allow-everything egress rule alongside the deny — egressDeny takes precedence — so that the manifest's blast radius matches its intent.

Failover

Environment.spec.dataPlaneRef is immutable (self == oldSelf), and there is no failover, placement or affinity concept anywhere in OpenChoreo. An environment cannot be moved to another cluster, so cross-cluster failover has to be a second environment on the DR data plane, deployed to from the start and held at state: Undeploy. Flipping that field is the entire mechanism — which is genuinely good (the DR binding already points at the same immutable release, so no build and no chance of drift) and genuinely limited (nothing detects, decides or acts on its own).

The planeID is the routing key, not the CR name. Install the data-plane chart with its default and the DR agent connects as planeID: default, competing with the real primary for the same identity while the ClusterDataPlane is called dr. --set clusterAgent.planeID=dr is what actually separates them.

A new data plane needs five prerequisites the chart does not install, each failing in a way that names the wrong thing:

missing symptom
Gateway API + kgateway CRDs no matches for kind "Gateway" — reads as a chart bug
CoreDNS rewrite agent loops on lookup cluster-gateway.openchoreo.localhost
CNPG image catalogs Cluster has incomplete or invalid image catalog — does not say which
/etc/machine-id fluent-bit stuck in Init:0/1, no logs ship
agent RBAC for emitted CRDs resources apply silently to nothing

no agents found for plane dataplane/dr is what a resource looks like when the target cluster's agent has not connected yet. It is reported against the ResourceReleaseBinding, so it reads as a problem with the resource.

The image registry lives inside the workflow plane. It is published on host port 10082, so stopping the workflow plane to free memory takes the registry with it and every data plane starts failing ImagePullBackOff — on clusters that have nothing to do with builds. Nothing in the error points at the workflow plane.

Measured, on this hardware:
CNPG replica promotion ~36s, no user-visible error, write-after-failover accepted
Application failover (either direction) ~60s from field change to 200

The application failover is honest about what it is: the two databases do not replicate to each other, so failing over gives a running application on an empty database. The mechanism is sound; the data story is the hard part and is not built here.

Provisioning through a Job, and swapping an implementation

A resource type can call an API during provisioning, because a Job is just a Kubernetes object. zulip-bot creates the bot account in Zulip; objectstore applies Garage's cluster layout and mints an access key. readyWhen then gates every consumer on the Job succeeding, which is what makes it safe: a bot cannot start before its identity exists, and an application cannot start against a Garage node that has no layout and rejects every request while answering health checks.

But a Job's spec.template is immutable. Change the provisioning script and the controller re-applies the new rendering forever while the API server refuses it:

Job.batch "...-provision" is invalid: spec.template: ... field is immutable

The resource sits at ResourceApplyFailed indefinitely, with a Complete Job beside it that did its work correctly. The fix is to put a version in the Job's NAME — both types take a provisionerVersion parameter — so changing the script creates a new Job rather than trying to mutate one.

Swapping a resource type's implementation is invisible to consumers, but the old workload is not pruned. Replacing MinIO with Garage kept the output contract identical — endpoint, bucket, region, accessKey, secretKey — so inkwell-enrich and Outline consumed the new store with no repository change, no rebuild, and no notification. What did NOT happen automatically is removal of the old MinIO Deployment: the new rendering emits a StatefulSet under a different resource id, so the Deployment stayed, kept matching the Service selector, and the provisioning Job got Connection refused from a pod that happened to answer on the wrong ports.

A provisioner must retry every call, not just the first. Applying Garage's layout makes it reload, so calls after it are refused for a few seconds. A Job that waits for the service once and then proceeds fails halfway through, having already created the bucket, which reads as a permissions problem rather than a restart.

CEL will not put a ${...} inside a YAML flow mapping. limits: {memory: ${environmentConfigs.memory}} fails to parse as YAML before any CEL runs; the value has to be in block style. This is the third distinct place YAML and CEL have collided in this project, after the ": " in a ternary and quoting an integer parameter.

Naming a Dagger module node breaks the build inside generated code. It collides with the SDK's own Binding.AsNode, and the failure is method Binding.AsNode already declared pointing at dagger.gen.go — nothing mentions the module name. Renaming to frontend fixed it, and renaming taught the second lesson: git mv moves the directory but go.mod keeps the old module dagger/node path, after which Dagger types every parameter as Void and no error says why.

gRPC and the proxy workload type

gRPC is a first-class endpoint type and needs no special component type machinery — Workload.spec.endpoints[].type accepts gRPC alongside HTTP, GraphQL, Websocket, TCP and UDP. What a grpc-service component type contributes is the probing: Kubernetes has spoken the standard gRPC health-checking protocol natively since 1.24, so

readinessProbe:
  grpc: {port: 9090, service: ledger.v1.Ledger}

replaces the grpc_health_probe binary that most charts still ship in the image purely to be checkable.

Trace context crosses HTTP → gRPC only if the gRPC instrumentation is installed on both ends. gRPC carries context in request metadata, not headers. Without it the callee's spans start a fresh trace and the hop disappears from the waterfall, with nothing erroring — the same failure as an unpropagated traceparent, in a transport where it is easier to miss. Working:

inkwell-enrich  POST /embed
  inkwell-enrich  /ledger.v1.Ledger/Record     (client)
    ledger        /ledger.v1.Ledger/Record     (server, same trace)

workloadType: proxy is real, undocumented, and unsampled. It is the only value exempt from the CRD's rule that a component type must declare a primary resource matching its workload type:

self.workloadType == 'proxy' || self.resources.exists(r, r.id == self.workloadType)

so a proxy component runs no pod. It is how a platform puts something it does not operate behind the same front door as everything it does.

But a proxy component still requires a Workload, and the Workload still requires a container. Both, in order:

Ready=False WorkloadNotFound: Workload for component "weather-api" not found
The Workload "weather-api-workload" is invalid: * spec.container: Required value

A component whose entire purpose is having no container must therefore name one. platform/openchoreo-config/proxy-components/ uses pause:3.9 — never pulled, no pod ever created — because a real inert image is more honest than a fabricated string if anything ever does try to run it.

What a proxy component is worth, and what it is not. Worth: a third-party dependency gets a stable internal name, so the services calling it do not each hardcode a vendor hostname and replacing that vendor is a platform change rather than N pull requests. Not worth: pretending the dependency is yours. There is no pod, so there are no logs, no metrics, no profile — the only telemetry is what the gateway records about the hop, and availability is still the vendor's.

A conditional migration that records itself as done

Inkwell reported capabilities.vector: true next to search_mode: "keyword" and its worker never embedded anything. Both halves of that were true, which is what made it hard to see.

Note.semantic_search? requires two things: the vector extension and the notes.embedding column. The extension was there; the column was not, and pg_indexes for notes was empty.

The migration that adds it opens with

return say("vector extension absent; skipping embedding column") unless extension?("vector")

which is reasonable on its own — the column genuinely cannot exist without the extension. But a skipped migration is still written to schema_migrations, so it never runs again. The extension arrives seconds later, the health endpoint starts reporting vector: true, and the column is gone for the life of the database.

The race is between the component's db-prepare initContainer and CloudNativePG's Database reconciler. Our postgres ResourceType gated the Cluster on Ready but had no readyWhen on the Database resource at all, so outputs resolved — and the app started — before CREATE EXTENSION had run. The CNPG Database CR carries exactly the status needed:

readyWhen: >-
  ${has(applied.database.status) && applied.database.status.applied == true
    && applied.database.status.extensions.all(e, e.applied == true)}

Three things worth carrying to any similar case:

The general shape — a health endpoint that truthfully reports a capability the application has independently decided not to use — is worth watching for. The probe was right, the feature flag was right, and the conjunction was the bug.

Metrics shipped all the way to a backend that drops them

Prometheus had container CPU and memory and essentially nothing else: no http_server_*, no Beyla metrics, no node_filesystem_*. Beyla was running cluster-wide with otel_metrics_export configured and features: [application, application_span, network]. The metrics were being collected and exported.

Alloy's batch processor routed them here:

metrics = [otelcol.exporter.otlphttp.tempo.input]

Tempo is a trace store. It answers /v1/metrics with 404, and the only evidence was an Alloy log line at level=error that scrolled past among healthy ones:

Exporting failed. Dropping data. component_id=otelcol.exporter.otlphttp.tempo
error="... request to http://.../lgtm/tempo/v1/metrics responded with HTTP Status Code 404"

Two further things were needed once the destination was right:

No new Prometheus was needed. OpenChoreo's own already has enableRemoteWriteReceiver: true, and the data plane was already remote-writing to it, so Grafana and the portal's metrics adapter read the same series.

Nothing prunes build pods, at any layer

144 finished pods had accumulated in workflows-default. There was no leak:

The only cleanup is OpenChoreo's WorkflowRun.spec.ttlAfterCompletion, "1d" on our builders, which deletes the whole WorkflowRun CR after a day and takes its pods with it. 144 pods is one day of builds at four to six pods each, held whole.

The fix is Argo's controller default, set through the subchart:

argo-workflows:
  controller:
    workflowDefaults:
      spec:
        podGC:
          strategy: OnWorkflowCompletion

It renders inside the ConfigMap's single config key, not as a top-level key — worth knowing if you patch the ConfigMap directly rather than upgrading.

podGC only, deliberately. ttlStrategy deletes the Workflow object and would race OpenChoreo's own retention loop; podGC reclaims the pods and leaves the workflow queryable for the portal's build history for the full day. Retention stays OpenChoreo's decision, pod reclamation is Argo's.

The portal cannot be made a single pane of glass without a custom image

Two independent blockers, both verified rather than inferred.

The bundled app has no Grafana plugin. Extracting ghcr.io/openchoreo/openchoreo-ui and reading packages/app/dist/.config-schema.json — which lists every config key the bundle understands — gives:

app, auth, aws, backend, catalog, discovery, events, gitlab, home, homepage,
integrations, kubernetes, openchoreo, organization, permission, search, techdocs

kubernetes and techdocs are there. grafana is not, and neither is proxy, so a dashboard can be neither embedded nor proxied. The backend package confirms it: @backstage/plugin-kubernetes-backend is present, nothing Grafana-shaped is. Note also that jenkins and github are absent from the frontend schema even though the chart ships a jenkins: config block.

backstage.appConfig is a genuine, image-free extension point — it mounts an overlay at /app/app-config.extra.yaml and appends --config — but it configures plugins. It cannot add one.

Arbitrary annotations do not survive catalog sync. Setting grafana/dashboard-selector, backstage.io/kubernetes-id and a private key on a Component CR, then waiting a full 300s poll cycle, none of the three reach the entity. The provider emits a fixed set of openchoreo.io/* and backstage.io/* keys and drops the rest. That also rules out the metadata.links fallback, since links come from the same provider.

So the honest arrangement is the other way round: Grafana is the observability pane — it holds metrics, logs, traces and profiles already — and links back to the portal, which keeps the catalog, builds and promotion. The OpenChoreo overview dashboard is the index.

To query the catalog from a script at all, add a static token via backstage.appConfig; every other route in is an interactive OAuth flow, and BACKEND_SECRET signs service tokens rather than being one.

Alerts that are worse than no alert

Two of these were live before this phase and neither was noticed.

ObservabilityAlertRule with type: log has never worked here. Both auto-generated slo-errors rules sat in phase: Error:

observer API POST .../api/v1alpha1/alerts/sources/log/rules returned status 500:
failed to create alert rule: logs adapter returned HTTP 500

Worth knowing before treating the removal of OpenSearch as a loss of log alerting: there was none to lose.

A PersistentVolumeFillingUp rule is a trap on k3s. The obvious expression over kubelet_volume_stats_* looks right and fires 22 identical alerts the moment the disk fills, burying the one that means something. k3s provisions PVs with local-path — unquota'd hostPath directories — so every PVC reports the host filesystem: all 22 of ours claim 1007G capacity. The rule belongs on a cluster whose storage class can enforce a size, and nowhere else.

Also worth recording: count() over an empty selector returns no series, not zero, which a Grafana stat panel renders as "No data" and a reader interprets as broken instrumentation. count(...) or vector(0) is the difference between "there are no build pods" and "we cannot see build pods".

A ResourceQuota is a deploy-time admission rule, not a ceiling

Adding limits.cpu to a project ResourceQuota broke every deploy in the project, with a message that names the real cause but is easy to misread:

Error creating: pods "inkwell-web-..." is forbidden:
failed quota: project-quota: must specify limits.cpu for: db-prepare

A ResourceQuota that constrains a resource rejects any pod that does not declare that resource. It is not a ceiling you bump into when full; it is an admission requirement that applies from the moment it exists. A LimitRange supplying defaults is what makes it safe, and the two have to cover exactly the same resources.

The part that actually caught us: initContainers count. The application containers had explicit CPU and memory limits from the component type, so the quota looked satisfied. db-prepare — the initContainer that runs migrations — had neither, and one missing limit on one initContainer is enough to reject the whole pod. The old ReplicaSet keeps serving, so the symptom is a deployment that never rolls while the app stays up: Available=True, ReplicaFailure=True, and a ReleaseBinding reporting ResourcesDegraded with the reason four levels down in events.

We resolved it by dropping limits.cpu from the quota rather than defaulting a CPU limit in the LimitRange. A default CPU limit means CFS throttling on anything that occasionally bursts — slow migrations, flaky probes — for no benefit, because CPU is compressible and memory is what actually kills a node. Memory, pod count and storage are quota'd; CPU is not.

Worth knowing alongside it: kubelet_volume_stats_* cannot backstop the storage quota on k3s, because local-path PVs are unquota'd hostPath directories that all report the host filesystem. requests.storage in the quota is the only thing bounding a project's disk use.

Building review environments on a platform that has no concept of them

OpenChoreo has neither review environments nor promotion automation, so both are ours. Five things had to be discovered by watching them fail.

A successful build does not produce a ComponentRelease. It produces a Workload. The release is cut by the Component controller and only when autoDeploy is set — internal/controller/component: "When autoDeploy is enabled, this means ComponentRelease and ReleaseBinding are created/updated." A ReleaseBinding with no releaseName is not resolved to "latest" either; it fails with ComponentRelease "" not found. So waiting for a release before creating a binding deadlocks: nothing will ever create one. Review components must set autoDeploy: true and accept that OpenChoreo also binds them into the pipeline's first environment, which the controller then deletes.

ResourceReleaseBinding is the opposite — resourceRelease must be pinned explicitly, and an unset one says so plainly: "spec.resourceRelease is unset; pin a ResourceRelease to deploy this binding". Two sibling CRDs, opposite defaults.

A review environment needs the whole project, not the changed components. Connections resolve within an environment, so inkwell-web bound alone sits at ReleaseBinding not found for component inkwell/inkwell-enrich forever. It also needs its own ProjectReleaseBinding — that is what creates the cell namespace, and without it every component fails with namespaces "dp-default-inkwell-review-pr-N-<hash>" not found. The upside: the review environment inherits the project's ResourceQuota automatically, which is what bounds the whole feature.

Components from other repositories must be bound, not forked. A fork gets a new component name, so it has no Workload, and a Workload only comes from a build triggered by a push matching that component's branch — there is no push to another repo just because a pull request opened elsewhere. Binding the original at its development release is also better semantics: "this change against the rest of the project as it stands".

Teardown deadlocks, and it is the common case. A review binding that never resolved keeps openchoreo.dev/releasebinding-cleanup forever, because the cleanup waits on state that will never arrive; the Environment's own finalizer then waits on that binding. Everything sits in Terminating indefinitely. The controller deletes in dependency order, waits a grace period, and then strips finalizers — but only from objects carrying its own ownership label.

The cost is real and worth stating: a working review environment for a six-component project is a full Postgres, a full object store and six workloads per pull request, around 3GB. That is why the cap is one.

Two mistakes of my own, worth keeping

podGC: OnWorkflowCompletion deletes the pods of FAILED builds. "Completion" includes failure, so the first review build failed and its logs were already gone — kubectl logs returned nothing and all that survived was Argo's main: Error (exit code 1). The failed build is precisely the one whose output you need. OnWorkflowSuccess keeps failures and still reclaims the bulk, since successful builds are almost all of the volume.

A repair migration inherits the flaw it repairs. Phase A fixed the missing notes.embedding column with a second, conditional migration. A fresh review environment then came up with vector: true next to search_mode: "keyword" — the original bug — with both migrations recorded and the column still absent.

Capability-dependent schema cannot be a migration at all. A migration runs once; a conditional one that finds the capability missing is still recorded and never runs again, and a repair migration is just the same latch one version later. Gating the platform side does not save it either: a component waits for its database resource's outputs — the connection string, available as soon as the cluster secret exists — not for CloudNativePG to finish applying extensions.

The fix is an idempotent task hung off db:prepare, running on every boot and converging whenever the extension appears. The readyWhen gate stays because it narrows the window, but it was never sufficient.

A service binding is not a ready service

The Ruby build's test step bound a Postgres service and immediately ran rails db:prepare. Dagger starts the container; it does not wait for the process inside to accept connections. db:prepare then does something worse than fail — it reads a connection error as "the database does not exist yet", takes the create-and-load path, fails that too, and still exits 0. rspec runs against an empty database and the suite dies with relation "notes" does not exist, which reads like a broken migration rather than a race.

It only appears under load. Three builds starting at once — which is exactly what a review environment produces — failed every time, while the same source passed when run alone. pg_isready with a bounded retry makes it deterministic. libpq-dev ships the headers, not the client binaries, so postgresql-client has to be installed for it.

Silent list failures hide missing RBAC

The controller's Kubernetes list helper returned [] on any non-200. A missing RBAC rule for resources then presented as "this project has no resources" rather than as a 403, and the review-environment code quietly did nothing at all. Anything that swallows an API error and returns an empty collection will eventually cost you an afternoon; log the status.

Public endpoints: what OpenChoreo gives you and what it doesn't

OpenChoreo has no DNS concept at all — repo-wide, "external-dns" and "cloudflare" appear only in one comment repeated across three values files, offering gateway.annotations as the sole extension point. TLS is different: the chart has always known how to render an HTTPS listener, but every shipped k3d values file sets gateway.tls.enabled: false, so the live gateway had one listener — http/19080, no hostname. The default is enabled: true with an empty certificateRefs, which renders a listener referencing nothing, and that is almost certainly why upstream disables it everywhere rather than leaving it on.

Turning it on takes three things that must agree: helm gateway.tls.*, an https: block on ClusterDataPlane.spec.gateway.ingress.external, and a Secret holding the certificate. Adding the https: block is also what makes status.endpoints[].externalURLs.https populate and what makes the component types emit a second hostname per route.

DNS records as a resource type, and why not a Job. The obvious implementation — a provisioning Job calling the Cloudflare API, in the style of zulip-bot.yaml — creates a record fine and can never remove one. With review environments appearing and disappearing that means a zone filling with orphans. external-dns with --source=crd and a DNSEndpoint-emitting ClusterResourceType gets creation, drift correction and deletion, through ownership TXT records that stop it touching anything it did not create. Verified end to end: a DNSEndpoint became a live record in public DNS within a minute, and deleting it removed both the record and its TXT.

Two things to get right in that config. policy: sync (not upsert-only) is what permits deletion. And sources: [crd] alone — adding service or gateway-httproute would make external-dns publish a record for anything that grows an annotation, which is not what you want pointed at somebody's real domain.

ClusterResourceType.resources[] has no targetPlane. Unlike ClusterProjectType, whose resources take one, a resource type's entries are id, includeWhen, readyWhen and template only — they always go to the data plane. The API rejects the field outright rather than ignoring it, which is the good outcome.

Wildcard certificates require DNS-01. HTTP-01 would appear to work for a single hostname and then fail on the pattern that actually matters: a per-pull-request hostname has to work the moment its DNS record appears, and an ACME order per review environment would mean every one of them starting life with a browser warning. One *.jung.town wildcard, renewed by cert-manager, covers all of them — and wildcards can only be validated by DNS-01.

Worth knowing: a Let's Encrypt order can validate every challenge and still fail at finalization with 404 urn:ietf:params:acme:error:malformed: Certificate not found. Both DNS-01 challenges showed valid and the order was errored. It is transient; cert-manager then backs off, and the fastest way to retry is to delete and recreate the Certificate rather than wait.

A Cloudflare token that fails /user/tokens/verify is not necessarily invalid. An account-scoped token returns 401 Invalid API Token from that endpoint while working perfectly against /zones and /accounts/{id}/tokens/verify. Verify against something you actually intend to call.

Cloudflare Access cannot be enabled by API. Zero Trust needs a one-time dashboard onboarding that asks for a team name, and every Access API call returns access.api.error.not_enabled until that is done — including the call that would create the organization. It is a genuine human step, not a permissions problem, though the token needs Access permissions on top.

Three ways one delivery pipeline can fail silently

Push → build → promote had three independent defects at once. Each answered 200, logged at INFO, and surfaced nothing as wrong. Together they meant no push had produced a build, and no merge had advanced staging, for over a day — on a platform whose whole subject is delivery.

A webhook that matches nothing still answers 200. OpenChoreo decides which components a push affects by comparing the webhook's repository URL against each Component's workflow.parameters.repository.url. The comparison is exact string equality; normalizeWebhookRepoURL strips a .git suffix, a trailing slash and case, and nothing else — scheme, host and port must all match (internal/openchoreo-api/services/autobuild/webhook_processor.go:382).

Gitea derives the clone_url it sends from ROOT_URL, and ROOT_URL has to be https://git.jung.town/ because Gitea builds its OAuth callback from the same value and a callback on host.k3d.internal is one no browser can follow. Every Component, meanwhile, has to declare http://host.k3d.internal:3001/..., because that is the only address a build pod can clone from — git.jung.town is behind the gateway's OIDC filter. The two can never agree. The comment in infra/compose.yaml reasoned that changing ROOT_URL was safe because "every clone, package push and webhook in this repo carries an ABSOLUTE host.k3d.internal:3001 URL of its own". That is true of the webhook's target and false of the webhook's payload, which is the one that matters here.

What it looks like: Skipping component: repository mismatch at INFO, "affectedComponents": 0, HTTP 200, and a green delivery in Gitea's webhook history. Nothing anywhere is coloured like a failure.

The fix is in platform/project-flow/controller.py, forward_to_autobuild(): receive the push, rewrite repository.*_url onto the internal host, re-sign with the shared secret and forward to /api/v1alpha1/autobuild. Rewriting and re-signing keeps OpenChoreo's own matching — appPath filtering, branch equality, per-component WorkflowRun creation — instead of reimplementing those rules and owning a second, subtly different copy of them.

Retrying on a substring of a human-facing message. promote_to_staging returned a status string, and the caller decided whether to retry by looking for "not ready" or "no release" inside it. The overwhelmingly common case — the build has not landed yet, so staging still legitimately matches development — returned "staging already on <release>", which contains neither, so the promotion was read as complete and dropped. Staging fell three releases behind with nothing logged as wrong.

Two things were needed, and only the first is obvious. The function now returns (status, message) so control flow never parses prose. But status alone is not enough: "the build has not arrived yet" and "there is genuinely nothing to do" produce identical observations from outside. So handle_push records what development pointed at when the push arrived, and promotion treats "still on the baseline" as wait, not done.

A cleanup that manufactures the deadlock it exists to break. unwedge() strips finalizers from objects stuck Terminating. Doing that to a ReleaseBinding first releases the Environment's block on it, so the Environment is garbage-collected — and a RenderedRelease's obsplane-cleanup / dataplane-cleanup finalizer resolves its plane client through that Environment (internal/controller/renderedrelease/controller_finalize.go). It can then never finalize.

Thirteen RenderedReleases from two closed pull requests were reconciling at tens of requests per second with no backoff ceiling, and controller-manager had restarted three times. Teardown must run bottom-up, RenderedReleases first. unwedge also refused to touch anything without project-flow's ownership label, and a RenderedRelease — created by OpenChoreo, not by us — never has one; openchoreo.dev/environment naming a review-pr-* environment is the tighter guard anyway, since nothing outside a review environment matches it.

The leak was observability-plane only. The data plane tore down correctly, so kubectl get ns on the data plane looked clean while the observability plane still held two namespaces and five ObservabilityAlertRules, themselves stuck on alertrule-cleanup — the log-alert path recorded elsewhere in this file as never having worked here. Checking the plane where the workloads run is not the same as checking every plane.

What generalises. All three failed upward: a mismatch became a 200, an ambiguous string became success, a cleanup became a loop. A delivery pipeline needs at least one assertion that runs the whole length of it, because every individual step can report health while the chain does nothing. The delivery index at project-flow's /delivery is the right place for it — it already joins commit to build to binding to URL, and a promotion pending past its deadline should be a series in it rather than a line in a log nobody reads.

Four ways a platform drops something on the floor without saying so

Everything below was found while building templates and a chat bot on top of the platform. They share a shape: a layer accepted input it did not implement, and reported success.

occ workload create silently drops configurations.secrets. The workload descriptor appears to support pulling a value from the shared secret store — configurations.secrets.envs[].remoteRef is in the render context's type (ContainerConfigurations) and the Workload CR carries it. But occ workload create, which the build's generate-workload step actually runs, emits a Workload with configurations: {} for secrets while the plain configs survive into a ConfigMap.

The component then deploys perfectly, with every config env var present and the one that was supposed to be a secret simply absent. Nothing errors at any layer. The fix is the shared-secret-env trait: a trait creates the ExternalSecret and patches the env in, the same way rails-runtime-secrets already did for a generated value. No app had used the descriptor's secrets block before, which is why the gap survived this long.

A ResourceReleaseBinding requires a release name that cannot be known in advance. Unlike a component's ReleaseBinding — where omitting releaseName lets OpenChoreo resolve the latest — a resource binding left unpinned reports spec.resourceRelease is unset and provisions nothing. And the release name is a content hash computed when the Resource is applied, so no template can contain it.

The failure surfaces two layers away: the database is never created, the component that depends on it starts anyway, and its db:prepare init container dies with PG::ConnectionBad ... socket /var/run/postgresql — a message about Unix sockets that mentions neither resources, nor bindings, nor releases. new-project.sh therefore applies the resources, waits for status.latestRelease.name, and patches it in.

Dagger caches a build step by its inputs, and a service binding is not an input. The Rails pipeline runs rails db:prepare and rspec as separate steps. Two components building from the same commit and the same appPath — which web and worker do by design, since they share an image — produce the same cache key for db:prepare. The second build reuses the first's cached layer without executing it, against its own freshly-started and completely empty Postgres.

The result is a suite that fails with relation "widgets" does not exist on one component and passes on the other, from an identical commit. It reads exactly like a flaky test. The template's spec/rails_helper.rb now migrates before the suite runs: idempotent, milliseconds when current, and impossible to cache apart from the tests it protects.

Solid Queue's schema is not in db/schema.rb. It ships as db/queue_schema.rb, loaded into a dedicated queue database. The platform provisions one database per Resource, so a Rails app here needs the queue tables as an ordinary migration against the primary — and without it the worker crash-loops on relation "solid_queue_processes" does not exist while the web component, which never touches those tables, is perfectly healthy. Half a deployment working is the confusing part.

What generalises. Three of these four were invisible because the thing that failed was not the thing that reported. The secret was dropped by occ and noticed by a bot; the resource was unpinned by a template and noticed by a Postgres client; the cache was reused by Dagger and noticed by RSpec. A platform that composes this many layers needs its assertions at the top — which is why the verification for every phase of this work was an end-to-end run rather than a unit test.

The delivery index answered the one question it existed to answer, wrongly

openchoreo_deployment_info is the join that makes the Delivery and Component 360 dashboards possible: commit, build, binding and URL in one series. Two more consumers were added on top of it — an Environments dashboard with a drift panel, and a chat bot that answers "where is this running". Then the drift panel would not fire, on a platform that demonstrably had drift.

The index read the deployed image from the component's Workload. There is exactly one Workload per component and every build rewrites it, so the image it holds is the image of the LATEST BUILD — not of whatever each environment is pinned to. Every environment therefore reported the same commit, always.

The consequences compound in a way worth spelling out:

The fix is to read the image from the environment's pinned ComponentRelease, which is immutable and carries its own rendered workload. spec.releaseName on the binding names it; spec.workload.container.image inside it is the image that environment actually runs.

Sharing a join between consumers does not make it correct — it makes every consumer wrong in the same way, and their agreement look like corroboration. The mistake survived a dashboard, a second dashboard and a bot, precisely because each new consumer was checked against the others rather than against the cluster. What caught it was pushing a commit to staging and asking whether production had moved.

appPath: "/" matches nothing, and the default is "/"

The path filter that decides which components a push rebuilds:

func (s *webhookProcessor) isComponentAffected(appPath string, modifiedPaths []string) bool {
	if appPath == "" {          // <-- the shortcut, taken BEFORE normalizing
		return true
	}
	appPath = strings.TrimPrefix(appPath, "/")
	appPath = strings.TrimSuffix(appPath, "/")
	for _, path := range modifiedPaths {
		path = strings.TrimPrefix(path, "/")
		if path == appPath || strings.HasPrefix(path, appPath+"/") {
			return true
		}
	}
	return false
}

"/" is not "", so it misses the shortcut. It is then trimmed to "", and the loop compares each modified path against the empty string: path == "" is false for every real path, and strings.HasPrefix(path, "/") is false because the path has just had its leading slash removed. A component with appPath: "/" can never be affected by a push that reports modified paths.

It builds under exactly one condition — a push with no modified paths at all, because the caller short-circuits that case:

if len(event.ModifiedPaths) == 0 || s.isComponentAffected(appPath, event.ModifiedPaths) {

which in practice means an empty commit. That is why the bug hid: every time it was tested with git commit --allow-empty to force a rebuild, it worked.

"/" is the default in the workflow's parameter schema and was set on six of this platform's components — all three bots, inkwell-enrich, ledger and storefront. Every one of them had silently stopped rebuilding on a real code change. The fix is appPath: "", which takes the shortcut and also resolves to the repository root everywhere else it is used.

Three things about this are worth keeping:

A build had no root span, its logs went to a trace store, and the noise filter ate the tree

Three findings from making build traces first-class (2026-09-02), each of which had looked like a limitation of Dagger or Grafana and was not.

The workflow minted a TRACEPARENT and nothing ever emitted that span. Every build therefore reached Tempo as <root span not yet received>: search results had no root service or name, Traces Drilldown could not group builds, and a traces panel fed from a search drew nothing -- which is why every dashboard had fallen back to tables that regex-matched span names. The fix is twenty lines of shell in the Argo step: derive the trace id from the run name (sha256, first 32 hex) so every consumer can compute it, and post the root span as one OTLP/JSON request from an EXIT trap, whatever the outcome. The trap runs set +e first; a trap that trips -e dies before posting, and this span is the one that records the failure.

Dagger's per-exec logs were configured away twice. Dagger sets up no log exporter from OTEL_EXPORTER_OTLP_ENDPOINT alone ("we can't assume all OTLP endpoints support logs", dagger/otel-go init.go); it needs OTEL_EXPORTER_OTLP_LOGS_ENDPOINT. And Alloy's batch processor sent OTLP logs to the Tempo exporter -- the same bug the metrics route had earlier, one signal over. With both fixed, a failing build's reason is one Loki query: {namespace="workflows-default"} | trace_id="<id>" |= "Error" found "go.mod requires go >= 1.25.0" attributed to the exec's span id, which is what "drill into just this stage's logs" was always supposed to mean.

A collector filters span by span and cannot re-parent. The first replacement for the whole-service drop used Dagger's own dagger.io/ui.internal| encapsulated|passthrough markers and produced 63 orphans, because the marked spans are mostly parents of unmarked children -- and POST /query (passthrough) is the structural parent of every stage's exec. What works is dropping whole subtrees: by instrumentation scope (dagger.io/filesync, dagger.io/codegen, the otelhttp/otelgrpc contrib scopes), by ui.encapsulated/ui.internal, by CLI ceremony name, and by DURATION for POST /query / resume <op> / API-call spans under 50ms -- the long ones are structure, the short ones are lazy calls and the module runtime's own queries. Measured against one captured unfiltered build: 1049 spans to 53, five orphans, all harmless -- after one more correction: the Container.withExec call span is a millisecond long and is the span Dagger attributes the exec's output to (and the parent the test container inherits through TRACEPARENT), so it is exempt from the duration rule. The first cut dropped it, and thirty rspec spans and every line of their output pointed at a span that no longer existed. The method mattered more than the rules: capture one build with no filter, simulate rules offline against /api/traces/<id>, and count orphans, not just spans.

And the span-profile labels disagreed by language. Ruby's pyroscope-otel wrote span and profile_id (and its tags are not a stack, so a child span ending erased the request's label); Python's upstream processor wrote span_name but for root spans only; Go wrote nothing. One contract now -- span_name as a series label, span_id and trace_id per sample, innermost span, probes skipped -- implemented once per language in platform_telemetry (gem), platform-telemetry (wheel) and otel-profiling-go, and selected by one name in every dashboard.

Spans that were created, propagated, logged, and never exported

Five findings from making the traces real again (2026-09-02). Each one had a symptom that pointed somewhere else.

OpenTelemetry Ruby drops the env-configured exporter the moment you add a processor by hand. Configurator#configure_span_processors reads processors = @span_processors.empty? ? wrapped_exporters_from_env.compact : @span_processors. inkwell added the span-profile processor (first pyroscope-otel, then PlatformTelemetry::SpanProfiler) and from that deploy on shipped no spans at all -- while still minting trace ids, propagating them to enrich and ledger, and printing them in every log tag. Tempo showed the downstream halves under <root span not yet received>, which reads as a collector problem, and orchard (same template, no gem in its Gemfile, so the rescue LoadError path skipped the processor) kept working, which reads as an inkwell problem. It is neither: the exporter has to be added explicitly whenever anything else is, and it goes first so a later processor's on_finish cannot stop it. The CI rspec plugin already knew this and said so in a comment nobody read back into the application.

Beyla traces the pods that already trace themselves, and the two never meet. eBPF sees the socket, not the trace id the process minted in memory, so every request to an SDK-instrumented service produced a second, one-span trace with its own id. The obvious fix -- Beyla's exclude_instrument on a pod label the otel-instrumentation trait stamps -- was in place for an hour and reverted, because Beyla is also the only source of http_server_request_duration on this platform (no SDK here emits metrics) and excluding a pod blanked its RED tiles. The duplicate spans are dropped in Alloy instead: k8sattributes now associates by the k8s.pod.name Beyla writes (association by connection tagged every Beyla span with Beyla's own pod), reads the trait's otel-sdk label, and the noise filter drops telemetry.sdk.name == "beyla" spans from labelled pods. Metrics never pass that filter.

Grafana's "Logs for this span" is a line filter. tracesToLogsV2 with filterByTraceID appends |= "<trace id>" to the query, so it finds only lines that print the id in the body. Alloy lifts trace_id out of the line into structured metadata for every tailed log, and Dagger's exec output arrives over OTLP with the id only ever in metadata -- so on a build span the button returned exactly one line, the Argo step's own build trace <id> echo. A custom query, {cluster=~".+"} | trace_id="${__trace.traceId}", matches the metadata instead; the pod/namespace tags were dropped with it, because an engine-origin build span names the engine pod while its log stream is labelled with the build pod, and an ANDed tag that disagrees returns nothing.

Span ids are not hierarchical, so "this stage's logs" needs a second writer. The build dashboard's stage click ran span_id=~"<stage>.*" and matched nothing: the output belongs to the exec spans under the stage, whose ids share no prefix with it, and Loki cannot be asked for "the descendants of this span". The pipeline module now re-emits each stage's output as log records of the stage span with ci_stage (and ci_origin, project or platform) in structured metadata, through dagger.io/dagger/telemetry's writer, and the rspec/pytest plugins post each example's captured output as records of the example's span straight to Loki's OTLP endpoint -- hand-rolled JSON, so no application gains a dependency. One stage or one test is now one query. The duplication is paid once per stage, not per exec.

Garage does not continue a caller's trace. Tested with a signed S3 HeadBucket carrying traceparent from the enrich pod: Garage's own span (trace_sink to Alloy) appeared under a fresh trace id, and no Beyla span joined either. So the S3 client span nests and Garage's server-side span is a separate trace, correlated by time and bucket -- the objectstore-garage resource type's comment was right, and it is now confirmed rather than assumed.

The build's root span was stamped in seconds, its children in nanoseconds. The Argo step's now_ns() guarded against a date that prints a literal %N; the build image's date drops %N silently and prints the seconds alone, so every root span started 1.8 seconds after the epoch and the waterfall's axis read 20698d 11h with the root's own duration as 0.05µs -- fifty seconds, in the wrong unit. The guard is now on the length of the number: anything under sixteen digits is seconds and gets nine zeros. Worth keeping because the failure mode was a dashboard that looked broken in a way that pointed at Grafana or Tempo, when the only wrong thing was one shell function in a YAML file.

Two smaller ones from the same day, both about "reliable" links: Grafana's Prometheus label_values() variable is bound to the dashboard time range, so a build dashboard whose trace id came from one went blank (trace id is required) for any build outside the range -- the id is now carried on every link and typed into a textbox variable; and Tempo's 48-hour retention means a build older than that has steps, logs and metrics but no waterfall, which the dashboard now says instead of failing.

2026-09-03: state after the observability and promotion work

Everything below was checked at 03:40 UTC after the session's changes; each item is either open or worth knowing. The memory notes in the session index carry the same findings with the reasoning.

Open

Fixed this session, easy to regress

The shared secret store is in memory, and losing it is silent

The shared OpenBao runs server -dev with no volume (infra/compose.yaml, command: server -dev). Dev mode's storage is in memory, so every restart of that container discards the whole store: the kv mount, all four kubernetes-* auth mounts, every policy and every secret.

That is a reasonable choice for a demo — dev mode is unsealed with a known root token, and managing unseal keys would be its own chore. What is not reasonable is that losing it says nothing.

How it presents. Nothing goes down. Every workload keeps running, because ExternalSecrets have already materialised real Kubernetes Secrets and those survive a reboot perfectly well. Every ExternalSecret on the platform goes to SecretSyncedError and stays there, which no dashboard shows and no alert reports — Alertmanager's default route is the null sink. The first symptom arrives only when something new needs a secret, and it arrives as

CreateContainerConfigError: secret "console-development-1edb32ac-flow-signing-key" not found

which names neither OpenBao nor the reboot that emptied it.

Found by deploying a new component. The host had rebooted at 04:07; the store had been empty for fourteen hours; thirteen ExternalSecrets across three clusters had been failing that whole time; and the thing that noticed was a brand new pod refusing to start. platform/cloudflared and platform/project-flow were simply gone.

Three levels of cached failure, and each needs its own nudge. Restoring the mounts is not enough:

  1. Re-running 35-shared-secrets.sh recreates the mounts and policies, but
  2. the ClusterSecretStore stays InvalidProviderConfig — the controller caches "unable to create client" and does not retry. An annotation change is a spec-level event, so it requeues and reports store validated; and
  3. the ExternalSecrets stay SecretSyncedError even once their store is Valid and even past their own refreshInterval. One on a 30-second interval had been failing for fourteen hours and went green the moment it was annotated.

Nudging only the store fixes the store and nothing else. This is the third distinct place on this platform where a terminal condition needs an annotation to requeue, after the RenderedRelease case above — it is a pattern, not a one-off, and it is why a reconciler that only reports stuck objects is worth less than one that nudges them.

platform/bootstrap/restore-shared-secrets.sh does all of it, idempotently: checks the mounts, rebuilds every secret whose value also exists on disk, and nudges both levels. Run it after any reboot.

The constraint that makes recovery possible at all is worth stating on its own: every value the platform keeps in the shared store also exists in a gitignored file or can be re-read from the service that owns it. A secret that lives only in this store cannot survive a reboot. That is a rule to follow when adding one, not a property to assume.

platform/cloudflared is the exception and is deliberately left for a person: it can only be re-fetched from the Cloudflare API by 30-cloudflared.sh, and the running tunnel is serving fine on its existing Secret, so the repair is not urgent and the blast radius of getting it wrong is every public hostname.

Still open: the root cause. Either give OpenBao a file backend and a volume — which buys persistence at the price of an unseal step and somewhere to keep the keys — or accept dev mode and run the restore script from a boot unit beside openchoreo-hostnames.service. The second matches what this platform already does about host.k3d.internal after a reboot, and is the smaller change; the first is what a real installation would do.

Also worth knowing: the OpenChoreo-shipped default ClusterSecretStore has been InvalidProviderConfig independently of all this. It authenticates against a kubernetes auth mount with role openchoreo-secret-writer-role, and no script in this repository creates either — 35-shared-secrets.sh only creates the four per-cluster mounts. One ExternalSecret (observer-secret) depends on it.

A firing disk alert with 86GB free

NodeDiskFillingUp fires at 85% and NodeDiskCritical at 93%, on a 938GB filesystem. It has been firing at 91% — which is 86GB free, more headroom than most of this platform will ever need at once. prune.sh --apply reclaims essentially nothing, because the space is genuinely in use by build caches and volumes that are load-bearing.

A percentage threshold is the right shape on a small disk and the wrong one here: it pages long before anything is at risk, and an alert that is always firing is an alert nobody reads. The threshold should be on absolute free space (or on both, whichever is tighter). Recorded rather than silently retuned, because "the alert is wrong" and "the disk is fine" are different claims and only the second one is comfortable.

Pyroscope wedged a second time, and the tell is still zero restarts

2026-09-05, and the same fault as 2026-09-04: pyroscope-0 at 0/1 Running for 17 hours with zero restarts, logging

rpc error: code = Unavailable desc = node is not the leader
failed to prepare compaction plan

several times a second. A single-node Raft metastore that loses leadership does not re-elect; /ready waits on the metastore, so the readiness probe fails forever. kubectl delete pod bootstraps an election and it is Ready in about 80 seconds -- exactly as recorded the first time.

The console reported it correctly on both occasions. Could not read pyroscope. HTTPStatusError: 503, with the URL. The failure was never in the reporting; it was that a fault whose only remedy is a human noticing had no probe that would notice.

Now it has one, and the chart cannot express it. helm template grafana/pyroscope 2.2.1 with a components.all.livenessProbe renders no liveness block at all -- the chart emits a readinessProbe and nothing else, and there is no values key for the other. So 26-lgtm.sh patches the StatefulSet immediately after the helm upgrade that creates it. That keeps it in git and re-applied by every run of the script, rather than in one operator's shell history, which is the distinction that matters: a kubectl patch nobody wrote down is a fix that survives exactly until the next install.

The thresholds are deliberately slack: 180s of grace, then ten failures at 30s. A wedged metastore never recovers, so nothing is gained by reacting fast; a Pyroscope replaying a write-ahead log is merely slow and must not be killed for it. About eight minutes of continuous unreadiness before a restart, against a healthy start of roughly 90 seconds -- and against 38 hours.

What the outage was hiding, found by querying the labels the moment it came back, and all three confirm what the profiling work is for:

One label, three stores, and the console pivots on a fourth

Closing the ci_stage gap above looked like a one-line change: the rspec and pytest plugins already configure Pyroscope, so give rubocop, bundler-audit and pip-audit the same treatment and tag the profiles with the stage. That part worked on the first try. It also produced profiles that no page could display, and the reason is worth keeping.

ci.stage is the bare step name. stage() in platform/main.go takes "rails-ci/lint" for the span and sets ci.stage="lint" from everything after the last slash, deliberately, so a dashboard can ask for a build's stages without matching span names against <lang>-ci/.*. Loki's structured metadata carries the same bare name. Tagging the profile ci_stage="lint" therefore lines up with both stores exactly as intended.

The console does not query any of that. It finds a build's profiles by asking Pyroscope for the values of span_name scoped to ci_build_run, and matching them against the span names in the trace (app/sources/pyro.py, profiled_span_names). build_profile then hard- requires span_name="…" in its query. A profile tagged only ci_stage is correct, queryable by hand, and invisible in the build viewer.

So the profiles now carry both: ci_stage for the cross-store query, and span_name for the console. They are different strings and neither is redundant. The span name is a const beside the stage() call that opens the span, because a mismatch between them is not an error anywhere -- it is a stage that reports "no profile" forever, which is indistinguishable from a profiler that was never installed.

The general shape: a label that exists in two of three stores looks like it only needs adding to the third. Check what the reader actually selects on first. Here the reader selected on a label the producer had never been asked for, and the whole change would have shipped as a silent no-op.

A profiler that loads before bundler cannot see the bundle

The stage profiler is injected with RUBYOPT=-r/opt/ci_stage_profile, the same argument that puts the rspec plugin behind --require: it reaches bundle exec rubocop without the platform reconstructing rubocop's own command line, and the application changes nothing.

It never once profiled anything, and nothing failed.

RUBYOPT's -r files are required as the interpreter starts — before bundler/setup has put the bundle's gems on the load path. The bundle here lives in BUNDLE_PATH=/bundle-baked, which is on no default load path at all, so require "pyroscope" raised LoadError on every stage however present the gem was. The file took its "gem not in the bundle" branch and returned. The rspec plugin never meets this because RSpec's --require runs long after bundler has set up.

The fix is to try bundler/setup and retry the require, kept inside the one file that has the problem rather than by putting -rbundler/setup on every exec in the container — where it would turn an unsatisfied bundle into a crash in stages that currently do not care.

The diagnostic was there and could not be seen. The file warns on the LoadError path, and Dagger does not surface an exec's stderr when the exec SUCCEEDS. A lint that passes prints nothing. So the warning existed, was correct, was never read, and the symptom — no profiles — is exactly what having no profiler at all looks like. It was diagnosed as that twice. A diagnostic on the success path of something that does not fail is not a diagnostic.

Getting it out took replicating the container by hand — with-deps, then the same withNewFile and env vars withStageProfile sets, then a script that runs bundle exec ruby with 2>&1 and returns stdout. That is worth doing early rather than late: two rounds of plausible theories (the gem is missing, the agent needs a flush, --parallel forks away from the profiler) cost more than the probe did.

Nothing flushes at exit, and short stages are all exit

Second reason the same file pushed nothing: the Pyroscope agent uploads on an interval. A test suite runs for tens of seconds and crosses several of them. bundler-audit check takes three seconds, exits, and every sample it collected dies with the process.

Both existing plugins already handle this — ci_span_profiles.rb:445 and ci_span_profiles.py:527 call Pyroscope.shutdown at the end of a run — but that is hung off the suite's own end hook, and a stage has no suite. It goes on at_exit instead.

Worth reading together with the sampling floor already documented here: a process that runs for less than ~10ms at 100Hz has no samples, and a process that runs for three seconds may have samples and no opportunity to send them. The two look identical from the console.

Pyroscope.shutdown returns before the upload finishes

The rspec suites had no CI profiles at all — not a thin one, not a short-suite one, none — while pytest on the same platform had them. Found on the first real Rails build after the stage profilers went in, where lint and audit on the same component in the same build reported has and test reported none.

Pyroscope.shutdown returns before the agent has finished sending. The after(:suite) hook called it and rspec exited, and the profile went with the process. Measured against the running Pyroscope, five runs each, identical 2s of CPU, the only difference being what happens after shutdown:

after shutdown landed
exit at once 1/5
sleep 2 5/5

The lost runs are lost permanently — still absent an hour later — so this is not ingestion lag. With the grace in place three consecutive rspec stages land 3/3, and a real build now renders test as has with 24 of its 34 examples carrying their own profile. (The other ten are under the sampling floor, which is the documented and correct answer.)

Two wrong turns, both of which are what a race looks like from the inside:

Adding warn calls to the flush chain made it start working. That reads as evidence about the chain and is evidence about timing — the extra I/O was the fix. Instrumenting a silent failure can move it.

Pyroscope.stop looked like the reliable alternative on a single trial, then landed 1/3 on repetition while shutdown landed 3/3. n=1 on a race is worth nothing. The two methods are the same coin flip; the grace is the variable that matters. Both of these had a clean story attached and both were wrong, which is the argument for repeating a timing experiment before believing it.

pyroscope 1.1.2 has no flush API, so a sleep is the instrument available: CI_PROFILE_FLUSH_GRACE, two seconds, once per stage against a four-minute build. The stage profiler gets it too — it had been passing on luck, because interpreter teardown happens to take long enough often enough, and that is the kind of luck that becomes "the profiler is broken" on a faster machine.

Python is deliberately left alone: no loss has been observed there across several real builds, and a speculative fix with no measurement behind it is how the next confusing thing gets added.

The general shape, and it is the third time in this one piece of work: a telemetry path that fails silently is indistinguishable from one that was never installed. This one additionally fails intermittently, so it was indistinguishable from a short suite as well.

The outage test had never simulated an outage

CI failed on test_no_json_endpoint_returns_500_when_every_upstream_fails. The test was right; it had simply never run. Its patch loop read

mod = getattr(_http, "__module__", None) and __import__(...)

and _http is a module. Modules have no __module__, so the guard was None, the and short-circuited, mod was None on every iteration, and dir(None) is all dunders — nothing was ever replaced. It passed by hitting the real upstreams, which are reachable from the build container, and failed the one day CI lost DNS, which is the only day it did what it was written to do.

Working, it found two bare 500s immediately: /api/deploys/{c}/{e} iterating model.deployments() unchecked three lines below /api/deployments doing that check and explaining why, and /api/components/{c}/graph calling gitgraph.for_component unguarded.

A test that passes for a reason you have not checked is not evidence. This one had a green tick for as long as it had existed.

The console had the discipline; the sibling app had none of it

Running the same outage sweep against the other apps' pages found that buildviewer had no soft() at all. _prom and _trace_cached call httpx with raise_for_status() and nothing catches, so an unreachable Prometheus or Tempo made five of its ten routes a bare 500 — including /builds, the page the app exists for.

The console's whole "an unreadable panel must never render like an empty one" discipline had simply never been carried across. Worth knowing generally: a convention that lives in one application's code is not a platform property, and nothing on this platform was checking.

What the sweep could and could not reach:

app result
console 49 pages, 27 API routes — clean after this session's fixes
buildviewer 5 of 10 routes 500 — fixed
chess/engine no outbound HTTP on any GET route; the sweep proves nothing, and says so
chess/render needs libcairo, absent from the uv container — would need its own image
enrich, ledger, zulip-bots no page routes worth sweeping
inkwell, orchard (Rails), rideshare (Go), brochure, storefront (static) different runtimes; this technique does not apply

Patch httpx's transport CLASSES, not one client. The console's version fails the single client its source modules share, which works because it has one. buildviewer builds its own, so the generic sweep patches httpx.HTTPTransport.handle_request — every client, made anywhere, at any time. That is the only way to say "the network is down" to code you have not read.

And the fix's own guard needed the same lesson twice: `

{% if unavailable %}

never fires, becauseUnavailableis deliberately **falsy** so that

{% if rows %}

treats it as nothing-to-show. The guard against rendering an outage as an empty list rendered it as an empty list.is not none` is the test.

A test that cannot detect its own vacuousness

The API outage sweep patched nine of the eleven source modules that exist. catalog and upgrades were never in the list, despite 17 references from main.py, and nothing would ever have said so.

That is a symptom. The disease is that with every upstream down, seventeen of twenty-seven API routes still answer 200 — correctly, because their soft default is an empty list — and every page answers 200 with an "unavailable" panel. So "nothing failed" is not evidence of anything. It is equally consistent with a program that handles outages and a test whose patching does nothing.

Which is exactly how that test passed for its entire life.

Both sweeps now count transport calls and assert the count is non-zero, and that guard was verified by sabotage: removing the patching — reproducing the silent no-op the __module__ bug produced — makes both fail with "the transport was never called: this test proved nothing" rather than passing.

The general rule: a test whose subject can succeed without the condition it is testing must assert the condition was applied. Otherwise its green tick carries no information, and the day it starts lying is the day nobody notices.

Patch the transport, not the module functions

Extending that sweep to the 49 HTML pages found five broken. Four of them were fine.

Replacing every public function in every source module also replaces the pure ones — alerts.undelivered is a list comprehension over its argument — so a handler that calls one looks exactly like a handler with an unguarded network call. That is simulating a broken program, not an outage, and it accused the home page.

Failing the single shared httpx.Client instead is precisely "the network is down" and nothing else. It reported one page, and that one was genuinely broken: /projects/{p}/environments/{e} iterated the Unavailable that deployments() returns — the third instance of that same bug in one session.

The fix filled the unavailable slot the page already carried and the template already rendered, rather than returning an empty list: "nothing is deployed here" and "I could not find out" must never look the same.

A link from a server-rendered row to a client-rendered widget rots silently

Adding "Memory by test" meant each row naming a span, and the only place a span can be inspected is the waterfall — which is drawn by JavaScript from a JSON fetch, so there is no #span-<id> element for an anchor to reach. The first version linked to one anyway. It would have rendered, looked right, and done nothing.

Three things had to be true for that link to work, and only the first is obvious:

The same audit found the retained profile was summarised in ten rows and otherwise unreachable — thousands of frames collected, uploaded, stored, and with no way to be seen. "The data is there" and "the data is reachable" are different claims, and only the second is worth anything to a reader.

pprof carries its units; folded stacks do not

The memory profiles used to arrive as folded stacks, which carry no units. Pyroscope filed them under the default CPU profile type and scaled them by the sample rate, so a 57MB retained profile read as 755,624 CPU seconds — which is why every build-scoped CPU query in the console carried service_name!~".*-ci-(mem|alloc)". That matcher is now deleted, because a pprof declares its own sample type and a CPU query cannot see it.

Verified at both ends. Before writing an encoder: a real Go heap profile pushed to this Pyroscope lands under memory:inuse_space:bytes:space:bytes with the right byte total and leaves process_cpu at zero. After: the totals the console shows are byte-for-byte what the uploader reported — 75,598,134 for the console's own suite — where before they were that number times ten million.

Two hand-rolled encoders, Ruby and Python, deliberately twins. The schema needed is five messages with no extensions, which is smaller than adding a protobuf dependency to every application's lockfile, and both are validated against Go's own pprof parser rather than against themselves.

Three things the format does not forgive:

There are two 4 MiB limits and they are not the same limit. One is on the decompressed profile — measuring the gzip instead yields decompressed size exceeds maximum allowed size from a body that looked comfortably small — and the other is an ingest rate, shared with every CPU profile the same build is pushing. Both encoders halve and retry with backoff rather than guess a constant that satisfies both. Coverage went from 79.2% of bytes on the folded path to 96.3% (Ruby) and 100% (Python).

And once more, the trap of the day: the Python converter reports what it uploaded on stderr, and Dagger does not surface an exec's stderr when the exec succeeds. Without 2>&1 the step was silent whether it worked or not. Third occurrence today.

The vernier converter, and what retained memory says about bootsnap

vernier's trace_retained writes Gecko (Firefox Profiler) JSON; Pyroscope ingests folded stacks. Nothing existed in between, which is the only reason Ruby had been reporting allocation churn instead of the question Python answers.

Forty lines close it. weightType is "bytes", so this is a real byte figure rather than a count wearing a byte label. stackTable is a prefix tree — entry i holds one frame and the index of its parent — so resolving one stack is a walk to the root, memoised, or a deep Rails stack is re-walked once per sample sharing it, which for a suite is most of them.

Validated against an oracle before being trusted. A workload retaining 200 × 2048 bytes and churning 500 × 512 folds to 418,040 bytes — vernier's own sum exactly — with 417,800 of it on the retaining stack and the churn correctly absent. Getting the prefix tree backwards yields plausible-looking nonsense, so "it produced output" is not evidence here.

Pyroscope refuses rather than truncates. The first real retained profile was 26,693 stacks and came back HTTP 400; the limit is a 4 MiB push. The producer now sends the largest stacks that fit — 632 of 26,517, holding 79.2% of the bytes — and logs what it dropped. The profile exists to answer "what is holding the memory"; the answer is the top of the ranking and the tail is thousands of stacks holding a few bytes each.

What it then said, on the first real build:

81.2%  RubyVM::InstructionSequence.load_from_binary
 4.6%  Thread.new
 4.1%  Module#module_eval

Four fifths of the suite's retained memory is bootsnap's loaded instruction sequences — the cache added earlier the same day to cut the boot CPU. That is the trade stated in bytes: ~23% less boot CPU for ~46MB held. Neither number was visible before today, and a CPU profile alone would never have shown the second.

Ruby still pushes churn separately, under -ci-alloc. A suite can be fine on one and bad on the other, so both selectors have to be excluded from every build-scoped CPU query, not just -ci-mem.

The cost is real and worth stating: with both profilers on, the test stage's CPU went from 3.05s to 8.10s. That is what profileRich buys and why it is opt-in.

Ruby memory profiling: what three profilers can actually do

Evaluated against the real inkwell suite, because the rejections are the useful part.

ruby-prof 2.0.5 is out on capability. RubyProf.constants offers ALLOCATIONS, PROCESS_TIME and WALL_TIME — the MEMORY mode older versions had is gone. So it cannot report bytes, only object counts, which is what stackprof does by sampling instead of by instrumenting every call. No collapsed printer either.

vernier 1.11.0 is the interesting rejection, because its answer is better. trace_retained reports what is still LIVE — the direct analogue of memray's high-water mark on the Python side, and the more useful question. It writes Gecko-format JSON, so it needs a converter that does not exist yet, and it needs Ruby 3.2+. This is the upgrade to make, not a dead end.

stackprof 0.2.28 wins on plumbing. print_stackcollapse emits exactly the folded format Pyroscope ingests — no converter to write, none to keep in step. Three traps in making it work:

What it measures, stated plainly: sampled object allocations — churn, and therefore GC pressure. Not residency. The Python side reports bytes live at the high-water mark. The two do not convert, and the console names which it is showing rather than calling both "memory".

A frozen bundle is the only step that notices a Gemfile/lock mismatch

Adding stackprof with bundle add wrote gem "stackprof", "~> 0.2.28" into the Gemfile while the lockfile's DEPENDENCIES section recorded a bare stackprof. Lint passed. Both audits passed. All four test runs passed. The image build failed, because the Dockerfile is the only step that runs bundler in deployment mode, where the two must match exactly:

failed to content hash dockerfile copy: exit code: 16
  ... remove the Gemfile.lock freeze by running `bundle config set frozen false`

The fix is a bare gem "stackprof", require: false, which is also the house style — every other DEPENDENCIES entry in these Gemfiles carries no constraint.

The lesson is about where to look: a green test stage says nothing about whether the bundle is internally consistent, and the error surfaces four stages later wearing a BuildKit hash failure as a disguise.

bootsnap was pure overhead, and the profile said which half

The cost panel said "outside any span" was 44% of inkwell's test CPU and 60% of orchard's -- more than every example put together. The flame graph named it:

430ms  Bootsnap::CompileCache::Native.fetch
 80ms  RubyVM::InstructionSequence#to_binary

to_binary is the serialise side. It only runs on a cache MISS, which is what turns "bootsnap is busy" into "bootsnap's cache is cold on every build". It lives under the app's tmp/cache, which arrives empty from the source directory every time -- so bootsnap was doing all of the bookkeeping and delivering none of the benefit, on every Rails build this platform has ever run.

Fixed by mounting a Dagger cache volume at /bootsnap and pointing BOOTSNAP_CACHE_DIR at it. Outside /app because WithDirectory replaces the app directory wholesale; keyed by Ruby version because compiled iseq is version-specific; shared across applications on purpose, since entries are content-keyed and two Rails apps compiling the same Rails is the case worth sharing.

cold  boot 0.75s   fetch 100ms   to_binary 20ms
warm  boot 0.58s   fetch  40ms   to_binary  0ms

to_binary reaching zero is the proof it is a cache hit rather than a faster miss. Boot CPU down ~23%, the bootsnap portion of it by ~84%.

The general shape: a cache that is present, configured and cold is worse than no cache. Nothing reports it, because nothing is broken.

Pyroscope's folded ingest carries no units, and it will not tell you

The opt-in memory profiles reach Pyroscope as folded stacks, because memray's own transform offers gprof2dot, csv and speedscope and Pyroscope reads none of them. Folded works -- but it carries no units. Pyroscope stores the values under the default CPU profile type and multiplies by the sample rate.

So a 70MB high-water mark arrives looking exactly like a CPU profile. Grouped by service, on a build that had both:

console-ci-mem   755624.5   (as CPU seconds)
console-pytest        0.6   (as CPU seconds)

It does not add a wrong row to the cost ranking. It dwarfs every true one, and build_profile would have merged bytes and nanoseconds into a single flame graph. Every build-scoped selector now excludes -ci-mem as a matcher, not as a filter on the results, so the store never sends them and no later caller can forget to drop them.

Two things follow. The memory panel says to read the ranking rather than the axis, and points at the build log for the true byte figure. And emitting real pprof with a declared inuse_space sample type is the fix that would make the units honest -- folded stacks are what let this work at all without generating protobuf from Python.

An empty flamebearer is truthy

Pyroscope answers a query that matches nothing with a well-formed flamebearer carrying one frame and zero ticks. It is a truthy dict. So if memory: drew a memory panel -- ranking one empty frame -- on every build that never asked to be profiled that way.

profiles.shape() has known the difference between "empty" and "absent" since the flat-flame-graph work; the new code simply did not ask it. This is the "empty vs refused" rule in this document meeting a third store, and it has now been reintroduced once per store.

Found by writing the test, not by reading the code. The test asserted the panel was absent and the page said otherwise.

A profiler that reaches every process reaches the ones being parsed

The Python stage profiler is injected as sitecustomize on PYTHONPATH, because any interpreter imports sitecustomize at start if it is importable. That is precisely why it was chosen: it reaches uv run pip-audit without the platform reconstructing pip-audit's command line, and the application changes nothing.

It reaches the children too, and that failed a real build:

ERROR:pip_audit._cli:failed to list installed distributions

pip-audit enumerates the environment by running python -m pip list -v and parsing its stdout. The Pyroscope agent writes a line to stdout when it shuts down. So the moment an at_exit flush was added — which short stages need, because otherwise they exit before the agent's first upload — every one of those children appended a line to the output being parsed. Observed argv during one run:

/opt/venv/bin/pip-audit   <- the process worth profiling
-m  ['list', '-v']        <- its stdout is parsed by the above
-m  ['--version']         <- same

Note the shape of it: the profiler worked, the profile arrived, the stage failed. The ci_build_run label for the failing run is in Pyroscope, because the main process was profiled perfectly well before pip-audit gave up on its own subprocess.

Two fixes, either of which would have been enough:

The general shape: an injection mechanism chosen for its reach — RUBYOPT, sitecustomize, LD_PRELOAD — has exactly that reach, including into processes whose output someone is parsing. Ask what the tool shells out to before deciding the injection is free.

Ruby was left alone on purpose: bundle exec execs rather than spawns, bundler-audit shells out to git rather than to ruby, and rubocop's parallel workers were checked and produce clean output. A speculative version of the same fix, with no evidence behind it, is how the next confusing thing gets added.

"No profile" was three different statements wearing one badge

The build page had three profile states: has, short (under ~10ms at 100Hz there is nothing to sample), and none for everything else. none renders as "Long enough to sample, but nothing arrived. Usually means the profiler is not installed in that process" -- accurate, specific, and actively misleading for two stages that could never have had a profile:

The engine profile comes with a caveat that is stated every time it is drawn rather than once in a doc: the engine runs at max-parallelism = 4, so the window can contain three other builds' work. It is the CPU the engine burned while the stage ran, not the CPU the stage caused. Presented as the latter it would be a confident wrong number, which is worse than the empty panel it replaces.

has is still checked before any of these tables. They describe today's pipeline, not a rule the data has to obey: if someone wires a profiler into one of these stages, the profile that actually arrived has to win, or the table quietly hides the work it was meant to prompt.

A rotated secret does not reach a running pod

infra/rotate-webhook-secret.sh moves the webhook/HMAC secret in all four places it is trusted — the file, git-webhook-secrets, project-flow-secrets and every repository webhook — and FEEDBACK.md records the two Gitea traps that made the first attempt lie. It moves every store. It does not restart a single consumer.

The bots take that secret as an environment variable, and an environment variable is fixed when the container starts. So after the 16:07 rotation the walkthrough bot went on signing with the pre-rotation value while the Secret object beside it held the new one. Every call it made to project-flow came back

401 {"error": "bad signature"}

and the first thing to notice, three hours later, was a walkthrough that would not start. All four stores agreed with each other; only the running process disagreed, and nothing compares those.

Worth being precise about the general rule, because it is not "restart everything after a rotation": an ExternalSecret refresh updates the Secret object, and that reaches a pod only where the value is mounted as a file and the consumer re-reads it. envFrom/env bindings are copied once, at container start. So a rotation has to name its consumers and roll them.

Here that is three deployments — openchoreo-bot, walkthrough-bot and console — and they are findable, which is the useful part:

kubectl --context k3d-openchoreo-dp get deploy -A -o json \
| python3 -c 'import json,sys
for i in json.load(sys.stdin)["items"]:
    for c in i["spec"]["template"]["spec"]["containers"]:
        if any(e.get("name") == "PROJECT_FLOW_SECRET" for e in c.get("env") or []):
            print(i["metadata"]["namespace"], i["metadata"]["name"])'

Confirming a consumer is actually on the new value needs the running process, not the Secret:

kubectl exec <pod> -- sh -c 'printf "%s" "$PROJECT_FLOW_SECRET" | sha256sum'

Three ways a teardown can leave the platform worse

All three found in one afternoon, running @walkthrough teardown rails.

It waited for a build that cannot exist. bakery-web owns the build and names bakery-worker as a siblingWorkload; the worker carries autoBuild: false and runs the same image with a different command. Two builds serve three components. The walkthrough still looped over all three, polling /builds?component=bakery-worker every ten seconds for the full 720s timeout, while staging and production had been Ready for ten minutes. The dry-run test asserted three build phases and had passed the whole time, because it fakes the build lookup and the answer always arrives.

It deleted the resources of a walkthrough it was not tearing down — or none. The resource cleanup asked whether the project still held any walkthrough component. That is the right question only while a project hosts one walkthrough. showcase hosts two: tearing down rails found brochure standing, called it a survivor, skipped the block, and left Resource/bakery-db plus two CloudNativePG clusters running with nothing pointing at them. Scoping by REPO_LABEL instead is correct — both Components and Resources carry it.

...and then that fix uncovered a destructive one. In the same block, the ProjectReleaseBinding deletion was scoped by project alone. Repo-scoping the survivor test made the block reachable on a no-op teardown, and it deleted showcase-staging and showcase-production — the cell namespaces — taking brochure's pods, routes and namespace with them. The control-plane objects survived, so kubectl get component brochure looked perfectly healthy while the site returned 404: an outage with no un-Ready object anywhere to point at it.

The lesson is not "be careful". It is that the two things have different scopes and cannot share one test: a Resource belongs to the repo that scaffolded it, a cell namespace belongs to the whole project. They now have one test each, and an empty repo set short-circuits both — without that guard, a teardown that matches nothing concludes "no survivors" and deletes everything.

And a retained binding is deleted without being released. The rails scaffold sets retainPolicy: Retain in production, which is right for a real database and wrong for a walkthrough's. Deleting such a binding is accepted and does nothing: the finalizer holds until someone flips the policy back. So teardown reported Done while a production Postgres kept running and its Resource sat Terminating indefinitely. Teardown now releases the retain first, scoped to the walkthrough label and the repos being removed.

A build that did everything right and reported Failed

openchoreo-bot-run-4e0cb757: lint, audit, test and publish all succeeded, the image was pushed, and all five generate-workload-cr fan-out steps updated their Workload with HTTP 200. The run is Failed.

The last thing each fan-out step does is annotate the shared WorkflowRun with its result. Five steps run concurrently and all five write the same object; four won and one got

Failed to update WorkflowRun (HTTP 500): {"code":"INTERNAL_ERROR"}

A lost write on a bookkeeping annotation, surfaced as a 500 rather than a conflict the step could retry, fails the step and therefore the build. Every consumer of that verdict is then wrong: the Gitea commit status says failure, openchoreo_build_* records a failure, and anything gating on build result blocks a build whose artefacts are all present and correct.

Two things follow. The step should retry a conflict on an annotation it does not own exclusively — it is idempotent by construction. And a build's verdict should not be decided by a write that happens after the image is published: the fan-out is a side effect of a build that has already succeeded.

Independently confirmed the same day: the engine's DNS failure at publish is still the one open item of the three in FEEDBACK.md. A console build lost all of its work to lookup host.k3d.internal on 10.87.0.1:53: no such host, retried three times by the module, and failed. Cluster DNS on the workflow plane answers that name correctly and a busybox pod resolves it; only buildkit's embedded resolver inside the engine does not, and restarting the engine did not clear it. The next push succeeded. Until the registry is addressed by IP from inside the engine, this stays a coin flip on every build.

Two credentials of the same shape, and a failure three layers away

Restoring the wiped store, platform/zulip-admin was rebuilt from infra/.zulip-announcer. Both are (email, apiKey) pairs, both are Zulip credentials, and the file was right there. They are not the same identity: the announcer is a bot, and Zulip answers a bot caller asking GET /bots with

400 {"msg": "This endpoint does not accept bot requests."}

The zulip-bot resource type's provisioning script is idempotent by lookup-first — it lists the bots, reuses an existing one's key, and only creates when absent. Take away the lookup and idempotent becomes always-failing. What that looked like, in order:

  1. the identity Job exits 1 on an HTTPError: 400 with no body printed;
  2. readyWhen stays false, so ResourceReleaseBinding never goes Ready;
  3. the consuming component reports ResourceDependenciesPending, so its env bindings are never injected;
  4. chess-bot crash-loops on KeyError: 'ZULIP_EMAIL'.

Four layers, and the message at the bottom names a missing environment variable while the cause is the wrong kind of account at the top. Nothing in between says "credential".

Worth generalising past this one secret: restoring a store is not the same as restoring a service, and a value that merely has the right shape is not the right value. A restore should re-derive a credential from the thing that issues it wherever it can — restore-shared-secrets.sh now logs in as the documented realm owner rather than copying a lookalike file — and it should expect that putting a secret back RE-RENDERS the resources that consume it, which re-runs their provisioning Jobs. The restore is when idempotence gets tested, and it is the worst time to discover it was never real.

Two smaller things confirmed on the way. The Job object is immutable, so a corrected credential does not reach it: the Job has to be deleted and the RenderedRelease annotated before the controller rebuilds it — the same nudge pattern, for the fourth time. And docs/ACCESS.md's admin@openchoreo.dev is a delivery address; the account behind it is user8@zulip.TAILNET.ts.net, which is why it does not appear in a user list you search by email.

A new OIDC client is two registrations, and the second one is silent

The console could not be signed into. It reached Thunder, showed a perfectly good username-and-password page, and never came back — /auth/callback was not hit once. Most of the console is behind that sign-in, so what a person reports is "I cannot log in and most clicks go to a white page", which names neither Thunder nor the file responsible.

Registering an application (71-console-app.sh) is only half of it. An application with no auth_flow_id falls back to default-basic-flow: local username and password, no GitHub button at all. Assigning the flow is a second, separate step, and 69-auth-flows.sh did it for a hardcoded list — Backstage Outline Gateway Zulip Grafana Gitea — that a new application does not join by existing.

The remarkable part is that this is the second time, and the file says so. Its own header records the gateway and Zulip clients being added later and nobody adding them here, with the sentence "it fails silently, because a perfectly good login page appears". A comment describing a trap does not stop anyone falling into it; only the list being derived, or the trap being impossible, would.

Two things worth carrying:

The publish check for a library had never run

pypi_check on platform_telemetry failed on two things at once, and both had been true for a while, because publishing is manual and rare.

Two lines over the 100-column limit — ordinary, and only surprising because the check that would have caught them was itself broken.

The real one: tests/test_link_spec.py computed the repository root as parents[4] and read files from platform/portal-image/ and platform/techdocs/. Those assertions are worth having — they are the agreement between this package's dashboard spec and the two places that hand-write dashboard URLs — but the publish pipeline mounts the package, not the repository, so parents[4] walked off the top of the filesystem and raised IndexError at import. pytest reports that as a collection error, the check step as a failure, and the package as unpublishable.

Fixed by locating the repository if it is there and skipping if it is not. A skipped cross-repo test in a package-only container is honest; an import error is not. The general shape is worth remembering: a test that reaches outside its package silently couples the package to its checkout, and the first place that bites is the one place you want to be able to build from source alone.

Dagger's cache signal: what it says, and what it only appears to say

The build page could tell you a stage took 1.3s and not whether that was work or a cache hit. Dagger already annotates its own operation spans, so the signal was there; the work was in reading it without overclaiming.

The trace is the wrong primary source, and the log is the right one. This was found by being asked how a high-level operation could cache while the image underneath it did not — a good question, and the answer was that the trace only carries part of the story. Dagger prints a verdict for every operation to its own output, and that output ships to Loki like any other build log. On one build the trace exports 17 operations and the log reports 88, of which 14 CACHED. So the headline number now comes from the log, where anyone can go and read the line that produced it, and the trace is kept only for per-stage attribution — which the log cannot give, because its nesting is drawn for a terminal and rows are re-printed as they resolve, so a depth stack over the shipped lines mis-attributes nearly everything (tried, measured, discarded).

The join, for the part the trace still does. Dagger's spans carry no ci.stage and the platform module's stage spans carry no cache verdict, so attributing an operation to a stage is a walk up the parent chain. Every Dagger operation does have a ci.stage ancestor, so that attribution is exact.

The digest is not stable across builds. dagger.io/dag.digest is a content digest and the obvious key for "how often does this step cache". It closes over the source tree and the session, so it changes with every commit: measured over five real builds, 2 of 79 digests recurred. Grouped by digest, everything reports "cached once, out of once" — a number that is both meaningless and reassuring. Cross-build identity has to be the normalised argv instead, and the normaliser has to stay narrow (session token, image tag, run name, nothing else) or it starts averaging genuinely different operations together.

dag.pending is not a cache miss. This is the one that would have put a wrong number on a dashboard. Dagger's DAG is lazy: an operation span is emitted when the node is declared, and the work is attributed to whatever forces it — which is why uv run pytest appears as a 0.19ms span while the 2.7s of running the tests sits on the Container.stdout beside it. Over 45 real build traces:

ran 525   cached 60   pending 180

uv sync --locked        cached 24x, pending 11x
uv run ruff check .     pending 32x, never resolved

The same operation appears in both states, so pending is the absence of a verdict, not a negative one. And there is no later version to prefer: Tempo holds one span per id, and 0 of 45 traces contained a duplicate span id, so Dagger's live update to the span never reaches the store.

Counting pending as a miss would have understated every cache ratio on the platform by roughly a quarter. It is counted as unknown, excluded from the ratio, and displayed beside every ratio it was excluded from.

The consequence, stated because it limits the feature. The commands a developer cares about — ruff check, pytest, rspec — are exactly the ones that are always pending. So these ratios describe container setup and dependency installation, not test commands. That is still the half worth measuring, because dependency installs are what caching is for, but it is not "percent of the build cached" and nothing in the UI says it is.

What it appeared to find, and why that was wrong. The first version of this page reported that ghcr.io/astral-sh/uv:0.5.29 and python:3.12-slim "cache 0 times in 25" and concluded that every build re-pulls its base images. That conclusion was wrong, and the way it was wrong is the more useful finding.

Container.from never carries dag.cached because resolving a tag to a digest is a registry round-trip Dagger does not cache-mark. The layers are an entirely separate question, and the registry mirrors answer it directly: over the window of a build, openchoreo-dockerhub-cache logs 14 HEAD requests for library/python:3.12-slim manifests and not one blob GET, and the ghcr mirror the same for astral-sh/uv. Nothing is re-pulled. The 0.2-0.4s Container.from spends is the manifest HEAD.

The lesson is about the shape of the mistake: a cache ratio computed over a denominator that includes operations which cannot report a hit reads as a finding, and it is an artefact. Container.from belonged in neither column.

The Python module was then read to check whether anything else was wrong, and it is already doing the right things — one shared base() across lint, audit and test; the uv cache volume keyed by a digest of uv.lock; only pyproject.toml and uv.lock mounted before uv sync, so a code change does not invalidate the dependency layer. There was no caching bug to fix.

A fallback order that looks like a fact

The project pipeline grid rendered production | staging — the promotion path apparently running backwards. The DeploymentPipeline is correct and says staging -> production; it simply was not read, because it is fetched with the viewer's OpenChoreo token and an anonymous visitor has none. With no edges the column order falls back to sorted(), and sorted() puts production first.

Worth recording as a shape rather than a bug: a fallback that produces plausible output is more dangerous than one that fails, because a grid with its headers in the wrong order is indistinguishable from one with them in the right order. The fix is not to remove the fallback but to make the page say which of the two it is showing.

Private repositories, and the two things that made the first attempt fail

Making the twelve application repositories private is a four-part change, and three of the parts are invisible until something breaks.

The credential has to land before the visibility flips. checkout-source mounts its git secret with optional: true, so with the repos public it logged Authentication: none (public repository) and worked. The moment they are private, an absent credential fails at git clone — so the SecretReference, the OpenBao entry and the secretRef on every Component go first, and are verified by a build that still had a public repo to fall back on.

allowInsecureAuth is the part that is easy to miss. With secretRef set and nothing else, the checkout printed:

>> Authentication: Basic Authentication
fatal: could not read Username for 'http://host.k3d.internal:3001'

Both lines are true and together they are misleading. The credential is only applied to https:// URLs unless allowInsecureAuth is set, and every repository here is http:// on the Docker bridge. The error names neither authentication nor the flag.

The manifest is the source of truth, not the live Component. The rollout was done with kubectl patch to test it, but manifest sync applies whatever the repo holds — so a cluster-only patch is reverted by the next push, breaking the very build that push triggers. Both fields are in each app's openchoreo/component*.yaml. apps/bakery has no local checkout, so its manifest was edited through the Gitea API rather than left behind.

platform-eng/dagger-modules cannot be private, and this was measured. Flipping it and pushing gave:

>> fetching platform CI module: http://host.k3d.internal:3001/platform-eng/dagger-modules.git#main
curl: (22) The requested URL returned error: 404

That fetch is dagger call -m <url>, not checkout-source, so it never sees the git secret. Reverted, and a build confirmed green again afterwards. The same applies to platform-eng/platform, openchoreo-config and templates: they are consumed by the build machinery over plain git and would need their own credential path. The application code is private; the platform's own modules are not, and that is a real remaining gap rather than a decision.

A new way for a reboot to break everything. OpenBao is in-memory here, so after a reboot default/generic/gitea-build is gone and every build fails to clone with an error that names neither the token nor the store. Before this change the same missing secret was invisible, because the clone fell back to anonymous. platform/bootstrap/restore-shared-secrets.sh now checks and restores it — verified by deleting the path and restoring it, not by reading the code.

One unrelated thing this surfaced. bench-stock could not be patched at all: it predates openchoreo-require-owner-label and carries no app.openchoreo.dev/owner, so the admission policy refused every update. A resource can sit in violation of a policy indefinitely as long as nobody edits it, and then the policy surfaces as an unrelated failure. Labelled.

"Is this up to date with the platform?" was unanswerable, and now is not

The question sounds like a lookup and was not one. Kubernetes records no modification time for a ClusterResourceType, metadata.generation counts edits but is not exposed on anything the type renders, and nothing recorded which version of a type produced a given consumer. There was no fact to read.

Two changes make it a fact rather than an inference:

Verified end to end: versioning postgres at 1.1.0 produced inkwell-db-5c8fc65f5 carrying the stamp, and all three inkwell-db bindings correctly reported BEHIND against it while continuing to serve.

That a platform change does not re-render existing consumers is correct, and is the point. A database should not be rebuilt because a template moved. The drift is real, it is now visible, and moving a consumer onto a new release stays a deliberate act.

One caveat worth keeping: the stamp only reaches objects rendered after the type was versioned. Everything rendered before it reads (none), which the console shows as unknown rather than as behind — an absence of evidence, not evidence of drift.

Choosing not to ship dagre

The dependency graph needed a layout, and dagre and d3-dag were the obvious candidates. Measured rather than assumed:
dagre (minified) 93 KB
every script the console ships today 63 KB (htmx 50, spans 10, flame 3)
d3-dag larger, and needs d3 beside it

The graph is a few dozen nodes with a layering the domain already implies — construct, instance, project. No edge routing around obstacles, no compound nodes, no cycles. That is the easy case, and doing it server-side in ~120 lines buys three things worth more here than polish:

The honest cost: edges are cubic curves with no obstacle avoidance and will occasionally pass near an unrelated node. Cosmetic, against a dependency four times the size of the app's own scripts.

The formatter and the linter disagreed, and CI found it

The lint stage failed on a line that ruff format produces and ruff check rejects: an SVG path f-string is 107 characters joined, over the 100-character limit. Splitting it into two named parts satisfies both.

The process lesson is the useful half. Running ruff format locally rewrites the file and reports success; CI runs ruff format --check, which fails on exactly the rewrite the formatter wanted to make. Running the mutating command locally hides the failure the pipeline will hit. The local gate is now --check, the same command CI runs.

Caching: the question is not age, it is whether it can still change

A finished build's trace, logs and Dagger verdicts are immutable and were being re-fetched on every render — the build page reads its trace to draw the pipeline, again for the cache figures, and again for the span viewer's JSON.

The previous fix was @lru_cache keyed on int(time.time() // 60), which works and has two faults: it keeps every minute's entry forever, and it gives a finished build exactly the same lifetime as a running one — both far too long and much too short.

So settled() and fresh() are separate calls and the caller must say which it means. A bug found while writing the tests, and the one direction this must never fail in: the first _put took a TTL and treated ttl <= 0 as forever, so fresh(..., ttl=0) — which reads as "do not cache" — cached permanently. Expiry is now an absolute time and a non-positive TTL means do not cache.

Nothing user-scoped is cached: every key is global to the process, so caching an ocapi response fetched with one viewer's token would serve it to the next. A test asserts _cache never appears in ocapi.py.

JSONResponse does not run FastAPI's encoder

Found by calling a new endpoint against a binding that does not exist and getting Internal Server Error.

JSONResponse(content) serialises with json.dumps directly. It never runs jsonable_encoder, which is what handles dataclasses — so a dict carrying an Unavailable raises inside the response and the caller gets a bare 500 with no logged cause. /api/deploys/{b}/events and /api/deploys/{b}/logs had done this since they were written, and they fail for any binding the platform API refuses — precisely the moment someone is clicking those buttons.

The conversion happens only at the JSON boundary, and that restriction is the finding rather than a detail. Unavailable is deliberately falsy and templates depend on it; a dict is truthy. Converting everywhere would silently flip every `

{% if not rows %}

` in the codebase — trading a 500 for a class of wrong pages, which is worse.

Two habits this argues for: a route returning JSONResponse is not covered by the same encoder as one returning a plain dict, and "returns 200 anonymously" is not a test of an endpoint whose interesting behaviour needs a token.

The container could not resolve the platform index, and the error blamed DNS

uv lock inside the build container failed with:

Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name does not resolve

Every obvious reading of that is wrong. The host resolves pypi.org fine. A fresh container resolves pypi.org fine too. --network host does not help, and neither does --dns 1.1.1.1.

The name that does not resolve is not pypi. pyproject.toml declares a second index for platform packages:

[[tool.uv.index]]
name = "platform"
url  = "http://host.k3d.internal:3001/api/packages/platform-eng/pypi/simple"

host.k3d.internal is wired up by k3d inside the cluster and inside the build pods that consume it. In a plain docker run it exists nowhere, so no nameserver can help — which is why pointing at a different DNS server changed nothing. --add-host host.k3d.internal:host-gateway maps it to the Docker host, which is where Gitea actually serves that index. Verified: the index answers HTTP 200 from inside the container with that one flag and no other change.

The trap is that it almost never fails. uv sync --locked works without the flag, because the lockfile pins exact artifacts and the cache already holds them. The failure appears only when uv has to resolve — that is, when you add or change a dependency — which is exactly when you are least expecting a network problem and most likely to conclude the dependency itself is the issue.

platform/dev/uv.sh now wraps this for any app in the repo, and runs uv as the invoking user: the first version wrote a root-owned uv.lock into the working tree, which is its own small trap for whoever edits it next.

A list endpoint returns 200 and an empty page when you may not read it

GetComponentRelease has an ErrForbidden branch and answers 403. ListComponentReleases has no such branch: the service filters the list by entitlement and the handler returns whatever survives. So a caller with no matching ClusterAuthzRoleBinding gets

HTTP 200  {"items": [], "pagination": {}}

which is byte-identical to the answer for a platform that genuinely holds no releases. Measured on this cluster with one machine token (openchoreo-backstage-client, sub matched by no binding), against the same namespace at the same moment:

collection status items actually on the cluster
projects 200 8 8
components 200 21 21
componentreleases 200 0 720
releasebindings 200 0 34

Note that it is per-collection, so "I can read some things" is not evidence that the caller can read the rest — the partial success is what makes this hard to spot. And the failure is not detectable by the client at all: there is no header, no warning, and no count to compare against.

What it costs. Any UI over these endpoints has to treat "empty" as ambiguous, and cannot say "this component has no releases" from a 200 alone. This console's whole _http.soft / Unavailable discipline exists to keep an unreadable panel distinguishable from an empty one, and this defeats it at the one layer below where the distinction is made: the read succeeded.

Workaround. None that is honest. GET a single object by name to distinguish 403 from 404 when you have a name to try; otherwise word the empty state so it does not assert absence. Upstream, a list response either wants a 403 when entitlement removed everything, or a field saying how many objects were filtered out.

The Rails memory leak was the profiler, and a controlled experiment says so

Two Solid Queue workers in different projects on the same Rails + Solid Queue + Pyroscope stack — inkwell-worker and bakery-worker — both grew ~45 MiB/hour with a rising floor, hit their 768Mi limit and were OOMKilled, on a loop. Web pods on the identical gem stack leaked ~3 MiB/h; the Python services leaked ~0. The stranded-span bug fixed in platform_telemetry 0.2.0 was ruled out on arithmetic: ~2,000 spans/hour at ~700 bytes each is ~1.4 MB/hour against an observed 46, more than thirty times short.

So on 2026-09-06 04:51 UTC inkwell-worker in production had traitEnvironmentConfigs.profiling.enabled set to false — which gates BOTH the Pyroscope agent and PlatformTelemetry::SpanProfiler, so it tests the whole profiling path with one switch — and bakery-worker was left untouched as the control. Read 37 hours later:

before the switch after
inkwell-worker memory +47.0 MiB/h +1.9 MiB/h
inkwell-worker CPU 0.020 cores 0.0055 cores
inkwell-worker OOMKills on a loop, never surviving a day 0 restarts in 37.4h, flat at 321 MiB
bakery-worker (control, untouched) +44.5 MiB/h +42 to +65 MiB/h, 4 OOMKills in 24h

The profiling path is the leak. Not "correlated with" — the control was running the whole time and is still dying.

The confound was checked and closed. A worker that leaks less because it is doing less work would show the same memory curve, and the CPU drop above is exactly what that would look like. It is not that: inkwell-worker's log volume is 54 lines/hour before the switch and 58 after. Same traffic, same job mix, same component — only the profiler removed. The CPU fell because a 100Hz sampling profiler and a span processor were most of what a mostly-idle worker was doing.

Which half — the Pyroscope agent or the span processor — is not yet known, and that is the next experiment: they are separable, and PYROSCOPE_ENABLED gates both together only because the initializer was written that way.

What this costs, immediately. inkwell-worker-profiled was built to profile live traffic harder — Vernier and StackProf around every job, at a raised sample rate. On this evidence it will leak at least as fast as the thing it is diagnosing, in a production cell that already holds ~2.7 Gi of a 6 Gi quota on a host with ~5 GiB free. It is safe in staging and it should not be promoted to production until the leak is attributed. The component and its bindings are in the repository; the promotion is deliberately not done.

Pyroscope was not OOMing, and the collector that starved it was

Reported as "the Pyroscope OOM", and it was neither Pyroscope's nor an OOM. The chain is worth writing down because every link looked like the cause of the one before it.

Pyroscope's memory was never the problem. Over six hours it held 84–109 MiB against a 1 GiB limit — a tenth of its ceiling — and used 0.03 cores. The exit=137 on its container is real and is not a kernel OOM kill: the pod's lastState says reason: Error, not OOMKilled, and the events say

Liveness probe failed: Get "http://10.42.0.71:4040/ready":
  context deadline exceeded (Client.Timeout exceeded while awaiting headers)

A timeout, not a refusal. /ready answers in 2–4 ms when the host is idle; the probe allows 5 s and failed ten times in a row.

The liveness probe is not the bug either. It was added deliberately (26-lgtm.sh) because Pyroscope's single-node Raft metastore can lose leadership and never re-elect, and that wedge cost 38 hours once and 17 more the second time. It did its job here: the previous container's log ends with

raft: failed to commit logs: "non-monotonic log entries:
  tried to append index 790757 after 790757"
raft: entering follower state ... raft_state=Candidate ... Follower

What wedged the Raft log was starvation. Six minutes earlier the process could not service its own loopback gRPC — metastore client error … context deadline exceeded against 10.42.0.71:9095, its own address — segment writes failed, and profiles were rejected with ingestion_delay=11.9s. A process that answers in 2 ms was 12 seconds behind itself.

And the starvation was one collector. The data-plane Alloy went from its usual 0.07 cores to 2.2 at 23:28, climbed to 10.4, and held 9–10.5 of this machine's 12 cores for ninety minutes, ending when it was OOMKilled at 01:05. Per-pod, over the same six hours:

alloy-z9k56 (data-plane)   0.04 -> 10.36 cores
beyla-xbcxj                0.03 ->  2.81
everything else in grafana-lgtm     ~0.1

It was emphatically not the builds, which is where suspicion naturally falls on this host: the Dagger engine peaked at 0.39 cores over the same window and the build pods at 0.04.

The fix, and what it does not fix. lgtm-alloy.yaml now sets limits: {cpu: "2", memory: 1Gi}. Two cores is roughly twenty times steady state and still leaves ten; 1 GiB replaces a 512 MiB ceiling that had OOMKilled the data-plane Alloy eight times while it carried four signals for the busiest plane — the same under-provisioning raise_agent_memory already corrected for the PrometheusAgent one layer along ("it OOMKilled 23 times before this was raised"). Verified at the cgroup: cpu.max on the container reads 200000 100000.

The ceiling is a blast radius, not a cure. Two things remain open, and both are Alloy's rather than Pyroscope's:

Two repositories declared one Project, and the last push won

Found by opening a real pull request, which is the only way it could have been found: the review-environment path had never been exercised against a live PR, and docs/FEEDBACK.md recorded the PR lane as designed-but-unverified for exactly that reason.

apps/brochure and apps/bakery both ship an openchoreo/project.yaml for the same Project, showcase. Manifest sync applies whatever the pushed repo holds, so the last push wins. brochure corrected its copy on 2026-09-06 ("The application pipeline is called applications, not default"); bakery's copy still said default, and bakery's next push — 11:36:45 on 2026-09-07 — put the broken value back.

There is no DeploymentPipeline named default on this platform. The two that exist are applications and platform-services.

The Project still reconciled Ready: True / Reconciled. Nothing at the project level says the reference is dangling. What breaks is one level down:

bakery-docs    Ready False  DeploymentPipelineNotFound
bakery-web     Ready False  DeploymentPipelineNotFound
bakery-worker  Ready False  DeploymentPipelineNotFound
brochure       Ready False  DeploymentPipelineNotFound   since 11:36:53
brochure-pr-4  Ready False  DeploymentPipelineNotFound

No new ComponentRelease is cut for any of them, while the releases already deployed carry on serving. So the project looks healthy, the sites stay up, and every push silently stops producing a release — for fifteen hours, across five components, with nothing anywhere saying so.

showcase was the only project affected: every other one names applications or platform-services, and both exist.

Fixed in apps/bakery/openchoreo/project.yaml, with the reason written beside it so the next person to copy that file does not reintroduce it. Within seconds of the sync all five went Ready: True / ComponentReleaseReady and brochure-pr-4-6d4fd494f5 was cut.

What to take from it. Two repositories owning one Project is legal and silent, and the failure it produces is invisible at the level where the mistake was made. If a second repository must declare a shared Project, the copies have to agree — and the cheap check is that a Project naming a DeploymentPipeline that does not exist should not report Ready.

The capacity number was memory, and the constraint was CPU requests

The same pull request produced its review environment and nothing in it could schedule:

r-bakery-db-review-pr-4-...        Pending
brochure-pr-4-review-pr-4-...      Pending
0/1 nodes are available: 1 Insufficient cpu

Meanwhile the console's /pull-requests page said "4.6 GB free", from project-flow's /host, which reported memory and nothing else. Both statements were true. Memory was at 53% of requests; CPU requests were at 99% — 11915m of 12000m allocatable — and requests are what the scheduler compares against.

The expression that reproduces kubectl describe node exactly, which took three attempts:

sum(kube_pod_container_resource_requests{resource="cpu",cluster="data-plane"}
    and on(namespace,pod) (kube_pod_info{cluster="data-plane",node!=""})
    unless on(namespace,pod)
      (kube_pod_status_phase{cluster="data-plane",phase=~"Succeeded|Failed"}==1))

returns 11.915 against kubectl's 11915m (99%). The obvious simpler forms do not. Phase Running alone gives 11.515, because a scheduled pod that is still initialising is Pending and its requests are held. Running|Pending gives 13.015, because it also counts the pods that are Pending precisely because they did not fit — adding their requests to the total is counting the thing you are trying to explain.

/host now reports cpu_requested_cores and cpu_allocatable_cores beside available_bytes, and the console states both, saying plainly when the node is out of requests that a new environment will sit Pending however much memory is free.

A review binding created 52 seconds after its teardown, and nothing collected it

Found while verifying that closing the pull request tore the review environment down — it did, cleanly, and one object from an earlier pull request was still sitting there:

releasebinding/ledger-pr-1-review-pr-1   owned=true  pr=1   3h59m
  state: Active
  Ready: False  ComponentReleaseNotFound: "ledger-pr-1-7dfb4d4b98" not found

Its Component, its Environment and its ComponentRelease were all gone. It had no deletionTimestamp — nothing was trying to delete it.

The timestamps name the race exactly. apps/ledger #1 was merged and closed at 22:59:46; the binding was created at 23:00:38, fifty-two seconds after the teardown that was supposed to have removed everything.

teardown_pr() deletes by label selector and then waits for the finalizers. What it did not touch was STATE["pending_reviews"] — one in-memory entry per forked component waiting for autoDeploy to cut a release. The reconcile loop picks those up on its next tick and calls ensure_binding(comp, project, review_env_name(pr), ...). So a build still in flight when the pull request closed produced a binding into an environment that had just been deleted, after the sweep that would have caught it had already finished.

Fixed with two guards, because they fail differently:

  1. teardown_pr() drops that pull request's pending_reviews entries first, before deleting anything, so a slow finalizer cannot let the loop tick in between.
  2. The loop refuses to bind into an Environment that does not exist and drops its own entry. Teardown is not the only way an environment goes away — the walkthrough's teardown() and a person with kubectl reach the same end state, and neither touches this process's memory.

The orphan itself was deleted by hand; it had no pods, because the namespace it named was gone.

openchoreo-config is a real submodule, and the mirror it replaced was a trap

platform/openchoreo-config was a write-only mirror: sync-to-gitea.sh pushed the working tree to platform-eng/openchoreo-config as a fresh commit and nothing ever read it back. The loop does, per target:

find "$clone" -mindepth 1 -maxdepth 1 -not -name .git -exec rm -rf {} +

then rsyncs over the top. So a pull request against that repository merged, looked finished, and was deleted by the next --apply — the worst available failure, because the loss is silent and the author believes the work landed.

upgrade-watch already knew this and refused to offer the button (MIRRORED, Finding.openable), which was the right call and left every pin in that directory with no route but "edit the laptop's checkout".

Measured after the conversion rather than quoted from the brief. NEXT-SESSION.md said "15 of the 23 actionable upgrades", which was true of an older scan and is not true now — the app-level pins added on 2026-09-04 grew the population and shifted the ratio. The live scan says:

platform-eng/platform            actionable  9   openable 0   (still a mirror)
platform-eng/dagger-modules      actionable  7   openable 7
platform-eng/openchoreo-config   actionable  6   openable 6   <- was 0
apps/*                           actionable 15   openable 15
TOTAL                            actionable 38   openable 28  (was 22)

So the conversion moved six findings, not fifteen, and the six are the ones that had no other route:

garage                v1.0.1                       -> v2.4.0
redis                 8.2-alpine                   -> 8.10-alpine
postgres-legacy       16-alpine                    -> 18-alpine
adminer               4.8.1-standalone             -> 6.0.1-standalone
alpine-git            v2.52.0                      -> v2.54.0
artifact-store-minio  RELEASE.2024-12-18T13-15-44Z -> RELEASE.2025-09-07T16-13-09Z

platform keeps the largest remaining block at nine, and converting it is the same shape of change for a bigger blast radius.

Converted on 2026-09-08. It is now a submodule at the same path, with Gitea as its origin, exactly like platform/dagger-modules and every apps/*.

The path mapping did not have to change, which is what made this cheap: the mirror already collapsed that directory to the repository root, so platform/openchoreo-config/resource-types/redis.yaml resolved to resource-types/redis.yaml before and resolves to the same thing now. Every bootstrap script that names $DEMO_ROOT/platform/openchoreo-config/... is untouched, because a submodule sits at the path it replaces.

Both halves are required and the self-test enforces it. Removing it from MIRRORED without removing it from sync-to-gitea.sh's targets would re-arm the trap. watcher.py --self-test asserted openchoreo-config is a write-only mirror and failed on exactly this commit — which is what that assertion is for. It now asserts the opposite, plus that platform still is one, plus that a resource-types/*.yaml pin is openable.

What changed for a person running the platform, written into 40-platform-config.sh where they will be standing:

git submodule update --remote platform/openchoreo-config
platform/bootstrap/40-platform-config.sh

A change made in Gitea is not in the checkout until it is fetched, and a change made in the checkout is not in Gitea until it is committed and pushed from inside the submodule and the pointer bumped. component-types/generate.sh writes into the submodule's working tree, so its output is committed there.

platform and templates stay mirrors, deliberately. templates is read at RUNTIME by project-flow to scaffold projects, so it genuinely is generated from here; platform holds bootstrap, docs and infra, where the laptop is still the source of truth. Between them they carry two pins — versions.env and one k3d config — so converting them buys much less for the same care.

Verified end to end, on the first pull request that repository has ever had

Not argued — done, through the console's own button path (POST /git/commit then POST /git/pr on project-flow, HMAC-signed, exactly what actions.py:open_pr sends):

platform-eng/openchoreo-config #1
  alpine-git: v2.52.0 → v2.54.0   workflows/checkout-source.yaml
  merged 29ae8ff

Then the thing that used to destroy it:

platform/sync-to-gitea.sh --apply
  ==> platform-eng/templates   in sync
  ==> platform-eng/platform    pushed
  (openchoreo-config is not a target any more)

GET .../openchoreo-config/raw/workflows/checkout-source.yaml
  32:        image: alpine/git:v2.54.0        <- still there

Before the conversion that same sequence would have left v2.52.0, silently.

Carried the rest of the way so the platform actually runs it:

git submodule update --remote platform/openchoreo-config
kubectl --context k3d-openchoreo-wp apply -f .../workflows/checkout-source.yaml
kubectl get clusterworkflowtemplate checkout-source -o yaml | grep alpine/git
  alpine/git:v2.54.0

The comment that documented a fix deleted the fix

Caught by re-running 28-beyla.sh for an unrelated memory change, an hour after the duplicate-sample fix was committed and verified.

The exclude_instrument block was added, applied, and verified live: port 9800's series went stale, port 80's kept flowing, the 400s stopped for 28 consecutive checks. Then the long explanatory comment was written into the same file with a Python replace over the region between the block's opening comment and the next one — and the replacement text contained the comment and not the two lines of YAML underneath it. yaml.safe_load was run afterwards and passed, because a file with the key removed is still valid YAML. Nothing else re-read the file, so the running config kept the exclusion and the repository lost it.

committed:  the comment explaining exclude_instrument, and no exclude_instrument
running:    exclude_instrument, applied before the comment was written

It survived undetected until the next helm upgrade from that file, which silently put the duplicate back.

Restored, and the checks that would have caught it, in the order they should have run:

The general shape: a large comment written by a script that replaces a region can eat the code it describes, and every downstream check — the YAML parse, the running system, the metrics — agrees that everything is fine.

Cgroup OOM and node OOM are different failures, and the log line says which

Beyla was OOMKilled again after its limit went 512Mi -> 1Gi, and the fix was written up as "the request was the worse half": 192Mi against an 800 MiB working set, the kubelet deriving oom_score_adj from the request, so the container is the kernel's preferred victim. That is Tempo's story, told about Beyla, and it is wrong here.

journalctl | grep -c constraint=CONSTRAINT_MEMCG   ->  10
journalctl | grep -c constraint=CONSTRAINT_NONE    ->   0

Every kill in eight hours was CONSTRAINT_MEMCG, with oom_memcg=.../kubepods/burstable/pod<uid>/<container> — the container hitting its own limit, where oom_score_adj plays no part at all. The score only decides anything when the NODE is out of memory and the kernel has to choose a victim, and that has not happened once. Tempo's kills were node-level — 801Mi resident against a 3500Mi ceiling — which is exactly why the argument is right there and reaching for it here was a mistake.

The effect is small even where it applies. The kubelet's formula for a burstable pod is

oom_score_adj = 1000 - 1000 * memory_request / node_allocatable

so on this 32 GiB node, 192Mi -> 768Mi moves the score from 995 to 977. Confirmed at /proc/<pid>/oom_score_adj inside the node container after the roll, and against the kernel's own log line, which printed oom_score_adj=995 at the moment of each kill. Eighteen points of a thousand.

So: the limit is what fixes the OOM. The request is what stops the scheduler believing the pod is a tenth of its real size. Both were worth changing; only one of them was the fix, and the write-up said the other.

The check to run first, before any reasoning about requests, limits or victims:

journalctl --since "8 hours ago" | grep -E "constraint=CONSTRAINT_(MEMCG|NONE)"

MEMCG means raise the limit or use less. NONE means the node is genuinely out and the request — and everything else competing for it — is the argument.

An alert on a sticky gauge fires for ever, and re-fires when its exporter restarts

PodOOMKilled was firing for zulip-0. zulip was OOMKilled on 2026-09-06 and has been healthy since; the alert claimed to have been active since 01:59 on 2026-09-08, which is when kube-state-metrics last restarted.

Both halves of that are the same bug. The rule alerted directly on

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0

which reports the last reason and keeps reporting it until the container terminates for a different one. So the gauge reads 1 for ever after one kill, the alert never resolves, and when the exporter restarts the series is recreated and the for: clock starts again — a two-day-old kill presented as a live incident.

Measured over 24 hours, old expression against new:

zulip-0                   289 of 288 samples  ->    0   killed two days ago
bakery-worker-production  273                 ->    5
bakery-worker-staging     272                 ->    5
alloy-z9k56               252                 ->    6
beyla-xbcxj               263                 ->   18
inkwell-worker-profiled     9                 ->    3
inkwell-worker-staging      -                 ->    1

Seven series still fire; the one that stops is the one with nothing to say. This is the same alert without the latch, not a narrower one.

The restart counter is the event; the reason gauge is the filter.

(increase(kube_pod_container_status_restarts_total[15m]) > 0)
and on (namespace, pod, container)
(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1)

increase(...[15m]) > 0 is true for fifteen minutes after a restart actually happens and then goes away by itself, which is what makes the alert resolve. and, not *, because nothing from the right-hand side needs carrying. A restart for some other reason after an OOM correctly does not match — the reason gauge has moved on by then.

The general shape, stated more carefully than the first draft of this note. The distinction is not "kube-state-metrics gauges are unreliable" — most of them describe the CURRENT state and clear on their own. kube_pod_status_ready and kube_pod_status_phase are current state, which is why PodNotReady in this same file alerts on them directly and resolves correctly.

last_terminated_reason is different in kind: it describes a past event and has no way to clear, because there is no such thing as "not recently terminated". A series like that can be a filter and must not be the subject. The test is whether the metric can go back down on its own — if it cannot, gate it on something that can.

Confirmed after the reload: 0 alerts firing, from 2.

deriv() over a sawtooth returns nonsense, and it returned it confidently

Asking whether the profiled worker leaks in production, the obvious query gave answers that were not merely imprecise but sign-inverted:

deriv(container_memory_working_set_bytes{...}[8h]) * 3600 / 1024/1024

inkwell-worker-profiled-production   -11.5 MiB/h     <- actually climbing
inkwell-worker-production            +55.5 MiB/h     <- actually flat

Plotting the same series at thirty-minute resolution says the opposite of both:

profiled (profiling ON)   410 450 476 496 519 543 | 302 352 379 406 | 295 … 459
worker   (profiling OFF)  277 294 307 316 300 316 314 304 302 304 307 307

deriv fits a least-squares line through the window. Over a sawtooth — a process that climbs and then sheds — the line it fits depends entirely on where the sheds fall relative to the window edges, and can slope either way regardless of the trend between them. Neither pod restarted; both are single, continuous series. The number was not noisy, it was wrong, and it was wrong with the same confidence a right one would have had.

Use deriv/predict_linear only on a series you have already looked at and know to be monotonic between resets. For leak questions, plot it, or measure peak-to-peak between sheds. Everything the experiment concluded still holds — profiling ON climbs 26–39 MiB/h, profiling OFF is flat at about +4 — and the first pass of this check said the reverse.

Same family as the Grafana format:table trap: a query that answers, and answers wrongly, is more expensive than one that fails.

The leak experiment had no second arm, because one variable gated two things

The controlled experiment finished on 2026-09-07 with a clear answer — turning profiling off takes a Rails worker from +47.0 to +1.9 MiB/h with its traffic unchanged — and an open question it could not touch: profiling is two things, and only one switch existed.

config/initializers/pyroscope.rb       the sampling agent, a thread in the process
config/initializers/opentelemetry.rb   PlatformTelemetry::SpanProfiler, a span processor

Both read PYROSCOPE_ENABLED. Not a decision — an accident of how the two initializers were written, and the same accident copied into every Python service, each of which constructs NestedSpanProfiler() inside its own PYROSCOPE_ENABLED guard.

spanProfiles is the missing arm. continuous-profiling 2.2.0 adds it as an environment config defaulting to true, ANDed with enabled in the patch so there is no state where the processor runs without the agent it depends on:

enabled=false                     both halves off, as before
enabled=true                      both halves on, as before
enabled=true, spanProfiles=false  agent only — the comparison that was missing

A MINOR, so an edit of the object rather than a v3, and every consumer that ignores it is unaffected.

Two implementations, one of them in the right place. Ruby reads the variable in inkwell's own initializer, because that is where the Ruby caller is. Python reads it in platform_telemetry.profiler: NestedSpanProfiler() returns an inert but valid SpanProcessor when the switch is off, rather than asking six services to check. That is this package's whole reason for existing — the same argument that put the boolean spellings in one tuple after five of six services ignored PYROSCOPE_SAMPLE_RATE — and returning a real processor rather than None means no caller has to handle a value it was not written to expect. Published as platform-telemetry 0.6.5.

It is in place and it is not yet live, which is the honest state. A ComponentRelease freezes the type it rendered from, so the new environment variable appears on the next release cut for a component and not before; the running staging pods still show PYROSCOPE_SPAN_PROFILES unset, and unset means on. The next inkwell build carries it into staging by autoDeploy, and production only on a promotion.

The arm is running, from 04:54 on 2026-09-08. On inkwell-worker in staging, not on inkwell-worker-profiled: the plain worker runs exactly the platform's two halves and nothing else, so a null result is unambiguous. The profiled worker adds Vernier and StackProf around every job — constant across a before/after, so it would also work, but "no change" there would leave a second profiler in the frame with no way to rule it out from the numbers.

inkwell-worker-staging            ENABLED=true  SPAN_PROFILES=false   <- the arm
inkwell-worker-profiled-staging   ENABLED=true  SPAN_PROFILES=true    <- reference
inkwell-worker-production         ENABLED=false                       <- both off

The baseline, taken immediately before, working set in MiB every thirty minutes with both halves on:

538 563 584 609 658 | 356 384 409 431 |

Two sawtooth ramps at roughly +45 MiB/h, each ending in a reset. That is the number the arm has to move, and it is written into apps/inkwell/openchoreo/bindings-staging.yaml beside the switch, because an experiment whose "before" is not recorded is one nobody can check.

Reading it: if the ramps flatten to the +1.9 MiB/h the first arm produced, the leak is PlatformTelemetry::SpanProfiler. If they continue at ~+45, it is the Pyroscope agent. Plot the series — deriv returned both signs backwards on this exact measurement earlier today.

Take it down when it answers. The binding document says so: an experiment left running is a configuration nobody can explain.

An object nothing could collect, because the guard was written for the wrong place

2026-09-08. ledger-pr-1-staging had been Terminating for fourteen hours and the console was drawing it as an inkwell deployment in staging.

What the cluster said:
created 2026-09-07T22:51:34Z
deletionTimestamp 2026-09-07T22:52:04Z, thirty seconds later
finalizers openchoreo.dev/releasebinding-cleanup
labels none at all
Component, ComponentRelease, Deployment none existed

So reconcile() did its job. A finalizer caught the delete, and then nothing on the platform could free it. unwedge() accepts an object only if it carries project-flow.openchoreo.dev/owned=true or its openchoreo.dev/environment starts review-pr-. It had no labels, and it lived in staging — the ROOT environment, where autoDeploy insists on writing. teardown_pr's label selector could not see it either.

Beside it, RenderedRelease/ledger-pr-1-staging-observability, deleted one second later, held by openchoreo.dev/obsplane-cleanup, in exactly the same position for exactly the same reason.

The guard was written for objects inside a review environment. The side-effect objects it also has to collect are outside one by construction. Moving the pipeline root to development would not have fixed it — it would have moved the wedge.

Labels cannot carry this even in principle: apply() falls back to a full PUT on a 409, so a body assembled without them replaces them. The sweep added for this judges by shape — a *-pr-<n> subject whose Component does not exist is garbage wherever it lives and whatever it is labelled. On its first run it collected a third orphan nobody had found, brochure-pr-4-staging-observability.

project-flow's root environment was a constant; OpenChoreo's is per project

Found while fixing the above. ROOT_ENV was one value meaning "where autoDeploy writes, and therefore where a review environment forks from". OpenChoreo derives that per project, from the project's own DeploymentPipeline: the source environment that appears in no target list (rootEnvironment, internal/controller/component/controller.go).

This platform runs two pipelines — applications roots at staging, platform-services at development — so one constant agreed with OpenChoreo for half the projects and silently disagreed for the other half. For a chatops or devtools pull request, project-flow pre-claimed <component>-staging while OpenChoreo went on to create <component>-development: the claim missed, and the side effect the claim exists to prevent was left unowned and unlabelled, which is the shape of the object above.

A development tier does not fit, and here is the arithmetic

Asked for, measured, refused. development -> staging -> production for the four application projects means standing up a development cell for each. Their staging cells cost, in CPU requests: inkwell 2.55, showcase 1.10, chess 0.85, rideshare 0.20 — 4.70 cores. The data plane node sits at 11415m of 12000m allocatable: 0.585 free.

Over budget by a factor of eight, and the failure mode is not a warning — every pod would sit Pending on Insufficient cpu, which a review environment on this host has already hit once while a page nearby reported 4.6 GB of memory free.

The arithmetic is in the comment beside ROOT_ENV so the next person to want three environments finds it rather than the idea. root_env_for() already returns the first stage of a three-stage pipeline, with a test for that case, so the derivation survives the day the host grows.

Two renderings have been Terminating since 31 August, and are not pull requests

Found by the sweep above, which correctly left them alone. inkwell-web-staging-observability and inkwell-enrich-staging-observability have carried openchoreo.dev/obsplane-cleanup since 2026-08-31T23:49:05Z — eight days — with Finalizing: Cleaning up dataplane resources.

They are not review-environment objects: their components exist and are running, and inkwell-web-staging (the workload rendering beside them) is 30 hours old and healthy. So this is a different fault: the observability-plane cleanup for those two renderings has never completed. Nothing is broken for a user — the components serve — but the objects leak, and the sweep must not collect them because the shape test that makes it safe does not apply.

Open. Worth one debugging session on the obsplane cleanup path before anything is stripped by hand.

Opening one pull request builds the same commit twice — FIXED

2026-09-08, reproduced on two consecutive walkthrough runs. kubectl get workflowruns after a single @walkthrough static:

brochure-pr-1-run-4e215d28  commit=6a125015  created 17:21:30Z  ok=True
brochure-pr-1-run-7faea761  commit=6a125015  created 17:21:32Z  ok=True

Two builds, one component, one commit, two seconds apart. project-flow's own log shows both triggers:

17:21:30 PR #1: review build for edit-hero@6a125015: autobuild: forwarded
17:21:32 autobuild: forwarded, ['default/brochure-pr-1'] ... nothing further
         for a push to edit-hero

The pull_request webhook forks the component and triggers a review build. The push webhook for the same branch then arrives and autobuild forwards it to the component that has just been created -- so the branch that the pull request was opened FROM now matches a component that did not exist when it was pushed.

Three costs, in increasing order of how much they matter:

  1. Twice the build load, on a host that reached 99% of CPU requests during this very run.
  2. A redundant rollout. Both builds cut a release and both bind it: bound brochure-pr-1 into review-pr-1 at brochure-pr-1-654485b7c5, then ninety seconds later repointed brochure-pr-1-review-pr-1 -> brochure-pr-1-66757cbd6b. The review environment runs one image and is then moved to a byte-identical other one.
  3. It doubled the exposure to any flaky build, and the walkthrough bot followed the wrong one. On the first run, 1557e6b7 failed on a DNS lookup and 0f08a88c succeeded two seconds later. The bot reported "7 · Build failed" at 17:13:10 and stopped. At 17:14:09 project-flow bound the component into the review environment from the build that had succeeded. The platform did the right thing and the walkthrough said it had not.

The walkthrough CAN finish -- but only with room to spare

Corrected the same day. After the duplicate-build fix and a teardown that reclaimed CPU from 99% to 78%, @walkthrough static ran clean end to end in 585 seconds: scaffold, build, staging, soak, auto-promote to production, pull request, ONE review build, review environment Ready, merge, teardown of the review environment, rebuild of main, and promotion of the merged commit. Nothing was left terminating and no namespace leaked.

So the section below is not "the walkthrough is broken". It is "a review environment needs about 1.7 cores that this host does not always have", and the walkthrough is the thing that notices.

A review environment does not fit when the host is full

Not a regression, and not the bot's fault. A review environment is a whole project cell -- for showcase that is seven pods including a Postgres, asking 1.70 cores of CPU requests. Measured at the moment it was refused:

node allocated: cpu 11915m (99%) of 12000m   -> 0.085 cores free

Every pod of it sat Pending on 0/1 nodes are available: 1 Insufficient cpu, and the walkthrough waited at step 8 for a readiness that could never arrive.

This is the same ceiling that made a development -> staging -> production pipeline impossible earlier the same day, measured the same way. The walkthrough is not broken; the host is full. platform/capacity/hibernate.sh is the lever, and OC_MAX_REVIEW_ENVS=1 already assumes there is room for one.

A review environment leaks its observability namespace and its metric rule

Every teardown observed on 2026-09-08 left behind, on the observability plane, the dp-default-<project>-review-pr-<n>-<hash> namespace AND a working slo-memory ObservabilityAlertRule inside it. The teardown's label sweep is control-plane scoped; nothing collects the observability-plane side, and the RenderedRelease that could have is deleted first.

Distinct from the finalizer deadlock fixed the same day -- these objects are not wedged, nothing is waiting on them, and they are simply never collected. Four such namespaces had accumulated from closed pull requests before this was noticed, and two more were created and removed by hand during the checks.

Not reproduced on the clean run at 18:14Z, and that is not the same as fixed. That walkthrough's review environment existed for two and a half minutes -- created 18:18:40Z, torn down 18:21:00Z -- and the observability plane was left with no review-pr namespace at all. Nothing shipped that day claims to collect it, so the likeliest reading is that the namespace was never created inside so short a window rather than that it was collected. Worth re-checking against a review environment that lives long enough to be used.

The engine's DNS failure was not a flake, and not DNS

Earlier in this file the publish failure is recorded as buildkit's embedded resolver being intermittent under load, "a coin flip on every build". The first half is right about where it happens and wrong about why, and the second half is wrong outright. Measured from inside the running engine on 2026-09-08:

host.k3d.internal                       0 ok, 25 failed
kubernetes.default.svc.cluster.local   25 ok,  0 failed

Nothing intermittent about it. The engine could resolve the cluster fine and could never resolve that one name, so every explanation that turned on load -- one CoreDNS replica, a 170Mi limit, eight concurrent builds, load average 102 -- was explaining a symptom that was not there.

host.k3d.internal is not a DNS record. k3d writes it into /etc/hosts on the node so that containers on the node's network can reach the host. A pod does not read the node's hosts file, and no nameserver in the cluster has ever been asked to serve the name. Anything that reached it did so because its own hosts file had been given the entry, and the engine's had not.

That also explains why the earlier [dns] options = ["ndots:1"] fix verified perfectly and changed nothing: [dns] in engine.toml shapes the resolv.conf BuildKit writes for exec containers, and publish runs in the engine. The measurement proving 480 of 480 lookups was taken in the half of the system that was never failing.

The fix is one hostAliases entry on the engine's StatefulSet, mapping host.k3d.internal to the node's gateway address, with ndots:1 alongside it as a pod dnsConfig. It takes DNS out of the path rather than making DNS work. platform/bootstrap/30-dagger-engine.sh applies it after the Helm release, because the chart has no hostAliases field and a hand patch is what a helm upgrade reverts. Verified after the roll: 25 of 25, and a walkthrough build published on the first attempt.

The transferable part is the measurement, not the fix. "Intermittent" was an inference from two failures under load; one loop of 25 lookups from inside the failing process, next to a control lookup that was expected to succeed, told a different story in ten seconds and would have told it a week earlier.

The dedup that ate the build it was written to keep

Same day, and self-inflicted. A pull request opened by the walkthrough produced a review environment with no build at all -- brochure-pr-2 existed, was never built, and the walkthrough waited at step 7 for a readiness nothing was going to deliver.

The cause was the fix for the duplicate-build problem landed an hour earlier. Gitea sends two webhooks when a pull request opens: a push for the branch and a pull-request event. Both used to forward to autobuild, so both built. The dedup keyed on (repo, ref, sha) and suppressed the second arrival within 180 seconds -- correct when the two arms are interchangeable, and they are not.

The push arm forwards a plain branch build and legitimately affects zero components on a fork branch. The pull-request arm creates the *-pr-<n> component first and forwards that. Whichever arrives first records the key, so the arm that mattered was decided by webhook delivery order, and on this run the push won: autobuild: forwarded, [] component(s) affected, then the pull-request arm suppressed against its own key.

The fix is that the arms are not equal. forward_to_autobuild takes authoritative: bool, the pull-request path passes authoritative=True, and an authoritative call records the key and forwards regardless of what came before. The push arm still collapses against itself, which is the duplicate this was written to stop.

Verified on a live run at 18:18:41Z: the pull-request arm forwarded ['default/brochure-pr-1'], the push arriving mid-flight was refused with already forwarded ... 0s ago, and exactly one review build existed.

Two tests pin it -- one asserting the pull request survives either arrival order, one asserting two pushes at the same commit still collapse. Both were written after the failure, which is the honest order but not the useful one: a dedup key that spans two callers doing different work is worth a test before it ships, and the comment on forward_key() had already said so in words.

A promotion renders the app before its database dependency resolves

The rails walkthrough passed end to end on 2026-09-08 -- twelve minutes, every step green, "Ready in production", **Done**. Underneath it, both Rails components crashlooped four times each in production and nobody was told.

db-prepare failed with the socket fallback this file already documents:

PG::ConnectionBad: connection to server on socket
"/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory

and the existing entry for that message says to check the ResourceReleaseBinding, because the env var was never injected. Correct diagnosis, and here it points at a resource that is perfectly healthy. The first rendering of the production Deployment carried an init container with zero environment variables and a main container with twenty, none of them DATABASE_URL. A second rendering a minute later carried four and twenty-five.

The timing is the whole finding:

18:32:07  bakery-db-production ResourceReleaseBinding created
18:36:27  bakery-worker production ReplicaSet #1   <- 77s early, no env
18:37:00  bakery-web    production ReplicaSet #1   <- 44s early, no env
18:37:44  bakery-db-production goes Ready
18:37:50  bakery-worker production ReplicaSet #2   <- full env
18:38:02  bakery-web    production ReplicaSet #2   <- full env

The component was rendered and applied before the resource it depends on had resolved its outputs, so the dependency contributed nothing and the render went out anyway. The next reconcile after the binding went Ready produced the correct manifest and the rollout converged on its own.

Staging was clean, and that is not reassurance -- it is the same race won. Its database went Ready at 18:33:48 and its ReplicaSets were created at 18:34:05 and 18:34:09, seventeen seconds later. One ReplicaSet each, no restarts. Same code, same pipeline, opposite outcome, decided by which finished first.

Two things this changes:

The shape of the fix is the one already recorded for the vector extension race: gate on the dependency rather than on wall-clock luck, so the component is not rendered into an environment until the resource binding it consumes reports Ready.

FIXED the same day, and the machinery was already there. resolveResourceDependencies returns a PendingResourceDependency for a provider that is missing, not Ready, or short an output, and the reconcile guarded on it -- after the render instead of before, so the guard blocked ReleaseSynced on a manifest it had already applied. The condition was accurate the whole time and the wrong Deployment was in the cluster underneath it.

The guard was copied from the connections guard above it, whose own comment explains why it must run late: two components can each depend on the other's endpoint, so refusing to render until resolved would leave neither ever publishing one. Resource dependencies cannot form that cycle, and the code says so twenty lines from the bug -- the watch registration in SetupWithManager carries a comment stating a ResourceReleaseBinding is reconciled by its own controller and never consumes a ReleaseBinding. The late placement was inherited from a sibling that needed it, not chosen for this one.

Moving it before the render, as platform/openchoreo-patches/resource-dependency-gate.patch, on the same walkthrough rerun:

19:09:11  bakery-db-production goes Ready
19:09:29  bakery-web    production ReplicaSet   <- 18s AFTER, and the only one
19:09:32  bakery-worker production ReplicaSet   <- 21s AFTER, and the only one

One ReplicaSet per component instead of two, zero restarts instead of four, no Init:Error anywhere in the run, and the guard logged six refusals to render while the two databases bootstrapped. Both walkthrough URLs serve 200 and all 38 ReleaseBindings on the platform stayed Ready across the controller roll.

Two things worth keeping from the fix rather than the bug: