Credentials on this page are redacted. This is the public export; the real values live in docs/ACCESS.md in the private repository.

Observability

Logs, traces, metrics and profiles, and why one stack — OBSERVABILITY.md

Observability

One stack. Loki, Tempo, Prometheus, Pyroscope and Beyla, with Grafana as the UI. OpenChoreo's OpenSearch-backed logging and tracing has been removed — this document records what that cost and how to get it back if you need it.

Why one stack

Running both was deliberate at first: it was the only honest way to see what OpenChoreo's own observability gives you. Having seen it, keeping both cost about 4GB of RAM — three OpenSearch masters at 900Mi, a 1Gi data node, dashboards, two adapters, an operator, and a fluent-bit DaemonSet on two planes — to answer questions Loki and Tempo already answer better.

What still works. The Observer is adapter-based and contains no OpenSearch code at all: it speaks HTTP to logs, tracing and metrics adapters against the contracts in vendor/openchoreo/openapi/observability-*-adapter-api.yaml. The metrics adapter is Prometheus-backed and untouched, so the portal keeps its Metrics tab, RuntimeHealthCard, runtime topology, FinOps and the SRE agent.

What went dark, and came back. Removing OpenSearch took the portal's Runtime Logs and Traces tabs with it, along with archived build logs and the Kubernetes events view. Logs and traces have since been restored by writing the two adapters the Observer expects — see below. Archived build logs and events are still gone; live Argo tail and kubectl get events cover them.

What was already dark. ObservabilityAlertRule with type: log routes through the logs adapter, and it never worked here — both auto-generated slo-errors rules sat in phase: Error with the adapter answering rule creation HTTP 500. Nothing was lost with them, and the current adapter returns a 501 for those endpoints rather than pretending.

The adapters

The Observer is adapter-based: it speaks HTTP to whatever implements the contracts in vendor/openchoreo/openapi/observability-*-adapter-api.yaml, and upstream's own charter names "swapping OpenSearch for Loki" as expected. Two small services in platform/loki-adapter/ do exactly that:

Together about 64Mi, against roughly 4GB to reinstate OpenSearch.

Both are stdlib-only Python with the code mounted from a ConfigMap, so there is no build step. ./platform/loki-adapter/install.sh deploys them and points the Observer at them.

They identify components by UID, not name. The Observer's search scope is {namespace, projectUid, componentUid, environmentUid}, so Alloy promotes component_uid, project_uid and environment_uid onto both log streams and span resource attributes. Matching on names would work by luck until two projects shared a component name.

Three things the contract does not advertise, all learned the hard way:

• The spans call requires startTime, endTime and searchScope, not just the search call. An empty body is rejected by the Observer before the adapter is reached.
• Attributes are omitted from the Observer's response unless the request sets includeAttributes: true. The adapter always returns them.
• The logs metadata object is a fixed struct and the tracing one is not. This is the difference between the two adapters, and it cost a session.

The blank Runtime Logs tab

Worth writing down in full, because every signal pointed away from the cause. Loki had the labels. The adapter found them. The Observer returned 200 with well-formed JSON, correct log text, correct timestamps, correct levels. There was no console error, nothing in any log, and the tab rendered nothing.

The adapter passed Loki's stream labels through as metadata. The Observer decodes that field into a fixed Go struct whose JSON tags are camelCase and whose names differ from Loki's anyway:

Not one key matched. Go's encoding/json drops unknown keys silently rather than erroring, so every field decoded as "", every field is tagged omitempty, and the browser received "metadata": {} on every row. The view keys its pod and container display off exactly that identity, so it had nothing to draw.

Traces were never affected because the tracing contract declares additionalProperties: true — the Observer decodes those attributes as an open map, so there are no fixed names to miss.

The subtle part of the fix: the two namespaces are different values. Loki's namespace label is the data-plane cell (dp-default-inkwell-staging-97325adc); the contract's namespaceName is the OpenChoreo namespace (default), which only the request scope knows. They are both plausible and swapping them is silent.

If you would rather reinstate OpenSearch

The adapter contract is public and upstream expects this. From the observability working group's charter, listing what is explicitly out of scope for them because the module model already handles it:

Selection of specific backend technologies where the existing module model already handles this (e.g., swapping OpenSearch for Loki).

The Observer takes LOGS_ADAPTER_URL and TRACING_ADAPTER_URL as plain configuration (observer.logsAdapter.url, observer.tracingAdapter.url), so pointing it at a different implementation is a values change. What does not exist is the implementation: no Loki or Tempo adapter is vendored, and the OpenSearch ones are pulled as OCI charts at install time.

Writing one means implementing about six endpoints per adapter:

logs:    /health, /api/v1/logs/query, /api/v1/events/query,
         /api/v1alpha1/alerts/rules, /api/v1alpha1/alerts/webhook,
         /api/v1alpha1/alerts/rules/{ruleName}
tracing: /healthz, /api/v1alpha1/traces/query,
         /api/v1alpha1/traces/{traceId}/spans/query, .../spans/{spanId}

That is the supported path and it is not large. It was skipped here because it is a bridge built solely to keep two tabs alive when Grafana already answers those questions — but if the portal must be the only UI, this is how.

Note one gap if you go that way: the chart has no observer.metricsAdapter value and never emits METRICS_ADAPTER_URL; the metrics adapter address is a compiled-in default. Override it through observer.extraEnvs.

Getting in

https://grafana.jung.town     sign in with GitHub, or admin / REDACTED

Four datasources, provisioned and health-checked: Prometheus, Loki, Tempo, Pyroscope.

A provisioned datasource cannot be retired by deleting it from the values file, and the API refuses DELETE even as admin (datasources:delete permission). Use deleteDatasources: in the provisioning block — it is the only mechanism.

Dashboards

Version-controlled JSON in platform/grafana-dashboards/<folder>/, one Grafana folder per directory, one ConfigMap per folder (platform/grafana-dashboards/apply.sh, server-side applied — the set is past the 256KB limit of a client-side apply). Re-run apply.sh after editing any of them; a browser edit is not persisted.

Two are landing pages — Transaction view for one request and Build for one build — and every link the platform hands out (chat, Gitea, the portal, the build viewer, the other dashboards) points at one of those two.

Finding your way around

Every dashboard links to every other one and to the four Grafana Drilldown apps — Traces, Logs, Metrics and Profiles — pinned in the sidebar (navigation.app_sections in lgtm-grafana.yaml). Those are the queryless browse-by-service views and are the way in when you do not already know what you are looking for: Traces Drilldown groups every build under the ci-build root service and every request under its service, and one click on a root opens the waterfall. Two honest limits: the Drilldown apps and TraceQL metrics answer only the last 30 minutes (Tempo keeps raw spans resident that long on this host — complete_block_timeout), so older builds are reached through the Build pipeline dashboard by run name; and they need the Editor role, which every account here has.

Tags group them: entry (overview, transaction view), delivery, ci, profiling, capacity.

Three dashboards used to have no links in or out at all and were reachable only by browsing the dashboard list.

The delivery index, which is what makes the first two possible

Everything needed to answer "what is running here and what built it" already existed on the control plane, and nothing joined it:

project-flow now publishes that join at /metrics as openchoreo_deployment_info, and at /delivery as JSON. An _info gauge is the standard idiom for this and was chosen for a specific reason: it needs no new datasource and no Grafana plugin. Its labels join to the existing RED metrics on project/component/environment and to logs, traces and profiles on the UIDs, so one dashboard can hold all four pillars at once.

curl -s http://<project-flow>:8080/delivery | jq '.deployments[0]'

Two honest limitations, visible in the data rather than hidden:

• A manually submitted build carries no commit. Only webhook-triggered runs record one, so those rows show the short sha recovered from the image tag and no build link. Inventing a link would be worse.
• state: Undeploy rows have no URL, because there is nothing running. That is correct, not missing data.

It is deliberately not pushed into the Backstage catalog. The catalog provider emits a fixed annotation allow-list, so CR annotations never reach the entity — a route already tried twice, and it does not work.

Editing one in the browser is not persisted. Export the JSON back to the repo and re-apply the ConfigMap:

kubectl --context k3d-openchoreo-op create configmap openchoreo-dashboards \
  -n grafana-lgtm --from-file=platform/grafana-dashboards/ \
  --dry-run=client -o yaml | kubectl --context k3d-openchoreo-op apply -f -
kubectl --context k3d-openchoreo-op rollout restart deploy/grafana -n grafana-lgtm

Why Grafana is not the index either, any more

This section is history. It was written when the choice was "portal or Grafana", and the answer turned out to be neither: the console is the index. Every bot reply, every build announcement and every alert now lands on a console page, and the five hand-encoded Grafana Explore URLs that used to be scattered through the templates were rewired.

The reason is the one this section already gestures at and did not follow through: a Grafana link needs a signed-in Grafana session, and Explore does not open at all for a Viewer. So the deep link that was supposed to be the payoff of an alert was, for most of the people receiving it, a login page.

Grafana is still underneath everything here and every console page offers a way through to the dashboard it was built from. It is the escape hatch rather than the front door. The rest of this section is kept because the two findings in it are real and were expensive.

The portal would have been the natural single pane, and it cannot be one without a custom image. Its bundled app has the Kubernetes and TechDocs plugins but not Grafana and not proxy, so a dashboard can be neither embedded nor proxied; and its catalog provider allow-lists annotations, so grafana/dashboard-selector and friends set on a Component CR never reach the entity. Both were verified rather than assumed — see FINDINGS.md.

So Grafana held the signals and linked back to the portal for catalog, builds and promotion, with Delivery and Component 360 as the entry points. The console carries all of that now, and platform_telemetry/links.py is the one place allowed to build a Grafana URL — with a checker that fails the build if anything else does.

The portal does link out now, and what it took

A component's page in the portal carries a Links card pointing at its dashboards. Getting there took three attempts, and the two that failed are worth knowing because both look like they should work:

So the portal runs a custom image: openchoreo/backstage-plugins v1.2.0 with a patch teaching the catalog provider to emit metadata.links. The patch is small because EntityLinksCard already ships in the app — the only thing missing was entities that carry links. It is gated behind one key, so with openchoreo.observabilityBaseUrl unset the entities are byte-for-byte upstream:

backstage:
  image:
    repository: host.k3d.internal:10082/openchoreo-ui
    tag: links-v1
  appConfig:
    openchoreo:
      observabilityBaseUrl: "https://grafana.jung.town"

The patch and build script live in platform/portal-image/ — an image built from an untracked local clone is an image nobody can rebuild. Two things the script encodes: the build needs the workflow plane stopped for memory, and the app registry is served through that same cluster's load balancer, so the push has to come after it restarts.

Grafana is still where the signals are. The portal links to them rather than embedding them: grafana and proxy are absent from the frontend config schema, so a dashboard can be neither embedded nor proxied without going considerably further than this.

Alerts

Platform alerts are PrometheusRules in platform/openchoreo-config/alerts/platform-alerts.yaml: node disk at 85% and 93%, node memory, crash loops, OOMKills, pods stuck not-ready, and build pods accumulating. They fire into the Alertmanager on the observability plane.

Application alerts are ObservabilityAlertRule with type: metric, which the portal still renders. type: log is not usable — see above.

Two rules deliberately absent, both documented in FINDINGS.md: anything over kubelet_volume_stats_* (k3s local-path PVs all report the host filesystem, so one full disk fires 22 identical alerts), and anything relying on count() of an empty selector without or vector(0) (renders as "No data", reads as broken).

The one change that moved everything

Applications were not touched. The otel-instrumentation trait had its endpoint changed from OpenChoreo's collector to Alloy:

collectorEndpoint:
  default: "http://alloy.grafana-lgtm.svc.cluster.local:4318"

Every component re-rendered and restarted pointing at the new collector. This is the argument for making the collector address a platform concern rather than an application one, and it is worth doing the thought experiment of what this migration costs when each of forty repos has the endpoint in its own config.

Alloy then forwards:

app ──OTLP──> Alloy ──> Tempo   (Grafana)

This used to fan out to OpenChoreo's own collector as well, so the portal's Traces tab and Grafana showed the same spans. That collector went with OpenSearch; the portal reads Tempo through platform/loki-adapter now, and Alloy has one trace backend.

What Alloy collects

One DaemonSet per plane — control, data, workflow, observability — each stamping a cluster label so four planes can share one Loki without blurring.

• Logs: tails /var/log/pods, parses CRI framing, and relabels OpenChoreo's own pod labels (openchoreo.dev/component, /project, /environment) into Loki stream labels. A Grafana query and a portal component therefore line up without anyone maintaining a mapping.
• Traces: an OTLP receiver, k8sattributes enrichment by source IP, then the fan-out above.
• Profiles: scrapes pprof endpoints from pods annotated pyroscope.io/scrape: "true".

Profiling

Two ingest models, because languages differ:

• Pull — Alloy scrapes /debug/pprof. Right for Go, where the runtime already serves it.
• Push — the SDK samples in-process and ships. Necessary for CPython and Ruby, which have no equivalent endpoint.

inkwell-enrich (Python) and inkwell-web / inkwell-worker (Rails) use push, enabled by the continuous-profiling trait:

traits:
  - kind: ClusterTrait
    name: continuous-profiling

It is a separate trait from otel-instrumentation rather than a flag on it, because a sampling profiler in your process is a different cost and usually a different decision from exporting spans.

The trait is half of it. It injects PYROSCOPE_*; something in the process has to read them. Applying it to a service with no profiler SDK gives you environment variables nothing reads and a profiles panel that looks enabled and stays empty — which is worse than one that is honestly absent, because it looks broken rather than off. Each language needs its own initializer: apps/enrich/app/profiling.py, apps/inkwell/rails/config/initializers/pyroscope.rb. Both guard on PYROSCOPE_ENABLED rather than on the address being set, so a profiler never starts during assets:precompile, a test run, or a rake task.

Profiles carry the same environment / namespace / pod tags the spans do, which is what makes Grafana's trace → profile jump land somewhere useful: from a slow span to what the CPU was doing while it ran.

How to actually get to a profile

Three routes, shortest first. This is written down because the data existed for a long time before anything pointed at it, and "no idea how to access pyroscope" is a fair description of a stack where every flame graph is two correct guesses away.

1. Profiles dashboard — https://grafana.jung.town/d/oc-profiles/profiles. Pick a service. It opens with a table of what is profiled and how, because profiling is opt-in and an empty flame graph is usually correct.
2. From a component — Component 360 has a Profiles for this component link that carries the service through.
3. From a trace — Grafana's trace view offers profiles for this span where the tags line up, which is what the matching environment / namespace / pod tags on both are for.

Twelve services have data. Which PANELS can fill, though, depends on the language rather than on anything configurable:

This table was wrong in two directions for months, and both are worth naming.

It said pyroscope-io ships no memory profiler. It does: the pinned version accepts mem_enabled, mem_max_nframe and mem_heap_sample_size, and its native library carries alloc_objects / alloc_space / inuse_objects / inuse_space. It had simply never been asked — and on six of the seven Python services the switch that would have asked it was read by nothing at all.

It also credited the pull path for Go and listed only dagger-engine. rideshare carries the Go SDK and pushes ten profile types itself (internal/telemetry/telemetry.go): CPU, alloc and inuse objects and space, goroutines, and mutex and block counts and durations. It does not need the scrape path and never did.

What remains true is Ruby: the pyroscope gem is genuinely CPU-only, no setting changes that, and a Rails leak is found with the CI memory profilers or a heap dump.

Anything else is empty because it is not profiled, not because the stack is broken. Adding one means the trait and a profiler SDK in the app — the trait alone only sets PYROSCOPE_*, which a process with no profiler in it ignores without complaint.

Span profiles: the profile for THIS span

One contract, every language. Every profiling sample taken while a span is active carries three labels — span_name (a series label), span_id and trace_id (per sample) — attributed to the innermost active span, and the span carries pyroscope.profile.id, which is what makes Grafana offer a flame graph scoped to the span you clicked rather than to the whole service over a window. Health probes are never tagged.

The stack is the part that matters. Pyroscope's thread tags are not a stack: tag on a child, remove on its end, and the parent's remaining samples are attributed to nothing. Both upstream processors dodge that by profiling root spans only, which is why a request used to be one bucket named for its route. The platform's processors restore the parent's tag when a child ends.

Selecting one: {service_name="chess-engine", span_name="engine.root_move"} for the operation across a window, or the span selector (Grafana's Profiles for this span, or the Transaction view's Span id variable) for exactly one span's samples. A span shorter than ~10ms has no samples at 100Hz; that is not a broken profiler.

CPU by default everywhere. Heap profiles where the SDK has one, which is not where this document used to say:

• Python can profile the heap. The pinned pyroscope-io accepts mem_enabled, mem_max_nframe and mem_heap_sample_size, and its native library carries alloc_objects / alloc_space / inuse_objects / inuse_space. It was never asked. On by default since the trait's 2.0.0 — "what is growing" is the question people arrive with, and the OOMKilled walkthrough should not have to begin with a redeploy. Turn it off for a component that does not want a hooked allocator with memoryProfiling: false per environment; shape it with memoryMaxFrames and memoryHeapSampleSize, which 2.1.0 added and which are the same dial for cost and for detail.
• Go already pushes the full memory set itself (rideshare/internal/telemetry.go). It does not need the pull path and never did.
• Ruby genuinely is CPU-only. The pyroscope gem ships no heap profiler and no setting changes that, so a Rails leak is found with the CI memory profilers or with a heap dump. That is the one true instance of the limitation this paragraph used to claim for all three.

See the OOMKilled runbook for what to do with any of it, and Profiles and flame graphs, in CI and in production for how to read one — which questions a CI profile answers that a deployed one does not, why a stage can say engine profile or no profile without anything being broken, and why a flame graph's first sixty levels are framework.

Build traces, and pairing them with the build profile

Every build is one trace with a root span — build <component> under service.name="ci-build" — whose id is derived from the run name, so the Build pipeline dashboard, the chat message and the Gitea commit status all point at it without asking Tempo first. See "Tracing the build" below for the shape.

The pairing is the useful part, and each half is useless alone:

• the trace says which step was slow — checkout, bundle install, rspec, publish
• the profile of dagger-engine over the same window says why — which function burned the CPU while that step ran

A build that is slow because it is waiting on the network looks identical to one that is slow because it is compiling, until you put those two side by side.

Profiling the build itself

The pull path had no subscribers at all until now — every profile in Pyroscope arrived by push. The Dagger engine is Go and buildkit serves pprof behind a flag, so platform/dagger/values.yaml turns it on and opts the pod in:

engine:
  args: ["--debugaddr", "0.0.0.0:6060"]
  annotations:
    pyroscope.io/scrape: "true"
    pyroscope.io/port: "6060"

This is the dimension tracing could never give. A build's spans tell you which step was slow and nothing about why; the profile says where the engine's CPU actually went. You can tell the pull path is working because goroutine and memory series appear in Pyroscope — the push SDKs here only ever send process_cpu.

Going from a build to its profiles

Two kinds, and they live in different places.

The build's own profiles carry the run name. Every CI profile is pushed under {ci_build_run="<run>", span_name="<span>"}, and there are two producers of them, under three application names:

The suites keep their own application names because renaming them would orphan every profile already stored under them. The stage is a tag, never part of the application name — putting it there would make each stage its own series and defeat the grouping.

Both producers also tag ci_stage with the bare step name (lint, not rails-ci/lint), matching the ci.stage span attribute and the Loki structured metadata, so one query works across all three stores. span_name and ci_stage are different strings on purpose and neither is redundant: the console selects on the former, dashboards on the latter. See FINDINGS.md, "One label, three stores, and the console pivots on a fourth".

The Build pipeline dashboard's flame graph selects on exactly that, and the Span profiles dashboard compares two of them. A sub-second suite produces no profiles at all — nothing to sample — and says so on the panel.

Which stage cost the most is a panel, not an exercise. The build page's "Where the CPU went" ranks the stages by CPU (with wall time beside it, because the stages run concurrently and CPU can exceed wall) and then ranks the spans inside them. /api/builds/{run}/cpu carries the same numbers.

Both rankings come from querier.v1.QuerierService/SelectSeries with groupBy: ["ci_stage"] and ["span_name"] — one request each, rather than a /render call per stage and per example. Worth knowing when adding any other "top N by profile" view.

Read the two rankings as answering different questions; they do not sum to each other. A stage's CPU includes everything it did while no span was open, and Pyroscope returns that as a series with the grouped label absent. The panel names it "outside any span" rather than dropping it, because on a real Rails build it is the largest single row — 44% on inkwell, which is Rails booting. No per-test view can show that, since it happens outside every test.

Go is profiled by its own toolchain, not by an agent. go test writes a pprof with -cpuprofile, and the platform uploads it. The toolchain refuses the flag with multiple packages ("cannot use -cpuprofile flag with multiple packages"), so the stage loops over go list ./..., merges with go tool pprof -proto, and pushes once. No agent means nothing to flush, nothing to race and no overhead when the flag is absent — the opposite of the Ruby and Python story above. go-ci/lint has no profile at all: gofmt and go vet expose no hook to attach one to.

Memory is opt-in, in both languages, and they measure different things. Set profileRich: true on a component's workflow parameters. The suite then also runs under a memory profiler and the result is pushed to <component>-ci-mem.

<component>-ci-mem is bytes still live in both languages: memray's high-water mark for Python, vernier's trace_retained for Ruby (whose Gecko output the platform folds — weightType is bytes). Ruby additionally pushes allocation churn to <component>-ci-alloc from stackprof, because "what churns" and "what is held" are different questions.

A retained profile too large for Pyroscope's 4 MiB push limit is capped to its largest stacks — the log line says how many and what fraction of bytes — because Pyroscope refuses an oversized body rather than truncating it.

ruby-prof is not an option for this: version 2 dropped its MEMORY mode.

Ruby needs gem "stackprof" and gem "vernier" in the application's bundle: bundler purges the default gem home from the load path, so a platform-side gem install is invisible under bundle exec. Same arrangement as gem "pyroscope". Both are require: false and loaded only when a build asks to be fully profiled. Measured cost on the console's own suite: 25s to 26s, about 4% — the reason it is opt-in is that it installs memray from the public index at build time, the only step in the pipeline that fetches a package, not the runtime.

Per-test memory is on every build, not just fully profiled ones, because the counters are free: the rspec and pytest plugins read them around each example and put them on the test span. test.memory.rss_delta_bytes is the number that compares across languages — resident change, signed, so a test that hands memory back is visible doing it. The churn counter beside it is NOT the same measurement in the two languages and is never compared across them: Ruby reports test.memory.allocated_objects (objects allocated during the test), CPython reports test.memory.retained_blocks (allocator blocks still outstanding after it).

The build page ranks them under Memory by test, by size of change rather than by growth — a −8 MiB row is usually the test cleaning up after the others, and sorting descending buries it. Each row links into the waterfall, which is where a span's logs and flame graph live. /api/builds/{run}/memory is the twin.

The axis is bytes and means it. These arrive as pprof declaring inuse_space in bytes with a space/bytes period type, so they land under memory:inuse_space:bytes:space:bytes — the same type the Go runtime's heap profiles use — and a CPU query cannot see them. No exclusion is needed anywhere; the service_name!~".*-ci-mem" matcher that every CPU selector used to carry is gone. A very large profile is capped to its biggest stacks to fit Pyroscope's 4 MiB limits, and the coverage achieved is logged by [ci-rich-profile]. The whole thing is drawable from the build page — /api/builds/{run}/memory-profile feeds the same flame canvas the CPU profiles use, labelled in MiB because that canvas is shared and the unit would otherwise change silently between clicks.

Two stages have no application profile, and the console says which kind of nothing that is. The Python lint runs ruff, a Rust binary a Python sampling profiler cannot attach to (n/a, with the reason). publish builds the application's own Dockerfile inside the engine, so the stage offers the engine's own CPU over its window instead (engine) — approximate, because the engine runs four builds at once, and labelled as such wherever it is drawn.

The deployed service's profiles are attributed to a component and environment, not to a build. The chain runs through the delivery index:

1. Delivery — find the build, and the component and environment it deployed to
2. click through to Component 360 for that component
3. the profiles panel is that component's, for the version currently running

Which is exact as long as nothing has deployed over it since. If it has, the commit shown in Component 360's header is the one you are actually looking at — that header, not the build you started from, is the authority.

The gateway hop

Every trace on this platform used to begin at the application's own server span. Envoy with no tracer configured emits no span and does not synthesise a traceparent — it passes through whatever the client sent and generates only x-request-id, which nothing collected. Beyla could not cover it either: its discovery is scoped to k8s_namespace: dp-* and the proxy pod lives in openchoreo-data-plane. So gateway latency, gateway 5xx, the OIDC redirect and the whole 15-second-timeout failure class sat outside every trace, and "the request took four seconds and the application says it took forty milliseconds" had no answer anywhere.

platform/openchoreo-config/gateway-tracing.yaml closes it: a second HTTPListenerPolicy on gateway-default with Envoy's OpenTelemetry tracer pointed at Alloy, plus a JSON access log carrying the traceparent. It is a second policy rather than an edit of the chart's own because that one is Helm-owned; kgateway merges policies field by field and reports Attached=True / Merged.

{resource.service.name="gateway"}      # the north-south hop, name `ingress`

Two things had to be true and only one of them was obvious.

The tracer speaks OTLP over gRPC only — tracing.provider.openTelemetry takes a grpcService and nothing else — so it dials Alloy's 4317 rather than the 4318 every application uses. Nothing had ever dialled that port, and nothing had noticed that a consumer discovering it through Kubernetes cannot know it needs HTTP/2. kgateway built the upstream cluster from the Service, defaulted to HTTP/1.1, and Alloy rejected every export as a protocol error.

The way that failure presents is worth remembering, because it says the opposite of what is happening:

tracing.opentelemetry.spans_sent: 37
tracing.opentelemetry.spans_dropped: 0

…with nothing in Tempo. spans_sent counts spans handed to the exporter, not delivered. The truth is one level down:

cluster.kube_grafana-lgtm_alloy_4317.upstream_cx_http1_total: 24
cluster.kube_grafana-lgtm_alloy_4317.upstream_cx_protocol_error: 24
cluster.kube_grafana-lgtm_alloy_4317.upstream_rq_502: 24

The fix is one line of chart values — appProtocol: "kubernetes.io/h2c" on Alloy's otlp-grpc port (lgtm-alloy.yaml). kubernetes.io/h2c and not the bare h2c, which is Istio's spelling; kgateway follows the Gateway API convention.

And the log half, which works when sampling does not. The access log is JSON carrying %REQ(TRACEPARENT)%, and Alloy's pipeline gained a second regex stage to unpack it — the existing one requires a trace_id-shaped key and finds the version byte 00 where the id should be. This matters more than one emitter's spelling: the gateway is the only thing that logs a request Envoy refused. A 502 that never reached a component has no application span and no application log line, and this is the only way it is findable from the trace it belongs to.

{pod=~"gateway-default.*"} | trace_id="<id>"     # container is `kgateway-proxy`

What is still outside a trace, and honestly:

• The browser. apps/storefront and apps/brochure have no OTel JS, so a user's request arrives with no traceparent and Envoy starts the trace. That is the right root for now, and it means the trace begins at this network's edge rather than at the user's.
• Credentialed cross-origin readers. The applications set X-Trace-Id and traceresponse on every response and the component types' CORS filter already carries exposeHeaders: ["*"], so a plain SPA on another origin CAN read the id its own request produced. The wildcard is ignored for a request sent with credentials, though — that is the Fetch standard, not a gateway setting — so a cross-origin call with a cookie or an Authorization header still cannot see it, and would need the two names listed explicitly.
• Garage. It starts a fresh trace rather than continuing its caller's, so an object-store hop correlates by time and bucket only.

eBPF, and what it is honestly worth

Beyla runs on the data plane and produces HTTP spans and network flows from processes with no instrumentation at all.

That matters because the traces elsewhere in this demo are good because I wrote the instrumentation. Most estates have a long tail of services nobody will retrofit, and for those the choice is not "SDK or eBPF", it is "eBPF or nothing".

What it gives you:

• RED metrics and spans per HTTP route, no code change, no restart of anything but Beyla.
• Network flows with Kubernetes attribution — which Deployment talked to which, in bytes — the cheap version of the Cilium/Hubble story, without replacing the CNI.

What it does not give you:

• Kernel probes see sockets, not intent. Beyla can say a request took 300ms and returned 500; it cannot say which of your functions was slow.
• Spans join a distributed trace only where Beyla can read traceparent off the wire. For Go binaries it can propagate context, but only with CAP_SYS_ADMIN, which is close enough to root that granting it to a DaemonSet on every node is a real decision. It is deliberately not granted here.

Tell the two sources apart in Tempo:

{resource.telemetry.sdk.name="beyla"}          # eBPF
{resource.telemetry.sdk.name="opentelemetry"}  # the application SDK

Network flows: Cilium and Hubble

The data plane runs Cilium as its CNI, with Hubble exporting flow metrics into the same Prometheus everything else uses.

https://hubble.jung.town — the service map, built from the kernel.

What that buys over the tracing already in place: a trace tells you what a request meant, a flow tells you what actually crossed the wire — including everything no SDK will ever report. DNS lookups, kubelet probes, the connections that were refused, and traffic from workloads nobody instrumented.

# the raw flow log, live
POD=$(kubectl --context k3d-openchoreo-dp get pods -n kube-system -l k8s-app=cilium         -o jsonpath='{.items[0].metadata.name}')
kubectl --context k3d-openchoreo-dp exec -n kube-system $POD -c cilium-agent --   hubble observe --namespace dp-default-inkwell-staging-97325adc --last 20

# the service graph, as a Prometheus query
topk(10, sum by (source,destination) (rate(hubble_flows_processed_total[15m])))

# what is being dropped, and why
sum by (source,destination,reason) (rate(hubble_drop_total[15m]))

hubble_drop_total is the one to keep. A dropped flow is invisible to application telemetry by definition — the request never arrived, so nothing logs it and no span is created. It is exactly the failure that otherwise gets diagnosed as "the network".

What it cost

The CNI is chosen when a cluster is created and cannot be changed afterwards. Adding Cilium meant destroying and rebuilding the data plane — see platform/bootstrap/k3d/config-dp.yaml and 34-cilium.sh.

That turned into an accidental test of whether the platform is actually declarative, and it passed: with the cluster empty and re-registered, the control plane re-created all three projects, their namespaces, workloads, databases, object stores and gateways with no manual intervention. What did NOT come back was data — every Postgres volume, MinIO bucket and Zulip realm was gone, and one secret had to be re-pushed to OpenBao because the SecretReference on the control plane outlived the value it pointed at.

Which is the honest summary of what declarative infrastructure gives you: the shape of the system, not its contents.

Tracing the build

Dagger is an OpenTelemetry producer, and the build step makes it one trace:

push apps/inkwell main by thomas  service.name=gitea      (HEAD: the webhook, written by project-flow)
│   events: webhook received · forwarded to autobuild · WorkflowRun created ·
│           commit status: pending · build succeeded · commit status: success ·
│           manifests: 2 updated
├── deploy staging                project-flow            (the release bound and Ready; url.full)
├── promote production            project-flow            (when the policy or a person promotes it)
└── build <component>             service.name=ci-build   (root of the build proper: the Argo step; result, commit, branch, Argo URL)
├── rails-ci --source=...         dagger-build            (the CLI's first call)
│   └── Platform.railsCi          dagger-engine           (the module function)
│       ├── rails-ci/lint         dagger-go-sdk           ci.stage=lint     ← stage spans, one per step
│       │   └── exec rubocop      dagger-engine           ← owns the command's output
│       ├── rails-ci/audit
│       ├── rails-ci/test         ci.test.capabilities=true
│       │   └── rspec <example>   inkwell-web-rspec       test.outcome, one span per test, one profile per test
│       ├── rails-ci/test         ci.test.capabilities=false
│       └── rails-ci/publish
└── rails-image-ref --source=...  dagger-build            (the second call; a cached publish)
    └── rails-ci/image-ref

The head of the trace is the push. Gitea cannot emit a span and OpenChoreo's autobuild keeps nothing of a webhook but the commit, so project-flow — which sees the raw delivery and the WorkflowRun it becomes — writes the head span itself (tracing.py, stdlib OTLP/JSON like the Argo step's root). Its id is derived, sha256("trigger:"+run)[:16], and the Argo root names it as parent by the same derivation, so neither side waits for the other; if project-flow was restarted between the push and the build the head says manual run and carries only what the run knows. Its events are the record of what the controller did, in order and with timestamps: that is what makes a build trace readable at a glance. deploy <env> and promote <env> land later as children of the head, so one trace runs push → build → deploy → promote. Gitea shows two statuses per commit, openchoreo/build/<component> (the Build dashboard) and openchoreo/trace/<component> (the raw trace), plus openchoreo/manifests for the sync.

Three more things make it findable rather than searchable:

• The trace id is sha256(<run name>)[:32], minted by the step (dagger-build-template.yaml) and recomputed by project-flow, the bots and the dashboards (platform_telemetry.links.build_trace_id), so "the trace for this build" never needs a lookup. The root span is posted by the same step on exit, whatever the outcome, which is what gives Tempo a root service and name to group by.
• Every exec's output is a log record attributed to its span. The step sets OTEL_EXPORTER_OTLP_LOGS_ENDPOINT (Dagger configures no log exporter from the base endpoint alone), Alloy sends OTLP logs to Loki's native endpoint, and the record's trace_id/span_id land in structured metadata: {namespace="workflows-default"} | trace_id="<id>" | span_id="<exec span>" is one command's stdout and stderr.
• Engine noise is dropped at the collector by Dagger's own markers (lgtm-alloy.yaml, otelcol.processor.filter "noise"), not by service, so the execs that own the logs survive.
• Every stage's output is also recorded on the STAGE span, and every test's on the TEST span. Span ids are not hierarchical, so a dashboard holding a stage span cannot ask Loki for the execs under it. The pipeline module re-emits each stage's output with ci_stage and ci_origin (project or platform) in structured metadata; the rspec/pytest plugins post each example's captured output and failure with the example's span id. So | trace_id="<build>" | ci_stage="test" is one stage, | span_id="<test>" is one test, and the Build pipeline dashboard's stage and test rows are one click to either. The failed-build announcement in chat uses the same query for its log tail.

Build latency and request latency are now the same question asked of the same tool. This is also exactly the accident the application's tracing initializer guards against: those OTLP variables are visible to every container Dagger runs, including the test suite, which is why the app requires an explicit OTEL_SDK_ENABLED instead of keying off endpoint-presence.

The build page, and Grafana inside the portal

Grafana is the right tool for a question that is a query; a build is a tree whose leaves each have their own output and flame graph, and that is a click-to-drill page. The console (https://console.jung.town/builds/<run>) reads Tempo, Loki, Pyroscope and project-flow's metrics directly and draws one page per build: the push at the top with its events, the waterfall, each stage's lines expanded on load, and on click, the span's flame graph (Pyroscope's own flamebearer, drawn on a canvas). It is a platform component like any other — its first page is its own build — and it links out to Grafana for the full tools.

apps/buildviewer served a subset of this at builds.devtools.jung.town and is deleted: same path shapes, strictly less on each page, and on a hostname only this LAN could resolve.

The portal embeds Grafana on every entity's Dashboards tab through the community Grafana plugin (platform/portal-image/grafana-plugin.patch): the catalog provider emits grafana/overview-dashboard (Component 360 or Project 360 in kiosk mode), grafana/dashboard-selector and grafana/alert-label-selector on every Component and Project, and the static platform-services catalog points each Resource (Gitea, Argo, Zulip, …) at its infrastructure dashboard. The plugin reaches Grafana through the backend proxy with a Viewer service-account token (26-lgtm.sh mints it into Secret grafana-sa); the iframe is the reader's own Grafana session, which is why allow_embedding is on.

Querying by hand

# Loki: which planes are shipping logs
curl -s -H 'Host: loki.lgtm.openchoreo.localhost' \
  'http://localhost:11080/lgtm/loki/loki/api/v1/label/cluster/values'

# Loki: one component's logs
curl -s -G -H 'Host: loki.lgtm.openchoreo.localhost' \
  'http://localhost:11080/lgtm/loki/loki/api/v1/query_range' \
  --data-urlencode '{component="inkwell-web"}'

# Tempo: needs a port-forward; the query API is on 3200, not 3100
kubectl --context k3d-openchoreo-op port-forward -n grafana-lgtm svc/tempo 3200:3200 &
curl -s -G localhost:3200/api/search --data-urlencode 'q={resource.service.name="inkwell-web"}'

Loki omits the data field entirely when a query matches nothing, so an empty result reads as {"status":"success"} and looks like a broken endpoint. It is not.

Cross-cluster plumbing

Loki, Tempo and Pyroscope live on the observability plane; Alloy runs on all four. The clusters reach each other through the observability plane's gateway on :11080, routed by path (/lgtm/loki, /lgtm/tempo, /lgtm/pyroscope) rather than by Host header.

Path routing because Alloy's loki.write and pyroscope.write build their own requests, and overriding the Host header from a config map is not something to rely on. OpenChoreo's shipped collectors do use the Host-header form — they can, because the OTel Collector exposes an explicit header override.

Cost

Roughly 3.5GB of RAM for the whole addition: Loki single-binary, Tempo monolithic, Pyroscope, Grafana, four Alloy DaemonSets and Beyla. Every component runs in its single-binary or monolithic mode; the microservices deployments of Loki and Tempo would be several times that for a demo whose entire data volume fits in a PVC.

Getting a link to the thing you just did

The stack had four signal stores, correct correlations between them, and no way to hand anyone a link to a specific trace. Two link builders existed — one in TypeScript in the portal's catalog provider, one in Python in the TechDocs generator — and neither could produce one, so the single most useful link on the platform was the one nobody could give you.

Three things changed.

Every HTTP response carries its own trace id.

curl -si -H 'Host: production-inkwell.jung.town' \
  http://localhost:19080/inkwell-web-http/widgets | grep -i trace
# X-Trace-Id: 4bf92f3577b34da6a3ce929d0e0e4736
# traceresponse: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

traceresponse is the W3C header a standards-aware client looks for; X-Trace-Id is the bare id, because every ad-hoc curl -I and every bot wants the 32 hex characters without parsing a version field out of them. Set by ApplicationController#expose_trace_context in the rails-service template, inside a rescue — telemetry must never take a response down.

Every conversation is one trace. Every bot reply ends with a traceparent: line, and a message that carries one is answered under it, so the conductor's scripted exchanges in #bot-arena (apps/zulip-bots/bots/conductor.py) are one waterfall each: conductor → chess-bot → chess-engine → chess-render, or → inkwell-bot → inkwell-web → worker → enrich → ledger and Garage. The closing message of each conversation links the whole trace, its log lines and the health page of each project it touched.

Every chat message opens a root span. apps/zulip-bots/bots/framework.py previously instrumented httpx and nothing else, so a bot command produced a floating client span: a request with no parent and no server span, appearing in Tempo as an orphan trace that says a call happened and nothing about what asked for it. A span opened per message makes the chat message the root, so one command is one waterfall from Zulip through every service it touches — and it is what makes a trace id exist to put in the reply.

There is one link builder. platform_telemetry.links (platform/pylib/platform_telemetry) turns a trace id, a service or a commit into a Grafana, Argo or Gitea URL. The datasource UIDs it hardcodes (tempo, loki, pyroscope, prometheus) are literals in the Grafana values file, so that is repeating a constant rather than guessing one, and tracesToProfiles / tracesToLogsV2 are configured on the Tempo datasource — so a trace link lands somewhere that can pivot to the flame graph and the logs without any of that being encoded in the builder.

Profiles that show something

The continuous-profiling trait injects PYROSCOPE_* and nothing reads them unless the process carries an SDK. Both bots and every rails-service project now do (bots/profiling.py, config/initializers/pyroscope.rb).

A bot is idle almost always, which is exactly what makes it worth profiling: the baseline is flat, so the work a slow command does stands out with no filtering. On a busy service you hunt for the signal; there it is the only thing in the graph.

The rails-service template ships four endpoints that are all slow for different reasons, because a flame graph only becomes readable once you have seen what each shape looks like. docs/WORKING-ON-IT.md has the table; the short version is that /diagnostics/memory produces a nearly empty CPU profile, and concluding "profiling says it is fine" from that is the mistake the endpoint exists to teach.

Environments

https://grafana.jung.town/d/oc-environments/environments — one row per component per environment, the commit each is running, and a drift panel that lists components whose environments disagree. Driven entirely by openchoreo_deployment_info, the join project-flow publishes, so it cannot disagree with Delivery or with the chat bot.

Blanks are the point: an empty production column means nothing has been promoted there, which is a different statement from a failure and used to be indistinguishable from one. state is shown alongside the commit because a binding can hold the newest release and be asleep — spec.releaseName says what was promoted, spec.state says whether it runs, and reading Undeploy as "not promoted" is a mistake worth naming.