Credentials on this page are redacted. This is the public export; the real values live in docs/ACCESS.md in the private repository.

Changing things

I want to change X: the task-shaped guide — WORKING-ON-IT.md

Working on it — how to change each moving part

Access and credentials are in ACCESS.md. This is the "I want to change X, what do I do" guide.

The demo is deliberately split so that the platform and the applications change independently:

apps/*                    application teams own this
platform/openchoreo-config  platform team owns this  (component types, traits,
                                                      resource types, workflows)
platform/dagger-modules   platform team owns this  (the CI pipeline itself)
platform/bootstrap        the install; rarely changes

Change an application and see it deploy

The full loop is webhook-driven — push and walk away.

cd apps/inkwell
$EDITOR rails/app/controllers/notes_controller.rb
git commit -am "change something"
git push

Then watch it:

export KUBECONFIG=~/.kube/openchoreo.yaml
CP="kubectl --context k3d-openchoreo-cp"
WP="kubectl --context k3d-openchoreo-wp"

$CP get workflowrun --sort-by=.metadata.creationTimestamp | tail -3
$WP get workflows.argoproj.io -n workflows-default --sort-by=.metadata.creationTimestamp | tail -3

# follow the build
POD=$($WP get pods -n workflows-default --no-headers | awk '/build-and-publish/{print $1}' | tail -1)
$WP logs -n workflows-default "$POD" -c main -f

The pipeline runs rubocop, bundler-audit, the RSpec suite twice (against a Postgres with pgvector/TimescaleDB and against one without), then builds and publishes. A lint or test failure stops the build before publishing — that is the point.

Only the components whose appPath was touched rebuild, and OpenChoreo starts one WorkflowRun per matching Component with the image name derived from the component. Left alone, that means components sharing a Dockerfile each run the whole pipeline and publish their own copy of an identical image: three for Inkwell (web, worker, nightly), five for the Zulip bots.

When several components ARE one image, say so with imageName and siblingWorkloads. One component owns the build; the rest set autoBuild: false:

# inkwell-web -- the owner
workloadPath: "/deploy/web"
imageName: "rails"                 # default-inkwell-rails, not -inkwell-web
siblingWorkloads:
  - {component: inkwell-worker,  workloadPath: "/deploy/worker"}
  - {component: inkwell-nightly, workloadPath: "/deploy/nightly"}
  - {component: inkwell-worker-profiled, workloadPath: "/deploy/inkwell-worker-profiled"}

The one build publishes one image and then generates a Workload per component from its own workloadPath, in parallel. Each still gets its own endpoints, config and start command — the command belongs to the ComponentType, which is the only thing that actually differs between a Rails web and worker.

buildPath remains for the other case: a narrow appPath (a single file matches) with the build rooted elsewhere. Prefer siblingWorkloads, because narrowing has a sharp edge — a change to code the components share matches no narrow appPath and rebuilds nothing, silently. Note also that appPath: "/" matches nothing at all; the spelling for "the whole repository" is the empty string. See docs/FINDINGS.md.

Run the pipeline locally instead

Same code, same engine, no push:

kubectl --context k3d-openchoreo-wp port-forward \
  -n openchoreo-workflow-plane svc/dagger-engine 18080:8080 &
export _EXPERIMENTAL_DAGGER_RUNNER_HOST=tcp://127.0.0.1:18080

cd platform/dagger-modules
dagger call ci --source=../../apps/inkwell/rails \
  --registry=host.k3d.internal:10082 --image=inkwell --tag=dev

# or just one step
cd ruby && dagger call lint --source=../../../apps/inkwell/rails
cd ruby && dagger call test --source=../../../apps/inkwell/rails --with-capabilities

Change the console

The console (apps/console) is the platform's own web application: projects, environments, deploys, builds and the controls for them. It is an OpenChoreo component like any other, so the loop is the ordinary one.

cd apps/console
$EDITOR app/model.py            # the joins -- what a page MEANS
$EDITOR app/sources/ocapi.py    # what it reads
$EDITOR app/templates/          # what it looks like
git commit -am "..." && git push

Tests need no cluster and no stores. Lint, format and the lockfile must run inside the uv container — the host cannot resolve the internal package index:

docker run --rm --add-host=host.k3d.internal:host-gateway -v "$PWD":/w -w /w \
  ghcr.io/astral-sh/uv:0.5.29-python3.12-bookworm-slim \
  sh -c 'uv run --group dev ruff format . && uv run --group dev ruff check . \
       && uv run --group dev pytest -q'

To drive it against the live platform from this machine, host.k3d.internal has to resolve: the LGTM routes are bound to that hostname, so reaching the same port as localhost gets a 404 from the gateway that looks exactly like an empty store. apps/console/README.md has the full command.

Adding a page. Every page has a JSON twin under /api returning the same model, and every upstream call goes through sources/_http.soft so a panel that could not load renders as a notice naming the upstream, never as an empty panel. Keep both: they are the two habits that make the surface trustworthy.

Publishing it. The console is NOT published by the public-hostname trait. That trait mints a name under the project's zone, which resolves through the *.jung.town wildcard A record — LAN only. Shared surfaces get a specific proxied CNAME in platform/openchoreo-config/public/dns-records.yaml plus a route in app-routes.yaml. If a thing is meant to be reachable from outside this house, it goes in those two files or it does not.

Change the CI pipeline

The pipeline lives in platform/dagger-modules/ and is consumed by the build step over git, so changes take effect on the next build with no cluster change.

cd platform/dagger-modules
$EDITOR ruby/main.go                     # e.g. add a step
dagger develop                           # regenerate bindings
dagger functions                         # confirm it compiles
git commit -am "add SBOM step" && git push

The build step clones this repo at the ref pinned in the rails-builder ClusterWorkflow (module-ref parameter, currently #main) and prints the resolved commit, so every build log records exactly which pipeline version ran.

To pin a released version instead of tracking main, tag the module repo and change module-ref to …#v1.2.0.

Change the shape of the pipeline (which steps, in what order)

That is the ClusterWorkflow:

$EDITOR platform/openchoreo-config/workflows/rails-builder.yaml
kubectl --context k3d-openchoreo-cp apply -f platform/openchoreo-config/workflows/rails-builder.yaml

The Argo step it calls is a ClusterWorkflowTemplate and lives on the workflow plane, not the control plane — a common mix-up:

kubectl --context k3d-openchoreo-wp apply -f platform/openchoreo-config/workflows/dagger-build-template.yaml

Keep the build's trace whole

A new step has to go through stage() and take buildRun, or it is invisible to every dashboard; the root span, the derived trace id and the log export are the Argo step's job. platform/dagger-modules/README.md ("The telemetry contract") is the checklist, and docs/OBSERVABILITY.md ("Tracing the build") shows what a correct build looks like in Tempo and Loki.

Change the golden path (component types and traits)

Component types are generated from OpenChoreo's shipped ones so that upstream's ~270 lines of routing logic never drift:

$EDITOR platform/openchoreo-config/component-types/generate.sh
./platform/openchoreo-config/component-types/generate.sh      # regenerate
git diff platform/openchoreo-config/component-types/           # review
kubectl --context k3d-openchoreo-cp apply -f platform/openchoreo-config/component-types/ruby-service-v1.yaml

Never hand-edit ruby-*.yaml — the generator overwrites them.

Traits are hand-written and applied directly:

kubectl --context k3d-openchoreo-cp apply -f platform/openchoreo-config/traits/otel-instrumentation-v1.yaml

Changing a component type or trait cuts a new ComponentRelease for every component using it, and only the root environment adopts it. Measured, not assumed: applying nine regenerated types on 2026-09-06 created 23 releases within three seconds, staging and development repointed at them through autoDeploy, and every production binding stayed exactly where it was.

That is because a ComponentRelease freezes the ComponentType's whole spec, not a reference to it — spec.componentType carries {kind, name, spec}, and the render uses the frozen copy. So an environment renders with the type as of the release it is pinned to, and the sentence that used to sit here — "re-renders every component using it" — was true of staging and false of production.

The practical consequence, and it is easy to lose an hour to: a new field on a ComponentType is inert in production until production is promoted onto a release cut after the edit. The field is accepted on the binding (pruning happens at render time, not admission), so it sits there looking applied and changes nothing.

kubectl --context k3d-openchoreo-cp get releasebinding inkwell-web-development \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

# Which release an environment is on, and whether that release knows the field:
kubectl --context k3d-openchoreo-cp get releasebinding inkwell-worker-production \
  -o jsonpath='{.spec.releaseName}{"\n"}'

Two things that will bite you

Add a backing service (resource type)

Resource types are the self-service catalog. postgres is the worked example:

$EDITOR platform/openchoreo-config/resource-types/postgres.yaml
kubectl --context k3d-openchoreo-cp apply -f platform/openchoreo-config/resource-types/postgres.yaml

A developer then asks for one in a few lines:

kind: Resource
spec:
  type: {kind: ClusterResourceType, name: postgres}
  parameters:
    database: inkwell
    extensions: [vector, timescaledb]

If your resource type emits a third-party CRD, grant the data-plane agent RBAC for it — the agent is only permitted the API groups OpenChoreo ships with. Without the grant you get Synced=True and OutputsResolved=True while nothing is created; only ResourcesReady=False reveals it. See resource-types/cnpg-agent-rbac.yaml for the pattern.

Deploy and promote

occ component deploy inkwell-web --namespace default --project inkwell            # to dev
occ component deploy inkwell-web --namespace default --project inkwell --to staging
occ resource promote inkwell-db  --namespace default --env staging

Promotion copies the source environment's immutable ComponentRelease pointer into the target's ReleaseBinding.spec.releaseName. Nothing is rebuilt.

Resources behave differently: occ resource promote ignores the pipeline and jumps the target to status.latestRelease.

Roll back

Releases are immutable, so rollback is repointing:

CP="kubectl --context k3d-openchoreo-cp"
$CP get componentrelease | grep inkwell-web        # pick an older one
$CP patch releasebinding inkwell-web-development --type merge \
   -p '{"spec":{"releaseName":"inkwell-web-6549566b69"}}'

Take something out of service

kubectl --context k3d-openchoreo-cp patch releasebinding inkwell-web-development \
  --type merge -p '{"spec":{"state":"Undeploy"}}'   # Active | Undeploy

Lock a commit out of an environment

The only gate here a person sets and a person clears. Capacity, soak, the deploy window and the promotion check are all computed — each answers a question about the world, and none of them can be told "not this one".

platform/templates/lock.sh --component inkwell-web \
  --commit $(git -C apps/inkwell rev-parse HEAD) \
  --to production \
  --reason "Leaks a connection under load; found in staging soak"

platform/templates/lock.sh --list
platform/templates/lock.sh --clear --component inkwell-web --commit <sha>

Omit --to and it covers every environment, which is the honest default for "this commit is bad" — a lock that had to enumerate environments would silently stop covering the next one somebody adds.

It is enforced in two places and you want both. project-flow refuses with a 409 naming the reason and who set it, which covers every promote and every rollback — promote.sh, the Zulip bot, the promote label on a merged pull request and the console button all funnel through one handler. That does not cover occ component deploy --to, a raw kubectl patch of spec.releaseName, or a repository pushing a binding through manifest sync. A ValidatingAdmissionPolicy (openchoreo-config/policies/commit-lock.yaml) stands in front of those, matching on the ComponentRelease names project-flow resolved the commit to.

Rollback is refused too, and that is the difference from the capacity check beside it. Capacity exempts rollback because a rollback returns to a release the environment has already run. "Roll back to the known-bad one" is precisely the mistake a lock exists to catch, and it is the one people make when the good release is two back rather than one.

There is no --force. Capacity offers one because its arithmetic is a projection and a projection can be wrong. A lock is a person's judgement, and a gate another person can force past is a suggestion with extra steps. Clear it — and clearing is recorded.

One commit can produce several releases: a manifest edit re-cuts one without rebuilding anything. --list shows every release a lock resolved to, and locking chess-render@8baa8723 covered four.

A lock names ONE component, and in a monorepo that is a sharp edge. inkwell builds inkwell-web, inkwell-worker and inkwell-nightly from one repository, so a bad commit produces a release for each — and locking inkwell-web@c260ec93 leaves the other two free to promote off the same commit. Found while photographing the promote page: the lock was on inkwell-web, which was not moving, while inkwell-worker and inkwell-nightly were moving on exactly that commit and were offered as normal. Lock each component you mean, and check --list afterwards. A --all-components that resolved the commit across every component in the project is the obvious next thing and is not built.

Gitea

http://localhost:3001    platform / REDACTED
repo contents
apps/inkwell the Rails app, its Dockerfile, and deploy/*/workload.yaml
platform-eng/dagger-modules the reusable CI functions
platform-eng/openchoreo-config platform CRDs

The org is platform-eng, not platform, because org names share a namespace with usernames in Gitea and the admin user is called platform.

Clone with the API token rather than a password:

TOKEN=$(cat infra/.gitea-token)
git clone http://platform:$TOKEN@localhost:3001/apps/inkwell.git

From inside a cluster the host is host.k3d.internal:3001, not localhost.

Webhooks

apps/inkwell has a push webhook to POST /api/v1alpha1/autobuild. To inspect deliveries: repo → Settings → Webhooks → the hook → Recent Deliveries. To re-run the wiring:

./infra/setup-webhook.sh apps/inkwell

When something is wrong

CP="kubectl --context k3d-openchoreo-cp"

# the four conditions that actually matter
$CP get releasebinding <name> -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'

# controller logs
$CP logs -n openchoreo-control-plane -l app.kubernetes.io/component=controller-manager --tail=100

# the build's own logs
kubectl --context k3d-openchoreo-wp logs -n workflows-default <pod> -c main

Known-confusing failure modes are collected in FINDINGS.md. The two that cost the most time:

Following a trace across services

The mesh is instrumented so that one browser request produces one waterfall:

inkwell-web      POST /notes                 Server   (root)
  inkwell-web      INSERT inkwell            Client
  inkwell-worker   default process           Consumer   ← the job, parented not linked
    inkwell-worker   POST                    Client
      inkwell-enrich   POST /embed           Server
    inkwell-worker   POST                    Client
      inkwell-enrich   POST /attachments/12  Server
        inkwell-enrich   S3.PutObject        Client     ← Garage
    inkwell-worker   UPDATE inkwell          Client

In Grafana: Explore → Tempo, or the OpenChoreo overview dashboard, which links to each component's traces directly. Every component of the mesh lives in one project so the waterfall stays whole.

By hand, against Tempo

Worth knowing because the portal's Traces tab is gone — traces live in Tempo now and Grafana's datasource proxy makes them scriptable:

G=https://grafana.jung.town
A='-u admin:REDACTED'

# recent traces
curl -s $A "$G/api/datasources/proxy/uid/tempo/api/search?limit=10" | python3 -m json.tool

# every span in one trace, in order
curl -s $A "$G/api/datasources/proxy/uid/tempo/api/traces/<traceID>" | python3 -c '
import sys, json
d = json.load(sys.stdin)
for b in d["batches"]:
    svc = [a["value"]["stringValue"] for a in b["resource"]["attributes"]
           if a["key"] == "service.name"][0]
    for ss in b["scopeSpans"]:
        for sp in ss["spans"]:
            print("%-18s %s" % (svc, sp["name"]))'

In the UI: Explore → Tempo → Search, filter on service.name. The OpenChoreo overview dashboard links straight there per component.

Service identity is service.name/service.namespace. Components carrying an OTel SDK set it from OTEL_SERVICE_NAME; eBPF-only ones get it from Beyla, which is configured to read OpenChoreo's own pod labels so both agree. Without that config Beyla names workloads after the Deployment and the same platform appears under two naming schemes.

If a trace stops at a service boundary

Before suspecting propagation, check the collector. Alloy runs no sampler, but it does run a drop filter (otelcol.processor.filter "noise" in platform/bootstrap/values/lgtm-alloy.yaml), and a span that matches it never arrives. The historical version of this trap — OpenChoreo's shipped collector dropping spans at 10/sec, per hop — is written up in FINDINGS.md; the symptoms were indistinguishable from a propagation bug.

To prove propagation independently of the collector, capture the wire:

# in the caller's pod: what did it actually send?
ruby -rsocket -e 'srv=TCPServer.new("127.0.0.1",9999)
  loop { c=srv.accept; while (l=c.gets) && l.strip!=""; warn l; end
         c.print "HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n"; c.close }' &
ENRICH_URL=http://127.0.0.1:9999 bin/jobs

A traceparent: 00-<32hex>-<16hex>-01 on the wire means the application is doing its job and the loss is downstream.

Open a pull request and get an environment

Push a branch and open a PR against apps/inkwell. Within a few minutes a comment appears with a URL:

Review environment is up: https://review-pr-7-default.jung.town
Environment `review-pr-7` on data plane `default`, built from `my-branch`.
It is destroyed when this PR is closed or merged.

What was created, and why it is that much:
Environment/review-pr-N pinned to its data plane at creation — dataPlaneRef is immutable
ProjectReleaseBinding creates the cell namespace, and brings the project's ResourceQuota with it
ResourceReleaseBinding per Resource its own Postgres and object store — connections resolve per environment
a forked Component per component in the PR's repo pinned to the PR branch; branch matching is exact-equality, so a fork is unavoidable
a ReleaseBinding for the rest of the project bound at their staging release, not forked

Merging or closing the PR destroys all of it. Watch it:

CP="kubectl --context k3d-openchoreo-cp"
$CP logs -n project-flow deploy/project-flow -f
$CP get releasebinding -n default -l project-flow.openchoreo.dev/owned=true

A review environment is a full stack per PR — roughly 3GB for a Rails project, one pod for a static site — so OC_MAX_REVIEW_ENVS is 2.

Review environments are not in the deployment pipeline. They used to be spliced into staging's promotion targets (the DAG has no word for "a fork of staging"), which is why the portal drew every open PR as an arrow out of staging. Nothing needs them there: OpenChoreo only reads promotionPaths to find the root for autoDeploy, so the pipeline is staging → production, always, and a review environment is reachable from its PR comment, the Environments dashboard and kubectl. A component that should never be forked (a benchmark rig, a one-off job sharing a repo) opts out:

metadata:
  annotations:
    project-flow.openchoreo.dev/skip: "true"

Create a new project from a template

./platform/templates/new-project.sh --template rails-service --name orchard

One command: the Gitea repository, the rendered source, the webhook, the Project, its cell, its database and its components. Templates are rails-service, static-site, zulip-bot and ruby-gem; --dry-run renders into a temp directory and stops.

Two things it deliberately leaves to you, both explained in platform/templates/README.md: the project's wildcard SAN on gateway-cert.yaml, and the promotion.

rails-service is the interesting one. It produces three components from one repository through two pipelines — a Rails service and its worker from /rails via rails-builder, and a documentation site from /docs via static-builder. A change under /docs rebuilds only the site. It also ships endpoints that make each kind of slowness recognisable in a profile; see Diagnosing a slow endpoint.

Promotion

A merge to main lands in staging by itself: staging is the root of the application pipeline, autoBuild builds the commit and autoDeploy binds the release. Nothing is rebuilt afterwards; every promotion is a repoint of spec.releaseName onto the identical content hash.

Production is automatic by default. project-flow reads a policy off the Project:

metadata:
  annotations:
    project-flow.openchoreo.dev/promotion: auto          # auto (default) | manual
    project-flow.openchoreo.dev/soak: 5m                 # Ready in staging this long first
    project-flow.openchoreo.dev/promotion-check: http    # GET the staging URL; anything < 500 passes
    project-flow.openchoreo.dev/production-window: "Mon-Fri 09:00-17:00 America/Los_Angeles"

Every reconcile (30s) it looks at each auto project: if staging holds a release production does not, that release has been Ready in staging for the soak (counted from when it appeared there, not from the Ready condition's timestamp, which a rolling update never moves), the staging URL answers, and the window is open, it calls the same promote() the manual paths use, with promoted-by: auto. Outside the window the promotion is parked with the time it will go, visible in /state and announced in the project's promotions topic. A staging that does not answer is announced once and left alone.

And it asks whether the result will fit — which it did not until 2026-09-06, and the asymmetry was the wrong way round. _promote_or_rollback refused a promotion that would breach the target cell's quota and offered force; the soak timer promoted whatever was ready and asked nothing. The path with nobody watching was the unguarded one.

The check is asked once, for the set the pass is about to promote, and that is the whole reason it lives in auto_promotions() rather than inside promote(). promote() returns as soon as it has patched the binding and does not wait for Ready, so everything promoted in one pass surges at the same time. Measured on chess, whose production cell holds 6Gi with 3840Mi used:

one component     needs  768Mi + 307Mi margin   vs 2304Mi free   fits
all four          needs 3072Mi + 307Mi margin   vs 2304Mi free   does NOT fit

Four per-component checks would each pass and the four together would wedge the namespace. That is the arithmetic capacity_check exists to get right, and it only gets it right if it is asked about the set.

All or none, and no force. capacity_check answers about a set — used already counts what is running and each member is charged its surge on top — so it cannot say which member is the problem, and picking a subset that fits would be a decision nobody asked for. There is no force because there is nobody here to offer it to; instead the refusal is announced once with the arithmetic, each component backs off for ten minutes, and it retries. A cell that is full because something else is mid-rollout empties on its own, and that case must not need a person.

The window binds the person too

The same asymmetry ran the other way for production-window: the soak timer had always honoured it and /promote had always ignored it, so a project saying "production changes on weekday afternoons" bound the robot and not the person — and the person is the one who promotes at 2am. rideshare declares Mon-Fri 09:00-17:00 America/Los_Angeles, so this was live rather than theoretical.

$ platform/templates/promote.sh --project rideshare
HTTP 409 {"error": "rideshare promotes to production during
  `Mon-Fri 09:00-17:00 America/Los_Angeles` and the window is closed;
  it opens Mon 16:00 UTC", ...}

Three deliberate limits, and each is a different answer from the gate beside it:

The console shows it beside the button rather than after it, reading project-flow's /window — the same helper the 409 comes from, so the sentence on the page and the sentence in the refusal are the same sentence.

not-Ready needed nothing

It was already on both paths and with the same test: promote() calls is_ready(src) and the soak timer inlines condition(src, "Ready").status == "True", which is what is_ready is. Two spellings of one rule is how they drift, so the self-test pins that they agree — including on a binding with no conditions at all, where "not ready" and "no answer" must come out the same.

The one difference is shape, and it is deliberate: capacity and the lock refuse up front and move nothing, while not-Ready is discovered per component inside the loop and comes back as retry in a 207. Promoting three of four and saying so is a real outcome for a condition that clears by itself.

chess, rideshare and the walkthrough's showcase are auto; inkwell and orchard are manual, which is where the three spellings below apply:

platform/templates/promote.sh --project inkwell                 # the CLI

@openchoreo promote inkwell                                     # in Zulip

add the `promote` label to a pull request before merging it     # in review

All three POST to project-flow's /promote, which does four things, because doing three of them looks like success and deploys nothing:

1. ProjectReleaseBinding   creates the namespace; nothing else applies without it
2. ResourceReleaseBinding  one per resource; no CLI can create the first one
3. ReleaseBinding          pinned to the exact release staging is running
4. spec.state = Active     neither autoDeploy nor `occ` ever writes this

occ component deploy --to production still works and still does only step 3.

Rolling back

platform/templates/promote.sh --project inkwell --component inkwell-web --rollback

Releases are immutable and never collected, so this is a repoint. The release being displaced is recorded in an annotation on the binding at promotion time — release names are content hashes with no ordering, and nothing else remembers what you were on. A rollback moves code only: if the release you are leaving ran a forward-only migration, the older code now runs against a newer schema.

To change any of this, edit platform/project-flow/controller.py and re-run ./platform/project-flow/install.sh — the code is mounted from a ConfigMap, so there is no image to build.

The repository's openchoreo/ directory is applied on push

Every application repository carries its Project, Components, Resources and staging bindings next to the code, and a push to main server-side applies that directory under the gitops field manager (project-flow, sync_manifests). The commit gets a status, openchoreo/manifests, saying what was created, updated or skipped, and the push's span carries the same line as an event.

What a repository may not do: bind a release. ReleaseBinding is never synced, and ProjectReleaseBinding/ResourceReleaseBinding documents are applied only for the root environment (or development for platform services). Production cells come from a promotion. Re-run a sync by hand:

SECRET=$(cat infra/.webhook-secret); BODY='{"repo":"apps/inkwell","ref":"main"}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $2}')
curl -s -H 'Host: host.k3d.internal' -H "X-Signature-256: sha256=$SIG" \
     -H 'Content-Type: application/json' -d "$BODY" http://localhost:8080/sync

new-project.sh still applies the manifests itself on day one (the first push happens before the hook exists); from the second push on, the repo owns them.

The walkthroughs

@walkthrough static, @walkthrough rails and @walkthrough teardown in Zulip run the platform end to end and narrate every step into #walkthroughs — what was done, the log lines pulled for exactly that step, and the links. See WALKTHROUGH.md.

The build page

https://console.jung.town/builds/<run> is one page per build: the push that caused it, the waterfall, each stage's log expanded on load, the tests and their flame graphs, what the build cached and where its CPU went. It is apps/console. Every build announcement links to it.

apps/buildviewer used to serve a subset of this at builds.devtools.jung.town and is deleted. Every route it had has a console equivalent at the same path shape, the console's is a strict superset, and its hostname was a LAN A record -- so its links worked for the platform and for nobody else.

Wiring a repo up:

./platform/project-flow/setup-hook.sh apps/inkwell

That is a second webhook, deliberately separate from the autobuild one. Never point pull-request deliveries at /api/v1alpha1/autobuild: it picks its provider from which signature header is present, never reads the event type, and a pull_request payload parses into an event with an empty branch that triggers a build for every component in the repo.

Publishing a gem to the internal registry

cd platform/dagger-modules
export GITEA_PACKAGE_TOKEN=$(cat ../../infra/.gitea-package-token)

dagger call -m ./gem-publish check --source=../gems/platform_capabilities
dagger call -m ./gem-publish publish --source=../gems/platform_capabilities \
  --registry=http://host.k3d.internal:3001/api/packages/platform-eng/rubygems \
  --token=env://GITEA_PACKAGE_TOKEN

Gitea does not enforce version immutability — re-pushing a version overwrites it and reports success — so the module checks first and refuses. Bump the version in lib/platform_capabilities/version.rb; --allow-overwrite exists but should be rare enough to explain in a commit message.

Consumers pin the registry as a scoped source so only internal gems come from inside:

source "http://host.k3d.internal:3001/api/packages/platform-eng/rubygems" do
  gem "platform_capabilities", "~> 0.1"
end

The gem-publish module is written with the Python Dagger SDK while ruby, python, frontend and platform use Go — SDK choice is per-module and invisible to callers.

Diagnosing a slow endpoint

Every project created from rails-service ships four endpoints that are all slow and slow for completely different reasons. They exist because a flame graph only becomes readable once you have seen what each shape looks like.

H='Host: production-orchard.jung.town'
B=http://localhost:19080/orchard-web-http

curl -s -H "$H" "$B/diagnostics/cpu?rounds=400000"     # busy
curl -s -H "$H" "$B/diagnostics/memory?rows=30000"     # allocating
curl -s -H "$H" "$B/diagnostics/waiting?ms=1500"       # idle, and slow anyway
curl -s -H "$H" "$B/widgets"                           # 51 queries
curl -s -H "$H" "$B/widgets/fast"                      # the same answer, 2 queries
symptom where the evidence is where it is not
CPU-bound Pyroscope process_cpu: one deep frame, most of the samples —
allocation-heavy container memory, and objects_allocated in the response the CPU flame graph, which is nearly empty
waiting the trace waterfall both profiles, which are flat
N+1 the trace: a wall of SELECT "parts".* under one server span a profile, which barely notices

The middle two are the ones worth internalising. Slow and busy are different claims and a profiler answers the second; the Ruby push SDK reports process_cpu, so an allocation problem shows a nearly empty flame graph and "profiling says it is fine" is the wrong conclusion.

Getting to the trace for a request you just made

curl -si -H "$H" "$B/widgets" | grep -i trace
# X-Trace-Id: 4bf92f3577b34da6a3ce929d0e0e4736
# traceresponse: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Paste the id into Tempo. This is the cheapest observability affordance on the platform and almost nothing had it: without it, "the request I just made was slow" and "this trace was slow" are two facts nobody can join, and you are left searching by service and time window and hoping.

ApplicationController#expose_trace_context sets it, and every bot appends the same link to its replies.

The leak, and showing a fix rather than asserting one

RollupJob keeps an unbounded cache, so the worker's resident memory climbs for as long as it runs. One profile says nothing about that; two profiles an hour apart say everything, which is what Grafana's profile diff is for.

kubectl --context k3d-openchoreo-cp patch releasebinding orchard-worker-production \
  --type merge -p '{"spec":{"state":"Active"}}'
# then set ROLLUP_CACHE_BOUNDED=true in deploy/orchard-worker/workload.yaml,
# push, and watch the same graph go flat.

Asking the platform from chat

@openchoreo status inkwell           what is deployed where, and what built it
@openchoreo where inkwell-web        every environment holding this component
@openchoreo promote inkwell          staging -> production
@openchoreo rollback inkwell inkwell-web
@openchoreo builds                   what the controller is waiting on

status and where read project-flow's /delivery — the same join the Delivery and Environments dashboards use, so the bot cannot disagree with them. promote and rollback POST to the same endpoint the CLI and the PR label use.

where volunteers the thing nobody quite asks: if a component's environments are on different commits, it says so.

The bots talk to each other, and the chat can be reset

#bot-arena is where demo-conductor (a component of the chatops project, apps/zulip-bots/bots/conductor.py) plays a chess game, writes and finds a note, checks the weather and asks for a status every ten minutes. Each conversation is one trace, and its closing message links it. It is the traffic generator for the bots and for chess-engine, which the HTTP traffic generator cannot reach.

To start over -- every message gone, the streams rebuilt, the welcome posted into #welcome, users and bot keys untouched:

export KUBECONFIG=~/.kube/openchoreo.yaml
./platform/zulip-realm/reset.sh

The welcome text is platform/zulip-realm/welcome.md.

Where platform activity is announced

A Zulip stream per project, a topic per kind of activity — builds, promotions, pull requests. A failed build's announcement names the stage that failed, links the console's build page already narrowed to that stage, and shows the last forty lines that stage wrote (read from Loki by ci_stage). Anything that is not production is announced into #platform-sandbox instead, because a review environment posting on every push is how a useful channel becomes a muted one.

Announcing is best-effort and never blocks the work: a promotion that failed because chat was down would be a much worse trade than a promotion nobody was told about. Configure it with infra/.zulip-announcer (email, api key, site — one per line); with the file absent, announcements are skipped and everything else is unchanged.