Credentials on this page are redacted. This is the public export; the real values live in docs/ACCESS.md in the private repository.

Getting at it

Every UI, credential, CLI and API, and which names are public — ACCESS.md

Access — how to get at everything

Every UI, credential, CLI and API in this demo. If something here does not work, that is a bug in this file; tell me rather than working around it.

TL;DR

export KUBECONFIG=~/.kube/openchoreo.yaml     # put this in your shell rc

open https://portal.jung.town     # the portal -- sign in with GitHub
open https://grafana.jung.town/d/oc-delivery/delivery         # every deployment, and what built it
open https://grafana.jung.town/d/oc-component-360/component-360   # one component, everything about it

These work from anywhere, not just this network — see the access model. Sign in with GitHub once and the portal, Grafana, Gitea, Zulip and Argo all accept the same session.

Start at Delivery if the question is "what is running, and what built it", and at Component 360 if the question is "what is going on with this one thing". Between them they carry every link you would otherwise have to construct by hand: the commit in Gitea, the build in Argo, the running URL, and that component's logs, traces, metrics and profiles.

The local-only entry points still exist:

open http://localhost:10081                   # Argo Workflows (builds)
open http://localhost:3001                    # Gitea, platform / REDACTED

The public names, from any device — these go out to Cloudflare and back through the tunnel, and each authenticates at the gateway:

https://console.jung.town     the console — projects, deploys, builds
https://portal.jung.town      the portal
https://thunder.jung.town     the identity provider
https://argo.jung.town        Argo Workflows
https://grafana.jung.town     Grafana

The tailnet is gone. This block used to list *.TAILNET.ts.net MagicDNS names, each on its Service's own port. They resolve nowhere now. Every name above is a jung.town record; which of them reach the internet and which stay on the LAN is the whole model below.

.localhost names resolve to ::1 through systemd-resolved, so browsers reach them with no /etc/hosts editing. curl needs -H 'Host: …' for the gateway-fronted ones over localhost — see Applications.

Web UIs

The portal's canonical login is https://portal.jung.town. Backstage's baseUrl is single-valued and is both the OAuth redirect_uri and the API base the frontend is told to call, so it has to be exactly what you type. Serving on one hostname while naming another gives every request a cross-origin call — and an HTTPS page naming an HTTP base is a mixed-content block on top. Both present as "Sign in failed: Failed to fetch", with nothing in any server log.

The tailnet and localhost origins remain in the CORS list, so those entry points still work for everything that is not the login redirect.

Sign in with GitHub, or with one of the four seeded personas. If the redirect URIs ever drift, re-run ./platform/bootstrap/rerun-thunder-bootstrap.sh — Thunder's setup Job is a pre-install hook, so helm upgrade never re-runs it.

Signing in with GitHub

GitHub is federated through Thunder, not wired into each application. Every service trusts one issuer, https://thunder.jung.town, and Thunder is the only thing that has ever heard of GitHub. Adding a second provider later is a change in one place.

The flow, when you click Sign in with GitHub on the Thunder login page:

portal.jung.town  ->  thunder.jung.town  ->  github.com/login/oauth/authorize
                                         <-  thunder.jung.town/gate/signin?code=...
                                             POST /flow/execute   (code -> token -> identity)
                                         ->  back to the portal with an OpenChoreo session

The callback registered on the GitHub App must be https://thunder.jung.town/gate/signin. It is deliberately not /auth/oauth/github/finish: that endpoint is POST-only and answers a browser redirect with 405 Method Not Allowed. The gate opens GitHub in a popup and reads code off the popup's own URL, so the callback has to be same-origin and return a page.

The allowlist

Self-registration is off on every application (is_registration_flow_enabled: false). With it on, any GitHub account on the internet is provisioned a platform user on first login — which is unacceptable the moment anything is published through the tunnel. With it off, federation can only bind to a user that already exists, which is what makes this an allowlist rather than a suggestion.

Thunder matches on the GitHub numeric user ID, not the email address. This is the single least obvious thing about the whole setup, and getting it wrong produces a login that authenticates perfectly against GitHub and then fails, with nothing in any log to say why. From the v0.28.0 source:

The email is fetched — GitHub returns email: null for a private profile, so the executor falls back to /user/emails for the primary — but it is used only when provisioning a new user. It plays no part in matching.

Keying on the numeric ID is also right on its own merits: it is stable across renames of both the login handle and the email address, neither of which GitHub guarantees.

The list lives in platform/bootstrap/thunder-scripts/66-allowlist.sh as githubId|email|Given|Family|group, semicolon-separated:

ALLOWLIST="${THUNDER_ALLOWLIST:-<github-id>|<email>|<given>|<family>|admins}"

Find someone's ID with no credentials at all:

curl -s https://api.github.com/users/<login> | grep '"id"'

sub is not in the stock openchoreo-user schema, so the script adds it — optional, and not unique, since the four seeded personas have no GitHub identity and sign in with a password.

Add a person, then reconcile:

./platform/bootstrap/rerun-thunder-bootstrap.sh

The script is idempotent — it reports already bound / already in <group> and makes no writes — so it is safe to run on every bootstrap.

The group matters as much as the account. Every human ClusterAuthzRoleBinding matches on claim: groups; a user in no group authenticates perfectly and is entitled to nothing. The four groups are admins, developers, platform-engineers and sres.

What each group can do, as of 2026-09-04. Everyone on the allowlist can fully use the console; there is no deny binding on this install. admins hold actions: ["*"]. developers hold the shipped developer role plus project-author (project:create, project:update) so the scaffold control works, and the two reader roles so cluster-scoped constructs are visible. platform-engineers and sres are as shipped. Local bindings are in platform/openchoreo-config/authz/scoped-bindings.yaml, which also records a deny expression that is deliberately not applied, and why.

Shell access is not one of these decisions. The OpenChoreo API serves no exec endpoint, so the console prints a kubectl exec line rather than running one — your own kubeconfig is the control. The console offers that line to platform engineers and admins anywhere, and to everyone in staging and below.

To see what the platform says about you rather than what this file says, the console's Access page renders GET /api/v1/authz/profile with your token: every action, the resource paths it is allowed and denied on, and any CEL attached. When that page and this file disagree, that page is right.

When it says "User not found"

{"flowStatus":"ERROR","failureReason":"User not found"}

after a successful GitHub authorization is the allowlist doing its job, not a bug — GitHub proved who you are and no local user carries that sub. Add the account's numeric ID to ALLOWLIST and re-run the bootstrap.

If you added someone and it still says this, check the ID, not the email: the email is not consulted during matching, so an entry with the right address and the wrong (or missing) ID fails exactly the same way as no entry at all.

A trap worth recording, if you ever script group membership yourself. PUT /groups/{id} returns 200 and silently ignores a members field — and GET /groups/{id} does not return members at all, so the obvious read-modify-write splices into nothing, reports success, and changes nothing. Membership has its own routes: GET /groups/{id}/members to read, and POST /groups/{id}/members/add to add. The additive route is also the safe one: replacing the list would evict the seeded persona that the chart's own ensure_group put in each group.

Two more, found the same way. A user cannot be updated from what you read back. PUT /users/{id} built from GET /users/{id} fails schema validation: the object validator rejects any attribute the schema does not declare, the read shape is not the write shape, and password is required but is a credential so it never comes back. Every one of those returns the same opaque USR-1019 "Schema validation failed", so the cause is not recoverable from the error — which is why 66-allowlist.sh deletes and recreates rather than patching. And deleting a user does not remove it from its groups: the membership row survives pointing at nothing, so the script purges members whose id 404s before adding anyone.

Personas

Thunder seeds four users with different roles. Logging in as each is the quickest way to see OpenChoreo's authorization model, since the portal shows different surfaces to each.

Where to look in the portal

• Catalog → Component → a component → Runtime Logs / Metrics / Traces tabs (logs and traces are served by the Loki and Tempo adapters in platform/loki-adapter/)
• Catalog → System → inkwell → Traces tab — traces are grouped per project, not per component, so a multi-service trace appears here
• Create Component — the dropdown of CI workflows is filtered by the component type's allowedWorkflows, which is the platform's enforcement point

Applications

Deployed apps are served by the data-plane gateway on :19080, routed by Host header. The generated hostname is <environment>-<project>.<gateway host>, and the gateway host is jung.town — so inkwell's staging web component is at staging-inkwell.jung.town and its production one at production-inkwell.jung.town.

There is no development-default any more. It was a single name for every application back when one namespace held them all; hostnames are per project and per environment now, and platform/openchoreo-config/public/dns-records.yaml is the list of the ones with records.

Over the LAN, no Host header is needed — these are A records at the LAN address, so they resolve everywhere and connect only from this network:

curl https://staging-inkwell.jung.town/inkwell-web-http/health/ready

Straight at the gateway on localhost, which works wherever DNS does not:

H='staging-inkwell.jung.town'
B='http://localhost:19080'

curl -H "Host: $H" "$B/inkwell-web-http/health/ready"
# {"status":"ready","capabilities":{"vector":true,"timescale":true,
#  "continuous_aggregates":false},"search_mode":"semantic"}

curl -H "Host: $H" "$B/inkwell-web-http/notes"
curl -H "Host: $H" "$B/inkwell-web-http/notes/search?q=vector"
curl -H "Host: $H" "$B/inkwell-web-http/activity/hourly"

curl -H "Host: $H" -X POST "$B/inkwell-web-http/notes" \
  --data-urlencode 'title=A note' --data-urlencode 'body=Some text'

The path prefix is /<component>-<endpoint>, generated by OpenChoreo from the component name and the endpoint key in its workload.yaml.

Tailscale is already configured, so the tailnet form above works today from any device on your tailnet.

kubectl

Five clusters, one per plane. Always set KUBECONFIG — the demo deliberately does not touch your existing ~/.kube/config.

export KUBECONFIG=~/.kube/openchoreo.yaml

kubectl --context k3d-openchoreo-cp get components,projects       # control plane
kubectl --context k3d-openchoreo-dp get pods -A | grep dp-        # workloads
kubectl --context k3d-openchoreo-wp get workflows.argoproj.io -A  # builds
kubectl --context k3d-openchoreo-op get pods -n openchoreo-observability-plane

Useful shapes:

CP="kubectl --context k3d-openchoreo-cp"

# what the platform offers
$CP get clustercomponenttypes,clustertraits,clusterresourcetypes,clusterworkflows

# what is deployed and whether it is healthy
$CP get components,workloads,releasebindings
$CP get releasebinding inkwell-web-development \
   -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

# why a resource is not ready (the useful one)
$CP get resourcereleasebinding inkwell-db-development \
   -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'

Status lags by up to 5 minutes. OpenChoreo polls applied resources rather than watching them (requeueAfter ~5m), so a healthy database reads ResourcesProgressing for a while. Do not debug a "stuck" resource for less than five minutes.

The occ CLI

occ login                      # browser OAuth against Thunder
occ project list --namespace default
occ component list --namespace default --project inkwell
occ component get inkwell-web  --namespace default --project inkwell

# builds
occ workflowrun create --namespace default --project inkwell --component inkwell-web

# deploy / promote (there is no `component promote` -- promotion is `deploy --to`)
occ component deploy inkwell-web --namespace default --project inkwell
occ component deploy inkwell-web --namespace default --project inkwell --to staging

# resources promote differently, and jump straight to the latest release
occ resource promote inkwell-db --namespace default --env staging

# secrets
occ secret create generic my-cred --namespace default \
  --target-plane ClusterWorkflowPlane/default --from-literal=token=...

The REST API

105 paths under /api/v1/namespaces/{namespace}/…. The OpenAPI document is vendored at vendor/openchoreo/openapi/openchoreo-api.yaml.

curl -H 'Host: api.openchoreo.localhost' http://localhost:8080/openapi.json | jq '.paths | keys'
curl -H 'Host: api.openchoreo.localhost' http://localhost:8080/api/v1/namespaces/default/projects

Note the webhook endpoint is on a different version prefix — POST /api/v1alpha1/autobuild — and is the only v1alpha1 path.

Observability by hand

Everything is queryable through Grafana's datasource proxy, so one credential reaches all four stores:

G=https://grafana.jung.town
A='-u admin:REDACTED'

# Prometheus: which components are serving traffic
curl -s $A --get --data-urlencode \
  'query=sum by (service_namespace, service_name) (rate(http_server_request_duration_seconds_count[5m]))' \
  "$G/api/datasources/proxy/uid/prometheus/api/v1/query"

# Loki: what labels exist
curl -s $A "$G/api/datasources/proxy/uid/loki/loki/api/v1/labels"

# Loki: one component's logs
curl -s $A --get --data-urlencode 'query={component="inkwell-web"}' \
  "$G/api/datasources/proxy/uid/loki/loki/api/v1/query_range"

# Tempo: recent traces
curl -s $A "$G/api/datasources/proxy/uid/tempo/api/search?limit=5"

Start from the OpenChoreo overview dashboard rather than these — it lists every component with links straight to its logs, traces and RED view.

Tailscale (removed — kept as a record)

None of this works now. The tailnet is gone, every *.TAILNET.ts.net name below resolves nowhere, and platform/bootstrap/05-tailscale.sh is not in the tree. The section is kept because the two operational points in it — proxy-authenticated identities need their own RBAC, and the proxies listen on the Service's own port — generalise to any setup of this shape. Read it as a postmortem, not a guide. The public names in the access model are what replaced it.

Gave every UI a stable MagicDNS name on TAILNET.ts.net, reachable from any device on your tailnet including your phone, and lets kubectl reach every cluster with no port-forward.

You have to do two things in the admin console first:

1. Tailnet policy → tagOwners:

   "tag:k8s-operator": [],
   "tag:k8s":          ["tag:k8s-operator"],
   

2. Settings → OAuth clients → Generate, with scope auth_keys write and tag tag:k8s-operator. Save it as infra/.tailscale-oauth (gitignored):

   CLIENT_ID=k123...
   CLIENT_SECRET=tskey-client-...
   

Then — and note that the first script below does not exist in this repository. The Tailscale operator is installed on no cluster (kubectl get ns tailscale is empty on all four) and the *.ts.net contexts in ~/.kube/openchoreo.yaml are leftovers from an earlier attempt that no longer resolve to anything. This section is the shape of the setup, not a transcript of a working one; the live public path is Cloudflare, above. Nothing in the install depends on it.

./platform/bootstrap/05-tailscale.sh   # NOT PRESENT -- see the note above

# register the API server proxy contexts
for c in cp dp wp op; do tailscale configure kubeconfig openchoreo-$c; done
kubectl --context openchoreo-cp.TAILNET.ts.net get nodes   # links-ok: historical

# gitea is a plain container, not in a cluster, so it goes through the host
tailscale serve --bg --https=443 http://localhost:3001

Two things the operator does not do for you, both already handled by the script:

• RBAC. The API server proxy authenticates you as your tailnet identity, which Kubernetes has never heard of, so every call is Forbidden until a binding exists. That binding used to live in platform/openchoreo-config/tailnet-rbac.yaml, which was removed with the tailnet — this whole section describes a path that no longer exists, and is kept only because the RBAC point generalises to any proxy-authenticated identity.
• Host-header routing. See tailnet-routes.yaml.

The proxies listen on the Service's own port, not 80/443. Hitting port 80 times out with no error, which is a confusing five minutes if you do not know.

Those were verified working at the time. They are not now: see the banner at the top of this section.

If the proxies never appear

The operator joins the tailnet itself but every proxy fails with

requested tags [tag:k8s] are invalid or not permitted (400)

when tag:k8s is missing from tagOwners or is not owned by tag:k8s-operator. Fix the policy, then restart the operators — by that point they are deep in exponential backoff and will otherwise sit idle for minutes:

for c in cp dp wp op; do
  kubectl --context k3d-openchoreo-$c rollout restart deploy/operator -n tailscale
done

The application gateway works differently from the rest: rather than hand-writing routes, the DataPlane's gateway host is pointed at the tailnet so that the hostname OpenChoreo already generates (<environment>-<namespace>.<host>) is one MagicDNS resolves. Adding a component needs no Tailscale change; adding an environment needs one more exposed node.

jung.town — the canonical access model

Everything is served at https://<name>.jung.town on 443, with a real Let's Encrypt wildcard certificate. There are now two classes of name, and the difference is the whole security model:

The shared surfaces are shared so the platform can actually be shown to someone. Every one of them is behind the same Thunder login and therefore the same GitHub allowlist — either because the application authenticates against Thunder itself, or because the gateway does it on the application's behalf.

Why the gateway does the authenticating

Four of those had authentication that was fine on a LAN and unacceptable on the internet: Grafana was admin/REDACTED with anonymous viewing enabled, Gitea and Zulip each had one shared password, and Argo had none at all.

The gateway can authenticate in front of a service — kgateway ships Envoy's OAuth2/OIDC filter, so it is a property of the route: a TrafficPolicy per HTTPRoute, in platform/openchoreo-config/public/gateway-oidc.yaml.

But prefer the application's own OAuth where it has one. Grafana and Zulip started behind the gateway filter and both failed, differently:

• Grafana got CSRF token validation failed from Envoy. The filter sets a nonce cookie before redirecting and matches it against a token in state; Thunder's gate finishes the GitHub leg in a popup rather than a plain top-level navigation, so the cookie does not come back — and the nonce expires after 600s regardless, which a real person signing in to GitHub can exceed.
• Zulip reached its own login and then had to be given a second password, because a proxy in front of an app does not give the app a user.

Both now run their own OAuth against the same Thunder. That is strictly better: one login instead of two, a real per-user identity in each tool instead of a shared admin session behind a gate, and one less thing between the browser and the app. The gateway filter is the fallback for services with nothing usable of their own — today Argo and Gitea.

Either way the allowlist in 66-allowlist.sh is the one place access is granted or revoked, because everything ends up at the same Thunder.

Cloudflare Access would also have worked — and the older claim in this document that it was a paid feature was wrong; the free tier covers 50 users with GitHub as an identity provider. It was still the worse fit: a second allowlist to keep in step with the first, a one-time manual step in a dashboard with no API, and it only protects traffic arriving from the internet. The gateway filter protects LAN access too.

Three services are deliberately not behind it, each for a reason worth knowing before you "helpfully" add them:

• thunder is the issuer. Putting the filter in front of the thing the filter redirects to is a loop.
• portal and wiki already authenticate against this same Thunder. Stacking a second flow means two cookie jars and two redirects for no additional gate.
• observer is not a browser surface. The portal's own JavaScript calls it for the Runtime Logs and Traces tabs, so a browser redirect in front of it would break those tabs rather than secure them. It validates bearer tokens.

Two consequences

git clone https://git.jung.town/... no longer works for a human. A browser redirect flow and a git client do not mix. CI is unaffected: every clone, package push and webhook in this repo uses host.k3d.internal:3001, which never passes through this gateway. For a human, clone over the tailnet name or the LAN address.

A proxied name hairpins. Traffic from this LAN to portal.jung.town now leaves the house, reaches Cloudflare, and comes back through the tunnel. It is correct and slower, and it means these names stop working when the internet does. The tailnet names remain the local-only path and are unchanged.

The applications are not published

The per-project application names and the wildcard stay on LAN-only A records. Being deployed should never be the same thing as being on the internet. To publish one service, give it a public-endpoint resource — which is a decision someone makes per service, and is exactly what the storefront does for shop.inkwell.jung.town.

curl https://staging-inkwell.jung.town/inkwell-web-http/health/ready
curl https://production-inkwell.jung.town/storefront-http/

Project-scoped names

A project asks for a public hostname by declaring a resource:

kind: Resource
spec:
  owner: {projectName: inkwell}
  type: {kind: ClusterResourceType, name: public-endpoint}
  parameters:
    hostname: api          # -> api.inkwell.jung.town

Names are scoped to the project by default, so two teams can both want api and neither has to win. Taking a name at the top of the zone needs scope: zone, which an admission policy refuses unless the resource carries the platform-eng owner label.

Deleting the resource removes the record: external-dns owns creation, drift correction and deletion through ownership TXT records.

Publishing a service to the internet, and taking it back

The records above are LAN-only. To put something on the public internet it goes through the Cloudflare Tunnel instead — cloudflared runs on the data plane with no inbound ports, so this works behind NAT and survives the residential IP changing.

# public: external-dns writes a proxied CNAME to the tunnel
kubectl --context k3d-openchoreo-cp patch resource storefront-public \
  --type merge -p '{"spec":{"parameters":{"publish":true}}}'

# private again: the record and its ownership TXT are removed
kubectl --context k3d-openchoreo-cp patch resource storefront-public \
  --type merge -p '{"spec":{"parameters":{"publish":false}}}'

Only publish things that should be public. With Access dropped, anything routed through the tunnel is reachable by the entire internet. The storefront is a static site with nothing behind a credential, which is why it is the example. Do not do this to the portal, Grafana, Outline, Zulip or Gitea.

Three things that path needs, none of them obvious:

• cloudflared refuses to start if an ingress rule without a hostname precedes another — it would shadow everything after it.
• A CNAME to <tunnel>.cfargotunnel.com only resolves when proxied. Cloudflare resolves that name internally and it has no public address, so grey-clouded the record exists and nothing answers.
• The proxied flag must be providerSpecific on the endpoint. external-dns's CRD source ignores the annotation form its service and ingress sources read. It is conditional on record type, because proxying the LAN A records would turn working internal hostnames into 522s.

Setup: ./platform/bootstrap/30-cloudflared.sh, with a Cloudflare API token in infra/.cloudflare-token carrying Tunnel and DNS edit permissions. It fails closed with the exact permission list if run without them.