Credentials on this page are redacted. This is the public export; the real values live in docs/ACCESS.md in the private repository.

Secrets

Application secrets, CI secrets, and the cross-cluster path — SECRETS.md

Secrets, three ways

Three different problems that all get called "secrets management", and how each is solved here.

1. Application secrets — generated, never written down

A secret the application needs but nobody should ever choose: a session key, an encryption key, a database password.

These are generated in the data plane by External Secrets, so the value never transits the control plane, never appears in a manifest, and never exists anywhere a human could copy it.

traits:
  - kind: ClusterTrait
    name: generated-app-secrets
    parameters:
      primaryEnvName: SECRET_KEY
      secondaryEnvName: UTILS_SECRET
      hexOnly: true

refreshInterval: "0" on the ExternalSecret is load-bearing: the generator is stateless and returns a different value on every call, so any refresh would rotate the key out from under a running app. For Outline that would invalidate every session and every encrypted field at once.

The same mechanism serves charts. helmchart generates values and injects them into the chart's values tree through Flux's valuesFrom, which is how Zulip gets a SECRETS_secret_key and its Postgres/RabbitMQ/Redis passwords without any of them being in this repository.

2. CI secrets — a credential the build needs

apps/enrich is a private Gitea repository. Cloning it needs a credential the build has and this repo does not.

# a read-only, repo-scoped deploy token (Gitea requires basic auth to mint one --
# it refuses to create a token when authenticated with a token)
curl -u 'platform:...' -X POST http://localhost:3001/api/v1/users/platform/tokens \
  -H 'Content-Type: application/json' \
  -d '{"name":"ci-read","scopes":["read:repository"]}'

# publish it to the workflow plane's secret store
occ secret create generic enrich-git --namespace default \
  --category git-credentials \
  --target-plane ClusterWorkflowPlane/default \
  --from-literal=username=platform --from-literal=password="$TOKEN"

The component then names it, and never holds it:

repository:
  url: http://host.k3d.internal:3001/apps/enrich.git
  secretRef: enrich-git
  allowInsecureAuth: true

allowInsecureAuth is ours, not upstream's. OpenChoreo's shipped checkout step applies git credentials only to https:// URLs; for anything else it prints a warning and configures nothing, so the clone falls through to prompting for a username and fails with

fatal: could not read Username for 'http://...': No such device or address

which never mentions authentication. That default is right — the token would cross the wire in cleartext — so rather than removing the guard, platform/openchoreo-config/workflows/checkout-source.yaml adds an opt-in that has to be set per run and is recorded on the WorkflowRun.

3. Cross-cluster — one secret, many clusters

Each k3d cluster sits on its own Docker network and cannot reach the others, so an OpenBao inside the data plane is reachable only from the data plane. That forces a separate store, and a separate copy of every secret, per plane.

Instead a single OpenBao runs on the host (infra/compose.yaml) beside Gitea and the registry mirrors, reachable from every cluster at host.k3d.internal:8200 — the same shape as a real deployment, where the secret manager is a service the clusters consume rather than something each cluster owns a copy of.

What makes it more than a shared password is a Kubernetes auth mount per cluster:

auth/kubernetes-cp   -> validates against host.k3d.internal:6550
auth/kubernetes-dp   -> ... :6551
auth/kubernetes-wp   -> ... :6552
auth/kubernetes-op   -> ... :6553

OpenBao verifies each presented ServiceAccount token against that cluster's API server, so the data plane authenticates as the data plane and cannot assume the workflow plane's identity, even though both read from one store.

Proof, and the thing worth running yourself:

# write once
curl -H "X-Vault-Token: REDACTED" -X POST -d '{"data":{"message":"hello"}}' \
  http://127.0.0.1:8200/v1/secret/data/platform/shared-demo

# read from two clusters, each authenticating as itself
for c in dp wp; do
  kubectl --context k3d-openchoreo-$c get secret shared-demo -n external-secrets \
    -o jsonpath='{.data.message}' | base64 -d; echo
done

Both print the same value. Setup is platform/bootstrap/35-shared-secrets.sh.

The DR cluster has its own mount, auth/kubernetes-dr against host.k3d.internal:6554. openchoreo-dp2 is currently stopped, so nothing is listening on 6554 and that mount cannot validate a token today — start the cluster before testing the failover half of this. The mount itself is configured and needs no change.

When it is running, the same secret is readable from dp2 after a failover with no extra copy and no re-issuing.

What failover actually does to secrets

This is the part worth reading before trusting a failover drill, because the platform has two kinds of secret and they behave in opposite ways.

Shared secrets are identical everywhere. Anything read from the OpenBao on the host resolves to the same bytes on dp and dp2. A Zulip bot token (resource-types/zulip-bot.yaml) is minted once against the Zulip API, stored in the shared store, and works on either cluster — the bot keeps its identity across a failover.

Generated secrets are per-cluster by construction. The generated-app-secrets trait uses an ESO Password generator with refreshInterval: "0" and creationPolicy: Owner. The generator is stateless: it returns a fresh random value on every call, and "sync once" means once per cluster. Failing over does not copy the value — the DR cluster generates its own.

For most components that is harmless: the secret is a per-instance session key that nothing outside the pod has seen. For Outline it is not. Its SECRET_KEY and UTILS_SECRET encrypt data at rest and sign sessions, so a DR cluster that generates its own pair comes up unable to decrypt the rows the primary wrote — the database replicated cleanly and is now unreadable. The failure looks like application-level corruption, not a secrets problem.

The rule that falls out of this:

A generated secret is safe to regenerate only if nothing outside the process that holds it depends on its value. The moment it encrypts persisted data or signs something a client keeps, it belongs in the shared store.

Where our components sit today:

Outline is the one to move before a real failover, by publishing both values into the shared store and switching its component from the generator trait to a SecretReference. It is left as-is deliberately so the failure mode is demonstrable rather than theoretical.

To see which side of the line a component is on:

kubectl --context k3d-openchoreo-dp get externalsecret -A -o json | python3 -c '
import sys, json
for e in json.load(sys.stdin)["items"]:
    sp = e["spec"]; df = (sp.get("dataFrom") or [{}])[0]
    gen = ((df.get("sourceRef") or {}).get("generatorRef") or {}).get("kind", "-")
    print("%-46s %-10s %s" % (e["metadata"]["name"][:46],
                              (sp.get("secretStoreRef") or {}).get("name", "-"), gen))'

A GEN of Password means per-cluster. A STORE of shared means identical everywhere.

Two traps

occ secret create reports success when the value goes nowhere. If the target plane has no usable ClusterSecretStore, the SecretReference is created on the control plane, the CLI says Secret 'x' created, and the value is never written. The failure surfaces much later as an ExternalSecret in SecretSyncedError on a different cluster.

The writer policy needs create/update on secret/metadata/*, not just secret/data/*. OpenChoreo pushes through an ESO PushSecret, which writes custom metadata alongside the value. With read-only metadata the push fails with could not write remote ref ... PUT /v1/secret/metadata/... while the CLI has already reported success.

Two things that have cost hours

Neither is a design point; both are operational facts that look like something else when they bite.

OpenBao is in-memory

A reboot empties the entire secret store, silently. Nothing reports it as a secret problem: every build then fails to clone, with an error naming neither the token nor the store.

platform/bootstrap/restore-shared-secrets.sh --apply

Run that after any restart of the data plane, before concluding that Gitea or the build image has broken.

The webhook secret is in git history, and is dead

infra/.webhook-secret was committed twice, before .gitignore covered it — and .gitignore never untracks. The value is in two old commits and will stay there.

It was rotated on 2026-09-03 (infra/rotate-webhook-secret.sh), so what is in history is a dead credential. Rewriting ninety commits to remove it would change every SHA for no security gain.

Rotation keeps the previous value beside the new one as infra/.webhook-secret.*.bak, which is gitignored for the obvious reason: committing that would recreate exactly the problem rotation exists to fix.

Nothing else has ever been tracked. .gitea-token, .github-oauth, .cloudflare-token, .zulip-announcer and tailscalecreds.env are gitignored and have never been in a commit.