Two environments

Staging beside production, as it happened — ENVIRONMENTS-WALKTHROUGH.md

One project, two environments

What actually differs between staging and production, how the platform lets you say so, and what the console can and cannot show you about it.

docs/VERSIONING-WALKTHROUGH.md is the same shape for a construct changing version. This one is about the other axis: the same release, deployed twice, configured differently.

The substrate had to be built first

The honest starting position was that this could not be demonstrated at all.

inkwell is the project with the most per-environment configuration, and its entire staging side is state: Undeploy — hibernated to save host memory on a five-cluster laptop. An undeployed binding renders nothing, so there is no manifest on one side to compare against, and the compare page said so.

Of the projects live in both environments — chess and showcase — none declared anything per environment. The compare page reported two environments rendering identically, which was true, and useless.

So apps/chess/openchoreo/bindings-production.yaml now declares three differences, and they are three deliberately different classes of knob:

Each is a decision, not decoration. Two replicas because a rollout with one has nowhere to send traffic while the new pod starts. Profiling off because chess-engine's readiness probe runs a real depth-1 search and opens a span per legal move. Retain because the Zulip identity is an account on a real chat server and deleting a binding should not orphan every message it posted.

Staging declares nothing throughout, which is what makes the difference visible as a difference — and is the platform's normal state: 33 of 34 bindings declare nothing at all.

What it looks like

The counts across the top are the whole idea:

2 substantive differences   2 declared on purpose   61 environment-scoped   24 identifiers

61 and 24 are the reason this page exists. Two renderings of the same component differ in eighty-odd fields, and almost all of it is the environment name, the environment UID, a namespace and a content hash — noise that a text diff would report and a person would learn to scroll past. Classifying it is what leaves four rows worth reading.

spec.replicas 1 → 2 carries a declared chip, and the Deployment above it says 1 declared. PYROSCOPE_ENABLED true → false carries another. The two substantive rows are the public hostname dropping its suffix — board-staging.chess.jung.town → board.chess.jung.town — which is the public-hostname trait keying off isProduction, and is exactly the kind of thing a reader should look at rather than scroll past.

?basis=live asks the other question — what is running, rather than what was rendered — and the page says which it drew, because the two can legitimately disagree (screenshot).

The classifier was wrong, and wrong in the direction that hides things

The first capture of this page read 4 substantive, 0 declared — about a project that had just written both differences down.

Two bugs, and both failed toward "look at this" for something somebody had already decided:

• declared_values skipped booleans outright. Every trait toggle on this platform is a boolean, so the single most common way to say "different here on purpose" could never be recognised.
• the rule required the value on each side to have been declared by that side. The ordinary shape is "production says something, staging takes the default" — with 33 of 34 bindings declaring nothing, requiring a declaration on both sides meant the common case could never match.

Both are fixed, with the narrow rule kept narrow: where both sides declare things the stricter both-sides test still governs, and the residual over-claim — an unrelated row that happens to move to the same value — is written down rather than papered over.

The other pages

Adding a trait

There is no "add a trait to a running component and preview it" page. app/compare.py's header names it as the third thing the module is shaped for and says plainly that it is not built.

What the platform does offer is the same question asked before anything exists — the scaffolding preview, where every input is a GET parameter, so the preview is a URL somebody can paste into a review.

Adding slo-alerts and pprof-scrape changes all three components the template creates, and the diff names each field: traits[slo-alerts].name — slo-alerts added. Removing a trait has no preview at all; it is an edit to a manifest and the change shows up as an ordinary release diff afterwards.

The trait selector on that page listed object names until this was written — otel-instrumentation-v1 and otel-instrumentation-v2, as though they were two traits a component could have both of. It cannot: traits are keyed by logical name and fetchAllTraits says so. It offers logical names now, once each.

Adding a component

chess-opening is a real fourth service — an opening book, which answers the first ten moves from a table so the engine is not asked to rediscover them, and which exists here because it is the flat profile next to a search whose shape changes with its input. It was written, built and autoDeployed like any other component.

0 on different releases · 1 only in staging · 0 only in production · 4 identical · 1 uncompared

chess-opening only in staging — not promoted to production 47407a37 → —

The page is careful with the wording: "staging is the source and production the target, so 'only in staging' reads as not promoted yet rather than as missing." Those are different facts and only one of them is a problem.

The render diff does something better than refuse. It projects what production would get and says so in as many words:

PROJECTION — not a rendering. The control plane renders with CEL against the live environment, and there is no dry-run on this platform. This is derived from what staging actually rendered, with the production namespace, environment name and environment-uid substituted in. Anything a ComponentType or trait decides conditionally on the environment cannot be seen from here, and nothing below will hint that it is missing.

It even names the field it cannot know — a ConfigMap called chess-opening-production-env-configs-<hash>, because the hash is of a value that does not exist yet.

The gap is two minutes wide, and these screenshots do not re-capture

chess carries promotion: auto with soak: 2m. chess-opening went Ready in staging at 02:53:58 and project-flow auto-promoted it at 02:56:01. The four screenshots above are a record of that window; re-running the shot list now produces four different pages, all of them correct, because the component is in both environments.

That is worth knowing rather than working around: on this platform the never-promoted state is a transient, not a condition. Seeing it again means a project whose promotion is manual, or catching the soak.

A promotion refused on capacity — which I did not have to manufacture

I said in the previous pass that this could not be photographed without inventing a quota. Adding a fourth component to chess made it real:

Will not fit. Needs 3840Mi of 3072Mi free in dp-default-chess-production-a386b579
(quota 6144Mi, used 3072Mi).

  chess-bot      1 x 768Mi steady, +768Mi while rolling (maxSurge 1)
  chess-engine   1 x 768Mi steady, +768Mi while rolling (maxSurge 1)
  chess-opening  1 x 768Mi steady, +768Mi while rolling (maxSurge 1)  [new in this environment]
  chess-render   1 x 768Mi steady, +768Mi while rolling (maxSurge 1)

  Short by 1075Mi including a 307Mi margin. maxSurge is not optional at
  replicas 1 -- maxUnavailable rounds down to 0, so the new pod must exist
  before the old one goes.

The same page shows chess-opening as new in production, replacing nothing.

And it is a projection about promoting the WHOLE project at once, which is what that button does. The auto-promotion that actually happened moved one component and fit — production settled at 3840Mi of 6Gi with all five pods running. The refusal is not wrong; it is answering a different question, and it says which ("This reads a quota snapshot and cannot see a deploy that lands in the next second").

A finding, from reading why the promotion succeeded anyway. capacity_check runs in _promote_or_rollback — the HTTP handler — and auto_promotions() calls promote() directly. So capacity is checked when a person promotes and not when the soak timer does. That is the same asymmetry the commit lock was written to avoid, and it is the wrong way round: the unattended path is the one nobody is watching. Fixing it belongs in auto_promotions() rather than in promote(), because the handler asks once for the whole set and asking per-component would under-count by N-1 — which the handler's own comment says.

What is still not shown

Removing a trait, or removing a component. Neither has a preview: removal is an edit to a manifest, and what it produces is an ordinary release diff afterwards rather than a page of its own.

A refused promotion for a source that is not Ready. Implemented on both paths, and it needs the platform to be in a state it is not currently in.

A closed deploy window is now shown — see the rideshare promote page. rideshare is the one project here that declares production-window, so capture it outside Mon-Fri 09:00-17:00 America/Los_Angeles or the page correctly shows an ordinary button.

Reproducing this

platform/dev/console-local.sh                                    # capture + serve on :18000
platform/dev/shoot.py platform/dev/shots-environments.txt  docs/screenshots/
platform/dev/shoot.py platform/dev/shots-customising.txt   docs/screenshots/
platform/dev/shoot.py platform/dev/shots-adding.txt        docs/screenshots/   # see the note in it

shots-adding.txt is the exception: it photographed a two-minute window and says so at the top. The other two re-capture.

The capture step now also assembles the k8sresources tree — the endpoint that makes the render diff possible — from the RenderedRelease plus the live data-plane objects. Without it the whole rendered half of the compare page 404s, which reads as the platform being broken rather than as the fake being partial. It synthesises no health: the real endpoint computes one per node and the snapshot cannot, so it omits the key rather than reporting everything Healthy and blinding the page that exists to show trouble.