Skip to content
SHC Docs

Estate app refresh: August 2026

This is the estate runbook for the pass described in Refreshing app packaging. That page is the mechanism; this page is the specific pass on the two live estates, with every current version measured on the boxes on 2026-08-04 rather than inferred from a catalogue head.

Estatespromanager (leader 204.168.129.34), bitspur (leader 37.27.215.210)
Daemonv0.0.16 on both: this constrains almost everything below
Deployments12 each: 8 in tenant corp / env prod, 4 in tenant master / env default
Cataloguefrozen at the shc repo’s SYNCED AT = 2026-08-01T12:00:01Z; the builtin repo re-crawled 2026-08-03T10:24:04Z
Version pinsall 12 are unpinned (app_version is latest or NULL; bitspur’s master/postgres reads 16, which is also unpinned; see below)
Backups of these 12zero, on either estate

Both estates measured identical in every rendered app package, so one plan serves both. Only the deployment row ids and catalogue row ids differ.


A platform roll never re-renders app packaging, so both estates are still running recipes rendered between 2026-07-27 and 2026-08-01. Separately, reviving the task worker (ce 9c805ce, shipping in the next release) drains the backlogged repo-sync tasks, the catalogue un-freezes, and because every deployment is unpinned, latest moves under all of them.

Three findings from this pass change the shape of the work. Read them before running anything.

0.1 A bare shc update does not hold a pin; it destroys one

Section titled “0.1 A bare shc update does not hold a pin; it destroys one”

The documented contract is that a bare shc update <stack> replays the row’s app_version. That is true of the daemon-side resolver in isolation, and true over the API when the request omits version. It is false from the CLI, because the CLI never omits it:

// ce modules/app/commands/update.go
if args.Version == "" {
args.Version = "latest"
}

BuildUpdateBody ships that, phase 1 of the update handler folds it onto the row (if req.Version != "" { current.AppVersion = &v }), and only then does phase 2 call resolveUpdateAppVersion(req.Version, current) against the already-merged row. Measured, not read:

row pinned at "16" → shc update <stack> → row pin becomes "latest",
re-render resolves NEWEST

Consequence for this runbook: once a deployment is pinned, never run a bare shc update against it again. Every refresh command must name the version token explicitly. A bare update silently un-pins and takes the hop.

0.2 A bare-major pin does not freeze the build

Section titled “0.2 A bare-major pin does not freeze the build”

The catalogue predicate is:

// ce daemon/services_seams_repo_crawler.go
func catalogVersionPredicate(version string) (string, []any) {
if version == "" || version == "latest" || version == "stable" {
return "", nil
}
if strings.Contains(version, "+") {
return " AND a.version = ?", []any{version}
}
return " AND (a.version = ? OR a.version LIKE ?)", []any{version, version + "+%"}
}

With ORDER BY a.id DESC LIMIT 1. So a pin of 16 matches 16 and every 16+N, and picks the last row the sync inserted. Only the <upstream>+<build> spelling (16+0, 0.7+2, 2025.4.3+0) contains a +, takes the exact-equality branch, and actually freezes.

This is why bitspur’s master/postgres reading app_version = "16" is not a pin. It floats across every published 16+N.

Row-id order was checked against the estates and ascends with version on four independent apps (huly 1126→1128 = 0.7+0→0.7+2; keycloak 1137→1142 = 26.6+0→26.6+5; traefik 1195→1203 = 3.6.6+0→3.6.6+8; keygen 1143→1146 = 1.7+0→1.9+1), so ORDER BY a.id DESC does mean “newest version” today. It is an insertion-order proxy, not a version comparison: one more reason to pin exactly.

0.3 Setting a pin costs one re-render: there is no metadata-only pin verb

Section titled “0.3 Setting a pin costs one re-render: there is no metadata-only pin verb”

updateHandler always launches phase 2. Phase 1 writes the row, phase 2 re-renders and runs docker compose up -d. There is no verb that writes app_version without re-rendering, and there must not be a direct row write (the estates are live and the migration baseline is append-only).

So “pin first, then refresh” is really “the pin is the first refresh.” The whole plan therefore turns on one question per deployment: is there a published version whose bytes are what this deployment is already running? If yes, the pin re-renders to identical bytes, compose up recreates nothing, and the pin is free. If no, the pin is a refresh and has to be judged as one.

That question was answered by measurement, below.


Yes for corp. No for master. The X409039APPAMB ambiguity is specific to the master-tenant modules, and the reason is mechanical.

install_runner branches on whether the re-render carries a repo name:

if opts.RepoName != "" {
appDir, err = ResolveRepoQualifiedAppDir(opts.RepoName, opts.AppName, opts.AppVersion)
} else {
appDir, err = ResolveAppDirVersioned(opts.AppName, opts.AppVersion, opts.BaseDir, opts.StateHome)
}

and the update path passes RepoName: diskRepo, read from the deployment’s manifest.json sidecar. ResolveRepoQualifiedAppDir joins on the repo name and has no len(res.Repos) > 1 check, so it cannot raise X409039APPAMB. ResolveAppDirVersioned does have that check, and at v0.0.16 it calls the catalogue with an empty repo, hardcoded.

Measured on both estates:

Deploymentmanifest.json repo_namePinned resolve path
corp/affine, corp/huly, corp/keygen, corp/minio, corp/affine-db, corp/huly-db, corp/keygen-dbshcrepo-qualified: safe
corp/license-brokershc-infrarepo-qualified: safe
master/keycloak, master/traefik, master/openobserve, master/postgresabsent (null)unqualified: ambiguous

The master rows have no repo because the master-tenant guard refuses one outright (X403979SYSSRC), and all four module names are carried by two registered repos:

postgres shc:16+0 builtin:16
keycloak shc:26.6+0 … 26.6+5 builtin:26.6
traefik shc:3.6.6+0 … 3.6.6+8 builtin:3.6.6
openobserve shc:0.91.4+0 … 0.91.4+2 builtin:0.91.4

So a pinned lookup for any of the four returns two repos and refuses. Note that corp’s three *-db deployments are also app_name = postgres, and postgres is also one of the two-repo names; they are safe only because their sidecar records repo_name: shc. Assert that before running, not after:

Terminal window
sudo python3 -c "import json;print(json.load(open('/var/lib/shc/deployments/corp/affine-db/prod/manifest.json')).get('repo_name'))"

1.1 The v0.0.16 trap: a refused pin is sticky

Section titled “1.1 The v0.0.16 trap: a refused pin is sticky”

SettleFailedUpdate (the rollback that makes an abandoned update a no-op on the row) does not exist at v0.0.16. The v0.0.16 handler calls svc.Update (not UpdateWithPrior) and keeps no pre-merge snapshot. So on these daemons:

A pin attempt whose phase 2 refuses leaves the new pin written on the row and the deployment armed to render that version on the next successful update.

That is what makes the master apps dangerous rather than merely blocked. A shc update traefik 3.6.6+8 refuses with X409039APPAMB and leaves master/traefik pinned at 3.6.6+8, which the next unattended traefik reconcile would then take. Do not attempt to pin any master-tenant module until the estate is on v0.0.17 or newer, where the exact-match shortcut and the refusal rollback both exist.

Both forms exist at v0.0.16, for install and update, and both carry + safely (probed against the real SplitSourceVersion):

Terminal window
shc update -t corp -e prod affine-db 16+5 # positional
shc update -t corp -e prod affine-db@16+5 # glued; SplitSourceVersion peels it

shc install <app> <version> / shc install <app>@<version> are the same grammar. Two caveats for scripted runs:

  • A pin can only name a version the estate’s catalogue already carries. While the sync is frozen, huly 0.7+3 and postgres 16+5 are published but not present, and naming them fails X404012APPDIR, which at v0.0.16 leaves the pin written (§1.1). Pin only from the present set in §2.
  • Run non-interactively. With no --var on a TTY, shc update opens a prefilled var form. Over nssh.sh there is no PTY so the form is skipped; do not run these from an interactive shell on the box.

2. Per-app assessment, all 12, both estates

Section titled “2. Per-app assessment, all 12, both estates”

manifest.json at v0.0.16 carries only app_name, compose_path, environment, project, repo_name, stack_name, tenant: no app_version and no app_digest (that is the v0.0.17 feature). And the package’s app.yaml never carries the +N build, so the deployment’s own copy cannot name it either.

So the running package was identified by content. The daemon caches each extracted package at /var/cache/shc/repos/<t>/<e>/<repo>/http-<sha256(source)[:16]>, and the deployment’s app/ directory is a faithful copy of it, verified: the only file that differs is vars.yaml, which the render rewrites with resolved values. Every published version of every relevant app was downloaded from the public index, tree-hashed, and matched. The method validated itself on four apps whose downloads reproduced the estate cache hashes byte-for-byte (license-broker 0.1+7, affine 0.26+0, keygen 1.9+0, postgres 16+0).

What that turned up: five of the eight corp deployments are running packages that no longer exist at any published version. Their cache directories are still on disk, and their source URLs are absent from the index. The catalogue has moved underneath them since they were rendered.

StackAppRunning (measured)Still published?Catalogue offers todayAfter un-freeze latest →One-way?pre_update?
affineaffine0.26 tree, image affine:0.26.5no, withdrawn0.26+0 only0.27+2YESno (has post_update)
affine-dbpostgres16 tree, image postgresql:16 (server 16.14)no, withdrawn16+0 only16+5nono → gains one at 16+3
hulyhuly0.7 tree, account:v0.7.426no, withdrawn0.7+0, 0.7+1, 0.7+20.7+3nono
huly-dbpostgresas affine-dbno, withdrawn16+0 only16+5nono → gains one at 16+3
keygenkeygen1.9+0 exactlyyes1.7+0, 1.8+0, 1.9+0, 1.9+11.9+3nono (has post_update)
keygen-dbpostgres16+0 exactlyyes16+0 only16+5nono → gains one at 16+3
license-brokerlicense-broker0.1+7, image held by a --var overrideyes0.1+0 … 0.1+70.1+7 (already head)nono
miniominio2025.4.3 tree, RELEASE.2025-04-03T14-56-28Zno, withdrawn2025.4.3+0 only2025.9.7+0YESno (has post_update)

Nothing currently running carries a pre_update hook. Every gate in this table is one the refresh brings with it. Since a pre_update refusal now abandons the update, an app that gains a gate and then refuses stays on its old recipe rather than going degraded (the safer failure), but that is v0.0.17+ behaviour and these daemons are v0.0.16.

  • affine → 0.27+2: affine:0.26.5 → 0.27.3. Upstream’s affine-migration service runs prisma migrations 103 → 115 ending at 20260714000001_drop_legacy_permission_and_subscription. flags.allowGuestDemoWorkspace appears as false (absent today), so logged-out visitors get a sign-in page instead of a browser-local demo workspace; restore with --var flags.allowGuestDemoWorkspace=true. The pgclient sidecar goes postgres:16-alpine → 16.14-alpine digest-pinned (same major as the 16.14 server; the pgclient-major rule holds).
  • minio → 2025.9.7+0: RELEASE.2025-04-03T14-56-28Z → RELEASE.2025-09-07T16-13-09Z, mc 2025-04-03 → 2025-08-13. Upstream strips the admin Console from the community build. The shipped oidc.plug.yaml says it plainly: the server half of OIDC survives and serves STS AssumeRoleWithWebIdentity to programmatic clients, but oauth_callback has 0 occurrences in this line’s console bundle. redirectUris is registered against a handler that no longer exists. Browser sign-in to MinIO is gone permanently. The console-carrying line is the sibling app minio-classic (published at 2025.4.22+0; not in either estate’s catalogue yet).
  • *-db → 16+5: the server binary does not move: all four postgres containers on both estates already report postgres (PostgreSQL) 16.14 - Percona Distribution, and 16+5 pins that same 16.14 by digest (sha256:d329970c…) instead of the floating :16 tag. What 16+5 adds is a postgres-guard service (8 references; absent in 16+0 and in the v0.0.16 deb tree) that refuses to start a server whose data directory is a different major, plus the pre_update: [pg-upgrade-stage, scale{postgres:0}, pg-upgrade] ladder. The ladder no-ops when the running cluster already matches vars.majorVersion (16 does), but step 2 stops the postgres service as part of every update from 16+3 onward.
  • huly → 0.7+3: no image change at any service. The delta is comment blocks, a raised nginx memory limit (from 128M), and tests/. This is precisely the class of catalogue fix that a platform roll cannot deliver.
  • keygen → 1.9+3: keygen/api:v1.7.2 in both 1.9+0 and 1.9+3. No image change. The deployment’s accountId (55dbdfb5-…) differs from the package default (1fddcec8-…, identical in both builds), which proves it is a persisted --var override and is replayed; a refresh does not touch it.
  • license-broker → 0.1+7: already at head. Its rendered image is license-broker:944493f8@sha256:33c576d8… (confirmed against the live container’s RepoDigest) while 0.1+5/+6/+7 all default to 09fd72f3@sha256:0d766af3…. The only cached license-broker package on the node is 0.1+7, so the deployment was rendered from 0.1+7 and the old image is a persisted --var images.licenseBroker= override, exactly the pattern the package’s own vars.yaml documents. It is replayed; a refresh does not move the broker image.

2.3 ★ affine cannot be pinned: the pin token is mislabelled

Section titled “2.3 ★ affine cannot be pinned: the pin token is mislabelled”

Measured across every published affine build:

Publishedapp.yaml version:images.affine
0.26+00.26affine:0.27.3
0.26+10.26affine:0.27.3
0.27+00.27affine:0.27.3
0.27+10.27affine:0.27.3
0.27+20.27affine:0.27.3
running (withdrawn)0.26affine:0.26.5

shc update affine 0.26+0 reads as “hold affine at 0.26” and deploys 0.27.3: the prisma migration chain and the guest-workspace flip, i.e. exactly the one-way hop the pin was meant to prevent. There is no published affine package that ships 0.26.5.

So corp/affine is excluded from the pin pass. Its exposure cannot be closed by pinning; see §4.

2.4 master: the four, and why they are a separate problem

Section titled “2.4 master: the four, and why they are a separate problem”

Unpinned master deployments do not resolve from the catalogue. With no RepoName, ResolveAppDirVersioned on a newest selector falls through to ResolveAppDir, which walks the static roots first: the deb’s /usr/lib/shc/apps/<app>. The catalogue un-freeze is therefore irrelevant to them; the .deb is what moves them.

Measured against /usr/lib/shc/apps on the box:

StackRunningDeb shipsRe-render wouldpre_update in deb tree
traefik3.6.6, tree identical to the deb3.6.6nothingnone
keycloak26.6, tree differs from deb26.6packaging only, no image changenone
postgres16, tree = the same withdrawn package corp’s affine-db runs16packaging only; no postgres-guard, no pg-upgrade ladder (the deb tree predates 16+3)none
openobserve0.80.20.91.4openobserve v0.80.2 → v0.91.4, dex v2.37.0 → v2.45.1, busybox 1.37 → 1.38.0none

master/openobserve is a live standing exposure, not a future one. It is eleven minors behind what the installed deb would render, its row is unpinned, and the unattended re-render seams that route a system stack through the update path are all master-tenant: modules/app/openobserve_basepath_reconcile.go, modules/app/keycloak_basepath_reconcile.go, modules/app/keycloak_admin_password.go, and modules/cluster/ca_rotation_global.go. Any of those firing today takes the 0.80.2 → 0.91.4 hop with nobody watching. No unattended seam touches a corp deployment, which is the one piece of good news in §4.

And openobserve cannot be pinned to what it runs at any daemon version: 0.80.2 is absent from the catalogue entirely (X404012APPDIR), and the only thing a pin could name is 0.91.4, which is the hop.


3. The order, with what to verify between steps

Section titled “3. The order, with what to verify between steps”

shc update --check and the app.package-drift doctor sweep do not exist at v0.0.16 (modules/app/commands/update_check.go is absent). Every verification below uses a surface these daemons actually have.

A pin whose target bytes equal the running bytes re-renders to an identical compose.yaml, and compose up -d recreates only services whose rendered spec changed, so nothing restarts. Prove it rather than assume it. Before each step, on the leader:

Terminal window
S=corp/keygen/prod # tenant/stack/env
sudo sha256sum /var/lib/shc/deployments/$S/compose.yaml
sudo docker ps --format '{{.Names}} {{.ID}} {{.CreatedAt}}' | grep "$(echo $S | tr / _ | sed 's/_prod//')"

After: the compose hash and every container ID must be unchanged for a free pin. A changed container ID on a step billed as free means the render resolved something you did not predict, stop and re-measure.

Step 0: take a backup, because there is not one

Section titled “Step 0: take a backup, because there is not one”

shc backup ls returns 5 archives on promanager, all pre-uninstall for retired test stacks (vaultprobe, s3front, s3proof-db, s3proof-nc), and 0 on bitspur. Not one of the 12 live deployments has an archive. Eleven of the twelve ship a backup.yaml and can be captured; corp/license-broker does not and cannot.

Terminal window
for s in affine affine-db huly huly-db keygen keygen-db minio; do
shc backup create -t corp -e prod -s "$s"
done
shc backup ls --all-tenants # confirm each one, and note its archive id

Backup stops the deployment (owner ruling: SHC would rather stop than write an untrusted archive), so schedule it. Record each archive id; the rollback column in §3.4 is only real once you can name one.

Step 1: the free pins (six stacks, expect zero restarts)

Section titled “Step 1: the free pins (six stacks, expect zero restarts)”

These pin to a published version whose bytes are what is already rendered. Run them one at a time, verifying with the primitive between each.

Terminal window
# tenant corp, env prod, on the leader, non-interactive
shc update -t corp -e prod keygen 1.9+0 # running tree == 1.9+0 byte-for-byte
shc update -t corp -e prod keygen-db 16+0 # running tree == 16+0 byte-for-byte
shc update -t corp -e prod license-broker 0.1+7 # non-vars identical; image held by --var
shc update -t corp -e prod huly 0.7+2 # compose delta is a template rewrite of
# SMTP_TLS_MODE with identical output;
# vars differ only in comments
shc update -t corp -e prod huly-db 16+0 # compose.yaml identical to running
shc update -t corp -e prod affine-db 16+0 # compose.yaml identical to running

huly-db and affine-db are running the withdrawn postgres tree; 16+0 differs from it in database.v{app,plug,socket,vars}.yaml, eight migrations/*.job.yaml files, tests/ and vars.yaml, but compose.yaml is identical, so the render output is unchanged and nothing recreates. This also normalises all three corp databases onto the one published package keygen-db already runs.

Verify after each: compose hash unchanged, container IDs unchanged, and the pin actually landed:

Terminal window
shc ls -t corp -e prod

Every one of the six must show its exact <upstream>+<build> token. A row still reading latest means phase 2 refused and phase 1’s write was rolled back, or, worse on v0.0.16, that it refused and the write stuck at something else. Read the row, do not infer from the exit code.

Terminal window
shc update -t corp -e prod minio 2025.4.3+0

This one will recreate the minio container, because 2025.4.3+0 adds a digest to the image string (RELEASE.2025-04-03T14-56-28Z → …@sha256:a640662d…) and the rendered image line changes. The bytes do not: the live container’s RepoDigest is already minio/minio@sha256:a640662d97632f7b94e9dee8cbb7da5c20db24879725cb4fac36f1e220cd528a which is exactly the digest 2025.4.3+0 pins. Confirm that before running:

Terminal window
sudo docker inspect --format '{{index .RepoDigests 0}}' corp_prod_minio-minio-1

compose.yaml is byte-identical between the running tree and 2025.4.3+0; only vars.yaml, s3.socket.yaml, the four bucket.v*.yaml files and tests/ differ. One restart buys permanent immunity from the console-removal hop. Take it.

Verify: the S3 API answers, the bucket vapp still resolves for every consumer (affine and huly both hold s3 plugs), and the console still answers, as it must, since the image did not change.

Step 3: stop, and do not go further on v0.0.16

Section titled “Step 3: stop, and do not go further on v0.0.16”

After steps 1–2, seven of eight corp deployments are pinned to exact <upstream>+<build> tokens and the un-freeze cannot move them. That is the whole objective. The actual version refreshes are a separate decision and a later pass, and three things must land first:

  1. The estates must be on v0.0.17 or newer, for SettleFailedUpdate (so a refused update is a no-op on the row), for the exact-match static shortcut (so modules can be pinned at all), and for shc update --check and the app.package-drift doctor sweep (so drift is observable rather than inferred from tree hashes, as it had to be here).
  2. master/openobserve needs a ruling before any system-stack work. It is the only deployment whose unattended re-render is a real version hop, it cannot be pinned to what it runs, and its exposure is live today.
  3. A repo sync, because huly 0.7+3, postgres 16+5, affine 0.27+2, minio 2025.9.7+0 and minio-classic 2025.4.22+0 are published but not present in either catalogue. Confirm SYNCED AT moved before trusting any verdict downstream.
StackRollback after the §1/§2 pinRollback after a version refresh
keygen, keygen-dbnot needed: identical bytes, nothing changed1.9+0 / 16+0 are still published; re-pin. Plus the step-0 archive.
huly, huly-db, affine-dbnot needed: identical rendered composere-pin to 0.7+2 / 16+0. Plus the archive.
license-brokernot neededone-way: no backup.yaml, so no archive exists or can be made. Acceptable: it is stateless, its image is held by a --var override, and its state lives in keygen + the broker’s postgres.
miniore-pin is impossible: the withdrawn tree is gone. But the image is unchanged, so there is nothing to roll back to that differs.one-way: 2025.4.3+0 remains published, so the package can be re-pinned, but a 2025.9.7 MinIO that has rewritten anything in its data dir is not guaranteed readable by the April build. Rely on the step-0 archive, and treat the console loss as irreversible in place: the migration is a minio-classic reinstall, not a downgrade.
affinenot pinned; see §4one-way: prisma 103 → 115 includes drop_legacy_permission_and_subscription. No downgrade. The step-0 archive is the only route back, and it must be verified before the hop, not after.

  • ★ corp/affine: do not pin and do not refresh. Every published affine build ships the 0.27.3 image, including the two labelled 0.26+N, so the pin that looks like a freeze is the one-way hop (§2.3). Leaving it unpinned means the un-freeze arms it, but no unattended seam touches a corp deployment, so nothing takes the hop without an operator typing shc update. The mitigations are procedural, and all three are needed:

    1. never run a bare shc update affine (§0.1);
    2. keep refresh_app_package = false on shc_deployment.affine in both estates’ OpenTofu (the default, but assert it);
    3. if the hop is ever taken, --var flags.allowGuestDemoWorkspace=true restores upstream guest behaviour.

    The real fix is upstream of the estates: the 0.26+N rows are mislabelled and should either be relabelled or withdrawn. Until then no affine pin is trustworthy.

  • corp/minio → 2025.9.7+0: not without a decision. It permanently removes the admin Console and every browser SSO login. If the console is wanted, the path is a minio-classic reinstall (shc uninstall <stack> without --data retains every mount, and ref+vault://…?autogen secrets are fetched, not regenerated, under the same stack name), done before any refresh, not after.

  • All four master modules: do not pin on v0.0.16. The pin refuses X409039APPAMB and, with no SettleFailedUpdate, the refusal leaves the pin written and the stack armed (§1.1). This is strictly worse than leaving them unpinned.

  • master/traefik: nothing to refresh. Its rendered tree is already byte-identical to the deb’s /usr/lib/shc/apps/traefik. A re-render is a no-op; the only thing it could do is churn the container.

  • corp/*-db → 16+5: defer, do not refuse. The server binary does not move (16.14 either way) and the guard plus the pg-upgrade ladder are wanted. But step 2 of that ladder stops the postgres service on every subsequent update, and three consumers (affine, huly, keygen) depend on these. Take it deliberately, on v0.0.17+, with --check available and an archive in hand.