Skip to content
SHC Docs

shc update must say what it did

Status: built; this note is still its only description. Implemented (ce fix/update-noop-honesty, extended by §8 in ce lane/three-honesty-defects), except §5 which is a recommendation with a measurement behind it.

Origin: two live measurements on the night of 2026-08-10/11, on two estates and two different apps.

estatecommandprintedactually happened
promanagershc update traefik -t masterupdated stack "traefik", exit 0container unchanged at 15h uptime; rendered compose byte-identical; the release’s traefik plugin never reached the node
bitspurshc update windmill -t corp -e produpdated stack "windmill", exit 0containers unchanged at 42h uptime; moved only when handed 1.784.0+0 explicitly, and only after shc repo sync shc (index was stale, 769 → 784 apps)

Two lanes spent a night on the gap between those two columns.


Three distinct defects share one symptom. They are listed in the order the command hits them.

1a. A bare update replays the deployment’s pin (correctly) and never says so

Section titled “1a. A bare update replays the deployment’s pin (correctly) and never says so”

shc update <stack> with no version is defined as “re-render this deployment as it is configured”. resolveUpdateAppVersion (ce/modules/app/system_app_pin.go) replays deployment.app_version, and ExecuteCompose re-renders that same package. When nothing else moved, the compose comes out byte-identical and docker compose up recreates nothing.

That is correct behaviour. The defect is that the command cannot tell you it happened. The completion payload carried {"success": true}, which means no error was raised, and the CLI’s result sentence (ce/modules/app/commands/operation_sse.go) printed the past tense of the verb the operator typed. A no-op and a successful upgrade were byte-identical on stdout and in the exit code.

This is the whole of the bitspur windmill case.

1b. A version-string edit strands every deployment pinned to the old string

Section titled “1b. A version-string edit strands every deployment pinned to the old string”

resolveAppDirVersionedProv (ce/modules/app/source_qualifier.go) resolves a pinned version to the node’s shipped tree only on an exact string match:

if dir := firstStaticAppDir(appName, baseDir, stateHome); dir != "" &&
declaredAppVersion(dir) == version {
return dir, ResolvedProvenance{}, nil
}

ce/builtin/traefik/app.yaml declared version: "3.6.6" from 2026-07-26 until 2026-08-08, when commit 65eb0e7 corrected it to "3.6.25", a manifest fix, because no build of the package had ever shipped a 3.6.6 Traefik. Every system traefik installed before that date carries the pin 3.6.6.

From 65eb0e7 onward the match fails, resolution leaves the shipped tree, and the deb’s drop is never read again by that deployment, however many releases ship. shc update --check reports CURRENT because both halves of its comparison resolve the same non-deb package, and the deb’s drop being correct on disk is not a contradiction of that: it is the diagnosis.

The class is general. Any catalogue app_version string edit does this to every deployment pinned to the old string.

1c. A pin the catalogue cannot serve was silently substituted

Section titled “1c. A pin the catalogue cannot serve was silently substituted”

The last branch of resolveAppDirVersionedProv (“not in the catalog at all”) delegated to the unversioned resolver, which walks the static roots and returns the shipped tree. Read the reachability: the exact-match short-circuit above it already returned when the shipped tree declares the pin, so this branch can only be entered when the shipped tree declares something else. Its one possible effect was therefore to render a version the operator had pinned away from, under the pin’s name, writing nothing anywhere. The comment justified it with the daemonless-CLI case, which is unreachable here; a nil CatalogAppDirResolver short-circuits at line one.


2. The honesty fix (the one that matters most)

Section titled “2. The honesty fix (the one that matters most)”

A verb must report its effect, not its attempt.

ce/modules/app/deploy_effect.go records what a render actually did, and ComposeRunOptions.EffectOut carries it out of ExecuteCompose:

facttaken
AppDirthe package directory the render actually read
PriorPackageDigest / PackageDigestapppkg identity before / after
PriorComposeSHA256 / ComposeSHA256rendered compose bytes before / after

Verdict() is tri-state (changed / no-op / unknown) and routes the digest comparison through apppkg.Comparable, which refuses absence. Two empty digests compare equal, so the natural spelling of this comparison would report a confident “nothing changed” precisely when the render knew the least. That is the same anti-absence rule package_identity.go already enforces, applied to a second comparison.

Both facts are compared, because either alone lies: same package + different vars is a real change the package digest misses; a doc-only package edit is not a change to the running stack.

The verdict rides the operation’s terminal frame and the audit row, and the CLI prints it, in the live TTY view too, where there used to be no result sentence at all:

traefik: NO CHANGE: nothing was updated; the re-render read the same app
package and produced the same compose the deployment already had; package
apppkg-v1:5dd22040… read from /usr/lib/shc/apps/traefik. A bare update
replays the deployment's own version pin, so if you meant to take a NEWER
package, name the version (`shc update <stack> <version>`); if the version
you want is not offered, this node's catalogue index may be stale;
`shc repo sync` is read-only and re-renders nothing.

The exit code is unchanged. A no-op is not a failure, and converge-shaped callers (terraform, the provider converge, CI) must keep working. What changed is that the command says what it did.

An absent verdict (a daemon predating this change) keeps the old sentence. Reading absence as “no-op” would print “nothing was updated” for every update against an older node, the same anti-absence rule at the wire.

X409042APPVSUB replaces the silent substitution of §1c: a pin the catalogue cannot serve, on a node shipping a different version, is refused naming both versions, with both remedies (shc repo sync first; it is read-only).


3. The resolution fix: a system app re-resolves to the deb’s drop

Section titled “3. The resolution fix: a system app re-resolves to the deb’s drop”

applySystemAppVersionRefresh (ce/modules/app/system_app_pin.go) is applySystemAppVersionPin’s update-lane twin. On a bare update of a canonical system stack it moves the pin onto the version this node’s own package ships, announced by X200972SYSDEB naming both versions.

This honours the [[system-pin-ruling]] rather than reversing it. The ruling’s stated purpose is that a re-render must not silently start a new-major server against an old-major data directory. It is a rule about silence and about the data, not about freezing a string:

  • it is not silent: X200972SYSDEB names both versions before the render;
  • it is not the catalogue head: the version taken is ShippedAppVersion, static roots only, no network: the artifact apt install shc just placed, which is what every system seam is built against;
  • the data-directory guard is untouched and still upstream of any harm; this lane fires pre_update, which is where postgres-guard refuses;
  • it is the system stacks only. IsCanonicalSystemStack’s four clauses keep an operator’s own shc install postgres -t master --name mydb untouched, as the standing unpinned means unpinned, no auto-pin ever ruling requires. An unpinned system stack also stays unpinned; the newest selector already walks the static roots first, so it already lands on the deb’s drop and a new auto-pin would have no defect to justify it.

Leaving the pin frozen is not the neutral option. It strands the control plane on a package the node no longer carries, reports that as CURRENT, and makes every future platform fix hollow, which is the outcome the ruling exists to prevent, arrived at from the other side.


4. shc update --check now names the tree it read

Section titled “4. shc update --check now names the tree it read”

PackageDrift.ResolvedFrom carries the directory the catalogue half resolved, and --check prints it on every verdict:

traefik: app package is CURRENT; `shc update` would re-render the same package
app: traefik 3.6.25
running: apppkg-v1:5dd22040…
catalogue: apppkg-v1:5dd22040…
read from: /usr/lib/shc/apps/traefik

“CURRENT” and “the deb dropped new bytes” read as a contradiction without that line, and as a diagnosis with it. It is one line and it is the single measurement that would have ended the promanager investigation immediately.


Section titled “5. Should a bare shc update <app> imply a catalog sync? No. Recommended instead: report, don’t fetch.”
measurement
public index (selfhostedcloud.org/apps/index.yaml)744 KB, 90 apps, ~1.2 s fetch (3 runs: 1257 / 1229 / 1169 ms from a warm box)
a real syncphase 1 of syncHTTPIndexRepo calls fetchHeadArchives(ctx, &m, repoURL, idx.Apps); it downloads every app’s head .app archive before writing a row
DB costphase 2 issues applicationDeletes(repoID) and reinserts the repo’s entire application row set; on the estates that is an rqlite cluster write of ~800 rows

So the index fetch is the cheap part and it is not the operation. A sync is 90 archive downloads plus a full catalogue rewrite, per update.

The correctness argument, which is the stronger one

Section titled “The correctness argument, which is the stronger one”

A bare shc update <stack> means “re-render this deployment as it is configured”. An implicit sync would make the same command mean different things depending on what the index published in the last minute. For an unpinned deployment it would literally mean “take whatever head is now”, the silent acquisition the codebase already refuses to do quietly (X200971UPDHED exists to warn about exactly that). Making a mutation’s outcome depend on a network fetch the operator did not ask for is a larger footgun than the stale index, and it is one that fires on production data.

  1. Say the no-op out loud: done (§2). The bitspur operator’s actual problem was not the stale index; it was that the command told them nothing. The no-op sentence now names both remedies, and names shc repo sync as read-only so it is safe to reach for first.
  2. shc update --check resolves the catalogue without mutating and now names the directory (§4). That is the read-only “what would happen” verb, and it is the right place to spend the network.
  3. Optional follow-up, not built here: an opt-in shc update --sync and a per-repo index-age warning on the update lane (the repo row already carries SyncedAt; the seam needed is a RepoSyncAge lookup inside modules/app). Recommended as a small, separate change; it is additive and does not alter what a bare update means.

ce/modules/app/update_says_what_it_did_test.go and ce/modules/app/commands/operation_change_line_test.go. Every assertion names the mutation it dies on; all were run against a deliberately broken tree:

mutationreds
Verdict compares digests with == instead of apppkg.Comparable4 rows of TestTheVerdictRefusesAbsence (absence became agreement)
the router drops the effect fields from the terminal frameall 3 rows of TestANoOpUpdateIsReportedAsANoOp
the silent delegation is restored in resolveAppDirVersionedProvTestAPinTheCatalogueCannotServeIsRefusedNotSubstituted
the applySystemAppVersionRefresh call is deleted from Service.updateTestTheUpdateDoorWritesTheRefreshedPin
the refresh’s IsCanonicalSystemStack clause is dropped2 rows of TestASystemStackRepinsToTheDebsDrop (a user’s deployment gets auto-moved)
the CLI’s no-op arm is removedTestTheResultSentenceNamesTheEffectNotTheAttempt, TestTheSentenceFallsBackToAStackNameItAlwaysHas
the CLI folds unknown into the no-op armTestTheResultSentenceNamesTheEffectNotTheAttempt
the CLI reads an absent verdict as no-op2 rows of the same
the prior compose hash is taken after WriteRenderedComposeTestTheBeforeHalfIsTakenBeforeTheOverwrite

The last one is a source-level ordering guard on purpose: ExecuteCompose needs a docker daemon, so no unit test in the package can drive it, and reading the prior hash after the overwrite would make every update report no-op, silently. A guard that only fires under a docker daemon is a guard that never fires.

  • The promanager traefik deployment’s actual state. §1b is the mechanism the evidence fits and it is now closed in code, but the estate-side confirmation (which directory that node’s daemon resolved) was not taken by this lane. With §4 shipped it is one command: shc update traefik --check -t master now prints read from:.
  • system.apps.<app>.vars.images.* config rows destroy the persisted deployment var on daemon-driven recreates. A hand-set image override on promanager’s traefik would pin the container image regardless of which package resolves, and would show up here as changed on the package with no change to the running container. Out of scope for this branch; it is a separate defect on the same estate.

8. The third fact: a var that feeds a job (2026-08-15)

Section titled “8. The third fact: a var that feeds a job (2026-08-15)”

Status: implemented (ce lane/three-honesty-defects). This EXTENDS §2; it is the same doctrine reaching a change the two original facts are structurally blind to.

shc update mailcow --var ssoOnly=true
mailcow: NO CHANGE: nothing was updated; the re-render read the same app
package and produced the same compose the deployment already had …

exit 0, and the deployment’s manifest came out carrying ssoOnly: "true".

ssoOnly feeds the oidc consumer plug’s jobs.post_deployment env and nothing else (apps/flagship/mailcow/oidc.plug.yaml). It never reaches compose.yaml, and it does not change which app package is read. So both halves of the §2 verdict (package identity and compose bytes) are equal by construction on the very run the operator cares about, and “both known, both unchanged” fired.

This is the doctrine’s own anti-absence rule turned up one level. §2 refuses to read a missing comparison as agreement. Here both comparisons are present and both agree, and the verdict is still false, because the state that moved is in neither of them. A verdict that reads only the compose is a verdict about docker, not about the deployment.

The result was worse than the silence §2 replaced: the deployment was DECLARED ARMED and REPORTED UNARMED. The operator re-types the command or gives up, while the persisted override means a later, unrelated reconcile arms the flag unattended.

DeployEffect.ChangedVarKeys is the third fact, recorded at the ONE line where both sets exist: install_runner.go’s MergeVarOverrides(priorState.VarOverrides, …) seam, immediately after the merge (before it, opts.VarOverrides is still the request’s set and every replayed override reads as moved). KEYS, not values: the list rides the wire to the terminal, the audit row and -o json.

Verdict() consults it FIRST, ahead of both existing halves, because it is positive evidence. The two halves below it are comparisons that can come out UNKNOWN; a recorded var move is a fact this render established about state it just persisted. Ordering it after them would ask the two facts that cannot see the change to rule on it.

A bare update replays the persisted set unchanged, so nothing is recorded and the §2 no-op sentence keeps firing for the run that genuinely did nothing.

The sentence, because “CHANGED” alone is the other wrong answer

Section titled “The sentence, because “CHANGED” alone is the other wrong answer”

An operator handed a bare updated stack "mailcow" goes looking for the container recreate, and, finding none, reaches the same conclusion the false NO CHANGE produced. So the var-only move is the one CHANGED run that carries a reason:

mailcow: CHANGED: var ssoOnly moved and the deployment now carries it. The
app package and the compose bytes are UNCHANGED; this var reaches the
deployment through its integration jobs, not through compose, so no container
was recreated and none needed to be. If the reconcile that consumes it
failed, that failure is named on this same line

The CLI arm fires on the reason’s PRESENCE, not on a second flag: an ordinary changed run sends no reason and keeps its existing sentence, and a daemon predating the reason keeps the old sentence too, the same anti-absence rule at the output surface. The last clause is not decoration: §2’s failed-job names ride this same frame, so the two halves of “what did this run do” print together.

The exit code is unchanged, as in §2.

The obvious optimisation (“the compose is identical, so skip the consumer plug reconcile”) is exactly what would make a var-only change a declared-armed, never-armed deployment. ExecuteCompose therefore never branches on its own change verdict: it RECORDS the effect and runs the reconcile unconditionally, and judging what that adds up to belongs to the surface that reports it. TestTheReconcileIsNotGatedOnTheChangeVerdict is a source-level guard on precisely that (same reasoning as §6’s last row; the function needs a docker daemon, so a runtime guard here is a guard that never fires).

ce/modules/app/var_only_change_is_a_change_test.go and the new row in ce/modules/app/commands/operation_change_line_test.go.

mutationreds
the len(e.ChangedVarKeys) > 0 arm is deleted from VerdictTestAVarOnlyChangeIsAChangeNotANoOp, TestTheMovedVarKeysRideTheWire
RecordVarOverrideMoves drops its equality skip4 rows of TestRecordVarOverrideMovesNamesOnlyWhatMoved (a bare update’s replay becomes a change)
the sort is droppedthe multi-key row of the same
WireFields drops the changed_var_keys foldTestTheMovedVarKeysRideTheWire
WireFields returns the effect’s own slicethe aliasing assertion of the same
the record is moved ABOVE the mergeTestTheVarMovesAreRecordedAtTheMergeSeam
the reconcile is wrapped in if …Verdict() != EffectNoOpTestTheReconcileIsNotGatedOnTheChangeVerdict
the CLI’s changed arm is removedTestTheChangedSentenceNamesTheVarThatMoved
that arm’s reason == "" guard is droppedTestTheResultSentenceNamesTheEffectNotTheAttempt, TestTheChangedSentenceNamesTheVarThatMoved

The two sibling defects reported with this one, both already closed

Section titled “The two sibling defects reported with this one, both already closed”

Named here so the next reader does not re-open them:

  • A failed integration job swallowed by a green update: closed by §2’s FailedIntegrationJobs (deploy_effect.go), the CLI’s operationJobFailureLine, and operationJobFailureError, which makes the exit code non-zero. That last part is the deliberate asymmetry: a no-op is not a failure, an unconverged edge is. On the update door only; see §9.
  • shc adapter migrate --from <tenant>/<stack>@<env> ignoring the @env: closed in ce/modules/adapter/commands/migrate_from.go: stackSource cuts the @env suffix and threads that environment through to Pause, and the CLI pauser routes through node.ComposePauseVerified, which reports an EMPTY container set as failure rather than the exit-0 docker compose pause gives it. Covered by TestStackSourceEnvSuffixAndFallback and TestComposePauseVerifiedFailsWhenTheProjectHasNoContainers.

9. The other door: shc install (2026-08-16)

Section titled “9. The other door: shc install (2026-08-16)”

Status: implemented (ce lane/install-door-says-what-it-did). This EXTENDS §2 the way §8 did, same doctrine, same fields, the door §2 did not reach.

The measurement is §8’s own closing note, re-read

Section titled “The measurement is §8’s own closing note, re-read”

§8 records the failed-integration-job swallow as closed. Verified by content, it was closed on the update door only:

doorbuilds a DeployEffectterminal frame carries the failed jobsexit code
shc updateRunUpdateWithObserver → (*DeployEffect, error)effect.WireFields() folded in updateHandlernon-zero
shc installnonenothing0

RunInstallWithObserver returned a bare error and built no effect, so ExecuteCompose ran the entire consumer-plug reconcile with opts.EffectOut == nil. RecordFailedIntegrationJob is nil-safe by design (the daemonless and test paths carry no effect) so every failure on the install door was recorded onto that nil and discarded without a word. The terminal frame was a bare {"success": true, "stack": …, "environment": …}, and the CLI printed installed stack "x" and exited 0.

The CLI half needed nothing: verbNounOr already maps "installed" → "install", and operationJobFailureLine / operationJobFailureError already fire off failed_integration_jobs regardless of verb. The frame was the only thing missing.

An update over a failed reconcile leaves the previous working configuration in place; the estate keeps whatever it had. A first install over a failed reconcile leaves an app that has never had its OIDC client secret written at all, and reports it as installed. It is the bitspur frappe-set-social-login-key shape with no prior good state to mask it.

RunInstallWithObserver now returns (*DeployEffect, error), allocating the effect up front and returning it on every path, mirroring RunUpdateWithObserver, for the reason that function already states. The dispatched-elsewhere path returned an empty effect: the work happened on another node, and claiming this one’s silence as a clean bill would be the anti-absence rule broken across a network hop. Since SHC-2426 it returns the effect that node answered, and the empty one only when the answer carried none.

RunInstall is the discarding wrapper, exactly as RunUpdate is for the update lane, and carries the same warning. It routes through the InstallDeploy package var rather than calling the runner directly, so the placement lane’s test seam still describes a call that happens.

JobFailureWireFields, and why the install door does not take WireFields

Section titled “JobFailureWireFields, and why the install door does not take WireFields”

The install door folds only the failed-job half. An install has no prior package digest and no prior compose by construction, so Verdict() is unknown on every successful install, and folding the whole projection would print

CHANGE UNKNOWN: this run cannot say whether anything was installed

after every shc install on the estate. That is the anti-absence rule pointed at the wrong absence: it exists for a measurement a run FAILED to take, and a missing prior on an install is not that; it is what the word means. The failed-job fact carries no such assumption (a job either failed or it did not, on either door), so it is the half that crosses.

It adds no key when nothing failed, because an install that reconciled cleanly must send no opinion rather than an empty claim, the same rule operationJobFailureLine reads at the far end, where an absent key and an empty list both mean “named no failed job”. An empty list here would make every clean install exit non-zero.

Both install doors take it: the stack door (installHandler) and the vdeployment door (installVdeploymentAsync), which binds its own plug edges and can fail the same way.

success: true is unchanged on every arm, including the failed one. The stack really is up; the asymmetry lives in the CLI’s exit code, taken off the job names, not in that flag.

ce/modules/app/install_door_reports_failed_jobs_test.go.

mutationreds
the JobFailureWireFields fold is dropped from installHandlerTestTheInstallDoorReportsAFailedIntegrationJob (frame returns to the bare map[environment stack success])
EffectOut: effect is deleted from RunInstallWithObserverTestTheInstallDoorBuildsAnEffectToRecordOnto
JobFailureWireFields returns e.WireFields()TestJobFailureWireFieldsCarriesOnlyTheFailure (a clean install starts sending changed:unknown)
the len(e.FailedIntegrationJobs) == 0 clause is droppedTestJobFailureWireFieldsCarriesOnlyTheFailure and the clean-install row of TestTheInstallDoorReportsAFailedIntegrationJob

TestTheInstallDoorBuildsAnEffectToRecordOnto is source-level for §6’s stated reason (ExecuteCompose needs a docker daemon) and it windows the scan to RunInstallWithObserver’s own body: the update lane’s EffectOut: effect is in the same file, so an anchor that merely proved “EffectOut appears somewhere” would stay green with the install door blind.

  • CLOSED by SHC-2426: a placement-DISPATCHED install reported green over a failed reconcile. Every install typed off the leader is dispatched, so this was every follower’s shc install, and it lost the exposure: line and the kept groups the same way. The receiving node now runs InstallDeploy and puts its effect in its answer (app.HolderEffectReply), and the dispatching door returns that effect with the holder’s settle (app.ReportHolderEffect), as the update door always has. InstallDispatcher still answers (dispatched bool, err error).