Config versus state, the migration (proposal)
Status: open design. The first migration,
backup.scheduleto thebackup_scheduletable, is built. The discriminator and the verdict table are not ruled.
As written: the discriminator and the verdict table are a proposal awaiting
owner ruling. The first migration (backup.schedule → the
backup_schedule table) is BUILT AND LANDED, and is described here as
built, not as proposed.
Written 2026-08-07. The backup.schedule migration landed on ce b2322ca;
the verdict table was compiled against ce 7ee33bc..8b0b8b4 and shc
819becc62, and every file:line citation below was read in a worktree at
those commits. No
production estate was contacted; the one measured estate fact quoted
(zero backup.schedule rows on both estates) is carried from the lane
that measured it.
The ruling
Section titled “The ruling”“If a command is gonna set state and you’re storing that state in config, you’re doing it wrong. Either have it be fully driven by config where you don’t actually have a special command that sets it, or you store the state separately from config, just in the database. It could still be scoped.”
And, when the alternative of validating harder inside the config write path was put to the owner:
“If you have to guard config because some other command is using it, doesn’t that mean it should be some kind of state in the database instead?”
Needing a guard is the tell. A guard requirement is not answered by adding a guard.
The discriminator
Section titled “The discriminator”The ruling turns on what a key’s checks are ASKING. There are only two kinds, and the difference is not one of severity; it is one of what information the question needs in order to be answered.
| the check asks | example | what it needs to answer | verdict |
|---|---|---|---|
| is this a well-formed VALUE? | the cron parses; the enum member exists; the retention count is a non-negative integer; the port is in range | the value, and nothing else | TYPE CHECK. Belongs in the key’s schema, where it protects every writer at once. The key STAYS CONFIG. |
| is this consistent with something ELSE? | the destination exists; a sibling key is also set; the application being hidden was crawled; the deployment named here is real | facts about other keys, other rows, or the live estate | CROSS-FACT INVARIANT. No key’s schema can carry it, because it is not a statement about this value. The key is STATE and moves to a table. |
The reason the second class cannot live in config is structural, not a matter of effort. Config has four writers and no common gate:
shc config setagainst a live daemon- the daemonless overlay (
ce/modules/config/local_store.go:222) POST /api/method/shc.config.set(ce/modules/config/router.go)- the terraform
shc_configresource, andshc install/update -c
A validation attached to any one of them is a validation the other two
route around. A validation attached to Service.Set itself could only
see the value, which is precisely the information a cross-fact invariant
does not have. So the invariant has no owner, and a key that needs one
has no business being a config row.
There is a third property that keeps showing up alongside the second, and
it is worth naming because it is diagnostic: a key whose scope is
narrowed on read is doubly a state candidate. A reader that empties the
Ctx makes finer-scope rows inert, so the store silently accepts writes
it will never honour. A dedicated table has no visibility walk to opt out
of.
The verdict table
Section titled “The verdict table”Verdicts are per key, with the evidence that decides them. Paths are ce unless marked.
IS-STATE: migrate to a table
Section titled “IS-STATE: migrate to a table”| key | owning writer | the cross-fact invariant | evidence |
|---|---|---|---|
backup.schedule ✅ MIGRATED | shc backup schedule set → POST shc.backup.schedule.set | CheckScheduleDestination asks whether backup.password is set for the backup.repo this scope resolves to. A repo with no password can never seal, so every fire lands plaintext on local disk, on a cron, forever, unwatched. | check: modules/backup/seal.go:246; refusal: X400077BKPSCD, modules/backup/errors.go:1713 |
lease.reap_attempts | the lease-retry cron, no operator surface at all | It is a running tally the daemon increments. Nobody authors it; there is no value an operator could type that would be correct. Every increment is a config-change event that wakes every config watcher on the estate. | modules/app/crons_lease_retry.go:66, written at :187 |
lease.reap_next_attempt | same cron | An RFC3339 backoff stamp, bookkeeping about a retry that is in progress. | crons_lease_retry.go:70, written at :204 |
lease.reap_abandoned | same cron | The terminal marker set when the retry budget is exhausted. A record of what happened, not a setting. | crons_lease_retry.go:76, written at :196 |
lease.expires_at | the lease stamp on install | Names a deployment that must exist, at deployment scope, and is written by the system rather than authored. It is the lease. | modules/app/lease.go:31 |
deployment.protected | shc protect / shc unprotect | ★ The flag lives in TWO stores (a deployment-scoped config row and a node-local sidecar mirror), and the command already has to report which half landed, because a deployment placed on another node arms the row and nothing this node can read. Two-store agreement is the definition of a cross-fact invariant. | modules/app/protected.go:102; the ProtectionStores type and its comment at :112-140 document the split in the code itself |
dns.owner_id | minted by shc cluster init (mintDNSOwnerID) | The cluster’s identity for DNS claim attribution. Two clusters sharing a zone are separated by it, and a destroy-and-rebuild must not silently reuse one. Identity is not a setting. | modules/dns/service.go:34; X503038DNSOID, modules/dns/dnsapi/errors.go:170 |
STAYS-CONFIG, but the type check moves into the schema
Section titled “STAYS-CONFIG, but the type check moves into the schema”These keys are genuine operator inputs. The ruling’s other half applies: the check they need is a type check, so it belongs in the schema where every writer gets it, rather than in the one command that happens to have it today.
| key | today’s check | where it lives now | where it belongs |
|---|---|---|---|
backup.retention.{daily,weekly,monthly,yearly,bundle_days} | validateFields: range checks on integers | modules/retention/service.go:372, reachable only through shc retention set | the schema. shc config set backup.retention.daily -5 bypasses it today. |
crons.<name> | the cron expression must parse | the cron runner, at read time; a bad override is a warn nobody reads | the schema, plus the existing registry-derived ReadClaim that already catches an unregistered job name (modules/config/keyspace.go:133) |
backup.repo | none | n/a | the schema: a destination URL has a form (s3:, b2:, an absolute path) |
backup.password, dns.<provider>.<credential>, backup.{aws,b2,azure,google}_* | none | n/a | secret-class handling, not a type check. Value-shaped: a credential is whatever the provider says it is. |
lease.protected_environments | none | n/a | a list of environment names, layered over a hardcoded set that config can never remove (modules/app/lease.go:37, :59). Pure operator policy. |
system.apps.<name>.host / .base_path / .vars.* | none | n/a | an app’s own variable namespace; AnyMemberRead is correct here for the reason the keyspace doc gives |
catalog.hidden.<qualified> | none | n/a | ★ the interesting near-miss. “Does that application exist?” IS a cross-fact question, but hiding an application that has not been crawled yet is legitimate, so absence of the referent is not an error. It stays config, and the honest response to an unknown qualified name is the warning it already gets, not a refusal. |
Not applicable
Section titled “Not applicable”quota.* is not a config family in ce; quota is derived from the
authored compose figure rather than stored. Nothing to classify.
Independent confirmation, arrived at from the other end
Section titled “Independent confirmation, arrived at from the other end”While this table was being written, a sibling lane landed ce 3d8b2f5,
which adds modules/config/command_owned.go, a marker declaring a key
“command-owned state”, enforced at the shc install/update -c passthrough.
It registers exactly three paths:
backup.schedule(modules/backup/config_reload.go)lease.expires_atanddeployment.protected(modules/app/register.go)
All three are on the IS-STATE list above, reached by a different route and
by a different author. That lane’s own comment on backup.schedule states
the limit of a marker and predicts this document’s conclusion:
“Per the owner’s ruling that is the tell that a schedule is not config at all; it wants its own table. Until it has one, the registration stops
-c backup.schedule=…becoming a SECOND way to arm a schedule…shc config set backup.scheduleremains the FIRST way, and remains a known defect: closing it means moving the storage.”
The storage moved. backup.schedule’s command-owned registration is
replaced by the retirement in this landing; the marker mechanism stays
and now covers lease.expires_at and deployment.protected, which is a
good description of what it is for: the holding position for a key that
is state and has not been migrated yet. Read the two registries together
and the remaining migration queue is legible from the code.
Ordering, and why
Section titled “Ordering, and why”backup.schedule: first, and already done. It is the only key on the list whose bad state arms a recurring action: a config row written raw registered a cron with none of the owning command’s checks. Every other candidate stores a wrong fact; this one stored a wrong fact that then did something, repeatedly, unattended.lease.reap_attempts/reap_next_attempt/reap_abandoned: next, and cheapest by a wide margin. There is no operator surface to preserve, no intent to migrate, and no compatibility story: the daemon writes them and the daemon reads them. They should land in onedeployment_leasetable together. Doing them second proves the pattern on a case with zero migration risk.lease.expires_at: rides into the same table as (2). Split out here only because it has an operator-visible origin (--lease) and therefore a real value to carry.deployment.protected: after the pattern is proven twice, because it is the one with a second store. The table becomes the truth and the sidecar becomes an explicitly-named cache, which letsshc protectstop reporting two-valued success.dns.owner_id: last of the state set. Smallest change, largest blast radius: it is cluster identity, and the write-once mechanism (modules/config/write_once.go) may be the better answer for it than a table. Worth its own ruling.
The STAYS-CONFIG schema work is independent of all five and can proceed in parallel; it has no migration and no ordering constraint.
What landed for backup.schedule
Section titled “What landed for backup.schedule”The table
Section titled “The table”backup_schedule (ce ent/schema/backup_schedule.go), created by
migrations/20260807170745_auto.sql. Still scoped, as the ruling
allows: scope carries the tier, tenant_name / stack_name /
environment carry the target.
Two decisions worth stating:
scope_keyis the uniqueness key, not the scope tuple. SQLite and rqlite treat NULLs as distinct inside a composite UNIQUE, which is why theconfigtable’s own composite constraint is inert for global rows and why its upsert has to be a hand-written guarded INSERT (modules/config/service.go:379). The scope columns here default to''andscope_keycarries the canonicalbackup.ScopeKey()rendering, so one-registration-per-scope is enforced by the database on every backend.- It is a NEW table, not a reuse of
schedule. The existingscheduletable is the per-stack JOB scheduler: its rows name a<job>.job.yamlinside a deployed stack and its fires become task rows. Itstenant_name/stack_nameare NOT NULL and its unique key is(tenant, stack, environment, job). A backup schedule is scoped at five tiers including global, where there is no stack, so the global and tenant registrations would collide on that key. Overloading it would have meant a sentinel-stack convention plus two columns meaningless to job rows.
The migration is a COPY, not a move
Section titled “The migration is a COPY, not a move”make migrate generated the DDL; the backfill is hand-appended beneath
it, which is the established shape here (see
20260805004250_application_table.sql, whose header explains why Atlas
cannot sequence a data-carrying transition).
The config rows are copied and left in place. Two independent reasons:
- The daemon-side loader takes the union of the table and any
leftover
backup.schedulerows, so a row the backfill could not see (an older CLI against a newer daemon, or a schedule armed concurrently with this landing) still fires. Deleting here would make that safety net unreachable for exactly the rows it exists to catch. - A migration cannot see a row written after it runs.
Three classes are deliberately not carried, and each stays in config
where it remains visible and honoured: a value that is not valid JSON
(json_extract would abort the migration and take daemon boot with it); a
value with no cron in either accepted shape (it could never have
registered); and a row whose scope columns are malformed: a stack or
environment recorded with no tenant, which scope.Classify reads as
GLOBAL and which would therefore collide with the real global row.
Nothing is silently dropped.
Measured: both estates carry zero backup.schedule rows, so the backfill
is empty in practice. It is written as though it were not.
The old path refuses
Section titled “The old path refuses”modules/config/retired.go adds RegisterRetiredPath, a sibling to the
existing RegisterWriteOncePath and to RegisterCommandOwnedPath. Service.Set and the daemonless overlay
both refuse a retired path with X409073CFGRTD, naming the table and
the command that replaces it.
The asymmetry with write-once is deliberate and is the opposite way round:
| write-once | retired | |
|---|---|---|
| SET | create and identical re-state allowed; change refused | always refused |
| DELETE | refused: delete-then-set would be a flip through the back door | allowed: a leftover row must be removable, or it is stranded forever |
| READ | allowed | allowed: the loader is still honouring it, so calling it inert would be a lie |
One more surface needed the tolerant form: the deployment-config restore
replay (daemon/services_backup_config.go:74) now SKIPS a retired path
with a warning instead of failing. An archive is a record of the past; a
key that has since moved stores will be in any archive taken before the
move, and letting that fail a restore would mean a schema migration
retroactively invalidated the estate’s backups.
Two defects fixed on the way past
Section titled “Two defects fixed on the way past”Both are consequences of the same root cause: a check that lived inside an optional live-apply rather than in front of the write.
- The cron parse was conditional on a scheduler existing. It rode on
Scheduler.RegisterScopeinsideif sch != nil, so a request served before thebackup_schedulersubsystem published its instance (a partial boot, a follower, a test root) persisted an unparseable cron and returned 200. It is nowbackup.ValidateCron, called unconditionally in front of the write. This is the type check the discriminator says belongs in front of every writer, and naming it as one is what made the bug visible. shc backup schedule lsrendered nothing on a follower. The list came from the in-memory cron map, and the subsystem is LeaderOnly, so any node that was not the leader reported “no backup schedules” for an estate that had them. It now reads the table;next_runstays best-effort, because when a schedule next fires genuinely is a property of the node that would fire it.
migrations/backup_schedule_carry_test.go: applies the real catalogue in two halves, seeds one config row per tier plus the three skip classes, and asserts the carry, the skips, that theconfigrow count is unchanged, and that the SQL’s scope-key rendering equals whatbackup.ScopeKeyreturns for the same scope.subsystems/impl/backup_schedule_loader_test.go: table-only, config-only (must still fire), and both-for-one-scope (must register once, and the table must win).modules/config/retired_test.go: SET refused, READ and DELETE still work, exact-path granularity.modules/backup/schedule_store_test.go: round-trip at all five tiers, upsert replaces rather than duplicates, andbackup.scheduleis registered retired.
Open questions for the owner
Section titled “Open questions for the owner”- Is
dns.owner_ida table or a write-once config row? It is identity, not configuration, so the ruling points at a table, butwrite_once.goalready exists for facts whose CHANGE needs machinery that does not, and identity may be exactly that shape. - Should the retired-path refusal ever expire? Today a retired path refuses writes forever and the loader honours leftovers forever. An alternative is a release in which the loader stops reading them and the rows become ordinary dead config. That is a decision about how long an upgrade window is, and it is not ours to pick.
catalog.hidden.<qualified>for an application that does not exist: confirmed as a warn, not a refusal? The table above rules it stays config on the grounds that hiding something not yet crawled is legitimate. If that is wrong it becomes the sixth migration.