Skip to content
SHC Docs

Squashing the migration catalogue, and why it needs no redeploy

★ SUPERSEDED, 2026-08-19, by owner ruling, see 2026-08-19-migrations-reset-doctrine.md. The bridge this document proposed (the manifest, the record-without-executing baseline, the ledger compaction, the flatten guard) was built, shipped in ce, and then removed whole. The owner’s reasoning: a reset is a reset; the entire point of resetting the migration history is that it is NOT safe for an existing deployment; the remedy for an existing deployment is rebuild + external import, never a bridge. This page stays as a dated record of the mechanism as it existed on 2026-08-18. Nothing below is current behaviour: squashes.json no longer exists, make migrate/flatten is gone, and the error codes it cites for the bridge (X500973MIGSQH among them) are retired.

Status: decision document, awaiting ruling. Nothing in this document changes behaviour, and no migration file was touched to produce it. Scope: ce/migrations/ (34 Atlas-managed SQL files), the runner in ce/core/db/, and the question of when the catalogue may be frozen. Evidence: ce e926c249, shc b40c0714, both estates’ live ledgers read at level=strong, plus eight executable probes and one full migrate/flatten dry run whose results are reproduced below.


The backup / redeploy / restore exercise is not required. It was never required.

The ask was:

“get all of our SQL migrations and little Golang migration files all squashed down to a single auto migration… delete the files and run make migrate. However… I’m gonna actually have to back up all of my deployments in [both estates] and completely redeploy and then restore those apps.”

The premise of the second sentence is false. This repository already implements the same baseline mechanism Rails, Django, Flyway and Liquibase use to make a squash survivable: a squashed file is recorded as applied without being executed on any database that already carries its predecessors.

The mechanism is migrations/squashes.json (a map of squashed file → the historical files it replaces) consumed by db.ApplyMigrations at ce/core/db/migrate.go:180-207:

“Squash baseline: a regenerated catalogue replaces history it squashed. When the ledger shows EVERY predecessor applied, the schema is already at (or past) this file’s content; record it as applied without executing. Without this, the squashed initial schema re-ran against live databases and every node failed to boot with ‘table already exists’ (observed on the proxmox cluster after a squash).”

The file exists, is embedded in the binary (migrations/embed.go, //go:embed … squashes.json), is loaded on the production path (subsystems/impl/migrations.go → db.MigrateFS → db.LoadSquashes), and today contains {}.

So the whole exercise reduces to:

the owner’s planwhat is actually required
back up every deployment on both estatesnothing
tear both estates downnothing
redeploy from scratchnothing
restore every app from archivenothing
noneone repo commit: delete 34 files, add 1, populate squashes.json, re-hash atlas.sum
noneroll the resulting release; each estate records one extra ledger row and executes no SQL

One thing in that commit is load-bearing and one thing only: squashes.json must be populated in the same commit that deletes the files. A squash that ships with squashes.json still {} re-runs the baseline against live databases and bricks every node with “table already exists”. That is the entire risk surface, it is a single JSON file, and section 4 gives the exact contents.


2.1 The runner does what it claims: eight probes

Section titled “2.1 The runner does what it claims: eight probes”

Probes were compiled against the real core/db package and deleted afterwards; the clone is byte-clean at e926c249. Five ran with the dev escape hatch on, three with production verification (atlas.sum + per-row ledger checksums) fully enabled.

#scenarioresult
1full history applied, then the squashed cataloguebaselined; one row inserted, no SQL executed, the pre-existing column added by an ALTER survives untouched
2squashed catalogue against a virgin databaseexecuted in full; fresh installs get the whole schema
3a node holding only 2 of 3 predecessorsrefused, loudly: X500973MIGSQH … conflicts with an existing schema
4old files deleted from the catalogue, names still in the ledgerno drift error; reboot idempotent; a later migration stacks on top cleanly
5reusing an already-applied filename as the squash nameaccepted, because the escape hatch disables ledger checksum enforcement
1vprobe 1 again, production verification onbaselined; reboot idempotent; the baselined row’s stored checksum matches the shipped file forever after
5vprobe 5 again, production verification onrefused: X500985MIGLDG … applied migration drifted, ledger checksum … but on-disk content is …
6van estate that skipped the squash release and takes squash + next in one stepbaselined, then the next migration ran

Probe 3 is the pre-flight requirement (section 4.2). Probe 5/5v is the naming trap (section 4.3). Probe 6v matters because it means an estate does not have to stop on the squash release.

2.2 The flattened baseline is semantically identical to replaying history

Section titled “2.2 The flattened baseline is semantically identical to replaying history”

make migrate/flatten was run in a throwaway copy of the repo. It produced one file, 20260818023245_squash.sql: 1,040 lines / 46 KB / 44 CREATE TABLE / 167 indexes, replacing 34 files / 2,663 lines / 146 KB.

Both catalogues were then applied to separate in-memory databases and compared, not by DDL text but by semantics: every column’s name, type, nullability, default and primary-key flag, every foreign key with its ON UPDATE/ON DELETE, and every index with its uniqueness and column list:

tables identical: 44
objects: replayed=211 flattened=211
ONLY IN REPLAYED HISTORY: 0
ONLY IN FLATTENED BASELINE: 0
SEMANTIC MISMATCHES: 0
PHYSICAL COLUMN-ORDER DIFFERENCES (cosmetic): 7

The seven are container, deployment, node, service, store, volume, volume_attachment, tables whose history includes an ALTER … ADD COLUMN that appended where the flattened generator interleaves. Nothing in the tree depends on ordinal position: SELECT * appears zero times in non-test Go source, as does any column-list-less INSERT. Ent always names its columns. The divergence is real but inert, and it is permanent; the two estates keep the replayed order forever while every public install gets the flattened order.

A generated baseline is DDL only, so anything a fresh install needs that is not DDL would vanish. Twelve of the 34 files carry INSERT/UPDATE/DELETE statements, which looked like a serious hazard until each was read. Every single INSERT in the catalogue is an INSERT … SELECT table-rebuild carry. There is not one literal VALUES seed anywhere in the 34 files, and every UPDATE is a backfill of existing rows (WHERE … IS NULL, a value rename, a de-duplication). A database with no rows needs none of them.

That is what makes the flatten sound, and it is the same invariant TestFromEmptySchemaEqualsTheMigratedSchema asserts today. That test must survive the squash in adapted form; it is the guard that keeps the ent schema and the catalogue honest.

2.4 Both estates are exactly where they need to be

Section titled “2.4 Both estates are exactly where they need to be”

Both estates carry byte-identical 34-row ledgers, sha256 a3b74dc4…, newest row 20260817170000_addon_rename.sql. There is no divergence to reconcile, and every predecessor the baseline will name is present on both. The compatibility window for this change is therefore exactly two databases, both already current, which is the whole argument for doing it before launch rather than after.


3. The redeploy path is not merely unnecessary; it is unsafe

Section titled “3. The redeploy path is not merely unnecessary; it is unsafe”

Stated plainly, because the owner’s plan bet both production estates on it. The restore path he intended to use has three confirmed defects on current main:

defectconsequence
multi-node restore recovers into the wrong environment, ignores --node, and leaves the deployment marked UNUSABLEa restore that “succeeds” puts data somewhere it was not asked to go and marks the result unusable
--paths is inert: accepted, parsed, never consumedask for a subset, get everything
archive checksums are written but never compared on reada corrupted archive restores silently and looks fine

Independently, docs/decisions/2026-08-16-durability-not-sagas.md already ranks “recover leaves a live, healthy-looking, EMPTY stack” as the third-worst reachable state in the system, and “restore stranded mid-wipe” as the fourth.

Because the baseline path never restores anything, these stop being blockers for this decision. They remain blockers for any future exercise that does rely on restore, and they are the reason the answer to “should I rehearse a full estate rebuild to de-risk the squash” is no; the rehearsal is riskier than the thing it would de-risk.


4. The procedure, if the ruling is to proceed

Section titled “4. The procedure, if the ruling is to proceed”
  1. cd ce && make migrate/flatten I_UNDERSTAND_FLATTEN_BREAKS_LIVE_ESTATES=1 NAME=squash deletes migrations/*.sql and atlas.sum, regenerates one baseline from ent/schema, and re-hashes. Verified working; output measured in §2.2.
  2. Populate migrations/squashes.json in the same commit. Contents below, with the generated filename substituted for the placeholder. All 34 predecessors, strictly; a shorter list would let a database that is genuinely behind baseline onto a schema it does not have.
  3. Delete or adapt the 21 test files in ce/migrations/ (5,545 lines, 43 funcs). Five hard-code SQL filenames and break the moment the files go, including application_backfill_test.go, which carries TestFromEmptySchemaEqualsTheMigratedSchema, the invariant that makes squashing sound at all. Adapt that one against the baseline; do not drop it.
  4. Do not touch embed.go, core/db/migrate.go, or subsystems/impl/migrations.go. No code change is required anywhere.
{
"<GENERATED_BASELINE>.sql": [
"20260727022830_auto.sql",
"20260728080502_tenant_tombstone.sql",
"20260730032524_node_cpu_cores.sql",
"20260731192534_fabric_posture_pins.sql",
"20260802033902_node_voter_slot.sql",
"20260802201225_service_resource_columns.sql",
"20260802201638_deployment_pinned_node.sql",
"20260803002303_container_exit_cause.sql",
"20260803090000_vault_envelope_only.sql",
"20260803150452_deployment_app_digest.sql",
"20260803152000_service_quota_columns.sql",
"20260803170107_connection_tombstone.sql",
"20260804022217_auto.sql",
"20260804050215_deployment_node_set.sql",
"20260804170000_oidc_socket_rename.sql",
"20260804183840_vdeployment.sql",
"20260804183842_app_browse_columns.sql",
"20260804185355_preset_rows.sql",
"20260804200106_app_images_recorded.sql",
"20260804221823_app_requirement_declaration.sql",
"20260804222619_usage_vdeployment_count.sql",
"20260805004250_application_table.sql",
"20260805031927_drop_deployment_instance_name.sql",
"20260805054457_drop_deployment_parent_index.sql",
"20260807042540_task_affinity_node.sql",
"20260807064858_node_mesh_ip_unique.sql",
"20260807170745_auto.sql",
"20260807223000_backup_schedule_tier_key.sql",
"20260812023430_store_registry_and_seed.sql",
"20260814200549_node_architecture.sql",
"20260815060000_datetime_text_utc_format.sql",
"20260815070000_deployment_lease.sql",
"20260815182958_volume_attachment_state_reason.sql",
"20260817170000_addon_rename.sql"
]
}
  • Both estates read exactly the 34 names above from _shc_migrations via the live rqlite API at level=strong. The on-disk db.sqlite is a lagging raft snapshot and must not be used for this check. A missing predecessor means the baseline executes and the node bricks (probe 3).
  • squashes.json is non-empty in the diff. If it is {}, stop.
  • The generated baseline filename is new; it must not appear in either ledger (§4.3).
  • atlas.sum is regenerated and lists exactly the one file.
  • TestFromEmptySchemaEqualsTheMigratedSchema passes in adapted form.
  • The two guard tests in §6 exist and pass.
  • feat/payload-interning has been ruled on (§5.3).
  • CI is green. Do not merge red.

The baseline filename must be new. Reusing an existing name (the obvious instinct, “make the first migration the baseline”), is fatal: the runner finds the name already in the ledger, compares the stored content checksum against the new file, and refuses to boot with X500985MIGLDG (probe 5v). atlas migrate diff stamps a fresh UTC timestamp, so following §4.1 avoids this automatically. Hand-editing the filename afterwards reintroduces it.

Roll nodes one at a time, leader first. There is no leader gate, no advisory lock and no INSERT OR IGNORE around the ledger write (core/db/migrate.go:201,243). Every node runs the runner at boot against the shared raft database. Two nodes booting together can race the single baseline INSERT and one will lose on the primary key. Restart=always / RestartSec=5 in shc.service means the loser recovers in about five seconds (this is a transient, not a brick) but serialising removes the noise entirely. Wait for the new ledger row before moving to the next node.

The ledger is per-cluster, not per-node. Migrations run against the raft-replicated handle after the engine swap, so each estate has one ledger, not one per node. The 7-node, 2-estate framing collapses to two databases.

Verify by reading the ledger, not by exit status. Restore is not the only thing in this estate that exits 0 without having done the job. The proof that the roll worked is:

SELECT name FROM _shc_migrations ORDER BY name;

35 rows on each estate, the 34 originals untouched plus the baseline, and every deployment still running. If the row count is 34, the release did not take. If any node logs X500973MIGSQH, stop the roll: that node’s database was behind and §4.2 was not honoured.

Rolling back is a package downgrade and nothing else. The previous release ships all 34 files; the ledger still has all 34 rows; the orphan baseline row sits lexically after them and outside the drift check’s window, so assertNoDrift does not fire (probe 4). If a belt-and-braces cleanup is wanted, delete the one row through the rqlite API. No schema is ever mutated by this change on an existing estate, so there is nothing to reverse.

If squashes.json is ever wrong in a shipped build, the baseline can be recorded by hand, per estate, through the live rqlite API:

INSERT INTO _shc_migrations (name, checksum)
VALUES ('<baseline>.sql', 'sha256:<sha256 hex of the file body>');

The checksum is db.ContentChecksum, plain sha256 of the file contents with a sha256: prefix. This is the “one-row insert per estate” fallback. It should never be needed; it exists so that a mistake is a two-minute fix rather than an outage.


The honest answer: converging fast, not yet frozen, and three known migrations are still owed.

Measured from the UTC timestamps in the filenames:

windowmigrations
Jul 27 – Aug 312
Aug 4 – Aug 1016 (peak)
Aug 11 – Aug 186

Supporting evidence, all pointing the same way:

  • Across 217 commits in the last three days, exactly one touched ent/schema/.
  • 1 of 77 ce merge requests merged in the last 21 days added a migration.
  • Zero open merge requests in either project touch ce/migrations/ or ce/ent/schema/.
  • ce!88 (backup console editor, fork-on-write lineage, the newest schema-adjacent workstream) changes 13 files and lands no schema at all.
  • Longest quiet streak observed to date: 4 days (Aug 8–11).

5.2 The convergence is architectural, not luck

Section titled “5.2 The convergence is architectural, not luck”

This is the part that should raise confidence rather than merely describe it. New persisted state is being deliberately routed into the already-migrated generic kv and config tables rather than into new ones: auth_session_revocation, launcher.layout, portal.registration, license.estate all live there, a pattern stated flatly in shc/modules/portal/registration.go. Features that would once have cost a migration now cost a key. That is why several of the workstreams that sounded like guaranteed schema (credential revocation, launcher_placed on the console wire, install reporting) produce none.

#migrationconfidencenote
1an installed_by-style column✅ LANDED 2026-08-18ce!114, migrations/20260818140325_installed_by.sql: two plain ADD COLUMNs (deployment, vdeployment), no backfill, no index, no rebuild vocabulary. Shape recorded in 2026-08-16-marketplace-identity-and-install.md §9.1
2repo.origin → repo.adapter rename✗ NEVER OWED; struck 2026-08-18the column has been spelled adapter since the FIRST migration. This row was a prior audit misreading a CLI verb rename as a column rename. Do not re-add it
3backup lineage (derived artifacts, parent/child)✗ NO MIGRATION; resolved 2026-08-18the lineage rides the existing manifest Text column. This row’s own note anticipated it: “ce!88 lands none today, which is evidence the lineage may fit in existing columns”
4feat/payload-interning✅ MERGED as ce!111, 2026-08-18landed on main as migrations/20260818071711_payload_interning.sql. The choice this row said a squash would force was made BEFORE the squash: rebase, not abandon
5marketplace community layer (reviews, ratings, threaded comments, votes, install counts)none; wrong systemsee §5.4
6credential revocation · launcher_placed · install reporting · backup and exec permission gates · consensus guardrailslowsurveyed; each lands in kv/config or is derived, not persisted

feat/payload-interning is the one genuinely unresolved item. RESOLVED 2026-08-18, merged as ce!111. It is on main as migrations/20260818071711_payload_interning.sql (a payload_blob table plus payload_ref columns on events_log and audit), so the decision this paragraph said a squash would force (rebase onto the new baseline, or abandon) was taken before the squash and in the cheaper direction. The paragraph is kept because its argument still holds for the next parked branch: a squash does not create that choice, it merely stops letting anyone defer it.

The original finding, for the record: an unmerged branch parked ten days carrying a 35th migration 20260808080553_auto.sql, with no merge request ever opened, a stated ≥72-hour soak gate that had not been run (docs/internal/payload-interning-soak-runbook.md), and a changelog recording it deferred for three consecutive release cycles.

5.4 The community layer is a different migration system; say this to users

Section titled “5.4 The community layer is a different migration system; say this to users”

The marketplace community layer (reviews, ratings, threaded comments with voting, report flows) is Frappe DocTypes. It migrates through bench migrate, inside the Frappe app container, against that app’s own database. It generates zero files in ce/migrations/ and is completely untouched by anything in this document.

This matters beyond bookkeeping. The owner’s concern is “I don’t really want users having loads of migrations they’re gonna have to go through.” A user’s real migration burden is the sum of two independent systems, and squashing one says nothing about the other. Freezing ce/migrations/ perfectly still leaves the Frappe side migrating on its own cadence. Any public claim about migration burden has to name both systems or it will be wrong.

5.5 What “delete the Golang migration files” means: it is three different things

Section titled “5.5 What “delete the Golang migration files” means: it is three different things”
categorycountwhat happens
tests in ce/migrations/21 files, 43 funcs, 5,545 linesdeleted or adapted. Five hard-reference SQL filenames and break the moment the files go, and one of the five, application_backfill_test.go, is where TestFromEmptySchemaEqualsTheMigratedSchema lives. It must be adapted, not merely kept
runtime backfills outside migrations/7 files, 1,688 linesmostly survive; they migrate on-disk state, not schema. But ce/modules/app/vdeployment_dir_migration.go:81-82 states outright: “What WOULD license removal: a flatten, or a daemon-enforced release floor at >= v0.0.22.” The squash unlocks deletions that are currently blocked
the runner itself (core/db/migrate.go, verify.go, pool.go, subsystems/impl/migrations.go)4 filesstays. Untouched.

Not a date, a state. The window opens when all of the following hold:

  1. Ten consecutive days with no new file in ce/migrations/. The longest quiet streak so far is four days, so ten is roughly 2.5× the observed noise floor and is a real signal rather than a lull.

  2. The three owed migrations have landed: installed_by ✅ ce!114, the repo.origin → repo.adapter rename, and whatever backup lineage finally needs (or a written finding that it needs nothing).

    Two of the three are now struck rather than done, and the difference matters. The repo.origin → repo.adapter rename was NEVER OWED: the column has been spelled adapter since the first migration, and this row was a prior audit misreading a CLI verb rename as a column rename. Backup lineage rides the existing manifest Text column, the finding this condition allowed for. So after ce!114 this gate is CLEAR, and the long pole is condition 3 alone, which ce!111 also closed by merging feat/payload-interning.

  3. feat/payload-interning is ruled on: merged, or abandoned and its branch deleted.

  4. Both guard tests from §6 exist and pass.

  5. Both estates read the full predecessor list at level=strong on the day of the change.

Condition 1 is the clock; 2–5 are gates. On the current trajectory conditions 2 and 3 are the long poles, not the cadence.

One correction to the framing of the original ask, and it changes the stakes entirely. The owner said “I don’t wanna have to continuously go through it again and again.” With the baseline mechanism, a second squash costs exactly what the first one costs: a commit and a ledger row. The dread is proportionate to the redeploy plan, not to the change. There is no reason to hold the squash hostage to certainty that no migration will ever follow, only to the three that are already known to be coming, because those are cheap to wait for.


6. Owed work, small, and it should land before the squash

Section titled “6. Owed work, small, and it should land before the squash”

Two guard tests, both absent today:

  1. The baseline branch has zero test coverage. Every test in the tree passes db.LoadSquashes(migrations.FS), which is {}, so the branch that makes this entire decision safe has never been executed by CI. The probes in §2.1 are the shape such a test should take. This is the single highest-value item in this document.
  2. squashes.json is unvalidated. A typo in a predecessor name degrades silently to “no squash”; the baseline then executes and bricks the estate. Nothing checks it. A test asserting that every name listed matches the set of files deleted in the same commit would convert the only real failure mode into a red pipeline.

One CI gap worth naming: nothing asserts that ent/schema and the migration head agree. grep 'migrate diff' .gitlab-ci.yml returns nothing, so a schema edit merged without its migration is caught only at boot, on an estate.

And a ruling is requested on the flatten guard. make migrate/flatten is disarmed behind I_UNDERSTAND_FLATTEN_BREAKS_LIVE_ESTATES=1 by the owner’s own 2026-07-30 append-only ruling, a ruling made before the baseline mechanism existed, and whose premise (“this destroys applied migration history”) is no longer true when squashes.json is populated. The recommendation is not to remove the guard but to re-arm it against the invariant that actually matters: refuse unless migrations/squashes.json is non-empty and names every file the flatten is about to delete. That turns a variable asserting an intention into a guard asserting a fact, which is the pattern the target already argues for in its own comment.


Proceed with the squash as a repo change. Do not back up, do not redeploy, do not restore.

Land the two guard tests first, wait for the three owed migrations and a ruling on feat/payload-interning, then take the ten-day quiet window whenever it next appears. The change is one commit, one release, and one extra row in each of two ledgers.

The exercise the owner was bracing for (the one he described as “a huge change”) is a git rm, a JSON file, and a dpkg -i.