Migrations
This workflow runs in the ce repo, not here. The split moved
ent/,migrations/andatlas.hclthere, and themake migratetarget moved with them; every command below is run from a ce checkout. This repo consumes ce as a pinned Go module and the daemon applies the embedded migration at startup, so there is nothing schema-related to run on the shc side. File paths below are ce-repo-relative.
SHC uses Atlas (ariga/atlas) with Ent (entgo.io/ent) as the
schema source. Tables are declared as Go structs under ent/schema/.
make migrate NAME=<slug> diffs the catalogue against those declarations
and writes the next migration. The SHC daemon applies the catalogue at
startup through the runner at core/db/migrate.go.
make migrate needs the Atlas CLI (make prepare). Operators don’t need
Atlas on production hosts.
One first file, then appended migrations
Section titled “One first file, then appended migrations”The store starts from one first file (SHC-2197), Atlas’s plan from an
empty database to the tables ent declares. A schema change appends a
numbered migration on top of it. make migrate is the only migration
target in ce.
ent/ generate.go # go generate ./ent entrypoint entc.go # codegen driver schema/ <table>.go # one file per table <table>.go, <table>_create.go, … # generated typed client (don't edit)core/db/released.sum # atlas.sum lines of every migration a release shippedmigrations/ <UTC-timestamp>_auto.sql # the first file <UTC-timestamp>_<slug>.sql # appended by make migrate NAME=<slug> atlas.sum # integrity manifest for all of them embed.go # //go:embed *.sql atlas.sument/schema/*.go is the source of truth. It drives both the typed CRUD
client (used by every module’s repo via core/entclient) and the
migrations. Nobody edits a migration or atlas.sum by hand.
Making schema changes
Section titled “Making schema changes”- Edit
ent/schema/<table>.go(add, remove or change fields, edges, indexes, CHECKs). - Run
go generate ./entto regenerate the typed client. - Run
make migrate NAME=<slug>. Atlas diffs the catalogue against ent and writesmigrations/<UTC-timestamp>_<slug>.sqland a newatlas.sum. - Commit the schema change, the client and the new file together.
$EDITOR ent/schema/deployment.gogo generate ./entmake migrate NAME=deployment_domainAn appended migration is additive. A live cluster takes a new build one
node at a time, so the previous build still runs against the store the new
one migrated: a new table, a nullable or defaulted column and a new index
are fine. Removing or reshaping something the previous build reads takes
two releases. The first stops reading it, the second removes it and names
the first in a line -- shc:last-reader-removed-in vX.Y.Z.
A merge conflict in atlas.sum is resolved by taking either side and
running atlas migrate hash (the first line of make migrate). Two
branches that both appended keep both files.
Released migrations
Section titled “Released migrations”core/db/released.sum lists the atlas.sum line of every migration a
release has shipped. The release commit writes it in ce, from the
catalogue the tag ships:
grep -v '^h1:' migrations/atlas.sum >core/db/released.sumTestEveryReleasedMigrationShipsUnchanged in tests/_ce/migrations fails
when a recorded line is missing from the catalogue or hashes differently,
so a shipped migration is never edited, renamed or removed. A release
tag’s pipeline refuses a pinned ce whose released.sum is not exactly the
file lines of its atlas.sum (release/freeze-check,
scripts/ci-freeze-check.sh), so a tag cut without the record fails its
own pipeline.
A reset
Section titled “A reset”A reset replaces the catalogue with one new first file. It ships in a
release whose clusters are reinstalled, never upgraded. In ce, on that
day’s dev: rm migrations/*.sql && make migrate, then move
ResetLine in core/db/reset_line.go to the reset release, and rewrite
core/db/released.sum from the new atlas.sum in the same commit. That
commit is the one sanctioned change to a released line.
atlas migrate diff against ent://ent/schema is not empty on today’s
catalogue: ent names an index past 64 characters
<first 31 characters>_<md5 of the name> at runtime, which is what the
first file holds, while Atlas’s ent loader keeps the long name. Today
that is integration_tenant_name_consumer_deployment_id_consumer_plug_name
and replicalabel_tenant_name_environment_stack_name_service_name_ordinal_key.
Anything else in that diff is real drift.
The gate
Section titled “The gate”The drift tests live in the shc repo’s test store, tests/_ce/migrations/
(run overlaid into ce), and tests/unit/migrations/:
TestEveryReleasedMigrationShipsUnchanged: every line ofcore/db/released.sumis in the catalogue, byte for byte.TestEveryMigrationAfterTheFirstFileIsAdditive: the additive rule above.TestFromEmptyMigrationsMatchTheEntDesiredStateandTestEntSchemaAndMigrationHeadAgree: the store the catalogue builds from empty is the store ent declares, table by table and column by column.TestEntIndexNamesAreTheNamesMakeMigrateWrites: no index name ent declares is over 64 characters. ent’s Create hashes a longer name andmake migratewrites it in full, so give such an index aStorageKeyof at most 64 characters.
What a new first file does to an existing store
Section titled “What a new first file does to an existing store”A store remembers the files it was made from in its _shc_migrations
ledger. A build refuses, at start and before any SQL runs, a store whose
ledger names a migration it does not ship: X409990MIGRST when the store
was made from a history this release no longer ships (“reinstall from this
release”), X409991MIGNEW when a newer build migrated it. A store made
from the first file and earlier appended migrations is upgraded in place.
A store made before a reset is not: the cluster is reinstalled from the
reset release and its apps’ data brought back from backups.
A reset moves the reset line, ce core/db/reset_line.go ResetLine, to the
reset release. A node records the line it was formed on,
and shcd refuses one formed before its own with X409992PRELIN, exit 65,
before anything starts (see reset lines).
Schema authoring conventions
Section titled “Schema authoring conventions”Every ent/schema/<table>.go:
- Type name = table CamelCase (
mesh_token→MeshToken,store_node→StoreNode). Annotations()pins the SQL table name explicitly viaentsql.Annotation{Table: "<name>"}; Ent pluralizes by default.- VARCHAR(N) UUID PKs: redeclare via
field.String("id").MaxLen(N).Immutable(). - VARCHAR(N) non-UUID PKs: use
field.String("id").StorageKey("<col>").Immutable(). - Field names match column names verbatim (snake_case); Ent derives Go field names from those.
- A closed set of values carries a CHECK in the table’s annotation.
- Edges: parent declares
edges.To("children", Child.Type)withAnnotations(entsql.OnDelete(...)); child declaresedges.From("parent", Parent.Type).Ref("children").Field("col").Unique().
The canonical exemplar lives in ent/schema/kv.go.
SQLite on both engines
Section titled “SQLite on both engines”rqlite replicates plain SQLite statements over raft, so the one file is
plain SQLite: CREATE TABLE and CREATE INDEX from an empty store. It
never sends a PRAGMA and never drops a table, which
tests/_ce/migrations/catalogue_shape_test.go holds.