Huly

Huly is an all-in-one project management platform (a Linear / Jira / Slack / Notion alternative). SHC packages it as a 13-service stack behind a bundled nginx that path-muxes every platform service, so one --host serves the whole product.

Huly’s database and blob store are integrations, not bundled services. Stand both up first, then wire them in:

shc install postgres --stack huly-db
shc install minio    --stack huly-blobs

shc install huly --stack huly \
  --integrate huly-db \
  --integrate huly-blobs \
  --host huly.example.com

Optional integrations:

  -i keycloak=<stack>   # native OIDC on the huly sign-in page
  -i mail=<stack>       # outbound signup/invite mail through mailcow or the mail app

To point at a database or object store SHC does not manage, set vars.postgres.hostname / vars.s3.endpoint instead of attaching the plug — each plug is required only while its var is empty.

A single-stack backup of huly captures only elastic and redpanda. To capture the product, back it up together with its providers:

shc -s huly backup create --all-connected   # -> <group>.bkg
shc recover --file <group>.bkg -y            # restores postgres, minio and huly

Upstream huly-selfhost ships CockroachDB. SHC runs huly on plain PostgreSQL 16. Huly speaks the Postgres wire protocol either way, and at the pinned v0.7.426 plain PostgreSQL is a first-class target:

• account and workspace carry a flavor table (dbTypes = { cockroach: {...}, postgres: {...} }) and choose a branch from getDbFlavor(), which runs SELECT version() and matches /cockroach/i against /postgresql/i. There is no scheme or env to set — detection is automatic. The postgres branch emits TEXT/BYTEA/BIGINT, creates pgcrypto and a gen_random_bigint() helper in place of cockroach’s unique_rowid(), uses GENERATED ALWAYS AS (...) STORED generated columns, and guards enum/constraint changes with DO $$ ... $$ blocks.
• hulykvs (Rust) ships two refinery migration runners, migrations_crdb and migrations_pg, and picks one from the detected backend.
• The per-workspace platform tables are engine-generic (text, bigint, bool, JSONB, USING gin on text[]), and the transaction-retry path keys off SQLSTATE 40001, which both engines raise.

The communication messaging backend is genuinely CockroachDB-only — its package is literally foundations/communication/packages/cockroach and has no flavor detection. Its migrations use DEFAULT unique_rowid(), ALTER TABLE ... ALTER PRIMARY KEY USING COLUMNS, DROP INDEX <unique constraint> CASCADE and a schema-qualified RENAME TO — none of which PostgreSQL accepts — and the runner rethrows on the first failure.

It is reached only through the COMMUNICATION_API_ENABLED gate in the transactor and the fulltext service, which SHC pins to "false". That is what upstream does too: its own dev/docker-compose.pg.yaml pairs postgres:16 with COMMUNICATION_API_ENABLED=false on transactor, fulltext and front, while the cockroach compose sets it true.

Do not set it to true while the database is PostgreSQL. Schema init would fail its second migration and retry-loop forever. This is not a regression from the CockroachDB packaging — SHC never enabled it there either.

An install created before huly’s database and blob store became integrations runs a bundled CockroachDB (cr-data, cr-certs) and a bundled MinIO (files). Those three volumes no longer exist, so a plain shc update will not carry the data across. This is a deliberate, one-time data migration.

This runbook has not been executed against a production estate. Rehearse it on a copy before running it anywhere you care about, and keep the pre-cutover backup until the verification steps below pass.

The obvious instinct — pg_dump the old engine, psql it into the new one — does not work here, for three independent reasons:

1. cockroach dump no longer exists. Verified against the pinned image: docker run --rm --entrypoint /cockroach/cockroach cockroachdb/cockroach:v24.2.10 dump --help answers ERROR: unknown command "dump" for "cockroach". CockroachDB’s own BACKUP/RESTORE only restores into CockroachDB.
2. The schemas are not the same schema. The account service emits different DDL per flavor: key STRING AS (CONCAT(type::TEXT, ':', value)) STORED on cockroach versus key TEXT GENERATED ALWAYS AS (global_account.social_id_type_to_text(type) || ':' || value) STORED on PostgreSQL, unique_rowid() defaults versus gen_random_bigint(), and so on. A row-level copy would fight the generated columns and the differing defaults.
3. The blob layout changes too. The bundled MinIO ran without rootBucket, so huly minted a bucket per workspace with objects at the bucket root. The integrated store uses one bucket with objects keyed <workspaceDataId>/<name>. No database dump addresses that.

The correct vehicle is huly’s own logical backup tool, which reads and writes through the platform adapters and is therefore both engine-agnostic and storage-layout-agnostic. It solves all three problems in one pass.

hardcoreeng/tool:v0.7.426 (same version train as the rest of the stack). Relevant commands, and the env each reads:

Env contract: DB_URL, ACCOUNT_DB_URL (note: not ACCOUNTS_DB_URL), ACCOUNTS_URL, SERVER_SECRET, STORAGE_CONFIG, QUEUE_CONFIG, and FULLTEXT_URL for the reindex commands. Copy the values straight off the running containers (docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' <deployment>-transactor-1) so the tool sees exactly what the stack sees.

Pinning dataId in step 5 is not optional: it is both the old bucket name and the new object-key prefix, so reusing it is what makes the restored blobs land where the restored rows expect them.

The database half is a cross-engine logical dump/restore either way — there is no shortcut for it. The blob half has two routes:

Route A — let the tool carry the blobs (default, fewest moving parts). backup downloads every blob into the dump directory and backup-restore re-uploads them through the storage adapter, which writes the new rootBucket-prefixed layout automatically. Nothing else to do. The cost is that every byte makes a round trip to local disk and back, and the dump directory has to be big enough to hold the entire blob set.

Route B — sync the buckets directly (for large blob sets). The layout change is a pure key remap: the old bundled MinIO ran without rootBucket, so workspace X had its own bucket named X with objects at the bucket root; the new store puts those same objects at huly/X/<name>. That is exactly a per-workspace mc mirror:

mc alias set old http://<old-minio>:9000 <old-ak> <old-sk>
mc alias set new http://<new-minio>:9000 <new-ak> <new-sk>

# one per workspace dataId from step 3
mc mirror --preserve old/<data_id> new/huly/<data_id>

Then skip the blob domain in the dump so you are not moving the bytes twice — huly’s blob domain is literally named blob:

  ... backup /backup/<ws-url> <ws-url> --skip blob

Route B keeps the byte movement server-to-server and can run before the cutover while the old stack is still serving reads, which is the main reason to choose it: it turns most of the copy into pre-work and shortens the downtime window to the database migration alone. Re-run the mc mirror once more after writes stop to pick up anything that landed in between.

Route B is the less-rehearsed path. If the instance is small enough that Route A finishes in an acceptable window, prefer Route A.

Downtime: the whole window, from step 2 to step 8. Huly is not usable mid-migration; a partially-migrated workspace is worse than a stopped one. Budget roughly the time of one full workspace read plus one full write, plus the reindex — for a large workspace that is hours, not minutes. Announce it.

1. Take a safety net. Full group backup of the current stack, on the old packaging, before touching anything:

shc -s huly backup create --all-connected

Keep the .bkg. It is the only rollback that restores the old shape.

2. Stop writes. Scale the front door to zero so no one can mutate state while you are copying it, but leave the backends up — the tool needs the transactor, cockroach and minio reachable:

shc -s huly scale nginx=0

3. Enumerate the workspaces. From the old cockroach:

docker exec <old-deployment>-cockroach-1 \
  cockroach sql --insecure -d huly \
  -e "SELECT url, data_id, uuid FROM global_account.workspace;"

Record url and data_id for every row. You need both later.

4. Dump each workspace with the tool pointed at the old stack. Attach it to the old stack’s network and give it the old DB_URL / ACCOUNT_DB_URL / STORAGE_CONFIG:

mkdir -p /var/backups/huly-migration
docker run --rm \
  --network <old-deployment>_default \
  -v /var/backups/huly-migration:/backup \
  -e DB_URL='postgres://huly:<pw>@cockroach:26257/huly' \
  -e ACCOUNT_DB_URL='postgres://huly:<pw>@cockroach:26257/huly' \
  -e ACCOUNTS_URL='http://account:3000' \
  -e SERVER_SECRET='<vars.secret>' \
  -e STORAGE_CONFIG='minio|minio?accessKey=<ak>&secretKey=<sk>' \
  -e QUEUE_CONFIG='redpanda:9092' \
  hardcoreeng/tool:v0.7.426 backup /backup/<ws-url> <ws-url>

Repeat per workspace. Verify each dump before moving on:

docker run --rm -v /var/backups/huly-migration:/backup \
  hardcoreeng/tool:v0.7.426 backup-check /backup/<ws-url>

Do not proceed past a failing backup-check.

5. Stand up the new providers and reinstall huly.

shc install postgres --stack huly-db
shc install minio    --stack huly-blobs

shc uninstall huly -y
shc install huly --stack huly \
  --integrate huly-db \
  --integrate huly-blobs \
  --host huly.example.com

Two notes on secrets. The vault keys were renamed with the plugs (cockroach_password -> postgres_password, minio_secret_key -> s3_secret_key), so the new stack mints fresh credentials — that is fine, nothing outside huly consumed the old ones. But vars.secret (the shared inter-service secret) and the Keycloak client secret are not regenerated by this migration and should be carried over if you want existing sessions and the existing OIDC client to keep working:

shc vault get huly/secret        # on the old stack, before uninstall
shc vault set huly/secret <value>

6. Recreate accounts and workspaces on the new database, reusing the dataId values from step 3:

docker run --rm --network <new-deployment>_default \
  -e DB_URL='postgres://huly:<pw>@<pg-host>:5432/huly' \
  -e ACCOUNT_DB_URL='postgres://huly:<pw>@<pg-host>:5432/huly' \
  -e ACCOUNTS_URL='http://account:3000' \
  -e SERVER_SECRET='<vars.secret>' \
  -e STORAGE_CONFIG='minio|<minio-host>:9000?accessKey=<ak>&secretKey=<sk>&region=us-east-1&useSSL=false&rootBucket=huly' \
  -e QUEUE_CONFIG='redpanda:9092' \
  hardcoreeng/tool:v0.7.426 \
  create-workspace '<ws-name>' '<owner_social_id>' -d '<data_id>'

Accounts come back with --accounts on the restore in the next step; use create-account only for accounts that own no workspace and therefore appear in no dump.

7. Restore each workspace with the tool pointed at the new stack (same env as step 6):

  ... hardcoreeng/tool:v0.7.426 \
  backup-restore /backup/<ws-url> <ws-url> --accounts

8. Rebuild the fulltext index. Nothing does this automatically after an out-of-band restore:

  ... -e FULLTEXT_URL='http://fulltext:4700' \
  hardcoreeng/tool:v0.7.426 fulltext-reindex-all

9. Bring the front door back.

shc -s huly scale nginx=1

Do all of these before you delete the old volumes.

1. Rows landed. The account schema exists and carries the expected identities:

   docker exec <pg-deployment>-postgres-1 psql -U postgres -d huly -c \
     "SELECT count(*) FROM global_account.workspace;"
   docker exec <pg-deployment>-postgres-1 psql -U postgres -d huly -c \
     "SELECT type, value FROM global_account.social_id ORDER BY value;"

   The workspace count must equal step 3’s, and every user’s email must be present.

2. The postgres branch ran, not the cockroach one. A direct fingerprint — this function only exists on the PostgreSQL migration path:

   docker exec <pg-deployment>-postgres-1 psql -U postgres -d huly -tAc \
     "SELECT count(*) FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
      WHERE n.nspname = 'global_account' AND p.proname = 'gen_random_bigint';"

   Expect 1. Expect kvs in information_schema.tables too — that proves hulykvs took migrations_pg.

3. Blobs are keyed, not bucketed. One bucket, workspace-prefixed keys:

   docker exec <minio-deployment>-minio-1 ls /data          # -> exactly: huly
   docker exec <minio-deployment>-minio-1 ls /data/huly     # -> one dir per dataId

   If you see one bucket per workspace, rootBucket did not take and the restore wrote the old layout — stop and re-check STORAGE_CONFIG.

4. The product works. Log in as a real user, open a project that existed before the migration, open an issue with an attachment and confirm the attachment downloads (that is the blob path end to end), then search for a word from inside a document (that is the reindex).

5. Only then remove the old volumes.

Before step 5, rollback is free: shc -s huly scale nginx=1. After step 5, roll back by recovering the step-1 .bkg, which restores the old bundled shape wholesale. There is no partial rollback — do not run a half-migrated stack.