System-app overrides + Keycloak's own host (2026-07-15)

STATUS: IMPLEMENTED (2026-07-17). Landed on main with one rename delta: the daemon env var is now SHC_MODULES (F9 modules-map rename), not SHC_SYSTEM_APPS as written below — internal/modules/app/system_plan.go reads SHC_MODULES, and shc.service sets it. The operator surface landed as shc init --modules-config (a compact JSON object keyed by app name; the key set is the install selection, values seed system.apps.<name>.{host,base_path,vars.*} — internal/modules/cluster/commands/init.go, mirrored by shc reconcile), with the terraform modules map on shc_cluster/shc_node wiring through it. Read SHC_SYSTEM_APPS mentions below as the historical name.

Status: DESIGN (implementation-ready). Branch feat/system-app-overrides.

“Allow a way to override config and vars of the apps that get initialized by shc init. Then you can override the host for keycloak.”

Two layers, one mechanism:

1. GENERAL — a declarative surface where the operator overrides the install-time vars (and host/basePath) of the SYSTEM apps shc init installs on the master tenant, so init (and every idempotent re-init / self-heal reconcile) installs them with the overrides instead of the built-in defaults.

2. MOTIVATING CASE — use that surface to give the SYSTEM keycloak its OWN host (auth.example.com) with its own cert, INSTEAD of today’s hardwired Host(SHC_HOST) && PathPrefix(/auth) path overlay. Unset ⇒ behaviour unchanged (SHC_HOST + /auth), byte-for-byte.

The master-tenant platform stacks are fixed at four, installed in dependency order (internal/modules/app/system_plan.go:33):

var systemAppInstallOrder = []string{"postgres", "keycloak", "openobserve", "traefik"}

shc init runs h.Init (bootstrap: CA, node cert, swarm, start shcd, unlock vault) and only THEN calls appcommands.PlanAndInstallSystemApps (internal/modules/cluster/commands/init.go:324). That POSTs shc.system.plan (daemon resolves the app set from its own SHC_SYSTEM_APPS, re-projected onto the canonical order — system_plan.go:58) and streams each install back through the same daemon install API a manual shc install <app> -t master uses.

Every system install passes through the install router (internal/modules/app/router.go:1020):

applySystemAppDefaults(&body)   // seeds integrations, fixed vars, basePath, publicUrl
installed, err := svc.Install(r.Context(), body)   // Phase 1: durable row + pre_install
// Phase 2: RunInstallWithObserver → ExecuteCompose (async goroutine)

applySystemAppDefaults (internal/modules/app/system_apps.go:93) is the ONE merge point. It consults the canonical spec table (system_apps.go:48):

For keycloak it additionally seeds, ALL derived from SHC_HOST:

• basePath = systemAppBasePath("keycloak") = /auth when SHC_HOST set, else "" (system_apps.go:315)
• publicUrl = systemKeycloakPublicURL() = https://<SHC_HOST> (system_apps.go:298)

SHC_HOST itself is not in the daemon env by default — it is seeded from node_bootstrap.json at boot, ONCE, before any system-app install reads it (internal/daemon/shc_host_seed.go:36). This is the ordering pin the whole feature must respect.

The precedence contract inside applySystemAppDefaults is “operator wins, spec fills the gap” — every seed is guarded by if _, set := req.Variables[k]; !set. The override surface plugs in at exactly this seam.

SHC has two disjoint config surfaces (internal/modules/config/service.go:167-172, the “no hybrids” rule): the ambient koanf file tree (internal/core/config, default.yaml ⊕ config.yaml ⊕ env, Global tier only, read via cfg.String/Bool) and the dqlite scoped store (5-level cascade, read via config.Service.Get). A key resolves from EXACTLY ONE surface.

Decision: the override block is scoped config, keys:

system.apps.<name>.host                 # bare host (keycloak only, for now)
system.apps.<name>.base_path            # optional path prefix override
system.apps.<name>.vars.<dotted-var>    # one row per overridden install var

Rationale (this is the load-bearing decision):

• Every read site is already a scoped-config reader. The install seam (applySystemAppDefaults, internal/modules/app), the ingress reconcile (MaterializeIngress / writeKeycloakSystemRoute, internal/modules/app), and the issuer derivation for consumers (injectDerivedKeycloakIssuer via the keycloak socket) all live in internal/modules/app, which reads scoped config through ONE seam — configResolver() / scopedConfigValue (internal/modules/app/dns_target.go:579,649). The daemon read sites (wireAuth, the dialer, systemHostRows in services.go) can read the same service via d.services.ConfigService(). Choosing scoped config means zero new plumbing — it reuses the exact machinery that already carries acme.email and launcher.

• acme.email is the precedent, feature-for-feature. systemAcmeEmail (internal/modules/app/dns_target.go:629) reads a GLOBAL-scope scoped-config key, and that ONE value feeds BOTH the install seed (applySystemAppDefaults → system_apps.go:214) AND the running-daemon ingress sweep (ingress_materialize.go:395, leEnabled) AND the recreate restate (SystemTraefikInstallVariables). The override surface needs the identical “read at init-install AND at every reconcile tick” property, and acme.email proves the scoped seam delivers it. The key is declared in default.yaml (acme.email: null, default.yaml:217) purely for keyspace/discoverability — it is never read via koanf.

• Live re-read powers the migration/reconfigure story. The ingress sweep re-reads scoped config every 60s tick (internal/subsystems/impl/ingress_materialize.go:64). A post-init shc config set system.apps.keycloak.host … therefore converges through the SAME self-heal reconcile that already exists for basePath drift (ReconcileKeycloakBasePath, keycloak_basepath_reconcile.go:246) — no new reconfigure path.

The one gap scoped config has is the first-boot chicken-and-egg: shc config set needs the daemon up, and within a single shc init the daemon comes up and installs the system apps in one shot, so there is no operator step in between. Two clean answers, both recommended:

1. Post-init (works today with zero new surface): set the key after init and let the reconcile converge (brief window at the old address). This IS the migration flow (§5).
2. First-boot genesis flag (recommended add): shc init --keycloak-host auth.example.com seeds the scoped row system.apps.keycloak.host right after h.Init and BEFORE PlanAndInstallSystemApps (init.go:324) via shc.config.set. This mirrors how --host seeds SHC_HOST, and it slots straight into the Terraform shc_cluster genesis-flag pattern (docs/proposals/2026-07-15-shc-node-per-node.md §2: host_domain / system_apps / nebula are already init-time genesis inputs) — keycloak_host becomes one more. No koanf/scoped hybrid, no chicken-and-egg.

The scoped Set accepts any dotted path but WARNS (X400033CFGUNK) on a path outside the known key space (internal/modules/config/keyspace.go:72). Register system.apps as a dynamic prefix so operator sets are clean:

// internal/modules/app/register.go (package init)
config.RegisterDynamicKeyPrefix("system.apps")

and add a documented (null) system: block to internal/core/config/default.yaml for shc config get discoverability. (Dynamic-prefix registration is the existing pattern for code-owned name→value sections — keyspace.go:38 seeds crons/timeouts the same way.)

(a) Override a system-keycloak install var + give keycloak its own host:

shc config set system.apps.keycloak.host auth.example.com
shc config set system.apps.keycloak.vars.ldap.enabled true      # re-enable a spec-disabled face
shc config set system.apps.keycloak.vars.javaOpts "-Xmx512m"
# or, at genesis:  shc init --host example.com --keycloak-host auth.example.com

(b) Override an openobserve install var (host untouched — control plane stays on SHC_HOST):

shc config set system.apps.openobserve.vars.sso.defaultRole viewer

The effective config view (shc config get) renders these as ordinary scoped rows, e.g.:

system.apps.keycloak.host            auth.example.com   (global)
system.apps.keycloak.vars.ldap.enabled   true              (global)

applySystemAppDefaults (system_apps.go:93) gains a config read that overlays the operator’s system.apps.<name>.vars.* over the built-in spec.variables BEFORE the existing “operator --var wins” fold:

// pseudo, inside applySystemAppDefaults, after resolving spec:
overrides := systemAppVarOverrides(ctx, req.AppName)   // ListMerged prefix system.apps.<name>.vars.
effectiveFixed := merge(spec.variables, overrides)     // config override beats built-in fixed var
for k, v := range effectiveFixed {
    if _, set := req.Variables[k]; !set { req.Variables[k] = v }   // explicit --var still wins
}

Precedence, narrowest-wins: --variable (this invocation) > system.apps.* config > built-in spec default. That keeps the existing “operator flag wins” guarantee while inserting the config layer between it and the hardcoded spec.

The host/basePath/publicUrl seeds (system_apps.go:162,194) switch from raw SHC_HOST to the new keycloak-host resolvers (§3).

The seeded vars land in req.Variables, which becomes ComposeRunOptions.VarOverrides (install_async.go:103), and ExecuteCompose folds them into the deployment’s persisted override set and replays them on every later render (install_runner.go:203):

opts.VarOverrides = mergeVarOverrides(priorState.VarOverrides, opts.VarOverrides, opts.RemoveVars)

persisted as persistedState.VarOverrides (install_runner.go:1722). So an override survives a bare shc update/recover automatically — the exact same replay contract normal --var deployments use (var_overrides.go:7). No new persistence.

But config is the source of truth, not the persisted cache. The recreate / self-heal path re-reads config each time it fires. The factored restate functions (SystemKeycloakInstallVariables, keycloak_basepath_reconcile.go:87; SystemOpenObserveInstallVariables; SystemTraefikInstallVariables) — which the daemon’s recreate seams call (services.go:1744,1769,1709) and which restateSystemAppVariables folds on every update (install_async.go:157) — extend to overlay system.apps.<name>.vars.* too. That closes the window where a persisted cache goes stale after shc config set: the next tick’s reconcile re-applies the live config value.

• Marker set (system-apps.initialized, system_plan.go:100) ⇒ re-init is a friendly no-op (init.go:153); overrides untouched.
• Marker unset (partial init) ⇒ re-init resumes the MISSING apps through the same applySystemAppDefaults path ⇒ override re-read from config, re-applied.
• Post-init change ⇒ shc config set + the 60s reconcile converges.

SHC_HOST is seeded from node_bootstrap.json at daemon boot (shc_host_seed.go:36), long before PlanAndInstallSystemApps. The scoped config service is likewise up before the install pass (proven: acme.email is read there today). If --keycloak-host is used, its shc.config.set runs after h.Init and before the install pass (init.go:324), so the row is present when applySystemAppDefaults reads it. No new ordering hazard — the override reads sit exactly where acme.email and the SHC_HOST-derived seeds already read.

Introduce, in internal/modules/app/system_apps.go (replacing the raw-SHC_HOST readers for keycloak only):

// bare host keycloak serves under: override system.apps.keycloak.host, else SHC_HOST.
func systemKeycloakHost() string {
    if h := scopedString(ctx, "system.apps.keycloak.host"); h != "" { return h }
    return bareSHCHostEnv()      // today's systemIngressDomain() value
}
// path prefix: explicit base_path override; else "" when a host override is set
// (dedicated host ⇒ root); else "/auth" (today's default, only when SHC_HOST set).
func systemKeycloakBasePath() string {
    if bp := scopedString(ctx, "system.apps.keycloak.base_path"); bp != "" { return normalize(bp) }
    if scopedString(ctx, "system.apps.keycloak.host") != "" { return "" }
    if bareSHCHostEnv() == "" { return "" }
    return "/auth"
}
// public origin: https://<systemKeycloakHost()>, "" when neither host resolves.
func systemKeycloakPublicURL() string { … }

SystemKeycloakBasePath() (system_apps.go:334) delegates to systemKeycloakBasePath(). openobserve’s systemAppBasePath("openobserve"), systemOpenObservePublicURL, and the /otel route are untouched — they keep reading SHC_HOST. This is what keeps the openobserve /otel overlay and the SHC_HOST control plane (the /api + dashboard routes, writeTraefikShcAPIRoute inject_traefik.go:1318) unaffected.

Today writeKeycloakSystemRoute (inject_traefik.go:1455) always builds Host(SHC_HOST) && PathPrefix(/auth) with a priority-100000 router and serves the SHC_HOST leaf. Change it to source host + basePath from the resolvers:

• host = systemKeycloakHost(), pathPrefix = systemKeycloakBasePath().
• When pathPrefix == "" (own-host default): the rule is a bare Host(auth.example.com) → keycloak overlay :8080, still ungated. Drop the priority bump when there is no PathPrefix (a dedicated host has nothing on it to shadow); keep it only for the SHC_HOST + /auth shared-host case where it must out-rank tenant routers. (writeKeycloakRouteFile already parameterises rule + priority — inject_traefik.go:1488.)
• Own cert: the LE gate becomes per-keycloak-host. Today the caller computes systemLE = systemIngressDomain() != "" && systemAcmeEmail() != "" (ingress_materialize.go:395) and passes it to BOTH keycloak and openobserve. Split it: keycloak’s LE gate is systemKeycloakHost() != "" && systemAcmeEmail() != "", and the served SHC leaf is signed for systemKeycloakHost() (its own SAN) via the existing serveSHCLeaf (inject_traefik.go:734). With LE on, Traefik solves HTTP-01 for auth.example.com and serves a real public cert; the leaf path already renews via RenewExpiringLeaves keyed by the served host (ingress_materialize.go:610). openobserve keeps its own systemLE on SHC_HOST.
• The wanted[…] orphan-sweep key and the injectSystemIngressNetwork overlay join (inject_traefik.go:408) already key off the keycloak PROJECT (master_default_keycloak), not the host, so the upstream path is unchanged — only the front-door rule/cert move.

DNS for auth.example.com must resolve to the cluster ingress. That is the operator’s zone (managed-DNS --host-style, or manual) — the same public- reachability requirement SHC_HOST already carries; call it out in docs.

Because the keycloak socket publishes publicUrl/basePath straight from vars.publicUrl/vars.basePath (apps/keycloak/keycloak.socket.yaml data:), and applySystemAppDefaults now seeds those from the resolvers, the issuer recomputes with ZERO changes to the consumers:

• injectDerivedKeycloakIssuer (integration_render.go:1063) builds <publicUrl><basePath>/realms/<realm> = https://auth.example.com/realms/master. No code change — it already reads the socket fields.
• The magic-token keycloak provider (cross-tenant SSO consumers, integration_magic.go) resolves the SAME socket, so every tenant’s consumer gets the new issuer automatically.
• wireAuth (services.go:1160) must switch its three keycloak reads from bareSHCHost() to the keycloak-host resolver:
  • derivedURL = "https://" + systemKeycloakHost() (was https://<SHC_HOST>)
  • BasePath = firstNonEmptyStr(configString(d.cfg,"keycloak.base_path"), app.SystemKeycloakBasePath()) — SystemKeycloakBasePath() now returns "" under an own-host, so auth.Service.PublicURL()/expectedIssuer() (service.go:251,453, built from rawSingleURL() ⊕ BasePath) yield https://auth.example.com/realms/<realm>. Then shc user / SSO / forward-auth all validate against the new issuer with no further change (they consume auth.Service).

The dialer rewrites exactly <shcHost>:443 → the live Traefik node (keycloak_dialer.go:82), fail-closed on anything else. Its shcHost field is supplied at construction (daemon.go:560 host: bareSHCHost(); services.go:1207 newKeycloakDialer(shcHost, …)). Change both call sites to pass systemKeycloakHost(). The request URL + TLS SNI stay the keycloak host (they ride derivedURL), so verification rides auth.example.com and its leaf; the trust pool anchors on the GLOBAL root (keycloakGlobalRootFn services.go:1369), which the auth.example.com leaf chains to exactly as the SHC_HOST leaf does — so pinned verification keeps working, and with LE it’s publicly trusted outright. keycloakBackchannelPort stays 443 (keycloak_dialer.go:37).

Note: the OTLP exporter dial reuses this dialer (services.go:1220, otel_dial.go) to un-hairpin https://<SHC_HOST> telemetry — that must keep targeting SHC_HOST, not the keycloak host. Since openobserve’s endpoint is still SHC_HOST, and the dialer only rewrites its configured shcHost, split the concern: keep the OTLP hairpin resolver on SHC_HOST (it computes its target from the openobserve endpoint), and give the keycloak dialer the keycloak host. These are already two logical concerns sharing one placement resolver; the fix is to not conflate their rewrite key.

writeOpenObserveSystemRoute (inject_traefik.go:1577), systemOpenObservePublicURL, SystemOpenObserveBasePath, writeTraefikShcAPIRoute, and systemHostRows’ openobserve row (services.go:1329) all keep reading SHC_HOST. Only the keycloak row in systemHostRows (services.go:1328) switches to systemKeycloakHost() + systemKeycloakBasePath() so the HUD paints the real address.

system.apps.keycloak.host / .base_path  (scoped config, GLOBAL)   ┐
SHC_HOST (fallback, node_bootstrap seed)                          ┘
                    │
                    ▼   systemKeycloakHost() / systemKeycloakBasePath() / systemKeycloakPublicURL()
   ┌────────────────┼───────────────────────┬──────────────────────┬───────────────────────┐
   ▼                ▼                        ▼                      ▼                       ▼
applySystemApp   wireAuth (services.go)   keycloak dialer     writeKeycloakSystemRoute   systemHostRows
Defaults         derivedURL + BasePath    (daemon.go:560,     (ingress reconcile:        (HUD row)
(seeds vars      → auth.Service           services.go:1207)   own Host rule + own cert)
 publicUrl,        issuer/PublicURL/           │                     │
 basePath)         expectedIssuer          rewrites <kchost>:443     └── ReconcileKeycloakBasePath
   │                    │                                                (self-heal: host/path drift)
   ▼                    ▼
keycloak socket    shc user / SSO / forward-auth
publishes          (validate against new issuer)
publicUrl+basePath
   │
   ▼
injectDerivedKeycloakIssuer  →  every consumer app's keycloak.issuer  (+ magic-token cross-tenant)

Nothing else reads it — this is the complete set (confirmed by grepping bareSHCHost / SystemKeycloakBasePath / systemKeycloakPublicURL / systemAppBasePath("keycloak")).

Unset ⇒ today’s exact behaviour, byte-for-byte. With no system.apps.keycloak.* rows: systemKeycloakHost() returns bareSHCHostEnv() (the current systemIngressDomain() value), systemKeycloakBasePath() returns /auth (only when SHC_HOST set), systemKeycloakPublicURL() returns https://<SHC_HOST> — the literals the code emits now. Every resolver’s “no override” branch reproduces the current constant, so the route stays Host(SHC_HOST) && PathPrefix(/auth) at priority 100000, the issuer stays https://<SHC_HOST>/auth/realms/<realm>, and the headless (no SHC_HOST) posture (root-served, no path route) is preserved. The general var-override path is a no-op when no system.apps.<name>.vars.* rows exist (empty overlay). This is a clean break in CAPABILITY, not in the default path (house doctrine: no_backcompat_no_direct_dns — clean breaks OK, but here the default is untouched by construction).

Setting system.apps.keycloak.host on a live cluster whose keycloak is at SHC_HOST/auth triggers, on the next reconcile tick:

1. Container re-render + basePath move. ReconcileKeycloakBasePath (keycloak_basepath_reconcile.go:246) sees Desired basePath change (/auth → "") vs Deployed (/auth) and recreates keycloak through the update path with SystemKeycloakInstallVariables restated (services.go:1739) — KC_HTTP_RELATIVE_PATH and the pinned KC_HOSTNAME (publicUrl) re-render to the new host. The reconcile’s divergence detector must widen from basePath-only to (host, basePath) so a host change with an unchanged path (/auth→/auth on a sub-path host) still heals.
2. Route move + cert re-issue. The ingress sweep rewrites the keycloak route file from Host(SHC_HOST)&&PathPrefix(/auth) to Host(auth.example.com) and signs (or HTTP-01-issues) a cert for the new SAN; the old SHC_HOST-leaf keycloak route file is orphan-swept (cleanupOrphanRouteFiles ingress_materialize.go:461).
3. Issuer change → SSO invalidation (call this out loudly). The OIDC issuer changes https://SHC_HOST/auth/realms/X → https://auth.example.com/realms/X. Consequences the operator MUST accept:
   • Every live SSO session / already-issued token is invalidated — strict issuer-match (auth.Service.enforceIssuer service.go:437) rejects the old iss. Users re-login.
   • Every consumer app must re-derive its issuer — this happens on the consumer’s next install/update/integration-job run (the client’s redirect/discovery re-renders from the new socket publicUrl). Until a consumer re-renders, its configured issuer is stale and its SSO is broken (confirmed failure mode: MAS mas-cli config check hard-fails on an issuer mismatch — integration_render.go:401). Operators should shc update (or reconcile) consumers after the switch, or expect a transient SSO outage per consumer until it next re-renders.
   • DNS for the new host must resolve to the cluster before the switch, or the login endpoint is unreachable.

Because of (3), recommend setting the keycloak host at INIT (before any tenant/consumer integrates) via --keycloak-host, and treat the live switch as a deliberate, disruptive maintenance action. The change is only ever triggered by an explicit operator shc config set (or genesis flag) — it never flips on an arbitrary tick — which IS the gate. Do NOT auto-honor it silently on an already-integrated cluster without surfacing the re-login/reconcile cost (a loud X2 warn on the first reconcile that detects the host change, naming the old→new issuer, is the minimum).

• internal/daemon/services_test.go — extend the existing SHC_HOST→issuer coverage: with system.apps.keycloak.host=auth.example.com (and empty base_path), wireAuth’s derivedURL + BasePath yield auth.Service.expectedIssuer("master") == "https://auth.example.com/realms/master"; with no override it stays https://<SHC_HOST>/auth/realms/master.
• internal/modules/app/system_apps_test.go — applySystemAppDefaults with system.apps.keycloak.vars.ldap.enabled=true re-enables the spec-disabled var; explicit --variable still beats the config override; host override seeds publicUrl=https://auth.example.com and basePath="".
• internal/modules/app/keycloak_route_test.go / inject_traefik_test.go — own-host route emits a bare Host(auth.example.com) rule (no PathPrefix, no priority bump) and signs a leaf for auth.example.com; the SHC_HOST/auth overlay is emitted only in the no-override case; openobserve /otel route is identical in both cases.
• internal/modules/app/integration_issuer_test.go — injectDerivedKeycloakIssuer builds the new-host issuer from the socket fields.
• internal/modules/app/keycloak_basepath_reconcile_test.go — host/basePath change is detected as divergence and heals; no-override steady state is a strict no-op.
• internal/modules/config/keyspace_test.go — system.apps.keycloak.host and system.apps.keycloak.vars.* resolve as KNOWN (no X400033CFGUNK).
• Proxmox e2e (standing gate) — the daemon-mediated install + real Traefik + real cert path can’t run in-sandbox; --keycloak-host on a live init is the acceptance test.

Order: 1 → 2 (general mechanism ships + tests, keycloak host still SHC_HOST) → 3 → 4 → 5 (own-host ships end-to-end) → 6 (first-boot ergonomics) → 7 threaded throughout. Each of 1–2 and 3–5 is independently green; the default path is untouched until an override row exists, so every step is safe to land incrementally. Estimated total: ~2–3 focused days plus the Proxmox e2e gate.

• First-boot surface: ship the --keycloak-host genesis flag now, or defer and rely on post-init shc config set + reconcile for v1?
• Live issuer change gating: honor a live host change automatically (converge via reconcile, loud warn) — or refuse it post-integration and require --force / a dedicated shc keycloak set-host verb that also kicks consumer reconciles?
• Scope of .host override: keycloak only (this proposal), or generalise system.apps.<name>.host to openobserve too (own telemetry host)? The general var surface is app-agnostic already; only the host-decoupling wiring is keycloak-specific here.
• General override reach: limit system.apps.<name>.vars.* to the four known system apps, or allow it for any master-tenant -t master install that hits applySystemAppDefaults?