Edge-CA trust: making server-side OIDC survive a non-public ACME directory

STATUS: LANDED, all runtimes. The CE core change is ce@8bc38bc (modules/app/edge_ca.go, modules/app/inject_ca_trust.go, modules/app/system_ca_root.go), the probe-backoff follow-up ce@2c62b54, and the JVM leg ce@6e5d9aa (modules/app/jvm_ca_trust.go). go.mod is pinned at the last of those. This file records the design, the per-app consumption survey, and what remains.

InjectDerivedKeycloakIssuer hands every SSO consumer a public https issuer — the pinned Keycloak origin on SHC’s own edge — and roughly two dozen of those apps run OIDC discovery server-side at boot.

When the estate sets acme.providers.<name>.directory to anything other than Let’s Encrypt production (LE staging, an in-house Pebble or step-ca), the edge leaf chains to a root that no container image ships. The server-side dial then fails with UNABLE_TO_GET_ISSUER_CERT_LOCALLY, and because discovery typically runs once at boot with no retry, the app’s SSO is broken permanently, from first boot, with no self-heal.

Live-hit on huly’s account service against a certResolver: acme-staging-dns-hetzner estate. Nothing in SHC knew the estate was on a staging CA, and nothing gave any app the anchor for it.

MaterializeTLSTrustAnchors writes the node’s trust bundle: the host’s public CA bundle ∪ the SHC global root. It now also carries the CA chain the estate’s own edge actually presents.

Where the CA comes from, and why.

The served chain is not sufficient on its own. Measured against a live LE-staging edge, traefik serves

leaf → (STAGING) Ersatz Emmer YR2 → (STAGING) Yonder Yam Root YR

and that last certificate is cross-signed by (STAGING) Pretend Pear X1 — a CA, but not self-signed. OpenSSL (and therefore Node, PHP, Ruby, Python, .NET) will not stop at a non-self-signed anchor unless X509_V_FLAG_PARTIAL_CHAIN is set, which none of them set. Verified: a trust store holding the complete served chain still fails openssl verify with error 2 ... unable to get issuer certificate.

So the collector continues through the topmost certificate’s AIA caIssuers pointer until it reaches a self-signed root — precisely the hop the verifier cannot make offline. Go’s own verifier would have accepted the cross-signed anchor, which is exactly why this had to be measured against OpenSSL rather than reasoned about.

AIA is plain HTTP by protocol. That is made harmless by a cryptographic gate rather than by transport security: a fetched candidate is accepted only when child.CheckSignatureFrom(candidate) succeeds. An attacker can substitute bytes on the wire but cannot produce a certificate whose key verifies the real intermediate’s signature.

Making the union richer is only half a fix — before this, exactly two of the ~32 SSO-consuming recipes read it. Requiring the other thirty to each opt in is how a systemic bug stays systemic: every future SSO recipe would have to remember, and none of them can reproduce the failure without a staging-CA estate.

So the platform injects it. Any service whose rendered spec references an https:// URL on one of the estate’s own ACME-covered hostnames gets the bundle bind-mounted at /etc/shc/tls.ca plus the trust-file environment variables its runtime reads:

No single variable covers that set, which is why four are set; a runtime that does not know a variable simply ignores it.

The JVM reads none of those four. It reads exactly one keystore, named by javax.net.ssl.trustStore, in JKS or PKCS12 form — so it gets its own leg (below).

InjectJVMEdgeCATrust runs immediately after the PEM leg, over the same compose, and covers the same predicate. For every matching service whose image is JVM-based it stamps a read-only bind of /etc/shc/tls.p12 plus

JAVA_TOOL_OPTIONS=-Djavax.net.ssl.trustStore=/etc/shc/tls.p12
                  -Djavax.net.ssl.trustStorePassword=changeit
                  -Djavax.net.ssl.trustStoreType=<detected>

JAVA_TOOL_OPTIONS is the one variable the JVM itself reads, before main() and therefore before the first SSLContext — the app-specific knobs recipes set (JAVA_OPTS, JAVA_OPTIONS, EXTRA_JAVA_OPTIONS, JAVA_OPTS_APPEND) are consumed only by that image’s launcher script. It is prepended to any value a recipe already set, so a recipe that names its own trustStore still wins (within one JAVA_TOOL_OPTIONS the last -D takes effect).

The store is built inside the app’s own image, not converted on the host:

cp $JAVA_HOME/lib/security/cacerts   →  copy
keytool -importcert (one per anchor) →  copy

one throwaway docker run --network none --entrypoint= per (image, anchors), content-addressed and cached under CacheHome, result written to <deployDir>/files/ and bound read-only exactly like the PEM union. The build runs before injection: a javax.net.ssl.trustStore pointing at a path that does not exist makes the JVM trust nothing, and docker would helpfully materialise the missing bind as an empty directory. Every failure mode (no docker, no keytool, unpullable image, unrecognised keystore) fails soft to exactly the pre-change behaviour.

Why not simply convert the PEM union to PKCS12 host-side. Two independent measured reasons, either one disqualifying:

Copying the image’s own cacerts is instead strictly additive: 144 → 147 entries for dependency-track, 109 → 112 for penpot. Nothing is ever removed.

The JVM-ness test is a cheap local docker image inspect over Config.Env/Entrypoint/Cmd, deliberately generous — an image that is not pulled yet answers “maybe”, and the builder decides. A false positive costs one short offline container; a false negative would silently re-open the bug.

Both halves are gated on an ACME provider declaring an explicit directory, read off the TRAEFIK_ACME_PROVIDERS carrier the daemon already produces for the system traefik render (no new daemon wiring, no config import in modules/app, and an empty directory is already SHC’s own encoding of “Let’s Encrypt production”).

Consequences:

• Stock estates are untouched. No probe, no fetch, byte-identical composes and a byte-identical CA union. Unit-pinned.
• Where it fires, SSO is already broken. The change can only move those estates from “cannot validate the edge” to “can”.
• Trust is only ever added. The union is a superset of the host’s public roots, so pointing SSL_CERT_FILE at it never removes trust a container had. Nothing is replaced and nothing else is broadened — the anchors are the estate’s own edge chain, each link cryptographically verified.
• Recipes win. A service that already mounts the union is skipped whole; an environment variable the recipe set is left alone.
• The system ingress stack is excluded. traefik and forward-auth get their material through the dynamic-dir bundle (which gained the same anchors), and injecting into the stack that terminates the edge would churn cluster ingress for no benefit.
• Idempotent. A second pass over an already-injected compose is a no-op, so shc update does not recreate containers forever.

Everything fails soft. A wedged edge, a firewalled AIA URL, or a chain that stops early yields the last known anchor set (or none) — never an error, and never a failed render.

/etc/shc is a reserved, platform-owned mount namespace. Inside a compose recipe an absolute source under it is not a host path — it is a name the platform rewrites to whichever host file backs it on this node — so a recipe writes the same string on both sides of the bind and never learns the host layout:

volumes:
  - /etc/shc/tls.ca:/etc/shc/tls.ca:ro

Named for what the file is to the container, not for how the platform builds it. Three spellings of one file (/etc/shc/ca-union.pem, /etc/shc/ssl-union.pem, /etc/dex/shc-ca-union.pem) collapse into tls.ca; /etc/shc/ca-truststore.p12 becomes tls.p12.

core/volume resolves the namespace and fails closed on an unknown leaf. That matters: on a root install the host’s real /etc/shc is SHC’s own config home, so a typo like /etc/shc/tls-ca would otherwise bind daemon configuration into a tenant container. Both mounts also require :ro — the host files are node-scoped and shared by every tenant, so a writable bind would let one tenant rewrite the anchors every other tenant’s containers load.

The “should I materialize the file” trigger keys on the container path, the one token that survives volume resolution (the source half is rewritten, the target half never is). The old trigger keyed on the host file name, which would have stopped firing the moment the host location moved — leaving the mount present and the file absent, i.e. a directory where a trust store should be.

Both artefacts moved from <deployDir>/files/ to <CacheHome>/mounts/.

A stack backup archives the deployment dir wholesale — the deployment tar carves out only mounts/ and app/, and the default restic snapshot path is the deployment dir — and modules/backup has no per-file exclusion mechanism at all (backup.yaml’s exclude.mounts reaches mount names only). So every archive carried a derived, node-specific trust bundle and truststore, and restore copied the stale ones back over the fresh ones. Restoring one estate’s edge anchors onto another estate is actively wrong.

Living outside deployments/ is the only thing that makes that impossible, and it needs no exclusion list anyone could forget to maintain. CacheHome is also the layout’s documented home for regenerable material, survives reboots (unlike RuntimeDir, a tmpfs wiped on unit stop) and the tmpfiles age sweep (unlike <StateHome>/tmp), and is untouched by removeRenderedArtifacts, which only ever walks a deployment dir.

Writes are skip-if-identical then atomic-rename, so a repeat render does not touch the inode — docker binds a file by inode, and a live container must not have its trust store swapped or torn mid-read.

The chain was derived from a 127.0.0.1 handshake against traefik’s websecure listener, using hostnames read from the traefik dynamic dir — both of which exist only on the Traefik host. The ingress sweep writes the dynamic dir behind AmITraefikHost, and nothing listens on the websecure port anywhere else. On every other node the hosts came back empty, the dial was refused, the anchors were nil, and the app rendered with no edge trust and no error.

That is not a corner case: stack.install is dispatched to the node a stack targets, so a worker-hosted app is rendered on the worker.

Derivation does not move. The Traefik host still probes its own listener and walks AIA; it publishes the result — anchors and the ACME hostnames — on /internal/rpc/edge_ca.anchors, and every other node fetches it.

Anchors only exist once the edge is serving a real ACME certificate. An app installed before the estate’s first successful issuance renders without them; the next update/recreate picks them up, because the bundle is re-materialized on every render. Anchors are cached in-process for 30 minutes, so a CA rotation self-heals within one TTL.

Under the swarm runtime a task may schedule on a node other than the one that rendered, and the platform mounts are host binds materialized on the rendering node. This is a pre-existing, systemic property of every FILE/SYSTEM mount in the catalogue (only DATA mounts are kept as docker-managed named volumes for swarm), not something this change introduces — but it is worth naming, because for the JVM leg the consequence is harsher: an absent truststore becomes an empty directory and the JVM then trusts nothing. Under compose, the supported multi-node path, the renderer and the runner are the same node, so the artefacts are always local. Closing the swarm case properly means teaching swarm about platform mounts (a config/secret, or a per-node materialize pass), and is deliberately out of scope here.

32 apps ship apps/<name>/sso.plug.yaml. Before this change, two mounted the CA union.

Recipe opt-in, kept — now on the canonical path:

Why openobserve’s opt-in cannot simply be deleted in favour of the injection. serviceDialsEdge does match both services — openobserve via O2_DEX_BASE_URL, dex via the issuer inside its command heredoc — so on a staging-CA estate the injection would cover them. But the injection is gated on non-empty anchors, i.e. it fires only on an estate whose ACME provider pins a non-production directory. openobserve needs the bundle on every estate: the enterprise image is distroless, so it cannot build one at entrypoint, and dex’s connector rootCAs replaces the system pool — on a stock public-CA estate, dropping the mount would leave dex unable to validate Keycloak’s ordinary public leaf. The recipe gate (shc.ca_root_b64) is a strict superset of the injection’s, which is exactly why it stays.

huly’s account is in the same position for the weaker reason that its NODE_EXTRA_CA_CERTS is additive; it is kept for symmetry and because the recipe documents which of huly’s services actually dials the edge.

Why affine had to be moved OUT of the injected set (live, 2026-07-27). The survey below originally listed affine as covered. It is not, and the reason generalises: serviceDialsEdge searches the service subtree, and AFFiNE is the one app in the catalogue whose OIDC issuer is not in its service at all — it lives in a top-level configs: entry (the rendered config.json, since AFFiNE dropped OAUTH_OIDC_* env in its 0.2x config rewrite). The service’s only own-edge https:// URL is AFFiNE’s own AFFINE_SERVER_EXTERNAL_URL, and on a first install that host is not an ACME router host yet — its route is created BY the install — so the predicate matches nothing and the app renders with no trust. Observed on the promanager hel1 corp estate (LE staging):

ERROR [OIDCProvider] Failed to validate OIDC configuration
TypeError: fetch failed / Error: unable to get local issuer certificate
LOG   [OAuthProviderFactory] OAuth provider [oidc] unregistered.
→ graphql serverConfig.oauthProviders = []   (sign-in modal shows no SSO button)

Keycloak had minted the client correctly and config.json carried the right issuer/clientId/secret — the only missing piece was the anchor. The recipe now opts in, which also removes the first-install ordering dependency. See “Still owed” for the predicate fix that would cover the class.

Now covered by platform injection (no recipe change needed). These consume a public-https issuer, had no CA trust at all, and run on a runtime one of the four injected variables reaches:

blockscout, defectdojo, gitea, gitlab, highlight, kutt, librechat, mailcow, matomo, matrix, minio, monica, netbird, nextcloud, paperless, peertube, postiz, supabase, vaultwarden, windmill

(affine was in this list and did not belong — it moved to the recipe opt-in table above. Every app that stays here carries its issuer in the SERVICE, which is the only place the predicate looks.)

Unaffected as wired — these resolve an internal http:// issuer, so no CA is involved: immich, mautic, wordpress, frappe. If any is ever repointed at the public issuer it falls into the injected set automatically.

The catalogue was swept for services that are BOTH JVM-based AND match the injection predicate (their rendered spec names an https:// URL on one of our own ACME hostnames):

JVM but no own-edge https anywhere, so out of scope entirely: sonarqube, wazuh-indexer, graphiti/neo4j, paperless/tika, paperless/gotenberg, huly/elastic, highlight/kafka, highlight/zookeeper.

So dependency-track and penpot are the complete set that genuinely needs this; the other three take a superset of their own trust and are unaffected either way. No recipe changes — the platform covers them, and a JVM app added to the catalogue tomorrow is covered for free.

1. minio’s sso.plug.yaml claims the derived issuer is internal, which contradicts InjectDerivedKeycloakIssuer’s public derivation. Worth reconciling — the injection covers it either way, but the comment is misleading.
2. Port the CE-tree unit coverage (modules/app/edge_ca_test.go, edge_ca_peer_test.go, inject_ca_trust_test.go, jvm_ca_trust_test.go, platform_mount_test.go, system_ca_union_edge_test.go, core/volume/platform_mount_test.go) into tests/unit/ if useful.
3. Live validation on a staging estate. The chain walk, the resulting trust decision, and both consuming runtimes were proven read-only against a live staging edge (below); an end-to-end shc install on such an estate has not been run, and neither has a live multi-node fetch of edge_ca.anchors from a worker.
4. serviceDialsEdge is blind to a service’s configs:/secrets:. A compose top-level configs: entry is app configuration in every sense that matters — affine’s OIDC issuer lives in one — but the predicate only walks the service subtree, so a service that references such an entry (configs: [{source: X}]) is judged on its env/command alone. Resolving the referenced top-level entries into the searched text would have covered affine for free and closes the class. Until then the affected recipe opts in by hand.
5. Inert predicate matches. keycloak, metabase and the two jitsi services match serviceDialsEdge on a self-advertised or link-generation URL rather than a real dial, so on a staging-CA estate they take a container recreate (and, for keycloak, a truststore) they do not need. Harmless — the truststore is a superset of their own — but tightening the predicate to ignore self-advertisement keys would save the churn.

Against a live LE-staging edge, read-only, driving the production collector and AIA fetcher:

anchor: subject="CN=(STAGING) Ersatz Emmer YR2,O=Let's Encrypt,C=US"      selfSigned=false
anchor: subject="CN=(STAGING) Yonder Yam Root YR,O=ISRG,C=US"             selfSigned=false
anchor: subject="CN=(STAGING) Pretend Pear X1,O=(STAGING) ISRG,C=US"      selfSigned=true

openssl verify of the served leaf:

host bundle only            → error 20 ... unable to get local issuer certificate
host bundle + edge anchors  → OK

Node v25 TLS against the same edge:

default trust                              → authorizationError = UNABLE_TO_GET_ISSUER_CERT_LOCALLY
NODE_EXTRA_CA_CERTS=<the new union>        → authorized = true

The first line is verbatim the failure huly’s account service died on.

Driving the production builder (buildJVMTrustStore) on the anchors the production collector had just walked, then a real JVM in the real app image against https://auth.corp.example.com/realms/corp/.well-known/openid-configuration:

store  name=shc-ca-truststore-72f46018958e.p12  type=PKCS12  bytes=192245
second call → cache hit, 0ms, byte-identical

dependency-track (Temurin 25)
  before → javax.net.ssl.SSLHandshakeException: PKIX path building failed:
           SunCertPathBuilderException: unable to find valid certification
           path to requested target
  after  → OK http=200 {"issuer":"https://auth.corp.example.com/realms/corp",…

penpot (Zulu 25)
  before → same SSLHandshakeException
  after  → OK http=200

Additivity, through the merged penpot store — public TLS is untouched:

https://letsencrypt.org/   OK http=200
https://api.github.com/    OK http=200
https://pypi.org/          OK http=200

And the disqualifying counter-measurement for the host-side conversion, same JVM, same anchors, store produced by openssl pkcs12 -export -nokeys:

keytool -list → "Your keystore contains 0 entries"
handshake     → SSLException: InvalidAlgorithmParameterException:
                the trustAnchors parameter must be non-empty