Google Cloud

SHC runs on GCE as a private-plane cluster: a custom-mode VPC network + one subnetwork in a single zone, N Compute Engine instances, and tag-targeted firewall rules — all folded behind one module "shc" front door (terraform/modules/shc/gcp, or the published shc-gcp-<ver>.tgz tarball). One tofu apply builds the substrate (Layer 1), installs the shc deb and runs init/join over the provider’s SSH transport (Layer 2), and — if you wire the platform block — lights DNS + Let’s Encrypt off a single connection.

The live reference estate is a gcp/us-central1 root kept in a private ops repo (a real, running cluster). This page documents what that estate actually does, plus the runnable terraform/examples/gcp-shc example and the tests/clouds/gcp smoke-test lane.

GCE is the one substrate with no cloud-init — provisioning is a startup-script shell translation of the cloud-init recipe. Most of the first-live findings below trace back to that fact.

There are two distinct credential sets, and they must not be confused.

The google provider reads these natively from the environment — never from a tfvars file or state:

The admin identity needs enough IAM to create instances, networks, firewalls, a Cloud DNS zone (if you use the native-DNS path), and a scoped service account — roles/compute.admin + roles/dns.admin + roles/iam.serviceAccountAdmin covers it, or roles/owner on a throwaway project.

The SHC daemon does not reuse your admin key. The estate mints a dedicated, least-privilege service account as a terraform resource and hands only its key to the gcp connection:

resource "google_service_account" "shc_daemon" {
  account_id   = "shc-daemon-gcp"
  display_name = "SHC daemon — scoped platform credentials"
}
# One project role per provider slot the daemon uses:
#   roles/dns.admin                 -> Cloud DNS provider (zone list + record writes)
#   roles/compute.loadBalancerAdmin -> gcp-tcp-lb proxy provider (target pools, forwarding rules)
#   roles/compute.instanceAdmin.v1  -> gcepd volume provider (disk create/attach/detach)
resource "google_service_account_key" "daemon" { service_account_id = google_service_account.shc_daemon.name }

The connection then carries base64decode(google_service_account_key.daemon.private_key) as its credentials_json — see §2. GCP IAM grants take ~a minute to propagate; if the connection’s connect-time prober fails on the first apply, just re-apply.

Connection type: gcp (modules/connections/capabilities.go).

One gcp connection lights three provider slots:

Create it at system scope (provider ≥ 0.0.322 does this automatically), so the tenant-less cluster-host DNS surface resolves it directly and default-tenant apps ride it with no per-tenant duplicate:

resource "shc_connection" "gcp" {
  name = "gcp"
  type = "gcp"
  credentials = {
    project          = local.project
    credentials_json = base64decode(google_service_account_key.daemon.private_key)
    region           = module.shc.region
    zone             = module.shc.zone
  }
  zones   = ["gcp.example.com"]      # connect-time prober validates zone visibility
  cluster = { ssh = local.app_ssh }
}

GCP is one of the few substrates that bundle managed-zone hosting for free (Cloud DNS), so it supports the native path. Both are valid; the right one depends on your domain.

• Native Cloud DNS, NS-delegated — what the live reference estate uses. You stand up a google_dns_managed_zone (gcp.example.com), delegate it out of a parent zone with four NS records pointing at the Cloud DNS name servers, and point dns.provider at the gcp connection. The daemon’s scoped SA writes records straight into that zone. Why: you get a real, isolated subdomain zone at no extra cost, and the daemon self-registers the cluster host into it. Best for a durable estate under a subdomain you control.

• Cloudflare-parent-mix — the default for the example + test lane. A single cloudflare connection (type = "cloudflare", cred token) writes records straight into your one Cloudflare-hosted zone. No Cloud DNS zone, no NS delegation. Why: simplest path for a tester who has exactly one Cloudflare domain and doesn’t want to provision or delegate a Cloud DNS zone. This is the convergent “cloudflare-mix” estate shape the multi-cloud campaign settled on for substrates without free bundled zone hosting — GCP supports it too.

NS delegation is a Cloud DNS fact worth repeating: Cloud DNS returns name servers with a trailing dot (ns-cloud-a1.googledomains.com.). trimsuffix(...) it before writing the parent-zone NS records — Cloudflare (and most parents) want the bare name.

Using terraform/examples/gcp-shc (cloudflare-mix DNS, one public demo app):

cd terraform/examples/gcp-shc

# admin GCP creds + the shc provider mirror
export GOOGLE_APPLICATION_CREDENTIALS=~/keys/gcp-admin.json
export TF_CLI_CONFIG_FILE="$PWD/registry.tfrc"     # teaches tofu the shc provider mirror

# inputs (never in tfvars)
export TF_VAR_project=my-gcp-project
export TF_VAR_dns_zone=example.com                 # your Cloudflare-hosted domain
export TF_VAR_cloudflare_token=...                 # Zone:DNS:Edit on example.com
export TF_VAR_acme_email=ops@example.com

tofu init
tofu apply

# verify
tofu output -raw ssh_private_key > /tmp/cluster.key && chmod 600 /tmp/cluster.key
ssh -i /tmp/cluster.key admin@$(tofu output -json node_public_ipv4s | jq -r '.[0]') \
  'sudo shc cluster status; sudo shc connection ls'
dig +short git.example.com
openssl s_client -connect git.example.com:443 -servername git.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer            # expect: issuer=...Let's Encrypt...

The example pins module "shc" to the shc-gcp-0.3.0.tgz Pages tarball (the SSH user is admin on that release), exactly like the estate. The tests/clouds/gcp lane instead builds the in-repo module + provider from the working tree (SSH user admin too, module pins >= the admin-user rename) — see the test README.

Every item below was observed on a real GCP boot during the cross-cloud campaign. Each is a one-layer fix, already landed.

Symptom. google-startup-scripts.service runs late in boot, after apt-daily/unattended-upgrades has already grabbed the dpkg lock. The startup-script died with apt exit 100 — Could not get lock /var/lib/dpkg/lock-frontend, held by apt-get, leaving docker installed but the compose plugin + git/gnupg/restic never installed — so the daemon’s docker compose -p … then failed with unknown shorthand flag: 'p' (compose v1 vs v2).

Fix. GCE Debian images run google-guest-agent, not cloud-init, so the recipe is a shell translation (modules/shc/gcp/templates/startup-script.sh.tftpl). Before any apt-get it stops and masks unattended-upgrades + the apt-daily[-upgrade] services/timers (a disable alone leaves the in-flight run holding the lock), and passes -o DPkg::Lock::Timeout=300 on every apt-get so a lock it didn’t win is waited for (up to 5 min), not failed on. set -e guarantees the /var/lib/shc-node-prepared marker is touched only after docker compose version succeeds — so a half-provisioned node never looks ready.

Symptom. The connection registry declares the cred field project, but the DNS factory’s buildGCP used to read project_id — a name mismatch that was papered over by a dns-local connCredAliases rename table (audit over-fit #7). A gcp connection carrying only project produced an empty project on the DNS path.

Fix (562a81811). buildGCP now reads project first, falling back to project_id only for older inline configs. The estate connection drops the project_id duplicate entirely. If you see empty-project zone misses on an old cluster, re-apply shc_connection to rewrite it at the current shape.

Symptom. Traefik solves the ACME DNS-01 challenge with lego, whose Google provider code is gcloud, not gcp. Rendering the SHC impl name (gcp) verbatim into dnsChallenge.provider made lego reject it and cert issuance silently fail (audit over-fit #2). Separately, the pre-0.3.0 estate had to SSH-upload the SA key file into the traefik ACME mount and set a credentials_file — brittle.

Fix (0fe8033c2 + 562a81811). A LegoProvider() method maps gcp → gcloud (also powerdns → pdns, azure → azuredns), so the resolver renders the code lego expects. And the gcp DNS provider’s TraefikEnv() now forwards credentials_json directly as lego’s GCE_SERVICE_ACCOUNT (inline JSON, no file) — the terraform_data.traefik_dns01_sa_key file-upload workaround is dead code. Note: this only bites the native gcp-DNS path; the cloudflare-mix path uses lego code cloudflare (identity), so it’s unaffected.

Symptom / behaviour. In the native-DNS estate, gcp.example.com is a real Cloud DNS zone delegated out of the Cloudflare-hosted parent. Cloud DNS assigns exactly four name servers under *.googledomains.com, and returns them with a trailing dot.

Fix / recipe. Create four NS records in the parent (use count = 4, not for_each — the name-server list is computed and unknown at plan time), and trimsuffix(..., ".") each name server before writing it. The daemon then self-registers the cluster host into the delegated zone, not into Cloudflare.

The whole cluster is pinned to one zone on purpose: gcepd provisions zonal persistent disks and refuses without a zone (X400344GPDCFG). Multi-zone node groups would strand volumes. The gcp connection’s zone cred is the placement pin (the EBS availability_zone analogue). Default regional static-IP quota is 8/region — a 4-node cluster with assign_static_ips uses 4.

Unlike EC2 user-data (first-boot-only but adoptable), GCE’s metadata_startup_script is baked at create time. Editing ssh_public_key, extra_authorized_keys, or flipping create_ssh_key on a live cluster replaces every node. To authorize a new key without a rebuild, append it by hand over an existing key (see the module README’s rotation recipe), and mirror it in terraform only alongside a planned roll.

The shc deb is amd64. Use x86 machine families (e2/n2/c3/…), never Arm (t2a/c4a). image_family defaults to debian-13; fall back to debian-12 if debian-cloud hasn’t published 13 in your project. The plane NIC is ens4 (virtio-net) — verify with ip a and override nic_name if a gVNIC-only family differs.

See also: the module README (GCP-vs-aws/hetzner delta table + first-boot verify checklist) and the live reference estate in your ops repo.