Vultr
Operator guide for standing up an SHC cluster on Vultr
cloud compute. Built from a live reference estate (region ewr,
first applied 2026-07); every quirk below was found on real hardware, not
guessed.
At a glance
Section titled “At a glance”| Concern | Value |
|---|---|
| Terraform module | terraform/modules/shc/vultr (front door) |
| Example root | terraform/examples/vultr-shc |
| Substrate provider | vultr/vultr ~> 2.0 |
| SHC connection type | vultr, CredFields api_key, region |
| DNS approach | Cloudflare-parent (Vultr DNS refuses subdomain zones; see gotchas) |
| DNS connection type | cloudflare, CredFields token |
| Volume provider | vultr-volume (block storage; region-scoped) |
| Load-balancer provider | vultr-lb (SharedVIP; tcp/http/https only) |
| Node image | Debian 13 x64 (trixie) (x86 / amd64 plans only) |
| Credentials (env) | VULTR_API_KEY, plus a Cloudflare token for DNS |
One Vultr API key drives both the substrate (the vultr Terraform
provider) and the daemon’s vultr connection (dns/proxy/volume). A Vultr
API key is account-level (it cannot be scoped into per-surface sub-keys), so
the daemon reuses the account key exactly as the DigitalOcean estate reuses its
account token. There is no scoped daemon sub-credential the way AWS/GCP have.
Credentials
Section titled “Credentials”Vultr API key
Section titled “Vultr API key”-
Vultr customer portal → Account → API.
-
Enable API access and copy the Personal Access Token.
-
Turn off the IP allow-list (portal: “Access Control”); see the gotchas below. The default allow-list contains only your current IP, which silently 403s any CI runner or shared build host.
-
Export it:
Terminal window export VULTR_API_KEY=... # substrate + the vultr connectionexport TF_VAR_vultr_api_key="$VULTR_API_KEY"The
vultrTerraform provider readsVULTR_API_KEYnatively; the estate also wires the provider block fromvar.vultr_api_keyso the same key feeds the SHCvultrconnection with one source of truth.
Cloudflare token (for DNS)
Section titled “Cloudflare token (for DNS)”Because Vultr DNS cannot host subdomain zones, managed DNS runs through
Cloudflare on the parent zone. Mint a token scoped Zone:DNS:Edit on the one
zone you will serve records under:
export TF_VAR_dns_token=... # Cloudflare API token, Zone:DNS:EditKeep both out of *.tfvars files and out of Terraform state variables; pass
them as TF_VAR_* / environment only.
The SHC connection
Section titled “The SHC connection”Two connections light the platform. The estate wires them as first-class
shc_connection resources at system scope (provider ≥ 0.0.322), so the
tenant-less cluster-host DNS surface resolves them directly:
# Proxy (vultr-lb) + volume (vultr-volume). Its DNS surface goes UNUSED:# Vultr DNS can't host subdomain zones (see gotchas).resource "shc_connection" "vultr" { name = "vultr" type = "vultr" credentials = { api_key = var.vultr_api_key # the same account key region = module.shc.region # required by vultr-lb AND vultr-volume }}
# Managed DNS on the parent zone.resource "shc_connection" "cloudflare" { name = "cloudflare" type = "cloudflare" credentials = { token = var.dns_token } zones = ["example.com"] # YOUR Cloudflare-hosted parent zone}Then the post-init defaults, one YAML document (provider ≥ 0.3.0):
resource "shc_config" "platform" { yaml = <<-EOF dns: provider: cloudflare # the connection NAME public_address: ${local.bootstrap_ip} storage: volumes: provider: vultr # vultr-volume dispatch (carries region) acme: email: [email protected] # arms public Let's Encrypt (DNS-01) EOF depends_on = [shc_connection.vultr, shc_connection.cloudflare]}region is required in the vultr connection credentials: both vultr-lb and
vultr-volume are region-scoped and provision in it.
DNS: why Cloudflare-parent, not Vultr DNS
Section titled “DNS: why Cloudflare-parent, not Vultr DNS”The obvious design (delegate vultr.example.com into Vultr DNS with a
vultr_dns_domain + parent NS records) does not work. Vultr DNS accepts
only registrable domains; a subdomain zone is rejected at create:
Error: error while creating domain : {"error":"Unable to create domain: Subdomains are not permitted","status":400}So every SHC estate on Vultr hosts its *.<sub>.example.com records as ordinary
rows inside the Cloudflare parent zone, the same shape the Hetzner and
DigitalOcean estates use. The vultr connection still lights vultr-lb and
vultr-volume; only its DNS surface goes unused.
SHC writes only records for hosts it claims, each guarded by a companion ownership TXT: it never touches records it did not create, so pointing it at a zone you also use for other things is safe.
With dns.provider set and bootstrap.host seeded, the daemon
self-registers the cluster-host A record (cloud.<sub>.example.com) every
leader tick (~60s) and fails closed rather than publish a LAN address; there is
no hand-managed shc_dns_record resource to maintain.
Volumes and load balancers
Section titled “Volumes and load balancers”vultr-volume: block storage, region-scoped. Setstorage.volumes.provider: vultr(as above); the daemon provisions, attaches (by node name), formats, and mounts a volume for any app declaringx-shc-volumes. Recreate caveat: Vultr assigns each node’s VPC plane IP itself (nocidrhostpinning on this substrate), so treat node replacement as remove-node + join-node, not an in-place recreate. Detach caveat: Vultr often refuses a live detach from a running instance (“Unable to unmount block device”, for as long as it is asked) and takes one from a stopped instance at once. A non-forced detach from a running node therefore fails withX409351VLTLIVand never reloads the node; a forced detach (the node is asserted gone) follows a refusal with Vultr’s reloading detach, which restarts that instance (SHC-1800).vultr-lb: a SharedVIP Load Balancer in front of the system Traefik. It is off by default; flip it on with anshc_stack_portsattach (vultr@80/vultr@443).vultr-lbis tcp/http/https only: a UDPconn@mapping is rejected at install. For a single-node or direct-ingress cluster you don’t need it: Traefik on 80/443 on the bootstrap node, with the cluster host pointed at that node’s IP, serves the edge directly.
Minimal end-to-end deploy
Section titled “Minimal end-to-end deploy”Full worked root: terraform/examples/vultr-shc.
It mirrors the live estate: one module "shc" (infra + bootstrap) plus the
vultr + cloudflare connections and shc_config, with the DNS provider, base
domain, and name prefix all exposed as variables so you can point it at your own
single Cloudflare zone.
cd terraform/examples/vultr-shc
export VULTR_API_KEY=...export TF_VAR_vultr_api_key="$VULTR_API_KEY"export TF_VAR_dns_token=... # Cloudflare Zone:DNS:Edit token
tofu init # needs the provider network-mirror config (~/.tofurc); see belowtofu apply \ -var dns_zone=example.com \ -var cluster_name=shc-vultr \tofu apply alone provisions the instances + VPC + firewall groups, installs
the .deb and runs shc init / shc node join over the provider’s SSH
transport, wires both connections and the platform config, and (via the demo
shc_deployment in the example) installs a public whoami app that gets a real
Let’s Encrypt certificate through Cloudflare DNS-01.
Verify:
tofu output -raw ssh_private_key > /tmp/vultr-key && chmod 600 /tmp/vultr-keyssh -i /tmp/vultr-key admin@"$(tofu output -json node_management_ipv4s | jq -r '.[0]')" \ 'sudo shc cluster status; sudo shc connection ls'
curl -fsS https://whoami.example.com/ # 200 + request dumpecho | openssl s_client -connect whoami.example.com:443 \ -servername whoami.example.com 2>/dev/null | openssl x509 -noout -issuer # Let's EncryptProvider mirror:
terraform-provider-shcis not on the public registry.tofu initneedsTF_CLI_CONFIG_FILE/~/.tofurcpointing at the SHC network mirror (https://selfhostedcloud.org/terraform/0.1/); a baretofu initfails withregistry.opentofu.org does not have a provider named .../shc. See the repo’sPUBLISHING.md.
First-live gotchas
Section titled “First-live gotchas”Each of these was found on a real Vultr apply. Symptom first, then the fix.
2. Vultr DNS refuses subdomain zones → Cloudflare-parent
Section titled “2. Vultr DNS refuses subdomain zones → Cloudflare-parent”Symptom: creating a vultr_dns_domain for sub.example.com fails with
400 "Subdomains are not permitted".
Fix: don’t delegate a subdomain into Vultr DNS. Host the records in the
parent zone’s own provider (Cloudflare) and point the daemon at a cloudflare
connection instead; see DNS above.
This is the documented default estate shape on Vultr.
3. vultr_ssh_key trailing-newline perpetual diff
Section titled “3. vultr_ssh_key trailing-newline perpetual diff”Symptom: every tofu plan shows a benign in-place update to the
vultr_ssh_key resource, forever, even with no change.
Fix: trimspace() the key material. tls_private_key’s
public_key_openssh carries a trailing \n, but the Vultr API returns the
stored key normalized without it, so the values never compare equal. The
module already wraps ssh_key in trimspace(); do the same in any hand-rolled
vultr_ssh_key.
4. The API key is IP-allow-listed by default
Section titled “4. The API key is IP-allow-listed by default”Symptom: Terraform / the daemon get 403 from the Vultr API from a CI
runner or shared build host, even though the key is valid; it works from your
laptop but not from automation.
Fix: in the Vultr portal (Account → API → Access Control) set the
allow-list to allow-all (0.0.0.0/0), or add the runner’s egress ranges.
Shared CI runners have non-allowlistable, rotating egress IPs, so allow-all is
the practical setting for automation. (Rotate/retire the key when the estate is
torn down.)
5. Plane NIC names and no plane-IP pinning
Section titled “5. Plane NIC names and no plane-IP pinning”Symptom (minor): the VPC interface is enp8s0 (public is enp1s0) on
Debian 13; set nic_name if your plan/image differs. And because Vultr assigns
the VPC address (no cidrhost pinning), a node recreate may hand it a
different plane IP that stored membership won’t follow.
Fix: the module derives enp8s0/enp1s0 automatically (override via
var.nic_name). Treat node replacement as remove + rejoin, not in-place
recreate.
Backups
Section titled “Backups”Backups to Vultr Object Storage are not wired by the module: a known gap.
Vultr Object Storage is S3-compatible, so restic can target
s3:<region>.vultrobjects.com/<bucket>/shc with
backup.aws_access_key_id / backup.aws_secret_access_key set to a Vultr
Object Storage key pair. But those keys are minted per-object-store in the
Vultr console, not through the Terraform provider, and the same keys are
needed just to create the bucket, so there is nothing to wire from Terraform.
Provision the bucket + key by hand and set the backup.* config if you need
off-cluster backups.
Teardown
Section titled “Teardown”Vultr is the substrate that named a whole class of bug: tofu destroy removes
every instance, exits, and leaves the network tier (VPC, firewall groups,
firewall rules, SSH key) standing in the cloud and described in state. It
happened twice on the same estate (promanager ewr, 2026-07-14 and
2026-07-19) and both times the teardown was recorded as successful.
Why it happened
Section titled “Why it happened”DELETE /v2/instances/<id> returns while the instance is still in
destroying, and the provider’s instance delete does not wait that out; it
detaches the VPC, issues the delete, and returns. tofu destroy walks
dependents-first, so the network tier destroys last, which means its
deletes fire the instant the instance delete returns: straight into the purge
window, against objects Vultr still considers attached. vultr_firewall_group’s
delete carries no retry at all and vultr_vpc’s retry engages on one exact API
message, so those deletes simply fail.
Both halves are now fixed in terraform/modules/shc/vultr:
time_sleep.instance_purgesits between the two tiers. Itsdepends_onnames the whole network tier andvultr_instance.nodedepends on it, so on destroy the order isinstances -> 120s settle -> reserved-IP / firewall / VPC / SSH-key deletes. Create-side cost is zero (onlydestroy_durationis set).- The settle’s membership rule is enforced, not remembered. Every resource the
instance block references must be behind the settle: a referenced resource
is one the instance is attached to, so its delete is the delete the
half-purged instance blocks.
TestCloudModuleSettleCoversEveryAttachedResource(tests/unit/ci/cloud_module_teardown_settle_test.go) derives that set from the.tfsource for every cloud module and fails if anything is missing. It was written because the list had already fallen behind twice: vultr shipped the settle with the SSH keys omitted (promanager-vultr-keywas found stranded beside the VPC the settle had been written for), and scaleway shipped it withscaleway_instance_ip.node(the server’s ownip_id) omitted.
tofu destroy exiting 0 is not proof
Section titled “tofu destroy exiting 0 is not proof”The lane shells used to treat the destroy’s exit code as the only evidence, so
a destroy that deleted the instances and stranded the tier reported CLEAN.
teardown_verdict now reads state back (teardown_state_residue in
tests/lib/tf-harness.sh): an address only leaves state when its Delete
succeeded, so anything still listed after a “successful” destroy is something
the destroy did not delete, and the lane exits INCOMPLETE.
That is still only a state check. State and cloud can disagree in both
directions (drop_platform_from_state divorces them deliberately), so for a
real teardown verdict the cloud has to be read through the provider API:
tests/clouds/lib/reap-run.sh --cloud vultr --prefix <cluster-name>tests/clouds/hostinger/scripts/destroy.sh has the shape to copy: it queries
the provider API after the destroy and fails the lane on what it finds, rather
than on what the destroy claimed.
The ewr residue, and how to clear it by hand
Section titled “The ewr residue, and how to clear it by hand”Two tofu states in promanager-ops still describe an estate that no longer
exists. Cloud pruning is deferred by owner ruling, so this is the sequence,
not a thing that has been run:
| state | serial | what it still describes |
|---|---|---|
infra.ewr.vultr.promanager.vip | 14 | 13 resource blocks / 31 instances: 1 vultr_vpc, 2 vultr_firewall_group, 24 vultr_firewall_rule (4 ssh + 4 icmp + 16 public), 1 vultr_ssh_key, plus tls_private_key.cluster, random_password.vault and local_sensitive_file.cluster_key. No vultr_instance block at all. |
deployments.ewr.vultr.promanager.vip | 4 | 3 shc_deployment (gitea, postgres, whoami), 1 shc_user, 1 shc_repo, 1 random_password, all against a cluster with no nodes |
Two things to understand before touching either.
The state carries live secrets. tls_private_key.cluster,
random_password.vault and local_sensitive_file.cluster_key are stored in
plaintext in the state JSON. The nodes they unlock are gone, but
vultr_ssh_key.this still holds the matching public half in the Vultr account,
so the pair is not inert until the key is gone from the account too. Treat the
state as a credential, and revoke the Vultr API key used to create the estate
when you are done.
The 2026-07-31 sweep already deleted 8 Vultr objects: two VPCs, four firewall groups and two SSH keys, covering both generations. So the cloud is expected to be empty and the state is expected to be a stale description of nothing. Expected is not verified: nothing on the dev box holds a Vultr API credential, so this has not been checked against the Vultr API.
The sequence:
-
Read the cloud first, through the API, not through tofu. With a Vultr API key that can list (
Account -> API, and mind the IP allow-list; see First-live gotchas #4):Terminal window for p in instances vpcs firewalls ssh-keys reserved-ips; doprintf '== %s\n' "$p"curl -sS -H "Authorization: Bearer $VULTR_API_KEY" \"https://api.vultr.com/v2/$p?per_page=500" | jq -r '..|.label?//.description?//.name?|strings'doneAnything matching the estate’s
cluster_nameis real residue. Nothing matching means the state is describing objects that are already gone. -
If the cloud is empty, do not run
tofu destroyto “clean up” the state. A destroy against absent objects depends on every provider delete being 404-tolerant, and the one thing this module has proven is that its deletes are not uniformly forgiving.terraform state rmthe addresses, or delete the state outright; the estate root is already removed frompromanager-ops, so there is nothing left to plan against:Terminal window # GitLab-backed http backend; the state name is the dotted form abovecurl -sS --request DELETE \--header "PRIVATE-TOKEN: $GITLAB_TOKEN" \"https://gitlab.com/api/v4/projects/<id>/terraform/state/infra.ewr.vultr.promanager.vip"Delete
deployments.ewr.vultr.promanager.vipthe same way. Do the deployments state first: it holds adata.terraform_remote_state.infrapointing at the other one. -
If the cloud is not empty, delete the objects in dependents-first order (firewall rules, then firewall groups, then the VPC, then the SSH key and any reserved IPs), re-reading the API after each step. A
DELETEreturning 204 is not evidence either; absence in a follow-upGETis. -
Verify by absence, then remove the state. Re-run step 1 and confirm it returns nothing before deleting the state, or you lose the only record of what is left.
The general rule this section exists to state: a teardown is verified by reading the provider, not by reading an exit code. Every deletion in the 2026-07-31 sweep was confirmed by absence in a follow-up read, and that is why its ledger can be trusted.