DigitalOcean

Standing up an SHC cluster on DigitalOcean droplets with the terraform/modules/shc/digitalocean front door, plus the platform wiring (DNS, Let’s Encrypt, block volumes) the daemon needs afterward. This page is the operator reference for the live-verified DO estate; the runnable mirror of everything below is terraform/examples/digitalocean-shc/, and the standing live smoke test is tests/clouds/digitalocean/.

DigitalOcean is the most complete substrate in the daemon: one account token lights all three provider slots — DNS, the load-balancer proxy, and block volumes. AWS and GCP each leave a slot dark; DO does not.

You need one DigitalOcean API token, and — unless you delegate a native DO DNS zone — one Cloudflare token.

Mint a Personal Access Token with read + write scope at cloud.digitalocean.com → API → Tokens. DO tokens are account-level: unlike AWS/GCP there is no scopeable daemon sub-credential (no per-surface IAM user / service account), so the daemon’s digitalocean connection reuses the same account token terraform itself runs with — exactly as the Hetzner estate reuses its hcloud project token.

The token is consumed under two spellings of the same value:

Gotcha — the TF_VAR name is do_token, not digitalocean_token. The SHC connection’s token arrives via TF_VAR_do_token. Setting TF_VAR_digitalocean_token does nothing; the connection’s token credential stays empty and the connect-time prober fails. Set both spellings to the one PAT:

export DIGITALOCEAN_TOKEN="dop_v1_..."
export TF_VAR_do_token="$DIGITALOCEAN_TOKEN"

Keep the token in the environment (or CI/CD variables) only — never in a .tfvars file or in terraform state.

If you delegate a native DO DNS zone out of a Cloudflare parent (the estate shape), you need a Cloudflare token scoped Zone:DNS:Edit on the parent zone to write the delegating NS rows — TF_VAR_cloudflare_token / CLOUDFLARE_TOKEN. Testers who point straight at a Cloudflare zone they own (the example’s default) give the same token to the SHC cloudflare connection instead. See DNS.

resource "shc_connection" "digitalocean" {
  name = "digitalocean"
  type = "digitalocean"
  credentials = {
    token  = var.do_token       # TF_VAR_do_token — same PAT as DIGITALOCEAN_TOKEN
    region = module.shc.region  # REQUIRED by dolb + dovolume; ignored by dns
  }
  zones = ["do.example.com"]    # the DO-hosted zone the daemon manages
}

region is a required credential field: both the dolb proxy and the dovolume volume providers provision region-scoped resources and need to know which region. The dns slot ignores it. The connect-time prober validates the token and confirms zone visibility, so credential mistakes fail loudly at apply, not silently at runtime.

Since provider >= 0.0.322 the connection is created at system scope, so tenant-less surfaces (the cluster host’s own DNS record) resolve it directly, and every tenant’s apps see it by longest-suffix zone match — no per-tenant duplicate connection is needed.

The estate uses native DigitalOcean managed DNS. DO bundles managed DNS with every account at no charge, so — like GCP and Azure, and unlike the Exoscale / Scaleway / OpenStack / Vultr estates that pivoted to a Cloudflare parent because their DNS is a paid add-on or absent — the DO estate keeps the cloud-native shape:

• a DO-hosted do.example.com zone (created empty by terraform; the daemon fills every record through the account token),
• NS-delegated out of the Cloudflare-hosted parent example.com.

DigitalOcean uses three fixed anycast nameservers for every domain — ns1.digitalocean.com, ns2.digitalocean.com, ns3.digitalocean.com — there is no per-zone computed nameserver list to read back (the delta vs GCP’s google_dns_managed_zone.name_servers), so the delegation targets are constants:

resource "digitalocean_domain" "do" {
  name = "do.example.com"        # empty — the daemon owns every record
}

resource "cloudflare_record" "do_ns" {
  count   = 3
  zone_id = data.cloudflare_zone.parent.id
  name    = "do"
  type    = "NS"
  content = ["ns1.digitalocean.com", "ns2.digitalocean.com", "ns3.digitalocean.com"][count.index]
  ttl     = 3600
  proxied = false
}

Once dns.provider names the connection, the daemon self-registers the cluster host’s A record every leader tick and mints/maintains each public app host’s record — no per-record terraform.

Delegating a native DO subzone is estate-grade. A tester who just wants a cluster on DO but only has a single Cloudflare-hosted domain should skip DO DNS entirely and let the daemon manage records straight in their Cloudflare zone. That is exactly what the example does by default — dns_provider = "cloudflare":

resource "shc_connection" "cloudflare" {
  name        = "cloudflare"
  type        = "cloudflare"   # CredFields: [token]
  credentials = { token = var.dns_token }
  zones       = [var.dns_zone] # the Cloudflare zone you already control
}

The DO substrate still runs on the DO token; only the DNS slot moves to Cloudflare. Volumes and the load balancer stay on the digitalocean connection. Both cloudflare and digitalocean are in the set of DNS impls whose SHC name equals their lego provider code, so DNS-01 Let’s Encrypt issuance works on either without translation.

Public certs are armed by two shc_config keys — the ACME contact email and the daemon’s public address (the bootstrap node’s reserved IP, which the DNS-01 challenge and the browser SSO redirect both land on):

resource "shc_config" "platform" {
  yaml = <<-EOF
    dns:
      provider: digitalocean        # or "cloudflare" — the connection name
      public_address: ${local.bootstrap_ip}
    acme:
      email: ops@example.com
    storage:
      volumes:
        provider: digitalocean      # dovolume dispatch (region rides the connection)
  EOF
  depends_on = [shc_connection.digitalocean]
}

Intra-key ordering does not matter: the traefik re-render after acme.email lands is daemon-owned (the ingress sweep recreates traefik within ~60s), so at worst a route serves the default cert for one sweep tick before the real LE cert.

• dovolume — DO block storage, region-scoped. Wire it with storage.volumes.provider = digitalocean (the connection carries the region it needs). The volume provider attaches by resolving the SHC node name against the droplet name, so keep droplet/node names stable.

• dolb — the DO Load Balancer in front of the system traefik. The digitalocean connection already lights the proxy slot; attach it with a port-grammar mapping (dolb supports UDP, unlike haproxy):

  resource "shc_stack_ports" "traefik_lb" {
    tenant     = "master"
    stack      = "traefik"
    ports      = ["digitalocean@80:traefik:80", "digitalocean@443:traefik:443"]
    depends_on = [shc_connection.digitalocean]
  }

  The estate runs traefik directly on the bootstrap node’s reserved IP and leaves dolb off, so this is the documented enable-path, not the default.

The runnable version of this is terraform/examples/digitalocean-shc/ — copy it, set the variables below, and apply. The example folds substrate + bootstrap + platform + two demo apps into one root.

1. Credentials.

   export DIGITALOCEAN_TOKEN="dop_v1_..."   # DO PAT, read+write
   export TF_VAR_do_token="$DIGITALOCEAN_TOKEN"
   export TF_VAR_dns_token="cf_..."         # Cloudflare token (default dns_provider)
   export TF_CLI_CONFIG_FILE="$PWD/registry.tfrc"   # the shc provider network mirror

2. Point it at your domain. In terraform.tfvars (or -var):

   cluster_name = "demo"
   region       = "nyc3"          # has block storage + LBs
   dns_provider = "cloudflare"    # or "digitalocean" for a delegated DO zone
   dns_zone     = "example.com"   # a zone your dns_token controls

3. Apply in two phases. The provider "shc" SSH host is the bootstrap node’s reserved IP — a module output that is unknown until the droplets exist, so apply the substrate first, then the platform:

   tofu init
   tofu apply -target=module.shc   # droplets + VPC + firewalls + bootstrap
   tofu apply                      # connection + config + demo apps

4. Verify. tofu output prints the node IPs; the demo whoami app comes up at whoami.<dns_zone> with a real Let’s Encrypt cert, and postgres runs LAN-internal. SSH in with the generated key (tofu output -raw ssh_private_key) as admin.

Every item below was hit on the first live apply (2026-07-13/14) against a real DigitalOcean account. AWS and Hetzner do not exhibit these — they are DO-substrate-specific.

Symptom. Droplets boot, but Docker / the shc daemon never appear; cloud-init status --long reports unacceptable character #x0080 and the whole user_data document was rejected. On the first apply this hit 3 of 4 debian-13-x64 nodes and installed nothing.

Cause. DO delivers user_data through a ConfigDrive whose pipeline double-encodes UTF-8: a comment em-dash (—, bytes E2 80 94) came back as C3 A2 C2 80 C2 94, embedding a raw U+0080 control byte that made cloud-init reject the entire YAML — so it ran none of it. AWS and Hetzner carry the same em-dashes in identical templates but deliver user_data via base64 / the metadata API, which preserve UTF-8; only DO trips this.

Fix. Keep cloud-init templates pure ASCII — including comments. The module’s templates/cloud-init.yaml.tftpl is ASCII-clean; if you author your own DO user_data, use - / -- and plain quotes, never typographic punctuation.

Symptom. A rebuild dies before it creates anything:

Error: Error creating VPC: POST https://api.digitalocean.com/v2/vpcs: 422 ...
a VPC with the same name already exists

DO namespaces VPC names per account, so <cluster_name>-vpc can exist exactly once. Retrying never helps — the name is taken until the old VPC is gone or adopted.

Cause. Almost always a previous tofu destroy that did not finish. Two ways to get there:

• The destroy failed early (a wedged cluster, an unresolvable provider) and never reached the VPC.
• Someone ran tofu state rm 'module.shc.digitalocean_vpc.this' to get past a VPC delete failure. Do not do this. The module has no data "digitalocean_vpc" fallback, so nothing re-adopts the VPC on the next run — the state-rm divorces a live VPC from terraform permanently and manufactures this exact 422.

What DO actually refuses is deleting a VPC that still has members (403 Can not delete VPC with members) — a race against droplet deletion that clears itself within seconds. Wait it out; do not state-rm.

Fix. Either adopt the existing VPC back into state:

tofu import 'module.shc.digitalocean_vpc.this' <vpc-uuid>

(The import plans clean because the module’s name + ip_range match the existing VPC.) Or delete it once its members have drained:

curl -sS -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  https://api.digitalocean.com/v2/vpcs/<vpc-uuid>/members   # must be empty
curl -sS -X DELETE -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  https://api.digitalocean.com/v2/vpcs/<vpc-uuid>

The test lane automates the second path — tests/clouds/digitalocean/scripts/destroy.sh sweeps <prefix>-vpc over the API after tofu destroy, waiting out the member drain, and test.sh re-runs that sweep before its first apply so an interrupted run cannot wedge the next one.

The one genuinely undeletable VPC is a region default ("default": true from /v2/vpcs) — the first VPC in a region, which DO will not delete and which there is no terraform-expressible way to demote. Leave it in place and let the cluster keep using it; it is free and shared. A cluster VPC created in a region that already has a default is not the default and deletes fine.

Symptom (historical). The DO DNS API rejects a record with ttl <= 30; an early daemon build sent ttl <= 0 and every record create failed.

Fix. Fixed in the daemon as of the 0.3.0 deb — the DNS self-registration path now sends a correct default TTL, so there is no terraform-side TTL knob to carry forward. Pin the 0.3.0 (or newer) provider and module tarball together and this is a non-issue. It is documented here only so an operator on an older build recognizes it.

Symptom. Records created in the DO zone don’t resolve on the public internet until the parent zone delegates to DO — and there is no per-zone name_servers attribute to wire (the GCP habit).

Fix. Delegate with the three constant anycast nameservers ns1/ns2/ns3.digitalocean.com (see DNS). Scope the parent-zone edit strictly to the NS rows of your subdomain — never touch other records in the parent zone.

For completeness, these DO assumptions held on first live apply and need no workaround: the debian-13-x64 image slug exists and lands the account key on root; cloud-init runs natively; the VPC-private interface really is eth1; reserved-IP inbound NAT reaches the droplet (egress keeps the droplet’s own public IP); and the admin user (root login disabled) with NOPASSWD sudo is what the provider’s SSH engine dials.

Backups to DO Spaces. DO Spaces is S3-compatible, so restic would target s3:<region>.digitaloceanspaces.com/<bucket>/shc with a Spaces key pair set as backup.aws_access_key_id / backup.aws_secret_access_key. But Spaces access keys are account-level and minted in the DO console, not via terraform, and the digitalocean_spaces_bucket resource itself needs those same keys on the provider block just to create the bucket — so the estate omits the whole chain (same disposition as the GCP/Exoscale estates). To close it: add TF_VAR_do_spaces_key / TF_VAR_do_spaces_secret, set them on the provider, create a digitalocean_spaces_bucket, and wire the backup.* keys into shc_config.

Volume failover classifier. The daemon’s cross-node volume-move / dead-node-failover / restore-override layer currently hardcodes hcloud as the sole location-scoped block-volume arm, so dovolume degrades to backup+restore for those specific orchestration paths even though it implements the same capability interfaces. Provisioning, attach, and detach work; the gap is only the failover/restore-placement seam. See the over-fitting audit.