Skip to content
SHC Docs

Private names through the node resolver

shcd answers DNS itself on every node. It listens on the node’s mesh address (the mesh0 interface, in the mesh range: 10.64.48.0/20 on the default block, see the address block), UDP and TCP port 53, and answers from the node’s clusterfs revision, so it keeps answering from the last state it saw when the node is cut off. No container is involved: name lookup comes up with shcd, before anything it installs. It answers eight kinds of name:

NameAnswer
<node>.nodes.shc.internalthe node’s mesh address, to every asker but a tenant’s containers
<service>.<stack>.<environment>.<tenant>.shc.internalto a container it is in view of, the service’s VIP while the cluster runs VIPs, else the fabric addresses of its ready replicas; to the node itself, always the replicas’ addresses, because the kernel resets a packet the host sends to a VIP; TTL 5 s
<service>-r<ordinal>.<stack>.<environment>.<tenant>.shc.internalone replica’s fabric address, for a replica pinned under its own entry; never a VIP
a routed hostname, such as git.corp.example.comthe ready nodes in the app’s location, whose Traefiks all carry every route, at the address their published ports take: their underlay addresses, never a mesh address; to a tenant’s container the system Traefik service’s VIP instead, and without one the nodes’ public addresses; TTL 30 s
a name an app calls an integration provider by, such as prod.postgres.mydb or postgresthe same answer as the provider’s service, to the consumer’s own containers only
collector.shc.internalthe system collector deployment’s VIP; nothing while it has no replica
ingress-sources.shc.internalevery node’s ingress address, the one fabric address of each node that only the system Traefik holds, to every asker; NXDOMAIN while no node has one. An app trusts X-Forwarded-For from these
_replicas.<service>to a container of the service’s own deployment, the fabric addresses of the service’s ready replicas, never its VIP; also asked as _replicas.<service>.<stack>.<environment>.<tenant>.shc.internal; NXDOMAIN to every other asker; TTL 5 s

A service’s name also answers SRV, one record per port the service declares, asked and answered in view like its A record. The bare name gives the TCP ports; _tcp. or _udp. before it picks the protocol. Each record names the service itself, with its addresses in the additional section. A port range has no SRV record:

Terminal window
dig +short SRV postgres.mydb.prod.corp.shc.internal
0 0 5432 postgres.mydb.prod.corp.shc.internal.
dig +short SRV _udp.statsd.metrics.prod.corp.shc.internal
0 0 8125 statsd.metrics.prod.corp.shc.internal.

Every other name, including other records under a routed domain, goes to the node’s own upstream nameservers.

The names under shc.internal are answered by who asks. shcd reads who that is from the address the question comes from, never from the name asked.

AskerNames under shc.internal it is answered
the node itself (its own mesh address, or loopback)every name
a container, known by its fabric addressits own tenant’s services, the platform’s shared providers (Keycloak in master/default), the providers its deployment is plugged into, _replicas.<service> of its own deployment’s services, collector.shc.internal and ingress-sources.shc.internal
any other addressthe node names, collector.shc.internal and ingress-sources.shc.internal

A name outside the asker’s view is NXDOMAIN, the same answer a name that does not exist gets, so a tenant cannot learn another tenant’s deployments, services or addresses by asking for them. Node names are not answered to a tenant’s containers. Routed hostnames are answered to everyone, as they are public names, but not with the same addresses. mesh0 feeds no published port, so a routed hostname never answers a mesh address. A tenant’s container is answered the system Traefik service’s VIP, which it reaches wearing its own fabric address, so a LAN-only route of its own tenant answers it and another tenant’s does not (see published ports). While the node routes no such VIP it is answered the nodes’ public addresses only, and nothing where they have none. The node itself, the system tier’s containers and every other asker are answered the nodes’ underlay addresses.

A container is known only by the fabric address shcd has filed under its deployment. One that asks from another address is answered as a stranger. One case asks as the node and gets its view: a service with network_mode: host. Another node’s address is a stranger’s too. An install or update on the fabric has the node’s DNS take up its pins before it starts a container, and fails with X503273RSVNAM when the node cannot, so a new container is never answered as whoever held its address before. For the same reason a node whose answers have fallen behind the control plane for more than five minutes (half the fabric’s pin quarantine), because its clusterfs cannot build or its DNS cannot load the revision, answers its containers as strangers until it catches up. A node cut off from the control plane sees nothing newer and keeps its answers. Each node lists what it answers whom in /var/lib/shc/clusterfs/resolver/current/views.json.

To see what a container is answered, ask from inside it. dig on the node shows the node’s view, which is everything.

The private zones are shc.internal plus the zones of every private DNS connection: the ones it declares, and each public zone auto-zone mirrored into it so a host gets a private record beside its public one. A mirror lasts as long as a public connection owns its zone. When the public connection drops the zone, every private plane drops it too, and each node stops routing it within a minute. Each node lists them in /var/lib/shc/clusterfs/resolver/current/domains.json.

The DNS sockets, like shcd’s unix socket and API listeners, are handed to systemd’s file descriptor store (FileDescriptorStoreMax= on shc.service). A restart never closes them: queries that arrive while shcd restarts wait in the kernel and the next shcd answers them, late but not refused. The listener binds with free bind, so it holds the mesh address from boot, before mesh0 brings the link up. A node without systemd (the Mac) binds directly and retries until the address exists; X503272RSVBND in the log names an address it could not bind yet.

Each source address may send 2,000 queries a second, with bursts to 4,000. Past that its UDP queries are dropped and its TCP queries go unanswered, so one noisy container cannot crowd out the others or the control plane. Queries wait in a bounded queue that reports into the node’s pressure registry as resolver.dns, like every other queue: depth, oldest age, rates and refusals, exported as the shc_pressure_* metrics. A query a quarter of a second old grades the queue amber and one a second old red; refusals grade it amber.

Check that a node answers:

Terminal window
dig +short @<mesh address> <node>.nodes.shc.internal

Until a node answers, shcd routes nothing to it, and hosts and containers keep their own nameservers.

shcd routes the private zones to its own listener through systemd-resolved, once a minute on every node:

Terminal window
resolvectl dns mesh0 <mesh address>
resolvectl domain mesh0 ~shc.internal ~corp.example.com
resolvectl default-route mesh0 false

Only the private zones take that route; every other name keeps the node’s own nameservers. It is applied again when the mesh link comes back after a mesh0 restart, which drops those settings. If the listener stops answering, shcd runs resolvectl revert mesh0 and the node resolves the private zones through its own nameservers until it answers again.

A node without systemd-resolved is left alone. Install systemd-resolved on it to have the private zones routed.

Check a node:

Terminal window
resolvectl status mesh0
resolvectl query n1.nodes.shc.internal

X500271RSVHST in the daemon log means systemd-resolved refused the change. The node keeps its own nameservers and shcd retries a minute later.

When an app is installed or updated on a node whose listener answers, each service that has its own network namespace and declares no dns: gets dns: set to the node’s mesh address, followed by the node’s own nameservers as a fallback. Docker’s embedded resolver still answers the compose network’s own names first and forwards the rest there. A service with network_mode: or an authored dns: is left as it is. Apps installed before the listener answered pick it up on their next update.

A container asks its own deployment’s services by the bare service name. The node’s DNS looks a name with no dot up in the asking container’s deployment zone first, so db-1 from a replica of pgha answers db-1.pgha.<environment>.<tenant>.shc.internal, whatever the image’s libc and ndots. SHC states no ndots, so on docker the embedded resolver keeps its own ndots:0 and forwards such a name from the container’s own address. A name with no dot that the deployment has no service of goes on as asked, to the names its plugs gave it and then to the node’s own nameservers, so a LAN host such as nas keeps resolving.

It asks them by <environment>.<service>.<stack> too, the name {{ hosts.<service> }} renders in an app’s templates. From a container of that environment and stack the node’s DNS answers it exactly as it answers the bare name, on every runtime, so prod.valkey.wiki from a replica of wiki in prod is valkey.wiki.prod.<tenant>.shc.internal. Another deployment’s containers are not answered it unless a plug gave them the name. On a Docker node the embedded resolver answers it first, with the replica’s own address on the node’s bridge, when a replica of the service runs on the same node.

SHC writes no extra_hosts into an app’s compose. The ones an app’s author wrote stay as written.

An app’s jobs (hooks, socket and integration jobs run in a one-off container) ask the same DNS. On the SHC fabric each run joins the node’s fabric network with a pin of its own, filed under the job’s deployment, with dns: set to the node’s mesh address and the deployment’s zone as its search domain. The node answers that address what it answers the deployment’s own containers, including the names its plugs call providers by. The address is published to the node’s DNS before the container starts, and goes into quarantine like any released address when the run ends. Two runs of one job get two addresses. While the node’s DNS is not answering, a job of a deployment that calls providers by name is refused with X503275RSVDWN, the way its install would be.

An app plugged into a provider (-i db=mydb) keeps the hostnames its config has always carried: the socket’s hostname, the <environment>.<service>.<stack> name and the bare service name. On the SHC fabric the app is not joined to its provider’s network, so the node’s DNS answers these names the way it answers the provider’s service name: with the service’s VIP while the cluster runs VIPs, else with the fabric addresses of the provider’s ready replicas, whether the provider runs on the same node or another (the bare name has the limit described below). Which provider service each name means is written when the app is rendered, in the deployment’s stored spec, which every node’s DNS reads, so it follows the app’s -i flags and its plugs’ services: pins. The node answers the new names before the install or update starts a container, and keeps answering the names the previous render gave until compose-up has replaced its containers. A VIP stays the provider’s address while its service exists, through a move, a restart or a stop, and the kernel takes each new connection to a ready replica or resets it when there is none. Without VIPs, a provider that moves or stops changes the next answer within seconds. Either way the app is not re-rendered. These names carry no tenant, so the node answers them by who asks: only the consumer’s own containers, recognised by their fabric addresses, get the provider’s address. Anyone else asking the same name gets the upstream’s answer. An app rendered before its node answered these names gets them on its next update.

While the node’s DNS is not answering, an install or update whose services call providers by these names is refused with X503275RSVDWN. Nothing starts and a running app keeps its containers. shc doctor shows resolver.revision until the DNS answers again; retry then. Every node answers these names from the deployment’s stored spec, wherever its replicas run, so a replica that moves to another node keeps them. If the store cannot be read, the node keeps answering every name from its last revision until it can.

The bare service name is a known limit on a Docker node. Docker’s embedded DNS answers a bare name itself, before the node’s DNS sees the question, when any container on a network the asker shares has that service name, and every deployment on a node joins the node’s fabric network. A consumer whose provider is on another node should use the socket’s hostname or the <environment>.<service>.<stack> name.

Podman’s own resolver, aardvark-dns, answers a container on any network that has DNS enabled and forwards every other name to the container’s dns: servers from the host. The node’s DNS would then see the node’s own address and answer the node’s view, every tenant’s names.

So a Podman node of the SHC fabric has no aardvark-dns. That is the guarantee, and it does not depend on how a compose document is written or read: with none on the node, nothing forwards a container’s question to the node’s DNS from the node’s address.

  • Remove the package before shc init or shc node join: apt-get remove aardvark-dns (dnf remove aardvark-dns). Both commands refuse a node whose podman info reports a DNS backend, and one where an aardvark-dns process still runs, with X400398FABAVK. A process started before the package was removed keeps answering every container on its networks, new ones included, until the last container on those networks stops. Then it exits and nothing starts it again.
  • Every start asks again: an install, an update, a job, shc start, shc restart, a rollback, the health reconcile, a replica project and the restart of one container. A package manager that brings the package back as a recommended dependency makes the next start refuse with X400398FABAVK, and shc doctor run reports it as fabric.engine before that.
  • Your own Podman containers on that node, outside SHC, lose aardvark’s names too. The node is SHC’s.

The rest is what gives a container a resolver at all, not the guarantee. On a network with DNS enabled and no aardvark-dns, a container’s resolv.conf points at the network’s gateway, where nothing answers. So the engine’s resolver is off on every network a container joins:

  • SHC creates its own networks (the node’s fabric network, the shared ingress network, a deployment’s networks on the node) with --disable-dns.
  • A network podman-compose creates carries x-podman.disable_dns: true. podman-compose reads that key from 1.4.0. An older one drops it without an error, so shc init and shc node join refuse it by name (X400396FABPDC) and shc doctor run reports it as fabric.engine. Debian 13’s podman-compose is 1.3.0.
  • A start is refused with X409429PODDNS while a network its compose could name still has the resolver on, with X503429PODDNQ when podman cannot list its networks, and with X400396FABPDC when the node’s podman-compose is older than 1.4.0 by then, on the fabric or off it. podman-compose versions name a network differently, so every name it could have is asked about: its key, its name:, the name under external:, and the project’s name joined to the key.
  • podman-compose 1.4.x keeps the resolver off but ignores --no-recreate: an update that changes one service creates every container of the deployment again, and an up over containers that exist fails on the first name it finds in use. shc init, shc node join and every start refuse it with X400397FABPRC, and fabric.engine reports it as an error naming the version found and 1.5.0. Containers that already run are left as they are.
  • A document rendered before SHC wrote the key is refused with X409430PODDNR: run shc update for the deployment. One that cannot be read is refused with X503430PODDOC. One with a top-level include: is refused with X409431PODINC, because a network an included file creates would not carry the key: inline what it includes.
  • Nothing turns the resolver off on an existing network. Remove a deployment’s network once no container names it (podman network rm) and install again.

Where SHC and podman-compose still read a document differently, the result is a container that cannot resolve names. It is never a container answered another tenant’s names.

With the resolver off, a container’s resolv.conf is the node’s DNS and the deployment’s search domain, and it asks from its own fabric address, as on Docker. What this takes away: the names only the engine knows. On a Podman node a compose container name (wiki_db_1) and an app-authored network aliases: entry do not resolve. A service name still does, through the search domain, with the answer the node’s DNS gives everywhere: the service’s VIP, or its ready replicas. On a Docker node the embedded resolver still answers container names and aliases, so the two runtimes differ there until the node’s DNS answers aliases itself.

A single Podman node resolves no service names at all. The node’s DNS listens on the mesh address, and a node gets one when a second node joins. Until then nothing answers on the node, containers keep the host’s nameservers, and with Podman’s resolver off a deployment’s containers cannot find each other by name. A single Docker node is not affected: the embedded resolver answers the compose network’s names.

An app with -i otel=OTEL exports to http://collector.shc.internal:4318. While the system collector deployment (master/default/collector) has a ready replica, every node answers that name with the deployment’s VIP, to every container, whatever the node’s runtime. The deployment runs one replica on each ready node, so the export goes to one of them, and the fabric lets every tenant reach its two OTLP ports and nothing else of it.

Until that deployment has a replica, no node answers the name: a node’s own collector project reads that node’s logs and metrics and takes no OTLP. An install or update of an otel-integrated deployment then emits X200182OTLNIL once. The app still runs; its OTLP export has no destination until shc install collector -t master. See the telemetry integration for the system collector and the warning text.

A service that sets its own dns: asks its own nameservers and sees none of this. Forwarding shc.internal from them to the node’s mesh address brings back the cluster names. It does not bring back the provider names: the node answers those by the asker’s fabric address, and a forwarder asks from its own, so the service has to ask the node’s DNS directly for them.

People’s devices reach the resolvers over NetBird. Two objects in the NetBird management console carry it: a network route to the mesh through routing peers, and a nameserver group for the private zones. SHC manages neither, so set them up by hand once per estate and again when you add a location.

  1. Routing peers. Run the NetBird agent on at least one node per location and add those peers to a group, for example shc-routers. Two per location keeps the route up while one reboots.

  2. Network route. Under Network Routes, add the cluster’s mesh range with the shc-routers group as routing peers, masquerade on, and distribute it to the group your people’s devices are in. Add an access control policy that lets that group reach shc-routers. The mesh range is the address block’s fourth sixteenth: 10.64.48.0/20 on the default 10.64.0.0/16, 10.80.48.0/20 on a 10.80.0.0/16 block (see the address block). On any node, ip -4 addr show mesh0 prints the node’s mesh address with the range’s prefix length, for example 10.64.48.1/20.

  3. Nameserver group. Under DNS → Nameservers, add a group whose nameservers are the mesh addresses of the routing nodes, port 53, UDP. Set the match domains to every entry in domains.json, with search domains off, and distribute it to the same people’s group. Every node’s resolver gives the same answer, so any node’s address works.

  4. Verify from a device. First the route, straight at a routing node’s mesh address. The first node of a cluster on the default block holds 10.192.0.1:

    Terminal window
    dig +short @10.192.0.1 n1.nodes.shc.internal

    Then the nameserver group, through the device’s own resolver. dig without @ reads /etc/resolv.conf, which a match-domain nameserver never appears in, so ask the system resolver instead:

    Terminal window
    dscacheutil -q host -a name git.corp.example.com # macOS
    resolvectl query git.corp.example.com # Linux
    Resolve-DnsName git.corp.example.com # Windows

    Each should return mesh addresses.

With masquerade on, Traefik sees a device’s requests coming from the routing peer’s mesh address. An app restricted to the LAN admits them only if its allowlist covers that address; see Gated apps. To the resolvers a device is a stranger whichever address its question arrives from: it is answered the routed hostnames and the node names, and no tenant’s services. Ask from the node for those.

A private zone added later, for example a new private DNS connection, appears in domains.json within a minute. Add it to the nameserver group’s match domains.