Nodes on a hypervisor
A node that runs the Incus runtime on the bare host already sees everything Incus runs there. It counts every instance SHC did not start as outside use and leaves that room alone. Nothing needs setting up. See Resource model.
A node that is itself an Incus instance sees only itself. It cannot see how big its host is, or the build VMs and the firewall running beside it. When the host is promised to more instances than it has cores and memory for, SHC would still count the whole node as room.
--hypervisor closes that gap. It names a connection, and the node
reads its host through it.
| What you want | Command |
|---|---|
| Tell a node which connection reaches its host | shc node update shc01 --hypervisor ogden-view |
| Name the instance when it cannot be found | shc node update shc01 --hypervisor ogden-view/shc01-vm |
| Set it as the node joins | shc node join <token> --hypervisor ogden-view |
| See what the node read | shc node info shc01 |
| Stop reading the host | shc node update shc01 --hypervisor "" |
What the node reads, and what it does with it
Section titled “What the node reads, and what it does with it”Every 30 seconds the node asks its host four things, all of them reads:
- every instance the certificate may see, in every project;
- the server’s name;
- the certificate’s own entry, to learn whether it is restricted;
- the cores and memory of the cluster member it runs on.
It finds itself among the instances by its addresses, then by its name.
If neither matches, name the instance after the connection:
ogden-view/shc01-vm, or ogden-view/infra/shc01-vm for one in another
project.
Then it does one sum. The member’s cores and memory, less the limits of every other running instance on that member, is what the host has left for this node. If that is less than the node’s own size, the difference is room the node does not really have. It is added to the node’s outside use, with those instances named, so placement and admission leave it alone.
An instance counts by limits.cpu and limits.memory. A virtual machine
with no limit counts 1 core and 1 GiB. A container with no memory limit
counts its measured memory.
A host with room for the node adds nothing. shc node info still shows
the host and its instances.
$ shc node info shc01... OUTSIDE OUTSIDE INSTANCES HYPERVISOR HOST MEMBER HOST CAPACITY HOST INSTANCES... cpu=2,memory=4Gi default/build01 ogden-view incus01 cpu=32,memory=128Gi default/build01, infra/pfsenseSHC only reads through this connection. The reader in the daemon has one method and it is a GET.
Connect the hypervisor
Section titled “Connect the hypervisor”The connection is the incus type. It needs the API URL of every
cluster member, a client certificate and key, and the server
certificate to pin. A guest usually reaches the API only on its own
host, so list every member and the node uses the one that answers.
shc connection connect ogden-view --type incus \ --endpoint https://10.20.0.11:8443,https://10.20.0.12:8443,https://10.20.0.13:8443 \ --cred client_cert=- --cred client_key=- --cred server_cert=-project and pool are for block volumes. A connection used only for
--hypervisor needs neither.
Which certificate to give it
Section titled “Which certificate to give it”This was measured on Incus 7.5.1, not taken from its documentation.
| Certificate | Host’s cores and memory | Instances it is shown | Can it change anything? |
|---|---|---|---|
| Client, restricted to some projects | yes | only those projects’ | yes, in those projects |
| Client, unrestricted | yes | every project’s | yes, everything |
| Metrics | refused, 403 | refused, 403 | no |
Three things follow.
A certificate restricted to one project reads the host’s size but is
shown only that project’s instances. Everything else on the host goes
uncounted, and nothing in the API’s answer says so. SHC reads the
certificate’s own entry to find out, and shc node info says it under
HOST NOTE:
the certificate is restricted to project default, so instances of otherprojects on this host are not shown or counted; a certificate restrictedto every project, or an unrestricted one, shows themNo Incus certificate measured here is both complete and unable to write. A restricted certificate can change instances in its projects. An unrestricted one can change everything. An authorization scriptlet was set and Incus never called it for a certificate client, so it restricted nothing.
A metrics certificate cannot change anything, but Incus refuses it the
instance list and the host’s size, so --hypervisor cannot use one.
HOST ERROR says so when you try.
So pick by what you need counted:
# every project on the host counted: trust the certificate for all of themincus config trust add-certificate shc-view.crt --name shc-view \ --restricted --projects default,infra,ciAdd a project to that list when you create one. An unrestricted certificate needs no list and is a full administrator’s key, held in SHC’s vault.
When the host cannot be read
Section titled “When the host cannot be read”The node’s last outside report stands. It is not replaced by a report
with the host’s share missing. HOST ERROR in shc node info says what
failed, and the next read clears it.
| HOST ERROR says | What to do |
|---|---|
| the hypervisor refused this certificate | trust it as a client certificate; a metrics certificate is refused these reads |
| the hypervisor’s instances could not be read | the endpoint is wrong or closed to this node; a host that is itself an SHC node closes it, see below |
| no instance … holds this node’s addresses or name | name it: --hypervisor <connection>/<instance> |
| the host’s cores and memory could not be read | the certificate may not read server resources; nothing is counted until it can |
| connection … is of type … | connect an incus connection and name that one |
A host that is itself an SHC node
Section titled “A host that is itself an SHC node”An Incus host that is also an SHC node keeps every container on it away
from its own services, the Incus API included. That is SHC protecting the
host, and it means a node in a guest on that host cannot read it. Measured
on a test cell: the guest’s reads timed out until SHC was stopped on the
host. Give --hypervisor to nodes whose hosts run Incus and nothing of
SHC, which is how Gorilla’s hosts are built.
What it does not do
Section titled “What it does not do”SHC does not resize or move the virtual machine. It does not read the host’s disk. Both would be writes or new reads, and neither is here.