Upgrade
The cluster record and every shc_node carry package, and its version is
the shc release the node installs. Changing it plans an in-place upgrade. The
provider installs the new version by apt on each node over SSH and holds it
there, one node at a time. The leader goes last, whichever node leads when
the roll starts, so the cluster elects a new leader once, at the end. After
each node the roll waits until that node reads healthy at the new build,
quorum is healthy and shc doctor run shows no new failure. Only then does
the next node restart, so quorum holds throughout.
The cluster runs 0.1.2. Change shc_version to 0.1.3, the release of the
modules and the provider.
home_region = "nbg1"far_region = "hel1"home_count = 4server_type = "cx23"shc_version = "0.1.3"$ tofu -chdir=learn/cluster plan... # module.shc.module.core[0].shc_cluster.main[0] will be updated in-place... # module.shc.module.core[0].shc_license.estate[0] will be updated in-place... # module.shc.module.core[0].shc_node.bootstrap[0] will be updated in-place... # module.shc.module.core[0].shc_node.workers["far01.learn"] will be updated in-place... # module.shc.module.core[0].shc_node.workers["home02.learn"] will be updated in-place... # module.shc.module.core[0].shc_node.workers["home03.learn"] will be updated in-place... # module.shc.module.core[0].shc_node.workers["home04.learn"] will be updated in-place...Plan: 0 to add, 7 to change, 0 to destroy....No replacement: a node keeps its identity, its disk and its place in the cluster. The cluster record and the five nodes change their package. The licence is in the plan because it reads the bootstrap node’s endpoint, which is known only after the apply; it does not change, so the apply counts six. Apply and the roll runs.
$ tofu -chdir=learn/cluster apply -auto-approve...Apply complete! Resources: 0 added, 6 changed, 0 destroyed....$ ssh -F learn/cluster/ssh/config home01 sudo shc versionshc version 0.1.3...$ ssh -F learn/cluster/ssh/config far01 sudo shc versionshc version 0.1.3...$ ssh -F learn/cluster/ssh/config home01 sudo shc node ls --fields name,state,condition,build,voterNAME STATE CONDITION BUILD VOTERfar01.learn healthy ready 0.1.3 nohome01.learn healthy ready 0.1.3 yeshome02.learn healthy ready 0.1.3 yeshome03.learn healthy ready 0.1.3 yeshome04.learn healthy ready 0.1.3 no$ ssh -F learn/cluster/ssh/config home01 apt-mark showholdshchome01 led when the roll began, so it restarted last and one of the other two voters leads now. Which one won the election varies from run to run.
$ ssh -F learn/cluster/ssh/config home01 sudo shc cluster statuscluster name: shc-clusterleader: home02.learnnodes: 5 healthy of 5voters: 3quorum: healthy, 2 voters requiredleader-only work, recorded by home02.learn 34s ago:...Every node runs 0.1.3, apt holds shc there so an apt upgrade never moves
it, and the cluster never lost quorum. The provider’s log shows the order and
each node’s health gate: run the apply with TF_LOG_PROVIDER=INFO and look
for shc upgrade lines with phase=roll-order and phase=gate-pass. A node
that fails its gate stops the roll there, and the apply names it and the
check it failed. A node that answers over SSH but stays
not ready for ten minutes fails the upgrade; a node that stops answering SSH
while its daemon restarts gets twenty minutes before the roll reports that it
could not confirm the upgrade. Both budgets are package attributes,
ready_timeout and transport_timeout.
Next: Destroy.