Restarting what is unhealthy
This page decides what happens to a replica whose healthcheck keeps failing.
A check only reports it; the node restarts it once it has been unhealthy
for long enough, five minutes unless the service says otherwise.
x-shc-health.restart_after is that time for one service.
Give web thirty seconds: a page that has not answered for half a minute is
better restarted than waited on.
services: web: image: nginx:1.27-alpine expose: - "80" volumes: - ./site:/usr/share/nginx/html:ro healthcheck: test: ["SHC-HTTP", ":80/"] interval: 10s timeout: 3s retries: 3 deploy: replicas: 2 x-shc-primary: true x-shc-health: restart_after: 30s api: image: traefik/whoami:v1.10 command: ["--port", "8080", "--name", "guestbook-api"] expose: - "8080" healthcheck: test: ["SHC-HTTP", ":8080/"] interval: 10s timeout: 3s retries: 3 x-shc-path: /api x-shc-path-strip: true counter: image: redis:7-alpine command: ["redis-server", "--appendonly", "yes"] expose: - "6379" volumes: - data:/data healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 5s timeout: 3s retries: 12 x-shc-scale: max: 1volumes: data:$ shc update guestbook...Now make one replica stop answering without stopping. pkill -STOP freezes
every nginx process in its container. A frozen process still exists, so the
container stays up while every request to it hangs. That is the failure a
healthcheck is for.
$ shc exec -s guestbook -S web -- pkill -STOP nginxThree failed checks ten seconds apart make it unhealthy, and thirty seconds later its node restarts it. The node says so in its journal.
$ journalctl --no-pager -u shc -g X201026CTRRST -o cat -n 1...A restart starts nginx afresh, and its healthcheck passes again.
$ shc status -s guestbook...prod guestbook prod web-r0 home01 ready...Next: The same app in shc.yaml.