Multi-node deployment testing

The following changes have been successfully implemented for multi-node deployment:

• NodeModel: Added has_sudo, data_dir, state_dir, runtime_dir, config_dir fields
• Migrations: Applied successfully via the ce module’s migrations/

• Per-environment app directories: {tenant}/stacks/{stack}/environments/{env}/app/
• Deployment output directories: {tenant}/stacks/{stack}/environments/{env}/deployment/
• Node-specific path helpers: Support for remote node path resolution

• Tar.gz-based transfer: Efficient compression for remote file sync
• Dual directory sync: Both app/ (source) and deployment/ (rendered) directories
• Base64 encoding: For binary-safe transfer over SSH

• Deterministic SHA256 hashing: Detects app file changes
• Install-time hash calculation: Stored in database
• Upgrade-time comparison: Automatic re-copy when hash changes

• Sudo privilege detection: Automatic during node registration
• Path auto-configuration:
  • Root/sudo: /var/lib/shc, /etc/shc, /var/run/shc
  • Regular user: ~/.local/share/shc, ~/.config/shc, /tmp/shc-{user}

• Access to Proxmox host (not inside a VM)
• PROXMOX_HOST environment variable set
• Terraform and Packer installed

cd tests/proxmox
bats test_multi_node_ssh.bats

This will:

• Provision node VMs (primary and additional nodes)
• Install SHC on all nodes
• Register nodes with auto-detected paths

SSH into the primary VM:

source tests/proxmox/terraform/env
ssh -i "$SSH_KEY_PATH" user@$PRIMARY_IP

Check node 2 configuration:

curl -s http://localhost:8000/api/resource/Node/shc-node-2 | jq '{
  id,
  address,
  user,
  has_sudo,
  data_dir,
  state_dir,
  runtime_dir,
  config_dir
}'

Expected output:

{
  "id": "shc-node-2",
  "address": "10.1.1.3",
  "user": "user",
  "has_sudo": false,
  "data_dir": "/home/user/.local/share/shc",
  "state_dir": "/home/user/.local/share/shc",
  "runtime_dir": "/tmp/shc-user",
  "config_dir": "/home/user/.config/shc"
}

# Install test app targeting node 2
shc install test-simple --node shc-node-2 -y

On node 2 (10.1.1.3):

# Check app source directory (per-environment)
ls -la /home/user/.local/share/shc/tenants/default/stacks/test-simple/environments/default/app/
# Should show: app.yaml, compose.yaml, etc.

# Check deployment output directory
ls -la /home/user/.local/share/shc/tenants/default/stacks/test-simple/environments/default/deployment/
# Should show: compose.yaml (rendered)

# Verify container is running
docker ps | grep test-simple

On the primary node:

# Get stack details
curl -s http://localhost:8000/api/resource/Stack | jq '.data[] | select(.name=="test-simple") | {
  name,
  app_name,
  app_version,
  node_id
}'

Expected:

{
  "name": "test-simple",
  "app_name": "test-simple",
  "app_version": "1.0.0",
  "node_id": "shc-node-2"
}

# Modify the app in the repo
echo "# Modified" >> /home/user/projects/selfhosted-cloud/shc/apps/test-simple/README.md

# Sync the repo to update the app
shc repo sync

# Upgrade the stack
shc upgrade test-simple

Expected behavior:

• App files are re-copied from source
• Deployment files are re-rendered
• Both directories synced via tar.gz

Check primary node logs for transfer evidence:

tail -f ~/.shc/logs/server.log | grep -i "sync\|tar"

Should show:

• “Syncing app directory to remote”
• “Syncing deployment directory to remote”
• “Successfully synced app directory”

# Uninstall the app
shc uninstall test-simple -y

# Verify cleanup on node 2
ssh user@10.1.1.3 "docker ps | grep test-simple"
# Should return nothing

# Verify files removed
ssh user@10.1.1.3 "ls /home/user/.local/share/shc/tenants/default/stacks/test-simple"
# Directory should be gone

• Node 2 registered with auto-detected paths
• Node has has_sudo=false and user-specific paths
• App deployed successfully to node 2
• App files in /environments/{env}/app/ (per-environment)
• Deployment files in /environments/{env}/deployment/ (sibling dir)
• App hash stored in database (64-char SHA256)
• Hash changes trigger re-copy on upgrade
• Files transferred via tar.gz (not file-by-file)
• Docker containers running on node 2
• Cleanup removes files from node 2

# Check SSH connectivity
ssh -i ~/.ssh/id_rsa user@10.1.1.3 echo "OK"

# Check vault SSH key storage
curl -s http://localhost:8000/api/resource/Vault | jq

# Check node service logs
tail -100 ~/.shc/logs/server.log | grep -A 10 "sync_deployment_to_remote"

# Manually test SSH command execution
curl -s -X POST http://localhost:8000/api/method/shc.node.exec \
  -H "Content-Type: application/json" \
  -d '{"node": "shc-node-2", "command": ["ls", "-la", "/home/user"]}'

• Tar.gz compression: ~5-10x faster than file-by-file copying
• Sync trigger: Only on install/upgrade, not on every command

• Binary files: Transferred via base64 encoding (slight overhead)
• Network: Requires reliable SSH connectivity to remote nodes

• Implementation: modules/app/service.go
• Path helpers: core/paths/paths.go
• Node registration: modules/node/router.go
• Database models: modules/node/models.go, modules/app/models.go