Multi-node deployment testing
Implementation Summary
Section titled “Implementation Summary”The following changes have been successfully implemented for multi-node deployment:
1. Database Schema (Completed)
Section titled “1. Database Schema (Completed)”- NodeModel: Added
data_dir,state_dir,runtime_dir,config_dirfields - Migrations: Applied successfully via the ce module’s
migrations/
2. Path Management (✅ Completed)
Section titled “2. Path Management (✅ Completed)”- Per-environment app directories:
{tenant}/stacks/{stack}/environments/{env}/app/ - Deployment output directories:
{tenant}/stacks/{stack}/environments/{env}/deployment/ - Node-specific path helpers: Support for remote node path resolution
3. File Synchronization (✅ Completed)
Section titled “3. File Synchronization (✅ Completed)”- Tar.gz-based transfer: Efficient compression for remote file sync
- Dual directory sync: Both
app/(source) anddeployment/(rendered) directories - Base64 encoding: For binary-safe transfer over SSH
4. App Hash Tracking (✅ Completed)
Section titled “4. App Hash Tracking (✅ Completed)”- Deterministic SHA256 hashing: Detects app file changes
- Install-time hash calculation: Stored in database
- Upgrade-time comparison: Automatic re-copy when hash changes
5. Node Auto-Detection (✅ Completed)
Section titled “5. Node Auto-Detection (✅ Completed)”- Sudo privilege detection: Automatic during node registration
- Path auto-configuration:
- Root/sudo:
/var/lib/shc,/etc/shc,/var/run/shc - Regular user:
~/.local/share/shc,~/.config/shc,/tmp/shc-{user}
- Root/sudo:
Manual Testing Instructions
Section titled “Manual Testing Instructions”Prerequisites
Section titled “Prerequisites”- A rig cell (SHC-1123):
tests/rig/cells/<cell>.envnames its nodes
Step 1: Bring the Cell Up
Section titled “Step 1: Bring the Cell Up”make -C tests/rig up CELL=cell1 DEB=dist/shc_<version>_amd64.debmake -C tests/rig health CELL=cell1This rolls the deb onto every node of the cell, forms the cluster and checks that every node is healthy.
Step 2: Verify Node Configuration
Section titled “Step 2: Verify Node Configuration”SSH into the cell’s first node (named in tests/rig/cells/cell1.env).
Check node 2 configuration:
curl -s http://localhost:8000/api/resource/Node/shc-node-2 | jq '{ id, address, user, data_dir, state_dir, runtime_dir, config_dir}'Expected output:
{ "id": "shc-node-2", "address": "10.1.1.3", "user": "user", "data_dir": "/home/user/.local/share/shc", "state_dir": "/home/user/.local/share/shc", "runtime_dir": "/tmp/shc-user", "config_dir": "/home/user/.config/shc"}Step 3: Deploy Test App to Node 2
Section titled “Step 3: Deploy Test App to Node 2”# Install test app targeting node 2shc install test-simple --where 'node=shc-node-2' -yStep 4: Verify File Synchronization
Section titled “Step 4: Verify File Synchronization”On node 2 (10.1.1.3):
# Check app source directory (per-environment)ls -la /home/user/.local/share/shc/tenants/default/stacks/test-simple/environments/default/app/# Should show: app.yaml, compose.yaml, etc.
# Check deployment output directoryls -la /home/user/.local/share/shc/tenants/default/stacks/test-simple/environments/default/deployment/# Should show: compose.yaml (rendered)
# Verify container is runningdocker ps | grep test-simpleStep 5: Verify App Hash Tracking
Section titled “Step 5: Verify App Hash Tracking”On the primary node:
# Get stack detailscurl -s http://localhost:8000/api/resource/Stack | jq '.data[] | select(.name=="test-simple") | { name, app_name, app_version, node_id}'Expected:
{ "name": "test-simple", "app_name": "test-simple", "app_version": "1.0.0", "node_id": "shc-node-2"}Step 6: Test App Upgrade
Section titled “Step 6: Test App Upgrade”# Modify the app in the repoecho "# Modified" >> /home/user/projects/selfhosted-cloud/shc/apps/test-simple/README.md
# Sync the repo to update the appshc repo sync
# Upgrade the stackshc update test-simpleExpected behavior:
- App files are re-copied from source
- Deployment files are re-rendered
- Both directories synced via tar.gz
Step 7: Verify Tar.gz Transfer
Section titled “Step 7: Verify Tar.gz Transfer”Check primary node logs for transfer evidence:
tail -f ~/.shc/logs/server.log | grep -i "sync\|tar"Should show:
- “Syncing app directory to remote”
- “Syncing deployment directory to remote”
- “Successfully synced app directory”
Step 8: Test Cleanup
Section titled “Step 8: Test Cleanup”# Uninstall the appshc uninstall test-simple -y
# Verify cleanup on node 2# Should return nothing
# Verify files removed# Directory should be goneArchitecture Verification Checklist
Section titled “Architecture Verification Checklist”- Node 2 registered with auto-detected paths
- Node has user-specific paths
- App deployed successfully to node 2
- App files in
/environments/{env}/app/(per-environment) - Deployment files in
/environments/{env}/deployment/(sibling dir) - App hash stored in database (64-char SHA256)
- Hash changes trigger re-copy on upgrade
- Files transferred via tar.gz (not file-by-file)
- Docker containers running on node 2
- Cleanup removes files from node 2
Troubleshooting
Section titled “Troubleshooting”Issue: Node registration fails
Section titled “Issue: Node registration fails”# Check SSH connectivity
# Check vault SSH key storagecurl -s http://localhost:8000/api/resource/Vault | jqIssue: Files not syncing to remote node
Section titled “Issue: Files not syncing to remote node”# Check node service logstail -100 ~/.shc/logs/server.log | grep -A 10 "sync_deployment_to_remote"
# Manually test SSH command executioncurl -s -X POST http://localhost:8000/api/method/shc.node.exec \ -H "Content-Type: application/json" \ -d '{"node": "shc-node-2", "command": ["ls", "-la", "/home/user"]}'Performance Notes
Section titled “Performance Notes”- Tar.gz compression: ~5-10x faster than file-by-file copying
- Sync trigger: Only on install/upgrade, not on every command
Known Limitations
Section titled “Known Limitations”- Binary files: Transferred via base64 encoding (slight overhead)
- Network: Requires reliable SSH connectivity to remote nodes
Related Files
Section titled “Related Files”- Implementation:
modules/app/service.go - Path helpers:
core/paths/paths.go - Node registration:
modules/node/router.go - Database models:
modules/node/models.go,modules/app/models.go
shc@docs:~$