Troubleshooting SSE streaming

This guide helps diagnose and resolve common issues with SSE (Server-Sent Events) streaming for real-time operation tracking.

1. Connection Failures
2. Missed Events (Gap Events)
3. High Memory Usage
4. Slow Clients
5. Metrics Interpretation
6. Log Analysis

Client cannot connect to SSE endpoint:

GET /api/method/shc.operation.stream/{operation_id}
→ 404 Not Found

1. Operation expired: Operation completed >5 minutes ago and was cleaned up
2. Invalid operation_id: Wrong operation_id or typo
3. Server restart: Operations lost (stored in memory, not persisted)

# Check if operation exists
GET /api/method/shc.operation.state/{operation_id}

# If 404, query current stack state instead
GET /api/stacks/{tenant}/{stack}/{environment}/status

Prevention:

• Reduce sse.operation_ttl if clients need longer replay windows
• Persist operations to database (future enhancement)

Client receives a gap event:

{
  "event_type": "gap",
  "sequence": 99,
  "missing_sequences": [50, 51, 52, ..., 99],
  "message": "Missed 50 events. Query current state to resync."
}

1. Slow client: Client couldn’t process events fast enough, buffer overflowed
2. Network disruption: Client disconnected for too long
3. Small buffer: sse.buffer_size too small for operation event rate

Client-side:

if event.EventType == "gap" {
    // Query current state from database
    state, _ := api.GetStackStatus(ctx, tenant, stack, env)
    updateDisplay(state)
    // Continue streaming (already at latest buffer)
}

Server-side:

• Increase sse.buffer_size in config.yaml (default: 1000)
• Check metrics: shc_sse_slow_clients_total

Prevention:

• Size buffer for expected event rate: events_per_operation × 2
• Example: 50 containers × 5 events each = 250 events → buffer_size: 500

Alert: SSEHighMemoryUsage fires

shc_sse_memory_bytes > 100MB

1. Too many operations: More than 100 concurrent long-running operations
2. Large buffer size: sse.buffer_size too high
3. Operations not cleaning up: Bug preventing cleanup

Check current usage:

# Query Prometheus
shc_sse_memory_bytes
shc_sse_operations{status="running"}

Tune configuration:

sse:
  buffer_size: 500           # Reduce from 1000
  max_operations: 500        # Reduce from 1000
  operation_ttl: 180         # Reduce from 300

Emergency cleanup:

• Emergency cleanup triggers automatically at limit
• Removes oldest 25% of completed operations
• Running operations are NOT affected

Prevention:

• Set buffer_size based on workload
• Monitor shc_sse_operations regularly

Metric increasing: shc_sse_slow_clients_total

Logs show:

[WARN] SSE buffer near full for operation abc-123

1. Network latency: Slow network between client and server
2. Client processing: Client can’t process events fast enough
3. Server overload: Too many concurrent operations

Check client:

• Profile client event processing
• Add async processing if blocking
• Reduce display refresh rate

Check network:

# Test latency
curl -w "@curl-format.txt" http://server/api/method/shc.operation.stream/{operation_id}

Check server:

# Check concurrent operations
shc_sse_operations{status="running"}

# Check event emission rate
rate(shc_sse_events_emitted_total[1m])

Prevention:

• Clients should process events asynchronously
• Batch display updates (max 10 Hz)
• Filter unnecessary events client-side

# Active connections by operation type
shc_sse_connections{operation_type="install"}

# Alert when approaching limit
shc_sse_connections > 450  # 90% of 500 default

# Events emitted per second
rate(shc_sse_events_emitted_total[5m])

# Events by type
rate(shc_sse_events_emitted_total{event_type="container_status"}[5m])

# Event latency (p95)
histogram_quantile(0.95, rate(shc_sse_event_latency_seconds_bucket[5m]))

# Slow clients detected
rate(shc_sse_slow_clients_total[5m])

Operation lifecycle:

# Operations created
grep "Operation created" shc.log

# Operations completed
grep "Operation completed" shc.log

# Operations failed
grep "Operation failed" shc.log

SSE events:

# Event emissions (DEBUG level)
grep "SSE event emitted" shc.log

# Gap detections
grep "Gap detected" shc.log

# Memory warnings
grep "SSE memory usage" shc.log

Client connections:

# New connections
grep "Client reconnecting" shc.log

# Connection errors
grep "Too many concurrent SSE connections" shc.log

• DEBUG: Event emissions, reconnections, cache cleanup
• INFO: Operation lifecycle, stack status changes
• WARN: Buffer near full, memory limit approaching
• ERROR: Operation failures, cleanup errors

{
  "level": "INFO",
  "message": "Operation created: install",
  "operation_id": "abc-123-def",
  "operation_type": "install",
  "timestamp": "2026-01-24T12:00:00.123Z"
}

Problem: Client disconnected for 2 minutes, received gap event on reconnect

Solution:

1. Client queries /api/stacks/{tenant}/{stack}/{env}/status
2. Updates display with current state
3. Continues streaming from current sequence
4. ✅ No action needed server-side

Problem: Deploying 100-container stack triggered memory alert

Solution:

1. Emergency cleanup triggered automatically (oldest 25% removed)
2. Large deployments generate many events
3. Consider increasing buffer: sse.max_memory_mb: 200
4. Or reduce buffer size per operation: sse.buffer_size: 500

Problem: 1000+ operations tracked, server slowing down

Root Cause: Operations not completing or cleanup not running

Solution:

# Check operation cleanup task
grep "Cleaned up" shc.log

# Manually trigger cleanup via API (future enhancement)
# POST /api/admin/sse/cleanup

# Restart server to clear all operations
systemctl restart shc

Problem: Client connected, but not receiving container/service events

Checklist:

1. ✅ Is SSE enabled? Check config.yaml → sse.enabled: true
2. ✅ Is operation_id set? Check operation context in compose operations
3. ✅ Are events being emitted? Check shc_sse_events_emitted_total metric
4. ✅ Is client parsing events correctly? Check event_type field

Debug:

# Enable DEBUG logging
export SHC_LOG_LEVEL=DEBUG

# Check if events are being routed
grep "SSE event emitted" shc.log | grep "operation_id=YOUR_OP_ID"

If SSE is causing issues, disable it:

sse:
  enabled: false

Then restart server:

systemctl restart shc

Clients will use DB polling (DeploymentTracker) instead of SSE.

Enable SSE for specific operations only:

// In code (future enhancement)
import "gitlab.com/bitspur/selfhosted-cloud/ce/modules/app"
if config.SSEEnabled && (operationType == "install" || operationType == "backup") {
    context.SetOperation(operationID)
}

If this guide doesn’t resolve your issue:

1. Check metrics in Grafana

2. Collect logs (DEBUG level for 5 minutes)

3. Run diagnostics:

   # Check active operations
   curl http://localhost:8000/api/method/shc.health.get
   
   # Check memory usage
   ps aux | grep shc

4. File issue with:

   • Log excerpt (DEBUG level)
   • Metrics snapshot
   • Reproduction steps