# Troubleshooting Guide

This guide covers common issues encountered when running the DevOps AI Toolkit Controller and their solutions.

## Common Issues and Solutions

### 1. Controller Pod Not Starting

**Symptoms:**
```bash
kubectl get pods --namespace dot-ai
# Shows controller pod in CrashLoopBackOff or ImagePullBackOff
```

**Diagnosis:**
```bash
kubectl logs --selector app.kubernetes.io/name=dot-ai-controller --namespace dot-ai
kubectl describe pod --selector app.kubernetes.io/name=dot-ai-controller --namespace dot-ai
```

**Common Causes:**
- **RBAC Issues**: Missing leader election permissions (we encountered this during testing)
- **Image Issues**: Wrong architecture or missing image
- **Resource Constraints**: Insufficient memory/CPU limits

**Solution:**
```bash
# Check if leader election RBAC is missing (error we fixed during testing):
# "leases.coordination.k8s.io is forbidden"
kubectl get clusterrole dot-ai-controller-manager-role --output yaml

# Add missing leader election permissions if needed:
kubectl patch clusterrole dot-ai-controller-manager-role --type='json' \
  --patch='[{"op": "add", "path": "/rules/-", "value": {"apiGroups": ["coordination.k8s.io"], "resources": ["leases"], "verbs": ["create", "get", "list", "update"]}}]'
```

### 2. Events Not Being Processed

**Symptoms:**
```bash
kubectl logs --selector app.kubernetes.io/name=dot-ai-controller --namespace dot-ai --tail 50
# Shows: "No RemediationPolicies found - event will not be processed"
```

**Diagnosis:**
```bash
# Check if RemediationPolicies exist
kubectl get remediationpolicies --all-namespaces

# Check policy selectors
kubectl get remediationpolicies --namespace dot-ai --output yaml
```

**Common Causes:**
- No RemediationPolicy created
- Event doesn't match policy selectors
- Policy in wrong namespace

### 3. MCP Connection Failures

**Symptoms:**
```bash
# Controller logs show:
# "❌ HTTP request failed" or "Failed to send MCP request"
```

**Diagnosis:**
```bash
# Check MCP pod status
kubectl get pods --namespace dot-ai --selector app.kubernetes.io/name=dot-ai

# Test MCP connectivity from controller
kubectl exec --namespace dot-ai deployment/dot-ai-controller-manager -- \
  curl -v http://dot-ai-mcp.dot-ai.svc.cluster.local:3456/health
```

**Common Causes:**
- MCP pod not running
- Wrong MCP endpoint URL in RemediationPolicy
- Network policies blocking communication

### 4. Slack Notifications Not Working

**Symptoms:**
```bash
# Controller logs show:
# "failed to send Slack start notification"
```

**Diagnosis:**
```bash
# Check Slack webhook configuration
kubectl get remediationpolicies --namespace dot-ai --output yaml | grep --after-context 5 slack

# Test webhook manually
curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test message"}' \
  YOUR_SLACK_WEBHOOK_URL
```

**Common Causes:**
- Invalid Slack webhook URL
- Slack webhook disabled (`enabled: false`)
- Network connectivity issues

### 5. Rate Limiting Active

**Symptoms:**
```bash
# Controller logs show:
# "Event processing rate limited" and "cooldown active for Xm Ys more"
```

**This is Expected Behavior:** Rate limiting prevents spam processing of duplicate events. The default settings are:
- `eventsPerMinute: 5`  
- `cooldownMinutes: 15`

**To Adjust:** Modify your RemediationPolicy:
```yaml
rateLimiting:
  eventsPerMinute: 10    # Increase if needed
  cooldownMinutes: 5     # Decrease if needed
```

### 6. MCP Analysis Failures

**Symptoms:**
```bash
# Controller logs show:
# "MCP remediation failed" or "McpRemediationFailed" events
```

**Diagnosis:**
```bash
# Check MCP logs for detailed error messages
kubectl logs --namespace dot-ai --selector app.kubernetes.io/name=dot-ai --tail 50

# Check RemediationPolicy status
kubectl describe remediationpolicies --namespace dot-ai
```

**Common Causes:**
- Invalid Anthropic API key
- API rate limits exceeded
- Network connectivity to Anthropic services
- Malformed event data

### 7. ResourceSyncConfig Not Syncing

**Symptoms:**
```bash
# ResourceSyncConfig status shows syncErrors or not active
kubectl get resourcesyncconfigs --output yaml
```

**Diagnosis:**
```bash
# Check ResourceSyncConfig status
kubectl get resourcesyncconfigs --output jsonpath='{.items[*].status}'

# Check controller logs for sync errors
kubectl logs --selector app.kubernetes.io/name=dot-ai-controller --namespace dot-ai | grep -i "resourcesync\|sync"

# Verify MCP endpoint is reachable
kubectl exec --namespace dot-ai deployment/dot-ai-controller-manager -- \
  curl -v http://dot-ai-mcp.dot-ai.svc.cluster.local:3456/api/v1/resources/sync
```

**Common Causes:**
- MCP resource sync endpoint not available
- Wrong `mcpEndpoint` URL in ResourceSyncConfig
- Network policies blocking communication
- RBAC permissions missing for resource discovery

**Solution:**
```bash
# Verify the MCP endpoint URL is correct
kubectl get resourcesyncconfigs --output jsonpath='{.items[*].spec.mcpEndpoint}'

# Check if watcher is active
kubectl get resourcesyncconfigs --output jsonpath='{.items[*].status.active}'

# Check watched resource types count
kubectl get resourcesyncconfigs --output jsonpath='{.items[*].status.watchedResourceTypes}'
```

### 8. CapabilityScanConfig Not Scanning

**Symptoms:**
```bash
# CapabilityScanConfig status shows errors or not ready
kubectl get capabilityscanconfigs --output yaml
```

**Diagnosis:**
```bash
# Check CapabilityScanConfig status
kubectl get capabilityscanconfigs --output jsonpath='{.items[*].status}'

# Check controller logs for scan errors
kubectl logs --selector app.kubernetes.io/name=dot-ai-controller --namespace dot-ai | grep -i "capabilityscan"

# Verify auth secret exists
kubectl get secret dot-ai-secrets --namespace dot-ai
```

**Common Causes:**
- MCP endpoint not available
- Wrong `mcp.endpoint` URL in CapabilityScanConfig
- Missing or invalid `mcp.authSecretRef` secret
- Resource filters excluding all resources

**Solution:**
```bash
# Verify the MCP endpoint URL is correct
kubectl get capabilityscanconfigs --output jsonpath='{.items[*].spec.mcp.endpoint}'

# Check if initial scan completed
kubectl get capabilityscanconfigs --output jsonpath='{.items[*].status.initialScanComplete}'

# Check last error
kubectl get capabilityscanconfigs --output jsonpath='{.items[*].status.lastError}'

# Verify include/exclude filters aren't too restrictive
kubectl get capabilityscanconfigs --output jsonpath='{.items[*].spec.includeResources}'
```

### 9. GitKnowledgeSource Not Syncing

**Symptoms:**
```bash
# GitKnowledgeSource status shows errors or Synced condition is False
kubectl get gitknowledgesources --output yaml
```

**Diagnosis:**
```bash
# Check GitKnowledgeSource status
kubectl get gitknowledgesources -n dot-ai -o jsonpath='{.items[*].status}'

# Check controller logs for sync errors
kubectl logs --selector app.kubernetes.io/name=dot-ai-controller --namespace dot-ai | grep -i "gitknowledge\|clone"

# Verify MCP endpoint is reachable
kubectl exec --namespace dot-ai deployment/dot-ai-controller-manager -- \
  curl -v http://dot-ai.dot-ai.svc:3456/health
```

**Common Causes:**
- **CloneError with "read-only file system"**: Controller deployment missing `/tmp` volume mount
- **Authentication failure**: Invalid or missing token for private repositories
- **MCP unreachable**: Wrong MCP server URL or network issues
- **Invalid path patterns**: Glob patterns not matching any files

**Solution:**
```bash
# Check for read-only filesystem error (needs /tmp volume)
kubectl get gitknowledgesources -n dot-ai -o jsonpath='{.items[*].status.lastError}'

# Verify the controller has /tmp volume mounted
kubectl get deployment dot-ai-controller-manager -n dot-ai -o jsonpath='{.spec.template.spec.containers[0].volumeMounts}'

# If missing, patch to add /tmp volume:
kubectl patch deployment dot-ai-controller-manager -n dot-ai --type='json' -p='[
  {"op": "add", "path": "/spec/template/spec/volumes", "value": [{"name": "tmp-dir", "emptyDir": {}}]},
  {"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts", "value": [{"name": "tmp-dir", "mountPath": "/tmp"}]}
]'

# For private repo auth issues, verify secret exists
kubectl get secret <secret-name> -n dot-ai -o jsonpath='{.data.<key>}' | base64 -d
```

### 10. ResourceSync High Traffic or Performance Issues

**Symptoms:**
- High CPU/memory usage on controller
- Frequent sync requests to MCP
- Slow cluster performance

**Diagnosis:**
```bash
# Check sync frequency and resource counts
kubectl get resourcesyncconfigs --output yaml | grep -A5 status

# Check debounce and resync settings
kubectl get resourcesyncconfigs --output yaml | grep -E "debounceWindowSeconds|resyncIntervalMinutes"
```

**Solution:**

Adjust debounce and resync intervals in your ResourceSyncConfig:
```yaml
spec:
  debounceWindowSeconds: 30   # Increase to batch more changes
  resyncIntervalMinutes: 120  # Increase to reduce full resyncs
```

## Getting Help

### Collect Diagnostic Information

When reporting issues, include this diagnostic information:

```bash
# Controller status and logs
kubectl get pods --namespace dot-ai
kubectl logs --selector app.kubernetes.io/name=dot-ai-controller --namespace dot-ai --tail 100

# MCP status and logs
kubectl logs --namespace dot-ai --selector app.kubernetes.io/name=dot-ai --tail 50

# RemediationPolicy configuration
kubectl get remediationpolicies --namespace dot-ai --output yaml

# ResourceSyncConfig configuration and status
kubectl get resourcesyncconfigs --all-namespaces --output yaml

# CapabilityScanConfig configuration and status
kubectl get capabilityscanconfigs --all-namespaces --output yaml

# Recent events
kubectl get events --namespace dot-ai --sort-by='.lastTimestamp' --field-selector type=Warning
```

### Enable Debug Logging

For more detailed troubleshooting, you can increase log verbosity:

```bash
# Edit the controller deployment to add debug flags
kubectl patch deployment dot-ai-controller-manager --namespace dot-ai --patch='
{
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "name": "manager",
            "args": ["--leader-elect", "--health-probe-bind-address=:8081", "-v=2"]
          }
        ]
      }
    }
  }
}'
```

## Resource Requirements

The default resource limits are:

**Controller:**
- Limits: 500m CPU, 128Mi memory
- Requests: 10m CPU, 64Mi memory

**MCP:**
- Limits: 1 CPU, 2Gi memory  
- Requests: 200m CPU, 512Mi memory

These should be sufficient for most use cases, but may need adjustment for high-volume environments.