# Remediate Guide

**Complete guide for AI-powered Kubernetes issue analysis and remediation with the DevOps AI Toolkit.**


> **Note**: The examples below use MCP, but this tool is also available via the [CLI](/docs/cli), [Web Dashboard](/docs/ui), and [Headlamp Plugin](/docs/headlamp).

## Prerequisites

Before using this guide, complete the [Deployment Guide](../setup/deployment.md) to set up:
- DevOps AI Toolkit deployed
- AI model API key configured (see [AI Model Configuration](../setup/deployment.md#ai-model-configuration) for supported models and setup)
- `KUBECONFIG` pointing to your Kubernetes cluster (optional for pure analysis)
- kubectl command-line tool installed

**Required Dependencies:**
- kubectl access to cluster for investigation (get, describe, logs, events, top, explain operations)

**Optional Dependencies:**  
- Write permissions for remediation execution (patch, apply, delete, scale, etc.) - only needed if you want to execute recommended fixes

## Overview

**What it does:**
- **AI-powered root cause analysis** - Multi-iteration investigation loop to understand Kubernetes issues
- **Helm-aware diagnostics** - Detects and diagnoses issues with Helm-managed applications, including stuck releases and failed upgrades
- **GitOps-aware remediation** - Detects Argo CD Applications and Flux Kustomizations, clones source repos, and automatically creates GitHub pull requests with the corrected manifests instead of running kubectl commands
- **Intelligent data gathering** - Safe, read-only cluster investigation with targeted operations
- **Comprehensive remediation plans** - Step-by-step solutions with risk assessment and validation
- **Dual execution modes** - Manual approval workflow or automatic execution based on confidence/risk thresholds
- **Iterative remediation support** - Handles complex issues requiring multiple sequential fixes
- **Complete audit trail** - Full investigation history and remediation tracking
- **MCP server integration** - Augment investigations with tools from external MCP servers (e.g., Prometheus metrics, distributed traces). See [MCP Server Integration](../setup/deployment.md#mcp-server-integration)

**Use when:**
- Investigating any Kubernetes resource issue — deployments, services, ingresses, storage, networking, and more
- Diagnosing Helm release failures or stuck upgrades
- Troubleshooting GitOps-managed resources where direct kubectl fixes get reverted

## Related Projects

**[dot-ai-controller](/docs/controller/remediation-guide/)** - A Kubernetes controller that transforms reactive infrastructure management into proactive, AI-enhanced operations. The controller monitors Kubernetes cluster events and automatically triggers this project's remediation tool for intelligent analysis and resolution. It supports both automatic remediation (where AI directly implements solutions) and manual remediation (where AI provides recommendations for human review), with configurable filtering, rate limiting, Slack notifications, and confidence/risk thresholds. This enables continuous cluster health monitoring with intelligent auto-healing capabilities.

### How AI-Driven Investigation Works

**Traditional Problem**: Basic kubectl commands show symptoms but require manual analysis to identify root causes.

**AI Investigation Solution**: Multi-step analysis loop gathers relevant data and reasons through complex interdependencies.

**Investigation Flow**:
1. **Initial Analysis**: AI reviews issue description
2. **Data Gathering Loop**: AI requests specific kubectl operations to gather targeted information
3. **Iterative Analysis**: Each data point informs the next investigation step
4. **Root Cause Identification**: AI synthesizes findings into comprehensive analysis
5. **GitOps Detection**: If the resource is managed by Argo CD or Flux, AI clones the source repo and reads the manifests
6. **Remediation Generation**: Automatic PR creation (GitOps-managed) or kubectl commands (standard), with risk assessment and validation instructions

<!-- 
Test Scenario Setup for Complex Cross-Resource Example:
Create a database pod that references a non-existent PVC to demonstrate resource creation remediation

kubectl create namespace postgres-demo
kubectl apply -n postgres-demo -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:13
        env:
        - name: POSTGRES_PASSWORD
          value: testpass
        - name: POSTGRES_DB
          value: testdb
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: postgres-storage
        persistentVolumeClaim:
          claimName: postgres-pvc
EOF

This creates a deployment where:
- Pod will be Pending due to missing PVC "postgres-pvc"
- Tool should investigate and recommend creating the missing PVC 
- Demonstrates cross-resource problem (pod issue) requiring solution in different resource (storage)
- Uses separate namespace to keep demo isolated
-->

## Complete Workflow Examples

This section demonstrates both manual and automatic execution modes:

- **[Manual Mode Example](#manual-mode-example)** - User controls execution timing with approval choices
- **[Automatic Mode Example](#automatic-mode-example)** - Tool executes automatically based on confidence/risk thresholds
- **[GitOps-Aware Remediation Example](#gitops-remediation-example)** - Tool detects Argo CD/Flux management and automatically creates a GitHub PR with the fix


### Option 1: Automatic Execution

**User selects Choice 1** - Tool executes commands internally and returns complete results

```text
User: 1
```

```text
Agent: I'll execute the remediation automatically and validate the results.

[Uses remediate tool with executeChoice: 1, sessionId: "rem_2025-09-20T1653_818d80901b10ce62", mode: "manual"]

✅ **Issue Resolved Successfully!**

**Executed Command:**
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
  namespace: postgres-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard
EOF

**Result:** `persistentvolumeclaim/postgres-pvc created`

**Validation Confirmed (95% confidence):**
- postgres-pvc is now bound to persistent volume
- PostgreSQL pod successfully transitioned from Pending to Running state  
- Pod successfully mounted postgres-pvc volume at /var/lib/postgresql/data
- Database is ready and accessible

Your postgres-demo namespace database is now working correctly!
```

**What happened behind the scenes:**
- Tool executed the apply command internally
- Post-execution validation ran automatically (4 iterations, 6 data sources)
- AI confirmed PVC creation and binding to persistent volume
- AI verified PostgreSQL pod transition from Pending to Running state
- Complete end-to-end resolution in a single tool call

#### Post-Remediation Visualization

After the remediation completes, click the **Reload** button in the Web UI to see the updated visualization with remediation steps and validation results:



<!--
Test Scenario Setup for Automatic Mode Example:
Create a SQL custom resource with configuration issues to demonstrate automatic mode execution

kubectl create namespace sql-demo
kubectl apply -n sql-demo -f - <<'EOF'
apiVersion: devopstoolkit.live/v1beta1
kind: SQL
metadata:
  name: test-db
spec:
  size: small
  databases: ["myapp"]
  region: us-east-1
EOF

This creates a SQL custom resource that will likely have configuration issues requiring remediation.
The automatic mode will demonstrate the tool executing fixes without user interaction when confidence/risk thresholds are met.
-->

### Automatic Mode Example {#automatic-mode-example}

**Automatic Mode**: Tool executes first remediation automatically, then falls back to manual mode when validation discovers additional issues

```
User: Something is wrong with my database in sql-demo namespace. Remediate it automatically.
```

```
Agent: I'll help you remediate the database issue in the sql-demo namespace. Let me use the remediate tool to automatically analyze and fix the problem.

[Uses remediate tool with issue: "Database issue in sql-demo namespace", mode: "automatic", maxRiskLevel: "medium", confidenceThreshold: 0.7]

The AI analysis has identified the root cause of your database issue with 95% confidence:

Root Cause: The SQL resource 'test-db' is failing because the KCL function is trying to concatenate 'POSTGRES_' with a missing 'version' field in the resource specification.

Recommended Action:
- Add version field to the SQL resource with PostgreSQL version 13
- Risk Level: Low
- Command: kubectl patch sql/test-db -n sql-demo --type=merge -p '{"spec":{"version":"13"}}'

Choose execution method:

1. Execute automatically (Recommended)
- Run the command automatically
- Risk: Low

2. Execute via agent  
- I'll run the kubectl command using the Bash tool
- Then validate the results
- Risk: Low

Which option would you prefer? (1 or 2)
```

**What happened behind the scenes:**
- Agent interpreted "automatically" and set confidence/risk thresholds (70% confidence, medium risk allowed)
- Tool executed first remediation automatically (compositionRef issue): `kubectl patch sql/test-db -n sql-demo --type=merge -p '{"spec":{"crossplane":{"compositionRef":{"name":"google-postgresql"}}}}'` 
- Post-execution validation discovered a new issue (missing version field)
- **Safety mechanism activated**: Tool switches to manual mode when validation finds additional issues
- User sees current issue analysis but tool includes `"executed": true` and `"results": [...]` showing previous automatic execution
- This prevents potentially dangerous infinite execution loops while maintaining transparency

**Key Automatic Mode Features Demonstrated:**
- **Single-issue auto-execution**: First issue (compositionRef) was fixed without user intervention  
- **Intelligent thresholds**: Agent interpreted "automatically" as moderate risk tolerance (70% confidence, medium risk)
- **Safety fallback**: After first fix, requires manual approval for additional issues discovered during validation
- **Complete transparency**: JSON shows both previous execution results and current issue analysis
- **User control**: Users can specify execution preferences like "high confidence only" or "allow risky fixes"