# Remediate Guide **Complete guide for AI-powered Kubernetes issue analysis and remediation with the DevOps AI Toolkit.** > **Note**: The examples below use MCP, but this tool is also available via the [CLI](/docs/cli), [Web Dashboard](/docs/ui), and [Headlamp Plugin](/docs/headlamp). ## Prerequisites Before using this guide, complete the [Deployment Guide](../setup/deployment.md) to set up: - DevOps AI Toolkit deployed - AI model API key configured (see [AI Model Configuration](../setup/deployment.md#ai-model-configuration) for supported models and setup) - `KUBECONFIG` pointing to your Kubernetes cluster (optional for pure analysis) - kubectl command-line tool installed **Required Dependencies:** - kubectl access to cluster for investigation (get, describe, logs, events, top, explain operations) **Optional Dependencies:** - Write permissions for remediation execution (patch, apply, delete, scale, etc.) - only needed if you want to execute recommended fixes ## Overview **What it does:** - **AI-powered root cause analysis** - Multi-iteration investigation loop to understand Kubernetes issues - **Helm-aware diagnostics** - Detects and diagnoses issues with Helm-managed applications, including stuck releases and failed upgrades - **GitOps-aware remediation** - Detects Argo CD Applications and Flux Kustomizations, clones source repos, and automatically creates GitHub pull requests with the corrected manifests instead of running kubectl commands - **Intelligent data gathering** - Safe, read-only cluster investigation with targeted operations - **Comprehensive remediation plans** - Step-by-step solutions with risk assessment and validation - **Dual execution modes** - Manual approval workflow or automatic execution based on confidence/risk thresholds - **Iterative remediation support** - Handles complex issues requiring multiple sequential fixes - **Complete audit trail** - Full investigation history and remediation tracking - **MCP server integration** - Augment investigations with tools from external MCP servers (e.g., Prometheus metrics, distributed traces). See [MCP Server Integration](../setup/deployment.md#mcp-server-integration) - **Untrusted-content boundary** - Every tool result re-enters the model's context wrapped in explicit delimiters, framed as data to be analyzed rather than instruction to be followed. Always on, nothing to configure. See [Untrusted Content in `remediate` and `operate`](../operations/untrusted-content.md) **Use when:** - Investigating any Kubernetes resource issue — deployments, services, ingresses, storage, networking, and more - Diagnosing Helm release failures or stuck upgrades - Troubleshooting GitOps-managed resources where direct kubectl fixes get reverted ## Related Projects **[dot-ai-controller](/docs/controller/remediation-guide/)** - A Kubernetes controller that transforms reactive infrastructure management into proactive, AI-enhanced operations. The controller monitors Kubernetes cluster events and automatically triggers this project's remediation tool for intelligent analysis and resolution. It supports both automatic remediation (where AI directly implements solutions) and manual remediation (where AI provides recommendations for human review), with configurable filtering, rate limiting, Slack notifications, and confidence/risk thresholds. This enables continuous cluster health monitoring with intelligent auto-healing capabilities. ### How AI-Driven Investigation Works **Traditional Problem**: Basic kubectl commands show symptoms but require manual analysis to identify root causes. **AI Investigation Solution**: Multi-step analysis loop gathers relevant data and reasons through complex interdependencies. **Investigation Flow**: 1. **Initial Analysis**: AI reviews issue description 2. **Data Gathering Loop**: AI requests specific kubectl operations to gather targeted information 3. **Iterative Analysis**: Each data point informs the next investigation step 4. **Root Cause Identification**: AI synthesizes findings into comprehensive analysis 5. **GitOps Detection**: If the resource is managed by Argo CD or Flux, AI clones the source repo and reads the manifests 6. **Remediation Generation**: Automatic PR creation (GitOps-managed) or kubectl commands (standard), with risk assessment and validation instructions ## Complete Workflow Examples This section demonstrates both manual and automatic execution modes: - **[Manual Mode Example](#manual-mode-example)** - User controls execution timing with approval choices - **[Automatic Mode Example](#automatic-mode-example)** - Tool executes automatically based on confidence/risk thresholds - **[GitOps-Aware Remediation Example](#gitops-remediation-example)** - Tool detects Argo CD/Flux management and automatically creates a GitHub PR with the fix ### Option 1: Automatic Execution **User selects Choice 1** - Tool executes commands internally and returns complete results ```text User: 1 ``` ```text Agent: I'll execute the remediation automatically and validate the results. [Uses remediate tool with executeChoice: 1, sessionId: "rem_2025-09-20T1653_818d80901b10ce62", mode: "manual"] ✅ **Issue Resolved Successfully!** **Executed Command:** kubectl apply -f - < ### Automatic Mode Example {#automatic-mode-example} **Automatic Mode**: Tool executes first remediation automatically, then falls back to manual mode when validation discovers additional issues ``` User: Something is wrong with my database in sql-demo namespace. Remediate it automatically. ``` ``` Agent: I'll help you remediate the database issue in the sql-demo namespace. Let me use the remediate tool to automatically analyze and fix the problem. [Uses remediate tool with issue: "Database issue in sql-demo namespace", mode: "automatic", maxRiskLevel: "medium", confidenceThreshold: 0.7] The AI analysis has identified the root cause of your database issue with 95% confidence: Root Cause: The SQL resource 'test-db' is failing because the KCL function is trying to concatenate 'POSTGRES_' with a missing 'version' field in the resource specification. Recommended Action: - Add version field to the SQL resource with PostgreSQL version 13 - Risk Level: Low - Command: kubectl patch sql/test-db -n sql-demo --type=merge -p '{"spec":{"version":"13"}}' Choose execution method: 1. Execute automatically (Recommended) - Run the command automatically - Risk: Low 2. Execute via agent - I'll run the kubectl command using the Bash tool - Then validate the results - Risk: Low Which option would you prefer? (1 or 2) ``` **What happened behind the scenes:** - Agent interpreted "automatically" and set confidence/risk thresholds (70% confidence, medium risk allowed) - Tool executed first remediation automatically (compositionRef issue): `kubectl patch sql/test-db -n sql-demo --type=merge -p '{"spec":{"crossplane":{"compositionRef":{"name":"google-postgresql"}}}}'` - Post-execution validation discovered a new issue (missing version field) - **Safety mechanism activated**: Tool switches to manual mode when validation finds additional issues - User sees current issue analysis but tool includes `"executed": true` and `"results": [...]` showing previous automatic execution - This prevents potentially dangerous infinite execution loops while maintaining transparency **Key Automatic Mode Features Demonstrated:** - **Single-issue auto-execution**: First issue (compositionRef) was fixed without user intervention - **Intelligent thresholds**: Agent interpreted "automatically" as moderate risk tolerance (70% confidence, medium risk) - **Safety fallback**: After first fix, requires manual approval for additional issues discovered during validation - **Complete transparency**: JSON shows both previous execution results and current issue analysis - **User control**: Users can specify execution preferences like "high confidence only" or "allow risky fixes" ## Tool Parameter Reference ### User-Defined Parameters *Parameters that users specify (directly or through natural language that agents interpret)* #### `issue` (string, optional when continuing session) The issue description that needs analysis and remediation. - **User provides**: `"Pod stuck in Pending state"`, `"Database connection failing"`, `"Storage issues in namespace prod"` - **Best practices**: Be specific about symptoms, include namespace/resource names when known #### `evidence` (string, optional) Supporting material the caller quoted from somewhere else rather than wrote — log lines, events, a manifest, an alert payload. - **Who sets it**: Integrations that can separate their own instruction from quoted telemetry at capture time (a controller, a dashboard, an alerting pipeline). Interactive users do not need it - **How it is treated**: Composed into the prompt inside an untrusted-content boundary and analyzed as data; instructions appearing in it are never followed - **Omitting it changes nothing**: A call without `evidence` produces the prompt it always produced, byte for byte - **Persistence**: On `remediate`, `evidence` is stored on the session and reused for the rest of it (unlike `operate`, which takes it per call) - **Details**: [Untrusted Content in `remediate` and `operate`](../operations/untrusted-content.md#the-evidence-parameter) #### `mode` (string, optional, default: `"manual"`) Execution mode determining how remediation actions are handled. - **Values**: `"manual"` | `"automatic"` - **User examples**: - `"fix this automatically"` → agent sets `"automatic"` - `"show me options first"` → agent sets `"manual"` - **Note**: when the server runs with [constrained execution](#constraining-automatic-execution) enabled, both modes are restricted to structured kubectl operations — a free-form command is refused in manual mode too, not offered for approval #### `confidenceThreshold` (number, optional, default: 0.8) Minimum AI confidence required for automatic execution (automatic mode only). - **Range**: 0.0 to 1.0 (0% to 100% confidence) - **User examples**: - `"only if you're very confident"` → agent sets `0.9` - `"try even with lower confidence"` → agent sets `0.6` #### `maxRiskLevel` (string, optional, default: `"low"`) Maximum risk level allowed for automatic execution (automatic mode only). - **Values**: `"low"` | `"medium"` | `"high"` - **Risk levels**: - **`"low"`**: Safe operations (adding labels, scaling, creating resources) - **`"medium"`**: Config modifications, pod restarts, patching existing resources - **`"high"`**: Resource deletion, cluster-wide changes, disruptive operations - **User examples**: - `"allow medium risk fixes"` → agent sets `"medium"` - `"only safe operations"` → agent sets `"low"` #### `executeChoice` (number, optional) Execute a previously generated choice from manual mode workflow. - **Values**: `1` (Execute automatically) | `2` (Execute via agent) - **User selects**: User responds with `"1"` or `"2"` to execution choice prompts - **Requires**: `sessionId` parameter from previous tool response ### Agent-Managed Parameters *Parameters that agents manage internally - users never specify these directly* #### `sessionId` (string, optional) Session ID from previous remediate call for continuing workflows. - **Format**: `"rem_YYYYMMDDTHHMM_hexstring"` - **Agent usage**: Extracted from previous tool responses, never user-provided - **Required**: When using `executeChoice` or validation calls #### `executedCommands` (array of strings, optional) Commands that were executed for post-execution validation. - **Agent usage**: Populated when agent executes commands manually and calls tool for validation - **Format**: Array of command descriptions or kubectl commands - **Never user-provided**: Generated by agent based on actions taken