agency-agents/strategy/runbooks/scenario-incident-response.md

🚨 Runbook: Incident Response

Mode: NEXUS-Micro | Duration: Minutes to hours | Agents: 3-8

---

Scenario

Something is broken in production. Users are affected. Speed of response matters, but so does doing it right. This runbook covers detection through post-mortem.

Severity Classification

Level	Definition	Examples	Response Time
P0 — Critical	Service completely down, data loss, security breach	Database corruption, DDoS attack, auth system failure	Immediate (all hands)
P1 — High	Major feature broken, significant performance degradation	Payment processing down, 50%+ error rate, 10x latency	< 1 hour
P2 — Medium	Minor feature broken, workaround available	Search not working, non-critical API errors	< 4 hours
P3 — Low	Cosmetic issue, minor inconvenience	Styling bug, typo, minor UI glitch	Next sprint

Response Teams by Severity

P0 — Critical Response Team

Agent	Role	Action
Infrastructure Maintainer	Incident commander	Assess scope, coordinate response
DevOps Automator	Deployment/rollback	Execute rollback if needed
Backend Architect	Root cause investigation	Diagnose system issues
Frontend Developer	UI-side investigation	Diagnose client-side issues
Support Responder	User communication	Status page updates, user notifications
Executive Summary Generator	Stakeholder communication	Real-time executive updates

P1 — High Response Team

Agent	Role
Infrastructure Maintainer	Incident commander
DevOps Automator	Deployment support
Relevant Developer Agent	Fix implementation
Support Responder	User communication

P2 — Medium Response

Agent	Role
Relevant Developer Agent	Fix implementation
Evidence Collector	Verify fix

P3 — Low Response

Agent	Role
Sprint Prioritizer	Add to backlog

Incident Response Sequence

Step 1: Detection & Triage (0-5 minutes)

TRIGGER: Alert from monitoring / User report / Agent detection

Infrastructure Maintainer:
1. Acknowledge alert
2. Assess scope and impact
   - How many users affected?
   - Which services are impacted?
   - Is data at risk?
3. Classify severity (P0/P1/P2/P3)
4. Activate appropriate response team
5. Create incident channel/thread

Output: Incident classification + response team activated

Step 2: Investigation (5-30 minutes)

PARALLEL INVESTIGATION:

Infrastructure Maintainer:
├── Check system metrics (CPU, memory, network, disk)
├── Review error logs
├── Check recent deployments
└── Verify external dependencies

Backend Architect (if P0/P1):
├── Check database health
├── Review API error rates
├── Check service communication
└── Identify failing component

DevOps Automator:
├── Review recent deployment history
├── Check CI/CD pipeline status
├── Prepare rollback if needed
└── Verify infrastructure state

Output: Root cause identified (or narrowed to component)

Step 3: Mitigation (15-60 minutes)

DECISION TREE:

IF caused by recent deployment:
  → DevOps Automator: Execute rollback
  → Infrastructure Maintainer: Verify recovery
  → Evidence Collector: Confirm fix

IF caused by infrastructure issue:
  → Infrastructure Maintainer: Scale/restart/failover
  → DevOps Automator: Support infrastructure changes
  → Verify recovery

IF caused by code bug:
  → Relevant Developer Agent: Implement hotfix
  → Evidence Collector: Verify fix
  → DevOps Automator: Deploy hotfix
  → Infrastructure Maintainer: Monitor recovery

IF caused by external dependency:
  → Infrastructure Maintainer: Activate fallback/cache
  → Support Responder: Communicate to users
  → Monitor for external recovery

THROUGHOUT:
  → Support Responder: Update status page every 15 minutes
  → Executive Summary Generator: Brief stakeholders (P0 only)

Step 4: Resolution Verification (Post-fix)

Evidence Collector:
1. Verify the fix resolves the issue
2. Screenshot evidence of working state
3. Confirm no new issues introduced

Infrastructure Maintainer:
1. Verify all metrics returning to normal
2. Confirm no cascading failures
3. Monitor for 30 minutes post-fix

API Tester (if API-related):
1. Run regression on affected endpoints
2. Verify response times normalized
3. Confirm error rates at baseline

Output: Incident resolved confirmation

Step 5: Post-Mortem (Within 48 hours)

Workflow Optimizer leads post-mortem:

1. Timeline reconstruction
   - When was the issue introduced?
   - When was it detected?
   - When was it resolved?
   - Total user impact duration

2. Root cause analysis
   - What failed?
   - Why did it fail?
   - Why wasn't it caught earlier?
   - 5 Whys analysis

3. Impact assessment
   - Users affected
   - Revenue impact
   - Reputation impact
   - Data impact

4. Prevention measures
   - What monitoring would have caught this sooner?
   - What testing would have prevented this?
   - What process changes are needed?
   - What infrastructure changes are needed?

5. Action items
   - [Action] → [Owner] → [Deadline]
   - [Action] → [Owner] → [Deadline]
   - [Action] → [Owner] → [Deadline]

Output: Post-Mortem Report → Sprint Prioritizer adds prevention tasks to backlog

Communication Templates

Status Page Update (Support Responder)

[TIMESTAMP] — [SERVICE NAME] Incident

Status: [Investigating / Identified / Monitoring / Resolved]
Impact: [Description of user impact]
Current action: [What we're doing about it]
Next update: [When to expect the next update]

Executive Update (Executive Summary Generator — P0 only)

INCIDENT BRIEF — [TIMESTAMP]

SITUATION: [Service] is [down/degraded] affecting [N users/% of traffic]
CAUSE: [Known/Under investigation] — [Brief description if known]
ACTION: [What's being done] — ETA [time estimate]
IMPACT: [Business impact — revenue, users, reputation]
NEXT UPDATE: [Timestamp]

Escalation Matrix

Condition	Escalate To	Action
P0 not resolved in 30 min	Studio Producer	Additional resources, vendor escalation
P1 not resolved in 2 hours	Project Shepherd	Resource reallocation
Data breach suspected	Legal Compliance Checker	Regulatory notification assessment
User data affected	Legal Compliance Checker + Executive Summary Generator	GDPR/CCPA notification
Revenue impact > $X	Finance Tracker + Studio Producer	Business impact assessment