Systematic incident triage protocol: Log Analysis → Minimal Reproduction → Root Cause Isolation → Regression Test → Hotfix.
Production Bug Triage & Root Cause Analysis Workflow
Follow this step-by-step protocol to isolate and resolve production defects without introducing unintended side effects.
Phase 1: Ingest & Triage
1. Extract Error Telemetry: Collect stack trace, error message, HTTP status code, and client user-agent.
2. Determine Blast Radius:
- Is it affecting all users or a subset (e.g. specific browser, authentication tier, payload size)?
- Classify severity: P0 (Service Down), P1 (Core Function Broken), P2 (Degraded UX).
Phase 2: Create a Minimal Reproduction Case
- Create an automated test in
tests/repro/ that replicates the exact failure condition.
- Run the test to confirm it fails with the expected error.
Phase 3: Root Cause Isolation (5 Whys)
- Trace the code execution path from input entry to exception throw.
- Identify whether the fault is in validation, state race condition, network timeout, or schema mismatch.
Phase 4: Implement Surgical Fix
- Apply the fix with the minimal necessary code diff.
- Verify the reproduction test now passes.
Phase 5: Regression Check & Post-Mortem Note
- Run the full test suite to guarantee zero regressions.
- Document the root cause, fix rationale, and prevention measures.