Self-repair Phase 5: failure cluster store with PVC backing

[rockfordlhotka]

Closes #349

Summary

• In-process ConcurrentDictionary<ClusterKey, FailureCluster> keyed by (server, tool, errorClass) for hot reads/writes.
• PVC-backed persistence: append-only failure-clusters.jsonl plus a periodic snapshot at failure-clusters.snapshot.json, both under /data/agent/telemetry. On startup, snapshot is loaded then JSONL events with at >= snapshotWrittenAt are replayed; on flush, the snapshot is written atomically and the JSONL is truncated.
• McpRecoveryExecutor records every post-recovery failure (chain-exhausted, Stage A retry-fail, Stage B retry-fail, Stage B fill-fail, and the no-provider/no-StageB short-circuit) with the originating session id from ToolInvokeRequest.SessionId. Auto-recovered calls do NOT record — those remain in RecoveryDiagnostics metrics only.
• GetEscalatableAsync filters to Count >= 3 && SessionIds.Count >= 2 && (now - LastSeen) < 24h so Phase 4 (Self-repair Phase 4: closed-loop repair tickets in DreamService #348) can iterate ready-to-ticket clusters without re-implementing the threshold.
• Bounds on per-cluster state: 5 most-recent sample messages (truncated at 512 chars each), 64 distinct session ids; configurable via FailureClusterOptions.
• Recording and the snapshot/JSONL writes share a single file lock, so concurrent records can't be silently dropped between flush capture and JSONL truncation.

Test plan

• FailureClusterTests — ClusterKey lowercases server/tool, rejects blanks, equality is canonical.
• FailureErrorClassifierTests — extracts field names from each Phase-1 pattern, falls back to unknown.
• FileFailureClusterStoreTests (12 cases) — escalation thresholds (count, session count, recency window), bounds enforcement, snapshot+JSONL persistence round-trip, JSONL truncation, corrupt-line recovery, ordering by LastSeen desc.
• McpRecoveryExecutorFailureClusterTests (9 cases) — auto-recovered does NOT record; each exhausted-recovery branch DOES record with the right errorClass and forwarded sessionId; null IFailureClusterStore keeps Phase-1 contract; throwing store does not break recovery.
• Full suite: 16 test assemblies, 0 failures.

Acceptance criteria (from #349)

Criterion	Coverage
3 same-class failures across 2 sessions in 24h → one escalatable cluster	GetEscalatable_ThreeFailuresAcrossTwoSessions_InWindow_ReturnsCluster
Restart preserves cluster state via PVC	Persistence_SnapshotAndJsonl_RestoreClusterState
JSONL truncated after each snapshot	Flush_TruncatesJsonl
Auto-recovered calls do NOT show up	AutoRecovered_StageA_DoesNotRecord

Out of scope

• DreamService reading the store and creating tickets — Phase 4 (Self-repair Phase 4: closed-loop repair tickets in DreamService #348).
• Canary failures with source=canary — Phase 6.

🤖 Generated with Claude Code