Zoekt Production Readiness: Testing, monitoring, docs, deployment

[Anthony-Bible]

Parent: #40

Goal

Harden the Zoekt integration for production: comprehensive testing, monitoring, documentation, and safe deployment.

Tasks

• Comprehensive integration testing across scenarios
• Load and performance testing at scale (1M+ LOC repos)
• Chaos testing
  • Zoekt webserver down -> degraded mode works
  • Network issues between services
  • Partial availability (one engine indexed, other not)
  • Concurrent failure scenarios
• Concurrency stress testing (max concurrent indexings, resource exhaustion)
• Race condition validation: go test -race ./...
• Search throughput validation: 100+ queries/second per Zoekt instance
• Update API documentation (OpenAPI spec, usage examples for all 3 modes)
• Create deployment guides (Docker Compose, Kubernetes)
• Set up production monitoring and alerting
  • Zoekt uptime > 99.5%
  • Indexing failure rate < 1% per engine
  • Latency SLA monitoring
• Security review (Zoekt access controls)
• Feature flag for search modes (safe rollout)
• Deploy to staging, run canary tests
• Monitor concurrent pipeline behavior (throughput, resource utilization)
• Validate concurrent performance improvements (30%+ throughput gain)
• Production deployment with rollback plan
• Monitor success metrics for 30 days
  • Text search <50ms for 95% of exact-match queries
  • Search success rate: 95%+ for exact queries
  • Zero API breaking changes
  • 100% of processed repos have both indexes
  • 30%+ of searches use text or hybrid mode within 3 months

Deliverables

• Production-ready deployment artifacts
• Complete documentation (API, deployment, configuration)
• Monitoring and alerting stack
• Race-free concurrent code validated
• Stable release in production
• Success metrics validated

Dependencies

• Zoekt Foundation: Deploy webserver, basic client, API mode parameter #41 (Phase 1: Foundation)
• Zoekt Concurrent Indexing: Parallel worker pipeline for dual-engine indexing #42 (Phase 2: Concurrent Indexing)
• Zoekt Search Integration: Text search, query translation, hybrid ranking #43 (Phase 3: Search Integration)

Out of Scope

• Symbol search / ctags
• Distributed Zoekt clusters
• Advanced hybrid ranking