feat: Adaptive per-model capability detection

[nielspeter]

Summary

This PR replaces hardcoded model pattern matching with a fully adaptive per-model detection system that automatically learns which parameters each (provider, model) combination supports. This eliminates special-casing and works with any current or future OpenAI-compatible provider without code changes.

Key Changes

Code Refactoring (~100 lines removed):

• Removed IsReasoningModel() function with hardcoded gpt-5/o1/o2/o3/o4 patterns
• Removed FetchReasoningModels() function and OpenRouter API integration
• Removed ReasoningModelCache struct and related code
• Removed unused imports (encoding/json, net/http from config.go)

New Implementation:

• Added per-model capability caching with CacheKey{BaseURL, Model} structure
• Implemented broad keyword-based error detection (no status code restrictions)
• Added thread-safe in-memory cache with sync.RWMutex
• Added debug logging for cache hits/misses (visible with -d flag)

Architecture

Adaptive Detection Flow:

1. First Request (Cache Miss): Try max_completion_tokens → Error → Retry without it → Cache result
2. Subsequent Requests (Cache Hit): Use cached value immediately (instant)
3. Works with OpenWebUI, OpenRouter, OpenAI Direct, Ollama, and any OpenAI-compatible provider

Cache Structure:

type CacheKey struct {
    BaseURL string  // Provider base URL
    Model   string  // Model name
}

type ModelCapabilities struct {
    UsesMaxCompletionTokens bool      // Learned via retry
    LastChecked             time.Time // Timestamp
}

Benefits

• ✅ Zero user configuration - No need to know which parameters each provider supports
• ✅ Future-proof - Works with any new model/provider without code changes
• ✅ Per-model granularity - Same model name on different providers cached separately
• ✅ Auto-adapts to provider quirks - Handles OpenWebUI, misconfigured providers automatically
• ✅ Fast - Only 1-2 second penalty on first request, instant thereafter
• ✅ Thread-safe - Protected by mutex for concurrent requests

Testing

Tested with:

• ✅ OpenWebUI GPT-5 (gpt-5-2025-08-07)
• ✅ OpenRouter GPT-5 (openai/gpt-5)
• ✅ OpenAI Direct
• ✅ Multiple provider configurations

Debug logging example:

./claude-code-proxy -d -s

# First request:
[DEBUG] Cache MISS: gpt-5 → will auto-detect (try max_completion_tokens)
[DEBUG] Cached: model gpt-5 supports max_completion_tokens (streaming)

# Subsequent requests:
[DEBUG] Cache HIT: gpt-5 → max_completion_tokens=true

Documentation

• ✅ Added "Adaptive Per-Model Detection" section to README.md with comprehensive details
• ✅ Updated CLAUDE.md with technical implementation documentation
• ✅ CHANGELOG.md updated for v1.3.0 release
• ✅ Cleaned up docs/ folder (removed planning artifacts and obsolete docs)

Philosophy

This release embodies the project philosophy:

"Support all provider quirks automatically - never burden users with configurations they don't understand."

The adaptive system eliminates special-casing and works with any current or future OpenAI-compatible provider.

Files Changed

8 files changed, 477 insertions(+), 467 deletions(-)

CHANGELOG.md                    |  57 ++++++++
CLAUDE.md                       |  84 ++++++++++++
README.md                       |  80 +++++++++++
cmd/claude-code-proxy/main.go   |  13 +-
docs/OPENWEBUI-GPT5-FIX.md      | 290 ----------------------------------------
internal/config/config.go       | 152 +++++++++------------
internal/converter/converter.go |  27 ++--
internal/server/handlers.go     | 241 +++++++++++++++++++++++++--------

Breaking Changes

None. This is a drop-in replacement that maintains full backward compatibility while improving functionality.

Ready for v1.3.0 Release

This PR is ready to be merged and released as v1.3.0.

[claude]

Pull Request Review: Adaptive Per-Model Capability Detection (v1.3.0)

Overall Assessment

This is an excellent refactor that significantly improves the codebase architecture. The replacement of hardcoded model patterns with adaptive capability detection is a smart design choice that makes the proxy future-proof and more maintainable. ✅

---

Strengths

1. Architecture & Design ⭐⭐⭐⭐⭐

The adaptive detection system is well-architected:

• Zero-configuration approach eliminates user burden
• Per-model caching with struct-based keys (type-safe, zero collision risk)
• Thread-safe implementation with proper RWMutex usage
• Error-based retry mechanism is elegant and robust

2. Code Quality ⭐⭐⭐⭐½

• Clean separation of concerns between detection, caching, and retry logic
• Excellent code documentation and inline comments
• Good use of Go idioms (defer for unlocks, proper error wrapping)
• ~100 lines removed while adding functionality is impressive

3. Documentation ⭐⭐⭐⭐⭐

Outstanding documentation updates:

• Comprehensive CHANGELOG.md with clear migration path
• Detailed CLAUDE.md technical documentation
• README.md explains the philosophy and benefits clearly
• Debug logging examples help users understand the system

---

Issues & Concerns

1. CRITICAL: Test Coverage Gap 🔴

The tests are now outdated and will fail. The test files reference removed code:

// internal/config/config_test.go:628
reasoningCache = &ReasoningModelCache{  // ❌ This type no longer exists
    models:    make(map[string]bool),
    populated: false,
}

// internal/config/config_test.go:887
err := cfg.FetchReasoningModels()  // ❌ This method was removed

Impact: Lines 581-1037 in config_test.go test IsReasoningModel() and FetchReasoningModels(), both of which were removed in this PR.

Recommendation:

• Remove or rewrite tests for IsReasoningModel() (lines 581-1037)
• Add new tests for ShouldUseMaxCompletionTokens()
• Add tests for GetModelCapabilities() / SetModelCapabilities()
• Add integration tests for the retry mechanism

2. Test File Mismatch 🟡

internal/converter/reasoning_model_test.go still calls cfg.IsReasoningModel() (lines 10-166), which no longer exists. This file needs to be:

• Removed entirely, OR
• Refactored to test ShouldUseMaxCompletionTokens() instead

3. Potential Race Condition in Cache Initialization 🟡

// internal/config/config.go:46
var (
    modelCapabilityCache = make(map[CacheKey]*ModelCapabilities)
    capabilityCacheMutex sync.RWMutex
)

This is fine for most cases, but in a concurrent environment with many requests on first startup, multiple goroutines might simultaneously:

1. Check cache (miss)
2. All send requests with max_completion_tokens
3. All detect error and retry

Impact: Minor - just duplicate work on first request, not a correctness issue.

Recommendation: Consider using sync.Map or a more sophisticated cache with single-flight pattern to deduplicate concurrent first requests for the same model.

4. Error Detection May Be Too Broad 🟡

// internal/server/handlers.go:917-927
func isMaxTokensParameterError(errorMessage string) bool {
    errorLower := strings.ToLower(errorMessage)
    
    hasParamIndicator := strings.Contains(errorLower, "parameter") ||
                        strings.Contains(errorLower, "unsupported") ||
                        strings.Contains(errorLower, "invalid")
    
    hasOurParam := strings.Contains(errorLower, "max_tokens") ||
                   strings.Contains(errorLower, "max_completion_tokens")
    
    return hasParamIndicator && hasOurParam
}

Potential Issue: This could match errors like:

• "Invalid API key when using parameter max_tokens" (auth error, not param error)
• "Rate limit exceeded, parameter max_tokens cannot be processed" (rate limit, not param error)

Recommendation:

• Add negative checks for auth/rate limit keywords ("authentication", "rate limit", "quota")
• Log warnings when retrying so users can report false positives

5. Cache Persistence & Expiry 🟢 (Design Choice)

The cache is:

• ✅ In-memory only (cleared on restart)
• ✅ Never expires (LastChecked field is set but not used)

Consideration: For long-running proxies, if a provider adds support for max_completion_tokens after deployment, the cache won't pick it up until restart.

Recommendation (Optional): Consider adding a TTL (e.g., 24 hours) where cached capabilities are re-validated. This is likely over-engineering for v1.3.0 though.

---

Code Quality Observations

Positive

• ✅ Proper error wrapping with %w for error chains
• ✅ Consistent debug logging throughout
• ✅ Clear function naming and documentation
• ✅ Struct-based cache keys (better than string concatenation)

Minor Nits

1. Redundant debug check (handlers.go:787-789):

if cfg.Debug {
    fmt.Printf("[DEBUG] Detected max_completion_tokens parameter error...")
}

This pattern repeats 4 times. Consider a helper: debugLog(cfg, format, args...)

2. Magic strings for cache misses:
   The default behavior on cache miss is to try max_completion_tokens. This is implicit. Consider adding a const:

const defaultAssumeMaxCompletionTokens = true

---

Security Considerations

✅ No security concerns identified:

• Cache doesn't store sensitive data (only boolean capabilities)
• Thread-safe with proper locking
• No injection vulnerabilities in error string matching
• No changes to auth logic

---

Performance

✅ Performance improvements:

• Eliminates OpenRouter API calls on startup
• Cache lookups are O(1) map operations
• Only 1-2s penalty on first request per (provider, model) pair

⚠️ Minor overhead:

• Every request now acquires a read lock for cache lookup
• Impact is negligible compared to network latency

---

Recommendations for Merge

Must-Fix Before Merge 🔴

1. Update or remove failing tests:
   • Remove tests for deleted IsReasoningModel() and FetchReasoningModels()
   • Add tests for new ShouldUseMaxCompletionTokens() logic
   • Verify go test ./... passes

Should-Fix Before Merge 🟡

2. Add integration test for retry mechanism:

   • Mock an error response with "unsupported parameter max_completion_tokens"
   • Verify retry happens and cache is populated correctly

3. Improve error detection robustness:

   • Add negative keywords to avoid false positives
   • Add debug logging when retry is triggered

Optional (Can be Follow-up) 🟢

4. Consider TTL for cache entries
5. Consider deduplication for concurrent first requests
6. Extract repeated debug logging to helper function

---

Documentation Completeness

✅ Excellent documentation:

• CHANGELOG.md clearly explains breaking changes (none)
• CLAUDE.md provides implementation details
• README.md explains user-facing behavior
• Code comments explain complex logic

---

Final Verdict

Code Quality: 8.5/10

• Excellent design and implementation
• Test coverage needs updating (-1.5)

Architecture: 9.5/10

• Future-proof, maintainable, elegant
• Minor potential for race on first request (-0.5)

Documentation: 10/10

• Outstanding documentation at all levels

Recommendation: APPROVE with required test fixes ✅

---

Action Items

Before Merge:

1. Fix or remove outdated tests in config_test.go (lines 581-1037)
2. Fix or remove reasoning_model_test.go
3. Run go test ./... and ensure all tests pass
4. Add basic test for ShouldUseMaxCompletionTokens()

Post-Merge (Optional):

1. Monitor for false positive retries in production logs
2. Consider adding cache TTL in future version
3. Consider single-flight pattern for first-request deduplication

---

Great work on this refactor! The adaptive detection system is a significant improvement over hardcoded patterns. Once the tests are updated, this is ready to merge. 🚀