configuration reference

Configuration Reference

This comprehensive reference documents all available configuration options for the CodeChunking system, including their possible values, defaults, and environment variable mappings.

Overview

The CodeChunking system uses a hierarchical configuration approach:

CLI flags (highest priority)
Environment variables (CODECHUNK_ prefix)
Configuration files (base → environment-specific)
Default values (lowest priority)

Configuration Files Structure

configs/config.yaml - Base configuration with defaults
configs/config.dev.yaml - Development environment overrides
configs/config.prod.yaml - Production environment overrides

Core Configuration Sections

API Server Configuration (`api`)

Setting	Type	Default	Possible Values	Environment Variable
`host`	string	`"0.0.0.0"`	Valid IP address or hostname	`CODECHUNK_API_HOST`
`port`	string	`"8080"`	Valid port number (1-65535)	`CODECHUNK_API_PORT`
`read_timeout`	duration	`"10s"`	Go duration format	`CODECHUNK_API_READ_TIMEOUT`
`write_timeout`	duration	`"10s"`	Go duration format	`CODECHUNK_API_WRITE_TIMEOUT`
`enable_default_middleware`	bool	`true`	`true`, `false`	`CODECHUNK_API_ENABLE_DEFAULT_MIDDLEWARE`
`enable_cors`	bool	`true`	`true`, `false`	`CODECHUNK_API_ENABLE_CORS`
`enable_security_headers`	bool	`true`	`true`, `false`	`CODECHUNK_API_ENABLE_SECURITY_HEADERS`
`enable_logging`	bool	`true`	`true`, `false`	`CODECHUNK_API_ENABLE_LOGGING`
`enable_error_handling`	bool	`true`	`true`, `false`	`CODECHUNK_API_ENABLE_ERROR_HANDLING`

Worker Configuration (`worker`)

Setting	Type	Default	Possible Values	Environment Variable
`concurrency`	int	`5`	≥ 1 (recommended: 1-50)	`CODECHUNK_WORKER_CONCURRENCY`
`queue_group`	string	`"workers"`	Valid NATS queue group name	`CODECHUNK_WORKER_QUEUE_GROUP`
`job_timeout`	duration	`"30m"`	Go duration format (≥ 1s)	`CODECHUNK_WORKER_JOB_TIMEOUT`

Database Configuration (`database`)

Setting	Type	Default	Possible Values	Environment Variable
`host`	string	`"localhost"`	Valid hostname or IP	`CODECHUNK_DATABASE_HOST`
`port`	int	`5432`	1-65535	`CODECHUNK_DATABASE_PORT`
`user`	string	required	Valid PostgreSQL user	`CODECHUNK_DATABASE_USER`
`password`	string	required	Valid PostgreSQL password	`CODECHUNK_DATABASE_PASSWORD`
`name`	string	`"codechunking"`	Valid PostgreSQL database name	`CODECHUNK_DATABASE_NAME`
`sslmode`	string	`"disable"`	`"disable"`, `"require"`, `"prefer"`	`CODECHUNK_DATABASE_SSLMODE`
`max_connections`	int	`25`	1-1000	`CODECHUNK_DATABASE_MAX_CONNECTIONS`
`max_idle_connections`	int	`5`	1-100	`CODECHUNK_DATABASE_MAX_IDLE_CONNECTIONS`

NATS Configuration (`nats`)

Setting	Type	Default	Possible Values	Environment Variable
`url`	string	`"nats://localhost:4222"`	Valid NATS URL	`CODECHUNK_NATS_URL`
`max_reconnects`	int	`5`	0-100	`CODECHUNK_NATS_MAX_RECONNECTS`
`reconnect_wait`	duration	`"2s"`	Go duration format	`CODECHUNK_NATS_RECONNECT_WAIT`
`test_mode`	bool	`false`	`true`, `false`	N/A

Search Configuration (`search`)

Setting	Type	Default	Possible Values	Environment Variable
`iterative_scan_mode`	string	`"relaxed_order"`	`"off"`, `"strict_order"`, `"relaxed_order"`	`CODECHUNK_SEARCH_ITERATIVE_SCAN_MODE`

Scan Mode Descriptions:

📚 Related Documentation: pgvector 0.8.0 Performance Guide | Iterative Search Overview

"off": Disable iterative scanning, use exact k-NN search only
- Best for: Small datasets, when perfect accuracy is required
- Performance: Slower, but most accurate results
- Memory: Lower memory usage
"strict_order": Strict similarity ordering in iterative scans
- Best for: Medium-sized datasets where accuracy is priority
- Performance: Balanced performance with strict similarity ranking
- Behavior: Always returns results in strict similarity order
"relaxed_order": Relaxed ordering for better performance (default)
- Best for: Large datasets, production workloads
- Performance: Faster search with slightly relaxed ordering
- Behavior: May trade perfect ordering for significant speed improvements
- Recommended: Best choice for most production scenarios

Gemini Configuration (`gemini`)

Main Settings

Setting	Type	Default	Possible Values	Environment Variable
`api_key`	string	required in prod	Valid Gemini API key	`CODECHUNK_GEMINI_API_KEY`
`model`	string	`"gemini-embedding-001"`	Valid Gemini embedding model	`CODECHUNK_GEMINI_MODEL`
`max_retries`	int	`3`	0-10	`CODECHUNK_GEMINI_MAX_RETRIES`
`timeout`	duration	`"120s"`	Go duration format (≥ 1s)	`CODECHUNK_GEMINI_TIMEOUT`

Batch Settings (`gemini.batch`)

Setting	Type	Default	Possible Values	Description
`enabled`	bool	`true`	`true`, `false`	Enable Gemini Batches API
`input_dir`	string	`"/tmp/batch_embeddings/input"`	Valid directory path	Directory for batch input files
`output_dir`	string	`"/tmp/batch_embeddings/output"`	Valid directory path	Directory for batch result files
`poll_interval`	duration	`"5s"`	Go duration format	Job status polling interval
`max_wait_time`	duration	`"30m"`	Go duration format	Maximum wait for batch completion

Batch Processing Configuration (`batch_processing`)

📘 Detailed Guide: 📦 Batch Processing Guide - Complete guide to batch processing architecture, priorities, and optimization

Core Settings

Setting	Type	Default	Possible Values	Description
`enabled`	bool	`true`	`true`, `false`	Enable queue-based batch processing
`threshold_chunks`	int	`50` (dev: 10)	≥ 1	Minimum chunks to trigger batch processing
`fallback_to_sequential`	bool	`true` (dev: false)	`true`, `false`	Fall back to sequential processing on failures
`default_priority`	string	`"background"`	Priority levels	Default job priority
`use_test_embeddings`	bool	`false`	`true`, `false`	Use synthetic embeddings (development only)

Priority Levels

🎯 Priority Selection Guide: Choose the right priority based on your use case requirements for latency, throughput, and resource utilization.

🚀 `"realtime"` - Ultra-Low Latency

Purpose: Immediate processing for time-critical operations
Use Cases:
- Real-time code completion suggestions
- Live code analysis in IDE plugins
- Interactive debugging tools
- API endpoints requiring < 100ms response times
Characteristics:
- Small batch sizes (1-25 chunks)
- Shortest timeouts (2-5 minutes)
- Highest queue priority
- Optimal for: Immediate user interactions
Performance Profile: Lowest latency, lowest throughput

⚡ `"interactive"` - Low Latency

Purpose: Responsive processing for user-facing applications
Use Cases:
- Web application search queries
- Code exploration interfaces
- Documentation generation
- Pull request analysis
Characteristics:
- Moderate batch sizes (3-50 chunks)
- Short timeouts (5-10 minutes)
- High queue priority
- Optimal for: User-driven operations
Performance Profile: Low latency, good throughput

🔄 `"background"` - Balanced Performance

Purpose: Bulk processing with balanced efficiency
Use Cases:
- Repository indexing jobs
- Batch code analysis
- Scheduled processing tasks
- Historical data processing
Characteristics:
- Larger batch sizes (10-200 chunks)
- Medium timeouts (15-30 minutes)
- Standard queue priority (default)
- Optimal for: Most back-end processing
Performance Profile: Balanced latency/throughput (recommended default)

📦 `"batch"` - High Throughput

Purpose: Maximum efficiency for large-scale processing
Use Cases:
- Mass repository ingestion
- Historical migration jobs
- Overnight batch processing
- Analytics and reporting
Characteristics:
- Largest batch sizes (25-500 chunks)
- Longest timeouts (20-60 minutes)
- Lowest queue priority
- Optimal for: Non-urgent bulk workloads
Performance Profile: Highest latency, maximum throughput

Batch Size Configuration (`batch_sizes`)

Priority	Min	Max	Timeout	Optimal Use Cases	Resource Profile
`realtime`	1-5	5-25	2m-5m	IDE integrations, live search	Low memory, high CPU priority
`interactive`	3-5	10-50	5m-10m	Web apps, API endpoints	Moderate memory, high CPU priority
`background`	10-50	50-200	15m-30m	Repository indexing, jobs	Higher memory, standard CPU
`batch`	25-100	200-500	20m-60m	Mass ingestion, migrations	High memory, batch-optimized

🔧 Configuration Guidance:

Development: Use smaller batches for faster iteration
Production: Scale up batches for cost efficiency
High Traffic: Favor realtime/interactive priorities
Off-Peak: Use background/batch priorities to maximize throughput
Resource Constraints: Reduce max batch sizes to limit memory usage

Retry Configuration

Setting	Type	Default	Possible Values	Description
`max_batch_size`	int	`500` (dev: 300)	1-10000	Maximum chunks per API batch
`initial_backoff`	duration	`"30s"` (dev: "5s")	Go duration format	Initial retry backoff
`max_backoff`	duration	`"300s"` (dev: "60s")	Go duration format	Maximum retry backoff
`max_retries`	int	`3` (dev: 2)	0-10	Maximum retry attempts
`enable_batch_chunking`	bool	`true`	`true`, `false`	Enable batch chunking strategy

Submitter Configuration

Setting	Type	Default	Possible Values	Description
`submitter_poll_interval`	duration	`"5s"`	Go duration format	Batch submission check interval
`max_concurrent_submissions`	int	`1`	1-10	Maximum parallel API submissions
`submission_initial_backoff`	duration	`"1m"`	Go duration format	Rate limit backoff
`submission_max_backoff`	duration	`"30m"`	Go duration format	Maximum submission backoff
`max_submission_attempts`	int	`10`	1-100	Max submission retry attempts

Token Counting Configuration (`token_counting`)

Setting	Type	Default	Possible Values	Description
`enabled`	bool	`true`	`true`, `false`	Enable token counting
`mode`	string	`"all"`	`"all"`, `"sample"`, `"on_demand"`	Token counting strategy
`sample_percent`	int	`10`	1-100	Sampling percentage for "sample" mode
`max_tokens_per_chunk`	int	`8192`	1-32000	Maximum tokens per chunk

Token Counting Modes:

"all": Count all chunks (accurate but slower)
"sample": Sample percentage of chunks (faster, approximate)
"on_demand": Count only when requested (lazy evaluation)

Logging Configuration (`log`)

Setting	Type	Default	Possible Values	Environment Variable
`level`	string	`"info"`	`"debug"`, `"info"`, `"warn"`, `"error"`	`CODECHUNK_LOG_LEVEL`
`format`	string	`"json"` (dev: `"text"`)	`"json"`, `"text"`	`CODECHUNK_LOG_FORMAT`

Log Levels:

"debug": Detailed debugging information
"info": General information (default)
"warn": Warning messages
"error": Error messages only (production recommended)

Git Configuration

Setting	Type	Default	Possible Values	Description
`default_depth`	int	`1`	≥ 0	Git clone depth (0 = full clone)
`shallow_clone_threshold_mb`	int64	`100`	≥ 0	Repository size threshold for shallow clone
`default_timeout`	duration	`"30m"`	Go duration format	Default git operation timeout
`max_concurrent_clones`	int	`3`	1-50	Maximum concurrent git clone operations
`retry_attempts`	int	`2`	0-10	Retry attempts for git operations
`retry_backoff_duration`	duration	`"5s"`	Go duration format	Backoff between git retries
`enable_progress_tracking`	bool	`true`	`true`, `false`	Enable git operation progress tracking
`enable_performance_monitoring`	bool	`true`	`true`, `false`	Enable git performance monitoring
`auto_select_strategy`	bool	`true`	`true`, `false`	Automatically select clone strategy
`workspace_cleanup_enabled`	bool	`true`	`true`, `false`	Enable workspace cleanup
`workspace_cleanup_interval`	duration	`"24h"`	Go duration format	Cleanup interval

Security Configuration

Setting	Type	Default	Possible Values	Description
`MaxURLLength`	int	`2048`	256-65536	Maximum URL length
`MaxBodySize`	int64	`64KB`	1KB-10MB	Maximum request body size
`EnableXSSProtection`	bool	`true`	`true`, `false`	Enable XSS protection
`EnableSQLInjection`	bool	`true`	`true`, `false`	Enable SQL injection protection
`EnableControlCharCheck`	bool	`true`	`true`, `false`	Enable control character validation
`EnableUnicodeCheck`	bool	`true`	`true`, `false`	Enable Unicode validation
`EnablePathTraversal`	bool	`true`	`true`, `false`	Enable path traversal protection
`LogSecurityViolations`	bool	`true`	`true`, `false`	Log security violations
`LogLevel`	string	`"INFO"`	`"DEBUG"`, `"INFO"`, `"WARN"`, `"ERROR"`	Security log level
`EnableValidationCache`	bool	`true`	`true`, `false`	Enable validation caching
`CacheSize`	int	`1000`	100-100000	Validation cache size
`CacheTTL`	duration	`"5m"`	Go duration format	Cache TTL

Rate Limiting Configuration

Setting	Type	Default	Possible Values	Description
`RequestsPerMinute`	int	`60` (dev: 300)	1-10000	Rate limit per minute
`BurstSize`	int	`10` (dev: 50)	1-1000	Burst capacity
`WindowSize`	duration	`"1m"`	Go duration format	Rate limit window
`Enabled`	bool	`true`	`true`, `false`	Enable rate limiting

CLI Configuration

Global Flags (apply to all commands)

Flag	Type	Default	Environment Variable	Description
`--config`	string	`"./configs/config.yaml"`	N/A	Path to configuration file
`--log-level`	string	`"info"`	`CODECHUNK_LOG_LEVEL`	Log level
`--log-format`	string	`"json"`	`CODECHUNK_LOG_FORMAT`	Log format

Commands and Their Options

`api` Command

Start the HTTP API server.

Usage: codechunking api [flags]
Configuration: Inherits global flags, uses API config section

`worker` Command

Start the background worker service.

Usage: codechunking worker [flags]
Configuration: Inherits global flags, uses worker config section
Requirements: CODECHUNK_GEMINI_API_KEY environment variable required

`chunk` Command

Parse and chunk a single source file.

Usage: codechunking chunk --file <path> [flags]
Flags:
- --file string: Source file path (required)
- --lang string: Language hint (default: "go")
- --out string: Output JSON file path (optional)

`migrate` Command

Database migration management.

Usage: codechunking migrate [command]
Subcommands:
- up: Apply pending migrations
- down [steps]: Rollback migrations (default: 1 step)
- create <name>: Create new migration
- version: Show current migration version
- force <version>: Force migration to version

`version` Command

Display version information.

Usage: codechunking version [flags]

Environment Variables Summary

Required Environment Variables

CODECHUNK_GEMINI_API_KEY: Required for production embedding generation
CODECHUNK_DATABASE_USER: Database username (production)
CODECHUNK_DATABASE_PASSWORD: Database password (production)

Optional Environment Variables

All configuration values can be overridden using environment variables with the CODECHUNK_ prefix. Nested configurations use underscores:

# Examples
CODECHUNK_API_PORT=9090
CODECHUNK_DATABASE_HOST=db.example.com
CODECHUNK_WORKER_CONCURRENCY=10
CODECHUNK_GEMINI_MAX_RETRIES=5
CODECHUNK_LOG_LEVEL=debug
CODECHUNK_BATCH_PROCESSING_THRESHOLD_CHUNKS=25

Duration Format

All duration values use Go's time.ParseDuration format:

ns, us, µs, ms: Nanoseconds/Microseconds/Milliseconds
s: Seconds
m: Minutes
h: Hours

Examples: 30s, 5m, 2h, 1h30m, 45ms

Configuration Validation

The system validates configuration values on startup:

Required Fields

database.user and database.name
gemini.api_key in production environments

Numeric Constraints

worker.concurrency ≥ 1
database.port: 1-65535
Various maximum/minimum values as documented above

File System Requirements

Gemini batch directories must exist and be writable
Configuration files must be readable

Best Practices

Development

Use config.dev.yaml for development overrides
Set use_test_embeddings: true to avoid API costs during testing
Enable debug logging: log.level: debug and log.format: text

Production

Use config.prod.yaml for production overrides
Set all required environment variables
Use SSL for database connections: database.sslmode: require
Increase connection limits for performance
Use JSON logging: log.format: json
Set appropriate log level: log.level: error

Security

Never commit API keys or passwords to configuration files
Use environment variables for all sensitive data
Enable security features in production
Set appropriate rate limits
Monitor security logs

Performance

Tune database.max_connections based on your database capacity
Adjust worker.concurrency based on available CPU/memory
Set appropriate batch sizes for your workload
Consider enabling parallel batch submissions for high throughput

Example Configurations

Minimal Development Config

database:
  user: dev
  password: dev

log:
  level: debug
  format: text

batch_processing:
  use_test_embeddings: true
  threshold_chunks: 10

Production Ready Config

api:
  read_timeout: 30s
  write_timeout: 30s

worker:
  concurrency: 20
  job_timeout: 60m

database:
  sslmode: require
  max_connections: 100
  max_idle_connections: 20

nats:
  max_reconnects: 10
  reconnect_wait: 5s

gemini:
  max_retries: 5
  timeout: 60s

log:
  level: error
  format: json

High Throughput Config

api:
  read_timeout: 60s
  write_timeout: 60s

worker:
  concurrency: 50
  job_timeout: 120m

database:
  max_connections: 200
  max_idle_connections: 50

batch_processing:
  max_batch_size: 1000
  max_concurrent_submissions: 5
  threshold_chunks: 25

Refer to this guide for all configuration needs and ensure proper validation for production deployments.

Home

Configuration

[📖 Configuration Reference](configuration reference) - Complete reference guide
Configuration
API Configuration
Database Configuration
Gemini Configuration
Git Configuration
Logging Configuration
Middleware Configuration
NATS Configuration
Worker Configuration

Usage Guides

Performance

- vector storage performance

configuration reference

Configuration Reference

Overview

Configuration Files Structure

Core Configuration Sections

API Server Configuration (api)

Worker Configuration (worker)

Database Configuration (database)

NATS Configuration (nats)

Search Configuration (search)

Gemini Configuration (gemini)

Main Settings

Batch Settings (gemini.batch)

Batch Processing Configuration (batch_processing)

Core Settings

Priority Levels

🚀 "realtime" - Ultra-Low Latency

⚡ "interactive" - Low Latency

🔄 "background" - Balanced Performance

📦 "batch" - High Throughput

Batch Size Configuration (batch_sizes)

Retry Configuration

Submitter Configuration

Token Counting Configuration (token_counting)

Logging Configuration (log)

Git Configuration

Security Configuration

Rate Limiting Configuration

CLI Configuration

Global Flags (apply to all commands)

Commands and Their Options

api Command

worker Command

chunk Command

migrate Command

version Command

Environment Variables Summary

Required Environment Variables

Optional Environment Variables

Duration Format

Configuration Validation

Required Fields

Numeric Constraints

File System Requirements

Best Practices

Development

Production

Security

Performance

Example Configurations

Minimal Development Config

Production Ready Config

High Throughput Config

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

API Server Configuration (`api`)

Worker Configuration (`worker`)

Database Configuration (`database`)

NATS Configuration (`nats`)

Search Configuration (`search`)

Gemini Configuration (`gemini`)

Batch Settings (`gemini.batch`)

Batch Processing Configuration (`batch_processing`)

🚀 `"realtime"` - Ultra-Low Latency

⚡ `"interactive"` - Low Latency

🔄 `"background"` - Balanced Performance

📦 `"batch"` - High Throughput

Batch Size Configuration (`batch_sizes`)

Token Counting Configuration (`token_counting`)

Logging Configuration (`log`)

`api` Command

`worker` Command

`chunk` Command

`migrate` Command

`version` Command