Feat 6567 memory leak

[Mzack9999]

Proposed changes

Closes #6567

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Summary by CodeRabbit

• Performance

  • Optimized memory efficiency when processing large HTTP responses through intelligent data handling
  • Enhanced response processing performance for improved handling in high-volume scenarios

• Refactor

  • Refined HTTP response protocol architecture with improved internal tracking and resource management capabilities

[coderabbitai]

Walkthrough

Adds memory optimization infrastructure to HTTP protocol handling, including verbose target tracking, hash-based body representation for large responses exceeding thresholds, and dynamic body read size limits. These changes prevent memory exhaustion when processing large HTTP responses during template scanning.

Changes

Cohort / File(s)	Summary
Body size constants pkg/protocols/http/body_heuristic.go	Introduces MinimalBodySize constant (4 KB) for defining minimal body size when body is not needed.
Verbose target tracking pkg/protocols/http/http.go	Adds concurrent-safe verbose target registry with public functions MarkVerboseTarget(), IsVerboseTarget(), and CheckAndMarkVerbose() to track and query targets requiring full response bodies. Exports VerboseTargetThreshold and VerboseTargetMaxSize configuration constants.
Response DSL mapping pkg/protocols/http/operators.go	Modifies responseToDSLMap() to accept interface{} types for response and body parameters. Changes string conversion and content-length calculation to use ToString() utility, enabling hash-marker support for large bodies. Adjusts status matching and body/headers handling logic.
Request execution & response handling pkg/protocols/http/request.go	Introduces HashThreshold constant (5 MB) for hash-marker representation. Integrates verbose target marking before/after response handling, applies conditional body read size limits for verbose targets, and passes separate response and body values to DSL builder with explicit hash-marker handling for memory efficiency.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Request as request.go
    participant VerboseReg as Verbose Registry
    participant HTTP as http.go
    participant Operators as operators.go

    Client->>Request: Execute HTTP request
    Request->>VerboseReg: MarkVerboseTarget(url)
    Request->>Request: Read response with conditional size limit
    Request->>HTTP: CheckAndMarkVerbose(response)
    HTTP->>HTTP: Check Content-Length vs Threshold
    alt Content-Length exceeds threshold
        HTTP->>VerboseReg: MarkVerboseTarget(url)
    end
    Request->>Operators: Build DSL with response & body
    Operators->>HTTP: IsVerboseTarget(url)?
    alt Body size > HashThreshold
        Operators->>Operators: Use hash marker instead of full body
    else Body size ≤ HashThreshold
        Operators->>Operators: Use full body string
    end
    Operators->>Client: Return optimized DSL map

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

• Concurrent registry implementation in http.go requires verification of thread-safety patterns and synchronized map usage
• Signature changes to responseToDSLMap() in operators.go need careful review of all call sites and type conversions
• Interaction between verbose tracking and body size limits in request.go — ensure precedence order (MaxSize → ResponseReadSize) is correctly implemented across conditional branches
• Hash-marker logic in operators.go and request.go — verify that HashThreshold is consistently applied and hash-marked values don't break downstream template processing
• Memory efficiency trade-offs — confirm that hash markers are generated/resolved correctly and don't introduce new memory overhead

Poem

🐰 A rabbit's ode to memory peace:
Large responses were stealing our RAM,
Now verbose targets mark where we stand,
Hash markers shrink bodies so grand,
Five megabytes—our new gentle dam,
Memory leaks fade, scan complete, hooray! 🎉

Pre-merge checks and finishing touches

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The pull request title 'Feat 6567 memory leak' is vague and uses generic terms that don't clearly convey what specific aspect of the memory leak is being addressed. While it references the issue number and topic, it doesn't provide meaningful information about the actual changes—such as whether the fix optimizes memory usage, prevents unbounded growth, or addresses a specific component causing the leak.	Consider revising the title to be more specific about the memory optimization approach, such as 'Use hash markers for large HTTP response bodies to reduce memory usage' or 'Implement verbose-target tracking to optimize memory for large responses.' This would make the change immediately clear to reviewers scanning the commit history.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check	✅ Passed	The pull request addresses the core requirements from issue #6567 by implementing memory optimizations in the HTTP protocol handling. The changes introduce hash marker thresholds, a verbose-target tracking mechanism, and per-request max body size limits to prevent unbounded memory growth during critical-severity scans. The implementation includes new constants (MinimalBodySize, HashThreshold, VerboseTargetThreshold, VerboseTargetMaxSize) and functions to intelligently mark and handle verbose targets, which directly mitigates the 100% CPU/RAM spike described in the issue.
Out of Scope Changes check	✅ Passed	All changes in the pull request are directly scoped to addressing the memory leak in issue #6567. The modifications focus on HTTP response handling, body size management, and memory optimization through hash markers and verbose-target tracking. No unrelated changes or scope creep is evident—the updates to body_heuristic.go, http.go, operators.go, and request.go are all necessary components of the memory optimization strategy.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat-6567-memory-leak

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3a44eaf and fc602c4.

📒 Files selected for processing (4)

• pkg/protocols/http/body_heuristic.go (1 hunks)
• pkg/protocols/http/http.go (3 hunks)
• pkg/protocols/http/operators.go (4 hunks)
• pkg/protocols/http/request.go (4 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt
Run static analysis with go vet

Files:

• pkg/protocols/http/body_heuristic.go
• pkg/protocols/http/request.go
• pkg/protocols/http/http.go
• pkg/protocols/http/operators.go

pkg/protocols/**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

Each protocol implementation must provide a Request interface with methods Compile(), ExecuteWithResults(), Match(), and Extract()

Files:

• pkg/protocols/http/body_heuristic.go
• pkg/protocols/http/request.go
• pkg/protocols/http/http.go
• pkg/protocols/http/operators.go

🧠 Learnings (4)

📓 Common learnings

Learnt from: Deamhan
Repo: projectdiscovery/nuclei PR: 0
File: :0-0
Timestamp: 2025-11-01T09:00:13.291Z
Learning: In nuclei's headless processing (pkg/protocols/headless), memory leaks can occur when the Run method fails after browser page initialization (e.g., navigation errors on heavy SPA sites). These manifest as "zombie" pages that remain open and consume RAM, potentially leading to OOM conditions.

📚 Learning: 2025-07-16T21:27:14.937Z

Learnt from: hdm
Repo: projectdiscovery/nuclei PR: 6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.

Applied to files:

• pkg/protocols/http/request.go

📚 Learning: 2025-06-30T16:34:42.125Z

Learnt from: dwisiswant0
Repo: projectdiscovery/nuclei PR: 6290
File: pkg/protocols/http/build_request.go:457-464
Timestamp: 2025-06-30T16:34:42.125Z
Learning: In the projectdiscovery/retryablehttp-go package, the Request struct embeds URL fields directly, making req.Scheme, req.Host, and other URL fields accessible directly on the Request object instead of requiring req.URL.Scheme, req.URL.Host, etc.

Applied to files:

• pkg/protocols/http/http.go

📚 Learning: 2025-09-05T14:23:55.257Z

Learnt from: CR
Repo: projectdiscovery/nuclei PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-09-05T14:23:55.257Z
Learning: Applies to pkg/protocols/**/*.go : Each protocol implementation must provide a Request interface with methods Compile(), ExecuteWithResults(), Match(), and Extract()

Applied to files:

• pkg/protocols/http/http.go
• pkg/protocols/http/operators.go

🧬 Code graph analysis (3)

pkg/protocols/http/request.go (1)

pkg/protocols/http/http.go (5)

• CheckAndMarkVerbose (570-583)
• IsVerboseTarget (560-566)
• VerboseTargetMaxSize (545-545)
• VerboseTargetThreshold (544-544)
• MarkVerboseTarget (555-557)

pkg/protocols/http/http.go (1)

pkg/protocols/http/httpclientpool/clientpool.go (1)

• Get (171-181)

pkg/protocols/http/operators.go (3)

pkg/protocols/http/http.go (1)

• Request (38-243)

pkg/output/output.go (1)

• InternalEvent (93-93)

pkg/protocols/utils/utils.go (1)

• CalculateContentLength (44-49)

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}