[codex] Improve Nettacker HTTP detection accuracy and explicit port handling

[kerberosmansour]

Summary

This PR improves Nettacker's HTTP scanning accuracy for several cases that showed up during a lab assessment against modern Node/SPA applications (bkimminich/juice-shop and nirocr/nodegoat). The main theme is reducing false positives caused by status-code-only matching while also fixing silent false negatives in header-based vulnerability modules.

The changes are intentionally split between two layers:

• small HTTP engine primitives that module YAML can reuse (content_length, content_sha1, missing-header matching, and catch-all baseline comparison)
• targeted module updates for directory discovery, security headers, OPTIONS/CORS handling, and WAF detection

This should preserve the current YAML-driven module model while giving module authors safer matching tools for modern web applications.

Problems addressed

1. URL probe modules false-positive on SPA catch-all routes

dir_scan, admin_scan, and pma_scan previously treated 200, 401, or 403 as enough evidence that a path existed. That produces very noisy output against SPAs and catch-all routes that return the same index.html for every unknown path.

For example, Angular/React/Vue/Svelte applications commonly route unknown paths back to the frontend shell with 200 OK. In that situation, status-code-only matching cannot distinguish a real /admin page from /nettacker-random-nonsense.

This PR adds an opt-in baseline_response condition for HTTP modules. When a module uses it, Nettacker performs a low-impact request to a random sibling path and only reports a probe hit if the probe response differs from the random baseline by status code, body length beyond tolerance, or body SHA-1.

Applied to:

• dir_scan
• admin_scan
• pma_scan

2. Header absence was not matchable by YAML header conditions

Several header vulnerability modules tried to detect unsafe header values, but a completely missing header could fail to match and therefore produce no finding. Missing security headers are often the common case, so this silently under-reports real issues.

The HTTP condition matcher now evaluates missing response headers as an empty string ("") instead of a non-string falsey value. That lets existing regex-based YAML conditions explicitly match absence with ^$.

Updated modules:

• clickjacking_vuln
• content_type_options_vuln
• x_xss_protection_vuln

content_security_policy_vuln already had an absence-compatible regex, but it benefits from the engine fix because missing headers can now match consistently.

3. OPTIONS method detection missed CORS preflight method lists

http_options_enabled_vuln only looked at the legacy Allow header. Modern web frameworks frequently expose allowed methods through Access-Control-Allow-Methods on OPTIONS preflight responses instead.

This PR makes either header sufficient evidence for the module:

• Allow
• Access-Control-Allow-Methods

4. WAF detection produced false positives from status-code deltas

waf_scan had a generic fallback heuristic: compare the baseline request status code to an XSS-payload request status code, and report "WAF detected" if they differ.

That is too weak as a WAF signal. Normal application routing, caching, redirects, frontend fallbacks, and framework behavior can all produce status differences without a WAF or CDN in front of the app.

This PR removes the status-delta-only heuristic and leaves the existing positive-signature checks in place. WAF findings now require a vendor/header/body/status signature from the existing iterative_response_match database rather than a generic status-code difference.

It also removes the previous typo-bearing fallback log (differenet).

5. Explicit URL ports should not be blocked by default service discovery

When a user provides a URL with an explicit scheme and port, such as http://jshop:3000, or provides -g 3000, Nettacker already has enough user intent to scan that port. The prior flow could run service discovery first, fail to classify the service, and stop with "no live service found" even though the requested HTTP service was reachable.

This PR updates target expansion to preserve explicit URL scheme/port into the parsed runtime options and skip the service-discovery gate for explicit user-provided ports. This means modules such as http_status_scan can run against common dev/test ports like 3000, 4000, 8000, and 8080 without requiring -d as a workaround.

It also fixes the English message:

• before: no any live service found to scan.
• after: no live service found to scan.

6. Add a focused CORS misconfiguration module

This PR adds cors_misconfiguration_vuln for common unsafe CORS responses:

• reflected or wildcard/null Access-Control-Allow-Origin combined with credentials
• reflected or wildcard/null origins combined with broad methods such as PUT, PATCH, or DELETE

This complements the existing http_cors_vuln module by adding checks for wildcard/reflected-origin and broad-method combinations observed in modern APIs.

Implementation details

HTTP response fingerprints

nettacker/core/lib/http.py now records two extra response fields for HTTP responses:

• content_length
• content_sha1

These are exposed as normal YAML matchable response conditions. The YAML schema test was extended so module definitions can use those fields.

Missing header matching

Missing headers are now passed to regex matching as "". This keeps the matching model simple and makes absence explicit in YAML:

headers:
  X-Content-Type-Options:
    regex: ^$|^((?!nosniff).)+$
    reverse: false

Baseline comparison condition

A module can now opt into catch-all filtering with:

baseline_response:
  max_content_length_delta: 64

For such modules, Nettacker requests a random sibling path and compares the probe to that baseline. The condition passes only when at least one of these differs:

• status code
• content length beyond max_content_length_delta
• SHA-1 of the response body

This keeps the feature opt-in so it only affects modules that are vulnerable to catch-all false positives.

Explicit port handling

Nettacker.expand_targets() now uses urllib.parse.urlsplit() for URL targets. It extracts:

• normalized hostname for scan target grouping
• explicit URL port into arguments.ports when -g/--ports was not separately supplied
• explicit URL scheme into arguments.schema when --schema was not separately supplied
• base path into url_base_path

If arguments.ports is present, the default service-discovery pre-pass is skipped because the user has already selected the port set to scan.

Files changed

Core behavior:

• nettacker/core/lib/http.py
• nettacker/core/app.py
• nettacker/locale/en.yaml

Module definitions:

• nettacker/modules/scan/dir.yaml
• nettacker/modules/scan/admin.yaml
• nettacker/modules/scan/pma.yaml
• nettacker/modules/scan/waf.yaml
• nettacker/modules/vuln/clickjacking.yaml
• nettacker/modules/vuln/content_type_options.yaml
• nettacker/modules/vuln/http_options_enabled.yaml
• nettacker/modules/vuln/x_xss_protection.yaml
• nettacker/modules/vuln/cors_misconfiguration.yaml

Tests:

• tests/core/lib/test_http.py
• tests/core/test_app_targets.py
• tests/test_yaml_schema_and_regex.py

Compatibility notes

• The baseline comparison is opt-in and only applied to modules that add baseline_response.
• Existing header regex behavior still works; missing headers are simply represented as an empty string during matching.
• The WAF module still contains the existing vendor-specific signatures. This PR removes only the generic status-code-delta fallback.
• Explicit -g/--ports and explicit URL ports now bypass the service-discovery gate for downstream modules. Default scans without explicit ports still use the existing service-discovery pre-pass.

Validation

I ran the focused regression and schema checks in the repository virtualenv:

.venv/bin/python -m pytest -o addopts='' \
  tests/core/lib/test_http.py \
  tests/core/test_app_targets.py \
  tests/test_yaml_schema_and_regex.py -q

Result:

125 passed, 7 skipped, 2 warnings

I also ran Ruff on the changed Python files and new tests:

.venv/bin/python -m ruff check \
  nettacker/core/lib/http.py \
  nettacker/core/app.py \
  tests/core/lib/test_http.py \
  tests/core/test_app_targets.py

Result:

All checks passed!

And checked whitespace:

git diff --check

Result: clean.

Reviewer notes

The most important design question is whether baseline_response belongs in the generic HTTP matcher as implemented here, or whether maintainers would prefer a more explicit condition name or a module-level option. I kept it as an opt-in condition because it fits the existing YAML condition model and avoids changing unaffected modules.

A second review point is CORS severity. The new module currently treats credentialed cross-origin access and broad unsafe methods as reportable. If the project prefers more granular severities for CORS combinations, this module can be split into multiple YAML steps or separate modules.

Finally, the WAF change intentionally favors false-positive reduction over broad heuristic detection. A status-code delta alone is not strong enough evidence for "WAF detected" on modern apps; vendor/header/body signatures remain the safer path.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ce89012f-f114-45fb-a60c-1e347e8c9354

📥 Commits

Reviewing files that changed from the base of the PR and between a2157ee and 851451d.

📒 Files selected for processing (45)

• .gitignore
• README.md
• docs/Home.md
• docs/Modules.md
• docs/RUNBOOK-pqc-compliance-scanner.md
• docs/Usage.md
• docs/slo/completion/pqc-scanner-m1.md
• docs/slo/completion/pqc-scanner-m2.md
• docs/slo/completion/pqc-scanner-m3.md
• docs/slo/critique/pqc-compliance-scanner.md
• docs/slo/design/pqc-compliance-scanner-architecture.md
• docs/slo/design/pqc-compliance-scanner-interfaces.md
• docs/slo/design/pqc-compliance-scanner-overview.md
• docs/slo/design/pqc-compliance-scanner-security.md
• docs/slo/design/pqc-compliance-scanner-stack-decision.md
• docs/slo/design/pqc-compliance-scanner-threat-model.md
• docs/slo/idea/pqc-compliance-scanner.md
• docs/slo/lessons/pqc-scanner-m1.md
• docs/slo/lessons/pqc-scanner-m2.md
• docs/slo/lessons/pqc-scanner-m3.md
• docs/slo/research/pqc-compliance-scanner/dossier.md
• docs/slo/research/pqc-compliance-scanner/sources.md
• docs/slo/research/pqc-compliance-scanner/synthesis.md
• nettacker/core/app.py
• nettacker/core/lib/http.py
• nettacker/core/lib/pqc.py
• nettacker/core/module.py
• nettacker/locale/en.yaml
• nettacker/modules/scan/admin.yaml
• nettacker/modules/scan/dir.yaml
• nettacker/modules/scan/pma.yaml
• nettacker/modules/scan/pqc.yaml
• nettacker/modules/scan/waf.yaml
• nettacker/modules/vuln/clickjacking.yaml
• nettacker/modules/vuln/content_type_options.yaml
• nettacker/modules/vuln/cors_misconfiguration.yaml
• nettacker/modules/vuln/http_options_enabled.yaml
• nettacker/modules/vuln/x_xss_protection.yaml
• tests/core/lib/test_http.py
• tests/core/lib/test_pqc.py
• tests/core/test_app_targets.py
• tests/core/test_module_pqc.py
• tests/e2e/__init__.py
• tests/e2e/test_pqc_scan_smoke.py
• tests/test_yaml_schema_and_regex.py

---

Summary by CodeRabbit

Release Notes

• New Features

  • Added Post-Quantum Cryptography (PQC) compliance scanner module for auditing TLS and SSH endpoints, providing per-host readiness verdicts and advertised PQ algorithm enumeration
  • Added HTTP baseline response matching to reduce false positives in directory scanning

• Improvements

  • Enhanced HTTP header validation logic for empty header values
  • Improved URL target parsing for explicit port and scheme handling

• Documentation

  • Comprehensive documentation added for PQC scanning module, including CLI usage examples and compliance framework mapping

Walkthrough

This PR introduces a complete post-quantum cryptography (PQC) compliance scanner module for OWASP Nettacker. It adds a new pqc_scan module that audits TLS 1.3 and SSH endpoints for PQC algorithm support via passive SSH KEXINIT enumeration and active TLS ClientHello probing, alongside supporting infrastructure for HTTP baseline response matching. The change spans comprehensive design documentation across five design artifacts, core library implementation, YAML module configuration, extensive testing, and user-facing documentation.

Changes

PQC Compliance Scanner Implementation

Layer / File(s)	Summary
Research, Design Contracts & Architecture docs/slo/{idea,research,design}/*	Research dossier with market/tech/legal context; synthesis of implementation direction; five core design documents establishing feature scope, API contracts, architecture, security constraints, threat model, and stack decisions (Python stdlib-based).
PQC Library Implementation nettacker/core/lib/pqc.py	1041 LOC: SSH passive KEXINIT probing, TLS 1.3 active ClientHello construction (RFC-8446 compliant, empty key_share for HRR signaling), total-function response parsing, verdict classification (pqc_ready/hybrid_only/classical_only/unknown), and operator opt-out via --modules-extra-args pqc_no_active_probe=true.
HTTP Baseline Response Matching nettacker/core/lib/http.py	Content fingerprinting (length + SHA1), randomized baseline URL generation, differential response matching, and follow-up baseline request flow in HttpEngine.run for false-positive reduction.
PQC Module YAML Configuration nettacker/modules/scan/pqc.yaml, nettacker/core/module.py	New pqc_scan module with ssh_pqc_scan (ports 22/2222) and tls_pqc_scan (ports 80/443/etc) steps, 5-second timeout, scan_succeeded pass condition; registration in ignored_core_modules.
App URL Parsing nettacker/core/app.py	Enhanced expand_targets() using urlsplit() to extract hostname/scheme/port/path from explicit URLs; conditionally sets arguments.ports and arguments.schema; skips port-scan when URL ports are explicit.
HTTP Module Updates nettacker/modules/scan/{admin,dir,pma}.yaml	Admin, directory, and PMA HTTP scanning modules updated with baseline_response: {max_content_length_delta: 64} for baseline matching support.
Unit & Integration Tests tests/core/lib/{test_http.py,test_pqc.py}, tests/core/test_{app_targets,module_pqc}.py	Comprehensive coverage: SSH/TLS wire format validation, verdict classification, exception safety, resource invariants (FD leaks, connection caps), fake SSH/TLS servers for e2e validation, opt-out mechanism, URL parsing, and module wiring.
End-to-End Smoke Tests tests/e2e/test_pqc_scan_smoke.py	Network-dependent smoke tests against public PQC-aware endpoints (GitHub SSH, Cloudflare TLS) with reachability checks and conditional skip logic.
User-Facing Documentation README.md, docs/{Home,Modules,Usage}.md	Feature overview, quick-start CLI/Docker examples, verdict semantics, CI gating patterns, compliance mapping (NIST FIPS 203, OMB M-23-02, CNSA 2.0), and opt-out flag documentation.
Milestone & Process Records docs/slo/{completion,lessons}/*	M1/M2/M3 completion records with scope, file changes, test coverage, and invariants; lessons learned capturing design decisions, mistakes, and forward-looking rules; four-persona adversarial critique with findings and recommendations.
Runbook & Execution Template docs/RUNBOOK-pqc-compliance-scanner.md	AI-first development runbook specifying feature purpose, architecture, stable interfaces, milestone contracts with BDD scenarios, bounded-resource rules, and global execution discipline.
Configuration & Cleanup .gitignore, nettacker/locale/en.yaml, nettacker/modules/vuln/*, tests/test_yaml_schema_and_regex.py	Ignore .venv/ and evidence/; locale string adjustments; vulnerability module regex updates for empty header values; HTTP condition schema extended for baseline/content fields.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• OWASP/Nettacker#1113: Shares skip_service_discovery behavior; this PR implements core URL parsing and skip logic in nettacker/core/app.py while the related PR handles UI/API surface submission of the flag.

Suggested labels

new module, enhancement

Suggested reviewers

• arkid15r
• securestep9

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

⚠️ This pull request might be slop. It has been flagged by CodeRabbit slop detection and should be reviewed carefully.

[github-actions]

PR validation failed: No linked issue and no valid closing issue reference in PR description

[kerberosmansour]

Superseded by #1545. This first draft was opened before I rebased the branch onto OWASP/Nettacker:master, so it briefly included fork-local history that is not part of the intended review. The clean maintainer-facing PR is #1545.