Migrate integration and provider HTTP types (PR 13)

[prk-Jr]

Summary

• Replace the remaining Fastly request and response boundaries in the integration and auction provider layers so PR13 runs on platform-agnostic HTTP types.
• Move provider dispatch onto RuntimeServices HTTP client primitives with async request_bids, PlatformPendingRequest, and PlatformResponse while preserving orchestrator deadline handling.
• Finish the review hardening for Testlight response normalization and convert the migrated integration tests to HTTP-native fixtures.

Changes

File	Change
crates/trusted-server-adapter-fastly/src/main.rs	Route integration proxy requests through the HTTP-native registry boundary and pass RuntimeServices into the PR13 entrypoint.
crates/trusted-server-core/src/auction/README.md	Update provider guidance and examples to PlatformPendingRequest, PlatformResponse, and the platform HTTP client flow.
crates/trusted-server-core/src/auction/endpoints.rs	Thread RuntimeServices into the auction entrypoint context.
crates/trusted-server-core/src/auction/orchestrator.rs	Await async provider launches, operate on PlatformPendingRequest/select, and update the remaining PR13 migration comments.
crates/trusted-server-core/src/auction/provider.rs	Convert AuctionProvider to async request_bids with PlatformPendingRequest/PlatformResponse and refresh the trait docs.
crates/trusted-server-core/src/auction/types.rs	Add RuntimeServices to AuctionContext.
crates/trusted-server-core/src/integrations/adserver_mock.rs	Migrate the mock provider to platform HTTP pending/response types.
crates/trusted-server-core/src/integrations/aps.rs	Migrate APS provider request dispatch and response parsing to platform HTTP types.
crates/trusted-server-core/src/integrations/datadome.rs	Convert DataDome proxying to HTTP-native request/response handling via the platform client.
crates/trusted-server-core/src/integrations/didomi.rs	Convert Didomi proxying to HTTP-native request/response handling via the platform client.
crates/trusted-server-core/src/integrations/google_tag_manager.rs	Remove Fastly shims, keep bounded POST handling on HTTP bodies, and convert GTM tests to HTTP-native fixtures.
crates/trusted-server-core/src/integrations/gpt.rs	Remove Fastly request/response compat from GPT asset proxying and convert GPT tests to HTTP-native fixtures.
crates/trusted-server-core/src/integrations/lockr.rs	Convert Lockr SDK/API proxy handling to HTTP-native request/response types.
crates/trusted-server-core/src/integrations/permutive.rs	Convert Permutive SDK/API proxy handling to HTTP-native request/response types.
crates/trusted-server-core/src/integrations/prebid.rs	Move Prebid provider dispatch to async platform HTTP requests and parse PlatformResponse results.
crates/trusted-server-core/src/integrations/registry.rs	Migrate IntegrationProxy/routing types to http types and thread RuntimeServices through proxy dispatch.
crates/trusted-server-core/src/integrations/testlight.rs	Convert Testlight to HTTP-native request/response handling, add the stale Content-Length regression test, and drop stale length headers when rebuilding JSON responses.

Closes

Closes #494

Test plan

• cargo test --workspace
• cargo clippy --workspace --all-targets --all-features -- -D warnings
• cargo fmt --all -- --check
• JS tests: cd crates/js/lib && npx vitest run
• JS format: cd crates/js/lib && npm run format
• Docs format: cd docs && npm run format
• WASM build: cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1
• Manual testing via fastly compute serve
• Other: cargo doc --no-deps --all-features (passes with pre-existing unrelated rustdoc warnings outside this PR's scope)

Checklist

• Changes follow CLAUDE.md conventions
• No unwrap() in production code — use expect("should ...")
• Uses repo-standard logging macros, not println!
• New code has tests
• No secrets or credentials committed

[ChristianPavilonis]

PR Review — Migrate integration and provider HTTP types (PR 13)

1 blocking finding, 5 non-blocking suggestions.

Blocking

🔧 Missing Content-Type forwarding for Didomi POST/PUT — see inline comment.

Non-blocking (inline)

• 🤔 Prebid still imports from fastly::http — last remaining use fastly in the integration layer
• ♻️ Duplicated body collection helpers (collect_response_body / collect_body_bytes)
• 🌱 Unbounded response body collection (matches pre-migration, but could use a size cap)
• ⛏ Dead "X-" uppercase branch in custom header copy (http crate lowercases names)
• ⛏ Duplicated backend_name_for_url helper across 4 integrations

[ChristianPavilonis]

ignore this

Replacing with full review

[ChristianPavilonis]

Summary

Solid migration of the integration and auction provider layers from Fastly-specific HTTP types to platform-agnostic http crate types. No blocking issues found. CI is passing.

Non-blocking

♻️ refactor

• Duplicated body_as_reader helper: proxy.rs:24 and publisher.rs:23 define identical body_as_reader functions. Consider extracting into a shared location (e.g., an into_reader() method on EdgeBody, or a helper in http_util.rs).

📝 note

• PR description file table incomplete: proxy.rs and publisher.rs have changes in this diff (the body_as_reader extraction and test type alias cleanups) but are not listed in the PR body's change table.

📌 out of scope

• Orchestrator tests disabled pending mock PlatformHttpClient: Several provider integration tests and timeout enforcement paths in orchestrator.rs:751-765 are disabled pending a mock for PlatformHttpClient::select(). This is acknowledged in the PR, but the scope of untested paths has grown — a follow-up to add thin mock support for select() would significantly improve coverage of the deadline enforcement logic.

[aram356]

Summary

The refactor is clean and the ensure_integration_backend / collect_body helpers extracted in the fix commits are a genuine improvement. The main risks are latent panics and missing POST size bounds that were introduced in the base migration commit. None are new in the fix commits, but this PR is the natural owner of the HTTP-type boundary, so they should land before merge.

Blocking

🔧 wrench

• EdgeBody::into_bytes() panics on streaming bodies at auction/endpoints.rs:42 — the inbound auction POST body is user-controlled. Body::into_bytes() panics on Body::Stream(_) (unit test into_bytes_panics_for_stream asserts this). Today Fastly's adapter happens to return Body::Once, so nothing panics — but that invariant is undocumented and the /auction handler is the first publicly-reachable crash if it ever changes. Fix: switch to Body::into_bytes_bounded(max).await with a documented cap (e.g. 256KB).
• EdgeBody::into_bytes() panics across every migrated provider/proxy — same root cause, see inline comments on prebid.rs:1157, permutive.rs:151, lockr.rs:149, datadome.rs:301, aps.rs:554, adserver_mock.rs:373. Make parse_response async and read bodies with into_bytes_bounded.
• Unbounded inbound POST body forwarding on the datadome / didomi / permutive / lockr proxies — only GTM enforces a cap. Promote the GTM size-bounded reader into integrations/mod.rs as collect_body_bounded and apply it uniformly. Inline comments on datadome.rs:367, didomi.rs:245, permutive.rs:211, lockr.rs:203.
• Lockr .expect() on user-controlled Origin header — reachable worker panic at lockr.rs:266.
• GTM stream-read errors mis-classified as PayloadTooLarge — returns 413 for transport errors at google_tag_manager.rs:313. Add a StreamRead variant and map to 502.

Non-blocking

🤔 thinking

• collect_body has no size cap (integrations/mod.rs:101) — silently drains whole bodies to RAM. Same concern applies to the GTM response-rewrite path at google_tag_manager.rs:486.
• Orchestrator select_result.ready error path drops provider identity in run_providers_parallel — when ready is Err, the provider identity is lost so no AuctionResponse::error(...) is pushed and the failing backend vanishes silently from the result set. Track a (backend_name, provider_index) pair alongside fastly_pending so failures are attributable.
• Deadline enforcement relies on every provider honoring backend_name(effective_timeout) — Phase 1 computes remaining_ms.min(provider.timeout_ms()) correctly, but nothing guarantees each provider actually threads that timeout through to the backend config. Not a new regression, but the async migration makes it more load-bearing. Add a unit test that asserts the select loop returns before deadline + epsilon.

♻️ refactor

• aps.rs / adserver_mock.rs still build BackendConfig inline — extend the new ensure_integration_backend helper to cover providers too.

🌱 seedling

• Make parse_response async in the provider trait (auction/provider.rs:48) — needed anyway once the into_bytes_bounded fix lands; zero cost under #[async_trait(?Send)].

📝 note

• Testlight stale Content-Length test is narrow (testlight.rs:425) — good unit-level coverage, but the two handle() rebuild call sites would benefit from an end-to-end assertion.

👍 praise

• Clean helper extraction in integrations/mod.rs — real dedup across six integrations.
• Dropping the dead || name_str.starts_with("X-") branch in lockr.rs / permutive.rs — HeaderName::as_str() is always lowercase, so it was unreachable.
• Adding CONTENT_TYPE to didomi's forwarded headers — POST bodies can now be interpreted by upstream.

CI Status

• browser integration tests: PASS
• integration tests: PASS
• prepare integration artifacts: PASS

[aram356]

Summary

Re-review of the EdgeZero PR13 integration/provider HTTP type migration after the latest "Address review findings" commit. The async parse_response migration and the new collect_body / collect_body_bounded helpers landed cleanly and resolved the prior into_bytes()-on-Stream panic class across all providers. Two blockers remain plus one CI/process question that affects whether the existing test plan can be trusted.

Blocking

🔧 wrench

• cargo fmt --all -- --check fails locally in 7+ files. The PR description's [x] cargo fmt checkbox is incorrect — see the CI question below for why this wasn't caught. Affected files:

  • crates/trusted-server-core/src/auction/endpoints.rs:39-47
  • crates/trusted-server-core/src/auction/orchestrator.rs:12-14, 402-405
  • crates/trusted-server-core/src/integrations/google_tag_manager.rs:16-25, 39-45
  • crates/trusted-server-core/src/integrations/lockr.rs:282-285
  • crates/trusted-server-core/src/integrations/permutive.rs:208-219, 278-281
  • crates/trusted-server-core/src/integrations/prebid.rs:5-13, 1154-1162
  • crates/trusted-server-core/src/integrations/testlight.rs:11-16

  Fix: run cargo fmt --all.

• Testlight POST body is unbounded (inline at integrations/testlight.rs:181).

❓ question

• CI gates aren't enforced for this PR. .github/workflows/format.yml and .github/workflows/test.yml only fire on pull_request: branches: [main]. PR 626's base is feature/edgezero-pr12-handler-layer-types, so on commit 94ec9477 only "Integration Tests" ran — fmt, clippy, and cargo test --workspace did not. Combined with the fmt failure above, every stacked EdgeZero PR ships with broken formatting and an unverified test plan. Was this intentional, or should the workflow triggers be widened (or the gates duplicated under the integration-tests umbrella)? Until that's resolved, the PR description's CI checkboxes can't be relied on as evidence.

  Verified locally on this commit: cargo test --workspace passes (821 tests), cargo clippy --workspace --all-targets --all-features -- -D warnings passes, cargo build --release --target wasm32-wasip1 passes, cargo fmt --all -- --check fails.

Non-blocking

🤔 thinking

• Unbounded response-body reads via collect_body (inline at integrations/mod.rs:93 for stream-read error fidelity; cross-cutting concern listed here). The bounded variant exists for inbound bodies, but every upstream response site still drains an unbounded body to a Vec<u8>:

  • integrations/prebid.rs:1157 — Prebid Server response (highest risk: third-party hop, large payloads)
  • integrations/aps.rs:555 — APS response
  • integrations/adserver_mock.rs:375 — Mediator response
  • integrations/lockr.rs:150 — Lockr SDK fetch
  • integrations/permutive.rs:152 — Permutive SDK fetch
  • integrations/datadome.rs:302 — DataDome SDK fetch
  • integrations/google_tag_manager.rs:491 — GTM script

  Provider responses are the highest risk — a misbehaving Prebid Server / mediator can ship a multi-GB body and OOM the worker. Recommend routing all of these through collect_body_bounded with a per-callsite cap, or removing the unbounded variant in favor of a single bounded helper.

♻️ refactor

• Auction providers still construct BackendConfig inline while integration proxies route through ensure_integration_backend. Reraising the prior-review concern with current locations:

  • integrations/aps.rs:517-528
  • integrations/adserver_mock.rs:338-348
  • integrations/prebid.rs:1131-1135

  The two paths differ on timeout (providers need a configurable per-request timeout; the proxy helper hardcodes 15s). Consider widening ensure_integration_backend to accept a timeout and reusing it from the providers, or at least documenting the divergence.

CI Status

• fmt: FAIL locally (CI did not run for this commit — see ❓ question above)
• clippy: PASS locally (CI did not run)
• rust tests: PASS locally — 821 tests (CI did not run)
• wasm32-wasip1 build: PASS locally
• Integration Tests workflow: PASS (only workflow that ran on 94ec9477)

[aram356]

Summary

Re-review of PR13 after the latest "Address PR review" commit (4b27087e) and the PR12 merge (a2036eb5). The async parse_response migration, the dedicated AUCTION_MAX_BODY_BYTES, the README updates, and the testlight body cap all landed cleanly. One blocker remains plus several non-blocking re-raises from prior reviews that haven't been addressed.

Local verification on this commit:

• cargo test --workspace — PASS (952 tests, 0 failures)
• cargo clippy --workspace --all-targets --all-features -- -D warnings — PASS
• cargo build --release --target wasm32-wasip1 -p trusted-server-adapter-fastly — PASS
• cargo fmt --all -- --check — FAIL (4 PR-introduced sites + 11 inherited from PR12 base)
• GitHub CI (integration tests workflow) — PASS

GitHub's format.yml / test.yml still only fire on PRs targeting main, so the fmt/clippy/test gates don't run for this PR's stack. The PR description's [x] cargo fmt / [x] cargo clippy / [x] cargo test checkboxes therefore can't be relied on as evidence — see the fmt blocker below.

Blocking

🔧 wrench

• fmt failures still present in PR-modified files. cargo fmt --all -- --check reports diffs at:

  • crates/trusted-server-adapter-fastly/src/main.rs:209 and :236
  • crates/trusted-server-core/src/integrations/prebid.rs:1057 (the line at ~1066 is too long inside the if let block)
  • crates/trusted-server-core/src/integrations/registry.rs:673

  These four sites did not exist on the PR12 base — PR13 introduced them. (11 additional fmt diffs are inherited from the PR12 base and are not this PR's responsibility, but the project-wide cargo fmt --all -- --check does fail.) Fix: cargo fmt --all.

Non-blocking

🤔 thinking

• Provider/integration response bodies still unbounded (re-raise; inline at integrations/mod.rs:84). 7 callsites still drain upstream responses with collect_body instead of a bounded helper. Provider responses are highest risk (prebid.rs:1160, aps.rs:555, adserver_mock.rs:375).
• Stream-read errors classified as Integration (re-raise; inline at integrations/mod.rs:93 and the bounded variant at :137). Conflates transport failure with integration semantic error; both surface as 5xx-server with the same shape. GTM has a bespoke 502 path that the shared helper doesn't.
• Orchestrator backend-name fallback can silently lose bids (inline at auction/orchestrator.rs:337). When pending.backend_name() is None and the platform sets a different PlatformResponse::backend_name, the lookup at line 392 fails and the bid is dropped with a "Received response from unknown backend" warning. Consider a log::warn! when the fallback fires.
• collect_body_bounded checks size after reading the chunk (inline at integrations/mod.rs:143). The effective bound is "≤ max + one_chunk", not strict ≤ max. Acceptable in practice (chunk sizes are small) but worth a comment on the helper.

♻️ refactor

• Auction providers still construct BackendConfig inline (re-raise; inline at prebid.rs:1134, also aps.rs:517-528, adserver_mock.rs:338-348). Diverges from ensure_integration_backend because providers need a per-request timeout and the helper hardcodes 15s.
• AuctionContext.client_info is redundant with services.client_info() (re-raise; inline at auction/types.rs:107).
• run_auction takes services as a separate arg even though context.services exists (inline at auction/orchestrator.rs:64). They must stay in sync; an inconsistent caller breaks the auction silently.
• Inconsistent services.client_info access (inline at didomi.rs:259). Two sites use the pub(crate) field (didomi.rs:259, auction/formats.rs:145); the rest use the method services.client_info(). Standardize on the method.

📝 note

• Lockr/Permutive/Didomi handlers contain unreachable PUT/PATCH branches (inline at lockr.rs:202). Routes register only GET/POST.
• StubHttpClient::send_async does not capture request headers (inline at platform/test_support.rs:297). Limits header-shape assertions for prebid/aps/adserver_mock tests.

🏕 camp site

• testlight::Default impls can be derived (inline at testlight.rs:271).

👍 praise

• compat:: shim eliminated from integration dispatch — IntegrationProxy::handle is now http::Request<EdgeBody> end-to-end (main.rs:195, registry.rs:266). Removes a non-trivial conversion overhead per request and a class of header-mapping bugs.
• Async parse_response migration with PlatformResponse resolves the prior into_bytes()-on-Body::Stream panic class for every provider (auction/provider.rs:38). The trait docs explicitly note why it's now async — exactly the right context for future implementors.
• GTM 413 vs 502 distinction (google_tag_manager.rs:511-535). Clean separation between client-error and transport-error; this is the pattern the shared collect_body* helpers should follow.
• EC-ID header-injection guard (registry.rs:673) plus regression test at registry.rs:1406. Defensive HeaderValue::from_str blocks \r/\n/\0 injection on a user-influenced ID before it lands in either the request or the response.

CI Status

• fmt: FAIL locally (CI did not run for this commit because workflow triggers don't match the PR's base branch — see header above)
• clippy: PASS locally (CI did not run)
• rust tests: PASS locally (CI did not run; 952 tests)
• wasm32-wasip1 build: PASS locally
• Integration Tests workflow: PASS

[aram356]

Inline annotations for the prior REQUEST_CHANGES review

Adding the per-line comments now that I've filtered them against the PR diff (the previous submission was body-only because GitHub rejected the batch when 3 of 17 comment anchors fell outside the changed hunks). These reference the same findings as #626 (review) — no new findings, just inline anchoring.

The three findings that couldn't be anchored to a diff line are tagged below for reference:

• ♻️ refactor — run_auction takes services separately even though context.services exists. Anchored as a top-level note rather than inline because auction/orchestrator.rs:64 (the run_auction signature) is not in the diff, but the dual-pass pattern is visible at the response-handling sites (e.g. orchestrator.rs:337).
• 📝 note — StubHttpClient::send_async does not capture request headers. crates/trusted-server-core/src/platform/test_support.rs is not in this PR's diff scope at all; the asymmetry between send (lines 266–280) and send_async (lines 297–320) limits header-shape assertions for prebid/aps/adserver_mock tests. Worth a follow-up issue.
• 🏕 camp site — testlight::Default impls can be derived. testlight.rs:271-285 is outside the changed hunks; deriving #[derive(Default)] would replace both manual impls.

[aram356]

Inline annotations for the prior REQUEST_CHANGES review

Adding the per-line comments now that I've filtered them against the PR diff (the previous submission was body-only because GitHub rejected the batch when 3 of 17 comment anchors fell outside the changed hunks). These reference the same findings as #626 (review) — no new findings, just inline anchoring.

The three findings that couldn't be anchored to a diff line are tagged below for reference:

• ♻️ refactor — run_auction takes services separately even though context.services exists. Anchored as a top-level note rather than inline because auction/orchestrator.rs:64 (the run_auction signature) is not in the diff, but the dual-pass pattern is visible at the response-handling sites (e.g. orchestrator.rs:337).
• 📝 note — StubHttpClient::send_async does not capture request headers. crates/trusted-server-core/src/platform/test_support.rs is not in this PR's diff scope at all; the asymmetry between send (lines 266–280) and send_async (lines 297–320) limits header-shape assertions for prebid/aps/adserver_mock tests. Worth a follow-up issue.
• 🏕 camp site — testlight::Default impls can be derived. testlight.rs:271-285 is outside the changed hunks; deriving #[derive(Default)] would replace both manual impls.

[aram356]

Summary

Re-review of the EdgeZero PR13 integration/provider HTTP type migration after the latest commit (96949807). The async parse_response, collect_response_bounded, and dedicated AUCTION_MAX_BODY_BYTES constant landed cleanly and resolved several prior findings. Three blocking issues remain, all rooted in the same anti-pattern: change_context over collect_body_bounded discards the RequestTooLarge error variant, so oversized inbound bodies surface as 502 BAD_GATEWAY instead of 413 PAYLOAD_TOO_LARGE at every site except testlight. A test added in this PR (auction_rejects_oversized_body) already proves the bug.

CI status (verified locally on 96949807)

• fmt: PASS
• clippy: FAIL — 3 doc_markdown errors at integrations/mod.rs:172, 175
• rust tests: FAIL — auction_rejects_oversized_body panics (asserts 413, gets 502)
• wasm32-wasip1 release build: PASS
• JS tests: PASS — 282 tests
• Integration Tests workflow: PASS (only workflow that ran on this commit)

The PR description's [x] cargo clippy and [x] cargo test --workspace checkboxes are not corroborated by CI on this branch — .github/workflows/{format,test}.yml only fire on pull_request: branches: [main], so on commit 96949807 only the integration-tests workflow ran. Both gates fail when run locally.

Blocking

🔧 wrench

• cargo clippy --workspace --all-targets --all-features -- -D warnings fails at integrations/mod.rs:172, 175 (3 doc-markdown errors).
• auction_rejects_oversized_body fails because auction/endpoints.rs:46 change_contexts RequestTooLarge (→ 413) into Auction (→ 502).
• Same bug at every other collect_body_bounded caller except testlight: datadome.rs:362-364, didomi.rs:244-246, permutive.rs:213-217, lockr.rs:205-207. All produce 502 instead of 413 for oversized bodies, defeating the bounded-read security goal. testlight.rs:182-184 is the only correct caller — its ?-only pattern is the model.

Non-blocking

🤔 thinking

• Unbounded upstream-response read in testlight (testlight.rs:214). Every other RTB integration migrated to collect_response_bounded; testlight is the last unbounded site.
• Predicted-backend-name fallback at log::debug (orchestrator.rs:339-344). Failure mode is silent bid loss; recommend log::warn! so operators can see the fallback firing.

♻️ refactor

• Three providers still inline BackendConfig::from_url_with_first_byte_timeout (aps.rs:519-528, adserver_mock.rs:340-348, prebid.rs:1141-1145). Justification comments are reasonable; widening ensure_integration_backend with a first_byte_timeout: Duration parameter would let all four sites share one helper.

📝 note

• Testlight stale-Content-Length test is unit-only (testlight.rs:388-412). Extending handle_uses_platform_http_client_with_http_request to inject a stale Content-Length on the stubbed upstream would close the integration-level gap.