Gateway API + Istio Ambient: traffic split to canary bypasses Service-bound HTTPRoute header injection (downstream header routing breaks)

[opschronicle]

What I’m trying to do

Use Flagger (Gateway API provider) with Istio Ambient Mesh + Waypoint to canary main-service (v1→v2). I need header-based fan-out: when a request lands on main-service v2, it should call secondary-service v2. The header propagation is done with a Service-bound HTTPRoute that injects headers for traffic to main-service-canary, and a second HTTPRoute on secondary-service that matches those headers.

Expected

When Flagger’s split sends N% of traffic to main-service-canary, those requests traverse the canary Service’s waypoint and the Service-bound HTTPRoute (main-service-canary-injection) runs, injecting:

x-target-version: v2

x-service-version: v2

x-routing-source: flagger-canary-injection

secondary-service-routing (Service-bound HTTPRoute) sees those headers and routes to secondary-service-canary.

Actual

Flagger’s split works (weights change as expected), and main-service v2 is reached some percentage of the time.

But the follow-on call from main→secondary stays on v1 during split (no headers present).

If I bypass Flagger and hit canary directly (or send x-target-version: v2 at ingress using a separate header-bypass route), everything works: secondary goes to v2 as expected.

Adding a ResponseHeaderModifier to main-service-canary-injection (e.g., x-injected: main-canary) is never observed on responses that came via Flagger’s split. This strongly suggests split traffic is not passing through the canary Service’s waypoint route where the injection lives.

Environment

Istio 1.27.0, Ambient mode (ztunnel/waypoint)

Ingress: istio-waypoint class for mesh waypoint; istio class for north-south HTTP Gateway

Waypoint Deployment image: docker.io/istio/proxyv2:1.27.0-distroless

Flagger: Gateway API provider (current release as of Aug 2025)

Kubernetes: EKS (AWS), services are ClusterIP; ALB Ingress → Istio Gateway service

Namespace: istio-test2 labeled for ambient:

metadata:
  labels:
    istio.io/dataplane-mode: ambient

Both main-service-{primary,canary} and secondary-service-{primary,canary} are ClusterIP.

Minimal manifests (trimmed to essentials)
1) Waypoint (mesh L7)

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-test2-waypoint
  namespace: istio-test2
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

2) Ingress Gateway (north-south)

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-test2-ingress
  namespace: istio-test2
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    hostname: istio-test2.mywebsite.com
    port: 80
    protocol: HTTP

3) Flagger-managed HTTPRoute (this is the splitter, owned by Flagger)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: main-service
  namespace: istio-test2
  ownerReferences:
  - apiVersion: flagger.app/v1beta1
    kind: Canary
    name: main-service
spec:
  hostnames:
  - main-service.istio-test2.svc.cluster.local
  - istio-test2.mywebsite.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-test2-ingress
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - kind: Service
      name: main-service-primary
      port: 9090
      weight: 60   # ← Flagger updates these
    - kind: Service
      name: main-service-canary
      port: 9090
      weight: 40

4) My Service-bound HTTPRoute to inject headers at the canary

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: main-service-canary-injection
  namespace: istio-test2
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: main-service-canary
    port: 9090
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: x-target-version
          value: v2
        - name: x-service-version
          value: v2
        - name: x-routing-source
          value: flagger-canary-injection
    # (optionally also add a response marker for debugging)
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        add:
        - name: x-injected
          value: main-canary
    backendRefs:
    - kind: Service
      name: main-service-canary
      port: 9090

5) Secondary routing (follows headers)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: secondary-service-routing
  namespace: istio-test2
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: secondary-service
    port: 9090
  rules:
  # match ANY of these headers -> route to canary
  - matches:
    - path: { type: PathPrefix, value: / }
      headers:
      - name: x-target-version
        type: Exact
        value: v2
    - path: { type: PathPrefix, value: / }
      headers:
      - name: x-service-version
        type: Exact
        value: v2
    - path: { type: PathPrefix, value: / }
      headers:
      - name: x-routing-source
        type: Exact
        value: flagger-canary-injection
    backendRefs:
    - kind: Service
      name: secondary-service-canary
      port: 9090
      weight: 100
    - kind: Service
      name: secondary-service-primary
      port: 9090
      weight: 0

  # default -> primary
  - matches:
    - path: { type: PathPrefix, value: / }
    backendRefs:
    - kind: Service
      name: secondary-service-primary
      port: 9090
      weight: 100

6) Service annotations/labels to force waypoint processing from ingress

(tried these to make sure ingress traffic hits the waypoint)

kubectl -n istio-test2 annotate svc main-service-canary \
  networking.istio.io/ingress-waypoint="enabled" --overwrite
kubectl -n istio-test2 annotate svc main-service-primary \
  networking.istio.io/ingress-waypoint="enabled" --overwrite
kubectl -n istio-test2 label svc main-service-canary \
  istio.io/use-waypoint=istio-test2-waypoint --overwrite
kubectl -n istio-test2 label svc main-service-primary \
  istio.io/use-waypoint=istio-test2-waypoint --overwrite

# (also set HBONE on the ingress pod and restarted it)
kubectl -n istio-test2 patch deploy istio-test2-ingress-istio --type='json' -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/env","value":[
    {"name":"ISTIO_META_ENABLE_HBONE","value":"true"}
  ]}
]'
kubectl -n istio-test2 rollout restart deploy/istio-test2-ingress-istio

Repro / Verification commands

Flagger split is active:

kubectl -n istio-test2 get httproute main-service -o yaml | yq '.spec.rules[0].backendRefs'
# shows primary/canary weights (e.g., 60/40)

Bypass check (works end-to-end):

curl -s -H 'x-target-version: v2' http://istio-test2.mywebsite.com/ \
| jq '{main: .name, secondary: .upstream_calls["http://secondary-service.istio-test2.svc.cluster.local:9090"].name}'
# Expected: main-service-v2 + secondary-service-v2 

Split traffic check (fails to carry headers):

HOST=http://secondary-service.istio-test2.svc.cluster.local:9090
for i in $(seq 1 80); do
  curl -fsS http://istio-test2.mywebsite.com/ \
  | jq -r --arg h "$HOST" '(.name)+" -> " + (.upstream_calls[$h].name // "none")' || true
done | sort | uniq -c
# Typical result during 40% canary:
#   45 main-service-v1 -> secondary-service-v1
#   35 main-service-v2 -> secondary-service-v1   <-- missing headers, stayed on v1

Response marker from main-service-canary-injection is never seen:

for i in $(seq 1 120); do
  curl -sI http://istio-test2.mywebsite.com/ | awk -v RS='\r\n' 'tolower($0) ~ /^x-injected:/{print}'
done | sort | uniq -c
# (no lines printed)

Waypoint/Ingress route dumps (evidence that split exists and that the injection route exists):

Ingress shows the split (weightedClusters to main-service-{primary,canary}); the header-bypass route also present:

ING=$(kubectl -n istio-test2 get pod -l gateway.networking.k8s.io/gateway-name=istio-test2-ingress -o jsonpath='{.items[0].metadata.name}')
istioctl -n istio-test2 pc routes "$ING" --name http.80 -o json

Shows:
Route name istio-test2.main-service.0 with weightedClusters primary/canary
The header-bypass route with requestHeadersToAdd (works when used)

Waypoint has the inbound virtual host for main-service-canary and includes the requestHeadersToAdd from main-service-canary-injection:

WP=$(kubectl -n istio-test2 get pod -l gateway.networking.k8s.io/gateway-name=istio-test2-waypoint -o jsonpath='{.items[0].metadata.name}')
istioctl -n istio-test2 pc routes "$WP" -o json

Contains virtualHost:
name: inbound-vip|9090|http|main-service-canary.istio-test2.svc.cluster.local
And shows requestHeadersToAdd: x-target-version=v2, x-service-version=v2, x-routing-source=flagger-canary-injection
(and optional responseHeadersToAdd: x-injected=main-canary if added)

Endpoints at ingress show CONNECT originate nodes (HBONE path), but headers still absent:

istioctl -n istio-test2 pc endpoints "$ING" --cluster "outbound|9090||main-service-canary.istio-test2.svc.cluster.local"

ENDPOINT … envoy://connect_originate/:9090 (HEALTHY)
Hypothesis

Flagger’s split at the Ingress HTTPRoute sends traffic to main-service-canary, but that path does not traverse the Service-bound HTTPRoute (main-service-canary-injection) at the canary’s waypoint for split traffic. (Bypass and direct service calls do traverse it.)

Practically: split traffic → no header injection at main canary → secondary stays v1 even though main is v2.

Questions for Flagger maintainers

With Gateway API provider + Istio Ambient, is Flagger expected to:

Ensure the generated canary/primary Services are annotated/labeled so that ingress traffic to those Services must traverse their waypoint (e.g., networking.istio.io/ingress-waypoint=enabled, istio.io/use-waypoint=)?

Or is that considered out-of-scope for Flagger?

Is there a supported way in Flagger’s Canary spec to attach header injection to the canary path when using the Gateway API provider?

e.g., “add these RequestHeaderModifier filters when backendRef is the canary” (today filters live at rules[*], not per-backend, so two rules would be needed)

Or generate (optionally) a companion Service-bound HTTPRoute for canary that injects headers (like main-service-canary-injection) and ensure split traffic goes through it.

Would you accept an enhancement to allow custom labels/annotations on Flagger-generated Services (primary/canary), so users can declaratively enforce waypoint processing (istio.io/use-waypoint, networking.istio.io/ingress-waypoint) as part of the Canary?

[stefanprodan]

Would you accept an enhancement to allow custom labels/annotations on Flagger-generated Services

We do allow this https://docs.flagger.app/main/usage/how-it-works#canary-service

From the docs:

You can configure Flagger to set annotations and labels for the generated services with:

spec:
  service:
    port: 9898
    apex:
      annotations:
        test: "test"
      labels:
        test: "test"
    canary:
      annotations:
        test: "test"
      labels:
        test: "test"
    primary:
      annotations:
        test: "test"
      labels:
        test: "test"