nginx active health checks use pod ip as host header by default, causing 502s with strict backend validation

[u/zero_backend_bro]

Universal 502 Bad Gateway responses across production API. Nginx ingress controller log: [error] 45#45: *1890201 no live upstreams while connecting to upstream, client: 10.1.4.55, server: api.prod.internal, request: "GET /v2/metrics HTTP/1.1", upstream: "http://api-backend".

K8s endpoints showed api-backend pods as Running and 100% Ready.

Assumed Nginx failed routing packets to pod IPs. Exec'd into Nginx pod, ran curl -I <pod-ip>:8080. TCP connection succeeded but returned 400 Bad Request.

Suspected asymmetric routing drops or MTU truncation on the Calico VXLAN tunnel corrupting the payload. Deep-dived iptables -t filter -L and tcpdump for 90 minutes looking for dropped fragments.

Looked at Envoy sidecars rejecting plain HTTP originating from Nginx due to PeerAuthentication strict mTLS policy. Verified destination rules. Applied PERMISSIVE mode. Upstream pool remained empty.

Nginx active health checks (health_check directive) were failing due to strict HTTP Host-header validation at the Go Fiber layer. Standard Nginx health checks request the URI using the upstream Pod IP as the default Host header (Host: 10.244.1.55).

Go application rejected requests lacking exact FQDN (Host: api.prod.internal) with 400 Bad Request. Nginx received 400, failed match condition expecting 2xx/3xx.

Incremented failure count to max_fails=3. Evicted every backend pod from the upstream pool. Round-robin found zero valid targets.

Fixed by defining a raw HTTP request string in the Nginx match block via the send directive. Forces health-check daemon to inject correct Host header expected by backend validation.

upstream api-backend {
    zone api-backend 64k;
    server 10.244.1.55:8080 max_fails=3 fail_timeout=10s;
    server 10.244.2.12:8080 max_fails=3 fail_timeout=10s;
}

match server_ok {
    # Send explicit HTTP literal to bypass IP-based Host header defaults
    send "GET /healthz HTTP/1.1\r\nHost: api.prod.internal\r\nConnection: close\r\n\r\n";
    expect ~* "200 OK";
}

server {
    listen 80;
    server_name api.prod.internal;

    location / {
        proxy_pass http://api-backend;
        proxy_set_header Host $host;
        health_check uri=/healthz match=server_ok mandatory;
    }
}

[u/One-Department1551]

You didn't post your Probes so it's harder to understand if you missed the HTTP settings there or not.

    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: Host
          value: api.prod.internal

should be somewhat enough to validate this.

[u/zero_backend_bro → u/One-Department1551]

k8s probes were passing fine (mentioned in the post, endpoints were 100% ready).

the issue isn't kubelet failing the pod, it's the nginx ingress controller's internal active health check (`health_check` directive) marking the upstream as down in its own shared memory zone. adding the host header to the deployment's livenessProbe does absolutely nothing to change how nginx formats its own internal routing check packets.