How do I debug Kubernetes ingress controllers for HTTP 502 or 504 errors?

1. Understand the error codes

HTTP 502 (Bad Gateway): The Ingress controller reached the backend, but the backend returned an invalid response.
HTTP 504 (Gateway Timeout): The Ingress controller didn’t receive a response from the backend within the configured timeout.

2. Check the Ingress resource

Start with the Ingress manifest:

kubectl describe ingress <ingress-name>

Verify:

Host and path rules match your intended routing.
The backend service.name and service.port (or serviceName/servicePort on older APIs) are correct.

Check annotations — they control timeouts, load balancing, and other behavior. For the NGINX controller, common ones include:

annotations:
  nginx.ingress.kubernetes.io/proxy-connect-timeout: "60"
  nginx.ingress.kubernetes.io/proxy-read-timeout: "60"

3. Verify the backend Service

kubectl get service <service-name> -o yaml

Confirm:

The service type (ClusterIP, NodePort, etc.) fits your topology.
The service port matches what the Ingress references.
If the service uses a selector, it matches the target pod labels.

4. Inspect the backend Pods

kubectl get pods -l <label-selector>

Pods should be Running and Ready. Check readiness and liveness probes. Pull logs:

kubectl logs <pod-name>

Test connectivity from inside the cluster:

kubectl exec -it <pod-name> -- curl http://<service-name>:<service-port>

5. Check the Ingress controller itself

Get the controller logs:

kubectl logs -n <ingress-namespace> <ingress-controller-pod>

Look for errors or warnings tied to the 502/504. Also inspect the deployment:

kubectl describe deployment -n <ingress-namespace> <ingress-controller-deployment>

6. Validate networking and DNS

From a pod, verify service DNS resolution:

kubectl exec -it <pod-name> -- nslookup <service-name>

From the Ingress controller pod, test the backend directly:

kubectl exec -n <ingress-namespace> <ingress-controller-pod> -- curl http://<service-name>:<service-port>

7. Check external load balancer and firewall rules

If a cloud load balancer sits in front, confirm it forwards traffic to the Ingress controller and its health checks pass.
Review any network policies or firewall rules that might block traffic between the Ingress controller and backend services.
If using a cloud provider’s load balancer, inspect its logs for errors or misconfigurations.

8. Adjust timeouts (NGINX controller)

504s often mean the backend is slower than the proxy allows. Increase the NGINX proxy timeouts via annotations:

annotations:
  nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "120"

Also verify the application can actually respond within that window.

9. Debug with port-forward and curl

Bypass the Ingress entirely to test the backend:

kubectl port-forward svc/<service-name> <local-port>:<service-port>
curl http://localhost:<local-port>

Hit the Ingress directly with verbose output:

curl -v http://<ingress-host>/<path>

10. Review custom controller configuration

If you’re running Traefik, HAProxy, Istio, or another controller, check its specific configuration and logs. Some require custom annotations or CRDs to behave as expected.

11. Monitor metrics

Enable Prometheus/Grafana monitoring for the Ingress controller. Watch for high latency, connection errors, or dropped requests.

12. Test with a minimal setup

Create a bare-bones Ingress pointing to a simple backend (e.g., an nginx pod) to rule out complex configuration as the culprit.

13. Check for known issues

Search the documentation and GitHub issues for your specific Ingress controller (NGINX, Traefik, HAProxy, etc.) for relevant bugs or limitations.