How do I troubleshoot Kubernetes networking issues?

Troubleshooting Kubernetes networking issues is challenging because it spans multiple abstraction layers: pods, services, network policies, and the underlying infrastructure. Here’s a systematic approach.

1. Understand the Scope

Identify the specific symptom:
Pods can’t talk to each other?
Services unreachable?
External access broken?
DNS lookups failing?
Determine whether the problem hits one pod, one namespace, or the whole cluster.

2. Verify Pod Networking

a. Check Pod Status

Run kubectl get pods and confirm the pod is actually running.
If it’s not, the issue may be scheduling, image pulls, or resource limits — not networking.

b. Inspect Pod IP Addresses

Run kubectl get pods -o wide to see assigned pod IPs.
Missing IPs usually point to a CNI (Container Network Interface) plugin problem.

c. Test Pod-to-Pod Connectivity

From inside a pod, use ping, curl, or telnet to reach another pod’s IP.
Example:
bash kubectl exec -it <pod-name> -- ping <target-pod-ip>
If this fails, check network policies, firewalls, or routing rules.

3. Verify Service Networking

a. Check Service Configuration

Run kubectl get svc and verify the type (ClusterIP, NodePort, LoadBalancer) and settings.
Example:
bash kubectl describe svc <service-name>

b. Test Service Connectivity

Access the service from inside and outside the cluster with curl or wget.
Example:
bash kubectl exec -it <pod-name> -- curl <service-cluster-ip>:<port>
If it fails, check:
Service selectors — are they matching the right pods?
Endpoints — run kubectl get endpoints <service-name> and confirm endpoints exist.

c. Check External Access

For LoadBalancer or Ingress, verify the external IP or DNS name.
Example:
bash kubectl get ingress
Test connectivity to that external endpoint from outside the cluster.

4. Verify DNS Resolution

a. Test DNS in Pods

From a pod, run nslookup or dig against a service name.
Example:
bash kubectl exec -it <pod-name> -- nslookup <service-name>

b. Check CoreDNS Logs

Inspect the CoreDNS pods for errors.
Example:
bash kubectl logs -n kube-system <coredns-pod-name>

c. Verify CoreDNS Configuration

Check the CoreDNS ConfigMap.
Example:
bash kubectl get cm -n kube-system coredns -o yaml

5. Check Network Policies

List policies in the namespace: kubectl get networkpolicy
Look for restrictive rules blocking expected traffic.
Example:
bash kubectl describe networkpolicy <policy-name>

6. Verify CNI Plugin

Confirm your CNI plugin (Calico, Flannel, Cilium, etc.) is healthy.

a. Check CNI Pod Logs

Inspect logs of the CNI pods.
Example:
bash kubectl logs -n kube-system <cni-pod-name>

b. Ensure CNI Configuration

Verify CNI config files on nodes (typically /etc/cni/net.d/).

c. Restart CNI Pods

A restart can clear transient issues.
Example:
bash kubectl rollout restart deployment -n kube-system <cni-deployment-name>

7. Check Node Networking

a. Verify Node IPs

Confirm nodes have valid IPs and can reach each other.
Run kubectl get nodes -o wide to see node IPs.

b. Inspect Node-Level Network Config

Check firewall rules, routing tables, and network interfaces on the nodes.
Ensure kube-proxy is running:
Example:
bash kubectl get pods -n kube-system | grep kube-proxy kubectl logs -n kube-system <kube-proxy-pod-name>

c. Test Node-to-Node Connectivity

From one node, ping or telnet another node to verify baseline connectivity.

8. Debugging Tools

a. Tcpdump/Wireshark

Capture traffic on nodes or inside pods with tcpdump or Wireshark.

b. Tracing and Logging

Use traceroute or mtr to trace network paths.
Increase kubectl verbosity for detailed API output:
Example:
bash kubectl get pods -v=9

c. Kubernetes Debugging Tools

Install tools like k9s, kubectl-debug, or kubectl-trace for deeper inspection.

9. Check Underlying Infrastructure

Verify the infrastructure layer (VMs, bare metal, cloud networking) is healthy.
Check security groups, firewalls, and routing in your cloud provider or on-prem environment.