How do I troubleshoot pod deployment issues in Kubernetes?

1. Check the Pod Status

Use kubectl get pods to see the pod state:
bash kubectl get pods -n <namespace>
Statuses you’ll see:
- Pending: Waiting for resources or scheduling.
- Running: Pod is running, but may still have issues.
- CrashLoopBackOff: Repeatedly crashing and restarting.
- Error: Failed during startup.

2. Inspect Events for Errors

Check events in the namespace for scheduling, resource, or other errors:
bash kubectl describe pod <pod-name> -n <namespace>
Key messages to look for:
- FailedScheduling: Node affinity, taints, or lack of resources.
- FailedMount: Persistent volume or storage issues.
- ImagePullBackOff: Problem pulling the container image.

3. Verify Pod Logs

Check the failing container’s logs for runtime errors:
bash kubectl logs <pod-name> -n <namespace>
If the pod has multiple containers, specify the container name:
bash kubectl logs <pod-name> -n <namespace> -c <container-name>
Look for stack traces, configuration issues, or application errors.

4. Check Deployment Configuration

Review the deployment YAML for misconfigurations:
bash kubectl get deployment <deployment-name> -n <namespace> -o yaml
Common problems:
- Missing or incorrect environment variables.
- Wrong image name or tag.
- Missing volume mounts or incorrect mount paths.
- Resource requests/limits exceeding node capacity.

5. Check Node Capacity and Scheduling

Verify nodes have sufficient resources (CPU, memory, GPU) to schedule the pod:
bash kubectl describe nodes
Check for:
- Insufficient CPU or memory.
- Taints and tolerations blocking scheduling.
- Node affinity/anti-affinity rules preventing placement.

6. Inspect Persistent Volume Claims (PVC)

If the pod uses storage, verify the PVCs:
bash kubectl get pvc -n <namespace> kubectl describe pvc <pvc-name> -n <namespace>
Typical issues:
- PVC not bound to a Persistent Volume (PV).
- Storage class misconfiguration.
- Insufficient storage capacity.

7. Check Image Configuration

Verify the container image is correct and accessible:
Confirm the image name and tag are correct.
Ensure the image exists in the container registry.
If the image is private, confirm the proper image pull secret is configured:
bash kubectl get secret -n <namespace>

8. Inspect Network Policies

If the pod relies on network communication, verify NetworkPolicies aren’t blocking traffic:
bash kubectl get networkpolicy -n <namespace>
Ensure required ports and communication paths are allowed.

9. Examine Pod Health Probes

Check readiness and liveness probes in the deployment configuration:
yaml livenessProbe: httpGet: path: /healthz port: 8080 readinessProbe: httpGet: path: /ready port: 8080
Misconfigured probes cause pods to restart or fail to become ready.

10. Validate ConfigMaps and Secrets

Ensure ConfigMaps and Secrets referenced in the pod are correctly defined:
bash kubectl get configmap -n <namespace> kubectl get secret -n <namespace>
Verify the pod has permissions to access them.

11. Check Cluster-Wide Issues

Investigate cluster-wide issues affecting pod deployment:
Review key component status:
bash kubectl get nodes kubectl get --raw='/readyz?verbose'
Check for resource constraints or node failures.

12. Debug with Temporary Pods

Spin up a temporary pod to inspect the environment:
bash kubectl run debug-pod --rm -i --tty --image=nicolaka/netshoot -- sh
Use busybox if you only need basic shell tools.

13. Review Kubernetes Logs

Examine kubelet or control plane logs for scheduling or runtime errors.
On the node, check kubelet logs directly.
Use your monitoring/logging stack (ELK, Prometheus, Grafana, Loki) for deeper investigation.

14. Scale Down and Restart

If the deployment has multiple replicas, try scaling to zero and back up:
bash kubectl scale deployment <deployment-name> --replicas=0 -n <namespace> kubectl scale deployment <deployment-name> --replicas=<desired-number> -n <namespace>
Note: this causes downtime. Use kubectl rollout restart for zero-downtime restarts.

15. Leverage Debugging Tools

Use Kubernetes-native debugging tools:
kubectl exec to inspect containers interactively:
bash kubectl exec -it <pod-name> -n <namespace> -- /bin/bash
kubectl port-forward to test application endpoints locally:
bash kubectl port-forward <pod-name> <local-port>:<container-port> -n <namespace>

16. Review RBAC and Permissions

Ensure the pod has necessary permissions if it interacts with the API or other cluster resources:
bash kubectl auth can-i <verb> <resource> --as=<service-account>