How do I implement custom metrics in Kubernetes Horizontal Pod Autoscaler (HPA)?

Custom metrics let you scale Kubernetes workloads on signals that matter to your application, not just CPU or memory. Here’s how to wire them into the Horizontal Pod Autoscaler.

---

1. Understand HPA and Custom Metrics

HPA pulls metrics from the Kubernetes Metrics API. For custom metrics, you need a component that serves those metrics to the API — typically the Custom Metrics API.

---

2. Set Up a Metrics Adapter

Pick an adapter that matches your monitoring stack:

• Prometheus Adapter: Exposes Prometheus metrics to the HPA.
• Stackdriver Adapter: For GCP, integrates with Cloud Monitoring.

---

3. Install and Configure the Metrics Adapter

a. Prometheus Adapter Installation

If you’re on Prometheus, install the adapter via Helm:

bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

bash
helm install prometheus-adapter prometheus-community/prometheus-adapter --namespace kube-system

Then edit the adapter’s values.yaml to map your Prometheus metrics to Kubernetes custom metrics.

b. Verify Adapter Installation

Confirm the custom metrics API is live:

bash
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq

You should see a list of exposed metrics.

---

4. Expose Custom Metrics

Your application or monitoring pipeline must push the metrics to your monitoring backend (e.g., Prometheus). Options include:

• Application instrumentation (e.g., prometheus-client for Python, prom-client for Node.js).
• Exporters such as kube-state-metrics or custom exporters.

---

5. Create an HPA Configuration for Custom Metrics

Define an HPA that references your metric:

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: custom-metric-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: your-deployment
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: your_custom_metric_name
target:
type: AverageValue
averageValue: 100

Replace your_custom_metric_name with the metric name from the adapter. Set averageValue to the threshold that should trigger scaling.

---

6. Test and Monitor Your Setup

• Deploy the HPA:
  bash
kubectl apply -f custom-metric-hpa.yaml
• Confirm it’s reading the custom metric:
  bash
kubectl describe hpa custom-metric-hpa
• Generate load or tweak the metric to verify scaling behavior.

---

7. Troubleshooting

If scaling doesn’t work:

1. Check metric availability:
   bash
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/your_custom_metric_name" | jq
2. Validate the Prometheus query in the adapter config.
3. Inspect HPA events and adapter logs:
   bash
kubectl describe hpa custom-metric-hpa
kubectl logs -n kube-system -l app.kubernetes.io/name=prometheus-adapter

---

8. Best Practices

• Choose metrics that directly reflect application performance.
• Watch HPA behavior over time; adjust thresholds as needed.
• Set reasonable min/max replicas and thresholds to avoid thrashing.