第 12 天:AI 推理服務如何用 HPA 應對突發 API 流量
Day 12: Auto-Scaling AI Inference — Using Horizontal Pod Autoscalers for Spiky API Traffic
Kubernetes HPA 可透過自訂指標,依等待佇列深度為 AI 推理服務增減 Pod,以應對 API 流量突增。範例透過 Prometheus Adapter 暴露 vllm_num_requests_waiting,將目標設為每個 Pod 平均 10 個等待請求,副本數介乎 1 至 4。設定將擴容穩定窗口設為 0,縮容則採 300 秒窗口,並限制每 60 秒最多減少 25%。
AI traffic is volatile, and idle GPU or high-memory instances are expensive. You want replicas to follow demand. The obvious tool is the Horizontal Pod Autoscaler (HPA), but its default signals, CPU and memory, mislead you for inference servers.
The pipeline
Diagram: FIG 1 · The inference engine exports queue depth, Prometheus stores it, an adapter exposes it to Kubernetes and the HPA adds or removes pods based on it. See the animated version.
Requests reach the inference engine, which exports a queue metric. Prometheus scrapes it, an adapter exposes it to Kubernetes as a custom metric and the HPA adds or removes pods based on it.
Why CPU and memory fail
Diagram: FIG 2 · Inference engines pre-allocate memory and saturate devices even under light load, so CPU and memory utilisation say little about demand. Queue depth and time per output token track what users experience. See the animated version.
- Memory. Runtimes such as vLLM reserve 85 to 90% of GPU memory at start-up for the weights and the KV-cache pool. That figure is high even with no traffic, so a memory-based rule would scale out to the maximum and stay there.
- CPU and GPU use. A single request can push a device close to 100%, while continuous batching may still have room for many more. The signal is noisy.
A better signal is the depth of the waiting queue, or the time per output token (TPOT). Both track the wait users feel, which is what scaling should protect.
Diagram: FIG 3 · As the waiting queue grows, the HPA adds replicas, and it removes them again slowly once the queue drains. The numbers are illustrative. See the animated version.
The HPA manifest
This scales on the number of waiting requests per pod. vLLM exports this metric as vllm:num_requests_waiting, and Prometheus Adapter exposes it to Kubernetes under a name you configure, shown here as vllm_num_requests_waiting.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: inference-engine-hpa
namespace: ai-platform
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: inference-engine-gemma
minReplicas: 1
maxReplicas: 4
metrics:
- type: Pods
pods:
metric:
name: vllm_num_requests_waiting
target:
type: AverageValue
averageValue: "10"
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- { type: Percent, value: 100, periodSeconds: 15 }
scaleDown:
stabilizationWindowSeconds: 300
policies:
- { type: Percent, value: 25, periodSeconds: 60 }
Exposing a custom metric to the HPA needs an adapter such as Prometheus Adapter, or you can use KEDA, which can scale directly on a Prometheus query. Check the exact metric name your vLLM version exports.
Scale up fast, scale down slowly
Diagram: FIG 4 · Scale out aggressively to absorb a spike, and scale in cautiously, because a model pod takes minutes to load and flapping wastes that work. See the animated version.
A new model pod takes minutes to load, so you want to react to a spike immediately and avoid throwing away warm pods the moment traffic dips. The behavior block encodes that: no stabilisation delay when scaling up, and a five-minute window with gentle steps when scaling down.
Scaling on the queue ties your replica count to the user experience, and it keeps expensive capacity from sitting idle.
Run it yourself
This is the exact example from the video. A small Python model of the autoscaler's rule, using the lesson's numbers. It is not the real HPA.
import math
target, low, high = 10, 1, 4 # waiting requests per pod
window = 5 # minutes, like 300 seconds
recs = []
for minute, waiting in enumerate([5, 30, 60, 20, 5, 5, 5, 5]):
want = math.ceil(waiting / target)
recs.append(min(high, max(low, want)))
pods = max(recs[-window:])
print(f"min {minute}: waiting {waiting:2d} -> {pods} pods")
Run it:
python hpa.py
Output I got while recording the video:
min 0: waiting 5 -> 1 pods
min 1: waiting 30 -> 3 pods
min 2: waiting 60 -> 4 pods
min 3: waiting 20 -> 4 pods
min 4: waiting 5 -> 4 pods
min 5: waiting 5 -> 4 pods
min 6: waiting 5 -> 4 pods
min 7: waiting 5 -> 2 pods
Coming Up Next
Day 13: Helm charts: packaging, versioning and updating AI deployments.
HPA #Autoscaling #MLOps #Prometheus #CloudCostOptimization #SRE #Kubernetes
Originally published at https://sureshpallapothu.in/blog/ai-int-day-12-autoscaling-inference, where this post includes animated diagrams.
來源:DEV Community · dev.to