Course outline▾
Week 1 · Bare-Metal Infrastructure & Advanced Containerization
- Day 1The AI Infrastructure Shift — Why the Model is Only 10% of the Production System
- Day 2Advanced Docker for AI — Multi-Stage Builds for Heavy Python Runtimes
- Day 3Hardware Acceleration in Docker — GPU Passthrough and Limits for Constrained Hardware
- Day 4Optimizing Model Weights — Quantization Basics (GGUF, AWQ) for Efficient Inference
- Day 5Local AI Runtimes — Deploying Ollama and vLLM for Bare-Metal Execution
- Day 6Containerizing AI Microservices — Building and Packaging APIs with FastAPI
- Day 7Docker Compose — Architecting a Multi-Container Database and Inference Stack Locally
Week 2 · Kubernetes Foundation for MLOps
- Day 8K3d and Lightweight Kubernetes — Running Multi-Node Clusters on 16 GB of RAM
- Day 9Kubernetes Core for Machine Learning — Pods, Deployments and Headless Services
- Day 10Persistent Volumes (PV/PVC) — Managing Heavy Model Weights and Vector Stores
- Day 11StatefulSets for Vector Databases — Deploying Milvus or Qdrant in Kubernetes
- Day 12Auto-Scaling AI Inference — Using Horizontal Pod Autoscalers for Spiky API Traffic
- Day 13Helm Charts: Packaging, Versioning, and Updating AI Deployments
- Day 14Week 2 Capstone: Deploying a Highly Available Local AI Application on K3d
Week 3 · High-Performance Networking & Security Mesh
- Day 15AI Network Bottlenecks: Transitioning from iptables to eBPF
- Day 16Cilium & eBPF: Sub-Millisecond Routing for Heavy AI Microservices
- Day 17Service Mesh 101: Installing Istio over Your Local Cluster
- Day 18Envoy Proxy as an AI Gateway: Load Balancing and Terminating LLM Prompts
- Day 19Zero-Trust mTLS: Encrypting Traffic Between Autonomous Agents
- Day 20API Rate Limiting & Egress Control: Preventing Agents from Exfiltrating Data
- Day 21Distributed Tracing: Debugging Agent Workflows with Jaeger and OpenTelemetry
Week 4 · Enterprise Agents & CI/CD Workflows
- Day 22Introduction to MLOps: Bridging AI Development with Continuous Integration
- Day 23GitOps with ArgoCD: Automated, Self-Healing AI Deployments
- Day 24Building Ephemeral AI Sandboxes: Secure Execution Environments for Agent Code
- Day 25Multi-Agent Orchestration Frameworks: Exploring Mastra, LangGraph, and CrewAI
- Day 26Tool-Calling in Production: Connecting Agents Securely to Internal Databases
- Day 27MLOps Guardrails: Evaluating Agents and Preventing Hallucinations in Real-Time
- Day 28Continuous Training Pipelines: Connecting Data Ingestion to Automated Model Retraining
- Day 29Managing Technical Debt and Infrastructure Drift in Enterprise AI
- Day 30The AI-Native Platform: Scaling from a Sandbox Environment to Global Cloud Architecture
Day 12: Auto-Scaling AI Inference — Using Horizontal Pod Autoscalers for Spiky API Traffic
2026-11-18 · 8 min read
AI traffic is volatile, and idle GPU or high-memory instances are expensive. You want replicas to follow demand. The obvious tool is the Horizontal Pod Autoscaler (HPA), but its default signals, CPU and memory, mislead you for inference servers.
The pipeline
Requests reach the inference engine, which exports a queue metric. Prometheus scrapes it, an adapter exposes it to Kubernetes as a custom metric and the HPA adds or removes pods based on it.
Why CPU and memory fail
- Memory. Runtimes such as vLLM reserve 85 to 90% of GPU memory at start-up for the weights and the KV-cache pool. That figure is high even with no traffic, so a memory-based rule would scale out to the maximum and stay there.
- CPU and GPU use. A single request can push a device close to 100%, while continuous batching may still have room for many more. The signal is noisy.
A better signal is the depth of the waiting queue, or the time per output token (TPOT). Both track the wait users feel, which is what scaling should protect.
The HPA manifest
This scales on the number of waiting requests per pod. vLLM exports this metric as vllm:num_requests_waiting, and Prometheus Adapter exposes it to Kubernetes under a name you configure, shown here as vllm_num_requests_waiting.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: inference-engine-hpa
namespace: ai-platform
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: inference-engine-gemma
minReplicas: 1
maxReplicas: 4
metrics:
- type: Pods
pods:
metric:
name: vllm_num_requests_waiting
target:
type: AverageValue
averageValue: "10"
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- { type: Percent, value: 100, periodSeconds: 15 }
scaleDown:
stabilizationWindowSeconds: 300
policies:
- { type: Percent, value: 25, periodSeconds: 60 }
Exposing a custom metric to the HPA needs an adapter such as Prometheus Adapter, or you can use KEDA, which can scale directly on a Prometheus query. Check the exact metric name your vLLM version exports.
Scale up fast, scale down slowly
A new model pod takes minutes to load, so you want to react to a spike immediately and avoid throwing away warm pods the moment traffic dips. The behavior block encodes that: no stabilisation delay when scaling up, and a five-minute window with gentle steps when scaling down.
Scaling on the queue ties your replica count to the user experience, and it keeps expensive capacity from sitting idle.
Run it yourself
This is the exact example from the video. A small Python model of the autoscaler's rule, using the lesson's numbers. It is not the real HPA.
import math
target, low, high = 10, 1, 4 # waiting requests per pod
window = 5 # minutes, like 300 seconds
recs = []
for minute, waiting in enumerate([5, 30, 60, 20, 5, 5, 5, 5]):
want = math.ceil(waiting / target)
recs.append(min(high, max(low, want)))
pods = max(recs[-window:])
print(f"min {minute}: waiting {waiting:2d} -> {pods} pods")
Run it:
python hpa.py
Output I got while recording the video:
min 0: waiting 5 -> 1 pods
min 1: waiting 30 -> 3 pods
min 2: waiting 60 -> 4 pods
min 3: waiting 20 -> 4 pods
min 4: waiting 5 -> 4 pods
min 5: waiting 5 -> 4 pods
min 6: waiting 5 -> 4 pods
min 7: waiting 5 -> 2 pods
Coming Up Next
Day 13: Helm charts: packaging, versioning and updating AI deployments.
#HPA #Autoscaling #MLOps #Prometheus #CloudCostOptimization #SRE #Kubernetes