AI at Scale: Kubernetes + vLLM 🚢

Naive LLM serving allocates a contiguous KV-cache slot per request, sized for the worst case (max output length). Most requests use a fraction of it. Result: 60–80% of GPU memory is wasted, and you can only batch 8–16 concurrent requests on an A10 GPU.

vLLM's two innovations:

# 1. Start vLLM server with Llama-3-8B
docker run --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.92

vLLM exposes an OpenAI-compatible API. Existing OpenAI client code works with one line change:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy",  # vLLM doesn't check keys by default
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a Hindi study tutor."},
        {"role": "user", "content": "प्रकाश संश्लेषण समझाइए"},
    ],
    temperature=0.7,
    max_tokens=400,
)
print(response.choices[0].message.content)

# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3
spec:
  replicas: 2
  selector:
    matchLabels: {app: vllm}
  template:
    metadata:
      labels: {app: vllm}
    spec:
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-A10
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model=meta-llama/Meta-Llama-3-8B-Instruct
        - --max-model-len=4096
        - --gpu-memory-utilization=0.92
        ports: [{containerPort: 8000}]
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: "32Gi"
        readinessProbe:
          httpGet: {path: /health, port: 8000}
          initialDelaySeconds: 120  # vLLM takes ~2 min to load 8B model
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: Pods
    pods:
      metric: {name: vllm_requests_running}
      target: {type: AverageValue, averageValue: "60"}

kubectl apply -f vllm-deployment.yaml
kubectl get pods -l app=vllm
# vllm-llama3-7c6f4d89b8-x2k4p  Running