Running LLM Workloads on Kubernetes — GPU Scheduling, vLLM, and Autoscaling

Introduction

Kubernetes orchestrates stateless services beautifully. LLM inference is different: GPUs are scarce, models consume gigabytes of memory, and batching requests yields 10x throughput gains. Running LLM workloads on Kubernetes requires dedicated GPU node pools, careful resource requests, and autoscaling beyond CPU metrics. The payoff is cost-effective, highly available inference at scale.

GPU Node Pools in EKS, GKE, and AKS

Create dedicated GPU node pools separate from CPU workloads. Mixing workloads wastes expensive GPU capacity on pods that do not need it.

EKS (AWS):

# Create on-demand GPU node group
aws eks create-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name gpu-nodes-a100 \
  --subnets subnet-12345 \
  --instance-types p4d.24xlarge \
  --desired-size 2 \
  --min-size 1 \
  --max-size 5 \
  --tags "workload=gpu-inference"
 
# Create spot GPU node group (70% cheaper, interruptible)
aws eks create-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name gpu-spot-v100 \
  --subnets subnet-12345 \
  --instance-types p3.8xlarge \
  --capacity-type SPOT \
  --desired-size 3

GKE (Google Cloud):

gcloud container node-pools create gpu-pool \
  --cluster my-cluster \
  --num-nodes 2 \
  --machine-type n1-standard-8 \
  --accelerator type=nvidia-tesla-a100,count=2 \
  --enable-autoscaling \
  --min-nodes 1 \
  --max-nodes 10

AKS (Azure):

az aks nodepool add \
  --resource-group rg-name \
  --cluster-name aks-cluster \
  --name gpunodes \
  --node-count 2 \
  --node-vm-size Standard_NC24ads_A100_v4 \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 10

Taints on GPU nodes prevent CPU-only pods from consuming GPU capacity accidentally.

NVIDIA Device Plugin for Kubernetes

Install the NVIDIA Device Plugin to expose GPUs as schedulable Kubernetes resources. Without it, the scheduler cannot see GPU availability.

# Install the plugin
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.0/nvidia-device-plugin.yml
 
# Verify GPUs are visible to the scheduler
kubectl get nodes -o wide
# GPU nodes should show nvidia.com/gpu: 8 in their capacity

The plugin discovers GPUs on each node and registers them with kubelet. Pods can then request nvidia.com/gpu as a resource.

vLLM Deployment on Kubernetes

vLLM serves large language models with continuous batching and KV cache optimization, delivering significantly higher throughput than naive serving.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama-70b
  namespace: llm-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-server
      model: llama-70b
  template:
    metadata:
      labels:
        app: vllm-server
        model: llama-70b
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: workload
                operator: In
                values:
                - gpu-inference
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - "--model=/models/meta-llama/Llama-2-70b-hf"
          - "--tensor-parallel-size=8"
          - "--gpu-memory-utilization=0.9"
          - "--max-model-len=4096"
        ports:
        - name: http
          containerPort: 8000
        resources:
          requests:
            nvidia.com/gpu: "8"
            memory: "100Gi"
            cpu: "16"
          limits:
            nvidia.com/gpu: "8"
            memory: "120Gi"
            cpu: "16"
        livenessProbe:
          httpGet:
            path: /v1/models
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        volumeMounts:
        - name: model-cache
          mountPath: /models
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-storage
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-llama-70b
  namespace: llm-inference
spec:
  selector:
    app: vllm-server
    model: llama-70b
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8000

Tensor parallelism distributes computation across all 8 GPUs. GPU memory utilization at 90% maximizes throughput without OOM errors. The remaining 10% headroom handles activation spikes.

Horizontal Pod Autoscaling for Inference

Scale inference pods based on GPU utilization or request queue depth rather than CPU, which is rarely the bottleneck for GPU workloads.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-queue-scaler
  namespace: llm-inference
spec:
  scaleTargetRef:
    name: vllm-llama-70b
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: vllm_request_queue_length
      threshold: "50"
      query: |
        rate(vllm_request_queue_length[1m])

KEDA (Kubernetes Event Driven Autoscaling) queries Prometheus for queue depth. When the queue exceeds 50 requests, new replicas spawn within seconds. This prevents latency spikes during traffic surges.

Resource Requests and Limits for GPU Pods

Specify GPU, memory, and CPU carefully. Requests drive scheduling decisions; limits enforce runtime boundaries.

resources:
  requests:
    nvidia.com/gpu: "8"         # 8 GPUs required for tensor parallel
    memory: "100Gi"             # Model weights + KV cache
    cpu: "16"                   # CPU for tokenization and I/O
    ephemeral-storage: "50Gi"   # Temp files during inference
  limits:
    nvidia.com/gpu: "8"
    memory: "120Gi"             # 20Gi headroom above request
    cpu: "16"

Unlike CPU, GPU resources are not compressible. Setting requests equal to limits for GPU ensures the pod is Guaranteed QoS class and will not be evicted under memory pressure.

Spot GPU Instances for Cost Reduction

Mix on-demand and spot instances to cut GPU costs by 60 to 70 percent. Spot instances can be reclaimed with two minutes notice, so configure pod disruption budgets.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-pdb
  namespace: llm-inference
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm-server

When a spot instance is reclaimed, the PDB ensures at least one inference replica remains available while pods migrate to on-demand nodes. In-flight requests drain gracefully using a preStop lifecycle hook with a sleep before termination.

Monitoring GPU Utilisation With DCGM

NVIDIA's DCGM (Data Center GPU Manager) exports GPU metrics to Prometheus. Install the exporter on all GPU nodes.

helm repo add nvidia https://nvidia.github.io/gpu-monitoring-tools/helm-charts
helm install dcgm-exporter nvidia/dcgm-exporter --namespace monitoring

Key metrics to track in Grafana dashboards:

GPU utilization: target 80 to 90 percent for maximum efficiency
GPU memory usage: alert above 92 percent to prevent OOM
Tensor core utilization: indicates whether the workload is compute-bound or memory-bound
Inference latency P95: primary SLO metric for end users
Request queue depth: triggers autoscaling decisions

Multi-Model Serving With Request Routing

Route requests to different models based on task requirements. A lightweight 7B model handles simple queries; a 70B model handles complex reasoning.

const express = require('express')
const axios = require('axios')
const app = express()
 
app.use(express.json())
 
app.post('/infer', async (req, res) => {
  const { prompt, model_type } = req.body
 
  let serviceUrl
  if (model_type === 'fast') {
    serviceUrl = 'http://vllm-mistral-7b:8000'
  } else if (model_type === 'powerful') {
    serviceUrl = 'http://vllm-llama-70b:8000'
  } else {
    serviceUrl = 'http://vllm-llama-13b:8000'
  }
 
  const response = await axios.post(`${serviceUrl}/v1/completions`, {
    model: 'default',
    prompt,
    max_tokens: 256,
  })
 
  return res.json(response.data)
})
 
app.listen(3000)

This pattern is simpler than a service mesh for most use cases. Add latency-based routing later if needed.

Key Takeaways

Dedicated GPU node pools prevent CPU workloads from consuming scarce GPU capacity; use taints and tolerations to enforce isolation.
Install the NVIDIA Device Plugin before scheduling any GPU workload or the scheduler cannot see GPU availability.
vLLM with tensor parallelism delivers 3 to 5x higher throughput than naive model serving through continuous batching and KV cache reuse.
KEDA queue-depth autoscaling outperforms CPU-based HPA for inference: scale on request backlog, not on CPU utilization.
Spot GPU instances reduce costs 60 to 70 percent with PodDisruptionBudgets ensuring graceful failover to on-demand capacity.
DCGM exporter is the correct tool for GPU observability — track utilization, memory, and temperature in Grafana.
Set GPU requests equal to limits for Guaranteed QoS to prevent eviction under node memory pressure.
Model routing routes cheap queries to small models and expensive queries to large models, cutting inference costs without degrading quality.

Introduction#

GPU Node Pools in EKS, GKE, and AKS#

NVIDIA Device Plugin for Kubernetes#

vLLM Deployment on Kubernetes#

Horizontal Pod Autoscaling for Inference#

Resource Requests and Limits for GPU Pods#

Spot GPU Instances for Cost Reduction#

Monitoring GPU Utilisation With DCGM#

Multi-Model Serving With Request Routing#

Key Takeaways#

Related reading