Kubernetes Persistent Volumes 2025 — Storage for Stateful Workloads

Introduction

Why This Matters

Kubernetes was designed for stateless workloads, but real applications need stateful storage — databases, message queues, object stores, and log files. Without proper storage management, data is lost every time a pod restarts or is rescheduled to a different node.

Kubernetes Persistent Volumes decouple storage from pod lifecycle. A database pod can be killed, rescheduled to a different node, and come back using the same data — if storage is configured correctly. StorageClasses and dynamic provisioning make this automatic on cloud platforms like AWS EKS, GKE, and AKS.

Understanding Kubernetes storage is essential for anyone running databases (PostgreSQL, MySQL, MongoDB, Redis) or message queues (Kafka, RabbitMQ) on Kubernetes.

Storage Concepts

PersistentVolume (PV): A piece of storage provisioned in the cluster (by an admin or dynamically by a StorageClass). Independent of any specific pod.

PersistentVolumeClaim (PVC): A request for storage by a pod. Specifies size, access mode, and StorageClass. Kubernetes binds PVCs to matching PVs.

StorageClass: Defines the provisioner (AWS EBS, GCE PD, NFS) and parameters. Enables dynamic provisioning — PVs are created automatically when a PVC is applied.

Access Modes:

ReadWriteOnce (RWO) — one node can mount read/write (EBS, standard disks)
ReadOnlyMany (ROX) — multiple nodes can mount read-only
ReadWriteMany (RWX) — multiple nodes can mount read/write (EFS, NFS, CephFS)

StorageClasses

# AWS EBS StorageClass (gp3 — faster than gp2, same cost)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer  # provision in the same AZ as the pod
reclaimPolicy: Delete  # Delete PV when PVC is deleted (use Retain for production databases)
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"

# AWS EFS StorageClass (ReadWriteMany — shared across pods/nodes)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-0123456789abcdef0
  directoryPerms: "700"

# List available storage classes
kubectl get storageclass
kubectl describe storageclass ebs-gp3

PersistentVolumeClaims

# PVC for a PostgreSQL database
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: production
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: ebs-gp3
  resources:
    requests:
      storage: 50Gi

kubectl apply -f pvc.yaml
kubectl get pvc -n production
# STATUS: Bound (PV has been provisioned and bound)
# VOLUME: pvc-abc123 (the auto-created PV name)
 
kubectl describe pvc postgres-data -n production

Using PVCs in Pods

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: production
spec:
  replicas: 1  # databases should be StatefulSets, not Deployments
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16-alpine
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 2
            memory: 4Gi
      volumes:
      - name: postgres-data
        persistentVolumeClaim:
          claimName: postgres-data

StatefulSets — The Right Way for Stateful Apps

StatefulSets are designed for stateful workloads. They provide:

Stable, predictable pod names (pod-0, pod-1, pod-2)
Ordered startup and shutdown (pod-0 first, then pod-1, etc.)
Stable network identities (DNS: pod-0.service-name.namespace.svc.cluster.local)
Per-pod PVCs via volumeClaimTemplates

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: production
spec:
  serviceName: postgres  # headless service for stable DNS
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16-alpine
        env:
        - name: POSTGRES_USER
          value: app
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: POSTGRES_DB
          value: mydb
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 2
            memory: 4Gi
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "app", "-d", "mydb"]
          initialDelaySeconds: 15
          periodSeconds: 5
  # Each pod gets its own PVC automatically
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: ebs-gp3
      resources:
        requests:
          storage: 50Gi

# Headless Service for StatefulSet DNS
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: production
spec:
  selector:
    app: postgres
  clusterIP: None  # headless — no load balancing, direct pod DNS
  ports:
  - port: 5432
    targetPort: 5432

Expand PVC Storage

# Edit the PVC to increase storage (only works with allowVolumeExpansion: true in StorageClass)
kubectl patch pvc postgres-data -n production \
  -p '{"spec": {"resources": {"requests": {"storage": "100Gi"}}}}'
 
kubectl get pvc postgres-data -n production
# SIZE will show 100Gi once expansion is complete

Backup Strategies

VolumeSnapshot (CSI)

# Create a snapshot (requires CSI driver with snapshot support)
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-snapshot-20260301
  namespace: production
spec:
  volumeSnapshotClassName: csi-aws-vsc
  source:
    persistentVolumeClaimName: data-postgres-0
 
---
# Restore from snapshot into new PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-restored
  namespace: production
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ebs-gp3
  resources:
    requests:
      storage: 50Gi
  dataSource:
    name: postgres-snapshot-20260301
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

pg_dump for PostgreSQL

# Logical backup (works across versions)
kubectl exec -it postgres-0 -n production -- \
  pg_dump -U app mydb | gzip > postgres-backup-$(date +%Y%m%d).sql.gz
 
# Restore
gunzip -c postgres-backup-20260301.sql.gz | \
  kubectl exec -i postgres-0 -n production -- \
  psql -U app mydb

Reclaim Policies

# Delete (default): PV deleted when PVC deleted — use for ephemeral data
reclaimPolicy: Delete
 
# Retain: PV preserved when PVC deleted — use for production databases
reclaimPolicy: Retain
 
# To reuse a Retain PV after PVC deletion:
# 1. Delete the PV's claimRef
kubectl patch pv <pv-name> -p '{"spec":{"claimRef": null}}'
# 2. Create a new PVC that references this PV

Common Mistakes

Using a Deployment (not a StatefulSet) for databases — Deployments do not guarantee stable pod names or ordered scaling
Not setting reclaimPolicy: Retain for production databases — Delete removes data when the PVC is deleted
Using ReadWriteMany when ReadWriteOnce is available and sufficient — RWX (EFS, NFS) has higher latency than RWO (EBS)
Not specifying PGDATA to a subdirectory inside the mount — PostgreSQL fails to initialize in a non-empty directory
Forgetting to configure allowVolumeExpansion: true in the StorageClass — blocks PVC expansion later

Best Practices

Use StatefulSets for all stateful workloads — databases, Kafka, ZooKeeper, Elasticsearch
Set reclaimPolicy: Retain for any StorageClass used by production databases
Use WaitForFirstConsumer volume binding mode to provision storage in the same availability zone as the pod
Automate backups with VolumeSnapshots or logical backup jobs (CronJobs with pg_dump)
Monitor PVC usage with kubelet_volume_stats_used_bytes in Prometheus — add alerts at 80% and 90% usage

Key Takeaways

PersistentVolumes (PV) are storage resources; PersistentVolumeClaims (PVC) are requests for storage — StorageClasses automate PV creation
StatefulSets are the correct controller for databases — they provide stable pod names, ordered scaling, and per-pod PVCs
ReadWriteOnce (EBS, standard disks) is for single-pod access; ReadWriteMany (EFS, NFS) enables shared access across pods and nodes
Set reclaimPolicy: Retain on production StorageClasses — the default Delete removes data when PVCs are removed
WaitForFirstConsumer volume binding ensures storage is provisioned in the same availability zone as the scheduled pod
VolumeSnapshots enable point-in-time backups of PVC data using the CSI snapshot API
Monitor PVC disk usage with Prometheus — without monitoring, PVCs silently fill up and crash databases
PVC storage expansion requires the StorageClass to have allowVolumeExpansion: true — plan for this from day one

Introduction#

Why This Matters#

Storage Concepts#

StorageClasses#

PersistentVolumeClaims#

Using PVCs in Pods#

StatefulSets — The Right Way for Stateful Apps#

Expand PVC Storage#

Backup Strategies#

VolumeSnapshot (CSI)#

pg_dump for PostgreSQL#

Reclaim Policies#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading