Networking & Storage

Persistent storage for datasets and model checkpoints, Ingress for exposing model APIs, NetworkPolicies for securing ML pipelines, and StorageClasses for different workload needs.

Persistent Storage for ML

ML workloads need persistent storage for datasets, model checkpoints, training logs, and saved models. Kubernetes provides PersistentVolumes (PV) and PersistentVolumeClaims (PVC) to decouple storage from Pods.

PersistentVolumeClaim for a Dataset

# PVC for training dataset
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: imagenet-dataset
  namespace: ml-training
spec:
  accessModes:
    - ReadOnlyMany       # Multiple training Pods can read simultaneously
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 500Gi

Access Modes for ML

• ReadWriteOnce (RWO) — Single node read-write. Use for model checkpoints where only one training Pod writes.
• ReadOnlyMany (ROX) — Multiple nodes can read. Ideal for shared datasets that multiple training jobs access simultaneously.
• ReadWriteMany (RWX) — Multiple nodes can read and write. Use for shared model registries or experiment logs. Requires NFS or a distributed filesystem.

StorageClasses

StorageClasses define different tiers of storage. For ML workloads, you typically need at least two classes.

# Fast SSD storage for training data
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iopsPerGB: "50"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
# Standard storage for model archives
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-hdd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: st1
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Ingress for Model APIs

Ingress exposes HTTP/HTTPS routes from outside the cluster to Services inside. For ML, Ingress is used to expose model serving endpoints with TLS, path-based routing, and load balancing.

# Ingress for multiple model endpoints
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ml-api-ingress
  namespace: ml-serving
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.ml.example.com
    secretName: ml-api-tls
  rules:
  - host: api.ml.example.com
    http:
      paths:
      - path: /v1/sentiment
        pathType: Prefix
        backend:
          service:
            name: sentiment-model-svc
            port:
              number: 80
      - path: /v1/translate
        pathType: Prefix
        backend:
          service:
            name: translation-model-svc
            port:
              number: 80
      - path: /v1/embeddings
        pathType: Prefix
        backend:
          service:
            name: embedding-model-svc
            port:
              number: 80

NetworkPolicies for ML Security

NetworkPolicies control traffic flow between Pods. In ML environments, use them to isolate training from serving, restrict access to data stores, and secure inter-service communication.

# Allow only serving Pods to access the model store
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: model-store-access
  namespace: ml-serving
spec:
  podSelector:
    matchLabels:
      role: model-store
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: inference-server
    ports:
    - port: 9000
      protocol: TCP

Common ML Network Policies

• Isolate training namespace — Prevent training Pods from accessing the internet (data exfiltration prevention)
• Restrict data access — Only authorized Pods can access dataset storage
• Allow distributed training — Permit inter-Pod communication within a training Job for gradient synchronization
• Expose serving only — Only model serving Pods can receive external traffic via Ingress

Volume Mounts in ML Pods

# Training Pod with multiple volume mounts
apiVersion: v1
kind: Pod
metadata:
  name: training-pod
spec:
  containers:
  - name: trainer
    image: myregistry/trainer:v1.0
    volumeMounts:
    - name: dataset
      mountPath: /data
      readOnly: true        # Dataset is read-only
    - name: checkpoints
      mountPath: /checkpoints  # Save model checkpoints
    - name: logs
      mountPath: /logs         # TensorBoard logs
    - name: config
      mountPath: /config       # Training configuration
  volumes:
  - name: dataset
    persistentVolumeClaim:
      claimName: imagenet-dataset
  - name: checkpoints
    persistentVolumeClaim:
      claimName: checkpoint-pvc
  - name: logs
    emptyDir: {}             # Ephemeral, lost when Pod dies
  - name: config
    configMap:
      name: training-config

Practice Questions