Kubernetes AI Cluster Architecture: Model Pre-Loading, Shared PV/PVC Storage & InitContainer Sync

☸️ Cold start challenges of cloud-native large-scale AI clusters

In a modern cloud-native environment (Kubernetes/K8s), when a business traffic burst triggers HPA (Horizontal Pod Auto-scaling), the newly scheduled GPU Worker Pod must be ready in the shortest time.

If a cluster with 100 GPUs automatically expands to 20 Pods, each Pod pulls 70GB of weight from the public network. Not only will the computer room's export bandwidth be completely full in an instant, but it will also encounter strict concurrency limits from official sources (HTTP 429 Too Many Requests), causing the Pod to be in a long-term CrashLoopBackOff.

🏗️ Three-tier architecture design for cloud native model distribution

1. Distributed shared storage (RWX)

Build a cross-node ReadWriteMany storage pool based on CephFS, JuiceFS or enterprise-level high-performance NFS. The model only needs to be pulled globally once to be visible to the entire cluster.

2. Node local NVMe cache

There is an upper limit on distributed storage network IOPS. Use DaemonSet to distribute high-frequency hotspot models to the local NVMe SSD (Local PV) of each GPU node to achieve microsecond-level weighted mmap loading.

3. InitContainer verification guard

The local weight integrity hash is checked by the lightweight Init container before the main inference container starts. If it is not ready, use the HF-Mirror domestic mirror channel for quick completion.

📝 Production-grade K8s deployment manifest (Manifest)

Here is an industry-proven production Kubernetes deployment manifest, including PVC declarations and InitContainers with image-accelerated pull logic:

k8s-deployment.yaml: InitContainer + RWX PVC

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference-deepseek
  namespace: ai-serving
spec:
  replicas: 4
  selector:
    matchLabels:
      app: deepseek-service
  template:
    metadata:
      labels:
        app: deepseek-service
    spec:
      # 1. Preload initialization container: ensure model integrity through mirror station
      initContainers:
      - name: model-preloader
        image: python:3.11-slim
        command: ["/bin/sh", "-c"]
        args:
          - |
            pip install huggingface_hub
            export HF_ENDPOINT="https://hf-mirror.net"
            python -c "
            from huggingface_hub import snapshot_download
            snapshot_download(
                repo_id='deepseek-ai/DeepSeek-R1-Distill-Qwen-32B',
                local_dir='/models/DeepSeek-R1-Distill-Qwen-32B',
                local_dir_use_symlinks=False
            )
            "
        volumeMounts:
        - name: shared-model-storage
          mountPath: /models

      # 2. Core inference service container: read-only loading, extremely fast readiness
      containers:
      - name: vllm-engine
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: 2
            memory: 64Gi
            cpu: "16"
        env:
        - name: HF_HUB_OFFLINE
          value: "1"
        command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
        args:
          - "--model=/models/DeepSeek-R1-Distill-Qwen-32B"
          - "--tensor-parallel-size=2"
          - "--port=8000"
        volumeMounts:
        - name: shared-model-storage
          mountPath: /models
          readOnly: true

      volumes:
      - name: shared-model-storage
        persistentVolumeClaim:
          claimName: nfs-model-pvc