Zhipu CogVideoX-5B: 3D Variational Autoencoder, Diffusers Pipeline & LoRA Fine-Tuning

📹 CogVideoX core space-time compression mechanism

CogVideoX-5B uses a customized 3D variational autoencoder (3D VAE) to simultaneously compress video data in both time and space dimensions, with a compression rate as high as 2x8x8. At the same time, it uses the expert Transformer backbone network to understand extremely complex physical motion trajectory prompts.

⚡ Image pulling and very low memory inference

Download CogVideoX-5B weights

export HF_ENDPOINT="https://hf-mirror.net"
huggingface-cli download THUDM/CogVideoX-5b \
  --local-dir /data/models/CogVideoX-5b \
  --local-dir-use-symlinks False

cogvideo_pipeline.py: production calling code

import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

pipe = CogVideoXPipeline.from_pretrained("/data/models/CogVideoX-5b", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()

video = pipe(
    prompt="A soaring eagle gliding gracefully over snow-capped mountains at sunset.",
    num_videos_per_prompt=1,
    num_inference_steps=50,
    guidance_scale=6
).frames[0]

export_to_video(video, "output_video.mp4", fps=8)
print("Video generation completed! Exported to output_video.mp4")