Meta SAM 2 / SAM 2.1 Video & Image Segmentation: Memory Optimization & TensorRT Acceleration

✂️ SAM 2 Spatiotemporal Memory Attention Architecture

SAM 2 introduces Memory Attention: During the video processing process, the model temporarily stores the interactive clicks and mask feature encodings of past frames into the spatiotemporal memory queue. Even if the target is severely occluded, deformed, or exits and re-enters the video, accurate and continuous tracking can be achieved.

⚡ Image pull and video segmentation tracking call

Download SAM 2.1 weights

export HF_ENDPOINT="https://hf-mirror.net"
huggingface-cli download facebook/sam2.1-hiera-large \
  --local-dir /data/models/sam2.1-hiera-large \
  --local-dir-use-symlinks False

sam2_video_track.py: interactive tracking code

from sam2.build_sam import build_sam2_video_predictor

checkpoint = "/data/models/sam2.1-hiera-large/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = build_sam2_video_predictor(model_cfg, checkpoint)

# Initialize video prediction state
inference_state = predictor.init_state(video_path="input_video_frames/")
# Click the prompt target at frame 0
_, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=0,
    obj_id=1,
    points=[[210, 350]],
    labels=[1]
)
print("SAM 2 first frame prompts success, prepare to mask the entire video for dissemination!")