HF-Mirror Logo
HF-Mirror Mirror Gateway
🌐 δΈ­ζ–‡
Datasets Proxy πŸ•’ Reading Time: ~10 min read πŸ“… Updated 2026-11

Hugging Face Datasets Mirror Guide: Fast load_dataset, Parquet Shards & datasets-server Alternative

Accelerate Hugging Face datasets with HF_ENDPOINT=https://hf-mirror.net. Download Parquet shards via CLI, stream load_dataset, and avoid datasets-server limits.

πŸ“Š Why Not Poll datasets-server /rows API for Bulk Data?

Many engineers searching for Hugging Face dataset endpoints look for datasets-server or the /rows viewer API. In practice, datasets-server is designed strictly for web browser previewsβ€”it enforces strict HTTP 429 rate limits, paginates only small batches of JSON rows, and is not mirrored by file-level gateways.

For training and evaluation pipelines, nearly every public Hugging Face dataset exposes raw Parquet, JSONL, or CSV shards under /datasets/<org>/<repo>/resolve/main/. By setting HF_ENDPOINT="https://hf-mirror.net", both huggingface-cli and the Python datasets library stream those binary shards directly through Anycast edge PoPs with full HTTP Range resume support.

⚑ Method 1: Bulk Download Parquet Shards with huggingface-cli

For multi-gigabyte corpora (such as FineWeb-Edu, UltraChat, or OpenMathInstruct), downloading selected Parquet shards directly to local NVMe or shared cluster storage with resumable CLI transfers is the most reliable production workflow:

CLI mirror download for dataset Parquet shards
# 1. Point HF_ENDPOINT to HF-Mirror
export HF_ENDPOINT="https://hf-mirror.net"
#
# 2. Pass --repo-type dataset and filter specific Parquet shards with --include
huggingface-cli download \
  --repo-type dataset \
  --resume-download \
  HuggingFaceFW/fineweb-edu \
  --include "sample/10BT/*.parquet" \
  --local-dir /data/datasets/fineweb-edu-10BT \
  --local-dir-use-symlinks False
Common Pitfall: Omitting --repo-type dataset causes the CLI to query the /models namespace and fail with 404 Repository Not Found.

🐍 Method 2: Python load_dataset() & Zero-Disk Streaming via Mirror

When calling datasets.load_dataset() inside Python scripts, set os.environ["HF_ENDPOINT"] before importing datasets so the underlying huggingface_hub client routes all file resolution calls to hf-mirror.net:

Accelerating load_dataset and streaming mode in Python
import os
# Must be injected BEFORE importing datasets or huggingface_hub
os.environ["HF_ENDPOINT"] = "https://hf-mirror.net"
#
from datasets import load_dataset
#
# 1. Download and cache locally under ~/.cache/huggingface/datasets
ds = load_dataset("fka/awesome-chatgpt-prompts", split="train")
print(f"Loaded {len(ds)} rows; columns: {list(ds[0].keys())}")
#
# 2. Stream massive datasets without exhausting local disk space
stream_ds = load_dataset(
    "HuggingFaceFW/fineweb-edu",
    name="sample-10BT",
    split="train",
    streaming=True
)
for i, sample in enumerate(stream_ds.take(3)):
    print(f"Row {i}: {sample['text'][:120]}...")

πŸ’Ύ Loading Local Parquet Shards in Strict Offline Mode

In distributed multi-node training, sync Parquet files once to shared storage and force offline loading so worker ranks never stall on redundant upstream metadata checks:

Zero-network local Parquet ingestion
import os
os.environ["HF_DATASETS_OFFLINE"] = "1"
#
from datasets import load_dataset
#
# Load local Parquet shards downloaded via huggingface-cli
local_ds = load_dataset(
    "parquet",
    data_files="/data/datasets/fineweb-edu-10BT/sample/10BT/*.parquet",
    split="train"
)
print(local_ds)
⚑ Need Instant Mirror Download Commands for This Model?

Paste any huggingface.co model, GGUF, or dataset URL into our converter to generate ready-to-run huggingface-cli, cURL, Wget, and Python commands.

Open HF Link Converter β†’

πŸ”— Related Engineering Guides

← Articles Index Explore All 34 Open-Source AI Engineering Guides Quick Start &rarr; 5-Minute Zero-Friction Setup Tutorial