Hugging Face Datasets Mirror Guide: Fast load_dataset, Parquet Shards & datasets-server Alternative
Accelerate Hugging Face datasets with HF_ENDPOINT=https://hf-mirror.net. Download Parquet shards via CLI, stream load_dataset, and avoid datasets-server limits.
π Why Not Poll datasets-server /rows API for Bulk Data?
Many engineers searching for Hugging Face dataset endpoints look for datasets-server or the /rows viewer API. In practice, datasets-server is designed strictly for web browser previewsβit enforces strict HTTP 429 rate limits, paginates only small batches of JSON rows, and is not mirrored by file-level gateways.
For training and evaluation pipelines, nearly every public Hugging Face dataset exposes raw Parquet, JSONL, or CSV shards under /datasets/<org>/<repo>/resolve/main/. By setting HF_ENDPOINT="https://hf-mirror.net", both huggingface-cli and the Python datasets library stream those binary shards directly through Anycast edge PoPs with full HTTP Range resume support.
β‘ Method 1: Bulk Download Parquet Shards with huggingface-cli
For multi-gigabyte corpora (such as FineWeb-Edu, UltraChat, or OpenMathInstruct), downloading selected Parquet shards directly to local NVMe or shared cluster storage with resumable CLI transfers is the most reliable production workflow:
# 1. Point HF_ENDPOINT to HF-Mirror
export HF_ENDPOINT="https://hf-mirror.net"
#
# 2. Pass --repo-type dataset and filter specific Parquet shards with --include
huggingface-cli download \
--repo-type dataset \
--resume-download \
HuggingFaceFW/fineweb-edu \
--include "sample/10BT/*.parquet" \
--local-dir /data/datasets/fineweb-edu-10BT \
--local-dir-use-symlinks False
--repo-type dataset causes the CLI to query the /models namespace and fail with 404 Repository Not Found.
π Method 2: Python load_dataset() & Zero-Disk Streaming via Mirror
When calling datasets.load_dataset() inside Python scripts, set os.environ["HF_ENDPOINT"] before importing datasets so the underlying huggingface_hub client routes all file resolution calls to hf-mirror.net:
import os
# Must be injected BEFORE importing datasets or huggingface_hub
os.environ["HF_ENDPOINT"] = "https://hf-mirror.net"
#
from datasets import load_dataset
#
# 1. Download and cache locally under ~/.cache/huggingface/datasets
ds = load_dataset("fka/awesome-chatgpt-prompts", split="train")
print(f"Loaded {len(ds)} rows; columns: {list(ds[0].keys())}")
#
# 2. Stream massive datasets without exhausting local disk space
stream_ds = load_dataset(
"HuggingFaceFW/fineweb-edu",
name="sample-10BT",
split="train",
streaming=True
)
for i, sample in enumerate(stream_ds.take(3)):
print(f"Row {i}: {sample['text'][:120]}...")
πΎ Loading Local Parquet Shards in Strict Offline Mode
In distributed multi-node training, sync Parquet files once to shared storage and force offline loading so worker ranks never stall on redundant upstream metadata checks:
import os
os.environ["HF_DATASETS_OFFLINE"] = "1"
#
from datasets import load_dataset
#
# Load local Parquet shards downloaded via huggingface-cli
local_ds = load_dataset(
"parquet",
data_files="/data/datasets/fineweb-edu-10BT/sample/10BT/*.parquet",
split="train"
)
print(local_ds)
Paste any huggingface.co model, GGUF, or dataset URL into our converter to generate ready-to-run huggingface-cli, cURL, Wget, and Python commands.
π Related Engineering Guides
Why legacy Hugging Face mirrors drop large downloads with Connection Reset: comparing 302 S3 redirects against HF-Mirror Anycast HTTP Range streaming.
Use Jina Embeddings v3 task-specific LoRA adapters for retrieval, clustering, and code search with 8192 context and Matryoshka dimension reduction.
Eliminate slow GPU pod cold starts in Kubernetes. Design shared NFS/Ceph/JuiceFS ReadWriteMany PVCs, InitContainer mirror sync, and NVMe model caching.
Run Microsoft Phi-4 14B reasoning model locally with Ollama GGUF and ONNX Runtime. Fast HF-Mirror download commands, quantization choices, and benchmarks.