Zhipu GLM-4-9B-Chat: 1M Context Window Inference, Function Calling & vLLM Setup

📜 GLM-4-9B-Chat’s 1M context and comprehensive capabilities

GLM-4-9B-Chat With 9 billion compact parameters, it comprehensively surpasses many old 13B~34B models. It natively supports the input of long documents up to 1,000,000 Tokens (equivalent to two full texts of "Dream of Red Mansions"), and demonstrates excellent command compliance rate in complex structured data parsing and tool external calling (Tool Use).

⚡ Image pulling and Function Calling structured output

Download GLM-4-9B-Chat weights

export HF_ENDPOINT="https://hf-mirror.net"
huggingface-cli download THUDM/glm-4-9b-chat \
  --local-dir /data/models/glm-4-9b-chat \
  --local-dir-use-symlinks False

tool_call_demo.py: Practical operation of tool function call

from transformers import AutoModelForCausalLM, AutoTokenizer

path = "/data/models/glm-4-9b-chat"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype="auto", device_map="auto", trust_remote_code=True)

# Pass in a prompt containing the custom tool definition
tools = [{
    "name": "get_stock_price",
    "description": "Get current market quotes for stocks",
    "parameters": {"type": "object", "properties": {"symbol": {"type": "string"}}, "required": ["symbol"]}
}]

inputs = tokenizer.apply_chat_template([
    {"role": "user", "content": "Please help me check the current real-time stock price of Tencent Holdings (00700.HK)."}
], tools=tools, return_tensors="pt", add_generation_prompt=True).to("cuda")

output = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(output[0]))