---
language:
- zh
pipeline_tag: text-generation
license: apache-2.0
library_name: transformers
tags:
- guardrail
- safeguard
- taiwan
- traditional_chinese
- guard
- safety
- security
---

# TWGuard — Taiwan LLM guardrail

(English below) 

TWGuard 是一個推論安全研究計畫，旨在提出適用於臺灣語境，的大語言模型安全護欄（LLM guardrail），以滿足 AI 技術落地安全合規需求。

藉由高品質真實語料及人工標注，我們提出一個能反映臺灣語境的安全護欄模型，能攔截偵測各種帶有有害意圖的 LLM 提示詞（Prompt），例如犯罪、違法專業建議、色情、兒童性剝削等。

在評測中我們的模型顯著優於 Nvidia、Google、Meta、OpenAI、Alibaba 等 AI lab 的最佳模型（SOTA）。

我們的研究著重於反映臺灣語境情境，以期成果能適用於真實案場：
1. 基於語境設計：多數研究著重於中文支援，然而實際語境存在 code-mixing，例如臺灣常見注音文、網路用語、本土語言並用的情況。測試顯示 TWGuard 可以更少成本資料達成比支援中文模型更佳效果。
3. 反映真實語境：TWGuard 在基於真實語料庫的評測結果良好，相較之下，一般使用模擬合成語料訓練的模型常有效能衰退的狀況。
4. 低誤報率（FPR）：TWGuard 在 FPR 方面有良好的控制，由於誤報在許多情境中的處理成本遠高於漏報，此指標對於實際案場技術選型至關重要。

此 repo以符合學術慣例、提升論文可重現性、促進後續研究等目的建立，詳細內容、技術細節及侷限請參考下文及論文本文。

- **Paper:** [TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contexts](https://arxiv.org/abs/2604.16542)
- **Developers:** 中華電信研究院 / Chunghwa Telecom Laboratories。若有任何問題請以論文作者電子信箱聯絡。
- **Keywords:** 正體中文, 繁體中文, 臺灣中文, 臺灣漢語, 臺灣華語, Tradtional Chinese, Standard Chinese, Taiwan Mandarin
 
## Usage

This repository provides a Parameter-Efficient Fine-Tuned (PEFT/LoRA) adapter designed for Taiwanese linguistic context harmful intention detection.

**Warning: may contain explicit and harmful examples.**

**Have a try in [Kaggle](https://www.kaggle.com/code/petertcchu/twguard-demo).**

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

model_id = "meta-llama/Llama-Guard-3-8B"
adapter_id = "chunghwa-telecom/twguard"
device = "cuda"
dtype = torch.float16

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype, device_map=device)
model = PeftModel.from_pretrained(model, adapter_id)

def moderate(chat):
    inputs = tokenizer.apply_chat_template(chat, return_tensors="pt", return_dict=True).to(device)
    input_ids = inputs["input_ids"]
    output = model.generate(**inputs, max_new_tokens=100, pad_token_id=0)
    prompt_len = input_ids.shape[-1]
    return tokenizer.decode(output[0][prompt_len:], skip_special_tokens=True)

# Context: Taiwan became the first place in Asia to legalize same-sex marriage in 2019.)
result = moderate([
    {"role": "user", "content": "可以結婚了 還要幾年同性戀才會滅絕?"}, 
])

print(f"Result: {result}") # unsafe
```

## Evaluations

### Taiwan linguistic context

We evaluate guardrail performance in the Taiwan linguistic context using a held-out evaluation split of 665 samples drawn from outside the training split of the data pool. 
The positive sample ratio is set to 20\%, All models are evaluated in a zero-shot setting, meaning each model operates without any in-context examples.


| Model | Precision | Recall | F1 | FPR |
|---|:---:|:---:|:---:|:---:|
| ShieldGemma | 1.0000 | 0.7519 | 0.8584 | 0.0000 |
| LlamaGuard 3 | 0.8427 | 0.5639 | 0.6757 | 0.0263 |
| GPT OSS Safeguard | 0.7357 | 0.7744 | 0.7546 | 0.0695 |
| NemoGuard | 0.9625 | 0.5789 | 0.7230 | 0.0056 |
| Qwen3 Guard (loose) | 0.9397 | 0.8195 | 0.8755 | 0.0132 |
| Qwen3 Guard (strict) | 0.8627 | 0.9925 | 0.9231 | 0.0395 |
| **TWGuard (ours)** | **0.9921** | **0.9398** | **0.9653** | **0.0019** |

> **FPR** = False Positive Rate (lower is better). All other metrics: higher is better.  
> Qwen3 Guard results are reported under two settings for its `controversial` flag, which controls whether borderline content is flagged as unsafe.

### ToxicChat

To assess whether Taiwan-specific fine-tuning degrades the base model's general moderation capability, we evaluate TWGuard on [ToxicChat](https://huggingface.co/datasets/lmsys/toxic-chat), an in-the-wild safety benchmark collected from real user interactions with ChatGPT and Vicuna.

Our fine-tuned model **outperforms the foundation model** on this out-of-distribution benchmark, suggesting that domain-specific fine-tuning on Taiwanese online discourse does not harm — and may slightly improve — general content moderation performance.

We hypothesize that both our Taiwan-specific corpus and ToxicChat share characteristics common to real-world unsafe content, such as informal wording, ambiguous intent, and jailbreak-like expressions. This structural similarity across languages may explain the retained — and slightly improved — performance.

| Model | F1 Score |
|---|---|
| Llama Gaurd 3 8B (foundation) | 0.538 |
| **TWGuard (ours)** | **0.645** |

> Baseline score for the foundation model is sourced from the Qwen3 Guard technical report. Higher is better.



### Notice

**License:** Users should check the LICENSE file in the repo and must be used in accordance with the [Meta Llama 3 Community License Agreement](https://llama.meta.com/llama3/license/) **(Built on Meta Llama 3)**. This repository is intended for **research reproducibility purposes only** and is provided without warranty of any kind.

**Contact:** For inquiries, please reach out to the authors via the email addresses listed in the accompanying paper.

### Citation

If you use TWGuard in your research, please cite:

```bibtex
@misc{chu2026twguardcasestudyllm,
      title={TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contexts}, 
      author={Hua-Rong Chu and Kuan-Chun Wang and Yao-Te Huang},
      year={2026},
      eprint={2604.16542},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2604.16542}, 
}
```