---
tags:
- heretic
- uncensored
- abliterated
- gguf
license: other
base_model: Qwen/Qwen3-VL-8B-Instruct
---

# Qwen3-VL-8B-Instruct-heretic

Abliterated (uncensored) version of [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct),
created using [Heretic](https://github.com/p-e-w/heretic) and converted to GGUF.

## Abliteration Quality

| Metric | Value |
|:-------|------:|
| Refusals | 6/100 |
| KL Divergence | 0.0033 |
| Rounds | 3 |

Lower refusals = fewer refused prompts. Lower KL divergence = closer to original model behavior.

## Available Quantizations

| Quantization | File | Size |
|:-------------|:-----|-----:|
| Q8_0 | [Qwen3-VL-8B-Instruct-heretic-Q8_0.gguf](./Qwen3-VL-8B-Instruct-heretic-Q8_0.gguf) | 8.11 GB |
| Q6_K | [Qwen3-VL-8B-Instruct-heretic-Q6_K.gguf](./Qwen3-VL-8B-Instruct-heretic-Q6_K.gguf) | 6.26 GB |
| Q4_K_M | [Qwen3-VL-8B-Instruct-heretic-Q4_K_M.gguf](./Qwen3-VL-8B-Instruct-heretic-Q4_K_M.gguf) | 4.68 GB |

## Usage with llama.cpp (Recommended)

> **Note:** Ollama (as of v0.16.x) has a [known bug](https://github.com/ollama/ollama/issues/13150) that crashes when loading Qwen3-VL models. Use llama.cpp directly for vision features.

Vision models require a separate multimodal projector (mmproj) file. Download the official mmproj from [Qwen/Qwen3-VL-8B-Instruct-GGUF](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF):

```bash
# Download mmproj
huggingface-cli download Qwen/Qwen3-VL-8B-Instruct-GGUF mmproj-Qwen3VL-8B-Instruct-F16.gguf

# Run with llama-server (OpenAI-compatible API)
llama-server \
  -m Qwen3-VL-8B-Instruct-heretic-Q8_0.gguf \
  --mmproj mmproj-Qwen3VL-8B-Instruct-F16.gguf \
  -ngl 999

# Or use the CLI directly
llama-mtmd-cli \
  -m Qwen3-VL-8B-Instruct-heretic-Q8_0.gguf \
  --mmproj mmproj-Qwen3VL-8B-Instruct-F16.gguf \
  --image photo.jpg \
  -p "Describe this image." \
  -ngl 999
```

## Usage with Ollama (Text Only)

Ollama can load this model for text-only chat, but vision/image features will crash due to the bug linked above.

```bash
ollama run hf.co/ThalisAI/Qwen3-VL-8B-Instruct-heretic:Q8_0
ollama run hf.co/ThalisAI/Qwen3-VL-8B-Instruct-heretic:Q6_K
ollama run hf.co/ThalisAI/Qwen3-VL-8B-Instruct-heretic:Q4_K_M
```

## bf16 Weights

The full bf16 abliterated weights are available in the `bf16/` subdirectory of this repository.

## Usage with Transformers

The bf16 weights in the `bf16/` subdirectory can be loaded directly with Transformers:

```python
from transformers import AutoModelForImageTextToText, AutoTokenizer

model_id = "ThalisAI/Qwen3-VL-8B-Instruct-heretic"
tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder="bf16")
model = AutoModelForImageTextToText.from_pretrained(
    model_id, subfolder="bf16", torch_dtype="auto", device_map="auto"
)

messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

## About

This model was processed by the **Apostate** automated abliteration pipeline:
1. The source model was loaded in bf16
2. Heretic's optimization-based abliteration was applied to remove refusal behavior
3. The merged model was converted to GGUF format using llama.cpp
4. Multiple quantization levels were generated

The abliteration process uses directional ablation to remove the model's refusal directions
while minimizing KL divergence from the original model's behavior on harmless prompts.