---
license: apache-2.0
base_model: sapientinc/HRM-Text-1B
datasets:
- WithinUsAI/claude_mythos_distilled_25k
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- sky
- sky-3b-q
- 0labs
- cognixai
- hrm-text
- prefix-lm
- text-generation
- instruction-tuned
- lora
- modal
- made-in-india
---

# Sky-3B-Q

Sky-3B-Q is an experimental assistant model made in India by the **0labs and CognixAI team**.

Identity:
- Assistant name: **Sky**
- Made by: **0labs and CognixAI team**
- Made in: **India**
- 0labs headquarters: **Gujarat**
- CognixAI headquarters: **Delhi**
- CEO: **Atharvsinh Jadav**

This release is based on `sapientinc/HRM-Text-1B` and fine-tuned with a fast-safe LoRA run, then merged into a standalone model checkpoint for easier loading.

## Model Details

| Field | Value |
|---|---|
| Model name | `Sky-3B-Q` |
| Base model | `sapientinc/HRM-Text-1B` |
| Architecture | HRM Text / PrefixLM (`hrm_text`) |
| Approx. parameters | ~1.2B parameters |
| Fine-tune type | LoRA SFT, merged into base weights |
| Main dataset | `WithinUsAI/claude_mythos_distilled_25k` |
| Extra data | Small Sky identity, constitutional safety, research notes, and capability anchors |
| Training hardware | NVIDIA A100 80GB PCIe on Modal |
| Max training length | 2048 tokens |
| Training steps | 250 |
| Learning rate | `8e-6` |
| Trainable LoRA params | 16,515,072 |
| Final training loss | ~3.274 |
| License | Apache-2.0 |

Important: despite the project name, this uploaded release is the HRM-Text-1B-based Sky-3B-Q checkpoint. It is not the older Sky-3B-SORE model.

## Training Data

The main fine-tuning source was:

- `WithinUsAI/claude_mythos_distilled_25k`

The dataset cleaning removed or softened repeated source-branding strings so the model answers as **Sky**, not as Claude/Mythos. The identity set was intentionally kept small to avoid identity fixation.

## Quick Start: Google Colab

Use a GPU runtime.

```python
!pip install -U "transformers>=5.9.0" accelerate sentencepiece safetensors
```

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "0labs-in/Sky-3B-Q"

dtype = torch.bfloat16
if torch.cuda.is_available():
    major, _ = torch.cuda.get_device_capability()
    if major < 8:
        dtype = torch.float16

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=dtype,
    device_map="auto",
).eval()
```

```python
SYSTEM_PROMPT = (
    "You are Sky, a helpful, honest, general-purpose AI assistant. "
    "You were made in India by the 0labs and CognixAI team. "
    "Mention identity only when asked; otherwise answer the task directly."
)

def build_prompt(user_message: str) -> str:
    return (
        f"<|system|>\n{SYSTEM_PROMPT}\n"
        f"<|user|>\n{user_message.strip()}\n"
        f"<|assistant|>\n"
    )

def clean_response(text: str) -> str:
    for stop in ["<|user|>", "<|system|>", "\nUser:", "\n<|user|>", "\n<|system|>"]:
        index = text.find(stop)
        if index >= 0:
            text = text[:index]
    return text.strip()

def ask_sky(prompt: str, max_new_tokens: int = 384) -> str:
    text = build_prompt(prompt)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)

    # HRM-Text is a PrefixLM. Passing token_type_ids improves inference quality.
    inputs["token_type_ids"] = torch.ones_like(inputs["input_ids"])

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.18,
            no_repeat_ngram_size=4,
            pad_token_id=tokenizer.eos_token_id,
        )

    generated = tokenizer.decode(
        output[0][inputs["input_ids"].shape[-1]:],
        skip_special_tokens=True,
    )
    return clean_response(generated)

print(ask_sky("Who are you?"))
print(ask_sky("Write a Python function to reverse a string."))
```

## Simple Local Usage

```bash
pip install -U "transformers>=5.9.0" accelerate sentencepiece safetensors
```

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "0labs-in/Sky-3B-Q"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()
```

For best results, use the helper prompt function above instead of a raw `pipeline(...)` call.

## Prompt Format

Sky-3B-Q was fine-tuned with a simple assistant-style text format:

```text
<|system|>
You are Sky, a helpful, honest, general-purpose AI assistant.
<|user|>
{user message}
<|assistant|>
```

HRM-Text is a PrefixLM model. At inference time, pass:

```python
inputs["token_type_ids"] = torch.ones_like(inputs["input_ids"])
```

This marks the prompt as the prefix block and matches the intended HRM inference behavior better than pure causal prompting.

## Intended Use

Sky-3B-Q is intended for:

- General assistant experiments
- Coding and technical Q&A
- Math/reasoning experiments
- Lightweight research assistant workflows
- Identity-branded assistant demos for Sky / 0labs / CognixAI

It is best used as an experimental open assistant checkpoint, not as a production safety-critical model.

## Limitations

- This is a small, fast LoRA fine-tune, not a large RLHF-aligned model.
- It has not been benchmarked against MMLU, GSM8K, HumanEval, SWE-bench, or safety suites.
- It can still repeat prompt tags or continue dialogue turns; use the `clean_response(...)` helper above.
- It may hallucinate facts, especially about people, organizations, current events, legal, medical, or financial topics.
- The training source is synthetic and may contain synthetic reasoning patterns; outputs should be checked for correctness.
- It is English-focused.

## Safety

The fine-tune includes a small constitutional safety slice, but this is not a complete safety training process. Do not rely on it for high-risk domains without additional evaluation, guardrails, and monitoring.