Hugging Face's logo

TildeAI
/
TildeOpen-30b-64k

Text Generation
Transformers
Safetensors
llama
custom_code
text-generation-inference
Model card Files
xet
 Community
TBergmanis commited on
Commit
38bd281
·
verified ·
1 Parent(s): bd99ee4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +54 -53
README.md CHANGED
@@ -1,48 +1,50 @@
1
- ---
2
- license: cc-by-4.0
3
- language:
4
- - en
5
- - de
6
- - fr
7
- - pl
8
- - ru
9
- - it
10
- - pt
11
- - cs
12
- - nl
13
- - es
14
- - fi
15
- - tr
16
- - hu
17
- - bg
18
- - uk
19
- - bs
20
- - hr
21
- - da
22
- - et
23
- - lt
24
- - ro
25
- - sk
26
- - sl
27
- - sv
28
- - 'no'
29
- - lv
30
- - sr
31
- - sq
32
- - mk
33
- - is
34
- - mt
35
- - ga
36
- datasets:
37
- - HPLT/HPLT2.0_cleaned
38
- - HPLT/hplt_monolingual_v1_2
39
- - HuggingFaceFW/fineweb-2
40
- - allenai/MADLAD-400
41
- - uonlp/CulturaX
42
- - bigcode/the-stack
43
- - common-pile/arxiv_papers
44
- library_name: transformers
45
- ---
 
 
46
  **Developed by:** [Tilde.ai](https://tilde.ai/tildeopen-llm/)
47
  **Funded by:** European Commission via [EuroHPC JU Large AI Grand Challenge](https://www.eurohpc-ju.europa.eu/winners-announced-large-ai-grand-challenge-2024-06-26_en)
48
  **Model type:** A 30B parameter dense decoder-only transformer
@@ -52,7 +54,7 @@ library_name: transformers
52
  ## Info
53
  This is the large context version of [TildeOpen 30B](https://arxiv.org/abs/2603.08182) foundational model, featuring context extension from 8k to 64k tokens using [YaRN](https://arxiv.org/abs/2309.00071). The repsitory also contains patches to YaRN implementation for transformers versions < 5. Patches are not nescessery for transformers versions >= 5, since YaRN was reimplemented and fixed there. Running vLLM does not require patches either.
54
 
55
- For more detailed background information please refer to the original model repository: [https://huggingface.co/TildeAI/TildeOpen-30b](https://huggingface.co/TildeAI/TildeOpen-30b).
56
 
57
  ## Model Hyper-Parameters
58
 
@@ -85,27 +87,26 @@ We use the following YaRN configuration for RoPE scaling:
85
  | Original Max. Position Embeddings | 8192 |
86
  | Rope Theta | 200000 |
87
 
88
- Similarly to [Deespeekv3](https://arxiv.org/pdf/2412.19437) we slightly overscale the YaRN embeddings to 10x and not 8x; related PyTorch warnings can be ignored.
89
 
90
  ## Differences from Huggingface LLaMa model implementation for transformers <5
91
 
92
  - New rotary embedding class was written - `NeoXRotaryEmbeddings`.
93
  - Supports **YaRN**, implemented by analogy with vLLM’s YaRN approach and the [YARN paper](https://arxiv.org/abs/2309.00071)
94
- - Designed to match the rotary embedding behavior used during pretraining and context extension.
95
 
96
- - The attention implementation modified to more closely match attention behavior used during training.
97
 
98
  Running the code requires `flash-attn >= 2.0.6, < 3.0`.
99
  The model can still be run with the original Hugging Face LLaMa code. However, when using YaRN, we found that can lead to vastly different logit generation.
100
 
101
- **NOTE**: If you are using transformers >= 5 or vLLM this section does not apply and can safely be ignored.
102
 
103
  ## Running model using HF transformers < 5
104
 
105
- **NOTE**: The provided YARN patch was written specifically for transformers==4.46.3. It likely can support other versions, but that has not been thoroughly tested.
106
  We suggest avoiding patches and using transformers >= 5 or vLLM.
107
 
108
- Tokenizer now supports ```use_fast=True```, which is the default setting.
109
 
110
  ```python
111
  from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -133,7 +134,7 @@ outputs = model.generate(
133
 
134
  ## Running model using HF transformers >= 5
135
 
136
- Tokenizer now supports ```use_fast=True```, which is the default setting.
137
 
138
  ```python
139
  from transformers import AutoTokenizer, AutoModelForCausalLM
 
1
+ ---
2
+ license: cc-by-4.0
3
+ language:
4
+ - en
5
+ - de
6
+ - fr
7
+ - pl
8
+ - ru
9
+ - it
10
+ - pt
11
+ - cs
12
+ - nl
13
+ - es
14
+ - fi
15
+ - tr
16
+ - hu
17
+ - bg
18
+ - uk
19
+ - bs
20
+ - hr
21
+ - da
22
+ - et
23
+ - lt
24
+ - ro
25
+ - sk
26
+ - sl
27
+ - sv
28
+ - 'no'
29
+ - lv
30
+ - sr
31
+ - sq
32
+ - mk
33
+ - is
34
+ - mt
35
+ - ga
36
+ datasets:
37
+ - HPLT/HPLT2.0_cleaned
38
+ - HPLT/hplt_monolingual_v1_2
39
+ - HuggingFaceFW/fineweb-2
40
+ - allenai/MADLAD-400
41
+ - uonlp/CulturaX
42
+ - bigcode/the-stack
43
+ - common-pile/arxiv_papers
44
+ - HuggingFaceFW/finepdfs
45
+ library_name: transformers
46
+ new_version: TildeAI/TildeOpen-30b
47
+ ---
48
  **Developed by:** [Tilde.ai](https://tilde.ai/tildeopen-llm/)
49
  **Funded by:** European Commission via [EuroHPC JU Large AI Grand Challenge](https://www.eurohpc-ju.europa.eu/winners-announced-large-ai-grand-challenge-2024-06-26_en)
50
  **Model type:** A 30B parameter dense decoder-only transformer
 
54
  ## Info
55
  This is the large context version of [TildeOpen 30B](https://arxiv.org/abs/2603.08182) foundational model, featuring context extension from 8k to 64k tokens using [YaRN](https://arxiv.org/abs/2309.00071). The repsitory also contains patches to YaRN implementation for transformers versions < 5. Patches are not nescessery for transformers versions >= 5, since YaRN was reimplemented and fixed there. Running vLLM does not require patches either.
56
 
57
+ For more detailed background information, please refer to the original model repository: [https://huggingface.co/TildeAI/TildeOpen-30b](https://huggingface.co/TildeAI/TildeOpen-30b).
58
 
59
  ## Model Hyper-Parameters
60
 
 
87
  | Original Max. Position Embeddings | 8192 |
88
  | Rope Theta | 200000 |
89
 
90
+ We follow [Deespeek v3](https://arxiv.org/pdf/2412.19437) and slightly overscale the YaRN embeddings to 10x rather than 8x; related PyTorch warnings can be ignored.
91
 
92
  ## Differences from Huggingface LLaMa model implementation for transformers <5
93
 
94
  - New rotary embedding class was written - `NeoXRotaryEmbeddings`.
95
  - Supports **YaRN**, implemented by analogy with vLLM’s YaRN approach and the [YARN paper](https://arxiv.org/abs/2309.00071)
96
+ - Designed to match the rotary embedding behaviour used during pretraining and context extension.
97
 
98
+ - The attention implementation was modified to more closely match the attention behaviour used during training.
99
 
100
  Running the code requires `flash-attn >= 2.0.6, < 3.0`.
101
  The model can still be run with the original Hugging Face LLaMa code. However, when using YaRN, we found that can lead to vastly different logit generation.
102
 
103
+ **NOTE**: If you are using transformers >= 5 or vLLM, this section does not apply and can safely be ignored.
104
 
105
  ## Running model using HF transformers < 5
106
 
107
+ **NOTE**: The provided YARN patch was written specifically for **transformers==4.46.3**. It likely can support other versions, but that has not been thoroughly tested.
108
  We suggest avoiding patches and using transformers >= 5 or vLLM.
109
 
 
110
 
111
  ```python
112
  from transformers import AutoTokenizer, AutoModelForCausalLM
 
134
 
135
  ## Running model using HF transformers >= 5
136
 
137
+ Tokeniser now supports ```use_fast=True```, which is the default setting.
138
 
139
  ```python
140
  from transformers import AutoTokenizer, AutoModelForCausalLM