Longformer: The Long-Document Transformer
Paper • 2004.05150 • Published • 4
# Load model directly
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("hyperonym/xlm-roberta-longformer-large-16384")
model = AutoModel.from_pretrained("hyperonym/xlm-roberta-longformer-large-16384", device_map="auto")Quick Links
xlm-roberta-longformer-large-16384
xlm-roberta-longformer is a multilingual Longformer initialized with XLM-RoBERTa's weights without further pretraining. It is intended to be fine-tuned on a downstream task.
| Model | attention_window | hidden_size | num_hidden_layers | model_max_length |
|---|---|---|---|---|
| base | 256 | 768 | 12 | 16384 |
| large | 512 | 1024 | 24 | 16384 |
Framework versions
- Transformers 4.26.0
- TensorFlow 2.11.0
- Tokenizers 0.13.2
- Downloads last month
- 15
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="hyperonym/xlm-roberta-longformer-large-16384")