Swift-Support/labse-priority-1.0

Swift-Support LaBSE Priority Classifier (v1.0)

A fine-tuned LaBSE (Language-Agnostic BERT Sentence Embedding) model that assigns an escalation priority — Low / Medium / High — to a banking support ticket written in any of five language tracks. Built for the Swift support-ticket triage project, alongside Swift-Support/labse-intent-1.0.

Model details

• Base architecture: sentence-transformers/LaBSE (471M parameters, 501k vocabulary)
• Task: 3-class text classification (priority / urgency)
• Classes: Low, Medium, High
• Languages: English, Sinhala, Tamil, Singlish (romanized Sinhala), Tanglish (romanized Tamil)
• Regime: one multilingual model over all five tracks — not five per-language models

The headline metric is macro-F1, never accuracy: the class distribution is roughly 55% Low / 36% Medium / 9% High, so accuracy flatters a model that neglects High.

Evaluation

Trained on train+dev (49,990 rows = 9,998 tickets × 5 languages), scored once on the held-out test set (15,395 rows = 3,079 tickets × 5 languages). Frozen split e7b5934392cd; test tickets come from the official BANKING77 test file and were never used for model selection.

Pooled test macro-F1: 0.8901 (accuracy 0.9008)

Per-class F1: Low 0.9206 · Medium 0.8760 · High 0.8735

Against the alternatives (pooled test macro-F1)

model	macro-F1
gemma-3-1b multitask (shared head)	0.8904
LaBSE (this model)	0.8901
gemma-3-1b (LoRA, single-task)	0.8898
mmBERT	0.8887
XLM-RoBERTa base	0.8872
TF-IDF + LinearSVC (classical champion)	0.8722
TF-IDF + logistic regression	0.8683

The classical champion's 95% CI is [0.8605, 0.8831], so this model clears its upper bound — a real, if modest, win. The Gemma multitask variant is a statistical tie, not a better model.

Per language, on test

track	LaBSE (this model)	classical TF-IDF	delta
English	0.9229	0.9032	+0.0197
Sinhala	0.9179	0.8745	+0.0434
Singlish (romanized)	0.8817	0.8915	−0.0098
Tamil	0.9130	0.8905	+0.0225
Tanglish (romanized)	0.8142	0.7994	+0.0148
ALL (pooled)	0.8901	0.8722	+0.0179

Two things worth stating plainly:

1. The classical baseline still wins on Singlish. LaBSE gives back most of its native-script advantage on romanized text — a pattern that also shows up in linear probing, where LaBSE has the largest native-minus-romanized gap of any backbone in the roster.
2. Tanglish is the weak track, 7–10 points below every other language for both model families.

⚠️ The label ceiling — read this before quoting 0.89

The training labels were generated by an LLM prompt, not by human annotators. On a 500-ticket benchmark set that was annotated by hand, those prompt labels agree with human judgement at 0.7722 macro-F1 (95% CI [0.7263, 0.8147]; raw agreement 0.804, Cohen's κ = 0.644).

This does not cap the number above — against the prompt labels a model could in principle reach 1.0. It caps what the number means. This model has learned the labeling rule well; the rule itself agrees with a human 77% of the time. Quoting 0.89 as "priority accuracy" overstates what a human reviewer would call correct. Any external write-up should state the 0.7722 figure alongside it.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "Swift-Support/labse-priority-1.0"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

texts = ["Someone has taken money from my account and nobody is helping me!",
         "How do I activate my new card?"]

with torch.no_grad():
    batch = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=128)
    probs = model(**batch).logits.softmax(-1)

for text, p in zip(texts, probs):
    print(model.config.id2label[int(p.argmax())], f"{p.max():.3f}", "|", text)

Two things that will silently corrupt results if you get them wrong:

• max_length=128 must match training. It is not stored in the checkpoint.
• Read the label from model.config.id2label, never from a hardcoded index. This checkpoint carries an explicit mapping (0: Low, 1: Medium, 2: High). A wrong index guess does not raise — it silently returns the wrong priority.

Cost: ~1.9 GB resident, roughly 100–300 ms per ticket on CPU. Load the model once at process start, never per request.

Training

base	sentence-transformers/LaBSE
fit portion	train+dev, 49,990 rows
epochs	3 (best epoch: 3 of 3)
learning rate	2e-5
batch size	32
max sequence length	128
class imbalance	class_weight (balanced)
precision	fp16
hardware	Kaggle T4, ~112 rows/s, 22 min wall

best_epoch = 3 of 3 — the model was still improving when training stopped, which is what a consistent labeling target looks like.

Limitations

• Romanized text is synthetic. Singlish is rule-generated from Sinhala and Tanglish is machine-translated, so both are cleaner and more regular than text a human would type. The Singlish and Tanglish numbers above are an optimistic upper bound, and no romanized-specific conclusion from this model should be trusted until it is re-measured on human-typed data.
• Labels are LLM-generated — see the label ceiling section.
• Domain-bound. Derived from BANKING77; behaviour outside retail-banking support is untested.
• Not calibrated. The softmax scores are not probabilities you should threshold on without re-calibrating; a threshold tuned by cross-validation on a sibling task failed to transfer to test in this project.
• Trained and evaluated only on the five tracks listed. LaBSE covers 109 languages, but nothing here measures the other 104.

Citation & provenance

Derived from BANKING77 (PolyAI, CC-BY-4.0), translated into Sinhala and Tamil and romanized into Singlish and Tanglish. Priority labels were generated by an LLM prompt and benchmarked against human annotation as described above.

Training data: Swift-Support/swift-support-tickets-1.0