NKCZ/old-icelandic-facs2dipl2norm

Old Icelandic facs2dipl2norm

This repository contains a character-level transformer model for Old Icelandic manuscript normalisation tasks, specifically facsimile transcription to diplomatic transcription (facs → dipl) and diplomatic transcription to normalised form (dipl → norm).

The model was trained on all the available MENOTA texts by Andrea de Leeuw van Weenen (AM 132 fol., AM 519 a 4to., and AM 677 4to). This is around 75% of all the currently available MENOTA texts, which are normalised, lemmatized, and (at least partially) POS-tagged.

Old Icelandic manuscript normalisation tasks:

• facs → dipl: facsimile transcription → diplomatic transcription (abbreviation expansion, character normalisation)
• dipl → norm: diplomatic transcription → normalised form (orthographic regularisation)

Task routing is controlled by a prefix token prepended to the source sequence — no architectural changes were necessary between tasks.

Model Details

Property	Value
Architecture	Transformer encoder-decoder
Parameters	~10M
Vocabulary	~120 characters (data-derived)
Max sequence length	128 characters
Model dimension	256
Attention heads	4
Encoder / decoder layers	3 / 3
Feed-forward dim	512
Task tokens	<DIPL> (facs→dipl), <NORM> (dipl→norm)
Training data	~36k line-level triples (80-10-10 split)
Language	Old Icelandic (non)

Training Data

• Corpus size: 36240 text chunks of differing lengths, containing around 400k word tokens.

• Training-validation-test split: 80-10-10.

• Sources: AM 132 fol., AM 519 a 4to, and AM 677 4to, transcribed and edited by Andrea de Leeuw van Weenen.

Training

TODO

Fine-tuning

Fine-tuning is handled by finetune.py, which supports TSV, CSV, and JSONL input. The default input format is pairs, and pairs require a task flag:

# facs -> dipl pairs
python finetune.py --data facs_dipl.csv \ 
--data-format pairs \
--task facs2dipl  \
-- checkpoint best_model.pt \
--vocab vocab.json \
--out finetuned_facs2dipl.pt
--extend-vocab

# dipl -> norm pairs
python finetune.py --data dipl_norm.csv \
--data-format pairs \
--task dipl2norm \
--checkpoint best_model.pt \
--vocab vocab.json \
--out finetuned_dipl2norm.pt \
--extend-vocab

For data containing all three transcription levels, use triples. This expands each row into both tasks and trains them jointly:

python finetune.py \
    --data Kringla-Fragment-GT.csv \
    --data-format triples \
    --checkpoint best_model.pt \
    --vocab vocab.json \
    --out finetuned_model_kringla.pt \
    --epochs 50 --lr 3e-5 --batch-size 64 \
    --extend-vocab --val-split 0.1 --patience 5

Input formats:

• pairs: two columns, source,target, or JSONL objects with source and target keys. Use --task facs2dipl for facs -> dipl or --task dipl2norm for dipl -> norm.
• triples: three columns, facs,dipl,norm, or JSONL objects with facs, dipl, and norm keys. The --task argument is ignored.

CSV and TSV files may include the corresponding header row. Blank pair fields are skipped. In triples, a blank dipl makes the row unusable, while a blank facs or norm only removes that task's pair.

Useful options include --val-split (default 0.1), --patience (default 5), --seed (default 42), and --extend-vocab.

I recommend always using --extend-vocab, as it adds unseen characters to the model vocabulary and saves the expanded vocabulary beside the checkpoint as <checkpoint-name>_vocab.json. Without it, unseen characters are encoded as UNK.

The script automatically uses CUDA when available and otherwise falls back to CPU.

Performance

In-domain evaluation

Task	CER
facs → dipl	0.01
dipl → norm	0.03
facs → dipl → norm	0.04

Out-of-domain evaluation

For out-of-domain evaluation, the first 200 lines from GKS 2365 4to (EAE edition) and Egils saga from WolfAug 9 10 4to (MENOTA edition) were used. Furthermore, 500 subsequent lines were used to fine-tune the model for each manuscript.

Task	CER (pre-finetuning)	CER (post-finetuning)
facs → dipl (GKS)	0.11	0.05
facs → dipl (Wolf)	0.22	0.08
dipl → norm (GKS)	0.14	0.12
dipl → norm (Wolf)	0.10	0.07
facs → dipl → norm (GKS)	0.20	0.15
facs → dipl → norm (Wolf)	0.22	0.12

HTR Example

This model was tested on the Kringla-fragment (Lbs. fragm 82; 164 lines) from the MENOTA edition by Matteo Tarsi. Whole fragment results: CER for facs2dipl is 0.20 and for dipl2norm is 0.22. Fine-tuning in the second half of the manuscrpt, improves the results for the first half of the manuscript for facs2dipl from 0.17 to 0.10 and for dipl2norm from 0.20 to 0.09.

Intended Use

This model is intended for researchers and digital humanists working with Old Icelandic manuscript material who need to automate or assist with the production of diplomatic and normalised transcriptions from facsimile-level texts (e.g., from HTR output from models like OICEN-HTR).

Usage

Try it out in Google Colab!

Important for fine-tuned models to load the correct vocab.json file!!!

import json, torch
from model_def_multitask import CharSeq2Seq, encode_text, decode_ids, greedy_decode, DIPL_IDX, NORM_IDX
 
# Load vocab
with open("vocab.json", encoding="utf-8") as f:
    v = json.load(f)
c2i = v["c2i"]
i2c = {int(k): val for k, val in v["i2c"].items()}
 
# Load model
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
ckpt   = torch.load("best_model.pt", map_location=DEVICE)
hp     = ckpt["hparams"]
 
model = CharSeq2Seq(
    vocab_size = hp["VOCAB_SIZE"],
    d_model    = hp["D_MODEL"],
    n_heads    = hp["N_HEADS"],
    n_enc      = hp["N_ENC"],
    n_dec      = hp["N_DEC"],
    d_ff       = hp["D_FF"],
    max_len    = hp["MAX_LEN"],
    dropout    = hp["DROPOUT"],
).to(DEVICE)
model.load_state_dict(ckpt["model"])
model.eval()

facs → dipl

MAX_LEN = hp["MAX_LEN"]
 
def predict_dipl(texts):
    if isinstance(texts, str):
        texts = [texts]
    src = torch.tensor(
        [encode_text(t, DIPL_IDX, c2i, MAX_LEN) for t in texts],
        dtype=torch.long
    )
    return greedy_decode(model, src, MAX_LEN, DEVICE, i2c)
 
predict_dipl("koma egƚ. kappı þınu ⁊ ꝺırꝼð . en ſkaplynꝺı") # random line from test set
# → "koma eg(il)l kappi þinu (ok) dirfð . en ſkaplyndi"

dipl → norm

def predict_norm(texts):
    if isinstance(texts, str):
        texts = [texts]
    src = torch.tensor(
        [encode_text(t, NORM_IDX, c2i, MAX_LEN) for t in texts],
        dtype=torch.long
    )
    return greedy_decode(model, src, MAX_LEN, DEVICE, i2c)
 
predict_norm("koma eg(il)l kappi þinu (ok) dirfð . en ſkaplyndi") # using previous line as input
# koma Egill kappi þínu ok dirfð . En skaplyndi

Full pipeline: facs → dipl → norm

def predict_pipeline(texts):
    if isinstance(texts, str):
        texts = [texts]
    dipl = predict_dipl(texts)
    norm = predict_norm(dipl)
    return list(zip(dipl, norm))
 
predict_pipeline("koma egƚ. kappı þınu ⁊ ꝺırꝼð . en ſkaplynꝺı")
# [('koma eg(il)l kappi þinu (ok) dirfð . en ſkaplyndi',
#  'koma Egill kappi þínu ok dirfð , en skaplyndi')]