seqSight/seqlens-v2-micro-novel-16k

SeqLens v2 Micro Novel 16K

A compact genomic language model pre-trained with a novel recipe — span masking + complexity-weighted MLM loss — on prokaryotic genomes. Achieves 96.2% on coding/non-coding classification with only 10M parameters.

What's Different (Novel Pre-training Recipe)

This model uses the same BiMamba2 architecture as seqlens-v2-micro-16k but a different pre-training recipe:

Component	Standard	Novel (this model)
Masking strategy	Random 15% individual tokens	Span masking (contiguous regions, geometric distribution, mean length 10)
Loss function	Uniform cross-entropy	Complexity-weighted cross-entropy (Shannon entropy weighting, low-complexity regions downweighted to 0.1×)
Architecture	BiMamba2 + sliding-window attn	Same
Data	OpenGenome2 GTDB v220	Same

Why this matters: Random masking teaches the model to predict individual nucleotides from local context — a shallow task that saturates quickly. Span masking forces prediction over contiguous regions (codons, promoters, gene boundaries), learning higher-order structural patterns. Complexity weighting prevents the model from wasting capacity on repetitive DNA (~30% of prokaryotic genomes), focusing learning on biologically informative regions.

Results

Coding vs Non-coding (linear probe, frozen backbone)

Model	Params	Accuracy	F1
SeqLens v2 Micro Novel	10M	0.9623	0.9623
SeqLens v2 Micro Standard	10M	0.9111	0.9111
SeqLens v1	89M	0.6872	0.6871
4-mer baseline	—	0.5879	0.5879
Random init	10M	0.5904	0.5903

Genus Classification (50 held-out genera, linear probe, frozen backbone)

Model	Params	Accuracy	F1
4-mer baseline	—	0.8652	0.8381
SeqLens v2 Micro Novel	10M	0.8371	0.8005
SeqLens v2 Micro Standard	10M	0.7978	0.7300
Random init	10M	0.8371	0.7823

Embedding Clustering

Model	Silhouette	ARI
SeqLens v2 Micro Novel	0.0427	0.4026
SeqLens v2 Micro Standard	0.0298	0.3909

Extended Benchmark Comparison

All evaluations below use the frozen linear probe protocol (frozen embeddings → LogisticRegression). Published fine-tuned baselines (e.g., ProkBERT's fine-tuned MCC scores) are not directly comparable — this is an apples-to-apples comparison across models with a fixed downstream classifier.

ProkBERT Prokaryotic Benchmarks (accuracy)

Task	v2_novel (this model)	v2_standard	NTv3-8M	Caduceus	SeqLens_v1	NT-v2	DNABERT-2
Phage ID (L512)	0.751	0.730	0.762	0.653	0.690	0.626	0.652
Phage ID (L1024)	0.805	0.775	0.808	0.650	0.713	0.659	0.700
Phage ID (L2048)	0.838	0.815	0.834	0.675	0.770	0.713	0.723
Promoter (sigma70)	0.653	0.657	0.674	0.665	0.628	0.594	0.591
Promoter (multispecies)	0.582	0.584	0.630	0.608	0.588	0.572	0.545
Lifestyle (BASEL)	0.705	0.737	0.753	0.702	0.735	0.669	0.700
Lifestyle (E. coli, held-out)	0.648	0.667	0.706	0.627	0.660	0.607	0.638
Lifestyle (Extremophile)	0.813	0.871	0.835	0.852	0.775	0.823	0.797
Average	0.724	0.729	0.751	0.679	0.695	0.658	0.668

NTv3-8M leads this suite (trained on 9T bp across all species, ~36× more data than our 248B prokaryotic tokens). Interestingly, v2_standard edges out this model (v2_novel) on the ProkBERT suite — the opposite ranking from CDS classification and GenomicBenchmarks, where the novel recipe wins. See PROJECT_STATE.md §3.4 for a per-task recipe breakdown.

GenomicBenchmarks — Eukaryotic Tasks (accuracy)

Task	v2_novel (this model)	v2_standard	NTv3-8M	Caduceus	SeqLens_v1	NT-v2	DNABERT-2
Mouse Enhancers	0.822	0.826	0.810	0.744	0.806	0.802	0.727
Coding vs Intergenic	0.895	0.904	0.927	0.936	0.929	0.886	0.948
Human vs Worm	0.906	0.928	0.955	0.967	0.969	0.941	0.981
Enhancers (Cohn)	0.730	0.726	0.734	0.746	0.754	0.721	0.809
Enhancers (Ensembl)	0.715	0.729	0.735	0.747	0.771	0.749	0.758
NonTATA Promoters	0.830	0.842	0.852	0.865	0.864	0.831	0.891
OCR (Ensembl)	0.661	0.676	0.670	0.682	0.692	0.670	0.677
Average (excl. Regulatory)	0.794	0.805	0.812	0.813	0.826	0.800	0.827

The human_ensembl_regulatory task is excluded: our single-nucleotide tokenizer preserves exact sequence length, and length alone is a near-complete shortcut for this task's 3-way label (a Random Forest trained on sequence length alone reaches 91.3% accuracy). Our models score ~0.99–1.00 on this task as an artifact of that leakage, not because of learned regulatory biology. Despite training exclusively on prokaryotic genomes, our 10M models remain competitive with human/multi-species models up to 10× our size on the remaining 7 eukaryotic tasks.

Full results: experiments/eval_reports/genomic_benchmarks/comparison.json and experiments/eval_reports/prokbench/comparison.json.

Architecture

Property	Value
Parameters	10.3M
Hidden dimension	256
Layers	8 (BiMamba2) + 2 sliding-window attention (layers 3, 7)
Context length	16,384 tokens (single nucleotide)
Vocabulary	A, T, G, C, N, [CLS], [SEP], [PAD], [MASK] (9 tokens)
SSM	Mamba2 (d_state=64, expand=2, headdim=64)
Attention	Chunked sliding-window, w=512, RoPE
FFN	SwiGLU, 4× expansion
Pooling	Attention-weighted (for sequence-level embeddings)

Training

Pre-training Recipe

Parameter	Value
Objective	Masked Language Modeling
Masking	Span masking (geometric, mean=10, max=200)
Loss	Complexity-weighted cross-entropy (window=64, min_weight=0.1)
Mask rate	15% of nucleotides
Mask composition	80% [MASK], 10% random, 10% keep

Optimizer & Schedule

Parameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.98, ε=1e-8)
Learning rate	5e-4 (cosine decay to 1e-5)
Warmup	1,000 steps
Weight decay	0.1
Gradient clipping	1.0
Precision	BF16 mixed

Data & Compute

Property	Value
Training data	OpenGenome2 GTDB v220 (113,379 prokaryotic genomes)
Pre-tokenized	15.3M chunks × 16,384 tokens = 248B tokens available
Total steps	20,000
Effective batch	64 (8 per GPU × 8 GPUs)
Hardware	8× NVIDIA A100-SXM4-80GB
Framework	PyTorch 2.6.0, mamba-ssm 2.2.4

Usage

import torch
from model import SeqLensForMLM
from config import MICRO_CONFIG
from tokenizer import NucleotideTokenizer

device = torch.device("cuda")
ckpt = torch.load("seqlens-v2-micro-novel-16k.pt", map_location=device)
model = SeqLensForMLM(MICRO_CONFIG).to(device).to(torch.bfloat16)
model.load_state_dict(ckpt["model"])
model.eval()

tokenizer = NucleotideTokenizer(max_len=16384)
seq = "ATGCGATCG..."
token_ids = torch.tensor([tokenizer.encode(seq)], dtype=torch.long).to(device)

# Sequence-level embedding (for classification, retrieval)
with torch.no_grad():
    emb = model.get_embeddings(token_ids, pool="attention")  # (1, 256)

# Per-position logits (for MLM, motif detection)
with torch.no_grad():
    output = model(token_ids)
    logits = output["logits"]  # (1, L, 9)

Model Family

Variant	Params	Recipe	CDS Acc	Genus Acc	Status
Micro Standard	10M	Random MLM	0.911	0.798	Released
Micro Novel	10M	Span + CW	0.962	0.837	Released
Base	~100M	TBD	—	—	Planned

Citation

@misc{seqlens-v2-novel-2026,
  title={SeqLens v2: Span-Masked Complexity-Weighted Pre-training for Compact Genomic Language Models},
  author={SeqSight Team},
  year={2026},
  url={https://huggingface.co/seqSight/seqlens-v2-micro-novel-16k}
}

License

Apache 2.0