Where Does One Text End and the Next Begin?
A Tibetan volume may carry one work or a dozen, running one after another with no file-level marker between them, and around those works sit the structural parts of the book: front matter, a table of contents, a colophon, back matter. When such a volume is digitized, all of it arrives as one undifferentiated character stream.
Recovering that structure is a prerequisite for almost everything else. You cannot catalog, cite, or search a work you cannot locate the start of.
This post documents a fine-tuning run that trains a multilingual encoder to predict those positions directly from the character stream, and reports honestly on what worked, what broke, and what we plan to change next.
The goal
Take jhu-clsp/mmBERT-base — a ModernBERT-family multilingual encoder — and fine-tune it as a token classifier with two labels: B (boundary) and O (everything else).
It is worth being precise about what counts as a boundary here, because the obvious guess is wrong. This is not chapter or section segmentation. The model is not looking for internal divisions inside a text. It marks the seams between the top-level units of a volume:
- where one text or work ends and the next begins;
- the start of the front matter;
- the start of the table of contents;
- the start of the colophon;
- the start of the back matter.
All of these are collapsed into a single B class. The model’s job is to find the position. This means a volume containing one work with no apparatus legitimately has zero boundaries — a fact that comes back to bite us in the evaluation.
One more distinction shaped everything downstream, so it is worth stating up front: there are two different scores in this project, and they are not interchangeable.
- Token-level val F2 is what the training loop watches when deciding which checkpoint to keep. It scores individual token predictions, and F2 deliberately weights recall over precision.
- Character-level micro P/R/F1 is what actually matters in production. It is measured after the real decoder runs — and that decoder is several steps removed from the raw token predictions:
flowchart LR
tok["Token B probabilities"] --> cw["Center-weight<br/>across overlapping windows"]
cw --> th["Apply decode threshold"]
th --> mg["Merge predictions<br/>within 50 chars"]
mg --> mt["Match ground truth<br/>within 25 chars"]
mt --> sc["Character-level P / R / F1"]
Optimizing the first does not guarantee the second. A good part of this write-up is about the gap between them.
Data preparation
The dataset is split at the work level — not the document level — so that no work ID appears in more than one split. An 85/10/5 split was stratified by print_format with seed 42. Where a work mixed print formats, volumes in the minority format were dropped (154 documents), keeping each work’s formatting internally consistent.
| Item | Value |
|---|---|
| Tokenizer / window / stride | mmBERT-base · 8192 / 256 tokens |
| Boundary label radius | ±3 characters |
| Train / val / test works | 3662 / 430 / 215 |
| Train / val / test docs | 7887 / 430 / 215 |
| Strata (works) | pering 2790/328/164 · modern_book 872/102/51 |
| Prepare neg-sample | 0.6 of O-only train windows kept |
| Train / val / test windows | 254,968 / 15,633 / 7,113 |
| Labeled B : O (prepare) | 1.08M : 2.06B (~1 : 1900) |
Boundaries carry a ±3 character radius rather than a single labeled position, giving the model a small target to hit instead of a knife edge. Windows are 8192 tokens with a 256-token stride, so overlapping context is generous.
The class balance is the central difficulty. Because boundaries are top-level rather than sectional, they are genuinely rare: roughly one B for every 1,900 O tokens.
The downsampling that stacked
To fight that imbalance, O-only windows are subsampled. Two separate --neg-sample-ratio flags exist — one at prepare time, one at train time — and it turns out they stack. Both apply only to the training split; validation and test windows are never downsampled. At each stage, any window containing a B label is kept unconditionally; only windows that are entirely O are candidates for removal.
flowchart LR
raw["Raw train windows after sliding"] --> prep["prepare: keep 0.6 of O-only"]
prep --> disk["254,968 windows on disk"]
disk --> train["train: keep 0.6 of remaining O-only"]
train --> seen["184,386 windows seen by the model"]
| Stage | Ratio | Effect |
|---|---|---|
| Prepare | 0.6 | 254,968 train windows written to disk |
| Train | 0.6 | 254,968 → 184,386 (78,515 positive + 105,871 / 176,453 O-only) |
| Effective O-only keep vs prepare input | 0.36 | 0.6 × 0.6 |
| Val / test | — | left intact (15,633 / 7,113) |
The practical consequence: the model trained on a mix substantially denser in boundaries than the real corpus, and checkpoint selection used F2, which further favors recall. Both pressures point the same direction. We should expect a recall-friendly model whose dominant residual error is false positives — right up until the decode threshold is raised.
That prediction turned out to be correct.
Training
The run used a single A100-SXM4-40GB in bf16 with torch.compile, and focal loss (γ = 2.0, α_B = 0.75, α_O = 0.25). After train-time sampling, the effective B:O ratio inside the loader was 1 : 1374.
| Setting | Value |
|---|---|
| Epochs planned | 3 |
| Batch / accum / effective | 1 / 16 / 16 |
| LR / warmup | 1.5e-5 / 3457 steps |
| Selection | val F2; patience 8 evals |
| Eval / save | every 500 steps |
The run did not finish. It died at step 13500, roughly 38 hours in and early into epoch 2 — a crash, not a clean early stop. That sounds worse than it was: the best-F2 checkpoint had already been set by the end of epoch 1, and epoch 2 was trading precision against recall without improving F2.
| Checkpoint | Step | token P | token R | F1 | F2 |
|---|---|---|---|---|---|
| best (saved) | ~11,500 / epoch-1 end | 0.63 | 0.86 | 0.73 | 0.800 |
| checkpoint-12000 | 12,000 | 0.74 | 0.78 | 0.756 | 0.768 |
Nearly all of the learning happened in epoch 1: token F2 climbed from 0.13 to 0.80. The two surviving checkpoints have distinct personalities that persist all the way through to document-level evaluation — best is the recall-leaning weights, checkpoint-12000 the conservative ones.
Evaluation
Initial testing with the argmax evaluation method produced byte-identical results at thresholds 0.85 and 0.95 (P 0.11 / R 0.89). To resolve this, I switched to a tolerance matching method that aligns detected boundary points with the nearest ground truth point within a given character tolerance range. The results from this new evaluation method are presented below.
Validation sweep on best (430 docs, 6528 ground-truth boundaries)
| Threshold | P | R | F1 | pred / TP / FP / FN |
|---|---|---|---|---|
| 0.50 | 0.693 | 0.894 | 0.781 | 8416 / 5835 / 2581 / 693 |
| 0.60 | 0.810 | 0.844 | 0.827 | 6806 / 5511 / 1295 / 1017 |
| 0.70 | 0.908 | 0.720 | 0.803 | 5172 / 4698 / 474 / 1830 |
| 0.80 | 0.966 | 0.340 | 0.503 | 2297 / 2219 / 78 / 4309 |
| 0.90+ | ~1.00 | <0.01 | ~0 | decoder almost silent |
The threshold does real work now. F1 peaks at 0.60, and above 0.80 the model goes quiet — near-perfect precision on the handful of predictions it still makes, and almost no recall.
The averages hide the shape of the trade, though. Counting the actual boundaries makes it plainer: raising the threshold from 0.50 to 0.70 removes about 2,100 false positives, but it also converts roughly 1,100 correct detections into misses. Past 0.70 the exchange stops being worthwhile — going to 0.80 clears fewer than 400 remaining false positives at the cost of nearly 2,500 more missed boundaries.
Test holdout (215 docs, 2157 ground-truth boundaries)
| Model @ threshold | P | R | F1 | pred / TP / FP / FN |
|---|---|---|---|---|
| best @ 0.70 | 0.876 | 0.618 | 0.725 | 1522 / 1333 / 189 / 824 |
| best @ 0.85 | 0.976 | 0.167 | 0.286 | 370 / 361 / 9 / 1796 |
| ckpt-12000 @ 0.70 | 0.940 | 0.463 | 0.620 | 1063 / 999 / 64 / 1158 |
| ckpt-12000 @ 0.85 | 0.976 | 0.076 | 0.140 | 167 / 163 / 4 / 1994 |
Precision is high in all four settings — it is recall that separates them, and recall is what the threshold destroys.
Recommended operating point: checkpoints/best at a threshold of 0.60–0.70. Validation F1 peaks at 0.60; the point actually measured on test is 0.70, giving P 0.88 / R 0.62. A threshold of 0.85 is far too aggressive — precision barely improves while recall collapses.
What we learned
The production decoder works. Moving the threshold moves the curve, which was simply not true of the old argmax reports. This is the precondition for every tuning decision that follows.
Token F2 is not use-case F1. The best validation F2 is 0.80; the best document-level test F1 is 0.73 at threshold 0.70. Checkpoint selection was optimizing a proxy, and the proxy and the target disagree about which errors matter.
Validation is easier than test. At the same 0.70 threshold, validation F1 is 0.80 against test’s 0.73, and recall drops from 0.72 to 0.62. Tuning the threshold on validation alone and assuming the recall transfers to the 215-document holdout would be a mistake.
Precision is cheap; recall is expensive. Above 0.70, buying a little precision costs a great deal of recall. checkpoint-12000 sits further along that same trade — fewer false positives, more misses — and is the better choice if the downstream task needs a high-precision filter rather than full coverage.
Single-work volumes do disproportionate damage. Several test volumes contain zero boundaries — one continuous work, no front matter, no table of contents, no colophon — and yet receive dozens to hundreds of predictions. W8LS32787 alone contributes 547 false positives, W1KG17450 another 498. This is the failure mode the framing predicts: a model tuned to find seams will hallucinate them in a volume that has none, most likely mistaking internal divisions of a single text for the start of a new one. Micro-averaged numbers hide this; macro F1 and per-document inspection do not.
The stacked 0.6 × 0.6 downsampling is a likely culprit. The model saw a boundary-heavier mix than the corpus actually contains — in other words, less exposure to long uninterrupted stretches of a single work, which is exactly the context it fails on.
For now, the default for these weights is checkpoints/best at a decode threshold of 0.60–0.70: roughly nine correct boundaries in ten predicted, at the cost of missing about four in ten.
Credits
Developed by Dharmaduta based on specifications from the Buddhist Digital Resource Center for the project The BDRC Etext Corpus, funded by the Khyentse Foundation.