Annotation Layer Detection in Tibetan Texts — Experiment Report,
Classical Tibetan commentaries are layered documents. A single page interleaves the root text being commented on (tsawa), the author’s topical outline (sabche), interlinear notes in small script (yigchung), quotations of other scriptures (quotation), and the markers that frame the whole — chapter, author, book title. Digital editions record all seven as character-offset spans over a flat base text, and producing them by hand is the bottleneck in publishing critical editions.
All seven layers are the target. Sabche gives you the commentary’s argumentative structure; tsawa separates root text from comment; yigchung marks notes that flat OCR merges into the body; chapter, author and title supply navigation and catalogue metadata. Quotation is the one that reaches outside the book — citation graphs across the Kangyur and Tengyur — and it is the hardest of the seven.
We fine-tuned jhu-clsp/mmBERT-base as a 15-label BIO token classifier over 8,192-token sliding windows, on 266 annotated books split 217 / 25 / 24. Three runs are on the Hub (v1, v1.2, v1.3).
Where we are
The training loop reported v1.3 at 0.071 exact span F1, which reads as a failed project. It isn’t — the number was measuring the readout, not the model.
Re-scoring the same checkpoint with a proper harness, changing nothing but the decoding step:
| v1.3, identical weights and logits | span F1 | precision | recall |
|---|---|---|---|
| argmax decode | 0.134 | 0.081 | 0.399 |
| + Viterbi decode | 0.398 | 0.411 | 0.387 |
| + per-layer logit bias | 0.427 | — | — |
Precision rises 5.1× while recall holds. Token recall inside gold spans barely moves (0.583 → 0.582) because the logits are identical — all the movement is in span assembly. Per-token argmax was shredding each long span into fragments: v1.3 proposed 20,430 quotation spans against 3,008 in gold, a 6.8× over-proposal that Viterbi drops to 1.0×.
Per layer, that leaves three groups:
| layer | span F1 after Viterbi | status |
|---|---|---|
| Author | 0.962 | shippable |
| Chapter | 0.818 | shippable |
| Tsawa | 0.395 | not yet |
| Sabche | 0.394 | not yet |
| Yigchung | 0.389 | not yet |
| Quotation | 0.373 | not yet |
| BookTitle | — | masked out of the loss since v1.2; needs a decision |
The four middle layers land within 0.02 of each other despite having nothing in common structurally — under argmax they spanned 0.102 to 0.199. A shared ceiling, not four separate problems.
The answer key is damaged
P000218 carries 847,725 characters and three annotations, all start == end — length-one stray marks. P000126 has zero spans while marked in scope. 32 of 195 in-scope books sit below 0.5% quotation density against a 3.6% median.
The audit caught none of it because classify_span tests only geometric coherence — negative offsets, start > end, out of bounds. Under inclusive-offset semantics a one-character span is valid, and no minimum-length rule exists anywhere in the pipeline. Meanwhile clean_summary.csv recorded Quotation at 195 books in scope against 194 with any span — the zero-span book was detected, written to disk, and never asserted on.
So some unknown share of the 166,441 false-positive quotation tokens are real quotations nobody annotated. Quote precision is understated by an amount we cannot currently quantify, and this has only ever been checked for one of the seven layers.
Future implementation
1. Report two metrics, not one. The MUC-5 / SemEval-2013 taxonomy gives the error category — correct, partial, incorrect, missing, spurious — and there are mature Python implementations of it, so none of this needs writing from scratch. IoU or character-F1 gives the magnitude: how tight the boundaries actually were. Reported together they replace a misleading 0.071 with a picture someone can act on. The spurious count alone would have exposed the fragmentation at epoch one instead of months later.
2. Clean what can be cleaned. Two deterministic fixes first, since they need no model: a per-layer minimum plausible span length in classify_span, which currently has no lower bound at all, and an assert that in-scope books actually contain spans. Then automated annotation-error detection for the harder case — books that look normal by density but are quietly missing annotations in the middle. The standard recipe is k-fold: train on k−1 folds, predict on the held-out one, and treat confident model-vs-gold disagreement as a candidate error. Our book-level split already gives the fold disjointness this needs. Use it to rank books for human review rather than to auto-delete spans — at 0.373 span F1 the model’s opinion on any single span is weak, but its aggregate disagreement rate per book is not.
3. Train softly on the books we can’t fix. Damaged books still carry real annotations, so discarding them throws away signal. Partial-CRF masking treats unannotated tokens as “any label allowed” rather than forcing them to O, so the model is never punished for finding a quotation nobody wrote down. The alternative is an expected-entity-ratio loss, which takes a prior on what fraction of tokens should be spans — and we already know the median density per layer.
4. Try a better encoder. mmBERT’s own authors concede weak structured-prediction results, attributed to tokenizer limitations, and Tibetan sits in the thin tail of the 1,833 languages it covers. There is now a continual-pretrained variant of our exact base model, trained on 13.8 GB of Classical Tibetan with several thousand Tibetan-specific tokens added; a sibling model cuts tokenizer fertility from 2.07 to 1.43. The upgrade to our base model already exists — it needs testing.
5. Evaluate externally. DharmaBench ships a Tibetan quotation-detection task: 406 test items, scored with MUC-5 in type mode, best published result 76.23 F1 from a generative model. Its authors state they omitted encoder fine-tuning because a 512-token context window was smaller than the average text length of roughly 1,000 characters. Our windows are 8,192 tokens. This is a stated gap in the literature that our setup was built to fill, it costs one inference pass, and it gives us an evaluation set that is neither ours nor damaged.
Caveats. Validation only — 25 books, 6,552 gold spans. Test has never been run. Author rests on 25 gold spans and no number here carries a confidence interval. All of it is measured against the answer key described above, so absolute values are likely pessimistic; the argmax → Viterbi comparison is unaffected, since both sides use identical labels.
Sources
Project artifacts:
- Dataset, karma689/layer_detection
- Models, v1.2 ·
v1·v1.3(private) - Base model, jhu-clsp/mmBERT-base, arXiv:2509.06888
Credits
Built on the OpenPecha / tsadra annotated corpus, whose seven annotation layers over 266 Tibetan books are the ground truth for everything above.
Developed by Dharmaduta based on specifications from the Buddhist Digital Resource Center for the project The BDRC Etext Corpus, funded by the Khyentse Foundation.