Annotation Layer Detection in Tibetan Texts — Experiment Report

[Karma_Tashi1]

Annotation Layer Detection in Tibetan Texts — Experiment Report

Most tibetan text are layered document. A single page interleaves the root text being commented on (tsawa), the author’s topical outline (sabche), interlinear notes in small script (yigchung), quotations of other scriptures (quotation), and the markers that frame the whole — chapter, author, book title. Digital editions record all seven as character-offset spans over a flat base text, and producing them by hand is the bottleneck in publishing critical editions.

All seven layers are the target. Sabche gives you the commentary’s argumentative structure; tsawa separates root text from comment; yigchung marks notes that flat OCR merges into the body; chapter, author and title supply navigation and catalogue metadata. Quotation is the one that reaches outside the book — citation graphs across the Kangyur and Tengyur — and it is the hardest of the seven.

We fine-tuned jhu-clsp/mmBERT-base as a 15-label BIO token classifier over 8,192-token sliding windows, on 266 annotated books split 217 / 25 / 24. Three runs are on the Hub (v1, v1.2, v1.3).

Where we are

The training loop reported v1.3 at 0.071 exact span F1, which reads as a failed project. It isn’t — the number was measuring the readout, not the model.

Re-scoring the same checkpoint with a proper harness, changing nothing but the decoding step:

v1.3, identical weights and logits	span F1	precision	recall
argmax decode	0.134	0.081	0.399
+ Viterbi decode	0.398	0.411	0.387
+ per-layer logit bias	0.427	—	—

Precision rises 5.1× while recall holds. Token recall inside gold spans barely moves (0.583 → 0.582) because the logits are identical — all the movement is in span assembly. Per-token argmax was shredding each long span into fragments: v1.3 proposed 20,430 quotation spans against 3,008 in gold, a 6.8× over-proposal that Viterbi drops to 1.0×.

Per layer, that leaves three groups:

layer	span F1 after Viterbi	status
Author	0.962	shippable
Chapter	0.818	shippable
Tsawa	0.395	not yet
Sabche	0.394	not yet
Yigchung	0.389	not yet
Quotation	0.373	not yet
BookTitle	—	masked out of the loss since v1.2; needs a decision

The four middle layers land within 0.02 of each other despite having nothing in common structurally — under argmax they spanned 0.102 to 0.199. A shared ceiling, not four separate problems.

The answer key is damaged

P000218 carries 847,725 characters and three annotations, all start == end — length-one stray marks. P000126 has zero spans while marked in scope. 32 of 195 in-scope books sit below 0.5% quotation density against a 3.6% median.

The audit caught none of it because classify_span tests only geometric coherence — negative offsets, start > end, out of bounds. Under inclusive-offset semantics a one-character span is valid, and no minimum-length rule exists anywhere in the pipeline. Meanwhile clean_summary.csv recorded Quotation at 195 books in scope against 194 with any span — the zero-span book was detected, written to disk, and never asserted on.

So some unknown share of the 166,441 false-positive quotation tokens are real quotations nobody annotated. Quote precision is understated by an amount we cannot currently quantify, and this has only ever been checked for one of the seven layers.

Future implementation

1. Task-Specific Fine-Tuned Encoders (BIO Tagging)

Instead of a single backbone, assign distinct fine-tuned encoders to specific annotation layers, keeping the exact same BIO token classification scheme (B-SPAN, I-SPAN, O) across all tasks:

                               ┌── Task 1: Quotation Detection (Long-Range Context)
                               │   ├── Model: Continual-Pretrained mmBERT (13.8 GB Classical Tibetan)
Fine-Tuned Encoder Allocation ─┤   └── Output: Standard BIO token classification via Linear-CRF head
                               │
                               └── Task 2: Yigchung Detection (Interlinear / Small-Print Text)
                                   ├── Model: Low-Fertility Sibling Variant (1.43 Fertility Ratio)
                                   └── Output: Standard BIO token classification over 8,192 token windows

• Task 1: Quotation Detection (Direct Quotes & Embedded Citations)

  • Model: Continual-Pretrained mmBERT (trained on 13.8 GB Classical Tibetan with expanded vocabulary).

  • Setup: Fine-tune with a standard BIO token classification head (via Linear-CRF). Uses differential learning rates ($1\text{e-}5$ for backbone, $1\text{e-}4$ for CRF head) to preserve long-range contextual dependencies.

• Task 2: Yigchung Detection (Interlinear Annotations & Glosses)

  • Model: Low-Fertility mmBERT Sibling Variant (tokenizer fertility reduced from 2.07 to 1.43).

  • Setup: Fine-tune with the identical BIO tagging head over 8,192-token windows. The lower fertility ratio prevents character-dense Tibetan annotations from splitting across token boundaries.

2. Task-Specific LLM Zero-Shot Baselines

Evaluate frontier and open-weight LLMs zero-shot on full 8,192-token context windows for each target annotation task:

                            ┌── Candidate 1: Google Gemini (e.g., Gemini 3.5 Flash / Pro)
                            │   └── Targets: Quotation & Yigchung Detection over long contexts
                            │
                            ├── Candidate 2: Anthropic Claude (e.g., Claude 3.5 Sonnet / Opus)
LLM Zero-Shot Candidate Set ┼── └── Targets: High-precision Tibetan span extraction
                            │
                            ├── Candidate 3: OpenAI (e.g., GPT-4o / GPT-o3)
                            │   └── Targets: Standard instruction-following benchmark
                            │
                            └── Candidate 4: Open-Weight (e.g., Qwen2.5-72B / DeepSeek-R1)
                                └── Targets: Local batch processing baseline

• Target Tasks: Run separate zero-shot passes specifically for Quotation Detection and Yigchung Detection.

• Zero-Shot Output Pipeline: Instruct models to extract exact quoted sub-strings or offset pairs per task, then convert the returned text spans directly into the primary token-level BIO format (B-SPAN, I-SPAN, O) to evaluate against fine-tuned encoders using identical metrics.

Sources

Project artifacts:

• Dataset: https://huggingface.co/datasets/karma689/Bo-MultiLayer-Detection
• Models: https://huggingface.co/BDRC/Bo-Multilayer-Detection
• Base model, jhu-clsp/mmBERT-base, arXiv:2509.06888

Credits

Built on the OpenPecha / tsadra annotated corpus, whose seven annotation layers over 266 Tibetan books are the ground truth for everything above.

Developed by Dharmaduta based on specifications from the Buddhist Digital Resource Center for the project The BDRC Etext Corpus, funded by the Khyentse Foundation.