Annotation layer detection in Tibetan texts, part 2: one model per layer, and zero-shot LLMs
This continues Annotation Layer Detection in Tibetan Texts — Experiment Report from August.
A digitised Tibetan book is one long string of text. The structure a reader sees on the page is gone: where a chapter starts, which lines are outline headings, which verses are root text, where the author quotes another work, and which words were printed in small type as notes. OpenPecha annotators have marked these “layers” by hand in several hundred books. We want a model that marks them in any new book.
Where we left off
In the first report we trained one model for all seven layers at once: jhu-clsp/mmBERT-base as a 15-label BIO tagger over 8,192-token windows, on 266 books (217 train, 25 validation, 24 test). Its first scores looked hopeless (exact-span F1 0.071). Most of that turned out to be decoding, not the model: tagging each token on its own shredded long spans into pieces (20,430 predicted quotation spans against 3,008 gold). Viterbi decoding and overlap scoring brought it to:
| Layer | Joint model v1.3, span F1 |
|---|---|
| Author | 0.962 |
| Chapter | 0.818 |
| Tsawa | 0.395 |
| Sabche | 0.394 |
| Yigchung | 0.389 |
| Quotation | 0.373 |
We also found problems in the gold itself: books with no spans at all and books where quotation was barely marked. The report ended with two next steps: a separate fine-tuned model for each layer, and zero-shot LLM baselines with frontier models. This post is both.
What changed
- One model per layer. Each layer now has its own mmBERT model, its own cleaned data and its own split, with settings chosen for that layer (window overlap, class weights, labels).
- More books and more cleaning. The corpus grew from the 266 books of the first round (now the “old” batch) to 539, adding a “new” batch of 273, and each layer went through its own audit before training (below).
- Whole-book scoring. Every score here is on whole test books, with a span counted correct when it overlaps a gold span by at least half.
- LLMs as a baseline. Claude Sonnet 5 and Gemini 3.1 Flash Lite ran each layer with a prompt and no training.
Author and book title are not in this round. They appear about once per book and the joint model already handled author well.
We looked at five layers:
| Layer | What it marks |
|---|---|
| Tsawa (རྩ་བ) | root text that a commentary copies piece by piece and then explains |
| Sabche (ས་བཅད) | outline headings, such as གཉིས་པ་༼ཚུལ་ཁྲིམས་ཕར་ཕྱིན་གྱི་རབ་དབྱེ་༽ནི |
| Chapter (ལེའུ་) | the title line that opens a work in a collected volume, a numbered chapter, a section or front matter |
| Quotation | a passage the author copies from another work or puts in someone else’s mouth |
| Yigchung (ཡིག་ཆུང) | text printed in small type: ritual instructions, colophons, glosses squeezed into a verse, editor’s notes |
For each layer we fine-tuned a separate model, and we also asked two large language models (LLMs), Claude Sonnet 5 and Gemini 3.1 Flash Lite, to do the same job with a prompt and no training. This post covers both, and where each one wins.
The short version: the trained models win clearly on sabche, chapter and yigchung. On tsawa the LLMs do better. On quotation the best LLM is close, but we cannot yet line the two up fairly.
The data
All books come from OpenPecha, in two annotation batches that we call “old” (P books) and “new” (I books). Each layer is a list of character positions in the book text.
| Tsawa | Sabche | Chapter | Quotation | Yigchung | |
|---|---|---|---|---|---|
| Books with the layer | 212 | 333 | 394 | not recorded | 254 |
| Books kept | 124 | 323 | 378 | 254 | 232 |
| Spans kept | 17,999 | 42,292 | 3,042 | 23,095 | 8,495 |
| Median span length (characters) | 96 | 40 | 43 | about 110 | 25 |
| Share of the text | about 4 to 5.5% | about 4% | about 0.18% | about 3.6% | about 1.8% |
The layers are very different. Chapter titles are rare and short. Quotations are common and long. Yigchung notes are mostly tiny (a quarter are 9 characters or less) with a long tail of colophons running to thousands of characters.
Cleaning the data
The raw annotation is never edited. Every fix is written to a separate file, so anyone can see what changed and undo it. The steps are the same in outline for every layer: fix what can be fixed without changing meaning, drop what cannot be trusted, then split by book.
- Span edges. Many spans start or end in the middle of a syllable. We moved those edges to the nearest syllable boundary: 8,513 tsawa spans, about 1,700 sabche spans and 316 chapter spans. For yigchung, 1,395 spans that covered only part of a bracketed note (
{མི་ལྷག},༼ ༽) were widened to the whole note. - Joining neighbours. This is a per-layer decision. For tsawa, annotators often cut one root-text passage into several spans sitting side by side, and we joined them (806 joins). For sabche we did not join, because in the new batch every heading is on its own line and two neighbours are two headings. For chapter we joined only on the same line: an early version that joined across lines glued separate titles together.
- Drifted positions. In some old-batch books the span positions had slid away from the text. For sabche we lined 30 books up again (4,173 spans moved). For chapter, 15 such books could not be recovered and were left out.
- Broken spans. 480 quotation spans and 45 yigchung spans ran backwards (end before start) and were dropped.
- Books left out. Tsawa: 88 books with fewer than 10 spans. Yigchung: 22 books that a reviewer rejected after checking books whose spans often started mid-syllable, a sign of careless marking. Several of those were very large, so this removed about 11,000 yigchung spans.
- Unsure spans. 436 tsawa spans looked like quotations rather than root text. We did not guess; the model ignores them during training.
Splitting the books
Every split is by whole book, so no book is in two splits, and books that share a text are kept together. The test split was locked away and scored once at the end.
| Tsawa | Sabche | Chapter | Quotation | Yigchung | |
|---|---|---|---|---|---|
| Books train / val / test | 84 / 18 / 22 | 261 / 33 / 29 | 308 / 34 / 36 | 210 / 25 / 19 | 184 / 24 / 24 |
| Gold spans in test | 2,734 | 4,079 | 258 | 2,337 | 873 |
Tsawa, sabche and chapter share one base split, so most books stay in the same split across those three layers. Quotation and yigchung were built separately and have their own test books, so scores should be compared within a layer, not across layers.
The models
Every model is jhu-clsp/mmBERT-base, a multilingual ModernBERT that reads 8,192 tokens at once. We fine-tune it to put a tag on every token: O (outside the layer), B (first token of a span) and I (inside a span). The quotation model adds an E tag for the last token, which gives it a direct signal for where a long quotation stops.
Books are much longer than 8,192 tokens, so they are cut into overlapping windows. For yigchung and quotation the overlap was chosen from the longest span, so that every span sits whole inside at least one window.
| Tsawa | Sabche | Chapter | Quotation | Yigchung | |
|---|---|---|---|---|---|
| Labels | BIO | BIO | BIO | BIOE | BIO |
| Window step (tokens) | 5,120 | 5,120 | 5,120 | 4,579 | 4,914 |
| Class weights | O 1, B 1,190.7, I 13.5 | O 1, B 1,298, I 25 | O 1, B 142.5, I 24.1 | O 0.07, B 5, I 1, E 5 | O 0.07, B 5, I 5 |
| Epochs | up to 15, best at 6.9 | 8 | 8 | up to 15, best at step 6,562 | up to 15, best at epoch 4 |
All five use learning rate 1e-5 and an effective batch of 8. The class weights matter because every layer is rare: without them a model learns that “not this layer” is almost always right. Chapter is so rare that plain inverse-frequency weights would have given B a weight of about 20,300, so it uses the square root instead.
Token tags are turned back into spans with Viterbi decoding, which only allows legal tag sequences (no I without a B before it) and charges a small cost each time a span ends, so one passage is not broken into many pieces. For quotation and yigchung that cost was chosen on validation and then frozen before looking at test.
How we score
A predicted span counts as correct when it overlaps a gold span by at least half (character IoU of 0.5 or more), with each gold span matched to at most one prediction. Matches are added up over all test books, and we report precision, recall and F1.
We score whole books, not windows, and we use IoU rather than exact match. Exact match was a bad choice for long spans: our first quotation model scored an exact-match F1 of 0.013, and most of its “misses” were off by a syllable at one end.
Asking an LLM instead
For each layer we wrote one prompt and gave it to Claude Sonnet 5 and Gemini 3.1 Flash Lite. Each call gets one 16,000-character window of a book. LLMs are bad at counting character positions, so the model returns the first and last 20 or so characters of each span, and a small script finds them in the text.
Each prompt was built from measurements on the gold data rather than from intuition. A few examples:
- Tsawa. In 70% of cases, the prose right after a root-text passage repeats and explains its words. We made this “gloss test” the main rule.
- Sabche. 70% of headings open with a number word, 74% end in
ནི, and 98% start a line. - Chapter. About three windows in four have no title at all, so the prompt says first that an empty answer is often right.
- Quotation.
ཇི་སྐད་དུ།really opens a quotation only 48% of the time, so the prompt calls a marker “a reason to look, not proof”. - Yigchung. The prompt describes four kinds of note and says yigchung is only about 2% of the text.
How much text the LLM sees at once matters a lot. On quotation validation books, Gemini given whole-book pieces (up to 250,000 characters) scored F1 0.377 and never touched 74% of the quotations. The same model on 16,000-character windows scored 0.615.
Results
Test split for each layer, F1 with precision and recall.
| Layer | mmBERT (trained) | Claude Sonnet 5 | Gemini 3.1 Flash Lite |
|---|---|---|---|
| Tsawa (22 books) | 0.521 (P 0.485, R 0.562) | 0.603 (P 0.705, R 0.526) | 0.556 (P 0.650, R 0.486) |
| Tsawa, without one interlinear book | 0.596 | 0.714 | 0.661 |
| Sabche (29 books) | 0.962 (P 0.954, R 0.970) | 0.681 (P 0.613, R 0.766) | 0.668 (P 0.600, R 0.754) |
| Chapter (36 books) | 0.829 (P 0.768, R 0.899) | 0.302 (P 0.196, R 0.651) ¹ | 0.540 (P 0.400, R 0.829) ¹ |
| Quotation (19 books) | 0.853 (P 0.831, R 0.876) ² | 0.454 (P 0.714, R 0.333) ³ | 0.696 (P 0.595, R 0.838) |
| Yigchung (24 books) | 0.431 (P 0.464, R 0.402) | 0.254 (P 0.170, R 0.502) | 0.199 (P 0.131, R 0.411) |
- The two chapter LLM numbers used different prompts. Claude ran first, and 88% of its wrong answers were outline headings. We then added one paragraph to the prompt (a window full of short heading-like lines is almost certainly an outline) and ran Gemini with it. That paragraph was written after reading test errors, so the Gemini number is not fully independent of the test set.
- The quotation mmBERT score is measured per window and in tokens, not per book in characters like every other number here. We have not rescored it the same way yet, so it should not be read as a win over Gemini.
- Claude was run on quotation with whole-book pieces, the setup that also hurt Gemini badly. A windowed Claude run has not been done.
Compared with the joint model. Separate models are far ahead on every layer:
| Layer | Joint model v1.3 (August) | Separate mmBERT (this post) |
|---|---|---|
| Tsawa | 0.395 | 0.521 (0.596 without the interlinear book) |
| Sabche | 0.394 | 0.962 |
| Chapter | 0.818 | 0.829 |
| Quotation | 0.373 | 0.853 ² |
| Yigchung | 0.389 | 0.431 |
Treat this as a direction, not an exact gain: the test books, the cleaning and the corpus are all different, and the quotation number is scored per window (note 2). The largest jump is sabche. We have not tested why, but two likely reasons are that it no longer shares one model with six other layers, and that the 30 old-batch books with drifted positions were repaired before training.
Tsawa. This is the one layer where the LLMs beat the trained model. Claude wins on 16 of the 22 test books. The two kinds of system also fail in opposite directions: mmBERT marks too much (3,169 spans for 2,734 gold), while the LLMs are careful and miss about half. One test book, IF3ACC3E1, is an interlinear commentary where the root text is marked a word or two at a time. It holds 27% of the test spans, and every system scores zero on it.
Sabche. The trained model is close to perfect, above 0.9 on 26 of the 29 books. The LLMs mark about 1,000 more headings than there are.
Chapter. The hard part is telling a title from an outline heading. Both are short lines at the start of a line, and a commentary has hundreds of headings for every few titles. The trained model has seen thousands of headings marked “not a title” and learned to skip them. The LLMs find most titles (recall 0.65 and 0.83) but mark far too many.
Quotation. Gemini on windows finds 84% of the quotations. Quotations whose text also appears in a training book are easier for it (recall 0.94 against 0.82). Short quotations (45 characters or less) and very long ones (over 400) are the hardest. On an outside benchmark, DharmaBench QUDT (406 short documents with different annotation rules), the trained model did poorly (span F1 0.130).
Yigchung. This is the hardest layer for every system. The trained model scores 0.666 on validation but 0.431 on test. The LLMs find half the notes but mark three times too many, worst in books that have almost no notes: in five test books with a single gold note, Gemini marked between 111 and 352 spans. One book, P000030, holds 64% of the test gold, so the test score is mostly a score on that book.
Inference cost
| One test run | |
|---|---|
| mmBERT | one GPU pass over the test books; a 24 GB card is enough |
| Claude Sonnet 5 | about $25 (tsawa), $15 (sabche), $12 (chapter), $7 (quotation, whole-book pieces), $13 (yigchung) |
| Gemini 3.1 Flash Lite | about ₹500 per layer run (tsawa, sabche, chapter); not recorded for quotation and yigchung |
The Claude runs for sabche, chapter, quotation and yigchung used the Message Batches API at half price. Some of these figures are worked out from token counts, not checked against the bill.
Next steps
Cleaning the tsawa data
- Decide about interlinear books. One test book (
IF3ACC3E1) marks root text a word or two at a time and holds 27% of the test spans. Four books like it are in training. Either move them to a separate task or drop them, then retrain and rescore. - Review the root text and quotation mix-ups. 436 spans looked like quotations and were masked from training. A person should decide each one, since this is the mistake that costs the most.
- Re-audit the weakest books. Five test books score under 0.31 for every system (
P000269,ICDC84458,P000242,P000144,IE5895799). Check whether the gold in them follows the same rules as the rest before blaming the model. - Look at the new batch. Every system does worse on new-batch books (Claude 0.531 against 0.672 on old). Find out whether the annotation rules differ between batches.
Cleaning the yigchung data
- Fill in missing notes. The gold is incomplete: in 19 of 129 books with 5 or more notes, all the notes sit in less than 30% of the book, the shape of someone who stopped partway. In a sample of 60 books, 67 note strings appear again in the same book without being marked. Use the model’s and the LLMs’ extra predictions as a list of candidates, and have a reviewer confirm or reject them.
- Stop treating unmarked text as “no note”. Only 48 of the 232 books are known to be fully annotated. In the other 184, unmarked text should be left out of the loss or weighted lower, instead of being taught as a confident “not yigchung”.
- Fix the test split. One book holds 64% of the test gold. Rebuild the split with a cap on how much any single book can hold, so the score reflects more than one book.
- Test the newer prompt. A tightened LLM prompt (headings and outline labels are not notes, an empty answer is often right) is written but has not been run on the test set yet.
Fine-tuning an open LLM such as Gemma
The results point two ways. The LLMs read meaning well (they won on tsawa), but they do not know how rare a layer is and mark far too much. The trained encoder knows the base rate but cannot read meaning as deeply. Fine-tuning an open LLM such as Google’s Gemma on our own gold could get both:
- Same task, same format. Train it to return spans in the same head-and-tail anchor format the prompts use, on the same 16,000-character windows, so the locator, the gold and the scorer stay unchanged and the results line up with everything in this post.
- Train on real answers, including empty ones. Most windows in the training data have no span for a given layer. Seeing thousands of empty answers is how the model learns to stop over-marking.
- Start where mmBERT is weakest. Tsawa and yigchung first, since those are the layers where meaning matters most and the encoder scores lowest.
- Cheap to train and run. A parameter-efficient fine-tune (LoRA) of a small Gemma model fits on one rented GPU, and the result runs locally with no API bill.
- Compare fairly. Score it on the same frozen test books with
src/score_spans.py, next to mmBERT, Claude and Gemini.
Links
| Link | |
|---|---|
| Zero-shot prompts, LLM results and scoring for all five layers | layer-detection-llm-zeroshot |
| Tsawa pipeline and model card | tsawa-layer-detection, model https://huggingface.co/BDRC/Bo-Tsawa-Detection |
| Sabche pipeline | sabche-layer-detection, model https://huggingface.co/BDRC/Bo-Sabche-Detection |
| Chapter pipeline | chapter-layer-detection, model https://huggingface.co/BDRC/Bo-Chapter-Detection |
| Quotation pipeline, model and LLM benchmark | Tibetan-quotation-detection, model https://huggingface.co/BDRC/Bo-Quotation-Detection |
| Yigchung pipeline and model | tibetan-yigchung-detection, model https://huggingface.co/BDRC/Bo-Yigchung-Detection |
Some of these repositories, models and datasets are private.
Credits
Source texts were digitised and made available by the Buddhist Digital Resource Center (BDRC). Annotations were prepared through OpenPecha with support from the Tsadra Foundation.