Testing a Tibetan quotation-detection model on DharmaBench QUDT

🚀 WG སྡེ་ཚན། 📑BDRC-Etexts WG
1

What we ran

We have been training a token-classification model for structural annotation of Tibetan texts — quotations, root text, sabche, yigchung, and other layers — on OpenPecha/tsadra commentary annotations. The base is mmBERT-base (~300M parameters), with an 8,192-token context window.

DharmaBench (Hashiloni et al., IJCNLP-AACL 2025) includes QUDT, a Tibetan quotation-detection task: 406 test passages of roughly 1,000 characters each, cut from 14 classical works, with 335 gold QUOTE spans. The paper evaluates twelve LLMs zero-shot and reports no encoder baseline at all — the 512-token context of the usual multilingual encoders was too short for ~1,000-character samples, so the QUDT train split (600 samples) went unused.

We evaluated our checkpoint zero-shot and cross-scheme: trained only on commentary annotations, never shown a single QUDT example, scored with the benchmark’s own eval4ner MUC routines in type mode.

The headline number, with the row that has to sit beside it

system MUC type F1
Gemini 2.0 Flash 76.23 ±0.47
ours — mmBERT v1.3 + Viterbi (zero-shot) 65.34
Gemini 2.5 Flash 61.81 ±0.93
baseline — predict nothing 60.09
Llama 4 Scout 59.61 ±0.25
Claude 3.7 Sonnet 57.63
Qwen2.5-72B 57.32 ±1.00
Gemini 2.5 Pro 57.22
GPT-4o mini 55.82 ±0.49
Claude 4 Sonnet 55.24
GPT-4o 52.86
DeepSeek-R1 49.30
Claude 3 Haiku 22.16 ±1.01

To the credit of the DharmaBench authors, they publish that italicised row themselves. 60.09 is exactly 244/406 — the fraction of test passages containing no quotation. The MUC score is a per-document macro-average in which an empty gold and an empty prediction score a full 1.0, so emitting nothing scores 60.09, and most of the leaderboard is below it.

So the defensible claim is not “beats GPT-4o and Claude.” It is narrower and still worth stating: a 300M-parameter encoder, run zero-shot on a different annotation scheme, on CPU, is second on the board and one of only three systems that beat the do-nothing baseline. Our number is a single point estimate against 3-seed means, and we have not yet bootstrapped a confidence interval.

What the score is actually made of

Decomposing the 265.3 document-points behind 0.6534:

  • 231 of 244 empty passages correctly left empty → 231 points, 87% of the total.
  • 65 of 335 gold quotations touched by any prediction at all.
  • 270 of 335 gold quotations (81%) missed with zero overlap.

Span-level, the picture is plain: IoU@0.5 F1 = 0.159. Measured in characters, the model marks 4.1% of the corpus as quotation; the answer key marks 33.7%. Character recall is 0.070 — and identical across all seven Viterbi penalty settings we swept. No decoding change moves it.

Two useful sub-findings fell out of that:

  • Viterbi decoding is a genuine gain (+10.3 type F1 over argmax), but it works by consolidating — it collapses 1,635 shredded fragments into 80 whole spans and cleans false fires off 25 empty passages. Characters marked barely change (17,122 → 16,979). It improves assembly, not detection.
  • Selecting the decoder on MUC type F1 selects for silence. By IoU — the metric that measures actually finding quotations — the optimum is a lower penalty that finds nine more real quotations while scoring lower on the headline metric.

Why recall collapses: anthology vs. commentary

This is the part we think is of general interest, because it is about the texts rather than the model.

Our training data is commentary. There, a quotation is a short insert embedded in surrounding prose and framed by citation particles — ཞེས་ / ཅེས་ and their relatives. The model learned, correctly for that genre, that a quotation is a marked insert.

QUDT’s positives come overwhelmingly from anthologies: Śikṣāsamuccaya, Sūtrasamuccaya, and mDo kun las btus pa chen po supply 69% of the data and essentially all the quotations. These are texts that are mostly quotation, cut into ~1,000-character windows. The consequence is structural: 202 gold quotes start at character 0 and 209 end at the passage’s last character — the citation frame is stranded in the neighbouring chunk. In 170 of 393 quote-bearing chunks the quotation covers more than 90% of the window. The gold answer is “this entire passage is a quotation,” with no surface marker present.

Recall follows that gradient exactly:

gold quote occupies… n character recall
< 20% of the passage 146 0.275
20–50% 96 0.076
50–90% 62 0.034
> 90% of the passage 31 0.013
n character recall
quote sits fully inside the passage 163 0.146
quote runs off the window edge 172 0.027

21× better on short quotes than passage-length ones; 5.4× better when both edges are visible. On the half of QUDT that is our task — short, framed quotes — recall is 0.275: weak, but an ordinary number with ordinary levers. On the other half it is close to zero, and we do not think that half is a fair test of a commentary-trained model.

A separate weakness shows up on our own data and compounds this: token recall inside annotated spans is 0.58–0.61 and degrades with span length (1.000 on short Author spans, ~0.55 on long Tsawa spans). The model fires at onsets and loses the thread past roughly 100 characters. Our corpus’s mean quotation is 134 characters; DharmaBench’s is 417.

Notes for anyone else evaluating on QUDT

Things we hit that are worth knowing before you compare numbers:

  1. The metric is a document-level macro-average, not span micro-F1, despite the table caption. Empty-vs-empty scores 1.0. Always report the empty / non-empty split alongside any aggregate.
  2. In the released code, gold is not filtered to QUOTE — all five labels (QUOTE, OP, CP, TITLE, GEN_SRC) go into scoring while the prompt asks only for QUOTE. A perfect QUOTE detector therefore caps around 80.6, not 100.
  3. The offsets field is internally inconsistent for ~16% of QUOTE spans: text[start:end] reproduces the annotated string for 281 of 335; 28 need end+1; 26 match neither. Normalise before measuring boundary error.
  4. Malformed model output becomes an empty prediction, which on this test set is a good answer 60% of the time. Failing to emit valid JSON is rewarded.
  5. Train and test are a random chunk-level split of the same 14 works, so this measures in-distribution performance; there is no work-level held-out set.
  6. The prompt has some rough edges — it asks for “the exact text span that contains the simile” (copy-paste from another task), asks for the “minimal required span” while gold quotes are the full quotation, and its single worked example is a closing particle rather than a quote.

None of this makes QUDT less valuable. It is a first benchmark in a genuinely under-resourced area, and the window-truncation artifact is an honest consequence of building one out of anthology texts. It does mean a leaderboard position is a weaker signal than the raw number suggests — in both directions.

Happy to share the evaluation harness and the per-document prediction files if anyone wants to reproduce or push on this.

Credits

Built on the OpenPecha / tsadra annotated corpus, whose seven annotation layers over 266 Tibetan books are the ground truth for everything above.

Developed by Dharmaduta based on specifications from the Buddhist Digital Resource Center for the project The BDRC Etext Corpus, funded by the Khyentse Foundation.

Sources