Tibetan texts have a distinct visual structure — running headers (མགོ་བྱང་), dense body text, footnotes, and footers — that differs significantly from the Latin-script documents most document AI models are trained on. As part of the BDRC Etext Corpus project at OpenPecha, we set out to answer a simple question: can off-the-shelf layout detection models reliably parse Tibetan book pages?
To find out, we built a benchmark dataset of Tibetan modern book pages annotated with four layout classes — header, text-area, footnote, and footer — and evaluated 9 models against it. The results reveal both promising capabilities and significant blind spots.
The Benchmark
Our primary evaluation dataset contains 879 ground-truth bounding boxes across four classes: header (270), text-area (312), footnote (82), and footer (215). During benchmark preparation, all nearby same-class bounding boxes were merged into single consolidated boxes, ensuring each annotation represents a complete layout region rather than fragmented sub-regions. We also prepared a second split benchmark (1,240 boxes) where annotations are kept as individual bounding boxes, even when adjacent and of the same class, to study the impact of annotation strategy on evaluation. All evaluations in this report are conducted against these benchmarks. The dataset is publicly available on the complete training and benchmark data can be found at BDRC/TDLA-Training-Dataset. All model inference code can be found on GitHub.
We tested a mix of model types:
- Dedicated layout detection models: Surya, PaddleOCR, PaddleVLM, DocYOLO, YOLO11, Eric YOLO
- Document OCR pipeline: Dots.OCR
- Vision Language Models (zero-shot): Gemini 2.5 Flash and Gemini 3.1 Pro
- Fine-tuned models (on our Tibetan training data): TDLAv9 (based on YOLO26m) and DocLayout-ft (based on DocLayout-YOLO).
For models that produce fragmented output, we applied a post-processing step that merges same-class bounding boxes within a 1500px margin. All models were evaluated using standard COCO-style metrics — mAP@0.50 and mAP@0.50:0.95.
Results at a Glance
Benchmark A: Merged annotations (879 GT boxes)
In this benchmark, nearby same-class bounding boxes are merged into single consolidated regions. Results shown use post-processed (merged) output where applicable. Vision LM models (Gemini) did not require post-processing.
| Model | Type | mAP@0.50 | mAP (COCO) | Header AP | Text-area AP | Footnote AP | Footer AP |
|---|---|---|---|---|---|---|---|
| Surya (merged) | Off-the-shelf | 81.9% | 49.0% | 77.6% | 95.8% | 81.7% | 72.4% |
| TDLAv9 (merged) | Fine-tuned | 73.2% | 41.8% | 83.5% | 95.9% | 68.9% | 44.4% |
| DocLayout-ft | Fine-tuned | 72.0% | 44.9% | 83.6% | 98.5% | 74.2% | 31.6% |
| PaddleVLM | Off-the-shelf | 68.9% | 48.9% | 51.3% | 96.6% | 82.6% | 45.1% |
| Eric YOLO | Off-the-shelf | 65.1% | 46.3% | 52.4% | 87.9% | 66.7% | 53.4% |
| Dots.OCR | Off-the-shelf | 61.3% | 37.1% | 60.2% | 89.7% | 74.9% | 20.3% |
| DocYOLO (merged) | Off-the-shelf | 48.9% | 41.1% | 62.7% | 77.2% | 0.0% | 55.6% |
| Paddle (merged) | Off-the-shelf | 38.2% | 20.6% | 40.4% | 65.0% | 0.0% | 47.4% |
| Gemini 2.5 Flash | Vision LM | 28.3% | 12.2% | 5.3% | 90.7% | 15.8% | 1.4% |
| Gemini 3.1 Pro | Vision LM | 26.9% | 13.9% | 38.8% | 23.7% | 24.3% | 20.9% |
| YOLO11 | Off-the-shelf | 18.4% | 10.8% | 0.2% | 53.3% | 1.2% | 18.9% |
Benchmark B: Split annotations (1,240 GT boxes)
In this benchmark, all bounding boxes are kept as individual annotations — nearby same-class boxes are not merged. All results are raw inference with no post-processing.
| Model | Type | mAP@0.50 | mAP (COCO) | Header AP | Text-area AP | Footnote AP | Footer AP |
|---|---|---|---|---|---|---|---|
| TDLAv10 | Fine-tuned | 73.4% | 39.6% | 84.4% | 74.8% | 88.5% | 45.9% |
| TDLAv9 | Fine-tuned | 46.4% | 22.3% | 38.4% | 34.8% | 71.1% | 41.4% |
| DocLayout-ft | Fine-tuned | 45.0% | 24.4% | 37.8% | 36.4% | 75.3% | 30.6% |
| Surya | Off-the-shelf | 41.5% | 17.2% | 40.9% | 35.7% | 28.3% | 61.0% |
| Eric YOLO | Off-the-shelf | 37.3% | 21.0% | 55.3% | 0.0% | 47.3% | 46.6% |
| DocYOLO | Off-the-shelf | 34.6% | 26.1% | 48.7% | 31.6% | 0.0% | 58.3% |
| Paddle | Off-the-shelf | 25.9% | 12.2% | 24.3% | 36.3% | 0.0% | 42.9% |
Key Findings
- Surya leads the pack. With an mAP@0.50 of 81.9%, Surya was the strongest overall performer on the merged benchmark, achieving balanced performance across all four classes. It was also the only model to exceed 70% AP on every single class (Header: 77.6%, Text-area: 95.8%, Footnote: 81.7%, Footer: 72.4%), demonstrating that its architecture generalizes reasonably well to Tibetan page layouts.
- Text-area is the easy class; footnote is the divider. Nearly every model could detect the main text body — even YOLO11, which struggled elsewhere, managed 53.3% AP on text-area, while both fine-tuned models exceeded 95%. Footnote detection, however, sharply separated the capable models from the rest. DocYOLO, Paddle, and YOLO11 scored 0% to 1.2% AP on footnotes, failing to reliably detect them. In contrast, Surya (81.7%), PaddleVLM (82.6%), and DocLayout-ft (74.2%) handled footnotes well.
- Fine-tuning delivers strong, consistent results. TDLAv9 and DocLayout-ft, fine-tuned on our Tibetan training data, both reached or exceeded 72.0% mAP@0.50 and achieved the highest header AP scores in the benchmark (83.5% and 83.6% respectively). DocLayout-ft also achieved the best text-area AP at 98.5%. While they don’t quite beat Surya’s overall mAP, they produce cleaner output that requires no post-processing.
- Fine-tuned YOLO models show promise. Eric YOLO, a YOLO model fine-tuned for document layout, ranked 5th overall (mAP@0.50: 65.1%) and achieved a footer detection AP of 53.4%. However, this came at a cost — it produced a high volume of predictions (605 raw), resulting in elevated false positive counts, particularly on headers and footers.
- Vision LMs struggle with spatial precision. Gemini 2.5 Flash achieved a surprisingly strong 90.7% AP on text-area, rivaling the best specialized models. But it collapsed on headers (5.3% AP) and footers (1.4% AP), and its COCO mAP of just 12.2% reveals that even its correct detections have imprecise bounding boxes. Gemini 3.1 Pro showed more balanced but uniformly mediocre performance across all classes. Both models were prompted with structured output schemas, yet the spatial grounding required for tight bounding boxes remains a clear weakness for current VLMs.
- Localization quality matters. Comparing mAP@0.50 with COCO mAP tells an important story. PaddleVLM nearly matches Surya on COCO mAP (48.9% vs 49.0%) despite a lower mAP@0.50 (68.9% vs 81.9%), suggesting its detections — when correct — are more precisely localized. Meanwhile DocYOLO’s mAP barely drops from 0.50 to 0.95 thresholds, indicating that while it misses many elements, the ones it finds are localized very tightly.
Error Patterns
Exploring the false positive and false negative distributions across models reveals distinct failure modes. Gemini 2.5 Flash produces 228 false positive headers — nearly as many as the 270 actual headers in the dataset — suggesting it frequently misclassifies other elements as headers. PaddleVLM generates 136 false positive footers, likely splitting or duplicating footer regions. On the false negative side, YOLO11 misses 267 of 270 headers entirely, and models without footnote support (DocYOLO, Paddle) naturally miss all 82 footnotes.
Fine-Tuned Models: TDLAv9 and DocLayout-ft
We trained two models specifically on Tibetan layout data to see how domain-specific fine-tuning compares against off-the-shelf models:
- TDLAv9 — a YOLO26m model fine-tuned on our Tibetan training set
- DocLayout-ft — a DocLayout-YOLO model fine-tuned on the same data
Both fine-tuned models achieved strong results, with DocLayout-ft reaching 72.0% mAP@0.50 and TDLAv9 at 73.2%. Notably, both scored above 83% AP on headers and above 95% AP on text-area, demonstrating that even modest fine-tuning on domain-specific data yields competitive performance. Their main weakness remains footer detection (44.4% and 31.6% AP respectively), where off-the-shelf Surya still leads.
The Impact of Post-Processing: Box Merging
Many layout detection models produce fragmented predictions — splitting a single text region into multiple overlapping boxes. We applied a simple post-processing step to test whether merging same-class bounding boxes within a 1500px margin could improve results.
| Model | mAP@0.50 (raw) | mAP@0.50 (merged) | Change | Preds (raw) | Preds (merged) |
|---|---|---|---|---|---|
| Surya | 34.3% | 81.9% | +47.6 | 1,787 | 862 |
| Eric YOLO | 35.1% | 34.0% | −1.2 | 605 | 567 |
| Paddle | 26.9% | 38.2% | +11.3 | 689 | 567 |
| DocYOLO (YOLO) | 27.1% | 48.9% | +21.7 | 1,258 | 722 |
| TDLAv9 | 73.1% | 73.2% | +0.1 | 835 | 832 |
| DocLayout-ft | 72.0% | 71.8% | −0.2 | 819 | 817 |
- Surya benefits the most. Raw Surya produces 1,787 predictions — over twice the 879 ground-truth boxes — with heavy fragmentation especially on text-area (1,050 predictions for 312 ground-truth boxes, yielding just 10.2% AP). After merging, predictions drop to 862 and text-area AP jumps to 95.8%. This suggests Surya’s underlying detection is strong but its raw output is highly fragmented for Tibetan pages.
- DocYOLO also improves significantly, going from 27.1% to 48.9% mAP@0.50 as its 1,258 raw predictions consolidate to 722. Footer detection in particular reaches a solid 55.6% AP with much tighter boxes.
- Fine-tuned models are already clean. TDLAv9 and DocLayout-ft show virtually no change after merging — their predictions are already well-consolidated, producing close to one box per ground-truth element. This confirms that fine-tuning on domain data not only improves accuracy but also produces cleaner output that doesn’t require post-processing.
- Eric YOLO slightly degrades, suggesting the merge step occasionally combines boxes that should remain separate, particularly for footnotes where AP drops from 46.9% to 36.3%.
The takeaway: post-processing can rescue models with fragmented output (especially Surya), but it’s no substitute for domain-specific training. The best results come from fine-tuned models that produce clean predictions natively.
Annotation Strategy Matters: Merged vs Split Benchmarks
Our initial benchmark merged nearby same-class bounding boxes into consolidated regions (879 ground-truth boxes). But this raises a question: does the annotation style itself influence evaluation outcomes? To test this, we prepared a second benchmark and training data where all bounding boxes are kept as individual split annotations, even when they are adjacent and of the same class. This resulted in 1,240 ground-truth boxes — a 41% increase — reflecting a more granular view of the page layout.
We re-evaluated several models on the split benchmark using raw inference output (no post-processing), and also trained a new model — TDLAv10 — on training data annotated in the same split style.
Split benchmark results (1,240 GT boxes, no post-processing)
| Model | Type | mAP@0.50 | mAP (COCO) | Header AP | Text-area AP | Footnote AP | Footer AP |
|---|---|---|---|---|---|---|---|
| TDLAv10 | Fine-tuned | 73.4% | 39.6% | 84.4% | 74.8% | 88.5% | 45.9% |
| TDLAv9 | Fine-tuned | 46.4% | 22.3% | 38.4% | 34.8% | 71.1% | 41.4% |
| DocLayout-ft | Fine-tuned | 45.0% | 24.4% | 37.8% | 36.4% | 75.3% | 30.6% |
| Surya | Off-the-shelf | 41.5% | 17.2% | 40.9% | 35.7% | 28.3% | 61.0% |
| Eric YOLO | Off-the-shelf | 37.3% | 21.0% | 55.3% | 0.0% | 47.3% | 46.6% |
| DocYOLO | Off-the-shelf | 34.6% | 26.1% | 48.7% | 31.6% | 0.0% | 58.3% |
| Paddle | Off-the-shelf | 25.9% | 12.2% | 24.3% | 36.3% | 0.0% | 42.9% |
The results tell a clear story about annotation consistency:
- TDLAv10 dominates the split benchmark. Trained on split-style annotations matching the evaluation format, TDLAv10 achieves 73.4% mAP@0.50 — nearly double the next best model. It scores above 73% on header, text-area, and footnote detection, with footnote AP reaching 88.5%. This demonstrates that aligning training and evaluation annotation strategies is critical for performance.
- Models trained or tuned on merged data struggle. DocLayout-ft, which scored 72.0% on the merged benchmark, drops to 45.0% on the split benchmark. Its text-area AP falls from 98.5% to 36.4% — the model learned to predict consolidated regions and can’t match the granular split annotations.
Similarly, TDLAv9, which achieved 73.2% on the merged benchmark, drops significantly to 46.4% on the split benchmark (with its text-area AP collapsing from 95.9% to 34.8%). Both models were fine-tuned on the merged annotation style, causing them to generate consolidated regions that mismatch the granular split layout ground truth. Interestingly, TDLAv9 still edges out DocLayout-ft and raw Surya on this split benchmark, showcasing solid underlying baseline performance despite the style mismatch.
The annotation gap is the key insight. When a model’s output style matches the benchmark’s annotation style, performance jumps dramatically. TDLAv10 on the split benchmark (73.4%) closely mirrors TDLAv9’s performance on the merged benchmark (73.2%). The underlying detection capability is similar — what changes is whether the model’s predictions align with how the ground truth is annotated.
This has practical implications for building Tibetan document processing pipelines: the choice between merged and split annotations should be driven by the downstream task. If the goal is to extract complete text regions for OCR, merged annotations and models like Surya (with post-processing) or TDLAv9 are ideal. If the goal is fine-grained structural analysis — identifying individual text blocks, line groups, or separate footnote entries — split annotations and models like TDLAv10 are the better fit.
What This Means for Tibetan Document Processing
No single model solves Tibetan layout detection out of the box, but the results are encouraging. On the merged benchmark, Surya with post-processing and our fine-tuned models (TDLAv9, DocLayout-ft) all achieve over 70% mAP. On the split benchmark, TDLAv10 — trained on matching split annotations — reaches 73.4% mAP, demonstrating that annotation consistency between training and evaluation is as important as model architecture.
The choice of benchmark and model should be driven by the downstream task. For extracting complete text regions for OCR or e-text production, the merged benchmark and models like Surya (with post-processing) or TDLAv9 are the right fit. For fine-grained structural analysis — identifying individual text blocks, separate footnote entries, or distinct header segments — the split benchmark and TDLAv10 are better suited.
For the BDRC Etext Corpus pipeline, these benchmarks inform which model to deploy. The full interactive benchmark dashboard, dataset, and inference code are all open source. We welcome contributions — particularly additional model evaluations and expanded annotation coverage across different Tibetan text traditions.
Appendix: Vision LM Prompt
For the Gemini models, we used structured prompting with a Pydantic output schema to enforce consistent bounding box output.
System prompt
Role: Expert Document Layout Analysis AI.
Task: Analyze book page images and detect all distinct layout elements using bounding boxes.
Classes: 0 = Header, 1 = Text-area, 2 = Footnote, 3 = Footer
Instructions:
1. Scan the provided image and identify all headers, main text areas, footnotes, and footers.
2. For every detected element, assign the correct integer class_id.
3. Calculate the bounding box coordinates (x1, y1, x2, y2) normalized between 0.000 and 1.000, where x1, y1 is the top-left corner and x2, y2 is the bottom-right corner.
4. Output the exact structured data adhering to the required schema. Do not include any conversational text.
User prompt
Analyze the attached book page image. Detect all layout elements and return their class IDs and normalized bounding box coordinates (x1, y1, x2, y2) strictly following the required output schema.
Output schema (Pydantic)
from pydantic import BaseModel, Field
class LayoutElement(BaseModel):
class_id: int = Field(
description=“Must be exactly one of: 0 (Header), 1 (Text-area), 2 (footnote), 3 (footer).”
)
x1: float = Field(description=“Top-left X coordinate, normalized between 0.0 and 1.0”)
y1: float = Field(description=“Top-left Y coordinate, normalized between 0.0 and 1.0”)
x2: float = Field(description=“Bottom-right X coordinate, normalized between 0.0 and 1.0”)
y2: float = Field(description=“Bottom-right Y coordinate, normalized between 0.0 and 1.0”)
class PageLayoutDetection(BaseModel):
elements: list[LayoutElement] = Field(
description=“A comprehensive list of all detected layout elements on the page.”
)
Credits
Developed by Dharmaduta based on specifications from the Buddhist Digital Resource Center for the project The BDRC Etext Corpus, funded by the Khyentse Foundation.