The problem
We had a large batch of scanned Tibetan texts. All of them had already been run through four different OCR systems. None of the four was good enough to use on its own.
- BDRC OCR (
ocrv1) is our older line-detection engine. It is strong on clean printed uchen. It gets fragile when the page layout is messy or the script turns cursive. - Qwen OCR (
ocr_qwen_v1) is a vision-language model. It is usually the sharpest reader of dense Tibetan. But like most VLM OCR it sometimes breaks down. It can get stuck repeating one syllable for the rest of the page. It can also run out of its generation budget partway through and leave English status text like “tokens exhausted” sitting in the middle of the transcription. - Google Vision OCR (
gv) is steady and dependable. It is weaker on cursive or heavily styled Tibetan. It is our best engine on non-Tibetan and Latin pages. - PaddleOCR (
paddle_ocr_v1) is another strong reader. What matters most is that it fails in a different way from Qwen. Paddle rarely produces a cleanAAAA…loop. Instead it produces drifting mantra storms, where a mantra-like fragment mutates as it repeats. It can also invent text on a page the other engines read as empty.
Every page also came with a script label from an earlier pass. That label told us in advance whether the page was blank, non-Tibetan (a title page or a library stamp), or which Tibetan script it used.
The problem was simple to state. For every page in the corpus, pick one transcription worth keeping.
We already had a first answer to that. The earlier prototype ran on three engines, with no Paddle in the mix. It routed blank and non-Tibetan pages by label, then trusted Qwen on every Tibetan page unless a hallucination check fired. If the check fired, uchen pages fell back to ocrv1 and everything else fell back to gv. It worked. Qwen shipped on 4,016 of the 4,119 pages, or 97.5%, and only 54 pages tripped the detector.
But that design had a ceiling. Qwen was the default, so a page was either all Qwen or all fallback. When two good engines disagree on a single syllable, picking a favourite engine is a coin flip. Adding Paddle gave us a second strong reader that fails in its own way, so the real question changed. Can we catch each VLM’s own failure mode, and then let the engines vote when nothing has failed?
What we did about it
We built a merge pipeline that does three things. It routes each page using the script label. It runs a separate hallucination detector against each VLM. And when both VLMs come back clean, it collates the surviving engines into a token-level majority vote instead of picking a winner. When only one VLM survives, we first check that it actually produced something.
The workflow
Four steps drive this.
Step 1. Route by page type. The script label settles the easy cases on its own.
| Page labelled… | What we output | Reasoning |
|---|---|---|
| blank | whichever engine returned the shortest text | An empty or near-empty result is the honest answer for a blank page, and the rule is simple and predictable. |
| non-Tibetan (title pages, stamps) | Google Vision (gv) |
Google Vision is our strongest engine on ordinary printed Latin and mixed-script pages. |
| Tibetan content (uchen and other scripts) | continue to steps 2, 3 and 4 |
Step 2. Catch each VLM’s own failure mode. The two detectors are built differently, because the two engines break differently.
Qwen (src/qwen_hallucination.py) looks for two things:
- Repeating loops. A sliding window scans the whole page for a character or short phrase repeated at least four times in a row. This catches a flood of the Tibetan tsek, or a syllable cycling on and on, anywhere on the page rather than just at the end. We only flag it once the repeated stretch is long enough and covers a real share of the page. That way normal repetition in ordinary prose does not trip it.
- Leaked stop messages. English fragments like “tokens exhausted” or “finish_reason=length” that Qwen leaves behind when it runs out of budget mid-page. The phrase list is narrow, so a real English catalog page is not flagged.
Paddle (src/paddle_hallucination.py) needs a wider net, because exact loop matching misses drifting storms. It flags a page when the text is long and one of these holds:
- Low variety. A low unique-3gram rate or unusually high gzip compressibility. This is the rule that actually catches mutating mantra storms.
- Text on an empty page. Paddle writes a lot where
ocrv1andgvboth see almost nothing, and the text is repetitive. Without the variety condition this would wrongly flag title pages thatocrv1simply missed. - Length inflation. Paddle runs far longer than the other engines and is less varied at the same time. Length alone is never enough, since
ocrv1is honestly short on cursive pages. - Exact loops. The same repeat rule Qwen uses, plus a softer span rule for long inflated pages.
Paddle returning almost nothing is not treated as hallucination. That case is handled in step 3.
Step 3. When one VLM is flagged, check the other one is not empty. The obvious move is to hand the page to the other VLM, and most of the time that is right. A clean engine beats a vote that includes a broken one. But passing the detector and reading the page are two different things. A VLM that returned nothing passes every check for free. There is no loop to find, no leaked phrase, no inflated length. Sending the page there would ship a blank transcription for a page with text on it.
So we test the surviving VLM for empty text before we accept it. If it is empty we treat the page as though both VLMs failed and drop to the classical engines. That means ocrv1 for uchen and gv for everything else. The check is strict. Only a truly empty or whitespace-only string counts. A short but real transcription on a sparse page still wins the page.
The guard runs the same way on both sides. Qwen flagged with Paddle empty falls back, and Paddle flagged with Qwen empty falls back. So the classical engines are now reached by two routes. Either both VLMs hallucinated, or one hallucinated while the other said nothing. In both cases the reason is the same. No VLM gave us a usable reading.
Step 4. When both VLMs are clean, vote instead of choosing. We hand three witnesses to Pydurma. Those are Paddle, Qwen, and a third engine picked by script (ocrv1 for uchen, gv otherwise). Pydurma normalizes and tokenizes the Tibetan, aligns the witnesses with its FDMP aligner, and weighs each token position with a count weigher. We take the highest-weight reading at each token. Ties keep witness order, so Qwen first, then Paddle, then the third engine. We label the output pydurma rather than an engine name, because it can mix readings and is not a copy of any single input.
Every check runs on text we already have. There are no external calls and no extra ML. So the merge stays fast and predictable, and we can re-run it any time the upstream OCR or classification data changes.
Results: running it across all 10 volumes
We ran the pipeline over the full corpus. That is all 10 scanned works we have classification labels for, and we let it make the real final call on every one of the 4,119 pages.
| Work / volume | Pages | Blank | Non-Tibetan | Qwen flagged | Paddle flagged | → Paddle | → Qwen | → ocrv1 | → gv | → Majority |
|---|---|---|---|---|---|---|---|---|---|---|
| W3KG424 / I3KG1177 | 30 | 1 | 0 | 2 | 0 | 2 | 0 | 0 | 0 | 27 |
| W1LT0482 / I1LT0482 | 200 | 6 | 0 | 2 | 2 | 2 | 2 | 0 | 0 | 190 |
| W8CZ237 / I8CZ786 | 623 | 3 | 4 | 40 | 13 | 35 | 8 | 0 | 5 | 568 |
| W8LS67515 / I8LS67520 | 596 | 2 | 4 | 3 | 10 | 3 | 10 | 0 | 0 | 577 |
| W8LS26135 / I8LS26137 | 122 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 120 |
| W1KG22442 / I1KG22493 | 712 | 7 | 1 | 4 | 25 | 1 | 22 | 0 | 3 | 678 |
| W8LS32582 / I8LS32754 | 582 | 1 | 1 | 2 | 1 | 2 | 1 | 0 | 0 | 577 |
| W8LS18063 / I8LS18085 | 50 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 48 |
| W3PD1002 / I1KG81117 | 640 | 7 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 631 |
| W3PD1002 / I1KG81118 | 564 | 7 | 1 | 1 | 1 | 0 | 0 | 1 | 0 | 555 |
| Total | 4,119 | 35 | 14 | 54 | 54 | 45 | 45 | 1 | 8 | 3,971 |
(The flagged columns count detector hits. The routing columns count final decisions. A page where both engines are flagged, or where one is flagged and the other came back empty, shows up in the flag columns but is routed to ocrv1 or gv.)
What the corpus looks like. Of 4,119 pages, 35 are blank, 14 are non-Tibetan, and 4,070 hold Tibetan content. Those Tibetan pages are mostly uchen at 3,371, with 501 druma, 100 danyig pedri, 68 multiscript, 25 gyuyig tsugdri, and 5 non-plain-text pages. There is enough cursive and mixed material here that trusting the line-based engine alone was never an option.
What the pipeline decided.
| Outcome | Pages | Share of Tibetan-content pages |
|---|---|---|
| Pydurma majority vote (both VLMs clean) | 3,971 | 97.6% |
| Qwen alone (Paddle flagged, Qwen non-empty) | 45 | 1.1% |
| Paddle alone (Qwen flagged, Paddle non-empty) | 45 | 1.1% |
gv fallback (no usable VLM, non-uchen) |
8 | 0.2% |
ocrv1 fallback (no usable VLM, uchen) |
1 | 0.02% |
Against the old prototype. The two runs cover the same 4,119 pages, so they line up directly.
| Three-engine prototype | Four-engine pipeline | |
|---|---|---|
| Pages on a single VLM’s raw output | 4,016 (97.5%) | 90 (2.2%) |
| Pages on a three-engine vote | 0 | 3,971 (97.6%) |
| Pages on a classical engine | 103 (2.5%) | 9 (0.2%) |
| Tibetan pages with a VLM flagged | 54 (1 in 75) | 99 (1 in 41) |
The number that matters is the 97.6%. Under the old design those pages shipped Qwen’s reading as-is. Now they ship a token-level consensus of three engines, and Qwen only wins the spots where it is not outvoted. The fallback rate also drops hard, from 103 pages down to 9, because a flagged Qwen page no longer has to go to a classical engine when Paddle read it fine.
More pages get flagged now, 99 instead of 54, but that is two detectors working rather than one engine getting worse. Both fire at the same rate, 54 pages each, and they overlap on only 9. That is the whole point of running both. The failures are mostly independent, so on 90 of those 99 pages there was still a healthy VLM to fall back on.
The per-work spread is worth a look too. W8CZ237 accounts for 40 of the 54 Qwen flags, since it is a dense and difficult volume where Qwen loops far more than average. W1KG22442 accounts for 25 of the 54 Paddle flags. Neither engine is worse across the board. They are each worse on different material, which is why we route per page rather than per work. The detector behaviour itself is pinned down by unit tests. There are 13 on the Qwen loop and stop-phrase rules, 7 on the Paddle drift and inflation rules, and a set on the routing tree, covering the empty-survivor fallback on both sides plus negative cases for ordinary Tibetan prose, ordinary English catalog text, and honestly short cursive pages. Real flagged examples from this corpus live in Data/output/paddle_drifted_mantra_examples.md.
Code & resources
Previous version: Prototype for merging OCR result, the three-engine pipeline this one replaces.
GitHub repository: github.com/OpenPecha/ocr_merger
Collation library: Pydurma, which handles Tibetan normalization, tokenization, FDMP alignment, and token weighing.
Data source: the scanned works and all four OCR outputs come from BDRC, the Buddhist Digital Resource Center. BDRC is an archive of Tibetan and wider Buddhist-canon texts, formerly known as TBRC. Its public S3 bucket (bec.bdrc.io) hosts the Parquet files this pipeline reads from.
Running it: Python 3.12+, pip install -r requirements.txt, then
python scripts/download_bec.py # Parquet from S3
python scripts/convert_parquet_to_text.py # Parquet → page .txt + label TSVs
python -m src.merge_pipeline # merge → Data/output/merged/