Title: Fine-Tuning a Multi-Dialect Speech Recognition Model for Tibetan Languages with Balanced Dialect Data
Introduction
To address the challenge of balancing cross-dialect performance in Tibetan speech recognition, we refined our previous approach by incorporating balanced dialect data into the fine-tuning process. Our aim was to develop a speech recognition system capable of handling Utsang, Khampa, and Amdo dialects while mitigating catastrophic forgetting in the base model trained on Utsang data. By adding 50 hours of Utsang data during fine-tuning, we significantly improved model retention of Utsang dialect knowledge while enhancing recognition performance for Khampa and Amdo dialects.
Background
Tibetan speech encompasses three major dialects—Utsang, Khampa, and Amdo—with considerable phonetic variations. While the base wav2vec2 model performed well on Utsang (CER: 15.23%), extending its capabilities to other dialects resulted in knowledge degradation. The inclusion of balanced training data across dialects provided an effective strategy to address catastrophic forgetting and improve cross-dialect performance.
Methodology
Dataset Preparation
To ensure balanced training, the fine-tuning dataset included:
- Utsang: 50 hours
- Khampa: 80 hours
- Amdo: 53 hours
Model Architecture
- Base Model: wav2vec2 pretrained on 1500 hours of Utsang data.
- Fine-tuning: Balanced data across dialects, including additional Utsang data for catastrophic forgetting mitigation.
Training Strategy
- Combined training with balanced sampling across all dialects.
- Retention-focused training with gradient accumulation and warmup steps.
Implementation Details
Training Parameters
training_args = TrainingArguments(
per_device_train_batch_size=8,
gradient_accumulation_steps=2,
evaluation_strategy="steps",
save_steps=1000,
eval_steps=1000,
logging_steps=100,
learning_rate=3e-5,
num_train_epochs=20,
warmup_steps=500,
fp16=True,
dataloader_num_workers=4
)
Hardware Configuration
- 4x NVIDIA L4 GPUs (24GB each)
- Training Duration: ~15 hours
- Using pytorch DDP (python -m torch.distributed.launch --nproc_per_node=4 finetune.py)
Results
Utsang Benchmark
| Model Type | Mean CER | Analysis |
|---|---|---|
| Base Model | 15.23% | Strong initial performance |
| Fine-tuned Model (unbalanced) | 29.62% | Catastrophic Forgetting on Prior Knowledge |
| Fine-tuned Model (balanced) | 25.47% | Improved retention compared to prior fine-tuning runs. |
Cross-Dialect Performance
| Dialect | Base Model | Fine-Tuned Model (unbalanced) | Fine-Tuned Model (balanced) |
|---|---|---|---|
| STT_AM | 44.29% | 17.23% | 17.26% |
| STT_AM_AB | 35.48% | 7.58% | 7.44% |
| STT_KH | 36.73% | 20.86% | 20.77% |
| STT_KH_AB | 31.10% | 10.46% | 10.42% |
| Mean | 36.90% | 14.03% | 13.97% |
Discussion
Positive Outcomes
- Improved Performance Across Dialects: The balanced fine-tuning approach significantly improved recognition accuracy for Khampa and Amdo dialects, with a mean CER of 13.97%, outperforming both the base model (36.90%) and the unbalanced fine-tuned model (14.03%).
- Mitigated Catastrophic Forgetting: While Utsang CER increased to 25.47% compared to the base model’s 15.23%, it showed a noticeable improvement over the unbalanced fine-tuned model (29.62%).
- Stability Across Dialects: The inclusion of balanced data resulted in consistent improvements, highlighting the importance of equal representation in multi-dialect fine-tuning.
Challenges
- Utsang Performance: Despite improvement, the balanced model’s CER for Utsang (25.47%) remains higher than the base model’s original CER (15.23%). This suggests further work is needed to retain Utsang knowledge without compromising gains in other dialects.
Conclusion
Balancing training data across Tibetan dialects proved effective in mitigating catastrophic forgetting and enhancing multi-dialect speech recognition. Key takeaways include:
-
Improved Multi-Dialect Performance:
- Balanced fine-tuning achieved a mean CER of 13.97% for Khampa and Amdo, a significant reduction compared to the base model (36.90%) and unbalanced fine-tuning (14.03%).
-
Utsang Retention:
- While Utsang CER increased to 25.47%, this represents a considerable improvement over the unbalanced approach (29.62%) and highlights the effectiveness of balanced fine-tuning in mitigating catastrophic forgetting.
This study underscores the critical role of balanced data in building robust, multi-dialect speech recognition systems. Future directions include:
- Expanding the dataset for all dialects to enhance generalization and ensure equitable representation.
- Investigating data augmentation or knowledge distillation techniques to further improve Utsang performance without sacrificing gains in other dialects.
Balanced fine-tuning lays a strong foundation for multi-dialect Tibetan speech recognition, setting the stage for more advanced models and broader language support.
Resources
- Base Model: Model Link
- Fine-tuned Model (Unbalanced): Model Link
- Fine-tuned Model (Balanced): Model Link
- Utsang Benchmark: Benchmark Link
- Amdo and Kham Benchmark: Benchmark Link
Citations
[1] Improving accuracy of speech recognition for low-resource accents
[2] Tibetan Multi-Dialect Speech and Dialect Identity Recognition