Pecha AI Studio: Buddhist AI Evaluation Platform - Product Requirements Document (PRD)
| Column 1 | Column 2 |
|---|---|
| Owning Group | Pecha AI Studio Team |
| Status | Draft |
| GitHub Project | Pecha AI Studio WG · GitHub |
| Last Updated | 2025-09-22 |
Purpose and Demographic
The purpose of the Buddhist AI Evaluation Platform is to establish a centralized, open, and transparent hub for benchmarking AI models developed for Buddhist studies. The platform will host standardized datasets for critical tasks—including Optical Character Recognition (OCR), Speech-to-Text (STT), and Machine Translation (MT)—allowing developers to evaluate their models against a common benchmark and track the state-of-the-art in the field.
✦ Mission Statement
Our mission is to accelerate innovation in Buddhist AI by providing a reliable and accessible evaluation platform. By offering standardized datasets and transparent leaderboards, we aim to foster a collaborative and competitive environment that drives improvements in model accuracy and performance. This will empower researchers, developers, and institutions to build more effective tools for the preservation, study, and dissemination of Buddhist heritage.
✦ Target Demographic
-
AI/ML Researchers and Developers: Individuals and teams building models for Tibetan OCR, STT for Dharma talks, translation of scriptures, and other related tasks who need a standardized way to measure performance.
-
Academic Institutions and Digital Humanities Scholars: Universities and research centers that require a clear understanding of the best available tools for their digital preservation and textual analysis projects.
-
Monastic and Lay Communities: Buddhist organizations and practitioners who wish to leverage AI tools and need a trusted resource to identify the most accurate and reliable models for their needs.
-
Language Technologists: Specialists focused on under-resourced languages who can use the platform to test and refine models for Tibetan, Sanskrit, and other Buddhist canonical languages.
✦ Problem Statement
-
Lack of Standardization: Researchers and developers working on Buddhist AI models lack a common ground for evaluation, making it difficult to compare results, track progress, and identify the true state-of-the-art.
-
Fragmented Efforts: Without a central hub, development efforts are often siloed, leading to duplicated work and slower innovation across the community.
-
Informed Decision-Making: End-users, such as academic institutions or monasteries, have no easy way to determine which AI models are the most accurate and best suited for their specific use cases.
-
Barriers to Entry: New researchers entering the field face a significant hurdle in finding quality benchmark datasets and understanding the current landscape of model performance.
Product Objectives
✦ Core Objectives
-
Establish a Benchmark Hub: Develop a robust platform to host, manage, and serve standardized evaluation datasets for key Buddhist AI tasks.
-
Automate Evaluation: Implement automated pipelines that take user-submitted model outputs, calculate standard performance metrics (e.g., CER, WER, BLEU), and update results in real-time.
-
Provide Transparent Leaderboards: Create clear, public-facing leaderboards for each challenge, allowing users to easily compare the performance of different models.
-
Ensure Ease of Use: Design an intuitive user experience for discovering challenges, downloading data, submitting results, and viewing performance scores.
✦ Non-Goals
-
Compute Provisioning: The platform will not provide computational resources for model training or inference. Users are expected to run their models on their own infrastructure and submit the final outputs.
-
Model Hosting: We will not host the AI models themselves. The platform is designed to evaluate the inference output of models, not to serve them via an API.
-
Dataset Creation: While we will host datasets, the platform’s primary function is not the creation of new benchmark data. We will rely on partnerships to source and validate high-quality datasets.
✦ Impact Areas
-
AI Research: Accelerate the pace of innovation in natural language processing and computer vision for Buddhist and other under-resourced languages.
-
Digital Humanities: Provide scholars with a clear guide to the most effective tools for digitizing and analyzing textual and audio-visual archives.
-
Cultural Preservation: Improve the quality and scale of digitization efforts, helping to preserve and provide access to invaluable Buddhist texts and teachings.
-
End-User Applications: Enable the development of higher-quality downstream applications (e.g., translation apps, study tools) by making it easier to identify top-performing models.
Example Use Cases
✦ A Researcher Improving an OCR Model
A PhD student has developed a new transformer-based architecture for Tibetan OCR. She navigates to the “Tibetan OCR Challenge” on the platform, downloads the standardized test set, and runs her model on the images. She then uploads the resulting text file. The platform automatically calculates the Character Error Rate (CER) and Word Error Rate (WER), and her model appears as the new #2 entry on the leaderboard, validating the effectiveness of her approach for her dissertation.
✦ An Institution Selecting an STT Service
A Buddhist dharma center has thousands of hours of recorded teachings they want to transcribe. Before investing in a transcription pipeline, their technical lead visits the evaluation platform. They review the “Dharma Talk STT Challenge” leaderboard and see that a particular open-source model significantly outperforms others on similar audio. They use this data-driven insight to select the right tool for their project, saving time and resources.
Architectural Considerations
✦ Tech Stack
-
Programming Language: Python (for backend and evaluation scripts) and Javascript (for frontend)
-
Web Framework: FastAPI + Alembic
-
Frontend: React
-
Database: PostgreSQL for storing user data, submissions, and leaderboard results.
-
Memory: Redis caching
-
Authentication: Auth0 service
-
CI/CD: Render.com auto deploy hook with git integration.
✦ Security & Privacy
-
User Data Protection: All user account information, including passwords, will be securely hashed and stored.
-
Submission Integrity: Submissions will be isolated, and evaluation scripts will run in a sandboxed environment to prevent abuse.
-
Dataset Access: While some datasets will be public, access controls will be in place to manage access to test set ground truth data to ensure fair evaluation.
✦ Dependencies
-
AI/ML Research Partners: Sourcing and validating high-quality benchmark datasets will depend on partnerships with academic and research institutions.
-
UI/UX Design: Requires design resources to ensure an intuitive and user-friendly interface.
✦ Scalability & Maintenance
-
Modular Evaluation Scripts: Each challenge’s evaluation logic will be modular, allowing new challenges and metrics to be added easily.
-
Asynchronous Processing: Submissions will be processed by background workers to ensure the UI remains responsive and can handle multiple concurrent evaluations.
-
Community Contributions: The platform can be designed to allow trusted partners to propose and help manage new challenges and datasets.
✦ Point of Contact
Tenzin Dhakar | dhakar@dharmaduta.in
Project Status
✦ Current Phase
We are in the initial planning and requirements-gathering phase. This PRD serves as the foundational document to guide the design and development of the platform’s MVP.
✦ Roadmap
-
Q4 2025: Finalize architecture and tech stack. Develop core backend services (user auth, database) and onboard the first internal benchmark dataset for Tibetan OCR.
-
Q1 2026: Implement the core user workflow (challenge discovery, data download, submission upload) and the automated OCR evaluation pipeline. Launch an internal alpha.
-
Q2 2026: Launch a public beta with the OCR challenge. Begin work on onboarding the second challenge (e.g., STT).
-
Q3 2026: Incorporate user feedback from the beta. Add support for the STT challenge and leaderboard.