Strategic Meeting Minutes: 13 Jul 2025 - Automatic Translation Evaluation project for 84000

🧩 SIG 📊 MT Evaluation SIG
1

Attendees: @Trinley, @Tenzin_Gayche, @billingsmoore

Agenda: Collaboration on a MT evaluation project for 84000

Key Discussion Points

  • OpenPecha will support the translation evaluation effort by providing the Pecha AI Studio platform for evaluators.

  • The project will proceed in phases:

    1. Train and guide evaluators to ensure standardized annotation.
    2. Compare human evaluations with existing automatic metrics to assess correlation.
    3. If current metrics underperform, explore training a new model to build a state-of-the-art automatic evaluator.

  • Proposed roles:

    • NT will join the Translation & Evaluation team.
    • Gayche will be part of the Tech team.
    • Both will collaborate across technical and evaluation tasks.

Action Items

Decisions Made

  1. Decision: OpenPecha will provide infrastructure and platform support for the evaluation effort.
  2. Decision: Project will move forward in a phased approach, starting with evaluator training and standardization.
  3. Decision: Teams and roles