Pterodactyl: Pushing the Limits of Efficient Automated Translation Evaluation
Evelyn Y. Garland, Carola F. Berger
Abstract
In this paper, we present 1020 Group's submission to the WMT26 shared task on automated quality evaluation. Our system, Pterodactyl, was engineered specifically for subtasks 2 and 3, that is, segment-level quality score prediction and the detection of error-free segments. Pterodactyl is a highly customizable, ultra-low cost system optimized for binary classification in translation triage and data curation. It is based on an LLM-as-a-judge approach, with prompt engineering and model selection guided by ROC analysis. We also developed several variants of Pterodactyl that were specifically adapted for subtasks 3.
Create a lesson
Related papers
Represented but misread: Post-training changes how language models read out what they represent
Jochen Laubrock
AutoMaster: Autonomous Post-Training by Making the Same Model a Better Teacher
Guoxin Chen, Shuang Sun, Fanzhe Meng et al.
Multimodal Error Detection in Chest X-ray Reports: A Large Language Model Reading Image Evidence
Yanda Cheng, Bo Wen, Yuhao Lu
MemFolder: Structure-Guided Memory Folding for Efficient Agent Memory Management
Xuyang Liu, Chenwen Lu, Junxi Wang et al.
Beam Model Technical Report
Reflection Team, Adam Abdulhamid, Osama Abuelsorour et al.
One Framework to Diagnose Them All: A Stratified Benchmark for Bengali LLM Hallucination
Asif Shahriyar Sushmit, Shefayat E Shams Adib, Abdullah Al Jubaer Gem et al.