Towards a Reliable and Practical Eval Pipeline
Emma Thuong Nguyen, Abhishek Ghose
Abstract
LLM-based software systems increasingly require effective "evals" as quality gates in the development lifecycle. However, existing work typically addresses individual aspects of eval reliability rather than the full set of practical requirements. We present an end-to-end eval pipeline that combines eval checklist creation, with learned aggregation for checklist responses, to improve agreement across LLM judges and accuracy against human judgments. The framework additionally pro- vides self-consistency, explanations, and prediction uncertainty, and we empirically demonstrate its effectiveness.
Create a lesson
Related papers
Discriminative World Models for Web Agents
Kelvin Li, Dhruv Pendharkar, Anish Pahilajani et al.
AI Contextual Measurement for Recovering Individual and Group-Level Effects: Validation Against Survey Measures and an Occupational Application
Wenxin Jiang, Xuyang Wang, Yuxiao Wu
Large Language Models (LLMs) for Telecom Root Cause Analysis (RCA): A Structured Reasoning Framework for Evidence-Grounded Diagnosis
Hao Zhou, Mandar Kulkarni, Hao Chen et al.
SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment
Qinghua Mao, Wanying Qu, Dadi Guo et al.
Measurement-Driven Sub-Network Selection for On-Premise Retrieval-Augmented Factory Agents
Vasileios Rizeakos, Georgios Paisios, Alexandros Machairas et al.
Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems
Yihang Chen, Yuxiang Chen, Yuxuan Huang et al.