Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian
Abstract
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Create a lesson
Related papers
Your Transformer Is Secretly Elastic
Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts et al.
Continuum Attention
Afshin Khadangi
Learnable Hodge-Laplacian Driven Mixture of Experts for Steady-PDE Surrogates on Coarse Unstructured Meshes
Stavros Nousias, Akis Nousias, Konstantinos Gkrispanis
One Task, Many Interfaces: Simple Cross-Interface Consistency Training Improves LLM Agent Robustness
Artin Tajdini, Manpreet Kaur, Peng Qi
Detection of Potato Plant Leaf Diseases using Convolutional Neural Networks
Haimanot Bitew Alemayehu, Simachew Melaku Ayalew, Chilotie Desalew Mengesha et al.
Revisiting Simple Baselines for Relational Deep Learning
Viet The Nguyen, Vijay Prakash Dwivedi, Gunther Gust