Correcting What You Cannot See: Credit Assignment for Perception Distillation in Multimodal Reasoners
Feng Xiong, Leyan Xue, Hongyu Lin
Abstract
On-policy distillation provides dense supervision for multimodal reasoners, but its trajectory-level reward cannot determine whether a failed answer arose from perception or subsequent reasoning. Perception Success Rate (PSR), estimated from multiple reasonings sharing one perception, remains ambiguous because low success conflates perceptual insufficiency with reasoning difficulty. We introduce Perception-Correction Distillation (PCD), a label-free method that identifies correctable perception failures using downstream failure and teacher--student disagreement as complementary witnesses. Their product, , forms a soft AND gate that strengthens distillation only when both witnesses are present. We motivate this rule through Bayesian evidence combination and show that multiplication is the unique normalized bilinear gate that vanishes when either witness is absent. PCD uses separated perception--reasoning rollouts and mean-preserving weights, leaving the reasoning objective unchanged. Across eight benchmarks, PCD improves the 8B 2B macro average from 44.50 with OPD to 47.28 and the 32B 8B result from 56.94 to 61.22. In matched 2B ablations, removing PCD and separated rollout reduces held-out average by 2.22 and 0.88 points, respectively. Effective multimodal distillation therefore depends not only on what the teacher predicts, but also on identifying when perception is the appropriate target of correction.
Create a lesson
Related papers
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
Luyao Zhu, Xun Wei Yee, Wei Li et al.
Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Jinli Hu, Ross M. Clarke, Yichuan Zhang et al.
Compositional Policy Violations: When Step-Level Compliance Fails In Agentic AI Workflows
Ashwini Kurady, Sri Sai Charith Grandhi, Rajesh Gupta et al.
CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents
Jiaxuan Jiang, Liyuan He, Zhixuan Fang
Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale
Liuyin Wang, Shuaipeng Jin, Jiwei Shi et al.
Clueing up LLMs with Tool-Augmented Deductive Reasoning
Rebecca Ansell, Autumn Toney-Wails