Using Experience Twice: Dynamic Process Rubrics for Privileged Token-Level Credit Assignment
Shumao Wu, Kongcheng Zhang, Jianqiang Huang, Baisheng Lai
Abstract
Agent trajectories often contain both successful decisions and failed attempts, yet optimization with sparse trajectory-level rewards struggles to identify which critical decisions should be reinforced or suppressed. We observe that meaningful distinctions among on-policy trajectories can be distilled into natural-language process rubrics and used as process supervision signals. Based on this insight, we propose Online Rubric Self-Distillation (ORSD), which leverages self-derived process rubrics for dense token-level supervision, thereby augmenting outcome-centric reinforcement learning. Specifically, the current policy independently generates multiple rubric bundles at each step according to the same group of trajectories, extracting candidate process rubrics from cross-trajectory behavioral differences. A frozen verifier then evaluates all rubric items against each trajectory in the group, and selects one candidate bundle based on bundle-level alignment between rubric satisfaction and task success. ORSD treats the selected rubric bundle as privileged information during training, then compares the log-likelihoods of the original tokens with and without the rubric. The resulting likelihood gap provides token-specific process credit, which is added to the original trajectory-level GRPO advantage through additive advantage shaping. ORSD thus bridges trajectory-level outcome supervision and fine-grained process supervision, enabling token-level credit assignment using process signals. Extensive experiments on ALFWorld and WebShop show that ORSD accelerates policy improvement and boosts final task performance for both Qwen3-1.7B and Qwen3-4B under the same environment-interaction budget. Our code is available at https://github.com/AnnaXChun/ORSD.
Create a lesson
Related papers
AgentSSL: Can MLE Agents Leverage Unlabeled Data?
Ethan Lin, Akanksha Sarkar, Ziang Liu et al.
CoReMem: Cost-Internalizing Memory for Successive Coding Tasks with Prospective Valuation
Peiwen Li, Harshit Verma, Sophia Zhou et al.
Classical Energies Learn to Reason 1,000x Faster than Looped Models
Benjamin Hoover, Hugh Blayney, James C. R. Whittington
LiveGeo: Benchmarking Geographic Agents in Continuously Evolving Worlds
Pengcheng Zhou, Xuanyu Liu, Yanchen Yin et al.
Ballad: Bandit-Based LLM Routing for Automated Heuristic Discovery
Samidha Verma, Ankit Anand, Sayan Ranu
Al-Ghasali: A Computational Framework for Simulating Islamic Jurisprudential Rulings via Finite State Automata and Large Language Models
Elnaserledinellah Mahmoud Abdelwahab