REAL Loss: Optimizing Real Expected Acceptance Length for Speculative Decoding
Yu Tian
Abstract
Speculative decoding accelerates LLM inference by having a draft model propose multiple tokens for the target model to verify in one pass. The speedup is governed by the expected acceptance length (EAL), yet standard training objectives such as cross-entropy, KL divergence, and total variation focus only on distribution matching or single-step overlap. Recent studies instead propose block-level training objectives, aiming to optimize EAL directly. However, these objectives remain surrogates rather than exact formulations of EAL. For example, some average later acceptance over target-distributed continuations rather than the verification branches selected by accepted tokens, introducing a covariance gap that can reverse the gradient direction. We analyze this gap and derive REAL, a real EAL training loss that uses path weights for unbiased estimation and Rao-Blackwellization for variance reduction. In a controlled loss-level comparison with a Qwen3.6-35B-A3B target and a Domino drafter, REAL outperforms WTV, VSD, DBLast, and the static LK baseline on probabilistic serving acceptance length across 9 task subsets and 5 evaluation seeds. These gains require only a change to the training loss, with no modification to the drafter architecture or inference procedure, demonstrating the practicality of real EAL training. A reference implementation of the REAL loss is available at https://github.com/King-ty/speculators/tree/real-loss.
Create a lesson
Related papers
Bio-PM-Guided Synthetic IMU Generation
Deepali Singh, Deepesh Parpani, Aviral Gupta
Active Feature Acquisition by Distilling an Oracle Teacher
Yang Meng, Zixin Ding, Robert L. Grossman et al.
The Window Lever: Longer Observation Windows Outweigh the Observed Effects of Data, Capacity, and Architecture in Customer Behavior Forecasting
Tara Sowrirajan, Yanqiao Wang
Hybrid Retrieval for Government Scheme Search in English and Marathi
Aishwarya Gunjal, Kanchan Khemnar, Pawan Poojary et al.
Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
Élodie Côté-Gauthier, Doina Precup, Isabeau Prémont-Schwarz
A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes
Francesco Caporali, Jason M. Altschuler, Boris Hanin