Continuous Speculative Decoding for Autoregressive Image Generation

Abstract

Continuous visual autoregressive (AR) models have demonstrated promising performance in image generation, but their inherently sequential nature results in slow inference speed. Speculative decoding, a successful acceleration technique for large language models (LLMs), has effectively accelerated discrete visual AR models. However, the absence of an analogous theory for continuous distributions precludes its use in accelerating continuous AR models. To fill this gap, this work presents continuous speculative decoding, and addresses challenges from: 1) low acceptance rate, caused by inconsistent output distribution modeled by target and draft models, and 2) modified distribution without analytic expression, caused by a complex integral. For challenge 1), we address low acceptance rates through an approximated criterion, a novel denoising trajectory alignment strategy based on reparameterization proximity, and token pre-filling. For challenge 2), we introduce acceptance-rejection sampling algorithm with an appropriate upper bound, thereby avoiding explicitly calculating the integral. Furthermore, our denoising trajectory alignment is also reused in acceptance-rejection sampling, effectively avoiding repetitive diffusion model inference. Extensive experiments on various models at 256x256 and 512x512 resolutions demonstrate that our approach achieves over 2x wall-time speedup while preserving the image generation quality. Codes is available at: https://github.com/MarkXCloud/CSpD