Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression
Manikanta Kotthapalli, Banafsheh Rekabdar
Abstract
Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~K imposes a hard information ceiling of 2 K bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning. Building on the MS-VQ-VAE architecture introduced in~kotthapalli2026msvqvae, we sweep K ∈ \128, 256, 512, 1024\ under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at K ≤ 512, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips (64\!×\!64, 32~frames), our models operate at 0.043-0.064~bpp -- 3.3-5× below H.264's practical floor and 5-7.6× below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF\,36 on perceptual quality (LPIPS) despite using 5-7.6× fewer bits. At K=1024, the model surpasses H.265 CRF\,36 on LPIPS by a margin of 0.072 absolute while using 5.1× fewer bits. Codebook analysis confirms power-law index distributions and 70-85\% entropy efficiency, establishing the pipeline as a principled learned entropy coder.