Rethinking Post-Training for Expressive Zero-Shot Text-to-Speech
Jingbin Hu, Xinfa Zhu, Hao Wang, Wenjie Tian, Kangxiang Xia, Hangrui Hu, Kexin Huang, Jin Xu, Liumeng Xue, Lei Xie
Abstract
Zero-shot voice cloning typically emphasizes preserving the timbre and prosodic characteristics of a reference speaker. However, in real-world use, speech should adapt its delivery to the target text rather than remain tied to a fixed speaking style. In this work, we formulate this problem as text-adaptive prosody synthesis for zero-shot voice cloning and study how post-training can endow a cloning model with this capability. We develop a three-stage framework. First, Direct Preference Optimization (DPO) provides a cold start by aligning the model with text-appropriate expressive speech. Second, Group Sequence Policy Optimization (GSPO) jointly optimizes prosodic expressiveness, acoustic quality, intelligibility, and speaker similarity. Finally, context-conditioned On-Policy Self-Distillation (OPSD) enables self-improvement by transferring expressive behavior from strong-reference to weak-reference conditions. On Qwen3-TTS, our framework improves Emotion from 18.5 to 81.2 and Paralinguistics from 38.4 to 73.9. Extensive ablations validate the role of each stage.
Create a lesson
Related papers
Blind tests of a simple-body model of submarine target strength on open benchmarks
Sunil Tyagi
UMelCodec: An Ultra-Low-Bitrate and Lightweight Mel-Domain Neural Speech Codec
Hui-Peng Du, Yang Ai, Zhen-Hua Ling
Joint Language Modeling for In-the-Wild Speech-Scene Generation with Distinct Token Representations
Semin Kim, Ju Yeon Kang, Sunghwan Ahn et al.
Sex Differences in the Detection of Parkinson's Disease from Speech
Hiba Akhaddar, Peter Plantinga, Mirco Ravanelli et al.
Location Matters: Layer-wise Prompt Conditioning for Voice--Style Disentanglement in Text-to-Speech
David Lindevelt, Suzan Verberne, Joost Broekens
21Codec: Decoder-side Speaker Control for Text-to-Speech
David Lindevelt, Suzan Verberne, Joost Broekens