21Codec: Decoder-side Speaker Control for Text-to-Speech
David Lindevelt, Suzan Verberne, Joost Broekens
Abstract
We propose 21Codec, a streaming codec model with reduced recoverable speaker information from the token stream and decoder-side speaker control. Existing codec models infer speaker identity from token stream. Making them streamable is not sufficiently robust to speaker drift, because each chunk re-infers the identity from the local chunk context alone. Our adversarial speaker classifier and NANSY-style formant-shift data augmentation pushes the model to rely on the provided speaker embedding while reducing the recoverable speaker information from the token stream. We show that 21Codec can control speaker identity through the conditioning vector, and that the speaker identity is maintained across chunks during streaming. With the ability to control speaker identity even in mid-stream, 21Codec is a promising candidate for streaming TTS systems for better controllability of speaker identity.
Create a lesson
Related papers
Blind tests of a simple-body model of submarine target strength on open benchmarks
Sunil Tyagi
UMelCodec: An Ultra-Low-Bitrate and Lightweight Mel-Domain Neural Speech Codec
Hui-Peng Du, Yang Ai, Zhen-Hua Ling
Joint Language Modeling for In-the-Wild Speech-Scene Generation with Distinct Token Representations
Semin Kim, Ju Yeon Kang, Sunghwan Ahn et al.
Sex Differences in the Detection of Parkinson's Disease from Speech
Hiba Akhaddar, Peter Plantinga, Mirco Ravanelli et al.
Location Matters: Layer-wise Prompt Conditioning for Voice--Style Disentanglement in Text-to-Speech
David Lindevelt, Suzan Verberne, Joost Broekens
Beyond Intelligibility: Fine-Grained Perceptual Assessment of Dysarthric Speech with Auditory Large Language Models
Haotian Wei, Can Long, Wen Xian Jim et al.