Location Matters: Layer-wise Prompt Conditioning for Voice--Style Disentanglement in Text-to-Speech
David Lindevelt, Suzan Verberne, Joost Broekens
Abstract
Independent control of voice and emotional style characteristics of speech is a long-standing problem in text-to-speech (TTS) systems. We investigate how the conditioning location of voice and style prompts across TTS decoder layers affects the trade-off between preservation and expressiveness for voice identity and emotional style. We introduce a layer-dependent gated fusion mechanism and systematically study which layers should be influenced by which prompt type. Our findings show an asymmetric relationship. Voice benefits from early conditioning for identity preservation, while style benefits from early conditioning for expressiveness. Our findings provide practical guidance that helps neural network architects make an appropriate trade-off for the architecture of TTS systems according to their use cases.
Create a lesson
Related papers
Blind tests of a simple-body model of submarine target strength on open benchmarks
Sunil Tyagi
UMelCodec: An Ultra-Low-Bitrate and Lightweight Mel-Domain Neural Speech Codec
Hui-Peng Du, Yang Ai, Zhen-Hua Ling
Joint Language Modeling for In-the-Wild Speech-Scene Generation with Distinct Token Representations
Semin Kim, Ju Yeon Kang, Sunghwan Ahn et al.
Sex Differences in the Detection of Parkinson's Disease from Speech
Hiba Akhaddar, Peter Plantinga, Mirco Ravanelli et al.
21Codec: Decoder-side Speaker Control for Text-to-Speech
David Lindevelt, Suzan Verberne, Joost Broekens
Beyond Intelligibility: Fine-Grained Perceptual Assessment of Dysarthric Speech with Auditory Large Language Models
Haotian Wei, Can Long, Wen Xian Jim et al.