Textraction: Unified Text-Guided Speaker Extraction and Enhancement for Single- and Multi-Channel Audio
Aviad Eisenberg, Sharon Gannot, Shlomo E. Chazan
Abstract
We present Textraction, a text-guided speaker extraction framework that enables the selective extraction of desired speakers or noise sources from an audio mixture using only natural-language descriptions. The proposed method operates within a single unified model that supports both single-channel and multi-channel inputs and it allows conditioning on either semantic textual descriptions of the target source or on its spatial description. Dedicated single- and multi-channel encoders feed a shared stack of time--frequency transformer blocks, which is conditioned on the prompt and on the recording configuration in every block through cross-attention and estimates the target directly from the mixture in one forward pass, without a separation stage or a source-selection stage. The system can extract an arbitrary subset of entities from a mixture, assuming up to two simultaneously active speakers and two noise sources, and interprets prompts specifying which entities to keep, which to remove, or both. In addition, we show that the framework naturally applies to a speech-enhancement setting, in which a single target speaker is present alongside background noise. To circumvent the application of computationally costly large language models used in prior work, we introduce a lightweight, task-adapted text encoder obtained by contrastively fine-tuning a compact text-retrieval model, which learns to be sensitive to instruction polarity (e.g., keep vs. remove) while remaining robust to paraphrasing. Experiments across attribute-based extraction, spatially guided extraction, and speech enhancement show that the proposed approach effectively isolates the desired entities under various acoustic and spatial conditions, and matches or surpasses dedicated spatial-extraction and enhancement baselines while offering substantially greater flexibility.
Create a lesson
Related papers
ZoionNET: Low-Power Animal Intrusion Detection Using an Acoustic Deep-Learning Classifier for a Cellular Remote-Sensing System
Prasanna Vijay, Adarsh Gupta, Vansh Panwar et al.
When Sounds Turn Heads: Recovering Event Timing That Audio LLMs Fail to Say
Jongbhin Woo, Jaewook Jeong, Joon Son Chung
From Attention to Semantics: Detecting and Mitigating Sound Event Hallucinations in Large Audio-Language Models
Tianyi Sun, Minghao Lv, Chunyang Yu et al.
The landscape of AI speech generation and deepfake detection
Simon King
Exploring Speaker-Aware Engagement Modeling for Children with Autism Spectrum Disorder
Kaleen Shrestha, Austin Joel Dympep, Young Suh Hong et al.
Harmonic Circuit: Chord Progression Lies in a Sparse Principal Information Flow in Symbolic Music Models
Ruiji Yu, Gus Xia