Direct Conditioning of Audio Diffusion Transformers on fMRI Reveals Cortical Contributions to Sound Reconstruction
Matteo Ciferri, Tonio Weidler, Matteo Ferrante, Nicola Toschi, Elia Formisano
Abstract
Reconstructing audio from brain activity remains challenging. Existing brain-to-audio pipelines typically map neural responses to predefined stimulus representations before synthesis. While recent work in the visual domain has explored direct conditioning on brain activity, this approach has not yet been extended to audio diffusion models. We introduce MAAS (Multi-Area Auditory Synthesis), an end-to-end framework that conditions a pretrained audio diffusion transformer on ROI-wise fMRI responses without first supervising a decoder to predict an external stimulus representation. MAAS uses region-specific projections and learned brain tokens to condition Stable Audio through a dedicated cross-attention pathway. On a 7T fMRI dataset, we demonstrate consistent improvements over baselines, including two-stage fMRI-to-latent conditioning, across both acoustic and semantic evaluation metrics. In addition, we introduce a fidelity metric grounded in a computational model of auditory neural processing, which allows us to quantify the contribution of individual auditory cortical regions to reconstructed audio representations.
Create a lesson
Related papers
ZoionNET: Low-Power Animal Intrusion Detection Using an Acoustic Deep-Learning Classifier for a Cellular Remote-Sensing System
Prasanna Vijay, Adarsh Gupta, Vansh Panwar et al.
When Sounds Turn Heads: Recovering Event Timing That Audio LLMs Fail to Say
Jongbhin Woo, Jaewook Jeong, Joon Son Chung
From Attention to Semantics: Detecting and Mitigating Sound Event Hallucinations in Large Audio-Language Models
Tianyi Sun, Minghao Lv, Chunyang Yu et al.
The landscape of AI speech generation and deepfake detection
Simon King
Exploring Speaker-Aware Engagement Modeling for Children with Autism Spectrum Disorder
Kaleen Shrestha, Austin Joel Dympep, Young Suh Hong et al.
Harmonic Circuit: Chord Progression Lies in a Sparse Principal Information Flow in Symbolic Music Models
Ruiji Yu, Gus Xia