When Sounds Turn Heads: Recovering Event Timing That Audio LLMs Fail to Say
Jongbhin Woo, Jaewook Jeong, Joon Son Chung
Abstract
Audio LLMs can know when a queried sound occurs but fail to say it. We find a few attention heads that track the queried sound: they shift when the query changes, follow the sound when it moves, and removing or blocking them often hurts the model's timestamps. Yet the answer does not express what they track. Intervals read directly from these heads outperform the model's original answer and recover events that its timestamps miss. To bring this timing into the answer, we introduce head-guided decoding, which needs no training. It keeps the model's decision on whether the sound is present and rescores its candidate timestamps with a prior from these heads. Heads selected on development data transfer unchanged to new recordings and free-text queries, where head-guided decoding improves the timestamps of three audio LLMs, scores above their head readout on every benchmark, and, in most settings, outperforms random-head guidance. On recordings released after the models were trained, head-guided decoding raises mIoU from 0.07-0.36 to 0.48-0.56. These results suggest that poor temporal answers in audio LLMs partly reflect a failure to say rather than to hear, and that the model's own attention can close part of this gap.
Create a lesson
Related papers
ZoionNET: Low-Power Animal Intrusion Detection Using an Acoustic Deep-Learning Classifier for a Cellular Remote-Sensing System
Prasanna Vijay, Adarsh Gupta, Vansh Panwar et al.
From Attention to Semantics: Detecting and Mitigating Sound Event Hallucinations in Large Audio-Language Models
Tianyi Sun, Minghao Lv, Chunyang Yu et al.
The landscape of AI speech generation and deepfake detection
Simon King
Exploring Speaker-Aware Engagement Modeling for Children with Autism Spectrum Disorder
Kaleen Shrestha, Austin Joel Dympep, Young Suh Hong et al.
Harmonic Circuit: Chord Progression Lies in a Sparse Principal Information Flow in Symbolic Music Models
Ruiji Yu, Gus Xia
RespCCT and FreqCCT: Compact, Explainable and Cost-Aware Convolutional Transformers for Multi-Class Respiratory Disease Classification
Auntor Chakma, Mst. Farhana Akter, Abu Henaf Rashid Ahmad Mufti et al.