Understanding Personalization Gains in Whisper
Sergei Katkov, Antonio Liotta
Abstract
Whisper-style ASR models provide strong zero-shot transcription, but their residual errors are not evenly distributed across datasets, recording conditions, and metadata-based groups. We study when lightweight low-rank adaptation (LoRA) improves Whisper and whether observed gains require group-specific adapters. Across Common Voice, LibriSpeech, and VoxPopuli, we compare zero-shot Whisper, pooled LoRA, and group-specific LoRA under matched within-group splits. We find that adaptation is not uniformly beneficial: gains are concentrated in difficult Common Voice groups, are consistent on VoxPopuli, and largely disappear on low-error LibriSpeech. Crucially, pooled LoRA often matches, and sometimes clearly outperforms, group-specific LoRA. Gains over zero-shot should therefore be tested against matched pooled adaptation before being interpreted as evidence of personalization.
Create a lesson
Related papers
Neos: Learning to Interleave Search and Visual Reasoning
Ekko He, Jack Hong, Shilin Yan et al.
Benchmarking Quantum Kernel SVMs for Credit Card Fraud Detection: Performance Realities Under NISQ Constraints
Hitanshi Arora, Mohit Swarnkar, Aswani Kumar Cherukuri
Glasseek: A Transparent Recipe for Training and Trustworthy Evaluation of DeepSearch Agents
Cong Pang, Jiaqi Su, Chengrui Wu et al.
PRAISE: A Reproducible Workflow for Non-Programmer Scientists to Implement Published AI Models
Max Achterweust, Stefan Groot Nibbelink, Vincent J. de Beer et al.
MADDIE: Multi-Agent Drug Discovery Engine
Ye Liu, Reza Averly, Avery Meyer et al.
WeFair: Fidelity-Aware Alignment from Human Feedback for Personalized Query Rewriting in Short-Video Search
Yingtao Ren, Aolin Li, Cheng Cheng et al.