Nepali-English Code-Mixed Speech Recognition: Data Preparation and Adaptation of a Speech Foundation Model
Jeevan Neupane, Janardan Bhetwal, Kshitiz Paudel, Anup Aryal, Bipin Bashyal
Abstract
Nepali speakers regularly mix English words into conversation, but multilingual speech recognisers often struggle with language switches and the writing script used for English words. We study how to adapt SraVaani, a 449M-parameter hybrid TDT-CTC model, using a 172.8-hour corpus covering six Nepali speech domains. Its pretrained tokenizer does not support digits and several common symbols. Rather than replace it or discard these recordings, we generate Nepali and English readings for numbers and use zero-shot ASR output to select a plausible form. Full fine-tuning reduces test word error rate (WER) from 48.6% to 17.1%; a 12-epoch run reaches 16.3%, compared with 16.4% for fine-tuned Whisper large-v3-turbo. Freezing the encoder increases WER to 25.5%, while excluding numeric clips raises WER on numeric test speech from 14.3% to 21.3%. We also evaluate on FLEURS using both its original references and mixed-script alternatives. The comparison shows that transcription conventions explain part, but not all, of the out-of-domain performance loss.
Create a lesson
Related papers
PlayCraft: Designing Engaging Games from Agent Play Experience
Weihao Cheng
Probing the "Conscious" Mind: Selective Access and Global Workspace in Vision-Language Models
Jiahao Huang, Jun Wang
ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images
Adrià Ciurana Lanau, Harleen Hanspal, Lusine Abrahamyan et al.
MM-SystemOneBias: Evaluating Bias in Multimodal Jev-Style System One Decision Models
Jinzhe Li, Qi Wang, Dandan Guo et al.
The Geometry of Lies: Knowledge and Persona Directions for Detecting Deception in Large Language Models
Md. Abdullah Al Zami, Suraiya Shanaz, Mahdi Hasan et al.
LLM GerryMandering: Separating Interactions from Additive Effects in LLM Audits
George Rahul, Ronit Jahilal, Bhaskara Hanuma Vedula et al.