SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces
Ruidong Zhang, Jiacheng Liu, François Guimbretière, Cheng Zhang
Abstract
Wearable silent speech interfaces (SSIs) are limited to small, closed vocabularies. Approaches achieving larger vocabularies require obtrusive hardware such as facial electrodes. We present SoniSpeech, the first large-scale, open-vocabulary, trimodal dataset for wearable SSI using acoustic-sensing eyewear. It contains 34 hours across 18,000 utterances with three synchronized modalities: ultrasound echo profiles, voiced audio, and frontal video, in both voiced and silent modes. The corpus draws from the SODA dialogue dataset, providing contemporary conversational English with 5,356 unique words and full phoneme coverage. A CTC-based ResNet-34 baseline achieves 26.3% word error rate (WER) on open-vocabulary silent speech recognition, the first benchmark for this task. Dataset is available at https://doi.org/10.7298/xjjr-9m85
Create a lesson
Related papers
FRAUDSkill: Structured Frozen-Weight Skill Optimization for Audio Anti-Fraud Detection
Chengxian Hu, Zhiming Ma, Mingjun Pan et al.
TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection
Huiyuan Liu, Zhiming Ma, Yanxing Liu et al.
Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding
Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf et al.
Beyond EER: Multi-Dimensional Evaluation of Information Leakage in Speaker De-Identification
Seungmin Seo, Oleg Aulov, P. Jonathon Phillips et al.
TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking
Giorgia Adorni, Michela Papandrea, Battista Rimoldi et al.
VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval
Aaron Yee, Fengjie Lu, Jiarui Hai et al.