A Low-Latency Interactive System for Real-Time Video Understanding Based on VLMs
Punan Dai, Jun Xu, Bingcong Lu, Zhengxue Cheng, Hongwei Hu, Ronghua Wu, Li Song
Abstract
Vision-language models are extending video understanding from offline clip analysis to continuous interactive streaming, but most research still emphasizes model capability rather than deployable low-latency interaction. This paper presents a unified edge-cloud system for real-time video VLM applications. Lightweight phone, smart glasses, PC, and pseudo-replay clients publish video and speech to a server runtime that provides shared ASR/TTS, session orchestration, backend adaptation, response delivery, and archive-backed measurement. The system integrates six representative video VLM backends with streaming or interaction-oriented capabilities and evaluates them across backend runtime, media transport, client-observed latency, and interaction behavior. With suitable backend selection and the WebRTC path, the tested system reaches approximately 0.9 to 1.0 s to first VLM text and 1.3 to 1.5 s to first non-silent TTS audio, while exposing backend adaptation costs and differences in real-time interaction behavior.
Create a lesson
Related papers
MoQSplat: Adaptive Progressive Streaming of 3D Gaussian Splatting via MoQ
Emanuele Artioli, Mohammadreza Ghafari, Md Tariqul Islam et al.
Divide and Conquer: Mixture-of-Bottleneck Experts in Informative Ordinal Space for Video-based Multimodal Sentiment Analysis
Ronghao Lin, Qiaolin He, Zefeng Lu et al.
Multimodal Aspect-Level Sentiment Analysis Based on Gated Noise Filtering and Emotion-Relevance Interaction
Chen Huang, Liangwei Guo, Yamin Li et al.
SemABR: Measuring Video Semantic Fidelity with Multimodal LLMs for Adaptive Bitrate Streaming
Shiqi Xu, Soung Chang Liew, Yuyang Du
Mechanism-Level Evaluation for Vision-Language Models: Controlled Activation-Replacement Diagnosis of Gender Bias
Zhipeng Zhao, Wenxu Wang, Peishun Liu et al.
Multimodal Emergency Vehicle Classification via Audio-Visual Transformers and Knowledge Distillation
Vijay John, Amar Dabaja