Deep Learning-based OTFS Channel Estimation and Symbol Detection with Plug-and-Play Framework

Abstract

Orthogonal Time Frequency Space (OTFS) modulation has recently attracted significant interest due to its potential for enabling reliable communication in high-mobility environments. However, the effectiveness of OTFS receivers relies on the inherent characteristic of the Delay-Doppler (DD) domain channel, where the sparsity of the discretized channel varies across different communication scenarios. For instance, the fractional Doppler effect reduces the inherent channel sparsity, which consequently degrades channel estimation accuracy and increases the complexity of symbol detection. Traditional algorithms relying on fixed sparsity priors often require manual design, while purely data-driven deep learning (DL) methods typically struggle to generalize across diverse channel conditions. To address these challenges, we propose a novel unsupervised DL-based plug-and-play (PnP) framework that provides a flexible solution for OTFS receiver design. The proposed framework can be applied to both channel estimation and symbol detection, jointly leveraging the flexibility of optimization-based methods and the powerful generalization capability of data-driven models. Specifically, a lightweight encoder-decoder network (EDN) is incorporated as an implicit channel prior for channel estimation, enabling robust performance across varying levels of channel sparsity. Furthermore, for symbol detection, we realize the PnP framework with a time-domain matrix inversion for model-based equalization, followed by a small multi-layer perceptron (MLP) pre-trained for specific constellations, thereby achieving low complexity and enabling flexible adaptation to various modulation formats. Finally, numerical results demonstrate the effectiveness and robustness of the algorithm.