Escaping High-order Saddles in Policy Optimization for Linear Quadratic Gaussian (LQG) Control
Yang Zheng, Yue Sun, Maryam Fazel, Na Li
Abstract
First order policy optimization has been widely used in reinforcement learning. It guarantees to find the optimal policy for the state-feedback linear quadratic regulator (LQR). However, the performance of policy optimization remains unclear for the linear quadratic Gaussian (LQG) control where the LQG cost has spurious suboptimal stationary points. In this paper, we introduce a novel perturbed policy gradient (PGD) method to escape a large class of bad stationary points (including high-order saddles). In particular, based on the specific structure of LQG, we introduce a novel reparameterization procedure which converts the iterate from a high-order saddle to a strict saddle, from which standard random perturbations in PGD can escape efficiently. We further characterize the high-order saddles that can be escaped by our algorithm.
Create a lesson
Related papers
Two Communities, One Problem: Branch-and-Bound vs. A* for Resource-Constrained Project Scheduling
Izack Cohen, Dominik Kress, Lara Nehrke et al.
Cost controllability and predictive control of input-constrained monotone systems on Hilbert spaces
Till Preuster, Karl Worthmann, Lars Grüne et al.
Pareto Front Tracing in Higher Dimensions Using Numerical Integration
Lara Löhken, Matthias Bolten, Hanno Gottschalk et al.
A New Computational Model for the Spin-Free Sum-of-Squares Energy Bounds of Electronic Hamiltonian
Yuming Bao, Yancheng Wu, Dongdong Ge et al.
From Worst-Case to Smoothed Complexity in Continuous Optimization
Daniel Cortild, Coralia Cartis
On Federated Averaging with Partial Participation under a General Variance Condition
Ziming Wang, Kun Huang, Karl Henrik Johansson et al.