Algebraic Decompositions of DP Problems with Linear Dynamics

Abstract

Inspired by rational canonical forms, we introduce and analyze two decompositions of dynamic programming (DP) problems for systems with linear dynamics. Specifically, we consider both finite and infinite horizon DP problems in which the dynamics are linear, the cost function depends only on the state, and the state-space is finite dimensional but defined over an arbitrary algebraic field. Starting from the natural decomposition of the state-space into the direct sum of subspaces that are invariant under the system's linear transformation, and assuming that the cost functions exhibit an additive structure compatible with this decomposition, we extract from the original DP problem two distinct families of smaller DP problems, each associated with a system evolving on an invariant subspace of the original state-space. We propose that each of these families constitutes a decomposition of the original problem when the optimal policy and value function of the original problem can be reconstructed from the optimal policies and value functions of the individual subproblems in the family. We derive necessary and sufficient conditions for these decompositions to exist both in the finite and infinite horizon cases. We also propose a readily verifiable sufficient condition under which the first decomposition exists, and we show that the first notion of decomposition is generally stronger than the second.