New Modelling and Solution Frameworks for Multiple Objectives, Decomposability and Impulsivity in Continuous-time Markov Decision Processes

Abstract

Impulsive control, in the context of continuous-time Markov decision processes (CTMDPs), refers to actions that instantaneously update the state of the process, leading to theoretical complications. Previous work has overcome these complications, but at the expense of approachability and ease of implementation: those models are mathematically complex and incompatible with pre-existing solution methods. This paper presents an alternative formulation of impulsive control for CTMDPs that is easier to employ in the modelling of real-world problems and can be solved by pre-existing solution methods. This formulation is then extended to allow for multiple objectives and decomposability. A decomposition-based heuristic method is developed that leverages Dantzig-Wolfe decomposition to find approximate Pareto fronts for large multi-objective decomposable impulsive CTMDPs. The modelling framework and solution methodology are applied to a dynamic maintenance problem to demonstrate their computational efficiency, solution space coverage, and solution quality.