MDP Abstractions from Data: Large-Scale Stochastic Networks

Abstract

This work proposes a compositional data-driven technique for the construction of finite Markov decision processes (MDPs) for large-scale stochastic networks with unknown mathematical models. Our proposed framework leverages dissipativity properties of subsystems and their finite MDPs using a notion of stochastic storage functions (SStF). In our data-driven scheme, we first build an SStF between each unknown subsystem and its data-driven finite MDP with a certified probabilistic confidence. We then derive dissipativity-type compositional conditions to construct a stochastic bisimulation function (SBF) between an interconnected network and its finite MDP using data-driven SStF of subsystems. Accordingly, we formally quantify the probabilistic distance between trajectories of an unknown large-scale stochastic network and those of its finite MDP with a guaranteed confidence. We illustrate the efficacy of our data-driven results over a room temperature network composing 100 rooms with unknown models.