Uncertainty-Aware Hybrid Offline Reinforcement Learning with Online Action Selection Enabled by Sparse Gaussian-Process Critics
Marco Chan, Christos Evripidou, Liqiu Dong, Benoit Chachuat, Mehmet Mercangöz
Abstract
Start-ups and grade changes remain largely operator-driven, while developing and maintaining dynamic models for model-based control requires substantial engineering effort. Hybrid Offline Learning and Online Optimisation (HOFLON) uses one-step offline reinforcement learning to select actions from historical transition data. In its original implementation, the online objective combines the predicted return from a tree-ensemble critic with an adversarial-autoencoder (AAE) penalty for unfamiliar state-action pairs. We examine the effect of this penalty and replace the critic and AAE with a sparse variational Gaussian-process (sGP) critic. Actions maximise μ- κσ, an uncertainty-adjusted estimate of the return from one action followed by the historical operating practice. On the HOFLON polymerisation-reactor benchmark, removing the AAE from HOFLON led to 4 failed start-ups in 15 runs over five critics. At its published weight, the variation in the AAE penalty is four orders of magnitude smaller than the variation in the tree-ensemble critic. Nevertheless, it changes actions when the predicted returns of competing candidates are nearly equal, and paired runs with and without the penalty diverged from the first move. With a Matérn-5/2 sGP critic, κ= 0 led to failure in 13 of 15 runs, whereas with κ= 3 none of 15 runs (five critics, three noise realisations) failed. The median and lowest returns were 13,464 and 13,398 (HOFLON: 13,393 and 12,411). A squared-exponential sGP critic with learned inducing inputs needed κ≥ 3 to remain within the empirical data support in steady state, and then settled with a temperature offset of 2.0 K.
Create a lesson
Related papers
Bio-PM-Guided Synthetic IMU Generation
Deepali Singh, Deepesh Parpani, Aviral Gupta
Active Feature Acquisition by Distilling an Oracle Teacher
Yang Meng, Zixin Ding, Robert L. Grossman et al.
The Window Lever: Longer Observation Windows Outweigh the Observed Effects of Data, Capacity, and Architecture in Customer Behavior Forecasting
Tara Sowrirajan, Yanqiao Wang
Hybrid Retrieval for Government Scheme Search in English and Marathi
Aishwarya Gunjal, Kanchan Khemnar, Pawan Poojary et al.
Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
Élodie Côté-Gauthier, Doina Precup, Isabeau Prémont-Schwarz
A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes
Francesco Caporali, Jason M. Altschuler, Boris Hanin