Entropy Minimization for Optimization of Expensive, Unimodal Functions

Abstract

Maximization of an expensive, unimodal function under random observations has been an important problem in hyperparameter tuning. It features expensive function evaluations (which means small budgets) and a high level of noise. We develop an algorithm based on entropy reduction of a probabilistic belief about the optimum. The algorithm provides an efficient way of estimating the computationally intractable surrogate objective in the general Entropy Search algorithm by leveraging a sampled belief model and designing a metric that measures the information value of any search point.