Prior Ordering and Monotonicity in Dirichlet Bandits

Abstract

One of two independent stochastic processes (arms) are to be selected at each of n stages. The selection is sequential and depends on past observations as well as the prior information. Observations from arm i are independent given a distribution P_i, and, following Clayton and Berry (1985), P_i's have independent Dirichlet process priors. The objective is to maximize the expected future-discounted sum of the n observations. We study structural properties of the bandit, in particular how the maximum expected payoff and the optimal strategy vary with the Dirichlet process priors. The main results are (i) for a particular arm and a fixed prior weight, the maximum expected payoff increases as the mean of the Dirichlet process prior becomes larger in the increasing convex order; (ii) for a fixed prior mean, the maximum expected payoff decreases as the prior weight increases. Specializing to the one-armed bandit, the second result captures the intuition that, given the same immediate payoff, the more is known about an arm, the less desirable it becomes because there is less to learn when selecting that arm. This extends some results of Gittins and Wang (1992) on Bernoulli bandits and settles a conjecture of Clayton and Berry (1985).