Watching a drunkard for ten nights: A study of distributions of variances

Abstract

For any physical observable in statistical systems, the most frequently studied quantities are its average and standard deviation. Yet, its full distribution often carries extremely interesting information and can be invoked to put any surprising properties of the individual moments into perspective. As an example, we consider a problem concerning simple random walks which was posed in a recent text. When a drunk is observed over L nights, taking N steps per night, and the number of steps to the right is recorded for each night, an average and a variance based on these data can be computed. When the variance is used to estimate p, the probability for the drunk to step right, complex values for p are frequently found. To put such obviously nonsensical results into context, we study the full probability distribution for the variance of the data string. We discuss the connection of our results to the problem of data binning and provide two other brief examples to demonstrate the importance of full distributions.