Symmetry in Shannon's Noiseless Coding Theorem

Abstract

Statements of Shannon's Noiseless Coding Theorem by various authors, including the original, are reviewed and clarified. Traditional statements of the theorem are often unclear as to when it applies. A new notation is introduced and the domain of application is clarified. An examination of the bounds of the Theorem leads to a new symmetric restatement. It is shown that the extended upper bound is an acheivable upper bound, giving symmetry to the theorem.The relation of information entropy to the physical entropy of Gibbs and Boltmann is illustrated. Consequently, the study of Shannon Entropy is strongly related to physics and there is a physical theory of information. This paper is the beginning of of an attempt to clarify these relationships.