Differential Shannon and R\'enyi entropies revisited

Abstract

Shannon entropy for discrete distributions is a fundamental and widely used concept, but its continuous analogue, known as differential entropy, lacks essential properties such as positivity and compatibility with the discrete case. In this paper, we analyze this incompatibility in detail and illustrate it through examples. To overcome these limitations, we propose modified versions of Shannon and R\'enyi entropy that retain key properties, including positivity, while remaining close to the classical forms. We also define compatible discrete functionals and study the behavior of the proposed entropies for the normal and exponential distributions.