Entropy (information theory)

Lore & Background

The core idea of information theory is that the informational value of a communicated message depends on the degree to which the content is surprising. If a highly likely event occurs, the message carries very little information; if a highly unlikely event occurs, the message is much more informative. The information content, also called surprisal or self-information, of an event is a function that increases as the probability of the event decreases, defined as the logarithm of the reciprocal of the probability.

Reader's Guide

Entropy measures the expected amount of information conveyed by identifying the outcome of a random trial. For a discrete random variable X with probability distribution p, entropy is defined as H(X) = -∑ p(x) log p(x). The choice of logarithm base varies: base 2 gives bits (shannons), base e gives nats, and base 10 gives dits, bans, or hartleys. Shannon proved in his source coding theorem that entropy represents an absolute mathematical limit on how well data from a source can be losslessly compressed onto a perfectly noiseless channel, and strengthened this for noisy channels in his noisy-channel coding theorem. Entropy in information theory is directly analogous to entropy in statistical thermodynamics, where Gibbs's formula is formally identical to Shannon's formula. The definition can be derived from a set of axioms establishing that entropy should be a measure of how informative the average outcome of a variable is.

Did You Know?

• Entropy is the expected value of the self-information of a variable.
• A fair coin flip has an entropy of one bit.
• English text, treated as a string of characters, has fairly low entropy because it is fairly predictable.
• The definition for a continuous random variable is differential entropy, given by E[-log p(X)].