← Back

Musings on Entropy

2026-06-02
Machine LearningEntropyMaths

Why?

I wanted to learn about how cross-entropy loss works in Machine Learning.

What is a random variable X?

An assignment of numbers to outcomes of a random experiment. The reason we care about a random variable is that it quite elegantly maps to a probability distribution, be it discrete or continuous.

Regardless of the type, it describes how the probabilities are distributed over the possible outcomes (random values, i.e x_1, x_2 \dots x_n) of the random variable, X.

What is meant by the entropy of a distribution?

Intuitively: events that are rare are more surprising, and therefore more valuable than ‘common’ ones in terms of the information they bring.

  1. Likely events should have low information content (guaranteed event = no information)
  2. The less probable an event is, the more surprising it is and the more information it yields.
  3. Independent events are measured separately and should thus have additive information (since two successful independent occurrences carry twice as much information as one)

Self-information

(AKA the information content/self-entropy) of an event X = x is

I(x) = -\log(P(x))

-and is measured in ‘nats’ (assuming the log applied is a natural log w/ base e; different bases output units other than nats).

The key takeaway here is that the self-entropy is simply the ‘negative log probability’ of some arbitrary event x, from the random variable X.

Why use the log-probability in particular?

Log probability is an alternate representation of probabilities on a logarithmic scale (-inf,0] rather than b/w [0,1].

Since it beautifully maps the desired requirements of our information-accumulating system, or more pedantically, the three axioms required to fulfill Shannon’s definition of self information.

  1. As the probabilities of independent events are generally supposed to multiply, and logarithms convert multiplication to addition, the log probabilities of independent events instead add!
    • This is exactly in line w/ our third requirement from B.!
  2. As the probability (input to log probability) decreases, the absolute magnitude of the logarithm increases, satisfying our second requirement i.e rare events should carry more information content.

What is meant by the entropy of a distribution (02)?

H(X) = - \sum(p(x) * I(x)) where x (event) belongs to X (random variable).

Kullback-Leibler (KL) Divergence

KL Divergence is a metric of measure for how different two separate probability distributions over the same random variable are.

I’m not going into the derivation of this expression, but it’s not incredibly challenging to comprehend, and Sebastian Hoenig’s site in the acknowledgements links has a decent proof, should you wish to dive deeper.

For now, just know that the KL-Divergence states:

D_{KL}(P || Q) = \sum P(x) * \log \frac{P(x)}{Q(x)}

-for all x within X, where P and Q are different probability distributions.

Since the KL-divergence is a measure of distance, it is ostensibly sensible to start w/ attempting to minimise this value in an attempt to minimise the distance b/w the true distribution and our model’s predicted distribution.

Replacing P and Q w/ y and \hat{y} respectively, and expanding the log out (as per the log division rule) out, we ultimately obtain:

D_{KL}(y || \hat{y}) = \sum{y_i \log y_i} - \sum{y_i \log \hat{y}_i}

That is to say:

D_{KL}(y || \hat{y}) = -H(y) + H(y, \hat{y})

FIXME:

Notice how the first summation term (i.e -H(y)) has no sign of \hat{y}. This allows us to affirm that minimising the cross-entropy loss is equivalent to the minimising the KL divergence, thereby bringing our prediction distribution, closer to the true distribution.

Acknowledgements