So, I recently took professor Andreas Krause’s really well thaught course on Probabilistic Artificial Intelligence, and while the script exhibited remarkable clarity and mathematical soundness, I feel that it could have used some more straight-forwardness in explaining some notions to poor non-mathematicians as yours truly. One of such notions is ELBO. In no way I’m implying that I will do a better job (or a good job at all) at explaining this, but I already had the LaTeX laying around for formulas, so might as well put it to use.

How does one derive ELBO?

We’re gonna approach the discussion from a different way than it’s typically done, but bear with me for a second and I think you will appreciate its rigor as well.

Say we got some random variable and some other random variable , no one can stop us from also taking into consideration some different random variables and such that has the same support as and as . Then (denoting by the p.d.f. of the RV ):

Now, let’s map this general identity to the specific context of Bayesian inference:

• let be the joint distribution of , i.e.
• from the joint distribution we can define all marginal/conditional probabilities, this justifies denoting them using the same letter, e.g. , etc.
• likewise we define as the joint distribution of .
• furthermore, and might be parameterized, so we identify with and with .

Finally, we get:

is called the Evidence Lower BOund (ELBO) because we define as evidence, and we have that:

admits different forms:

Variational inference

The primary application of the ELBO is in Variational Inference (VI). Suppose we have a model and observing data , we want to infer the latent variables . Ideally, we would compute the posterior . However, applying Bayes’ rule requires calculating the evidence , which is often an intractable integral.

Variational inference bypasses this by picking a family of approximate posterior distributions and trying to find the member of this family that is “closest” to the true posterior. If we choose the KL divergence as our measure of closeness, we want to solve:

Problem is, we can’t minimize this directly as it depends on the intractable . Here is where the decomposition we derived earlier shines:

Notice that the evidence is constant with respect to the variational parameters . Therefore, maximizing the ELBO is equivalent to minimizing the KL divergence between the approximate and true posterior. The ELBO provides a computationally tractable proxy for the intractable inference problem.

This transformation is powerful because it turns an integration problem (inference) into an optimization problem, which we are generally better equipped to solve (e.g., via stochastic gradient descent).

The tightness of the bound

The “gap” between the evidence and the lower bound is exactly the approximation error . The tighter the bound (higher ELBO), the better our approximation is to the true posterior . If our variational family is rich enough to contain the true posterior, the gap becomes zero and the ELBO equals the evidence.

Diffusion generative models

Here, we have some data distributed according to some real process we want to imitate. We learn a denoising Markov chain and want to maximize the probability it produces that real data ():

whereas our is actually gonna be the fixed forward (noising) Markov chain. Then:

By maximizing this lower bound with respect to , we effectively train the reverse process to invert the diffusion process . This objective is commonly decomposed into a sum of KL divergence terms matching the forward posterior transitions with the learned reverse transitions at each step, making the training of deep generative models stable and efficient.