为什么情报中的信息有其代价——Alexander Mattick

Effectively, information can be very expensive. In fact, I would say most information is very, very expensive. If you completely say, “Constraints don’t matter. I don’t need any prior information. I don’t want to encode anything in there,” we still have the problem of needing to relearn everything from scratch.

He believes that you can figure out a suitable reward function for every task, and that is enough. I think that energy-based models are not really useful, or at least not useful anymore. I think that world models and—so, both JEPA and world models are ultimately branding. There were JEPA-like models before JEPA, and there were world models before world models.

Right now, all of these methods are not set up like this. I think they are set up to look very nice in demos and to actually do some really impressive stuff. How do you know? How many times did it fail to do something—

Amazing. Alexander Mattick, it’s amazing to have you back on MLST. It’s been many years since you were on the show, and we were so grateful to you when you last came on the show because you helped us do some research for our interview with Yann LeCun and Randall Balestriero about the spline theory of neural networks. We did that intro all about interpolation and extrapolation.

For folks at home, your nickname is ZikZak on Yannic Kilcher’s Discord server, and you are by far the smartest person on Yannic’s Discord server. That is saying something, believe you me. Alexander, it’s wonderful to have you on the show. Can you give us a bit of a background?

My name is Alexander Mattick. I first did my bachelor’s at FAU in Erlangen, Germany. Then I also did my master’s there. My bachelor’s was in computer science, and my master’s is in artificial intelligence.

Then I went over to Fraunhofer, which is one of the biggest applied research institutes, and now I’m also dual-affiliated with the Technical University of Nuremberg, which is still under construction and very much in the process of becoming a real thing. It’s really exciting to be one of the first people to do my PhD there.

Folks, today we’re going to get into savant levels of technical detail. We’re going to talk about inference—things like energy-based models, diffusion, flow matching, and normalizing flows. We’re going to talk about theories of deep learning, constrained reinforcement learning, and world models, and maybe some other stuff as well.

Why don’t we start with inference? Can you sketch out the history? Back in the olden days, we might have parameterized a basic normal function, and then we had things like energy-based models, flow matching, and normalizing flows. Sketch out the whole history.

1. The Inference Problem

In the very beginning, one of the first ideas was that if we have these nice parameterized densities where we know that, regardless of which parameters we put in, we will end up with a normalized density, then we can work with those. Imagine a normal distribution, but there are plenty of others. A categorical distribution is based on the same idea.

The core problem when we try to do inference is that the density has to be normalized at some point. The first idea was, of course, to say, “Let’s just pick a group of functions that is always going to be normalized.” This gives us the traditional variational inference problem. Effectively, we have a normal distribution with unknown parameters, and now we try to fit it to the data.

The issue with this, which people noticed very quickly, is that you are now very constrained by the actual density. Let’s say your data is not normally distributed—let’s say it’s bimodal. Your actual density no longer fits. If you imagine that you have one mode here and another mode here, fitting a normal distribution would put all the mass basically in the middle and try to split the difference.

The first idea that came up more or less at the same time was something like rejection sampling. Rejection sampling is something you may have already seen in school, where you paint a square, put a circle inside it, and generate random points. The fraction of random points that falls inside the circle can be used to compute pi.

You can use exactly the same idea with any kind of density, as long as you have something that is above it. Let’s say we have a box that is essentially a uniform distribution. You scale it up so that it surrounds your density, and then you randomly pepper it with points. That’s rejection sampling, and it works pretty well, assuming you have a relatively close envelope over your function.

It doesn’t scale very well, especially with high-dimensional data. It’s one of the quintessential examples of the curse of dimensionality. Then people said, “We can do a little bit better by framing this as the limiting process of a Markov process,” and this gives us MCMC.

The idea is that we may not be able to create the density, but we are able to sample from it. That’s one of the very interesting things about inference, or anything density-based: you can do a lot of things without actually knowing the density itself, just by sampling and rejecting samples.

The issue with this is that if you’ve ever run Markov chain Monte Carlo, you’ll notice that it’s insanely slow, especially in high dimensions. Again, we have the curse of dimensionality. Not only do you have these high-dimensional problems with many modes, where escaping from one mode to the next is extremely difficult, but because you still have an accept-and-reject step, you end up doing a lot of work for things that don’t go anywhere.

Alex, just a quick point of order on that. What about GFlowNets? Didn’t Bengio come up with some kind of amortized way to replace Monte Carlo sampling?

GFlowNets are an interesting case, but I don’t think they really solve the same problem. What a GFlowNet does is, let’s say you have a disentangled distribution. You effectively have a graph that you walk around, and you construct your sample slowly over time.

The idea of a GFlowNet is that if we have some final density in the end—for example, a reward function—the incoming and outgoing flow at every node has to be the same. Essentially, you intensify the paths that lead to a higher reward and reduce the ones that lead to a lower reward.

However, you are basically back to something closer to the variational notion, where you make the assumption that you have a specific structure and assume that structure is correct. If you have, for example, some molecular problem where you start building something up, we can assume that this process is accurate. But this is not a general way of doing inference.

The general way is still that you need to do sampling when you have a very, very general function family to actually be able to do that.

Very cool. Keep the story going. At some point, energy-based models and normalizing flows appeared. Explain that to me.

Energy-based models actually come directly from MCMC. The reason they’re called energy-based models is that they come from physics. In physics, they literally had a function for energy, and you would assume that particles conformed to a distribution proportional to the energy.

If you have a specific box with hot particles in it, you would imagine that they follow some form of distribution, because not every particle is moving at the same speed.

Some are, some are a lot faster, and some are a lot slower. That gives you this statistical-mechanics point of view. Indeed, I think it was Teller who made the first developments in MCMC and got into the idea of doing simulations to obtain the actual parameter or particle distributions. When you look at what MCMC does, you sometimes phrase it literally as energy-based sampling, but in other cases, it is effectively the exponential of the energy. It depends on exactly which algorithm you talk about and exactly who you have teaching you.

Essentially, this is exactly what you would imagine as the gold standard of an MCMC method. The MCMC method has an energy, which is an unnormalized density, and now we want to turn this into a normalized density, which is what the particles are actually doing.

An energy-based model is any model that parameterizes complex densities using an energy function. Energy functions correspond to an unnormalized density. The lower the energy, the more favored the state, just like in physics, where low-energy states are preferred. Being unnormalized makes EBMs quite easy to train, since no complex integrals need to be computed during training.

Instead, an energy-based optimization simply pushes up areas with low sample support and down areas with high sample support. This makes energy-based modeling quite flexible, with many different objectives, all targeting slightly different assumptions when it comes to, for instance, the availability of a reference energy to track.

While easy to train, this comes at a steep cost during sample generation. Since energies are unnormalized, we need to perform complex Markov chain Monte Carlo optimization to sample states proportionally to the energy without having access to a normalization constant. This makes energy-based modeling unfavorable for generative usage. The most expensive part of sampling gets executed over and over again during inference instead of being amortized during training.

Further, energy-based models do not offer access to the probability of any sample, which is helpful for many applications, such as anomaly detection or correcting accumulated errors when unrolling a dynamics model.

Just another quick point of order. Can you give a little bit of context here? Many of us are machine-learning people, and we are not in the mindset of working with densities and probability distributions. We are using models that give us point estimates.

What is the benefit of having these densities? Is it because they are composable? Now we have to do more work at inference, right? We have this density, and we actually need to search around on it and do things with it. That is what this inferencing thing is. What does it give us?

2. Why Densities Matter

I think there are 2 answers to this. One is that most of deep learning is actually doing inference, right? If you fit a categorical distribution, like you do with your large language model, you have your density, which is effectively an autoregressive structure. That is just a decomposition of your final density.

What you actually want is to model the probability of generating a certain sentence given, let’s say, a prompt. What you end up doing is producing the actual answer step by step, token by token. This works really nicely in something like language modeling because you ultimately still have a discrete measure. You have all your tokens, and you choose which one to get.

A categorical distribution is more or less universal. I can always just write down the probabilities of everything. Even if you look at something like regression or image detection—for example, depth estimation—with your mean squared error, you are still doing inference. Mean squared error is just the negative log-likelihood of your data under a normal distribution with a fixed standard deviation. So you are still doing inference at some point.

What you lose is the breadth of the density, but that is more an assumption. You can still do exactly what you are currently doing, but get an entire mean and standard deviation, and that can be super useful.

My background is reinforcement learning. In reinforcement learning, one of the best things you can do to improve the performance of your algorithm is to say, “I don’t actually want a point estimate. I want some breadth to do better exploration.” That is the big advantage of, let’s say, a policy-gradient method over a Q-learner.

A Q-learner only tells you what to do. It does not tell you, “If you do the different thing, I expect you to be worse by this or that amount.” If you had that information, you would be able to say, “For tactical exploration, I choose the one which might be slightly worse but ends up being better in the long run.”

If you are interested in something like active learning, then you also want densities. So I would push back on the fundamental notion that we are not doing inference when we look at normal deep learning.

As an aside, I interviewed Chris Watkins last week. He was my professor, and he invented Q-learning in the late 1980s. That was a great interview. Maybe you should briefly explain what Q-learning is, because this exploration thing is obviously the biggest problem in reinforcement learning.

But help me understand this: what is a good density? Bayesian folks, for example, valorize the idea of a normalized or calibrated density. You were talking about the softmax discrete distribution that we have at the back end of a language model. What does good look like? To what extent is that a good density?

That is a much bigger question because it really depends on what you end up using your density for. Let’s say you are completely deep-learning-pilled and want to have a large model that describes the density of images. You do diffusion—we are going to get to that—but we are doing diffusion for image modeling.

What you are ultimately interested in is a density that explains images the best. You want to have as high a likelihood as possible for real images, or, if you have a prompt, the highest likelihood for things that actually correspond to that prompt description.

Ultimately, what you are interested in is a density that is high enough on the good parts and low enough on everything else. That, of course, goes into the discussion of a prior. A prior effectively tells you that we do not want to memorize our data set. We want a smooth path from peak to peak so that we can get images in between the data points. That is a prior.

A prior is simply something that tells you, from a discrete data set, how you can get from, let’s say, the 50 points of your data set to the 51st point between 2 different samples. If you did not have a prior, then the exact solution would be to memorize the data, and it would be rather useless.

If you go to the other extreme, what the Bayesian people do is say, “I actually want information in my latent space,” or basically, “I want to reveal information that is not in the data by looking at latents.”

A classic example is that I flip a coin and observe heads or tails. What I am actually interested in is whether the coin is fair. If the coin is not fair, then you will see this in the latent, because the latent is effectively a distribution over the probabilities that it could have.

If there is a 99% probability that the coin is fair, I can accept the hypothesis that your coin is fair. If there is a 99% probability that it always lands on tails, then I can infer that it is actually an unfair coin. That is what statisticians are usually interested in.

That is the big divide between someone who does statistics in, let’s say, the medical domain, who wants to figure out, given those tests, what the likelihood is that you have a disease, and someone in deep learning who just cares about, “I want to generate pretty images of pink elephants.”

Yes. My intuition is that we are machine-learning people with a finite sample of data, and our purposes might be that, at some point, we have to interpolate between these data points. We have the curse of dimensionality and so on, so we have to put some kind of structure in place. We are amortizing this process, and we have to smooth it and make it amenable to the types of predictions that we are doing.

Exactly. This goes nicely back to the discussion of variational inference with sampling. Because, of course, we have this hybrid thing with energy-based models, where you effectively say, “I still want to do sampling. I still want to do MCMC, but I learn the energy.” That is what the energy-based people did.

And this was a very nice idea, at least at first, because it gives you the property that you can learn basically arbitrary densities. Assuming your energy function is rich enough that it can approximate the energy, then you can effectively just pawn off the problem of normalizing the density to sampling time. But if you set up your loss function in some clever way, you can do the training without needing to do sampling.

That is a huge advantage because now you can either directly fit to an energy, which you might have literally measured, or you can do this for samples of some density. You try to push down the energy where the samples are rich, and push it up where the samples are not rich. It is exactly the same way as you do with the density.

The problem you run into with this, of course, is that now you have the difficulty of both worlds. You have the problem of variational inference: What if my function is not strong enough? And then you have the problem of the sampling-based world, where suddenly you have the issue that my actual sampling process is really, really slow. This is what always limits energy-based models.

3. Diffusion And Normalizing Flows

The next idea, which came up more or less in parallel, is this idea: What if we simplify our problem a little bit and effectively give ourselves an infinite pool of latents? Because latents in inference are powerful. If you can split your problem up into smaller subproblems by conditioning on some information that can make it conditionally independent, then these subproblems can become a lot simpler.

In many cases, a model becomes cheaper the more latents you have. Of course, this is not always true, but generally speaking, you want the flexibility of taking tiny steps, which is what you, for example, parameterize. The actual big steps then come simply from the fact that you have latents which you condition on. The absolute limit would be to say, “What if I have infinite latents?” That is what gives you diffusion.

Diffusion is the idea that I have a Markov process similar to MCMC, but with infinite steps, so you can make it smooth and continuous. At every step, I now sample based on the previous step’s information, like in the normal Markov step. This has one big difficulty: You now actually need to model time, because what you are now saying is that we have latents, such as the latent at time step 0.5. If you go down to the latent at time point 0.6, you end up effectively having a different density.

Now we have this time dimension as an additional variable. This is what you pay for. But in general, this is still much, much cheaper than needing to do the actual MCMC process, because now you just follow essentially a density. You randomly inject more noise at every time step, but you no longer need to have an accept-reject step.

You no longer have the problem of local minima and so on, because essentially what you did is rephrase this as a stochastic differential equation, which then goes from the initial density to the final density. You are paying for it with one additional variable, which effectively indexes the latents. This gives you a huge improvement because you no longer need to do this at inference time. It still exists in MCMC as the sampling steps, but effectively you amortize that as well, and that makes it a lot cheaper.

Then you can go a little bit further and say, “What if we go away from this Markovian structure in the first place?” Or maybe not from the Markovian structure, but from a specific diffusion-style, Fokker–Planck-style formulation. What if instead we just phrase this as reparameterizing a density from beginning to end in tiny, tiny increments? That then gives you what is effectively these flow methods or, more generally, continuous normalizing flows.

Continuous normalizing flows are effectively phrasing the problem of finding a density as starting off with an initial guess. Just as in optimization, you start with an initial guess, which is like your normal distribution, and then you end up with your final density. What you end up doing is finding an optimization process that gets you from there to there, which is a gradient flow and gives you these optimal transport-style problems.

We say that we have the continuity equation, which says that the change over time is proportional to the divergence of the vector field of the current density. If you write this out without invoking densities, but instead invoke it with points, you see that this is just doing gradient descent in a continuous domain. Of course, it is the cheapest thing of all, because now you no longer do any sampling. Now you phrase the problem as: I do optimization in infinite-dimensional space.

You pay for the infinite dimensions, but you get the lack of sampling. Effectively, the entire trajectory was that you start off with the very expensive sampling and the variational models, which were very limited. Then you said, “Well, let’s do both in the middle,” which were these energy-based models. Then you said, “Well, can we make the energy-based models less like rejection sampling?” That then gives you diffusion models.

Now you say, “When we actually do this trajectory from an initial distribution to a final distribution, we do not need to do this Markov chain process at all. We can phrase it as just a pure optimization problem without any stochasticity by instead updating the density at every point.” The only thing you now need to pay for is invertibility, because you need to be able to go in one direction to generate samples and in the other direction to get the likelihood.

You need the likelihood to actually do inference. That is really cheap because a vector field, at least in a density sense, is invertible. Instead of going downhill, you just go back uphill.

Diffusion models do density optimization as an invertible noising process. During training, first, a sample gets repeatedly corrupted with normal noise until only normal noise remains. The training objective is simply to revert a single corruption step, for instance, going back from step t = 5 to t = 4. This corresponds to learning a vector field, Vₜ(x), that tells you how to change a sample to make it look more like a target sample.

During inference, for sample generation, we simply draw a sample from a normal distribution and follow the learned denoising process backward. Inversion like this works because, in the background, this corresponds to solving a stochastic differential equation. This equation is a reversible coupling between a normal distribution and your target distribution. This also allows you to compute the probability of any sample from your target distribution.

Simply follow the target sample through the noising process and accumulate the divergence of your vector field. This shows the major upside of diffusion models. Instead of needing to perform MCMC during inference time, we only need to follow our vector field across time, which can be further accelerated using sophisticated ODE solvers.

The downside of diffusion models is that we now bloat the representation with a synthetic time dimension, t. This adds an extra cost that the neural network needs to work with and also slows down training. We effectively solve many tiny problems of reverting individual noising steps. However, this is a one-time training expense instead of an inference-time expense, making this cheaper for real-world generative usage.

Normalizing flows are models that treat density optimization as a reparameterization process. Instead of trying to model a density from scratch, a normalizing flow starts off with a known base measure and transforms it into the target measure. If the series of transforms applied to the base measure is invertible, we can use this to essentially inherit the normalization of the base measure. We only need to track the local expansion or contraction, which corresponds to the Jacobian of a transformation and is trivial to do with any framework supporting backpropagation.

Sampling simply follows the flow forward. Starting with a base-measure sample, apply the model to get the target-measure sample. Computing probabilities goes in the other direction. Follow the flow backward from the target back to the base by rescaling based on the function’s Jacobian.

One core limitation of naive normalizing flows is the constraint that functions have to be invertible for sampling to work. The way around this is surprisingly easy. A continuous normalizing flow frames a normalizing flow as following a path from the base to the target measure, based on a vector field that gives local directions. The trick is that such a path, a gradient flow, is necessarily invertible, even if the function giving you the vector field is not.

Just walk the same path backward. This has the nice property of allowing arbitrary couplings between measures. The base measure could be a normal distribution, a Student’s t-distribution, or even another normalizing flow.

Anything where you can compute the likelihood of a sample works. However, this also comes at a non-zero cost. For training, we need to compute the full ODE from the target sample back to the base measure every single time, since that is needed to compute the likelihood of the sample for maximum-likelihood training.

Okay. I think there are quite a few things we need to unpack here because we've suddenly gotten very technical. So, if I understand correctly, in a normalizing flow, it does need to be invertible, but in flow matching, it doesn't need to be invertible.

Just to help the audience understand, what we're talking about here with flow matching is the ability to model a vector field or an optimal transport between noise and the data that we want to model. I still think we're missing the link here a little bit to energy-based models. LeCun, when he talks about energy-based models, talks a lot about contrastive learning and how you can build specific models to do specific types of things. Many folks would have seen the HJEPA architecture, for example. Maybe we should start with that: what kinds of prediction architectures and predictive tasks can you do with EBMs and these normalizing flows?

4. Energy Models And JEPA

I think JEPA is still a little bit further away. But what I would say is that the difference between something like an energy-based model and something like a diffusion model is that both of them are fundamentally generative models. I don't really get this idea of energy-based models not being generative. I've heard this a bunch of times, but it's just not true. The interesting thing is that you can maybe use them in settings where you don't need to be generative, but fundamentally, you are still generative. You still can generate samples. It's just very expensive, so nobody does it.

Sorry, can you just explain that? That is something LeCun says quite a lot. He says that a good property of these energy-based models is that, because they're not generative in some way, they're better. Can you explain that?

Fundamentally, you can still do exactly the same thing. You take your energy model, throw it into an MCMC sampler, and technically, you will get samples out. This is very technical. Of course, many of these energy-based models are not actually good enough to generate nice samples, right?

But at least in theory, if you had the perfect energy-based model—someone comes down from heaven and says, “These are the right parameters”—you plug them in, and now you have the actual data-generating process. You can use them to generate data, right? In fact, this isn't even that special of a property. Fundamentally, as soon as you have something that is proportional to your density, you can throw this into an MCMC sampler and generate samples.

So the distinction between having a generative model and having a non-generative model is really, really small, because the question is not whether they can do it; it's whether it's economical to do it. I would agree with LeCun that it's not economical to use energy-based models to generate data. But I wouldn't frame this as either a disadvantage or an advantage.

The nice thing about energy-based models is that you can do certain things more cheaply. Namely, the very big one is that you can do comparisons between data points, right? There is effectively an unknown scalar inside energy-based models. But you can still say, “Well, this point is three times as likely as this other point,” by effectively computing the ratio.

If you just need this—if you have a problem where you only need pairwise comparisons—then energy-based models can be quite good, right? You don't need to generate data points to produce something like—not a likelihood, but something like a score. “Score” is also maybe a bad word because it's also inference-coded. But you still get the relative difference between two points.

Of course, the downside, and what makes it hard for us to be generative, is that even if the ratio between two points might be really large, it still might be really, really bad because you don't have an absolute score. It's like saying, “Well, this movie is a 5.” Well, 5 out of 5, 5 out of 10, or 5 out of 100? It doesn't tell you enough to actually figure out whether it's a good sample or a bad sample.

Actually figuring this out in an absolute sense is the expensive part. Energy-based modeling is really nice if you never need to sample anything. This is what makes it really cool for, let's say, pre-training, right? If you do pre-training, then doing something like an energy-based model might make sense.

Of course, there are many other things that also make sense, and I think it's up for debate whether it's actually a good idea to do energy-based modeling over anything else. But I think there's a completely different discussion about whether these energy-based models are good at doing inference or bad at doing inference. Essentially, you're just saying, “Well, I train an energy-based model, but I don't end up using it for real inference. I use it to effectively measure differences in intensity.”

Okay, and just help me understand a little bit more. I think LeCun would say almost any model could be an energy-based model, and something I've struggled to understand is that you can design energy-based models and decide where to place the latents. You can make it quite domain-specific, or it could be quite general. You could have an energy-based model of a transformer or something like that. So help me understand that.

More generally, this is also one of my big problems with the JEPA-style things. With JEPA, or Joint Embedding Predictive Architecture, that could describe literally anything, right? It basically just says, well, it's not contrastive. Let's say I recently made this distinction in one of the paper discussions on Yannic Kilcher's Discord. I found generally that dyno embeddings are much better than JEPA-style embeddings, at least for the stuff that I tried them on. This might be very different for your application.

But I think this is a false comparison, because if DINO came out today, people would call it a JEPA architecture because it's such an open domain. The same thing is true with, let's say, variational inference or energy-based models. You have a lot of flexibility. Energy-based models are just a name for what you end up predicting.

What you end up predicting is effectively this unnormalized density map, right? It's like a heat map. Anything that has this can be framed as an energy-based model, which is also why they show up everywhere. If you want to, you can phrase it as an energy-based model, and that often makes it a less useful description.

Where I would draw the distinction is whether it's already normalized. Let's say in reinforcement learning, I predict the mean and standard deviation given the input data. Then I say, “Well, this is not really an energy-based model.” Technically, it is. Practically, you wouldn't think of it like that.

Then, on the other extreme, you have something like score-based models, which effectively model the gradient of energy and do tricks to also remove all the normalization. That then gives you something like these Langevin diffusion-style things that give you diffusion modeling and—well, they don't give you flow matching, but they give you something like flow matching. I would agree that the difference is really, really wishy-washy.

Can you just explain this non-contrastive thing? Randall was telling me at NeurIPS that apparently—I think Hinton was the big contrastive guy, and SimCLR was one of the original contrastive architectures. LeCun is big on this non-contrastive thing. He talks about it as shrink-wrapping the latent. What does he mean by all of that?

There are different ways of framing these unsupervised learning problems. One of the first ideas was, okay, we can phrase this as making similar things similar and dissimilar things dissimilar, right? This ultimately gives you a contrastive-style model.

Basically, what SimCLR does is, if you look at the loss, it's effectively like a cross-entropy, but over the batch. You then have the positive part, which is the different augmentations that get pulled together, and the dissimilar parts get pushed apart.

The downside with this is that it's ultimately a very, very moving target—a batch-to-batch moving target. Contrastive just means that somehow the right things have to be together and somehow everything has to be apart. This means there are many, many, many solutions that satisfy this condition but aren't actually useful.

So in practice, you want something that is, A, stronger, and B, does not have this instability that you fundamentally have with these SimCLR-style methods.

If you look at SimCLR, they only start making sense at huge batch sizes. Then there is follow-up work like MoCo, the momentum contrastive models, which essentially say, “Well, we can simulate huge batch sizes by effectively taking, let’s say, a buffer over time.” This, of course, adds a little bit of noise but tends to work pretty well. But fundamentally, I agree with LeCun that we need to move away from these contrastive models, because contrastivity alone is not a strong enough statement to actually learn necessarily useful models.

What LeCun now says is, “Well, we should revisit old ideas,” which is trying to map things into a compact latent. The first idea of this was a variational autoencoder. A variational autoencoder says, well, you have to be able to map the data to a normal distribution or, let’s say, a beta distribution—whatever you set in the middle—and then you have to be able to map this back. Effectively, this compresses everything down into this tiny bottleneck in the middle, and the top part is a generative model. The bottom part is effectively an embedding model.

A problem with these variational autoencoders, and something that had also been seen very early on, is that you effectively need to pay for things that don’t end up making a difference. I don’t know who did it, but there’s an example of, let’s say, a real dollar versus a painting of a dollar. They represent the same idea, but they’re very, very different on a pixel-to-pixel basis.

This then gives you something like DINO, something like VICReg, or something like JEPA-style architectures. Specifically, one of the newer ones that I really like is this Ledger-style thing, which effectively uses statistical tests to figure out, “Okay, am I different from a normal distribution, or am I the same as a normal distribution?” Based on that, they can differentiate through the test and move you closer to or farther away from the now-fixed goal of the density.

So you replace the contrastive pushing of things apart or pulling things together with something somehow based on this fixed target. Of course, you still have some flexibility, because if every point starts pushing into the mode of the density, at some point you would have a kind of pushing-away effect again. So you have something similar to a contrastive model, but now it is contrastive compared to a reference density and not compared to itself, which makes it a lot more stable.

So what does all of this mean for energy-based models?

In my opinion, energy-based models are not really useful, or at least not useful anymore. In general, of course, you can frame your model as being energy-based. If I’m a reviewer, I will not ding you for it. But ultimately, I don’t think it’s a useful way of thinking about the vast majority of problems.

There are exceptions to that. If you’re actually in a physics setting, then maybe it makes a lot of sense to phrase this as an energy-based model. But I would argue that it’s such a general statement that it’s not a useful way of thinking about the problem.

For things like contrastive or unsupervised learning—these self-supervised learning methods—I think there is, generally speaking, a better way of thinking about the problem than energy-based models. Ultimately, I think this is where these methods lie. If you want to do generative modeling, diffusion or flow matching is better. I don’t think anyone would argue with me about that. If you want to do unsupervised learning, I think you can phrase it as an energy-based model, but I don’t think it’s a useful framing.

Yeah. We’ll get to diffusion in a minute, but can you help me understand where flow matching is being used? Folks in the audience might not be very familiar with it. For example, I was interviewing Mistral a couple of days ago, and they were using it in their Voxtral model. There are people out there using flow matching, but where are they using it?

5. Where Flow Matching Fits

The big difficulty with doing something language-model-like is that, ultimately, everybody right now wants to do language-model-like things without language: big audio models, big image models, and big world models. We can get to that later as well. All of these models want to do exactly the same thing that large language models did for language.

The problem they quickly run into is that most domains are not actually that discrete. You can make them discrete by hitting them with a hammer long enough, but practically, they’re not discrete. The problem is that if something is not discrete, things like autoregressive decomposition become kind of dicey.

For language models, the nice part is that you can effectively go step by step and predict from left to right. If you have a continuous density like an image, you can still do autoregression, but it’s no longer a natural way of doing things. It’s not a natural way of decomposing a density.

Ultimately, that’s what all of these models do, whether they are diffusion, energy-based, or autoregressive. All of them try to take a huge density—an entire sentence—and decompose it into smaller parts that can actually be handled. The idea that most people settle on for continuous processes is that instead of separating from left to right, token by token, they say, basically, we separate from bottom to top. We start off with all the tokens, but corrupt them, and then kind of go back up.

That immediately allows you to do things like image generation because, as I explained just now with tokens to make the comparison sensible, ultimately you start off with something and end up with something. In fact, the discrete case is more difficult because with discrete diffusion, you end up with some dicey things if you actually sample from the density at every single point. It’s still possible, but you have to be a little more careful.

The nice part is that once you have this general recipe of decomposing your structure, you can apply it to anything. This is what makes diffusion models or flow matching useful. If you read about diffusion models and look into them, oftentimes it’s flow matching because it tends to be a little more stable. All of these methods are just ways of decomposing a density in a really general way.

What you do in diffusion is give yourself these infinite numbers of latents and say, “Well, given the infinite number, it has to be decomposable at some point, in some way.” In the case of flow matching, you literally give the reference: “Okay, this is what you need to do. Just do it now, but generally.”

In autoregressive structures, you do exactly the same thing, but instead of giving yourself the latent codes, you say, “Well, I have a very natural decomposition of my text already. Just use that.” Ultimately, both of these solve the same type of problem. It’s just whether you do the decomposition in the observable space or in the latent space.

For high-dimensional things, which themselves have less information per effective variable, it tends to make sense—or at least empirically tends to make sense—to go into diffusion instead.

Can we talk more about inference in general? I think this is something that people don’t think enough about, because we don’t really do things that resemble inference that much. With language models, we just do greedy sampling: we’ve got this density, and we just take a token, take a token.

In something like AlphaGo, for example, we had a structured inference process. We did Monte Carlo tree search, searching around and algorithmically updating nodes that we had visited, then feeding that back and retraining the model. With these vision models, and when we’re using flow models and things like that, we’re actually doing quite sophisticated inference processes.

I interviewed Clément Bonnet. He had this Searching Latent Program Spaces paper, and he was searching a decoder to find the best programs to generate for ARC. We’ve got this whole test-time computation, test-time adaptation, and so on. This whole thing about doing work at inference time is very exciting to me, and I just don’t think that many people are talking about it.

6. Inference As Search

Yeah. I think we give people not enough credit when we talk about doing inference. First of all, inference is a really, really overloaded term. If you go into logic, they say they do inference—logical inference. If you go into Bayesian methods, they also say they do inference. Now I say, well, we are also doing inference. So I think we have to be a little careful not to use the term too liberally, because I have to be careful myself: at some point, it loses all meaning.

At some point, it becomes the energy-based model, which you can apply to anything. Ultimately, I think in many cases it is very instructive to think about these types of models as just composing conditional dependencies and marginalization.

Something like AlphaGo is actually a little bit more difficult because what they do is effectively nonparametric. Arguably, they come less from the Bayesian side and more from the frequentist side, which is an entirely different can of worms I don’t want to open up right now.

At the end of the day, all of these methods—or at least all these mainstream methods—really benefit from this idea: we have a big problem, we need to decompose it, and every step is one step in a bigger inference process.

Let’s give an example from one of my papers. I did a paper about effectively doing mixed-integer programming—learning a heuristic for mixed-integer programming. Specifically, in mixed-integer programming, without going too deep, they have a process called branch-and-bound, which essentially means you start off with an approximation and then, based on a solution, you pick 2 different solutions, just like in a tree search—or it is a tree search—and you go further down from this node.

Similar to something like AlphaGo or alpha-beta pruning, you can prune away lots and lots of nodes if you know more information by basically saying, “Well, this node is worse than the solution that I already found, so I can remove that.” It becomes really important, if you have a huge tree with thousands of open nodes at the bottom, to choose which one to actually end up selecting.

What I did was frame this as an inference problem. Basically, I say, well, if we go down from the root to one of the nodes, you effectively do a conditional probability. You have a probability to choose either left or right, and then you have another choice between left or right, and that gives you a much more general model, which grows with your problem.

As your problem grows, your inference process grows and ingests more and more information. The same thing is true with something like a large language model, which ultimately does research, looks up more data, ingests it, and then the next step is conditioned on that information.

How useful this is for the viewer’s daily work really depends on what you end up doing. If your work is scaling up a large language model, then you probably don’t need to think about this. If your research is figuring out really, really efficient architectures, maybe—but I would also say it probably doesn’t matter.

But if you use them, or do something like reinforcement learning, where ultimately everything is like—you update some density, unless you do Q-learning. If you update densities, then you should be thinking about this as an inference process, because especially Bayesian inference gives you a lot more structure and a lot more information than we give it credit for.

Something like just decomposing a problem into conditional probabilities gets you 99% of the way there, but it can already help guide the actual problem. I think all these agentic things could really benefit from thinking about this as a Bayesian problem: going out, gathering data, and updating your conditional belief based on the information that you just ingested.

Very cool. We’ve mentioned Q-learning a couple of times. Can you just give us a quick refresher on that?

Okay. The idea of Q-learning is actually quite simple. The problem that you run into in reinforcement learning is that you don’t want to be greedy with your immediate actions, because what you do immediately could be really, really detrimental.

Let’s say you manage a warehouse, and the greedy thing to do to maximize profit right now is just to sell everything in your supply closet and never buy anything, because then the profit is maximal. That’s not necessarily the right thing in the long term.

What we need to do is not look at the current reward, but look at the future as well. That gives you a problem: I now need to model the future.

The way Q-learning does this is by saying, well, let’s decompose this as a head, which is the current reward, plus the tail, which is the rest. If we have such a decomposition, then we could effectively act greedily with respect to this combined metric.

If we have a Q function that tells us, “If you go down this path, the future will behave like this; if you go down this path, it will behave like this; and this path will behave like this,” then we can be greedy again with respect to the Q function. Effectively, the Q function fundamentally is a closure over the path that the policy can take.

The way you train this is by enforcing exactly this. You say the current reward plus the next state’s Q function should be equal to the Q function starting from the original state.

You effectively say that R plus Q from the next state, or for a specific action, has to be the same as doing a Q function from the original state. This then gives you 2 options.

The first option is that, for the next action, we could choose the action that was actually taken in the real world. This gives you SARSA: state, action, reward, next state, and action.

The other option is to say that, instead of just picking the next action that would have actually happened, I pick the optimal action. Let’s say we have a discrete number of actions; we just look at every single one and say, “Well, I pick the one that is the best.”

Effectively, you hallucinate a world in which you actually chose the correct path. It has the very nice property that you are no longer coupled to the specific policy.

If you choose the action that the policy chose, you effectively update with respect to the policy. If you choose the optimal action, effectively you are updating with respect to the optimal policy already, like Q*.

The problem with this is that you are no longer dealing with just one thing. Effectively, you have a counterfactual inference problem where you need to deal with what actually happened versus what happened in our, let’s say, golden state, where we walked along the optimal path.

Very cool. Coming back to EBMs for a second, we’ve been really negative about EBMs. LeCun actually spun out 2 companies, AMI and Logical Intelligence, doing this stuff. Are there any situations where you think EBMs are better?

Again, I think the big one is if you just need ratios, because then it can be a lot cheaper. You don’t end up generating the entire trajectory.

It can also be helpful if you have something energy-like already, such as physics, where, based on data, you try to learn the energy function. Other than that, I think it becomes really, really difficult to justify.

Of course, it really depends. As we said, “energy-based model” is a really overloaded term. If you use them, for instance, as a pretraining task, I don’t see a reason why not. I don’t see a reason why you should, but I don’t see a reason why it wouldn’t work.

The question really is: if we look at LeCun’s work—let’s say we jump forward 5 years from now and LeCun’s startup is doing whatever he is currently doing, which is research—whether what they end up doing might still be called energy-based models, but whether it is actually an energy-based model as you and I would think about it right now, I highly doubt that.

While it’s possible, I don’t think it’s the best way of doing this. The good thing and the bad thing about deep learning is that, if you do something wrong, it generally doesn’t just not work; it just has worse performance.

That makes it really, really difficult to do research, which is why there are thousands of papers that essentially claim more or less contradictory things. Depending on subtle details, things can just work or not work, and they can be really, really sensitive.

Even if you do something completely wrong—for instance, recently I had a paper that I tried to get done for ICLR where I had an implementation bug—it still kind of worked. There are plenty of other things where, on paper, they don’t work, but in practice they just work, because especially in reinforcement learning, it just hammers away at the problem until it figures out something.

I think many, many things right now are like this. They are not necessarily optimal, but they’re also not bad enough to actually bother fixing.

Very good. How do we go from flow matching to diffusion models?

So again, the jump from flow matching to diffusion models is a lot larger. Ultimately, flow matching comes from 2 ways of thinking about it. One is to think of it as a continuation of continuous normalizing flows. The idea of a continuous, or any, normalizing flow is that you have a base density, such as a normal distribution, and you frame the problem as trying to figure out a transformation that goes from that base density to your target density.

If you make sure that this density is invertible—technically, a little bit less is possible, but let's say it's invertible—then we can do both sampling and density evaluation. You go from the base density forward to the sample space; this is the sampling direction. If you have a sample, you can go backward and figure out the probability of generating something by looking at where it is on the normal distribution.

This is the first idea that you can branch off from. The second is optimal transport. In optimal transport, there's a very famous theorem by Brenier and Benamou, who basically said, "We can frame an optimal transport problem from one density to another as a kinetic problem," where effectively we follow this transport equation. The density change over time is equal to the negative divergence of the current density times the velocity field.

This is effectively like gradient descent, but in continuous time. Instead of having a finite number of parameters, you do it over the actual density. It's called a gradient flow. These have really nice properties, so even in finite dimensions, these continuous normalizing flows—these gradient flows—have super nice properties.

What you end up doing is using the fact that the velocity field and the density change are coupled together. Instead of learning the density, we learn the change in density, this vector field. This is the parameterization, at least for these flow matching methods. What you now say is: instead of picking any velocity field, I use a reference, say rectified flow, which is a straight shot, and then I fit to this.

We can go from this back to diffusion, specifically if you look at the Fokker–Planck diffusion process. The Fokker–Planck equation is effectively the PDE form of the stochastic differential equation describing diffusion. A diffusion process is the idea that we have both a mean change and a standard deviation, which adds noise. We can frame this as, instead of sampling our way through, changing the density from the start to the end.

This is the Fokker–Planck equation. Instead of sampling, we now have to solve a PDE. Usually, people go the other direction because sampling is much cheaper and much easier than solving a PDE. But let's say we look at this from a PDE point of view. If we actually look at it, we can rewrite it so that the velocity field inside it is actually very specific. The Fokker–Planck equation reduces back down to having a very specific velocity field.

If you look at this velocity field, you realize 2 things. One is that there's a fundamental connection between these 2 methods. The second thing is that you can see why diffusion models are kind of iffy to train, more so than flow matching. If you look at the velocity field, it actually depends on the density. Effectively, what we now have is this self-coupling, which originally, in flow matching, we can get rid of. We can completely get rid of ever needing to touch the density, while in diffusion you end up having to touch the density.

Of course, there's this trick with the score transformation, which means you kind of get around this. But ultimately, the vector field now depends on the current state of our diffusion. The nice part about the diffusion style is that it has some really nice properties, specifically that it's kind of minimizing the relative entropy, the KL divergence between your target density and your current density, in this optimal-transport sense.

Flow matching is a method straddling the space between diffusion processes and continuous normalizing flows. The upside of normalizing flows is the ability to couple arbitrary measures with each other, while diffusion is theoretically restricted to coupling with a normal density. The upside of diffusion is fast training, since we do not need to solve a full ODE for every optimization step.

Flow matching exploits the fact that the actual path taken from the base measure to the target measure is arbitrary. This means that we can simply impose a reference direction that we fix beforehand. This is quite similar to what we implicitly did for diffusion models. A diffusion model's reference is given implicitly through the noising process. In fact, a diffusion model is simply a very specific reference flow that aims to minimize the KL divergence.

Flow matching still frames sampling as a reparameterization problem, but now fixes the degree of freedom in a path a priori. Pick a path, for instance, a straight line between a real sample and a base sample, and train a model to emulate that. The model's task is now to simply learn a general parameterization of the example flows—a simple supervised regression problem.

The upside of fixing a path is 2-fold. First, it allows us to give supervision on every part of the flow without needing to simulate the entire ODE. The intermediary targets are known and can be used as supervision signals. Second, it allows us to shape the path based on properties we want. For instance, straight-line paths tend to be easier to sample from than highly nonlinear ones. As such, flow matching allows all the flexibility of normalizing flows while having the superior sampling properties of diffusion models.

Yeah, can we make this a little bit more concrete? For example, I interviewed John Jumper, and he was talking about how he used diffusion in the third version of AlphaFold, AlphaFold 3. He was talking about some really interesting properties because it's very much an iterative process. He was saying that we were running it through this diffusion model, and the big structure kind of converges first, and then it learns the details afterward.

Many folks would have used Stable Diffusion models, for example. You can actually see these images materializing over time. But be a little bit more concrete: where are these being used? How can people use them?

I think this is pretty hard to say because, ultimately, you would expect something like this for a general model. The reason is that while the latents themselves don't hold any information—a latent is completely unobserved—they don't hold fundamental information. If you want them to hold fundamental information, you need to go to the Bayesian people who do the parameterization of the coin flipping, which I had at the beginning. The actual latents don't hold any information.

But of course, in practice, if you enforce the fact that the change from latent number 1 to latent number 2 should be really small, then you would expect something like this, where the big structure appears first. If you don't have the big structures, the small changes later on cannot reconstruct them.

I don't really have a good argument for why it has to be like this, and I don't even think it has to be like this. I think this is more a notion of what happens to be easy for these models. But I would imagine that most sane models would structure it like this.

This doesn't mean that you couldn't build a diffusion model that doesn't do that. I could imagine enforcing a constraint that says, "You have to do the small things first." I don't think it would be a good model. You might need thousands of times more parameters to actually fit this. You still would be able to train this, at least I think so. Mathematically, there's no reason why it shouldn't work. Practically, there are lots of reasons why it shouldn't work.

That's actually really interesting because we see a similar thing with standard neural networks. You can look at the types of representations they learn during the training process in vision models. They learn low-frequency features early on, and then they learn higher- and higher-frequency features. You're saying it doesn't have to be a certain way; maybe it just depends on the architecture, the data, the domain, and things like that.

But it is quite interesting with these types of models that you can basically do an arbitrary amount of computation. In Stable Diffusion, you can do 25 diffusion steps, or you could do 150 diffusion steps.

And if you overcook it, the image starts to look weird because you get little microdetails on the skin, and it doesn't look very good anymore. So how should practitioners reason about this?

I think reasoning about architecture is really, really difficult. The more I think about architectures at scale—let's say, these billion-parameter models—the less sure I am that it matters as much as people think. I think it matters in the sense that you need to make sure your input data actually fits into the model. You couldn't do an MLP that does the same type of large-language-model things simply because it has a finite input space.

However, I don't think that there is a fundamental limitation on one of these architectures if you look at it at huge scales. At huge scales, all of these perform a lot less like parameterized models of a specific architecture and more like function spaces. Effectively, the actual change of the individual parameters does not really mean anything anymore; what matters more is the bulk change of the entire field around this. This is effectively the mean-field-theory approach to deep-learning analysis, where it doesn't make sense anymore to look at individual particles, as in physics.

You don't model a stream of billions, cajillions of hydrogen molecules inside your water stream or in a physics simulation. You model the actual bulk properties with Navier–Stokes. I think the same thing is true with large language models—or, let's say, any large model. That makes it really, really difficult to make insightful statements about what architecture leads to what, because I don't think it ultimately matters that much. I think what matters much more is the scale.

Of course, it might be that one architecture scales more nicely. One architecture might need a billion parameters, while another one needs a hundred billion parameters, just because the other architecture scales less nicely. That's probably the reason why we choose deep neural networks and not just a shallow one which is really, really, really wide. But I don't think that in the limit—and I don't mean the infinite limit, but the large-scale limit—it makes a difference.

I think it's much more about how you would expect the information to proceed through these models, especially in diffusion. In diffusion models, we have this structure where they are doing—how is it called?—KL divergence minimization in a Wasserstein space. So you would expect higher-information things to appear first because they are the free wins. I don't think it has to be like that, but I would expect it to be like that.

The same thing applies to rectified flow. In rectified flow, I don't think it should be that surprising that if you basically do a linear interpolation, big things appear first. But I'm not sure if I could advise people, in good conscience, and say, “You should do this and that for your architecture,” because I don't think it matters.

Very cool. So we should move on to theories of deep learning. Much to our chagrin, we've tried many times. I mean, God knows we've interviewed many people who have said that they've got a theory of deep learning. But you've said to me that, roughly, this can be broken down into a parametric view and a functional view. Explain.

7. Theories Of Deep Learning

If we look at deep neural networks, there is, of course, a very obvious first-instinct way of saying, “Well, this is just a huge nonlinear program.” You basically look at this as you would in the way optimization people do. The optimization people say, “Well, I have a huge polynomial or, in this case, a huge parameterized neural network.” Then we ask: What are the properties? What would need to happen for this to converge to something? What do we need to do to figure out a global optimum? What do we need to do to effectively model the change in parameters over time? This is ultimately a really, really interesting way of thinking about the problem, but it's not necessarily that useful.

There are, of course, counterexamples—things where we actually look at the parameterized nature of these models and can make some bigger inferences. That's effectively where the optimization people look into deep learning, especially if you look at spline theory in deep learning. If you look at what the mathematicians are doing, all the combinatorics people say, “That looks like something I can do.” Ultimately, there are really, really interesting things there as well, but I don't think they give you that much insight.

They can give you statements like, “Okay, where is the actual complexity of a learned model?” Basically, how many of these linear regions exist? They can also sometimes get some really nice explanations. I really like the paper by, I think by Lassorio, who did the explanation of why batch normalization works. He framed it as, well, batch normalization actually concentrates things in the middle, and therefore all the hyperplanes are more concentrated, and they're—effectively, it's more linear, more nonlinear.

But I don't think it's ultimately that insightful because, at some point, again, when you're looking at bulk properties, it doesn't matter anymore what an individual parameter does. Then we very quickly go into this domain of saying, “Well, if parameters don't matter, can we get rid of parameters in our analysis?” That lets us move into something like neural tangent kernel theory. It allows for some randomized-sketching-type results, and there is also mean-field theory, which I really like from an analysis point of view.

I'm not sure whether it's ultimately that useful, but I really like it as a mental model. The way they phrase it is: If we look at, let's say, a one-hidden-layer MLP—very simple—we can view this as effectively every input neuron being an inner product with an activation around it. We can view this as, let's say, 1,000 test functions. The activation function is, let's say, a sigmoid, to make it very, very instructive between 0 and 1, and it figures out whether this test, or this weight—this inner product—is successful. Then the next layer is a combination of all of these.

What we can do is phrase it as: What happens if, instead of having 50 input neurons, we have an infinite number? If you put this into the infinite limit, you effectively write this down as a density, because every single point exists; it just matters how present it actually is, how relevant it becomes. Now we have this structure of effectively fitting a density over weights and putting a density over these weights.

That is very nice to analyze because now we are back in literally a diffusion-style problem. We can model the change in density over time as a change in distribution over time, which is ultimately a diffusion-style problem. Neural tangent kernels are a very similar idea. Instead of looking at this from a density point of view, they simplify it a lot more and say, “Let's actually make it a function space.” The function space then spans the area.

There are some limitations with this analysis. For instance, neural tangent kernel theory assumes the kernel is stationary. But even that is an assumption that we make from an analysis point of view. There is nothing that fundamentally hinders us from looking at the kernel itself changing. Effectively, we have 2 dynamics: the kernel-change dynamics, and the kernel itself describing the local function.

This is also what mean-field theory does. In the back of their big paper, they have a section where they talk about the connection to neural tangent kernel theory: We just assume the kernel also changes over time. That makes the analysis way, way better, because you can now effectively model not only lazy learning but also the feature-learning domain. The limitation is that it becomes much, much harder to analyze.

Still, I think that from an analysis point of view, these end up being more useful, specifically because the predictions end up lining up a lot better than what we would expect from the parameterized world. In the parameterized world, we say, “An infinite, or a huge, number of parameters means this thing is going to overfit instantly. This thing is completely overparameterized. Why do you do this?” From a function point of view, you say, “Well, no, it's not overparameterized. I'm just going closer and closer and closer to the actual true function space I'm fitting to.”

A function which has, say, a billion parameters is much, much closer than a function with 50 parameters to being in this infinite-scaling domain.

So, effectively, functional analysis really explains this idea of overparameterization, while the parametric kind of view says, “Actually, this overparameterization might be bad, but we can exactly explain what the local architecture is.” However, I would imagine that, in the long term, the functional analysis people are going to be more useful just because they explain not a specific architecture, but architectures in general. A neural tangent kernel works for pretty much anything, right? Not every proof works for literally anything, but the idea of looking at this from a function-space point of view works even if, let’s say, we all go back and do MLP-Mixers now. We no longer do transformers; we all do MLP-Mixers. It still would work from a theory point of view, right?

Because we no longer use ReLU. In fact, we use gated activations. Lots of research in the parametric world no longer really makes sense because they all assume this spline-like structure. Instead, the function-space theorists say, “Well, the function space looks a little bit different.” It’s still ultimately something like a Barron space, and then it’s the analysis that works.

Yeah, I suppose I’m just curious to know: what would a good theory of deep learning resemble? Maybe it’s something that gives us a good explanation. With this functional analysis, the kinds of questions we might ask are, “Is this network overparameterized?” Or we might want to ask, “Is this architecture optimal?” It feels a bit like the blind men and the elephant.

We’ve got folks like Randall Balestriero coming at it from a geometric point of view. There are a lot of people doing that. We’ve also got our good mutual friend, Keith Duggar [?]. He’s coming at it from a computability point of view, and he’s talking about whether they are Turing machines. He’s talking about the space of algorithms that they could possibly represent.

Then there are complexity views. There are folks looking at statistical physics, talking about spin glasses and heavy tails. You see what I mean? Even effective-theory folks and complexity-theory folks are offering different perspectives on the blind elephant. Which one is best, and why?

Ultimately, I think that the most important thing is that it makes the correct explanations. It comes up with the correct solutions. I don’t care that much about the specific structures. Again, I don’t want to sound too negative about the parametric point of view. If it makes good predictions, then I’m pretty fine with this. I just don’t think it is as tuned to being productive.

The important thing, and I think it’s also missing from many of these theory papers, is that you actually check whether the test works. You try to falsify your theory or figure out where this theory actually breaks. In this case, I would think something physics-y is a lot more useful, just because of this loop: I make a prediction, then I do an experiment. I think this is much more useful.

I think the big thing is that theory will never be able to explain literally anything, because it might be able to explain things from a purely functional point of view. Let’s say the mean-field people can completely explain that, if I have a density composed of, let’s say, a Gaussian mixture model, then I can completely tell you, “Hey, you need this size,” or, “At this size, the error to the infinite model will be this size.” Completely fine.

The problem, of course—or the same thing with the SVM people, the kernel-based people—is that they have source and capacity conditions, which effectively ask how much your function has and how much your data has. The problem is that, in practice, we will never have that information for our real data. In practice, maybe we have a clue, right? I think we can often estimate. For instance, we can often estimate how many clusters there could be in our data with relatively simple methods.

But we can’t really have very deep information about our structures because ultimately it’s still the real world, and we still approximate everything as spherical cows. The physics-y side would then say, “Well, we try to do a best effort, and we try to solve everything that is solvable or predictable. Based on that, we can still make real predictions for the real system,” right? That is effectively the transition from physics over to engineering, and where the distinction lies doesn’t ultimately matter, even if you can actually make a hard cut.

I think the important thing is that we need to have something that makes accurate predictions. For that, the loop between the theory people, who do the really nice things like mean field and neural tangent kernels, and also stuff like diffusion, is important. Diffusion is also a nonparametric explanation of what these models do. You just say, “Well, they fit some density.” Reinforcement learning, what I do, is also fit some density and be happy.

The actual loop should be to take this and try to make inferences about the real world. I think this is where there should be a deep theory of deep learning that people look at. We should now go and say, “Exactly that. We need to explain this.” The problem is that right now it’s really, really difficult because we have so little theory that the reason something could not work, or did not work, could be just an implementation error, because we don’t have a good enough idea of what we do yet.

I previously mentioned that I wasted, let’s say, 2 or 3 months with an implementation bug in one of my papers. The fact that we cannot figure out whether there’s an implementation bug in a continuation of existing work—that alone should mean that the theory is not yet at the level, and this is what we should be looking at.

Yeah.

I don’t think there’s really a path right now that gives us that amount of progress. Not the nonparametric people, not the parametric people, not the mechanistic interpretability people. No one can tell us.

Yeah, I guess we live in really strange times, and so we don’t even know what to believe anymore because we had all of these things that we used to say confidently a few years ago. We’ve got things like the double-descent phenomenon. We’ve got the mystery of overcoming the curse of dimensionality.

From a sort of linguistics point of view, people talk about systematicity and productivity. Why are the models syntactically competent? Why can they produce all of these sentences that have never been uttered before? Why are they learning these abstract models? Are they too incomplete? All of these different things. We need to have a theory that kind of explains this phenomenon.

I think there’s also a big problem: a lot of the research is really disjointed. It’s not like in physics, where we have the Standard Model of physics and we work around this. If you go to school and learn the Standard Model, you can then basically start with a common starting-off point. A common example, for instance, is that people say, “Well, why does deep learning beat the curse of dimensionality?”

I can tell you. Barron literally proved in Barron’s theorem that the actual convergence is not dimension-dependent. It’s a dimension-free bound. It depends on the frequencies of the target function you’re fitting. Intuitively, once you think about this, it actually also makes sense. If you have a high-dimensional function which is constant almost everywhere, then it’s much, much easier to fit than a 2-dimensional function which looks like a heartbeat monitor.

In fact, I think the same thing is true for most other estimators as well. If you think about a decision tree, it has the same property. If the high-dimensional function is constant in all dimensions but 1, then it doesn’t care about all the other dimensions.

You would need to have something like a standard work that actually looks at this and combines all of it. Something that doesn’t figure out the next paper, but something that actually goes and writes down and tries to figure out how all of these things fit into 1 coherent narrative. I’m not sure whether machine learning as it’s practiced right now is actually equipped to do that, because right now the pace is so fast. Effectively, you have 3 big conferences every single year. If you don’t count AAAI, you have 3; if you count that, you have 4.

Effectively, you go from conference to conference to conference to conference, and all of these papers—I don't know, maybe it's just me. Maybe I'm just unlucky. But if I look at the papers I'm reviewing, they're really incremental, all of them.

I don't fault all of the people there for publishing like this because, first of all, it gets rewarded. Secondly, it doesn't make sense to say, “Well, I'm setting up a big deep learning theory. I write this book over my PhD. I don't do cumulative work; I actually write a manuscript. In the end, I have, say, my 5,000-page book where I categorize everything there is to know.” By the time you're done, it's going to be completely out of date and useless. Probably lots of things you write in there are completely wrong.

So I think that right now there is no way to have something like physics. Of course, we have a theory of deep learning. I talked about this paper like this. There will be a theory of deep learning. It's kind of a vacuous statement because we arguably already have plenty of them. The problem is more that, to have something like physics, I don't think the entire field is set up to do that, and I think that it will not be set up to do that until the point where things start slowing down.

Can we just have a quick digression on the manifold hypothesis? Our mutual friend Randall Balestriero isn't a fan of the manifold hypothesis. He's got this spline theory, which is basically that, in an input-sensitive way, the ambient space gets partitioned into these little honeycomb regions. You were helping me understand this at the time. You get this weird teleporting thing happening where, if you change the input a little bit, you'll teleport into another honeycomb somewhere else in the space, and it's kind of anti-manifold hypothesis.

But I spoke with Goodfire last week. They're an interpretability company in San Francisco, and they've got all of this work about the geometry that these models learn. For example, with days of the week and months of the year, they can take a PCA projection of the activations, and then they fit this kind of Bézier curve to it. They see these beautiful geometric structures in the activation space, and they're really manifold-pilled. What do you think about that?

8. The Manifold Hypothesis

I think it's really hard to make such a statement—whether it exists or not. Something like the manifold hypothesis: if it's not a low-dimensional manifold itself, something like this has to exist so we can actually compress things down. Ultimately, I don't even think that Randall would disagree with me on that. If we actually have, let's say, the world outside—the real world is the smallest thing there is, and there's no simplification—then what are we doing?

But it would also push back against all of these explainability techniques. Generally speaking, they're not very robust. Firstly, they often make extremely simplifying assumptions, like the circuit people and the Transformer Circuits. They usually make the assumption that the actual routing is constant as you change or permute your inputs. That's not true. You effectively model a transformer like a huge MLP-Mixer.

I think there are many silent assumptions, and I think there's even a paper where they show that, if you make some slight tweaks to the setup of these experiments, the results completely flip. All of these approaches aren't robust enough to make statements that the manifold hypothesis exists, but they're also not robust enough to say that it doesn't exist.

I think it's really, really difficult, and I think it's also difficult because I don't know whether the language exists to realistically talk about these problems. There are, of course, people who go very much into abstract or functional space. I don't think this is completely true. It's often very useful, but it's not completely true because we don't have infinite-dimensional functions, right? They're finite-dimensional, and there's a big difference between finite and infinite. If you have infinite dimensions, every problem becomes convex. In fact, every problem becomes linear.

On the other side, you have the parametric people who say, “Well, we just have these spline regions, and that explains everything.” But if it was like this, then we would need to be able to make predictions that say, “Well, if we go away from the splines, maybe that doesn't work anymore.” Or, to what extent does it stop working?

Then there are the explainability people who say, “Well, we can actually extract this or that circuit, or we can extract this or that representation.” Maybe one of the best papers I saw in explainability was the paper about grokking, where they extracted the modular arithmetic code. Look at this—it completely makes sense that there are sines and cosines everywhere, especially because they do it in the input. But if this is how these models actually describe modular arithmetic, there's not a chance we will explain how they model a duck.

I don't think there is currently a language for describing what these models do. For something like the manifold hypothesis, I don't think it's true, and I don't think it's wrong. I don't think it's a useful way—even if it's true—of thinking about the problem. The same thing applies to the extracted grokking-style representations: even if they're true, I don't think they're a useful way of figuring out what these models do, how they get the solution, and whether things work or don't work.

I don't know. It's such a tantalizing idea. There's this idea of a homunculus—that we've got a little man inside our brain—or these grid and place cells. Basically, we've got a little world inside our brain that captures some kind of geometric invariant, some kind of model of the world outside.

It's a beautiful idea, isn't it? When you visualize these representations—for example, geography—you actually have some kind of geometric invariant of how the different countries in the world are related to each other. It's fascinating that this would be convergent: that different model families would learn similar types of geometry that represent abstract structural invariances of the world. It's very, very exciting.

But, yeah, I agree. Maybe we're reading too much into that. We should move on to “Reward Is Enough” and constrained RL. Rich Sutton, the wonderful Rich Sutton, had this paper out called “Reward Is Enough.” What did he mean by that?

9. Reward Is Enough

The idea, especially among reinforcement learning maximalists, is that they say, “Well, technically, I can write down any objective I want as some form of reward.” You can literally just say, “Well, you should be moving the arm like this.” You can write this down as a reward. Basically, it's 1 everywhere it is. Or, if we move to a goal, it's 1 where the goal is and 0 otherwise.

This is a very true statement. If I have an optimization problem—let's say I have a really bad one, like a mixed-integer nonlinear program with millions of parameters and huge numbers of constraints—I can still write down the solution really easily. I have my X-star, my actual solution. I just write it down as minimizing the error between X and X-star. It's a convex problem. It instantly works.

If it's relatively low-dimensional, you can solve it by hand. Even if it's really, really high-dimensional, and the actual problem behind it is really, really high-dimensional and complex, I can still do that. In that sense, the idea of reward being enough is completely true. Technically, you can define a reward function for pretty much anything, and we'll somehow figure it out.

The question is more whether this is a realistic assumption to make—whether having a God-given reward function that just does something is realistic, whether it's even possible to find it, and, once we have it, whether it's realistically optimizable with any technique. That is where I think the practicality, or the reasonableness, of this breaks down.

Just help me understand: is Sutton advocating for the God reward function, or is he more practical? Is he saying that there are reward functions in different applications or whatever?

Of course, the God function was a very extreme example. Ultimately, what he believes is that you can figure out a suitable reward function for every task, and that is enough. We don't need to research all of these other complicated things. Instead, if we just become really, really good at maximizing reward, then everything else becomes obsolete.

I mean, I think it’s a very, very optimistic view of the world. I’m not sure how far he himself would be going with this, so I don’t want to put words in his mouth. But I don’t think it’s a reasonable thing to do even in the kind of infinite limit.

I mean, give us a bit more context here. When I was talking with Chris Watkins the other day, he was saying that he met Rich at some conference in the late 1980s or whatever, and there were people in his lab at Cambridge doing all sorts of control theory. For example, there’s this cart-pole balancing problem, and you can just solve that analytically. But this reward thing was different because, in this particular case, I think it was a binary signal, so it’s either balanced or it’s not, and it’s a signal from the future. So you have to do a lot of stuff before that thing says yes.

At the time, they thought this was an interesting problem to solve because maybe it’s a better representation of animal learning.

Yeah. I think, in general, there’s a big overlap between control theory and what reinforcement learning people do, but there’s also a huge distinction. Notably, the main difference between doing actual control and doing reinforcement learning is that control assumes that you have—essentially, you have a model of the plant, or at the very least, you have some idea of the behavior, like the actual response to an input. That is not true in reinforcement learning. In reinforcement learning, you say, “Well, I don’t have a model at all. I just have interaction with the environment.” And that immediately makes the problem more complicated.

First of all, it makes it more interesting, but it also makes it a lot more complicated because now you no longer are trying to solve the actual problem at hand. You also solve the problem of figuring out what to do in the first place, right? A controller—let’s say we have a reference which it tries to follow. We have some model of the plant dynamics. Let’s make it very extreme: we have an MPC controller, or we have whatever else, or a linear-quadratic regulator. We have some idea of what this thing does; or if we have a PID, we have a human who has an idea of what it should do and how it should behave. Then effectively you outsource all the exploration.

While in reinforcement learning, you say, “Well, I put this into the loop itself,” and that gives you a huge amount of power because now you can actually solve much more general problems. You can solve problems that we don’t even have an idea how to solve in the first place. And this is also maybe where the reward is enough. People then go in and say, “Well, if we can solve an arbitrary problem with an arbitrary definition, and we can do the exploration, the learning, and the optimal control, why do we need anything else?”

Again, while I think this is true in the limit, I also think it’s a very, very wasteful way of thinking about the world, especially since things like exploration become really dicey in many domains. I don’t want my open-heart-surgery robot to do exploration—first of all, not on me, but not on anyone else either.

I think that the actual real-world complexity of needing either not to allow too much exploration in the practical sense, or to give really, really hard guarantees, becomes difficult to handle in pure reinforcement learning. The classic example where we can actually prove that it doesn’t work for deep learning is that we cannot give L-infinity control guarantees. Because everything is sampling, sampling and absolute maximal errors don’t really play well with one another.

You can have a controller which, in expectation, is really, really nice but has one really, really huge failure case, right? On average, something which tracks it perfectly and has one huge error where the robot clocks you on the head is completely the same as one that just slightly wanders around.

In fact, there’s this nice paper about—what was it called?—“training deep neural networks to uniform accuracy is intractable,” or something like this. Put it into Google, and you will find it. They basically show that, for neural networks, it’s not even possible. Or, basically, it is possible, but it’s exponential in the number of parameters.

Because basically what they say is, when we have the ReLUs, if you have two, they effectively form a hat, something like this. Just figuring out how to place and how to scale these hats is already an NP-complete problem. Or rather, you need an exponential amount of data to figure out these deviations from your reference.

And I think having something like this is necessary for many domains where damage is really, really, really bad. Having just a chance, in expectation, that damage and good things happen most of the time is often not good enough. You want stronger things, where then suddenly you go into things like robust reinforcement learning or something like this.

Or what I do for my PhD: constrained reinforcement learning, where effectively you do a little bit more just to get this little bit of an extra boost in either safety or performance, or by just including prior data, as in imitation learning, where we just say, “Well, I actually have lots of data around, and I don’t want to do open-heart surgery from scratch.” So I think it’s very naive to think about this the right way.

Yeah, because I suppose this is the other vexed issue with our friend Rich Sutton: he has this “The Bitter Lesson” essay, which is basically that he doesn’t think we should be building knowledge and constraints into the system. He thinks we should just scale models of computation.

And I’m a big believer that constraints are incredibly valuable, and by constraints, more broadly, I’m just talking about knowledge. I think constraints happen at multiple levels of abstraction. So in the physical world, constraints might just be the laws of physics or whatever, and in our minds, constraints might be things that we already know.

Chris was saying to me the other day that one of the amazing things about RLVR is that it doesn’t solve, but it helps with, the exploration problem because the base LLM knows so much, so it has lots of conceptual understanding. Rather than just blindly searching all over the place, because obviously back in the days of deep Q-networks and whatnot, we actually needed to guide and put a lot of priors into the exploration process; otherwise, not very many interesting things would happen.

And now, of course, with RLVR on language models, they have a lot of base knowledge. But it feels to me that how you get there and how you constrain and shape the search space is critical to making it work.

Yeah. I, I have the nice property that I'm a very theoretical person, but I also work at, like, Fraunhofer, which is very applied. Uh, so I, I actually did some project which I actually applied reinforcement learning. And this is a big problem in reality, that effectively information can be very expensive, and in fact, I would say most information is very, very expensive. And reinforcement, if you completely say, "Well, constraints don't matter. Uh, I, I don't need any prior information. I don't want to encode anything in there," uh, assuming this is even possible, uh, the c- being completely, like, unconstrain- unconstrained et cetera, and completely, like, without prior knowledge, let's assume all of this is the case. We, we still have the problem of now needing to effectively relearn everything from scratch, and the reality is that we want to be gathering as much information that we already have paid for and gathered with our blood, sweat and tears. We want to put all of that in the model first, and then we go and we continue from there. There is... There's no point in basically starting from nowhere. There's also the reality that if you actually look at a reward function like a, an, an actual, like, industry-ready reward function, you will im- immediately see that they have constraints in there. Like, it's... They, they just, they just reframe them as, "Well, it's a reward function," because yeah, of course, like, you can put them into it as penalties. But ultimately they are just kind of... You're just doing a lot of work to not write down a constraint. The, the nice example of this, which, which I always like to use as a, uh, as like a motivation for constrained RL, is if you look, if you look into the paper about, uh, what is it called? OpenAI Five, which is the, like, Dota playing AI. If you look into the appendix, there's a huge table where effectively on, on the left they have all these different attributes which they put into the reward function. On the right they have some, like, explanations and so on. In the middle, they have just a bunch of random numbers. And these are... They have the weights for their reward function. Like, the, the... If you tell me that there is a way to actually figure this out, there, there's not a chance. So especially if you look actually at what these reward functions are. Like at the bottom for instance there's one which says, well, the H- like the, there's a negative reward for being in the wrong lane, uh, for a specific amount of time. That’s a constraint, because writing it down is exactly the same as saying, “Well, don’t be in there for more than 5 seconds,” or whatever. Or rather, in most constrained reinforcement learning, you say, “Well, in expectation, if you take 1,000 trajectories, don’t be in the wrong lane too often,” or something like this.

Another nice example is that we can always run into trouble with reinforcement learning. Above, there’s a section where killing an enemy hero actually gives a negative reward. The reason, as they explain next to it, is that killing an enemy hero gives you a lot of gold, and there’s already a bonus for getting gold. So if you also had a positive reward for killing an enemy hero, you would just sit in the middle of the map and kill the enemy hero over and over again without making any progress.

The problem there is that 2 of these quantities can be completely correlated. If you model constraints, you basically get rid of this correlation problem because, for example, you can write down the constraint explicitly: “I should not have—there should not be, in expectation, more than…” How was it? “I should not have, let’s say, 50% fewer enemy hero kills than my opponent in expectation.” It’s a small guarantee, but effectively, the thing that you want to get rid of is already removed by that.

Yeah. Leonardo da Vinci famously said that constraints give us freedom.

I’m a huge fan of that. But you specialize in this mode of reinforcement learning called constrained reinforcement learning. Can you explain that?

10. Constrained Reinforcement Learning

Constrained reinforcement learning is still a relatively up-and-coming field, I would say, at the very least, especially since there are many different formulations. The classical one is the constrained Markov decision process. Effectively, you have exactly the same setup: you maximize the cumulative rewards, but then you have constraints that say the expected cumulative costs should be below a threshold.

Ultimately, you have very similar structures between the reward and the constraints, but there are also other ways of framing this. One big disadvantage of this expected cost is, for instance, that you don’t get rid of long tails. If you say it should be less than 25 in cost, then it could be that you have 50 trajectories with a cost of 0 and 50 with a cost of 50, so it averages out to 25. That’s not necessarily what you always want.

On the other hand, for something like the OpenAI Five case, it would be exactly what you want. You want to have some games where everyone is in one lane, and then some games where it’s the normally distributed amount. If you want to go further, there are also solutions that use Conditional Value at Risk, where you effectively look at the probability of being in a tail and weight this by how far out on the tail you are.

There are percentile risks, such as: with 99% probability, you should be within, let’s say, 50% of some target—basically, PAC-like guarantees. All of these have different domains. I recently wrote a paper, which I have to rewrite and resubmit, about doing this if you actually know the constraints completely. You have a complete model, as in an operations research setting, but you don’t know the objective.

Let’s say you have a complete model of your local gas network or your local district-heating network, but you don’t actually know the exact costs of solar versus whatever, or the actual demands. Then you train, or try to train, based on that. In that case, you’re never allowed to have any failure. The pipes should not burst—not with 99% probability; they should simply not burst. You try to encode all of these guarantees into the training process itself.

There are different ways of doing that. One of the very earliest approaches looked at TRPO and said, “Well, we have this performance-improvement bound. We can do exactly the same thing for costs.” This gave us CPO, where effectively it says, “Well, if we are within the constraints, then with 99% probability, after taking that step, we’re still within them.”

The problem with that, of course, is what happens if we’re not in the constraint set? Then it effectively has to do greedy descent. It replaces the reward with the cost function and descends back down. You basically see oscillations on the boundary, something like that.

Further ideas asked, “Well, what if we do projection?” We effectively have an approximation of where we expect the cost surface to be, and we project back onto it. The nice property of constrained reinforcement learning is that it’s actually completely convex. Again, this is because we assume we have densities. We don’t have parameters; we have densities. In density space, everything becomes linear and convex and nice, so projection actually makes sense.

The downside is that you can have some really, really bad runs. The reason is: what if your effectively constrained system is hallucinated? What if you hallucinate a constraint somewhere? With something like my recent paper on Safe-MPO, you hallucinate a barrier, and suddenly your constraint model never walks in there because it’s very careful to project around it.

In fact, this is a problem that becomes worse the better your solver becomes. It’s this hallucination problem. Then there are the people who say, “Well, we have zero duality gap. Why not just use Lagrangian approaches?” This is effectively a min-max style approach.

That works, but then you also have the problem of oscillation. You go down and oscillate around because you’re below the cost—the Lagrangian was slightly too large. Then you’re very safe, so the Lagrangian drops, and you go above, then below, and so on. You have these large or small oscillations.

There are methods that try to get around this. There’s something that uses PID-style tuning, which is pretty interesting, but ultimately you still run into exactly the same issues: the oscillation errors are still there. That’s fundamentally because estimating Lagrangians from data—estimating constraints from data—is really difficult, because you have all this variance.

Then there’s a final family, which also includes my recent Safe-MPO paper. There’s CBPO, and what’s the other one? FOCOPS is another one. These methods basically try to use local approximations and solve the problem locally optimally, and then train the model to emulate this locally optimal solution.

That has some nice properties. It can be very, very precise, but now you need to be really, really careful not to maneuver yourself into an effectively unsolvable setting where your subproblem is unsolvable because you had a bad batch.

Of course, that’s model-based, but model-based is easy to explain. You effectively just do model-predictive control, but with constraints. It’s really hard to do that properly because you could be in a setting where almost every proposal you make is unconstrained, or outside the constraints. Ultimately, though, the algorithms are very similar.

Yeah. I suppose the million-dollar question is: given that you said before that we already have constraints in normal reinforcement learning, what’s the cash value of explicitly modeling the constraint?

The big advantage is that you first gain the ability to model things more accurately, especially if you have experts. You can go outside and talk to an engineer, and it will be much, much easier to talk to him about constraints than about tuning a specific weight of a reward function.

Constraints can often be known ahead of time with high certainty. For example, with a robot, you know that a certain load will break the robot. You know that even without trying it out. If you try to do this by tuning things, there’s a very high likelihood that you’ll accidentally have something go above the threshold. It’s often really, really difficult to think about these coefficients.

The second big thing is that you get composability. One of the problems I talked about with the OpenAI example was that if 2 elements of your reward function are correlated and, for example, you add something to it—let’s say that at first you just had a bonus for killing an enemy, and you did not have a gold bonus at first.

And now you add the gold bonus later because, while doing the engineering, you found that it would be a good idea as an additional guide. Suddenly, your entire system breaks, and you need to retune all the weights because the weights are tied relative to each other. If you model it as a constraint—let's say, “You should have more gold than your opponent in expectation”—that's a reasonable idea. Then you would not need to retune because, effectively, you still have these coefficients in the Lagrangian setting, but they get automatically tuned to enforce the actual target that you want, not the kind of target you implicitly encode with these tiny weights.

Very cool. I'm really interested in constraints. Do you know Jeremy Budd? He's also in Yannic's Discord server, and we wrote this article called “Why Creativity Cannot Be Interpolated.” Basically, the idea is that creativity is all about deeply respecting constraints and that representations are path-dependent in some way. You might have read that “Fractured and Entangled Representations” paper from Kenneth Stanley.

It's kind of interesting because neural networks seem to have soft constraints. They learn this statistical distribution, and then you sample from them and they kind of respect the constraints, but they don't have hard constraints. I visualize problem spaces a bit like a maze. A maze has walls in it, and the walls are the constraints. If you respect the constraints, then you don't walk through the walls, and you can be creative because you can take paths and trajectories through the maze and find new, undiscovered subspaces, if you like. You're not constantly wasting time walking through the walls and failing to respect the path that got you there.

I have this deep intuition that maybe it's neurosymbolic, or maybe it's some way of hardening the constraints that we learn within particular domains. It's unclear to me whether neural networks could, in principle, have these hard constraints, or whether we would always need to enforce them after the fact.

That's a really interesting question. Generally speaking, if you can encode your constraints into your architecture, you're probably going to be a lot better off. A classic example in reinforcement learning is that you do action masking, or you scale your actions to be in a reasonable range.

The question is usually whether there are things outside the model that actually inform the constraints. For example, in reinforcement learning, having action constraints is pretty simple: effectively, you just project onto some set. The difficult thing is having state constraints or something in the environment, because now you can no longer enforce these simply by modeling them in your architecture. You have to be able to interact with the state and the architecture together. There are sometimes ways of encoding this into your model, but whether this is generally possible, I would honestly doubt it, especially since there are plenty of constraints that are NP-hard to enforce.

Let's say we have a CSP-style problem that is NP-hard to enforce. How are you going to do that in a static architecture? Mathematically, it already breaks from a complexity point of view. Of course, you can then do things like a diffusion-esque approach or a Monte Carlo-esque approach. But fundamentally, you still need to externalize some structure if it's not the constraints and that's the way you actually generate the output.

I do think there is something to this idea that constraints can help with sensible exploration by basically removing irrelevant things from the state space. The complexity here is actually figuring out how to compare unconstrained versus constrained settings. Fundamentally, if you look at the literature, there is the paper by Altman, who did his PhD on constrained reinforcement learning, about effectively mixed-integer programming or LP-style methods.

One of the things that was known even back then is that constrained reinforcement learning is a lot harder, specifically because there isn't necessarily a stationary deterministic policy anymore. I have to be careful with this: there are non-stationary deterministic policies, and there are stationary stochastic policies. Basically, this means that doing something like Q-learning does not generally give you a solution, because you can get into some really iffy situations at the boundaries.

Technically, if you look at this and ask them, they would say, “Mathematically, it's a lot harder.” But of course, this is a wrong comparison because you're comparing a constrained problem with an unconstrained problem. A constrained problem is, of course, harder because it needs to do more things.

The actual comparison would be something like: I have an unconstrained problem and a constrained problem, both of which are potentially really difficult, and then I compare them on the same task. That becomes difficult because just designing a reward function that implicitly informs a specific constraint is difficult. Maybe we would need to do it the other way around: you would need to have the reinforcement learning agent figure out an optimal solution, and then, based on the violations and so on, define the constraints.

But then, of course, you start the constrained reinforcement learning method on the back foot because it basically needs to emulate the unconstrained model. So I think it's really hard to compare whether it makes an actual difference. It might just be that, for the solutions we see, it makes a difference, but it doesn't make a difference in the absolute sense. I can't think of an easy example—or an easy experiment—that does not fundamentally disadvantage either method.

I often think about when constraints can be harmful. I often use this expression that I got from Mahvita Chiramuta. She said that reality is protean. What she meant by that is there's an ancient myth in Greek mythology about this guy, Proteus. You could pin him down, and he'd have to answer your question, but when you let go, he would shapeshift.

There's this idea that problems in the real world shapeshift when we interact with them, and they're non-stationary. What we try to do as mathematicians and scientists, though, is learn knowledge and representations at an increasingly high level of abstraction. The purpose of that is for the knowledge to be less shape-shifty. It feels like we almost need to have a spectrum of representations where we admit fewer constraints because we allow for the fact that lower-level representations change as we interact with them. But maybe we privilege higher-level abstractions that are more known and invariant.

I think there's generally something to that. Once you have abstraction, it becomes really difficult. The fundamental constraint that you want for a hierarchical, abstracted system is that the lower systems guarantee some invariant that the upper systems can actually guarantee.

This is, in my opinion, one of the main reasons why hierarchical reinforcement learning never really took off: the lower level does not give you enough guarantees for the higher level to exploit. In control, you can do that. Control gives you very strong guarantees. In control, you can have an H-infinity controller, which gives you effectively worst-case, adversarial robustness, and you can still work with this.

The same thing cannot be done with reinforcement learning because the lower level just does whatever it does, and the upper level just needs to deal with it. I think there's something to the idea of, in an abstraction hierarchy, making a statement that the lower level has some guarantees. I don't fix what the lower level even does; I don't care. It just has to uphold a certain thing, and that's a constraint that I can rely upon when I build my abstraction on top.

We see this to some extent already, even in deep learning. Many people invoke the universal approximation theorem. Very few people have actually read what the universal approximation theorem does, but many people invoke it because they basically say, “Well, the invariant is that if I have a large enough model, it's just going to do the right thing.” Even if that's not technically what the universal approximation theorem says, you still need to have a stable interface in an abstraction.

What the lower-level things do, you don't care, but you need to be able to rely on them. This could also be seen as a constraint, right? The constraint comes from the prior methods and potentially also from the upper level. Generally speaking, you have a level below you that has a constraint in the form of capability, and you have a level above you that is a constraint in the sense of need.

A robot needs to not crush you.

Where do our constraints come from? How do we find these incredibly distilled, evolved, abstract representations?

I think the main difference between a good and a bad abstraction is, again, its usefulness, right? There are plenty of abstractions that never actually took off. We only see the good abstractions because the good abstractions are the ones that are actually used. There’s a long tail of abstractions that were started and then forgotten about very quickly.

Ultimately, what would be an example, so as not to piss off too many people? Fuzzy logic. It may piss people off, but there are too few fuzzy logic people to actually complain. Fuzzy logic was an abstraction for decision-making under uncertainty, which never really took off because it had some limitations in actual usage.

For instance, the reason we use, let’s say, Bayesian inference—or probability theory in general—over fuzzy logic is because fuzzy logic has an unpleasant property: you can’t do fuzzy logic to perform inference on fuzzy logic. With Bayesian inference, you can effectively do Bayesian inference to learn a Bayesian model.

I think the reason we see certain abstractions and not others is simply that someone picked them up and they worked really nicely. This also, of course, means that there are plenty of abstractions that would be really nice but never end up seeing the light of day because they were never discovered by the right person to actually use them.

I think this leads us on quite nicely to world models. What is a world model? Is it a compressed latent dynamics model, a density over observations, or something else entirely?

11. World Models Meet Reality

World models have exactly the same problem as JEPA because it’s such an absolutely overloaded term. It can mean literally anything, right? We’ve seen people talking about world models that are effectively sensorimotor models, where we have, let’s say, a robot with a tactile sensor or whatever, and it tries to predict what these things do. Or we have a robot that tries to make motor predictions, such as predicting where the end effector would be given some input.

We’ve seen world models that are effectively image or video diffusion models. Then we’ve seen world models that end up being JEPA-style models, where you have some internal state. That gives you the problem that you don’t know whether the internal state is actually useful. Many internal states are just not useful. They look nice, they’re consistent, and they’re hard to benchmark.

Then there are other people who say, “We actually do reconstruction.” They say, “We reconstruct things that we don’t actually end up using.” All of these are still world models.

I’ve also seen people refer to the new Dreamer paper—I don’t know what version they’re at—where they pretrain a large video diffusion model on all the data to play Minecraft. I’ve seen people call this a world model. I’ve seen people say, “It’s just model-based reinforcement learning.” I don’t think there’s a reasonable difference anymore.

The distinction that I personally would make is that if data collection is part of the loop, I would call it model-based reinforcement learning. If it’s out of the loop, then I would call it a world model.

The reason is that it’s basically the control-versus-reinforcement-learning setup that I mentioned previously. In control, or nonlinear model-predictive control, you assume you have all the data. You have your plans, and you don’t need to do any exploration. You just hammer on this.

But for model-based reinforcement learning, you would also discover the world model, and therefore you need to do exploration. Generally speaking, though—and there are also people who think differently—I would say a world model is simply something trained on a lot of data to predict something temporal that is not language. I don’t think there’s a better description.

So many people are talking about this. I interviewed NVIDIA about their Cosmos 3 Omni model, which they called a world model. They were saying, “It’s not just a world model. You can use this as the basis to train a specialized world model. You can use it to build your robot policy for doing whatever.”

Our friend Yann LeCun is talking about world models all the time. Is this just branding? What’s going on? Is this anything new?

I think so. Both JEPA and world models are ultimately branding. There were JEPA-like models before JEPA, and there were world models before world models. In fact, in the Schmidhuber paper World Models, this is model-based reinforcement learning. So, arguably, people who now use it for something other than model-based reinforcement learning are just using the name incorrectly.

I think it’s mostly branding for techniques that are very loosely defined. They’re just fitting some form of dynamics model, whether the dynamics model is probabilistic or deterministic, whether it’s fit on visual data, audio, or whatever. They’re all called world models.

It’s just a dynamics model that predicts the next state from the previous state and an action. I don’t think there’s much more to it. If that’s it, then people have been doing this for a very long time without calling it a world model.

How does this fit with the whole reinforcement-learning story?

I think it’s a little bit orthogonal. I always hear LeCun saying, “We don’t need reinforcement learning; we just need world models.” That’s basically saying, “If we have a world model that describes the world perfectly, then we can just use that and we don’t need to do reinforcement learning anymore. We can do planning in that world model, and we’re fine. We shouldn’t spend all our time doing reinforcement learning.”

The problem with that is, of course, that it’s a very, very simplified assumption. I can give you a perfect world model of chess, and you still cannot solve it. You still won’t beat Magnus Carlsen, because there’s a difference between being able to predict the future and being able to make inferences about the correct action to take.

I think this is very much underestimated, mostly because the tasks being done with world models right now are not that difficult from a control point of view. From a control point of view, I was at RIG, the German robotics conference, and talking to people there, one of the big challenges is doing the sensing part in a reasonable way while also getting relatively strong guarantees.

First of all, if you don’t get strong guarantees, you might crush someone. Secondly, getting sensor information was always the bottleneck, or for most of the time, the bottleneck. To give an example, there’s still no good way of doing high-quality tactile sensing.

The reason you see these Chinese, American, German, and other humanoid robots always doing things like backflips and sprinting, and never just doing the following task—stand up, go into the next room, open the fridge, look in the back, pick out a carton of milk, and bring it to me—is that this would be a much, much harder task.

You have this contact-rich environment, which is hard to model, and it’s hard to gather data. The bet for all these world-model startups and papers is that we can gather all the data you’ll need, distill all these dynamics into a model, and use that. I don’t think this is a reasonable expectation because the world is so strange.

On the other hand, I think you can get very, very far with this. I think especially with world models that have reinforcement learning inside them, you get nice information. You can do the simulation really fast, and you can potentially get things like gradient information, depending on how you set it up.

Mostly, you can do infinite rollouts. That’s a huge paradigm shift for reinforcement learning, which we can also see in something like large language models. Something like GRPO, which effectively doesn’t learn a value function, does advantage estimation on the fly by just simulating a bunch of things—that doesn’t make sense in the real world, where every simulation step is hugely expensive.

It makes sense if you have a simulator that’s really, really cheap and you just roll out everything. You run GRPO and get amazing results because you can hammer data on it.

And you still have the same exploration problems, right? You still need to figure out what the optimal strategy for chess is, but you have so much more ability to do experiments, and that changes the types of algorithms you want to use.

And just to play devil’s advocate, did you see last week—I forget the exact name of the company, but they were talking about physical prompting? They had basically built a foundation omni-model that handles actions, video, and God knows what. They had a robot arm and were teaching it to do things. They gave it a demonstration and said, “You pick up this ball and put it in the bowl.”

Apparently, it had these emergent abilities. For example, sometimes the ball would get stuck to its hand, and it would use its other hand to brush the ball off. Maybe that’s a different thing because you’re providing constraints and demonstrations, and it figures out how to perform particular tasks. But I was quite intrigued by that.

Yeah, I haven’t seen the video you’re talking about. But if we imagine that these models generalize, then yes, we could imagine things like that, especially if we do planning in a sufficiently large world model. Assuming compute and latency don’t play a role, and assuming we have all the data for the specific type of task, I can imagine NMPC ultimately being a solution.

If you go to a university and look for an advanced control theory lecture, the lecture after doing NMPC is completely NMPC-pilled. They say, “Well, I don’t need to do basically anything. I have my huge nonlinear model; I just throw something like MPPI against it, and it works. I don’t need to do anything else.”

Then they notice that this takes a while to run, even if the model is really small, and that they have all of these local optima. If they add constraints, it’s very easy to have a model where most of the trajectories they propose aren’t actually feasible. Then they can do optimization over this, but they need to make the model very small or give it a specific form, because if it doesn’t have that specific form, it doesn’t fit into a linear program or whatever. Suddenly, their opinion starts to dim the more they use it.

I have a feeling that the same thing is going to be true for world models. I think there are going to be a lot of really easy things—the low-hanging fruit—that are just going to be grasped. It might be that, for many tasks, this low-hanging fruit is already enough. But I have a feeling that we’re going to hit some really hard limitations, not necessarily because of the fundamental idea, but because of the nature of how the real world works, how diverse tasks can become, and how complex tasks can become.

Ultimately, with things like moving balls around on a table, I can imagine that working. I can also imagine the same thing working with an NMPC controller. For instance, in the example where the ball gets stuck and you use the other hand to move it around, an NMPC controller can come up with that, right? It would start planning and figure out, “Okay, I need to do something like this,” assuming that the model is accurate enough and that you actually do the complete NMPC.

I don’t think that’s the argument. The argument is: can we make a useful enough model with strong enough guarantees to actually deploy this in production? Production could mean at home. I don’t think this is where we’ll see these robots, at least not at first—not in the short to medium term, and probably not in the long term either. I don’t know.

If we look at it in a production setting, if you go to, let’s say, BMW and say, “I have a robot. It does the right thing 90% of the time, and the other 10% of the time it does something else,” they will say, “No, I will not pay you that money for a robot. I’d rather spend that money fine-tuning my CNC G-code for a couple of seconds,” because it would give them more effective speed. The level of guarantee you need in the real world is much higher than people actually think.

If you look at most of these things that do complex decision-making, like large language models, they’re all in a software setting where you can roll back whatever you want. Even there, if you look at Twitter, people routinely delete their computers because they didn’t sandbox them properly. But if we’re in the real world, the equivalent of deleting your computer would be crushing someone.

The level of safety is much higher. You need a much higher level of reliability, especially for tasks that you do over and over again and that need to work over and over again. Arguably, I would say that peak accuracy matters very little. What you need is some form of guarantee that, in every state, there is a minimum probability of being correct, because then you can start doing things like throwing a Kalman filter against it and taking multiple measurements to get the right result.

If you have 51% accuracy and take however many measurements, you can reduce your probability of error very quickly. I think that, right now, these methods aren’t set up like this. They’re set up to look very nice on demos and to do some really impressive things. But how do you know how many times they failed to do something?

Of course, the consequence of juggling balls is really low, but what are the consequences of doing something wrong? In the physical world, that’s a much bigger problem.

Wonderful. Well, Alexander Mattick, it’s been an honor and a pleasure. Thank you so much for joining us today. Folks at home, if you want to catch up and chat with Alexander, your natural habitat is Yannic Kilcher’s Discord. You do occasionally come into our Discord as well, and we’re very grateful when you do. Thank you so much for joining us today.

Thank you.