深度学习并没有那么神秘或不同——Prof. Andrew Gordon Wilson（NYU）

So when I say deep learning is not so mysterious or different, I'm not saying that deep learning is not mysterious or not different. I think it's actually both. Rather, the ways in which people often think it's mysterious can be relatively well understood, both intuitively through a notion of soft inductive biases, but also formally in terms of rigorous generalization frameworks that have actually existed for many decades. A lot of these phenomena can also be reproduced using other model classes.

I think deep learning really is distinguished by its relative universality—how broadly applicable it is relative to other model classes. That doesn't mean that it's anywhere close to being completely universal, but it's sort of a movement in the direction of universality, a significant movement. It also does representation learning incredibly effectively. It has properties of its optimization objective and its loss landscape that are relatively different and surprising, like mode connectivity.

Deep learning is certainly different and mysterious, but often not in the ways that people might believe. I'm very excited to be talking to you both, even though it can be challenging knowing that there are so many people watching. I think there are so many fundamental misconceptions in the way that people understand generalization, model construction, and artificial intelligence. It's really important to hear a different perspective around, for example, how it's completely fine to build a huge model that will also have a stronger bias for simple solutions, having more of an Occam's razor-like behavior than even smaller models.

These sorts of perspectives actually help us understand phenomena that are often seen as very mysterious, like double descent, benign overfitting, and overparameterization. They give us a principled approach for thinking about how we're going to build our own models for whatever properties we're interested in.

There's this fundamental trade-off between bias and variance. It feels like you're saying you can have your cake and eat it, and you can keep them in the mixture and still win. That just goes against most people's intuition.

1. The Bias Variance Misnomer

I think the bias-variance trade-off is an incredible misnomer. There doesn't actually have to be a trade-off.

Well, Andrew, much of your work challenges conventional wisdom. Is that hard to do? Is there resistance to challenging strongly held beliefs?

Yes. This is what happens when you challenge conventional wisdom. But in some sense, I think that we should always be trying to do that, because otherwise we're just preaching to the choir. What's the point? If you're not changing anyone's beliefs about anything, then maybe it doesn't make a difference.

I think it's important to really try to understand what a lot of people believe that might be wrong, and then unpack that. It's also very exciting and fun. But it's challenging because, of course, the initial instinct will be to resist whatever you're saying. Over time, if you try hard enough and if you talk to amazing communicators like you two, then you can start to have an influence.

I think that's really important, because so much progress has been stalled by getting stuck on misconceptions. Once a certain number of people believe something, it's very, very hard to change their minds, no matter what you say. As a consequence, we've been in all sorts of local minima in machine learning and AI research because we haven't been able to get unstuck from these erroneous beliefs.

There's a whole roster of things like this. The role of implicit biases in stochastic optimization and generalization is significant, but also significantly overstated. How we can have really large models that also generalize well, even when there's a small number of data points, is something that is not very well recognized. In fact, I think it's one of the primary drivers of scale being important for achieving good generalization—not just flexibility, but the simplicity bias that comes about through scale.

I think another misconception is this idea that we should change our model depending on how many data points we happen to have available. This might even be the most controversial one. The reason I don't think we should is because we should always honestly represent our beliefs, and our beliefs about the process that generated our data typically shouldn't change depending on how many data points we happen to have access to.

You can actually demonstrate that these principles work in practice. You can have models that will be very good when you have a small number of data points and also very good when you have a very large number of data points. This relates to not necessarily needing to have hard constraints, but instead combining expressiveness with Occam's razor.

You're only slowly starting to convince me to give up this 10,000-degree polynomial being bad. I'm only starting to change because you value simplicity, or a bias toward simplicity, as much as I do. The real key for me was understanding that somehow scale has a bias toward simplicity. I don't know why or where it comes from, but I believe it, and it's weird.

In your talk last year, your first slide had a bunch of students in the room, and you showed fairly simple, linearly correlated data.

Airline passenger data.

Airline passenger data. It was pretty linearly correlated.

Some seasonality.

You said, "Here's 3 models. One is basically y = mx + c," or something like that, just a straight line. I think the second one was something like 10 parameters, and 10,000 parameters was the third one. Almost everyone in the room said they preferred 1 or 2. You said, "At the end of this conversation, I'm going to convince you to prefer 3," which was 10,000 parameters.

I took a poll at the end, and it did shift, so that was promising. I joke sometimes that if I hadn't met the airline passenger data set, I don't know what my life would be like now, because it has driven so much of my research.

2. Parameter Count Misleads

It's just amazing how people are biased toward choosing the linear function or the cubic polynomial, even if in practice they're not making that choice. On CIFAR, for example, it's not uncommon to use a neural net with tens of millions of parameters to fit a training set with tens of thousands of data points.

Even before deep learning was popular, we were doing nonparametric statistics, where we were working with models like Gaussian processes that were inspired by taking infinite limits of neural nets—models that are more flexible than any neural net you can fit in memory. Other popular covariance functions as well, like the RBF kernel, really are like saying, "I want to use an infinite-order polynomial."

Even in these classical statistical models, we're implicitly saying, "If we're unhappy about the third choice, it's actually because it doesn't have enough parameters."

We want infinitely many parameters, not just 10,000 parameters. And I think another way to say this is that parameter counting is a very bad proxy for model complexity.

Right.

Really, what we care about is the properties of this sort of induced distribution over functions rather than just how many parameters the model happens to have. And so you can have a distribution over functions that is very flexible. It can represent many different solutions to a given problem, but it can also have very strong preferences for certain types of solutions over others. Strong preferences don't mean saying that certain things are necessarily impossible. They can just have epsilon probability.

I think that is meaningfully different from saying, “Okay, we’re going to have a hard constraint, and we’re not going to represent those solutions,” A, because it’s not an honest representation of our beliefs to have those hard constraints, and B, because we see in practice that when we do have these expressive models with simplicity biases, they’re much more adaptive. They’re much more automatic.

So when you have a small data set, it sort of does the right thing. When you have a large data set, it also does the right thing. And so you don’t need as much human intervention, and arguably, that’s the definition of what machine learning is really trying to achieve. It’s trying to build an intelligent system that doesn’t require manual intervention. And so I think this is an important principle towards that goal.

You had a paper kind of removing some of the mysteries of deep learning, but I think it’s still fair to say it’s a bit mysterious where the simplicity bias comes from at scale, isn’t it?

It is. There is some hand-wavy intuition around loss landscapes, and I think this is borne out empirically, so we can understand, for example, that the solutions that we’re finding when we’re building larger models are more compressible and are flatter, et cetera. But this is really ongoing research, and I think this is one of the most important questions to understand right now: Why does scale, rigorously speaking, produce a simplicity bias, and can we get that bias in a more elegant way than just building bigger models?

I was going to ask you: Are you a theory guy or an engineer? I know you’re a theory guy. Presumably, you’re both, but deep in your bones, are you an engineer or are you a scientist?

It’s really hard to choose. In some sense, both. I’m mostly driven by trying to understand things, and so this can be done in a variety of ways. A lot of our papers empirically try to understand model behavior, and so I feel like this is a scientific approach to machine learning.

One thing that really motivates me about this type of approach is that whatever you learn will never go obsolete. Quite often, newcomers to the field and even very experienced researchers feel distressed at the rapid pace in the discipline, where you see methods getting published at a conference and then becoming obsolete within a month. Or when you go to the conference, everything you’re seeing has been replaced by some other algorithm, and you wonder, “Okay, well, is there any point to me investing myself significantly in building a model if I know that it’s not going to be used by anyone for any long period of time?”

I think one way to address that is really to try to combine what you’re doing with an understanding of why things are working. So if you’re building a model that gets better performance on some problem, there’s a reason for that. And if you can understand the reason for that, that understanding will outlive that specific model and how widely it might be used.

I’m really hoping to do research that will be relevant hundreds of years from now. And so I think these questions around model selection, for example, and Occam’s razor, people will never stop asking. Hopefully, they’ll be able to go back and read not just my work, but work that’s been done in this space and think, “Okay, this is useful to me in thinking about how to approach some of these questions.”

In this respect, I would say I’m a scientist, and I try to combine classical theory with empiricism towards understanding model behavior. And I think if you really understand something, hopefully it’s something that you can demonstrate in practice. And so I also try to combine some practical demonstration with a lot of the work that I do.

This process also involves engineering, and sometimes understanding those low-level engineering details becomes really fascinating and leads to unexpected intuitions about the principles behind model construction. I think this is something that’s perhaps underappreciated. Quite often, you can have a great idea, and whether it works or not depends very significantly on all the low-level details, like numerical stability and other things like that. And when you get really deep into those details, sometimes you can discover things at a higher level that are also very significant in how we should think about model construction and algorithm design.

Very cool.

So maybe I was just going to share with you a duality that Shannon pointed out that you may like. It’s a duality between past and future, knowledge and control. He said, “We have no knowledge of the future, but we can control it. We have knowledge of the past, but we cannot control it.”

And I took that and related it to science and engineering. The way I look at the two sides of that coin is: Scientists leverage control to gain knowledge. Engineers leverage knowledge to gain control.

Mm-hmm.

Absolutely. Beautiful. Andrew, we haven’t even introduced you yet. Can you tell the audience about yourself?

I’m Andrew Wilson. I’m a professor at the Courant Institute of Mathematical Sciences and the Center for Data Science at New York University. My work focuses on having a prescription for how to build intelligent systems. What are the key principles involved in model construction?

I think, although the field has made an extraordinary amount of empirical progress towards building more performant machine learning systems, we’re still at early stages of understanding what principles we should broadly embrace when we’re approaching our own problems. And so this involves work on understanding inductive biases, so what assumptions we should be making.

This relates to symmetries like equivariances. Maybe we’re modeling molecules or rotation-invariant images; they could be translation-invariant. How do we represent those invariances? How do we learn them automatically? How do we discover interpretable scientific structure in our data that tells us something, maybe surprising, that we didn’t know before and that will go beyond a particular application?

How do we represent uncertainty towards decision-making? Arguably, a prediction that’s just a point estimate without any kind of error bars associated with it isn’t really actionable in the real world. If you have an autonomous car and it says there’s a stop sign 5 feet ahead, plus or minus 10,000 feet, you can’t really do anything with that information. But if it’s plus or minus 1 foot, then you can really act on that information.

Observing that almost makes you paranoid. It’s like, “Okay, now I really need to represent uncertainty,” because if I don’t have that uncertainty, then machine learning can’t meaningfully engage with the real world. And Bayesian methods, I think, are a really great way of reasoning about uncertainty, and so that also forms a big part of my research program.

Amazing. Welcome to MLST. We have Dr. Duggar in the house.

Thank you.

This is the first time we’ve met in person. We’ve been doing this for 8 years—5 years on this channel, but we had the previous channel as well. Keith came to my wedding on Friday. It’s good to have you here, man.

Yeah, it’s pretty crazy we haven’t met until now. It’s going to be a blast.

Absolutely. We’ve got some good stuff lined up. To kick this off, Andrew, I was inspired by geometric deep learning. I interviewed Michael Bronstein, Taker Cohen, Xuan Bruna, and Peter Velickovic. They had this geometric deep learning blueprint. Our video on that did 500,000 views. It was an amazing video.

The basic hypothesis, as Joan outlined in discussing the three curses in machine learning, is this: There’s the statistical curse, which is that you only have so many data points, and the distance to those data points, or the density of those data points, is cursed by the dimensionality of your data. There’s the optimization curse, which is that you get stuck in these local minima. And there’s the approximation curse.

That’s where they were driving to: You have this function class, and you can be quite opinionated in how you structure that function class. But if you make it too small, you incur approximation error, where the actual test sample is some epsilon distance from the approximation class.

They said that all of these things are cursed, and I guess their prescription was this Platonic idea that if we constrain the models—if we add bias to the models with these symmetries, because the generating function of the universe is using these symmetries anyway—then there’s no apparent approximation error in doing so, so why wouldn’t we do it anyway?

I think your ideas are tangentially related to that.

You still think we should have biases, but you also think we can have our cake and eat it, so to speak.

3. Soft Constraints Preserve Flexibility

Mm-hmm. So this is a wonderful question, and I've also done a fair amount of work on geometric deep learning, particularly in terms of how we should represent equivariant symmetries in scientific domains. I agree that it's very appealing to say, “Well, if we know that some constraint applies to our problem, then we should encode it in our model.” I don't think that's particularly controversial. There are some perhaps surprising results that, in some instances, even when you know what the constraint is, you can do as well or better when you don't represent that constraint. This can be due to the dynamics of how you train your model, et cetera.

But generally, the way I would respond to this is really to say 2 things. The first is there aren't that many instances in practice where we know exactly what constraints we want to have, even when it comes to things like physical conservation laws. So rarely are we modeling, for instance, closed systems. You can have a dynamical system, and maybe you have a pendulum with wind in the air or whatever, and now you have some sort of violation of conservation of energy. And so you want to perhaps instead build models that are just biased towards these constraints without being exactly constrained.

Secondly, when you represent so-called approximate constraints or you have soft constraints, you have a model which is very flexible, but it says, “Well, if we can fit the data in a particular way, then we want to do that.” Quite often, it will just collapse down onto those constraints if it provides a consistent explanation of what we observe. And so if you're paying any kind of penalty for deviating from those constraints and you can perfectly explain your data with those constraints, you'll just collapse down onto that.

And so I think, as a prescription for model construction, it's often going to be fruitful to try to embrace expressiveness, but at the same time have a simplicity bias, which can be formalized in terms of compression. And this is really an honest representation of our beliefs in many cases. Another way to describe my philosophy for model construction is just honestly represent your beliefs. We believe the real world is a complicated place, and if we combine that belief with the idea that simple solutions that are consistent with our observations are more likely to be true, then we can often see desirable behavior in quite a variety of different settings.

4. Double Descent Reverses Overfitting

And so, in terms of having cake and eating it too, I think one of the most surprising findings that we and others have had is that quite often you can increase model expressiveness while simultaneously increasing its biases, so larger models are often more inclined towards simple solutions. There are demonstrations of this that have been hiding in plain sight, like double descent: this idea that as you increase model flexibility, your generalization error first gets lower, so it improves. As you capture more structure in the data, it gets worse as you start to overfit, and then it gets better again.

And in that second descent, typically all of the different models that you're considering are fitting the training data perfectly, so the only possible way that larger models could be generalizing better is because they have some other sort of bias, like a simplicity bias, rather than being more expressive. And so I think time and again, researchers express surprise at the fact that they can have these massive models, billion-parameter-plus models trained on relatively small data sets that aren't overfitting. But in fact, if they've made the models even bigger, they would be less likely to overfit. And so quite often, expressiveness and soft constraints can be aligned.

So parameters don't solve your problem? You're not using enough of them.

We always want more.

In prediction with expert advice, we know theoretically and, for me, empirically, that by keeping all of these other experts around in the mixture, you pay a cost for that.

Mm-hmm.

When you have the historical experts in the mixture, they still had some weight on every single prediction. You have to give them some epsilon weight, because otherwise they would die. And that actually harms your performance. So the question is, is it better for you, when the new regime comes, to pay the cost of learning the regime versus the switching cost of bringing the old regime back?

And that's kind of the same with any ensemble or any set of restrictions on biases, really. There's this fundamental trade-off between bias and variance, and it feels like you're saying you can have your cake and eat it, and you can keep them in the mixture and you still win. And that just goes against most people's intuition.

So I think the bias-variance trade-off is an incredible misnomer. There doesn't actually have to be a trade-off. The idea behind the classical bias-variance trade-off is that your generalization error can be compartmentalized in these 2 terms. Bias is how well you're fitting the data, essentially, and variance is how your fits vary depending on whether you sample different points from the distribution that you're trying to model.

It's true that sometimes, if you naively build a really large polynomial, for example, you can have low bias and high variance, whereas if you build a small polynomial, maybe you have low variance and high bias. However, approaches like ensembling are actually a good way of getting low bias and low variance, and it turns out that building large neural nets is another way of getting both low bias and low variance. You actually have flexibility combined with a simplicity bias. And this is what's leading to good generalization, and it's another perspective on double descent.

I think here's the way I'll put the question: I ran into this as a practitioner back in the day. So maybe I was just stuck in the hump of having too many but not enough parameters to get to the double-descent phase. I'm not sure. But what I experience is that having parameters in a model, even if they're very, very small because I put in some term in the objective function that forced them to be small, is not the same thing as actually the simpler model that just didn't have them at all.

Right? We'll talk about marginalization and the Bayesian perspective on that. So overfitting can be a real problem. For example, you brought up conservation. If I'm doing a model and I don't enforce conservation of energy, and then, as a result, I end up with some small parameters that cause a little bit of feedback, increasing energy every single time a robot takes some action, it can cause it to spin out of control, right? In that case, I actually did need it to conserve energy and not have that positive feedback.

So maybe we're still struggling with the fact that overfitting can be a problem. It's a real problem. It's a known problem. How do we know if we're overfitting in a bad way? Maybe we don't have enough parameters and we're stuck in the area before we got to double descent. Or how do you, in practice, avoid the actual consequences of bad overfitting?

Right. So overfitting is real, absolutely, but the conventional wisdom about how we should approach it, I think, is fundamentally misguided.

Okay.

This is rooted in things like the bias-variance trade-off: let's constrain our hypothesis space so that we can't have a bad fit to the data that will make bad predictions and so on. Whereas instead, I think if we just embrace the honest belief that there are many possible solutions, even if they're not probable for any given problem, combined with this sort of simplicity bias, we won't tend to overfit.

And interestingly, the prescription is almost the opposite of what people think it perhaps should be in principle. Building a smaller model is usually the prescription for avoiding overfitting.

Right. Or to enforce simplicity.

Exactly, yeah. Whereas in fact, as we build bigger models, we often actually start to alleviate overfitting, and double descent is just a great example of this because that first ascent is from overfitting the data, but then it gets alleviated as we start to make our models bigger and bigger.

It happens by some phenomenon that we still don't really fully understand: this ability of simplicity to start to come back into the picture as you make it even bigger.

Right. So in that second descent, the models are typically fitting the training data perfectly, and so the loss is not really the decisive factor anymore in determining which models we're selecting. It's something else, and so those other biases start to dominate. This is actually really important, I think, when people talk about phenomena like flatness.

Flatness is the idea that if you perturb your parameters, you can still get a relatively low value of the loss. There are all sorts of debates about the role of flatness and how relevant it ought to be in understanding generalization, et cetera. But I think what a lot of these discussions miss is that it's just one of many properties that control generalization.

If I had to choose between a model that has very high loss but is very flat and a model that finds a low-loss solution that's relatively sharp, I would almost certainly choose the low-loss solution.

Mm-hmm.

And so when you're in that kind of second-descent regime, you're now controlling for the value of the loss, and it's just the flatness of the solutions that's increasing.

Mm-hmm.

That's sort of one way of knowing maybe what side of the curve you're on and thinking about how big you should make your model. But I would also just say: make your model always as big as possible. Try to combine what you're doing with some sort of simplicity or a compression bias. There's a question of how you do that, but I think there are good ways that we have of thinking about how to do that.

Trick question: Is predictive power the same as understanding?

5. Prediction Needs Representation

I agree with the idea that representation matters, representation meaning sort of how you're solving the problem, even if you're getting the same performance in a particular application. But the reason it matters is because different representations that are achieving the same performance might give you different performance on different problems. If we're trying to build more general agents, we want to understand what sorts of representations are going to provide a better general description of the real world.

That means we want to avoid things like shortcut learning and so on if it's just going to lead to good predictions in some contrived problem and not really in the real world. There's this question of whether we can understand what sorts of distribution shifts we might typically encounter, and whether we can build methods that have broadly more robustness to a variety of different types of realistic distribution shifts. I think this connects to things like no-free-lunch thinking.

The no-free-lunch theorems say that every model is equally good in expectation over all problems sampled uniformly from a distribution over all problems. There are other no-free-lunch theorems that say no single learner can be good on all problems. The issue, I think, with these theorems is not their mathematical validity. What they're saying is correct under the assumptions they're making, but rather that the assumptions they're making are not a good description of the real world.

The real world is a small corner of all possible data sets. It's not drawn uniformly from a distribution of all possible problems. If we were to do that, we would mostly just get noise. The question then is: To what extent is the structure across real-world problems shared, and at what level of abstraction can we represent that shared structure? My contention is that the distribution over real-world data is biased toward low Kolmogorov complexity, and so are some of the models that we started to develop.

Yeah, but can you give an example of where it was hard to confront a misconception, why it mattered, and what the process involved?

6. Deep Ensembles Are Bayesian

There had been this approach where you would take some approximate Bayesian inference procedure and pit it against deep ensembles as the non-Bayesian alternative.

Yeah.

Normally, I don't care that much about what's being called Bayesian or not. It's the same with intelligence. Whether something's a good representation—all these things—let's just connect this to whatever problem we're trying to solve.

But this was actually problematic because the takeaway seemed to be that if deep ensembles were working better than some Laplace approximation or some MCMC procedure, then the answer was to be non-Bayesian, to be less Bayesian than we had been historically. That turned out to be exactly the wrong directionality for how we should think about model construction, because for a given computational budget, those deep-ensemble procedures were actually doing a much better job of approximating the posterior Bayesian predictive distribution, so doing marginalization.

In fact, the prescription should have been that we actually need to be more Bayesian. This was a frustrating thing, but it was also very difficult to approach because there had been so many papers where people had just written that these deep ensembles were the non-Bayesian alternative.

Coming out and saying, “Well, actually, they're doing a better approximation of the Bayesian ideal than all these methods that are being called Bayesian,” is confronting hundreds of papers in some sense at once. But it felt like a very important thing to do. We had actually done it subtly in a lot of papers, where there'd be some subsection of some paper that mentioned something like this, but that was never really internalized because it was never front and center.

I thought, “Okay, a blog post is the right way to do this, and let's make it all about this.” I think in the end, actually, it was the blog post that changed people's minds. I never saw a paper after that where people were making this separation. But also, I think it did strike a nerve a little bit.

Yeah, but how should we approach model construction? How can we embrace expressiveness without overfitting?

When I say I want to embrace expressiveness, there is some subtlety associated with that idea. That basically means that, in some cases, maybe we're wanting to represent lots of solutions, but we're assigning them almost zero probability, but not zero probability. They're possible but not plausible solutions in our view.

If the data is telling us that we really should be paying attention to a certain type of structure that might surprise us, the model can actually respond to that. If that structure isn't actually there, then your model isn't going to perform a lot worse than the model that is exactly constrained in those ways.

In terms of how we should approach model construction in general, if you have a soft bias, a gentle encouragement toward certain types of constraints over others, quite often you can do as well as the perfectly constrained models. The reason is that you're paying some sort of penalty, even if it's small, for deviating from that constraint.

If you can fit the data perfectly with the constraint, you'll just collapse down onto that model. We noticed this in a work we had called *Residual Pathway Priors for Soft Equivariance*. This was a Bayesian mechanism, essentially, to have a distribution over solutions that would be concentrated in some way around certain types of equivariant constraints.

Just for the sake of the audience, equivariant is a generalization of invariance. It basically means if you have some transformation T, F of T x equals T of F of X, rather than F of T x equals F of X.

Like a CNN, for example. So it commutes in that case with the translation.

Exactly right. So if you translate the image in some way, the pattern of activations will translate in the same way across the different layers, rather than just staying exactly the same, which would be invariance.

In this paper on *Residual Pathway Priors*, we were interested in the strength of this soft bias. We basically had a distribution over neural-net parameters that had a covariance matrix that lived in some equivariant subspace of our choice, plus some orthogonal complement.

There would be these weighting terms that would represent how strong we wanted this bias to be for equivariance. Surprisingly to us at the time, it didn't matter very much. As long as you had a very soft bias for the constraint, it would often be as good as even a perfectly constrained model.

Basically, the reason is that you're still paying some penalty for deviating from the constraint. If it is a good description of the data and you would fit it perfectly, then you often collapse down onto that constraint. Although I wouldn't want to dismiss the importance of trying to calibrate these biases, it can matter in certain instances. But in a lot of instances, a very gentle bias is sufficient.

So maybe, just to put this into perhaps more familiar territory for other Bayesians out there, the assignment of priors for parameters: it's always the goal to try and find a prior that's relatively ignorant, but encodes some very soft type of constraint.

Maybe it's a scale-invariant prior or a location-invariant kind of prior. Overall, a lot of times these priors are maybe worth 1 or 2 data points, but they're small enough that they can be overridden quite easily by enough data. But even that small amount is enough to avoid stupid answers like the chance of a head is infinity or something like that. Is it kind of analogous to that?

I think that's a reasonable analogy. I would also add that we can't get away from making assumptions. So even though I'm in favor of embracing expressiveness and having relatively soft biases for certain types of solutions as opposed to hard constraints, machine learning means learning by example, and we can't do that without making assumptions. The question is just what assumptions should we be making and at what level of abstraction?

Perhaps it's enough in a surprisingly large array of different problems to embrace expressiveness in combination with some sort of simplicity, some Occam's razor bias that can be formalized in terms of compression.

Empirically, simple models work better.

Ironically, given the surprise people often express at this idea of wanting to always embrace flexibility, before deep learning, the community had started to come on board with this notion that we want arbitrarily flexible models. In fact, the class of models that I was working on in my PhD, Gaussian processes in machine learning, were kind of inspired by this idea that we want really, really large neural nets. Radford Neal, a statistician at Toronto at the time working in Geoff Hinton's group, was saying, “Okay, we wanna build models the size of a house, and I'm a Bayesian, and I'm gonna really embrace expressiveness, so I'm gonna take an infinite limit of a neural net with an infinite number of hidden units, and this is going to converge actually to a Gaussian process using a central limit theorem argument with a particular type of covariance function.”

People thought, “Oh, well, that's amazing. Let's just use Gaussian processes because they're so much more principled in a lot of other ways: they're less sensitive to a bunch of design decisions, et cetera. They can be written in a very small number of lines of code, and everyone anywhere in the world is gonna get basically the same answer, et cetera.” So people sort of moved in this direction of just embracing expressiveness.

But then, once we started working in these kernel formulations, I think people perhaps started to forget that there was this dual-space correspondence, and we were actually working with models that were more flexible than any neural net you can fit in memory and finding we were achieving very good generalization, especially on problems with a relatively small number of data points.

So now, to get back to your question, Radford Neal also had an interesting quote, I believe, in his PhD thesis: whenever you have a simple model that performs well, you can always build a more complicated model around it that will perform even better. And he gives this example of handwritten character recognition where you might have irregular writing styles, weird ink blots on the page, just some sort of structure that you probably haven't already accommodated in your model that you can try to accommodate, and you'll achieve better and better performance.

So what I would do in this situation you described is really try to understand what's the inductive bias there that's leading to good performance, and how can I soften it in some way? How can I generalize this in a way that still honestly represents my beliefs?

But there seems to be something wrong with this quote that you can always build a more complex model because maybe you can always build a more complex model that has a higher likelihood on the training data, but overfitting is a real phenomenon. Nothing in your work says that overfitting doesn't occur and can be harmful. So there have to be situations where, if you diverge from the ground-truth model, I'm sure I could just build a system that has a ground-truth model, simulate data, and provably show that a more complex inference doesn't generalize as well.

Mm-hmm.

So that has to be true. How do we know in reality when we've stepped too far?

It's a great question. I think we have to be careful about what we mean when we say a complex model. I think most people would not consider Gaussian processes with an RBF covariance function, just the standard covariance function kernel that's often used, to be a complex model, but it's highly expressive. So it's more expressive than any neural net we can fit in memory. It just has very, very strong preferences for certain types of solutions over others, and this enables it to be extraordinarily data-efficient.

So one of the main use cases these days for Gaussian processes is in something called Bayesian optimization, where you're trying to maximize some sort of black-box objective, so it's not something you have a closed-form expression for. It could be generalization performance of a neural net as a function of some of its hyperparameters, for instance, or some really costly physical simulation as a function of some parameters, and you basically want to query this objective as few times as possible in order to achieve a good result.

Gaussian processes are an amazing surrogate model for this objective, and you use the uncertainty to do the exploration efficiently. So I think that you can have expressive models that aren't necessarily complex. They still have very strong simplicity biases, very strong preferences for certain types of solutions over others, but at the same time, they're representing a wide array of possible solutions to the problem. And I think that's how you can kind of reconcile what Radford Neal was saying with what we see in practice around things like overfitting.

I think there's a common misconception that the expressiveness of a model and its inductive biases are at odds with each other. The more expressive the model, the weaker its assumptions in some sense—the fewer inductive biases it has, the less data-efficient it will be, et cetera. And what we found, which has been quite exciting, is that the larger you make, say, big transformers, actually, the stronger its inductive bias is.

The models get both more expressive and they have a stronger simplicity bias. So I think you can expand these two things together in some sense, and this is how you can avoid, say, overfitting and other sorts of issues with not achieving very good generalization. And I think one of the clearest demonstrations of this is in a phenomenon called double descent.

Hmm.

Double descent is this phenomenon where typically on the horizontal axis you have the expressiveness of the model, the number of units, for example, in each layer of a residual neural network, and on the vertical axis you have generalization error. Initially, generalization error decreases as you increase the expressiveness of the model and it's able to just fit the data better and capture more structure. Then it starts to go up, and that corresponds to some sort of overfitting. And then it decreases again, and that's why it's called double descent, because of that second descent.

In that second descent, all the models typically have about 0 training loss. The training loss just keeps going down as you increase the expressiveness of the model until roughly the number of parameters equals the number of data points. What that means is that the larger models in that second descent cannot be generalizing better because they're more flexible. They're all fitting the training data perfectly. It has to be that the larger models have some sort of bias which is enabling better generalization, and it turns out that this is a simplicity bias, a compression bias that we can measure.

Yeah. Let's talk through that a little bit. So as I understand it, your thesis is that there is some kind of generating function of the universe, and in some sense it's quite simple. François Chollet talks about this. He talks about the kaleidoscope effect: we have the generating function, and then it gets composed together in a myriad of different ways, and we see the kaleidoscope, and intelligent people can decompose the kaleidoscope back into the original generating function.

You've also said that natural data in particular is quite low-dimensional, quite simple, and that neural networks prefer simple data. Even that, I want to take a slight issue with, because what I find is that neural networks in the early stages of training prefer simple data, and when you continue to train them, they seem to complexify and complexify and learn more high-frequency data. How do you think about that?

It's a great question. So it's a really important observation that neural nets tend to learn structure before they learn noise and things like this. There's a question around to what extent being able to fit noise is hurting their generalization capabilities.

So there’s this other phenomenon called benign overfitting, where the model typically fits a mixture of signal and noise, but fitting the noise doesn’t significantly degrade its generalization performance. This is often seen as something specific to deep learning and at odds with everything that we know about generalization.

That’s partly because classical frameworks for trying to understand generalization, like VC dimension and Rademacher complexity, are essentially measuring a model’s ability to fit noise. However, there are other generalization frameworks, like PAC-Bayes and countable hypothesis bounds, which we explore. These don’t penalize an expressive hypothesis space and instead try to understand what sorts of soft preferences the model has for certain solutions over others.

We’ve been able to achieve fairly tight bounds on the generalization performance of these large models using something called a Solomonoff prior. A Solomonoff prior says that we actually have a maximally overparameterized model: We can represent every possible program on a computer, but we have exponentially stronger preferences for solutions that have what are called low Kolmogorov complexity, and so are very compressible. The Kolmogorov complexity is the shortest possible program that can generate our hypothesis.

The fact that we’re able to get these tight generalization bounds for these very large models—and, in fact, the generalization bounds get better as we make the models larger—suggests that this is not a bad description of how these models are actually behaving. Doing induction with a Solomonoff prior is called Solomonoff induction. It seems that when we make these transformers, for instance, very large, we’re combining this expressiveness with this strong preference for low-Kolmogorov-complexity solutions.

Another observation that’s been made is that models are becoming increasingly general-purpose. So, 20 or 30 years ago, the typical prescription was to encode as much expert knowledge as possible into the model you’re constructing and tailor it very specifically to the problem that you’re considering, because of results like the no-free-lunch theorems. These say that every model is equally good in expectation over all problems sampled uniformly from the distribution over all problems.

There are several no-free-lunch theorems. Another one says that a single learner isn’t going to be good on all problems. These results are mathematically correct, but they don’t really correspond to the real-world data-generating distribution. The real world is a small corner of all possible data sets; it’s not drawn uniformly from that distribution. If you were to draw data sets from that distribution, you would mostly get noise.

I guess the question is: What is the real-world data-generating distribution really like? It seems like there is a bias toward generating data with low Kolmogorov complexity, and our models share that bias. This is why we’ve seen increasingly general systems. We’ve moved from feature engineering—basically hard-coding structure into our models—to more modality-specific models and architectures: convolutional neural nets for vision, recurrent neural nets for sequences and language, and MLPs for tabular data and regression, to transformers for almost everything.

This isn’t to say that transformers have achieved general intelligence—absolutely not. But they’re, relatively speaking, more general than their predecessors. We have seen this kind of movement toward increasingly general models, and our contention is that this has been made possible by aligning with the real-world data-generating distribution, which seems to have a bias for low Kolmogorov complexity.

We had this paper on no-free-lunch theorems, inductive biases, and Kolmogorov complexity. I think one of the most surprising findings in that paper was that convolutional neural nets, which were clearly designed for image recognition—they have locality and translation equivariance, and so on—provably have inductive biases for tabular data shaped as an image. The only possible reason that could be the case is because they both share this bias for low Kolmogorov complexity, and that bias gets stronger as we make the model bigger.

Yeah.

David H. Wolpert

Convolutional neural nets, which were clearly designed for image recognition, so they have locality and translation equivariance and so on, provably have inductive biases for tabular data shaped as an image.

Yeah, mm-hmm.

David H. Wolpert

The only possible reason that could be the case is because they both share this bias for low Kolmogorov complexity, and that bias gets stronger as we make the model bigger.

Okay. I have actually 2 questions about this, and they’re about the Kolmogorov complexity. I think, if I heard you correctly, on the one hand, you’re saying that just stock neural network training of today—just transformers, SGD, batch norm, whatever people are doing—seems to empirically exhibit a bias toward lower-Kolmogorov-complexity models, and this shows up by their generalization falling within this bound that you found.

Then you also mentioned Solomonoff induction, which I believe would be explicitly introducing a penalty term or a risk term—an objective, part of the objective function—that has to do with Kolmogorov complexity, maybe pushing the model a little bit further toward simplicity. I think you’re talking about both.

So maybe you could elaborate. Also, where is this bias toward simplicity coming from? Everybody knows the algorithms. Which part of the algorithm is inducing this simplicity bias? Is it because we’re using floating-point numbers, IEEE, or what? Where is it coming from?

7. The Source Of Simplicity Bias

David H. Wolpert

This is largely an open question, although there are some intuitions, and this is something I’m really excited about pursuing further in my research. Where does the simplicity bias, especially from scale, originate? There are some geometric intuitions around the loss landscapes for the objectives that we use to train these models.

When we’re minimizing training loss, we can try to geometrically understand the properties of this landscape that we’re minimizing. It’s been observed, for instance, that flat solutions—meaning solutions where you can perturb the parameters by some amount but retain low training loss—tend to generalize better than sharp solutions that have the same value of the loss. You could make a compressibility argument for why that’s the case. Flat solutions don’t need to be represented with as much precision, and so they’re more compressible.

As you grow the size of these neural nets, the relative volume of these flat solutions starts to exponentially dominate the volume of the sharp solutions.

Mm-hmm.

David H. Wolpert

You can imagine, heuristically—

Well, why is that? Do we know why, or is it just an empirical observation?

David H. Wolpert

Yes. We know why to some extent, but our understanding is somewhat heuristic. You could imagine, for instance, having a region of the loss surface with radius R_A that’s flat. You perturb your parameters within that radius, and you have a low value of the loss. Then there’s another region with R_B, where R_B is much less than R_A.

Mm-hmm.

David H. Wolpert

Now, as you grow the number of parameters in your model, D, R_A^D starts to really dominate relative to R_B^D.

Mm.

David H. Wolpert

We seem to observe this in practice. There was a result—not from my group, but from Tom Goldstein’s group at the University of Maryland—where they were trying to understand the role of the implicit biases of SGD and stochastic optimization in generalization.

Quite often, as you perhaps have alluded to, SGD is thought to be an integral component of achieving generalization in deep learning. There’s this idea that we have these very complicated objectives that we’re minimizing that are very nonconvex, and SGD somehow saves us: The implicit biases of SGD navigate our procedure through some region of the loss landscape that represents low-loss solutions that do generalize rather than low-loss—

Mm-hmm.

Solutions that don’t generalize well. It turns out you can actually do full-batch gradient descent and achieve pretty comparable generalization to what you would get if you were using SGD, even if you don’t try to make that implicit regularization explicit in the loss.

There was another paper that showed that if you even do guess-and-check—so you just randomly sample your solution vector and then stop when your loss is below a certain threshold—the generalization will also be fairly comparable to what you get if you use SGD or Adam.

That corresponds to this geometric intuition. If you’re just throwing darts at the loss landscape and then stopping as soon as you have loss below a certain threshold, you’re much more likely to be in this region of low loss and good generalization than in a region of low loss and bad generalization as you increase the number of parameters in the model.

This isn’t an airtight argument, so there are lots of ways that you can increase parameters in models without really influencing the geometric properties of the loss landscape. However, it does seem to correspond to what we observe empirically, and not just in terms of results like guess-and-check.

If we look at something like double descent, in that second descent we can measure something called the effective dimensionality of these models. That’s the number of relatively large eigenvalues of the Hessian, which is essentially the number of sharp directions in the loss landscape.

Right.

That decreases as we make the model bigger.

So interesting. And to the second question about explicitly introducing a penalty term for Kolmogorov complexity, have you done much of that or looked into that? Is that useful, or is it just not necessary?

Daniel Roberts

It's something I've been thinking about. It's very hard to operationalize. You can evaluate these bounds by computing an upper bound on Kolmogorov complexity, measuring the compressed file size of your model after training.

But in order to use that prior as some sort of regularizer, you would need to be considering the compression of a whole set of different hypotheses, not just a single hypothesis that's found by the model. And so there's an open question of how you could try to operationalize that. Solomonoff induction is sort of an idealized learning system.

It's not something that we can really do exactly in practice. It seems that neural nets are sort of approximating it, and that's evidenced by these bounds and how they're able to tightly characterize the generalization behavior of these models. But it's hard to turn into some kind of regularizer.

I'm also interested in how we can go beyond things like Kolmogorov complexity. Kolmogorov complexity doesn't distinguish between incompressibility due to randomness, such as noise in our data, versus structural complexity. Scott Aaronson actually had a really interesting blog post related to this about 10 years ago, where he was imagining a system. He had this kind of physics analogy where you have coffee and cream, and initially they're separated liquids, and you start to stir them together.

As you do this, the entropy of this system is increasing over time. The Kolmogorov complexity is increasing. But the intuitive sophistication of that system is kind of non-monotonic. Initially, it has low entropy and low sophistication, then sort of intermediate sophistication and entropy, and then high entropy and low sophistication.

Oh, I've seen that one. That was shared on our Discord recently.

Oh, very nice. Okay.

You should join our Discord.

I'd love to. Yeah. We actually have an idea, of course, for how we can do this. But I don't want to get ahead of myself. There was a blog post that sort of compares this to machine learning. The entropy of the system is increasing, and the Kolmogorov complexity is increasing. But the intuitive sophistication of that system is kind of non-monotonic.

You can think of this in a machine learning context in terms of reasoning about the value of data. If I sample data from a uniform random distribution, that's going to be very incompressible. I'm going to need to memorize it. It's sort of uncorrelated. This could be useless for learning a representation for training my model.

I could alternatively imagine some sort of sophisticated cellular automaton problem, some sort of Game of Life problem with very sophisticated generation rules. That data actually could have an extraordinary amount of value for learning a representation. There was a paper that looked at something briefly like this called “Intelligence at the Edge of Chaos,” and I think there are other results like that that are coming out. You might want to train your models on this data with a lot of structural complexity, even if, in the end, the model has some kind of Occam's razor bias.

We've been thinking about measures of information that might compartmentalize structural complexity and random complexity. This will help us reason better about the value of data and develop priors that are like Solomonoff priors, but might actually be more directly addressing the type of incompressibility that we're interested in.

Yeah, I mean, you said so many interesting things. First of all, to Keith's point, you were saying that this isn't a penalty term yet. Your hypothesis is that neural networks implicitly do this kind of compression, which might be correlated with or related to this Kolmogorov complexity.

Many folks just equate intelligence with compression, and when I spoke with David Krakauer, he took umbrage with that. He said, “Compression is a component of intelligence, but there are so many other things going on as well.” Certainly, when we look at things like the ARC Challenge, there are so many possible solutions. A naive heuristic of just selecting the simplest program isn't always the best thing to do. There are many possible selections that you could make.

You demonstrated this upper bound, which used this complexity term, and in a sense, that's saying that it could be no worse than this, but it could actually be so much better. Our empirical experience of deep learning models is that they seem to have found some superficial generalizing thing. We call it shortcut learning, basically. They seem to have found some superficial generalizing thing that does all the things you said when you mentioned the No Free Lunch theorem.

You could take a CNN and use it on tabular data. You could take a transformer and use it on audio data. It's almost like what we've seen is that we've hit this, for want of a better word, local minimum, and it feels like we need something more to get to the real understanding of some of these problems. Does that make sense?

Daniel Roberts

So, is compression intelligence? I think this is a big debate right now. I think it is very closely associated with intelligence in a lot of ways. If we can compress our data really effectively, then, in order to do that, we're discovering regularities that are typically going to help enable generalization.

In some sense, physical laws, for example, are a great compressed representation of reality. I think that compression is really intricately connected with what we mean when we talk about building intelligent systems.

There are instances in which this can go very wrong, like in shortcut learning, where, for instance, you might have some spurious correlation. Maybe every time there's a blue pixel in an image, the label is a bird or something like this. The model forgets about the foreground, and it just looks for some feature in the background.

In terms of generalizing on that distribution, that's actually not a bad idea. That's actually the right thing to do. Occam's razor is really a good principle. But if we go out of distribution, so we see a bird with a volcano or something behind it, or in a room, then it's not going to arrive at the right label.

So there's this question of whether there are other principles of induction that will lead to greater robustness under distribution shifts. I think, in general, Occam's razor still is the right thing to do. There are many cases where compression will actually not give you what you want more broadly if you move beyond that distribution.

In the absence of additional information, it seems like you can't really do better as a guess as to what's the right strategy. I still think Occam's razor is a very robust principle of induction, and perhaps one answer is just more data. I think this will work well in some cases and not in others.

We have some work in progress on trying to build transformers for matrix operations. This is actually analogous to the work that people have done on transformers for multiplication, addition, and things like this.

These systems, LLMs, seem very impressively effective in some instances in being able to write code and solve a variety of different problems we didn't really expect them to be able to solve. In other instances, they're just shockingly bad: how many Rs are in “strawberry,” reversing strings, counting, and basic addition and multiplication.

There's a debate about whether we should be giving them tools to do those things. Why not just give them a calculator? Whatever they learn to do in terms of adding and multiplying numbers is still not going to be as efficient as giving them access to a calculator, so why don't we just do that?

Or is there some greater value in having to learn a representation that can do some of these things? Even though we'll always want to use a calculator if we're multiplying or adding numbers, being able to do something like that reasonably well might transfer into other settings that we don't really anticipate.

I'm probably more in this category. I just think, intellectually, we should try to figure out how to do this without tools. This work that we're doing on transformers for matrix operations is quite analogous. You could argue that addition and multiplication are just fundamental primitives for trying to build intelligent systems.

We're not as good at it as a calculator is, but it might be important for us to have some ability to do these things. The same could be said of matrix operations. This is really the backbone of all sorts of different learning algorithms, like Gaussian processes, which we discussed a little, involving solving linear systems with a covariance matrix and computing log determinants.

Deep generative models, like normalizing flows, involve log determinants. Dimensionality reduction, like PCA, involves other matrix operations. They're just really ubiquitous as a primitive for trying to build learning algorithms and intelligent systems.

And so I think if transformers are ultimately going to become some sort of general intelligence, then they ought to be competent at these types of operations. We were representing matrices as sequences of numbers and having the outputs be things like the maximum eigenvalue, the solution to a linear system, or whatever other operation we were considering—the spectrum of eigenvalues. We found, interestingly, that when you train this approach on Gaussian random matrices, so you just sample every entry of the matrix from a standard normal distribution, it will do fairly well at in-distribution matrix operations: other matrices sampled from that distribution that it hasn't seen before, but extraordinarily poorly even if you go slightly outside of that distribution.

So give it an identity matrix—all 1s on the diagonal, 0 everywhere else—and that will have pretty low density. It has support under this Gaussian distribution of matrices, but pretty low density under it. It hasn't seen something very much like that; it will just completely fail. It won't do anything reasonable.

And so we considered a number of interventions, like looping, adaptive test-time computation, and enriching the training distribution very significantly. We had this einsum space of structured matrices that we were sampling from—all sorts of different matrix structures: Toeplitz, Kronecker, block diagonal, low rank, and a variety of other interventions. Interestingly, once we had done that, this approach actually was able to generalize even to matrices that were out of distribution for this fairly rich sort of training set that we'd created. And so it seemed to move more toward learning an algorithm rather than just doing statistical interpolation on the training data.

I'm an optimist by nature, so I am very happy about the fact that we can use data to move toward things like algorithm discovery. But now I remember why I started talking about this: I think there are instances where it's going to be hard. So autonomous driving is an example.

Right.

You have outliers, but they're different each time. Just training on the outliers isn't going to be useful because they're going to be new outliers that look very different from those outliers. I just have the intuition that more data alone is not really the answer to building robust autonomous driving systems.

Yeah, I just want to say, as far as why not just give them tools, my answer, folks, is because those have to be programmed and built by people, and the whole point here is to allow machines to do their own programming, right? Machine learning. And if they can't even learn to do multiplication reliably and to generalize from decimal multiplication to binary or hexadecimal, and from 9 digits to 36 digits, what hope do we have that they're going to discover relativity or non-Newtonian mechanics or any other kind of frontier things? I mean, isn't that part of the goal here?

Absolutely. I think being able to do certain things well, we're finding, might surprisingly relate to doing other things very well. And I think we have seen this to a large extent with LLMs. We had a paper where we just took a text-pretrained LLM off the shelf and applied it to time-series forecasting. We did everything naively; we weren't even really intending for this to be a proper project or a paper. We were just curious: if you just give GPT a sequence of numbers naively encoded as strings and have it extrapolate the next sequence of string tokens, how would it compare to purpose-built time-series forecasting procedures?

It just worked way, way better than we thought it could possibly work. It didn't even really make sense. And so we did turn this into a proper project. We did a little bit of work on trying to improve the tokenization and think about uncertainty representation, et cetera. But most of that paper, called Large Language Models or Zero-Shot Time Series Forecasters, was focused on trying to understand how this is even possible.

In the end, it did start to feel a bit more like maybe it's not just that you can do this; maybe you should do it in some instances. They did quite well on a variety of benchmarks. To me, this suggests that with a proper dedicated research effort on LLMs for time series, we could see these systems actually working a lot better than the purpose-built models. And so what that shows is being able to predict the next words in sentences can actually transfer to being able to do other things, like time-series prediction, really well.

And we had another paper on LLMs for materials generation, which was kind of similar. We took a text-pretrained LLM off the shelf, like a Llama 2 model at the time, and we fine-tuned it, in this case, on atomistic data represented as text: locations of atoms, energies, and things like this. The resulting system was able to generate inorganic crystals with favorable properties better than these purpose-built approaches and even foundation model approaches that had been trained on that domain-specific data.

One of the takeaways from that project was that the text-based pre-training was an indispensable component in being able to achieve good results on materials generation. We tried to understand also in that paper why that was the case. We were collaborating with some chemists at FAIR in California, and they were just very curious about LLMs and foundation models, so they were willing to humor us a bit and help us see what we could do. But they were skeptical throughout the project until we saw the results, and then it was like, “Well, you can't deny the results are great. Why is this happening?”

Part of it was that, in being able to predict the next tokens in strings, you are learning principles of induction like Occam's razor. So how do those manifest themselves in in-context learning and in fine-tuning? What it means is these models are going to be predisposed to discovering compressible representations, and that means, for example, salient symmetries. This was a problem where there was a rotation invariance, and these models actually were very quick to learn these kinds of invariances because of the text-based pre-training, and it sort of instilled this principle.

And so I think this is also an example of how compression can be a broadly applicable principle for induction. It's at the right level of abstraction that you can start to see more relatively universal behavior. So, for instance, there are some theories on the success of foundation models that suggest that different problems are just different projections of some underlying reality. The Platonic Representation Hypothesis is an example of this.

Mm-hmm.

You can represent an image with pixels or with words and train the respective models on those different modalities, and they learn similar representations. I think that can be true in some instances, but I also think different problems are often truly quite different from each other in terms of their low-level structure. If we have molecules, there's rotation invariance. Maybe in some other problem, some image recognition problem like character recognition, we might have translation invariance. It doesn't matter if the 2 is on the left of the screen or the right of the screen; the label is still a 2.

These are very different low-level feature representations. The architectures that you would typically use for each of those modalities, and to respect those different types of invariances, would look very different. But what they have in common with each other is that they're both ways to compress the respective problems that they're being applied to. So if you have a model that has this compression bias, then it can discover those salient symmetries.

And we had this really surprising finding that vision transformers actually can be more translation-equivariant than convolutional neural networks after training, which sounds impossible because convnets by design are translation-equivariant. But they're not exactly translation-equivariant because of aliasing artifacts, edge effects, and things like this. And so this other model, this transformer with no explicit constraint whatsoever, just a soft bias that manifests itself increasingly at scale, is able to discover a solution that has lower equivariance error than the convolutional neural network, which is just absolutely remarkable.

So to come kind of full circle to your question, I think that we're discovering more and more that being able to solve certain problems really well will translate in perhaps unexpected ways to being able to solve other problems well.

And so I think especially when it comes to things like addition, multiplication, and matrix operations, these are obviously fundamental primitives for building intelligent systems. Even if we can use tools for those things, we want our representations to be somewhat competent at them because it's going to be useful for all sorts of other things we probably haven't anticipated.

Occam's razor has been mentioned multiple times in our last few segments, so I want to dive into that a bit because I love Occam's razor. I first came to really understand it when I became a Bayesian, and it's nice to have a fellow arch-Bayesian to talk with here because, in Bayesian inference, Occam's razor has a very explicit mathematical form. It's marginalization, right?

It's saying that if you have all these parameters around and you compute the average—you do this integration over all these parameters—that's telling you the probability of the model, given your full parameter space having been integrated away. That's really where you get a penalty for complexity coming into play with marginalization. If you expand your model's flexibility or make it more complicated without any gain in inference, that counts against you, or without any gains from some kind of weird simplicity in the form of less curvature or things like that.

I was looking at some of your talks from 5 years ago, these Bayesian deep learning tutorials you had, I think when you first went to NYU. I really enjoyed them. There was a lot of talk about marginalization, its importance, and the intuitions that you gain from that.

More recently, though, it has played less of a role in the conversation, or people have just given up on trying to do the integrals and gone back to doing maximum likelihood, maybe with some hacks in the objective function. I'm curious, as a Bayesian, about your journey from understanding the beauty of marginalization and the Occam's razor built into Bayesian inference to what you do as a practitioner and what you see people doing in practice. How is it playing out in the field, or not? Maybe is there a future in which marginalization and doing these computationally intractable integrals could play a role in driving even better machine learning and inference?

8. Marginalization Encodes Simplicity

I'm so glad you asked. There's almost nothing I like more than talking about Bayesian inference, and I'm so glad you said marginalization because I feel like that's often overlooked. When someone says “Bayesian,” probably the word that comes to most people's minds is “prior,” followed by questions like, “Is the prior good?” and “How do we know what a good prior would be?” But really, the prior is not the defining feature of what it means to be Bayesian.

Instead, being Bayesian means that you want to represent the honest belief that you have uncertainty over which solution is correct, given a finite data sample. If I'm trying to estimate the bias of a coin that I'm flipping, and I flip it once or twice, regardless of whether it comes up heads twice or something like this, I ought to still have some uncertainty over the bias. That uncertainty is manifested through this procedure called marginalization.

We can think of regression to get some intuition for this. We could imagine a bunch of different points on a sort of y–x plot, and maybe the points look roughly like they're on a straight line, but not quite. We could also imagine that there are many different curves that will perfectly run through all of those points, and that all look different from each other. Some of them will be very wiggly, some of them will be slowly moving, and some of them will basically just be a straight line.

We wouldn't be able to say, “We know with 100% certainty that it's this curve that's the right description of our problem.” But that's exactly what we're doing 100 minus epsilon percent of the time when we're training models in deep learning. That is not an honest representation of our beliefs, and it's going to become a bigger and bigger problem the more expressive our model actually is, because that means there are going to be many more different settings, or parameters, that are consistent with what we observe, and we're just betting everything on one of them.

Probability theory says, “No, that's just wrong. That's not what you should be doing.” The sum and product rules of probability say you should be doing marginalization. So all of that is to say that Bayesian marginalization—basically looking at all possible solutions that can be expressed by your model class, weighted by their posterior probabilities—is going to be most important when we have a model that's very expressive and has a lot of parameters, so deep learning, for example, especially relative to the number of data points that we're considering.

I think people often think of it in the opposite way: “Maybe Bayesian methods are most relevant in the realm of classical statistics, like if you're doing some logistic regression and you want to represent some uncertainty, but it's not really suited for deep learning.” It's really the opposite. The challenge, then, is how we do this in a way that's computationally tractable.

With most things in life, the answer, of course, is nuance. It's not that we can either be fully Bayesian or not Bayesian at all. Let's just try to do what makes sense given the resources available to us. If we're just doing standard training, you can actually view that as a very crude form of marginalization, where you're saying, “I'm representing the posterior as a point mass around the most likely setting of parameters.”

Then we can say, “Maybe the posterior”—which is really what the loss functions that we're minimizing are, basically negative log posteriors—“maybe the posterior looks nothing like a point mass, and it looks nothing like a Gaussian. It's very multimodal and very messy.” But we can still do a better job of representing it with a Gaussian than with a point mass, so let's use a Gaussian approximation.

Indeed, when we do that, we often see better generalization because we're representing all these other complementary and compelling explanations for our problem. Then we can keep taking it from there. Maybe we can develop some MCMC procedure that will explore the loss landscape and capture something much richer than just unimodal Gaussian structure, and again we see improvements in performance from doing that.

I actually think that Bayesian methods have been an extraordinary success story in deep learning and beyond, but we don't hear as much about them now as we did maybe 10 years ago. I think there are a number of reasons for that. To some extent, they're a victim of their own success.

There was some low-hanging fruit in being able to do better approximate marginalization in neural nets around 2015, and there was really a lot of progress between about 2015 and 2020 in achieving increasingly better results that would also be computationally efficient. There are procedures, like one we developed called SWAG, for instance, which was based on some insights we had into the structure of these loss landscapes. It allows you to do Bayesian marginalization without any significant additional cost during training. It's a bit more expensive at test time, and so on.

There's also deep kernel learning, which basically just requires a single forward pass through the model, and you get some representation of epistemic uncertainty. These procedures were adopted in practice. If I go to some more domain-specific conference, like a workshop or conference on materials engineering and things like that, I'll see lots and lots of talks using Bayesian optimization, Gaussian processes, and neural networks with an epistemic uncertainty representation.

This is really useful in practice, but I think it's very hard to go beyond the useful approximations we developed without making a really significant, 10-year, moonshot-style investment in those directions, which I think is worth making.

Secondly, there's the advent of LLMs, and that did change things in practice. We went from million-parameter models to billion-parameter models, and the role of epistemic uncertainty is also a little bit less clear, I think, in some instances when we're working with LLMs. This is just a real technical challenge, and I think we're barely starting to scratch the surface of how to think about it.

But the motivation is absolutely there. I think it's really greater with these big models than it is for virtually any other model class. In this sense, I don't know how anyone could not be a Bayesian. How could anyone say, “I don't want to represent epistemic uncertainty”?

I don't like using jargon, so there's aleatoric uncertainty, which means irreducible uncertainty, often associated with noise. Given a data set, we have some noise on that data set, and there's nothing we can do about it. That's aleatoric uncertainty. Epistemic uncertainty is uncertainty that's reducible with more information.

And we believe that uncertainty is there, and there’s even an argument that that might be the only uncertainty in the world, right? Things that seem intrinsically random—like, I could roll a die, and it might seem like there’s a 1-in-6 probability that it’ll land on any one of the sides. But if I had enough information—if I could know exactly the strength of the throw, the wind in the air, the friction on the table, and I had the right physical model—I should be able to predict exactly what side it’s going to land on with each throw. So that’s just saying, okay, the more information we gather, the less uncertainty we have, even for this process that somehow seems intrinsically random.

Physicists, modern physicists now, probably do believe that some aspects of the universe are intrinsically random—radioactive decay and things like this. But many others have not historically believed that; Einstein was a determinist. He would have believed that the only uncertainty there is is epistemic uncertainty. And I think, for practical purposes, that’s not an unreasonable belief, so it’s absolutely crucial that we try to model it in some way. To not do it is just to do something that’s mathematically incorrect and could come at a huge cost.

There’s also a change, I think, that’s happened in this movement to LLMs in the way that people think about data. It used to be the case that you had a fixed data set, and then you threw everything you could at it to get the best possible performance. Maybe you had some scientific problem, and you just really wanted to do well, and you were limited by computation and other things to some extent. But it’s not the case that you have this trade-off to manage, for a given computational budget, between the size of the data and the size of the model. That’s described by these scaling laws, like the Chinchilla scaling laws.

That is starting to become the assumption in mainstream machine learning: that you have almost an arbitrary amount of data, and you have a fixed computational budget, and you want the best possible performance under that computational budget. In that case, rather than trying to be more Bayesian or something like this, it could actually make sense just to use more data.

Well, the other huge benefit, though, is because you mentioned that, for whatever reasons we’re still trying to understand, these large neural networks, these deep neural networks, have this kind of simplicity bias toward simplicity, right? Marginalization is the ultimate Occam’s razor. It’s like Occam’s guillotine or something that really can force models—or force you—to select, let’s say, the simplest model that’s consistent with the data, in a sense. So I think there’s possibly more to gain by trying to understand, in computationally tractable ways, how to pull in more of those effects of marginalization on inference. Is that true?

Absolutely, yes. Bayesian marginalization has this automatic Occam’s razor bias, and I actually started working on Bayesian deep learning, ironically, when I saw a talk on optimization, which might be viewed as almost the opposite of being Bayesian. Jorge Nocedal gave a talk at Cornell, where I was initially a faculty member, and he was presenting on small-batch biases of stochastic optimizers. He had this figure where he was contrasting flat optima with sharp optima, and he had this horizontal translation of the loss landscape. Under that translation, the flat minimum still had reasonably good test loss, but the sharp minimum had very bad test loss.

Right.

He was saying, “Look, this is why we want to do stochastic optimization with small batches, because it’s going to be more likely to find these flat solutions.” In some sense, the other optimizers, like his L-BFGS and so on, were too good. They were converging to these sharp minima. If you just care about minimizing loss, then maybe that’s not bad behavior. But generalization depends on more than just getting a low value of loss, at least with the loss functions that we’re using.

I saw that figure and thought, “Well, this is actually really great motivation to be Bayesian,” because if you’re being Bayesian, you’re not just betting everything on one solution. You’re integrating under a sort of flipped version of that curve. Automatically, most of the volume will be in these flat regions. That’s so much more elegant than approaching this from the optimization perspective, where you have to say, “Okay, I need to rigorously define what it means to be flat.” Maybe it’s going to be the largest eigenvalue of the Hessian, or maybe it’s going to be flatness in—

Each tensor or something like that.

Exactly. Maybe flatness in random directions versus the sharpest directions. Maybe it needs to be parameterization-invariant. That’s often been a criticism of discussions around flatness: most ways of measuring flatness aren’t parameterization-invariant. I have thoughts about that separately, but anyway, you might then look at Fisher matrices or something.

Once you’ve decided on what it means to be flat—and no one will completely agree with you; most people will strongly disagree with whatever you choose—then you have to decide how much you’re going to penalize sharpness. No one will really know what to do about that either. The idea is that you want to build a loss function that is a better proxy for generalization by accounting for things like flatness, but it’s just this really messy rabbit hole.

Whereas if you’re just doing marginalization, this is all happening under the hood. You don’t need to worry about it, and that’s really elegant. Then there’s Occam’s razor in terms of model selection, and I’d recommend that everyone check out Chapter 28 of David MacKay’s book Information Theory, Inference, and Learning Algorithms. It’s titled “Occam’s Razor,” and I don’t agree with some of the stuff in that chapter. In fact, we wrote a paper about Bayesian model selection and marginal likelihoods. But it’s a very beautiful description of automatic Occam’s razor, and he has these extraordinary visualizations to try to demonstrate how that’s possible.

He also has everyday examples, like this idea that maybe you have a block behind a tree. If you had X-ray vision, it could appear to be 2 blocks of equal height and color, or 10 blocks, or something like this. But given that you don’t have X-ray vision and you don’t think this is a trick question, you’d be pretty confident it’s got to be 1 block. This seems like some manifestation of Occam’s razor, and you might try to rationalize this by saying, “Well, it would be just a remarkable coincidence to have 2 blocks standing next to each other of equal height and color.” But he argues that this is actually a quantifiable consequence of doing Bayesian marginalization.

He basically has this conceptualization where you have all possible data sets on a horizontal axis and the probability of generating a particular data set under your model on the vertical axis. That’s the marginal likelihood: the probability that you would generate your training data under your model and prior. If you have the 1-block model, you’re not going to be able to generate very many different data sets, so most of the data sets on the horizontal axis have no support—no p(D | M). But for the ones it can generate, it’s going to have to give a lot of probability, because this is a proper, normalizable probability density.

Similarly, if you have the 10-block model, you can generate all sorts of different types of observations, but you’re going to have to spread that mass more thinly, because this is a proper, normalizable probability density. So for a particular data set that’s consistent with both of these models, like the block-behind-the-tree example, the 1-block model is actually going to have significantly more probability. This is even without factoring in the idea that maybe we have some prior preference for simplicity, and we think that 1 block is more likely than 10 blocks or something like that.

Let’s just forget about this ratio of prior odds and consider only this ratio of marginal likelihoods. It’s a beautiful demonstration of how Bayesian inference automatically encapsulates a notion of Occam’s razor, and this is a fundamental question in science. If you have arbitrarily many hypotheses that are consistent with any number of observations you can ever record, how do you choose between them? What’s the principled way of approaching that problem? The answer, to some extent, I think, is given by Bayesian marginal likelihoods.

There are very subtle issues, I think, with some of the ways that this can be done, and so we wrote a paper all about that. But largely speaking, it’s something that I think people should acquaint themselves with, because it really is getting at something very fundamental, and it has extraordinary practical value.

What was that paper?

The one that we were looking at. We have a paper called “Bayesian Model Selection, the Marginal Likelihood, and Generalization.” That paper is really trying to sell two sides of a story. On the one hand, it’s trying to convince the readers that the marginal likelihood is something quite extraordinary and that there’s a reason we should be interested in it.

Because if it isn't, then if you haven't heard of it, why even read criticisms of it if it's something that doesn't matter? The other part of the paper is questioning whether it's answering exactly the question we want to be asking when we're doing model selection toward trying to achieve the best possible generalization.

And so the question that the marginal likelihood answers is, “What is the probability that my prior generated the training data?” That's different from, “What is the probability that my posterior, after I've observed the data, is going to lead to reasonable predictions?”

Mm-hmm.

And so we can construct examples that really illustrate this difference. You could have a uniform prior over solutions that you expect to be easily identifiable from the data, and so the posterior will contract very significantly around something that will actually be quite reasonable and make reasonable predictions, but the marginal likelihood will be really bad.

And so we construct all sorts of examples where there's actually a misalignment between the marginal likelihood and generalization for these reasons. You could also overfit, even though you have some robustness to overfitting. If you're considering arbitrarily many models, you could just get super unlucky and have a model that's like a point-mass prior around something that can only generate that training data set but isn't going to do anything reasonable.

So the marginal likelihood might prefer that, but it's not going to lead you to a model that's going to make good predictions. It is very useful as a heuristic for model selection in many instances. It's got incredible practical value in learning things like hyperparameters that control complexity in Gaussian processes.

And I think it often is the right tool for scientific hypothesis testing, which is subtly different. This is actually a real historical example. There was a dispute between statisticians—I think Columbia University thought that general relativity was not the explanation for Mercury's irregular orbit. It's called Mercury's perihelion. There must be some hidden planet or some orbital debris or something like that.

And so Bayesian statisticians actually went ahead and computed the marginal likelihood associated with general relativity explaining Mercury's orbit versus these alternative hypotheses, like some modifications to Newtonian gravity, et cetera. And because general relativity was so sharply falsifiable—its predictions were so sharp and consistent with what we observed—it had orders of magnitude greater marginal likelihood than something like a modification to Newtonian physics, where you have to have some distribution over the modification.

That might enable you to explain what we see, but it's also going to generate other data sets, and so its mass is going to be spread more thinly. And so I think that's actually a beautiful demonstration of how the marginal likelihood can be used for scientific hypothesis testing.

Yeah. Are there any other heuristics? We've spoken about marginal likelihood and model complexity and so on. I'm just thinking of the Game of Life. You have all of these simple rules, and what we do there is this sequence of computations, and it's irreducible, as Wolfram would say. We just look at the dynamics, right?

Is there something to that? My intuition is that just by looking at the thing in isolation doesn't tell you something. Actually running it in the real world over several steps of computation—that's where the information is about whether the model is good or not. Is that a fair intuition?

That's a great intuition, and it's connected with the marginal likelihood, sequential coding, and ideas in information theory. This is something we've been thinking about a lot, actually, in trying to go beyond Kolmogorov complexity, and it's very much related to your earlier question about benign overfitting: this observation that models first tend to fit structure, and then they start to fit noise.

And so if we can think of some sort of compute-limited Kolmogorov complexity, maybe we could start to get some idea of how to do model selection. The marginal likelihood is the probability of the data under your model, conditioned on your model. And so you can use the chain rule of probability to write that as the product of P of Dᵢ given D less than i, essentially.

And so you take the log of that, and it looks sort of like the log loss under your training curve as you're observing more and more data. These things are all connected together. I think this is a very reasonable way of trying to understand how to do model selection properly: thinking about compressibility, but also thinking about the dynamics of training, how a model's representation evolves with increases in computation.

Yes. And David Krakauer, when he talks about intelligence, says it's—inference, adaptivity, and representation. But when he was talking about emergence, he said it's about a fundamental reorganization in the microsubstrate.

Taking Navier–Stokes, for example, it's a reorganization where this new high-level description is a better description than at the molecular level. And surely there must be something in training dynamics as well: if there is some kind of emergent behavior, there would be a fundamental reorganization.

Then, with some emergent test optimization where you are looking at the macroscopic behavior, you would actually select that underlying model which generated it.

Judea Pearl

Mm-hmm.

Yeah, I think that type of reorganization is what's happening in double descent and probably grokking too, right?

Oh, is it? That's interesting.

Yeah.

Yeah.

It starts to reorganize. It hasn't found lower loss or anything; it's just that, because of the simplicity bias, all the hyperplanes have started to adjust into a simpler phase of parameter space.

Yeah. We spoke to Daniel Roberts about that. He had that criticality thing during training.

Oh, right. Right.

It was a similar idea.

The theory of deep learning. We talked to these physics guys who were trying to start, if you will, a physics-based perspective on the theory of deep learning, and they had some really interesting points about criticality and things like that.

Yeah, I agree. I think grokking is similar, right? You start forcing more and more data, and essentially, because it can't memorize anymore, it's forced to reorganize into these simpler, more generalizable representations.

Yeah, more training?

Well, more training, and it's also related to the size of the model too, right? As it becomes—if it's too small to really memorize anymore, it has to reorganize.

Yeah.

Well, that's a good question, Jude, because you had this amazing paper out, and we were skimming it earlier. Oh, God, what was it? It's not “Deep Learning Is Not So Mysterious After All.”

Different.

“Not So Different After All.”

No, “Not So Mysterious or Different.”

Yes. You're talking about benign overfitting, double descent, and overparameterization, but would you lump potentially grokking in with that as well?

That's a great question. In the introduction, at the end of the introduction to that paper, I say that I'm not talking about grokking or scaling laws much in this paper because these phenomena are often not treated as particularly mysterious or distinct to neural nets.

But I would say that they fit into this suite of generalization phenomena that we can try to understand using some of the same tools. The generalization bounds that I present in that paper are now being used by us and others to try to understand the root of scaling laws.

I mean, they seem like remarkable laws of nature, almost. If you increase computation by a certain amount, you can predictably improve generalization on a set of tasks. But it's an empirical law, and so we want to understand why. This relates to why larger models might have stronger simplicity biases and things like that.

And so definitely the generalization frameworks that I present in that paper can be used to shed light on scaling-law behavior. Grokking is not something I've thought about too specifically, but it does seem to be the case that by training for longer, the model is doing some reorganization that enables a more compressible solution.

And so it would be very interesting—I'm sure people have done this—to measure the flatness of the solution and things like that as you proceed through grokking. I think this is actually older than people might realize, like many things. Double descent is thought of as a modern deep learning phenomenon, but it was actually first presented in the 1980s.

And so this has been around for a while. I remember Ilya Sutskever and others talking about behavior that was very similar to grokking, where the training loss isn't really changing, but training for longer actually leads to better generalization.

And this relates a little bit to a procedure we called stochastic weight averaging. The idea there is that you want to ramp up the learning rate to a relatively high constant learning rate and then maintain a running average of the parameters as you're traversing this loss landscape with SGD or Adam or whatever else. What happens when you do that is that you're spinning around the periphery of flat solutions. By taking an equal average, you move inside that region and get a much flatter solution.

That reliably led to better generalization, and it's quite convenient because you can load up a pretrained model, increase the learning rate, do this for a certain number of epochs, and get better generalization as a consequence. I think grokking might be related to that sort of behavior as well.

I think this ability to move in parameter space during optimization is also part of the explanation for double descent, right? Because as you provide more and more flexibility, it can shift around a bit. So I'll give you why I think that. You're familiar with integer programming, where you're trying to solve some system of equations and looking for a solution only among integers. You have thousands of integers—an extremely difficult combinatorial optimization problem—because what am I supposed to do? Try out all the different integers? There's no smoothness, et cetera.

People figured out really early on, "You know what? Let's just expand the parameter space from integers to floating-point numbers and do a floating-point optimization. When we get to the end, crystallize it to the nearest integer solution." That turns out to be pretty good.

Mm-hmm.

Pavel Izmailov

I think this ability to move within this numerical space smoothly is what allows it to find reasonable integer solutions. I think that's sort of what happens with double descent too, right?

You're preaching to the choir. So, adding flexibility.

Yes.

Yeah.

So we can have flexibility and biases.

So that you can move, right? You can move around a bit more, maybe by virtue of whatever the structure or the landscape or SGD is. I think there's a paper about these wormholes, about being able to wormhole to a better solution. Because you have a more complex space, and the higher the dimensionality is, the more likely there's a wormhole to a nearby good solution. Maybe it's related to these connected modes.

Exactly. It sounds a bit like mode connectivity.

Pavel Izmailov

Yeah.

So actually, after I heard that talk from Jorge Nocedal about flatness and its role in generalization and its connection to small-batch optimization, I thought, "Well, I want to do Bayesian deep learning." I had done a lot of Bayesian ML, and I had done some deep learning, but I hadn't thought about Bayesian deep learning until I saw that figure in his talk. Before we get into Bayesian marginalization, let's try to understand the geometric properties of these objectives that we're minimizing first, so that we can come up with good posterior approximations. In some sense, this is a bit Bayesian-agnostic. It should be useful even if you're just doing classical optimization.

9. Loss Landscapes Are Connected

The first thing we encountered was this discovery we called mode connectivity, which shows that if you retrain your neural net, for example, with different initializations and find seemingly different modes, you can actually walk from one mode to the other in a subspace without increasing the training loss at all along the way. Before that, it had been believed that the different solutions we would find, for example, by retraining our model, were isolated from one another. You walk in any direction, and you increase the loss a lot along the way. There were some results from Goodfellow and others to suggest that intuition.

We showed that you can introduce very simple parametric curves, like a polygonal chain with just a single turn or a quadratic Bézier curve. You can anchor the endpoints with whatever solutions you find in this procedure that finds your two parameter vectors, W1-hat and W2-hat. As you vary the parameter of this curve, T, from 0 to 1, you walk from one to the other.

The idea is, how do you learn this curve? You can discover these by minimizing your loss uniformly in expectation over the curve. If you're doing classification, this looks like a line integral of cross-entropy loss normalized by arc length. It's actually a pretty simple objective to try to minimize because you could sample uniformly along the curve, T, and then take gradient steps with respect to the parameters of the curve, theta, that you're trying to learn. You can always do this, and the larger you make the model, the less bending you need to do, and the more it looks almost like a straight-line path between the two solutions.

What this showed is that there were these regions within the loss landscape that were extraordinarily flat, so they all had sort of zero loss. What was especially interesting about them was that the different parameters in these mode-connecting curves actually led to models that made very different predictions on the test set. Of course, they're making the same predictions on the training set to have the same loss, but on the test set, they were different representations. This meant that you could ensemble them and get much better performance, for example, just by uniformly sampling on the curves.

This led to the stochastic weight averaging procedure, where we were thinking, "How can we spin around these contiguous regions of flat solutions and find something that's centered within them?" That ended up being fairly practical.

But even then, I suppose, also related to the grokking question, where does the gradient come from? You just said that it's the same on the training data, but on the test data, it's actually behaving differently. Where does the signal come from? What happens when you continue to train a model that has ostensibly converged but is behaving differently on validation?

Mm-hmm.

Pavel Izmailov

Right.

Well, I have a thought on that. Maybe this is helpful. I'm a big fan of Randall Balestriero. His work on A Spline Theory of Deep Learning helped me personally, at least, to understand it better. I think what's happening there is that, if you agree with that and think of these splines as essentially just this unbelievably hyperdimensional honeycomb, with all these shared hyperplanes activating, they're really slowly just moving.

The slopes are just slightly changing, and then they happen to hit a phase change where it's like, wow, now we can combine all these hyperplanes into a much less complex—or, you know, simpler, more parsimonious—kind of combination. That's what it is. I think that's what's happening. It's just slowly shifting, and then it snaps into place, right?

Well, yeah, and I don't know whether you saw his ICML paper from last year, but he had a spline interpretation of grokking. He said exactly that. Basically, during grokking, the splines just suddenly bzoop—

Pavel Izmailov

Mm-hmm.

—and because there are so many of them overlapping, what happens is that it forms this honeycomb where they just compress together.

Pavel Izmailov

Mm-hmm.

I should have said this to David Krakauer, because he said there's a reorganization in the microsubstrate. What's this if it's not a reorganization, where those little hyperplanes, if you like—the honeycomb—zhoop, it just does that during grokking.

Yeah, it's fascinating. Grokking is not something I have thought about too specifically, but it seems to me that you probably are entering some region of the loss landscape that doesn't have a different loss but is providing a representation with very different properties. This does relate to procedures like stochastic weight averaging, where, again, in the end, the solution doesn't really have a different loss than what you would have gotten by training in a standard way, but it has properties that will lead to better generalization, better compressibility, et cetera.

I still don't quite understand where the drive toward more compressible or simpler representations is coming from mechanistically, in the optimization process. I mean, I don't know. Maybe it's just like you said: maybe it requires less floating-point precision, and so that de-optimizes the jitter in the optimizer or something like that. I'm really curious about the actual mechanism that forces you there.

Like, if there is no change in loss—

Mm-hmm.

Or maybe it's so tiny that we just don't really pay attention to it. I don't know, but I'm super curious: what's really driving it downhill, if you will, to a simpler solution?

You might start to—I mean, this is really speculation, and I haven't thought a lot specifically about grokking, but it could be that you're just edging your way inside because of gradient noise and so on, toward a flatter solution as you continue to train.

Yeah, it's really fascinating.

This sort of made me wonder, actually, after the mode connectivity discovery: What do these loss landscapes really look like? The paths that we considered initially were just one-dimensional. Eventually, we started thinking about multidimensional loss volumes. And so we had a paper that was creating a simplex. We were basically adding vertices to the simplex and sampling uniformly within the resulting simplex, and trying to add vertices such that, when we did that, we would have low loss and also maximize the volume of the simplex.

And so that enabled us to find these multidimensional loss surfaces. Then we had these, which all had sort of low loss. We had this picture, I guess, in the front of that paper, where we started with this original understanding that all the local optima are kind of isolated to wormhole-like tunnels between the different optima, and moved to this idea that maybe everything is just connected together in some manifold that's embedded in this really high-dimensional space.

What seems like a sharp or a flat optimum might actually just be how converged you are within that manifold. Because if you're on the edge, it'll look sharp because you move in most directions and you increase the loss quite a bit. But if you move in specific directions, it's very flat. And so I think we still don't have a full understanding of what that looks like, and I think it has really fascinating implications for the generalization behavior.

It's really hard to think in a billion dimensions. Yeah, it really is. Andrew, let's sum up a little bit. Your philosophy is that we should have maximally flexible models with soft regularization. How is that actionable to practitioners? You also gave many practical empirical examples, ensembling models together, and you spoke about this residual pathway prior and so on. Help us understand.

10. Scaling Requires Honest Assumptions

My philosophy is that we should honestly represent our beliefs in the way that we do model construction, and our honest beliefs are usually that the real world is a nuanced place, and we're going to want to have model expressiveness in order to represent that nuance. At the same time, it can't just be flexibility. It has to be flexibility combined with some sort of simplicity bias. It should have some kind of Occam's razor bias.

It turns out, perhaps surprisingly, that making transformers and other types of neural net models larger often actually enhances rather than reduces a simplicity bias. And this is, in many instances, mostly what's been responsible for the better generalization behavior of larger models. So we had this example with double descent, where in the second descent, all the models have basically zero training loss, but the larger models are generalizing better. It can't be because they're more flexible. It has to be because they have some other bias, some sort of simplicity bias.

So how do we operationalize this? I think one way is to embrace expressiveness. Choose a model class that's going to be able to represent lots of different solutions. In terms of the simplicity bias, we're finding empirically that increasing model size can help with that. And so if you're able to build a really big model, you probably should, both for the expressiveness and the simplicity bias.

But what if I'm a researcher who just wants the simplicity bias to go to 11, but I can't afford more parameters? Is there something I can tweak in my objective function to just push me a little bit more toward simplicity without breaking things?

Yes. So there are a lot of things you can do. I think—and we'll talk about them—but the question of how can you more elegantly encode this compression bias beyond just making the model bigger is really a fascinating open research question.

Yeah.

So my contention, and I might be wrong, is that in many cases, a 7-billion-parameter model is not doing better than a 1-billion-parameter model primarily because it's more expressive. It's actually because of the simplicity bias.

Mm-hmm.

And perhaps things like knowledge distillation can help make this argument. If it's possible to really distill a large model into a much smaller model, then it means that the small model has some setting of its parameters that can provide a good approximation to the large model. It's just not able to find those parameters when trained directly on the data. It needs help from the teacher model. And so the question is, can we build the 1-billion-parameter model with some sort of explicit bias that would have otherwise come about through scale in the 7-billion-parameter model—

Mm-hmm.

—and find those solutions itself? I don't think we're anywhere close to being able to do that. I think it's—

Yeah.

It's sort of an open research program. However, there are little things we can do that will help. So there are ways of intervening in the training procedure, like the stochastic weight averaging we discussed, that will help find a flatter, more compressible solution. There are, of course, all sorts of regularizers that can be useful in certain instances. Bayesian marginalization can be very helpful in sort of encoding an automatic simplicity bias in what we do. And so there are all sorts of interventions that will help us with this.

But I think the dream is that maybe we can embrace flexibility in 15 or 20 years from now by building these nonparametric models that really do have an infinite number of parameters and are more expressive than any model we're using right now. But then they have this more explicit compression bias that is interpretable and is getting us what building huge models is inelegantly giving us right now.

Oh, that's interesting. Yeah. I mean, the other kind of interpretation that some folks at home might have based on what you've said is going back to Rich Sutton's “The Bitter Lesson,” right? So he said that designing things is bad. Don't put your symmetries in there. Don't create these multifaceted systems. Just scale and lots of computation is the way forward.

Because it seems like one potential interpretation of what you're saying is that we should create hybrid systems that make models more flexible by combining different modalities. And what you're saying is we should just have bigger models. So are you a Sutton guy or not?

Boris Hanin

I think The Bitter Lesson is widely misunderstood and incomplete. The Bitter Lesson describes how, over short timescales, it's been appealing to try to encode our knowledge into our procedures in order to achieve good results, and that this is also a cognitive bias that people have. They like encoding expertise and things like this, and being thoughtful and elegant about problem-solving.

But over longer timescales than a typical research project, computation becomes cheaper. And so quite often, more brute-force-seeming approaches based on search and learning end up working a lot better than elegantly encoding our priors and our constraints, et cetera. There are a number of examples in the essay, like Deep Blue for playing chess in the late 1990s, and speech recognition, where some researchers were trying to model things like the physiology of the voice box, et cetera, to try to get any possible advantage.

But then they were beaten out by statistical procedures like hidden Markov models and, of course, AlphaGo and other procedures like this, too. And so the takeaway seems to have been that, because computation is becoming cheaper over longer timescales, quite often it's going to be more practical to try to build procedures that are based on search plus learning as opposed to feature engineering.

I think to a large extent this is true, but what it doesn't say is that in order to learn, you need to make assumptions. And so machine learning, as we discussed, is learning by example, and we can't do that without making assumptions. So if we go to this coin-toss example where we're trying to estimate the bias of the coin, you have to make some assumption.

Before I start doing that experiment—

Yeah.

Boris Hanin

—are we assuming a uniform bias? Are we thinking it's looking sort of more like, you know, centered around 0.5 or something like that, like being unbiased, but we have support for other things? These assumptions are going to influence how we do induction. We just can't get away from making assumptions. And so the question is just what assumptions should we make, and to what extent can they be universal?

And when it comes to things like scaling laws that describe how we can reliably improve performance with increases in computation, if we're able to make better assumptions, we can actually change the scaling exponents, which mean that we'll get exponential improvements in performance with increases in computation, which is just remarkable motivation for trying to do this.

And it’s not just a contention that this might be possible. We’re actually starting to see some evidence of this. In our own work, we’ve been very interested in how we can produce structured representations for linear layers in neural nets.

This might sound very abstract, but there’s a very well-known example of doing this. You can start with a fully connected layer, having every possible connection between the nodes in 2 layers. Remove a bunch of the connections and enforce parameter sharing, and you have a convolutional layer.

Mathematically, what you’ve done is replace a dense matrix multiply with a matrix that has sparsity and locality, and that’s a Toeplitz matrix. There’s a question of whether you could actually systematize this process of creating structured layers toward better compute-optimal efficiency, and, if you were to do that, what sorts of principles should we embrace.

Should we embrace things like parameter sharing? Convolution, for example, has parameter sharing. Should we embrace sparsity? Should we embrace other sorts of features?

We introduced this sort of Einstein notation over structured matrices that contains all sorts of different structures as special cases, as well as all sorts of different novel structures, and this taxonomy of interpretable hyperparameters that control the properties of these structures. How fast is it to do a matrix multiply? What’s the rank of the matrix? How much parameter sharing is there? We’d have continuous values for these parameters that would control these things.

What we found is that, in general, toward compute-optimal efficiency, assuming we have as much data as we ever need, parameter sharing was actually not a great principle.

Right.

Boris Hanin

That was a bit surprising. You also want to have full-rank structures, and that moved from surprising to sort of disappointing. It’s like, well, can we go beyond what we’re doing with dense matrices then? Because that’s what we’re using now.

The answer is yes, if you can get faster multiplies. We proposed a structure called a block tensor train, which is related to another structure called a tensor train and another structure called a Monarch matrix. It’s basically a sum of Monarch matrices.

This is full rank, doesn’t have parameter sharing, but you can do a multiply faster than you can with a dense matrix. This allows you to build wider layers for a given computational budget. This did have a meaningful effect on the scaling exponents.

This is at a proof-of-concept level, and a lot of work would need to go into parallelized implementations and so on of these structures. But it showed that it’s possible, and we’re not the only ones.

There are other groups that have been looking at neural tangent kernel–inspired ideas to understand different regimes of learning, like easy versus hard feature learning, and trying to think about what principles are actually going to modify these scaling exponents. Other things, like chunking in transformers and the inductive biases that come with that, are being studied to see whether we can change these inductive biases toward better scaling exponents.

In short, learning requires assumptions. I don’t think we can neatly compartmentalize elegant ideas from having successful learning combined with computation. We really do need to—these things not only aren’t at odds with one another, they strongly go together.

Yeah. When we spoke, out of all of those assumptions, sparsity seems like a really good one. There’s always one thing where you think, “That seems like a really good one.” Daniel Roberts was saying that, in many effective field theories in physics, that’s one of the earliest assumptions: sparsity.

But then you get to other factors to consider, like computational complexity and training tractability, and so on, because in an ideal world, we would make these things sparse. What’s stopping us?

Boris Hanin

It’s a great question. I’ll say that I think convolutions are a great idea. If you’re in the setting I described earlier, where you have a fixed data set and you want to do something reasonable in order to achieve the best performance, this will often work really well.

Even if we’re moving toward soft inductive biases from hard constraints, I would normally advocate for having some sort of convolutional inductive bias, even if it’s not a hard constraint anymore. That’s what we were doing with residual pathway priors, and there’s been some interesting work on convolutional ViTs and things like this to try to have these sorts of soft biases for more efficient learning.

The reason we found, in this instance, that parameter sharing didn’t seem to be a good principle toward better compute-optimal scaling is that we were in a setting where we could have as much data as we ever needed, and there wasn’t much of a generalization gap. We basically just needed to fit the data as efficiently as possible.

This led to the principle of having as many possible parameters per flop. You might wonder, then, “Well, can you go beyond 1 parameter per flop?” I think things like sparse mixtures of experts actually allow you to do this, because you have some sort of gating function that might be acting over E different MLPs, and you only have some subset of them that are active.

You divide your computation by K over E, where K is the number of active experts relative to a model with the same number of parameters. We also tried to push that principle a bit further in this work. Rather than having the mixture-of-experts gating function operate over whole MLPs, it was actually operating over individual linear layers within the MLPs and the attention projection matrices.

You would represent these layers as sums of structured matrices, and then you would have this gating function acting over the rank index of this sum. This would allow much finer-grained routing decisions across the experts and did lead to better efficiency for a given amount of computation.

The short reason, I think, is that we found parameter sharing not to be helpful because you basically just want to reduce your loss as efficiently as possible, and you have as much data as you could ever need, so there isn’t much of a generalization gap.

Also reminiscent of ViT, which surprised everyone because it did so well compared to CNNs. But final question—and we were talking about this in the car on the way over, Andrew—the elephant in the room is GPT-5. We have all these huge, overparameterized models, and on the surface they seem to be doing very well. They’re benchmarking brilliantly, and they’re brilliant to use.

But it feels like there’s something missing. I think they’re not intelligent. I believe you would agree with that statement. What’s missing, and what’s next?

11. AI Should Discover Science

One of the things I’m most excited about is developing AI systems that can discover new scientific theories at the level of general relativity or quantum mechanics. We haven’t even really scratched the surface in being able to do this.

It’s not even clear how data-driven that process would be, or how much it would look like symbolic logic, if we were to think about what Einstein did when he proposed relativity and how we might want to write that down as an algorithm and automate it on a computer.

I would love to see progress in this direction. I think some of the ideas we’ve discussed around compressibility could play an important role in how we think about selecting for scientific hypotheses. There are also ideas around universality, like what sorts of assumptions might be more universal than others and at what level of abstraction.

It’s something that we haven’t really made progress on at all, despite a number of very exciting research projects in AI for science, where neural nets, for example, are being used as black-box function approximators in some sort of pipeline targeted at a very specific application.

I think this is extraordinary work, and it’s really a way in which machine learning is clearly doing a lot of good in the world. But I think it’s time to try to go beyond that paradigm toward really giving us new scientific insights into the data that we didn’t have before.

In fact, that’s something that I’m more excited about than anything else in terms of how technology might develop in the future.

If I were to go a thousand years into the future, one of the first questions I would have is, well, do we understand something about physics that we didn't before? Do we understand how the brain works? This is a conventional approach to science, where the theory is really the quantity of primary interest, and the applications, of course, are important, but they're not primarily why we care about the theory.

GPS would break within minutes if we weren't accounting for gravitational time dilation and general relativity, but Einstein wasn't thinking about GPS when he proposed relativity. If you have the theory, then you can suggest all sorts of applications that otherwise wouldn't have been on the horizon. We probably could train a neural network to correct for gravitational time dilation without understanding what's going on, but that wouldn't be nearly as exciting or useful as having the theory of relativity.

Yeah. Professor Wilson, thank you so much for joining us today. It's been amazing.

Thank you.

Thanks so much. It's been a real pleasure.