🔬搜索所有可能材料的空间——CuspAI 的 Max Welling 教授

I want to think of it as what I would call a physics processing unit, like a PPU, right? You have digital processing units, and then you have physics processing units. It’s basically nature doing computations for you. It’s possibly even the fastest computer known.

swyx

Yeah.

It’s a bit hard to program because you have to do all these experiments. It’s also quite bulky. It’s a very large thing you have to do. But in a way, it is a computation, and that’s the way I want to see it.

You can do computations in a data center, and then you can ask nature to do some computations, right? Your interface with nature is a bit more complicated, but these things will have to seamlessly work together to get to a new material that you’re interested in.

swyx

It’s a pleasure to have Max Welling as a guest today. Max has done so much over his career that I’ve been excited about. If you’re in the deep learning community, you probably know Max for his work on variational autoencoders, which has literally stood the test of time—officially stood the test of time.

If you are a scientist, you probably know him for his pioneering work on graph neural networks and equivariance. And if you’re in materials science, you probably know him for his new startup, CuspAI. Max has a long history of doing lots of cool problems. You started in quantum gravity, which is very different from all these other things you’ve worked on.

The first question, for AI engineers and scientists: What is the thread in how you think about problems? What is the thread in the types of things that excite you? And how do you decide what the next big thing you want to work on is?

1. Curiosity Meets Real World Impact

It has actually evolved a lot. In my younger days, let’s put it that way, I would just follow what I found super interesting. I have this sensor that I think many people have, but maybe don’t really use very much, which is that you get this feeling of getting very excited about some problem.

It could be what’s inside of a black hole, what’s at the boundary of the universe, or what quantum mechanics is actually all about. And so I followed that basically throughout my career, but I have to say that as you get older, this changes a little bit.

swyx

Mm-hmm.

In a sense, there’s a new dimension coming to it, and this is impact.

swyx

Mm-hmm.

Working in 2D quantum gravity, you pretty much guarantee there’s going to be no impact in what you do relative to this world. Maybe a few papers, but not in this world, at this energy scale.

As I get closer to retirement—which is fortunately still 10 years away or so—I do want to make a positive impact in the world. I got pretty worried about climate change, and politics seems to have a hard time solving it, especially these days. So I thought I’d better work on it from the technology side, and that’s why we started CuspAI.

But there are also a lot of really interesting science problems in materials science. It’s combining both the impact you can make with it and the interesting science. So it’s these 2 dimensions: working on things where you feel, “Oh, there’s something very deep going on here,” and, on the other hand, trying to build tools that can actually make a real impact in the world.

Alessio Fanelli

The thread, when I look back at the different things that you’ve worked on, is that some of them seem pretty connected, like the physics to equivariance and graph neural networks, maybe. And that seems to be somewhat related to CuspAI. Do you have a thread through there?

2. Physics Connects AI And Science

Yeah. I think physics is the thread. Having spent a lot of time in theoretical physics, I think there are, first, very fundamental and exciting questions—things that haven’t actually been figured out in quantum gravity—so that is really the frontier.

There are also a lot of mathematical tools that you can use. In particle physics, but also in general relativity, symmetries play an enormously important role, and this goes all the way to gauge symmetries as well. Applying these kinds of symmetries to machine learning was a very deep and interesting mathematical problem.

I did this with Taco Cohen, and Taco Cohen was the main driver behind this. It went all the way from simple rotational symmetries to gauge symmetries on spheres and stuff like that. Maurice Weiler, who’s also here, wrote an entire book when he was a PhD student with me, which I can really recommend, about the role of symmetries in AI and machine learning. I find it’s a very deep and interesting problem.

More recently, I’ve taken a different path: the relationship between diffusion models and a field called stochastic thermodynamics. This is basically thermodynamics, which is a theory of equilibrium, but formulated for out-of-equilibrium systems.

It turns out that the mathematics we use for diffusion models—and even for reinforcement learning, Schrödinger bridges, and MCMC sampling—has the same mathematics as this physical theory of nonequilibrium systems. That got me very excited.

Actually, when I taught a course in Muizenberg, South Africa, close to Cape Town, at the African Institute for Mathematical Sciences, or AIMS, I turned that into a book. Two years later, the book is finished. I’ve sent it to the publisher, and this is about the deep relationship between free energy, diffusion models—basically generative AI—and stochastic thermodynamics.

I find physics very deep. I also think a lot about quantum mechanics.

swyx

Yeah.

It’s a completely weird theory that nobody really understands.

swyx

Mm-hmm.

There’s a very interesting story that may be good to tell to connect my PhD back to where I am now. I did my PhD with a Nobel laureate, Gerard ’t Hooft. He’s just the most brilliant man I’ve ever met. He was never wrong about anything, as long as I’ve seen him, and now he says quantum mechanics is wrong and has a new theory of quantum mechanics.

Nobody understands what he’s saying, even though what he’s writing down is not mathematically very complex. But he’s trying to address the understandability, let’s say, of quantum mechanics head-on. I find it very courageous, and I’m completely fascinated by it.

So I’m also trying to think: Can I actually understand quantum mechanics in a more mundane way, without all the weird multiverses and collapses and stuff like that? Physics has always been the thread, and I am trying to apply physics to machine learning to build better algorithms.

swyx

You’re still very involved in understanding physics and the world, even beyond just applications to machine learning or introducing new formalisms. That’s really cool.

Yes, I would say I’m not contributing much to physics, but I’m contributing to the interface between physics and science, and that’s called AI for science or science for AI. It’s actually a new discipline that’s emerging.

It’s not just emerging; it’s exploding, I would say. That’s the better term, because investments have gone from the hundreds of millions into the billions. There’s now actually a startup by Jeff Bezos that has a $6.2 billion Series C round, right? It’s insane. This is the largest startup ever, I think, and that’s in this field, AI for science. It tells you something: we’re creating a new bubble here.

swyx

Yeah.

Right?

swyx

So why do you think it is? What has changed that has motivated people to start working on AI-for-science-type problems?

There are 2 reasons, actually. One is that people have been applying the new tools from AI to the sciences, which is quite natural. Of course, I think there are 2 big examples.

Protein folding is, of course, a big one, and the other one is machine-learning force fields, or what are sometimes called machine-learning interatomic potentials. Both of them have been very successful. Both also have something to do with symmetries, which is also cool.

People in AI for science saw an opportunity to apply the tools they had developed beyond ad placement or multimedia applications to something that could actually make a very positive impact in society, like health, drug development, materials for the energy transition, and carbon capture. These are all really cool, impactful applications.

swyx

Yes.

Besides that, the science itself is also very interesting. The fact that these 2 fields are coming together, and that we’re now at the point where we can actually model these things effectively and move the needle on some of these scientific methodologies, is also a very unique moment, I would say.

And people recognize that, okay, now we're at the cusp of something new. This is also what our company is called after. We're at the cusp of something new, and of course, that always creates a lot of energy. It's like, okay, there's something. It's sort of a virgin field, right? It's like nobody's been there. I can rush in, and I can start harvesting there, right?

Yeah. Yeah.

And I think that's also what's causing a lot of enthusiasm in the field.

If you're an AI engineer—which many of the people who listen to this podcast will be—and you maybe don't have a strong science background but are excited, how does somebody who is not a scientist on a day-to-day basis get involved? Most, I would say, most AI practitioners, be they engineers or scientists, would consider themselves scientists, and they have some background—a little bit of physics, a little bit of chemistry—in college, maybe even graduate school, and have been working or are starting out. How does somebody who is not a scientist on a day-to-day basis get involved?

Well, they can read my book once it's out. We should create curricula that are on this interface. I'm not sure. There are possibly already, at some universities, actual courses you can take—maybe online courses you can take. These workshops where we are now are actually very good as well, and we should probably have more tutorials before the workshop starts. Actually, I've proposed this at some point: maybe first have an hour of a tutorial so that people can get new people into the field.

But yeah, there's a lot out there. Most of it is, of course, inaccessible. But I would say we will create many more books and other content—

Alessio Fanelli

Yeah.

—that is more accessible, including this podcast, I would say. I think it will come. These days, you can watch videos and things. There's a huge amount of content you can go and see.

swyx

So maybe a follow-up to that. How do people learn and get involved, but why should they get involved? I mean, a lot of people in our audience will be interested in AI engineering, but they may be looking for bigger impacts in the world.

Yeah.

swyx

What opportunities does AI for science provide to make an impact, to change the world, that working in the world of pure bits would not?

3. Materials Become A Search Engine

My view is that underlying almost everything is a material.

swyx

Yeah.

We're focusing a lot on LLMs now—

swyx

Yeah, yeah.

—which is kind of the software layer. But I would say, if you think very hard, underlying everything is a material. I was saying, you know, there's the LLM, and underlying the LLM is a GPU on which it runs. In order to make that GPU, you have to put materials down on a wafer and shine it with EUV light in order to etch the structures in. But that's now an actual material problem because, more or less, we've reached the limits of scaling things down, and now we're trying to improve further with new materials. So that's the fundamental materials problem.

We need to get through the energy transition fast if we don't want to mess up this world.

swyx

Mm-hmm.

For instance, there are batteries. That's a complete materials problem, right? There are fuel cells. There are solar panels. They can now make solar panels with new perovskite layers on top of the silicon layers that can capture, theoretically, up to 50% of the light, whereas now we're at, I don't know, maybe 22% or something.

swyx

Yeah.

Right? So these are huge changes, all through material innovation. And yeah, I think wherever you go, I can probably dig deep enough and then tell you, “Well, actually, the very foundation of what you're doing is a material problem.”

I think it's just very nice to work on this very foundation, also because I think this is maybe something that's happening now: we can start to search through this material space. This has never been the case, right? The normal way of working is you read papers, and then you come up with a hypothesis, you do an experiment, and you learn, et cetera. So that's a very slow process.

Now we can treat this as a search engine. Like we search the internet, we can now search the space of all possible molecules—not just the ones that people have made or that are in the universe, but all of them.

swyx

Yeah.

Right? And we can make this kind of fully automated. That's the hope, right?

swyx

Yeah.

We can just type what we want. It becomes a tool where you type what you want, and something starts spinning—

swyx

Mm-hmm.

—and some experiments get going.

swyx

Yeah.

Right? Then out comes a list of materials, and you look at it and say, “Maybe not,” and then you refine your query a little bit.

swyx

Yeah.

You kind of do research with this search engine, where a huge amount of computation and experimentation is happening somewhere far away, in some lab or some data center or something like this. I find this a very, very promising view of how we can build a much better materials layer underneath almost everything, and also more sustainable materials.

Our plastics are polluting the planet, right? If you can come up with a plastic that destroys itself after, I don't know, a few weeks, and actually becomes a fertilizer, these are things that are not impossible at all. These things can be done, right? And we should do it.

swyx

Can you tell us just generally a little bit about CuspAI? Then I have a ton of questions.

4. CuspAI Targets Carbon Removal

Yeah. So CuspAI started about 20 months ago because I was worried about—I’m still worried about—climate change.

swyx

Mm-hmm.

I realized that in order to stay within 2 degrees, let's say—

swyx

Mm-hmm.

—we would not only have to reduce our emissions to zero by 2050, but then have another half-century or even a century of removing carbon dioxide from the atmosphere—not by reducing our emissions, but actually removing it at a rate that's about half the rate at which we now emit it.

That is an unsolved problem. And if we don't solve it, 2 degrees is not going to happen, right? It's going to be much more. I don't think people quite understand how bad that can be—4 degrees, very bad.

So this technology needs to be developed, and this was my motivation, along with my co-founder, Chet Edwards, to start this startup. We also saw that the technology was ready, which is very good. If you're going to do it, the time is right.

In the meanwhile, we've grown to about 40 people. We've collected $130 million in investment into the company, which for a European company is quite a lot. It's interesting that right after that, other startups got even more, so that tells you how fast this is growing.

We are now at the point where we've built the platform, of course, but it's for a series of material classes, and it needs to be constantly expanded to new material classes. It can be more automated because we're now putting LLMs in, so the whole thing gets more and more automated.

Now we're moving to high-throughput experimentation, connecting the actual computational platform to the experiments so that you can also get fast feedback from experiments. I kind of think of experiments as something you do at the end, although that's what we've been doing so far. I want to think of it as what I would call a physics processing unit, like a PPU.

swyx

Yeah.

You have digital processing units, and then you have physics processing units. It's basically nature doing computations for you. It's the fastest computer known, possibly even.

It's a bit hard to program because you have to do all these experiments. It's also quite bulky. It's a very large thing you have to do.

swyx

Yeah.

But in a way, it is a computation, and that's the way I want to see it. You can do computations in a data center, and then you can ask nature to do some computations, right? Your interface with nature is a bit more complicated. But then these things will have to seamlessly work together to get to a new material that you're interested in. That's the vision we have.

We don't say superintelligence because I don't quite know what it means, and I don't want to oversell it. But I do want to automate this process and give a very powerful tool to chemists and materials scientists.

swyx

That actually brings up a question I wanted to ask you. First of all, can you talk about your platform to whatever degree you can? Explain how it works and what your thought processes were in developing it?

5. Inside The Automated Materials Platform

Yeah. Actually, it's been surprisingly—it's not rocket science, I would say.

It's not rocket science in the sense of the design. Basically, the design that I wrote down at the very beginning is still more or less the design, although you add things. I wasn't thinking very much about multi-scale models, and it came on our radar that multi-scale is actually very important. In the beginning, I wasn't thinking very much about self-driving labs, but now I think we're at the stage where we should be adding that. So there are bits and details that we're adding.

But more or less, it's what you see in the slide decks here as well: there's a generative component that you have to train to generate candidates, and then there's a digital twin, multi-scale, multi-fidelity—

swyx

Mm-hmm.

…digital twin, which you walk through the steps of the ladder. They do the cheap things first. You weed out everything that's obviously not useful, and then you go to more and more expensive things later. So you narrow things down to a small number. Those go into an experiment: you do the experiment, get feedback, et cetera.

Now, things that have also been added more recently are more agentic parts. We have agents that search the literature and come up with chemical suggestions for doing experiments. We have agents that autonomously orchestrate all of the computations and experiments that need to be done. They're in various stages of maturity, and they can be continuously improved, I would say. So that's basically it. I don't think that part is rocket science. But the design of that thing—

swyx

Mm-hmm.

…is not surprising. What's surprising is that it's surprisingly hard to actually build it, right? So that's the thing where the moat is in the data that you can get your hands on and in actually building the platform.

swyx

Yeah.

And I would say there are 2 people in particular I want to call out: Felix Hunke, who is actually building the scientific part of the platform, and Alessandro De Maria, who is building the MLOps part of the platform.

Recently, we also added Aaron Walsh to our team, who is a very accomplished scientist from Imperial College. We're very happy about that. He's going to be our chief science officer. We also have a partnerships team that seeks out all the customers, because I think this is one thing I find very important.

It's so complex to actually bring a material into the real world that you must do this in collaboration with the domain experts, which are typically the companies. So we only start to invest in a direction if we find a good industrial partner to go on that journey with us.

swyx

Makes a lot of sense. Over the evolution of the platform, did you find that human intervention—human feedback—was needed at different steps? I guess you could start out with a purely automatic process. You could imagine 2 directions: 1, you start out making everything purely automatic—automated, agentic, and so on.

🔬Searching the Space of All Possible Materials

Mm.

swyx

Or maybe did you start out with having human feedback at lots of steps and then figure out ways to—

🔬Searching the Space of All Possible Materials

Yeah.

swyx

…remove—

🔬Searching the Space of All Possible Materials

That's it. It's the second one. So you build tools.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

It's much more modular than you think.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

We need these tools for this application; we need these tools for that application. So you build all these tools, and then you go through a workflow.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

Actually, in the beginning, manually. You put them together: “Okay, now first this tool, then run this tool, then run this one,” et cetera. So you put them in a workflow.

swyx

Mm-hmm.

🔬Searching the Space of All Possible Materials

Then you figure out, “Oh, actually, this porous material that we're trying to make collapses if you shake it a bit. Okay, then you add a new tool that says, ‘Test for stability,’” right?

swyx

Yeah.

🔬Searching the Space of All Possible Materials

So there are more and more tools, and then you build the agent, which could be a Bayesian optimizer, or it could be an actual LLM, maybe trained to be a good chemist, that will then start to use all these tools in the right way, in the right order.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

Right? But in the beginning, it's like you, as a chemist, are putting the workflow together.

swyx

Yeah, yeah.

🔬Searching the Space of All Possible Materials

Then you think about, “Okay, how am I going to automate this?” right?

swyx

Yeah.

🔬Searching the Space of All Possible Materials

One very easy question you can ask yourself is: every time somebody who is not a super expert in DFT wants to do a calculation—

swyx

Yeah.

🔬Searching the Space of All Possible Materials

…they have to go to somebody who knows DFT.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

Could you start to automate that away? Make it so user-friendly that you actually do the right DFT for the right problem and for the right length of time, and you can actually assess whether it's a good outcome, et cetera. So you start to automate smaller pieces and bigger pieces, et cetera. In the end, the whole thing is automated.

swyx

So your philosophy is that you want to provide a set of specific tools that make it so the scientists making decisions are better informed, rather than trying to create an automated process.

🔬Searching the Space of All Possible Materials

I think this is sort of the same as what you're saying because, yes, we want to automate. But we don't see a scenario very soon where the chemist and the domain expert are out of the loop.

But it's a retreat, right? It's like, okay, so first you needed an expert to tell you precisely how to set the parameters of the DFT calculation.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

Okay, maybe we can take that out.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

We can maybe automate it, right? So increasingly, more of these things are going to be removed.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

In the end, the vision is that it will be a search engine where somebody—a chemist—will type things in and get a list of candidates, but the chemist will still decide what is a good material and what is not a good material out of that list, right?

And so the vision of a completely dark lab—

swyx

Uh-huh.

🔬Searching the Space of All Possible Materials

…where you can close the door and you—

swyx

Yeah.

🔬Searching the Space of All Possible Materials

…and you just say, “Just find something interesting,” and then it will figure out what's interesting, and it will say, “Oh, I found this new material to blah, blah, blah, blah,” right? That's not the vision I have.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

At least not for—I don't know—a long time. So for me, it's really about empowering the domain experts who are sitting in the companies and in the universities to be much faster in developing their materials.

I should say it's also good to be a little humble at times, because it is very complicated to make a material and bring it into the real world. There are people who are doing this for their entire lives.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

Right? And I wonder if they scratch their heads and say, “Well, how are you going to completely automate that away in the next 5 years?” I don't think that's going to happen at all. So, to me, it's an increasingly powerful tool in the hands of the chemists.

Alessio Fanelli

I have a question. You've talked before about getting people interested based on having a big breakthrough in materials versus incremental change. I'm curious what you think about the platform you have now and are stepping toward. Are you chasing the big change, or is this incremental? They're not mutually exclusive, obviously, but what do you think about that?

🔬Searching the Space of All Possible Materials

We follow a mixed strategy, so we're definitely going after a big material. Again, we do this with a partner. I'm not going to disclose precisely what it is, but we have our own long-term goal. You could call it a lighthouse, or a moonshot, or whatever.

It is going to be a really impactful material that we want to develop as a proof point that it can be done, that it will make it into the real world, and that AI was essential in actually making it happen.

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

At the same time, we're also quite happy to work with companies that have more modest goals.

Alessio Fanelli

Mm-hmm.

🔬Searching the Space of All Possible Materials

One is a very deep partnership where you go on a journey with a company.

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

That's a long-term commitment together. The other one is somebody saying, “I need a force field. Can you help me train this force field and then maybe analyze this particular problem for me?”

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

“ I'll pay you a bunch of money for that, and then maybe after that we'll see.” And that's fine, too, right? But we prefer the deep partnerships—

where we can really change something for the good.

Alessio Fanelli

Yeah. And do you feel like, from a platform standpoint, you're ready for that? Or what are the things that—

🔬Searching the Space of All Possible Materials

Mm.

Alessio Fanelli

And, again, not asking you to disclose proprietary secret sauce, but—

🔬Searching the Space of All Possible Materials

Yeah.

Alessio Fanelli

What are the things, generally speaking, that need to happen from where we are to get those big breakthroughs, I guess?

🔬Searching the Space of All Possible Materials

What I find interesting about this field is that every time you build something, it's actually immediately useful.

Alessio Fanelli

Mm-hmm.

🔬Searching the Space of All Possible Materials

Right? And so, unlike quantum computing or nuclear fusion, where you work for, I don't know, 20, 30, 40 years and nothing, nothing, nothing, nothing happens. And then it has to happen. Right? And when it happens, it's huge.

Alessio Fanelli

Mm-hmm.

🔬Searching the Space of All Possible Materials

So it's quite different here. You go to a customer and say, "What do you need?" We work, let's say, on a problem like water filtration. We want to remove PFAS from water.

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

So we do this with a company, Kamira. They are a deep partner for us, right? We're on a journey together. I think that the breakthrough will happen with a lot of humans in the loop because there are chemists who have a whole lot more knowledge of their field, and it's us who will help them with AI, training AI, and new methods.

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

In that interface and these interactions, something beautiful will happen. And that will have to happen first before this field will really take off, I think. So, in that sense, it's not a bubble, let's put it that way.

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

As people see that what's happening is actually real, in the beginning it will be very much with a lot of humans in the loop.

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

I would say, and I would hope, we will have this new sort of breakthrough material before everything is completely automated, because that will take a while. Also, it is very vertical-specific. Completely automating something for problem A, you can probably achieve it.

Alessio Fanelli

Mm-hmm.

🔬Searching the Space of All Possible Materials

But then you'll have to start over again for problem B because your experimental setup looks very different. The machines that you use to characterize your materials look very different. Even the models in your platform will have to be retrained and fine-tuned to the new class. So every time you have a lot of learnings to transfer, but the problems are actually different.

Alessio Fanelli

Yeah. Yeah.

🔬Searching the Space of All Possible Materials

And so I would want that breakthrough material before it's completely automated, which I think is kind of a long-term vision. Every time you move to something new, you'll have to start retraining, and humans will have to come in again and say—

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

"Okay, so what does this problem look like?" And now—

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

—you point the machine again in the new direction, and then use it again.

Alessio Fanelli

For the non-scientists amongst us, me included, and a bit of a scientist, there's a lot of terminology. You mentioned DFT. Equivariance we've talked about.

🔬Searching the Space of All Possible Materials

Mm.

Alessio Fanelli

Can you explain, in engineering terms, or at the level of sophistication of engineering, what equivariance is?

6. Equivariance Cuts The Data Burden

🔬Searching the Space of All Possible Materials

Equivariance is the infusion of symmetry in neural networks. If I build a neural network, let's say, that needs to recognize this bottle, and then I rotate the bottle, it will actually have to completely start again because it has no idea that the rotated bottle—the input that represents a rotated bottle—is actually a rotated bottle. It just doesn't understand that.

Whereas if you build equivariance in, once you've trained it in 1 orientation, it will understand it in any other orientation. That means you need a lot less data to train these models. These are constraints on the weights of the model. Basically, you have to constrain the weights such that it understands it, and you can build it in; you can hardcode it in.

The symmetry groups can be translations and rotations, but also permutations. In graph neural networks, there are permutations. In physics, of course, there are many more of these groups.

Alessio Fanelli

To play devil's advocate, why not just use data augmentation by—

🔬Searching the Space of All Possible Materials

Yeah.

Alessio Fanelli

—your bottle is in all the different orientations?

🔬Searching the Space of All Possible Materials

It's an option, but it's just not exact. Why would you go through the work of doing all that when you would really need an infinite number of augmentations to get it completely right, when you can also hardcode it in?

Now, I have to say, sometimes data augmentation actually works even better than hardcoding the equivariance in. This has to do with the fact that if you constrain the weights before the optimization starts, the optimization surface or objective becomes more complicated, and so it's harder to find good minima. There is also a complicated interplay, I think, between the optimization process and these constraints you put in your network.

You'll hear contradicting claims in this field. Some people say that, for certain applications, it works just better than not doing it. Sometimes you hear other people say that if you have a lot of data and you can do data augmentation, then actually it's easier to optimize them, and it actually works better than putting the equivariance in.

Alessio Fanelli

Do you think there's kind of a bitter lesson for mathematically founded models and strategies for doing deep learning?

🔬Searching the Space of All Possible Materials

Ultimately, it's a trade-off between data and inductive bias.

Alessio Fanelli

Yes.

🔬Searching the Space of All Possible Materials

If your inductive bias is not perfectly correct, you have to be careful because you put a ceiling on what you can do. But if you know the symmetry is there, it's hard to imagine there isn't a way to actually leverage it.

But, yeah, there is a bitter lesson. One of the bitter lessons is that you should always make sure your architecture scales, unless you have a tiny data set, in which case it doesn't matter. The same bitter lessons you can draw in LLM space are eventually going to be true in this space as well, I think.

Alessio Fanelli

Yeah.

🔬Searching the Space of All Possible Materials

Yeah.

swyx

Can you talk a little bit about your upcoming book and tell the listeners what's exciting about it? Why they should read it?

🔬Searching the Space of All Possible Materials

Yeah.

swyx

So this book is about—

7. Generative AI Meets Thermodynamics

🔬Searching the Space of All Possible Materials

It's called Generative AI and Stochastic Thermodynamics. It basically lays bare the fact that the mathematics that goes into both generative AI, which is the technology to generate images and videos, and this field of nonequilibrium statistical mechanics, which deals with systems of molecules that are just moving around and relaxing to their ground state, or that you can control to have them be in a certain state, is actually identical. That's fascinating.

In fact, Geoff Hinton and Radford Neal already wrote down the variational free energy for machine learning a long time ago, and there's also Karl Friston's work on the free energy principle and active inference. But now we've related it to this very new field in physics called stochastic thermodynamics, or nonequilibrium thermodynamics, which has its own very interesting theorems, like fluctuation theorems, which we don't typically talk about but can learn a lot from.

I think it can now start to cross-fertilize. When we see that these things are actually the same, we can, like we did for symmetries, look at this new theory that's out there, developed by these very smart physicists, and say, "Okay, what can we take from here that will make our algorithms better?" At the same time, we can use our models to help the scientists do better science. It becomes a beautiful cross-fertilization between these 2 fields.

swyx

Yeah.

🔬Searching the Space of All Possible Materials

The book is rather technical, I would say. It takes all sorts of things that have been done in stochastic thermodynamics and all sorts of models that have been done in the machine learning literature, and it basically equates them to each other. Hopefully, that sense of unification will be revealing to people.

swyx

Yeah. Wait, and when is it out?

🔬Searching the Space of All Possible Materials

Well, it depends on the publisher now.

swyx

Okay.

🔬Searching the Space of All Possible Materials

But I hope in April. I'm going to give a keynote at ICLR, and it would be very nice if I have this book in my hand, but it's hard to control these kinds of timelines.

swyx

Yeah. I'm looking forward to it.

🔬Searching the Space of All Possible Materials

Great.

Alessio Fanelli

Likewise.

🔬Searching the Space of All Possible Materials

Thank you very much.