He won a Nobel here for AlphaFold. Then he left. - John Jumper

I don't really love the bitter lesson as people try to apply it. In fact, AlphaFold 2 is the opposite of that.

Protein folding is one of these holy-grail-type problems in biology.

We predict nature-level science with the press of a button, in a very narrow category of nature-level science: the structure of a specific protein.

John Jumper led the team behind AlphaFold, the system that predicted 200 million protein structures. In 2024, he won the Nobel Prize in Chemistry, and now Jumper is leaving DeepMind. But what did AlphaFold solve? What remains unsolved? And could AlphaFold be the template for AI for science?

We are not trying to tell you everything. We are not a model of the entire cell. You try it, you measure. Nine times out of 10, you find out you're wrong, right? If you're wrong nine times out of 10, you're a very successful machine learner. You're incredibly productive.

For half a century, structural biology had a massive bottleneck. DNA was easy to read, but protein structures were not. A protein structure begins as a chain of amino acids, and then, often with help from the cell, they settle into a three-dimensional shape. That shape determines what it binds, what chemistry it catalyzes, where it sits in the cell, and whether it even works at all. But from a machine-learning perspective, if you only have the sequence, can you predict the fold? Can you predict the structure?

We've discovered more about the world than any other civilization before us, but we have been stuck on this one problem: How do proteins fold up?

Every 2 years, there's a big scientific experiment called CASP. Essentially, teams from around the world gather to see if they can predict protein structure from sequences, based on recently completed but not yet publicly available experiments. For many decades, the progress was incremental, until 2020, when John Jumper's team, AlphaFold, produced a result that was significantly better than the competition. For many single-chain targets, the predictions from AlphaFold were so close to the targets that the organizers of the event said the problem had been essentially solved. A protein structure that might have taken a year of specialist work can now be predicted and operationalized in minutes.

Good job, everyone. The whole team. It's been an incredible effort.

John Moult

Congratulations on this work. It is really outstanding.

David Baker

AlphaFold represents a huge leap forward that I hope will really accelerate drug discovery and help us to better understand disease. It's so mind-blowing.

These results were, for me, having worked on this problem for so long, after many, many stops and starts and wondering whether this would ever get there, suddenly a solution. We solved the problem.

Fair play to DeepMind. They could have kept this close to their chest, but they decided to release it. They released a database with over 200 million predicted protein structures. Now, it's important to emphasize the word “predicted.” These are not experiments, but they are the basis for a lot of interesting new research that the world of biology can now perform. Today, AlphaFold is used by more than 3 million people in over 190 countries.

In 2024, the Nobel Prize Committee made a formal verdict. Half of the Chemistry Prize went to David Baker for computational protein design, and the other half went to Demis Hassabis and John Jumper for protein-structure prediction. AI had become a new tool for chemists, a way of seeing molecular structure at a level of resolution that structural biologists couldn't even have dreamed of before.

It's absolutely wonderful to be here. It's truly an extraordinary honor to tell you about this work, to tell you about the work of our team in protein-structure prediction. By about 10:30, I said, “Oh, well, I guess not this year,” and I told my wife. She goes, “No, no, wait.” As she's telling me to wait, my phone lights up with a phone call from Sweden. Thankfully, it was not the world's meanest prank call.

All of this makes John's next move very interesting. Just a few days ago, he announced his departure from Google, and he's going to Anthropic. It's important to note that John Jumper was not building generic prediction architectures like Claude or Gemini. It was extremely structured, designed, and engineered for the purpose of doing a specific thing. Why might that be interesting to Anthropic? We can only guess.

This isn't just about Google DeepMind, the CASP competition, and whatnot. There are now structural biologists around the world who can innovate, build new products, and potentially save lives because they have access to this protein database. I spoke with Emmanuel Nee. He is a structural biologist based in Africa.

Emmanuel Nee

I went back and did just one protein purification, collected the data, and we AlphaFolded. In combination, I got the structure in less than 2, 3 months.

We're talking about potentially years of work compressed into several months. That's pretty good.

Agents are getting smarter every day, but even the best agents get stuck without good context, and this is where Notion comes in. With the recent launch of custom agents, Notion becomes the collaborative AI platform where agents and humans work side by side. And now their new developer platform is turning that into infrastructure that developers can work on. The way I think about it is it is the kind of curated materialized memory plane for all of my work. And the great thing about Notion is it's so easy to work with programmatically. It has a CLI, it has an MCP server, um it has agents built into it, right? So, that means using my phone, I can ask an agent to go and do some research or to put some information in there. I can sync things to my calendar. Um without Notion, I honestly don't think I'd be able to do anything that I do on MLST, so it's really, really good. So, I highly recommend you give Notion's platform a go. You can sign up at notion.com/mlst, and you'll be supporting the show if you do. And now, let's get back to John Jumper. We filmed this before the Anthropic announcement, so it was really cool to speak with John. He's such an inspirational guy. I was lucky enough to have dinner with him the night before, so we had a bit of a warm-up conversation, and we drilled into the various topics we wanted to discuss.

One thing that struck me is that John is unusually careful about what AlphaFold does and does not solve. He doesn't sell it as a model of life or as a model of curing disease. He sells it as something a little bit narrower, and possibly more radical, actually: a machine that predicts one class of structural-biology measurement well enough to change what scientists can do next. So now I give you John Jumper.

AlphaFold itself is this kind of landmark in AI and science, but it's really about how we use AI to solve problems that humans can't, that are really hard, where we go and do years-long experiments. In the case of AlphaFold, it's this problem of protein-structure prediction.

This is Machine Learning Street Talk, not Biology Street Talk. DNA is the instruction manual for life, but what does it actually tell you to build? One of the many things it tells you is how to build proteins. These are little nanomachines, a couple thousand atoms in the cell, that actually do the work of the cell.

Three letters of your DNA tell you how to add one extra piece to this protein. This protein is kind of a long string, and there's a tiny machine itself, made of proteins and RNA, that's built one kind of string at a time. You make this rope of 20 types of chemical groups, so it's kind of 20 types of letters, and people of course use the alphabet for these things.

Each of them is quite different, right? My PhD supervisor could tell you lovingly about what's special about each one. But you build this kind of rope of the protein, and then it assembles itself. It twists, it curls, it folds up into a really compact and interesting shape. It has these helices, sheets, and all these things. That's actually what works.

The analogy I always like to use is that it's like you have an IKEA bookshelf, and you open the box and it builds itself. This really central problem for maybe 70-plus years in biology is: I can read DNA—in fact, I can read DNA really well now. You probably have many people among your listeners who have had their DNA sequenced. But understanding the structure of even one protein is extraordinarily difficult, right?

That's a worthy PhD project. I would say maybe a typical time frame is a year. If you want to put money on it, maybe $100,000 to get one answer. This is really important to biology because we want to understand how these proteins work. When they misfold, sometimes it's disease. Even when they work, they are the parts of the cell. They do all the things of the cell.

They're beautiful proteins. The reason that cells can move, right, is this giant protein machine whirling around, driving the force to move cells. All of the functions of the cell are basically in these proteins. Humans have about 20,000 different types in different locations in your genome.

What scientists have done is they've gone to these enormous machines—synchrotrons, normally, the size of small towns—in order to produce extraordinarily bright X-rays. Even that only comes after they've done really hard experiments trying to figure out how to crystallize a protein. This takes years and years. Once they do that, they solve another mathematical problem that maybe we'll talk about, maybe we won't.

They get one picture of a protein. They get this kind of progress and often this whole wealth of understanding: “Oh, okay, I can understand how this DNA change that was found in the population might affect Parkinson’s, because look, it’s right here on this protein, and that makes so much more sense.” People have studied this problem for a long time. There have been almost innumerable Nobel Prizes given for individual proteins, right? Ribosomes and many others.

People have an incredible societal investment in collecting around 200,000 of these structures—about 140,000 at the time we did AlphaFold. Each one is still extraordinarily difficult. I remember seeing people talk about their PhD and give one of their talks near the end of their PhD: progress toward crystallizing whatever protein. So, I did my PhD, and I’m going to be a doctor, and I’m probably not going to crystallize this protein.

I guess I’m telling you all about proteins and nothing about what we did, but what we did was develop a new deep-learning system from the publicly available experimental data—all very public data—that was vastly more accurate at predicting protein structure. It predicts it to something like within the radius of an atom, in typical accuracy. That’s an accuracy that starts to rival at least some experimental methods, but more importantly than that, it’s extraordinarily fast: it takes 5–10 minutes to get the structure of a protein instead of a year. I should at some point figure out what that ratio is in terms of time.

But then also, of course, it’s incredibly scalable. We’ve predicted the structure of 200 million proteins—basically every protein from an organism whose genome has been sequenced. We’ve made this widely available, and scientists are using it like crazy.

It’s absolutely amazing. You have released a database of all of these proteins, and the map lit up. Now scientists from all around the world can access these protein structures for many downstream tasks. But to bring this to life, we have proteins doing things in the body, and we can use these structures to do things like drug discovery and whatnot. But what’s the gap? What can people do now that they have these structures?

I think the right way to think about this is that it’s a starting point for biological research. If you think about what people do and some of the beautiful studies that people have done, we see it all the way. One that just came out was a scientist trying to understand how cholesterol is moved about in the body. What actually is the thing that takes cholesterol and moves it from one place to another? How might mutations in that affect high cholesterol, heart disease, et cetera? There’s this beautiful, weird protein that wraps around it in a shape that we really didn’t know until a few months ago, when this paper came out.

What they were able to do, actually, is one of the ways in which scientists, I think, really commonly use AlphaFold: they use both experimental techniques and AlphaFold. They used an experimental technique, cryo-electron microscopy, to take an incredibly blobby picture. They used to call cryo-EM “blobology.” It’s gotten much better, but it’s still an incredibly rough picture, and they don’t really know the atomic details. Then they also run AlphaFold, and they see, “Well, actually, AlphaFold has this shape that almost exactly fits within this kind of blob.”

So, you get both confirmation and more detail. Suddenly, you have this beautiful atomic model where you can start to say, “Now, where are the changes in this protein? What might it affect? How might it affect how it takes cholesterol from one place to another?” Then you have to go figure out, “Now, how do I make drugs for this? Do I bind to this protein?”

I think when you see it in drug development, there are actually 2 or 3 ways in which it’s used. The first, I will say, is that the hardest part of drug development is that we do not know how biology works very well. The thing preventing us from curing, say, autism is not that we know exactly one protein and, if we just had its structure, then autism would be cured. It’s a huge disease that involves the whole body.

We’re trying to unwrap and unravel biology well enough to figure out which proteins are involved, how these proteins interact, and how that ultimately contributes to these phenotypes. People do biology across all these length scales. The contribution of AlphaFold is to say, “This protein, for example, that you didn’t even know was important…”

There was one study from a few years ago on a protein. There are all sorts of recycling mechanisms in the body that take proteins the body doesn’t need or doesn’t want anymore and get rid of them. There were hundreds of genes, in fact, that people found were turned off at a certain phase in cell development, and they didn’t exactly know what protein was involved. They did some genetics and found this protein that had essentially never been studied before, a human protein called Midnolin.

If you knocked it down, these proteins didn’t get recycled, and that’s more or less what they knew. They knew it didn’t work in the standard way. They ran AlphaFold, and they looked at it and saw some pieces that were suggestive. Then they ran AlphaFold together with almost all 500 proteins that were responsive to knocking this protein down. This is kind of how biologists develop evidence.

They found that in about 40% of these, when they ran AlphaFold, there was this very, very specific pattern where one part of that protein was trapped between 2 parts of Midnolin, grabbing it like clamps. Then they would go and do experiments, because they would say, “What happens if I take this bit of protein and remove the place where AlphaFold says it’s clamped by Midnolin?” Suddenly, that protein doesn’t drop down in the cell.

They found this on maybe 10 examples: 9 of them worked exactly this way. One of them was only partially reduced in how much it was knocked down, but then they looked at the AlphaFold predictions and found that AlphaFold actually put it in 2 places. If they took out that second place as well, then the degradation was completely abolished. Now they have this mechanistic understanding of this new protein they had never thought about before, and now they know exactly how it recognizes what’s to be degraded in this really important stage of cell division.

So, now the question becomes: okay, now how do you take that knowledge and do drug development? AlphaFold 2 came out 5 years ago. What we’ve done more recently, about a year ago, is AlphaFold 3, which says, “Well, let’s not just do proteins; let’s do the protein cinematic universe.”

I said proteins bind cholesterol, right? This is a non-protein kind of fatty molecule. Very importantly, they also bind drugs. Drugs are small molecules, maybe 20–50 atoms, that stick to proteins and change how they behave. You couldn’t even ask this question of AlphaFold 2. You couldn’t say, “How does this drug stick?” It would say, “You’d better give me a protein”—unless your drug is a protein, which some are.

In AlphaFold 3, we expanded it to do the whole universe of things that appear in the PDB, the whole universe of cells. Now we can say, “This is exactly where that drug sticks.” People around the world are using these ideas and building others.

For example, Isomorphic Labs, inside Alphabet, which was inspired by the AlphaFold breakthrough, is trying to say, “Okay, let’s really use this to start to do drug design. Let’s start to take these technologies that finally work, that are finally predictive about this, and now let’s see if I can design a small molecule with it, or design a drug that binds, that changes how this machine works,” in a way that hopefully makes someone healthy.

I think the best kind of analogy for how you should think about drug development, really—or maybe the way to think about how hard it is—is this old joke. Do you know this joke? There’s this giant factory, and one of the most important machines in this factory has stopped working. They call in a technician who comes, looks at it, goes to some screw or some nut, and turns it a quarter turn. The factory comes back to life.

They say, “That’s wonderful. Thank you so much. Can we have a bill?” And he says, “$10,000.”

Yeah.

And they say, “What?”

Knowing what to turn.

Yeah. Yeah. There’s knowing what to turn, or turning this for 50 cents and knowing what to turn for all the rest. I think this is the right analogy: we are both learning how to turn this, how to build drugs, and also learning, in this big, complex factory of the cell, what we need to do to cure a disease.

Yes. But it’s so incredibly complex, isn’t it? The human body is alive, and there is a symphony of complex, adaptive, compensatory mechanisms. I guess the idea here is that we’re proposing a mechanistic understanding of how this works, which means we can design interventions that are very effective. But in machine learning, we’ve kind of learned the opposite lesson: all of our intuitions about how things work don’t really work, and we need lots of data, and we need to test lots of things. Could it be a similar thing here? It’s like whack-a-mole: you do one thing, and then something else compensates.

I think what’s really important, in a certain sense, is almost the humility of AlphaFold. People say, “We are trying to predict what this experiment will give you. We are not trying to tell you everything. We are not a model of the entire cell; we are a predictor of this experiment that you did all the time and that took you a year.” And so, in a certain sense, I think we have validity in that I can characterize very well how well we will reproduce that experiment.

Then people figure out how to take this machine and use it in other ways that we didn’t expect: to discover new mechanisms, to try thousands of AlphaFold predictions to find 2 proteins that stick together, and to find this unknown component of a complex system. So people are finding ways to push this further.

But, in a certain sense, we are narrow. We predict the result of a scientific paper that often appears in Nature, Science, and Cell, in these big journals, right? We predict Nature-level science with the press of a button, in a very narrow category of Nature-level science: the structure of a specific protein. But there’s this enormous, wide universe of biology that ultimately we’re going to have to figure out and understand.

What data will we pin ourselves to? What experiments will we predict, and predict really, really well? Maybe the other story of machine learning is that predicting things adequately is all right, but predicting things extraordinarily well starts to produce amazing machines. We see this, of course, in language models and image generation, but also in protein. So I think we aren’t building just 1 universal biology machine—or at least, if we do, it will have to look a lot more like a language model than a narrow predictor—but we are doing something truly useful.

Can we talk through the predictive architectures of the different versions of AlphaFold? The first version was a CNN; the last version is a diffusion model. The second version—we spoke about this last night—had a structure module, and obviously geometric deep learning is spoken about a lot. I think people misattributed the benefit of having these symmetries—it had these SE(3) symmetries. Just talk me through that process, because you were saying at the beginning that you were really trying to imbue your human understanding of this, and then experience told you differently.

I think there are 2 or 3 things. First, I would almost object—not technically, but thematically—to AlphaFold 3 being a diffusion model. We love to stick things in boxes. We love to have the highest-level bit be the answer for why these things work: “Oh, they switched from a CNN to a diffusion model.”

I think the answer is that AlphaFold 1, as a network, predicted a subpart of the problem. It started from biological data—evolutionary correlations—and ended in geometric-ish data: the distance between atoms. In between was a CNN. It was actually an off-the-shelf CNN from computer vision that someone else had developed. That was a CNN, but all the protein-specific pieces were wrapped around the machine learning.

I would say AlphaFold 2 was, “Let’s build the science,” instead of building the science of image recognition and then applying it to proteins. The human visual system is not exactly what we needed to fold proteins. Humans were bad at predicting protein structures. How were we going to actually build all the pieces?

There was an SE(3) component. In fact, AlphaFold 2 was built iteratively; there were many stages. The SE(3) component was the first part of AlphaFold that we built. At the end, though, AlphaFold 2 was really this giant trunk of an architecture we called Evoformer, which is axial attention plus a bunch of other things. That is 90-plus percent of the compute and the accuracy.

It produces this kind of N-by-N bit. You start off with 2 pieces of data: the protein sequence, and then the sequence of every protein evolutionarily related to it. Protein structure changes slowly. The structures of proteins are, in most cases, similar to the structures of proteins in yeast, sometimes even out in E. coli. So you grab many related structures; you often have hundreds or thousands.

You provide this information to a specialized architecture called Evoformer, which had 2 forms of axial attention that were having a conversation between what we believed about geometry and what we believed about evolution. We had these 2 representations.

We ended up taking the geometric bit—the N-by-N—which we had as an intermediate loss, and asking, “What are the distances between these atoms?” We made categorical predictions and then handed it to what we called the structure module, which is best thought of as a geometrization engine. If you have N-squared predictions about N positions, somebody is going to have to harmonize this thing.

This used an SE(3)—I guess it was invariant in the sense that we collapsed it on every layer—SE(3)-invariant attention. This was actually one of my ideas when I was starting at DeepMind. I was thinking about protein residues even then. You have this backbone, which has 3 atoms, and you can align the frame to it. It’s extraordinarily rigid, and the business end—the places where all these residues differ—is kind of just off that.

If you align them to reference frames and operate on points in those reference frames, it’s natural. You can just take attention and let it project points in its local frame. You can transform it, and then use the distance of those points to bias your attention. This is invariant point attention; this is what we called it in the end.

More important than that, probably—almost certainly—was defining the frames. Defining frames let us write down a really interesting loss function. We called it Frame-Aligned Point Error, or FAPE. This was saying, in the reference frame of the ith residue, “Where is everyone else?” It’s locally registered, and then you have N-squared errors that you average together. I think that was really, really important; one of the breakthroughs early on was this loss.

Of course, the really fun part is the SE(3) invariance. But remember, we didn’t start with any geometric data; we started only with nongeometric data. So our geometry emerged in the middle. We started with what we would call black-hole initialization, where we just stuck all the residues on top of each other in the world’s least physical structure.

It was important also that we disrespected the known symmetries of a protein. For example, the atoms—this atom and that atom in a residue—are separated by 1.3 angstroms, plus or minus 0.015. And in fact, even in AlphaFold 1, when we did the optimization, we would use something like a jointed robot arm to optimize these, say, typically 300 residues.

You would do all this twisting, and you would actually have very ugly geometry. The geometry of a 300-joint—or actually, no, sorry, it wouldn’t be 300; it would be a 900-joint—robot arm is really bad. That means that your optimizer has to take many steps.

One important thing was, “Let’s just break it up. Let’s just treat them as a residue gas,” we called it, “so that this can proceed in 4 or 8 steps instead of the number of steps required by this twisty geometry.” Then we used equivariance, and it helped.

But one of the things that was really surprising—I think maybe because of the early talk, or maybe because people were working on equivariance—was that geometric deep learning had become very popular, and people said, “Ah, they mentioned my keyword. That must be the reason it worked.”

I remember being a little confused. I thought, “Okay, but we’ll very carefully ablate this. We did quite a few ablations for AlphaFold 2, and we’ll publish the paper, and everyone will realize.” And we published the paper. I remember the 5th row was called “No IPA.”

AlphaFold 2 was about 30 points on the GDT scale better than AlphaFold 1, right? That’s the kind of 30 points you’re talking about. We did these ablations, and they were all small—almost all small. Removing the invariant point attention, and therefore the equivariance, cost about 2 points. You could measure it—maybe 2.5 points. So it contributed, but it contributed 2.5 out of 30.

I thought that would put it to bed. It didn’t put it to bed at all. People still talked about AlphaFold 2 as the great victory of equivariance. They never talk about FAPE. They talk about an equivariant transformer that they think came from others.

Actually, we did this equivariant transformer in 2018. I remember it was October 2018. We tried a little to improve it, but it didn’t make AlphaFold better, so we went on to the next thing. We’re kind of ruthlessly empirical about it.

But it’s a very cool thing, and so people really hooked onto it. I think what really happened is that we were talking about it at dinner last night, at this AlphaFold dinner, but global SE(3) symmetry is not a very powerful symmetry. It’s not nearly as big and powerful as a symmetry like, “All the residues are permutation-invariant.”

We do still have permutation invariance as probably the big symmetry of AlphaFold. We have a transformer whose position is encoded only by relative position.

We clipped the relative-position encodings. But I think this particular symmetry isn't like physics, where you write down the symmetry group, derive the laws of physics from your big symmetry group, and get the Standard Model. This is a symmetry of a messy real-world problem that probably doesn't pin it down so much. So I think it's good, but we shouldn't obsess about one good thing. You don't want to valorize things.

My favorite review of AlphaFold 2 came when we got the reviews back after submitting the paper. One of them said, “This is 6 or 7 papers’ worth of ideas.” I think that was right. There are many ideas that added up to a transformative system. To use a baseball analogy, it's not 1 or 2 home runs; it's 18 doubles. These midsize wins stack together and make a transformative system.

We would sometimes find in our ablations—we ran a double ablation, I think it was no recycling and no IPA—that we turned off 2 things and performance cratered. There are many problems we need to solve, and we solved most of them 2 ways because it was better than solving 1. If you knock out both things, then your building may collapse. That was maybe a 12- or 15-point drop, which was our biggest ablation, and it was still only half the gap to AlphaFold 1. I remember doing the ablations and saying, “Guys, we've never crossed AlphaFold 1 performance.” But a lot of those ablations actually went into AlphaFold 3.

So we said, “Okay, equivariance isn't super important.” We had another ablation: if we took out giving the raw genetic information and gave it the pairwise correlations, that was 1 or 2 points worse. So maybe the fact that we're processing these all the time is not so good. We made this interpretability projection of each layer into a structure and made movies, and we could see that most of AlphaFold's capacity was spent optimizing the structure geometrically. Outside the first few layers, it's much more a geometry engine than an evolution engine.

So we said, “Okay, why don't we just cut back the Evoformer to operate for a few layers, and then we'll do a much simpler version called a Pairformer?” That improved performance. We basically used these ablations to say, “This is what the machine learning is telling us.” Whatever we may feel like, being a machine learner is all about thinking about the data, looking at it, coming up with hypotheses—maybe equivariance is important—trying it, and measuring. 9 times out of 10, you find out you're wrong. If you're wrong 9 times out of 10, you're a very successful machine learner; you're incredibly productive.

You build this local intuition, this notion of what the problem needs and how it works, and, in my view, a kind of science local to your area—a local manifold of ideas in protein-structure prediction near these architectures. We would develop intuitions like, “Thou shalt not put a 1D representation rather than a 2D representation anywhere near the Evoformer, or your performance will go down.” Maybe a year or 6 months into AlphaFold 2, at some point we had a mix of axial attention and convolutions in our pairwise processing. It was a somewhat different architecture.

I remember someone did an experiment where they just deleted the convolutional layers—not adding attention layers, just deleting the convolutions, adding no parameters, and using strictly fewer parameters—and the model got more accurate. The validation loss improved. That doesn't normally happen in machine learning. They don't say, “Remove parameters and your generalization will improve.” Convolutions were probably actively harmful to learning what we wanted to learn, and I have a hypothesis about what that is. But all of these lessons and explorations were about how deep learning generalizes in proteins, and you have to build that knowledge and expertise.

What's really special in AlphaFold—especially in AlphaFold 2, and then I guess I should talk about AlphaFold 3 in a minute—is that we have a mix of biological hypotheses, physical and geometric hypotheses, experience, and the tactile feel of years of banging our heads against this particular problem and data set.

We trained AlphaFold 1 and AlphaFold 2 with the exact same data. We decided just to have more evaluation data and not bump up our training data at all. The effect of that was really large. There was a wonderful study by the AlQuraishi lab, which retrained AlphaFold 2 models on 1% of the PDB. So instead of 100,000–150,000 structures, it was something like 15,000. They found that AlphaFold 2 trained on 1% of the PDB was more accurate than AlphaFold 1. So you can really say that the architecture and training ideas we put into AlphaFold 2 were worth a clean 100× in data.

We were speaking last night about all of the tacit knowledge that you guys acquired, but maybe we'll park that for just a second. The enterprise of machine learning is about building models of understanding for things that we don't understand. We're building these alien artifacts, and you gave a wonderful example: imagine I'm generating some text. We might think naively that the way it's rendered around the edges is actually the thing doing the heavy lifting, but the load-bearing thing might be something completely different. Many of our intuitions don't really work.

For me—and I know you're allergic to the word “understanding”—understanding is the possession of a generative model that can do the thing. If you could create a physics simulator of some phenomenon, assuming that it's not lossy, I would say that you understood that thing. We think of machine learning as modeling existing examples, which means that, rather than modeling how something is constructed or built, it's modeling the thing at the end.

But you were describing the Game of Life, which is a great example of this. It's modeling—it's learning the path, not the destination. You were describing something fascinating last night: we have, for example, this recycling mechanism in AlphaFold, and you can pass things through the structure many, many times. What seems to happen is that, at the beginning, it solves the most complex problem, and then it's refining, refining. This is a little bit like the Game of Life. It's not simulating the creation; it's almost, at any point, learning how to refine and optimize the structure.

Okay, so I think we should distinguish 3 things: predict, control, and understand. Predict means that you say, “I'm going to do a thing. What will be the value of my machine? What will appear on my computer screen in the future?” That is prediction. Control is, “I want to measure this thing in the future, and I want it to come out 17.” That's control.

Understanding is a lot like prediction, except there's a human in the loop. Understanding means that I have such a small collection of facts that you can predict, and you will do it with facts that I can communicate to another human in a compact form that fits on an index card. That's almost understanding. So I think these machines let us predict. They let us control. We have to derive our own understanding at this moment.

We can experiment now on the artifact. We can look at the 200 million predicted structures, not just the 200,000 experimental structures, in order to help us understand. But it doesn't do the act of understanding for us. It does the act of prediction and maybe control.

There's maybe one other thing. There's the algorithm, and it's a really important concept in machine learning. There's the algorithm you program and the algorithm you get. Machine learning is code meets data, which produces weights. One of the lasting debates in machine learning is how much work is done by the code and how much work is done by the data that ends up in the weights.

What I think we see in AlphaFold, in a certain sense, is a very beautifully intuitive algorithm—an algorithm we can, in some sense, understand. It does successive geometric refinement. I communicated that to you in a few words, and you probably almost saw it in your head, even though I don't think you've seen these videos. They're in the supplement to our Nature paper, but that is an algorithm that humans already came up with.

Maybe we should do gradient descent in some empirical model that makes each thing more correct. Maybe that's how AlphaFold should work, but that wasn't what we programmed. We still thought about it in a certain sense. We were thinking about things like recycling in terms of, “Wow, isn't it weird that AlphaFold, in layers, has to give an answer no matter how hard this problem is? Maybe we should give it some more layers.”

My GPU is out of memory, so maybe I should just run it back through so I don't have to use more memory. But even without that, I think AlphaFold, even without recycling, was learning this kind of iteration. Then we put in a kind of code idea—an architectural idea—to help this process that it was going to learn from the data.

Going back to the earlier thing about exactly how far apart residues are, we didn't tell AlphaFold that. We knew that the data would scream at it that i and i + 1 were 1.3 angstroms apart. So I think when we think about our human understanding, one of the things—I don't really love The Bitter Lesson as people try to apply it.

In fact, AlphaFold 2 is the opposite of that. We did a whole bunch of specialty stuff because our data is not finite. And in fact, now that we’ve gone to language models, we’ve found our data is still finite. The internet is finite.

So, I think “don’t do architectural research” is the wrong thing to draw from it, but have some humility about which things go into your code and which things will be derived from your data. Look at what’s missing, and understand the algorithm that deep learning is trying to learn. How can you accelerate it? How can you add hypotheses, and where especially can you add some kind of communication?

The most important thing we would do within the architecture is modify which units communicated and how. I think all of these are kind of how we derive understanding to ultimately make an iterative process. And it should shock no one that if you’re trying to make an intricate geometric object, you are going to iterate.

Similarly, if you think about generating text, one kind of naive assumption that people will make is that these are next-word generators, so they have no idea what’s going to happen 2 words ahead or 3 words ahead. But of course, you can’t write down the next word without knowing how it’s going to end. I don’t start a sentence not knowing how it’s going to end most of the time. At some point I change, but I think ahead a little bit in order to accomplish my task.

And so, the understanding that we see built into these models—the structures that we want—sometimes emerge and sometimes don’t. I think we valorize the high-level ideas that emerge. For example, coming back to AlphaFold 3, you said, “It is a diffusion model.” But I would argue it’s a different diffusion model than an image model. There are some differences.

For one thing, there’s a huge trunk that is not in any way a diffusion model; that’s only run once. That trunk is probably where the structure is actually determined, and the diffusion is just like the structure module was: a geometrization engine that took a set of really quite good constraints that had a very clear notion of the structure within those constraints and solved the details.

I think AlphaFold 3’s diffusion is similar, and it’s especially similar because, in images, you start generating an image and you see, especially in these early-trained diffusion models, kind of colored blobs. They start to decide what those colored blobs mean, and they pretty clearly decide what those colored blobs will mean later because you could stop them in the middle of the process and run them again and get a somewhat different interpretation of those colored blobs.

In AlphaFold 3, you actually have an interesting thing. If you look at AlphaFold 2, we can, through this process of projecting out intermediate layers, see what it solves first, and it basically solves local details, local pieces. It starts to put local pieces together. It’s agglomerative in how it solves the structure, as is kind of natural. The easiest thing to predict is your local structure; the hardest thing to predict is your largest-scale structure. That’s how AlphaFold 2 works.

If you look at AlphaFold 3 and you take coordinates to which you’ve added a very large amount of noise, the very first thing you have to solve is, say, if you have 2 proteins, how do they associate? Where are their 2 blobs relative to each other? What are their Gaussians? So, the problem that AlphaFold 2 is solving last is the problem that AlphaFold 3’s diffusion has to realize first.

And how does it do it? The answer is not that it comes up with an orientation and builds the protein around it, because of course it’s going for 1 correct answer, or at least a very narrow distribution. The answer is really that the big network before it, plus the first pass through the diffusion network, are solving the overall structure, and then the diffusion is realizing any details it couldn’t solve before. It is diffusion technically, but it’s much closer to AlphaFold 2.

I think there are very specific technical reasons around laziness and geometry that made diffusion a really good choice for AlphaFold 3. It made it easier to handle ligands and some bond distances and local things. But it’s not like diffusion in the same way as, “Oh, it’s drawing up blobs and deciding what they mean at the end.”

I think all of these details are far more important than this high-level bit of, “Is it a transformer? Is it a diffusion model?” that we want to talk about. Even these diffusion mechanisms don’t work in the way of progressive refinement that makes sense for images. Maybe you’ll make colored blobs and decide what those colored blobs mean. Even that, I think you can argue, is maybe not entirely the story, but it’s definitely not the story for proteins, because the hardest problem is the large-scale structure.

I mean, in a sense, this is leaning toward this idea of constructive complexity that we were talking about before. I’d love to get your general take on what this means for artificial general intelligence. With language models, for example, we train them basically with behavior cloning. We have this rich, adaptive, generative process, we generate all of this language, and we train language models on it.

For me, intelligence is the adaptive acquisition of coarse-grained representations. Culture and language are changing all the time, so we invent the word “unalive” to get around the filters on social media platforms. That’s an example of linguistic agency. With language models, we notice that when we do this iterative, adaptive refining with active fine-tuning and adaptation, they become a bit intelligent. They learn new representations and adapt.

In a way, what they’re doing is—even though they’re ungrounded from the world—they can take a code solution like AlphaFold and refine it, and refine it. It seems to work really, really well. But are we in this regime, do you think, where we’re not necessarily building artifacts that have the same type of generality? What do you think about intelligence in general?

So, this question of representations is very, very important—and far less important than people believed 5 years ago in the explicit sense. Just like we were talking about the things that AlphaFold does and the things that AlphaFold is forced to do by its code: obviously, everything that it’s forced to do by its code, it does, but many things it does without being forced. It had to learn them to make a good predictive model of the data. It had to find good intermediate representations.

In a certain sense, I think the most seductive idea in machine learning is always, “There’s this thing I know will have to be there in the end. So, I’m going to have a place in my code that is named that, and then force the mechanism to be a high-level concept-builder thingamajigger.” That was a very popular kind of idea: “I’ll make the concepts units. I shall force disentangled representations via this loss, sometimes on the intermediate layer. This is where it will store those.”

That’s reasonable to test, but what we’ve seen a lot of is that many of the things that you would imagine are needed for intelligence are developed by desperately trying to predict the next token really, really well. They’re not developed because you predict next tokens at all. They develop because you do a really, really good job at it.

These generalized spaces, representations, and understandings of concepts are forced very slowly with data. There are a lot of log-linear relationships—or, you know, everyone’s least favorite function, right? Things go up linearly with the log of effort, which we see all the time in our scaling laws. But we do get these concepts and representations, and what we don’t really have an answer for is how we get them more cheaply.

Sometimes we can get them via programming, right? We get memory-like things. Now we have language models writing notes for themselves and then retrieving those notes, or we find out it’s better to keep reminding agents what they’re doing so they don’t forget over long trajectories. So, we build weights, we build artifacts, and we find efficiencies.

We can often paper over those deficiencies in some kind of software harness, but then we don’t yet know how to drive that back into exactly our machine learning. You don’t just put a harness with external memory around it and then distill it back into the network and have amazing memory that no longer needs this harness. We haven’t figured out how to do that.

Unfortunately, John, we have to wrap it. Dr. John Jumper, it’s been an honor to have you on the show. Thank you so much for joining us today.

Been tremendous fun. Thank you.

So, as I said earlier, Emmanuel is based in Africa, and he’s actually training scientists. He’s not just giving them access to AlphaFold; he’s training them how to use it, how to interpret the results, and how to help scientists build experiments using the database.

Emmanuel

Yes, my research focuses on drug discovery for malaria and enteric bacteria.

And then I'm also involved in capacity building for Africa-based researchers using tools like AlphaFold. Initially, African scientists didn't have access to expensive structural biology tools. With AlphaFold, these researchers can now do complex experiments that were not possible before and tackle diseases such as malaria, HIV, and other antibiotic-resistant infections. In my own research on drug discovery, I use AlphaFold in terms of solving structures from cryo-EM data, and I also utilize that to map out the mechanisms of the proteins.

So, for him, AlphaFold was so impactful. If you think about the before and after, we're living in a different world now.

Emmanuel Nee

At that time, phasing a protein was still really, really difficult. I tried for several years, close to 4 or 5 years, and it wasn't successful. With AlphaFold—I imagine this was more than 10 years ago—I went back and did just one protein purification, collected the data, and, with AlphaFold in combination, I got the structure in less than 2 or 3 months.

And now it's his goal to train as many scientists as he can in how to use this technology for the betterment of humankind.

Emmanuel

This year, with funding from Google DeepMind and the Swedish Research Council, we have scaled up to 100, and there's no drop in the quality of the training. In fact, there was an improvement. So, based on this, we want to train 100 scientists every year for the next 10 years. We're targeting close to 1,000 African scientists in the next decade to be able to utilize this tool effectively. And then we want to form an emerging community of structural biology practitioners working on prevalent diseases in Africa.

So, that was the AlphaFold show. Thank you very much to John and Emmanuel. The conversation with John was very interesting. He's so inspiring because I think he is testament to the fact that even though we talk about all of these general-purpose foundation models, to really advance the frontier and build cutting-edge applications in science, we need to do a lot of engineering. We need domain knowledge. We need serious expertise. And a lot of our models will actually look quite hybrid. They'll look quite customized. I think AlphaFold is a kind of proof of existence for the types of hybrid models that we can deploy to further the field of science. John, I wish you the very best of luck in your new position at Anthropic. And thanks for watching the show.