🔬 The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub

So ESM-C is also approaching programmable biology, but I would say in a very different way. It's approaching it from this kind of world-modeling perspective, where the idea is basically that you have a predictive model and you're going to search the world model to find protein molecules that satisfy whatever design criteria you have. We've been able to use this to design many protein binders. But I think, most excitingly, we've been able to use this to design antibodies, scFvs.

Thank you for having me here. It's great to be here. I'm head of science at EvolutionaryScale. I'm a computer scientist, and I work on AI for biology. A lot of my work has been on language models for biology.

swyx

1. The Bitter Lesson For Proteins

By the time this podcast is released, you will have put out several new, exciting, interesting models. Going over them, I couldn't help but have the thought that you might be the most Bitter Lesson person in protein biology right now. Can you give a little context about what that means for biology and why you're so committed and excited about this route?

Well, I'll take that. I believe in scaling laws. I guess I've been working on this since the summer of 2018. My team, when we were at Meta AI, trained really the first transformer language model for protein biology. I've always thought that there would be an emergence of biological information as you train a model to predict the next token that evolution creates.

So our team has really explored that idea over a number of years, and we've really seen the scaling curve. We've seen that, as we have increased models by an order of magnitude in each generation, there's this emergence of new capabilities.

swyx

Yeah. So you've been talking about the emergence of capabilities and scaling over generations. You've been working at this for, I guess, 8 years now or something like that. It didn't always work that way, right? There were signs that scaling might work.

We'll be getting to some new results where I think you've really clearly demonstrated this hypothesis in a way that hasn't happened before. But you seem to have a strong commitment to this in a way that I'm not necessarily sure I would have been so convinced would work in the same way. Protein language is not the same thing as natural language. There are similarities, but if you start sampling a normal language transformer at temperature, you're going to get gibberish.

If you sample a protein language model at infinite temperature, you're going to get something that is a valid protein, if not an interesting protein, despite the fact that it's a different domain for a different reason. So what is specifically about proteins that you thought was special, or that would make this valid as well?

2. Protein Sequences Encode Biology

Yeah, I mean, it's a really interesting question. I think it's a deep question across AI right now more broadly. What's so interesting is that AI right now is such an empirical science, and we don't have theory that can always guide us in these things. But we have this really strong empirical evidence of scaling.

The thing that motivated me is, if you think about evolution and the data that we have around proteins, we have databases that have billions of protein sequences. Those sequences contain patterns, and it had long been known—decades before we started working on this with language models—that there are patterns in the sequences of protein families that arise because of the constraints that evolution is operating under.

So you can think about a protein sequence that folds into a three-dimensional structure in space. You can imagine that there are 2 residues, or amino acids, that are in contact in that folded structure. Evolution isn't free to choose those independently from each other. If it makes a choice at one position, it has to make another choice that's going to be compatible at the next position.

Going back all the way to the beginning of gene sequencing, when people first began to be able to look at the same protein in related organisms, you could start to see these patterns that reflect the fundamental underlying biology.

So the idea behind ESM was, what if you were to apply this principle across all of evolution, across the vast diversity of proteins that have been generated across all of life, and basically have a language model predict the amino acids that evolution will choose to place in proteins across all of those biological contexts?

You can think that there's just this incredible amount of information in that total picture about the underlying biology of proteins. That was really the idea that sparked this. As a model is having to predict the next token—and we train these models with masked language modeling, so they're predicting tokens that are masked out of various parts of the sequence—it would have to learn something about the underlying constraints that are shaping which tokens evolution can choose.

3. Building The Protein World Model

Alessio Fanelli

Yeah. So maybe for a bit of history: you just released ESM Cambrian, right? Is that what it's called?

Well, they were called ESM from the start. We had various branches of the different models. This one, I would say, is a fourth-generation model. It's actually a model that we trained a little over a year ago. Now that we're at EvolutionaryScale, we're open-sourcing this model fully under an MIT license for the first time, so we're really excited to do that.

The big thing that's new here is that we've really built a world model of protein biology. The foundation of that is ESM-C. Using the representations of ESM-C, we've now built a structure-prediction model. This is the next-generation ESMFold model.

We've also used the techniques of mechanistic interpretability and sparse coding to really start to look deeply into the representation space of the language model and pull out the underlying features that the model actually uses to represent protein biology.

Bringing all of this together, we're able to make predictions for protein structure and predictions about the underlying features that proteins are made out of. That allows us to build linkages across evolution. We're able to take this model and invert it to design proteins, and we've used this to create a comprehensive picture of protein biology.

We put together the world's largest protein sequence databases. That amounts to 6.8 billion nonredundant proteins. We've resolved or predicted structures for 1.1 billion of those, and we've also computed features across all of them so that we can make these linkages basically all across evolution and protein biology.

Alessio Fanelli

6.8 billion, of which you've resolved structures for 1.2 billion—is that right?

1.1.

Alessio Fanelli

1.1. So what about the others?

Basically, what we did is take that database and cluster it at 70% sequence identity. It's really resolving structures for everything in the sense that, for each cluster, we have a cluster center. We're predicting the structure there, and we can expect that the other proteins are going to have a similar template structure. There will be small variations, but they have the same fold.

Alessio Fanelli

1.2 billion or so clusters that are covering the 6.8 billion?

That are kind of covering the 6.8 billion, yeah.

Alessio Fanelli

Okay, interesting. Since we're talking about scaling, how do you know that this is the right number? How do you know that focusing on these 1.1 billion is the right resolution for this model?

We've chosen them so that they really cover that entire space. What I can say about this database is that it's really the most comprehensive picture of protein structure and function that's been created. It's adding hundreds of millions of structures to our knowledge of the diversity of protein structure, and it's also creating this feature space that allows us to find these linkages between proteins across evolution.

We can see really interesting themes emerging across evolution, linking, for example, gene-editing systems that are very far apart in sequence but share some kind of underlying functional patterns or structural homology that the model is able to bring together and find those connections.

Alessio Fanelli

Now we're talking about the mechanistic-interpretability part. If I understand correctly, you use sparse autoencoders and other techniques to understand what the patterns of outputs are when you activate the network using a protein, and how they relate to each other.

You have these sequences that are unrelated or only partly related based on the actual sequence, but in terms of behavior they have similar behavior and therefore activate similar networks. Is that right?

swyx

Is that kind of the summary of what you just said?

Yeah. Basically, what we've done is train sparse autoencoders across all the different layers of the ESM-C model family. There are actually 3 models in that family: a 300 million parameter model, a 600 million parameter model, and a 6 billion parameter model. We've done a very deep analysis of the feature space of that 6 billion parameter model, which is really the state-of-the-art protein language model.

What we find that's really interesting is that there's this hierarchy of features that emerges. What's really interesting about it is that it corresponds to the reductive picture of biology that has been developed over many decades—a century—of biological experiments. What's so cool is that this is emerging without any prior knowledge; it's been learned by the language model.

The interesting thing about SAEs, right, is that they're really just revealing the intrinsic structure of the representation space. This model's been trained on protein sequences. It's been trained just to predict the amino acids that evolution will choose. Somehow, this is leading to the emergence of this very ordered feature space that has a hierarchical structure, where you can really see everything from the basic biochemical properties and the basic structural building blocks of proteins to these very large functional themes, these abstract concepts that connect to the human picture of protein function.

swyx

Do you have a hypothesis or feel for why there are relationships between the sequences themselves, even if they're shifted, cut up, and recombined in different ways? I can imagine that might work because proteins are hierarchical in their nature as well. So maybe the hierarchy moves around, but the functional units, I guess, have related structures. What is the hypothesis here?

I mean, it's a really interesting question. I think I can speculate about it. I don't think we completely understand this, but let me give a concrete example.

The nucleophilic elbow is this core functional motif that people have thought maybe has emerged independently in evolution, at different times in different protein families. But it has this very clear structural motif that you can see in a crystal structure. What we found, basically, is that the model has a single feature for this nucleophilic elbow, and it's activating across these very evolutionarily diverse families—really completely different structural topologies, proteins that probably evolved entirely independently from each other. But the model is using this one feature to represent that.

Why does it do that? I think it's a really interesting question. One answer is the idea of compression and the idea that the model needs to have some kind of underlying latent variables that it develops to help solve this sequence prediction task. What's so interesting is that the choice of any amino acid is completely entangled with the choice of all the other amino acids in the sequence. This is a very complex task: trying to predict what amino acids should be where in a protein.

To really do this well, the model would have to have these hidden variables that are representing the biology, which allow it to look at a protein and say, “Okay, what amino acids should be there in all these different contexts?” That's the intuition. I would draw the parallel to language modeling.

I was very influenced by a paper by Zellig Harris called “Distributional Structure” from 1954. I think that paper influenced a lot of people in the language-modeling field as well. It focuses on language and really articulates this idea that the set of contexts in which a word appears is determined by the meaning of that word. What Zellig Harris imagined is that, as you looked at the statistical patterns of what words appear in what context sets, you would be able to derive the meaning of language. You would have this statistical structure that would mirror the underlying meaning of language.

For me, at least, that's one of the most convincing explanations for why a language model that's trained on the text of the internet is going to learn something about meaning. It's going to learn something deeper and more fundamental. I think you can think about the same thing in biology, where the contexts in which an amino acid can occur are really determined by the structure and function of the protein, its biological roles—these very complex phenomena, both the intrinsic biology of the protein and its relation to all of the other proteins and their function and evolution. Those are what determine the context sets, and so you would imagine that those statistical patterns in the use of amino acids directly reflect those underlying hidden variables. The model is going to learn something about those hidden variables.

swyx

4. Metagenomics Unlocks Scaling

I definitely buy that. It seems plausible. I want to be clear: I actually do really believe in this direction, but there are a lot of ways I think about this where maybe I could say, maybe I would imagine it wouldn't work. One of them is data availability. What type of data do we normally have? What type of sequence data do we normally get?

I think ESM-C in particular has some new data sources compared to previous models, which might be helpful, but oftentimes the types of sequences we have available have a very strong bias toward certain specific needs for medicine or human biology or disease biology. It's not necessarily the case that, if you take just a naive data set, you're going to necessarily get an interesting scaling law.

I'm curious about what in particular was the breakthrough in ESM-C. Maybe we can go back a bit and talk about some of the other ESM predecessors, which got here before ESM-C, and how their strengths, but also maybe some of the limitations that ESM-C overcame, led to the developments there.

Alessio Fanelli

Yeah. Well, I'll admit that I am a bit less informed. I am a scaling fan.

I do think that just increasing the data and increasing the parameters and having that compression is going to lead to more powerful models. But it is also true—and I think you're absolutely right—that the underlying structure and distribution of the data is really critical. Some data sets will be far more valuable for learning these general principles than others.

But I think it goes against a lot of biological intuitions about collecting data, I guess, is what I'd say. Normally, when you think about what data you want, you're trying to answer a very specific scientific hypothesis. You want a very well-controlled experiment. You really want multiple replicates. It's something very focused, is the way that I would put it.

I think the change in the way of thinking is to think, okay, what you really want, if you want to learn a general representation of proteins, is to see amino acids in as many evolutionary contexts as possible. That's really what you want. That's really how I think about data.

If you look at what changed between ESM-2, which was kind of the previous-generation model, and ESM-C, which is this new-generation model, they're both at approximately the same scale. And—

Alessio Fanelli

The same scale of compute.

Same scale of parameters. Yeah, ESM-2 got a lot of compute, but ESM-C got even more compute. But it's not just the compute. The data was really the critical thing here, actually.

When we trained ESM-2, we observed 2 things. The first was that, as we increased the number of parameters and compute, we saw improvements. We had a model at the billion-parameter scale. We had a model at the 10 billion-parameter scale, and the larger-scale model is better than the smaller-scale model.

But if you look at a plot of parameter scale—sort of a log plot of parameter scale versus capability—and for capability, we're looking at representational fidelity: how well does it capture protein structure? You could see diminishing returns in ESM-2. ESM-2 is trained on UniRef. For ESM-C, we added metagenomics. We added billions more sequences to the training data.

swyx

Could you explain what UniRef and metagenomics mean?

Yeah. UniRef is, I'd say, the gold-standard data set of sequence biology. It's taking sequences from across a wide variety of different sequencing resources and clustering them to remove some of this redundancy that you were mentioning. It creates definitive coverage of protein biology.

In parallel to classical gene sequencing, there's this idea of metagenomic sequencing, where people go out into all kinds of different biomes and environments and collect samples from the world, then just sequence the natural diversity that's present there.

So, proteins from a hydrothermal vent, or proteins from a frigid environment near the South Pole, or the deep ocean, soil, the human gut—all kinds of different environments. This is a very different way of collecting data.

Instead of trying to understand the specific genome of a specific organism or a specific protein, you just collect a bunch of stuff, mix it up in a pot, and get the sequences out. You have no idea what organisms these are from. You don't necessarily even know if a given sequence is a protein, but you can guess based on certain contexts and say, “Okay, we threw these together. These are likely protein sequences we found. We're not assigning them to an organism. We're not assigning them to a larger context. We're just saying this is probably a protein. Let's train on it.”

swyx

That is right. Yeah. You don't even get the full genomes. You just get these kinds of contigs that are often broken and have even partial proteins, so the data is really noisy.

One more little nerdy question that I have here: If I understand correctly, you're not actually using a device that sequences proteins. You're sequencing the DNA that would manufacture those proteins, so you're finding DNA and then looking for markers that indicate the beginning and end of a protein sequence. Is that kind of right?

Yeah, that's exactly right. Basically, we're sequencing genetic sequences, and then we translate the proteins from those sequences.

swyx

So you're digging up, like, sewers—not you, but—

Me personally, there are sewers, like probably many New York City subways, all kinds of things.

swyx

Yeah. So the natural question to me is: You built this model and you think that you've de-duplicated it, so you have a good representational set without a lot of redundancy in it. How much more is there? If we had an order of magnitude more resources, do you think that there is an order of magnitude more proteins to discover?

I think so. I'm not entirely sure, but there are a lot of proteins, and I think we've barely scratched the surface of measuring Earth's biodiversity. There are core proteins that are conserved across all of life, so I think we know those. But as you go into these different environments, there are constantly new genes and new proteins being created by evolution.

swyx

A lot of my understanding is that this is viruses, bacteria, and other—

Microorganisms. Those guys are basically in this long-running conflict with each other that causes them to—

swyx

Recombine their DNA in ways that help them survive in these extreme, or whatever, environments. And so that's what's causing this incredible diversity of proteins.

That's right. Yeah. And just 4 billion years of life running experiments in parallel all across the Earth, in all kinds of different ecological niches. We just see the outcome of all of that.

Shawn Wang

And the combinatorial effect—that's why you believe that there's going to be so much more diversity, although maybe from a macroscopic perspective, when we look at it, there's not even nearly as much diversity as there will be at the microscopic scale, because you have this incredible combinatorial effect.

Alexander Rives

Yeah. I mean, there's just tremendous diversity there. So, kind of going back there—

Shawn Wang

Yeah, I know. It's great, right? And I think it's really—I mean, we could also talk about data and building models of the cell, and really going from the molecular level to higher levels of biological complexity. But, to complete the description of ESMC—

Alexander Rives

The big change was adding these metagenomic sequences. What we saw, basically, is that there are no longer diminishing returns to scale. That's really saying that ESM2 was data-limited rather than compute-limited for ESMC.

There's a really beautiful scaling law that we can plot. We can train models at the smaller scale, and we can look at the best representational fidelity they can achieve for a given compute budget. Then we can draw a line of extrapolation that beautifully predicts what the larger-scale models will be able to achieve in their representational fidelity.

There's this really beautiful scaling. The only changes to ESMC are some changes to make it a more efficient model for training, but I think the data is really the big thing that's driving that.

Shawn Wang

So it still is basically just a standard vanilla transformer, with a few tricks—everyone has a few tricks at this point—a language model, and just a lot of data?

Alexander Rives

So, I mean, this is very much in contrast to something like AlphaFold, where you have a lot of inductive bias built into the model in order to be able to—

Shawn Wang

Predict protein structure.

Alexander Rives

That's right. And the idea here is, can we just learn the right structure? Don't give any priors; just allow machine learning to figure out what that structure is.

Shawn Wang

You also had your own detour into priors with ESM3—or maybe not priors, but using more intuition or more human design. Do you think ESM3 was a detour? Did you just end up saying, “Okay, let's make ESMC bigger,” and then suddenly it worked, and now you learned that we actually don't need priors anymore? Is that a key insight, or do you still think there's room for priors?

Alexander Rives

I think we need both. I think there's a place for both of them. The goal for ESM3 was to really make biology programmable. We were trying to think, “Okay, what is the programming language? How are you going to be able to allow biologists to prompt a model and design structure and design function and all these things?” We really thought it needed the right tracks.

I would say that ESM3 was very consistent with the philosophy of ESM, because what we did was predict structures for this vast array of evolutionarily diverse proteins, and we used that as the training data. The model is now learning from sequence patterns, structural patterns, and functional patterns.

I think that same kind of synthesis—the model learning on sequences—could be extended by bringing in more multidimensional information. That would build an even better representation space.

Shawn Wang

If you're a coder, or if you're building language models and then building coding agents, you start with pre-training on everything, and then you go to doing the programming part through some sort of post-training, probably RL. Have you thought about post-training ESMC to try to give it the same abilities for programmability? Do you think you could get programmability without all of the inductive biases, which involve an atlas of structures and some sort of interesting distillation? I guess maybe that is some kind of post-training of a different model.

Alexander Rives

Yeah. I think it's a really interesting question: To what degree can you interconvert these models? I don't think that's fully understood yet, but I think it's a very promising direction to think about—what are the right ways to do that?

5. Searching The Protein World Model

Shawn Wang

So, ESMC is also approaching programmable biology, but I would say in a very different way.

Alexander Rives

It's approaching it from this kind of world-modeling perspective, where the idea is basically that you have a predictive model, and you're going to search the world model to find protein molecules that satisfy whatever design criteria you have.

We've been able to use this to design mini-protein binders, but most excitingly, we've been able to use it to design antibodies—scFvs—and we're seeing really exciting success rates in a small number of trials now.

Shawn Wang

Can you explain what those scFvs are?

Alexander Rives

Yeah. An scFv is basically a single-chain antibody. It's a kind of therapeutic modality. An antibody has a heavy chain and a light chain, and it basically has a pair—1 heavy chain and 1 light chain—that come together to recognize a target.

There are different variations of these kinds of modalities that are used therapeutically. What's interesting about the scFv is that it has 1 heavy chain and 1 light chain. It's able to form these very complex binding interfaces, where you can have 2 different subunits coming together to engage a target.

These are important therapeutic modalities. Something like a quarter of new drugs are antibodies, so it's really 1 of the critical modalities for medicine. What we're able to see is that you can search ESMC and actually find antibodies that are reaching the level of affinity needed for therapeutic function and activity.

Shawn Wang

The protein design space has kind of exploded in the last 5 years. Pretty much everyone is doing protein design, and many people are excited about protein design. My high-level, naive understanding of the field is that things like mini-binders are quite doable.

People have done that quite routinely and successfully in smaller proteins. By the time you get to nanobodies and scFvs, they’re a little bit harder to design, and antibodies are still actually quite out of reach oftentimes. One of the common reasons for this is that if you’re in the AlphaFold paradigm, you don’t have MSAs, right? The evolutionary pressure for antibodies is actually the opposite, in many ways, of what the evolutionary pressure is for everything else: they go for diversity rather than trying to evolve along a very constrained path. So I’m curious: did you try larger structures, and is that something that you’ve seen success on, or is this something that you still think, for some reason, might be hard to do?

Alexander Rives

We can actually take the scFvs and reformat them as antibodies. I think that would be the quickest approach to do that. We’ve not tried full IgGs. I don’t see any reason why that wouldn’t work. Actually, it’s something we haven’t done yet.

We’ve decided we’re basically releasing this now because we feel like it’s reached a point where we’re seeing a really significant step above what’s been possible in the past. We just wanted to get it out there. I think there’s a lot more progress that’s possible. We have a lot of collaborations to look at some of the other applications here.

The thing about it, right, is that it’s a general model. To me, that’s the most exciting thing about it: a general model for protein sequence, structure, and function. You can search it, and therapeutic design basically emerges from that search.

Shawn Wang

Yeah, the fact that you’re not using MSAs, or multiple-sequence alignments—which was one of the, or maybe the, critical insights that allowed AlphaFold to work really well—and the fact that you didn’t need that in order to make it work basically as well as AlphaFold 3 is really exciting to me. That means that your thesis is, let’s cover the space of possible proteins as well as we can and see what the emergent behaviors are. If this is an emergent behavior that we’re able to replicate, what happens with multiple-sequence alignments when we use multiple-sequence alignments? What are the other things that maybe we don’t have data for but that we’re able to also do in an emergent way?

Alexander Rives

I would say, actually, we’re doing significantly better on antibodies. I think that’s one of the theses that we had: antibodies are not going to benefit from evolutionary information probably in the same way that predicting the structural topology of a molecule will. I think you see that now, where the representation space contains something that’s really interesting about antibodies here.

6. Virtual Cells Need Generalization

Shawn Wang

I want to talk about something, because you mentioned something very interesting to me: virtual cell and how this maybe interfaces with this work here. I’m really interested to know: were you able to find other things in your mechanistic interpretability? What were some interesting things that weren’t just validating biology, but where there was a pattern that was unexpected? Did you find anything like that?

Alexander Rives

It’s complicated. We have to now actually go and validate some of these things, right? I think what we saw were interesting connections. What we can see, for example, is that distantly evolutionarily related gene-editing systems cluster together in this space in ways that are consistent with and reflect our knowledge of the origin of those gene-editing systems. So that’s really exciting.

But the thing is, there are a number of proteins in that map that are brought together in different ways where we just don’t know what they are right now. We don’t know what they do. One hypothesis is that these are novel gene-editing systems. I think in this atlas, there’s going to be some really interesting basis for scientific discovery there.

If you think about how people go out and look for new gene-editing systems, for example, they’re typically mining the large genetic sequence databases, and they’re looking for different sequence patterns or structural patterns that are linked to that. Actually, the first version of the ESM Atlas was used by Feng Zhang’s group to find a new gene-editing system. So I think there’s just a lot of biology out there that we don’t understand that’s waiting to be discovered, and being able to connect the dots between proteins so that we can go from what it is that we know today to make those inferences about the unknown—that’s what I’m excited about.

I think there are proteins for so many applications that nature has probably invented. You think about the thermostable polymerase, which enables PCR, that came from a bacterium living in a thermal hot pool. There may be a solution to climate change somewhere in protein biology. There are probably all kinds of building blocks for completely green chemistry infrastructure out there. There are probably new medicines and therapies, but the question is: how do you find those? I think being able to connect the dots is really one way to start opening up that space of protein biology to discovery.

Shawn Wang

I’m curious: one of the advancements of ESM-C is an improvement in multimer prediction—basically, protein–protein interactions, the ability to predict the way 2 proteins interact. I think you now claim to do better than anyone else, right? Correct me if I’m wrong.

Alexander Rives

Yeah. I think we’re state of the art for open models.

Shawn Wang

Okay. One thing that I know some people would find very useful for virtual cell is just an entire mapping of every single pair of proteins inside the human proteome. Have you thought about doing this as a beginning to a virtual cell—creating that map?

Alexander Rives

I think something like that would be really valuable. The other thing about ESMFold 2 is that it’s a really fast model because it doesn’t require multiple-sequence alignments. You can do inference directly from the sequence. It takes seconds; you can get an atomic-resolution prediction.

That’s one really interesting application at Biohub. The other thing that we’re thinking about is whether we can actually experimentally resolve this. One of the things that we’re building is cryo-electron tomography, and we’re really building systems that can greatly increase the contrast when you’re looking at the cell at the atomic level.

So I think one thing that I hope to see is actually structurally, empirically resolved interactomes at some point in the future. I think there are some pretty big technical hurdles and technologies that have to be developed to overcome that, but I think that’s something that’s going to be possible. We can use computational methods to start to get a proxy for that, and I think that’s going to be really powerful. But I think a lot of the future of structure prediction is going to turn into structure determination, actually—really bringing together these tools that we have for modeling proteins and bringing them together with experimental data so that we can start to develop this picture that’s informed by empirical biology, by what we can observe.

Shawn Wang

So is that the vision here, if I’m understanding correctly? You have maybe a lab-in-the-loop kind of thing where you have an agent that’s talking to your CRO and whatever, and then it predicts a property that you’re interested in. It sequences the genome, or it creates the genome. It creates the protein from the genome, and then it observes it with some version of this microscope. What did you call the microscope again?

Alexander Rives

Cryo-electron tomography.

Shawn Wang

Okay, okay. And then you do whatever experiments, or you observe it, and then you use this as a lab-in-the-loop to say, “Okay, this folds this way. Therefore, I want to check the next one that I want to check,” and use an active-learning system. Is that sort of the vision that you’re articulating here?

Alexander Rives

Well, I think there are going to be a few fundamental principles for the next era of biology. I think it’s such an interesting time right now because we’re really at the beginning of a new scientific paradigm. It’s really just the beginning of it. So what is defining that paradigm, right?

I think there are a few principles. Data generation is going to be really critical. The second is computational, predictive digital representations of biology. We can talk about that: you can think of ESM as being a first generation, AlphaFold as being a first generation of those kinds of approaches. You can start to think about what that looks like as we can model more and more biological complexity in that way.

Then you have the principle of feedback, and you have the principle that intelligence is now scalable and so can be applied to every unit of a biological problem. What would it mean for all of that to come together? I think we’re going to have increasingly capable and accurate digital representations of molecules, genomes, cells, and ultimately physiology.

That's where you want to get. We're going to have to go up that complexity scale, through the levels of biological complexity, which requires traversing a data barrier. There's, I think, data that does not exist that needs to be generated to achieve that level of predictive fidelity. And then we're going to have reasoning. I think what that will mean is that we can reason over thousands, millions, hundreds of millions of scientific hypotheses in parallel digitally, using predictive oracles which can actually predict the outcome of an experiment.

So the scale at which we can ask questions, and the kinds of questions that we can ask, will just fundamentally change. That feedback is going to be critical. The models are going to need to—there's going to be sort of a scaling dimension of this, which is building the data to have those accurate representations, and then a feedback dimension where the models can learn from biology, can reason digitally, can reduce that to a small number of experimental hypotheses, examine the outcome of each of those experiments, update their understanding, and build knowledge in that way. So I think that's what it's going to look like, and we kind of have to build each of those components.

What Biohub is really trying to do is bring together the experimental and technology layers that will actually allow us to have these AI models interact with biology and do experiments. We see incredible advances in areas where we can get feedback computationally, so in closed domains, but of course experimental biology is completely open-ended. The feedback principle there is going to be very different. But there's going to be something like RLVR with experiments, where we can have models that are really building knowledge and learning from that knowledge and being able to develop more and more accurate representations.

7. Biohub Builds The New Paradigm

Shawn Wang

You're the head of science at Biohub. Maybe a fun fact for those who don't know: the science section of Latent Space was basically launched after, or in response to, Mark Zuckerberg and Priscilla Chan appearing on this podcast about 6 months ago. It's actually very exciting to have you here and kind of come full circle. Mark laid out quite an ambitious vision for what Biohub wants to accomplish, and I think you just laid out a very natural successor to that. I think you had just joined—you were there 2 weeks.

Alexander Rives

I joined at the very end of October and launched at the beginning of November.

Shawn Wang

Yeah. One thing I'm curious about is, in your eyes, where is Biohub now? What do you want to accomplish? What are your big-picture goals for listeners who haven't watched the episode with Mark and Priscilla? Have you learned anything even in just the short time of 6 months you've been here? Has the vision evolved, and where do you see this going? How does ESMC fit into this? How does the virtual biology initiative that you recently announced fit into this? And then I think there's several other things that you're working on that we haven't even touched on.

Alexander Rives

Yeah. I'm learning things every single day. But the way I think about it, we're building a scientific institution for this new paradigm. To do that, it's an institution that's going to be powered by frontier experimental biology, frontier technology for measurement and observation, and frontier artificial intelligence.

Shawn Wang

And this is all open source, right?

Alexander Rives

It's a philanthropy. Our goal is to accelerate science. Our mission is to cure or prevent disease. To do that, our belief is that there's a fundamental gap in our understanding, and we need to accelerate science to traverse that gap.

We're really thinking about every layer of biological understanding, from the most basic level, like the atoms of a protein in a cell, all the way to systems of cells in physiology and disease. How can we create models that can capture that complexity and allow us to understand that complexity? And I think, if you think about what the cure to disease looks like, it's not a pill. It's not a medicine in the conventional sense. It's going to have to be a system that is capable of modeling and understanding the underlying physiology of disease in a way that's differentiated for every single human being, for every single different genome. And it's going to have to be able to link events all the way from the molecular scale to the manifestation of disease in physiology.

So it's an incredibly complex, incredibly hard problem. For us, we're trying to ladder up those layers of complexity, and we're trying to build the foundational tools that scientists can use to answer the fundamental questions there. We're creating atomic-level imaging. We're creating light-sheet microscopy that allows us to observe how all the cells move and develop in a developing organism. We're creating spatially and temporally resolved maps of inflammation.

We're creating cellular programming and immune-cell reprogramming to be able to actually design completely programmable therapies. We're creating these digital representations at each of these layers so that we can accelerate the science, simulate what's happening, and make biological matter—proteins, cells, and genomes—programmable. All of that has to come together. If you have the focus and you build the biology and the computational layers together so that they're tightly integrated, that's how we're going to make the fastest progress.

For the last 10 years, I think we've been one of the big champions of open science. We're an organization that does both: we fund and we build. In our funding, we've always supported open science, and in our building, we've always done open science. That's something that's going to continue. It's just really fundamental. We're not a drug development company. We're not trying to generate therapies. We're trying to build the technology that moves science forward.

swyx

I think Mark had this concept that if you provide the right tools, then the entire scientific community can leverage them. So obviously you believe strongly in protein language modeling as a tool. What is the next most important tool for advancing a general improvement in our ability to tackle human disease?

Yeah. I think the next level of complexity that we have to address is the complexity of the cell. This is going to be tremendously hard—billions of proteins.

swyx

So you say it's tremendously hard. If you came and said it was going to be easy-peasy—

Well, I think it's a worthy challenge, but it requires technology that doesn't exist today. It requires new modeling approaches and probably architectures and ideas in machine learning that probably don't yet exist. So there's deep and fundamental problems to solve. But again, I think you take it step by step. We start at the molecular layer, and we know that that is really fundamental, and we can begin to link that to observables in cellular biology.

swyx

I'm really curious because this has been the question that's been on my mind for a long time: we have virtual cell models, we have molecular-scale models, and I've seen a few papers about trying to link them. But what are you guys doing? Because it sounds like this is becoming top of mind for you.

So let's maybe make the analogy with protein biology. What I think makes our digital representations of proteins powerful and useful is that they generalize. They're able to make predictions for proteins that are entirely unlike the proteins in their training data. They're able to generalize so that you can design fundamentally new folds, new binding interfaces, and new structures.

So there's this degree of what we call generalization, or generality. In short, they can predict the outcome of an experiment that we haven't already made, that they haven't already been trained on. For digital representations to be valuable, they've got to be able to be used to answer a new question. I think that's the critical thing.

We're not there with cells. I think with the current generation of models that are being called virtual cells, they are good representations of the underlying data, but they have a very limited ability to predict what will happen when you make a novel intervention in a novel, unobserved context. To be able to answer the fundamental scientific questions about cellular biology, we need a model that can do that. Our thinking about this starts with that idea: what's it going to take to get there?

swyx

Going back to protein-protein interaction, the human interactome—if you had that, just predicting static structures—static structures are, in some sense, not enough for a lot of understanding biology.

swyx

Dynamics are probably, for most people, a much more useful tool to have. You can start with static; it can give you some insight, but it’s very rarely the full answer. So you have a model capable of predicting a lot of different proteins. We probably have many of these resolved in the PDB, and some of them we don’t.

Given that dynamics and interactions are more important, how do you bridge that gap? To me, that seems like maybe one of the key steps in going from a really microscopic model of things to something closer to a virtual cell. You actually have to be able to model local interactions of local proteins, RNA, DNA, lipids, or whatever else is floating in the cell. Is that sort of a goal that you would try to bridge, or maybe I’m misunderstanding? Is there another way you would imagine bridging these two?

I mean, one day it’ll probably be possible to have a computer that can simulate the cell from first principles, but we’re very far from that, right? I think that’s far beyond the reach of current computational technology. I mean, even simulating the physics of the folding of a single protein molecule—we could do it for a few fast-folding proteins, but that’s really about it.

Yeah. So there’s kind of this dual view of biology, this dual, complementary view of biology. One view is that kind of first-principles reduction, where all of biology is explainable in more basic terms—in basic physical, chemical, and biochemical terms. I think historically there’s a long line of research that’s really sought to understand biological phenomena and simulate biological phenomena in that way.

Historically, the field had believed that the solution to the protein-folding problem, or the protein-structure-prediction problem, would come from this kind of first-principles simulation. It really came out of nowhere that this could be solved using essentially pattern recognition, or this type of machine-learning approach. I think historically it has been productive to understand biology through information theory and information.

In informational terms, there are these very basic principles that link the information coded in the genome to the genes that are transcribed, to the phenotypes of the cell that will result. If we could model and understand the cell at the level of its underlying programs, that sort of gives, I think, the right abstraction.

What do I mean by the right abstraction? I mean the abstraction that is possible today, because we’re in the era of information theory at scale. Claude Shannon had this idea of the ideal predictor of the next character, and he had this really beautiful paper where he tried to compute the entropy of the English language and imagine taking an infinite context: What is the entropy of the next character?

At that time, it was unimaginable. I’d say it took a great leap of imagination to imagine that ideal predictor, but today we’re getting closer and closer to being able to build that, and we can do that for text. What would that predictor be for biology? That’s kind of the idea of ESM: It would learn the underlying structure of all biological phenomena.

If you think about that from the standpoint of the cell, if we can collect enough outputs of cellular biology that we can observe to reveal the underlying programs, patterns, and structure, then we could create the information-theoretic description of the cell. I think that would be sufficient for understanding disease.

swyx

This reminds me of a lot of the work that happens in signaling pathways right now, where you have a protein in a cascade of different protein–protein interactions that eventually cause a phenotypic change in the cell in some way. How do you translate that into something that can be scaled into a model, or maybe something else? How do you, for example, do that?

Yeah, going back to the Bitter Lesson.

swyx

Going back, let’s just get back to the Bitter Lesson.

We need data. I think the reason these advances in protein biology have been possible is because of decades—I mean, for protein structure, half a century—of work to experimentally determine the structure of proteins, and the effort across the scientific world to sequence genomes and metagenomes. That’s created a data set that you can really train at scale and learn these deeper principles.

swyx

But those 2 different data sets are actually, in many ways, quite different. The PDB is a bunch of very painstakingly constructed protein structures, many of which were the subject of an individual PhD thesis. Then maybe similar ones came later, which might have been 10 of them for a PhD thesis.

These people estimate it’s like $13 billion to create the PDB, some very large number. The reason people created the PDB was because each individual protein was independently useful. People didn’t create it for the sake of solving protein structure; they saw that, “This protein, we believe, is involved in this pathway. Let’s understand this protein so we can target it,” and so on.

Of course, there are some caveats here, but at a high level, a lot of this genomic data, especially for humans, viruses, or bacteria, was sequenced for a very specific reason as well, right? It’s great that these are useful after the fact, but I wonder if now, going forward, especially since the Virtual Biology Initiative—Biohub’s Virtual Biology Initiative—is like half a billion dollars, I think, and I’m sure there will be more large initiatives coming from Biohub in the future, you have the chance to be very specific and deliberate, and now collect data for the sake of solving a problem with ML rather than depending on a data set that was curated and created for some other purpose.

Given that new opportunity, how do you do things differently? How do you think about data collection to enable science broadly when you have the option of doing basically anything from first principles?

8. Building The Biology Data Flywheel

A little bit of context: We announced the Virtual Biology Initiative a few weeks ago. We basically said, “We’re going to invest $400 million internally in data creation and development of technology to scale data generation, to be able to increase the number of modalities that we can measure simultaneously.” We also announced that we’re going to commit $100 million to catalyzing efforts outside of Biohub to generate data.

We think that’s a fraction of what’s actually needed to do this, right? But the hope is that by making this initial commitment, giving starting funds to some of the groups that are really thinking about this and working to build different core areas of the data that’s going to be needed, that’s going to be a catalyst. That’s going to galvanize other groups to come in and contribute to this. That’s what we really hope to see.

The idea is that this is a broad-based effort, so it’s not just us. I can say what my perspective is on what data needs to be generated here or what can be generated, but we also want to approach this really collaboratively with the scientific community. Part of this is also hearing from scientists what they want.

From my view, there are a few key principles here. The first is speed. It took decades to build the data for proteins, and we can’t wait decades. We need to figure out how to do this in a couple of years. You look at the rate at which general AI is developing, and the limitation in biology is that we’re going to be fundamentally limited by experimental science and data. We really need to work to address that gap as quickly as possible.

One key thing is looking at what technologies we can scale up today to begin to give this picture of the information architecture of the cell. There’s speed, and then there’s also the idea of generalization. Going back to what I was saying before, we want models that can serve as oracles for biology. They can predict an experiment that you haven’t done.

How are we going to be able to do that? We’re going to need to look at a multitude of different interventions in a multitude of different contexts. It’s similar to the principle of training a language model on the internet or training a protein language model across all of evolutionary diversity. What does that look like for cellular biology? We have to scale interventional biology.

That looks like things such as perturbation biology and Perturb-seq measurements, where we can look at combined transcription, imaging, and other layers of the cellular information hierarchy. A number of groups—our teams, as well as groups across the scientific world—are working on problems like this and are ready to scale.

The second is spatial biology. I think that’s going to be really important, and it’s going to help us understand the cell in context. Understanding the cell in isolation is really not what we need; it’s not the goal. The cell is part of an incredibly complex system in the body, and to be able to understand disease, we have to understand how cells interact, the systems that they form, and the circuits that they form. We need to see that.

Spatial biology is undergoing rapid progress and is an area that’s really ready to scale up. That’s kind of what can scale now. Biohub has, over the last 10 years, made pioneering funding commitments in those areas.

We’ve funded efforts like the Human Cell Atlas, and we’ve built Tabula Sapiens, which is a large cell atlas. We’ve built CELLxGENE, which is kind of a database of single-cell transcriptomics. We’re really looking to build on that. I don’t know how many cells there are in the largest efforts; we’re probably around a billion cells or something like that today.

So we've got to go multiple orders of magnitude from that. That involves scaling the technologies that we have now, but it also involves new, next-generation technology. We're also funding and supporting efforts in that area. There, we really want to look more at cross-modality. Can you simultaneously see the phenotype, observe the transcriptional layer, understand what's happening proteomically, and link that to the genome? We'd like to see the epigenetic state as well. We'd like to be able to see all of that.

We're really pushing technology to be developed faster so that it can reveal more of those connections and more of that biology, and do that in a more scalable way.

Shawn Wang

It's interesting because when I hear most of those ideas, they're often the things that people already think about in terms of scaling biology. What is the next technology that's going to enable data collection? Going back to the theme of the Bitter Lesson for biology, you don't just have scaling laws on compute and parameters, but now the scaling laws are probably in data collection in some meaningful sense. Where are the next big opportunities there? You're talking about developing new technology as part of this initiative.

Alexander Rives

Yeah. I think it's basically the things that I'm saying: scaling what we have now, being able to expand the number of interventions that we can look at, expand the number of parameters that we can measure—really, more and more multidimensional measurement—and drive down the cost and all of that. Better gene sequencing, better ways of encapsulating cells, and being able to measure what's happening not just in the transcriptome but in other layers simultaneously.

Alessio Fanelli

There's an interesting Pareto frontier there: if you have a fixed budget, how much time do you spend on improving your assay versus how much do you spend on actually scaling it? Where do you weigh in there?

Alexander Rives

We have to do both of those things, right? I think with current technology, we can definitely get data 10× to 100× where it is today with relatively reasonable investments. But then to get another 10× or more beyond that, that's going to require a lot more technology development.

The other really big principle is going to be feedback, and I think that's going to be really critical. You can see that as a layer of technology development that's going to need to occur. There's a lot of great things happening right now—automation, flexible robotics—that's going to accelerate where that can go, and experimental design as well.

Alessio Fanelli

We typically ask our guests what bottleneck they would remove that would sort of unlock things, but we just spent a long time talking about that.

Shawn Wang

Yeah, I want to ask that question, but I'm going to give it a spin: maybe something a little bit outside of your domain, like language modeling or supply chain—something that is a bottleneck that is maybe nonobvious and not directly something that you are working on, but that maybe has an impact on the work of biology or Biohub in particular.

Alexander Rives

I mean, it's a hard question to answer because there are just so many bottlenecks. The one that I always think about is compute, but I think that's a pretty obvious one. It's the bottleneck for all of AI in many ways right now.

Especially because we're training these large-scale models, we're always focused on compute. I think we're limited both by the data and compute. We're in a position where we have incredible compute resources for a team working in biology. But, like all teams working in AI right now, really the limit is just how much compute power you—

Shawn Wang

So if you could 100× your compute, do you think that ESM-C would be way better?

Alexander Rives

It would definitely be way better. We also need to scale data, so both of those things would have to happen in tandem.

Shawn Wang

Have you basically exhausted what's available right now for—

Alexander Rives

I don't think so. No, I don't think so.

Shawn Wang

Okay. The large datasets out there, or—

Alexander Rives

Well, I mean, more parameters, you know. We trained ESM-C up to 6 billion parameters.

Shawn Wang

Oh, but I'm saying in terms of data available: have you exhausted most of what's publicly available in terms of—

Alexander Rives

No, not yet. And the atlas that we just built actually has more sequences and structures than ESM-C was trained on.

Shawn Wang

So you definitely have a little room to go. Is that an order-of-magnitude jump, or twice as much? How does that work?

Alexander Rives

Yeah, I mean, ESM-C is trained on, say, on the order of 1 billion sequences. There's definitely probably on the order of 100 billion sequences.

Shawn Wang

That's large, but a lot of them are largely redundant. 100 billion.

Alexander Rives

Yeah.

Shawn Wang

Okay. To get that billion, you whittle down from 6 billion—6.8 billion, right? So, of those 100 billion, if you were to similarly cluster and find unique ones, where do you think it would land?

Alexander Rives

The sequences aren't actually redundant, right? It really depends on what you mean by redundancy, because I think there's a tremendous amount that you can learn from small genetic variations. These are really revealing of the very basic determinants of protein structure and function at a very fine level.

As we think about protein space, having a vast diversity of sequences across a wide range of protein families is really critical for the emergence of this kind of structure-prediction capability, because I think large diversity is what trains the model to understand—to develop a representation of structure. But I actually think that to develop a representation of function, it's these very small variations that are important.

I do think that there's probably a lot more. The models haven't yet been trained at that level of really deep understanding of these very small but critical patterns in sequence. A single mutation is enough to destroy the function of a protein.

Shawn Wang

So you could conceivably actually take all 6.8 billion of those and retrain—everything's the same, but—

Alexander Rives

Yeah, you could train on more than that. Even that is kind of clustered down, so—

Shawn Wang

Yeah, maybe the question is: how far until you hit the law of diminishing returns here? It sounds like you have plans for an ESM-4 or an ESM-3, or whatever you want to develop. I'm just wondering: at some point, is this actually something that you could exhaust? People talk about exhausting the pretraining data in—

Alexander Rives

Yeah, at some point. Yeah, at some point.

Shawn Wang

But it's not actually something you could conceivably imagine doing in the next few years. Even if you don't exhaust it, you could hit a lot of diminishing returns for the applications that you're trying to predict here, where maybe your resources are better spent somewhere else.

Alexander Rives

I mean, it's basically an empirical question, right? It's truly an empirical question. And so we just don't know. With ESM-2, we weren't sure because there were some diminishing returns. With ESM-C, now there aren't, right? So you can kind of look at that, extrapolate from the scaling law there, and there is enough data to train that next model.

Alessio Fanelli

And the other question that we usually ask is: any call to action? What do you want people to go take action on? If the listeners want to get involved, get hired, or build things, what would you ask people to do?

Alexander Rives

Well, we just announced—or, I should say, at the time that this podcast comes out, we will have announced—ESMC and this world model for protein biology. It's going to be open source. It's MIT-licensed, and we want people to use it. We want this to be a tool that can unlock science.

We're excited to collaborate. We have a team that works on that, and we want to hear from people and understand what we can build that can help to accelerate their science.

Shawn Wang

Yeah, we might have a demo/paper club of some sort on this channel, so stay tuned.

Yeah, stay tuned for that. We'll invite you and your team, whoever can make it. We'll feature this paper once it's in final preprint form and spend some time on it for an hour on the Latent Space Paper Club.

Alessio Fanelli

Yeah. Thanks for chatting with us.

Awesome. Yeah, great to meet you guys.