🔬生物安全是一场 AI 军备竞赛——Eric Nguyen（Radical Numerics CEO）

As the design side becomes more capable, the defensive side needs to match every effort to push forward. I think there is essentially an arms-race dynamic when the defensive side is significantly behind. Therefore, we want to bring the defensive side into alignment.

We believed it was important that the same laboratories and teams developing design capabilities also be suitable for developing defensive capabilities, because they are essentially the same models. A model that is good at generating sequences can also distinguish or assess whether a sequence is pathogenic. For us as a company, it was very important to have a dual mandate: to be aware of and feel responsible for the capabilities we provide on the design side. If we are going to create models that can design functions in a sequence, we see a gap that companies could fill to protect this technology.

Welcome to Latent Space. I’m Brandon. I’m developing RNA therapy at Atomic AI. I was joined by my co-host, RJ Honicky, technical director and co-founder of Miro Omics. Today, we welcome Eric Nguyen, CEO and co-founder of Radical Numerics.

Eric started his doctoral degree in Chris Ré’s lab. He spent a lot of time thinking about how to create long-context genomic models, even before long-context or genomic models became cool. He was, I think, the first author of the generative model Evo, one of the first platforms for generative genomics. He developed Evo 2, which naturally led to Radical Genomics.

Did I miss something?

Sounds great.

Yes. Welcome.

Thank you.

Eric, let’s talk about Omni and your blog posts about benchmarking. But first, I want to hear: what is a genomic language model? Why do we care? What does it do? And then let’s talk about the main results from the blog post.

1. What Genomic Language Models Do

A genomic language model, or GLM, is a large language model trained on DNA sequences. It is very similar to the natural-language chatbots you see, but it is not trained on words or natural language. It is trained on DNA, the fabric of life—the sequence of letters that make up DNA.

Our company and our team are known for creating some of the first generative genomic models: models trained on DNA not just to read, but also to write—to generate new DNA sequences. We felt that this was an underestimated area. If AI could read and write DNA, it could make a major difference in scientific discovery, our understanding of human health, and the ways we develop treatments. We felt there was a great opportunity to train AI on the genome.

What is potentially possible to do with such a model?

2. The Long Context Breakthrough

A great starting point, when we first began working on DNA models, was a model called HyenaDNA. It is a large language model, but it uses a different operation instead of attention. Here are some more technical details: DNA has the property of being very long. At that time, large language models had limited context windows and limited ability to process long sequences.

We were looking for a more efficient algorithm that could process something like DNA. We came up with what we called the Hyena operator, which uses convolutions. In short, it allowed us to process longer sequences—in this case, up to 1 million tokens—which at the time was the largest context window for a language model.

What we did was essentially use it to read DNA and predict its functions. Given a DNA sequence, a string of characters, we could predict its regulatory function and its impact on the genome. This is interesting for scientists because there is a lot about the DNA in our bodies that we do not understand.

We know that DNA encodes information that makes us who we are and influences our traits and potential diseases. At the same time, the grammatical rules for how these letters form combinations, what they encode, and how they encode functions and traits are not completely understood. We hoped that using these DNA language models would allow us to infer a certain function from raw DNA sequences.

Our first-generation HyenaDNA models were able to show that we could teach AI to read and understand, in a certain way, the complexity of DNA, especially what we called long-range interactions across sequences. If you use chatbots, for example, you may have heard the phrase “context rot”: the longer you enter data into a language model, the more its performance begins to deteriorate.

We tried to give the model the ability to collect long-range information and identify patterns, motifs, and grammar across long sequences. In those first generations of HyenaDNA, we demonstrated that this was possible over contexts of more than 1 million tokens.

What can be achieved with that—something that cannot be done in laboratories?

3. Evo Starts Writing DNA

That led to the model called Evo. With Evo, we tried to demonstrate not just the ability to read DNA, but also the ability to generate it. We wanted to accelerate how biologists and scientists learn from biology, and genomics in particular.

Generative AI has clearly accelerated our understanding of and ability to manipulate natural language. But DNA is another language—a language of the genome that we do not yet understand—and, just a few years ago, it was barely being used with artificial intelligence.

The models we saw were very small and had short contexts, so they could capture only small patterns and limited context. None of them generated DNA; they could only read it. We felt that the idea of generation was so powerful and transformative in natural language that, if we could bring it into biology and DNA, it could achieve something that could not be done in the laboratory.

What does that open up for you?

One of the first things we demonstrated that interested people in this area was the CRISPR-Cas system. This is an enzyme capable of cutting DNA, and I think the Evo models particularly enabled the generation of not just one modality or one type of sequence, but several modalities and several scales.

CRISPR-Cas systems involve RNA, DNA, and proteins. At that time, we did not see models that could generate across several modalities. There were protein language models that could generate proteins, and sometimes RNA models that could generate RNA, but there was not a single system for jointly designing them.

We demonstrated that a DNA model could serve as the basis for generating DNA, RNA, and proteins. With DNA, you can obtain RNA and proteins, so we could develop a single system for generation and function in the real world.

We asked Evo whether it could create new CRISPR-Cas systems. We took a sample from the model we had trained, and we demonstrated that Evo was able to create a new CRISPR-Cas system. People were intrigued by it, and the work appeared on the cover of Science. Later, I had the opportunity to speak about it in a TED Talk, which was interesting because the audience was very broad.

I had to think about how to explain what a CRISPR-Cas system is, what DNA is, how DNA is generated, and why you would generate it. There were all sorts of interesting topics to cover.

I think an even more fascinating development was that scientists demonstrated only last year what could be done with generative models for DNA: the creation of the first genome from scratch using artificial intelligence. People usually imagine copying and inserting parts of other genomes or other DNA into something else. They may delete small motifs whose functions they understand and modify rules they already know.

But creating something from scratch, at the genome level, had not been done before. Evo was able to create a functional genome—in this case, a bacteriophage, also known as a virus.

4. Generative Biology Raises Stakes

That was a key turning point for the scientific community and for us at Radical Numerics. We felt it demonstrated both the potential to create an entire organism that does not exist in nature and the potential for harm. If you can control and manipulate the fabric of life and control its functions, what are the consequences? What are you bringing into the world?

We received many reviews, comments, and requests from people who were simultaneously excited and concerned about this capability and the trajectory of its development. These are early stages, and it is difficult to predict what they might lead to.

As a company, we felt it was important not only to advance the biological-design capabilities of these models, but also to use them as protective tools against potential misuse and the biological risks that could arise. Many companies and advanced laboratories are concerned about the risk that artificial intelligence models could design biological sequences.

At the same time, these systems are mostly approached from the perspective of natural-language-model safety, with safety filters and things like that, as with Claude. If you talk about viruses, those systems simply shut you down.

Hmm, which is great. I think that, to some extent, safety precautions are needed at the level of natural language. But precautions are obviously needed at the level of biological sequences as well. Therefore, you need models that not only understand language and the trajectory of your chat, but also understand the substrate itself. That’s the next step and the final frontier, right?

If you have models that can understand whether a sequence is pathogenic or viral, that’s one level of protection that you need. Another possibility is system observation for national security: having the opportunity to monitor new sequences in the surrounding environment. That makes this type of opportunity possible. We believe that bringing this advanced technology to this community is also important, just as important as using it for human health.

There was an evolution, as you mentioned. There was HyenaDNA, then Evo, Evo 2, and now Omni. Can you tell us about Evo and Evo 2, which were difficult to outperform specialized models in many tasks, while in the blog you talk about how Omni is actually able to surpass them in a wide range of tasks? Can you tell me about it?

5. Omni Makes Genomics Useful

Yes. Evo was interesting for people in many ways. It demonstrated potential for application to different types of modalities, but it was still behind many specialized DNA models, especially in genetics or human genomics. So, although Evo was competitive, it still wasn’t advanced enough to overturn the idea that specialized models might be better. Some parts of the community thought, “Why use a giant LLM when I can use these smaller, more specialized models?”

What we wanted to do with Omni, among many other things, was demonstrate the idea of intermediate and post-training, or general alignment. Our approach to language models in biology and genomics, in particular, is that we mostly saw basic pretrained models. They were pretrained, but not actually aligned. In a similar space for natural language, that would be like pretraining. But in order to make a model really useful in the real world and answer questions that users actually want to ask, you need a whole series of alignment, intermediate-training, and post-training steps to make the model ready for use.

We felt that Evo just demonstrated the potential of pretraining, but Omni is a step toward making it useful for people such as scientists. We spent a lot of time on alignment, as well as on intermediate and further training, which essentially means demonstrating tasks in the form that people usually want when they are trying to understand genomics: considering wild-type and disease mutations, helping find the causal variant, and so on. These are the types of questions and formats in which you may want to analyze genomics; they don’t necessarily arise by themselves from pretraining.

Pretraining is a task involving prediction of the following token or filling in masked tokens. I mainly think about it as the raw ability to create embeddings, which is what you teach in pretraining. But then, taking these learned embeddings or representations and applying them to specific tasks, or to a whole range of downstream tasks, and labeling them—showing you the result in a way that is meaningful to you—is necessary to fine-tune and manipulate the model.

So far, the advanced research laboratories are those that have moved this research forward in the natural-language community. That’s why we wanted to attract many of these researchers, and even more, to genomics. I think Omni is just a preview to demonstrate this potential. With Omni, we were surprised that it started to be state of the art and to expand the boundaries—not simply performing well on a few tasks, as Evo did, but actually expanding the limits of each of these areas: prediction of variant effects and causal mutations for diseases. It could be useful for human genomics.

We’re very happy to share this with people. That was just a preview in this sense, because we are still actively learning and implementing additional methods in the model, such as additional modalities. But I think we were basically too excited, and we wanted to convey this to people faster.

Some of the interest we received in the early stages came from clinical systems and nonprofit organizations that have a lot of genetic data. For example, they know that a patient has some condition or symptom but cannot find out which parts of their DNA cause it. For them, there are VUS, or variants of uncertain significance, and we are extremely happy to apply these models and actually help diagnose many of these patients. This is one example of a real application that I’d like to talk more about.

I’m interested, from a technical point of view, in what this looks like, or what it means, to train a genomic model. I can imagine that in large language models there is a certain natural chain of thought; with reinforcement learning, there is a clear paradigm. I wonder what it looks like for a genomic model, which perhaps is better not described as something analogous to a chain of thought in language.

Yes. I think many aspects can be described as analogous to language, but beyond that, I would say the key components are correctly structured input data. You feed them in a certain sequence so that the model knows what is required of it for a certain task. There is a combination of special tokens. In essence, you can imagine it as if you are going to do disease prediction: for disease A, you expect this special token.

It’s a way to indicate what you expect the model to do. If you expect it to do design, there is another special token, and then you demonstrate examples, as in a chain of thought. In other words, you demonstrate the desired sequence of results and their trajectory.

This is a little vague intentionally, because it is part of our secret sauce, which we are still developing. I think that over time we want to demonstrate more and more of this. But in many ways, it imitates a lot of natural language. A lot of this is fine-tuning, but it is also curating specific datasets on which you want the model to focus. Then you structure questions or tasks in a certain way, unlike pretraining.

Pretraining is just the introduction of data, and everything that happens is a prediction of the next token or masked filling if you are engaged in masked language modeling. Pretraining doesn’t have a question-and-answer structure in which you have a question, a hint, and then a result. You can imagine mid- or post-training as demonstrations: “Considering this type of input data, I’m expecting this type of result,” whether that is an assessment, a predicted score, or a design. This is basically mid- and post-training.

Post-training also includes things like reinforcement learning. But I think the bigger steps are implementing structures such as questions and answers.

Do the models have several heads that are specific to specific tasks? Do you train one set of heads, or something such that it can answer a few questions simultaneously? Do you simply change the input tokens or something like that?

Yes, in general, it’s a bit flexible. I think we’re flexible on this. Sometimes you need to use different heads, but the idea for us is to unify them more. In the early experiments with Omni, there were different heads, but in some cases different heads work better, and in some cases a unified setup—one head, one model—works better. So we’re flexible on this question.

But in general, the direction in which we’re moving is a single head. One special motivation for this is that we’re trying to cover many modalities and many generalizations. I think the more we can unify, the more capabilities emerge. This is largely motivated by our work with DNA.

We felt that many models specialize in other modalities a little further downstream from DNA, such as RNA, proteins, or small molecules. We mostly believe that DNA is the basis, and that from DNA you can learn many other modalities, potentially all of them. I think about other modalities as additional context that you show the model.

Yes, I’m philosophical about it; I perceive it in my head. But the idea of unifying models across different modalities is what the laboratory is working on. Can you tell us about several of the tasks you discuss in the blog, just to explain them for listeners? Tell us about some of the main results, and maybe we can delve into them a little.

6. Omni Predicts Noncoding Disease

Of course. One of the areas that we really believed was interesting to demonstrate in this specific release of Omni was understanding variants and their effects, which are essentially changes in DNA. Changing one letter to one of the other 3 letters in your genome can sometimes cause disease, and sometimes it does nothing. In fact, it often does nothing. But there are certain areas in your genome where, if you have another variant or another letter, it can cause disease.

In many cases, because of the combinations of these changes across a genome with more than 3 billion letters, the space is so huge that clinicians—and we, as scientists—know only a very small part of which variants cause disease and which ones do not. That is why there are benchmarks built from variants collected from hospital systems and clinics. Some of them are known, and some are still unknown. People have created benchmarks with ClinVar or TraitGym to explain whether, based on a mutation in DNA, it is possible to determine whether it will cause disease or not.

This is a very good framework for DNA models because they are probabilistic. When you make changes, they can change the model's confidence, or its probability of predicting the next letter. In this case, we used the predictability of these models. They have seen and studied such a large amount of DNA—particularly human DNA—that they understand what is common in human genetics and what is usually benign or conserved. If a variant occurs frequently among other people, it usually means that it is healthier.

If a variant occurs less often, the model can use that to detect and predict that it is potentially pathogenic or causes disease. We took some of these control metrics, and when you enter a variant or mutation, there are different types of mutations and variants. Sometimes you can completely delete a letter, reverse it, or delete large parts of DNA, but usually they are individual variants.

Previous DNA models had real difficulties, especially on people. We demonstrated that our model is not only competitive and capable of working with people, but also, in many cases, the most modern model. I think the most interesting thing is that in the areas where the other leading models are still significantly behind, from the point of view of where those regions are located in the genome, our model performs well.

The genome contains coding and non-coding regions. Coding regions are the areas that actually encode protein structure, unlike other areas that perform different functions, such as regulating gene expression. These non-coding regions are largely regulatory: they control how much, or when, a certain gene is used or turned on. In many cases, variants and mutations in these regulatory areas are much harder to predict as either disease-causing or benign.

I think the most interesting thing about the new generation of models that we created with Omni is that this is exactly where we shine. The models are capable of detecting mutations and distinguishing whether they cause disease in non-coding, and especially long-range, regions. This is particularly interesting for geneticists who have tried to use traditional bioinformatic tools or statistical methods, because those methods mostly focus on coding regions, which constitute only approximately 1.5% to 2% of the genome. It turns out that many, if not the majority, of diseases involve these non-coding regions. Therefore, there is a real desire to create models that can discover pathogenic variants in these non-coding regions.

I have several questions. First, while we're here, for the listeners, there is this column on the comparison chart labeled “zero-shot” in Figure 4 of the blog post. Could you tell me, for historical context, what this column represents? Maybe this could also help provide context for the Omni column.

Yes, that's a great idea. In this benchmark, we show some representative models, or the strongest models, in deep learning, as well as traditional methods. We have Evo 2, the latest genomic model that our team worked on. Then there is Borzoi, which is also a DNA model, but of a completely different type.

Essentially, Borzoi is a supervised model that provides functional genomic DNA tracks. It is also multi-task by nature, but it is not a language model, so it does not predict the next token. It moves directly from DNA sequences to functional genomic tracks, such as chromatin accessibility or gene expression. These tracks were extensively annotated by people doing their dissertations and so on.

A big difference is that it is a supervised task, right? It requires labeled data. In our case, these language models do not need labeled data, isn't that right? You pretrain on raw, unannotated sequences. Why is this desirable? There is a lot more data—much more genomic data—that is not annotated. In fact, most of it, almost all of it, is not annotated. So the opportunity to learn in an unsupervised way is extremely desirable, isn't that right?

For us, we wanted to demonstrate the advantage of pretraining on unprocessed genomic sequences and compare that with the most modern models on other DNA regions. When you direct the model to specific tasks, there is a supervised step. We take a smaller set of labeled data, but we actually use plausibility metrics.

The raw language-model results can be considered the probability of predicting the next letter. We can essentially calculate a probability measure: the probability for mutations compared with the wild type, or the reference genome compared with the mutated sequence. We then have 2 measures, and we can consider their ratio to show how much the mutation differs from the normal or baseline sequence. This can be considered a surprise factor, which the model can use to evaluate whether it actually forecasts disease or not.

Is Omni autoregressive? Is it diffusion, or is it something else that you can tell me?

Yes. This time, we do not describe the exact composition, but Evo was autoregressive, and I think it was the first large-scale autoregressive method. For us, we are not limiting ourselves to a specific training objective. We use essentially every tool from the toolkit.

For this specific benchmark, you simply use the distributional plausibility of tokens. Some tokens the model considers unlikely, probably because of some kind of evolutionary constraint. If something rarely manifests itself in genomes, it is likely to cause problems, and people will not survive, and so on. So this is essentially your metric for ranking or something like that?

Yes, that is one interpretation of how the model thinks about it. It is very similar to how, in natural language, we can describe the same paradigm.

There is additional training during and after pretraining, because we can teach the model a specific structure and set of guidelines. It can build on what you just described—what is common in nature—but because this is a specific task, forecasting pathogenic variants, the model also receives additional intermediate-task training to demonstrate and add that capacity.

What are some examples of this? I am probably asking for some of the secret ingredients again, but can you briefly describe what it looks like? What kinds of things would you add there?

We would actually add self-assessment. This is what I mentioned: the ratio comparing the wild type with the mutations. I would say that this is more of a zero-shot method, where you do not even need intermediate training. This is what Evo 2 does in particular in the zero-shot column.

There is also an Evo 2 column in which people actually fine-tuned the model using their own results from Evo 2. This work is from GoodAI. We respect these people very much, and they demonstrate that these models are capable of being state of the art when they are fine-tuned, rather than simply producing a zero-shot result.

Our model implements something like a step up: not just fine-tuning, but also implementing structure. In the format of these benchmarks, the model is trained on a question-and-answer style, which gives us an extra push.

I understand. That extra push will probably be beneficial. We did not emphasize this very much on the blog, but I think it is convenient for practical use by scientists, because you can do this without taking embeddings, putting a head on top, and then doing some regression. That is an additional step and an additional obstacle. Can you imagine ChatGPT if every time you asked a question, you had to configure it for a specific domain?

We made it so that the model is flexible during training and can be taught many tasks simultaneously. This is fine-tuning; that last step of learning embeddings does not need to be done. At this stage, the model is already ready. You simply ask in a certain format, and that particular format tells it which task you are going to perform. Then it outputs the answers in the desired format.

How carefully did you design this scheme after training to avoid data leakage from ClinVar, GENCODE, RNAcentral, and so on? How sure are you that there isn’t data leakage, random or otherwise, somewhere upstream? I wouldn’t be surprised if many of these sequences also appeared in your training data, even without supervision.

The short answer is that we were extremely aware of the risk of data leakage, and we were extremely careful not to mislead ourselves. We have bioinformaticians who can review and curate the data, delete duplicates, and deduplicate at the sequence level, to make sure that anything potentially similar to what is in the benchmark is excluded. If it is, we delete it.

We also have fairly thorough data-quality-control procedures.

Yeah, cool. Before we continue, I think it’s really cool to see that these existing control methods—which earlier were either within the margin of error or simply surpassed by previous ones—have now been improved significantly.

We’re very excited. For a long time, this was the domain of another method called Caduceus, which was state of the art—and state of the art for a reason. But what they did was take the best practices by design and combine them.

They would take another method, even if the best method was a previous model, combine it with an SVM, and just throw the kitchen sink at it. You can understand why that would perform so well. That was the bar for us. We said, “If they’re going to throw everything into it, we’re not going to choose one model and say that we’re better than it. We need to surpass that, maybe not just in one thing, but in everything.”

Our researchers aimed for this—to see whether they could actually improve the performance of each method.

I’d be interested to see whether there’s been any discussion of chain of thought. It was a little mind-bending when I first heard about it, and it would be very interesting to hear what it means. I had to read a blog post to really understand it.

7. Chain Of Thought For DNA

Yeah. I think it’s just a hint at where, in our opinion, we can move the design space and make it more comfortable for people. Chain of thought emerged from the natural-language community. I think Jason Wei from Google demonstrated the first examples.

The breakthrough was the demonstration that language models work better when you show your work, in essence. You show the steps that led to a conclusion or an argument. It turns out that even if those are simple steps, it gives the model a chance—maybe it’s a philosophical question, and who knows why it works—but it introduces more tokens and gives the model more room for reflection.

That’s why people think this is the beginning of thought for language models: the ability to find an answer and think independently. It seemed to work well in language, and in some cases extremely well, so there are many agents that spend a huge number of tokens simply demonstrating their work. A lot of this follows from the chain-of-thought paradigm.

In biology, we saw very little of this. We started to see something like it in protein design, and that’s why we wanted to push it further and investigate whether it might exist in DNA. What does this generally mean in genomics? There aren’t words to describe the kind of thinking you described.

So what happens if you take the paradigm itself and put it into a DNA language model?

What we did was a simplified version in many respects. We had an RNA-seq dataset of aptamers. You can simply imagine these as desired sequences with a certain fitness score associated with them. If the fitness score increases, that’s good.

It was a large dataset, and we wanted to demonstrate that if we showed the model progressively better RNA sequences in a series of steps based on their fitness score—starting with low scores and gradually moving up the chain—the model might continue that trajectory independently. On the last step, could it optimize itself to the point where it reached the best score it could get?

That was the experiment: could we do this? We took the large aptamer dataset and held out a portion—for example, the best sequences—while showing the model only the lower-scoring sequences. We ranked the RNA aptamers so that their scores gradually increased, and then asked the model to continue the pattern.

It turned out that the model was able to reproduce some of the higher scores that we hadn’t shown it yet. We’re currently checking those sequences in the laboratory, so we couldn’t show the results here. But we wanted to know whether the model was merely doing this in silico, or whether it had demonstrated that it could continue the trajectory and create plausible, designed aptamers with higher fitness scores.

Now we believe that, if this works in the laboratory, it could be an extremely valuable paradigm that could apply to practically any other type of biological sequence. Usually, you have a sequence and a certain fitness score or desired result. If we can eventually get models to learn this structure by showing them a series of progressively stronger sequences, the model can then predict the rest. It’s a very powerful paradigm.

To be honest, I’m a little surprised by this. Given the task at hand, that’s a significant improvement. Maybe I’m showing my own bias, but RNA is somewhat known for not having good coevolution data from the perspective of secondary structures. Viral genomes often experience strong evolutionary pressure, but mammalian RNA usually does not.

The prevailing view I’ve seen in the community is that RNA mostly encodes information. Genomes carry many different kinds of information: they encode proteins, they have regulatory elements, and different types of genomes have different kinds of structures. So I’m curious where, in your opinion, this ability could come from in the language model.

That’s a good question. To be honest, we were surprised too. First, the model was pretrained on DNA, and DNA is actually very similar to RNA, at least in a small sense.

I think your intuition that DNA contains a lot of evolutionary constraints and information is probably right. That hints at why we think it’s so important to pretrain on DNA genomes and then add additional modalities on top, because you get a lot of carryover. You also want generalization across modalities, and that’s what we’re working on.

8. Toward Biological General Intelligence

I’m not just talking about it as a company. It’s the idea of biological general intelligence, where we unite many different so-called languages or modalities of biology. Eventually, they all come from DNA. I don’t think people use that fact enough. Usually, these are very specialized, subject-area modality models, and they don’t make much use of the inherent common structure in other modalities.

One example is when people talk about virtual cells. This is a little outside the topic, but they tend to focus only on RNA and transcripts and want to generalize that to describe the whole cell. Obviously, a cell is much more than that. In my opinion, if you want to learn a system, you want to learn from all the signals or sensors of that system.

If it’s a cell, you want to unite and understand DNA, metabolomics, epigenomics, and proteomics. That’s when you’re approaching what people call a virtual cell. We don’t even want to stop at the cell; we want to unite all these sensors across all of biology.

Whether this will lead us to superintelligence—which everyone understands is complicated and tricky—who knows? But I’m sure this type of paradigm will take us much further than where we are now.

That’s my hunch, too. Can you create something much more useful than what we have now? I’m curious: do you focus on eukaryotic cells? Do you focus on human genomes? Have you gone as far as working with viral genomes?

I mean, there are many DNA viruses, but it seems possible that RNA viral sequences may not fundamentally differ from a learning perspective. I’m not sure how much of that you can talk about.

We’re interested in all areas of life. Here, we focused in particular on humans because we thought, what is the area in which the previous Evo and Evo 2 models were not as strong? We received a lot of feedback from people asking, “What are these models useful for?” They couldn’t understand human genomics because it was too complicated from the perspective of grammar and the rules of DNA.

It’s too large. There are too many repetitive sequences and all of that. We wanted to demonstrate that, in our opinion, this is really useful and may be applicable to people. Human genomics is probably the most difficult and complex area in a certain sense.

But I think there is a possibility of applying these models in general to every form of life. We are absolutely interested in prokaryotes and viruses. In particular, we are interested in viruses from the perspective of biosecurity and biosafety.

I think there are also a lot of therapeutic applications that we can learn from microbial life. Maybe this is obvious to some people. In particular, I remember that people used Evo to create the first AI-generated bacteriophage genome. It turns out that you can potentially use bacteriophages for AMR, or antimicrobial resistance. If you have a bacterial infection caused by superbugs—there are still about 2 million deaths from bacterial infections worldwide—the idea of using viruses, especially engineered viruses, to combat specific bacteria has existed for a long time.

This is especially relevant in Eastern Europe. There is potential for creating a new class of antimicrobial drugs that are not antibiotics, but are very similar and can be used in this way. That's why I think we are gravitating toward things that have a high impact and can potentially save lives. That's why we don't stop at only 1 type of genome. I think we are interested in everything that is useful for people.

Perhaps returning to my questions about SELEX and RNA, and following your chain of thought about the evolution of RNA, I wonder if this model was trained on RNA sequences, or on sequences that have a certain evolutionary pressure on RNA structure.

Only during pretraining. During the previous training, it was all genomes and DNA. The only time we injected RNA was for this specific task, where it was only RNA from this dataset—not even from outside it.

I understand. So there really was something along the lines of, “Is there something that encodes the structure of RNA in this model to some extent, perhaps implicitly?”

Yes, implicitly. I would say implicitly.

It is interesting because, as we know from proteins, sequences implicitly encode their structure.

Yes, and we don't add 2D or 3D information at the moment, but we absolutely plan to do it.

So, just so I can understand: first of all, the idea of chain of thought is that it is a demonstration of this. The idea is that, because you can get a training set with sequences that consistently get better at measuring a certain property, maybe there is a candidate for this technique.

Can you describe this particular experiment so that we can understand how you display the chain of thought on this dataset and what the dataset looked like? I know that this wasn't your dataset, but how was the data collected in this way so that you could accurately match some measure of suitability or something like that to a certain phase or part of the dataset?

I am less familiar with how the data was actually synthesized or generated from an experimental point of view. But they were tested in wet laboratories, in the real world, when they were gathered, so they have a certain indicator of suitability. I believe I can provide a little context for this, if you will.

The idea is that you generate a bunch of random sequences, and then you take these sequences and have a structure, an aptamer, which is essentially similar to an RNA switch. When something contacts it, it does something like split off sequences. You can use NGS, or next-generation sequencing, which reads with very high throughput.

So you create a lot of these different sequences. I think in this case they were aimed at a specific HIV protein, genome, or something like that. If it makes contact, you can get a signal that you are getting more reads. The more time the sequences are circulating, the greater the suitability and therefore the more likely the binding. Then you take them, mutate them again, and perform iterations.

Right. So this is an iterative experiment where you gradually use the method in a Petri dish to essentially determine the most suitable thing. You are doing the same in silico experiment, and they actually perform it at very high throughput. I think there are about 10¹¹ sequences—something like a very large number of sequences—being investigated in parallel for this experiment.

But the key is that the biological experiment actually performs filtration, right?

Yes. Can you imagine other types of experiments where you could apply the same idea—for example, the gradual improvement of something, which is very common in biological laboratory work? If you record these intermediate steps, then you can perhaps introduce them into the model and do something similar.

Absolutely. One example is resistance to antimicrobial drugs, which I think is also very interesting: the ability to selectively influence certain strains of bacteria or destroy bacterial strains.

It is measured, in essence, based on a score from 0 to 1, according to how effectively it is done. Therefore, demonstrating gradually more effective phage genomes and relating scores to them fits this paradigm well.

There is one more development task that we are working on with a national laboratory. It is completely different for us as a company, but it relates to developing proteins for extracting rare-earth elements. It turns out that you are interested not only in proteins that can bind to something, but also in selectivity. You want them to bind to one type of rare-earth element.

So there are points associated with how strong the affinity, or binding affinity, is for each type of element. We want to do a similar exercise with rare-earth elements and show a series of gradually desired points—not just binding with one element, but also lower binding scores for others, so that it can be selective.

I think this is a creative way in which you can demonstrate a sequence and a desired sequence with a certain kind of suitability or functional result. There is a relatively intuitive way for people to develop it. In essence, it is just protein engineering, which, in my opinion, is very interesting to explore.

Continuing the theme of mining rare-earth elements, I believe this is an interesting option. In fact, perhaps you don't have comparative advantages compared to some other methods, because it seems that your strengths probably consist of large-scale design of whole organisms, whereas focusing on individual proteins could be much more similar to a structural task.

I am also wondering, when I talk about ClinVar, whether the argument is that the model is a really good statistical reflection of what types of mutations are widespread or rare. I think that, for the design of structures, you need to understand structure. If you want to understand diseases, I think this is much more natural for many diseases from the point of view of population genomics.

So I am wondering: where, in your opinion, is your strongest competitive advantage in using these strategies? And do you think that your model understands structure in addition to function?

We are attracted to this application of rare-earth elements, and our strengths in general—why we thought that we could come up with this—come from 2 factors. First, I think it is in those areas where context has value. You are right: the task of protein design where you just design a structure perhaps isn't natural, and we don't see ourselves as competitively advantaged there.

But in the case of rare-earth elements, the hypothesis is that context has value. Maybe what is important here is certain microorganisms with proteins that have the desired function. We can potentially suggest and provide context for the area in which you should look for new proteins. So it really is something like an exercise in data mining.

What we can demonstrate with our models that other people can't, for example with protein structure models, is that we can model noncoding regions before the gene that we are interested in, or the protein of interest, and then ask the model to provide options.

For example, you could say, “Mutate this protein, but know that you are in this microorganism. Show me different options that you have seen in nature, or combine different things you have seen in nature, taking this context into account.”

I think this is one of the reasons why we can create very evolutionarily diverse sequences and potentially phages. People have used Evo models for this with toxins and antitoxins, using a very similar methodology. They prompted the model based on things upstream of the proteins they were interested in, and then requested that the model generate a heap of other plausible options.

I think this case is extremely fascinating for rare-earth elements because we can come up with plausible options and then relatively simply test them to see what they bind to and whether they bind selectively. In my opinion, that is really interesting for a partner who cares, for example, about the strategic supply-chain security of rare-earth elements for the USA.

So we thought it was totally worth supporting.

So, in your opinion, it is not just because you develop a protein, but because you develop the organism that generates the protein, and that protein has a certain action.

But is it you, or is it AI? What does this method development consist of? It is necessary to understand how, across the tree of life, proteins interact with rare-earth elements or with certain minerals.

Yes. I would say that, in this particular case, for rare-earth elements, we are very interested in organisms that have developed around them. But I think the model tells us which proteins are possible and what their selectivity is, as well as how they evolved—not only around the protein itself, but also around regulatory elements. This helps us narrow things down and provide additional context.

I think of the genome, or DNA in general, as a kind of imprint of the physical world. There are parts of DNA that we need, but the things around those parts matter too. They tell us about the context where it came from, how it arose, and which function it performs.

I do think so. Context has a big importance in biology. One of our largest limitations is a shortage of context and how we, as humans, for the most part, approach biology from a very engineering-oriented point of view. We isolate separate systems, and a systems biology approach is very difficult to achieve with significant quantitative predictive power.

Perhaps this is getting off topic, but I’m curious: if you go to context and think about how context scales to the genome, are you now talking about a context length of 2 million? The human genome is about 1,000 times longer, right? Or even bacterial genomes—if you try to design them, they’re much longer. How do you use something with a long but still limited context for synthetic biology, for example, with larger organisms? Perhaps the context used to design bacteriophage Evo 2 was also much longer than 2 million?

Yes, that’s why I would expect a little more from bacteriophage Evo, because it separated the group that was being worked on. But this context was actually quite short. The reason they started with phages is that they have some of the shortest genomes. I think it was something close to 6,000 base pairs.

Oh, that’s really small. It’s very little.

Yes, exactly. Extremely small. Viruses are crazy. They’re incredibly effective and can do a lot of things with very little material.

I think your question about how to get longer context with something smaller is a very important one. The artificial intelligence community is constantly trying to solve that problem and demonstrate creativity. This largely explains why we’re a research laboratory for artificial intelligence: We believe innovation needs to be continually advanced.

This isn’t a space where we can simply take models from open source and expect them to resolve all the tasks we care about. We want to constantly expand the boundaries. Context is one of the key research areas we’re working on. Perhaps that’s how we got our name, because we were working on long context before it became popular, probably in 2022 or 2023.

Long context. You and your team have a long history of using these state-space models and expanding the context in ways that seemed crazy at the time. Now I think about routine workloads, and books are large context windows, but at that time it was orders of magnitude longer.

I don’t know if you want to talk a little about that. One thing that I find really fascinating, both for these models and for others, is that you’re really following a first-principles approach. You immerse yourself in GPU architecture and design models that use the architecture of graphics processors. It’s not just, “We’ll build a longer model.” It’s, “What can we do to make something special, considering the computational restrictions we have to expand the boundaries?”

If you want to distract our researchers at Arc, that’s how you get them. You criticize them like a bore. You talk about long context and GPU kernels, and they say, “Wait, did someone say kernels?” Then they start trying to understand how to do things quickly.

Long context has a special place in my heart because I concentrated on it in my doctoral dissertation at Stanford. We started thinking about DNA, and, going back a little for fun, we considered language models in general and noticed that our models were good at handling long context. That’s how we started working in this space: These models seemed very effective with long context.

With Michael Poli, my colleague at the lab, who worked on the first Hyena project and its convolutional architecture, we started thinking, “Let’s take this further. What new applications will open up if we really turn to long context?” We asked ourselves, “What is the longest sequence?” Not surprisingly, we settled on DNA. We thought DNA had to be the longest: 3 billion base pairs.

We started thinking about what length context people actually use. They use very short contexts—1,000 or 2,000 base pairs, or tokens, at a time. That’s much less than what’s needed for DNA. Most human transcripts are about 3,000 bases long, so this isn’t even enough to represent, for example, a protein in many cases. Obviously, this was a need we hadn’t noticed.

We first approached it by saying, “Let’s see if we can do something. We don’t know if it will work, and we don’t know who needs it.” We just started experimenting. It turned out that the model performed surprisingly well at reading DNA.

Then people kept asking about DNA. Their questions weren’t really about language. They asked things like, “Can you make it longer? Can you do something with this?” We didn’t really know. Then, for some reason, we grabbed onto the idea: No one writes DNA. Can we make the model write DNA?

It was a very simple question, but looking back, it’s almost obvious that you would want to write DNA and develop it. When I first proposed the idea of Evo to people, I spent 6 months asking them, “If I generate DNA, would you find it useful? What would you do with it? Would you support us? Would you want to be part of this?” In the biological world, 6 months isn’t that long, but almost every scientist at Stanford I talked to thought it was a stupid idea.

I thought, “That would be so cool. We could generate DNA. How much could we accelerate the field?” Then people asked, “What would you do with this?” And I answered, “I don’t know.” I received comments like, “This is impossible. We humans don’t understand the rules, so how can you expect AI to learn them? You can’t even say whether something is correct or wrong. How can you expect AI to learn it?”

I heard that there were too many repetitive sequences, that DNA had a noisy distribution, that there were no real rules, and that it was just a mess. I heard all those reasons, and I was stubborn about it. I felt that there had to be some use case or opportunity to generate DNA. I just felt that it was right, even though I didn’t know what the use case was.

It was really an experiment: What would happen? When we first trained Evo, we didn’t know whether it would work. We didn’t know what would emerge. We just thought, “Let’s train a large model.” It was a little funny, but people at Arc said, “Of course. Why not? Why wouldn’t we? Let’s see what happens. Let’s spend a little money on GPUs and let these crazy children train the model.”

I remember when the first result came in, and it forced us to pause. We thought, “Maybe something is happening.” Someone took the first model checkpoint and ran it on ProteinGym, one of the protein benchmarks. It turned out to be competitive with protein-specific models, which was quite strange because we had never told the model that it was looking at proteins or anything like that.

In fact, DNA and protein are related, right? So it was all one thing. It was just strange that the model was genuinely competitive with protein-specific models. That was the first experience. Then results kept coming in one after another, and the model continued to appear competitive.

We realized that this was one of the most advanced RNA and DNA models, and we began to see that it could learn across different modalities, not just DNA. At that point, we felt that there was something there. The project continued to grow, and people at NVIDIA said, “Let’s support Evo 2. Let’s do it at a larger scale.”

Then Greg Brockman from OpenAI said, “I’ll do it. I’ll take a 4-month creative break from OpenAI and help with these crazy children.” We were messaging Greg Brockman on Slack at 3:00 in the morning, trying to debug our code, which was wild. So the development trajectory was very unexpected in many ways.

But at the same time, we received a lot of feedback, such as: “Why are these models useful? What are they useful for in the real world?” That motivated us to create a company. We thought that what we showed was simply an attempt at an academic approach.

It was similar to language models when they were first introduced. When natural-language models first appeared, people asked very similar questions: “Why are these things useful? Cool, they can write jokes for me.” Would this lead to artificial general intelligence, which could automate everything? They didn’t think so, did they? They thought it was a toy. But there were new opportunities, and they extrapolated the potential.

I think what we see here is, in many ways, even more exciting, or resembles the trajectory of DNA research. But either Evo or Evo 2 showed a bacteriophage, and it wasn’t convincing. I mean, it’s almost a terrible example, right?

As you mentioned, people didn’t see that this is not an easy problem.

Yes, I have to remind you that these are some of the criticisms, right? I think either fair or harsh criticism—you can play devil’s advocate and say that what it generates is close enough to nature, and that it just generates small variations. Therefore, there are many ways to criticize and minimize the potential, and that can be a fair argument.

At this moment, what we believe is more important is not just promoting the science and saying, “Cool, it can be done,” and leaving it as a kind of imaginary experiment. It’s about how we can actually use this to improve our understanding of diseases, improve treatment methods, and create better extractors for rare-earth minerals. How can we actually make it useful? That’s what concerns us as a company, apart from some of these more scientific questions.

Perhaps this is a good transition. That cheers me up. Two things come to mind. First, there’s the question of mechanistic interpretability, which appears at the end of the blog post, as well as biosafety issues. Both are important applications. Let’s start with mechanistic interpretability. Can you tell us a little about it? This is actually ongoing work that was also done with Evo, I think.

I remember that in the Evo paper, there was some work on mechanistic interpretability in which they discussed using perturbations that turn the question upside down and using models to obtain information about biology. Here, this has actually become a common theme, perhaps in biology more than in any other field: because it is a scientific question that studies the laws of the world, people can ask, “What regularities did you find?”

9. Mechanistic Interpretability Begins

I think mechanistic interpretability is a new branch of biology, which is extremely exciting. Honestly, we’re at an early stage. We’re building this team, but what we’ve demonstrated so far, and what makes us happy, is embedding analysis and the study of activations in models.

This idea of mechanistic interpretability for biology borrows a lot from what the natural-language community currently has. You can imagine what models do: they compress the information they see, right? Through that compression, they distill it into key components and key patterns that help us understand the data or study the distribution of the data.

We’re trying to explore models to see whether they distill something into their own weights and activations, or something like that. We started with model outputs—that is, activations—and wanted to see what structures and visualizations could help us understand some of the complexities of DNA, of which there are many.

One of these complexities could be GC content or certain motifs of transcription factors. There’s a whole series of regulatory patterns and motifs that, in our view, the model should take into account in order to perform its tasks, such as understanding whether a disease-causing variant is present. In general, the model compresses all these different motifs and distills them into its representations. Our task is to find them and see whether we can highlight a particular pattern or structure that could generalize to other cases where we don’t understand the patterns. That’s mostly the goal.

In this case, GC means 2 nucleotides that make up a fraction of these sequences, right?

That’s right. It’s the fraction of the letters G and C in the genome. I think that’s one of the simplest things, but other simple things include repeats—the number of repetitions.

I think transcription-factor motifs are another especially interesting area, because eventually we may be able to develop transcription factors. In other words, we’ll have certain transcription-factor patterns that we can study with modalities such as ChIP-seq.

Transcription factors are molecules that bind to DNA and, in fact, change the expression pattern of genes. They have a regulatory effect. They don’t change the DNA itself, but they can change the impact of the DNA and the products that the DNA produces.

That can have a lot of consequences for practically everything in your body. It can change body weight, and it can affect many studies related to aging that are dedicated to designing transcription factors. That’s why we believe that the ability to understand some motifs in DNA, as well as in additional modalities, will ultimately allow us to develop transcription-factor patterns.

That’s why I think it’s very exciting. In many ways, the combinatorial study space of transcription factors—what connects to what, where they connect, and what effects they cause—is far too large to study manually. We want to apply a data-based approach to study some of these motifs.

The difficulty here is partly that transcription factors themselves are encoded by genes. They can regulate one another, so you get this combinatorial effect.

Just to explain it for the listeners, we seem to be moving through increasingly complex and higher-level factors, starting with GC content. We started with the consideration of diseases, which is probably the most difficult thing you can point to in this analysis.

Yes. I think our first idea, before even building this, was the idea we’re working on as a whole: reflecting the variety of diseases. That diversity is its own kind of representational space of what a disease looks like to the model, from the standpoint of latent space or embeddings.

We believe there’s much more structure that can be drawn from understanding some of this original data. A reflection of that diversity, or the landscape structure of diseases, is obviously a large space because there are many diseases. I think it would be a great field of research for us and could actually stimulate interest in mechanistic biology.

We’re just beginning to explore the surface, because if we can obtain so much pure understanding potentially from DNA alone—which, in my opinion, is only one of the modalities that will eventually need to be united for biological modeling—then the possibility of doing something similar across all modalities is enormous.

That includes proteins, RNA, epigenetics, ATAC-seq, chromatin accessibility, methylation patterns, and all these different types of molecular phenotypes around DNA. It’s a huge opportunity that lies before us. It’s all green space, a green field.

I haven’t seen anyone use such complex methods from machine learning and deep learning to the point that, in my opinion, this could become the most important and influential field for understanding and applying AI.

Exactly. This makes us happy. I think we’re just showing the previous generation of models using DNA alone, but our next-generation models will be increasingly multimodal. We’re talking about dozens of modalities. This is a very exciting moment for us to demonstrate that.

Is natural language one of those modalities yet?

Not yet, but it will be.

There are many questions about how to combine this with biology, but I actually think it will be quite simple. I think the harder part is how to combine biological signals. We see many examples of combining language with images and video, so we feel pretty good about that.

We’ve done some early experiments with language, and I think this will make the models more accessible to people. When you can combine them with language, it becomes easier to work with them. But I think this is part of the recipe that people are still trying to figure out: how to do it across various biological modalities.

It seems natural to implement a chain of thought, right?

Exactly. Absolutely. Especially when you’re starting to create chains of thought from tools such as Claude, I think the idea of turning on language is already in the minds of many people.

If you don’t have any questions, let’s talk about biosafety.

10. The Biosecurity Arms Race

For us as a company, it was very important to have a dual mandate, as we call it. This is the idea of being conscious of, and feeling responsible for, the opportunities we provide on the design side.

Therefore, if we are going to create models that can design functions in sequence, we believe there is a gap in how companies can protect this technology and provide for its responsible use.

I was just on a panel last night about scientists working on artificial intelligence agents that can do scientific discovery. One of the last questions was, “What are the biggest risks or end-of-the-world scenarios associated with artificial intelligence that can be used for scientific discovery?” Every panelist spoke about biological weapons. At the same time, I was curious because I thought, “Okay, so what are these people doing about it?” In fact, I haven’t heard much about it. I won’t name names, but when I look at these companies, they don’t seem to be making a great effort in these areas.

In any case, we considered it important, as a laboratory and as a team, to simultaneously develop the best design capabilities and capabilities for developing defenses, because these are essentially the same models. A model that generates good results also distinguishes very well, or predicts, whether a sequence is pathogenic. From a principled point of view, we felt it was necessary to work on both biosecurity and design. Strategically, it also made sense.

That’s why we felt this resonated with our team and, in a wider sense, with communities of people in the U.S. government and abroad who believed there was a clear gap and a need for this type of organization. So, yes, we felt it was necessary to build this into our mission.

For us, biosecurity has 3 or 4 different pillars, or strategies. The first is detection and observation: can something be detected in the surrounding environment, or can you determine whether a sample contains a pathogenic sequence or something that can cause disease? That could mean swabs at an airport, a nasal swab, or a sample from a sewer system. The next is attribution. After detecting a danger, can you find out where it came from? Did it occur naturally? Did it come from some country? Or was it created by a person in a particular laboratory in a particular country? This helps you understand what to do about it.

The next idea concerns countermeasures. Once you have discovered something and found out where it came from, what do you do with it? Is it possible to create an antidote? Is it possible to create an antiviral or antimicrobial agent? The fourth idea is deterrence, but that is more of a government-level question. That’s why we focus mainly on the first 3.

We create tools that can determine whether something is pathogenic. But what we felt was lacking in the community was not just discovering whether something is pathogenic in a broad sense; it was also fully characterizing it—which parts of the sequence or which genes are dangerous, for example, as well as its origin and the possibility of attributing its signatures. Determining origin is not as easy as looking at a sequence, matching it against a database, and attributing its signatures.

Another important component is that the biosecurity community generally focuses on comparing sequences. They take a sequence, compare it with a known database, and ask, “Have you seen this before? Does it correspond to this list of known pathogens?” But I think what causes anxiety in many laboratories is, first, something new that isn’t on your list, and second, things that have been deliberately obscured so they don’t appear in sequence space. The letters may not exactly match, and the functions may not be obvious either.

The model we are launching now will be functionally and structurally aware. This means that you can have a sequence with the same function as a pathogen that looks different at the level of its letters. You can imagine that what we have on the screen is still similar to the original, while there is a diversity that the model constructs internally—something that, in a sense, encodes these functions.

What we are talking about in our blog about protection is the idea that things can function similarly while starting to differ in how their sequences look, all while maintaining the same functionality. This can happen naturally, but artificial intelligence models also allow you to do it intentionally: to preserve the function or functional ability while using different letters and a different spelling to describe essentially the same thing.

A Microsoft paper called “Paraphrases” demonstrates this. You can use protein language models to preserve the same structure, which provides a similar function, while changing the sequence. They wanted to check whether, if you had one sequence and sent it through existing detection systems, it would break the system—or whether the system would actually detect it.

One interesting fact that perhaps the wider public does not know, but most biologists do, is that there are companies that synthesize DNA. You can send them designed sequences and get DNA back—a DNA molecule delivered as if it were an Amazon package. You send it in, and they produce the physical DNA for you. This is a service used by the scientific community. It is a pipeline that allows people to conduct research, understand biology, create medicines, and do all of that.

One of the problems that concerned people, and one of the problems that worried us when we first started working on this, was the possibility of a world in which agents are actively working online—probably billions or trillions of agents—creating and carrying out all kinds of actions on the internet. It is strange that they don’t have tools to ask, essentially, “Is this creating something dangerous or not?” That was literally our first motivation: it was probably worth creating something that could detect whether something was dangerous.

You could have a filter for these manufacturers so they could ask, “What am I doing here? Is it dangerous?” There might need to be an exception if you are a licensed laboratory or something similar, and you are doing something dangerous that you know is dangerous. You could say, “Please allow me to do it anyway.” There are various cases like that.

For context, many companies involved in DNA synthesis have detection tools, but I would strongly suggest that they are not based on artificial intelligence and that they are probably unreliable. For the most part, they compare sequences against known samples.

Sorry, maybe I should save this question for later. I’m just curious: everyone who works in biology has tried to use Fable, and it instantly, literally, excludes everything you enter, for example. From the point of view of scientists who study biology and create or design new sequences, it seems as though it would be very difficult to obtain an ROC curve that you can live with—one that does not prevent legitimate scientific research.

How do you avoid that? Even an F1 score of 0.99 seems far away right now. I think there are billions of sequences, or at least sequences generated every day or every year. It seems very difficult to balance.

Yes, this is a complicated question. In general, if we are asking how to stop 100% of dangerous design, that is a more difficult question. The question we posed relates to developing many capabilities for people who are pushing the limits. When we look at the defense side, do we see advanced technology being applied? I think the answer was no.

We saw this huge gap and wanted to help address it from the defense side. This also helps give it a push. From this point of view, it would be easy for us to choose to move forward and prove that it works better through a direct comparison with the design side.

Will this solve everything? I think that is what we want, but realistically, there will always be cases where someone can get around it. That is one of the main reasons we believe that safety language models in chatbots are only one level. You are right: there may be things that simply slip past.

What do we do with those cases? They are already beyond chatbots, right? Chatbots may already have helped someone create dangerous sequences. I think the coolest thing about the tools we are creating is that we are building tools for people who care about things that already exist somewhere in the environment, somewhere out there.

And now there is an entire ecosystem in which we want to embed observation, attribution, and countermeasures. We want to develop this community and create stronger tools for this space.

Does everything need to be ideal to be useful?

I don't think so. I think we can be useful in moving the biosecurity community forward using artificial intelligence and advanced technologies without everything being ideal. That's exactly how we look at it. Maybe, in other words, the threat model you are facing is what you are actually trying to defend yourself against at the moment, but for now those possibilities generally do not exist.

So, you know what you're offering: something that can work from a broader perspective—with regulatory frameworks, the government, nonprofit organizations, and any other broader bases—and that provides a tool on which you can build a community?

Yes. Even if it's not perfect, this provides a starting point, and if you don't have a tool, then you can't make anything.

The bar is improvement, right? The bar is where you are now. Can we move forward? Can we go beyond criticism of sequence-based screening?

Absolutely. I think there are many, and I think interest only grows. We hear a lot of chatter from regulators, politicians, and people from think tanks. It seems that people are mobilizing, so we are optimistic that this will become known and more important to people, so that they actually take action rather than just talk about it.

One of the reasons we decided to create tools, release them, and give people access was this: a lot of companies engaged in artificial intelligence say, “We should engage in biosecurity,” but then they ask, “Okay, what does this mean? What should I do with this?” That was our approach.

So, I can just steal this for a moment. Can you go to another diagram where you have “Yes”? If you click on them, they will show other similar compounds or similar genomes. Maybe I can just steal some things from the opposite point of view.

If you're on the front line—and it seems that you are, and you definitely want to be—you are simultaneously moving the boundary between attack and defense. What you're doing now clearly helps, but maybe if you're on the front line, it doesn't really have value in the long-term perspective. How are you thinking about it?

I mean, do you just need to nerf what people do, or something like that?

I was hoping that people would start thinking about this not so much as something like, “Let's create a tool for protection and that's all.” That's right—essentially, it's an arms race, right? It's like community cybersecurity.

You will create better technologies, especially something like Fable, which can potentially be used to attack. This means that you will have conflicts. The offensive side will become more powerful; the defensive side must try to step forward, isn't that right? And it will motivate other people to overcome this on the design side.

So I think that, in my opinion, there is the essence of arms-race dynamics, which seems to exist because the defensive side strongly lags behind. That's why we want to bring the defensive side closer to parity. That's the way I think about it; that's the frame I use.

The analogy with cybersecurity is interesting, but I think it differs in some key ways. First, I think that with cybersecurity, with a strong enough model, you can actually close all the loopholes that aren't sociological. There are certain ones that will always be difficult to get around, and there will always be ways to get around them, but in principle, you can identify every exploit and then fix it. While people stay up to date, you're safe.

We have fixed genomes, right? So you can't patch a person. I think that, in a certain sense, the attack surface—or the way to protect against it—is much larger or much more complicated. But then, perhaps on the contrary, it seems a lot less likely that someone will have an incentive to go on the offensive to the same degree.

There is also a similar barrier to success. Even if you can print random DNA, the process of going from that to creating a successful virus, especially one that doesn't kill the person developing it, is actually quite large.

What, in your opinion, is the biggest threat for us, actually? What might keep you awake at night? Is there anything specific, or do you just think this is something we need to create, so let's create it?

First, it's difficult to imagine a person who wants to develop biological weapons. We have to think about all potential actors and scenarios in which intruders could operate. What worries us is that we lower the bar for the amount of experience required and the speed with which people can develop such things, which means that the volume of activity, at a certain point, will just grow exponentially.

There are deliberate threats, because of course there are state entities that have had very large biological-warfare programs. One of our company advisors personally saw these facilities and took them out of operation. So we heard many stories about how motivated states really are to create such weapons. This is one of the questions that raises concern. In peacetime, it's less scary; in wartime, it's especially scary.

Unintentional cases are also potentially more likely in the near term. When people are trying to create things and perform gain-of-function research, they may unintentionally cause something to leak. They may be trying to create a certain thing for understanding and study, but it leaks, doesn't it? These things can be difficult to contain, for example, and there is a leak. So I think these scenarios are potentially the most likely.

Does that keep me awake at night? Not necessarily. I am generally much more optimistic. I think that creating this type of technology is, after all, a game where you weigh all the pros and cons, costs and advantages. I think the advantages significantly prevail over the potential harm, and so we are working on this because, in the end, we think that it will be an engine of discovery and improve human health.

At the same time, we felt that the protective side seemed to be losing this arms race. So we are also working on this; this is what we are trying to do—to expand the boundaries. But I would say that, in general, as a community of researchers, as well as people from the policy side, I think the community is stable enough and has the ability to mobilize to get ahead of it. So I am very optimistic about this.

We have 2 questions that we would like to put to every guest. First: if you could, by fiat, eliminate the bottleneck that is important to you, what would that be?

Interesting. Can I give 2 answers to this?

Of course.

Good. The first is a kind of excuse, because every artificial intelligence laboratory says this: GPUs. We've answered this question a few times. We could use GPUs.

Another point that, in my opinion, is a little more philosophical: I think that in this area, and in what we are striving for, we need to rethink how scientists do their work in this field. One obstacle we face is people—you see it in many industries—who have more experience in an industry and are, for some reason, more pessimistic the longer they stay in it.

I think you especially see this in biology, where you know the area—a disease or a treatment method—so well, and then someone introduces something new and you think, “Oh, what about this, this, and that?” They are very pessimistic, and that's probably fair. But I also think that what I have noticed in the companies we are trying to build, and in the people we are trying to attract, is that they are experts in these industries who know the industry, but they are also dreamers. They still have this imagination and desire to change how things are being done.

I think this is a barrier we often see in this industry. If we are more accepting of this, we will be able to see much more progress and the rapid changes I would like to see.

That brings us to the last question, which goes like this: Is there anything you want the audience to learn? One message.

Yes. I think one of the messages is that people, especially in the artificial intelligence research community, sometimes feel that they have to make a choice: work on cutting-edge AI, for example, or work on consumer or corporate applications and just work on chatbots. But chatbots are advanced technology too.

When people think about the true potential of what artificial intelligence can do—and I think many of these concerns involve improving human health and understanding our biology—they feel that they have to choose. They think, “If I work in this area, I can't work on advanced AI.”

I would like people to learn that you don't need to choose. I think we can work on things that you really like or worry about, that you think will promote humanity, and work at the frontier of technology.

And this is exactly what we’re trying to create at Radical Numerics. I mean it clearly, and I urge people to read the posts on the blog. In the architecture—Mamba and Transformer—there are many innovations involved in creating these models, and I am firmly convinced that biology is really at the forefront of AI.

Perfect. Yes, I’m glad that you think so. Thank you for joining us.

Yes, thank you for that. I made a long trip—three-quarters of the way, I would say. But anytime. Thank you for the invitation. I had a great time.

Perfect. Thank you.