一项自带标准答案的 AI 科学家评测——Ian Quigley，Leash Bio

Hey there. Welcome to Lane Space Science. This is Brandon and RJ, joined with Ian Quigley from LeashBio. We recorded an episode earlier this summer with Ian, where he talked about the cool things Leash Bio was doing with DNA-encoded libraries and how they could screen millions of compounds against thousands of proteins in a high-throughput way. Around this time, Ian realized they could do fun things with this data. It essentially allows them to create a virtual sandbox where agents can explore and make decisions in real time, as if they were medicinal chemists, and get instant feedback while running a huge number of exploratory rollouts. Since that episode, Ian and Leash Bio have developed this idea much more fully. They’re now releasing a paper, a dataset, and a blog post, along with the episode we just released. We wanted to sit down with Ian, talk a bit more about what he’s done, and hear about the updates. Anyway, welcome to the podcast. Thank you, Ian.

So last time we talked, Ian, you were saying that you had some agents exploring a sandbox. You said, “Oh, this is just a fun idea. We don’t know what we can do with it, but it’s an interesting idea.” I think you’ve since fleshed this out into a full environment for agents to explore, maybe do RL on, and do scientific reasoning. Can you talk a bit more about that?

1. The Medicinal Chemistry Sandbox

Yes. The thing I talked about last time was the idea that we had these large collections of screens, and that they were big enough environments and counterfactually dense enough that you could drop agents in there, give them tasks, and have them do those tasks on a limited budget in a landscape that we had measured completely. The flagship problem we’ve spent a lot of time on over the past couple of months is a structure-activity relationship problem. The idea is: I’ve got some molecules, and I know that they bind—maybe they bind to the protein I want them to. How do I change them to make them better? This is a very common task for medicinal chemists.

SAR, as it’s abbreviated, also includes other things, like certain drug-like properties: Does it get metabolized? We’re not doing those things right now. We’re just asking whether we can find molecules that bind better to the thing we’re interested in. We take an agent, like a language model, and say, “Here’s a catalog you can look at. You have a budget of 100 molecules. We’re not going to tell you anything except the target. Go find the best binder you can.” Different agents try different things. They have different search policies and different ways of wanting to spend their budgets.

They put out a SAR hypothesis, in readable language, about why they think these molecules are or aren’t working. Then they test those hypotheses with this limited experimental budget. Because it’s all pre-measured, you can roll out a lot of these. We’re up to about 90,000.

2. Measuring Scientific Agent Performance

So you started doing these on foundation models. I think you have a paper or a preprint or something at this point that you can share. Do you want to share that or talk about it?

Yeah. The preprint will probably be out at the same time this episode comes out. It’s pretty ready. What we show in there is that we get great separation of agents across model families and at the levels, that kind of thing. Another thing that’s really interesting is that we have not just the results you could use to evaluate how the agent is doing the job; we also know the results of the entire measured landscape, or world. So then you can start to interrogate how good the agent is at making scientific decisions. Maybe it only got to the thing that’s half as good as the best thing in the entire environment.

We see all kinds of interesting behaviors: agents that give up early, ones that buy molecules that don’t agree with the rest of their hypotheses, ones that keep retesting the same molecules despite having a limited budget, and also when it’s good to explore versus exploit. It turns out that more performant models are much better at abandoning unproductive neighborhoods. That was a really interesting surprise for us.

Yeah.

Yeah. When the paper comes out, we’re going to release a bunch of these environments as hardware packages, and then many thousands of transcripts for people to play with and see how these agents are doing—how their science is doing.

That—

3. Why Better Models Move On

That’s super cool. Do you have a feel for why the smarter models are able to figure out that they’re in a bad landscape and move? Or that they’re in a bad part of the landscape and should move?

We don’t know that at all yet. All we know is that they’re abandoning these crappy neighborhoods more quickly. We definitely see a progression toward more willingness to jump around overall in more performant models, but they especially abandon the lousy places fast. The really poor ones stick around.

It’s not just that they jump more; they’re actually figuring something out and saying, “Hey, this is not good. Let’s get out of here.”

Exactly. And what happens with the dumber models is that they’ll just keep running little subtle changes on molecules that aren’t ever going to go anywhere. They’ll think, “These two have a low score, but I feel good about them, so I’m going to keep trying stuff.” Then the smarter models will abandon the entire process and go somewhere else.

Have you looked at the rollouts to see if it’s thinking about whatever?

Yeah. They think about all kinds of stuff. When we release all these transcripts for everyone to look at, one of my favorite things to do is take large numbers of transcripts, dump them into LLMs, and look at the different failure modes. You’ll see things like, “I think this ring is good,” or “I think this particular nitrogen is painful,” or whatever. They say these things, and they’re very plausible medicinal chemistry narratives. It’s pretty fun.

Yeah. That’s amazing.

4. The Medicinal Chemistry CAPTCHA

I’m curious: have you given humans these same questions?

I have, to have them look and evaluate.

Yeah.

Not a ton, because we have a large number of them. But when we get medicinal chemists to look at them, they’re like, “Yeah, those are very plausible thoughts about how to conduct this particular part of the SAR campaign.”

I don’t know, maybe I’m thinking more about whether you could make a game out of this. This is sort of a game, right? The medicinal chemistry video game, and then you can—

No, the CAPTCHA.

—put people in these environments and give them—

It is a CAPTCHA.

Give them a medicinal chemistry CAPTCHA?

Yeah.

That’s your new business model.

I like this plan. We did a game a little bit like this for Name That Chemist. We’d probably have to boil it down to a smaller number of building blocks, because you can imagine that if you’re a medicinal chemist, you might have a suite of things you want to try over time, like putting a methyl here or moving this around. With these pre-measured environments, we’re restricted to the things we actually measured. So then we’d have to ask the medicinal chemist, “You can’t do whatever you want; you can just do a lot of stuff,” and figure out a way to—

And then they have to know what stuff to—

They have to know—

Yeah.

—what the stuff is, yeah. We’d have to figure out a way to surface it to them in a meaningful way, but I kind of like where you’re going with this.

Yeah.

We could definitely shrink it down to a smaller number of options. So that’s possible. We could do that.

5. The Frontier Model Leaderboard

Yeah. What model is best among the frontier models?

Oh—

What are the best ones?

Let’s see. I was looking at the plot a minute ago. It’s kind of embarrassing that I forgot already. Right now, the champion is Astra 6.

Nice, nice.

Yeah. Not a big shock.

Not a big shock. Sonnet 5.5 is actually pretty good too.

Okay.

Yeah.

Yeah.

And Sol 6.1 is also pretty good, so—

And everything 5 is just a raging pile of crap, right?

Sonnet 5, yeah. Sonnet 5 is—

Yeah.

—both very expensive and not very performant.

Yeah, yeah. But I think 5 will be long considered a big mistake by Anthropic. The 5.5s are pretty good, yeah.

Yeah, no.

Do you have Opus 5.5?

We don’t yet, no. That’s one thing that’s actually been a little infuriating, as you guys can appreciate. When we talked two months ago, we were starting these rollouts to see what happened, and then there’s been a new frontier model drop, what, like once every 11 days?

11 days, yeah.

Every time I think, “We’re going to ship this; we need to drop this public stuff,” some other new model shows up. It’s like, “Ugh.”

You have to ship faster than Anthropic.

Yeah, yeah. Good luck, Ian. We’re excited to put more in here, and we’ve also run a large number of the open-source ones. We’ve thrown a bunch of smaller Qwen models in there, and GLMs, and Kimi, and DeepSeek, and that kind of stuff. So there are a lot of these.

Mm-hmm.

How do they do?

DeepSeek V4 Flash was pretty good. Not that bad. Maybe I should just send you guys the full chart of how things are going, the data.

Yeah.

Well, we can just pull this particular one up on the screen.

Yeah, we can put it into the—

Yeah, stick it up there. I can send it. It kind of looks a little bit like the ARC-AGI-2 plots, where you’ve got some crappy models on the right. It does kind of look like that, and that’s kind of why we made it like that.

6. Models Learn Molecular Language

One thing I mentioned earlier that has been very interesting to us is that about this time last summer, we started an internal project to do supervised fine-tuning on some of these language models to get them to understand SMILES, because none of the existing frontier models were any good at it.

Yeah.

None of the Anthropic or OpenAI models at the time could even convert SMILES to non-canonical SMILES very effectively.

So, just for non-chemists, SMILES is a string language that you can use to accurately describe a chemical, right?

Yes, that’s right. I think we mentioned it a little earlier when we last recorded, but it’s a string, and molecules can be branches or rings, or both put together. A SMILES string is basically a graph traversal across the molecule, written down as a one-dimensional string. You can imagine starting the graph traversal at a different point in the molecule, so there are multiple ways of representing the same molecule with SMILES. It’s useful to be fluent in transferring from one to the other.

There are a couple of open-source packages people use, like RDKit, to do this. But we wanted the language models to do it because it would go faster, and they weren’t as good at using tools on their own back then. Nobody could do it. Then around February, like a lot of other things—

Yeah.

—it became clear that they could start to actually work with chemistry in a meaningful way, which is one reason we started going in this direction.

That’s really interesting, because that means it’s internalizing some understanding of the physical nature of molecules.

I think a little bit, yep.

Yeah.

These are the tasks that we’re having these things do right now. They’re iterative tasks where they’re looking at molecules and learning something about those molecules to make choices that lead to better molecules next.

Yeah.

When we remove the information about the molecules, these decisions really suffer. So they’re learning something. This is very interesting.

Yeah.

Or at least they’re using it. I guess that’s a better way to put it. We’re not changing weights here, but they’re definitely using that information as they make future choices.

You mentioned this zoo of models that you’ve tried. Have you used one of these Bayesian hyperparameter algorithms as a deterministic baseline for your problem?

We think it’s too spiky of a landscape, and it’s very high-dimensional. We don’t do that, but we mostly think it’s not going to work. I think that’s the party line.

Okay.

You plan on releasing all this, so I guess we’ll put it in the show notes. You said you’re going to have a preprint, a blog, and some data rollouts and everything. Is there anything else?

Right. Yeah, that’s it. I think that’s the big thing. We’ve built this thing where, if you’ve got scientific agents and you want to know how they’re behaving, what mistakes they might make, or how they might learn from physical feedback, or whether a new training recipe improves scientific decision-making, we’ve got the capabilities to measure these things at scale.

Yeah.

And—

That’s really cool. Do you have plans for other similar problems using the same kind of thing?

We do, yes. Broadly, the way we think about this is as a lab in the loop with amortized latency. You go out and record the thing once, and then you can have a huge number of rollouts through it—effectively infinite rollouts.

We’re thinking hard about other types of biological or scientific questions that you could transform into these kinds of environments. With a conventional lab in the loop, it takes a long time, and because those things have limited budgets, it’s also not counterfactually dense. You have to keep running it and make different choices with every rollout to make the agents learn.

Right.

To make the agents learn.

Right. But with your multiplexed strategy, you can just amortize.

You can just amortize.

Yeah.

So it’s not quite accurate to say it’s a lab in the loop with zero latency, but it’s pretty close.

This is the same thing that Lila’s doing. I mean, it’s not the same thing, but they also do experiments and then use that in their rollouts, but they’re not doing them live.

Right. Doing it in the rollout one step at a time offers a lot more freedom, because you can choose almost anything. Here, you’re restricted to the number of choices we made when we pre-measured the environment.

Yeah.

That’s definitely true, but it’s still a pretty big number of choices. Right now, we’ve screened about 1,000 proteins against a couple of these chemical libraries since we started Leash, and we’re going to start to grow that. I think we’re going to try to get to maybe 20 of these new environments a day. That’s our plan for the beginning of 2026 or 2027. We’re doing 2 a day right now, so I think we could probably get to 20 a day without too much lifting. We’re pushing, and we’ll see if we can do it.

Wow. So you’re going to have this huge RL environment.

Yeah. It seemed like the thing to do. Informally, it seemed like a lot of these agents are making the best and fastest progress where there’s instant feedback that’s very scalable.

Yeah.

It’s really hard to do that in the physical world, but we think there are probably a handful of biological questions that you could rebuild in this way and go after. We don’t yet know if we’re able to take reinforcement learning progress from these kinds of environments and move on to new biology, but we’re trying. We’re going to find out. It won’t be long.

That’s the obvious goal. How does this transfer to other biology? You can imagine that if you did it for some sort of covering set, you might be able to get things in between that you could transfer to pretty well.

Yep, yep. One thing I need to do is talk to some more decision people, because a lot of this is about how you spend a limited experimental budget and what you can do to do better at that.

Yeah.

Beyond just medicinal chemistry, right? When do I stay? When do I go? How do I get better at that?

Interesting. Yeah.

Yeah.

You can imagine that if it knows how to connect a decision at that point to knowledge it has, then it could learn that skill for other problems.

Yes. So we’re incorporating more and more structural tool calls and things like that with this as well. It’s a pretty fun time right now.

We’ll be sure to link to all that cool stuff you have coming out. Thanks for taking the time to chat.

Cool, cool.

Yeah.

All right.

Thanks for taking the time for a follow-up chat. I think this will actually appear back to back with the rest of the episode.

Yeah, yeah.

So—

This was great.

All right.

Well, thanks, guys.

Always.

Yeah. Always fun. Talk to you later.