An Eval for AI Scientists That Knows the Answer Key - Ian Quigley, Leash Bio
- Leash Bio makes scientific-agent decisions auditable by premeasuring the full landscape. Agents get a target, a catalog, and a budget of 100 molecules to find the best binder; metabolism and other drug-like properties are outside scope. Across about 90,000 rollouts, Leash can compare results with the measured landscape.
- Better models’ clearest behavioral edge is leaving unproductive chemistry sooner. Agents may repeat tests, choose molecules at odds with their hypotheses, or keep modifying poor binders; stronger models abandon “crappy neighborhoods” faster. Why they do so remains unknown: “We don't know that at all yet.”
- Astra 6 leads, while expensive models can still underperform. Quigley also rates Sonnet 5.5 and Sol 6.1 well, and says DeepSeek V4 Flash was “pretty good.” Sonnet 5 is “both very expensive and not very performant”; Opus 5.5 had not yet been tested.
- Improved chemistry fluency changed what Leash could attempt with general models. After poor SMILES handling last summer, models began working meaningfully with chemistry around February. Removing molecular information hurts their decisions. Quigley says they may understand some of it, but qualifies that they are at least using the information—not changing weights during these tasks.
- Leash’s scaling plan reuses measured experiments across vast numbers of agent rollouts. Quigley calls it “a lab in the loop with amortized latency,” with choices restricted to measured molecules. Leash has screened about 1,000 proteins against a couple of chemical libraries and is doing two environments a day; it plans to try for maybe 20 a day, with that goal for the beginning of 2026 or 2027.
- Transfer to new biology remains unproven, and a proposed baseline is untested. Quigley does not know whether reinforcement-learning progress will transfer. Asked about a Bayesian hyperparameter algorithm as a deterministic baseline, he says they have not run one and think it probably will not work because the landscape is “too spiky” and “very high-dimensional.”
1. A measured chemical world makes scientific decisions auditable
- Quigley’s structure–activity relationship task gives agents a target, a catalog, and a budget of 100 molecules: “go find the best binder you can.” It measures binding improvement; metabolism and other drug-like properties are outside scope.
- Agents use different search policies, state readable-language SAR hypotheses about why molecules do or don’t work, and test those hypotheses. Across about 90,000 rollouts, Leash knows the entire measured landscape. An agent might find something only “half as good as the best thing in the entire environment,” making missed opportunities visible alongside final scores.
2. Better models abandon unproductive chemistry sooner
- Quigley sees agents that give up early, retest molecules, or spend budget on choices inconsistent with their hypotheses. Better models leave poor neighborhoods faster; weaker ones keep making “little subtle changes on molecules that aren't ever gonna go anywhere.”
- A host asks whether this means recognizing bad neighborhoods, beyond simply jumping more. Quigley agrees that they abandon them faster, but says the reason is unknown: “We don't know that at all yet.”
- Medicinal chemists have judged sampled rationales plausible. A host’s proposed human game faces a catalog constraint: “you can't do whatever you want, you can just do a lot of stuff.”
3. Astra 6 leads an already shifting model field
- Quigley names Astra 6 the champion, with Sonnet 5.5 and Sol 6.1 also good. Sonnet 5 is “both very expensive and not very performant”; Opus 5.5 has not been tested. DeepSeek V4 Flash was “pretty good.”
- Frontier releases roughly every 11 days keep disrupting publication plans. Quigley expects the preprint around the episode’s release, with a blog, data, environment packages, and many thousands of transcripts.
4. Chemistry fluency improves, but a proposed optimization baseline remains untested
- Last summer, Leash began supervised fine-tuning because frontier models struggled to convert SMILES into noncanonical strings representing the same molecule. A SMILES string traverses a molecule’s graph; starting at a different point can produce a different string for that molecule.
- Around February, models began working meaningfully with chemistry. Removing molecular information now hurts decisions. Quigley allows that models may understand some of it, but clarifies that the evidence shows they are using the information: “Or at least they're using it”—“We’re not changing weights here.”
- A host asks about using a Bayesian hyperparameter algorithm as a deterministic baseline. Quigley says they have not run one and think it probably will not work because the landscape is “too spiky” and “very high-dimensional”; he calls that the party line.
5. Reusable experiments scale feedback before proving biological transfer
- Quigley calls the design “a lab in the loop with amortized latency”: measure once, then run effectively unlimited rollouts. Live experiments allow more freedom; premeasured worlds restrict choices to what Leash already measured.
- Leash has screened about 1,000 proteins against a couple of chemical libraries. It is doing two environments daily and plans to try for maybe 20 a day, with that goal for the beginning of 2026 or 2027.
- Whether reinforcement learning in these worlds transfers to new biology remains unknown. A host suggests a covering set might help; Quigley also sees broader questions in experimental budget allocation: “when do I stay? When do I go?”
Full transcript
Hey there. Welcome to Lane Space Science. This is Brandon and RJ, joined with Ian Quigley from LeashBio. We recorded an episode earlier this summer with Ian, where he talked about the cool things Leash Bio was doing with DNA-encoded libraries and how they could screen millions of compounds against thousands of proteins in a high-throughput way. Around this time, Ian realized they could do fun things with this data. It essentially allows them to create a virtual sandbox where agents can explore and make decisions in real time, as if they were medicinal chemists, and get instant feedback while running a huge number of exploratory rollouts. Since that episode, Ian and Leash Bio have developed this idea much more fully. They’re now releasing a paper, a dataset, and a blog post, along with the episode we just released. We wanted to sit down with Ian, talk a bit more about what he’s done, and hear about the updates. Anyway, welcome to the podcast. Thank you, Ian.
So last time we talked, Ian, you were saying that you had some agents exploring a sandbox. You said, “Oh, this is just a fun idea. We don’t know what we can do with it, but it’s an interesting idea.” I think you’ve since fleshed this out into a full environment for agents to explore, maybe do RL on, and do scientific reasoning. Can you talk a bit more about that?
1. The Medicinal Chemistry Sandbox
Yes. The thing I talked about last time was the idea that we had these large collections of screens, and that they were big enough environments and counterfactually dense enough that you could drop agents in there, give them tasks, and have them do those tasks on a limited budget in a landscape that we had measured completely. The flagship problem we’ve spent a lot of time on over the past couple of months is a structure-activity relationship problem. The idea is: I’ve got some molecules, and I know that they bind—maybe they bind to the protein I want them to. How do I change them to make them better? This is a very common task for medicinal chemists.
SAR, as it’s abbreviated, also includes other things, like certain drug-like properties: Does it get metabolized? We’re not doing those things right now. We’re just asking whether we can find molecules that bind better to the thing we’re interested in. We take an agent, like a language model, and say, “Here’s a catalog you can look at. You have a budget of 100 molecules. We’re not going to tell you anything except the target. Go find the best binder you can.” Different agents try different things. They have different search policies and different ways of wanting to spend their budgets.
They put out a SAR hypothesis, in readable language, about why they think these molecules are or aren’t working. Then they test those hypotheses with this limited experimental budget. Because it’s all pre-measured, you can roll out a lot of these. We’re up to about 90,000.
2. Measuring Scientific Agent Performance
So you started doing these on foundation models. I think you have a paper or a preprint or something at this point that you can share. Do you want to share that or talk about it?
Yeah. The preprint will probably be out at the same time this episode comes out. It’s pretty ready. What we show in there is that we get great separation of agents across model families and at the levels, that kind of thing. Another thing that’s really interesting is that we have not just the results you could use to evaluate how the agent is doing the job; we also know the results of the entire measured landscape, or world. So then you can start to interrogate how good the agent is at making scientific decisions. Maybe it only got to the thing that’s half as good as the best thing in the entire environment.
We see all kinds of interesting behaviors: agents that give up early, ones that buy molecules that don’t agree with the rest of their hypotheses, ones that keep retesting the same molecules despite having a limited budget, and also when it’s good to explore versus exploit. It turns out that more performant models are much better at abandoning unproductive neighborhoods. That was a really interesting surprise for us.
Yeah.
Yeah. When the paper comes out, we’re going to release a bunch of these environments as hardware packages, and then many thousands of transcripts for people to play with and see how these agents are doing—how their science is doing.
That—
3. Why Better Models Move On
That’s super cool. Do you have a feel for why the smarter models are able to figure out that they’re in a bad landscape and move? Or that they’re in a bad part of the landscape and should move?
We don’t know that at all yet. All we know is that they’re abandoning these crappy neighborhoods more quickly. We definitely see a progression toward more willingness to jump around overall in more performant models, but they especially abandon the lousy places fast. The really poor ones stick around.
It’s not just that they jump more; they’re actually figuring something out and saying, “Hey, this is not good. Let’s get out of here.”
Exactly. And what happens with the dumber models is that they’ll just keep running little subtle changes on molecules that aren’t ever going to go anywhere. They’ll think, “These two have a low score, but I feel good about them, so I’m going to keep trying stuff.” Then the smarter models will abandon the entire process and go somewhere else.
Have you looked at the rollouts to see if it’s thinking about whatever?
Yeah. They think about all kinds of stuff. When we release all these transcripts for everyone to look at, one of my favorite things to do is take large numbers of transcripts, dump them into LLMs, and look at the different failure modes. You’ll see things like, “I think this ring is good,” or “I think this particular nitrogen is painful,” or whatever. They say these things, and they’re very plausible medicinal chemistry narratives. It’s pretty fun.
Yeah. That’s amazing.
4. The Medicinal Chemistry CAPTCHA
I’m curious: have you given humans these same questions?
I have, to have them look and evaluate.
Yeah.
Not a ton, because we have a large number of them. But when we get medicinal chemists to look at them, they’re like, “Yeah, those are very plausible thoughts about how to conduct this particular part of the SAR campaign.”
I don’t know, maybe I’m thinking more about whether you could make a game out of this. This is sort of a game, right? The medicinal chemistry video game, and then you can—
No, the CAPTCHA.
—put people in these environments and give them—
It is a CAPTCHA.
Give them a medicinal chemistry CAPTCHA?
Yeah.
That’s your new business model.
I like this plan. We did a game a little bit like this for Name That Chemist. We’d probably have to boil it down to a smaller number of building blocks, because you can imagine that if you’re a medicinal chemist, you might have a suite of things you want to try over time, like putting a methyl here or moving this around. With these pre-measured environments, we’re restricted to the things we actually measured. So then we’d have to ask the medicinal chemist, “You can’t do whatever you want; you can just do a lot of stuff,” and figure out a way to—
And then they have to know what stuff to—
They have to know—
Yeah.
—what the stuff is, yeah. We’d have to figure out a way to surface it to them in a meaningful way, but I kind of like where you’re going with this.
Yeah.
We could definitely shrink it down to a smaller number of options. So that’s possible. We could do that.
5. The Frontier Model Leaderboard
Yeah. What model is best among the frontier models?
Oh—
What are the best ones?
Let’s see. I was looking at the plot a minute ago. It’s kind of embarrassing that I forgot already. Right now, the champion is Astra 6.
Nice, nice.
Yeah. Not a big shock.
Not a big shock. Sonnet 5.5 is actually pretty good too.
Okay.
Yeah.
Yeah.
And Sol 6.1 is also pretty good, so—
And everything 5 is just a raging pile of crap, right?
Sonnet 5, yeah. Sonnet 5 is—
Yeah.
—both very expensive and not very performant.
Yeah, yeah. But I think 5 will be long considered a big mistake by Anthropic. The 5.5s are pretty good, yeah.
Yeah, no.
Do you have Opus 5.5?
We don’t yet, no. That’s one thing that’s actually been a little infuriating, as you guys can appreciate. When we talked two months ago, we were starting these rollouts to see what happened, and then there’s been a new frontier model drop, what, like once every 11 days?
11 days, yeah.
Every time I think, “We’re going to ship this; we need to drop this public stuff,” some other new model shows up. It’s like, “Ugh.”
You have to ship faster than Anthropic.
Yeah, yeah. Good luck, Ian. We’re excited to put more in here, and we’ve also run a large number of the open-source ones. We’ve thrown a bunch of smaller Qwen models in there, and GLMs, and Kimi, and DeepSeek, and that kind of stuff. So there are a lot of these.
Mm-hmm.
How do they do?
DeepSeek V4 Flash was pretty good. Not that bad. Maybe I should just send you guys the full chart of how things are going, the data.
Yeah.
Well, we can just pull this particular one up on the screen.
Yeah, we can put it into the—
Yeah, stick it up there. I can send it. It kind of looks a little bit like the ARC-AGI-2 plots, where you’ve got some crappy models on the right. It does kind of look like that, and that’s kind of why we made it like that.
6. Models Learn Molecular Language
One thing I mentioned earlier that has been very interesting to us is that about this time last summer, we started an internal project to do supervised fine-tuning on some of these language models to get them to understand SMILES, because none of the existing frontier models were any good at it.
Yeah.
None of the Anthropic or OpenAI models at the time could even convert SMILES to non-canonical SMILES very effectively.
So, just for non-chemists, SMILES is a string language that you can use to accurately describe a chemical, right?
Yes, that’s right. I think we mentioned it a little earlier when we last recorded, but it’s a string, and molecules can be branches or rings, or both put together. A SMILES string is basically a graph traversal across the molecule, written down as a one-dimensional string. You can imagine starting the graph traversal at a different point in the molecule, so there are multiple ways of representing the same molecule with SMILES. It’s useful to be fluent in transferring from one to the other.
There are a couple of open-source packages people use, like RDKit, to do this. But we wanted the language models to do it because it would go faster, and they weren’t as good at using tools on their own back then. Nobody could do it. Then around February, like a lot of other things—
Yeah.
—it became clear that they could start to actually work with chemistry in a meaningful way, which is one reason we started going in this direction.
That’s really interesting, because that means it’s internalizing some understanding of the physical nature of molecules.
I think a little bit, yep.
Yeah.
These are the tasks that we’re having these things do right now. They’re iterative tasks where they’re looking at molecules and learning something about those molecules to make choices that lead to better molecules next.
Yeah.
When we remove the information about the molecules, these decisions really suffer. So they’re learning something. This is very interesting.
Yeah.
Or at least they’re using it. I guess that’s a better way to put it. We’re not changing weights here, but they’re definitely using that information as they make future choices.
You mentioned this zoo of models that you’ve tried. Have you used one of these Bayesian hyperparameter algorithms as a deterministic baseline for your problem?
We think it’s too spiky of a landscape, and it’s very high-dimensional. We don’t do that, but we mostly think it’s not going to work. I think that’s the party line.
Okay.
You plan on releasing all this, so I guess we’ll put it in the show notes. You said you’re going to have a preprint, a blog, and some data rollouts and everything. Is there anything else?
Right. Yeah, that’s it. I think that’s the big thing. We’ve built this thing where, if you’ve got scientific agents and you want to know how they’re behaving, what mistakes they might make, or how they might learn from physical feedback, or whether a new training recipe improves scientific decision-making, we’ve got the capabilities to measure these things at scale.
Yeah.
And—
That’s really cool. Do you have plans for other similar problems using the same kind of thing?
We do, yes. Broadly, the way we think about this is as a lab in the loop with amortized latency. You go out and record the thing once, and then you can have a huge number of rollouts through it—effectively infinite rollouts.
We’re thinking hard about other types of biological or scientific questions that you could transform into these kinds of environments. With a conventional lab in the loop, it takes a long time, and because those things have limited budgets, it’s also not counterfactually dense. You have to keep running it and make different choices with every rollout to make the agents learn.
Right.
To make the agents learn.
Right. But with your multiplexed strategy, you can just amortize.
You can just amortize.
Yeah.
So it’s not quite accurate to say it’s a lab in the loop with zero latency, but it’s pretty close.
This is the same thing that Lila’s doing. I mean, it’s not the same thing, but they also do experiments and then use that in their rollouts, but they’re not doing them live.
Right. Doing it in the rollout one step at a time offers a lot more freedom, because you can choose almost anything. Here, you’re restricted to the number of choices we made when we pre-measured the environment.
Yeah.
That’s definitely true, but it’s still a pretty big number of choices. Right now, we’ve screened about 1,000 proteins against a couple of these chemical libraries since we started Leash, and we’re going to start to grow that. I think we’re going to try to get to maybe 20 of these new environments a day. That’s our plan for the beginning of 2026 or 2027. We’re doing 2 a day right now, so I think we could probably get to 20 a day without too much lifting. We’re pushing, and we’ll see if we can do it.
Wow. So you’re going to have this huge RL environment.
Yeah. It seemed like the thing to do. Informally, it seemed like a lot of these agents are making the best and fastest progress where there’s instant feedback that’s very scalable.
Yeah.
It’s really hard to do that in the physical world, but we think there are probably a handful of biological questions that you could rebuild in this way and go after. We don’t yet know if we’re able to take reinforcement learning progress from these kinds of environments and move on to new biology, but we’re trying. We’re going to find out. It won’t be long.
That’s the obvious goal. How does this transfer to other biology? You can imagine that if you did it for some sort of covering set, you might be able to get things in between that you could transfer to pretty well.
Yep, yep. One thing I need to do is talk to some more decision people, because a lot of this is about how you spend a limited experimental budget and what you can do to do better at that.
Yeah.
Beyond just medicinal chemistry, right? When do I stay? When do I go? How do I get better at that?
Interesting. Yeah.
Yeah.
You can imagine that if it knows how to connect a decision at that point to knowledge it has, then it could learn that skill for other problems.
Yes. So we’re incorporating more and more structural tool calls and things like that with this as well. It’s a pretty fun time right now.
We’ll be sure to link to all that cool stuff you have coming out. Thanks for taking the time to chat.
Cool, cool.
Yeah.
All right.
Thanks for taking the time for a follow-up chat. I think this will actually appear back to back with the rest of the episode.
Yeah, yeah.
So—
This was great.
All right.
Well, thanks, guys.
Always.
Yeah. Always fun. Talk to you later.