Why a 4-Billion-Point Kaggle Dataset Still Couldn’t Beat Chance — Ian Quigley, Leash Bio
Ian QuigleyBrandon AndersonRJ Honicky
- Leash’s reported advantage is inexpensive, carefully characterized experimental data rather than an elaborate model architecture. RJ Honicky describes the shop as a ten-person operation; Ian Quigley says Leash has developed about 2,000 protein constructs and run roughly 5,000 screens, each covering at least 6.5 million molecules. A one-off CRO screen might cost $100,000, depending on the handholding. Quigley’s defensibility argument is operational: “there's just so many things where it can go sideways.”
- Leash’s Kaggle competition exposed how quickly predictive performance disappears when chemistry genuinely changes. Its roughly 4-billion-interaction release dwarfed the then-largest aggregated binding database, but none of approximately 2,000 competing teams beat chance on the most distant chemical library. Quigley found that encouraging rather than discouraging: “wow, this is much harder than we thought” strengthened the case for collecting better, broader experimental data.
- Chemical benchmarks may reward recognition of scientists’ habits rather than understanding of physics. A classifier identified the correct chemist among its top five guesses with 60% accuracy across 1,815 chemists; representing molecules only through author probabilities predicted ChEMBL activity about as well as chemical representations. Combining the two did not meaningfully improve results—evidence, Quigley argues, of models “learning a lot of sociology.”
- Hermes demonstrates some external transfer, while its failure modes still shape Leash’s experiments. Trained exclusively on internally generated data, the simple embedding-and-cross-attention model performed about as well as Boltz on binding evaluations Quigley estimates covered 13 million tests and roughly 500 targets. Quigley interpreted attention maps as showing protein “nicknames” rather than functional understanding; engineered mutations broadened the model’s attention, but the scaling thesis remains “We hope so.”
- The newer opportunity is a physically grounded reasoning sandbox operating at database speed. Ian describes agents starting with ten binders and ten nearby non-binders, then choosing molecules to test their hypotheses. RJ Honicky says the agents get a 100-molecule lookup budget and find the best binders within a measured 6.5-million-molecule library. The experiments have already happened—“like it's a coding sandbox”—making this early evidence inside a finite measured space, not unrestricted drug design.
- Binding predictions remain constrained by chemical coverage, DNA effects and incomplete potency calibration. Off-DNA resynthesis works for about 90% of easier hits and 50% of harder ones in Leash’s hands, while sequencing readouts support relative rankings more readily than reliable binding constants. Asked whether current chemical coverage is enough, Quigley answers: “Not remotely.”
- Mouse results support a partnering strategy, not a clinical-efficacy claim. For a few series, Leash moved from library hits to good mouse PK/PD in six months, including inflammation-related cytokine changes consistent with the intended targets. Quigley describes potentially selective starting material for about 700 targets and partner discussions around new entities—“little bits of gold”—but presents no human results or firm clinical timeline.
- The broad AI upside depends on transferring knowledge beyond molecules that cheap screening already finds. Quigley concedes the Kosuri paradox for an already-screened target: “you don't need a model for the thing.” The hoped-for payoff is faster optimization and transferable chemical reasoning, while the hosts’ economic challenge remains: discovery may cost only a few million dollars against $200 million–$1 billion for a successful drug’s development.
1. Small molecules offer reach but complicate safety
- Leash measures small molecules binding to proteins, trains models on those measurements, and moves selected chemical material toward partners’ drug programs. RJ Honicky describes the shop as a ten-person operation; Quigley says Leash has developed about 2,000 protein constructs and run roughly 5,000 screens, each covering at least 6.5 million molecules. A one-off screen at a CRO, he estimates, could cost $100,000, depending on the handholding.
- The therapeutic attraction is reach and convenience: small molecules can become pills and enter cells, whereas larger antibodies and protein therapies struggle with oral delivery and intracellular access. Quigley thinks protein-drug design has advanced faster because sufficient data exist; comparable predictive capabilities for small molecules are “really not here.”
- RJ’s counterweight: going everywhere is “good and bad.” Cell penetration, tumor access and potentially easier blood–brain barrier passage also create opportunities for toxicity. Quigley names liver problems and heart arrhythmias; changing a molecule to remove one liability can compromise its intended activity. Screening potential off-target proteins is one way Leash hopes to anticipate these problems.
- The coverage ambition is explicit: “it is our aspiration to have screened every protein,” extending beyond humans. Species differences could inform whether mouse or rat experiments are more appropriate. Cellular behaviors such as membrane crossing sit another biological layer away; Quigley hopes biophysically informed models might eventually be fine-tuned on smaller datasets for those tasks.
2. DNA encoding turns pooled chemistry into sequencing data
- Quigley’s DEL explanation starts with a fishhook: attach protein to a magnetic bead, immerse it in pooled chemicals carrying DNA barcodes, retrieve the bead, wash it, and sequence the attached molecules’ barcodes. Unlike testing one chemical per tube or well, pooling removes much of the logistical overhead of screening large collections.
- Combinatorial construction supplies the scale. Thousands or tens of thousands of building blocks can produce millions of combinations. That restricts chemistry, but also creates useful comparisons: many molecules differ in only one component, making it easier to identify which part drives binding rather than merely observing that an entire molecule works.
- Conventional, individually arrayed libraries top out in the low single-digit millions, Quigley says; the largest DEL produced contained 40 trillion molecules. He cautions that libraries encounter problems at that size.
- Leash’s measurement philosophy is deliberately narrow: “Sequencing DNA is another way of taking pictures.” Quigley sees imaging and sequencing as the genuinely scalable readouts for biology-plus-ML, while acknowledging proteomics researchers may disagree. Falling sequencing costs directly influence Leash’s collection strategy; competition among newer sequencing companies has restarted price declines after a period of relative stagnation.
3. Barcode controls distinguish chemistry from assay artifacts
- The enrichment measurement needs comparisons, not just recovered DNA. Quigley describes negative controls for binding to the bead itself and for broad, nonspecific protein stickiness. PCR introduces another distortion because some sequences amplify faster; unique sequence tags let Leash collapse amplified copies back toward the original observed events.
- DNA is also an active participant: many proteins naturally bind it. Leash therefore uses 32 or 48 different barcodes encoding the same small molecule. If one encoding enriches much more strongly than the others, the signal may come from DNA recognition rather than the chemical. Sometimes the enriched sequence recovers a transcription factor’s known binding motif—an artifact that also provides evidence the protein was made correctly.
- Pooled synthesis can react imperfectly, leaving a mismatch between a barcode’s intended chemical and what was actually produced. Leash builds manufacturing controls around that risk. Quigley rejects the assumption that individually arrayed compounds are automatically cleaner: he says studies find their labels “typically off by twenty to thirty percent,” which he thinks is comparable to DELs, “maybe worse.”
4. On-DNA measurements and off-DNA drugs answer different questions
- A host presses the bulky-tag objection: DNA hanging from a molecule could obstruct a pocket and create a false negative. Quigley’s rebuttal depends on the modeled question: “we know it's there, and we don't pretend that it's not.” Leash includes a placeholder marking DNA’s attachment position rather than asking the model to treat a tagged compound as untagged.
- A drug program still requires removing that dependency. Leash manufactures interesting chemicals off DNA and tests them in functional assays; Quigley reports that about 90% work for easier cases and about 50% for harder ones, in his team’s hands. He explicitly acknowledges other groups have different experiences and that removing the barcode can change behavior.
- For training, the narrower claim is reproducible physical signal. Fresh protein batches and newly manufactured libraries, tested across many months, recover mostly the same hits in roughly the same order. That makes the tagged system useful even though it is not identical to the eventual medicine.
5. Screening breadth changes the incentives behind DEL experiments
- Historically, Quigley says, expensive sequencing encouraged treating a DEL screen as a major undertaking, often reserved for unusually difficult targets after other methods failed. Teams might spend two to six months producing and characterizing protein. Maximizing the chance of finding any usable hit then favored very large chemical libraries.
- Leash instead screens each library against many hundreds of targets, learning the library’s behavior rather than judging it from one difficult protein. One supposedly identical library behaved differently across manufacturing runs; only broad screening exposed the recurring motifs. Understanding those chemical patterns also helps troubleshoot protein production, making the assay itself an accumulating source of diagnostic information.
- Quigley and co-founder Andrew Blevins brought genomics-methods experience and years of Recursion mistakes to the problem. Their instinct is to hunt for cheating opportunities in noisy measurements—for example, whether a model can predict which instrument produced the data. “We had this real eye” for sources of noise that need chasing down.
- Brandon Anderson’s reversal is worth retaining: he once told a Leash angel investor, “that's money you'll never see again.” On the podcast: “I was wrong.”
6. Kaggle separated familiar chemistry from genuine generalization
- Leash’s competition release contained about 4 billion physical interactions, measured by one person in Quigley’s basement apartment. At the time, roughly two years earlier, BindingDB’s aggregated protein–small-molecule collection held a little under 3 million. The main competition library, supplied through AlphaMa in Shanghai, contained about 130 million molecules sharing a central triazine hub with three spokes.
- The easiest evaluation allowed one shared building block between training and test molecules, with the other spokes different. A harder split shared no building blocks but retained the triazine hub. These were designed to offer attainable wins as well as progressively harder tests, rather than presenting competitors with a single random train–test split.
- The hardest evaluation used a different Leash-designed library: no common hub, no shared building blocks, different attachment chemistry. That library was withheld from the public evaluation. Competitors were not told the three-part difficulty structure in advance.
- After about 2,000 teams competed, results were acceptable on shared-building-block chemistry and sometimes improved on the shared-hub tier, depending on the target. On the distant library, “no one did better than chance.” Even the winner failed that test; three of the top five were “the noisy tutorial notebook.”
7. The response favored protein breadth and better measurement
- Quigley admits Leash assigned about a 5% chance that contemporary co-folding advances would make the company unnecessary. Instead, the difficulty felt validating: “we are extremely bitter lesson pilled over here.” If scalable measurement could supply reliable signal, this looked like the moment to “pour gas on this.”
- The budget decision was not simply “more molecules.” Kaggle covered three well-studied proteins, reinforcing the need for many more targets. Better characterization of a smaller library looked more useful than the approximately 100-million-compound collection. Quigley contrasts three proteins against a billion chemicals with thousands of proteins against single-digit millions: variation matters on both axes.
- Molecular representation was another useful result. Basic SMILES strings “kind of beat everything else,” although not by a “resounding thumping.” SMILES encode a graph traversal as text, not intrinsic 3-D coordinates; more elaborate representations did not justify switching.
- Skeptics asked whether any strong competitors had participated. Leash responded by releasing everything—raw read counts, binary labels and evaluations—through Polaris, allowing independent processing and testing. Quigley’s invitation remains: “if anyone out there thinks they're awesome, then, like, please show us,” because the company wants the problem solved.
8. Hermes pairs simple architecture with demanding baselines
- Hermes was built to iterate quickly on training regimes, not initially to be a state-of-the-art model. It combines an ESM protein embedding with a ChemBERTa molecular embedding, joins them through cross-attention, and predicts a binary binding label. Quigley calls the architecture “incredibly simple”; ChemBERTa supplies a SMILES-token-based representation.
- RJ points out that an MLP would be simpler still. Quigley explains that earlier work had found some graph models retained performance after positional coordinates were scrambled. Brandon connects this to the “Picasso problem”: recognizing local features without understanding their arrangement. Cross-attention offers an opportunity—not a guarantee—for molecular features to interact meaningfully with protein residues.
- Evaluation therefore separates increasingly distant chemistry. Holding out building-block identities is insufficient when near-identical blocks remain on both sides. Leash clusters building blocks by similarity and prevents whole clusters from overlapping between training and test. Performance drops under that stricter regime, exposing how easy apparent generalization can otherwise be.
- XGBoost is the recurring sanity check. Its performance declines as shared building blocks go from two to one to none, then falls further with cluster-based separation. Hermes is “pretty steady” by comparison across these regimes. The lesson is not that simple baselines are weak: “you've really got to get pretty far out of distribution for XGBoost to really tank.”
9. A chemist’s signature can masquerade as chemical understanding
- The authorship project began when a medicinal chemist walked past Quigley and Blevins at Recursion and remarked, “Hey, looks like a Stuart Schreiber molecule.” Recognizing the creator from the structure alone haunted Blevins for six or seven years: could a model identify the scientist and infer their intent?
- Leash collected ChEMBL molecules from 1,815 chemists, using what Quigley recalls as a cutoff of more than 30 papers and 600 molecules. A classifier achieved 60% top-five accuracy at predicting authorship. Confusion often involved chemists who shared publications and institutions, consistent with distinctive styles and trainees inheriting some of their mentors’ preferences.
- The stronger test removed explicit chemical representations from the activity predictor. Molecules became roughly 1,800-element vectors of creator probabilities. Those vectors predicted ChEMBL activity about as well as chemical representations; combining the two did not meaningfully improve performance. Quigley’s interpretation: “It's not learning ground truth,” but scientists’ choices and fingerprints.
- This is not an accusation that chemists are doing their jobs badly. They aim to reach drugs with as few chemical series as possible, not generate unbiased training corpora. Unrecorded decisions about cost, feasibility, available building blocks and expected failure shape what gets made. His warning: “when you train models on the, on the exhaust, you might be in trouble.”
10. Clever Hans explains why impressive benchmarks can mislead
- A preferred building block can become a sociological feature. Quigley describes chemists reusing something that worked in a prior clinical program or seemed helpful for solubility. The model may learn the scientist’s confidence and habits rather than the physical reason a new molecule should bind.
- Brandon’s concrete shortcut is molecular weight. Chemists find a hit, add groups to improve binding, then add other groups to fix properties such as solubility; successful molecules accumulate elaboration. Quigley recalls a weight-based baseline tying for first at CASF-2016, or finishing very close to second. RJ stresses that this is a dataset property, not an intrinsic law of chemistry.
- Quigley’s historical analogy is Clever Hans, the apparently mathematical horse. It tapped out answers successfully until questioned by someone who did not know the answer. The horse was reading posture and microexpressions, not arithmetic. Likewise, a chemical model can appear competent while reading the creators’ faces, metaphorically, rather than solving the underlying problem.
11. External validation shows transfer without settling the physics question
- Because Hermes trains exclusively on internally generated measurements, Leash can use external collections as evaluations rather than training material. Quigley estimates the paper covered around 13 million tests across roughly 500 protein targets. On the binding-prediction task, Hermes did about as well as Boltz, then a leading structural model; he emphasizes that Boltz can do many things Hermes cannot.
- Brandon asks whether Boltz had trained on evaluation data. Quigley says some training-subset boundaries were unclear. Many evaluations came from the Boltz team, with Leash adding more molecules. Relative strengths differed by protein family, which he suspects reflects training distributions—or some memorization—rather than a universally superior architecture.
- BRD4 supplied a revealing case: an orthogonal library included an identical molecule already represented with that protein in a PDB crystal structure. Hermes performed poorly on the distant chemistry; Boltz was perfect. Quigley suspects Boltz had already seen that exact protein–molecule structure. “That checks out” is his conclusion, not a claim that the comparison proves either model understands physics.
- Asked whether this validates broader scaling, Quigley answers, “We hope so.” The next emphasis is intentionally chosen activity cliffs: nearly identical molecules or proteins with sharply different behavior. His molecular mnemonic is the “magic methyl”—“kind of mythical,” he cautions—where one tiny thorn’s presence, absence or position can determine whether a molecule works.
12. Mutant screens manufacture the missing hard negatives
- Quigley says current co-folding models can fail a direct pocket test: digitally fill a binding pocket “like you're filling it with cement,” remove it, or enlarge it, and the model may still place the molecule where the pocket used to be. Structural datasets lack strong negatives; successful protein–molecule complexes do not teach every way an interaction can be broken.
- Leash tests this physically. First screen an intact protein; then add a high concentration of a known pocket-binding compound and rescreen to see which chemical series it displaces. Next mutate an amino-acid residue in that pocket and screen again. If the same series disappear, the two interventions provide converging evidence that those chemicals use the identified pocket.
- The obvious objection is global protein damage. Quigley’s control is that other chemical series, binding elsewhere, remain intact: the mutation did not simply destroy every interaction. Leash runs many hundreds of these screens and can screen about 100 protein constructs on a big week. Mutations can also improve binding or create opportunities for different chemicals.
- The displayed plot makes that last point concrete: one mutation improves binding for colored populations of related molecules, while a second mutation expels those populations entirely. Because DELs contain hundreds of close relatives across several chemical series, repeated population shifts can connect one small-molecule side group to one protein residue much more convincingly than an isolated hit disappearing.
13. Attention maps caught Hermes inventing protein nicknames
- Leash expected Hermes to attend to binding pockets, or perhaps broadly across conserved functional domains. Instead, it focused on short, non-conserved stretches away from the functional regions—distinctive sequences that could identify a protein without explaining its behavior. Quigley’s interpretation was that Hermes had invented “a nickname for the protein”: remember ten or twelve distinguishing amino acids, then retrieve the associated binding behavior.
- Engineered mutants undermine that shortcut because almost identical proteins now have substantially different outcomes. After adding mutant data and retraining, Leash saw attention distributed much more evenly. The intended lesson extends beyond any one gene: the model should discover that the nickname strategy no longer works reliably, rather than simply memorize another identifier.
14. Language agents turn measured chemistry into compute-speed experiments
- Quigley’s newer direction uses readable English to represent molecules, proteins and behavior. With open-source language models including Qwen and GLM, Leash teaches structure descriptions, differences between molecules, and explanations of binders versus hard negatives. The growing activity-cliff corpus supplies reasoning traces grounded in reproducible physical changes. An agent starts with ten binders and ten nearby non-binders, then proposes why the binders work.
- It can select five or ten molecules to test that hypothesis, receive results, and revise its explanation. These are database lookups against experiments already performed: “It will get back the answer instantly, like it's a coding sandbox.” RJ Honicky says current agents get a 100-molecule budget and find the best binders among 6.5 million. Quigley describes the approach as encouraging early signal.
- The scale comparison is a medicinal chemist’s structure–activity relationship campaign: a person might see a couple dozen, perhaps 50, in a career; Leash extracted over 10,000 “last weekend.” Natural language “seems like” a useful regularizer, Quigley argues, because adding parameters to Hermes could let it memorize mutants too. He thinks the reasoning approach is “much more resistant to the memorization problem.”
- RJ presses why avoiding shortcuts should also improve answers. Quigley points to rapid rewards and choices humans would not make: “It's gotta have the counterfactuals. It's gotta make mistakes.” Pooled measurements provide those alternatives without recording only a chemist’s preferred path. The sandbox remains restricted to screened chemistry; ordering new catalog molecules and retesting them is a future extension.
15. Chemical coverage and potency remain unfinished work
- RJ asks whether DELs favor flat molecules. “It depends on the building blocks,” Quigley replies. Leash uses forked, Y-shaped designs, linear designs and macrocycle-like geometries. DNA placement, available building blocks and compatible assembly chemistry all constrain the result; a library can be good for one target class and poor for another.
- Brandon challenges whether 6.5 million molecules matter in chemical spaces often described as (10^{40}) or (10^{60}). Quigley points to transfer into external ChEMBL chemistry, but does not call current coverage sufficient: “Not remotely.” The roadmap balances coarse diversity for new target shapes against fine-grained changes that expose activity cliffs.
- Building-block availability is a real limitation: moving a nitrogen around every position of a ring may require custom synthesis. Quigley is less worried about fragile attachment chemistry. Over roughly the past four years, DNA-compatible methods have expanded beyond digestible amide bonds to carbon–carbon and carbon–nitrogen bonds.
- Relative potency is more accessible than absolute binding constants. Repeated screens recover similar rankings, but reads are not directly reliable (K_D) values. Leash is adding SPR biophysical measurements and exploring a potency ladder for calibration; “I wouldn't say that we're there yet.” Unexpected DNA binding by proteins not previously described as DNA binders further complicates translation.
16. Mouse results feed partnering rather than a clinical timeline
- For a few chemical series, Quigley says Leash progressed from library hits to good mouse PK/PD in six months with few medicinal-chemistry cycles. One inflammation compound changed blood cytokines as expected from its targets. That describes exposure and biomarkers of intended activity; the episode does not present a human efficacy result.
- Asked what enabled the speed, Quigley credits, in part, starting with drug-like chemistry. Library design uses a Thompson-sampling-like approach: generate possible libraries, score them against an optimization function, and choose a smaller collection. Objectives include desired binding, avoiding other targets and predicted ADME properties, rather than maximizing library size alone.
- Brandon’s “When people?” gets a process answer, not a date. Leash is discussing partnerships and new entities to advance selected programs. Declaring a development candidate, establishing manufacturing and completing additional safety work still take time; a fast discovery engine does not remove those subsequent requirements.
- The byproducts can themselves have value: “No one wants to sell reasoning traces about, like, political theory,” but Leash owns chemical starting material. Quigley thinks it has material that “looks pretty selective to us” for about 700 targets. These are opportunities with different economic value, not programs it intends to pursue simultaneously—“little bits of gold” generated along the way.
17. Discovery economics sharpen the case for transferable models
- A host’s challenge is that discovery is a small fraction of drug development: perhaps a couple million dollars against $200 million–$1 billion for the steps leading to a successful drug, depending on the indication and testing requirements. Even allowing for failed candidates, is Leash solving the economically important bottleneck?
- Quigley argues some early problems are valuable enough. Cytokine targets may have strong genetic evidence and successful antibody therapies yet remain hard to drug with small molecules. A pill replacing an infusion, without refrigerated transport, would be desirable. He says Leash has selective material against several such cytokines, while acknowledging substantial downstream capital requirements.
- He also concedes a “lamppost problem”: scalable measurement currently favors early biology, not millions of mice or clinical trials. A host suggests an eventual oracle would need mouse and clinical data; Quigley answers “maybe.” For a trusted target, sufficiently good ADME and off-target predictions might already justify going directly to a mouse study without requiring clinical data first.
- Brandon’s “Kosuri paradox” asks why models are necessary if cheap experiments already deliver the molecules. For one screened target, Quigley concedes, “you don't need a model for the thing.” The hoped-for payoff is faster optimization and transferable reasoning into unscreened targets, agriculture, veterinary science or materials. Only if chemical understanding generalized broadly enough would further screening eventually become unnecessary.
18. Process discipline is the moat, and purification is the next bottleneck
- Asked what prevents a better-funded lab from outscaling Leash, Quigley points to accumulated experimental mistakes and the culture required to detect them. Teams need frictionless metadata tools, historical comparisons and persistent batch-effect investigation. Leash rebuilt selection and much of protein expression, then added quality proxies. His claim is that replication would take time, not that more capital could never compete.
- His best example is lumacaftor-like chemistry. Compounds resembling the cystic-fibrosis drug repeatedly hit certain protein preparations; investigating expression conditions suggested those hits tracked misfolding. Leash now treats that pattern as one warning: “that protein probably didn't fold right.” Such data may be excluded from training and the experiment rerun under friendlier conditions. The diagnostic emerged from noticing an unexpected recurring chemical series.
- Quigley welcomes sequencing costs “drop like a rock”: “Cheaper sequencing, more molecules” directly serves the desire for more chemicals and more measurements. He treats the trend as something already helping, not an unsolved hypothetical.
- His more concrete wish is massively parallel small-molecule purification. HPLC and similar cleanup approaches do not deliver sufficiently clean compounds at the desired scale. Easier purification would let Leash take large numbers of DEL hits into different assays and “orthogonally validate all those measurements”—expanding confidence beyond the platform’s deliberately unusual primary readout.
Full transcript
My co-founder, Andrew Blevins, and I were at Recursion looking at a small molecule on a screen. Then a medicinal chemist walked by us, a guy named Mike Cucurisi, and Mike was like, “Hey, looks like a Stuart Schreiber molecule.” Then we were like, “Oh, no. You can tell just by walking by? Like, who made this?” That’s horrifying. If the models can identify the chemist by looking at the molecule alone, then maybe the models can infer chemist intent, which means maybe they don’t learn anything at all.
Well, they did learn something. They learned who the chemist was.
They learned something. Yeah, they don’t learn physics. We downloaded all the chemicals from all the chemists at ChEMBL, which is the biggest repository of these things and their activities. We included every chemist who published more than—I think it was—30 papers and 600 molecules. That gave us all the molecules and all the associated information for 1,815 chemists.
Then we trained a classifier to predict a chemist from looking at the molecule alone, and we got a top-five accuracy of 60%. We chose to represent the molecules not with any chemical structures or formulas, but just with a vector of probabilities for who made them. So now it’s this 1,800-element vector, and it’s got some probability of who made it. You can use that representation and then train a model to predict activity from ChEMBL, and it does as well as if you use chemical formulas. We really think that it’s learning a lot of sociology. It’s not learning ground truth. When you train models on the exhaust, you might be in trouble.
Welcome to the Latent Space AI for Science podcast. I’m RJ Honicky, CTO of MirOmics, and I’m joined by my co-host, Brandon Anderson. He develops RNA therapeutics at Atomic AI.
Super excited to have Ian Quigley in the studio with us today. This episode’s going to be a treat. Ian is the co-founder and CEO of Leash Bio, which, in my personal opinion, is one of my favorite young tech bios of the last few years. At a high level, Leash is combining epistemic rigor with data collection at a crazy scale. It has a strong commitment to open science and an incredible series of blog posts, a lot of which we’ll cover today. I think they really tell you how to do machine learning in a very hard domain in kind of the right way.
I think I’ll have some fun anecdotes to trickle in through the episode. But to start with, thanks for being here. Is there anything about us that you want to tell us before we get started?
No, thanks. That was very kind. We founded Leash after being confronted by a particular problem: trying to predict small molecules binding to protein targets, and then finding that the data that was out there was kind of wanting. So we went out and built this thing at a reasonable scale to try to go after it.
Yeah.
Yeah.
So, “reasonable scale.” You’re founded upon the idea of doing something called a DNA-encoded library. Every person I’ve ever met who’s touched a DEL has come out severely traumatized. I don’t know anyone who’s ever said, “DELs, that’s a great idea. Let’s do hundreds or thousands of these things.” So why? And also, please explain what a DEL is before you get started.
Yes. When you’re going after small-molecule drug discovery, the ultimate goal is to get a small molecule, or a dinky chemical, to bind to the protein that you think is causing the problem in a disease, and then go from there. Some of these proteins are pretty easy to find chemical material that sticks to, and some are harder.
One way to go about this is to have a lot of different chemicals in different tubes. You then dunk the protein in one tube at a time, and it kind of takes forever. A different solution that was proposed some years ago was to have a lot of chemicals in one tube at once. You’ve got a small DNA barcode physically attached to each small molecule. You dunk your protein in the tube, kind of like a fishhook, pull out the ones that bind, and then sequence their DNA. The idea is that you could pool this large number of molecules and avoid the logistical overhead of trying to go across large numbers of tubes or individual wells and plates.
What’s happened downstream of that is that chemical libraries made one tube at a time have a biggest size in the low single-digit millions. With DELs, the largest one that has been produced is 40 trillion molecules. There are some problems when they get that big, but that is the kind of scale you could try to attempt with this technology.
So the way you create these is you use combinatorial chemistry, where you have these building blocks, and then you build the library piece by piece, but combinatorially. So—
That is correct.
Yeah.
Yes. To unpack that a little bit, if you’ve got some number of millions of molecules, what this really means is it’s a combinatorial mixture of building blocks, and the number of building blocks is going to be a lot smaller than the final product of all of those. So you might be squared or cubed, right, in some different examples.
It’s not uncommon for there to be thousands or tens of thousands of building blocks that would then generate a chemical library of many millions of molecules. This is restrictive in some ways because you’re stuck with the smaller number of building blocks. But one thing that’s really nice about it for machine learning is that you’ve got all these different combinations of things that look very similar to each other, except for maybe one or two changes.
I’m sure we’re going to get into this later, but that really enables comparisons so that you can identify, “Oh, it’s this part of the small molecule that’s driving the binding because I see a lot of variations in this other part that’s not affecting the behavior too much.” That turns out to be incredibly useful.
The other aspect of this is how you do the readout, which is DNA sequencing. This has been a big theme on the podcast. We’ve had many guests who have had some variation of, “We do something crazy, and then we convert the reading into a DNA sequencing readout.” So how does this work, and why is this so useful?
I would definitely say that it’s a party line at Leash that if you’re going to go off and try to do life science stuff and couple it with machine learning approaches, there’s really only one scalable technology in town, and that technology is taking pictures. Sequencing DNA is another way of taking pictures, so it’s basically pictures in DNA sequencing.
If you can shovel your challenging problem into one of those approaches and use it as a readout—for example, Recursion takes actual pictures of cells, and a bunch of other groups do that, like Insitro. Almost everyone else with the spatial omics stuff or other approaches will do some mixture of pictures and sequencing, which is also pictures.
There are going to be some proteomics people out there who are going to disagree with me. I’d still argue that the scale is not there yet.
Hot take number one. And one of the really cool things about sequencing is that it is dirt cheap. It’s extremely scalable, a very well-established technique.
It is.
We’re very good at it.
It is very, very cheap. It is getting cheaper all the time. There was a little while where it kind of plateaued in terms of the price dropping, but now a bunch of new companies have started up—
Mm-hmm.
—and they’re driving it down again. We very much look at the trajectory of sequencing cost dropping, and it has implications for how we think about data collection and the scale that we’re doing. It’s really nice.
That’s cool. So you have these components that you mix together to form this combinatorial mixture of molecules, and each of those components is barcoded. Is that right?
That’s correct.
And then they kind of bind together because they’re these protein chains, or these amino acid chains, that all click onto the ends of each other and fold together. Then they either bind or don’t bind to your target. First of all, how do you know that they bound to your target?
Yeah. So first of all, it is very much like this fishhook analogy. The target is some protein or a protein complex that you’re going to try to find chemical material to bind to. It’s attached to a solid support. Some folks do it a variety of different ways, but the way that we usually do it is it’s attached to a magnetic bead, and so you dunk this bead in this complex mixture of all these small molecules.
In a very easy case, let’s pretend that the protein has a very distinctively shaped hole in it. You dunk it in there, and then the chemicals in that pool, some tiny fraction of them, have the appropriate shape and electronics that they’re going to bind into this very distinctively shaped hole.
So then you pull this protein back out, and you only pull out the things that are in that hole. You rinse it a couple of times, and then you sequence the DNA of the things that come out of that hole.
Okay. When you say “sequence of DNA,” you’re basically amplifying the barcodes in there.
That is correct.
And then you can—so “amplify” meaning PCR or something like that?
You do a little bit of PCR, and then you toss it into the sequencer. One thing you’re probably thinking right now is, “Okay, well, we need to compare it to something to get a good measure of relative enrichment.” You do have to engineer a bunch of good negative controls: if I just stuck the bead in there by itself, or whether there are pan-protein stickiness signals that are going to pull out certain chemicals from the library. You’ve got to think pretty hard about all those things to make a reasonable comparison to the thing you’re actually trying to get binders for.
For the things that bind, you sort of have these combinations of barcodes—
Mm-hmm.
—and if this one bound, then I have these… How many barcodes are we talking about for one protein?
Great question.
Or one small molecule.
For one small molecule. There are a couple of things. If we want to get into the PCR-based weeds, for the audience: if you’re making lots and lots of copies of DNA, there are little subtle biases in the DNA that might make some sequences amplify faster than others, so you want to control for that. You put in a little bit of unique sequence so that, when you have lots of copies, you can collapse them and know they all came from this one—you know, they’re very much just from this one event.
There are a bunch of other things about barcodes that we worry about at Leash. When you’re trying to stick proteins in there, some of those proteins are going to stick to DNA, because that’s their job, right? There are lots of proteins whose job it is to interact with the genome in cells and turn things on or off, or affect them in some other way. Then you’ve got this assay where you have a ton of DNA and you’re throwing them in there.
Yeah.
That can be risky.
And that can cause false positives, right?
Very much so.
Yeah.
A lot of people complain about this. This is kind of what I was getting at earlier, with folks having DEL nightmares. This is one of those problems. There are a variety of ways to address it. One thing we do is have large numbers of barcodes that each encode one particular type of small molecule. Here’s one small molecule, and now we have 32 or 48 different barcodes all encoding this one small molecule.
Mm. Oh, I see.
Right?
Yeah.
So there’ll be a single example: one single molecule has this barcode, and its neighbor, which is the same chemical, will have a different barcode.
Right.
And then so on for dozens—
So if there’s any differential there, then you know that you—
That is correct.
—that you’re binding to the DNA instead of the—
That’s right.
—the small molecule.
That’s right. For the DNA-protein aficionados in the audience, one really nice thing about this is that because there’s DNA in there, you can see if there’s a single DNA-encoding barcode that binds way better than everything else, even though it’s all the same small molecule. Then you can look at the sequence that you enriched and recover the described DNA-binding preferences of those proteins.
Mm.
It’s like, “Oh, here’s—”
Got it.
—a transcription factor.
Yeah.
It binds this series of nucleotides that is very well described. It’s 6 or 8 of them. Then you look at the—you recover the DNA sequences from this experiment. It’s like, “Oh, great, we found the consensus binding motif.”
Right.
So I think our protein probably got made right.
You have this barcode that you’ve identified, and you’ve kind of gotten rid of the false positives using the technique you just described.
Mm-hmm.
And so now I know that for a given combination—for a given small molecule that I’m interested in, I bind to my target, and it’s… But don’t you have all these DNA tags, barcode tags, hanging off your molecule?
You do.
Doesn’t that impact the—
Yes, they are hefty. They are hefty. This is another complaint people bring up about DELs. There are a couple of different answers here. The 10,000-foot philosophical answer is that there are multiple different reasons why you might want to run one of these experiments. One reason is that you’re trying to get a small-molecule material that behaves a certain way, that you’re going to progress into drug programs.
If that’s your main goal, you want to make sure you’re getting some kind of fidelity between having the DNA barcode on there and having it off there. When we try to pursue interesting chemical material at Leash, you do have to go through the work of manufacturing the chemical off DNA. You’ve got to run it in these functional assays. There’s a certain level of attrition. For things that are easy in our hands, about 90% of them work. For things that are hard, it’s about 50%, so there’s some loss. Other people have different experiences.
But if you’re thinking about doing this for machine-learning training purposes, all these chemicals with their DNA barcodes are in the tube. You ran the experiment, some of them bound and some of them didn’t, and that is real signal that you’re physically measuring from the world. Then you condition your models on the existence of this barcode and keep moving on with your model training.
In our hands, when we rerun experiments—make fresh protein batches and a new batch of the chemical library over many months—we get back the same hits in the same order, for the most part. It’s extremely reproducible and robust. In terms of ground-truth signal, we feel quite good about it.
Okay. You could be blocking the pocket or something with your DNA, so then you can get a false negative there.
You could. That’d be interesting. But you don’t, because you know that the DNA is there. It did not bind. The way you’re thinking about it is, “Oh, I’m thinking about how the molecule will behave missing the DNA,” but we know it’s there, and we don’t pretend that it’s not.
Oh, I see. So does your hit include the DNA, then?
Yeah. When we train models, we very much put in a placeholder that marks the position of the DNA.
If you’re going to synthesize this for a drug, will you leave the DNA on there?
No. Like I said, that’s where you get those attrition rates, right?
Okay, yeah. Then you’ve got to make it, and sometimes they behave differently.
Okay.
I see. Yeah, yeah. But let’s be conscious of the time. Got it. Got it, got it. So—
Sorry.
Oh, no, no, no. I just need to understand how you’re—
No, actually, those are really good questions. There are lots of objections, and these are many of the reasons why folks are trying to do the thing that we’re trying to do.
Yeah, I was going to say, going back to my original statement, I have never met someone who has worked on DELs and said, “We should do more of these.”
Oh.
I think, hopefully, we’ll get to why I’m actually excited that this route seems to be working. Hopefully the audience will agree with me before this is over. In fact, maybe even a personal anecdote: we have a mutual friend who is, I think, an angel investor in Leash. When he told me he was an angel investor in a DEL company, I jokingly said, “Okay, well, that’s money you’ll never see again.” I’m here in front of our entire audience to say I was wrong.
Oh, that’s great. I wish I had—
Oh, that’s funny.
I wish I had listened.
I’m curious, first of all, why are DELs so hard? I think we just covered a lot of these. What in the historical community of people doing DELs was making it so traumatic as an experience, and then why did you think you could overcome that?
1. Why DEL Screens Are Hard
One thing I do want to say is that when DELs were invented, it was definitely at a time when sequencing was more expensive, and doing sequencing experiments felt like a big logistical and financial swing. When you built the technology at the time and then pointed it at the problems it was pointed at, it was very much, “Oh, we want to find new chemical material binding to targets that are very challenging.” That brings up a couple of tactical issues. One of them is that, classically, folks don’t screen large numbers of protein targets.
They’re usually interested in finding chemical matter for something that’s really hard. They may have tried a bunch of other techniques, and those haven’t worked out, so they think, “Okay, let’s try a DEL screen because we don’t have anything.” Then they’ll make the protein they’re going to screen, and they’ll put a lot of work into it because it feels like a lift to do a DEL screen.
They might spend 2 to 6 months growing big batches of the protein they care about and doing a bunch of purifications. Maybe they’ll run functional assays to convince themselves it’s good enough to go ahead with the screen. Then, when they run the screen, they’ll try to maximize their chances of getting pretty good chemical material. The incentive drives them toward wanting a big DEL with a lot of diversity.
If you look at folks offering CRO services or internal DEL groups, a lot of times they’ll push toward these very large DELs. That’s their motivation, and it makes a lot of sense. What we’ve done that’s very different is screen tons and tons of proteins. We’re less driven by trying to find individual binders for a particular protein target that might be of high interest for a program. We just want tons of active chemical material across large numbers of proteins.
That lets us understand how our DELs behave very well, because every one of our chemical libraries gets screened against many hundreds of targets. So this—
So you’re characterizing the—
The chemical library better. We know it very, very well, and we start to encounter lots of motifs that are quirks of the chemistry we designed or maybe the manufacturing process. A great example is one chemical library we’ve had made a couple of times. Even though it’s the same chemical library, it behaves differently, and we didn’t know that until we screened it against hundreds of proteins.
Then you have to stare at those results and say, “Oh, now we’re seeing these motifs show up a little bit more.” As you characterize the library more and more, it really helps you evaluate and troubleshoot even your protein production process. One reason we felt this was pretty tractable overall is that a couple of people at the company and I come from genomics methods development backgrounds, as compared to chemistry backgrounds.
We knew what it might take to tweak experimental protocols to get good signal out of these funky assays. I’d say it was that, plus the many years my co-founder, Andrew Blevins, and I spent at Recursion, along with a number of other people on our team. We made a lot of mistakes in high-dimensional biology, bumping around in the dark and figuring out what different failure modes might allow models to cheat on the high-dimensional imaging data Recursion produces.
So we had a real eye for thinking, “There are these sources of noise this way; we’re going to have to chase those down,” and, “Can you predict the instrument this was run on? If you can, that’s really bad.” We incorporated all that thinking into going after this particular assay, and we had a feeling that people hadn’t really stared at it in that amount of detail before.
Yeah, this goes back to the Ron Alpha, Noetic episode, where I found—
Recursion Mafia?
Yeah, Recursion Mafia. I found some real spiritual parallels between both of your companies.
Mm.
This real epistemic rigor, checking very carefully everything you’re doing, doing things in different ways and sort of semi-overlapping ways. I’m curious—we asked Ron this.
Data focus too, right?
Yeah, very data-focused, at scale.
And automation.
And I’m curious, was this just core to Recursion’s culture, that sort of—
It is—
Like—
Yes.
Yeah.
Yes, especially in the early years. I think there are 4 ex-Recursion Mafia companies, and they all kind of rhyme. There’s us, there’s Noetic with Ron Alpha, there’s Invada with Vish Waloo, August Allen, Hannah Gordon, and all these great people, including Dave Healy. And then there’s Known Medicine, which was run by Katie Roskelley and got picked up by Pathos a couple of years ago.
All these companies are looking at different interesting biology problems with extremely noisy measurements, and they’ve pulled in people from Recursion to figure out, “We’re pretty good at noisy measurements. Let’s see if we can get a good signal out of this particular corner.” For us, it’s a small-molecule chemistry problem. Invada works with mixtures of natural products and does a lot of mass spectrometry work. Then there’s Ron’s group, and Katie Roskelley’s group. So, yeah, we’re all very similar.
So let me—let’s pause for just a second. What do you guys actually do?
That’s great.
Sorry, I’m so excited about this, I just jumped in. Sorry.
That’s great.
Yeah.
2. Small Molecules And Their Tradeoffs
We’re a machine learning small-molecule company. We collect massive numbers of small molecules binding to protein targets, and we use that to drive advances in machine learning and our understanding of this kind of thing. We also push small-molecule material into partners and move it forward toward patients.
Just for the audience that’s not familiar, what is a small molecule? Why do we care? How is it different from other therapeutic—
This is great.
—techniques?
This is great. When you’re trying to drug something, I talked a little bit earlier about how you’ve got a protein of interest in your body, and you’re going to try to affect its behavior in some way. There are a couple of different things currently on the market that people use. One common one is antibodies, which are made by cells. They’re large molecules, and they’re big enough that they have a really hard time getting inside cells. They also have a really hard time being in a pill.
If you have an antibody treatment or one of these protein treatments, like GLP-1 treatments, you’re going to need an injection. They can affect certain proteins that are easy for them to get to, but there are other proteins inside the cell that drive a lot of disease and are quite challenging for them to reach.
In a lot of ways, it’s more appealing to have a little tiny chemical instead of a big protein. Small molecules can be in pills. They can go inside your cells and affect behavior. They make up most drugs. If you’ve got any pills in your house, they’re almost certainly small molecules.
Life scientists in the 21st century have made pretty good progress in predicting and enabling the behavior of protein-type drugs. There are a bunch of antibody companies, and groups like Isomorphic Labs, Chai, or Novla. There are lots of very good, very smart people pushing on these problems. I think part of that is because we understand protein behaviors, or have enough data for them, and that makes it reasonably effective for these companies to do these designs.
Our thesis is that those capabilities aren’t really here for small-molecule approaches. That was one reason we built this company: we felt we could go after it in a meaningful way.
And small molecules, one of the key characteristics is that they go everywhere in your body, right? So that’s good and bad.
Mm-hmm.
They can penetrate into the cell, and they can also get through tumor defenses and cross the blood-brain barrier more easily. But at the same time, they do all those things, and therefore they can be more prone to toxicity and other side effects.
Very true. The path through the regulatory barriers for small molecules can be a little more fraught than for antibodies. We know pretty well right now how antibodies behave in the body, what problems they’re likely to cause, and what they’re not. For small molecules, there are a bunch of well-described problems they might cause. They might cause liver problems, for example, or heart arrhythmias.
As a medicinal chemist, a person whose job it is to develop these molecules and get them from something that looks okay in a tube to something you’d feel confident enough to give to a human being, there are a lot of pitfalls to avoid. You have to do a lot of work to make sure that your molecule doesn’t cause heart arrhythmias.
And then if I change it to stop this causing those problems, does it affect it doing the thing that I want it to do?
How do your models help with this larger problem of developing these and then maybe with these downstream problems?
One way to think about it is that a handful of these problems—these off-target toxicities—are driven by your small molecule binding to other proteins in ways that you might be able to predict or identify.
Mm-hmm.
That’s something we’re very much pushing forward: let’s also make those proteins, let’s also screen those proteins with our chemistry.
It seems useful if you have a giant assay of—
That’s right.
—different proteins and a library that you can—
Yeah.
Yeah.
It is not at all an exaggeration to say that it is our aspiration to have screened every protein. And then also move on to those outside of humans.
Mm-hmm.
Both to get variation in the proteins, but also to identify whether maybe you don’t want to run a mouse model; maybe you should run a rat one because your chemical material behaves differently in those different species, since those proteins might be slightly different.
Interesting.
Right, so you want to do that stuff. Going after large numbers of these things allows you to think, “All right, now I’ve got my chemical material, and I know which chemicals, and which variations, are likely to bind these things and cause other toxicity problems.”
Got it.
Yep. Downstream of that, there are a bunch of other assays that lots of people do, involving more emergent behaviors. It might not be a single protein, but rather how well it gets across cell membranes, or how it interacts in a more complex way that requires a cell to evaluate. Or it might be a cardiac cell, for example, for this heart arrhythmia stuff.
Right now, lots of groups are pushing on technologies to measure these at the cellular level, which is one level of biological abstraction away from what we’re focusing on. But we’d like to think that models with a good understanding of the biophysics themselves can be useful, and maybe fine-tuned on smaller datasets in the future.
Popping back one more, and maybe getting a little into the model: why DELs?
DELs really are, at least in our view, the most scalable technology for measuring small molecule interactions, because you can put many millions of them in a tube. One thing we haven’t touched on with these chemical libraries is that, when you make them in a pooled process, some may react imperfectly, and the label translation of what you think you have might be a little off. We’ve designed a lot of controls into constructing our chemical libraries to mitigate this.
When you’re using arrayed libraries, or libraries where there’s one chemical at a time in a tube or in a well in a plate, you hope the labels are good. But when people examine this, they’re typically off by 20 to 30%. It’s an enormous mislabeling problem. We think it’s probably comparable to mislabeling in DELs, maybe worse. DELs themselves are enormously scalable.
3. The Kaggle Benchmark
Maybe I can tell a quick story that I’m sure we’ll get into at some point anyway. We ran a Kaggle competition a couple of years ago, and we decided to generate the data for it and release it. There was an existing database called BindingDB, which at the time, 2 years ago, was the biggest aggregated collection of small molecules binding to protein targets, as opposed to cells or something else. It had a little under 3 million physical interactions. We released a dataset with about 4 billion physical interactions, and it was all measured by a single person in the basement of my house.
The early blog posts are really funny. You have a literal picture: it’s not quite a garage startup, but it’s in a basement. You’re—
It is a basement apartment with a 7-foot ceiling.
It looks claustrophobic.
Yeah.
I’ve never seen a biotech startup started in a basement.
It can be done.
Chris was like that, right?
Yeah.
Anyway.
But this Kaggle competition is a fascinating story because you had such high hopes going into it.
We did.
I think this is one of those cases of really nice epistemic rigor. You didn’t just provide a train-test split with a random split or something. You thought carefully about how to organize the Kaggle competition, how to run it, and how to evaluate fairly. Can you talk about that process, the decisions you made, and the result?
Yes, happy to. We were not by a long shot the first group to try to evaluate whether you could use computers to make reasonable predictions for small molecules binding to proteins, and then test those predictions. There are lots of papers on this. There’s a real classic paper that came out of X-Chem and Google some years before we founded Elish.
We had run a lot of our own experiments on public data, and we suspected that no one was really very good at it. We wanted to put together this contest and design the evaluations in a way that could really test it across a range of difficulties. We were thinking, “Okay, we’re going to have a lot of people compete. Some are going to be very good, and some are probably going to be beginners. We want to make sure there are some nice, easy wins, a bunch of stuff in the middle that’s kind of hard, and then some things at the very top end of what would be difficult, or what we imagine difficulty might be.”
To unpack that, we’ve talked a little bit about how these libraries are combinatorial. You’re going to have a smaller number of building blocks, but a large number of combinations. We got our hands on a chemical library from a wonderful sponsor, a guy named Xiaoji Lu, who runs a shop called AlphaMa in Shanghai. The chemical library had about 130 million molecules. You can imagine the center part of each molecule as a hub with 3 spokes, and the center hub was the same for every molecule. It’s a particular chemical motif called a triazine.
For the evaluation dataset, we had a couple of different ways to split this. One way was to have the chemicals share one building block, one spoke, while the other 2 spokes were different. That was the easy stuff. Another way was to have no shared building blocks, but still have the shared hub, since they came from this chemical library. The spokes might be similar, but they wouldn’t be exactly the same, and they wouldn’t be shared, so we thought this would be a little harder.
For the third, most difficult evaluation, we had a completely different chemical library that we had designed at Elish. There was no common hub, no common building blocks, different attachment chemistries—different everything. We thought we’d have these 3 evaluations in different proportions, and we wouldn’t tell anyone that’s what we did. We’d just say, “Here are the evaluations. Let’s see how it goes.”
We launched the Kaggle competition 2 years ago, and you can look it up if you want. We’ve written a lot about it. After 2,000 teams competed, the winner did okay on the data with shared building blocks. Depending on the target, we saw some boosts in performance when there were no shared building blocks but the core hub was still the same—the triazine was still there. On the most challenging evaluation, no one did better than chance.
The winner was, like, the 1,200th random person who submitted a Jupyter notebook, right? Or something—
The winner was a guy named Victor Shlapak, who put together a slightly different approach. His was good, but like everyone else, no better than chance. Three of the top 5 were using the noisy tutorial notebook.
But just to be clear, usually with a Kaggle competition, there’s a public evaluation for you to test your algorithm, and then there’s the private one that’s held out—
That’s right.
…
And the competition runs that, and all you know is the score.
Yep.
So was your library included in the public eval?
No.
Okay.
No.
Yeah.
It was hidden to make it spicy.
Right, okay.
Yeah.
And it was spicy.
It was very spicy.
So what did you take from this? What was the lesson? First of all, what did you hope to get from this? You kind of touched on that, but how did that affect your thinking about how to proceed?
Yeah. We had a couple of hopes, and some fears. One thing I’ll be candid about is that we had released this dataset that was 1,000 times bigger than anything anyone had ever seen. We gave it about a 5% chance that recent advancements in protein-folding approaches would mean it could obviate our company entirely.
Didn’t happen.
No, it didn’t happen. We thought there was a small chance that we were really messing up here, but that didn’t happen. We suspected it was more likely that people weren’t very good at it and that the data might help, so we were hoping to see that.
And anyway, you’d want to know that.
You’d want to know that.
Yeah.
Before you—
You’d want to know that.
—spend 10 years of your life on it.
Absolutely.
Yeah, yeah.
Absolutely. Another thing we wanted to know was whether the different ways of representing molecules and giving them to computers made a difference. You could use SMILES, which represents them one way, or 3D graphs. There are also unusual approaches like spherical convolutional networks. People have tried all kinds of things to represent molecules usefully, and people in the field will argue that one approach is better than another.
We encouraged competitors, especially those with a chemistry background, to try some of these different representations and see how they did. We were grateful to find that the approach we were using—basic SMILES—kind of beat everything else. It wasn’t a resounding thumping, but it was enough for us to keep using that approach.
And SMILES, if I recall correctly, is basically a textual sequence—
It’s a string.
—a sequence of—
It’s a string.
—strings. It doesn’t encode any 3D structure intrinsically.
It does not, yeah.
It’s all emergent from the—
It’s a graph traversal translated into a text string. You can start the graph at different points, so there are what they call canonical and non-canonical SMILES, depending on where you start. But it’s an extremely simple representation. We wondered if some of the fancier ways of representing molecules might beat SMILES on this particular problem, and they really didn’t.
So was your takeaway that you should go full force into this? Was this encouraging or discouraging? What was the collective emotional reaction at Elix after that? I’ve always wondered. You wrote several follow-ups, but I’m curious to hear.
Yeah, this is a great question. We’re extremely bitter-lesson-pilled over here. Part of our reaction was delight: “Wow, this is much harder than we thought.”
Yeah.
That’s kind of great. That’s kind of great.
Yeah.
Because if this problem is like all the others we’re aware of, and you can get a pretty good signal from a scalable measurement regime, then you should just try to get that.
Yeah.
For us, it was pretty validating: “Oh man, there’s an opportunity here.”
Especially if your scalable measurement regime is hard and you have specialized knowledge to do it.
Absolutely.
Yes.
Absolutely. It was like, “Oh man, this is the time to pour gas on this. This is great.”
All right. Wow. That’s awesome. So then you were just like, “Okay, let’s go full force. We’re going to start collecting data, collecting data, collecting data.”
Correct.
Yeah.
Yes.
So you did that for a while. What came next? First, what were the lessons you learned in the process, building toward your first useful outcome?
Yeah. There were a couple of things from the process that we found very useful. The Kaggle competition was based on 3 protein targets that were fairly well studied. It became clear to us that getting performance across novel proteins would probably mean we’d have to do a whole lot more. We were always motivated to do that, for lots of reasons, but this really underscored that we were going to have to make a lot of proteins.
Part of it was realizing that we had a very large number of small molecules—this 100-million-compound library. That didn’t help us as much as better characterizing a smaller library, so that was another strategic choice we made.
This is actually quite important because you have a fixed budget. You can either spend more on sequencing to get higher resolution for what you already have in a smaller set, or you can get more molecules and hope the noise averages out with more data points.
That’s correct. Having a fixed budget really affects your strategic choices. We wanted to make sure we were getting a lot of variation across both axes, rather than across one axis and a handful of proteins.
Sorry, which two axes?
The number of chemicals against the number of proteins.
Okay.
Right?
Yeah, yeah.
You can imagine, “I’m going to do 3 proteins and a billion chemicals,” or maybe, “I should do thousands of proteins and single-digit millions of chemicals.”
Yeah.
Or some other combination.
Right.
Single-digit millions would probably be one of the tiniest DELs ever constructed, right?
It’s pretty small.
Yeah.
The smallest one I’ve seen is 50,000 molecules.
Oh, okay.
Yeah.
I guess you can make a custom DEL if you want to do SAR or something.
You can do whatever you want.
Yeah. When you have the technology, just do it. Do it with two building blocks.
Yeah, right?
Yeah, yeah.
You can do whatever you want.
Yeah, yeah.
That’s the nice thing: you have enormous control over both the library size and its characteristics, as well as the barcode issue I talked about. You can expand that kind of thing, and we’ve put a lot of thought into the bells and whistles in our own DELs to boost performance and signal-to-noise.
Getting back to the Kaggle competition lessons, it was also clear to us that models could memorize individual building blocks, and that would boost their performance. They might also be able to learn, probably through memorization, that a very similar building block might behave similarly. That helped with some of the performance metrics. But when they had to work with something pretty far away, they simply couldn’t do it in that regime.
When we released the results, basically saying, “Hey, no one did better than chance,” we got some blowback. People asked, “Did anyone good compete in your DEL competition? Maybe everyone’s just bad. Maybe it’s all first-year grad students.” As a result, we released all the data. If anyone out there thinks they’re awesome, please show us, because we want this problem solved. But we haven’t seen anyone perform great on that stuff yet.
And when you say all the data, that includes all the raw read counts and everything?
That’s right, yeah.
So you can go through and say, “Oh, it’s not just a processing error.” You can start from the raw data and use it.
Yeah.
Yeah, you could take our binary labels. You could do it yourself. All the evals, everything. It is all on Polaris, which is a nonprofit group that was run by Recursion. It’s there for anyone to use. Please use it. We intentionally released it, hoping the community would make good use of it.
Sure. Do you think, though, that the structure of this dataset was fundamentally limited in a way that means, without some external augmentation, you’re not going to be able to do any better than what people have pretty much already done?
I think if you had really perfect structural understanding or really good structural models, then you ought to be able to do okay on this, right? If we have models that truly understand the physics of the interactions between the amino acids and the small molecules, then it doesn’t really matter what the distribution of the small-molecule training set looks like, if they’re really good at this.
Mm-hmm.
And I think when we see structural model predictions and folks say, “Oh, well, I’ve got this time split on the PDB, for example. Our model’s never seen these molecules or this protein before, and hopefully we can use this understanding to make reasonable folds,” that’s very much in the spirit of being able to make predictions on this stuff. No one has seen these molecules—especially our internal molecules. No one’s seen this before.
That’s cool. So, with the Kaggle competition, you learned some lessons. You went full force: data collecting, regrouping, committed to scaling.
Yep.
All right. Then, I think something like 6 months later, you had a blog post about your model, Hermes.
Yep.
Can you tell us about that, and some of the learnings there—especially how you evaluated it and all the care that went into that model?
4. Hermes Tests Generalization
Hermes is a model that was the original brainchild of the data scientist at Leash, Max Kleinsasser, and Sean Francis Lyon. Andrew Blevins, of course, is the CTO there. Originally, the thinking was that we had a bunch of data and were thinking about how to split it into different training regimes. We wanted to come up with a really simple model so that we could do lots of experimental iterations. The purpose wasn’t to be a state-of-the-art model. It was, “We want to try stuff fast.” So we built Hermes.
The basic architecture of Hermes has 2 different embeddings. One is an Evolutionary Scale Modeling, or ESM, embedding. That’s a protein embedding from a group that was at Meta, then left to start their own company, and more recently got picked up by Chan Zuckerberg.
We had Alex Reeves on the podcast.
Oh, right, right.
A throwback to that episode.
Hey, yeah.
If you want to hear more, we can go back a few weeks.
Lots of detail.
Yeah. Thank you, Alex, if you’re listening. We used that for the protein side, and then we had a different embedding called ChemBERTa, which is basically a next-token predictor, a chemistry embedding for this side.
Something that predicts SMILES?
That’s right.
Yeah.
It just predicts SMILES tokens. So that’s it: 2 very simple embeddings, and then we put them together with cross-attention. That was it. Then we used it to predict a binary label.
The simplest possible thing you could do.
It’s incredibly simple. They’re pretty small, and then we trained them on some of our own data.
Actually, not the simplest. You could have done an MLP after those instead of cross-attention, right?
Yes.
So why attention?
The thinking with the attention had to do with some other experiences we’d had with graph convolutional models—graph models for protein structure. One thing we’d noticed is that if you’ve got these sorts of graphs—and there are a couple of models that do this—in other work we’d done, we scrambled their positional coordinates, and performance for some of those models more or less stayed the same. That made us think attention was super critical. So we wanted to make sure the model would be looking at the different parts of the protein and small-molecule structures that could be useful here.
This is oftentimes known as the Picasso problem in image generation. CNNs, for example, were famous for a while: you could scramble the features of a picture, like Picasso—flip the nose around or rotate the eyes—and the CNN would still depict, “Oh, this is a Picasso,” because it just picked up on individual local features.
Mm-hmm.
Rather than some sort of global structure.
Mm-hmm.
And this has been a problem in small-molecule drug discovery for a while, where it often picks up on local motifs and doesn’t have any global understanding of the topology.
Yep.
So especially for something like a DEL, where you have this combinatorial structure of individual building blocks—
Yeah.
—you could imagine the model learning each building block independently and essentially just decomposing into the sum of 3 building blocks or something. I mean, it’s a simple concept.
Yep.
Yeah. So cross-attention, in some sense, forces certain molecules to interact with certain residues in a protein.
Yep.
Or at least maybe not forces, but—
Exactly.
—provides the opportunity.
If you think about it. Yeah.
It’s an inductive bias.
Yeah. What you’re getting at is that we were thinking about how to evaluate this. We’d done some similar tricks to the ones we used in the Kaggle competition, where we had training sets on data we generated internally, and then we’d look to chemical libraries that were further away. We also wanted to think about splits to probe Hermes’ behavior.
Because DELs have this combinatorial approach, you could share certain building blocks. We did that, and of course performance is better, which we suspect is from memorization. But you can imagine that some building blocks are very similar to one another. If you’re just trying to split on their identity, you might have things that are close enough that memorization or some other cheap mode could confer predictive power.
So we came up with a scheme where we clustered the different building blocks based on similarity. That way, we could avoid entire clusters overlapping between test and train. When you do that, performance does drop. One thing you might be alluding to is that we use lots of reasonable baselines, which we’re really concerned about. For almost everything we do, we’ll use XGBoost. You should do something like that just to see how it’s going.
Yeah, yeah.
Yeah, right?
Every classic young machine-learning practitioner spends 6 months optimizing their favorite model, and then someone walks by and runs XGBoost or a random forest, and they’re like—
XGBoost.
“Yeah, that just beat 6 months of work.”
Yeah, ruh-roh.
Oh, yeah.
And so we did these different split regimes to try to evaluate how well things would do relative to XGBoost. One thing that was really gratifying to see is that XGBoost drops steadily as you get further and further away with these building-block-cluster approaches. If you share 2 building blocks, it’s at a certain level. If you share only 1, it gets worse. If you share none, it’s worse; if you use the building-block-cluster approach, it’s even worse than that. We compared that to Hermes’ performance, which was pretty steady across all these different regimes.
I see.
It made us think, “Oh yeah, you’ve really got to get pretty far out of distribution for XGBoost to really tank.” That’s in a couple of plots and some work that we published.
We haven’t yelled about it too much.
Wait, didn’t you have a blog post like “XGBoost Is All You Need”?
Yeah, we did do that.
Yeah, yeah.
That’s right, we did that. Yeah.
Yeah.
Yep.
Yeah.
Running those splits is something we’ve put a lot of thought into, trying to figure out: How do you stop your models from memorizing and remembering things?
This might be a good time for a little bit of a detour, but I want to reference another one of your blog posts, which I found absolutely hilarious from my own personal experience. I’ve seen machine learning models pick up on so many weird predictors.
Uh-huh.
Your blog post about molecule ownership or authorship was—
Oh, God. Yeah, the—
—I think a really fun story. Yeah.
The Trevor Hans post, yeah.
Yeah.
Ugh. So many years ago, my co-founder Andrew Blevins and I were at Recursion, looking at a small molecule on a screen. A medicinal chemist walked by us, this guy named Mike Cucurisi, and Mike was like, “Hey, looks like a Stuart Schreiber molecule.” We were like, “Oh no, you can tell just by walking by? You know who made this?” That’s horrifying.
And then that haunted Andrew in particular for many years—like 6 or 7 years. We joked every couple of months when we were out for beers, “Man, we really should figure out how bad this problem is.”
Why is that a problem?
It’s easy to imagine you’re going to try to train models to understand chemistry. You’re going to get data from lots of groups, because it’s very tempting to get data from lots of groups. If the models can identify the chemist by looking at the molecule alone, then maybe they can infer the chemist’s intent, which means maybe they don’t learn anything at all.
Well, they did learn something.
They did learn something.
They learned who the chemist was.
Yeah, they don’t learn physics, right?
Yeah, yeah.
They’re not learning the thing you’re trying to get them to learn.
Yeah.
And so we were really afraid of this.
They learned that certain chemists are better than others.
Yeah.
Yeah.
We learned all kinds of stuff. But maybe, to set this up for the audience, we published this over Christmas—you can look it up. We have a leaderboard and a goofy online game you can play.
We downloaded all the chemicals from all the chemists at ChEMBL, the biggest repository of these things and their activities. We included every chemist who published more than, I think, 30 papers and 600 molecules. So you get all the molecules and all the stuff. It’s 1,815 chemists. Then we trained a classifier to predict the chemist from looking at the molecule alone, and we got a top-five accuracy of 60%. It’s a very, very easy problem.
Yeah.
Yeah, it’s super easy for the models to do. Where they get confused is when you have 2 chemists who were on the same publications and at the same institution. We think it’s very likely that individuals have a distinctive chemical signature or style when they make molecules. And when those individuals have students or trainees, they adopt some fraction of that. That seemed pretty clear to us. And so that was horrifying enough on its own.
And this is a data set that many academic projects, many PhDs, have trained on and tried to predict on—
Yeah.
—find patterns and—
Yep.
—and this is really a major data set for the field.
Yeah.
Yeah.
If you weren’t generating your own data, you were using this. And if you were generating your own data, you were probably a victim of this, because your organization probably has multiple chemists with their own distinctive chemical styles, and they’re using those styles to go after different types of projects.
But the key here, I think, is that those chemists also tend to solve similar problems. I might have a career here, and I have a style, but I’m also working on this class of problems.
Right.
So you’re learning their style instead of the problem.
You’re learning the style; you’re learning the sociology of the problem. And the other thing you’re learning is that when you’re a chemist at these organizations, ordering a molecule to be manufactured and tested is kind of a pain, right? So what happens is you make many, many choices of things you will not pursue, and you don’t write those down. It’s like, “Oh, yeah, it would be dumb to do that.” And it’s like, yeah, that’s their motivation.
Mm-hmm.
Like, “I can’t do that.”
Or it’s like our team just wouldn’t make it.
Yeah.
Like, I could say, “We could do this—”
“We could—”
“—but then they would cost too much,” or the lab team—the chemistry team—will say, “No, we can’t make it now,” or—
Or, you know—
—or, “We don’t have the building blocks on site.”
“I don’t think it’s going to work.”
Yeah.
There are all these reasons why you might not make a choice, and not making that choice is a fingerprint that’s going to land on what does get made. And when you talk to some medicinal chemists, bless them, some of them are superstitious people, and some of them will say—
Yeah, this comes up every single time. Medicinal chemists are some of the most superstitious people I’ve ever met.
Right. And so it’s like, “Oh, well, this building block worked in this program that made it to the clinic. I’m really fond of it. Maybe it helps with solubility, so I’m going to put it in things that I like.” Then your model will learn, “Oh, the chemist likes this because they put it in, so that means it’s probably working.” And so it learns all this weird stuff.
We were very afraid of this, so we ran a model to identify who the chemist was. Then we chose to represent the molecules not with any chemical structures or formulas at all, but just with a vector of the probability of creator. So now it’s this 1,800-element vector, with some probability of who made it. You can use that representation to train a model to predict activity from ChEMBL, and it does as well as if you use chemical formulas. And if you combine them—
You’re coding to sociology only.
Yes. If you combine them, it doesn’t do meaningfully better. We really think it’s learning a lot of sociology. It’s not learning ground truth; it’s just learning the choices and fingerprints of the people trying to develop these molecules. And it’s not a knock on them, right? That’s not what they’re trying to do. They’re trying to get 2 drugs by making the fewest number of chemical series possible. Of course they’re going to do it like that. But when you train models on the exhaust, you might be in trouble.
So this means that this is not a great machine learning training set.
We do not believe so.
It’s funny—in my own life, I’ve had many similar lessons. It’s well known that things like molecular weight: all you have to do is sort by molecular weight.
Oh, yeah, of course.
That’s one of the single biggest predictors you can possibly have. Wasn’t it CASF-2016 where molecular weight—
It was CASF-2016 where it was tied for first place.
Yeah, yeah. It’s like—
Or second place. It was really close.
It was really close.
Yeah.
Yeah, so all these people are pulling out their fancy machine learning models, but you can just sort the molecules by weight: heavier molecules are better binders.
Yeah.
Yeah.
Yeah.
Yeah, so—
Yeah, no.
But this isn’t an intrinsic property of chemistry. This is a property of the data set.
Yes, yes. This is a property of the sociology of what gets made, what gets tested, and the way medicinal chemists work.
Heavier molecules are more likely to bind.
A lot of times it’s because they’re seeing improvements to affect other properties. That’s part of it, and you probably have some parts you want to talk about, too.
Basically, you have a chemist who finds a hit, and then they start optimizing it. They’re like, “Oh, I have a hypothesis that this is the binding site, so let me add this functional group to bind here.”
Okay.
And then they’re like, “Okay, well, that was a stronger binder.” So they’re like, “Oh, well, what if we add this other functional group to bind to this thing?”
Oh, so they just add—
And what ends up happening is they just keep adding things—
Yeah, until it tastes really good.
And then, until it’s sufficiently good. Or, as Ian said, you have these other properties. You’re like, “Okay, now we have this really good binder, but it’s not soluble, so let me add some hydrophilic functional groups on the other side of the molecule, away from the binding pocket, to make it more soluble.” You just keep tacking these things on, and next thing you know, you get this big, good binder. So you sort by molecular weight, and it’s the single biggest predictor—
Yeah, probably.
Okay, I see.
—that you can have in most public datasets.
Yeah.
Yeah.
But that has nothing to do with chemistry.
No. I think a good way to think about it is, if something’s working, they’re going to do more stuff to it—
Yeah.
—because they like it.
Yeah.
Yeah. Yeah.
So you just do more stuff to it. It gets more baroque, or you have more complicated, expensive reactions, or you get weird building blocks that are hard to get. Things you wouldn’t do if you didn’t think it was working. And so these are all signatures that the models just pick up. It’s quite terrifying.
Yeah, this is like the Clever Hans problem.
Clever Hans.
Clever Hans.
Yeah.
Clever Hans, the horse.
Can you tell the story about Clever Hans?
Yes. A little history lesson: there was a horse trainer in Germany, I think in the late 1800s. I read about this; I should remember it. There was a horse named Clever Hans, and he seemed to be able to do complicated math. You’d show the horse a chalkboard with a math problem, and he’d stamp his foot on the ground, then stop when he reached the correct answer. Everyone was amazed that this horse was brilliant. Then someone had the good idea to get a questioner who didn’t know the answer to the math problem.
Yeah.
Then the horse completely failed. It turned out the horse was just looking at the owner’s posture and microexpressions. Let’s say the answer was 7: the owner would get really giddy when the horse got to 6, and then the horse would stop stamping. There’s a wonderful paper by a German psychologist that evaluates the microexpressions and postural changes he observed in the horse’s owner. It’s quite extensive. If you’re dying for a silly read, you can look it up.
Oh, wow.
So, if you’re dying for a silly read, you can look it up.
Okay. So the analogy is that you’re reading the face and not the math?
That’s right. You do not understand math. The models do not understand chemistry at all.
Yeah.
They’re just remembering the sociology of the creators.
Got it. Going back to Hermes and your epistemic rigor, and trying to build something useful out of all the data you trained on: you find public datasets are untrustworthy, and you have a bunch of data that you trust internally, but you don’t really have a good way of—now you need to understand how to accurately and carefully use this to do something useful in the real world. So how did you bridge that gap?
Yep. We did a bunch of these splits, like I described earlier. Then the next thing that seemed obvious was: we’ve trained models on internal data, and we don’t like training models on external data, so maybe we should try predicting on external data instead—
But without ever training on external data?
Without ever seeing it. That puts us in a nice position because I haven’t seen any other group train models for this task exclusively on internally generated data. It’s tempting to get these other sources, especially because there are so many possible protein targets, and people don’t typically screen as many as we do. So we took large collections of external molecules and predicted them to see how well we’d do across many different proteins, with lots of different positives and negatives that we chose very carefully. The total number of tests in the Hermes paper was, I want to say, around 13 million across many—something like 500 protein targets. It’s a huge number. We benchmarked it against the existing state-of-the-art structural model at the time, from the Boltz team. It’s a very good model, and ours can’t do a lot of the things that model can do. But we were able to do about as well as they did on those public datasets in terms of binding prediction.
When you compared to Boltz, it required some tricks to have a fair-ish comparison. I think Boltz also trained on this data. Was that right?
Sort of. There are a couple of large collections where we weren’t totally clear about the different subsets they had trained on. There was one example in particular where it was very clear, and I’ll talk about that in a minute. A lot of the evaluations we used were also surfaced by the Boltz team, but we put a lot more molecules in to see how it would do. If you look at the different performances between Hermes and Boltz, we’re better on some proteins and protein families where Boltz is worse, and vice versa. We suspect that’s because of different distributions in the training sets, so they understand a little bit more—or, honestly, maybe memorize a little bit more—and they’re able to extend that a little bit.
Yeah.
One thing that was very validating for us: one of our test sets was another orthogonal chemical library that our models hadn’t seen. We’d used it internally, but it was a copy of a chemical library that had appeared in the literature. There was one target in particular, a protein called BRD4, where an identical molecule from our screen was also in the PDB, the Protein Data Bank training set, which Boltz had almost certainly seen. Our models did kind of badly on that molecule. It was from a distant chemical library, and they weren’t very good at it, while Boltz was perfect. We suspect Boltz had already seen this protein and individual molecule in a crystal structure as part of the training set, so they were great at it. It’s like, “Oh, yeah, that checks out.” It feels pretty good.
So does this mean that, in your opinion, this validates your scaling hypothesis? Add more proteins and more diverse chemical libraries, and now you can basically fill in this space more broadly?
We hope so.
Yeah.
Yes, we hope so. We think that going after large amounts of diversity—and, in particular, intentionally chosen activity cliffs—is the way to do this. To unpack that a little bit: on either the protein side or the small-molecule side, do you have 2 things that look very similar except for a tiny change, and that tiny change drives outsized changes in behavior? A classic example medicinal chemists talk about is this idea—it’s kind of mythical, not exactly real—called a “magic methyl.” The idea is that you’ve got a molecule with a little thorn that is or isn’t there, and the presence or absence of that thorn means it either works great or doesn’t work at all. In those situations, maybe your model starts to learn, “Man, having a thorn could be really meaningful, so maybe I need to think about these very small changes.” One thing we do a lot, and that’s baked into DELs, is have 2 out of 3 building blocks that might be similar across large numbers of examples, with variations in behavior. On the protein side, we very intentionally engineer mutations in those proteins to do the same kind of thing.
I’m a little confused here, because if you have these building blocks and the size of your variation is multiple atoms—
Mm-hmm.
Right, how big is a building block in terms of atoms?
It might be 6 to 30 atoms or so, if we’re ignoring the hydrogens. You can imagine that the building blocks themselves are very similar, with a single-atom substitution.
Oh, I see. So maybe there are 2 building blocks, one with a fluorine and one without?
Right.
And then you can see that in the sequencing?
That’s right. And maybe that fluorine just changes position.
Okay.
As you think about this, you might have tension between your goals. You might want coarser diversity because you’re trying to find binders across a large number of protein targets that may be very different. Or you might want super fine-grained changes that are very subtle, to really teach the models those subtle differences.
Mm-hmm.
As we think about designing our own chemical matter, we have to span that tension. So, yeah, we think about it a lot.
And both of those matter.
Both of those matter.
Both of those matter.
Both of those matter.
And it also matters on the protein side too, right?
It matters very much on the protein side. A couple of different groups have shown this with current co-folding models. You can take a protein with an amino acid sequence that codes for a protein with a big pocket. Maybe that protein is represented in the training set, so the model has a pretty good idea of what it looks like when there’s a small-molecule chemical in that pocket.
You can change the amino acid sequence you give the model. You can use that to break the pocket: fill it in like you’re filling it with cement, remove it entirely, or make it much bigger. With current state-of-the-art models, what we’ve seen so far is that the model will shove the small molecule in where the pocket used to be.
Okay.
Because it just remembers where it is.
Yeah.
And that’s bad.
So it’s looking at the large-scale structural things? Or that’s the hypothesis, anyway?
Yeah. The hypothesis is that it doesn’t quite understand that when you have a pocket in a protein, a small number of amino acids will have hydrogen-bond positions sticking out. There might be a little bit of pi-stacking and other things that drive the chemistry. Those things ought to be very important, and they are very important in the physical world when you’re measuring them. If you break them and the model understands the physics, then it probably should know that it can’t shove a molecule in there.
That’s the thing: a lot of these structural training sets don’t have great negatives, because what makes it into them is the protein with the thing, right? They don’t have a lot of the variations that disrupt these behaviors. No one’s going to make crystal structures with a bunch of these mutants the way we’re currently doing for ourselves.
Yeah, I was going to ask about this. How do you get negatives?
The way we get negatives is by having a really good understanding of the distribution of those molecules in a baseline condition.
Okay.
When we do the screen, we know these bind a lot because they’re enriched when we pull the protein out. Then we break the pocket, and they’re no longer enriched.
Right.
That one amino acid substitution we made is making a bunch of these chemical series drop out.
But this is a one-off experiment. This is not a DEL.
No, this is a DEL.
Oh, so you’re not saying you found a case where it was broken?
No, we break it.
Oh.
We break it. Let’s say we screen a protein and the pocket is intact.
Yeah.
Sounds great.
Yeah.
We get a bunch of hits.
Yeah.
Let’s go even further. Let’s say we know someone else has already identified a chemical that goes in that pocket.
Mm-hmm.
Let’s buy it, stick it in there, and run the screen again.
Yeah.
That means all the chemicals that normally go in the pocket will be displaced by a high concentration of the one we know.
Oh, so you definitely know.
Oh, you have a competitive assay.
We do all the time.
Okay. Okay.
We do this all the time.
Yeah. Yeah.
Yeah.
Then you say, “Okay, that looks pretty good. I believe that.” We know this chemical that someone else has characterized goes in this pocket because maybe they made a crystal structure or something.
Yeah.
Great. Let’s look at the amino acids in that pocket, and then we’ll change the DNA construct that makes the protein for us. We’ll physically mutate this one residue.
Yeah.
Now we put a big, bulky amino acid in there, and the pocket’s full.
Or a different side chain or something.
A different side chain.
Yeah, yeah.
All these things you can do.
Yeah, yeah.
And for things that aren’t as well characterized, we use conservation. We have a structural biologist on staff, and we look at these things. “Oh, yeah, this guy, Edward Craft, is amazing.” So now we’ll make this one mutation and run another screen. All the chemicals that got booted out when we hit it with the thing blocking the hole are also gone when we broke the pocket. So now we have really good evidence that they’re all going into that pocket.
Right.
And here’s the point where the protein chemists come in and say, “If you made a mutation, how do you know you didn’t just break the whole thing?” That’s a reasonable question. You say, “We only lose the chemical series that got kicked out by that competitor. All the others binding elsewhere in the protein are intact, and they come back.”
They are intact, yeah. So you didn’t just break the protein entirely.
You didn’t just break it. And then you think, if you’re screening 100 protein constructs a week, which is what we do on big weeks—
Wait, sorry. So you’ve broken the pocket somehow?
Oh, yeah.
Then do you also find that other things start to bind to the protein?
When you change things around, you might change the shape in a way that encourages other things to bind.
Right.
We have definitely seen it.
And is that an information source for you?
It is. Another nice thing about this—and about DELs—is that if you’re not thinking in large numbers, you might think, “I have this one chemical that was binding, and now I broke the pocket, and it’s gone.”
But because of the combinatorial nature of DELs, maybe I have 5 different chemical series, with hundreds of molecules, and there’s another group of hundreds of molecules that are the same except for a single change in the building block. You have these repeated examples where one tiny thing shifted the population behavior enormously, across a big population. I can show you guys a plot. I don’t know if you put pictures on the podcast or whatever.
Yeah, sure.
We pulled this thing up because we think it’s really important to have a lot of variation in the proteins as well as the small molecules. It’s very obvious: this one side group on this building block is clearly interacting with this one amino acid residue, because we have about 5 independent examples of population behavior shifting because of it. It’s beautiful. It’s very clear.
Yeah.
Yeah.
Go ahead. Tell us about this plot.
We pulled this thing up, and the reason we pulled it up is we think it’s really important to have a lot of variation in the proteins as well as the small molecules.
A reason we think this is important is that you want a model trying to learn medicinal chemistry to put together the hydrogen-bond interactions between this part of the small molecule and this part of the protein. We’ve seen other work, which I talked about earlier, showing that if you digitally break a pocket and use current state-of-the-art models, they have a really hard time understanding the physics. They’ll just shove the molecule back into that pocket anyway, which suggests they’re memorizing. We thought one way to get around this was to experimentally engineer mutations in the binding regions of these proteins and then rescreen. Here’s what that looks like—
So, physically doing that?
Physically doing that.
Not virtually doing that.
You take the DNA of the wild-type protein that you’ve already screened, and then you use molecular biology techniques to change 1 or more amino acids in that protein, so that the functional part where the small molecules are binding is no longer the same. It’s different in some way. We have different methods to try to identify which amino acids are the smart ones to break. Sometimes you break ones you’re confident will do things; sometimes you don’t know. But we can do this at a pretty big scale, so it’s not a big problem for us to take a couple of swings at each one.
Then you run the screen, and now when the pocket is broken, what do the molecules do? Do they still go in there? Do some of them go in there? Is there something in common among the ones that stick around? Is there something in common about the ones that get dislodged or pitched out? Then you have lots of different examples where you might have 3 to 5 amino acids in a given pocket that are driving the binding behaviors, and maybe you try to break all of them independently. You redo the screens each time and get a good feel for what part of the small molecule is interacting with the part you’ve changed.
When you do this with a large DEL, you’ll have lots and lots of repeated examples of similar molecules that are all following the same behavioral trajectories. That way, you get a really good feel for it: “Oh, I’ve got lots of examples. They’re all pretty similar, but the one thing that’s super common is this one little bit on the small molecule, so I have a suspicion that it’s interacting with the pocket I just broke.”
What are we looking at on this diagram?
What we’re looking at on this plot is that all those little dots are small molecules from a chemical library that we designed and made. Along the axes is how well the small molecule binds. If it’s really down the axis, it’s binding well. If it’s more towards the origin, it’s not really binding at all. We’ve colored a couple of different blobs, and those blobs are different chemical series. The chemicals look very similar to each other, but the difference is where we pointed that kind of chemical-looking structure. If you squint and look at these, they’re all pretty similar, but there are a couple of subtle differences. We believe that whole population is driven by interactions between those differences and the one mutant we made.
So, in this plot, the wild-type target is on the bottom.
Yeah.
And then on the horizontal or vertical axis, you have the mutant target. There’s this red one, and the red one is pointing basically up.
It is binding better.
It is binding better.
That’s correct.
So that particular one is—yeah, the new mutation made that particular one bind better.
That’s correct.
Which speaks to the question I asked a few minutes ago.
Yep.
And then maybe most of these seem to be above the axis, so this mutation caused most of these to bind better.
That’s correct. There’s a second mutation in that pocket that I can also share with you, where all of those populations, all those blobs, get kicked out entirely, and they do not bind.
Okay.
So then the pocket was broken in a meaningful way, and here the binding is increased, as we can see.
And so then basically everything will be along the bottom.
That’s correct.
Okay.
That’s correct.
Yeah. You can give us—
Yeah.
—a post-production.
Yeah, I’ll have both.
Awesome.
So doing this, you can map out both—well, you can treat the activity cliff issue in compound space and in protein space.
That is correct.
And this is particularly useful if you’re, let’s say, working at kinases, where there are about 100 kinases that differ only by— the important binding site might differ by 1 residue at a time or something. So this can be incredibly powerful for certain classes like that, in addition to your own sanity.
Yep, that’s right.
Yeah.
That’s right. We do this a lot, as I mentioned.
Yeah.
Many, many hundreds of these types of screens.
One of the things you can do with this is that Hermes sees this cross-attention, so you can actually interpret where Hermes thinks the binding pocket is. It’s learning the binding pocket through these types of maps.
Yes. One thing we’ve written about is using these attention-mapping approaches to look at where Hermes is looking in amino acid space. When we first did this work, we expected it to either look at the binding pockets—that was our first intuition—or, with a more measured intuition, maybe look at all those residues because it probably didn’t really know what was going on yet.
When we looked, it wasn’t looking at either of those things. It was looking at really short sequences of amino acids that were away from the functional part of the protein and also away from parts of the protein that look similar across many proteins. For the non-biology people in the room, a lot of molecular evolution has worked this way: there’s a stretch of a protein that does a pretty good job and often gets shuffled among other proteins, so you have this conserved region. It’s called a domain, or a conserved protein domain. That part looks pretty similar across many proteins. Biologists use it to infer protein families, relationships, and functions. If they all have the same kind of piece, they’re probably doing something kind of similar.
We expected the models to look at that, but they don’t. They look outside it, at these little stretches that people don’t typically think do a whole lot. They’re a little different; they’re non-conserved, and there’s a lot more variation in them. We realized that Hermes was probably coming up with a nickname for the protein. It’s not thinking, “Oh, this is the functional part.” It’s like, “I don’t need to learn that. I can remember 10 or 12 amino acids that are different here than everyone else, and I can just memorize it.” We thought that was bad.
Even because maybe many of them—if you have variations in the pocket, but they all have these other parts that are similar, and the variations in the pocket mostly don’t matter, but some of them do, then maybe the model might learn, “Oh, I can just pay attention to these.” These are—
The concern was that the model, again, wasn’t really learning any physics.
Yeah.
It’s like, “Oh, I can just memorize this stuff.”
Right.
It’s very much in keeping with what happens when you try to break the pocket with the co-folding models—
Right.
—and they shove it in there anyway.
Yeah.
It’s like, “Oh, it doesn’t know what’s going on. It’s just made this nickname.”
Yeah.
It’s like, “Oh, man. So how do you fix that?” That’s when we hit upon trying to do these mutations at scale. What happens is that if you have a protein that looks almost exactly the same, but there’s 1 tiny difference, and the behaviors change a lot as a consequence of that difference, you’re encouraging the models to look at all the amino acids again. It can’t do that for the non-conserved section, because if it does that with these 2 things that have different behavior, it’s going to do a bad job.
Yeah.
So we went back and trained models with these additional mutations, like the one on this plot that you guys are seeing. Then we looked back at the attention maps, and now Hermes looks at all of these things much more evenly.
So you’re basically using the attention map to find these cases where it’s memorizing nicknames—
Yep.
—and then enrich the mutations in the pocket—
Mm-hmm.
—to encourage it to pay attention to the pocket instead.
Yep, and it’s not necessarily just across any one gene.
It’s more like the general problem of, “Oh, I can’t use this nickname strategy effectively anymore.”
You want to teach the model, “Don’t do that.”
You don’t do that. No, no, that’s lame. Don’t do that.
I love this because I think the most pragmatic use of mechanistic interpretability has been in biology and probably other related sciences. The reason why I think it’s qualitatively different is that you go backwards: you’re learning about biology from the model rather than just trying to interpret what it’s thinking. In this case, maybe it is actually that forward case, but I’ve seen so much mechanistic interpretability being applied to biology because you’re really trying to understand. You don’t have the language in these machine learning models. There is no language to lean on, so you really have no clue what it’s thinking unless you’re inspecting the model.
Well, it’s really interesting. Yes, and one thing we’ve found very fruitful in the very recent past is transitioning to language models.
Okay.
What we’re currently doing is using readable, interpretable English to represent the small molecules, the proteins, and their behaviors.
Wow, okay.
And that turns out to work pretty well.
Yeah, I mean, this is crazy because—
Yeah.
I want to frame this a little bit before you tell us about this, because there’s actually a really interesting paper in the single-cell domain called GenePT.
Mm-hmm.
I can’t remember the authors. They’re from Stanford, and one of them was a statistician, not even really in bio.
Mm-hmm.
Maybe they are now, but they took the textual embeddings of the descriptions of genes and used those.
Yep.
So, like, you’re using the ESM embeddings?
Yeah, for Hermes, yeah.
So you were using the ESM embeddings. They were just like, “Let’s use the textual embeddings.” The beauty of this is that you have a gazillion years of human time put into understanding each of these genes, and that’s all captured to some extent by these textual embeddings, that knowledge in the embeddings. So, yeah, please tell us about what you’re doing.
5. Language Models Enter The Lab
We started this project a little while ago because we realized we had this growing collection of activity cliffs and profiles of the same chemical matter across large numbers of proteins. We wanted to know if we could use reasoning agents to explore it and use it in a lab-in-the-loop kind of situation. That meant we started working with reasoning models and language models to let them go play. We took some open-source language models, like Qwen and GLM, and taught them medicinal chemistry. We’d say, “Here is a structure. Describe it in natural English. Give me the difference between 2 molecules in natural English.” Then we got to the point where we could say, “Here are some molecules that are binders, and here are some that are not binders. Can you tell us why you think the binders are binding?”
Mm-hmm.
Then you can also give it hard negatives.
Yeah.
You have these 2 collections, and the models think, “Okay, well, I think this chemical series has this,” and write these things down in readable English, which we think is great. Then we said, “Okay, now that you sort of know these basic concepts, we’re going to give you 10 binders and 10 non-binders that are close, and ask you to decide why you think the binders are binding. Now we’re going to give you the opportunity to run your own experiments. Here is a catalog of millions of molecules. Why don’t you pick 5 or 10 and test your hypothesis about why they’re binding?” The model will go into the data we’ve already collected and pull out what it thinks are the most informative choices. It will get the answer back instantly, like it’s a coding sandbox, and then update its hypothesis about the binding. The ultimate goal is to get the best binder in as few steps as possible, very much like a medicinal chemist would do. And then we just want to know—
But you’re doing this based on your a priori-run experiments, and it’s looking things up in a database.
That’s right.
Right? So you’ve basically created a reinforcement rollout.
It is a reinforcement rollout at a large scale that’s growing all the time, full of these activity cliffs and different protein variations. Since we know that when you change the molecules in these small activity-cliff ways, it’s a causal change—we know that when you put this thing on there, the behavior changes in this reproducible, measurable way—you can use these trajectories to come up with reasoning traces at a pretty good scale. Right now, we have these agents running through the dataset. We give them 10 binders and 10 non-binders and a budget of 100 molecules, and they’ll go through probably about 100 tool calls, working through this long, multistep reasoning problem and coming up with what they think are the best answers at the end.
Why do you think this is superior to something like Hermes, or scaling up the Hermes approach, where you’re building some kind of attention thing?
We think it’s much more resistant to the memorization problem. We’re very happy with where the attention goes with Hermes when you throw in all these mutants, but as you add more parameters to a model, it’s going to memorize the mutants too, right? We want this thing to really understand these causal changes, and it seems like using natural language is a pretty good regularizer for the problem. We’re unaware of other problems where you have a reinforcement learning loop that is physically grounded, this scalable, and this fast. When you’re doing lab-in-the-loop stuff, where you’re making robots and going after these problems, you’ll still have limited resources. You’ll still make choices about what to make in the first place and what not to make, and those loops will take a little longer. If there’s a scalable way to measure that landscape and continually make it bigger and bigger, it allows the agents to play in it. We think it’s probably going to go a lot faster for this particular problem.
That’s interesting. I would say Exscientia is getting there, but they’re not actually using language. They are encoding things as a sequence, so it’s… In some ways, the perturbation is similar, but this is different. Lila also rhymes with this, but—
It does. All these folks are really smart and pushing these automation things.
Yeah. But they’re not doing these pooled screens and then using that. As far as I know, they haven’t figured out how to make that into a database lookup, so you can do a huge number of rollouts.
Yes. Right now, the way we like to think about it is that if you’re a medicinal chemist working at a pharmaceutical company, you do something called a structure-activity relationship. That means, “Oh, when I change the molecule in this way, it changes the behavior,” and so I’m trying to optimize for these different properties. You go through what’s called an SAR campaign. That’s early drug development: making the molecule better. You might see a couple dozen of these, maybe 50 of them, in your career. And so right now—
Yeah.
Last weekend, we pulled out over 10,000 of them, and we’re not even trying that hard.
Right.
So now is the opportunity. It will be restricted to the space of things we’ve screened, and then at the end, you can ask the models to make additional choices from catalogs. In the future, you can order those and retest them, like you might in a traditional lab-in-the-loop approach. Right now, we give these agents 10 binders and a budget of 100, and they’ll get the best binders out of 6.5 million within that budget. It seems like it’s working.
Tell us more about why you think this is effective. I get the argument that it’s a better regularizer.
Mm-hmm.
But also, why do you believe this is effective? You can imagine it doesn’t get things wrong so badly, but it also doesn’t get things that right.
I think it’s because you give it the opportunity to make all of those choices.
Mm-hmm.
It’s like running these coding sandboxes. You get a really rapid reinforcement reward signal. But you also have to include choices that a human being would not have made, because we’re measuring all the molecules across all the proteins.
Yeah. So it has to be able to make the counterintuitive decision—
Yeah.
—and get feedback on that.
It’s gotta have the counterfactuals. It’s gotta make mistakes.
Yeah, yeah.
And you need to make sure that the mistakes—or potential future mistakes—are present in the data in ways that aren’t biased toward it learning chemist sociology fingerprints.
Yeah. That’s really fascinating.
So, through all of the language pre-training corpus, it kind of has an intuition for some basic chemistry that I think most medicinal chemists might have an idea of. And now you’re giving it sort of an infinitely fast lab-in-the-loop story—
That’s correct.
—story.
But maybe reasoning too.
Yeah, reasoning. Yeah.
I would say that the reasoning skills—
Yeah, yep.
—might even be more important, right?
Yeah.
The ability to say, “Oh, in my reinforcement campaign—my learning campaign—I’ve learned things, and now that allows me to reason about—
Yes.
—the next—
Yep.
—harder question.”
And then just keep making choices.
Yeah, yeah.
Right now, we’re really excited about the early signal. We’re also a tiny shop, right? There’s only 10 of us. I’m really looking forward to when our experimental corpus grows at 2×, 10×, or 100× the rate that it currently does, because then the amount of space for these agents to explore expands dramatically.
Let’s talk about that space. My understanding—and this is from me talking to Claude and Googling and whatever—is that DELs are limited in that they tend to create flat molecules or flatter molecules.
It depends on the building blocks. You can imagine that you’ll be somewhat constrained by the designs: you’re going to get some number of building blocks, usually 2 to 3, and then stitch them together in some way. When we make chemical library design choices, one that we run a lot right now is a forked design. It’s a Y-shaped molecule. There are also linear-shaped molecules, and others that are kind of circular, macrocycle-shaped things.
Then you have to think about where the DNA is going to go and how that’s going to cause problems, along with the commercial availability of building blocks and the chemistries required to stitch them together. You can go for custom stuff, but it takes longer. It’s more appealing to us to go faster. There are certainly corners of chemical space that we’re probably bad at.
Then you have to think about what geometry future libraries should have, and the different proteins they’re going to be tested against. A whole chemical library made this way might be really good for certain types of targets and really crappy for others. Then you have to think about what corners of protein space you’re optimizing for, and those kinds of tensions.
So it sounds like the argument is: we know we’re not optimizing the whole space, but it’s a lot more than it was before, so that’s better. As we get better at this and understand more—or have a bigger budget or whatever it is—then we’ll expand to the corners that we know we’re lacking in. Or maybe we don’t even know we’re lacking in them, but we’ll find that out.
Right. Exactly.
Yeah.
Exactly.
Do we want to move on to the last phase?
Yeah, you can. Go for it. Let’s see how it goes.
All right. For audience context, Ian requested—I want to confirm that this was his idea, not mine—what you might call a “don’t ask me” or a “come at me, bro” section, where we ask the hardest problems we can come up with.
It was my idea.
We only had about—
As you guys can probably tell, this is not very hard for me. I think I’ve asked my hard questions already.
We only had about 24 hours to prepare, so I think most of these will be Claude’s hardest questions. Anyway, we’ll see how this goes.
The context for this is I was at a Cancer Microtherapeutics meeting last week. The format of the meeting is half the time is for content presentation, and the other half is for very aggressive grilling. It was horrifying and inspirational. So here I am, willing to do it in public.
Nice.
Let’s do this. DELs, as RJ was just alluding to, often have very specific properties. You can get very good at optimizing in DEL, but ultimately they frequently fail at certain things that are hard to translate to actual medicine.
Yep.
Certain failure modes include that anything easy to assemble is easy to disassemble, which is bad for ADME properties. You only have so much room to build on because you’ve already used some of the combinatorial building blocks, and so on. How is this limiting in terms of long-term out-of-library generalization?
I think a lot of it is going to depend somewhat on building block availability and one’s willingness to go into custom synthesis. For the fine-grained mapping stuff, being able to move a nitrogen all the way around the ring to all of those positions is very desirable, but it’s pretty unlikely that you’re going to get all those building blocks commercially. Is that going to be a failure mode? For sure. Yes. I would love to get more of those and beef up our custom synthesis. That’s something we’re looking at, but right now, getting those is challenging.
One critique Claude might be thinking about is that molecules that are easy to put together are easy to disassemble. For many years, the most common way to attach building blocks in chemistry was with amide bonds or peptide bonds, and those are quite digestible in the body. But in the past 4 years or so, there’s been a really nice proliferation of chemistries that are friendly to DNA and don’t destroy it during the attachment process, which is very important. That gives you carbon-carbon bonds, carbon-nitrogen bonds, and these kinds of things. So it’s actually pretty good for workaday simple chemistry. I’m not too worried about that part. I’m more worried about building block diversity, and there are sources for that.
What about ADME properties in general? You’ve been focusing very much on binding. How do you—
We have, yeah. In some of the projects we’ve done, we’ve gone directly from chemical library hits to good PK/PD in a mouse in 6 months, with a very small number of med-chem cycles. At least the chemical material we currently have—or those few series we did that for—turned out pretty good in our hands. We’ll continue to see how that modeling looks. Of course, we also want to do more ADME target models and think about how to incorporate other types of more emergent models along with our designs.
Was that speed driven by your ability to create custom, local combinatorial libraries? What allowed you to get there in 6 months?
When we designed these chemical libraries at the beginning, we very much optimized that space for drug-likeness. We think that’s part of it: it was a really good starting point. When we do our chemical library designs, we have this kind of Thompson sampling-level approach where we come up with an optimization function. You say, “Okay, well, I want the libraries to have these properties.” Some of those properties are: I want to bind these things, and I want to avoid these things.
There are other ADME model approaches that you can use to make predictions about how your chemical material might turn out, and you put those into the optimization function. Then you say, “Of all the chemicals I could make given my geometry, I don’t want to make that many. I want to make a smaller number.” You can set up and generate a bunch of synthetic libraries, then score them on this optimization function. We did that with this library, and the starting material looked pretty good. I’m sure we’ll continue to do that.
Along those same lines, has anything you’ve done made it to animals? Has anything made it to people?
Yeah. So, PK/PD in a mouse—that’s what that means. Can you dose a mouse with the thing and then see how it gets into the mouse’s body and tissues?
Yeah.
And then—
So PK/PD is pharmaco—
Pharmacokinetics.
…kinetics.
That’s right.
Pharmacodynamics.
Correct. Yes.
Meaning, how does the body move the drug through the body? Or how does the metabolism move the drug through the body?
Yeah. And then, are there biomarkers of the drug’s intended activity? Here, this is a compound we’re working on for inflammation, so we were following cytokine levels in mouse blood, and they did what we wanted and expected them to do based on the drug’s targets.
Sean likes to ask, “When money?” I want to ask, “When people?”
Yeah. For those, we have a couple of things we’re talking to people about partnering up to spin up new entities and go chase them down.
Okay.
And so, yeah. Of course, all these things take some time. There are traditional steps where you declare what’s called a development candidate. Then you’ve got to do a bunch of other work on manufacturing and additional safety before it can get into people. All these things take a lot longer.
Yeah. And so you’re—
Yeah.
—going to spin out.
Yeah.
The idea is to spin out. People, maybe even internally, will say, “I really want to take this thing to market or try to get it—”
Yeah.
—like, take it to the next step.
Yeah. One way we think about it is that we built this really great engine for data generation and model training, and now, as we’ve talked about, reinforcement learning and reasoning approaches. In most other domains, when you build an engine to help with those problems, the subset you generate along the way isn’t intrinsically valuable. No one wants to sell reasoning traces about political theory. But there are going to be little bits of gold that we’ve generated as a consequence of doing all this screening.
Yeah.
Here is why that’s good, and here is chemical material that’s proprietary to us, with activity in protein target classes that others might find hard to drug. We have starting material that looks pretty selective to us for, I think, 700 targets. It’s an enormous number. You don’t want to chase all those down, but they’re all opportunities with different levels of economic value capture and desirability. So you think, “All right, we’ve got all this stuff. Let’s see what we can get out of it.”
Discovery is a small percentage of the total cost of developing a drug.
Yep.
You can argue that any given discovery is cheap, but because of the failures, you have a lot of them. Even then, to get one successful drug through all the steps, a single drug candidate might cost, on the low side, $200 million and, on the high side, $1 billion, depending on who you have to test on and lots of other factors. Discovery is a couple of million. So, are you tackling the right problem here?
I think there are a number of early discovery problems that are worth reasonable amounts. What does that mean? There are a bunch of targets right now with really good genetic evidence that are very challenging to drug with small molecules. A pretty straightforward example is that, for a lot of these cytokine inflammatory indications, people are using antibodies to drug them. Going after a small-molecule therapeutic that you could take as a pill instead of an infusion, one that doesn’t require cold-chain transport to reach different patient populations, would be enormously desirable.
Meaning you have to have refrigerated trucks.
Yeah. Right, yeah.
Yeah.
If you want to get these antibody-based drugs to some of these folks, this is a very hard problem.
Yeah. And very expensive.
And very expensive. People are really quite excited about finding a small-molecule therapeutic that does the same thing as an antibody against these cytokines.
Right. There’s an antibody, and it works.
Yep. We know it works.
And you’re like, “Well, why can’t we just go after the same target—”
That’s right.
—“with a small molecule?”
It turns out those proteins are very hard to drug for a variety of reasons, but we like to think that the technology we’re developing can get good material. We do, in fact, have selective material for a lot of these cytokines. That’s one example of a very desirable target where—
Yeah.
—there are going to be much higher capital requirements to get it across the line and actually get it to market, but the interest is sufficiently high.
So you’re going to ruin the antibody market, basically.
If you can, then that’s good for some things. Antibodies are great for some things, and there’s a lot of energy going toward those groups because we’re good at protein folding, right?
Yeah.
And so they’re making—
Yep.
—a lot of progress. More broadly, I’d say that going after the right problem is a bit of a lamppost problem. The things where we can make enough scalable measurements in the life sciences are these early problems. We can’t really run a million-mouse experiment, right? We can’t run many millions of Phase 1 trials. We’re going to get better at this stuff and start to go up these different layers of biological abstraction with better data, better reasoning, and physics models. But right now, the technologies we have to make the number of measurements we need are going to be these early approaches, I think.
Yeah.
Yeah, and—
And I’d also argue that if you have an oracle, some AI oracle that says, “All the ADME properties will be good, and this generalizes to exactly this population,” then you have this oracle filter, and all your failures in clinical trials go away.
Yeah. You can imagine a world where you come up with an oracle, you have some new target and some new evidence that covers it, and then the oracle could give you small-molecule material instantly that you’d feel comfortable running straight to a mouse study.
Right.
Right.
But that requires training on data from mouse studies, clinical trials, whatever.
I mean, maybe. For ADME properties, definitely, if you’re confident in the target and need to make sure it’s not hitting all these other off-targets.
Yeah.
But even then, you probably don’t need clinical results to feel confident about running a mouse study right away.
All right. We’re on everyone’s favorite pet question. My favorite pet question is what I call the Kosuri paradox of machine learning and discovery.
Oh, here we go.
Yes, yes. Okay, so Sri Kosuri—
Bless him.
—great tweet. I love this tweet.
Love this guy.
Yeah, he was on just a few weeks ago. We hopefully will release his episode in the next few weeks.
Oh, okay.
Sadly, I actually didn’t even bring this up on the episode.
I’m so disappointed. Anyway, the tweet is basically: What is the business model in machine learning for drug discovery? To build a good machine learning model, you need lots of high-quality data at scale. You clearly have that. But if you need all that data to train the model, what do you need the model for? You can just go collect the stuff, and you clearly can collect it quite well.
Mm-hmm.
We talked a lot about data collection and machine learning, and I’m curious: Where does the actual machine learning come in? Is it necessary for your business model?
Of course, it depends on the business model. If your only rationale is to get good chemical material for a particular target, then of course Sri is right. You’d have it; you could get started. But if you had enough chemical material spread across all these other targets in a way that could drive a deeper machine understanding of chemical behaviors, then you could optimize those early materials much faster—especially at the scale we’ve achieved and with the low capital infusions we’ve been able to do this with. We haven’t raised a lot of money at Leash, and we screen an enormous amount.
One reason I brought up the language model stuff is that we’re starting to wonder whether it could enable a much broader scope of capabilities than just identifying novel chemical material. If we get really good at chemical reasoning, we could get really good at lots of different industries and their applications without having to physically screen in all those domains.
What’s an example of that?
Let’s say we get really good at making these predictions. Then we can go work in agriculture immediately, right? We could work in veterinary science or materials science. If we’re really good at it, we don’t have to screen all those proteins, because we have a good fundamental understanding of them.
Yes.
And the same thing could be argued for all the protein-folding companies, right? Now that these models are good, there are teams going after protein targets that haven’t been screened yet or are very challenging to screen, and getting really good starting material. The Nova guys did it with a receptor on cells that has a tiny piece of a mutated cancer gene. It’s really hard to do, and they managed to pull it off.
You can imagine a world where we do this with small-molecule material as well. I don’t think Sri is wrong: if you’re just trying to drug one thing and you screen a lot and already have the thing, you don’t need a model for that thing. But that’s not the same as generalizing across all things. We’re hopeful that if you have enough variation in engineered activity cliffs, you’ll get to a point where you can do all things—in which case you don’t need to screen ever again.
Yeah. Well, arguably, I don’t know exactly how much you raised, but for a campaign, that’s like a couple of campaigns that would cost the same amount.
Yeah, yeah. Maybe to put some real perspective on it, we’ve developed and screened about 2,000 different protein constructs, and done a total of probably about 5,000 screens in the history of the company, across at least 6.5 million molecules each.
Mm.
I don’t want to get into a huge numbers game, but it’s certainly almost as much as anyone else on Earth, maybe more than everyone else. Probably not—there are probably a couple of groups that have done a lot. But the number of mutations is probably more than everyone else’s. That seems likely to me. We’ve put a huge amount of work into making this very cheap and very scalable.
And if I was doing one at a time—
Yeah, if you went to a CRO and did one at a time, it would probably cost you $100K each, depending on how much handholding you got.
Right.
Yeah.
You just said 6.5 million molecules?
Yep.
That’s not a lot of molecules, especially since it’s combinatorial space. Everyone will say 10^40 or something.
They will.
10^60—big numbers.
They will.
I mean, obviously, you don’t need to understand all of chemical space.
Nope.
You just need to make drugs consistently, repeatedly. That’s your goal. But is that actually enough chemical space to meaningfully do anything outside of your chemical space?
Great question. The pleasant answer is that I said earlier we’ve trained models on internal data only, and we’re able to extend and make generalized predictions into ChEMBL, which is in very different corners of chemical space. So, yes. Do we think it’s enough? Not remotely.
Part of our roadmap is making larger numbers of chemical libraries in different corners of chemical space, spanning what I talked about earlier: some very fine-grained changes to more pronounced activity cliffs, and then, on the other side, big, coarse changes to try to capture new proteins and protein shapes.
This is—sorry, not a hard question; I’m genuinely curious. Are you able to predict potency, like relative potency, meaningfully? Can you do KD curves, where you titrate larger and larger libraries?
People have wrestled a little with trying to get the KDs directly out of DELs. One thing people do is vary the protein concentration. When we do replicates of screens, we get back the same hits in roughly the same order, which makes us optimistic that we could get these kinds of things.
We’ve started moving more into SPR biophysics-related measurements to figure out whether we can come up with a potency ladder that we could use as part of the chemical material we’re screening. But I won’t say that we’re good at it yet.
Yeah. You can’t just use reads as a proxy for KD or something?
I think the relative ranking, yes. But I think the KD—
You can do it relatively.
Reads for KD, not really. Then you start running into weird stuff. We’ve encountered lots of proteins that aren’t described as DNA binders but that we’ve discovered do bind DNA. You can imagine that might affect the KDs of things, and we have to figure out how to mitigate those problems.
I think relative ranking is quite doable. Translating it into KDs is something we’re actively working on. We think we might be able to get a reasonable, somewhat sloppy translation, but I wouldn’t say we’re there yet.
Okay. This one was pretty brutal.
Oh, great.
The bitter lesson—yeah, I’m just going to read. This one is pure Claude. I had no thought in this: If the bitter lesson is really the law, what stops a better-funded lab from outscaling your DEL engine?
Of course, people are going to ask us this. What I would really say is that, both at Leash and at Recursion, my team has made so many mistakes by accidentally bumping into the cheating modes that come from doing laboratory work. There are so many of these, and you have to chase them down. A lot of it has to do with tracking batch effects with metadata, for example.
You have to build a culture at your organization where you say, “All right, we have to track all these things. We have to build tools to track them. We have to make those tools frictionless so the people doing the work will actually record these things.” Then you have to go back and look at all the data throughout the company’s history to make sure you’re getting really consistent results. When you see big changes, you have to identify what drove them and mitigate those. It’s a huge amount of work.
We also rebuilt the selection process from scratch and rebuilt a lot of the protein expression process from scratch. Then we have all these other proxies and filters that we use to evaluate how well the protein production went, and whether it’s good or bad.
It takes a unique culture, patience, and a willingness to make these mistakes, and then have the team think deeply about how to address them. Unless you've got folks who really have a nose for these kinds of experimental mistakes, it's going to take some time to replicate.
Process moat.
Yeah, I really think so. There are so many things that can go sideways, and you have to pay enough attention to identify them when they do. They might be things you can write automated filters for, but maybe not early on.
One pretty good example I'm happy to talk about is that people ask us a lot, “How do we know our protein's really good?” We have lots of ways to figure this out. One of my favorite examples right now is that we've noticed when proteins misfold, we're able to identify it in very crude cases. If you run the protein on a gel—which is a protein trick where you try to figure out how big it is—and it misfolds, you can detect that. But if it misfolds a little bit, it can be hard to detect. There are drugs whose job it is to refold misfolded proteins.
Correct.
They're called molecular chaperones.
Oh, okay.
You can call them creatures too. That's what Sri would call them.
Yeah.
There's a very famous one that treats cystic fibrosis patients called lumacaftor. We have a bunch of chemicals in our chemical library that look like lumacaftor, whose job it is to refold misfolded proteins. Guess what kind of proteins it hits when we screen them?
Cystic fibrosis? Sorry.
Misfolded proteins.
Misfolded proteins. Oh, yeah.
Misfolded proteins in there.
Oh, wow.
That's what it's for, and it's in our libraries. We discovered this by accident: we saw these chemical series keep popping up across all these different protein types. We wondered, “What is that? Is that lumacaftor?” Then you start looking at the protein expression and how it went, and it's like, “Oh, it comes up when things aren't going well.”
So now we can use it as one filter: if we get a bunch of lumacaftor hits, that protein probably didn't fold right. We probably don't want to use that as training data. We probably want to rerun it with slightly different conditions that are friendlier to the protein. Then we'll see what those hits look like.
Unless you're paying that kind of attention, it's really hard. You have to build tools so you can look at these things, and have people who know to look for them and recognize them. It's a hard problem.
Yeah, going back to everyone who touches DELs comes out traumatized, somehow you've figured it out.
I've probably supported this one.
Yeah, you're also traumatized, but in a good way.
Yeah. No, it's really hard. That's—
Yes.
And that's why—
Yes.
When you have the motivation to find just 1 or 2 really awesome hits to start off a drug program, then you're not going to chase down all these different aspects of the assay. I get it, but if you're motivated for it to be machine-learning fodder, then you have to pay attention to all this stuff.
One of the questions that we always ask—
Mm.
…at the end is: If you could, by fiat, remove a bottleneck—
Mm-hmm.
…what would that be?
When I think about “by fiat,” it's one of these things that's actually currently happening: watching sequencing costs drop like a rock. I'm thinking, okay, what do we really want to do? We want more chemicals and more measurements. Those are the things that would make my life a little easier—the problems that are immediately on the ground.
So you want— Sorry, I don't understand.
Cheaper sequencing, more molecules.
Okay.
Yes.
In sequencing, can you give some context for how much it's dropped?
Oh my God. Many, many, many orders of magnitude. I think—
The first genome cost $1 billion, and now we can do a human genome for $1,000, right?
I think it might be $100.
$100. Yeah, $100.
It's really cheap. There were stretches where it was dropping faster than Moore's Law.
Yeah, that was exciting. It's crazy. Why do you want even cheaper sequencing?
Oh, I'm just happy with the trend.
Oh, okay.
Yeah.
What would you remove by fiat that hasn't been removed?
Here's a good one. There are different gizmos you can use to purify small molecules. Right now, when you make them, you put some stuff in a pot and heat it up or use some other conditions, and you get the output. That works at some level of efficiency; it might be great or crappy.
In either case, especially if it's crappy, there are other gizmos you can use to clean it up. People use HPLC or some other tricks. None of them are really good and super-parallelizable. Being able to clean up stuff at a massive scale would be really nice, because we run all these molecules in this funky assay that we've already described as giving people lots of trauma.
It would be great to get large numbers of molecules in a different assay that's clean, purified, and easy to work with, and then do that next. Then you could really orthogonally validate all those measurements you're making.
Yeah, that'd be really, really nice. Of course, everyone complains about GPUs and stuff—
Yeah, GPU, well, that's the one—
Yeah.
…that everyone says, besides GPU.
Besides GPU.
Yeah. All right. And the other question is: What would you like people to take away from the episode?
I would like people to take away that, at least for us, we believe we're building something really exciting: what feels like a physically grounded substrate for all sorts of interesting reasoning and biomedical applications. If that’s of interest to them, please reach out to us. We’re hiring machine learning people. We want to talk to partners, and we’re building this thing that we think will be transformative. So come talk to me.
Amazing. Cool. That was great. So much fun.
Thanks, dude.
Thank you for coming.
Thank you.
Yeah. Yeah.