AI 驱动原子：Periodic Labs 联合创始人 Liam Fedus 如何重塑材料工程

Sarah Guo

We're talking with Liam Fedus. Liam is one of the co-creators of ChatGPT, which I think almost everybody uses at this point. He was the VP of post-training at OpenAI, and before that was at Google Brain, where he worked on a variety of really early AI innovations.

Liam will be telling us a bit about Periodic Labs, his company, which is focused on building an AI foundation lab for atoms. In other words, how do we impact the physical world—materials science, chemistry, and so on—using AI? It's a very exciting topic, and I'm excited to be talking with him today.

Liam, thank you so much for joining us today on No Priors.

Thank you so much for having me. It's great to see you.

Sarah Guo

I think you're doing incredibly interesting things in terms of alternative types of models, specifically for materials science and the physical world. Effectively, what you're building is an AI foundation lab for atoms, which I think is fascinating.

That's right.

Sarah Guo

Maybe we can start with a little bit more of your background. You were VP at OpenAI, and you worked on one of the first trillion-parameter models ever. Could you tell us a little bit more about what got you here?

Even further back, I was a physics major in undergrad. I spent some time doing dark matter research. We had an apparatus that was sensitive to the direction of dark matter, so it was very interesting.

Sarah Guo

Why are there so many—I’d love to come back to this—but why are there so many physicists in AI right now? You look at Dario Amodei, who runs Anthropic. You look at Adam Brown at Google, and a variety of other people, and they all have these physics backgrounds.

My old manager, Josh, also studied physics and is at Anthropic.

Sarah Guo

Why do you think that is?

I think it's a great way to think about the world. It's very principled, very hard-nosed, scientific, and careful. I don't know; I think it's just such an incredible field. You have such high leverage in computer science and AI, and I think a lot of physicists were seeing that.

Particularly in high-energy physics, with the discovery of the Higgs, I think a lot of high-energy physicists were looking for what's next. Ultimately, it becomes bottlenecked on the new apparatus for pushing the next energy frontier. I think a lot of physicists were looking at their skill set, looking at the progress elsewhere, and saying, "Hey, I think I could be a huge contributor elsewhere."

Sarah Guo

It's been fascinating to see string theorists, people working on black holes, and all sorts of physicists moving into AI. It's almost like we're recreating a Manhattan Project, except now what we're seeking is different forms of intelligence. It's an interesting perspective.

That's right.

Sarah Guo

Sorry to interrupt. You studied physics and worked on dark matter.

That's right. In grad school in physics, I was always gravitating toward machine-learning problems. I was looking at particle reconstruction, which is effectively a machine-learning problem. But it felt like, if I really wanted to push the frontier of machine learning, I should be in computer science.

I ended up at Google Brain, overlapping with the first-year residents there. It was an absolutely remarkable group of people and a remarkable period for Google Brain. This was the era of the creation of distributed-training strategies, mixture-of-experts, and the Transformer. It was a really rich period in that history, and it was a fun Cambrian era where people were really pushing the frontier with just a handful of GPUs and very small collaborations.

The field was at a much earlier stage, and I think there was a lot of diversity and entropy in the research. It was very fun.

Sarah Guo

So it was kind of the late 2010s or so, something like that?

This was 2016 and 2017. Google Brain at that point was still very small, and it was eventually combined with DeepMind.

Sarah Guo

Were you at Google for many years?

Mostly, I was doing architecture work. I was really pushing sparsity. That allows for more efficient serving of models at scale and really pushed the scale of what we could do.

Toward late 2022, I became excited about the creation of products. The technology was getting very compelling, so I ended up at OpenAI with some other Googlers as well.

Sarah Guo

What did you work on specifically at OpenAI?

The goal was to come up with some productionization of GPT-4. OpenAI had GPT-4. It was pre-trained, and there was some rough post-training on it. There were questions about, "Okay, how do we turn this incredibly powerful model into products?"

We were spitballing ideas like a writing bot and a coding bot, which were very natural at the time. Some of our least interesting ideas were a meeting bot. It would just sit in a Google Meet, take notes, and then send out to-dos afterward.

John Schulman was very opinionated. He said, "We think we should keep it very general. Let's do a chatbot." That became a large part of the effort for those few months.

Sarah Guo

That's right. So you worked on ChatGPT.

That's right.

Sarah Guo

Obviously, I felt like that was the starting gun of this whole AI revolution, or at least in terms of people's awareness. I'd started investing in the area beforehand, but it seemed almost like a secret until ChatGPT came out, and suddenly everybody realized that there's this powerful technology available.

Yes.

Sarah Guo

How did that lead you to materials and atoms and the physical world? I know that was your starting point academically, but what brought you back, given how much is being transformed right now through language?

I think it's just the inevitability of connecting these systems to the physical world. The view that I and others held at Periodic was that you're not going to see the same kind of acceleration in science and technology unless you start connecting these things to the physical world.

Science ultimately isn't sitting in a room thinking really hard. You have to conduct experiments, learn from them, and interface with reality. The creation of ChatGPT in late 2022 was an important technology, but it was still far too weak. We couldn't have done Periodic with the technology of that era.

Over the next few years after that, we saw ever-improving models. We saw reasoning, and test-time inference became really important. That led to more reliable error correction and more reliable tool use. We saw the rise of coding agents and other agents.

Sarah Guo

Mhm.

I think those were foundational technologies necessary to connect these systems to the physical world. It was just not possible with the AI technology of 2022.

Sarah Guo

I guess the other thing that's missing from the physical world is data, or at least data that's easily accessible. You look at something like the big foundation models on the language side, and they're basically trained on the internet as a major corpus. It's augmented in all sorts of ways with other data sources.

How do you think about that for what you're doing, where you're trying to model atoms in the physical world, and how all that stuff works?

We have simulations—physics simulations—and experiments. I think exactly as you're pointing out, machine-learning systems are good on the data you've trained them on and on the tasks you've trained them to do.

I think sometimes there's this mythology of AGI, ASI, and RSI. We see increasingly powerful systems, but they do become limited if they don't have access to the raw data to actually make informed decisions.

Sarah Guo

How much data do you need? I know that there's some data-scale-related research and other work on how you hill-climb toward a really good model.

Yes.

Sarah Guo

How many experiments do you need to run? How many data points do you need? How do you think about the diversity of data points you need to generate? I'm a little bit curious: what does that actually look like tangibly?

There is some generalization from the existing models. We don't need to reproduce a system that can understand and write English or write code, so we're leveraging—

Sarah Guo

Are you using open-source for that, or closed-source models, or some mixture?

A combination.

Sarah Guo

Uh-huh.

For example, Periodic spends zero effort on improving coding models. We're incredibly impressed by Codex and Claude Code, and those have been a huge accelerator for the company. But we focus our machine-learning efforts where the existing frontier isn't sufficiently good for us.

Going back to the data question, we're leveraging on the order of tens of trillions of tokens that went into open-source models. That's given us a very foundational understanding. But once we start moving into specific discovery areas and chemical spaces, we can see a very high level of sample efficiency.

The system isn't starting as a randomly initialized neural net. It has a strong prior on the world.

Sarah Guo

Where does that prior come from? What data informs it? Just general—

Papers, yeah—the internet, as you're pointing out.

Sarah Guo

Yeah.

However, that's insufficient.

One of the engineers on our team was looking at a reported material property, and they were just extracted values from the literature. It was really interesting to see that the reported value spanned many orders of magnitude.

And so you train a machine-learning system on that, and it's like, well, the best you can do is model this distribution, but you're no closer to a ground truth. And that's where experimental data comes in, where you now have a grounding in this.

But what's really important is that it's not just a pool of data. It's this interactive, closed-loop system that is so powerful. Once you have the experimental data, you can look through it for aberrations, patterns, and consistency with simulation data and literature, and then that helps drive the next set of experiments. So it's not just a pool of data; it's this very active loop.

I see. And then how do you think about diversity of data? I look at something like AlphaFold or some of the protein-folding-related models, which are amazing, right?

Yep.

If you think about it, I used to work as a biologist, and a crystal structure would take years, if it happened at all, because you wouldn't necessarily be certain you could crystallize a specific protein under certain reagent conditions in a way that would work for actual crystallography, or NMR, or whatever approach you took for structure. And then AlphaFold comes out, and you can just arbitrarily model anything in the protein world, which was amazing as a breakthrough.

Right.

But it was a very specific data set that already existed, with lots and lots and lots of structures.

Over decades.

Over decades of work. How hard do you have to bootstrap that for every single materials domain, or do you choose specific ones and then generalize?

We have seen internally the greatest advances where we have an abundance of data in some space, and that has led to the highest rate of acceleration internally. But I think you can think of different levels of generalization. For systems that are strongly governed by quantum-mechanical effects, there is some generalization there.

I see.

But if you produce a system that has modeled quantum-mechanical objects really accurately, it's not really helping much with fluid dynamics or another kind of level of abstraction. And so the generalization we're seeing is quite good.

Oh, that's so interesting. So you could do the basic steps of chemical synthesis, quantum mechanics, and different aspects of how atoms interact in general, or van der Waals forces, or things like that.

Absolutely.

Oh, that's so interesting. Yeah, that's cool. And then, from an architecture perspective, is there anything unique or interesting that you're doing? Or can you talk a little bit about how you're actually constructing some of these models?

Yeah, so language models are incredibly powerful. It's a very natural interface, and so we continue to use them. But we think about them almost as an orchestration layer. That's sort of a copilot assistant, but also a system that can direct experiments. It's almost orchestrating other specialized models as well.

So we do construct neural nets that are specially designed for atomic systems, where there's some symmetry awareness. Those have much lower latency, and they've been fine-tuned for that. Basically, you can think of this as an orchestrating layer that can ingest literature, go through our experimental data, and go through different modalities, but it can also use specialized neural nets—

Mhm.

—as tools and as reward functions. So it's an overall system.

Okay. Yeah, that makes a lot of sense. I've seen a lot of people architect those sorts of approaches, even for things like customer support or other areas. It seems like it's the common architecture that's emerging as you're doing these different use cases of these models.

Yeah. But transformers have been very powerful.

Yeah, yeah. And that's really cool. If I look at the language world, one of the things that was pretty unique about it, and the reason I think companies like OpenAI, Anthropic, and others are growing so fast, is that it plugged into a very big domain of human existence, which is all language. All language means enterprise software and enterprise interactions, and it means consumer behavior. It's basically how we interact with the world.

Yes.

It seems like there's a little bit more of a leap for other areas. For example, in robotics, there are really interesting different types of robots that exist in the world, but the footprint of that is quite limited relative to language. The same seems to be true for materials science. So how do you think about where you're going to commercialize this first, who you're going to work with, or whether there are specific domains of products that you're working on first?

We've begun working very closely with scientists. We've treated Periodic Labs as our customer zero, to see how we can transform how this field of science is done. But there are huge opportunities across all of these industries and enterprises that are interfacing with the physical world—people who are bottlenecked by materials engineering and process engineering.

Again, those are kind of the same natural interfaces where engineers are asking questions about their data, trying to find aberrations, trying to debug machinery, and trying to get to a better formulation. It's actually quite universal as well. And so we've created our little testing ground internally, and now we're sufficiently excited about the technology we've been building and seeing this acceleration for advanced manufacturing more broadly.

Is your model going to be developing materials for other third parties? Is it developing your own materials that you then sell in the market? Because it almost reminds me a little bit of a biotech model. In biotech, you can either partner with a big pharma company and effectively help them create a drug and take a royalty on it, or you can build your own drugs. How do you think about that in the context of what you're doing?

We're thinking about ourselves as an intelligence layer for these companies. You can think about it as a system of record and a control plane for different experiments and getting to solutions. But, like you're saying, there's a very interesting aspect: some breakthroughs here could have really high value, and it might be more akin to a discovery model, like we've seen in biotech and elsewhere. But to start, we're thinking about ourselves just as a software business.

Have you ever heard of The Diamond Age?

No, I haven't, actually.

It's a Neal Stephenson book. It was written in the 1990s, and there are 2 key concepts in it. One key concept is that there's effectively an AI tutor unleashed on the world, and it teaches huge numbers of young girls all sorts of skills. It's a very interesting thing about AI education, and then, in parallel—

Why young girls in particular?

Basically, an AI research scientist creates a Primer for his daughter, and the Chinese steal it, clone it, and distribute it across the country. Because he built it for young girls, suddenly every young girl in China has it.

Right, right.

That's the reason. It's this very China-theft-of-IP kind of thing.

Yes, right.

The other part of the book is about matter being piped into everybody's homes, and they all have 3D printers. You download blueprints, and it just creates whatever you need in the physical world. Some people start evolving different nanobots to do different things. It's this very advanced kind of AI-plus-materials future world.

Yes.

What is your vision or conception of what our world looks like in 10 years, assuming Periodic Labs is successful?

As you're pointing out, you're going from systems that aren't just writing essays or writing software, but are literally generating matter. I think it has pretty profound implications for semiconductors, aerospace, and energy, and I think it's incredibly important to ask whether we can increase the pace of the physical development of the world.

We see how quickly the digital realm is changing. Software engineering now looks wildly different than even 6 months ago. But I think we see similar opportunities in the physical world. Of course, atoms are hard, and so you will have some limits of physics, but just because atoms are hard doesn't mean there's not an order of magnitude or 2 to speed up just making sense of huge amounts of data and getting to solutions more quickly.

What we're trying to do is give humanity this agency for atomic rearrangement and synthesis, and we think it's going to be a huge accelerator. If our physical world could keep up at some fraction of our digital world, I think life will just feel dramatically different.

The kind of revolution that could really come. It reminds me almost of the materials equivalent of the agricultural revolution, where you suddenly had a massive spike in productivity and output.

Sarah Guo

Exactly. It seems like there have been all sorts of bottlenecks that have constrained us until now that you folks are trying to address.

That's right.

Sarah Guo

What aspect of the work that you're doing are you most excited about?

The iteration between these groups of people. This is just irreducibly a multidisciplinary problem. We have physicists and chemists working really closely with some of the top AI researchers in the world, working closely with some of the best engineers in the world.

Sarah Guo

Mhm.

This multidisciplinary, really close collaboration is just absolutely incredible because you're seeing firsthand how a field can fundamentally change. People who have been doing research for, in some cases, decades in a field are now seeing, "Oh, under these systems, under intelligent systems, it could look this very different way."

I use an analogy to machine learning a lot, going back to the early Google Brain days, where the frontier was pushed forward by a few GPUs and a few people. Now you look at this era where it's really industrialized, and there are dozens—hundreds—of researchers working together with hundreds of thousands, millions of GPUs, dictated and driven by scaling laws.

Sarah Guo

Mhm.

Everything is about scaling. It's given us that predictability. It's allowed us to put huge amounts of capital into this field. I think the physical sciences and physical engineering will have a very similar property, where we establish these scaling properties and bring that mindset.

Periodic Labs is really thinking about how we bring much larger-scale sets of experiments to bear on this. Intelligent systems have enabled this, automation has enabled this, and you really need both. With an improvement to automation, you can soon create bottlenecks in intelligence. The scientists very much feel this, where they're not used to working at that level of throughput, and they simply can't make sense of so much data.

Sarah Guo

It's interesting. I guess in terms of scale here, one of the real benefits—one of the things that's really benefited the frontier labs on the LLM side—is just scale of capital, and therefore scale of GPUs and scale of data.

Of course.

Sarah Guo

Is this similarly a capital-intensive area in your mind?

We will require more capital. GPUs are so extraordinarily expensive. What's interesting is that the compute cost relative to physical infrastructure is actually surprising. So much money is spent on compute that the physical infrastructure is sometimes actually lower, but it has very large lead times, and there's the intrinsic difficulty of having these well-calibrated, well-functioning physical systems. From a capital perspective, it's primarily a compute cost.

Sarah Guo

It's really interesting. If you look up the cost of a Stanford postdoc, for example, relative to a machine learning engineer, it's such a big difference. My takeaway is that many people working in science, particularly in an academic-center setting, are very undercompensated relative to their societal value.

Absolutely.

Sarah Guo

I always like it when companies help bring people into the fold, both in terms of human impact, but also that ability to do things at real scale and really do things a different way. It must be very exciting for the people on your team.

Some of the scientists who joined us are among the best in the world, and it's been absolutely incredible working with them.

Sarah Guo

It sounds like you've built such an amazing interdisciplinary team. Are there specific roles you're actively looking for right now, or key things that you really want to hire for?

Absolutely. On our site, we have decomposed the world into bits and atoms. It's a loose taxonomy, but on the bit side, we're really thinking about mid-training and pre-training roles from the AI side, and always more infrastructure roles. On the atom side, there are control engineering and systems engineering roles, but we're also now thinking about spanning that with product engineering.

Sarah Guo

Across the board.

A lot of active roles, et cetera.

Sarah Guo

That's really cool. I think one of the things that everybody's really thinking deeply about or is excited about right now is AGI, ASI—these advanced systems that are as good as humans or better than humans at different things, or are very generalizable in terms of their abilities to do a broad swath of things. How do you think about that within the context of what's happening over the overall foundation-model curve? Obviously, you were very integral to the development of some of these systems. How do you think about that applied specifically to some of the areas you're working in?

I think one fallacy is thinking about intelligence as a scalar. We've consistently seen these systems have a very odd spikiness, and it's actually possible to architect a system that is world-class in some math domain, but then you could perturb the questions and actually degrade it substantially. It's like a bad high school student. There's this odd spikiness to these systems.

Sarah Guo

So basically, you can make a system that's a genius at one thing and not very good at a bunch of other stuff.

I guess the point I was making is that those fields can actually be quite adjacent. Sometimes the generalization can be nonintuitive. One way I think about recursive self-improvement is really akin to neural architecture search from roughly 10 years ago.

I think there's a very clear path for software engineering. These systems have become so incredibly impressive in this domain as a result of huge amounts of data and really cheap, verifiable environments. You can check unit tests going from failing to passing with just a few CPUs. It's basically instantaneous. There's no domain-expertise gap between an AI researcher and a software engineer. Obviously, this will become—and is becoming—a larger contributor to the next generation of the system.

Sarah Guo

When do you think it flips into everything being machine self-improvement versus being human-directed or needing a lot of human intervention? Do you think that's 2 years away? Do you think it's 5 years away? Do you think it's 10 years away?

Building on what I was saying, I think there's a domain caveat to that.

Sarah Guo

Sure.

Rolling forward that software-engineering self-improvement, I think you're going to have a system that can write complete repositories, identify bugs, and refactor code.

Sarah Guo

Mhm.

But it doesn't suddenly understand biology.

Sarah Guo

Sure.

Right? There's a domain gap there in knowledge.

Sarah Guo

Yeah.

But even beyond that, there are sets of strategies used in software engineering that differ from scientific or engineering strategies. You're not operating under—it's not like decision-making under uncertainty to the same degree. It's very verifiable, and that's driven so much of our work.

Sarah Guo

Mhm.

In that domain, I think it's happening now-ish.

Sarah Guo

Mhm.

I think we'll see the same thing for AI research.

Sarah Guo

Uh-huh.

That's a slower outer loop because now the experiment isn't just checking whether some unit tests are passing; it's checking: What was the scaling property? Did this model converge? What's the generalization of the system? That requires GPUs and many hours of experiments.

Sarah Guo

Those are all evals that people use today as they're looking at existing models, so they do have that utility function, that feedback loop that can be driven by self-learning.

That's right. But again, the connection of these things to the physical world is going to be so critical because both of these systems are being trained in a closed loop against that domain. So it's a closed loop for doing software engineering, a closed loop for doing AI research.

That's the premise of Periodic Labs: We need to have these closed loops of actually doing science and actually doing engineering. These two domains are how I think the rest of the world will go, with some delay. This is, again, the foundational technology that we're building.

Sarah Guo

Super interesting. Do you think you need sufficiently good robotic systems in order to have that closed loop for what you're doing? In other words, do you need something like Pi or Skild, or something else, to work in order for Periodic Labs to hit that escape velocity in terms of a closed-loop system?

No, but it's a huge accelerator.

Sarah Guo

Mhm.

The goal for Periodic Labs is to generate high-quantity, high-quality, diverse data. Automation is assistance to that. Right now, we employ people as well, and we have autonomous parts that are very reliable. If you had a dexterous humanoid who could wander into an unstructured lab, make sense of it, and follow instructions reliably, that would be a huge accelerator.

Right now, automation of physical systems requires very careful design and is slow, but I think improvements in robotics are going to accelerate this. Already, the reliability of these hybrid systems is sufficient to produce huge amounts of reliable data, but it's just going to accelerate us further.

Sarah Guo

Yeah, one of the reasons I ask is that at a company called Color, we built our own liquid-handling robotic systems. We’d buy liquid-handling robots, but then we had to adjust them dramatically. We had cameras that used ML to monitor the system and make adjustments. We had to 3D-print parts to decrease vibrations on the platform because we were dealing with such small volumes of liquid.

And so there was an enormous amount of customization versus just having it in the firmware. It was awful, and writing against that was painful, versus just having a robotic system that would work like a modern system in all the ways that you’d conceive of. That’s the reason I was asking: if you really want to do high-throughput experiments, you need these underlying systems to be able to do all the liquid handling, the titration, and all the rest of it.

Yeah, that’s right. I think right now we’re using more like off-the-shelf robotics. It’s very simple, very commoditized, and we’re not doing a huge amount of innovation on that front. But again, as these more general robotic systems come to hit this reliability threshold, it’s going to be a massive accelerator for spinning up new labs as well.

Sarah Guo

Yeah, you’ve seen such a wide range of different things happen in the AI world since you were working at Google, I guess, about a decade ago at this point. You were there during the birth of the Transformer model, and you were there for the birth of ChatGPT. What are you most excited about outside of Periodic over the next few years in terms of what’s happening with AI?

I mean, of course, robotics. Again, I’m just so excited about the interface of AI systems with the physical world, and we’re approaching one angle of that, which is science and engineering. We need that data in order to make those advances. But simply having agency and control of the physical world via robotics is going to be transformative.

Sarah Guo

I’m very excited about these interface layers. I think that’s going to be such a massive opportunity. How many software engineers are there in the world versus people who are in the physical world?

Mm-hmm.

Sarah Guo

And there are labor shortages everywhere. So, yeah, I think it’s going to be a very interesting decade.

Oh, amazing. Well, thank you so much for joining us today.

Yeah, well, thank you so much. That was very good chatting today.