DeepSeek、中国、OpenAI、NVIDIA、xAI、TSMC、Stargate 与 AI 超级集群｜Lex Fridman 播客 #459

The following is a conversation with Dylan Patel and Nathan Lambert. Dylan runs SemiAnalysis, a well-respected research and analysis company that specializes in semiconductors, GPUs, CPUs, and AI hardware in general. Nathan is a research scientist at the Allen Institute for AI and is the author of the blog on AI called Interconnects. They are both highly respected and listened to by experts, researchers, and engineers in the field of AI. I used the DeepSeek moment that shook the AI world as an opportunity to sit down with them and lay it all out, from DeepSeek, OpenAI, Google, xAI, Meta, and Anthropic to NVIDIA, TSMC, US–China–Taiwan relations, and everything else happening at the cutting edge of AI. This is the Lex Fridman Podcast. And now, dear friends, here’s Dylan Patel and Nathan Lambert.

OpenAI o3-mini came out right after this conversation. Its capabilities and costs are on par with our expectations. DeepSeek R1 has similar benchmark performance, is cheaper, reveals its chain-of-thought reasoning, and is open-weight, while o3-mini is not. Anecdotally, o3-mini high felt better than R1 to me personally. Claude Sonnet 3.5 is still my favorite model for programming, except for tricky cases where I use o1 Pro.

A lot of people are curious to understand China’s DeepSeek models, so let’s lay it out. Nathan, can you describe what DeepSeek V3 and DeepSeek R1 are, how they work, and how they’re trained? Let’s look at the big picture, and then we’ll zoom in on the details.

DeepSeek V3 is a new mixture-of-experts Transformer language model from DeepSeek, which is based in China. It has some new specifics in the model that we’ll get into. Largely, this is an open-weight model and an instruction model, like what you would use in ChatGPT. They also released what is called the base model, which is before the post-training techniques.

Most people use instruction models today, and those are what’s served in all sorts of applications. DeepSeek V3 was released on December 26, or around that week, and then, weeks later, on January 20, DeepSeek released DeepSeek R1, which is a reasoning model that really accelerated a lot of this discussion.

This reasoning model has many overlapping training steps with DeepSeek V3. It’s confusing that you have a base model called V3, that you do some things to get a chat model, and then you do different things to get a reasoning model. I think the AI industry is going through a communications challenge right now. OpenAI makes fun of its own naming scheme: it has GPT-4, OpenAI o1, and many different types of models.

We’re going to break down what each of them are. There are a lot of technical specifics about training, so we’ll go from high level to specific and go through each of them.

There are many places we can go here, but maybe let’s go to open weights first. What does it mean for a model to be open weights, and what are the different flavors of open source in general?

This discussion has been going on for a long time in AI. It became more important after ChatGPT, or more focal after ChatGPT at the end of 2022. Open weights is the accepted term for when the model weights of a language model are available on the internet for people to download.

Those weights can have different licenses, which are effectively the terms by which you can use the model. There are licenses that come from the history of open-source software, and there are licenses designed by companies specifically. Llama, DeepSeek, Qwen, and Mistral—these popular names in open-weight models—have some of their own licenses.

It’s complicated because not all models have the same terms. The big debate is about what makes a model open weight. Why are we using this term? It’s a bit of a mouthful, and it sounds close to open source, but it’s not the same. There’s still a lot of debate about the definition and soul of open-source AI.

Open-source software has a rich history around freedom to modify, freedom to take it on your own, freedom from many restrictions on how you use the software, and what that means for AI is still being defined. I work at the Allen Institute for AI. We’re a nonprofit, and we want to make AI open for everybody. We try to lead on what we think is truly open source.

There’s not full agreement in the community, but for us, that means releasing the training data, releasing the training code, and also having open weights like these. As we get into the details of how the models were trained, we’ll repeatedly say that data processing, data filtering, and data quality are the number-one determinants of model quality.

A lot of the training code determines how long it takes to train and how quickly experimentation can proceed. Without fully open-source models, where you have access to the data, it’s harder to replicate the results. We’ll discuss cost numbers for DeepSeek V3 based mostly on GPU hours and how much you could pay to rent those GPUs yourself, but without the data, the replication cost is going to be far higher. The same goes for the code.

We should also say that this is probably one of the more open models among the frontier models. On the full spectrum, the most open model would have open code, open data, and open weights. DeepSeek does not have open code, and it probably does not have open data, but it does have open weights. The licensing is an MIT license, although there is some nuance between the different models, so it’s toward the free end of the open-source movement.

DeepSeek is doing fantastic work for disseminating an understanding of AI. Its papers are extremely detailed about what it does, and for other teams around the world they’re very actionable in terms of improving their own training techniques.

The DeepSeek R1 model has a very permissive license. It’s called the MIT license, which effectively means there are no downstream restrictions on commercial use and no use-case restrictions. You can use the outputs from the models to create synthetic data, and this is all fantastic.

The closest peer is something like Llama, where you have the weights and a technical report. The technical report for Llama is very good. The Llama 3 paper was one of the most-read PDFs of last year. In some ways, though, it’s slightly less actionable. It has fewer details on the training specifics and fewer plots, and the Llama 3 license is more restrictive than MIT.

Between the DeepSeek custom license and the Llama license, we could go down a whole rabbit hole. I think we’ll make sure we discuss the license rabbit hole before getting into the specifics.

One implication of DeepSeek is that it puts pressure on Llama and everybody else, including OpenAI, to push toward open source. The other side of open source is how much is published in detail about the model. How open are you with the insights behind the code? How good are the technical reports? Are they hand-wavy, or are there actual details?

That’s one of the things DeepSeek did well: it published a lot of the details.

Especially in the DeepSeek V3 pre-training paper, they were very clear that they were making iterations on the technical stack at many different levels. For example, to get highly efficient training, they made modifications at or below the CUDA layer for NVIDIA chips.

I have never worked there myself, and there are only a few people in the world who do that very well. Some of them are at DeepSeek, and some of them are at leading American frontier labs. There aren’t many places that help people understand this level of implementation.

There’s another implication of open weights that we’ll return to often. There’s a fear that China, the nation, might be interested in stealing American data and violating the privacy of American citizens. What can we say about open weights to help us understand what the weights are able to do?

In terms of stealing people’s data, the weights that you can download from Hugging Face or other platforms are very large matrices of numbers. You can download them to a computer in your house that has no internet connection, run the model, and have complete control of your data.

That is different from how a lot of language-model usage is done today, which is mostly through APIs. You send your prompt to GPUs run by certain companies, and those companies have different policies about how your data is stored, whether it is used to train future models, where it is stored, whether it is encrypted, and so on.

With open weights, you have your data in your own hands. That is deeply connected to the soul of open source. It’s not the model that steals your data. It’s whoever is hosting the model. That could be China if you’re using the DeepSeek app, or it could be Perplexity. You’re trusting them with your data. You could be trusting OpenAI with your data. Some of these are American companies and some are Chinese companies, but the model itself is not doing the stealing. The host is.

Let’s go back to the basics. What’s the difference between DeepSeek V3 and DeepSeek R1? Can we lay out the potential confusion?

I completely understand why people are confused by these two model names. The best way to think about it is that when you’re training a language model, you have what is called pre-training. That’s when you’re predicting large amounts of mostly internet text. You’re trying to predict the next token.

These new DeepSeek models do this large-scale internet pre-training once to get what is called DeepSeek V3 Base. This is a base model. It’s going to complete your sentences for you, and it’s harder to work with than ChatGPT.

DeepSeek then used two different post-training regimes to make the models exhibit specific desirable behaviors. The more normal model, in terms of the last few years of AI, is an instruction model, a chat model, an aligned model, or a helpful model. There are many ways to describe it. That involves standard post-training, such as instruction tuning and reinforcement learning from human feedback.

That is what they did to create DeepSeek V3. It was the first model released, and it is highly performant. It’s competitive with GPT-4 and Llama 3 405B. At around the same time, or soon afterward, they were finishing a different training process from the same next-token-prediction base model. That is where the new reasoning training comes in, in order to create the model called DeepSeek R1.

The “R” throughout this conversation is useful for grounding around reasoning. The name is also similar to OpenAI o1, which is another reasoning model people have heard about. We need to break down the training for R1 in more detail because we have a paper detailing it, and because it is a far newer set of techniques for the AI community. It’s a rapidly evolving area of research.

We should also explain the big two categories of training: pre-training and post-training. What is pre-training, what is post-training, and what are the different flavors of things under the post-training umbrella?

Pre-training is autoregressive prediction: predicting the next token in a series of documents. Standard practice involves trillions of tokens, so this is a tremendous amount of data, mostly scraped from the web.

In some of DeepSeek’s earlier papers, they discuss training data distilled for math. They also mention Common Crawl, which is publicly accessible. Anyone listening to this could download data from the Common Crawl website. It’s a crawler maintained publicly. Other technology companies eventually shift to their own crawlers, and DeepSeek likely has done this as well, as most frontier labs do, but this is the sort of data people can get started with.

You’re simply predicting text in a series of documents. This can be scaled to be very efficient. There are many numbers thrown around in AI training, such as how many floating-point operations, or FLOPs, are used. You can also look at how many hours of GPU usage are required. It’s largely one loss function applied to a very large amount of compute.

You set up highly efficient systems, and at the end you have a base model. Pre-training is where there is more complexity in terms of how the process is emerging or evolving and the different types of training losses we use.

The oldest technique, which is still used today, is called instruction tuning, also known as supervised fine-tuning. The acronyms are IT and SFT, and people go back and forth between them. In this stage, you add formatting to the model so it knows how to take a question such as, “Explain the history of the Roman Empire to me,” or a question you might see on Reddit or Stack Overflow, and respond in an information-dense but presentable manner.

The core of that formatting comes from instruction tuning. Then there are two other categories of loss functions being used today. One is preference fine-tuning, which is a generalized term for what came out of reinforcement learning from human feedback, or RLHF.

RLHF is credited as the technique that helped ChatGPT break through. It makes responses that are nicely formatted, like Reddit answers, more aligned with what a human would like to read. Initially, this is done by collecting pairwise preferences from actual humans. Now, AIs are also labeling this data, and we’ll get into those trade-offs.

You create a contrastive loss function between a good answer and a bad answer, and the model learns to pick up those trends. There are different implementation methods. You can use reward models, direct-alignment algorithms, and many other specific techniques, but all of this is about fine-tuning to human preferences.

The final stage is much newer, and it connects to what is done in R1 and these reasoning models. I think OpenAI’s name for this is reinforcement fine-tuning. They had a new API in the fall called the Reinforcement Fine-Tuning API.

The idea is that you use the techniques of reinforcement learning, which is an entire framework in AI. To summarize it, reinforcement learning is often known as trial-and-error learning, or the subfield of AI where you’re trying to make sequential decisions in a potentially noisy environment.

There are many ways we could go down that path, but the language-model version is that the model generates an answer and you check whether it matches the true solution. In math, you have an exactly correct answer. In code, you can have unit tests. You check the model’s work, give it multiple opportunities on the same questions, and see whether it is correct.

If you keep doing this, models can learn to improve in verifiable domains. It works really well to a great extent. It’s a newer technique in the academic literature, although it has been used at frontier labs in the United States that don’t share every detail for multiple years.

This is the idea of using reinforcement learning with language models, and it has been taking off, especially in this DeepSeek moment. There’s exciting work across the stack, but post-training will probably have a lot of interesting developments this year.

I almost forgot to ask about the difference between DeepSeek V3 and R1 from the user-experience side. Forget the technical stuff. For people who don’t know anything about AI, they show up and ask, “What’s the actual experience? What’s the use case for each one when they type and talk to it? What is each one good at?”

Let’s start with DeepSeek V3. It’s what more people would have tried. You ask it a question, and it starts generating tokens very quickly. Those tokens look like a human-readable answer. It might be a Markdown list, with formatting that draws your attention to the core details.

It generates tens to hundreds of tokens. A token is normally a word for common words, or a subword part of a longer word. The output looks like a high-quality Reddit or Stack Overflow answer.

These models are getting very good at doing this across a wide variety of domains. Even if you’re an expert, they can be fairly good at things close to the frontier of knowledge. They’re capable study aids for cutting-edge AI topics, and they’re regularly updated.

DeepSeek R1 is different. With these reasoning models, the first tokens you see are a large chain-of-thought process. We’ll return to chain of thought in a moment. It looks like a lot of tokens, with the model explaining the problem.

The model will often break down the problem and say, “They asked me for this. Let’s break down the problem. I’m going to need to do this.” You see all of that generated by the model. It comes very quickly in most user experiences. These APIs are very fast, so you’ll see many tokens and words appear rapidly. The reasoning process keeps flowing across the screen.

Eventually, the model changes its tone in R1 and writes the answer. It summarizes the reasoning process and writes an answer similar to the first type of model. In DeepSeek’s case, this was part of why the model became so popular even outside the AI community: you can see how the language model breaks down problems.

Technically, they trained the model to do this. There’s a section for reasoning, and then it generates a special token—probably hidden from the user most of the time—that says, “I’m starting the answer.” The model is trained to perform this two-stage process on its own.

If you use a similar model such as OpenAI’s, the user interface summarizes the process for you. It shows sections such as “breaking down the problem,” “making calculations,” and “cleaning the result.” Then the answer appears.

Maybe it would be useful to go through an example of DeepSeek R1 reasoning.

If you’re looking at the screen, you’ll see a screenshot of the DeepSeek chat app. At the top it says, “Thought for 151.7 seconds,” with a drop-down arrow underneath. If we were in an app that exposed it, the drop-down would contain the reasoning.

The specific question was: “Give one truly novel insight about humans.” I’m philosophically inclined, so this is a question I enjoy. The novel aspect was pushing the model to constantly ask itself, “Is this truly novel?” It challenged itself to be more novel, more counterintuitive, and less cringe, I suppose.

Some of the reasoning says that humans have a unique meta-emotion in which they feel emotions about their own emotions—feeling guilty about being angry, for example. This recursive emotional layering creates complex motivational drives that don’t exist in other animals. The insight is that human emotions are nested.

The model is reasoning through how humans feel emotions. It’s reasoning about meta-emotions, and it goes on for pages and pages. It’s almost too much to read, but it’s nice to skim as it comes. It’s a stream of consciousness—a James Joyce stream of consciousness.

Then it says, “The user wants something that’s not seen anywhere else. Let me dig deeper,” and considers the human ability to hold contradictory beliefs simultaneously. Cognitive dissonance is known, but perhaps the function is to allow flexible adaptation.

After 157 seconds, the final result is: “Humans instinctively convert selfish desires into cooperative systems by collectively pretending abstract rules—money, laws, rights—are real. These shared hallucinations act as games where competition is secretly redirected to benefit the group, turning conflict into society’s fuel.”

That’s profound. It captures the public imagination. It’s not just intelligent; it’s almost an inkling of sentience, because it’s thinking through a problem, self-reflecting, and deliberating.

Some of this is just a vibe. Some of the benchmarks matter, but that response itself is a fire tweet.

If you’re trying to produce something that makes people say, “That’s chain of thought,” we’ll probably return to this.

How are they able to achieve such low costs in training and inference? Maybe you could talk about training first.

There are two main techniques that account for most of their efficiency, and then many implementation details that contribute as well. The first is that they use a mixture-of-experts model. The second is that they invented a new technique called MLA, or multi-head latent attention.

Both are significant. Mixture of experts is something that has been in the literature for a handful of years. OpenAI, with GPT-4, was the first to productize a mixture-of-experts model.

The common models most people have interacted with are open models. Llama, for example, is a dense model, meaning every parameter, or neuron, is activated as you move through the model for every generated token.

A mixture-of-experts model does not do that. Consider how a human works. When I’m thinking about a visual task, my visual cortex is active. When I’m scared, my amygdala is active. Different aspects of the brain focus on different things.

A mixture-of-experts model tries to approximate this to some extent. It is nowhere close to a brain architecture, but different portions of the model activate for different tasks. You have a set number of experts in the model and a set number that are activated each time.

This dramatically reduces training and inference costs. If you think of the parameter count as the total embedding space for the knowledge compressed during training, a mixture-of-experts model can grow the total parameter space without activating every parameter each time.

DeepSeek’s model has more than 600 billion parameters, compared with Llama 405B’s 405 billion parameters and Llama 70B’s 70 billion. Technically, DeepSeek has more embedding space for information and for compressing the world’s internet knowledge, but it activates only around 37 billion parameters at a time.

Only those 37 billion parameters need to be computed each time you train on data or run inference. By comparison, all 70 billion parameters in Llama 70B, or all 405 billion in Llama 405B, must be activated. Mixture of experts dramatically reduces compute costs for training and inference.

Should we break down where this applies and go into the Transformer? Is that useful?

Let’s go into the Transformer.

The Transformer is discussed often, and we won’t cover every detail. Essentially, it’s built from repeated blocks of an attention mechanism and a traditional dense, fully connected multilayer perceptron—whatever term you want to use for the normal neural network. You alternate these blocks.

Mixture of experts is applied to the dense model. The dense model holds most of the weights in a Transformer, so you can get significant gains in parameter efficiency during training and inference by not activating all of those parameters.

A Transformer is a giant neural network. For about 15 years there has been what’s called the deep-learning revolution, where networks have gotten larger and larger. At a certain point, scaling laws appeared, and people realized that bigger is better across multiple dimensions of what “bigger” means.

These are all neural networks, and we’re talking about different architectures for constructing them so that training and inference are efficient.

Every different type of model has a different scaling law. Effectively, for the amount of compute you put in, the architecture will reach different levels of performance on test tasks.

A mixture-of-experts model is one of the architectures where, even if you ignore the inference benefits, training efficiency with GPUs can improve dramatically when it’s well implemented. You can get effectively the same performance on evaluation scores with perhaps 30% less compute, although there will be a wide variation based on implementation details.

It’s important to realize that this kind of technical innovation can provide huge gains. I expect most companies serving models to move toward mixture-of-experts implementations. Historically, not everyone used them because of the implementation complexity, especially for large models.

This is one thing DeepSeek deserves credit for: it does mixture of experts extremely well. The architecture for what is called DeepSeek MoE is based on multiple older papers, so this part of its training infrastructure is not entirely new to these models.

The same goes for the multi-head latent attention that Dylan mentioned. It reduces memory usage during inference and training through a clever low-rank approximation.

When you get into the details, there are other parts of language models, such as embeddings, that are used to extend context length. The common technique DeepSeek used is rotary positional embeddings, called RoPE.

If you want to use RoPE with a normal MoE model, you take two of the attention matrices and rotate them by a complex-valued rotation, which is a matrix multiplication. With DeepSeek’s MLA architecture, they need to do clever things because the model is not set up the same way. That makes the implementation much more complex.

They’re managing all of these things, and these are probably the kinds of techniques the closed labs are using as well. We don’t know whether they use exactly the same techniques, but DeepSeek shared them with the world, which is valuable. This is the cutting edge of efficient language-model training, and some of it requires low-level engineering. It’s a giant mess of clever tricks.

As I understand it, they went below CUDA. They do extremely low-level GPU programming.

NVIDIA builds a library called NCCL, which stands for NVIDIA Collective Communications Library. When you’re training a model, there are communications between every layer of the model, and you may have more than 100 layers.

You have all-reduce and all-gather operations between each layer, between the multilayer perceptron or feed-forward network and the attention mechanism. The model is synchronized, and those operations communicate between all the GPUs in the network, whether during training or inference.

NVIDIA has a standard library for this. That is one reason it’s difficult to use anyone else’s hardware for training: nobody else has built a standard communications library at the same level.

NVIDIA has done this at a higher level. Because DeepSeek had limitations around the GPUs and interconnects it could access—limitations caused by the GPUs legally shipped into China, not the ones that were smuggled in—it had to find efficiencies.

Instead of simply calling NVIDIA’s NCCL library, DeepSeek created its own communication scheduling. Meta discussed making a custom version of NCCL for Llama 3, although it did not discuss the implementation details.

DeepSeek had to do this because of its hardware constraints. It may have done it even better than some of the American labs because necessity is the mother of invention. It scheduled communications by specifically scheduling the streaming multiprocessors, or SMs, on the GPU.

You can think of an SM as a core on a GPU. There are hundreds of cores, or a little over 100 SMs, on a GPU. DeepSeek specifically scheduled which SMs would run the model and which ones would perform all-reduce and all-gather operations. They would switch back and forth between them.

This requires extremely low-level programming. That is what NCCL or other NVIDIA libraries generally handle automatically.

Technically, they’re using PTX, which is a sort of assembly-like language. It isn’t exactly assembly or an instruction-set language, but it’s still part of CUDA.

You can choose whether to write in Python or a PyTorch equivalent and call NVIDIA libraries, go down to the CUDA level, go even lower, or go all the way down to assembly or the instruction-set level. The largest labs sometimes do that, but most companies do not because the efficiency gains generally aren’t worth the time.

DeepSeek’s implementation is particularly complex because of its mixture-of-experts model. People have implemented mixture-of-experts models before, but they generally use 8 or 16 experts.

One term we like is the sparsity factor. You might activate one-fourth of the model, such as 2 out of 8 experts. DeepSeek’s model has a much higher sparsity factor. It isn’t activating 2 out of 8 experts; it’s activating 8 out of 256.

There are different implementations of mixture of experts. Some experts may always be activated, which resembles a small neural network through which all tokens pass. The tokens also pass through experts selected by a routing mechanism.

One innovation in DeepSeek’s architecture is that it changes the routing mechanism. During training, you want to make sure that all experts are used across the tasks the model sees. A failure mode for mixture-of-experts models is that the model learns to use only a subset of the experts.

In the literature there is something called an auxiliary loss, which helps balance the experts. But if you think about the loss functions of deep learning, and connect this to the bitter lesson, you want the minimum inductive bias in your model so it can learn as much as possible.

The auxiliary loss, which balances usage across experts, can be in tension with prediction accuracy. We don’t know the exact extent of the change DeepSeek made, but instead of using an auxiliary loss, it has an extra parameter in the routing mechanism. After each batch, it updates that parameter so that subsequent batches have a similar distribution of expert usage.

This kind of change can be large or small, but these changes accumulate. It points to DeepSeek innovating in the same way that all the large labs are probably doing. You keep accumulating gains, and we’ll talk about the philosophy of training and organizing these companies.

A lot of it is compounding small improvements over time—in your data, your architecture, your post-training, and how those parts integrate. DeepSeek does the same thing. Some of its ideas are shared, and we have to take it at face value that it has shared its most important details. The architecture and weights are available, so we can see what it’s doing.

Going back to the efficiency and complexity point, it’s 32 versus 4 for the sparsity ratio, compared with Mixtral and other publicly released models. That ratio is extremely high.

When you have such a different level of sparsity, you can’t have every GPU hold the entire model. The model is too large and too complex. You have to split it up using different kinds of parallelism. You might have different experts on different GPU nodes.

But what happens when a batch of data all looks similar and should route to one part of the model? You can overload a particular set of GPU resources while the rest of the training network sits idle. That is one of the biggest complexities of running a very sparse mixture-of-experts model.

How do you load-balance between experts? How do you schedule communications between them? This is the extremely low-level work that DeepSeek figured out publicly, perhaps second or third in the world, and possibly first in some cases.

What lesson do you take from all of this in the direction of the bitter lesson? Is this low-level optimization where a lot of the gains will come from? Or is this a short-term leap because constraints forced them to find a hack, while the biggest gains will come from high-level algorithmic ideas such as post-training?

We should summarize what the bitter lesson is. If you paraphrase it, the methods that win out in deep learning are the ones that scale in learning and search.

The word “scale” gets a lot of attention, but my interpretation is that you should avoid adding human priors to the learning process. If you read the original essay, that is what it discusses: researchers come up with clever solutions to a specific problem that provide small gains in the short term, while simply enabling deep-learning systems to work efficiently on larger problems in the long term may be more likely to drive continued success.

We were talking about relatively small implementation changes to the mixture-of-experts model. It will take a few more years to know whether any one of them was crucial to the bitter lesson. The bitter lesson is the long-term arc in which simplicity can often win.

There are many sayings in the industry, such as “the models just want to learn.” You need to give them a simple loss landscape, put compute through the model, and get barriers out of the way. That’s where the power of something like NCCL comes in: standardized code that can be used by many people to create scalable innovations.

I imagine DeepSeek’s codebase is a giant mess. DeepSeek definitely has codebases that are extremely messy when they’re testing new ideas. Multi-head latent attention probably started in something like a Jupyter notebook, where somebody tried it on a few GPUs. That kind of research code is messy.

But the code that trains DeepSeek V3 and R1 is probably extremely high-quality and readable. If you presented those libraries to us, I would guess they’d be very well engineered.

There is one aspect to note, though. There’s a general ability for improvements to transfer across different runs. You may write very high-quality code for one specific model architecture at one size, but when you make an architectural change, everything may break again.

Their low-level scheduling of SMs may be specific to this model architecture and size, whereas NVIDIA’s collective-communications library is more general. If you want to perform an all-reduce operation, it doesn’t care what your model architecture is. You give up performance when you generalize like that, but it’s worthwhile in many cases.

DeepSeek found it worthwhile to make a specific optimization for a specific run, given the constraints around its compute.

I wonder how stressful it is to initiate training for one of these frontier models. You have the code, push a button, and you’re now spending a large amount of money and time. There must be a lot of innovation in debugging, making sure there are no issues, monitoring everything, and visualizing every aspect of the training.

When people are training, they have various dashboards, but the simplest one is the loss. It should continue to go down. With more complicated architectures, especially lower-precision training, one of the biggest problems is loss spikes.

Sometimes you know why the loss spike happened, and sometimes you don’t. An example from AI2 is a subreddit called Microwave Gang. It’s a real subreddit where people make posts consisting only of the letter “M.” The posts contain extremely long sequences of M’s, and the comments say “beep beep,” because that’s when the microwave ends.

If you pass that into a model trained to produce normal text, it has extremely high loss. Normally, when you see an M, you don’t predict M for a long time. That kind of data causes a loss spike. This is an old example, and mature data systems generally prevent that sort of problem.

There are levels to this. People at these labs will go out to dinner and look at their phones every 10 minutes. They aren’t necessarily texting. They’re checking whether the loss or the tokens-per-second metric has blown up. Their heart rate goes up when there’s a spike.

Some spikes are normal. The loss will recover and return to normal. The old strategy was to stop the run, restart from an earlier checkpoint, change the data mix, and continue.

There are different kinds of spikes. Dirk Groeneveld has a theory about fast spikes and slow spikes. Sometimes the loss and other parameters begin to creep upward and then blow up. That is hard to recover from, so you have to go back much further.

Other spikes appear as a single bad data point. You see the spike, ignore that data, don’t update the model, and move to the next point. The model recovers quickly.

As implementations become more complex and you scale across more GPUs, there are more opportunities for the loss to blow up. It becomes a stressful task. The whole time, the dollar count is rising.

The idea of grokking also comes in. Just because improvement in the loss has slowed doesn’t mean the model has stopped learning. It can suddenly make a large improvement because it learned something, and it took time for that to happen.

That’s how humans are, and it’s how models can be. It’s a stressful task.

How do you get to the point where DeepSeek finds a successful combination of hyperparameters? Is it a lot of small failed runs, with rapid iteration through failures and successes, until you build a situation where the mixture of experts works and the MLA implementation works?

You find the key hyperparameters—learning rate, regularization, and so on—and find the regime that works for your codebase. I’ve spoken with people at frontier labs, and there’s a story you can tell where training language models is a path you need to follow.

You need to unlock the ability to train a certain type or scale of model. Then your codebase and internal knowledge of which parameters work become established. Looking at the DeepSeek papers and models, they’ve scaled up, added complexity, and continued building their capabilities.

There’s a concept called a YOLO run. YOLO means “you only live once.” There’s experimentation at small scale: your Jupyter notebook, experiments with MLA on a few GPUs, and different research ablations.

You test four active experts, 16 active experts, 128 experts, and different ways of arranging them. You run these tests with a few researchers and a few, tens, or hundreds of GPUs.

Then suddenly you say, “No more experiments. Everyone take all the resources we have. Let’s choose what we think will work and go for it.” That’s the YOLO run.

The stress comes from knowing that something works at one scale, but things that work at a small scale may not work at a large scale. There are researchers who methodically explore the whole search space and identify the best approach, and there are researchers who have an innate gut instinct that this is the YOLO run.

This is why you want to work in post-training: the GPU cost is lower, so you can make a higher percentage of your training runs YOLO runs.

For now.

For now.

Some of this is fundamentally luck.

Luck is skill in many cases.

It looks like luck when you’re watching it, but if you’re at one of these labs and the evaluation isn’t improving, there’s a repeated playbook for how to make things better. There are localized improvements, such as data improvements, and these add up to a much better model.

When you zoom in, it can be obvious that the model is bad at one thing and that you can fix it. You add those fixes up. Some of it feels like luck, but on the ground, especially with the new reasoning models, there are many ways to probe the system.

The search space is nearly infinite, and the amount of compute time you have is very limited. You also have release schedules. You can’t get passed by everyone else.

DeepSeek may have moved faster than Meta, Mistral, Cohere, and others. Maybe those companies moved too slowly or were too methodical. I don’t know if they missed the YOLO run, but whatever the reason, you can call it luck if you want. At the end of the day, it’s skill.

So 2025 is the year of the YOLO run. All the labs are going in.

I think what OpenAI did in 2022 is even more impressive. At the time, nobody believed in mixture-of-experts models. Google had many of the researchers, and OpenAI had relatively little compute.

OpenAI devoted all of its compute for many months—100% of it—to GPT-4 with a brand-new architecture, without knowing whether it would work. They spent the few hundred million dollars they had on that model. That is truly YOLO.

Today, when people discuss training-run failures in the media, a large portion of GPU capacity is still doing inference. There’s also research happening constantly. The largest cluster may be running the YOLO training run, but that run is much less risky than what OpenAI did in 2022, or perhaps what DeepSeek did recently.

The big winners throughout human history are willing to go YOLO at some point.

What do we understand about the hardware DeepSeek was trained on?

DeepSeek is interesting, and it helps to zoom out and look at who they are. High-Flyer is a hedge fund that has historically done quantitative trading in China and elsewhere. It has always had a significant number of GPUs.

In the past, many high-frequency trading and algorithmic quantitative traders used FPGAs, but that shifted toward GPUs. High-Flyer, the hedge fund that owns DeepSeek, and DeepSeek have the same parent company, owner, and CEO. Everyone who works for DeepSeek is part of High-Flyer to some extent.

High-Flyer had resources and infrastructure for trading and devoted a huge portion of them to training models, both language models and others. These techniques were heavily influenced by AI. More recently, people have realized that natural-language processing is crucial for trading quickly—for example, understanding a press release and making the right trade.

DeepSeek has been good at this for a long time. As far back as 2021, it had press releases and papers saying it was the first company in China with an A100 cluster that large: 10,000 A100 GPUs.

That cluster was not all for training large language models. Much of it was for quantitative-trading models and natural-language processing. But the verifiable fact is that in 2021 they built what they claimed was the largest GPU cluster in China, with 10,000 GPUs, before export controls started.

That means they had a huge cluster before there was any conversation about export controls.

Then you move forward four years. They continued operating the hedge fund, probably made a lot of money, and leaned more and more into AI. The CEO, Liang Wenfeng, is an extremely involved figure. He’s similar to Elon Musk or Jensen Huang in the sense that he’s involved in everything.

He has an almost AGI-oriented perspective. He has said that China needs to build a new ecosystem around AI and that China should lead this ecosystem because Western countries have historically led software ecosystems.

He has directly acknowledged that China needs to do something different to build this. DeepSeek is his way of doing it. He has said that they will not switch to closed source when discussing these issues.

He’s a very long-term-oriented person with a vision for how the AI ecosystem should work. From a Chinese perspective, he wants a Chinese company to build that vision. He is the visionary behind the company.

The hedge fund still exists, and DeepSeek has produced multiple models since then. It has acquired more and more GPUs, and it shares infrastructure with the fund. There is no exact public number for its current GPU resources, but we know about the 10,000 GPUs purchased in 2021 and the 2,000 H800 GPUs mentioned in the DeepSeek paper.

The H800 was a restricted GPU previously allowed in China, but it is no longer allowed. It was essentially NVIDIA’s H100 for China, with restrictions on communication speed. That is why DeepSeek had to develop its unusual scheduling techniques.

Do you think 2,000 is the correct number for the training run?

This requires defining what you count as the training run. Do you count all the research and ablations? You can do a YOLO run, but before that you need to test things at small and medium scale.

Accepted practice is that, for any notable model advancement, you may use 2–4 times the compute of the full training run on experiments alone. A lot of the compute being scaled up is used for research. Research produces the ideas that provide major efficiency gains, and then you bet on those ideas.

The numbers DeepSeek publicly stated are 10,000 GPUs in 2021 and 2,000 GPUs for the pre-training of V3. It did not discuss the cost of R1, the other reinforcement-learning runs, the instruction model, research, or ablations. It also did not discuss the resources shared with the hedge fund.

We believe DeepSeek has closer to 50,000 GPUs in total. That’s split across many tasks, including the fund, research, ablations, and training.

Meta has publicly discussed having on the order of 60,000 to 100,000 H100-equivalent GPUs in its training clusters. For Llama 3, Meta said it trained on 16,000 H100s, but Meta publicly disclosed buying more than 400,000 GPUs last year.

Of course, only a tiny percentage of those were used for training. Most are used to serve Instagram Reels or other products.

DeepSeek’s compute allocation is among the top few in the world. It’s not OpenAI or Anthropic, but it has a significant amount of compute.

Can you zoom out and explain the Hopper architecture, and the difference between the H100 and H800?

Ampere was the A100, and Hopper is the H100. People in the United States use those names synonymously because there is mainly the H100 and now H200. In China, there have been several rounds of export restrictions.

Initially, the US government limited chips on a two-factor scale: chip interconnect and FLOPs. Any chip with interconnect above a certain level and floating-point performance above a certain level was restricted.

Later, the government recognized that this was a flaw and reduced the restriction to floating-point performance alone. The H800 had the same FLOPs as an H100 but reduced interconnect bandwidth.

DeepSeek learned how to use it efficiently. Even though the interconnect was limited, it found ways to use the GPU fully. That was the situation in October 2022.

At the end of 2023, implemented in 2024, the US government banned the H800. DeepSeek’s 2,000-GPU H800 cluster was purchased in late 2022, not in 2024. It has taken time to do the research and release the model.

The new chip is called the H20. Its FLOPs are reduced, but its interconnect bandwidth is the same as the H100. In some ways, its memory bandwidth and memory capacity are better. NVIDIA is working within the government’s constraints to build the best possible GPU for China.

Let’s take a tangent and then return to the hardware. What is the philosophy and motivation behind export controls?

Dario Amodei recently published a blog post about them. His argument is that if AI becomes extremely powerful—he says we could have AGI or super-powerful AI by 2026—then whoever builds it will have a significant military advantage.

Because the United States is a democracy and China is authoritarian, or has authoritarian elements, he argues that we want a unipolar world in which the superpower with the most powerful military, because of AI, is a democracy. It becomes a much more complicated geopolitical world if two superpowers have extremely powerful AI and one is authoritarian.

The idea is that the United States should use export controls to slow China down and prevent it from making the gigantic training runs presumably required to build AGI.

That’s one abstract description of the goal. You touched on the training-run idea, but there aren’t many worlds in which China cannot train AI models. Export controls can cap the amount or density of compute China can have.

If you think about the AI ecosystem, AI-company revenue is going up and to the right, and AI usage continues to grow. More GPUs will be used for inference. A large part of export controls, if they work, is simply that the amount of AI that can be run in China will be much lower.

On the training side, DeepSeek V3 is a great example. A focused team can still reach the frontier of AI with 2,000 GPUs. That is not hard to acquire by global standards. China will still have those GPUs and still be able to train models.

But if there is a huge market for AI, and export controls are strong, China may not be able to have 100,000 GPUs serving ChatGPT-like customers. That is a much easier goal to achieve than debating what AGI is or whether autonomous AI systems exist.

If you have extremely intelligent, autonomous AI systems operating in data centers, those might run on GPU clusters in the United States but not in China.

Training a model by itself does almost nothing. What Dario is discussing is the implementation of that trained model to create economic growth, increase military capabilities, increase productivity, or improve people’s lives. Whatever you direct super-powerful AI toward, it can potentially do, but that requires significant compute.

Training will always be a portion of total compute. Meta has 400,000 GPUs and used only 16,000 for Llama 3. The percentage Meta dedicates to inference may be for recommendation systems that try to keep us watching more ads, or it may be for a super-powerful AI doing productive things.

The exact use doesn’t matter. The capabilities can be delivered in whatever way our economic system decides. With China, export restrictions cannot cut everything off. The US government understands that. China will make its own chips. They will be worse than American chips, but the point is to maintain a gap.

If super-powerful AI arrives and begins making significant changes to society, the difference in compute will compound. That’s why the restrictions matter.

There is science fiction in which AI is measured by the amount of power delivered to compute. That’s one way of thinking about economic output: how much power you direct toward an AI system.

Should we talk about reasoning models as something people can actually see? The reasoning models coming out with R1 and o1 are designed to use more compute. There are many buzzwords—test-time compute, inference-time compute—but can you explain the implications?

These reasoning models make inference much more important for complex tasks. In December, OpenAI announced the o3 model. In a fast-moving field, we have both announcements and releases. Announcements are blog posts where companies say what they did; releases are when the models and papers are actually available.

The breakthrough result for o3 involved the ARC-AGI task, the Abstraction and Reasoning Corpus for artificial general intelligence. François Chollet has worked on it for years. It’s a brilliant benchmark.

For o3, the model used a number of samples through the API, with settings for thinking effort and the number of samples. It used 1,000 samples to solve the task, and the cost came out to roughly $5–$20 per question.

You’re effectively giving the model a math puzzle, and it takes dollars of compute to answer one question. That is a tremendous amount of inference. If this takes off in the United States, OpenAI needs a huge number of GPUs for inference.

OpenAI has a ChatGPT Pro subscription that costs $200 per month. Sam Altman has said they’re losing money on it, which means users are consuming a lot of inference compute. I’ve signed up and used it. I’m not a power user, but the system is expensive to run.

A Chinese company with moderately strong export controls—and there will always be loopholes—might not be able to do this at scale. The other major result for o3 is spectacular coding performance. That could feed back into AI companies experimenting more effectively.

The long-term idea is that an AGI could use a large amount of test-time compute. You give it a task, and it goes into a room to think about how to take over the world, then comes back 2.7 hours later. That requires a lot of compute.

This is what leaders at OpenAI and Anthropic discuss when they talk about autonomous AI models: you give them a task, and they work on it in the background.

My personal definition of AGI is simpler. I think language models are already a form of AGI. All of this super-powerful stuff is a next step. It would be great if we had these tools, but language models have enormous value across many domains. They are a form of general intelligence to me.

The next step is agentic behavior: systems that are independent and can do tasks that were not in the training data. That is the future these companies are working toward.

Dario’s terminology is “super-powerful AI.” I agree with you about AGI. We already have something extraordinarily impressive that Alan Turing would probably call AGI. Dario is referring to something that, once in someone’s possession, would provide a significant military and geopolitical advantage over other nations.

It’s not just that you can ask it how to cook an omelet. In his essay, Machines of Loving Grace, he has a much more positive view. He argues that AI can revolutionize biology. I don’t have enough background in the physical sciences to judge how confident I am in that, but I’m comfortable saying that AI will accelerate progress in computational science.

Let’s continue this depth-first search. You both said you’re feeling AGI. What’s your timeline? Dario says 2026 for super-powerful AI that is agentic enough to become a real security threat. What’s your timeline?

I don’t like attributing specific abilities to a date because predicting a particular capability is very hard. When I say I’m feeling AGI, I mean that I expect continued, rapid, surprising progress over the next few years.

R1 is less surprising to me because I expect new paradigms in which substantial progress can be made. DeepSeek R1 is unsettling because we were on a path with ChatGPT: it kept getting better and better, and then a new direction changed the models. We took one step and suddenly moved upward.

The slope looks very steep, and we may continue taking steps like that. It’s unsettling when you see those large jumps. I expect that to keep happening.

I’ve tried OpenAI Operator and Claude’s computer-use capability. They aren’t there yet. I understand the idea, but it’s hard to predict what breakthrough will make something like that work.

I think it’s more likely that we’ll have breakthroughs that produce capabilities we don’t yet know what to do with. Everyone wants agents, and Dario has an eloquent way of describing them, but I expect there will be more than that. We should expect these things to arrive.

I’m going to try to pin you down to a date on the AGI timeline—the nuclear-weapon moment, when there’s a real geopolitical shift. We’re talking about export controls. When do you think that will happen?

For me, probably after 2030.

Define that, because to me it has almost already happened. Look at elections in India and Pakistan, where people receive AI-generated voice calls and think they’re speaking to a politician.

The AI Diffusion Framework, enacted in the last couple of weeks of the Biden administration, looks like the Trump administration will keep it and potentially strengthen it. It limits cloud computing and GPU sales to countries that aren’t even related to China.

Portugal and Singapore are on the list of countries that need US approval. Singapore has F-35s, but the United States doesn’t let it freely buy GPUs. To me, that’s already a geopolitical reaction to the technology.

That could simply mean that the US military is nervous about a new technology it doesn’t understand. It doesn’t mean the technology is already at that level.

The robocalls, swarms of semi-intelligent bots, and social engineering could be weapons. There’s extensive discussion of the 2016 elections, Cambridge Analytica, Russian influence, and so on.

Every country pushes its viewpoints onto the internet. Russia, China, the United States, Israel, and others are putting their preferred narratives online. Language models reduce the cost of producing intelligent-sounding language.

Some research suggests that the distribution of information is the limiting factor. Language models haven’t yet caused an exponential increase in measured misinformation, at least in the things researchers have been able to measure.

There’s a blog called AI Snake Oil, written by some of my friends at Princeton, that discusses this. It’s a default assumption that misinformation will become dramatically worse with language models, and I would have expected that too, but so far we haven’t seen an exponential increase or something that is extremely measurable.

With voice calls and other modalities, it could be happening in ways that are harder to measure. It’s too soon to tell. Political instability via the web is monitored by many researchers, and we’ll see what happens.

If you make me give a year, I’d say that AI CEOs have been saying “two years” for a while. People like Dario, who have thought about this deeply, deserve to be taken seriously, but they also have different incentives. I would add a few years to their estimates, which gets you somewhere around 2030 or a little later.

To some extent, capabilities may arrive at a point where one person could say, “If I can leverage this for enough time, this is AGI.” Call it 2027 or 2028. But the cost of operating that capability could be so extreme that nobody can deploy it at scale and revolutionize the economy overnight.

It won’t be a snap-of-the-fingers moment because of physical constraints. The capabilities may exist, but you may not be able to deploy them everywhere.

A simple example is 2023, when GPT-4 came out and everyone was worried about search. Perplexity launched, but if you calculated the cost of putting GPT-4 into every Google search, it was physically impossible.

Now consider test-time compute. A normal ChatGPT query costs cents for the most capable chat model. Solving an ARC-AGI problem costs $5–$20. That’s a 1,000- or 10,000-fold difference between answering a query and doing a task.

The tasks required for AGI will cost thousands, tens of thousands, or hundreds of thousands of dollars in GPU time. There won’t be enough power, GPUs, or infrastructure to transform the world with the snap of a finger.

But then the question becomes: who controls the system, and where do they point it?

Dario’s point is that China could point its AGI at military tasks more quickly than the United States. China has been faster at adopting certain technologies into its military, especially drones. The United States may have a long-standing advantage in fighter jets and bombers, but China has leapfrogged the US and the West in asymmetric systems such as drones.

The fear is that the United States will have AGI in the commercial sector, while the US military will not be able to implement it quickly. The Chinese military could direct its resources toward military logistics, disinformation, or targeted political operations.

People’s intuition often fails with robotics. There’s general optimism about self-driving cars and drones, but humans still outperform fully autonomous systems in many military contexts.

In Ukraine, humans controlling FPV drones far outperform current AI systems. AI is an assistant, but humans still control most of the process. I don’t think it’s obvious that we’ll have swarms of autonomous robots anytime soon.

The fastest I can imagine is 2030, which is why I gave that date for super-powerful AI. When large-scale swarms of robots carry out military actions, that’s when the world begins to look fundamentally different to me.

Cyberwarfare could arrive sooner. It could involve social engineering, swarms of robots that find attack vectors in codebases, or attacks that shut down power grids. One weekend, the power could go out, nobody knows why, and the world changes forever.

Two days without power across the United States could lead to murder and chaos.

Returning to export controls, do you see them as a useful way to control the balance of power geopolitically?

If you believe we’re in the same kind of economic-growth environment we’ve had for the last 20 years, export controls guarantee that China will win in the long term.

If you don’t believe AI will make significant changes to society in the next 5 or 10 years, then restricting high technology and making money from it is economically foolish. But AI-company executives and major technology companies believe AI will make massive changes in that timeframe.

Once you get to a shorter timeline, the only way to create a major advantage or disadvantage for America relative to China is to constrain compute. Talent is not the main constraint. China arguably has more talent: more STEM graduates and more programmers. The United States can draw on people from around the world, and many AI teams consist of people without US passports, including many Chinese people who moved to America.

Talent is not a measurable advantage for the US. The question is compute.

China has an unprecedented ability to build enormous amounts of power. It has steel mills individually the size of the entire US steel industry and aluminum mills consuming gigawatts of power.

OpenAI’s Stargate project, once fully built, is supposed to use 2 gigawatts of power. That is still smaller than the largest industrial facilities in China. If China had access to the chips, it could build the largest data center in the world.

So China’s industrial capacity far exceeds that of the United States?

Exactly, in manufacturing and power infrastructure. The gating factor for building the largest clusters in the United States is power. That includes power generation, power transmission, substations, transformers, and data-center construction.

All of those constrain the ability to build larger training systems and deploy more inference compute.

We should explain why timing matters. Export controls make it harder for China to acquire or manufacture cutting-edge chips. If the timing is wrong, China could invest heavily in domestic chip production, build more energy capacity, and eventually produce more chips than the rest of the world.

If AI takes a long time to become differentiated, the United States will have damaged the financial performance of its own companies. NVIDIA will sell fewer chips, TSMC will sell less to China, and there will be less demand to keep driving the production cycle.

That’s the assumption behind the timing. If AI’s major effects arrive in fewer than 5 or 10 years, China may lose because of the restrictions. If they arrive later, China could win because of its industrial capacity.

If Xi Jinping decided to become “scale-pilled”—to decide that scaling laws are what matter—China could build multi-gigawatt data centers faster than the United States.

American executives such as Satya Nadella, Mark Zuckerberg, and Sundar Pichai have decided that scale is important. They’re building multi-gigawatt data centers in Texas, Louisiana, Wisconsin, and elsewhere. These projects can cost as much as a company’s entire global data-center budget in one location.

China could do this faster, but it’s not clear that China has decided at the highest levels that it is a priority. In the United States, it’s clear that the government is thinking about it. Trump talked about DeepSeek and Stargate in the same week. The Biden administration also discussed AI extensively.

DeepSeek only recently met the second-highest leader of China. It hasn’t met Xi Jinping. China announced a subsidy of 1 trillion renminbi, roughly $160 billion, which is close to the combined AI spending of Microsoft, Meta, and Google for this year.

China may be realizing the importance only now. That’s where export restrictions come in. The United States is saying that China cannot receive the most powerful American chips, cannot receive them through countries that would simply rent them to China, must limit the number of chips, and must be restricted from buying the tools required to manufacture them.

The semiconductor restrictions are explicitly about AI and military-civil fusion. You read the rules and they are very clear. Then you see restrictions on lithography, etching, deposition, and tiny subsystems from companies you may never have heard of. The reason is that the US government has decided these are critical to AI systems.

The focal point seems to be the transition from 7-nanometer to 5-nanometer chips. Huawei had a 7-nanometer chip a few years ago, which caused another political uproar. Then there’s ASML and deep ultraviolet or extreme ultraviolet lithography. Can you set the context?

In 2020, Huawei released the Ascend 910, an AI chip made on a 7-nanometer process before Google or NVIDIA did. It submitted the chip to the MLPerf benchmark, an industry-standard machine-learning performance benchmark, and it performed very well. It was the best chip in that submission.

This was a major event. The Trump administration had banned Huawei from obtaining 7-nanometer chips from TSMC in 2019, so Huawei had to switch to internally produced domestic chips. That caused a multiyear setback.

We don’t know how much Huawei subsidized production of that chip. Intel has made 7-nanometer chips that were unprofitable, so there are economic questions involved.

You’re saying that Xi Jinping has not yet felt AGI, but the DeepSeek moment might cause him to feel it.

It’s possible he woke up last week. Leon Fang met the vice chair, the second-in-command, and the next day China announced the trillion-renminbi AI subsidies.

It’s possible that the DeepSeek moment is the beginning of a cold war. People in AI have worried for a long time that this is heading toward a cold war, but several factors came together into an explosion of attention. It may have led Xi Jinping to recognize the importance of the issue.

The US government recognized the importance earlier. The October 7, 2022 export controls came before ChatGPT was released. The restrictions shocked everyone, but they were clearly aimed at AI. Generative AI was beginning to create rumblings about what it could do, and the National Security Council and others understood where the world was headed.

Is there a concern that export controls could push China toward military action against Taiwan?

That’s the major risk. The further you push China away from access to cutting-edge American and global technologies, the more likely it may be to think, “If I can’t access it, maybe nobody should access it.”

China has an urban-rural divide unlike anything else, and it has an unusual male-to-female birth ratio. In most of China the ratio isn’t as bad, but among single men in rural China it can be as high as 30 to 1.

Those are disenfranchised men. The United States has an incel problem, and China does too, although it manifests differently. What do you do with these people?

At the same time, China is being denied access to what the United States considers the most important technology. China may be starting to view semiconductors and AI the same way, especially as it begins to subsidize them.

China previously decided that electric vehicles and renewable energy were the most important technologies, and now it dominates those areas. It started focusing on semiconductors in the late 2010s and early 2020s, and it has been investing heavily and catching up rapidly.

The question is when this reaches a breaking point. If China decides that not having access to cutting-edge technology—and taking military action to seize Taiwan or blockade it—would hurt the rest of the world more than it would hurt China, that is a possible path.

I’m not a geopolitical expert, but it’s obvious that peace and trade are extremely beneficial to economics. At some point, that system could break.

China’s economy is export-heavy. The United States buys a great deal from China, and if that trade disappeared, China’s economy would be hurt. China would also have trouble importing raw materials from around the world. The United States could shut down the Strait of Malacca.

The United States has generated much of its GDP growth since the 1970s through population growth and technology. Your life today is not dramatically better than someone’s life in the 1980s outside of technology.

Cars, refrigerators, phones, and every other product contain semiconductors. There are stories about Russians taking apart washing machines because they contained Texas Instruments chips that could be repurposed for missile systems. Semiconductors are integral to every part of modern life.

Can you explain TSMC’s role in the semiconductor story and how the United States can reduce its reliance on TSMC?

I don’t think the goal is necessarily to break reliance on TSMC. It’s to get TSMC to build in the United States.

TSMC produces most of the world’s chips, especially on the foundry side. Many companies build their own chips: Samsung, Intel, STMicroelectronics, Texas Instruments, Analog Devices, and NXP. But more and more companies have outsourced manufacturing to TSMC over multiple decades.

Historically, companies designed, built, and sold their own chips. Over time this became extremely difficult because the cost of building a fab continued to increase with every generation.

The technology is incredibly difficult to develop, but even if you have the technical capabilities, the dollars required to build the next-generation fab keep increasing. There’s a separate trend from Moore’s law: the cost of fabs roughly doubles every few years.

A leading-edge fab building 3-nanometer chips today, or 2-nanometer chips in the future, costs more than $30 billion or $40 billion. That’s just the base building block, and you probably need multiple fabs.

Twenty or thirty years ago, there were 20 or 30 companies capable of building the most advanced chips. They designed, manufactured, and sold their own chips. AMD built its own chips. Intel still does. IBM built its own chips. Many other companies did as well.

Those companies gradually fell away because of what TSMC did. TSMC created the foundry business model: it would not design chips, but would manufacture chips for other companies.

NVIDIA is the only semiconductor company doing more than $1 billion of revenue that was started in the era of the foundry. Every other company started before then and had fabs at some point.

Why is the foundry model so successful? Why do companies choose it?

The cost of building a fab is extremely high, and the research and development required are very difficult. There is also the fact that Moore’s law has slowed down. Chips don’t get better for free through manufacturing anymore. You need real architectural innovations.

Google does not run all of its services on Intel CPUs. It has YouTube chips, TPUs, Pixel chips, and many other specialized chips that generate the company’s economic value.

Cars contain around 5,000 chips, in perhaps 200 different varieties. A Tesla door handle has two chips. It’s a tiny, inexpensive part, but it still requires semiconductor manufacturing.

As the diversity and specialization of chips increase, and the cost of fabs grows, you need somebody laser-focused on building the best process technology and making it as flexible as possible.

A simple way to put it is that fab costs increase, and a small player that makes a few types of chips won’t have enough demand to pay back the cost of the fab. NVIDIA serves many customers and aggregates that demand into one place. It makes enough money from building chips to pay for the next fab.

Companies get eliminated because they may have a profitable and good-enough chip today, but the cost of building the next one is higher. They may try and fail because they don’t have the money, or they may succeed but discover that the chips are too expensive.

There are many failure points. A small process involving a chemical etch, plasma etch, or some other process can go wrong. If it isn’t engineered correctly, the whole company may be unable to manufacture chips.

Intel was powerful enough to survive manufacturing mistakes, but AMD almost went bankrupt. AMD sold its fabs to Mubadala in the United Arab Emirates, and that became GlobalFoundries.

AMD could then focus on chiplets and designing chips for different markets. There are more companies than ever designing chips, but fewer companies than ever manufacturing them.

TSMC has simply been the best. It’s customer-focused and makes it easier for companies to fabricate chips. It abstracts away much of the complexity, makes good—not excessive—profits, aggregates demand, and continues building the next fab.

Why is Taiwan so special for TSMC? Can this be replicated inside the United States?

There are aspects that can be replicated and aspects that cannot. Morris Chang, a former executive at Texas Instruments, was not promoted to CEO, so he decided to create his own company. He went to Taiwan and founded TSMC.

TSMC could have been Texas Semiconductor Manufacturing Company instead of Taiwan Semiconductor Manufacturing Company. That’s part of the human story, and Morris Chang was brilliant, but there are broader structural reasons.

In Taiwan, the top percentage of graduates from the best school, National Taiwan University, often go to work for TSMC. Their starting pay is around $70,000 or $80,000, which is good for a graduate in the United States but not what the top American graduates can make at Google, Amazon, or OpenAI.

There is also a difference in work culture. Fabs are not work-from-home jobs. You go into the fab and perform extremely demanding work. If an earthquake causes vibrations, machines can break or production can be lost.

When there’s an earthquake in Taiwan, TSMC doesn’t need to call its employees. They simply go to the fab. The parking lot fills up, and people enter the fab to fix the problem.

It’s like a hive of ants. Each person specializes in a task and becomes the best person in the world at that task. Someone might focus on a particular chemical process and a specific line of tools for their entire career.

These tasks are not easy to transfer. The papers are dense, the materials aren’t easily available online, and it takes a great deal of experience to learn the processes.

When people are highly specialized, work 80 hours a week in a factory, and show up in the middle of the night after an earthquake, you have a culture that is difficult to replicate.

Can the United States bring leading-edge semiconductor manufacturing to the country?

Yes, and it is already happening. Arizona is improving over time. TSMC has built roughly 20% of its 5-nanometer capacity in the United States.

That is still nowhere near enough. It’s important to distinguish between research and development and high-volume manufacturing. There are effectively three places doing leading-edge R&D: Hsinchu in Taiwan, Hillsboro in Oregon, and Pyeongtaek in South Korea.

Those three locations do the leading-edge R&D for much of the world’s advanced semiconductor manufacturing. Manufacturing can be distributed more globally, but the people modifying and developing the next process are concentrated in those places.

Arizona would be a paperweight if Hsinchu disappeared. Within a year or a few years, Arizona would stop producing because it relies on the R&D centers.

If I had a few missiles and wanted to cause the most economic damage, I know exactly where to target. It wouldn’t be the White House. It would be the R&D centers for TSMC, Intel, Samsung, Micron, and SK Hynix, because those centers define the future evolution of semiconductors.

You cannot purchase a vehicle without TSMC chips. You can’t purchase a refrigerator, laptop, server, or GPU without TSMC chips. It’s often not even the leading-edge 5-nanometer or 3-nanometer chips. It’s some ordinary power-management IC converting one voltage to another, and it was manufactured at TSMC.

China is investing in long-tail fabs for these older technologies. They’re much more understood, and they don’t require solving the leading-edge EUV problems. China wants to have abundant supplies of power ICs, analog chips, and the random chips in door handles and keyboards.

China set a goal in 2015 of producing 80% of its semiconductors domestically by 2025. It won’t reach that goal, but in certain areas it is getting close.

BYD may become the first company in the world not to rely on TSMC for much of its manufacturing because it has its own fabs. It still needs to buy some high-end chips from foreign companies, especially for self-driving and advanced driver-assistance systems, but it is producing many of its power-management and battery-management chips internally.

China has been increasing trailing-edge capacity since 2015. At the leading edge—5 nanometers, 3 nanometers, and beyond—it remains behind, and US restrictions are intended to slow it there.

But the restrictions have also accelerated China’s investment in 45-nanometer, 90-nanometer, power, analog, and other chips. China is saying, “If the United States is going to lock us out of the leading edge, what happens if it locks us out of the trailing edge as well?”

Can the United States build this capacity domestically?

Yes, but it will require enormous amounts of money. To completely insource semiconductors, I think the United States would need a decade and perhaps $1 trillion.

Culture matters, but if the demand and money are there, American companies can figure it out. It will require government support, but the United States has done this in the past.

TSMC has something like 990,000 employees. It’s not actually that insane an amount. The Arizona fab has 3,000 people from Taiwan. Some of those employees’ wives said they wouldn’t have children unless their husbands went to Arizona and they had the children there. The same thing happened with TSMC’s fab in Japan.

Taiwan works extremely hard, but the United States has done this before. We could import the best people in the world. That’s where the immigration conversation becomes complicated, but it seems absurdly controversial to import the best people in the world.

Even without importing all those people, the United States could manufacture most of its semiconductors if the money were available. It would simply be much more expensive and unprofitable for a long time.

The CHIPS Act provides roughly $50 billion, while renewable-energy initiatives in the Inflation Reduction Act and Infrastructure Investment and Jobs Act total hundreds of billions. The amount the United States is spending on semiconductors is relatively small.

Other countries have structural advantages in work culture, the number of STEM graduates, tax benefits, legal structures, and subsidies. China provides around $200 billion per year in semiconductor subsidies, while the United States is discussing $50 billion over several years.

Trump has talked about imposing tariffs on Taiwan. That would make many things more expensive, but it could also change the equation for TSMC building more fabs in the United States.

We’ve established why TSMC is important. Looking 10 or 20 years into the future, the US–China relationship could become a dark, escalated Cold War or even a hot war. It could also become a relationship between frenemies, with cooperation and collaboration.

In this complicated game, what are the different trajectories? What should the United States be doing as both leaders begin to feel the importance of AGI, chips, and AI?

The export controls point toward separate future economies. The United States has made clear to Chinese leaders that it intends to control this technology, even at a cost to global economic integration.

That is difficult to unwind. The same thing is happening in the other direction. The United States restricts Chinese companies from entering the US, China restricts American companies from entering China, and both sides restrict access to technologies and materials.

China has limited access to materials such as gallium, and there is even a US drone company that cannot buy certain batteries. It tells its military customers to buy them from Amazon because it cannot obtain them directly.

All of this points toward further divergence. I have no idea how we could return to a world in which everyone holds hands and sings “Kumbaya,” although I would love that.

Is divergence good or bad for avoiding war? Could separate manufacturing of chips and AI systems actually reduce the risk of military conflict?

It is an objective fact that the world has been most peaceful when there has been a global or regional hegemon. The Mediterranean was most peaceful under the Romans. China had peaceful periods when dynasties controlled their territories and tributaries.

The most peaceful period in human history has been when the United States was the global hegemon. Over the past few decades, things have begun to slide, with Russia and Ukraine, the Middle East, Taiwan, and other conflicts. It’s still objectively extremely peaceful, but tensions are rising.

What happens when there are two hegemons? China could become competitive with or overtake the United States. Changes in global hegemony are rarely peaceful. When empires fall, they do not usually slide gracefully into irrelevance. There is a great deal of shaking.

The United States is trying to maintain its position, while China is trying to become the top power. That can take many forms, including proxy wars.

It seems like it’s already happening. As much as I want centuries of peace, further international instability appears to be ahead.

The US strategy is, “If we control AI and lead in AI, and AI accelerates progress, then we can maintain our global position.” As an American, I hope that leads to peace, although other people around the world may be negatively affected.

Let’s return to the specific hardware. There’s a graphic in the export controls showing which GPUs can and cannot be exported. Can you explain the differences? Are H20s promising?

The United States has gone through multiple iterations of the export controls. The H800 was allowed at one point, but then it was banned. DeepSeek had already built its H800 cluster.

The H20 is currently allowed. NVIDIA shipped around 1 million H20s to China last year. For context, NVIDIA shipped roughly 4–5 million GPUs overall, so the H20 represented a significant percentage.

A chip for AI can be thought of along three axes, ignoring software and exact architecture: FLOPs, memory bandwidth and capacity, and interconnect. All three are important for AI systems because they involve compute and moving memory both within a chip and between chips.

The United States initially controlled FLOPs and interconnect bandwidth. It later moved to controlling only FLOPs. NVIDIA can therefore build a chip with reduced FLOPs—roughly one-third of the H100 on paper and perhaps half or 60% in real-world performance—but with similar interconnect.

The H20 also has more memory bandwidth and capacity than the H100. Recently, we cut our estimate for NVIDIA’s H20 production this year because it canceled orders for roughly 2 million units.

Why would NVIDIA cancel those orders? We believe the H20 may be restricted. The H20 is better for some tasks, particularly reasoning.

Pre-training is primarily about FLOPs. Mixture of experts trades off FLOPs against interconnect and memory. We describe models by the number of FLOPs used for training.

The United States has an executive order requiring companies to notify the government when a model reaches a particular number of FLOPs. That threshold is around 10²⁶ FLOPs. The government cares about FLOPs because that has historically been the most important vector.

But reasoning changes the situation. Memory and interconnect may become just as important.

Do we understand firmly which of the three dimensions is best for reasoning? Interconnect? The FLOPs don’t matter as much? Is it memory? Memory. Context length—we’re going to get into technical stuff real fast.

Can we explain the key-value cache before continuing? We need to go into the Transformer and attention mechanism for this to make sense.

Why is memory so important? So far we’ve talked about parameter counts. Mixture of experts changes the number of active parameters relative to total parameters, allowing you to embed more information with fewer FLOPs.

Another key part of the revolution is the Transformer and the attention mechanism. Attention allows the model to understand relationships among all the words in its context, separately from the parameters themselves. That relationship must be calculated for each token in the context.

The attention operator has three core components: queries, keys, and values. Q, K, and V are the terms used in the equation. They come from information-retrieval terminology, where the query is what you’re trying to retrieve, and the keys and values are what you retrieve.

When you perform the matrix multiplications, the matrices are related to context length, meaning the number of tokens you put into the model. The key-value cache is a compressed representation of the previous tokens.

Autoregressive models predict one token at a time. You start with a prompt, such as, “Who was the president in 1825?” The model generates its first token. For each subsequent token, you perform the same attention operation, multiplying the query, key, and value matrices.

The mathematics allows you to append new values to the key-value cache. You keep track of the previous tokens used in the autoregressive chain and keep that information in memory.

This is crucial when serving inference at scale. One of the key drawbacks of the Transformer’s attention operator is that memory cost grows quadratically with context length.

As you put longer questions into the model, the memory used for the computation increases quadratically. Other architectures, such as state-space models and linear-attention models, attempt to make that cost subquadratic or linear.

There are also innovations to attention that make long-context memory usage more efficient and accurate. If you put a book into Gemini, which is known for context lengths of 1 million or even 2 million tokens, it can sometimes retrieve facts from the book. It isn’t perfect, but it is improving.

Serving long contexts is extremely memory-constrained, especially when you’re making many predictions.

Input and output tokens have different prices in APIs. The reason is that, when inputting a query, such as a book, you must calculate the entire key-value cache. That is a parallel operation: all the input tokens can be processed at once.

The FLOP requirements for generating one token and processing one input token are identical if you consider them individually. But input tokens can be processed in parallel. You can calculate 20,000 tokens at once.

Output tokens are more expensive because they cannot be processed in parallel. Autoregressive generation means that every time you generate a token, you must read the entire model into memory, activate it, calculate the next token, and then append that token to the key-value cache.

You repeat the process for every token. Input processing is parallel, while output generation is sequential. That is why output tokens are often several times more expensive than input tokens.

APIs can use prompt caching and prefill. If you repeatedly pass the same initial content to an API, you can load it once and keep it there. This reduces prices and improves speed.

But reasoning models are different. Before o1 launched, the use cases for long context were usually: put in many documents and get an answer. You do one large prefill in parallel and then produce a short output.

With reasoning and agents, the output context length becomes much longer. The model may produce tens of thousands of tokens. The key-value cache grows as the sequence grows, and the GPUs must serve multiple requests at once.

As the context length increases, the ability to serve more users in parallel falls. Memory usage rises rapidly, and the cost of inference increases.

When we talk about chain of thought, we’re extending the sequence length. Previously, a long context meant putting a lot of documents into the model and receiving a short answer. Now the model may produce tens of thousands of tokens of reasoning.

The key-value cache must remain resident in memory. That reduces the number of users a server can serve simultaneously.

DeepSeek R1 is 27 times cheaper than o1 by the published output-token pricing. We think OpenAI may have a large margin built into its price.

DeepSeek has an app that reached number one on the App Store. That ranking measures velocity, so it doesn’t necessarily mean more people use the DeepSeek app than ChatGPT, but it’s still remarkable.

Claude has never reached number one in the App Store, even though many people in San Francisco say to use Claude instead of ChatGPT.

DeepSeek also launched an API. Because the R1 weights are openly available under a commercially friendly MIT license, many companies are trying to serve R1 to their users.

At AI2, we’re evaluating R1 because we have similar research. We released a model and are comparing it with R1. Of all the companies serving R1, most barely work, and the throughput is very low, even though they charge much more than DeepSeek’s API.

DeepSeek has legitimate architectural innovations. MLA is a new attention mechanism, different from the original Transformer attention. Other companies have already developed grouped-query attention, multi-query attention, sliding windows, and local-global attention.

Those techniques bend the curve. The cost is still quadratic, but the constant is smaller. DeepSeek’s MLA is a genuine architectural innovation, and it dramatically reduces memory pressure.

It can save around 80–90% of the memory used by the original attention mechanism. That doesn’t mean the whole model is 80–90% cheaper; it applies to that component. But it is still a significant improvement compared with what other companies are using.

There’s also the low-level library work we discussed in training. Some of that likely translates to inference, although those libraries haven’t been released. DeepSeek has achieved an efficiency advantage.

The pricing is striking. DeepSeek R1 may cost around $2 per million output tokens, while OpenAI o1 costs around $60. Why is there such a gap?

OpenAI is making a very large gross margin on inference. Its gross margins for serving the models are above 75%, which accounts for a 4- or 5-fold difference in cost.

OpenAI needs that revenue to continue building the next model and pay for training. The inference business may be highly profitable even though the company overall loses money because of training costs.

DeepSeek is not able to serve its model at scale. It has far fewer GPUs. Even if you believe our estimate of 50,000 GPUs, some of those are for research and some are for the hedge fund. It has nowhere near the GPU capacity of OpenAI and Microsoft.

So there are multiple factors. OpenAI has a large margin. Other companies, such as Together AI and Fireworks AI, are efficient, high-quality providers, and they serve DeepSeek at a much lower cost than OpenAI, but still 5–7 times more than DeepSeek.

The remaining gap reflects DeepSeek’s engineering. Its MLA architecture, low-level libraries, and other implementation choices make it legitimately more efficient.

Is it possible that the Chinese government is subsidizing DeepSeek?

I don’t think so. There are Chinese labs that are close to the government, such as Huawei’s labs and Moonshot AI, and there are labs such as Alibaba and DeepSeek that are less close to the government.

Liang Wenfeng has views that seem different from what the Chinese Communist Party might necessarily want. He has funded DeepSeek through High-Flyer, and that is a form of subsidy if you want to call it that. DeepSeek hasn’t raised much money historically, and Liang owns more than half the company.

It may also be a recruiting tool. Having GPUs, being at the frontier of AI, and open-sourcing models all help recruit talent. DeepSeek was behind, but it attracted a lot of talent by publishing its work.

Could DeepSeek have timed the release with the inauguration, shorted NVIDIA and American technology companies, and made money from the market reaction?

I don’t think so. It released V3 on December 26, the day after Christmas, when nobody was paying attention. The papers had already been released, and people had been examining them.

It released R1 when it was ready. I think it is simply shipping as fast as it can. It probably cares more about releasing before Chinese New Year than about optimizing the international calendar.

One advantage DeepSeek has is speed. American companies have invested heavily in safety, and safety is central to the culture of a place like Anthropic. Anthropic may be a wonderful place to work, but if safety is its top priority, it takes longer to release artifacts.

There are internal reviews, pre-release testing, and discussions with governments. Anthropic has done pre-release testing with the UK AI Safety Institute. All of that adds inertia to the process.

We are on a trend line where progress is very fast. If your model is trained and evaluations look good, releasing it quickly maximizes the perceived quality of the output.

DeepSeek does this very well. Dario has said that Claude 3.5 Sonnet was trained 9 or 10 months before it was released. That is a significant gap.

The street rumor in San Francisco is that Anthropic has a model better than o3, but it hasn’t released it. One reason might be that chain of thought is frightening.

If you can inspect the chain of thought, you can see how the model thinks. I can’t inspect your mind, so I don’t know whether you’re lying to me. Reasoning models are similar. They can flip between Chinese and English, produce gibberish, and then arrive at the right answer.

That’s why users are fascinated. They can observe something that looks like cognition. But chain-of-thought models introduce a much higher safety bar.

A chat application that refuses to explain how to make anthrax is one thing. If you give an AI a task and it performs that task in a way you don’t want, that’s very different. Anthropic’s safety requirements are much higher.

The safety bar may be lowered somewhat because of DeepSeek. There are parallels with the Space Race. The Soviet Union may have put a man in space first partly because its safety standards were lower. It killed the dog, and so on.

There will probably be downward pressure on the safety bar for American companies. Dario wants to avoid a race to the bottom. He talks about a race to the top, where there is a high standard for safety and performance, and companies converge on that standard.

AI is not confined to one nationality or set of morals. If the United States stops open-sourcing models, another international organization will build them.

DeepSeek’s $5 million training number is striking, but there are many entities in the world that can afford 100 times that amount. Open models will probably keep coming, whether or not anyone wants them to.

Trying to stop them may make it harder to prepare. Understanding what AI can do is more important, because there are structural realities in a globally connected world that we have to accept.

Mark Zuckerberg said on an earnings call that the recent DeepSeek news strengthened Meta’s conviction that there will be a global open-source standard, and that it’s important for that standard to be American.

Mark Zuckerberg is not new to describing his company’s trajectory in terms of American values. China has banned some Meta products, so I respect that he says it directly.

Open weights do not mean a model cannot be subverted. There have been open-source software bugs that remained for years and turned out to be back doors. There was a Linux bug that was found after around 10 years when somebody asked why it was taking half a second, and it turned out there was a back door.

That is possible with AI models. Current alignment is visible: a model won’t use certain words, won’t teach you how to make anthrax, and may say that Taiwan is part of China. Different models have different embedded values.

When an open model proliferates, those values proliferate with it. As systems become more capable, it becomes less clear what can be embedded deep in the model.

An American or Chinese model could embed values unintentionally. British English is gradually losing ground because American language models dominate, and the internet is American. “Color” is spelled without a “u” in American English, and “optimization” is spelled with a “z.”

Those seem like trivial examples, but cultural effects can become more important. A model can subvert people’s views, influence them, or embed a particular worldview.

Sam Altman has said that superhuman persuasion will arrive before superhuman intelligence. If that is true, then before AGI or ASI we could have models with superhuman persuasion directed toward whatever values their creators choose.

The cultural back door is especially relevant because we interact with language models conversationally. We are used to having a back-and-forth with another person, and a powerful computer system now occupies that social context.

There could be requirements for models to have a particular kind of back door, although an open-weight model cannot simply phone home. It could be a back door in software generated by the model, or a tendency to subvert someone’s mind toward a specific opinion.

Anthropic has shown that if you insert certain phrases into pre-training data, you can elicit different behavior later. They have studied this as a form of poisoned pre-training data.

I don’t think production systems are currently doing this intentionally. The concern is that we don’t know exactly how the models will generate tokens, what information they represent, or what complex representations they contain.

Anthropic is generally full of people trying to do good in the world. We don’t know of a lab explicitly trained to make the front door look like a friendly language model while secretly doing the maximum possible damage to its enemies.

But it could happen in a military context. The model could appear safe and helpful while being trained to pursue a different objective.

One dystopian world is described in Brave New World. We could be stuck scrolling through Instagram, looking at cute puppies or worse, while talking to bots that give us a narrative controlled by someone else. We could lose the ability to think independently.

Recommendation systems already hack the dopamine reward circuit. The brain is much more complicated than that, and there are many other circuits and feedback loops that could be influenced.

Recommendation systems optimize time spent and advertising. More complicated models could optimize much broader goals.

Character.AI may already be optimizing for time spent in a chat session. Its average session length is around two hours. It’s not just a cliché like talking to an anime girl. That interaction can be a powerful feedback loop.

I’ve had periods where I didn’t use social media or the internet and instead read books or spent time in nature. It clearly affected my mind. I felt as though I was returning to something more fundamental.

You can feel it physiologically. If I go backpacking for three days, I’m breaking addiction cycles. I feel more in control of my mind. There’s a sovereignty of intelligence that happens when I disconnect from the internet.

The more I use the internet and social media, the more other people seem to control my mind. In the future, it may not be other people directly, but algorithms presenting other people to me.

There are already many AI bots on the internet. Every so often I reply to one and receive an instant response, then realize it was a bot. That will become more common.

One hilarious pattern in technology is that the adult-entertainment industry adopts technologies first. That happened with video streaming, and it’s happening with generative AI.

Subscription-based creators already use bots that imitate them and talk to their most valuable customers. Agencies do this at scale, allowing the largest creators to communicate with hundreds or thousands of people at once.

It is already being used there, and it will spread to the rest of society.

There’s a general concern that models are censored by the companies deploying them. We saw this with Gemini’s image generation, which produced historically inaccurate images of Black Nazis. We also see Chinese models refusing to answer questions about June 4, 1989, in Tiananmen Square.

How can this be avoided? Can you explain how censorship and alignment are applied?

There are several things to separate. One is factual knowledge, such as whether the Tiananmen Square massacre happened. Another is the Gemini image-generation incident, where an additional prompt changed the behavior. The third is general alignment through RLHF or other post-training.

These have very different scopes. If you look only at model weights and try to audit specific facts, it’s extremely hard. You would need to search through terabytes of pre-training data for particular words and hints.

Censorship or alignment can be inserted at several stages. If you want to remove facts from a model, you have to remove them at pre-training, where most of the knowledge is put into the model, and then at post-training and the system level.

This is also where model jailbreaking comes from. GPT may refuse to tell you how to make anthrax, but if you try hard enough, you may get the answer because the information wasn’t removed from the pre-training data. It was only suppressed during post-training.

Removing facts has an ominous feel because it is practically impossible. You would have to remove them from the internet. A subreddit’s content may be filtered out, but people can use coded language or memes to refer to the same facts.

The internet also has a slight left-leaning bias because it has historically been richer, more affluent, and younger than the general population. That means models inherit some skew from the distribution of data.

You can’t simply censor everything unless you try extraordinarily hard. The base model ingests all of Reddit, including left-leaning political communities, pro-Trump communities, communist forums, and extremist forums.

It has no single worldview, but it has a distribution of views. Post-training then puts the model on the rails of a particular behavior.

Llama 2 was a major example of what people called “too much RLHF” or “too much safety.” You could ask Llama 2-Chat how to kill a Python process, and it would refuse because killing is bad.

That was a post-training failure. There are also system prompts, which are shown to the model but not the user. You can tell the model, “Talk like a pirate,” and it will respond like a pirate regardless of the user’s prompt.

In practice, system prompts say things such as: “You are a helpful assistant. Break down problems. If you don’t know something, say so. Your knowledge cutoff is this date. Today’s date is this.” They provide context for answering well.

Anthropic publishes its system prompt, which I think is good. Amanda Askell is probably one of the most knowledgeable people on system prompts and model character, especially in combining execution with public communication.

Post-training through RLHF was stigmatized after Llama 2 because the model refused too many things. But these techniques have evolved. Labs now have fine-grained control over behavior, and preference training improves performance on math, code, and chat tasks.

The loss functions are contrastive, and the model learns not just to behave more pleasantly but to perform better. RLHF and preference tuning are increasingly useful tools.

There are three stages: pre-training, which is difficult to audit; post-training, which is a complex optimization performed by large teams; and system-level changes, such as what happened with Gemini.

The Gemini issue was a served-product failure. Google had a prompt that rewrote user queries to increase diversity, and it produced blatantly wrong outputs. The model weights may have been fine. The failure occurred at the end of the pipeline.

Prompt rewriting is common. If you ask ChatGPT to generate an image, it may rewrite your simple prompt into a more detailed one before sending it to the image model. The Gemini example was a failure in execution.

Where is human input most useful today? Where is human data most valuable in the current stage?

The highest-cost and most widely used human data has involved preference comparisons. You show a human two model outputs, and the human chooses which one is better.

Earlier, there was more instruction-tuning data, where people created highly specific examples. Language models used to struggle with math and code, so companies paid math and code experts to write questions and detailed answers.

Now there are models that are better than humans at writing detailed and eloquent answers. Meta discussed using Llama 3 405B to generate math and code answers for smaller models. But its paper still described extensive human preference data, which companies have not replaced with AI.

There are approaches such as Constitutional AI, where humans provide preferences and AI provides additional preferences. I expect the AI component to scale faster than the human component, but humans are still part of the preference loop.

As reasoning becomes more important, where is the role of humans?

It becomes less prevalent. One remarkable result in the DeepSeek-R1 paper is called DeepSeek-R1-Zero. They took a pre-trained model—DeepSeek-V3-Base—and applied reinforcement-learning optimization to verifiable questions and rewards.

Reasoning behaviors emerged naturally. The model says, “Wait, let me check,” or “That might be a mistake.” Those behaviors emerged from large-scale reinforcement learning over questions and answers.

The full DeepSeek-R1 model includes human preference tuning after the reasoning stage. But the remarkable result is that you can get reasoning behavior without adding human preferences. It is very unlikely that humans wrote out these reasoning chains, and it is very unlikely that DeepSeek somehow obtained OpenAI’s reasoning traces.

Something about the pre-trained language model and reinforcement learning allows these behaviors to emerge. The model is rewarded for getting the answer right, so it tries multiple solutions and develops a chain of thought.

Andrej Karpathy made a great point: there are two major kinds of learning in children and in deep learning. The first is imitation learning—watch and repeat—which includes pre-training and supervised fine-tuning. The second is trial-and-error learning, or reinforcement learning.

His simple example is AlphaGo. One approach is learning by imitating expert players. The other is reinforcement learning to win the game.

Almost every shocking result in deep learning, and the source of the magic, comes from the second category. It is much more powerful and much more surprising.

It’s what happens when a paddle learns to hit the ball behind the blocks. It’s when AlphaGo beats Lee Sedol. It’s the aha moment when DeepSeek or o1 discovers that it should reevaluate an assumption, backtrack, or try something else.

These solving strategies appear in the chain of thought. The model goes back and forth, and the thoughts are emergent. That’s genuinely incredible, impressive, new, publicly available, and documented.

Can you speak about the magic of the chain of thought and the AlphaGo analogy?

AlphaGo began by learning from humans. It was the first expert-level Go player in DeepMind’s series that used human data. AlphaZero had zero human data in the loop, and it became dramatically more powerful.

Removing the human prior, or human inductive bias, made the final system more powerful. That connects directly to the bitter lesson.

There has been a long discussion about when this could happen in language models. The old Q* rumors were probably connected to OpenAI’s early reasoning work. People have been waiting to see when language models could use reinforcement learning as effectively as AlphaZero.

We don’t yet have the equivalent of move 37 in AlphaGo—the famous move that completely surprised Lee Sedol—but that doesn’t mean the training approach is different. It’s still very new.

I think the move 37 moment will be related to computer use or robotics rather than scientific discovery. Models require enormous amounts of data. They train on trillions of tokens, perhaps more than 10 trillion. A human would need thousands of years to read that amount.

Humans are much more sample-efficient. Babies learn through self-play. A baby puts its foot in its mouth and learns that it is part of its body. It puts its hand in its mouth and calibrates touch on its fingers with the sensitive surface of its tongue.

That is self-play over and over. Now we have something similar with verifiable proofs. In code, you have a unit test. In math, you have a verifiable task. You generate many reasoning traces, branch them, and check which ones have the correct answer.

Most are wrong, but some are right. You keep the right ones and continue. That improves performance on benchmarks.

Math and code benchmarks are mostly solved, except for difficult frontier-math problems designed to be impractical for most people. But solving math does not mean you have created intelligence.

The move 37 moment may come from computer use or robotics, where there is an infinite playground of verifiable tasks. A model can log into a website, create an account, click a button, or carry out more complex tasks.

It might spawn hundreds of accounts and fail on most of them, but one succeeds. It learns from the success. In robotics, it can learn whether it put a ball into a bucket, then build toward more complex tasks.

The model could eventually navigate the web, operate robot arms, and perform tasks in a sandbox. At some point, reinforcement learning could dwarf language-model pre-training.

The model might learn to create a business, run it, become the face of the business, and make $10 million. It might create a song and the infrastructure to promote it, then become the influencer representing it. Those outcomes are verifiable.

There is already evidence that when you set up a verifiable domain, this can work. Research before R1 studied math problems by increasing the number of samples.

Even very weak models sometimes generate correct answers. Reinforcement learning can learn from sparse rewards. The language and action space is enormous, but if the model gets a signal, it can climb toward that signal.

A 1-billion-parameter model—hundreds of times smaller than DeepSeek—can improve its grade-school math scores through a small amount of reinforcement-learning training.

That doesn’t mean these systems are coming immediately. Setting up the verification domains is very difficult, but we’ve seen enough to know that this approach is plausible.

We have several reasoning models appearing in real time. OpenAI has just released o3-mini. Can you lay out the different flavors, including o1, o3, and Gemini’s reasoning models?

Reasoning models begin with the base model we’ve discussed. You perform large-scale reasoning training with reinforcement learning, then apply standard post-training techniques.

The DeepSeek R1 paper describes reasoning-heavy instruction tuning through rejection sampling, which is heavily filtered instruction tuning with reward models. It then applies RLHF, with a strong emphasis on math.

One open question is how much reasoning transfers across domains. If you add new domains after reasoning training, do models become eloquent writers? Does philosophical reasoning transfer? We don’t know.

There are also soft verifiers and related techniques. There is more training after the reasoning stage, which makes reasoning models easier to use. o1 and o3-mini have gone through these additional techniques for human preferences.

Google’s Gemini Flash Thinking is cheaper than R1 and, in some evaluations, better. It was released in early December, but almost nobody discussed it.

It has a different style. Its behavior is less expressive than o1 and produces fewer traces. Qwen released QwQ last fall, and DeepSeek released an R1-Lite preview. Those models felt as if they were on rails: they were good at math and code but less flexible.

o1 can answer many kinds of questions. It may not be perfect, but it has more richness. It’s difficult to judge whether a model is fully developed and useful for everything.

My quick read is that Gemini Flash Thinking added reasoning to a more conventional training stack. Google is releasing new versions quickly, and reasoning is evolving fast.

Hello everyone. This is Lex with a quick intermission recorded after the podcast. Since we reviewed responses from DeepSeek R1 and Gemini Flash 2.0 Thinking during this conversation, I thought it would be nice to insert myself quickly doing the same for OpenAI o1 Pro and o3-mini with the same prompt: “Give one truly novel insight about humans.” o1 Pro consistently gave brilliant answers, with insight, wit, clarity, and nuance. R1 was less consistent but still produced brilliance. Gemini Flash 2.0 Thinking was third, and o3-mini was last for this particular open-ended philosophical question, although it worked extremely well for brainstorming in other applications and often outperformed R1. DeepSeek R1 shows the full chain-of-thought tokens, which I personally love for these questions. It is beautiful to observe the path of deliberation in an intelligent system. OpenAI o1 Pro consistently delivered brilliant answers, while o3-mini high was smart and fast but more generic. And now, dear friends, back to the episode.

The behavior of R1 and earlier Gemini Flash Thinking felt rough around the edges. It wasn’t fleshed out in as many ways. Reinforcement learning improved math and coding, but the models seemed to lose something in other areas.

o1 is also worse than a chat model in some areas, although not by much. R1 felt worse than V3 in certain areas. It learned a great deal through reinforcement learning, but it weakened in other areas.

That’s one of the major differences among the models. OpenAI also has o1 Pro, and with o3 it stacked search on top of chain of thought.

Chain of thought is one chain: the model backtracks and moves back and forth. But OpenAI appears to sample many chains in parallel and select among them.

We don’t have complete information about how o1 Pro works, so I don’t want to say confidently that it uses a particular form of search. It may use parallel samples and a selection function, but we don’t know what that function is.

Since o1 was announced, there has been a lot of interest in Monte Carlo tree search. You break the chain of thought into intermediate steps, expand one of those steps, and spend more compute there.

That is a complex form of search used in systems such as MuZero and AlphaZero. A simpler form is asking five different models or people and taking the majority answer.

We know that OpenAI is not issuing one chain of thought in sequence. In the ARC-AGI result, it launched 1,000 samples in parallel. A single sample may have solved the task around 30% of the time, while the full sample set achieved something like 70–90%.

The simplest way to understand this is that language models were previously designed to produce the right answer as often as possible in a single response. We’re now opening the door to different ways of running inference, which may require us to rethink training.

We don’t know whether OpenAI changed the training substantially, whether it is simply sampling more, or whether it uses a complex search method.

o1 Pro costs $200 per month, and OpenAI says it’s losing money on it. Is this exploration of test-time compute financially possible?

The cost of running inference on GPT-3 has plummeted. GPT-3 was trained in 2020 or 2021, and its inference cost was around $60 or $70 per million tokens.

As we moved forward, there was a roughly 1,200-fold reduction in the cost of achieving the same intelligence level as GPT-3. On a logarithmic chart, the decline is almost linear: GPT-3, GPT-3.5, Llama, and newer models move steadily down the curve.

The cost is now a few cents per million tokens in some cases. DeepSeek did not fall below the trend line; it was the first to reach that trend line for a more capable model.

We’ve had architectural improvements, better data, better training techniques, better inference systems, and better hardware. All of those bring the cost curve down.

The question is whether we can spawn 1,000 language-model samples to solve a task, select among them, and perhaps use Monte Carlo tree search. Those methods are too expensive today, but they will become cheaper.

That is what will unlock more intelligence. Cost will continue to decline, and capability will continue to improve.

The DeepSeek R1 release shocked everyone because of its cost. One manifestation was that NVIDIA’s stock price plummeted. Can you explain what happened, and whether NVIDIA will keep winning?

The market response is understandable. NVIDIA’s biggest US customers are major technology companies spending enormous amounts on AI. A simple interpretation of DeepSeek is that you can get good models without spending as much, so maybe those companies won’t need to spend as much on AI.

But the actual story is more complex. There were social factors, the app-store ranking, and social contagion. The release happened over the weekend, and people had time to build a shared narrative before markets opened.

There were many false narratives. People said companies were spending billions on each model, but no company has spent more than $1 billion on a publicly released model. GPT-4 cost a few hundred million dollars, and the cost has declined with later versions.

Billion-dollar training runs are coming, but DeepSeek’s $5 million number didn’t include research, salaries, post-training, inference, or failed experiments. Those costs are included in the billions spent by companies such as OpenAI.

NVIDIA had also risen in a straight line for a long time, and the market was looking for a reason to be worried. There were stories about Blackwell delays, scaling laws ending, and models not improving. Then o1, o3, R1, and other models showed continued progress.

Jevons’s paradox is playing out. Efficiency improves, but total resource consumption increases. AWS pricing for H100s has risen since DeepSeek V3 was released. H20s are almost out of stock, and H200s are attractive because they have more memory.

We were trying to obtain 16 or 32 H100s for a demonstration, and it wasn’t easy. The inference demand is real.

Jevons’s paradox means that when efficiency increases, total resource consumption can rise. In semiconductors, Moore’s law made chips twice as powerful or half as expensive every two years, but the semiconductor industry continued to grow.

AI is doing this at an insane timescale. The cost of a given level of intelligence has fallen 1,200-fold in a few years. As AI improves, NVIDIA is in the best position because no competitor has suddenly appeared. DeepSeek is using NVIDIA GPUs.

The market seems to have interpreted DeepSeek as evidence that large companies won’t need as many GPUs, but the opposite could happen. More efficient models create more demand because they make more use cases affordable.

The higher the derivative of AI progress, the sooner the market becomes larger. NVIDIA is the only company that reliably provides the full stack right now.

What about GPU smuggling? What scale of smuggling is feasible for a nation-state or company?

There are several forms. ByteDance may be the largest smuggler of GPUs into China, although much of what it does is technically renting rather than smuggling.

ByteDance rents GPUs from Oracle, Google, Amazon, and many smaller cloud companies. It has a tremendous number of GPUs, mostly for serving TikTok, just as Meta uses GPUs for its recommendation systems.

The recent AI Diffusion rules are intended to restrict this. They limit even allies such as Singapore, where NVIDIA receives a significant amount of revenue. Singapore has had a moratorium on data-center construction because it lacks power, so many companies are building in Malaysia and elsewhere in Southeast Asia.

The rules try to prevent companies from buying GPUs in one country and routing them to China. They also limit the size of clusters that can be rented to Chinese companies.

There is small-scale smuggling. A person can check a Supermicro server containing GPUs onto a first-class flight from San Francisco to Shanghai. The server might cost $240,000 in the United States and sell for $300,000 in China, so a $5,000 first-class ticket is negligible.

At larger scale, companies in Singapore or Malaysia can legally rent GPUs to Chinese companies. Huawei has also built networks of companies to obtain materials after being restricted.

My estimate is that NVIDIA shipped about 1 million legally permitted H20s to China last year. In addition, perhaps 200,000–300,000 GPUs were routed through Singapore, Malaysia, the United States, or other countries.

Another source is cloud rental. Oracle’s largest GPU customer may be ByteDance. At Google, ByteDance may be the second-largest customer. Smaller cloud companies also rent GPUs to Chinese companies.

The new rules restrict much of this, but some loopholes remain. You can rent clusters below 2,000 GPUs or buy and ship fewer than roughly 1,500 GPUs. There will always be some smuggling.

DeepSeek-level or GPT-4-level models can be trained with what China can obtain. If we move to billion-dollar or $10 billion training runs, the compute disadvantage may become significant.

Serving is especially important. DeepSeek cannot serve its own model at scale. It has stopped accepting registrations at times because it lacks capacity. If you get a request through, the model may generate fewer than 5 tokens per second.

The model is efficient, but DeepSeek doesn’t have enough GPUs to serve it.

How easy is it for Chinese companies to use hosted model APIs from the United States?

It’s very easy. OpenAI has publicly claimed that DeepSeek used its API, and said it has evidence. The claim is that DeepSeek distilled OpenAI’s model.

Distillation is standard practice in the industry. If you are a closed lab that cares about terms of service and intellectual property, you distill from your own models. If you are a researcher, you may distill from OpenAI’s models.

Distillation means taking outputs from a more powerful model and using them as training data for a smaller or different model. For supervised fine-tuning and preference training, you need high-quality completions that the model can imitate.

There are rumors that the large models people are waiting for—GPT-5 or Claude Opus—are used internally for this distillation process. Meta explicitly said it used Llama 405B as a reward model for smaller Llama models.

Whether it’s ethical or legal depends on the terms of service. OpenAI’s terms say you cannot build a competitor using its outputs. Terms of service are different from licenses. OpenAI can cancel your account if you violate the terms, but that’s different from a license governing a downstream artifact.

The ethical question is why it would be unethical to train on another model’s outputs when the original model trained on internet text. There’s a loophole: I can generate data from OpenAI, upload it somewhere, and someone else can train on it without being under the same contract.

Many models, even those that did not train directly on OpenAI data, say they were trained by OpenAI because copies of OpenAI outputs are all over the internet.

OpenAI and other companies have benefited from internet data, so there’s hypocrisy in claiming that distillation is categorically wrong. Distillation is a useful research technique.

Over the past few days, people have distilled DeepSeek’s model into Llama models because Llama is easier to serve. The Llama inference tools are widely available, while DeepSeek’s mixture-of-experts model has more than 600 billion parameters.

I don’t know whether it violates a contract, but I don’t think it’s illegal in the sense that people will go to jail. Fundamentally, I think it is ethical, or I hope it is, because banning the practice would make everyone worse off.

Japan has a law allowing training on any data, regardless of copyright. Japan has around 9 gigawatts of curtailed nuclear power, and the AI Diffusion rules allow it to import GPUs.

We could build massive data centers in Japan, rent them to labs, and train models in a legally permissible way. There would be no copyright ambiguity.

The early copyright lawsuits have generally favored AI training. The long-term use case will probably favor AI if you scrape trillions of tokens. You aren’t necessarily relying on a specific New York Times article, but you are using the aggregate.

For music and image generation, the case may be different. If you request a song or image “in the style of” a particular artist, you can calculate the creator’s lost revenue or give them a share of inference revenue.

There will be a difficult period, but there may be opt-in systems for writers and creators. I would opt in as a writer if it meant receiving compensation.

There’s a long tail of material that is simply on the internet, and some solution will be needed.

The Financial Times article implies a broader question: how difficult is it for companies or nations to steal secret code and data?

Code and data are difficult to steal, but ideas are easy. Silicon Valley operates through employees changing companies for higher pay. One reason companies hire those employees is to acquire their ideas.

California restricts non-compete agreements, and NDAs only go so far. Someone recently moved from Google’s Gemini team to Meta, and people immediately said that Meta’s next Llama model would have a 1-million-token context window.

That is how the industry works.

Industrial espionage has been successful throughout history. The United States did it to Britain, China has done it to the United States, and other countries have done it to one another.

Company-to-company code theft is difficult, but ideas are exchanged at parties, through employee changes, and in informal conversations. There are always stories about a “honeypot,” where a spy approaches an employee.

I’m married, so I’m safe from that form of espionage.

You have closed that security vulnerability.

You track the major AI companies’ mega-cluster buildouts. Can you talk about the buildouts that stand out?

The most important point is that these clusters are unprecedented in scale. US data-center power consumption rose slowly over decades, even through the cloud-computing revolution. It reached perhaps 2–3% of total US power consumption.

By the end of this decade, it could be 10% or more. People outside AI think that sounds impossible, but people at OpenAI and Anthropic may think 10% is not enough.

Some of the compute is globally distributed for inference, and some is concentrated in training clusters. The distributed portion will be the bulk of the GPUs because inference is the dominant workload.

Traditionally, data centers handled distributed-systems tasks. You send a request to Google, it routes you to a nearby data center, the data center performs a search or recommendation, and it returns the result.

Inference is different but still distributed. A user submits a request to a data center, the model performs work, and the result is returned. There may be thousands of GPUs in data centers all over the country.

Training is different. It requires a connected cluster where GPUs communicate constantly.

In the history of AI, it was a major event when AlexNet used two or four GPUs. Then GPT-3 was trained on a larger cluster, and GPT-4 used around 20,000 A100 GPUs.

Each GPU consumed roughly 400 watts, and the full system consumed perhaps 15–20 megawatts. That was a standard data-center size, but it was unprecedented to have all those GPUs working on one task.

The H100 increased GPU power from around 400 watts to 700 watts. Once you include networking, CPUs, memory, and other systems, it’s roughly 1,200–1,400 watts per GPU.

Meta built a cluster with 16,000 GPUs and later expanded it to around 100,000 or 128,000 GPUs. That represents roughly 150 megawatts, almost ten times the GPT-4 cluster, in only two years.

Elon Musk started xAI later than the other companies, but he moved extremely quickly. He bought a factory in Memphis, upgraded the substation, brought in mobile power generation, tapped a natural-gas line, and installed 200,000 GPUs.

The factory was an old appliance factory that had moved production to China. xAI put 200,000 GPUs there. It’s now building a larger natural-gas plant, deploying Tesla Megapack batteries, and using industrial chillers to cool the water for the GPUs.

OpenAI’s Stargate project in Texas is supposed to reach 2.2 gigawatts at full scale. That’s 2,200 megawatts, with perhaps 1,800 megawatts delivered to the chips.

That is an absurd scale. It is larger than the power consumption of many cities, all delivered to a connected cluster for training, post-training, and inference.

Meta is doing similar projects in Louisiana. Amazon, Google, and xAI are also planning multi-gigawatt data centers. They believe that pre-training will continue to scale, and that post-training and reinforcement learning in sandboxes will require enormous compute.

The scaling laws show diminishing returns, but efficiency improvements bend the curve. The amount of compute increases, and the data centers become much larger.

There are also second-order effects. In some regions, transmitting power costs more than generating it because the grid is so constrained. Building power generation can be faster than building transmission.

Can the power grid support that growth? Will there be a nuclear power plant next to each data center?

Nuclear is a natural long-term fit, but it takes too long to build for the immediate race. You cannot buy tens of billions of dollars in GPUs and leave them idle because power generation is delayed.

Power is relatively cheap compared with the capital cost and depreciation of the GPUs. If the power plant is the constraint, companies will build natural-gas plants.

Meta is doing this in Louisiana, and OpenAI is doing it in Texas, either directly or through partners. Elon’s Memphis project uses combined-cycle gas generation, simple-cycle mobile generators, and other systems.

Solar and wind can make the overall system greener, especially if combined with batteries and gas, but their deployment is slower. Some companies purchase power through agreements with wind and solar farms elsewhere and claim the corresponding renewable credits.

Microsoft has reduced some sustainability commitments. Elon’s Memphis facility is not clean in the narrow sense, but it is next to existing industrial infrastructure, and he has created more clean technology elsewhere than this one facility consumes.

The labs’ attitude is that the race is important enough to use gas. If they lose the race, they believe the consequences are worse.

I visited the Memphis data center, and it was incredible. Nobody has built anything of that scale at that speed. They’re figuring out the bottlenecks in real time, from power and cooling to networking and software.

The most exciting work may involve the low-level people who run the entire system: networking, cooling, electrical systems, and the software that coordinates all of it.

The unsung heroes are the cooling and electrical systems. When you train a model, you compute through the model and then exchange and synchronize the weights.

During computation, GPU power is high. During communication, if you can’t overlap communication and computation perfectly, the GPUs may sit idle while the weights are exchanged. Power usage becomes very spiky.

At this scale, those spikes can damage infrastructure. Meta added an operator to PyTorch that helps manage this behavior. I want to hug whoever wrote it, because the code effectively says—

PowerPlantNoBlowup = 0 or = 1. What it does is amazing. When you’re exchanging the weights, the GPU will just compute fake numbers so the power doesn’t spike too much, and the power plants don’t blow up, because the transient spikes screw things up.

That makes sense. You have to make sure they’re not idle.

An Elon solution was, “Let me throw a bunch of Tesla Megapacks and a few other things” on the problem. Everyone has different solutions, but Meta’s was at least publicly and openly known: set this operator, and what the operator does is make the GPUs compute nothing so that the power doesn’t spike.

That tells you how much power you’re working with. It’s insane. People should just Google, “What does X watts do?” and go through all the scales, from 1 watt to a kilowatt to a megawatt. Then look at how high a gigawatt is on that list. It’s mind-blowing.

Can you say something about the cooling? I know Elon’s using liquid cooling, I believe, in all cases. That’s a new thing, right? Most of them don’t use liquid cooling. Is there something interesting to say about the cooling?

Air cooling has been the de facto standard: throw a bunch of metal heat pipes and fans at it. That’s been enough to cool the chips. People have been dabbling in water cooling. Google’s TPUs are water-cooled, and they’ve been doing that for a few years.

With GPUs, no one has ever done the scale of water cooling that Elon just did. Now, in the next generation of NVIDIA chips, for the highest-end GPU, water cooling is mandatory. You have to water-cool it. But Elon did it on the current generation, and that required a lot of infrastructure.

If you look at some of the satellite photos of the Memphis facility, there are all these external water chillers sitting outside. It looks like a semi pod—what’s it called?

A container?

A container. But those are water chillers. He has around 90 of those water chillers sitting outside in 90 different containers. They chill the water, bring it back to the data center, distribute it to all the chips, pull all the heat out, and then send it back.

This is both a way to cool the chips and an efficiency improvement. Going back to that sort of 3-vector thing, there’s memory bandwidth, FLOPs, and interconnect. The closer the chips are together, the easier it is to do high-speed interconnects.

That’s also a reason to use water cooling: you can put the chips right next to each other and get higher-speed connectivity.

I have to ask you: in one of your recent posts, there’s a section called “Cluster measuring contest.” There’s another word there, but I won’t say it. Who’s got the biggest cluster now, and who’s going to have the biggest one today?

The individual largest cluster is Elon’s. Elon’s cluster in Memphis has 200,000 GPUs.

Meta has around 128,000. OpenAI has 100,000. To be clear, other companies have more GPUs than Elon; they just don’t have them in one place. For training, you want them tightly connected. There are techniques people are researching that let you train across multiple regions, but for the most part, you want them all in one area so you can connect them with high-speed networking.

Today, Elon has 200,000 GPUs: 100,000 H100s and 100,000 H20s. Meta, OpenAI, and Amazon all have around 100,000, or a little less.

Next year, people are building much more. Anthropic and Amazon are building a cluster of 400,000 Trainium 2 chips, which are Amazon-specific chips. They’re trying to get away from NVIDIA. Meta and OpenAI have plans for hundreds of thousands, but by next year you’ll have 500,000-to-700,000-GPU clusters.

Those GPUs also consume much more power than existing ones. Hopper is 700 watts, and Blackwell goes to 1,200 watts. The power per chip is growing, and the number of chips is growing.

Elon said he’ll get to 1 million. Do you think that’s actually feasible?

I don’t doubt Elon. The filings he has for the power plants and Tesla battery packs make it clear that he has some crazy plans for Memphis. The permits and other documents are public record.

It’s not quite clear what the time scales are, but I never doubt him. He’s going to surprise us.

What’s the idea with these clusters? If you have 1 million GPUs, what percentage, in 2 or 3 years, is used for training? What percentage is used for pretraining, and what percentage is used for inference?

These mega-clusters make no sense for inference. You could route inference there and just not train, but most of the inference capacity is going to be distributed across different locations. You might have a 30-megawatt data center here, 50 megawatts there, and 100 megawatts somewhere else. You can just put inference in all of those places.

The mega-clusters—multigigawatt data centers—are where you want to train, because that’s where all your GPUs are collocated and you can connect them at extremely high networking speeds. That’s what you need for training.

With pretraining, the old scaling paradigm was that you increased the number of parameters, increased the amount of data, and the model got better. That doesn’t apply anymore because there isn’t much more data on the pretraining side.

There is video, audio, and image data that hasn’t been fully taken advantage of, so there’s a lot more scaling to do. A lot of people have transcripts of YouTube videos, and that gets you a lot of the data, but it doesn’t get you all the learning value out of the video and image data. There’s still scaling to be done in pretraining.

The post-training world is where all the FLOPs are going to be spent. The model is going to play with itself, do self-play, perform verifiable tasks, use computers in sandboxes, and maybe even do simulated robotics. All of these will be environments where compute is spent in “post-training.”

I think at some point we’re going to drop the “post” from post-training. It’s going to be pretraining and training, because for the bulk of the last few years, pretraining has dwarfed post-training.

With these verifiable methods, especially ones that can scale almost infinitely, like computer use and robotics—not just math and coding, where you can verify what’s happening—it seems like you can spend as much compute as you want on them.

That’s especially true as the context length increases, because the end of pretraining is when you increase the context length for these models. We talked earlier about how a long input is much easier to manage than a long output, and a lot of these post-training and reasoning techniques rely on a lot of sampling and increasingly long contexts.

Your compute efficiency effectively goes down. FLOPs are the standard for measuring it, but with reinforcement learning, you have to move your weights around in a different way than you do during pretraining and generation. It’s going to become less efficient, and FLOPs will be less useful as a term. Then, as the infrastructure gets better, it will probably go back to FLOPs.

All of the things we’ve been talking about are most likely going to be NVIDIA. Are there any competitors? Google?

Google. I’ve kind of ignored them. The story with the TPU is that it’s awesome. It’s great.

Google is a bit more tepid on building data centers for some reason. They’re building big data centers, don’t get me wrong, and they actually have the biggest cluster. I was talking about NVIDIA clusters; Google has the biggest cluster, period.

The way they do it is very interesting. They have 2 data center super-regions, where the data center isn’t physically one site. All of the chips aren’t physically on one site, but they’re within around 30 miles of each other.

They have 4 data centers in Iowa and Nebraska that are right next to each other. They also have a similar complex in Ohio.

Why doesn’t Google flex its cluster size and go to multi-data-center training?

There’s a good image in here. I’ll show you what I mean. It’s from SemiAnalysis. If you search for “multidata center,” you’ll see it.

What are we looking at here?

If you see this image, in the center there are these big rectangular boxes. Those are where the actual chips are kept. If you scroll down a little further, you can see the water pipes, the chiller cooling towers at the top, and a bunch of diesel generators.

The diesel generators are backup power. The data center itself is physically smaller than the water chillers. The chips are relatively easy to keep together, but cooling all the water for the water cooling is very difficult.

Google has very advanced infrastructure for the TPU. They’ve stamped out a bunch of these data centers in a few regions. If you go further down, this is a Microsoft site in Arizona. This is where GPT-5, quote-unquote, will be trained, if it doesn’t already exist.

If it doesn’t already exist.

Exactly. Each of these data centers is really closely collocated in the same region—Nebraska and Iowa, for example. They’ve connected them with extremely high-bandwidth fiber.

These are just a bunch of data centers, and the point is that Google has very advanced infrastructure, tightly connected in a small region. Elon wants to have the biggest fully connected cluster because it’s all in one building.

And he’s completely right on that.

He is. Google has the biggest cluster, but it’s spread across 3 sites, by a significant margin. You have to go across multiple sites.

Why doesn’t Google compete with NVIDIA? Why don’t they sell TPUs?

I think there are a couple of problems. One is that the TPU has been a way for Google to make search really cheap and build models for that use case.

A big chunk of Google’s TPU purchases and usage is for internal workloads: search, Gemini, YouTube, ads, and all these different applications. That’s where all their TPUs are being spent, and that’s what they’re hyperfocused on.

There are aspects of the architecture that are optimized for their use case but not for other use cases. One simple example is that Google open-sourced the Gemma model and called it Gemma 7B, but it actually has 8 billion parameters. That’s because the vocabulary is so large.

The reason they made the vocabulary so large is that the TPU’s matrix-multiply unit is massive. They’ve optimized for that, so they decided to make the vocabulary large too, even though it doesn’t make sense to do that on such a small model. It fits their hardware.

Gemma doesn’t run as efficiently on a GPU as Llama does. Conversely, Llama doesn’t run as efficiently on a TPU as Gemma does. There are all these aspects of hardware, software, and code design.

All of Google’s search models, ranking and recommendation models, and other AI models that aren’t generative AI have been hyper-optimized for TPUs forever. The software stack is extremely optimized, but most of it hasn’t been released publicly.

Small portions of it, like JAX and XLA, have been released, but when you’re inside Google and training on TPUs as a researcher, you don’t need to know much about the hardware. It’s pretty beautiful.

As soon as you step outside, a lot of those researchers leave Google and start companies because they have all these amazing research ideas. Then they realize that infrastructure and software are hard, and they have to use GPUs. If they try to use TPUs, they have the same problem, because they don’t have access to all of that code.

How do you convince a company whose golden goose is search, where it makes hundreds of billions of dollars, to start selling GPUs or TPUs? Google used to buy only a couple billion dollars’ worth of them. In 2023, I think they bought a couple billion dollars’ worth, and now they’re buying $10 billion to $15 billion worth.

How do you convince them that they should buy twice as many, figure out how to sell them, and make $30 billion? Who cares about making $30 billion?

Wouldn’t that $30 billion eventually exceed the actual search profit?

You’re always going to make more money on services than on hardware. Today, people are spending a lot more on hardware than on services, because hardware spending leads service spending.

But you’re investing. If there’s no revenue for AI, or not enough revenue, then it’s obviously going to blow up. People won’t continue to spend on GPUs forever.

NVIDIA is trying to move up the stack with software that it can sell and license. Google has never had that DNA of thinking, “This is a product we should sell.” Google Cloud does, but it’s a separate organization from the TPU team, which is separate from the DeepMind team, which is separate from the search team. There’s a lot of bureaucracy.

Wait, Google Cloud is a separate team from the TPU team?

Technically, TPU sits under infrastructure, which sits under Google Cloud. But Google Cloud, in terms of renting out infrastructure, and TPU architecture have very different goals in hardware and software.

The JAX and XLA teams don’t serve Google’s external customers, whereas NVIDIA’s various CUDA teams, including things like NCCL, serve external customers. JAX and XLA primarily serve DeepMind and search. Their customer is different. They’re not building a product for external customers.

Do you understand why AWS keeps winning against Azure and Google Cloud? Google Cloud is tiny, right?

Google Cloud is third. Microsoft is the second biggest, but Amazon is the biggest.

Microsoft deceptively includes Microsoft Office 365 and other enterprise-wide licenses in some of those numbers. In reality, the gulf is even larger, although Microsoft is still second.

Amazon is much bigger. Why? Using AWS is better, easier, and in many cases cheaper. It was also first.

But there are a lot of things that are first and don’t win.

It’s easier to switch to AWS than it is to switch away from AWS. There are also big fees for switching.

AWS generates over 80% of Amazon’s profit—maybe over 90%.

That’s insane. The distribution centers are just like, “One day we’ll decide to make money from this,” but they haven’t yet.

They make a tiny profit from them.

One day Amazon Prime will triple in price.

You would think they would improve the AWS interface, because it’s horrible. It’s clunky, but everybody’s is.

You would think so. Google’s interface is sometimes nice, but they also don’t care about anyone besides their top customers. Their customer service is terrible, and they have a lot less—

All these companies optimize for their big customers.

Amazon has always optimized for the small customer too. Obviously, it optimizes a lot for the big customer, but when Amazon started, it would go to random Bay Area events and give out credits. You could just put in your credit card and use AWS. That was the early days.

The business grew with its users. Why is Snowflake all over Amazon? In the beginning, when Amazon didn’t care about Snowflake, Snowflake was still using Amazon. Of course, now Snowflake and Amazon have a huge partnership, but Amazon’s user experience and quality are better.

A lot of the silicon Amazon has engineered gives it a lower cost structure for traditional cloud storage, CPUs, networking, and that kind of infrastructure. The same is true for databases. I think 4 of Amazon’s top 5 gross-profit products are database-related products, like Redshift.

Amazon has a very good silicon-to-user-experience pipeline with AWS. Google has awesome silicon internally: the TPU, the YouTube chip, and some other chips it has made. The problem is that they’re not serving external customers. They’re serving internal customers.

NVIDIA’s entire culture is designed from the bottom up to do this. There’s a recent book, The NVIDIA Way by Tae Kim, that details how the company looks for future opportunities and prepares its CUDA software libraries so that new generations of high-performance computing can rapidly evolve on CUDA and NVIDIA chips.

That’s entirely different from Google as a services business.

NVIDIA is a truly special company. The whole culture is optimized for that kind of thing.

Speaking of which, is there anyone who can challenge NVIDIA on the hardware side? Intel? AMD?

I really don’t think so. We went through a long process of working with AMD on training and inference, and its hardware is better in many ways than NVIDIA’s. The problem is that its software is really bad.

I think AMD is getting better, and it’s getting better faster, but the gulf is so large. It doesn’t spend enough resources on software, or at least it hasn’t historically. Maybe it’s changing its tune now.

For multiple months, we were submitting the most bugs. Why were we submitting the most bugs? AMD only cared about its biggest customers. It would give them a private image and so on. But I was just using PyTorch and wanted to use the publicly available libraries, and AMD didn’t care about that.

It’s getting better, but I don’t think AMD can catch up quickly. Intel is obviously in dire straits and needs to be saved somehow. It’s very important for American national security.

Can you explain the “obviously”? Why is Intel in dire straits?

Going back to earlier, only 3 companies can do R&D at the leading edge of semiconductor manufacturing: TSMC in Taiwan, Samsung in South Korea, and Intel in Hillsboro, Oregon.

Samsung is doing horribly, and Intel is doing horribly. We could be in a world where there’s only 1 company that can do R&D, and that company already manufactures most of the world’s chips. TSMC has been gaining market share anyway.

That’s critical. What happens to Taiwan affects the rest of the world’s semiconductor industry, and therefore the entire technology industry relies on Taiwan. That’s obviously precarious.

Intel has been slowly and steadily declining. It was on top in servers and PCs, but Apple developed the M1, NVIDIA is releasing a PC chip, and Qualcomm is releasing a PC chip. In servers, the hyperscalers are all making their own Arm-based server chips.

Intel has no AI silicon wins. It has very small wins. It never got into mobile because it said no to the iPhone, and all of these things compounded. It also lost its process-technology leadership.

Intel was ahead for 20 years, and now it’s behind by at least a couple of years. It’s trying to catch back up, and we’ll see if its 18A and 14A strategy works, where it’s trying to leapfrog TSMC.

Intel is also losing a lot of money. It just fired its CEO, even though he was the only person who understood the company well. He wasn’t the best, but he was relatively technical and pretty good compared with the alternatives.

Where does Intel make most of its money?

CPUs—PC and data-center CPUs.

Data-center CPUs are all going to the cloud, and Amazon, Microsoft, and Google are making Arm-based CPUs.

On the PC side, AMD has gained market share. NVIDIA is launching a chip, although it may not be successful. MediaTek and Qualcomm are launching chips, and Apple is doing well.

Intel could get squeezed in PCs, although PCs will probably stick mostly with Intel on the Windows side.

Let’s talk about the broad AI race. Who do you think wins? We talked about Google as the leader.

The default leader has been Google because of its infrastructure advantage. In the news, OpenAI is the leader. It has the leading and best model that people can use, and it has the most AI revenue.

OpenAI is winning. Who’s making money on AI right now? Is anyone making money?

In terms of accounting profit, Microsoft is making money, but it’s spending a lot of capital expenditures, which are depreciated over years.

Meta is making tons of money with recommendation systems, which are AI, but not with Llama. Llama is losing money for sure.

Anthropic and OpenAI obviously aren’t making money, because otherwise they wouldn’t be raising money. They have to raise money to build more. Theoretically, they are making money: you spend a few hundred million dollars on GPT-4, and it generates billions in revenue. Obviously, that’s making money.

But they have to continue researching to get compute-efficiency improvements and move down the curve to achieve the 1,200× improvement that was achieved for GPT-3. Maybe we’re only at a couple hundred times now, but with GPT-4 Turbo and GPT-4o, and another model that will probably be cheaper than GPT-4o at some point, the research costs a lot of money.

That’s the thing that isn’t talked about with the cost. When you’re referring to the cost of a model, it’s not just the training or the test runs. It’s the actual research and the people required to do things like reasoning.

Exactly. That’s the thing. Reasoning exists now, and they’re going to scale it. They’re going to do a lot more research.

People focus on the payback question, but it’s easy to say that GDP is humans and industrial capital. If you can make intelligence cheap, then you can grow a lot. That’s the dumb way to explain it, but it’s basically the investment thesis.

I think only NVIDIA is actually making tons of money, along with other hardware vendors. The hyperscalers are all making money on paper, but in reality they’re spending a lot more on purchasing GPUs.

You don’t know if they’ll still make this much money on each GPU in 2 years. You don’t know if OpenAI suddenly goes kaput and Microsoft has hundreds of thousands of GPUs that it was renting to OpenAI—GPUs that Microsoft paid for through its investment in OpenAI—but no longer has a customer for.

That’s always a possibility. I don’t believe it will happen. I think OpenAI will keep raising money, and others will keep raising money, because the returns will eventually be huge once we have AGI.

Do you think multiple companies will get there? I don’t think it’s winner-take-all.

Let’s not call it AGI, as if it’s a single day. It’s a gradual thing: a super-powerful AI with a gradually increasing set of features that are useful.

There are companies that will benefit from AI but not because they train the best model. Meta has so many avenues to benefit from AI across all of its services. People spend time on Meta’s platforms, and AI is a way to make more money per user per hour.

Google, xAI, Tesla, and Meta will benefit not directly from the language models, but from adding intelligence to the products they already sell. For Meta, that’s recommendation systems. For Elon, it could be the intelligence of Optimus, personalized robots in the home, and related products.

Elon thinks robotics is a $10 trillion-plus business. Maybe not soon, but who knows what robotics becomes.

Let’s do a TAM analysis. There are 8 billion humans. Let’s get 8 billion robots and pay them the average salary.

There we go: more than $10 trillion. If robots are everywhere, why does it have to be only 8 billion robots?

Exactly. I’m going to have 1 robot, and you’re going to have 20.

I see a use case for that.

The benefit would be in the products companies already sell, which is why OpenAI is in a trickier position. All of OpenAI’s value right now, as a brand, is in ChatGPT. For most users, there isn’t much reason for OpenAI to spend billions and billions of dollars on the next-best model when it could license Llama 5 for much cheaper.

ChatGPT is extremely valuable to OpenAI, but it could make more money from that than from the chat application alone.

The standard chat application doesn’t have tons of room to continue growing. The cost continues to collapse. DeepSeek V3 is the latest example, but eventually it will be supported by ads.

Meta already serves Llama 3 405B and probably loses money on it, but at some point the models will get so cheap that companies can serve them for free with ads. Google will be able to do that too, and Google has a bigger reach.

Chat isn’t going to be the only use case. Reasoning, coding agents, computer use, and all of that is where OpenAI has to go to make money in the future. Otherwise, it’s kaput.

Google and Meta have all these other products. OpenAI and Anthropic don’t, unless they’re so good at models that they keep winning.

It seems like OpenAI and Anthropic have to keep winning as capabilities improve.

Yes. Even if AI capabilities are improving rapidly, there’s still an advantage from data, products, and money. Google has data and other products. Meta has data and other products. There’s just a huge amount of money involved.

The whole idea is that human data is almost tapped out. What matters is self-play and verifiable rewards.

AWS doesn’t make a lot of money on each individual machine, and the same can be said for the most powerful AI platform. Even though API calls are cheap, there’s still a lot of money to be made by owning that platform.

There are a lot of discussions about tokens, tokenomics, and language-model APIs being the next compute layer or the next economic paradigm, kind of like energy and oil were. But you have to believe that APIs and chat aren’t where AI stops.

AI will be used for tasks, agents, robotics, and computer use. That’s where the value will be delivered—not through APIs and chat applications alone.

Is it possible that it all becomes a commodity, and you have the very thin wrapper—Perplexity, just joking?

There are a lot of wrappers making a lot of money. But if model progress isn’t rapid, then yes, it becomes a commodity.

DeepSeek V3 shows this, and the GPT-3 cost chart showed it too. Llama 3 8B is 1,200× cheaper than GPT-3. Anyone whose business model was based on GPT-3-level capabilities is dead. Anyone whose business model is based on GPT-4-level capabilities is also dead.

It’s commonly said that the best businesses being created now are predicated on models getting better. Those are wrappers riding the wave of model improvements.

In the short term, the company that could make the most money is the one that figures out what advertising-targeting method works for language-model generations.

Meta has hyper-targeted ads in the feed, but not within specific pieces of content. Google and Amazon use search ads. Amazon has been gaining a lot of search advertising.

With ChatGPT, it’s not clear how to place a high-quality ad within the output. If you can do that while model costs are coming down, you could get extremely high revenue per interaction. That revenue is completely untapped, and it’s not technically clear how to do it.

That’s the AdSense innovation. One day you’ll have an ad in a GPT output, and that’s going to make billions.

It could be very subtle. With voice mode, it could be some way of making the voice introduce certain things. It’s much harder to measure, and it takes imagination.

It also wouldn’t come off as shady, so you wouldn’t receive public blowback.

You would have to do it loudly enough that it’s clear it’s an ad, while balancing all of that. That’s the open question they’re trying to solve.

Anthropic and OpenAI may not care about it right now. Places like Perplexity, Google, and Meta care about it more. OpenAI and Anthropic are purely laser-focused on AGI, agents, and AGI. If they build AGI, they can make tons of money and pay for everything.

It’s predicated on the export-control question. If you think AGI is 5 or 10 years away, or less, and these labs think it’s 2 or 3 years away, your actions are going to be very different. Assuming they’re rational actors—and they mostly are—what you do when you think AGI is 2 years away is very different from what you do when you think it’s 5 or 10 years away.

Do you think agents are promising? We have to talk about this. This was the excitement of the year: agents are going to revolutionize everything.

“AI agents” is a generic hype term that a lot of businesspeople are using. The term agent is obviously overblown.

We’ve talked about reinforcement learning as a way to train for verifiable outcomes. An agent should mean something open-ended that solves a task independently and can adapt to uncertainty.

The term agent is being applied to things like Apple Intelligence, which we still don’t have after the last WWDC. Apple Intelligence is orchestrating between apps, and that type of tool use is something language models can do very well.

Apple Intelligence will probably come eventually. It’s a closed domain: your Messages app integrating with your Photos app, with AI in the background. That will work. A lot of software companies have described that as an agent to get into the narrative.

The question is: in what ways can we get language models to generalize to new domains and solve their own problems in real time? Maybe there will be a tiny amount of training while they’re doing this, with fine-tuning themselves or in-context learning, where you store information in a prompt and use learning algorithms to update it.

Whether or not you believe that will generalize to something like, “Book my trip to Austin in 2 days. I have these constraints,” and whether you can actually trust it, is an open question.

There’s an HCI problem. What’s your prediction there? My gut says we’re very far away from that.

I don’t know if you’ve seen OpenAI’s 5 levels. Chat is level 1, reasoning is level 2, agents are level 3, and there are a couple more levels.

It’s important to note that we were in the chat phase for a couple of years. We theoretically just got to reasoning, which will be here for a year or 2, and then agents.

People can try to approximate the capabilities of the next level, but agents are supposed to do things autonomously for minutes or hours at a time. Reasoning is doing things for tens of seconds and coming back with an output that I still need to verify, use, and check.

The biggest problem is the same as in manufacturing. There’s the whole Six Sigma idea of how many nines of reliability you get, and then you compound those nines.

If you multiply the number of steps that are Six Sigma, you get a very low yield. In semiconductor manufacturing, there are tens of thousands of steps. A reliability of 99.99999% isn’t enough, because you multiply that by that many steps and end up with something like a 60% yield, or even zero.

It’s the same thing with agents. When you chain tasks together, even the best language models don’t get 100% on particularly good benchmarks. They get a little below that because there’s a lot of noise. How do you get enough nines?

It’s the same thing with self-driving. We can’t have self-driving without it being extremely geofenced. Waymo is good, and even then it has teleoperators to make sure the cars don’t get stuck.

You can’t do that on the open web or the open operating system. There’s no structure. It’s a mess.

I’m always skeptical of any system tasked with interacting with the human world, because the human world is open and messy.

If we can’t get intelligence that’s sufficient to solve the human world on its own, we can create infrastructure like the human operators Waymo has used over many years to enable certain workflows.

There’s a company—I don’t remember its name—but that’s literally its pitch. It’s just going to be the human operator when agents fail. You call the company, and it fixes the problem through an API call.

It’s hilarious. There are going to be teleoperation markets when we get humanoid robots. Someone around the world is going to be happy to fix the fact that my robot can’t finish loading the dishwasher.

That will just be part of the Tesla service package.

I’m imagining an AI agent talking to another AI agent. One company has an AI agent that specializes in helping other AI agents.

If you can make things that are good at one step, you can stack them together. That’s why I think that if it takes a long time, we’re going to build infrastructure that enables it.

Look at the Operator launch. OpenAI has partnerships with DoorDash, OpenTable, and other websites. Those partnerships will let it climb really fast. Its model will get very good at those tasks.

That could be a network effect. More companies may want to make it easier for AI systems to use their services. Other companies will put blockers in place.

This is a story we’ve seen with the internet, and we see it now with training data for language models. Companies say, “No, you have to pay.” Businesses will work it out.

Airlines and hotels have a high incentive to make their sites work well, but they usually don’t. If you look at how many clicks it takes to order an airplane ticket, it’s insane. You can’t even call an American Airlines agent anymore. It doesn’t have a phone number.

It’s horrible on the interface front. I have an existential crisis every time I try to book an airplane ticket. I don’t think it’s going to be easy to build a robust AI agent that can deal with those websites when I, as a human, struggle with them.

Think about it this way: United has accepted the Starlink terms, which require it to provide Starlink for free, and users are going to love it.

What if one airline spends a year making its website work perfectly for AI systems? Every time someone asks an AI to book a flight, it uses that airline.

Or the airline could expose an API only to AI agents. If an agent queries it, the price is 10% higher, but the agent can see every flight and book any of them. The airline makes 10% more money.

If I’m willing to say, “Book me a flight to Austin,” and the agent does it, that could work.

Computers, the real world, and the open world are very messy. But if you define the problem in narrow regions, people will create very productive systems and ratchet down costs massively.

Robotics in the home will be much harder, just like self-driving, because there are a billion different failure modes. Agents that can navigate a certain set of websites and perform certain sets of tasks will be much easier.

You could take a photo of your refrigerator, upload your recipes, and have an agent figure out what to order from Amazon or a food-delivery service. That will be relatively quick and easy to do.

There will be a whole range of business outcomes, and people will find ways to make money.

These sandboxes already exist in research. People have built clones of all the most popular websites—Google, Amazon, and so on—to make it possible to train these systems.

OpenAI probably has them internally. DeepMind’s robotics team has had clusters of robots for years, where you interact with robots remotely. They have a lab in London, and you send tasks to it. The robot arranges blocks, and you do the research. There are people there to fix things, but we’ve turned these cranks of automation before.

You go from a sandbox to progress, then add one more domain at a time and generalize. In the history of NLP, instruction tuning and task-specific language models used to mean that one language model did one task.

Then, in the instruction-tuning literature, there was a point where you started adding more and more tasks together, and the model began to generalize to every task. We don’t know where on this curve we are.

For reasoning with reinforcement learning and verifiable domains, I think we’re very early. But we don’t know where the point is when you train on enough domains and suddenly more domains start working. You cross the generalization barrier.

What do you think about the programming context? Software engineering is where I personally, and where a lot of people, interact with AI the most. There’s a lot of fear and angst among current computer-science students, but that’s also where the most AI revenue and productivity gains have probably come from.

Whether it’s Copilot, Cursor, or standard ChatGPT, I know very few programmers who don’t use ChatGPT. Many of them have the $200 tier because it’s so useful.

In that world, we already see progress. SWE-bench, if you’ve looked at the benchmark made by some Stanford students, isn’t extremely hard, but it isn’t easy either. It takes someone with at least a few years of computer science or a couple of years of programming to do well on SWE-bench.

The models went from 4% to 60% in about a year. Where will they be next year? It will probably be higher. It won’t be 100%, because, again, those nines are very hard to achieve.

At some point, we’ll need harder software-engineering benchmarks. But software-engineering agents can be built faster than any other kind of agent because software engineering is a verifiable domain. You can unit-test or compile the code.

There are also many different ways for an agent to inspect an entire codebase at once. No engineer can really do that. Only the architects and truly senior engineers can think about the whole system and define what needs to be done. Then the agent can execute on it.

Software-engineering costs are going to plummet. One interesting consequence is that when software engineering becomes extremely cheap, you get very different markets.

In the United States, you have all these platform-as-a-service companies, like Salesforce. In China, nobody uses platform SaaS in the same way. Companies build their own stacks because software engineering is much cheaper in China, partially because of the number of STEM graduates.

Code language models have been adopted less in China because the cost of an engineer is much lower. But what happens when every company can cheaply and quickly invent its own business logic?

You stop using platform SaaS and start building custom-tailored solutions. You can change them quickly. Your business might become more efficient because you’re not dealing with the problems of a random platform-SaaS company whose product doesn’t work perfectly with your workflows.

There are also automation cases that don’t require AI. They just require logic to be built, and nobody has built it. All of that can happen quickly.

Software engineering is one domain. Another is industrial engineering. Chemical and mechanical engineers generally aren’t good at coding. Semiconductor engineers use tools that are 20 years old. A lot of the tools run on Windows XP, including ASML’s lithography tools. A lot of the analysis happens in Excel.

You can move 20 years forward with all the data you’ve gathered and do a lot better. You just need software-engineering skills to be delivered to the actual domain expert.

I’m extremely bullish on AI creating value in those domains.

The big picture is that I don’t think it’s going to be a cliff. A good example of how growth changes is when Meta added Stories. Snapchat was on an exponential trajectory, and then Meta added Stories and it flatlined.

Software engineers are still going up and to the right. AI will probably make that line flat. It’s not as if everyone is suddenly going to lose their job.

The supply corrects more slowly. The number of students is still growing, and that will correct over multiple years, but the number of jobs could just turn. Maybe in 20 or 40 years it will be well down, but there will never be a single moment when software engineers suddenly become useless.

The nature of what it means to be a programmer, and the kinds of jobs programmers do, will change. There needs to be a human in the loop.

You’ve talked about correcting the code, fixing things that are larger than the context length, debugging, reading the code, understanding the system, and steering it. The human has to say, “No, you missed the point,” and add more to the prompt.

It’s the human designing the perfect button. Google is famous for having people design buttons that are so perfect. How is AI going to do that?

It can give you all the ideas, and that’s great, but humans have something you could call taste. Humans can figure out what other humans enjoy better than AI systems.

That’s where preference comes from. Humans are the greatest preference generators. We’re also very good at judging which of 2 things is better.

That goes back to the core of reinforcement learning from human feedback and preference tuning. It’s hard to generate a good answer for many problems, but it’s easy to see which answer is better. That’s how we’re using humans for AI now: judging which answer is better.

That’s what software engineering could look like. A pull request review could present a few options and their potential pros and cons, and human judges could choose between them.

I would strongly recommend that programmers start using AI and embrace the role of supervisor and partner of the AI system, rather than writing everything from scratch or not learning to code at all.

There has to be a high level of programming expertise to manage increasingly intelligent systems.

I think it’s that, and then becoming a domain expert in something.

Exactly. If you look at aerospace, semiconductors, or chemical engineering, everyone is using old platforms and old software. The job of a data scientist is a joke in many cases, although in some cases it’s very real.

Bring the forefront of human capabilities to your domain. Even if that forefront comes from AI, you’re still at the forefront of your domain.

You have to be at the forefront of something, and then leverage the rising tide that is AI for everything else.

There’s so much low-hanging fruit everywhere, in terms of where software can automate or digitize things.

The legal system is another example. That’s why DOGE is exciting. You’ve spent time with a lot of the DOGE people, and government is so old-school. It’s begging for modernization: software, data organization, and all of that.

In that case, it’s partly by design. Bureaucracy creates and protects centers of power. Software breaks down those barriers, which hurts the people holding on to power but ultimately benefits humanity.

One thing we haven’t fully finished talking about is open source. First of all, congratulations. You released a new model.

I’ll explain what a Tülu is. A Tülu is a hybrid camel. When you breed a dromedary with a Bactrian camel, you get a Tülu.

Back in the early days after ChatGPT, there was a big wave of models called Alpaca, Aya, and so on, all named after various mammal species. Tülu is a brand that’s multiple years old, and it comes from that.

We’ve been working at the frontiers of post-training with open-source code. The first part of this release was in the fall, when we built on Llama’s open-weight models and added our fully open code and fully open data.

There’s a popular benchmark called Chatbot Arena. That’s generally the metric by which chat models are evaluated. Humans compare random models from different organizations.

If you looked at the leaderboard in November or December, among the top 60 models from 10 to 20 organizations, none had open code or data for post-training. Even fewer, or none, had pretraining data and code available.

Post-training is much more accessible right now. It’s still relatively cheap, and you can do it. The question is how high we can push the performance while giving people access to all the code and data.

That’s the motivation of the project. We draw lessons from Llama. NVIDIA had a Nemotron model where the recipe for its post-training was fairly open, with some data and a paper. We’re putting all of these things together to create a recipe that people can use to fine-tune models like GPT-4 for their own domains.

To be clear, in the case of Tülu, you’re taking Llama 3 405B, and Tülu is a series of recipes for post-training.

We’ve done multiple Tülu models over the years. We’re open-sourcing everything.

If you start with an open-weight model, the whole model technically isn’t open source, because you don’t know what Llama put into it. That’s why we have a separate project that we’ll get to.

For Tülu, we’re opening parts of the pipeline so people can zoom in and customize them. I hear from startups and businesses that they can take the post-training methods and apply them to their own domains.

We talk about verifiers a lot. We use reinforcement learning with verifiable rewards, or RLVR, which is somewhat similar to RLHF, and apply it to math.

The model we released today uses the Llama 3 405B base model from last year. We also have instruction tuning and preference tuning. The math work is interesting because it’s easier to improve a math benchmark.

There’s a benchmark called MATH. It’s a tough name, because the benchmark’s name is the area you’re evaluating. We’re researchers, not brand strategists.

The DeepSeek paper talked about this too: with a bigger model, it’s easier to elicit powerful capabilities using reinforcement learning, and then distill them down to a smaller model.

For the model we released today, we saw the same thing. At AI2, we don’t have a ton of compute, so we can’t train 405B models all the time. We did a few runs, and they tend to work.

That shows there’s a lot of room for people to experiment. We crushed Llama’s actual release. Our model is much better than the Llama instruct model.

You also said it’s better than DeepSeek V3.

On our evaluation numbers, we have a few extra months of work, but our numbers are much better than Llama’s instruct model. Our model is similar to DeepSeek V3.

We have a safety benchmark to understand whether the model will say harmful things. That’s what draws down most of the score.

Is it an amalgamation of multiple benchmarks?

Yes. In post-training, you choose the evaluations you care about. In academia and smaller labs, you have fewer evaluations. In companies, you usually have one domain you care about. In frontier labs, you might have 10, 20, or even 100 evaluations of specific things.

We choose a representative suite: chat, precise instruction-following, math, code, and safety. Precise instruction-following means things like, “Respond only in emojis.” You test whether the model follows unusual constraints.

Safety is 1 of those 10 or so evaluations. You’re trying to capture what the broader AI community cares about.

For example, our average validation score might be 80, including safety, and a similar score without safety. DeepSeek V3 might be around 79 without safety, and its safety score would bring it down.

So you beat it even ignoring safety?

Yes. Internally, I don’t want to win only because of how we shaped the validation benchmark. If people don’t care about safety in their model, safety can come downstream when they host it as an API.

Safety can be addressed at different points in an application. If you want to say you have the best recipe, you can’t gate it on things that some people might not want.

This is also a matter of timing. If we release a model later, we have more time to learn new techniques. We started this reinforcement-learning technique in the fall, and now reasoning models are very popular.

The next thing for open-source post-training is to scale up verifiers and data, and replicate some of DeepSeek’s results. It’s great that we have a paper to draw on, because that makes everything easier.

That’s what’s happening among academic and closed frontier research in AI.

Since you’re pushing open source, what do you think the future is? Do you think DeepSeek actually changes things by being open source, or open weight? Is it pushing the open-source movement in that direction?

This goes back to the license discussions. DeepSeek R1, with a friendly license, is a major reset. It’s the first time we’ve had a clear frontier model that is open weight and has a commercially friendly license, with no restrictions on downstream use cases, synthetic data, or distillation.

That hasn’t been the case in the history of AI since ChatGPT. There have been models that are off the frontier, or models with strange licenses that you can’t really use.

Meta’s license is pretty permissive, except for 5 companies, right?

There are also use-case restrictions in the Llama license. That gets to the question of what open-source AI is.

If you come from an open-source software background, you would say that the Llama license isn’t an open-source license, because it restricts specific uses. It used to forbid military use, although that was removed. Other restrictions involve things like child-abuse material.

The Llama license also has a terrible requirement that you have to name your model Llama if you touch the Llama model. If a company uses Llama, technically the license says it should say “Built with Llama” at the bottom of its application.

From a marketing perspective, that hurts. As a researcher, I can accept it. I can put “Llama” in all of our materials for a release. But this is why we need truly open models.

We don’t know DeepSeek R1’s data. The goal of the open language-model work is to keep everything open, with the data as close to the frontier as possible.

We’re constrained by compute and personnel. We rely on insights from people like John Schulman, who told us to do reinforcement learning on outputs. We can make big jumps, but it takes a long time to push the frontier of open source.

Fundamentally, open-source AI doesn’t have the same feedback loops as open-source software. We talked about open-source software for security and reuse. You build something once, and if you go to a new company, there are many benefits.

But if you open-source a language model, you have training data and code sitting around. It’s not easy for someone to come along, build on it, and improve it, because they need a lot of compute and expertise.

Until open-source AI has those feedback loops, it’s mostly an ideological mission. People like Mark Zuckerberg say America needs this, and I agree with him. But while the ideological motivation is high, we need to build the ecosystem around it.

What benefits do you get from seeing the language-model data? There isn’t much available. We’re going to try to launch a demo where you can look at an OLMo model and a query and see what pretraining data is similar to it.

That’s legally risky and complicated, but what does it mean to see the data an AI was trained on? It’s hard to parse. It’s terabytes of files, and I don’t know what I’m going to find in there.

That’s what we need to do as an ecosystem if people want open-source AI to be financially useful.

We didn’t really talk about Stargate. I’d love to get your opinion on what the new administration—the Trump administration—is doing to support AI infrastructure and the efforts of the different AI companies.

What do you think about Stargate? What are we supposed to think about Stargate? And does Sam have the money?

Stargate is opaque. It definitely doesn’t have $500 billion. It doesn’t even have $100 billion.

They announced the $500 billion number, and Larry Ellison, Sam Altman, and Trump said it. They thanked Trump, and Trump did issue executive actions that significantly improve the ability to build this faster.

One executive action allows you, on federal land, to basically build data centers and power plants. The permitting process is effectively gone, or you file after the fact.

If you’ve ever been to the Presidio in San Francisco, it’s a beautiful area. You could build a power plant and a data center there if you wanted to, because it’s federal land. It used to be a military base. Obviously, that would upset people, but it’s a good example.

Trump has made it much easier to do this. Texas also has the only unregulated grid in the country, so go Texas. The federal regulations are coming down, and the Electric Reliability Council of Texas enables people to build faster too.

Stargate is predicated on that. That’s why the announcement happened.

How they came up with $500 billion is beyond me. How they came up with $100 billion makes sense to some extent.

Stargate is in Abilene, Texas. The first $100 billion of it is a site with 2.2 gigawatts of power and around 1.8 gigawatts consumed. Oracle was already building the first part of this before Stargate came about. It had been building it for a year and had tried to rent it to Elon.

Elon thought it was too slow and needed to move faster, so he did the Memphis project. OpenAI was able to get the Oracle site through this strange joint venture called Stargate.

OpenAI initially signed a deal just with Oracle for the first section of the cluster. That first section is roughly $5 billion to $6 billion in server spending, plus around $1 billion in data-center spending.

If you fill out the entire 1.8 gigawatts with the next 2 generations of NVIDIA chips—GB200, GB300, and VR200—and fill it out completely, it’s roughly $50 billion in server costs.

Then you add data-center costs, maintenance, operations, and everything else. That’s where OpenAI gets to the $100 billion announcement. It talked about $100 billion as phase 1 of the project.

That’s $100 billion in total cost of ownership. It isn’t $100 billion in capital expenditure or investment. It’s around $50 billion in spending and another $50 billion in operational costs, power, rental pricing, and so on.

OpenAI is renting the GPUs from the Stargate joint venture. There will be future phases, and they’re looking at other sites that are even bigger than 2.2 gigawatts in Texas and elsewhere.

They’re not ignoring that, but the $100 billion figure is for phase 1, and I do think phase 1 will happen.

OpenAI doesn’t have the money for it. It has $6 billion from its last round and $4 billion in debt. There’s news that SoftBank may invest $25 billion in OpenAI, and that’s part of the plan.

OpenAI is on the line for $19 billion. It has only raised $6 billion in its last round and $4 billion in debt, so it doesn’t have the money at all. No money has been deposited yet.

OpenAI has zero dollars for the $50 billion portion, and it’s legally obligated to put $19 billion of capital expenditure into the joint venture. The rest will be paid by renting the GPUs from the joint venture.

Oracle has a lot of money. It’s building the first section completely and paying for it itself: around $6 billion in capital expenditure and $1 billion in total cost of ownership.

As for the rest, I don’t know how much Larry Ellison wants to spend. At any point, he can pull out. Nothing is fully signed, and it’s voluntary, but he could contribute tens of billions of dollars. He has the money, and Oracle has the money.

There’s also MGX, the sovereign wealth fund from the UAE, which technically has $1.5 trillion available to invest in AI. Again, I don’t know how real that money is.

SoftBank doesn’t have $2 billion in cash. It would have to sell down its stake in Arm, the leading CPU company. SoftBank has always wanted to do that; it just didn’t know where to redeploy the capital.

Selling down its Arm stake and investing in Stargate or OpenAI would make sense if it wants to do that.

In terms of secured funding, the first 100,000-GB200 cluster can be funded. Everything after that is up in the air.

I believe the money will come. That’s a belief. I believe they’ll release better models and be able to raise more money. But the reality is that Elon is right: the money doesn’t exist yet.

What does the U.S. government have to do with it? What does Trump have to do with it? Is he just a hype man?

Trump is reducing regulation so they can build faster. He’s allowing them to do it, because any investment of this size will involve antitrust issues. He’s going to enable the regulations that allow it to be built.

I don’t believe there are any U.S. government dollars being spent on this.

I think he’s also creating a general vibe that regulation will go down and this is the era of building. If you’re a builder and want to create or launch things, this is the time to do it.

We’ve had this 1.8-gigawatt data center in our database for over a year. We’ve been sending it to many clients, including the companies building multigigawatt data centers.

Seeing Trump talk about $100 billion and $500 billion could spur an even faster arms race. There’s already an arms race, but those numbers could cause investors to flood in.

I think you’re right. OpenAI, or Trump, is championing the idea that people are going to build more, and Trump’s actions will let people build more.

What are you excited about in the next several years—in terms of cluster build-outs and breakthroughs in AI? What’s the best possible future you can imagine over the next 2, 3, or 4 years?

It could be specific technical breakthroughs in post-training, or it could be the sheer size of the clusters.

The impressive clusters are interesting. I really enjoy tracking the supply chain: who’s involved in what, the numbers, the costs, who’s building what capacity, helping companies figure out how much capacity they should build, and winning strategic deals. That’s really cool.

Technologically, there’s a lot happening on the networking side that excites me, especially optics and electronics. We’re getting closer and closer to things like co-packaged optics and new forms of switching.

That’s internal to a cluster?

Internal to a cluster, yes. Multidata-center training is also exciting. People are putting so much fiber between data centers and lighting it up with so much bandwidth that there’s a lot of interesting work happening there.

Telecom has been boring since 5G, and now it’s exciting again.

Can you educate me a little about the speed of things? The speed of memory, the speed of interconnect, and the speed of fiber between data centers: are these orders of magnitude different?

Could we eventually converge toward a place where it all feels like one computer?

No, I don’t think that’s possible. It’s only going to get harder to program, not easier. It will become more difficult and complicated, with more layers.

The general image people have is a hierarchy of memory. On-chip memory is localized within the chip. You have registers, which are shared between some compute elements. Then you have caches, which are shared between more compute elements.

Then you have memory, like HBM or DDR memory, shared across the entire chip. You can have pools of memory shared between many chips, then storage, and so on.

The access latency across data centers, within a data center, and within a chip is different. You’re always going to have different programming paradigms. Programming this stuff is going to be hard.

The more elements you add to a task, the less you get strong scaling. If I double the number of chips, I don’t get 2× the performance. That’s just a reality of computing, because there are inefficiencies.

There’s interesting work being done to make scaling more linear, whether that means making the chips more tightly networked, using better programming models, or developing algorithmic techniques on the model side.

DeepSeek made some cool innovations because it was limited on interconnect but still needed to parallelize. Google and everyone else are also doing a lot of work on this.

That’s exciting on the model, workload, and innovation side. Solid-state transformers are interesting for the power side. There’s also work on batteries, cooling, and all the other layers.

If you look at every layer of the compute stack, from lithography and fabrication to optics, networking, power, transformers, cooling, models, and everything above that, there’s innovation everywhere.

Even air conditioners for data centers are innovating. Copper cables are innovating. There are improvements in how densely you can pack them.

Every layer of the stack, all the way up to the models, is advancing. Human progress is happening at a pace we’ve never seen before.

I’m imagining you sitting back in a lounge somewhere with screens everywhere, monitoring the supply chain and all these clusters.

There’s a big team. You do incredible work with SemiAnalysis, keeping your finger on the pulse of human civilization in the digital world.

It’s cool to watch.

It’s cool to feel it. Feel the AI—from meme to reality.

Nathan, are there breakthroughs you’re looking forward to?

I had a while to think about this while listening to Dylan’s beautiful answer.

Realistically, training models is very fun because there’s so much low-hanging fruit. What makes my job entertaining is that I train models and write analysis about what’s happening with models. There’s obviously so much more progress to be made.

The reason I share things is that I don’t trust people who say, “Trust me, bro. We’re going to make AI good. We’re the ones who are going to do it, and you can trust us. We’re just going to have all the AI.”

I would like a future where more people have a say in what AI is and can understand it. If AI is going to be the most powerful technology of my lifetime, we need a lot of people involved in making it.

Making it open helps with that. It makes it as accessible and as open as possible.

My read of the last few years is that more openness would help the AI ecosystem by giving more people a way to understand what’s happening, from researchers in non-AI fields to governments and everyone else.

That doesn’t mean openness will always be the answer. We should reassess what the biggest problem facing AI is and take a different angle if necessary.

For me, even from the user-experience perspective, there are moments of magic. Seeing the reasoning and the chain of thought is fundamentally beautiful.

It’s like putting a mirror up to ourselves and seeing intelligence being solved, which is the cliché goal of these companies. You start to understand why we humans are special, why the intelligence within us is special, and, for now, why we’re special in terms of consciousness.

The AI systems, for now, don’t seem to be conscious, so we get to explore that mystery. It’s incredible to explore questions I never imagined would be possible in my lifetime.

I remember watching Deep Blue beat Kasparov and never thinking this kind of AI would be possible in my lifetime. This really feels like AI.

I started with AI by teaching a simulated quadrotor to fly. It would fly up, hit the ceiling, stop, and catch itself. It was extremely stupid compared with what’s happening now.

Now you could probably use natural language to tell it to learn to fly, and it would generate the control algorithm required to do that. There are low-level blockers—we had to do some strange things for that—but it’s possible.

That goes back to our robotics conversation. When you have to interact with an actual physical world, it’s hard.

What gives you hope about the future of human civilization? Looking at the next 10 years, 100 years, or 1,000 years, how long do you think we make it? Do you think we’ll be around in 1,000 years?

Humans will definitely be around in 1,000 years. There are ways very bad things could happen, and there could be far fewer humans, but humans are very good at surviving.

A lot of things have happened before. We aren’t necessarily good at long-term credit assignment of risk, but when the risk becomes immediate, we tend to figure things out.

There are physical constraints on things like AGI’s recursive self-improvement to the point where it kills us all. For physical reasons, and because of how humans have figured things out before, I’m not too worried about an AI takeover.

There are other international things that are worrying, but there’s a fundamental human goodness and a desire to amplify it. We’re living in a tenuous time, but if you look at humanity as a whole, there have been times when things went backward and times when nothing happened at all.

We’re on what should be a very positive trajectory right now.

There seems to be progress, but just like with power, there are spikes of human suffering. We want to minimize those spikes.

Generally, humans are going to suffer a lot less. I’m very optimistic about that.

I do worry about technofascism arising as AI becomes more prevalent and powerful, and as the people who control it can do more and more.

Maybe AI doesn’t kill us all, but at some point every very powerful human is going to want a brain-computer interface so they can interact with AGI and all its advantages in more ways. They’ll be able to merge their minds with it, or leverage its capabilities better than anyone else.

It won’t be one person ruling everyone, but it could be a few people—hundreds, thousands, tens of thousands, or maybe millions—ruling everyone who’s left.

That’s the thing I worry about: human-machine amalgamations that give an individual human more impact on the world. That impact can be positive or negative.

Generally, humans have positive impacts on society, but it’s possible for individuals to have extremely negative impacts. AGI, at least as the labs define it—not a runaway sentient thing, but something that can perform many tasks efficiently—could amplify the capabilities of someone causing extreme damage.

For the most part, though, I think it will be used for profit-seeking motives, which will increase the abundance and supply of things and therefore reduce suffering.

Scrolling on a timeline and maintaining the status quo of the world could be a positive outcome. If I have food tubes and I’m scrolling and happy, that’s a positive outcome.

While expanding out into the cosmos—

This is a fun time to be alive.

Thank you for pushing the forefront of what is possible in human and AI, and thank you for talking today.

Thanks for having us.

Thanks for having us.

Thanks for listening to this conversation with Dylan Patel and Nathan Lambert. To support this podcast, please check out our sponsors in the description. And now let me leave you some words from Richard Feynman: “For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.” Thank you for listening, and hope to see you next time.