No Priors Ep. 127 | With SemiAnalysis Founder and CEO Dylan Patel

Hi listeners, welcome back to No Priors. Today I'm here with Dylan Patel, the chief analyst at Semi Analysis, a leading source for anyone interested in chips and AI infrastructure. We talk about open source models, the bottlenecks to building a data center the size of Manhattan, geopolitics, and poker as a tell for entrepreneurship. Welcome, Dylan.

Dylan, thank you so much for being here.

Thank you for having me.

I've been really looking forward to this conversation. You're such a deep thinker about the space. It's also very odd that you clearly have the Samsung watch.

Yeah. I got the laptop and the Fold.

The Fold. Yeah, yeah. Tell me more.

Part of the origin story is that I was moderating forums when I was a child, and my dad's first Android phone was the Droid. For some reason, I was obsessed with messing with it—rooting it, underclocking it, improving the battery life, all these things—because when we went on a road trip, there was nothing to do besides mess around on this phone.

I posted so much about Android that I became a moderator of r/Android on Reddit, and many other subreddits related to hardware, NVIDIA, Intel, and all this stuff. Because of that, I've just always had Android. I've had work iPhones before, but I really love Android.

If you're going to like technology, I'm not someone who pushes it, but get the best stuff. I have the Galaxy Watch Ultra, which I think looks cool, and the Fold phone. It's fun. It's obviously different and weird. No, no, iMessage is attractive.

What does it dominate at, besides the openness and hackability?

I don't even hack that much stuff anymore. What do you use your phone for? I think the main thing is that you can have Slack and email up on 2 different parts of your phone. That's probably the main thing.

Or you can actually use a spreadsheet on a folding phone. You cannot use a spreadsheet on a regular phone.

And that's not even an Android thing. Apple's folding phone next year will be able to do that just fine, and I'll have no argument then.

But I just like it. You know, people have their preferences. People are creatures of habit.

You got to look at the GPU purchasing forecast on a sheet on your phone.

Yes, I do. It's like someone's telling you numbers, and you're like, "Wait, this is slightly different from my number."

Okay, so we have a week of big rumored announcements coming up. Tell me your reaction to the OpenAI open-source model.

In theory, it's going to be amazing. I assume this is releasing after it's been released.

Yes.

The open-source model is amazing, guys. I think the world is going to be really shocked and excited. It's the first time America has had the best open-source model in 6 months, 9 months, or a year. Llama 3.1 405B was the last time we had the best model, and then Mistral took over for a little bit, if I recall correctly. The Chinese labs have been dominating for the last 6–9 months.

It'll be interesting, and it'll also be funny because the open-source model probably won't be the best for regular chat. It is more reasoning-focused and all these things, but it'll be really good at code, so I'm excited for that. Tool use will be interesting, although that's going to be confusing. How do you use the tools if you don't have access to open tool-use systems, but the model is trained to do so? That'll be interesting for people to figure out.

I think the last thing is that the way they're rolling it out is really interesting. They accidentally leaked all the weights, but no one in the open-source community has figured out how to actually run inference on it because there's some weird stuff in the model architecture, like 4-bit and the biases and all this other stuff.

What's interesting is that other companies drop the model weights and say, "Go make your own inference implementation." OpenAI is actually dropping the model weights and all these custom kernels for people to implement in inference. Everyone has a very optimized inference stack on day 1.

They work with partners on it, too.

Yeah, they're working with partners on this. This is very interesting because when DeepSeek drops, Together and Fireworks are like, "Yeah, we're the best at inference because we have all these people who are really good at low-level coding." Whether it be Fireworks with all their former PyTorch and Meta people, or Together with Tri Dao, Dan Fu, and all these super-cracked kernel people, they have higher performance.

In this case, OpenAI is releasing a lot of this stuff, so it's interesting for the inference providers, too. How do they differentiate now?

My premise on this is that, in the end, a lot of the model-optimization and performance layer is open-source, and it's a commodity. It'll end up being a fight at the infrastructure level, actually. All of these inference providers—as you mentioned, Fireworks, Together, Baseten, and such—compete on both dimensions, and the question is what's going to matter in the long term.

Why would these model-level software optimizations all be open? They haven't been open so far, and the advancements are so fast.

I think a bunch of them have been partially open, and I think OpenAI is also pushing for them to be open. There's a lot of force in the ecosystem to open-source, both from the NVIDIA level up and from the model providers down. Today, these providers all fight on that dimension.

Yeah.

They also fight on the infrastructure dimension, and I think infrastructure is going to end up being a bigger differentiator. You can't open-source your actual infrastructure. You just have to have the network and run it.

Yeah, that makes a lot of sense.

Although, today, I see such a wide variance among inference providers. The ones you mentioned, especially Together and Fireworks, are on the leading edge with their own custom stacks, all the way down to a lot of people who just take the out-of-the-box open-source software. I think there's no market for that.

Yeah, I agree. There's no market. It's commoditized. They have way worse margins than the people who are very optimized.

When you see NVIDIA trying to open-source all this stuff around Dynamo, and OpenAI and all these other people trying to open-source stuff, the level of optimization is also really large. It's caching between turns, caching tool-use calls, and all these other things.

It's not just a single-server problem. The DeepSeek implementation of inference is something like 160 GPUs. That's over $10 million of hardware, and that's just 1 replica. Then you'll have a lot of replicas, and you share the caching servers between them.

The orchestration of that, as well as the infrastructure, is a very large amount of infrastructure. I don't know. That's an interesting thought—that the optimization layer would be completely commoditized.

I think there's optimization at the single-node level, and then there's the system software where you can orchestrate this. Owning the abstractions for it, having people use your tools, and having more sophisticated teams do that optimization is a very ugly distributed-systems problem. I think that will matter.

Okay. Yeah, I could agree with that. I could agree that the single-node level is not necessarily commoditized.

Let's move out and a layer down. What does having access to an American open-source model—or just more and more powerful open-source AI models—mean for the application ecosystem?

I know a lot of people, and some enterprises, are really iffy about using the best open-source model. They're worried that there's nothing wrong with them today, but there's a worry that one day they will—

How do you check?

I mean, you don't. You can just vibe it out. They're competing with each other to release as fast as possible. DeepSeek, Moonshot, Alibaba, and all these other companies are competing to release as fast as they can.

The Alibaba teams in Singapore—I don't think they're putting Trojan horses in these models. There are some interesting papers that Anthropic did on trying to embed some things in models, and it ended up being detectable pretty easily.

Again, I don't know how to do that. I'm not too deep into interpretability and evals, but I just don't think that they are. It's a vibes thing.

Some people are worried that they could be, or they're just iffy: "I don't want to use a Chinese model." Fine, but now you're going to go use a service that's backed by a Chinese model, which is fine. They're fine with that; they just don't want to use the model directly.

I think it's interesting for some enterprises that are still stuck on Llama, but it's mostly interesting because it continues to move the commodity bar up, now with this tier being open-source.

And sure, it probably won't be drastically better than Kimi, but Kimi is so big and so difficult to run that people aren't running it, whereas OpenAI's models are relatively small, so you can run them without being a gigabrain at infrastructure. You end up with that commoditizing so much more of the closed-source API market. I think that's just going to be great for adoption, right?

Yeah, one of my hopes is for our companies that are doing more with reasoning. They're still blocked on cost and latency. Something I've found very interesting is that, when we've been trying to build a lot of alternative data sources for token usage—who's using what tokens, what models, where, and why—it's very clear that people aren't actually using the reasoning models that much in the API.

Anthropic has eclipsed OpenAI in API revenue, and their API revenue is primarily not thinking. It's Claude 4, but it's not in thinking mode. Code is the biggest use case that's skyrocketing, and the same applies to OpenAI and Deepbind. From what we see when querying big users and through other ways of scraping alternative data, the latency and cost issues are especially significant. The cost is just ridiculous.

You're exactly right. I guess my view is, you're not allowed to have a tech podcast without saying the words “Jevons paradox” now. I think the behavior is going to be that we see a lot of people use reasoning because it's so much cheaper to run if you take out a big piece of the margin layer and make it smaller. We have a lot of companies that are at scale using it, but it's so expensive that they restrain themselves.

Elad Gil

For a long time, OpenAI was charging more per token for the reasoning models—o1 and o3—than they were for GPT-4o, even though the architecture is basically the same. The weights are different, and there's some reason for it to be a little bit more expensive per token because the context length is, on average, longer. But, in general, it made no sense for it to be 4× the cost per token. That didn't make any sense. Then, finally, they cut it. For a long time, not only were there way more output tokens, but the price per token was also much higher, and they were just taking that as margin.

Because they could, right? Because they had the only thing out there.

Elad Gil

Yeah. Then DeepSeek dropped, and Anthropic, Google, and others started releasing models. It commoditized things quite a bit, but this is going to kneecap—cut everyone off at the knees—and bring margins down again.

Who has an API business, you mean?

Elad Gil

Yeah, for APIs for models that aren't at the super-leading edge. What do you think evolves in the Neocloud layer over time?

It's funny. Every day, we still find a new Neocloud. We have around 200 now, and we still find new ones every day, right?

Should they all exist?

Obviously not. To some extent, it depends on what the Neocloud business is. Today, there's quite a bit of differentiation between the Neoclouds. It's not just “buy a GPU and put it in a data center.”

Otherwise, you wouldn't have some Neoclouds with horrible utilization rates, and you wouldn't have some Neoclouds that are completely sold out on 4-, 5-, and 6-year contracts. CoreWeave, for example, doesn't even quote most startups, or they just give them a stupid quote because they don't want the business. They want a long-term contract, which a lot of people don't want to sign.

There's quite a bit of differentiation in the financial performance of these Neoclouds, time to deploy, reliability, and the software they're putting on top. Many of them can't even install Slurm for you. It's like, what are you doing? You should have some sort of—

So, very low-level hardware management.

Yeah, it's very low-level. To some extent, from the investor side, we see a lot more debt and equity flowing in from the commercial real estate folks. Commercial real estate has been really poor over the last few years, so they've been starting to pour money into the cloud space.

Obviously, the return profile is quite different because it's a short-lived asset versus a longer-lived asset. But at the end of the day, these companies are okay with a 10% or 15% return on equity. Over time, that falling is not okay for venture capital, and yet a lot of these Neoclouds are backed by venture capital.

A lot of these companies will fail, either because it no longer makes sense for them to continue getting venture funding, or because they end up getting outcompeted because they just can't get their utilization up, unlike some other clouds, like CoreWeave, Crusoe, and others. There's sort of a rock and a hard place for 100 of these Neoclouds, and there are many of them who are like, "Oh no, I purchased these GPUs, I have a loan, it cost me this much, and because my utilization is here, I'm burning cash."

Right? And they should, at the very least, not be burning cash, right?

Some of them are desperate to sell their remaining GPUs, so they go out to companies and give them insanely low deals. There are some startups that I really commend because they've figured out how to get the desperate Neoclouds to give them GPUs. But those Neoclouds are going to go bankrupt at some point because their cash flow is worse than their debt payments.

At the end of the day, there's going to be a lot of consolidation. There is going to be differentiation, too. There's a lot of software today, but we have this thing called ClusterMAX, where we review all the Neoclouds and major clouds. Some of these Neoclouds are actually better than Amazon, Google, and Microsoft in terms of software.

In terms of uptime and availability, or however you—

Yeah, uptime, availability, reliability, network performance—there's just a variety of things. They don't have all the old baggage. But the vast majority are worse. We measure across a bunch of different metrics, including the ones I mentioned, security, and so on and so forth.

Our vision for ClusterMAX is that it starts at a really low stage today, which is: does the cloud work, and how long does it take the user to get a workload running? You have Slurm installed, or you have Kubernetes installed, your network performance is good, your reliability is good, and it's secure. These are table stakes.

What we consider gold or platinum tier today will just be table stakes in 6 months, a year, or a couple of years. There'll be a whole layer of software on top, and then the question is: do Neoclouds build this software?

Together AI and Nebius are offering inference services on top. They're saying, “Hey, we actually want to provide an API endpoint, not just rent GPUs.” CoreWeave is rumored by The Information to be attempting to buy Fireworks for the same reason.

Do you move up, or do you just slide down into “I'm making commercial real estate returns”? Or do you have to go crazy? Crusoe is saying, “We're going to build gigawatt data centers.” There are only a few companies doing that, so there's no competition there.

It's very different. You either have to go really big, you need to move into the software layer, you just make commercial real estate returns, or you go bankrupt. Those are the paths for all Neoclouds, I think.

Elad Gil

I really have to believe there's a reason for these companies to exist. My simple framework is that the software layer is really hard for people coming from operations to try to build, right? There's actually a lot of very specialized software, so I think people will buy or partner into it.

But if you think about other inputs, it could be that I'm very good at finding and controlling power agreements. It could be that I build at a scale other people are incapable of doing, as you mentioned—

Which is sort of what, like—

Or, like, what warrants me to exist, right? I can't think of a lot of arguments beyond that. I would agree with you that eventually we're going to see consolidation, either in this layer, or commoditization by the inference providers.

In the meantime, there's a lot of lunch to eat from Amazon, Google, and Microsoft, which continue to charge absurd margins for their compute because they're just used to doing that in the CPU world.

Yeah.

Their return on invested capital is extremely high on CPUs and storage. To assume that it can translate over to GPUs is a bit of a fallacy, which is why a lot of these companies are moving in.

In standard cloud, there's a lot more software that people can't just build out of nowhere. Yes, EC2 is a product that's pretty simple, but block storage and all these other things are actually quite difficult to do well at scale, as Amazon does. That's what enables them to charge these absurd margins on standard compute.

But now it's like, well, the cloud doesn't actually generate any software that the end user actually uses. Sure, I need Slurm and Kubernetes, but then I'm just using PyTorch, which is open source. I'm using a bunch of NVIDIA software, maybe, or I'm using open-source models. I'm using vLLM and SGLang, which are open source.

You just go down the list, and there's actually no software that the cloud can provide to deserve the margins that Amazon and Google's clouds have today if you're just an infrastructure provider.

Elad Gil

I think there is software that the cloud can provide, but the major clouds have not delivered that software.

Agree. Agree.

Okay. Same same, because it’s really hard to do this stuff, right? There’s no reason every single startup needs to have multiple people dedicated to infrastructure, figuring out how to run models, and managing their SLAs. Their reliability is just so low, right? So many random SaaS providers that are AI—they have GPUs and an open-source model, and it works great, except sometimes it fails and then it’s down for 8 hours. Why? This shouldn’t be a problem. It should be something you should just be able to pay away.

I feel like the multi-trillion-dollar question that you have thought about for perhaps longer than almost anyone else is: What does it take to actually challenge NVIDIA? You know, asking for a friend, what would it take?

The simple way to put it is that it’s a 3-headed dragon, right? They’re actually just really, really good at engineering hardware and GPUs. That is difficult. They’re really, really good at networking, and then I would actually say they’re okay at software, but everyone else is just terrible. No one else is even close on software. I guess in that argument, you can say they’re great at software, but actually installing NVIDIA drivers is not always easy, right?

Elad Gil

Well, there’s great, and there’s also just—well, there’s 20-plus years of work in the ecosystem, right?

Yeah. There’s today’s capability and usability, and there’s just this mass of libraries.

Yeah. So I think NVIDIA is really hard to take down because of those 3 reasons. It’s like, okay, as a hardware provider, can I do the same thing as NVIDIA and win? No. They’re an execution machine, and they have these 3 different pillars. I’m sure they have a lot of margin, but you have to do something different, right?

In the case of the hyperscalers—Google with TPUs, Amazon with Trainium, Meta with MTIA—they are making a bet of, “I can actually do something pretty similar to NVIDIA if you squint your eyes.” Now, Blackwell and TPUs are starting to converge. The NVIDIA and TPU architectures are actually converging: the same memory hierarchies and similar sizes of systolic arrays. It’s actually not that different anymore. It’s still quite different, but hand-waving, it’s pretty similar. Trainium and TPUs are very similar architecturally.

The hyperscalers are not doing anything crazy, but that’s okay because they can just do the margin game. That’s fine. But for a chip company to try and compete, they must do something very unique. If you do something unique, all your energy is focused on that one unique thing, but on every other vector, you’re going to be worse.

Are you going to be at the latest process node as fast as NVIDIA? No. Okay, that’s like 20% to 30% on cost, performance, and power, right? Are you going to be on the latest memory technology as fast as NVIDIA? No, you’ll be like a year behind. Great. Same penalty. Are you going to be the same on networking? No. You just stack all these penalties up. It’s like, oh, wait, your unique thing can’t just be 2 to 4 times faster. It has to be way faster.

But then the problem is, if you really look at it simplistically, a FLOP is a FLOP, right? Again, this is super simple, but there’s not 10 times you can get out of doing a standard von Neumann architecture on compute efficiency. In which case, you do all of these things that NVIDIA will engineer better than you because they have a team of 50 people working on just memory controllers and HBM, and thousands of people working on networking. Do they just cut you by a thousand? And that's like, oh, actually, what would have been 5 times faster is now only 2 times faster. Plus, if I misstep, I’m 6 months behind and now the new chip is there, right? And you’re screwed.

Or supply chain, or intrinsic challenges with getting other people to deploy it, or rack deployments—there are all these supply-chain challenges, right? Literally, in Amazon’s most recent earnings, they said their chip architecture is not aggressive. Their rack architecture is very simple; it’s not that aggressive. They were like, “Yeah, we have rack-integration yield issues,” which is why they blamed their miss on AWS for their Trainium not coming online fast enough because of rack-integration issues.

When you look at the architecture—we have an article on it—it’s not that crazy. It’s like what Google was doing 4 or 5 years ago, right? And it’s like, oh, wait, supply chain is hard, and Amazon couldn’t get everything in the supply chain to work. Therefore, they missed their AWS revenue by a few percent, right? Which caused the whole stock market to freak out.

But there are so many things that can go wrong in hardware, and the timescales are so long. The last thing is that model architecture is not stagnant. If it was, NVIDIA would optimize for it. Model architecture and hardware—software-hardware co-design—is the thing that matters, right? These 2 things, you can’t just look at one individually, right? There’s a reason why Microsoft’s hardware programs suck: They don’t understand models at all, right?

Meta’s chips actually work for recommendation systems and are deployed for recommendation systems because they can do hardware-software co-design. Google’s awesome because they do hardware-software co-design. Why is AMD not catching up despite being awesome at hardware engineering? Well, yeah, they’re bad at networking, but they also suck at software and they can’t do hardware-software co-design.

There are much deeper reasons why you can get into this, but you have to understand the hardware and the software, and they move in lockstep. Whatever your optimization is doesn’t end up working.

One example is all of the first-wave AI hardware companies: Cerebras, Groq, SambaNova, and Graphcore. All of them made a very similar bet—not that they were very different architecturally, right?

Some of these are architecturally pretty weird, relatively.

Right, they’re architecturally pretty weird, but they made the same bet on memory versus compute. They decided to have more on-chip memory and lower bandwidth off-chip, because that was the trade-off they decided to make.

All of them had way more on-chip memory than NVIDIA. NVIDIA’s on-chip memory has not really grown much from A100 to H100 to Blackwell; it’s up 30% in 3 generations, whereas these guys had 10 times the on-chip memory, all the way back when they were competing with A100 or even the generation before. But that ended up being a problem because they were like, “Oh, yeah, we could just run the model on the chip. We can put all the weights on there, and then we’ll be so much more efficient.” Then the models just got way too big, right? Cerebras was like, “Oh, wait, but our chip is huge. Oh, wait, but still, the model’s way too big to fit on it.”

This is very simple, right? The same thing’s happening in the other direction. Some companies are like, “Oh, we’re going to make our systolic array—your compute unit—super, super, super large because, let’s say, Llama 70B is an 8K hidden dimension, and your batch and all that. It’s a pretty large matmul. Oh, great. Okay, we’ll make this chip.”

And then all of a sudden, all the models get super, super sparse, right? The hidden dimension of DeepSeek’s models is really tiny because they have a lot of experts, right? Instead of 1 large matmul, it’s a bunch of small ones. You do routing, right? And all of a sudden, if I made a really, really large hardware unit but I have all these small experts, how am I going to run it efficiently?

No one predicted that the hardware would go that way, but it ended up going that way. This is actually the case with at least 2 of the AI hardware companies today. I don’t want to shut them just because—let’s be friendly—but this is clearly what’s happening, right?

So you can make a hardware bet that will actually be way better on today’s architectures, but then architecture evolves, and the generality of NVIDIA’s GPUs—or even TPUs and Trainium—is more general as an architecture, but then it doesn’t beat NVIDIA by that much, right? In which case, they’re just going to destroy you: They’re 6 months or a year ahead on every technology because they have more people working on it and their supply chain is better, right?

It’s kind of really tough to make the architecture bet and have the models not just go in a different direction that no one predicted, because no one knows where models are headed, right? Even if you could get Greg Brockman, and he might have a good idea, I’m sure he doesn’t even know what models will look like in 2 years. There’s got to be a level of generality, and it’s hard to hit that intersection properly.

I’m very hopeful people compete with NVIDIA. I think it’ll be a lot more fun. There’d be a lot less margin eaten up by the infrastructure; there’d just be a lot more deployment of AI, potentially, if someone was able to compete with NVIDIA effectively. But NVIDIA charges a lot of money because they’re the best, and if there was something better, people would use it, but there isn’t.

And it's just really hard to get better than them.

I mean, you have to give the first-generation AI hardware companies some credit because they made a secularly correct decision about the workload. But then the architectural decisions ended up being hard to predict correctly, right? Then you have the cycle of NVIDIA innovation, which is really hard to compete with—both hardware and, as you said, supply-chain issues.

Elad Gil

Even just putting together servers is hard.

Yes. I think the thing that people oversimplified with the current generation of AI chip startups was that they were saying, “We're betting on transformers.” It's a lot more complicated than that in terms of workload at scale and the continued evolution of model architecture. It's also not exposed, so if you're not working with the SOTA labs from the beginning, you can't make predictions, because nobody can make a lot of predictions right now.

Yeah.

With no other changes happening, I can't make that bet right now. One of the interesting things about OpenAI's open-source models is that they have all their training pipelines, but on a quite boring architecture, right? It's not their crazy, cool architecture advantages that they have in their closed-source models, which make them better for long contexts, more efficient KV-cache usage, and all these other things they're doing.

They're doing it on a standard model architecture that's publicly available. They intentionally made the decision to open-source a model with a boring architecture that's pretty much open-source already. People have already done all these things and kept all the secrets internal that they wanted to keep. What's in there? Are they even doing standard scaled dot-product attention? Probably, but there are probably a lot of weird things they're doing that don't map directly to hardware.

Like you mentioned, Transformer chip architecture is a lot more complicated than just saying, “Oh, it's optimized for transformers,” because so is an NVIDIA chip and a TPU, and their next generation is more optimized for it. They take steps toward it; they don't leap. But as long as they're close enough that you are architecturally optimized for the workload, they'll beat you because of all the other reasons.

I think your description of how a chip startup might win, or how any vendor might win by specializing, is actually really hard in this era. Generalization may continue to win to a degree.

Elad Gil

And it happened with all the edge hardware companies, too. We talk about the first-generation AI hardware companies for data centers. There were a handful, but for the edge, there were 40 or 50, and none of them are winning because it turns out the edge is just: take a Qualcomm chip or an Intel chip that's made for a PC or smartphone and deploy it on the edge, right? That ended up being way more meaningful.

So it ends up being the incumbents. They can take steps toward what you're going for, and if you didn't execute perfectly, or if the models did change the architecture away from what you thought it would be, you end up failing.

If you had to make a bet that something becomes competitive, what is the configuration or company type that does that?

I don't want to show any company that I've invested in or anything like that, and so therefore, this is not investment advice. No, but I would just say that I probably think AMD GPUs or Amazon's Trainium will be more likely to be the best second choice for people, or Google TPU, of course. But I think Google is just more interested in it for internal workloads.

I think those will be much more likely options to succeed than a chip hardware startup. Yeah, but I really hope they do, because there's some really cool stuff they're doing.

If we zoom out to the macro and think about just the scale of hardware and data-center deployment for these workloads, people talk a lot about the operational constraints on building data centers of this size. The power constraints, in particular, are very interesting in how they practically show up. Is it generation at scale and cost? Is it grid issues? How should more people in technology understand this?

Yeah. Supply chain is always fun because people want to point at one thing as the issue, but it always ends up being that these things are so complicated. If one thing were solved, you could increase production another 20%, and then something else would be the issue.

Or, for Company A, it's actually because their supply chain is this—that's the issue—and for Company B, this is the issue. But that's sort of in generalities.

Noahpinion had a really fun blog about, “Is this AI hardware buildout going to cause a recession?” I think it's actually funny because you could flip the statement and say, “Actually, the U.S. economy would not be growing that much this year if it weren't for all the AI buildouts and, as a result, data-center infrastructure.”

As a result, electricians' wages have soared. As a result, power deployments and other capital investments, which have 15- to 30-year lifespans, are being made, and all of this capex is, in turn, actually growing the economy. Maybe the economy wouldn't even be growing much or at all if it weren't for all of these investments.

Elad Gil

One thing that is perhaps overlooked from the White House AI Action Plan was the view that we're going to build these AI data centers in the United States. We're actually going to need a lot of general investment beyond the GPUs and the power, which are everybody's first 2 items, into labor, for example.

For simplicity's sake, you could say it's the size of Manhattan, and we have to run it. It's a new system with changing topology and a very high degree of relatively novel hardware with failures, and lots of networking. It kind of feels like we need to have a bunch of new capacity from a labor or robotics perspective.

In 2023, it was very simple: Nvidia can't make enough chips. Okay, why can't Nvidia make enough chips? Oh, CoWoS, right? Chip-on-wafer-on-substrate packaging technology. And I was like, oh, HBM, right? Those were—it was very simple in 2023 and 2024. All these tools were involved in that supply chain. It was great, but then it very quickly became much murkier, right? Then I was like, oh, data centers are the issue.

Oh, okay, we'll just build a lot of data centers. Oh, wait, substation equipment and transformers are the issue. Oh, wait, power generation is the issue. It's not like the other issues went away, right? CoWoS is still a bottleneck, and HBM is still a bottleneck. Optical transceivers are still a bottleneck, but so are power generation and data-center physical real estate.

Like I mentioned, Meta is literally building these temporary tent structures to put GPUs in because building the building takes too long and it takes too much labor. As you mentioned, labor—that's one way they were able to remove part of a constraint. They're still constrained on power, and they had to delay the bring-up of some GPUs in Ohio because the grid in Ohio had some issues. The utility had issues with bringing on a generator or something.

Oh, okay, great. We'll buy our own generators and put them on-site. Oh, wait. Now there's an 8-year backlog—or whatever, a 4-year backlog—for GE's turbines.

Yeah.

Oh, okay. I'm Elon. I'm going to buy a power plant from overseas that's already existing. I'm going to move it in. Okay, great. Now there are permits and people protesting against me in Memphis. There's a bajillion things that could go wrong.

Labor is a huge one. I've literally had people in pitches say, “No, no, no. We've already booked all the contractors, so no one else is going to be able to build a data center of this magnitude in this entire area besides us—”

Because we took all the people.

We took all the people. They're going to have to fly them in. But it's like, okay, fine. You can fly them in, but there are just not that many electricians in America. As a result, we've seen wages rise a lot for people building data-center infrastructure.

There's a group of Russian guys who used to work for Yandex, Russia's search engine, who now live in America and wire up data centers. They get paid a ton, and they get paid bonuses for being faster. Therefore, they do certain drugs to be able to finish the build-outs faster because they get bonuses based on how fast they build it, right? There is crazy stuff going on to alleviate bottlenecks, but there are bottlenecks everywhere.

It really just takes a hyper-competent organization tackling each of these things and creatively thinking about each of them, because if you do it the lame, old way, you're going to lose and you're going to be too slow. Which is why Microsoft is not building Stargate for OpenAI, right? It's because it would have just been too slow, and they're doing it the lame, old way.

You have to go crazy. That's why Microsoft rents from CoreWeave a ton, right? Because, oh wait, we need someone who can do things faster than us, and oh look, CoreWeave is doing it faster. Now OpenAI is going to Oracle and CoreWeave and others, Nscale in Finland, and all these other companies all around the world—the Middle East, G42—anywhere and everywhere they can get compute.

You put your eggs in many baskets, and whoever executes the best will win. This infrastructure is very, very hard. Software has fast turnaround times. It's still hard—software is not easy—but the cycle time is very fast: try something, fail, try something else.

It is not for infra, right? What has xAI actually done to deserve its prior funding rounds? They haven't released a leading-edge model, right? And yet their valuation is higher than Anthropic's today, right? At least Anthropic is raising, but whatever.

It's Elon, A. And B, they've tackled a problem creatively and done it way faster than anyone else, which is building Colossus. That's commendable because that is part of the equation of being the best at models, right?

Yeah, besides the talent.

Yeah, and Elon is known for being able to get talent. There is so much complexity in infrastructure that it would be nice to say there is one thing, but the White House action plan lists a lot of things.

I want to know, concretely, how do we solve the talent issue? There aren't enough people in trade schools. The pay will go up and that will help, but the timescale for that is too slow. Do we somehow import labor? That's how the Middle East is building all its data centers: they're just importing labor. Or is there something more intelligent we could do—robotics, right?

I just realized today—you told me just now about a company I'd seen or angel-invested in, and you led the round, right? It's really cool for data-center optimization and automation. There are all sorts of interesting problems on the infrastructure layer that could be tackled, and tackled creatively.

Speaking of the policy and geopolitical implications here, what do you think about the White House's implication that America needs to export the AI stack or control important components of it? Is it better for us to export NVIDIA chips than to foster a new industry? Is it better for us to have a globally leading open-source model? What actually makes sense to you there?

I want to tell a crazy story. I was in Lebanon for a week. It...

It was a good start.

This is completely unrelated, but it just popped into my head. I think it will be entertaining. I was in Lebanon with a few of my friends—2 Indian people, 2 Chinese people, and a Lebanese person, right?

These 12-year-old girls walked right up to the Chinese woman who was with us, my friend, and said, “Oh my God, your skin is so beautiful. Do you like sushi?” Fine, you’re just ignorant. But what was really interesting is that when they asked where we were from and we said, “San Francisco,” they said, “Do people get shot in the streets?” because their entire worldview was built from TikTok—

Okay.

—politics. And when you think about the global propaganda machine that is Hollywood, it’s not intentional; it’s just that American media is pervasive. It built such a positive image of America. Now, with monoculture broken and everything more social-media-based, a lot of the world thinks people in America are getting shot all the time, that it’s really bad, that people have bad lives and are working all the time, and that it’s unsafe.

Europe has a certain view of America, and I don’t think it’s accurate. A random Lebanese 12-year-old had a really negative view of America. They liked America, and they loved Target for some reason because some influencers posted TikToks about Target, but they had negative views of America. It’s like—

From a sense of what is important, the world should still run on American technology, right? They generally do still, in terms of the web, although ByteDance’s TikTok has broken that to a large degree.

But in this next age, do you want them to run on Chinese models, which now have Chinese values and then spread Chinese values to the world, or do you want them to have American models that have American values? You talk to Claude and it has a worldview, right? I don’t know if you want to call that propaganda or what, but there is a worldview that you’re pushing, right? So I think it makes sense that we need that worldview espoused now. How do you do that?

The prior administration and the current administration had different viewpoints on this. The prior administration said, "Yes, we would love for the whole world to use our chips, but it has to be run by American companies." And so it was like, "Microsoft, Oracle, we're cool with you building shitloads of capacity in Malaysia. We don't want random other companies doing it in Malaysia." So the prior diffusion rule had a lot of technical ways in which you could have these licenses and all this. It was very hard for random small companies to build large GPU clusters, but it was very easy for Microsoft and Oracle to do it in Malaysia. Of course, the current administration tore that up and they have their own view on things. I think there were a lot of things wrong with the diffusion rules. They were just too complicated and pissed a lot of people off. Now they have a different view, which is what they did in the Middle East with the deal they signed. Most of those GPUs are being operated by American companies or rented to American companies. G42 is operating them but renting them mostly to OpenAI and such, or Amazon and Oracle and others are operating the GPUs themselves in the Middle East. That's effectively the same thing but in a very different way. That is still a view, which is that we want America to be as high in the value stack as possible. If we can sell tokens or if we can sell services, we should.

Okay, but if we can’t sell the service, let’s at least sell them tokens. Okay, we can’t sell them tokens; at least sell them infrastructure, whether it be data centers, renting GPUs, or just the GPUs physically. It sort of makes sense in the value chain: give them the highest-value, highest-margin thing, where we capture most of the value, and squeeze it down to where, actually, for the bottom of the stack—the tools to make chips—maybe you shouldn’t sell.

Current export controls and policy dictate that. Yes, it’s better to sell them services, but sell them both. Give them the option, let us compete, and don’t let anyone else win.

There is a push forward. There are a lot of different sanctions, and many of these are not well enforced or have holes. A lot of people in San Francisco, frankly, don't sell China any GPUs, but then they cut off rare-earth minerals. Ostensibly, most people think that the deal was that you get GPUs and also EDA software, because the administration banned EDA software for a little bit, just for a few weeks, basically until China was like, "Okay, we'll ship rare-earth minerals." You can't just ban everything because China can retaliate. If they banned rare-earth minerals and magnets and such, car factories in America would have shut down, and the entire supply chain there would have had hundreds of thousands of people not working.

Yeah.

There is a push and pull here. Do I think China should just have the best NVIDIA GPUs? No, that would suck. But can you give them no GPUs? No, they’re going to retaliate. There is a middle ground.

Huawei is eventually going to have a lot of production capacity, but there are ways to slow them down, right? Properly ban the equipment, because there are a lot of loopholes there. Properly ban the subcomponents of memory and wafers, because Huawei is still getting wafers in Taiwan from TSMC through shell companies, right? There are a lot of enforcement challenges because parts of the government are not funded properly or are not competent enough, and have never been competent.

So how do you work within this framework? Fine, we should sell them some GPUs so that they—that kind of slows them down from a Huawei standpoint, although not really—but it also gets us the rare-earth minerals back. Don’t sell them too many. How do you find that massive gray line? That is what the administration is grappling with, in my view.

Implied in that opinion is your belief that China is going to be able to build NVIDIA-equivalent GPUs eventually if forced?

Maybe not equivalent.

Sorry—price-performance competitive.

There are interesting things here. If China has a chip that consumes 3× the power—

But they have 4× the power, then—

Yeah, who cares? Obviously, there are a lot of supply-chain challenges with building that, and maybe it’s on N−2 technology. It’s on 5-year-old technology or 4-year-old technology. Great. It only consumes 3× the power because they were able to do a lot of software optimization, architecture optimization, and so on.

They end up with something that maybe costs a little bit more, but when you think about the value of a GPU today, the GPUs dominate the cost of everything.

But over time, services will be built out that are high-margin. You can go look at Anthropic or OpenAI fundraising docs and see that their API margins are good. API margins are nothing compared with what service margins will be for people who use these APIs to build services. And that’s nothing compared with the net good to the economy from how much automation can happen and how much increased economic activity there is.

So this is the argument: even if their chips cost 3x as much, do you—

Elad Gil

They can subsidize that rationally.

They can subsidize that rationally because the end goal is, wait, actually, we can deploy a lot of Chinese AI and make money and gather data because people are sending us their prompts and all their databases and all this stuff to our models, controlled by our companies, et cetera. Plus, we’re just making money off it.

And they’ve done this in other industries. They rationally subsidize solar, and now no one can even compete on solar or EVs. It’s very close to no one being able to compete on EVs, besides Tesla, really. And even Tesla is adopting a lot of the Chinese supply chain.

It is rational to say you want America to have more AI prowess around the world, so that a random child in Lebanon doesn’t think America is bad, or so they’re using American products more than Chinese products. But how you get there is very difficult, and it’s a hard thread to weave.

Thread. You got it.

I don’t crochet, you know.

Oh my God. Crochet.

Crochet. There we go.

You clearly don’t.

Croquet is the—

Amazing. Croquet is the game. I want to ask you a wild-card question to finish out. We’re trying to get Mark to do the podcast.

Elad Gil

Zuck.

Yes. You can ask him any question. What would you ask Mark? You have to do the podcast.

I thought the page they put up was very interesting. They were like, “We want AI to be your companion.” So my question to him is not around his infrastructure stuff, because I feel like I know most everything—you can figure that stuff out from the supply chain, satellites, and all this stuff—but the interesting thing I’m curious about is, philosophically, what exactly does the world look like if everyone is talking to AIs more than other people, or interacting socially with AIs more than other people? Do we lose our human element? Do we lose our human connection?

It’s not the same thing as, hey, I’m posting on social media and we’re interacting with their social media posts, which already breaks the brain of a lot of people. What happens when it’s always on your face? His worldview is that Meta Reality Labs makes these devices that you wear, and they’re always on. They have all this AI on them, and you’re talking to the AI companion all the time.

How does that change the human psyche? This human-machine evolution—what are the negative ramifications of it? What are the positive ramifications? How are you going to make sure that there are more positive ramifications from this than the sloppification and complete brain rot of our youth? I love my brain rot, though. It’s like, okay—

Elad Gil

Obviously, the coding wars continue to be very central.

And we were talking about Cognition’s relevance and how to think about the strategy here, but I do think it’s really funny what flipped your bit on Cognition. Can you tell the story?

I thought Cognition was NGMI. OpenAI, Anthropic, xAI, et cetera, were just going to make better code models. They have way more resources. General models will win. I hadn’t really met too many people there; it was just a pure vibes-based thing. I had used a little bit of Devin, but it was whatever. Claude Code seemed better, and we use that internally.

But I went to KTO's East meets West event. It’s an awesome event where there are people from Asia. There were all these CFOs and CEOs of major Chinese companies, people from the East Coast of the US—all these finance bros—and from the West Coast, a lot of tech people. You and I were both there; there were people from governments and major companies, and Scott was there.

I spoke with him very briefly, but what was interesting is that they had a poker night one night and everyone got blasted. The CEO of Cognition is very good at poker. Hedge fund guys are generally good at poker, and people like poker as well. There’s a big poker culture in the Bay Area.

Okay, I was playing. I’m okay. But I look over at the super-high-stakes table, and Scott’s just dominating everyone. I’m like, what is going on? How are you taking chips from the CEO of a major Chinese company? I don’t want to name people because I think there are some terms around naming who was there, but you’re winning a lot of chips from a lot of big people.

All of a sudden, my vibes were like, I don’t know, maybe he can win. Maybe he can take from the lion, you know. So I was very excited about that. I thought it was funny.

I haven’t done much due diligence on their code product, nor have I done much on Claude Code, besides the fact that we use it. But, you know, cool.

Elad Gil

Well, I think the Windsurf acquisition, part 2, is a pretty good hand to play here. And as somebody who invests a lot at a violently competitive application layer—

Yeah.

The poker game is live, man. Everybody there—you just invest in live players. Exactly. And I just loved that, you know, that was how he dominated everyone. It’s such a stupid reason, because I pride myself on being analytical and data-driven. And it’s, you know, vibes.

Correct. For any entrepreneurs listening, I think Dylan might angel invest, or we might back you fully, if you win the Cognition poker game. We’ll host it at Conviction. Okay, we got it. Good. Awesome.

Yeah. Thank you.

Find us on Twitter at no prior pod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-bers.com.