(Preview) Inference in the Agentic Future, xAI Is Two Companies in One, Q&A on Elon’s Lawsuit, Intel, Apple

Hello, and welcome to a free preview of Sharp Tech. Hello, and welcome back to another episode of Sharp Tech. I’m Andrew Sharp, and on the other line is Ben Thompson. Ben, how are you doing?

The question is, how are you doing, Andrew? Unfortunately, we did not podcast last week. There was no chance it was happening once my travel plans actually worked out.

Hmm.

So, good thing we did the makeup in Taiwan. Just reminding everyone: we’re very conscientious of your time and attention.

Yeah.

In the meantime, your Washington Wizards got the number-one pick in the NBA draft. Is this making up for the fact that the NBA let Dallas win last year so that Luka could go to the Lakers?

Oh, my God, yeah.

Is this how they made it up to you the following year, after demoting you?

Hmm.

Thus demoting you, so they made it up for you the following year?

It doesn’t quite make up for it because there’s no Cooper Flagg-level prospect available this year. But look, I can’t get greedy. Does it make up for the last 25 years of Wizards fandom? Probably not. It’s not making up for that, either. However, it’s nice to have some hope in Washington, DC, for the first time in about 10 years or so.

I was very, very happy on Sunday afternoon. Now I’m, I would say, pretty anxious about what direction the Wizards are going to go over the next 7 weeks or—

I’m glad you had a few moments of happiness.

Exactly.

Hopefully that’s not the peak level of happiness in your entire adulthood of being a Wizards fan. But I’m happy that it happened.

You know what? That’s all that matters. There was a window of pure happiness. Now we’re back to anxiety. We’ll see where we end up by the end of June. But I am—

But look, here’s the deal. Here’s the deal.

Okay.

I hope they choose someone that you don’t like, because my experience of the current Wizards regime is that every time they do something you disagree with, they end up right—

It works.

—and you end up wrong. So—

Look, we’re not going to relitigate the Bradley Beal trade at the top of this Sharp Tech episode.

Hey, you’ve even—

We have a lot to cover.

—you’ve even become an Alex Sarr guy. You wanted—what’s his name in Houston? Reed Sheppard.

Reed Sheppard.

I think I’d rather have Sarr than Sheppard these days.

Look, I’m not giving up hope on Reed Sheppard. I’m not fully in on Alex Sarr, but I’m happy—

But you’d rather have Sarr than Sheppard.

—to be wrong on that point. I would definitely rather have Sarr, and he looked great this year. So things are looking up in our nation’s capital.

I’m happy for you. I’m happy for Charles. You know what? I’m a big believer in raising your son to support the hometown team.

Yes.

You don’t want your son to end up like Andrew. I guess if you want to be like Andrew and not cheer for the hometown team just because you’re a contrarian—

Mm-hmm.

—look, there’s potential for a big history in podcasting, so that’s fine as long as you did it yourself. But by and large, either raise your son to support the hometown team or let him rebel on his own. So we’ll see.

Indeed.

Well, that’s one of the reasons I’m excited. I can actually take my son to some Wizards games over the next several years with a clear conscience. I don’t have to worry about consigning him to decades of mediocrity, or at least the next decade of mediocrity.

But in any event, it’s great to see you. I did miss you last week. It’s good to see you on the other end of the video call here, and we have a lot to cover.

1. The Inference Shift

We’re going to begin with your article on Monday, which was headlined “The Inference Shift.” You mentioned that there have been 3 inflection points over the past 3 years of AI development. I’ll list those 3 inflection points for anybody who’s been asleep for the past few years: ChatGPT demonstrated the utility of token prediction; o1 introduced the idea of reasoning, where more tokens meant better answers; and then Opus 4.5 and Claude Code introduced the first usable agents, which could actually accomplish tasks using a combination of reasoning models and a harness that utilized tools, verified work, and so forth.

Reading your article on Monday, it seemed like the kernel of insight that spawned the article was that fast inference for coding is ultimately going to be a temporary use case. Can you explain what you mean by that? It was a bit of a light-bulb moment for me—something that seems obvious but hasn’t really been articulated, at least from what I’ve seen.

I don’t know. When it comes to AI, I think everything has probably been articulated to some extent.

Mm-hmm.

This is where the doomers get credit: a lot of the stuff they’ve talked about has come true. But let me set the stage for this idea overall.

Okay.

When it comes to computing, speed is always important. I’m going to make some assertions about speed and quickness that some people will say, “Ugh, why would you want a slow computer?” No. This entire discussion is about trade-offs, scaling, and all those sorts of things, so let’s have that upfront.

If you’re coding, of course you want the computer to be fast. But everything that we’ve done with computing, by and large, has had humans in the loop.

Mm-hmm.

As long as the human is in the loop, computers can basically never be fast enough, right? We’re always looking for them to be faster, or at least fast enough for the speed of human interaction.

Think about it from an enterprise perspective. Why would an enterprise update computers quite frequently back in the ’80s and ’90s, even though they were much more expensive?

Yeah.

Because the more expensive asset is the human worker. If they’re waiting around for the computer, you’re sort of wasting time and money.

You’re losing productivity, sure.

That’s right. An enterprise has always been willing to pay for productivity. At least for your coder or whatever, you’d typically get a very good computer, and it would be updated fairly regularly so that you could work, the computer could respond, and you could continue your work.

Mm-hmm.

2. Agents Can Wait

There’s a bit now where more and more of that work is obviously being done by the computer. As these agents become more capable, there are a couple of weird things going on.

One, you’re having the agent go and do something, and then what are you doing in the meantime? You could spin up another agent and do something else. But are you losing your own mental context? I know for me, I’m certainly having a hard time switching gears sometimes.

There’s also this weird bit where people have token budgets, and Claude Code will have limitations on how much usage you have.

Yeah.

Then they’ll use it up. It’s like, “Well, I guess I’ve got to go home for the day.”

Yeah.

What, you want me to sit here?

What do I do now? This is my new workflow.

That’s right. Am I going to go back to working the way I used to for the next 3 hours? No, of course not. This is ridiculous. We’ll just wait until tomorrow.

You have this idea that, even today, there’s a measure of how much work these systems can do, and it’s getting longer and longer and longer. I think that’s actually one of the more interesting benchmarks of these programs: how long can they do an autonomous task before they sort of lose the thread?

Mm-hmm.

That’s getting longer and longer and longer. But it still comes back to the human, and then the human has to—

There’s human oversight—

—tell it what to do next.

—in most of these tasks.

Yeah. Oversight.

Yeah.

That’s right, all these sorts of things. Obviously, it makes sense that that’s where we are. But as long as that’s the case, of course we want faster and faster inference, and it’s worth paying for that inference.

Okay.

Because if you can get that response much more quickly, the better.

I’ve been focused on the agent bit, but the thinking part is super important. ChatGPT, before they got the Spud model, was still running on a GPT-4-class model. The base model was horrible, but its reasoning was so good that you’d still get really good answers. It just took forever.

Yeah, you just—

And—

—had to wait 55 seconds for it to come back with an answer.

35 seconds if you’re lucky, right? Some of this stuff would just take a good few minutes. But it would come back with a really good answer. It’s like, what am I doing here? What am I waiting around for? So you think, oh man, wouldn’t it be great if that were faster? You see things like Cerebras or Groq or whatever, and it’s not just amazing in terms of spitting out an answer; if you’re reasoning, the more you reason, the more tokens you use, the smarter it gets. Wouldn’t it be great if that could be faster and faster and faster? Absolutely, that is the case.

3. Specialized Chips Target Inference

And Cerebras and Groq, just for anybody who’s not familiar, are chip companies that specialize in inference and specialize in speed—

Yeah, well, sort of accidentally specializing in inference. Both of them started kind of before the LLM moment and retrofitted what they’d been working on to this. I think the next-generation chips for both of them are going to be super interesting in terms of how you would change things now that you have that in mind. They’re architected a little bit differently. Cerebras is actually really interesting architecturally.

You have a wafer that’s 300 millimeters. Usually, in a wafer, you’re limited by the size—the reticle limit. The reticle limit is the lens, basically, for your lithography and how much of the chip it covers. Usually, that’s the size of a chip.

26 millimeters or something?

I don’t have the numbers in front of me. They’re in my article.

It was a tiny, tiny number in your article. It’s mind-boggling how infinitesimal all these measurements are.

Right. What you have to do, though, is, if you want a bigger and bigger chip, Blackwell is actually 2 chips fused together. Those 2 chips are defined—their size is defined—by the reticle limit. Then they have to put an interposer in to let them communicate and expose themselves to the system as 1 chip, even though they’re actually 2 chips linked together.

Apple’s Ultra chips, I think, are sort of something different. They’re limited by the reticle limit. The idea is that, on a big wafer, the bigger the chips, the more expensive they are because you’re more likely to have yield problems. If there’s a defect on the chip, you’re going to have a number of chips defined by the reticle limit.

What Cerebras has done is basically develop a technology to—I’m forgetting the name. There are scribe lines, I think, between all the reticle exposures. They basically run wires across that, so you do a bunch of exposures, limited by the reticle limit, over all the different parts of the chip. Then they do this additional step of adding all these lines across those boundaries, so the entire wafer is 1 chip.

Mm-hmm.

It’s wild stuff. It’s a really interesting approach to get a lot of compute and a lot of SRAM, the super-fast on-chip RAM, which is what Groq does also. But Groq is still limited by the reticle limit. Groq is more about systematizing the process of putting different ones together. Cerebras is like, no: 1 wafer, 1 chip.

And that allows them to serve stuff faster than other chips.

Unbelievably fast.

Yeah.

This solves parts of the inference process. There are different aspects of inference, but there are parts of it that are extremely limited by bandwidth—how fast you can get memory into the processor and move on to the next step. They’re unbelievably fast at that, orders of magnitude faster than other approaches.

There are limitations. You’re limited by how much memory you can fit on that chip, and the moment you’re going off the chip, your performance totally plummets.

Right.

It’s definitely a narrow use case, but there are situations where, if you want immediate response—and not just immediate response, but immediate thinking-through-things response—it makes a lot of sense. But if you’re reasoning and doing stuff, it’s not just the size of the model fitting on the chip; it’s also the KV cache, which is all the context of the conversation. That gets large very quickly. There are lots of limitations, but the larger the market is, the more room there is for different—

Niche use cases.

Sort of approaches. So we’ll see how it turns out. They’re IPO-ing this week, or at least I thought they were going to IPO this week. It hasn’t come out yet, I don’t think. Maybe it’s today. Of all the times to IPO, right now is a pretty great one. Not a bad stretch.

Yeah. Sure.

We’ll see.

Well, yeah. You can envision—I think you mentioned in your article that, if there are voice interactions with AI—

That’s the biggest one by far.

—speed is going to matter. For the consumer market, speed will always matter. But to the extent that we expect a lot of computing to just be done by the computers, certainly in the enterprise, that will probably be independent of humans. Optimizing for speed just doesn’t make as much sense, and the obsession with speed is sort of immaterial to the conversation. Nobody has to care about the speed for the robot users.

That was sort of the secondary implication of the agentic shift that seems inevitable but wasn’t immediately apparent to me over the last couple of months, as we’re all sort of obsessing over agents and what they mean in the enterprise.

Yep.

So, if you think about what agents are, what’s the upside? They never sleep. They’re always working, right? That’s like what he does—

24-hour employees, yeah.

Right. So we’re going to need all this compute, and I think that’s all completely true. But part of the implication of them always being awake and always being available is they can sit around for compute. It’s fine, right? There’s no loss in terms of them waiting around.

Particularly for these agentic workflows, at least right now—and I think there will be breakthroughs, algorithm breakthroughs, and architectural breakthroughs—for now, a lot of these agentic workloads are really limited by memory. It’s this KV-cache issue. It’s pulling in all this context and remembering state.

If you want these sorts of things that aren’t just useful for a task you define right now, but can be spun up suddenly—something comes up in a week, and it’s spun back up, and it has all the right context, and it knows what needs to be done, and it executes a job, and then it goes back to sleep or whatever it might be—all that stuff needs memory.

4. Memory Becomes The Bottleneck

The memory question is also interesting because everything’s been about HBM, high-bandwidth memory. The reason we want high-bandwidth memory is because we want—

What do you think we want from high-bandwidth memory?

Is that for training?

We want high bandwidth. High bandwidth is the answer.

Well, I gave it a shot. You just put me on the spot there.

Okay.

I know that China lacks high-bandwidth memory, and training is a problem in China, so that’s how I landed on that guess.

No, you’re totally right, actually, because the reason we need it for training is that everything in training is this highly distributed problem where we want the GPUs to do these calculations super fast. We want to keep the GPUs fed. It’s a multivariate problem that NVIDIA has solved much better than everyone else.

It’s not just the fast processors; it’s also loading them up with tons and tons of HBM, high-bandwidth memory.

Mm-hmm.

But it’s also developing all this crazy networking to tie all this stuff together. It’s not just that you have chips acting as 1 system; you have fleets of them—

Clusters, yeah.

—tens of thousands of them acting as sort of 1 chip, and it’s going to be hundreds of thousands of them. There’s a lot about the way development for AI has gone that has been very focused on this problem: how do we execute stuff quickly, but then keep the executors full so they’re being utilized all the time?

It turns out that, because they’re GPUs—and GPUs are fairly flexible, not as flexible as a CPU, but more flexible than an ASIC—this is also an architecture that works for inference, right?

What do we have? What’s the inference problem? You need to get the model into memory, and then you also need to house this KV cache. If you have all these GPUs linked together, you can solve both problems. You can get large models into a pod, and you can handle the KV-cache issue.

Yeah.

But as it gets larger and larger, the KV cache is actually becoming a problem even for GPUs. NVIDIA has announced its own thing called Dynamo for inference in general, but it has also announced a whole rack for its systems that’s just memory.

It's just SSDs.

Mm-hmm.

The whole point of that memory is for KV cache. But even then, if you fast-forward 5 years—fewer, I don't know—you have these agents where, again, there is no limit to how many agents you might want. There is a limit to how much compute you might want for humans. The limiter is how many humans there are—how much stuff they can come up with. But at least in theory, the limit for computers doing computing, and especially once they're doing their own programming and spinning up their own sorts of things, is effectively infinite.

You're going to need to store all this context. There are lots of innovations around here, whether you're doing batching or caching, or some prompts have the same sort of context and so you can put those together. There are all sorts of things that people are going to figure out and innovate on in lots of ways.

Along those lines, there's a concept called a memory hierarchy. With traditional computing, like your typical CPU, you have registers, which are the actual data being processed. Then you have L1 cache, L2 cache, maybe L3 cache. This is all storage on the chip itself. One of the reasons Apple's M chips are really fast is that they have a lot of cache.

Mm-hmm.

So it's right there on the chip, and a lot of the core, foundational operating system stuff is right there. They've done a lot of integration. This is where their integration has really paid off to make sure everything is super available and executes very, very quickly.

Yeah.

But then from there, you go out to RAM, and RAM is super fast relative to your hard drive, astronomically slow compared to cache, right?

Mm-hmm.

When you go past RAM, you go to your SSD. Remember when we got our first SSDs? Unbelievably fast compared to spinning disks, but SSDs are way slower than RAM. Then you can go out to spinning disks. You can go out to tape machines. There's still storage using magnetic tape—huge capacity, very, very slow.

In general, there's this capacity-speed sort of trade-off with memory, and so part of designing a computer is designing the memory hierarchy. We're figuring out, in general, what stuff should be super close to the processor and thus super fast, while knowing you have a limited amount of space there, and what stuff gets bumped down.

Mm-hmm.

When you say stuff, are we just talking about tasks and different applications on a computer?

No, literal ones and zeros. The actual bits that go into calculating this sort of thing, which ultimately is everything. Everything is a 1 or 0 at the end of the day.

Okay.

This is already happening for inference, and it's going to happen even more. So right now—

Mm-hmm.

Mostly everything's in HBM of a bunch of GPUs tied together. There's not much of a memory hierarchy. There is a little bit of one. We don't need to overcomplicate it, but generally speaking, there's been this one-size-fits-all approach: a bunch of GPUs tied together with a bunch of high-bandwidth memory. Put everything in there, and it'll do everything.

But we're already seeing this shift, particularly if you want a ton of context, to having other places to put stuff in memory that are slower.

Mm-hmm.

You get way more room—way more capacity, right?

Right.

As that increases over time, where the memory aspect becomes more important and stuff gets slower, that's okay because the agents—there's no human in the loop.

Computers are doing computing.

That's right.

Right.

That's right.

Well, if that's the world that we live in, who wins in that sort of world? How does that change how AI is served, and what sort of infrastructure is best to serve it? Does the infrastructure get more affordable? Is that bad news for NVIDIA in that scenario? What do you think?

I think the biggest winner is China.

Okay. Why is that?

5. China Could Win Inference

For a few reasons. Number one, you don't necessarily need HBM. Regular RAM is fine, and then you need a lot of storage beyond that. Relatively slower chips are okay because they're waiting around on memory anyway.

Mm-hmm.

China can make all that stuff, number one. They can make all that stuff and start selling it abroad, alleviating the sort of memory shortage that we're facing, particularly if SK Hynix, Samsung, and Micron are all focused on HBM. So who's going to make DRAM, right?

Yeah.

It's not just that they can probably self-supply for more AI workloads than you might think in this agentic workload, but also there's going to be a large market for their companies to expand and sell stuff.

Mm-hmm.

That market will exist even if it's relatively lower-end compared with Western companies.

Overall, the hyperscalers in general buying cheaper stuff is always better, and I do think NVIDIA is running as fast as they can. This whole Dynamo approach is basically like an operating system for inference, balancing different loads and adding on a Groq for the super-fast aspect of inference and their regular GPUs for some stuff. The way they tie that together is really interesting.

They have Groq just doing the sort of inference-speed-related aspect of inference, but then also they're shipping these new memory racks. They're certainly out there trying to get ahead of it, but it's going to be a challenge in the long run, which we saw with the cloud.

Mm-hmm.

6. AI Hardware Goes Modular

Back in the day, in the dot-com era, all the money you raised for a startup went to buy Sun systems. You would buy these incredible servers that were fully integrated, super reliable, the best there were, with a SPARC operating system. You had to buy that to actually run your website so you could then build a business.

It's funny, actually: Hotmail was one of the first ones to do this. Hotmail and Yahoo, but Google was the one that really did it at scale, taking commodity hardware, like Intel-based systems, and just building a ton of them and saying, "These are way less reliable, way flakier, much cheaper, but that's fine because once you get to our scale, even Sun systems are going to break down, and also we're not going to pay Sun all that money."

They designed an entire way of computing that assumed stuff was just way slower and way more fragile, but in software, we can work around that. We can have built-in resiliency, built-in fault tolerance, and all these sorts of things.

Mm-hmm.

Amazon took that concept, did it, and made it available to everyone. So now, if you're a startup, you could start a website right away with nothing up front.

In AWS, yeah.

That seems like what I'm talking about. You start out at the beginning. You have these dedicated, super-high-end systems that sort of do it all.

Mm-hmm.

In the long run, every piece of that system is going to get disaggregated and become sort of commodity markets in their own right.

Right. So it goes from integrated to modular in terms of what people are building?

Exactly.

Yeah.

Exactly. Because there isn't a user, I think holding on to the integration in the long run depends a lot on owning the user to a certain extent. Apple still owns the user. The user cares about the benefits of integration. No one cares about how their inference stack is constructed as long as it does the job, right?

Yeah.

Well—

Well, cost will obviously matter in the long run if there are more—

That's right.

—affordable, more durable options than the NVIDIA GPUs. It'll be very interesting to see where it all goes.

This isn't necessarily totally bearish for NVIDIA.

Mm-hmm.

NVIDIA's approach is still by far the best for training, and it's not like we're going to suddenly stop training.