Groq founder and TPU creator Jonathan Ross GPU ♥ LPU Everything You Wanted to Know Nvidia GTC 2026

NoRush Invest

I want to start out.

Thank you very much for asking me to do this with you. It's a great honor.

NoRush Invest

Let's get into what the partnership is and how this came about. And first of all, thanks for David coming. He normally doesn't do this kind of thing, so thank you very much.

1. NVIDIA Partners With Groq

It all started early in 2025, when NVIDIA released NVLink and was going to allow partners to connect to it. That's when Groq's COO, Sunny Madra—raise your hand—yeah, that guy right there—reached out to Jensen and said, “Hey, can we connect to this?” And Jensen said, “Sure, why not?”

So we got some GPUs. We didn't have NVLink at the time; we just had Ethernet. But we got some GPUs and started trying to get something working across GPUs and LPUs, where we took different portions of the workload that ran on different chips, or ran better on each different chip. It worked. We presented it to Jensen.

Three days later, Jensen called up and said, “Why don't we work more closely together?” Three weeks later, the deal was done, and 1 day after that, I was at NVIDIA working full-time. That was actually December 25th—Christmas. That's when I got my laptop and started working.

NoRush Invest

Perfect. So can you tell us what the product is?

Yeah. Let me start with—I can't see the slide, so I have no idea.

NoRush Invest

Oh, uh-oh.

Why don't we start off with me explaining it with words, then? All right, we're going to do this a cappella.

2. Hybrid Chips Split Workloads

The best way to describe it is: If you were building out a logistics network for the entire United States and I told you your 2 options were that you could either use all 18-wheelers or just delivery vans, which one would you pick? The 18-wheelers are great for long-haul freight transport. The delivery vans are great for the cities, but you certainly don't want to use 18-wheelers in cities.

The best answer is both. So the product is that we actually take the decoder layer of the LLMs, and we take the sort of weights and the FFN portion—

NoRush Invest

Oh, we can see it now? I don't think they can see it, though.

Oh. Okay, great. Perfect.

So we're going to start off a little bit backwards here. What we do is we take the FFN layers and put them on the LPUs. Then we take the attention layers and put them on the GPUs. If you have about 40 decoder layers, there's going to be about 40 round trips.

This is why we started talking with NVIDIA when NVIDIA offered NVLink to the ecosystem. This is a really important point: NVIDIA is all about growing its partners, making new partners, and building the rest of the ecosystem. So what I would recommend is, if you have an idea where you can use NVLink Fusion, reach out. That's how this whole thing got started.

This actually allows us to bend those curves that you saw in the presentation. Ruben is great if you want the lowest-cost tokens; it's your best option. It also bends the curve so that you can get faster tokens. But you see that LPU or LPX that's brought in, and it sort of flattens out the curve there and raises it up.

Now, when you put them together, you can go all the way to this ultra-performance level, which is otherwise just impossible: thousands of tokens per second. That little arrow there that points off to the right goes quite far off to the right.

So why is this important? I'm going to go back to an analogy. Let's talk about the economics here. When you buy energy, if you produce that energy from coal versus oil, it costs you about 7 times as much per BTU that you get out of oil versus coal.

People aren't dumb. They're not going to overspend on something. So why are people willing to spend 7 times as much per BTU from oil versus coal? There are things that you cannot do with coal. You can drive a railroad with coal; you cannot fly a plane with coal. You need velocity.

If you're trying to code and you have to wait 10 minutes to get a result, you'll accept it if you have no alternative. But if someone else is able to get the results in 5 minutes, they can get twice as many iterations per day. That means they can get to what the customers want in half the time.

That speed is very important. You're trying to get a report done. If you can get the report in half the time, or 1/10 the time, it fundamentally changes what you can do.

So what does this mean for everything else?

NoRush Invest

You were saying earlier—one of the things we were talking about on the phone—is that speed is really important, especially when AI starts talking to other AI.

3. Agentic AI Needs Speed

Yeah. Well, let's try and explain agentic AI really simply. Who here finds AI useful in getting tasks done? Okay. Would you be surprised to find out that AI finds AI useful in getting tasks done?

When AI invokes AI to solve a task, that's agentic AI. There are all sorts of reasons why it does this rather than just doing everything itself. One is that it can fork the context, right? There's a certain speed at which you can get those tokens out. If I can fork and try 10 things at the same time, then I can try those 10 things.

On the other hand, if I'm able to do something in 1/10 the time, I can change what I'm doing at each step based on what I just learned. So you get those feedback loops faster.

In agentic AI, there have been some breakthroughs recently where 100,000-plus-line browsers have been written by AI. If you could do that not in days but in a couple of hours, then you could start iterating on that design pretty fundamentally.

It's simply a way to do research, to explore. And, actually, just stepping back for a moment, we just don't know. The bottom line is, we don't know what we don't know. We don't know all the things that can be done with fast AI because we haven't had it yet.

It's like when you first got oil: What did everyone do? They started running ships with oil. You didn't need oil to run a ship. It was cleaner; it was better, right? But what we knew was that ships being powered by energy was useful.

We couldn't have imagined that we would have an internal-combustion engine that could fly a plane. We don't know everything that's going to be done with speed. We all know that we want it. We all know that we want the answers faster. We're going to want to code faster, and we're going to want our results faster from research reports.

But there are going to be things that can be done. Think about having a conversational agent speaking to you where, instead of when you ask it, “What is my bill?” it goes, “That's a very interesting question,” trying to buy time in order to generate an answer. What happens if it immediately answers? You're going to feel more engaged. You're going to feel like you're in flow, and that conversation is going to feel much better.

This is something I was actually talking to Jeff Bezos about, because the way he describes AI—you can go back on YouTube and see this TED Talk he gave 18–20 years ago—he was saying that there are these inventions of new technology that are thin, horizontal enabling layers.

He said electricity was one. He talked about the internet as another one. And the way he describes AI, he talks about this thin, horizontal enabling layer.

What he told me—and he said this in the talk—was, “Listen, when we invent electricity, we weren't inventing electricity. We were trying to put lighting in a house.” But then there's no way, after you did that, you could have possibly prevented all the different ways that electricity was going to be used. You just know it was going to go everywhere.

NoRush Invest

Yeah. And so your point was, with AI, we know it's there; we can't predict what is going to be created with AI, and we certainly the hell cannot predict what's going to be created with super-fast AI.

Correct. Okay. So, going back to that point, when electricity was first invented and put in houses, and you started getting other appliances like washing machines, they didn't have prongs where you'd plug them into an outlet. You screwed them into a light-bulb socket.

NoRush Invest

So you've seen this because he talked about that. Okay. I don't know if it's the same source, but yeah.

He talked about it in the TED Talk. You couldn't imagine that you would have other uses besides lighting. Yeah. But going back to this, AI is not like the internet. It's not like telephones. Those are information-age technologies.

Information-age technologies are about taking information, duplicating it, and distributing it. Some are better than others, but the internet is conceptually the same thing as the printing press. It's just faster. It's just cheaper. It's more convenient. But you're copying data and replicating it.

AI is not copying data. It's creating answers that didn't exist before. So AI is not an improvement like electricity. It's bigger. It's more foundational, because we've never had the ability to offload creativity and this other kind of mental labor before.

NoRush Invest

Okay, this is perfect, because we were talking earlier about how one of the biggest problems you had back at Groq was getting people to understand why speed matters. And the conversation we were having earlier was about the difference between a company trying to preserve what it has versus a company that is going to generate new value. So what will speed do to the second kind of company?

4. Value Creation Runs Faster

Well, there are 2 kinds of companies: value preservation and value creation. If you're in value preservation, you're going to use AI to reduce costs. If you are in value creation, you're going to use AI to increase your revenues.

One of the most important things when you're increasing your revenues is that you're going to want to iterate more quickly. A lot of SaaS companies, a lot of web companies, can push a new build once a week.

What if you could do it once a day? What if you could do it once an hour? To give an example of something that we did as a team, we were meeting with a customer and got a customer request in that meeting. I sent a message to 1 of our engineers who, using a coding agent, implemented that feature, and it was done, I think, less than an hour after the meeting was over.

Now, if we had more speed, that could have been done before the meeting was over. Just imagine you meet with a customer, they have a problem, and before the meeting is over, it's solved. We just don't—we're not able to wrap our heads around that because we can't do it yet. This is what that's going to enable.

NoRush Invest

Explain why people are always going to want to spend more money to go faster. Think of it as competition.

So, actually, let's take NVIDIA. NVIDIA went from us starting on December 25th to us unveiling the LPX rack onstage today. Do you want to be half that speed? Everyone's going to want to be as fast as they possibly can.

NoRush Invest

The best businesses move fast, and you're an expert in this. I'm going to ask you: Is that true?

Yeah, of course. What business wants to move slower than everyone else? Especially when there's value creation, the faster you can create those new features, those new products, and those new abilities, the faster you can bring that revenue in.

I think the important point you were making on the phone call we had earlier is that the product is already in production.

NoRush Invest

Yeah.

It's already being produced right now. As Jensen announced, Q3 is when it's going to be available. It's probably 1 of the fastest ramps of a semiconductor in history. NVIDIA's legal department wanted us to insert the word “probably” there, just so you guys know. That's not a joke.

NoRush Invest

Oh, I'm sorry. Not supposed to curse either. We have plenty of time. Why did you say that being the king of inference is a big deal at NVIDIA?

Everything started with training. We had to get models that worked. Training scales with the number of researchers you have, and inference scales with the number of users you have. Revenue comes from users, not researchers.

The better your training is, the better your inference is. But the better your inference is, the more revenue you generate, and the better your training gets. It's a virtuous cycle. If you are the best at inference, it means that you're going to be enabling that revenue for your customers, and they're also going to buy more training hardware.

But you can't be just 1. There's a virtuous cycle.

NoRush Invest

Can you explain in detail how you made it both faster and more cost-effective?

5. Hybrid Design Bends the Curve

The way we did that was—remember that Pareto curve? Let's go to this one. What we're really showing here is throughput per megawatt.

It goes back to what Jensen said: You're going to buy a gigawatt data center, and you're going to fill it with hardware. The question you're going to ask yourself is, what are you going to run in that data center? How are you going to run it to earn as much revenue as you can, or to get as much productivity as you can for your workforce?

What you see is that the faster those tokens are produced, the fewer tokens you get per megawatt or per gigawatt. That's the trade-off. The LPU is really good at low latency. The GPU is really good at high throughput.

Going back to the logistics network, if you deliver the first mile—the long-haul part—with the 18-wheeler, and then hand it off to delivery vans to deliver the rest of it, you're going to get the most cost-effective delivery network.

What you see here is that bend in the curve. If you compare Blackwell to Reuben with LPX, you see that it doesn't just keep going down like it did. It bends up as you start going faster. That's because the LPU is particularly good at the high-throughput matrix multiplies in the decode layer.

If you were to run something purely on GPU or purely on LPU, there would be different matrix multiplies that got different utilizations. If you ran everything on the LPU, you'd be underutilizing it on attention. If you ran everything on the GPU, you'd underutilize it on the projection, or the FFN, layers.

Putting them together, the utilization goes up for both. The speed also improves because the LPUs don't have to wait to fetch things from external memory. So now you actually get more throughput out of the same hardware while getting better latency.

It's not simply just faster is better. It's faster at what cost? It's faster at what capacity? The world right now doesn't have enough inference compute. It doesn't have enough energy to power enough inference compute. But if you can get more tokens out of the same amount of energy, then you can start to provide high-performance tokens to everyone.

One of the things at Groq that was great was that we had very fast AI, but we didn't have the throughput to deliver it to those customers. We couldn't run anything large, and customers always wanted to adopt us, but they couldn't.

Now you mix in the GPU, and you can do this at scale.

NoRush Invest

Can you tell the story that you told me earlier about using a model that's average and how you were using your technology to actually have it solve math theorems?

6. Fast AI Unlocks New Work

One of the points here is that speed is intelligence. Just think about it: The faster someone thinks, probably the smarter they are, right? It's the same with AI. But why is that?

There was a great example. We were doing formal reasoning on the LPUs, trying to prove correctness of portions of the circuitry. One of the folks at Groq did an experiment, and they realized that if they ran something on Anthropic's Opus, at the time they got the answer in the smallest number of iterations, but at the greatest cost.

When they ran Qwen-32B, a good model—not a great model, but a good, solid model, especially for its size—it would take more iterations, but it solved every single 1 of the math problems faster and cheaper than Opus did because it would just iterate until it got the answer.

On programming problems, where you can get a debug—for example, you get an error on compile, you test-run, and you get an error—you can actually iterate much more quickly with a less capable model. Now, if you have a more capable model and it can run fast, that's even better.

NoRush Invest

You were doing that internally at Groq, correct?

Yes.

NoRush Invest

What are some ways other people use this technology outside of Groq that have surprised you the most?

I would probably go back to voice again. Voice was pretty heavily used.

NoRush Invest

Can you tell them why your girlfriend gets mad at voice?

My girlfriend uses ChatGPT, and she mostly uses it in voice mode. When she asks it a question, it starts off by answering, “That is a very good question.” Then it'll go, “Hmm, let me think about that.” She's like, “I just want the answer. Stop telling me all this.”

It's because it's slow. You do that because you're adding filler words.

NoRush Invest

You're adding filler words.

Yeah. Those filler words are better than silence, but they're not much better. I don't know. I think I would prefer silence. She certainly would.

If you think about the most engaging conversations you've ever had, those are conversations where, as you are stopping speaking, the other person is saying something that answers your question. They're not going in a different direction. They understood you. They really got you. They're aligned.

But they're answering immediately. It feels like they're reading your mind. That's why speed is going to matter.

NoRush Invest

If you will, explain why this applies to voice and the other ways that you think it's going to apply to voice, besides customer service. We used the example of calling call centers, but what else?

Coding. Some of the engineers here at NVIDIA—1 of them on the LPU team—has connected up, I think, to either Codex or Claude Code. I don't know which 1, but they just have their phone connected to it, and they talk to it and have it make all the updates.

They're not typing anymore. They're just using the computer to interact with the applications and stuff, but all of the direction of what the AI should do is voice.

NoRush Invest

Our mutual friends Eric and Karim from Ramp told me the same thing. They have a bunch of engineers in their companies who say they're not typing anything.

Yeah.

NoRush Invest

Which is not going to be great for open-floor plans. If you created this technology and invented this technology, but you were sitting in the audience, what are some ways that you think you'd be using this if you were in their shoes?

I would probably focus on giving your premium customers speed. You certainly don't want to be slower than anyone else. In these charts, we refer to this as a free tier at the—

NoRush Invest

Go to the dollar.

Oh, yeah.

NoRush Invest

Yeah, that 1 should just be a permanent display. Look at this. I was asking you before we even came out here: Which plan would you use?

I always pay for time. Yeah, Ultra.

NoRush Invest

So basically the only thing I buy.

Exactly. So it's a little bit pricey.

NoRush Invest

There you go.

The thing is, speed matters. Another way to think of it is: If you have engineers who aren't worth giving the Ultra package to, are they good enough to hire? Don't you want all your engineers to be so good that it's worth speeding them up?

You should be giving the fastest-speed tokens to all of your best engineers. Then the other engineers are going to ask, “Why don't I get the best tokens?”

NoRush Invest

You were mentioning the other 2 companies, Anthropic and OpenAR. Are they doing this as well?

Yeah. They already offer plans where you can pay to get a speed-up, and I believe it's superlinear. They charge more than linear for the speed-up. So this is something that's already needed.

NoRush Invest

I was using open-source models on Groq LPUs because, even though when I would ask a question it normally wasn't perfect, I could immediately ask a follow-up and it would correct it. Rather than having to wait 5 or 10 minutes for my answer on a better model, if I could get a really good answer in a second, I'll take that every time.

There's one more analogy here. When we're looking at how much token capacity we're willing to give to engineers, there are people spending $10,000 a month right now. That's on the high end, but that might be on the low end of where this is going. In fact, there are plenty of careers where people are responsible for things that are worth tens or hundreds of millions of dollars. Pilots fly airplanes that are worth tens of millions—actually, hundreds of millions of dollars. If you can generate the revenue, that makes sense. It probably won't be long until engineers are actually using millions of dollars of tokens per year, and you're going to want to give them the fastest tokens possible because they're the best engineers.

I have a weird question that just popped to mind. Your name's Sunny, right? Okay. You're essentially encouraging them to find creative ways to use the latest technology, right? This is a very old idea from the history of entrepreneurship. Andrew Carnegie, in his autobiography, talked about the fact that you have to invest heavily in technology. The savings compound; it can be the difference between a profit and a loss and give you a massive competitive advantage over your slower-moving competitors, which he applied to one of the most valuable industries in the world at the time, during the Industrial Revolution: steel.

Do you remember the breakfast we had in New York, like, 2 years ago?

[Speaker?]

Yeah. It wasn't that long ago.

Or 1.5 years ago—I don't know. It feels like it does, right? This was before you called Jensen and asked for access, right? That one idea of, "Hey, this new thing came out. Let me figure out creative ways to use this new technology." What was the difference in value in your business? The state of Groq—I mean, Groq was doing okay, but certainly not... That was life-changing, correct?

Okay. So this is really important. Sunny proposed this to me, and at first I was against it—not because I didn't think it was a good idea. I just didn't know if it was going to work, and there were a whole bunch of other things that were going to work.

NoRush Invest

But what was the state of the business at that time?

We were hitting some revenue targets, but they weren't nearly as big as the deal that was done.

NoRush Invest

Exactly. So it might have increased the value of the business by 5× or 10×?

More. But I'm actually making another important point.

NoRush Invest

I hope he got paid.

Oh, yeah. Sunny's okay. But I'm actually making a more important point here, which is we almost didn't do it. Whereas if we could have had AI at the time—because AI wasn't at the state where it is today, which was less than a year ago, so it's crazy—if we had AI where it's at today, it would have just tried the experiment.

NoRush Invest

Wait, say it again.

We'd have just tried the experiment of seeing if we could get it to work. We'd have had AI go do it. I run experiments all day long. Today, it would have been no question. The only question was opportunity cost. We had a finite number of engineers. We were trying to deliver things for customers. Those customers told us how many dollars they were going to give us if we delivered, and so we were on a path.

Sunny advocated for this, and I said, "Fine, take a small contingent and do it." It took longer than it would have, but now with AI, you can just run experiments. It's cheap relative to the lost opportunity.

NoRush Invest

So imagine you advocated—I would imagine you advocated pretty hard. Imagine if you said no: the difference in the valuation of the company is 1/10, 1/20, 1/50?

I mean, it wasn't that much of a difference, but it was big.

NoRush Invest

Yeah, substantial. So AI is going to allow you to run experiments. It's going to allow you to run them quickly, and if we run the experiment faster than someone else, then we have the advantage. I guess this is the main point: you're developing new technology to make other businesses more valuable. You're almost like your own proof of concept for that.

Yeah, so I can't disagree with that.

NoRush Invest

Yeah, it just came to mind. Well, you want a different example.

Claude Code writes Claude Code.

NoRush Invest

Yeah. So Sunny just said, "Claude Code writes Claude Code." If you look at how fast it updates, more than once a day they're updating, right? He's saying more than once a day it's updating. I run experiments so quickly now on things that I just whispered in your ear. They were doing the introductions. I don't know if you want to talk about this or not. I was like, "What are you hacking on?"

Yeah, I'm not going to—

NoRush Invest

I was going to show you now.

Okay, you show.

NoRush Invest

Okay, show me after.

I'm just doing AI coding—nonstop. It's all I'm doing right now. You're like, "What's your schedule?" I wake up at around 6:00 a.m. and spend the first couple of hours doing... I start off with an AI-delivered daily briefing. It tells me all the most important things from my email, my calendar, and world events. It integrates Polymarket, Kalshi, everything—market numbers. It tries to give me an update on everything that's going on. It takes news and integrates it.

I read through that and ask it follow-up questions. In fact, I told it to stop writing long descriptions. Just give me the title, and if I'm interested, I'll ask a follow-up question. That's the first thing I do, and that takes me a little while to go through. It organizes all my emails for me and drafts things for me.

After I get past that, I'll start coding a little bit. Then I'll start sending some emails to people, and then I come into the office. When I'm in the office, I'm in meetings just dreaming of getting back to coding. I wasn't coding pre-AI—I had stopped coding. I just didn't have the time for it.

Now, even while I'm in a meeting, I'll have an idea, and I'll type it, run the experiment, and go back to the meeting.

NoRush Invest

How much sleep are you getting?

Oh. 6 hours a night.

NoRush Invest

A lot of people are coming here because they want to hear you speak. Right before we got on stage, you said, "Sometimes I wonder why anybody wants to hear me speak." I'm like, "Jonathan, are you crazy?" You have this very unique lived experience. You have stuff in your head—very important, valuable information about the most important thing going on. Of course people are going to hear you speak.

Who do you go to learn about other things that you don't know about AI? Where do you get that information?

Actually, I just go on X and hear things from Andrej Karpathy. I hear things from... Actually, the daily brief usually has a bunch of new papers in it, so I'll just read whatever's in there. So AI, I guess, is teaching me about AI.

NoRush Invest

So there you go. It's loops all the way down.

And also, of course, I listen to the Founders podcast a lot.

NoRush Invest

Of course. I was just telling David, "All of your mistakes are a form of leverage." I think I've listened to every single one that has made it to YouTube, and a lot of the lessons for Groq were actually in there.

Oh, that's incredible.

NoRush Invest

Yeah. How much of the purchase price are you going to give to me?

Well, I should have let you invest.

NoRush Invest

I think you did offer it. I said no. I'm pretty stupid with these kinds of things. We have 8 more minutes. I don't have any other questions. I know you said no Q&A, but can I just violate this and have us take questions for 8 minutes?

I'm sorry. I think I did 2 curse words. That's way less than I normally do. I see some nervousness from the press relations team, but yeah, let's do it. What could possibly go wrong?

NoRush Invest

They have microphones set up. Why do you say no questions? All right. So right now, as you guys have said, it's superlinear to get fast mode in Claude or Codex. Do you think that's going to continue into the future?

7. The Audience Tests The Thesis

I do. I think it will be superlinear because there is an increased cost. What we're doing is improving the cost significantly, but there is an increased cost. There's always a point on this chart—which goes way out, way off where this chart is—that some people will want and others won't.

But I just imagine if there's some sort of emergency situation, like a FEMA situation, you're going to want the fastest possible tokens responding to that and trying to action it. There's going to be use for the fastest possible token. But no matter what speed is available here—and it will be—you'll get more of them per gigawatt or per megawatt deployed.

NoRush Invest

Did you guys kill the Rubin CPX?

Well, hey, 1 question per person. What are we doing?

NoRush Invest

Yeah. Yeah. Yeah.

Audience member

Can I go next? Can you ask my question? I just do LPU stuff. So, I think the LPU rack is cheaper than NVL72. Why limit it to just a premium tier? Why not all workloads?

Yeah. So, the LPU rack is actually very dense and has a lot more hardware in it. So, I don't actually agree that it's less costly. There's a lot more silicon in that rack, or Groq chips in that rack.

Vinay

Hi, Vinay from Microsoft. Slightly technical question, but you spoke about the utilization of both GPUs and LPUs for different portions of the workload, which totally makes sense. But there is still the scale-out latency cost you have to pay to transfer the activations. What secret sauce do you actually use to push toward the right-hand side of the Pareto curve and hide that latency?

Well, one of NVIDIA's best-kept secrets is that we're really good at networking.

For scale-out, we bought this company called Mellanox and have some nice gear there. So, we've gotten the latency down quite a bit on that, but we do have some improvements coming that will make it even better, and so you'll see these tighten up a bit.

NoRush Invest

I actually have a question about something you said earlier that I think is important. Can you describe the speed at which NVIDIA operates? From the outside, you thought they were fast, but now you're inside. We were talking about this earlier—just how fast they actually are.

Yeah. So, at Groq, we were a startup of 450 people, and we moved incredibly quickly. Most people thought it was crazy how quickly we could move. NVIDIA—what are we, over 40,000 people?

Yeah, okay. Getting thumbs up. And we move just as fast. No slower. No real bureaucracy. Things just happen super quickly. When someone is needed for something, they jump on it. The culture is very unique in that way. I've been at other large companies that didn't work this fast. Google.

NoRush Invest

I was trying. I said it, not me. Or him, not him. Oh, yeah. I'm going to get in trouble. Thanks.

No, sorry. You shouldn't invite me. I don't know what to tell you.

Audience member

Regarding RSI, or recursive self-improvement, how does fast AI help with that? I'm assuming that since you can do so many fast experiments, that also blows up RSI.

I'm not on the AI research side, so anything that I say is purely a bunch of speculative tokens here, and some draft model or some verifier is going to have to come by and correct them. That's a joke for a very small number of the scientists.

The way to think about it is that one of the differences between the way that we teach models and the way that human beings learn, and why we're able to learn on so little data, is that we're curious. We're curious about anomalies. We see things that violate our expectations, and then we double-click on them.

When a baby sees something that doesn't make sense—like if you try to trick them and make them think something's hovering in the air or something—they'll look at it intently because they don't understand. That's a very innate thing, and the reason is that whenever something violates your expectations, that's a very interesting thing to generate more data about and to train on.

When we're training these models, we're just training them on a whole bunch of random things. We go, “What's 1 + 1? What's 2 × 3? What's the second derivative of the square of the hyperbolic tangent?” As if that should follow, but it's not ready to receive that information yet. It's too far out.

When you're improving a model based on its interaction with the world, you're actually able to set the difficulty of the problems right at where it's ready to receive that training data. So, what I would expect is that if you were able to generate tokens faster and do the rollouts faster in a sort of reinforcement context, at least, you could potentially narrow the number of cycles needed for training by targeting more carefully.

This is just speculation. This is not an area I'm aware of being done too deeply, but there are possibilities.

NoRush Invest

Makes sense. Thank you. We've got 2.5 minutes.

Nick

Awesome. Thanks for doing this, Jonathan. This is really cool. I'm Nick from Knight Capital on X.

So, basically, my question is: when we were listening to Jensen at CES recently, it sounded like the idea for Groq was, at some point, a little bit narrower. I have the Meta Ray-Ban glasses, and he was talking about how super-low-latency inference would be useful for this type of application, but it seemed like it was a more narrow thing.

Today, I think what we heard was a much bigger rollout of Groq—a platform-wide thing where it's integrated into the entire Rubin architecture. So, I'm just wondering: was the interpretation that I had initially correct, and then there was a change as far as how Groq was going to be integrated? If you can just talk a little bit about that. Thank you.

Well, Jensen likes to set expectations here and then overdeliver. So, I think he's done that again.

The expectations were that it would be used quite broadly, but there was a lot of work done internally to verify that that could be the case before it was rolled out more broadly. At this point, we're in production, so more can be said.

NoRush Invest

Thank you. Thank you. A quick question following up on the question by the Microsoft gentleman. On this Pareto curve, in terms of a performance improvement, can I guess that, right now, the main bottleneck is the rack-to-rack latency and the bandwidth? Or, in other words, how do we get back to over 1.5 million tokens across the X-axis in the future? What's the bottleneck?

Only an LPU could answer that in the amount of time left, but I'll give a high-level answer.

Oftentimes, when doing computer architecture, you get focused on one bottleneck. I did a bunch of spreadsheet analysis on some of the more recent models, and there was a point where, in a single layer of one of the open-source models with all the operations, one of the operations was bottlenecked on memory capacity. One was bottlenecked on memory throughput. One was bottlenecked on compute. One was bottlenecked on networking latency. One was bottlenecked on networking bandwidth.

Literally everything you can imagine was bottlenecking one of them, and if you improve that, something else would be bottlenecked. You're right that if we do that, it will speed things up, but these are fairly balanced architectures where you have to improve a lot of things in order to get noticeable improvement. But there's a lot of room for improvement going forward.