Chip design from the bottom up – Reiner Pope

I'm back with Reiner Pope, CEO of MatX, a new AI chip company. Last time, we were talking about what happens inside a data center. Now I want to understand what happens inside an AI chip. How does a chip actually work?

Full disclosure, by the way: I am an angel investor in MatX. So hopefully you have designed a good chip.

Hope so. I'll start with the smallest fundamental unit of chip design, and we'll build up to what an actual production chip is and what its components are.

At the very bottom level of a chip, the primitives we work with are logic gates, very simple things like AND, OR, and NOT. These are connected together by wires that have to be laid out physically as metal traces on a chip.

The main function that AI chips want to compute is the multiplication of matrices. Inside that, the fundamental primitive is a multiply-accumulate of pairs of numbers. We're going to demonstrate what that calculation looks like by hand, and then infer what a circuit would look like for that.

It'll be easiest if I do a multiply-accumulate of a 4-bit number with another 4-bit number. The clearest primitive is actually multiply-accumulate. So there's a multiply of these 2 terms, and then we're going to add in an 8-bit number.

Can I ask a clarifying question? Why is this the natural primitive for whatever computation happens inside a computer?

There are a few reasons. It's a little bit more efficient, but the reason it's natural for AI chips is that if you look at what's happening during a matrix multiply…

What is a matrix multiply in short?

There's a for-loop over i, over j, and over k, of `output[i, k] += input[i, j] × other input[j, k]`. A multiply-accumulate happens at every single step of a matrix multiply.

The other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is specific to AI chips. You're multiplying low-precision numbers, and then when you accumulate, errors accumulate quickly, so you need more precision there.

This is why we've chosen to do a 4-bit multiplication and an 8-bit addition.

Let me make sure I understood that. There are 2 ways to understand that. One is that the value will be larger than the inputs. The other is that if it was a floating-point number, it would be… Maybe that part is less intuitive to me. But maybe it's the same principle?

It really is the same principle. The separate principle is that as you're summing up this number, you're summing up a whole bunch of numbers, so you've got a lot of rounding errors accumulating. Whereas in this case, there's only 1 multiplication in the chain, so there aren't a lot of rounding errors accumulating in the multiplication.

Why are you summing up a whole bunch of numbers? There are just 2 numbers there.

This summation is repeated j many times. Any errors accumulate.

I see. So how would we perform this calculation by hand?

As a human, we would probably separate it into 2, but we can do it all in 1 using long multiplication. For the multiplication term first, we're going to multiply this 4-bit number by every single bit position in the other 4-bit number.

We write that out. First, 1001 multiplied by this bit position. That is the number itself. Then, shifted across by 1, we're multiplying by 0. That gives us an all-0 number. Shifted across 1 more to multiply by this 1, we get 1001. Finally, for this last bit position, we get an all-0 number again.

This gives us a bunch of terms that we have to add for the multiplication. While we're doing that summation, we might as well add in the actual accumulator term as well. So we just copy that directly across. This is the sum. It's a 5-way sum that we want to compute.

What logic gates did it take us to get to this intermediate step?

We needed to produce all 16 of these partial products. How do I produce 1 of these partial products? Let's take this number 1, for example, here. We produce it by multiplying this number by this 1 over here. We can produce that with an AND gate.

This number is 1 if both this bit is 1 and this bit is 1. If either of them is 0, then the multiplication of 0 times anything is 0. To produce all of this, we ended up consuming 16 AND gates. In the general case, if I were doing a p-bit multiply times a q-bit multiply, this will be p times q many ANDs.

Finally, I sum them. Most of the work is going to happen in the summing. Let me describe the other logic gate that we use here.

AND is almost the simplest logic gate that exists on a chip. It's almost the smallest. At the other extreme, the very largest logic gate you'll typically use is something called a full adder.

Coming from software, you might think that a full adder adds 32-bit numbers together. In this case, it just adds 3 single-bit numbers together, so you can think of it as adding 0, 1, and 1 together.

When I add these together, the result can be 0, 1, 2, or 3, so I can express that in binary using just 2 bits. As input, it has 3 bits. As output, it has 2 bits. The number 2 in binary is 10.

This is also known as a 3→2 compressor because it takes 3 bits of input and produces 2 bits of output.

Just to make sure I understood: the 2 inputs are an X and a Y value, and then some carry that came in…

The 3 inputs are all bits in the same bit position, like 3 bits in a column here. The 2 outputs, I've drawn them vertically here and horizontally here to match this vertical versus horizontal layout.

This expresses that things in the same column are in the same bit position, whereas things in adjacent columns are different. This is a carry out, whereas this was the sum.

So if the inputs in the full adder were, say, 101, then the output would be 10. If it were 111, it'd be 11. If it were 000, it'd be 00. If it were 010, it'd still be 01.

Got it. Yeah.

It's essentially just counting the number of things and expressing that in binary. This circuit captures what we as humans naturally do when we're summing along a column.

I'll show 1 iteration of using the full adder to sum. The way I sum here is going to be a little unnatural for humans. We would sum along the column and then remember the carry, but instead of remembering the carry, we'll explicitly write it out.

We proceed from the rightmost column toward the left. On the rightmost column, we sum the 1 and the 1, and that produces a 0 here and a carry of 1. We've used this full adder circuit on this pair of bits and produced a pair of bits as output.

Now we can do the same thing with this column. We have a column of 4 numbers, so we'll take the first 3 of them, run a full adder on them, and that gives us a 0 and a 0 as output. The sum of these is 00. That's the full adder applied to all these bits. As I've used up bits, I'll cross them out to indicate that I've handled them.

Let's keep going a little bit more. I take these 3 numbers, I add them, and that gives me a 1 and a 0. I've dealt with these 3 numbers. Now I take these 3 numbers and add them, and that gives me a 1 and a 0, and I've dealt with these numbers.

The way to view this is that I have this whole grid of numbers that need to be added. I'm going to keep applying full adders to all the bits here, constantly removing 3 numbers from a column and writing out 2 numbers as output. Keep going over and over again until I eventually get just 1 single number coming out.

This approach is called a Dadda multiplier. This is the standard for how you do area-efficient multipliers using full adders.

Let's try to quantify the circuit size of this so we have a sense of how big things are and can compare them later. How many full adders did I use? How many numbers did I start with?

I have the 16 partial products, which is a product of all of these terms with all of these terms, plus the 8 terms that I'm adding here. I started off with 24 bits. Eventually, I produced 8 bits on the output.

In every step, I was crossing off 3 numbers and writing 2 numbers out as a result. Every single use of a full adder eliminates 1 of the bits here. So how many full adders? It must be 24 minus 8, so there were 16 full adders in this circuit.

This is true in the general case as well. There will be p times q many full adders in this circuit.

Let me make sure I understand the logic of that. The input bits, 24, are p × q plus p + q. The output bits are just p + q. So p × q plus p + q, minus p + q, equals p × q.

That's right. I think this explains, or at least hints at, the second reason we chose to do a multiply-accumulate. The first reason is that it's what shows up in matrix multiplication. The second is that it gave us this very slick, simple p × q, very simple algebra.

We've described this whole procedure.

Every single atomic step that I took here becomes a logic gate, and then the wires are connected together. When I had these 3 inputs that I used to produce these 2 outputs, if I think of mapping this to a physical device, there would be a wire connecting all 3 of these things together into a logic gate that produced this output.

This is the main primitive, at different bit widths, that's inside an AI chip. We're going to build up from here to how you would use it to run all the other operations you might want.

This might be the wrong time to ask, but whenever NVIDIA reports that this chip can do X many FP4 or half as many FP8, it seems to imply those circuits are fungible, that there's not a dedicated FP4 versus FP8. But the way you're mapping it out here, it seems like if it has to be mapped out in the logic, you would need a dedicated FP4 multiply-accumulate and then a dedicated FP8 accumulate. Can you “funge” them?

As drawn, they're not particularly fungible. This is actually one of the main choices you have to make when designing a chip: how much of FP4 and how much of FP8 do I have?

Sometimes I'll make that consideration from the point of view of the customer requirement. Another angle is to equalize the power budget between FP4 and FP8. When they report those numbers and it just happens to be the case that they do 2× as many FP4 as FP8, they're just choosing to give equivalent die areas to all the floating points, and as a result it ends up being—

Why is the ratio exactly 2×?

Part of it is that surely it won't be exactly equivalent to die area. There's a data-movement reason. We'll maybe come back to this when we look at how it goes into and out of memories.

There's something really nice from a software level about the fact that I can pack 2 4-bit numbers into the same storage as an 8-bit number. When I store that to memory, the sizing of the buses that I wire out within the chip makes that work out really nicely.

Come to think of it, it's not just 2×. The amount of area it takes sounds like it's quadratic with the bit length. That's why smaller precision is even more favorable than you might think. This is a really big reason.

In fact, NVIDIA made a change. Historically, up until B100 or B200, every time you halved the bit precision, you doubled the FLOP count. For the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong. You should get an even bigger speedup than you might otherwise think.

NVIDIA's product specs have started acknowledging that in B300 and beyond, where FP4 is 3 times faster than FP8.

Though it should be 4×.

Yeah. What I've shown here is the simplest case of integer multiply. When you're dealing with floating point, as you do in FP4 and FP8, there's this other term, the exponent, that complicates the calculation.

What can we see already from this? I think the big observation you've made is that there's this quadratic scaling with bit width, which is very effective and is the single reason low-precision arithmetic has worked so well for neural nets.

1. Muxes and the cost of data movement

The other thing we're going to do now is compare the area spent on the multiplication itself with all the circuitry around it. We'll walk back in time a little bit and see how GPUs prior to Tensor Cores worked, which is in fact the same way CPUs worked.

Where do we stick this multiply-accumulate unit?

Generically, I'll describe a CUDA core or a CPU. You'll have some register file which stores some number of entries, maybe 8 entries of 4-bit numbers in this case, but typically 32-bit numbers.

Inside the CUDA core, I'll have some register file of some depth, and then I'll have my multiply-accumulate circuit. What it's going to do is take 3 arbitrary registers from this register file, perform the multiply-accumulate, and then write back to the register file.

It's going to write to this one, but it was able to read from this one, this one, and another random one. It will take 3 inputs like this. This is the core data path of many processors.

Most processors look like this. You've got some set of registers, and then you've got some set of logic units, or ALUs. We want to analyze the cost of the data movement from the register file to the ALU and back.

Ultimately, there's going to be some circuit that says, “Well, I don't always have to select this guy. I might select any of the registers at any point in time.”

The first question is: how can I build a circuit? The circuit I'm going to look for is a mux. In this case, it's going to have 8 inputs, 1 from each entry of the register file, and it's going to have 1 output, which is actually producing this output.

What is the cost of this thing? All we have to build it out of is AND and OR. How do we build it? We do the dumbest thing possible. We form a mask.

When we want to read the 3rd entry, we're going to AND every single entry with either 1 or 0 based on whether that's what we want to read, and then we're going to OR all of them together.

Just to make sure I understand the basics: what the mux is doing is just selecting an input?

Just selecting, invisible to software. You say, “I want input number 3,” and that means there's a mux here.

So what is the cost of this mux? For an n-input mux operating on p bits, I have n rows. That's just 8 rows, and each row is p bits wide. I have to AND every single bit, so I get n × p many AND gates.

For every single input, I have to decide whether I'm going to mask it out or not. Then I'm going to OR them all together. There's going to be (n − 1) × p many OR gates.

I've got all of these different things, almost all of them are 0s, but I need to collapse them from my 8 options down into 1 option. Every step, I need to OR 1 row into an existing row.

It's actually funny that you don't think at the level of hardware. You just think, “Oh, I'll just select element 3,” and something as simple as that is in and of itself quite a complicated circuit. This is the first step of all of the hidden data-movement costs that show up.

We're just going to compare. I have to pay this cost. I've got 1 mux here, and in fact I have 2 more copies of that for each of the 3 inputs to my multiply-accumulate operation.

I have this cost, which is 3 × n × p AND gates over here, compared to p × q gates in the actual circuit that is doing the thing I care about. If we plug in actual numbers, with n being 8, I get 24 × p gates just in the data movement, compared to—if q is 4—4 × p gates just in the multiply-accumulate.

Where is the 3 coming from?

3 different inputs here. What I'm hinting at is that all of this work, which scales as the size of the register file—and this is a very small register file—all of this work just moving the data from the register file to the logic unit is many, many times more expensive than the logic unit.

It may be helpful to just see what a mux looks like, maybe a 2-bit or a 4-bit mux. We'll do a 2-way mux. We've got 2 different numbers, these 2 inputs.

These are the inputs being selected between, and the selector can either be, “I want this one,” or, “I want the other one.” This is a one-hot encoding. This is what we start with.

Let's focus on this case. This is the actual input we got, and we want to produce this guy as the result. Very laboriously, we AND this bit with all of these. That produces the result of ANDing this bit with this row.

Likewise, we AND this bit with this row. That produces all 0s. There are 4 ANDs here. Finally, we OR these 2 together, and this gives a 1. We OR these 2 together, and this gives a 1. We OR these 2 together, and it gives a 0. We OR these 2 together, and it gives a 1. Those are the 4 ORs.

This actually ends up looking a little bit like addition. We did exactly the same set of ANDs. We've ANDed all of these things together, but then, instead of collapsing it by using full-adder circuits, we just get a very simple collapsing with OR gates. But that doesn't look like n times p. This was with n = 2 inputs. In the general case, we will have n rows, and we'll have p bits per row. That gives us n times p many AND gates.

In this circuit I've described, almost all of the cost—7/8 of the cost—is in reading and writing the register file, and only a tiny fraction of the cost is in the logic unit itself. This is the problem to solve. This essentially was the state of play prior to the Volta generation of NVIDIA GPUs. This kind of thing is what was inside the CUDA cores. This problem statement is what motivated the introduction of Tensor Cores, which are more generically called systolic arrays.

Think about how we're going to solve this problem. We're spending almost all of our circuit area on something that we really don't care about and is hidden to the software programmer, and the thing that we actually care about is not much of the area. Make this one bigger somehow while keeping this at the same size. That's the goal.

2. How systolic arrays work

The evolution was that we had baked this much into hardware at this stage. This single line is a multiply-accumulate, and this single thing was baked into hardware. The idea of a systolic array is to go 2 levels of loops up and bake this entire loop out here into hardware. The idea is that if we have a much bigger-granularity, fixed-function piece of logic, maybe the taxes we pay on the input and output are much smaller.

Interesting. It sounds like you're suggesting that if you go up 1 step in the matrix-multiply loop, you can tilt the balance more toward compute than communication.

That's right. There are 2 effects we're going to take advantage of here. One is that we can do more stuff per every trip through our register file. The other is that, in some of this loop, we can take advantage of certain things staying fixed.

Visually, we're going to look at this matrix multiplication. This portion of the loop corresponds to a matrix-vector multiplication. We'll take a matrix and multiply it by a vector. How do we do this? Every column gets multiplied by the vector and then summed. We're going to sum along columns.

This 0 and 3 gets multiplied by the 3 and 7 and gets summed, and then the 1 and 2 gets multiplied by the 3 and 7 and gets summed. There is a multiply-accumulate associated with every single one of these entries in the matrix. We'll draw out these 4 multiply-accumulates.

Just to make sure I understand why there are 4 multiply-accumulates: each entry in the column that corresponds to the output vector is a dot product, and in this case it will be 2 multiplications and then the addition of those 2 multiplications. You're accumulating—

Really, there's only 1 addition per dot product, but we like to start with 0.

But it includes the initialization of 0.

Yeah. We want to have quadratically more compute. We have x times y as much compute as we had before, but we want to aim for having only x times as much communication. The intention is to get this advantage term going as y.

We've laid down the multiplications. We want to bring in a vector of size 2, and that is already in line with our columns target. That's fine. However, we need to manage the communication of this matrix, which exceeds our budget of x.

The idea is that, in an AI context, this matrix is going to stay fixed for a long period of time. We've got some register files sitting over here. The amount of stuff coming out of this register file—this is the term that we want to go as x, in some sense. We don't want to bring this full matrix in from the register file every cycle, because that would cost too much in terms of wiring from the register file.

Our key trick is that this matrix can be stored locally in the systolic array. We'll store these numbers 0, 1, 2, and 3 in a gate called a register that physically stores these numbers, and we're going to reuse these numbers over and over again for a large number of different vectors.

The optimization here is that the nature of matrix multiplication is that you can store this square, quadratic thing directly where the logic is happening, which has an extra dimension compared to the inputs that you keep swapping in and out.

That’s right. This is the nature of what a matrix multiplication is. You do a lot of multiplication to get 1 value out. A dot product is the result of a lot of multiplications.

So that optimization means you can stuff a lot of multiplication in before you get some value out of it.

That's right. Just to complete the picture of concretely how that looks: I swapped the 3 and the 2 here. Just like this 0 and 3 is going to multiply by the 3 and the 7, we're going to form a dot product along columns here. We're going to feed a 3 and a 7 in here.

This feeds into this multiplication and also feeds into this multiplication. Likewise, the 3 feeds into here and also into here. Then we're going to sum along here. Starting at the top of a column, we feed in 0s, and then, coming out the bottom, we get results.

Visually, there's a dot product performed along columns in a matrix, and that maps exactly to what is done spatially in the systolic array. This is 1 dot product summed vertically, and this is a second dot product also summed vertically.

What is the data that needs to go into and out of the register file? We have x amount of data coming out on the output, and we also have x amount of data coming from the input. With respect to the input and output vectors, at least, we've met our goal of having only x as much data going in and out of the register file.

This leaves open the question: I said the weight matrix is stored locally in the systolic array, so how did it get there in the first place? At some point, you need to boot your chip and populate this data, so where did that come from?

The trick is that we just do it very slowly. We very slowly trickle-feed it into the systolic array. The simplest strategy is that we run this daisy chain: feed a number in here, and on the next clock cycle it will move down to the next entry of the systolic array.

We can do that in every column in parallel. This is also going to come from here, and that gives us another factor of approximately x units of bandwidth coming in.

Would you mind repeating that sentence 1 more time?

We know that we're going to be bringing numbers only rarely into the matrix. We just want to come up with any construction at all such that the amount of wiring that crosses the boundary of the systolic array is bounded to x and does not go as x times y.

A particularly simple strategy is that we bring a number into the top row of the systolic array in 1 clock cycle. Then, for y consecutive clock cycles, we bring in the top row every time and shift all the other rows down by 1. This keeps the wiring that needs to come from this expensive register file only down to a factor of x rather than x times y.

I see. There are 2 questions in terms of communication: communication time and communication bandwidth. You're saying that since we're only going to be loading this in once, let's minimize bandwidth, because bandwidth equals die area. We load it in slowly over smaller lanes because we're just going to keep this value in there for a while.

Exactly.

It's interesting to me that when we were talking last time about inference across many chips, the big high-level thing we're trying to optimize for is increasing the amount of compute per memory bandwidth—that is to say, per communication. Here also, we're trying to increase the amount of actual multiplies or additions relative to transporting information from registers to the logic. In both cases, you're trying to maximize compute relative to communication. This shows up all the way up and down the stack.

This is close to the bottom, to the gates. There's a version that's maybe even closer to the gates in the precision of the number format that you choose to use. We saw that same effect. There's a squared-versus-linear term going on both purely in the precision of the ALU, but also in the size of the matrix.

This unit is the next bigger unit. We had the multiplication circuit, and on top of that we have a pretty large systolic array. I drew it as 2 × 2, but older TPUs were described as 128 × 128 of this circuit shown here. This ends up being the most efficient known circuit for implementing a matrix multiply.

We've talked about how it seems obvious that you should try to maximize compute relative to communication.

What are non-obvious trade-offs that keep you up at night, about whether you should do X or Y when it’s not obvious what the answer is?

Most of the decisions in chip design are sizing decisions. Already in what we’ve drawn so far, AI chips all have this circuit in them. They have a systolic array and, somewhere near it, a register file providing inputs and outputs.

Even within this scope, the sizing questions you have are: How big should I make my systolic array, and how big should I make the register file? These 2 questions are coupled. One way to think of it is to set a budget for what percentage of chip area you want to spend on data movement.

Maybe I say that I want this to be 10% and the systolic array to be 90%. Then I can size my register file. Bigger register files are more flexible. They allow me to get more application-level performance out, but they take away from the area spent on the systolic array.

3. Clock cycles and pipeline registers

Where does the clock cycle of a chip come in? What determines what that is, and what is a clock cycle of a chip?

At baseline, it’s worth observing that chips are incredibly parallel. You’ve got 100 billion transistors in a chip. A key thing you need to do whenever you have massive parallelism is synchronize between the different parallel units.

In software, typically you have these very expensive synchronization methods, like a mutex. One thread will finish what it’s doing, grab a lock stored somewhere in memory, and notify the other thread that it’s done.

On chips, we take a very different approach. Every nanosecond or so, all circuitry in the chip will pause for a moment and synchronize. That is the clock cycle. The entire chip typically goes in lockstep to the next operation in one fell swoop.

What this looks like in circuitry is that the clock is mediated by registers, which are these storage devices we’ve drawn elsewhere. The way to think of it is: I have some storage holding a bit, which might be 0 or 1. Then I have some cloud of logic, which maybe is this systolic array or multiplier.

I have a bunch of inputs feeding into this cloud of logic, and eventually there’s going to be some output register that it writes to. There is a global clock signal driving all these registers. At a certain instance in time, when the clock strikes, whatever value happens to be on that wire at that instant is what gets stored.

The challenge is that I would like to have my clock speed run as fast as possible. If I run at 2 gigahertz, I get twice as many operations done per second as if I run at 1 gigahertz. But what that ends up meaning is that I’m very sensitive to the delay through this cloud of logic, because any computation happening in there needs to finish before the next clock cycle hits.

A major point of optimization on any chip is to make this delay as short as possible.

Interesting. The constraint here seems to be that if you add too much logic, you might risk missing the clock cycle. But if you don’t add enough, you’re leaving potential compute on the table.

Is there ever a situation where you take a probabilistic chance that a computation finishes, or is it strictly that it either finishes by the clock cycle or it doesn’t?

In standard chip design, you margin it such that there is a probability, but it’s many standard deviations out. For all intents and purposes, it is a reliable part and will always meet the clock.

There are some weird exceptions, like clock domain crossings, where you go from one clock to another. Then you actually do have to reason about this probability. But in the main path, you margin it such that you’ll get there 25% of the clock cycle in advance, making it very unlikely that it misses.

Where the clocks synchronize, where the registers are, is this something you determine as a chip designer? Or is it an artifact where you want a certain sequence of logic, and the software you use to convert your Verilog into what you send to TSMC just determines that, to make it work, you have to put a register here, here, and here, making sure no single step makes the whole chip’s clock cycle longer than it has to be?

Inserting them is actually a huge part of the work of designing a chip. It’s done by a combination of manual and automatic methods.

To show the very dumb version of what you can do here, you can take this logic and split it in half. Instead of just one cloud of logic, I can have 2 smaller clouds of logic that do the same thing, but split them up by a register.

If you split it in the middle, you can hit twice the clock frequency. That’s great—you get twice the performance, but at the cost of an extra register, which means more storage.

Stepping back, why do we need to synchronize the whole chip? If you imagine playing Factorio or something, there’s no global clock cycle. Things are just done when they’re done. There’s iron on the plate, and you can take it if you want.

Taking that analogy, the thing you need to be mindful of is if I have 2 different paths through some logic. Say I have to do computation f here and computation g here, and they’re going to meet for computation h.

There’s going to be manufacturing variance. In some chips, f will take a little longer; in some chips, g will take a little longer. If I have a signal propagating through, and the results from f and g have to meet up at h, what can go wrong is that f gets there early and it meets the previous value of g, or the next value of g.

Ah. And h needs to know when to start, when the next iteration has—

Exactly.

This explains why different chips made at the same process node, the same TSMC technology, can have different clock cycles. 2 chips made at 3 nm might have different clock cycles based on whether they were able to optimize to ensure no single critical path is so long that it slows down the whole chip’s clock cycle.

That’s right. This optimization I showed here is called pipeline register insertion. We’ve inserted a register in the middle of the pipeline. This is a pure trade-off between clock speed and area. That is the easy case.

There is a harder case too. I drew out a pipeline of logic, but in other cases you may have some calculation which actually feeds back in on itself. It runs some function f and then writes back to itself.

For example, this might be an addition where you’re adding a number every clock cycle. This little circuit essentially sums all the numbers presented on different clock cycles.

The challenge is, if this plus takes too long, what can I do? If I try to put a pipeline register right in the middle of it, it changes the computation that’s done. Instead of forming a running sum of everything that comes in, I will actually have 2 different running sums. I’ll end up with a running sum of the even numbers and a running sum of the odd numbers.

This constraint—where I have a loop in my logic, which all chips have somewhere—is the hardest thing to address and sets the clock cycle.

I don’t understand why it would be a problem to have that. I’m not even sure what it would mean to have a register there. Is it a sort of atomic operation?

Well, plus is not really atomic, as you just demonstrated. It took a whole lot of work to do a summation. You can take the early parts of that work, stick a register in the middle, and then take the late parts of that work.

Okay. TSMC offers a PDK, which specifies the primitives of logic they can grant you in the chip.

It's up to them to determine that no primitive is bigger than the clock cycle they're hoping a process node targets. But other than that, can't you just take all the primitives from TSMC and keep adding registers between them as much as needed until you get to your desired clock cycle?

As a logic designer, the chip architect sets the clock cycle. For example, the primitives you get from TSMC are on the order of AND gates or full adders. It depends a lot on voltage and which library you choose, but generally you can have about 10, 20, or 30 of these sequentially in a clock cycle.

These primitives are very fast, maybe 10 picoseconds. As a logic designer, in principle, if you just had a register and an AND gate in a loop, you could get an insanely fast clock speed—more than 4, 5, or 6 gigahertz. But if you take this really simple circuit and look at the area you're spending here, this is called 1 gate equivalent in size, so a unit of 1 in area. This thing is maybe a unit of 8 in area.

Again, almost all your cost becomes synchronization or communication cost compared to the actual logic. This would be a case where you've gone too far. You've made your clock speed really fast at the cost of spending almost all of your area on pipeline registers.

Interesting. So you're hinting at a dynamic where you can have a really fast clock speed, but you're not getting that much work done. You can have low latency but low throughput. It hurts your throughput, in fact, because the throughput of your chip is the product of how much you get done per clock cycle—which is based on area efficiency—times how many clocks you get per second.

This is actually so similar to the thing we were discussing last time about batch size, where if you have a low batch size, any one user can receive their next token really fast, but the total number of tokens processed in, say, an hour will be lower than it could otherwise be.

Exactly. You get less parallelism out if you drive your clock speed up really high.

4. FPGAs vs ASICs

I remember talking to an FPGA engineer at Jane Street, Clark, who helped me prep for the previous interview we did together. He was explaining why they use FPGAs. I imagine that for high-frequency trading, throughput is less important than latency, so having very specific control over the clock cycle in a deterministic way is the most important thing.

Maybe it'd be interesting to talk about why you can't just achieve that with an ASIC, or why you might use an FPGA to have deterministic clock cycles for high-frequency trading.

Let's consider the business case for an FPGA versus an ASIC. FPGAs and ASICs use largely the same conceptual model. You have a series of gates built from small primitives—ANDs, ORs, XORs—connected together with wires running in a fixed clock cycle. Anything you can express in an FPGA, you can express in an ASIC too.

It will be about an order of magnitude cheaper and have better energy efficiency on an ASIC than an FPGA. The trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs $30 million because it requires an entire tape-out.

The business use case for an FPGA is when you want something that has very deterministic latency, fast runtime, and high parallelism, but you are going to change the workload frequently, maybe every month. You don't want to pay that tape-out cost every time.

How does an FPGA actually emulate the ASIC programming model in a fixed piece of hardware?

At its core, it has the 2 components we just talked about. It has registers as storage devices, and it has lookup tables, or LUTs, which provide all of the gates. Then there's a 3rd component. We have a swarm of these registers and LUTs, and they are connected by a big set of muxes.

In front of every single one of these, we have a mux which selects an input from everywhere else. We have a whole bunch of different options feeding into all of these things. What this allows is essentially that when I program my FPGA, I can take all of these components and superimpose a particular wiring which goes through this LUT, feed it into another LUT, send it to this register, and then feed it into another LUT, or something like that.

FPGA means Field-Programmable Gate Array. The orange is what has been programmed in the field, whereas the white is all the wires that must exist in the FPGA in order to actually make the device in the first place.

What does it mean to be programmed in the field?

Programmed in the field means the device is being deployed in a data center. It's sitting out in the world, and then you can come and program it.

Ah, not field as in an electric field. Field as in out there in the world, okay. If I look at how the field programming comes out of the first lookup table and goes into a second one, how does that work? Where are the wires that make that happen?

I got a little bit lazy in drawing all of these. Every single device here has a mux sitting in front of it, which can select from all of the nearby circuits that are available. The actual configuration of the FPGA amounts to the mux control.

In this mux, we have the data inputs, and we have the control that selects. There's a little storage device sitting next to every single one of these muxes saying, “This is where you're going to source your input from.” Programming it consists of configuring every single one of these muxes.

That makes sense. What is happening inside of the lookup table?

The lookup table is also going to have a little bit of control telling it what to do. Its purpose is to configurably take the role of an AND gate, OR gate, XOR, or any of those different things. There are many ways you could consider doing that. The way it's done in traditional FPGAs is that a lookup table has 4 bits of input and 1 bit of output.

How many different functions are there from 4 bits to 1 bit?

There are 16 different functions. You can tabulate this as 16 different numbers. You've got a table of 0s and 1s, 16 entries. This table is stored in these blue configuration bits. It views these 4 bits as binary, looks up the relevant row of the table, and emits that bit. This is essentially a truth-table view of lookup tables.

Okay, so if you think about an AND gate, OR gate, NOR gate, or XOR gate, these all take 2 inputs.

Those are 2-input functions.

Sometimes we have a 3-input function, like a 3-way XOR, or a 4-way XOR. In this case, does it just depend on how big it is? There's another compute-versus-communication trade-off here.

If it has too few inputs, you need to use more LUTs. Basically, the lookup table is a truth table. With a truth table, you can program in any gate you want.

So instead of a lookup table, you can just think of it as a programmable gate.

That's right. One of the things you can do here is see where the rule of thumb that an FPGA is an order of magnitude more expensive than an ASIC comes from. You count how many gates would be inside this lookup table.

We can view this lookup table essentially as one of these muxes. It has to select between 16 different values, so it's a mux with n = 16 options and p = 1 bit. As we saw earlier, this circuit costs n times p gates. So it costs np, which is 16 AND gates, and also 16 ORs.

This circuit is the mux?

Exactly, the mux.

The mux that goes into the lookup table?

The lookup table itself you can think of as being a big mux that selects from all 16 rows down to 1 output.

That's the lookup table. But the way you've drawn it here, there's a mux and then a lookup table. It's muxes all the way down. There is a second mux that is inside here. This mux is this mux, and the other mux is just saying where it came from in this mess of gates.

Right, and the second mux is, “Okay, now you have one value, but that value is still a 4-bit value.”

Yeah, I've selected 4 bits from the soup. Then I use those 4 bits to select which entry in the lookup table I'm going to use.

Suppose in the first mux you're pulling from 8 nearby registers as input. That's a total of 32 bits going in. Out of that, 4 bits come out. Those 4 bits go into the second mux, which is inside the lookup table.

In this case, these registers are single-bit registers. If there are 8 nearby registers and lookup tables, then I have 8 bits total coming in nearby. I select from 8 down to 4 different values. There are actually 4 different muxes, a little mux associated with each of these input bits. Each of them is selecting 1 out of 8.

Where are those 8 coming from?

Nearby registers and other LUTs. Each register is 1 bit.

Yes. I guess AMD, or whoever makes these FPGAs, still has to be opinionated about which registers are connected to which registers. You can program in the actual gates, but they add a wire in the communication topology, right? You get flexibility at a local grain. There's a nearby neighborhood you can select from, but for more coarse, long-distance connections, they form an opinion on that.

And the reason it's 10× slower is why?

If you look at the cost of building this lookup table, it's 32 gates. It can give me the equivalent of—what's an interesting thing I can do here?—a 4-way AND gate. A 4-way AND means AND, AND, and then an AND of an AND.

This is a circuit I could implement in an ASIC directly using 3 AND gates. Using a LUT, I can also implement it, but it's going to take 32 gates instead of 3. The overhead is really coming from the fact that there's a more concise way to describe a truth table than listing out every single possible combination of inputs, which is just to write out the gate.

5. Cache vs scratchpad

Yes, to place down the polysilicon and the wires and so on. Interesting. One important point you made to me is that the reason they prefer FPGAs to CPUs is that they get deterministic clock cycles. They know when a packet will come in and go out. Why isn't that a guarantee in CPUs?

You can actually design a CPU that has deterministic latency as well. In fact, the processors inside a lot of AI chips also have deterministic latency. Groq has advertised this. TPUs have that in the core as well.

The challenge is getting deterministic latency and high speed at the same time. Non-deterministic latency comes from specific design choices in a CPU. It's actually possible to remove those design choices and make a CPU with deterministic latency, but those are not very attractive in the market, so people don't make those CPUs anymore.

In some sense, deterministic latency is a simpler starting point, and some chip designers have added things in to make it non-deterministic. To take a concrete example, probably the most important source of non-determinism on a CPU is the CPU cache itself.

In a CPU, you have the CPU die itself, and then DDR memory off on the side. You have a cache system inside that remembers recent accesses to DDR and stores them. When I'm running through my CPU instructions, every time I have an instruction that accesses memory, it first checks if the data was stored in the cache. If not, it fetches it from DDR.

This is a huge optimization. The cache is 2 orders of magnitude faster than the DDR. If you never used the cache, basically all programs would run 100 times slower. The presence of a cache is absolutely necessary for a CPU to run at a reasonable speed.

But whether or not you get a cache hit depends on the ambient environment of the CPU: what other programs are running, what has run recently, and what the random number generator inside the cache system is doing. That is a big source of non-determinism in the runtime of a CPU.

That is the memory system for a CPU. The big thing you can do differently is, instead of having the hardware say, “I'm going to read memory,” and then the hardware decides whether or not it comes from the cache, you can bake this decision into software—a different design philosophy. You see this in TPUs, for example. I'll draw the same diagram, but I'll call it a scratchpad.

This would be a TPU, and you have HBM in this case rather than DDR, but it's still an off-chip memory. Instead of the software saying, “First access memory,” and letting the hardware decide, you have one kind of instruction that goes to the scratchpad and a totally different kind of instruction that goes to HBM.

This style is generically known as scratchpad instead of cache. The key distinction is that you have one kind of instruction that says, “Read or write scratchpad,” and a totally different instruction that says, “Read or write HBM.”

So scratchpad being the cache.

Yeah, this thing here is the scratchpad.

6. Why CPU cores are much bigger than GPU cores

Stepping way back, people say computers have the “von Neumann architecture,” where there's this serial processing of information. Maybe it's just because we've been talking about parallel accelerators, but the FPGA is super parallel. The AI accelerators, the TPUs, are super parallel. Even CPUs are super parallel if you think about all the cores they have.

In what sense is modern hardware actually the von Neumann architecture? Is that actually a fair way to describe modern hardware?

I think it's a fair way to describe CPUs. The amount of parallelism you get on a CPU is about 100 cores times maybe 16-way vector units, so about 1,000-way parallelism on a CPU.

One question: there is a die being used for the CPU, and if there are fewer threads, just as a matter of transistor voltages switching on and off, is it that there's literally one control flow—a small part of the die—where voltages are switching on and off? How do you actually occupy the die area of a CPU? If there are so few cores, what are you spending all of the die on? Yeah, what is happening there?

The cores are just much bigger and more complicated. We should compare a CPU core, which takes up 1/100th of the die, to a LUT. A LUT is only 16 gates. It's clear why there are so many more LUTs in an FPGA than cores in a CPU.

But why are there more CUDA cores, for example, than CPU cores? What's the difference between a CPU and a GPU?

Inside the CPU, one big use of the area is the cache. Very little is actually the ALUs. Mostly it's these register files rather than the logic units. Both of those have equivalents in a GPU, so that's not a big difference.

But the thing that does not have an equivalent in a GPU is the branch predictor. There is a whole big area in the CPU which is just a bunch of predictors saying when the next branch will be and where the branch target is. Stripping a lot of that out, as well as making these register files tighter, drives a lot of the GPU gains over the CPU.

What is the purpose of the branch predictor? To execute both branches at once, or what does it do?

The issue is that when I've got a series of instructions, if I have a branch, the actual step of processing an instruction takes a really long time. It takes maybe 5 nanoseconds. The time to notice that I've got a branch, evaluate whether the Boolean is true, update the program counter to the new target, and then read from the instruction memory could take 5 nanoseconds to finish.

So in reality, this may finish somewhere down here. I want to run a clock speed that is much faster than what 5 nanoseconds allows. 5 nanoseconds is a 200 MHz clock speed. I would like to run at 1 or 2 GHz, so I need to run other instructions while the branch is being evaluated.

I just want to keep running the instructions that happen after me. But that might have been wrong. If the branch ended up being taken, then I need to know that instead of evaluating these instructions, I actually need to jump to wherever the target is and run those instructions instead.

The purpose of the branch predictor is to predict, 5 cycles earlier, that a branch is going to happen, before you even get to that instruction.

7. Brains vs chips

If I think about how the brain works versus what you're describing here, at a high level the differences might be that while you can do structured sparsity in these accelerators and save yourself some area that you would have otherwise had to dedicate to gates, in the brain there's unstructured sparsity.

Any neuron can connect to any other neuron, and not in ways where they have the columns aligned. Then there's the fact that memory and compute are co-located. Although I guess you could say, in a way, the memory and compute are co-located on these dies too. This is exactly the co-location, in some sense, of the memory and compute. So maybe that isn't a big difference.

Another big difference is that the clock cycle in the brain is much slower than on computers. Partly, that's to preserve energy, because the faster the clock cycle, the bigger the voltage needs to be in order for the signal to settle and to identify what state a transistor is in.

That's right.

I don't know if you have any commentary on what the brain might be doing versus how these chips work.

Let's take the clock speed one first. The clock speed is quite high on a chip because that drives higher throughput. When we compare a GPU running some workload, it's running at a batch size of 1,000, whereas the brain is not running at a batch size of 1,000. There's only one of me.

You could imagine saying, “Take a GPU and, instead of running at a gigahertz, run it at a megahertz,” and that would start to look a little more like the equivalent things you're talking about in the brain. But in the way silicon works, that does not give you a 1,000× advantage in energy efficiency.

What it ends up looking like is you just run this circuit once to stabilization, and then it will sit idle for a long period of time. It doesn't consume a lot of energy while it's sitting idle because most of the energy is consumed in toggling bits from 0 to 1 and back.

Let's talk about the energy consumption of a circuit like this. The way to think of a bit being stored is that you've deposited some charge in a capacitor sitting somewhere in the chip, implicitly. It becomes charged when the bit becomes a 1, and then it becomes discharged when it next goes to a 0. That cycle of charging the capacitor and then dumping that charge out to ground is where the energy is consumed.

This is called the dynamic, or switching, power, and it's most of the energy consumption of a chip. There is some other energy consumption just coming from the fact that insulators aren't perfect, but we'll discard that. Most of the energy consumption comes from toggling from 0 to 1 and back to 0.

If you run a chip much slower and you only clock it once every 1,000 clock cycles, you will have 1,000 times fewer transitions. It will be about 1,000 times less energy consumption. But it's not a substantial advantage in energy efficiency.

8. A GPU is just a bunch of tiny TPUs

Okay, so you described how a TPU works at a high level. What is the difference, at a high level, between how a GPU and a TPU work?

There is a high-level organization principle that is different, and then inside the cores, things are different. Looking at the high level, we'll take a GPU and a TPU and see what the top-level block structure looks like.

If you think of this as the whole chip in each case, the organization of the GPU is mostly a bunch of almost-identical units, which are the SMs. They've got an L2 memory in the middle, and then a bunch more of these SMs on the bottom. So there is a fairly regular grid of cores.

If we look at a TPU in comparison, you end up with much coarser-grained units of logic. You end up with just a few matrix units, which are the big systolic arrays. In the middle, you've got some vector unit, and then you've got your matrix units at the bottom. These matrix units, with a vector unit in the middle, make up the whole TPU chip.

You can think of scaling this thing down into a really tiny unit with a smaller matrix unit and a smaller vector unit, and that is sort of what an SM is. From a very high-level point of view, the GPU has a lot of tiny TPUs tiled across the whole chip.

Oh, interesting. You're suggesting the tensor core within a streaming multiprocessor is analogous to an MXU?

Yeah, it's all very similar.

I see. If you had more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you just have huge matrix multiplications, you might want to avoid the cost of having individual SMs with their own registers and warp schedulers. Why not just make a huge thing and amortize those costs across the whole thing?

This shows up in how large you can grow things. We've seen this theme, especially with the systolic array, where a larger systolic array amortizes the register file costs better. This design allows you to have larger systolic arrays, whereas the GPU design constrains you to having small units of everything.

There is a trade-off, however. Because of this coarse-grained separation of things, you need to move a lot of data from the vector unit to the matrix units through just 2 lines of perimeter here. If you look at the equivalent thing in a GPU, you've got vector units everywhere, and you can move data through many different lines.

The amount of data you can move between a vector unit and a matrix unit is actually much higher in a GPU than in a TPU. Instead of having to move all the data through just 2 lines, you're moving it through 16 lines of wiring in a GPU.

Right. But also, you might have to move across less area, which is an energy saving as well. So if you can operate entirely within an SM, the data movement is much smaller. But the moment you want to operate across SMs, it becomes more complicated and expensive.

So you don't have to comment, but one might expect that a thing MatX might try to do is get the GPU-like smaller structure of systolic arrays surrounded by SRAM, but at the same time make it so that the things you need in an SM to support the CUDA architecture—which take a bunch of space—you might discard.

We've talked publicly about something we call a splittable systolic array, which, in some sense, you can think of as big systolic arrays that can be small systolic arrays too.

Cool. Okay, I think that's a good note to close on. Reiner, thank you so much.

Thanks, Dwarkesh.