第035期 - 技术尽调、性能预测、供应链、投资逻辑（咨询）

Hello, everyone. Welcome back to SemiAnalysis. My name is Jordan, and I'm here with Abhilash. This week, we're going to do something a little bit different. We're not talking about an article. Instead, we're giving everybody an explanation of the SemiAnalysis consulting business.

Abhilash leads our consulting business. He's got a great team, and a lot of people have questions about what we do in consulting, what types of clients we serve, what projects we do, and what we've learned. This covers everything from diligence for investors or companies that are looking at M&A activity, to investors who want to go beyond the core products and customize something about one of our research products, like the accelerator model or data center model, as well as strategy guidance, feedback on product roadmaps, and go-to-market activities for lots of clients and industries.

So, Abhilash, welcome to the show. I'm excited to have you today.

Thank you, Jordan. I'm excited to finally make my debut on the renowned SemiAnalysis Weekly.

Abhilash and his team have lots of opinions that get expressed on this podcast, but for the first time, we're going to hear it from him directly, so I'm excited for that.

Absolutely.

Okay, give us a quick overview. What's the SemiAnalysis consulting business? What do we do?

1. Consulting Becomes A Dedicated Team

SemiAnalysis has been doing consulting ever since Dylan started SemiAnalysis, but it was only last year, I would say, when we set up a dedicated team. The reason is that SemiAnalysis does a lot of things. We publish our newsletter, and we have institutional models. I think we have close to 13 to 14 different institutional models right now.

A lot of this stuff is off the shelf. We have open projects and open research projects like Infant X and ClusterMax. There's a lot happening, but if any industry participant or investor wants to engage more deeply with SemiAnalysis, we didn't really have an operating model internally to service those requests. That's the genesis of SemiAnalysis Consulting.

What we do is work with various clients on a very dedicated basis. All the work that we do is custom. We typically staff a dedicated team for anywhere between 3 to 4 weeks, all the way to a year. I'm happy to talk more about it, but that's essentially what we do. All the work that we do is custom.

Yeah, awesome. Custom work.

Yeah.

Maybe start by talking about the types of clients that we serve. I gave a high-level overview of the different types of projects that we've done, but take us through that.

2. The Tech Finance Client Base

In my opinion, I think what we are witnessing is something called hardware-software co-design. We always talk about hardware-software co-design, right? I think in the world of AI infrastructure, there's something else that's also happening, which I call tech-finance co-design.

We need to teach tech bros more finance. We need to teach finance bros more tech, and that's essentially what we do. I think there are 2 sorts of client groups that we serve.

The first one is financial institutional investors. These are large private equity funds and large hedge funds who are very savvy investors and very excited about investing in AI infrastructure. When they're deciding to drop $1 billion or $2 billion into a chip company, a neocloud, or a data center, they want SemiAnalysis to come in and help them figure out various aspects of the deal.

We do various sorts of diligence for them. We also do a lot of thesis development for hedge funds. A lot of hedge funds are looking for the next wave. People got excited about memory, and people got excited about CPUs, so they're trying to think about what's next. We've also worked with a lot of hedge funds to combine various data sources and different market narratives to figure out what's the next wave.

On the other hand, we also serve a lot of industry clients. These are largely the hyperscalers, the AI labs, chip companies, and semiconductor companies. Here, the range of topics can be really, really large.

It could range from helping a semiconductor company figure out its product and engineering roadmaps. A common question is: If LLMs are evolving in a certain way, what products should we be thinking about 3 years, 4 years, and 5 years out?

It could also be helping a certain hyperscaler figure out data center sites in remote locations in India, for example. It could also be doing go-to-market strategy for various companies that have a strong product but don't know how to connect with and figure out their customer journeys.

It's a range of different options, but that's essentially the type of clients that we serve as part of SemiAnalysis Consulting.

Makes sense. And quick advertisement for Abhilash on his behalf. If you're listening and you're interested in tech finance co-design, you should consider applying to work on consulting at SemiAnalysis. It's a lot of fun.

We are hiring a lot, so shameless plug: if you're AGI built, excited about semis, and curious to learn, please hit us up. We are desperately in need of people.

Take me through some of these customization examples. The place where I get involved in some of this work is that we're taking research that other members of the team and I are doing and trying to package it up in a more digestible way, as well as customize it for somebody's specific interests. Take me through what that process looks like and how we lean on our existing data sources to execute on these projects.

3. Custom Models Answer New Questions

At SemiAnalysis, we have so many different things happening. Usually, if you're able to combine some of these data products in a more comprehensible manner, we're able to answer questions that each of these data products is not able to answer by itself.

One example was a hedge fund client that was very interested in understanding the total token supply in the world, assuming the chip install base. If you think about it, it's a fairly simple equation. What is the total install base in the world today, and how is it expected to grow in the next 5 years? How much of that capacity is allocated to training versus inference?

If you have the install base, the second part of the equation is the throughput per GPU. How many tokens can each GPU generate in a certain day in a given year? Finally, what's the pricing in terms of dollars per million tokens?

Each of these 3 different parts of the equation was answered in a different manner, but the most complex was estimating the throughput per GPU. We had to combine a lot of our inference X data that's already public with chips that are not out yet and models that are not out yet.

We had to build an inference simulator to estimate what it would mean if NVIDIA came up with Trainmen with a certain set of characteristics. What would it mean for the throughput? If memory gets de-specced, as has happened, what's the impact on throughput? If OpenAI trains a 10-trillion-, 15-trillion-, or 20-trillion-parameter model, what does it mean for throughput?

A lot of this is not readily available. It's something that we had to build out as custom work, and we had to work with various inference engineers at SemiAnalysis to do that. That's an example of a project that would be custom work.

Okay, take me through some of those assumptions in the inference simulator. Maybe explain how it gets delivered to the clients, because I think a lot of people hear “consulting” and think “slide deck.” We make plenty of slide decks, but for complicated questions like this, the trend for us has been to move toward live dashboards where people can edit their assumptions over time, and we have to maintain this. It's kind of like a software product that we're delivering to them, right?

Exactly. There were so many permutations and combinations in this product that it wasn't feasible for us to put it in a static Excel file. We had to build a dynamic dashboard, and that was essentially what was delivered to the clients.

This is a dashboard that they could use to play around with different sets of assumptions in a very visual manner and see literally every number changing. The key assumptions were, to start with, how much of your compute are you allocating toward inference versus training?

We know the existing chip specifications part; what are the chip specifications that you're imagining for the next set of silicon being developed by AMD, TPUs, and NVIDIA?

Yeah, we know from one of the episodes two or three episodes ago that those specs can change pretty quickly. Rubin Ultra just got de-specced from 1 terabyte of HBM down to 66, 128, or 192, so...

Yeah, yeah. Another set of assumptions could be around your input-output ratio. Are you using agent X harnesses? Are you using fixed sequence lengths, context lengths, or data types? What's your cache hit ratio? How does throughput change if you're hitting 80% cache versus 85% versus 90%? I think those are some of the assumptions that make it interesting.

Yeah. Makes perfect sense. Now, tell me more about an example that goes in a different category, which is strategy. This could be on product roadmap, or it could be a business decision about build versus buy versus partner. What are some of the domains that you're helping people with right now?

4. Strategy Guides Products And Sales

Yeah. I think strategy is a very broad term, I guess. It could mean product strategy, or it could mean go-to-market strategy, but I can give you an example of both.

As I said, one of the things that we have built internally is an inference simulator, and one of the memory storage companies is essentially using that simulator to replay agentic traces for a combination of different models and chips to figure out the next set of memory products that they should be developing. They're answering questions like, how do memory bandwidth versus capacity trade-offs play out in serving production inference workloads? That's one example of a product strategy project that we're doing, and we are working with the teams on a very dedicated basis.

The other example could be that we are helping one of the semiconductor companies figure out their ideal set of customers. They have a product, but they don't know who to sell to. There are probably 100 customers that could buy their product at any given point in time, but it's a long-lead-time product. It takes at least 3 to 4 months to qualify a customer, so it's not realistically possible to go and qualify your product with 100 different customers.

We are helping them figure out who the top 10 or 15 customers are that they should be prioritizing, what's the right way for them to approach sales conversations, what are the key technical topics that those customers would be interested in, and how should they position themselves to be able to deliver on those technical specifications that these customers want? It seems easy to say. It's very difficult to execute, because we need to go out and literally speak with 20, 30, or 40 different customers, get their perspectives, and aggregate a market view.

This is something that a lot of different companies are not able to do because SemiAnalysis is so well connected. We don't really use an expert agency to carry out these expert calls, or whatever you call them. Because we have such a wide network, we are able to just use our relationships to figure out important questions that would help the wider ecosystem.

Yeah, and this also informs our research. Maybe you can explain a little bit about how people who have been on this podcast in the past, like Eric a few weeks ago working on modular data centers, when they go and do this research effort, it contributes to SemiAnalysis products.

In many ways, some of the institutional products that we sell, the data products, originally started as a consulting project, or at least the inspiration to start working on them actually started because somebody was asking us about this. They clearly think it's important, so let's go do it, right?

Right. Because we work on a very custom basis with a lot of these clients, and we go very deep on a topic, oftentimes we are able to identify that there is a broader opportunity. If this question or problem is relevant for one client and we have been able to figure out a standardized solution for it, I think we're able to scale that up to a broader set of clients, and that becomes a product.

I think our data center model started as a consulting project for one of the clients. Maybe ClusterMAX started as a consulting project. I don't know. You have to tell me that.

No, not ClusterMAX, but the AI Cloud TCO model definitely—

Yeah.

—started with the consulting projects, which we continued to use to build analysis on how, what are the input costs for GPU clusters? How profitable are given companies that are serving tokens based on the choice of chips, based on the amount of OpEx, and based on the utilization ratio on an inference endpoint or something like that? The way we—

Yeah.

—we actually dig in and do that is that you get inspired to do the work because people that you trust, your clients, are taking some problem seriously and asking questions—

Correct.

—and you go get answers. That becomes a useful product. So anyway.

Exactly. I think the other example—the latest example of that—is the inference simulator that we're building, right? We started building that for one client, and not even for one client. I think it started as a cool science project within SemiAnalysis, and then we offered it to one client. They found value in it, and then a few other clients found value in it, and now it's becoming a consulting product by itself. So, absolutely.

Yeah. Makes perfect sense. So how about due diligences? That's obviously a big thing for investors, as well as some companies that are interested in M&A activity.

5. Due Diligence Protects Investors

I think Dan talks about $7 trillion of debt being issued to fund AI infrastructure, right? So there are a lot of investors coming into the game. There are a lot of newer players that are also coming into the game.

What a due diligence product essentially looks like is an investor coming to us and asking, “Hey, can you either help us…” Let's take a neocloud as an example, because that's what we do a lot of the time. If a private equity fund wants to put $1 billion into a neocloud, essentially they're looking for 2 things: a technical analysis and a commercial analysis.

On the technical side, I think what they're interested in is what SLAs the neocloud is offering to the customer, and whether this neocloud will be able to deliver on those SLA guarantees. If they don't, there are a wider set of implications that we can talk about later, but it's not good for any of the parties. They're also interested in understanding what the OEM support and warranty structure and contracting look like, because there's a lot of devil in the details there. If you are not transacting properly with the OEMs, you could expose yourself to a variety of risks.

They're also interested in understanding the cluster performance, so we are able to test their clusters with a ClusterMAX-inspired methodology and tell them exactly what the things are that they're good at and what the things are that they're really, really bad at, and what the 100- to 180-day plan is for them to improve on those things.

On the commercial side, because it's such a financially savvy decision, they need to understand what the CapEx and OpEx are for these builds and how that really compares to the market standard. As part of our TCO model, we are able to benchmark costs in real time, as well as performance through various other means.

Yeah.

We are able to benchmark these things in real time for these clients. So that's what our typical due diligence project looks like. It varies depending on whether you're diligencing a neocloud, a chip company, a model company, a data center, or powered land. We have literally done everything.

Yeah.

I think—

We've done a lot in just a year or so since you've been here, and we're just stacking up experience across it and building the team. It's pretty cool to see it grow as more and more people in the industry take this stuff seriously, because I think the need for technical due diligence on a lot of these projects is of the utmost importance.

All of it's new, right? It's not like certain investors can understand or have the capabilities to understand a type of model that's 6 months old or 3 months old. It's new for everybody, right? Anyway.

Exactly. It's new for everybody, and I also think what's happening is that, in my opinion, the offtaker base is diversifying beyond the top labs to also now include the Neo labs, which are actively transacting in the market.

The neocloud operator base is also diversifying. If you think about it, hyperscalers are the OGs, and then you have the crypto miners and specialist players like CoreWeave and others. But now you see a lot of real estate developers who have no experience operating cloud entering the scene. You see family offices of rich people trying to build a neocloud now.

SLAs become even more important because if you don't have experience operating these kinds of things, and if SLAs are not in your favor, the implications would be really massive.

Yeah. Let’s talk about the 2 types of them. Obviously, this is something I’m quite familiar with. There’s one type of SLA that’s related to delivery timelines. Really, an SLA is just that if something happens, a contract can be terminated. That’s roughly the legalese behind a service-level agreement.

“Contract termination rights” is maybe a broader definition, and that’s really what we’re talking about here, as opposed to SLAs. The first reason why you might be able to terminate a contract is because of late delivery: the cluster shows up months late, so therefore, you cancel the contract. But the most common one is downtime.

If somebody has GPUs installed, they expect to have 3 nines, 2 nines, or 1 nine of availability. If you’re under 90%—1 out of 10 days in the month, 3 out of 30, you don’t have access to the GPUs you’re paying for—generally, you’re getting some credits in response, or you have the right to terminate the contract.

Maybe talk to me about the motivation for the lenders, the neocloud itself, or even the buyer to have some sort of insurance against either of these possibilities of a contract termination. They want downside-risk protection, right?

Exactly. Jordan, you hit the nail right on the head. One, you asked why SLAs are important, and then I’ll talk about why insurance is important.

I think you said it right. The first thing is that, as a neocloud, without strong SLAs, you won’t be funded, period. You won’t be able to start your journey. All the investors and all the lenders care most about is their risk exposure. If they’re getting interest payments every month or every quarter, what is the chance of that payment not hitting their bank account because your cluster was down? That’s the number-one thing.

The second thing is that SLAs are really for the bad times. We have seen this in various settings: if things are going well, nobody talks about the SLAs. If things don’t go well, people talk about the SLAs, and it gives the parties a way to get out of the contract scot-free.

The third thing is that, as we’ve talked about, neoclouds typically sign a 15- or 20-year data center lease. They typically sign the GPU contracts for 5 years. If you’re not committing to or delivering on your SLAs, you carry a reputation risk, which means that you will not be able to get your second deal or your third lease. If you’re not able to do that, then how do you pay for 20 years of data center leases?

SLAs are important. The reason they’re important for lenders, as I said, is that they’re looking for downside protection. One way to protect your downside is by having SLA insurance.

We know a few companies now that are insuring SLA uptime for neoclouds. If you, as a neocloud, pay a certain number of basis points in premium, you get a certain level of protection: if you have to pay an X amount of penalty, the insurance company will take care of it.

It’s not easy, in my opinion, to get this insurance, because these insurance companies are also pretty savvy in terms of how they underwrite these neoclouds. It’s not an excuse to have bad performance—that you can continue to have bad performance and compensate for it by having insurance. If you have bad performance, you won’t even get insurance. But if you have good performance, I think one way to protect your downside and make your contract more investable is to consider some of these insurance products.

Now, we’ve thrown around the term SLA and talked about a few different things. Obviously, there are many different components of a given cluster, data center, or site that can have an SLA attached. This could apply to the GPUs or the nodes themselves, the racks, the clusters, or the site itself. It could also apply to the power systems and cooling systems in there.

When there are so many parties at play and so many contracts changing hands, there are actually a lot of reasons why you might want to have a risk assessment done for just 1 individual part that you’re less comfortable with, compared to some other part of the contract that you have more confidence in because maybe you’ve worked with that vendor before or because you understand the technology better.

Talk to me about the concept of a risk assessment and its scope, because I know I’m leading the witness here, but the scope can be pretty tight, pretty quickly, or it can be pretty big and pretty broad, right?

Right. As you said, a risk assessment could be fairly critical, especially in cases where lenders or investors are demanding it, and that’s a necessary condition for them to fund a particular cluster.

It could be as lightweight as doing a contract review of the most important contracts for a neocloud. These are the offtaker contract, the OEM contract, and the data center contract, because you’re promising something to the customer, and then you’re relying on the data center partner and the OEM to be able to service that. There are other side contracts as well that we can get into, but we don’t have to right now.

The second thing is a slightly higher-touch review, or risk assessment, which you can call data center readiness. That means understanding the milestones around when the data center will clear L3, L4, and L5 testing; when the data center will be made available to you to do a bunch of different other testing; when the racks will arrive at the data center; when you can install power and cooling; and when all of these things can be tested as a system, all at once.

We can talk about other aspects here as well. The third thing is doing a bare-metal review, which, Jordan, I think you are the master of with all the ClusterMAX testing you’ve been doing. I’ll let you do it, and then we can come back for the final one.

The summary is basically that we get access to the systems, log in, test them, simulate failures, and make sure that people have the monitoring systems set up so that they can identify that the failures occurred and how quickly they can recover from them.

There are many different ways in which things can fail in a cluster, and your monitoring systems have to have full coverage, because we see things fail in spectacular ways, even in just our few weeks of testing. This is in a fully contrived environment, where people have all hands on deck to set it up, just through ClusterMAX testing. Then we hear the stories from all the customers of these providers about how poor the quality of service they’re experiencing is.

Yeah. The root cause can be really varied, right? This assessment helps you understand what exactly is the root cause of bad performance. Is it a problem with the way your electricity is set up? Is it the way your cooling is set up? Is it challenges in the racks themselves? Is it challenges in the software?

Ultimately, even if there’s only 1 reason that your cluster is not performing well, that still does not mean that you don’t have to pay the SLA penalty. Only 1 thing has to go bad for you to be liable for paying all the SLA credits back to the customer.

Yeah. That’s why we go back to the contract review stage, which is the first one, right? People need lots of advice on how to set up these contracts because, as you said, contracts are for when there are bad times, not when there are good times.

Okay, let me ask a related question, but it can keep us going here. When we talk about giving strategy advice, a lot of people are getting into the space where they’re considering build versus buy, specifically with the neoclouds, because there are such long lead times and it’s so hard to find a data center site.

We’ve done a little bit of that consulting work, where we’ve helped people make that build-versus-buy-versus-partner decision. We’ve done it in more than just the neocloud decision—for other things as well, such as supply-chain management if you’re a chip company. Maybe you can talk a little bit about those examples too.

6. Build Versus Buy Follows Constraints

We are definitely seeing a lot of constraints around getting colocation capacity to do large site build-outs for GPU clusters. We’re also seeing some tightness around chips, and there are definitely shortages around labor.

In principle, even when a lot of companies want to eventually build their own clusters, that is often not the most feasible path to get near-term capacity. Ultimately, it boils down to a few factors.

The first is time to market. The second is your willingness to pay a premium to get compute. The third is how technically savvy your own team is, and whether you have any experience running cluster operations.

Finally, in terms of financing and everything else, do you have enough credibility in the capital markets to be able to raise funding to set up a cluster from scratch, or to be able to be an offtaker of a company?

Recently, we’ve been working with a few clients where we’ve arrived at a few different answers.

I think in one of the cases, we recently advised a company to just go out and rent because they wanted 100 megawatts literally in the next 6 months, and there was no way they would have been able to build a cluster that fast. But there have been cases where somebody was willing to wait until 2028 or 2029, and the recommendation was completely opposite. I'd say these are typically 3- to 4-week projects, but definitely very interesting from the range of content we are able to cover and the options we are able to explore with them.

Okay. Personal question. What's the project you find most interesting right now out of this whole suite we've been talking about? And what's your favorite? Is that the same thing, or is that a different project?

So you asked favorite and what else?

Most interesting. You just said it's an interesting project. I assume that means you're learning and doing interesting—

Yeah, yeah, yeah. I would say one of the most interesting projects right now that we're doing is where there's a bigger company that's looking to acquire a smaller company. That company doesn't have a very big team, but they have very sharp people. I get paid to speak to a lot of these very sharp people, and I get to learn a lot in the process about deeply technical projects, LLM architectures, kernel engineering, GPU optimizations, fast inference, slow inference, high-throughput inference, and whatnot. I think any of these topics definitely sound very interesting to me.

I think my favorite projects are the ones where we're able to use a repeatable process to get to an answer, because we've done so many of these neocloud diligence projects over the last year that it has just become muscle memory. We know exactly what information request list we need to send out on day 2, what interview we need to conduct on day 4, what 5 follow-up questions we always anticipate landing on day 6, what midpoint check-in and readout we need to do on day 8, and finally, what the conclusion is. So, yeah, it's—

No, no, no, Abhilash, the conclusions are not predetermined. You can't be saying that on the podcast, man.

You'd be surprised how often they are.

For the neocloud diligences, do you think it's pretty straightforward to analyze some of these contracts and come to pretty clear results right away?

Yeah.

Yeah. Kind of cookie-cutter at this point. But it makes sense. It makes sense that people want the peace of mind that we've reviewed a lot of the stuff. Okay, I'll ask you a broader question. What are you most excited about going forward with consulting?

I would say we're definitely a very, very fast-growing organization. I think the breadth of topics that we're covering today versus 6 months ago is a lot higher. What I'm more excited about is partnering with people who are building at the frontier of AI. We already partner with people who are building at the frontier of AI, but getting more opportunities to partner with such people is definitely the most exciting thing because you get to learn so much in such little time.

It's crazy how fast the knowledge graph and the learning graph here at SemiAnalysis are growing. The other thing I'd say is that Dylan has told me to hire 20 more people in the next 6 months. I don't know how I'm going to do that, but that'll be an exciting journey as well—interviewing a bunch of people and spending 50% of your day on hiring.

Yeah.

I know.

It sounds like your current day is pretty exciting because if you're excited for that, it's just more of the same.

Yeah. More of the same, man. More of the same. Yeah.

Cool. Okay. What do you think jumps to mind when I ask, “What did we not cover in this call so far?”

I think we've pretty much covered everything that I could publicly say, to be honest.

Okay.

Because a lot of the stuff—

Let's talk about what you can't say publicly, Abhilash. No, no, no.

There are a lot of things that I can't say publicly, but I can't say them.

Yeah. Cool, man. I appreciate you joining us today. It's a lot of fun. I love working with you on a lot of this stuff. Hopefully, next time we can have some more people on the team and give an update on the trends we're seeing as we execute on a lot more of these projects and keep learning.

Yeah, I think today was more about what consulting is. We should definitely do an episode where we talk more about what our learnings are, because there are definitely a lot of learnings we have as a team. I think the team will be excited to come and share everything that we've learned so far.

Yeah. Well, I'm sure the audience would be excited for that too, but we'll leave that for a little teaser. A teaser behind the podcast paywall. Coming soon.

Yeah.

15 minutes of podcast behind a paywall. How about that? Everybody will like the newsletter because it's free, but there's a paywall there. So how can we paywall this video?

Let's do it, man. Let's do it. Let those dollars come in.

We might need to watermark right over our faces too. I think that might be a better thing. Yeah, yeah.

Dollars. Dollars flowing.

Translation watermark. Yeah. Cool. Good job today. Thanks everybody for listening. See you next time.

Thank you, guys.