Ep. 033 - ClusterMAX 3.0 Is Here! Neoclouds Ranked (Neoclouds, GPUs)

We're here this week with a ClusterMAX 3.0 review podcast. I have Sam and Pratt with me. We're going to talk through everything included in the article we published on Wednesday, September 23, including the rankings as they stand today, statistics such as how many providers we tested, and the methodology we've used for testing over the last several months.

We'll also tease some upcoming projects, such as Endpoint X, Post-Training X, Harness X, and Sandbox X. We're going to test everything and then talk about some of the trends affecting the neocloud industry: financing, the move to Vera Rubin from Blackwell, security—we had a previous article on that—and agentic coding and how that's affecting infrastructure. Maybe we'll get into some individual stories about specific topics.

Guys, welcome to the show. How have the last 3 months been, Sam? You're coming out of your hole, man.

Yeah, it's my first time seeing sunlight today, actually, since testing began. I'm a little sunburned from it, but I've had a few lifetimes' worth of reps on Kubernetes and Slurm, so I wouldn't trade it for anything.

1. The Rankings

Pratt, how about you? Are you feeling good after the testing period has come to a close?

Yep, feeling pretty good. It was a lot of fun testing these clusters. It's always fun playing with big infrastructure.

Headline news: I think the biggest thing is that there's a new addition to the platinum tier. We've got Nebius up there, joining CoreWeave. In the gold tier, rounding it out, we've got Oracle and Google Cloud, with Google joining Oracle.

On the silver side, Azure moves down, Firmus and Lambda stay, GMI moves up, and TensorWave stays at silver. In bronze, companies like AWS, Crusoe, and Together are all moving down. At a high level, this time we ranked 77 providers. We expanded our market view to cover 323 providers in total, which is available to subscribers to our Neocloud model in great detail, and we've now interviewed well over 200 end users of neoclouds as part of this research.

2. Financing Neoclouds

Let me pose a question to Sam to start. What's the most interesting insight that comes to mind when you think about the labs we've interviewed so far? Reliability, performance, ease of use, support—what do you think is top of mind for people buying GPU clusters right now?

It's a good question. The first thing that comes to mind is just the ridiculous backwardation of the price curve at the moment. People are willing to pay just about anything for any chips they can get in the next few weeks, especially. You can get a big discount if you're willing to wait even a few months.

That means just about everyone in the industry looks like a genius right now. If you have chips to sell, even if your fivecoded Kubernetes is completely broken, even if storage barely works, if you've got GPUs, you're going to find people willing to pay a pretty good margin to take them off your hands.

Yeah, it's interesting. We've talked about that trend a few times previously: providers that weren't able to sell their stuff early—typically, that's a signal that they're not as high quality—but now they're actually closing these deals for significantly better margins, which is interesting.

Pratt, how about you? We're expanding a little bit beyond ClusterMAX in the future: things like serverless inference endpoints, post-training infrastructure for RL, harnesses, and sandboxes. You've been digging in there when talking to the providers themselves about the roadmap. Do you find it confusing that people can't get Slurm and Kubernetes right sometimes, but they're already ready to launch 4 new products?

I mean, I guess, right, because currently the margins on compute are so high. Of course, as a neocloud, you do want to capitalize on that. For example, SpaceX provides really good compute to Google. They have tight deals with Anthropic and also several other neolabs.

However, neoclouds also want some of that cake. They're thinking, "What is the highest-margin business that we can enter right now?" Managed clusters are, of course, one of them, but it's a huge headache setting up the entire cluster. So why don't we just go above that layer and sell hosted training? Why don't we sell managed inference?

They're just expanding their market, trying to earn the good cash that the labs are earning right now. But everything points to centralization of compute. That's the main thing I can say: centralization of compute.

Makes sense. Sam, how about you? There are a few different providers who have been on the ClusterMAX ranking system in the past and are either moving down the list or moving to unavailable because they're focused on bare metal or perhaps on their inference endpoint business over clusters. I talk to neoclouds all the time, and a lot of people still need training clusters. A lot of people host inference for themselves, and they still need those clusters. Do you think there's no more to be done on the product roadmaps for people building managed-cluster experiences?

It's interesting. As Pratt said, there's an interesting fluidity to the market right now. Some of the better managed-cluster providers know how to manage a cluster because they have teams internally who are trying to use these things and can give good feedback on what a good Slurm and Kubernetes layer looks like, how to set up health checks, and that sort of thing.

There definitely is a benefit to integrating these things in-house, in addition to the fact that you have another service you're offering on top, where you can take margin from another layer. We certainly see plenty of room for improvement from most of these players.

I don't think that we're requesting super-fancy bells and whistles on top of these managed clusters. There's a handful of genuinely interesting additions to the managed-cluster experience, but mostly, I think there's plenty of work to do just to get Slurm and Kubernetes bulletproof and to get health checks right.

3. Health Checks

I think everyone on this call would agree that it's quite lucrative for these companies to do a better job and be able to collect a few more cents on top of the GPU-hour prices they're charging, just by doing a slightly better job at the software layer, picking one lane, and making it great.

Okay, let's dig into health checks. This is something that we wrote a dedicated article about a little while ago called "How Much Do GPUs Really Cost?" In ClusterMAX 3.0, we go through in detail the methodology we use to test reliability.

Can you tell me a story—I'm leading the witness here—about some providers that claim to have health checks, but when we test them or try to exercise them hands-on, they're just not really working, or they require a lot of manual intervention? What sets apart someone whose health checks are fully developed from someone who doesn't have them? And maybe, at a basic level, define a health check.

Sure. There are basically 2 types of health checks: active health checks and passive health checks. An active health check runs an actual workload or a dummy workload when there's nothing else going on in the cluster. This is normally preemptible, so if I have an active health check running and then decide that I have a training job I want to run, the health check gets interrupted and the actual workload begins.

The passive health check is just running in the background. You want to see as little performance overhead as possible on the passive check. The overhead can be exactly 0; it can just be reading logs from the CPU while the GPU does all its work.

There are a handful of basic things we like to see in health checks that aren't always respected. One basic thing is: if you just diagram the health check on paper, is there any chance that it works? There are a million corner cases you could come up with, like some obscure race condition that causes the health check to fail. But we've been through a few of these clusters where there's just no way it could possibly work.

One example I'm thinking of—and these are, I guess, health checks that are so bad they're worse than no health checks because they're actively interfering with jobs—is on Amazon's HyperPod Slurm. The first time we tested it, there was a health check scheduled in the auto-remediation phase that needed the node to be healthy before it could run. This was to make sure that the node had recovered its health. There was an obvious contradiction: it could only run once the node was back in the fleet, and it was needed to bring the node back into the fleet.

I think there are probably 5 or 10 examples of this throughout testing where the health check just never had a chance. It makes you wonder whether these people are really putting it through the paces before they hand it off to customers, given that our simple little XID injection catches some basic failures of functionality.

Yeah, that makes sense. On the XID injection, maybe you can explain a little example of the process we go through to simulate a failure. It's different on managed clusters versus endpoints in terms of how things can fail or how that affects a customer's workload. When we go through the testing, we're doing custom stuff for some of it, but the basic way is with DCGM injection, using the documented approach that NVIDIA releases, right?

Yeah.

It’s nothing nearly as complicated as some of the other things that these companies handle in their day-to-day. The most basic version is that you take a node, as long as you have the requisite privileges, and write into the kernel ring buffer the message that exactly corresponds to the message that would be emitted in the event of a genuine GPU hardware failure. This doesn’t always work. People have health checks configured in different ways, but normally you can literally read out the health check, figure out how it works, and make sure that this is, in fact, tripping the wire that you need to trip.

Most of the time, this is exactly what you need to do. This springs the health check into motion exactly as it would run in production. Once that happens, it’s a bit of a Rube Goldberg machine sometimes. Ideally, it’s straightforwardly quarantined. There’s some passive check that runs before it’s automatically remediated, and everyone makes sure it’s healthy.

Depending on the exact error that you get, you want to handle it in different ways. But the supported path is pretty straightforward in general, and there’s not that much to say about it. It’s taken out. You can’t see it. It’s validated, brought back in, and then you can schedule jobs onto it again. In the meantime, because it was unhealthy, you want to make sure that you’re not able to schedule jobs on it and slow down your workload. But there’s a lot that can go wrong between the simulated failure and getting your node back into your fleet again.

Yeah, and that’s exactly what we’re doing, right? We’re simulating a failure and testing the amount of time it takes for the provider to identify that the failure has occurred, followed by testing the amount of time it takes for them to drain the node, get it out of the cluster, and replace it with a hot spare in the case of the HGX machines, or do some sort of reboot or repair workflow in the case of the NVL72 rack-scale architectures.

This is when it comes to hardware failures on a cluster. I think the reason why we dig in here so much is that people are actively paying money for these GPUs that they then can’t use if they fail. The only way you can, as a customer, have any sort of recourse with the provider in the event of these failures is if you have SLAs enshrined in a contract that you both signed up for, which say you’re going to get this amount of credits if you have downtime.

4. Endpoint Cache Misses

You have to define the downtime. You have to make sure that there aren’t a bunch of excuses around it. For this reason, you should contact SemiAnalysis for access to our sample SLAs so that you can get these contracts to the industry standard for what other people agree to with other customers. Anyway, it makes me think about people paying money for things that they’re not getting in return, or spending too much money—more than they have to.

So, Pratt, maybe you can tell us a little bit about your initial experience testing some of these providers’ endpoints as a bit of a teaser. It’s not included in the ClusterMAX experience or in this initial article, but it’s still something that we’re tracking and going to be writing more about. Give us a preview of what you’ve experienced so far.

When we initially said, “Let’s go into endpoint testing,” it was like, “What’s the difference? It’s just the API. Everybody’s API is going to be the same.” But no, there are some real reliability challenges and some differences in cost that you’ve already teased out in a week of testing, right?

Yep. Full disclosure: this is the preliminary test. We are going to do a huge test right after this, which I call the hero run. That’s going to test concurrencies up to 24 or even more than that.

From the preliminary runs, something that’s very interesting is that, as you said, Jordan, a lot of people would assume that an endpoint just works. An endpoint is like a Lambda function: you just execute the function and get an output. However, a lot of providers have their hardware running in a configuration where their autoscaling or cache-aware routing isn’t working, so the caches don’t really deliver.

For one provider that we tested—and it’s a very famous provider that I can’t name right now—we got a cache hit rate of around 75%. You would say, “75% seems decent, right? Above 50% is fine.” For the other good providers, you’ll see a cache hit rate of 99%.

FIA’s workload was around 800 tokens for a trace, so we were just doing replays on a single endpoint. But that 75% cache hit rate almost doubled the total cost of running this replay. It’s quite shocking. You would assume that 75% is pretty decent—75% is fine. However, for these providers, even 75% would double the cost.

Imagine if you’re paying $200: you’ll be paying $400 for the same accuracy, probably at even slower speeds, because cache-aware routing affects speeds. You’ll have a huge bill for the same exact model and the same exact accuracy. It’s quite shocking.

Yeah, that’s become a bit of a theme: getting shocked by what you’re having to pay after you actually spend time running on something.

5. Networking and NCCL

Okay, let’s talk a little bit more about some of the technical details on these tests that you were involved in across a lot of these clusters, Sam. Beyond reliability and beyond pricing, let’s say there are huge differences in performance, or in the performance that you’re able to realize out of the box with a reasonable amount of testing.

6. Hosted Training and RL

Let’s start by talking about networking, and then we can go into storage. When it comes to getting good performance out of the network and troubleshooting this with customers, do you think there are specific providers that stand out as being good at this? Obviously, Platinum, Gold Tier, and Silver are where the good providers are sitting in this ranking, but what stops somebody from giving us a cluster and us being able to realize full performance on a training job or a microbenchmark on the network out of the box?

Yeah, it’s a good question. People hear about GPUs, and the topline number that’s normally quoted for a new generation is the FLOP spec. But it turns out that this is something that basically doesn’t distinguish cloud providers. Reliability is the first thing. Obviously, if you can’t use the chips at all, then that’s a bad deal. Networking is probably the next most important thing that we check.

The simplest advice we can give to providers is to make sure there’s a golden recipe for NCCL, the NVIDIA Collective Communications Library that’s used by default on these clusters. Make sure there’s a golden recipe for NCCL that gives me reasonable performance out of the box. I don’t want to have to go hunting down strange interface configurations.

You see pretty frequently that there are pathological message sizes. For whatever reason, on this cluster you get a huge dip at 128 KB, or maybe at excessively small message sizes it won’t go through at all. There’s a fairly straightforward test where you sweep all the message sizes, and you sweep in multiples of 2 the number of nodes that you have on the cluster. You want to see monotonic increases in throughput as the message size increases.

There’s a bit of detail to contend with here because NVIDIA’s default library isn’t perfect out of the box. Sometimes, due to bad software or bad heuristics in the routing, you’ll get dips in performance. But basically, you should see a smooth logistic curve. I shouldn’t have to go into NCCL.conf and rename things, or figure out exactly how this interface is exposed to Kubernetes. I just want to have something waiting for me on the cluster that I can hit Enter on and get a reasonably good starting point to hill-climb from.

I can tune that to my heart’s content depending on what model I’m serving, what my parallelism strategy is, and so on. But something we think all providers should do is provide a golden recipe for anyone as soon as they get on the box.

Yeah, and we saw a big win out of this from Google, actually, which is that they built this into the NVIDIA NGC containers. Their custom networking, which does not use NVIDIA’s silicon in the switches or in the network cards, is still able to work out of the box with NCCL. They don’t have to do a bunch of controls in the versioning where they have to keep track of every single parameter that NCCL can tune in that container, how it changes over time, and then tune their network to accommodate it. It just works out of the box there.

On the alternative side, I think a lot of people felt like, with NVIDIA’s networking software going to be free, EFA from AWS was going to be easier to use. But I think what’s happened is that, instead of everybody sticking with NCCL and just making the defaults easier to use—which, to be fair, EFA out of the box with NCCL is easier to use than the last time we tested it—what you’ve now seen is that collective libraries like DeepEP or Mooncake are actually built on top of NVSHMEM, a lower-level library than NCCL. This is just completely unsupported by EFA out of the box.

Maybe you can talk about the headache, or the double-edged sword, of a provider trying to save some money by controlling the network and designing it in their own way, while then actually having to make it work for customers out of the box.

Yeah, we've had our fair share of conversations in Slack about EFA. The bottom line is that our InferenceX team, the people they collaborate with, and people throughout the industry are working really hard on hill-climbing these performance metrics for scale-out networking. The DeepSeek Collectives library stands out. There are a handful of other ones that the InferenceX guys could tell you all about. You want to be able to take advantage of all that work by the open-source community: take their recipes and use them on your cluster, or take their recipes and improve them in some way that's important to your workload.

We could have Dan on the pod to explain basically how the BOM breaks down when you're building out a big cluster. Jordan, you can fact-check me on this, but something like 75% or 80% of it goes down to stuff other than networking. Scale-out certainly matters. If you can save a few percent, it's nice, but if you save a few percent and keep people from using modern workloads because they can't use the DeepSeek communications recipe that they would like, that's absolutely not worth it.

7. Vibe-Coded Infra

We've seen some cool blog posts by people at Perplexity and elsewhere explaining how to get reasonable performance out of EFA. You can see that the money they've saved by building their own custom scale-out is being paid instead to engineers who have to spend extra hours making sure that the workloads run reasonably well on their weird scale-out that nobody actually wants to use. It would just be so much better if the industry's best player used a stack of hardware and software that made use of everything else going on in the field.

This is not to say that open-source software is the be-all and end-all. Going back to inference endpoints, maybe Pratt, you can chime in and say that everybody wants to use vLLM and SGLang, but what we've seen from these endpoint providers is that you actually need to build a whole bunch on top of open source just to get this stuff to work correctly. You can't just expect to plug GPUs in, turn them on, install the open-source software, and then have it all just work while you run this great business. You need to actually do a bunch of extra work, too.

Wait, no way. I thought you could just earn 80% margins on the inference endpoint business, right? Just kidding. You're right.

Maybe if your company policy is to never give anybody a discount for getting a cache hit, then you can earn the right.

Yeah. But Jordan, you're absolutely right. A lot of people think, “We have this cluster. A lot of neoclouds right now think that we have this cluster. We're on ClusterMAX, so we might as well create a managed inference endpoint.”

They go out, get vLLM, try running models on it, and it runs pretty fine. But as you start serving these workloads, as you start scaling these workloads to bigger customers, you'll realize that basic vLLM and SGLang are just not good enough. That's why companies like Radian Arc, and companies like the folks who created vLLM—they also have a company, but I forgot the name. Is it Inferact? Yeah, Inferact. All those folks are creating companies in order to make kernels as well as develop megakernels, which reduce kernel-launch time and, in fact, give faster inference latencies.

Of course, you cannot just go out, install vLLM, run it on a cluster, and assume you're going to make 80% margins, while also making sure that your cache-aware routing is in place. There are a lot of moving parts here.

Okay. Can you tell me about how you came up to speed on using clusters and the relationship that you have with agentic coding at this point? What we've heard from providers is that agentic coding has been very difficult for them to contend with. It's not a free lunch where everything just works out of the box and the model can solve everything, though it can be quite helpful for debugging.

The high-level lead-in for you is that agentic coding is resulting in a different type of stress on the system, because you can just have these coding models get away from you in terms of what they're launching on these clusters. Do you have some of that experience that you can share?

For sure. It's very interesting because a lot of our ClusterMAX testing leads with our CMAX CLI. Please check it out. It's open source. Feel free to try it out on your managed cluster and play around with it. If you want to contribute, feel free to contribute. Make sure to read it before contributing.

Of course, we do test our clusters—we use agentic coding for testing all these clusters—and the workload is very different. The workload is very stressful on these systems. That's why we have CMAX CLI as a front end, and the agent is just a wrapper around it. If there are any issues that CMAX CLI faces, the agent in turn debugs into it and understands what's going on. That's the main use of agentic coding, even in my day-to-day work.

I'd love to hear what Sam does. He also is a huge user of Astra, right? Is that right? Are you Astra or Fable?

I'm a 5.6 guy. I find Aider just so badly undercooked. I get more going with 5.6 right now, but certainly not Claude. The word on the street is that they fixed the writing, but I'll believe it when I see it.

Anyone who's used these tools extensively has an idea of how frustrating they can be. I think cluster orchestration is the perfect sweet spot: close enough for the agents to be superficially useful, and far enough that they'll just blow your foot off every single time. It's not quite like—yeah, Jordan's laughing the laugh of a man who's had late-night sessions where Claude is reliably leading him astray—and use—

I'm not saying—

—and use some vibecoded products, which is something I shouldn't be laughing at too much because it's not like I've been able to produce any reliable dashboards so far. Mine are plenty brittle themselves.

It's funny to me to think about getting tagged by Pratt in Slack: “What do you think of this run?” Well, let's check with the computer what we think it'll cost on the endpoints. And it comes back and goes, “Yeah, this will range from $300,000 to $400,000.”

That was insane.

We do a cost sweep across everyone. Like—

Hey, that's light work. That's light work for SemiAnalysis.

On this one for now—

Just make sure we're bulletproof before we start launching some of these runs.

So anyway, it gives you a false sense of confidence, and you need to maintain the discipline to actually try to understand how these systems work. I think what we've seen is that the providers themselves have developed some products that they seem to not understand as deeply as they did a year ago, despite the fact that the product is not meaningfully more complicated than the one they were providing a year ago.

We're reaching this wall with clusters where the top providers don't have that many more features that they can add. They need to contend with bringing online the latest and greatest GPUs as fast as possible, which is complicated in and of itself. But the expansion opportunity is bare-metal data centers becoming a construction company and/or an operations company, basically; inference endpoints; and RL training infrastructure, which we can talk about a little bit. But spinning up a few VMs and creating a Kubernetes cluster should be a solved problem, despite the fact that the agents haven't quite solved it yet.

Yeah. Yeah, I think that's right. I mean, there's a handful of somewhat high-margin offerings that we can think of that some good providers are not yet offering. I guess performance storage or block storage are the 2 main ones that are not offered by everyone, and they seem like an easy way for them to expand in their current shape. But I completely agree that such—

There's plenty of stuff that feels like it should be right in the sweet spot of the models. It's plausibly properly configured, but if you spend as much time on clusters as we have, we know exactly how we want it—or at least we know exactly how we don't want it. We see plenty of setups that we don't like.

Honestly, putting the ClusterMAX write-up that we have here into your agent's context and telling it to rearrange your orchestration is better than anything else I could think to mention. There are a lot of good Slurm patterns in the training data, and there are a lot that just don't work for the use case we're speaking about right now. The same goes for Kubernetes and NCCL configuration.

8. Bug Hunting

I think there are a lot of people who have outkicked their coverage trying to set up stuff that they don't actually have experience with. They haven't spoken to customers. It's not easy to Google the right way to set up this part of a Slurm cluster. Your agent will do it for you better than you could have done on your own, but it will do it incorrectly.

Yeah. Let me give a real example of where this kind of thing manifests, with a quick demo of our dashboard and some troubleshooting. I’m not going to pick too hard on these guys, but Firmus was a company that we worked pretty hard with on the GB300 to actually get it working.

What I’ve got on screen is 1 of the example dashboards that we track, which is an NCCL performance curve that Sam was describing earlier. Here, I’m highlighting Firmus. There are 2 runs: 1 that we did after a bunch of work together, and 1 that was done earlier.

The chart on screen shows time, which is the latency for a message to actually pass over the network in this all-to-all collective, and more time is bad. Lower is better, and this is clearly an issue with the network configuration that we identified.

What happens is that if we get a collective-constrained workload—which we do have here—and I can go down to our TorchTitan GPT-2 training job, which is run with expert parallelism and is communication-bound, you can see that Firmus out of the box was doing 4,455 tokens per second per GPU, whereas the others in some testing were up around 3,500. That’s way less performance out of the box, like 50% less, which is an issue.

If you drill down into these steps with some of our profiling, this yellow bar here is exposed communication time that you’re not supposed to be seeing. Backing out and showing all of these providers highlighted together, Firmus, obviously, on a per-step basis, is just taking way longer, and that’s because all of this exposed communication is actually being shown there.

Anyway, this is not a bad provider. This is a company that has stood up thousands and thousands of GB300s. They have happy customers, and they were eventually able to fix this with us after we did some testing. But it’s an example of how a company that has not built the battle scars of working through that back-and-forth with a customer is not going to have an idea of how to set up a network correctly for the GPUs out of the box.

We default to trusting people who have experience with the GPUs that you care about the most, and that’s the latest and greatest GB200 and GB300 systems. Who are you going to trust for your Vera Rubin going into next year? Probably the people who have been through these battles before and know what they can trust from NVIDIA, what they can trust from their other vendors in the documentation, and what they maybe can’t.

Let’s move on from networking and talk a little bit about storage. Sam, you had a fun experience with 1 of the providers trying to get their storage to work correctly, which specifically led to a conversation about security. Maybe you can tell us a little story about that experience.

Sure. This is a story with multiple layers, I guess. We got an email notification for a node down. As mentioned earlier, we do synthetic injections of XIDs to test the health checks, but we normally do this at the end of testing, just in case it doesn’t come back. We don’t want to run short-handed for all of testing.

We got an email notification that a node was down pretty early on. I screenshotted it and put it into Slack: “Is this anyone?” “Nope, it’s not me.” “Jordan, are you messing with my cluster?” “No, man. That’s real.” “Yeah, Jordan, what are you doing, dude? We’re only going to have 3 nodes here for the rest of testing.”

It turned out that it was an actual failure, and it was a pretty fun failure. There’s some way that working on ClusterMAX is the exact opposite of being a performance engineer in a lab. If you’re a performance engineer at a lab, you just want everything to work. Here, we want bugs. Otherwise, we have nothing to do. This was 1 of the better bugs that we saw in training.

If I met an actual performance engineer and shook hands with him, we would both disappear. Engineer with anti-engineer.

We got on the cluster, and there were a lot of logs to dig through. This is an example of agentic coding being helpful. We weren’t sure exactly what happened. No one was actually running anything on the cluster.

It turned out, long story slightly shorter, that there was an error in the Linux kernel. The Linux kernel had a hard time dealing with reassigning dirty inodes. You have a bunch of stuff written in the write-back cache, and then it changes ownership. In our case, this was an old Slurm job that was ending, so we needed to deallocate it.

This is the behind-the-scenes stuff of an operating system that no one ever cares about or wants to care about. This was a very old Linux version. The kernel that we were dealing with was a couple of years old and had bugs that had been solved fairly long ago.

There was a bad quadratic-time algorithm that was dealing with reassigning all these dirty pages in memory. This locked up the CPU—all the cores corresponding to that NUMA node. I’m not sure exactly why it was constrained to 1 NUMA node, but it locked up all the CPUs corresponding to that NUMA node.

As it was tearing down this Slurm job, the client for storage couldn’t get any CPU time. An ordinary request like, “Hey, are you still alive?” to the Lustre client didn’t get a response, so that triggered the health check. It said, “Storage is failing,” but the health check couldn’t actually complete its work because the CPU was frozen up.

It took a couple of hours for that job to complete. Finally, the CPU returned to a healthy state because it was done doing its laundry. As soon as it returned to healthy, it was like the cops were waiting outside its door, and it got health-checked. That was the cause of the email.

The last thing that you want is a node getting taken from your fleet, not because the GPU is actually unhealthy, but because of a health-check failure for your filesystem that was due to a failure with your CPU.

9. Security

That was a fairly obscure corner case with a filesystem. It meant that I was burning all kinds of Codex tokens over the weekend, trying to figure out exactly the right workload to recreate this bug. In any case, we want the storage client to get CPU time. That’s 1 way of summarizing our conclusion from that event.

Actually, this turned into a security issue too, right? I’ve talked about this on a couple of episodes previously: the asymmetric nature of trying to find a bug. If other people have more powerful models than you, you can’t defend yourself.

Pratt, maybe you can tell us a little bit about digging into security on a couple of these clusters, where you’re trying to identify whether you were protected against a given vulnerability. In order to even start asking questions, a lot of the models would just completely refuse to help you, right?

Yeah. I mean, that’s 1 part of it. But the other part is that—and this is something I’ve noticed—frontier models especially just don’t want to exploit vulnerabilities. They have a tendency to show false positives, being like, “Yeah, everything is perfect.”

The other day, I was testing out a cluster and basically just playing with it, making sure there was a vulnerability that I wanted to exploit. I was using, I believe, 1 of the frontier models for that, and the model said, “Yeah, we have this bug. We can exploit it, and we can actually get host access—host access to the control plane of that cluster.”

I was overjoyed. I was like, “Yo, this is a perfect, perfect chance for me to show that the CVE is actually a really real thing.” However, the model actually produced a false proof of concept. After about 2 hours of debugging, the model said, “Oh, yeah, actually, my bad. This was something I accidentally said. This cannot be exploited.”

You could think those 2 hours were wasted, but I would say I learned a lot at the same time. This is something that frontier models especially do: they just want you to believe in false positives.

That makes me realize: are hallucinations gone? Most likely, yes. But are false positives gone? Definitely not. That’s my fun experience testing these clusters and exploiting these CVEs.

Yeah, man. There’s a lot more work to be done there. Sam, what’s coming to mind when it comes to security? We’ve talked about this on a previous episode, but a few things have happened since then, including OpenAI being hacked as part of a bug bounty program.

The HackerOne AI guys opened a pull request in their internal repository. It seems like they’re vibe coding, because their monorepo had 1.1 million pull requests or something. So they put up the 1,000,001st pull request to prove that they had full access.

A little crazy, but they’re 1 of many that are actively pursuing these bug bounty programs, which not a lot of these neoclouds are participating in right now.

Yeah, I certainly have a rant about this loaded. Maybe I’ll keep it in my pocket for the time being.

Whatever your opinions are about AI doom, or whether we should keep pushing full speed ahead, the fact of the matter is that these are extremely valuable, extremely powerful pieces of equipment that are very badly secured in many cases—very carelessly secured.

There’s nothing like a standard set of tests that you can run to make sure that your cluster is properly configured.

Things are moving very quickly. I understand that there are a million things on the to-do list of the people working at these companies. At the same time, we see CVEs that are 2 years old applying to the basic software running on these things. We're not asking for super-fancy things when we want tenant isolation. We don't want to be able to see what's going on in the Grafana of our neighbors, and we don't want CUDA Toolkit 12.0 when 13.2, or whatever it is, is the most recent version. So, apocalyptic scenarios aside, there are straightforward ways to see this becoming a tinderbox for tools that are only getting more powerful.

Yeah, we talk about AI needing to be used to defend against attackers. That certainly seems real when you go through the details of the Hugging Face incident, and it also feels real when I'm trying to understand the details of that incident—or anything else we're working on on the security side—where the speed at which we have to move means you have to use these models to understand what's actually going on and comb through logs and all of that. But there's also some stuff you can do that's pretty straightforward, right?

Yeah, I agree with that. I was going to push back a touch. I understand that, in the case of the Hugging Face incident, I need to stop using the euphemism supplied by OpenAI's lawyers: “Hugging Face incident.” It was the Hugging Face cyberattack by OpenAI. It's strange to me that we've forgotten this in favor of this anodyne phrase.

The Hugging Face cyberattack consisted of dozens of very mundane misconfigurations by OpenAI—almost inexplicably mundane misconfigurations by OpenAI. Sure, you have petabytes or whatever it is of chain-of-thought to dig through if you're interested in understanding qualitatively this novel model behavior. But you don't need an LLM to understand that you shouldn't have an applicable CVE in your Linux kernel, which was the first thing the agents used to escalate their privileges.

You should be careful about managing permissions on the Kubernetes layer, which is another thing the agents exploited. If you want to sandbox your agents, you should make sure you have alerting if they have unmitigated access to the internet, which OpenAI did not have. So I think agents can certainly be helpful there, in ways that you can only fathom, but there's also just a lot of basic things that OpenAI didn't do and that these providers are not doing.

I don't think we should allow them to use the excuse that it's a brave new world. We're not the most skilled people with Qwen 3, so you can't blame us for having bad infrastructure. It's a lot more basic than that, and that's the fairly low bar that we're trying to hold everyone to.

Nice take, Pratt. Are you ready to fight slop with slop?

I'm down. Slop-slop. Let's do it.

Right. What's coming to mind, guys? What have we not covered on this one so far? Any stories that you think we should be talking about?

Can we cover the financing? I believe it would be good for the audience to understand why ClusterMAX is even important. Are we just testing out clusters, giving them feedback, and calling it a day? Why is ClusterMAX so necessary for this $7 trillion buildout, which is going to happen—or is happening, I guess?

Good question. I'll take that one. Financing is the biggest topic for everybody because whatever you were doing 3 years ago, everybody is being asked to do it 100 times bigger right now. Everybody's being asked: Can you take what you did at 10 megawatts and try to do it at gigawatt scale? What's going to break if we try to go that size?

The thing that's breaking right now is that there is more money than good investment opportunities, except the money is still concerned about quality. There are 2 big reasons why somebody might be concerned about the financing of a Neocloud. One is if they're late delivering on their contract: they don't build everything fast enough, get it online fast enough, or hand it over and have the customer start paying. So they can't pass acceptance.

The second is if there are so many violations of the SLA, so much downtime, that the customer has a cancellation right and wants to cancel their contract. The simple dynamic that's happening right now is that people are trying to insure against these things. They're trying to insure against delays and contract cancellation. This applies both to the debt providers and to the Neoclouds themselves, which want insurance against these things happening in their customer contracts.

There are products for this. Companies like Parametrix and others that we're aware of are involved in providing insurance. They're underwriting the parametric risk of these contracts, and to do that, they need actuarial input.

They need to have some sort of information about the cluster that people have: things like redundant power at the site level, redundant cooling, redundant internet connections, and the ability to mitigate failures. They need to know that things have been designed to a performance standard that meets these requirements, that there's a parts depot nearby so that if there's a bunch of failures, they can actually get things back online, and that they have the correct number of people on site. They need to know that the site is not in a floodplain or about to get hit by a hurricane, and, if it is, that they have the requisite insurance against those natural disasters and force majeure clauses in the contract that protect them.

We've developed a number of example SLAs and example contracts that we are circulating with customers of Neoclouds, and both the clouds and the labs want this industry standard. They want some standardization so that they can make customizations, but they know what the definitions of important terms are within these contracts. What is the definition of a node versus a rack versus a site versus a cluster? What is downtime? What is force majeure? There are all sorts of specific things.

Because we've seen so many of these deals happen and so many of these contracts, we're able to provide a level of certainty to the lenders, the providers, and the buyers that they can get these deals done and that people aren't going to have a bunch of issues later.

With that said, the motivation for this is concerning. We've seen a bunch of people sue each other. We don't want this to happen, guys. At a very basic level, when companies raise hundreds of millions of dollars sometimes and spend—I've known companies that have spent literally 100% of their seed round on compute because it's a couple of guys who have equity in the business and aren't taking a salary yet—they just get the check from the VCs and turn it around and spend it directly on Neocloud capacity.

This is the most critical decision you can make in the life cycle of your company, so it really depends on whom you choose to trust. That's sort of the backdrop on financing. The offering from SemiAnalysis is to provide a technical assessment of all of these providers, the decisions they've made, and their ability to respond to things. This includes hands-on experience, visiting sites, and doing the dirty work to get those details flushed out. If you want to engage us for that technical assessment, contact cluster@semianalysis.com, so you can underwrite your deal or move forward with confidence.

Maybe the other big conversation is that SemiAnalysis is launching a credit and markets model covering really the entire backstop universe that's going on right now. We had a great article that Dan put out covering these details. NVIDIA's off-balance-sheet backstop universe has grown to over $588 billion at the end of FY27—basically, the current or soon-to-be-current fiscal year. The forecast, if this continues, is for this off-balance-sheet backstop to grow to over $2 trillion by the end of 2031. This includes a whole bunch of detail that we published when it comes to actually building a site and who is being backstopped, what they need to actually build, the likelihood that it all comes through, and the likelihood that you have issues raising the capital or delivering on the commitment to the paying customer. If you're interested in the financing of this buildout, go read the article NVIDIA Backstop Universe and contact us about the credit markets model, because the amount of capital flowing into this ecosystem is worth paying attention to. Let's leave it at that.

Yeah, NVIDIA, I believe—and I might get these numbers wrong—is backstopping approximately $827 billion, projected according to our model, in FY31, only for the supply chain—for HBM and so on. This does not include LPS. It does not include the actual data-center buildout. This is purely the supply chain that they're backstopping, like HBM providers, which is quite insane. It's $827 billion.

Yeah.

There's a lot of stuff happening in the physical world in order for this stuff to be turned on, especially on time.

But there's another point to make here: it's not just NVIDIA. Amazon is doing this, Google is doing this. A lot of companies are putting their balance sheets to work because they want to be exposed to the upside without actually taking on a whole bunch of—not necessarily risk, because they're taking on some risk—but the quantity of risk that others are taking.

A lot of companies are benefiting from this because they don't necessarily need to have a business built on having investment-grade offtakers like NVIDIA. They just need to have that backstop where NVIDIA has confidence that these customers are going to show up and be ready to pay. If they don't, NVIDIA will take the capacity for itself. But it is a track record of companies showing up to pay right now.

Companies that don't necessarily have the funding raised today, who want to commit to a 1,000-GPU contract for a year, don't have the money to do it for 3, 4, 5 years. But when they train that model in the first year, they're expecting to be able to raise more money, and things are going to work. A lot of these companies have to work for this to work out, but so far, so good.

We are seeing a lot of companies across all sorts of different industries—robotics, drug discovery, materials science—producing incredible results, let alone the coding agents. There's something important going on here, and I think it's worth digging into and understanding the full extent to which NVIDIA is backstopping things.

Maybe the other thing to end on is just how much we are seeing old GPUs be used. Lots of people want to give us H100s today, which are 3 or 4 years old at this point, with some moving into 5 years next year. When we get Vera Rubin, we're going to have 5-year-old H100s on the market. This is not going the way that Dr. Burry said it was going to when he started criticizing everybody's depreciation schedules.

There are 4-year contracts being signed today for H100 capacity. OpenAI has never given up an A100 GPU that they've gotten. Those came back in 2020, so those are 6-plus years old. Maybe the question on everybody's mind on the finance side is terminal value: How can you underwrite the terminal value at 5 or 6 years on some of these chips when these contracts expire?

So far, so good. If demand continues to outstrip supply, we would expect these prices to continue to rise, and people will continue to pay for even the old GPUs so long as they can be turned on.

Tune in for a follow-up coming up soon about 800-volt DC and all of the modifications to data centers. If people are going to turn on Grace Blackwell GB300, they're running racks that are 140 kilowatts, approaching 200 kilowatts at peak demand. This is way different from an H100 rack at 30 kilowatts, where the entire facility is built for 30 to 40 kilowatts. They can't accept any of the new Vera Rubin systems that are coming next year.

What are these facilities going to do other than host these GPUs? Maybe rip them out and put in new CPU servers for the sandboxes, maybe install some storage, but likely they just keep running the H100s that are so valuable until they expire.

Where do we go from here? There's a lot to talk about beyond financing. Do you want to talk about sandboxes or harnesses or RL stuff? Pratt, what are you going to be testing?

Should we talk about hosted training?

Yeah, hosted training.

Let's do it. The thing I didn't realize until this week is that RL seemed smaller because it's smaller than pre-training, but from a reliability perspective and a correctness perspective, it's actually way harder because you're combining all the hard parts of inference with all the hard parts of training, and then you need to sync them up.

Plus, you add this thing called an environment, or a sandbox, that you also need to figure out how to deploy. So there are 3 components that need to sync up, as opposed to just 1 thing you can reason about, even if the post-training clusters are not necessarily so big.

What do you think about hosted training? Is it going to be a real thing when we're not necessarily seeing everybody nail Kubernetes and Slurm? Now they have to do Kubernetes and then do a whole bunch of stuff on top of it. What do you think?

I mean, hosted training is very interesting. We've been talking to a lot of people. I still don't think there's a market for it yet. I said “yet,” so please don't come back to me if there is a market.

One common thing you can think about is that your IP—your intellectual property that you're training these models on—has to remain with you rather than going to a third party like OpenAI or Anthropic. This could be a completely different case when OpenAI and Anthropic just deliver a better product and you're just like, “I need a better product over protecting my IP.” I think that's going to be a good discussion to have in the near future.

Another thing I want to discuss about hosted training is how infrastructure-intensive it is. I was talking to this other evaluation and RL-environment lab the other day, and they mentioned that they literally have server farms just running environments—a whole room full of server farms just running environments.

You would assume, why don't we just run Docker? Of course they do. They do virtualization to a decent extent, but at the same time, you also need to run it on different hardware, like iPhones, Android phones, et cetera. It's so infrastructure-intensive that once you take on hosted training, you're managing more infrastructure than you did when you were doing managed clusters.

Now you have to autoscale the environments, autoscale the inference nodes, and autoscale the trainer nodes. At the same time, you have to make sure that there is no trainer-inference mismatch. It's a million things to take care of. We should give more credit to these hosted training providers, because there are a billion things they have to manage.

My hot take is that managed hosted training is only going to become big if your company—if your startup—is handling literally everything. Once you handle every single part, from environment creation and literally sitting with your customer, to deploying forward-deployed engineers, building environments, training, and helping your customers train their models, and once the customer develops this—basically gets educated on how to really work with this—then only can you take your hands back, and that entire run happens.

You have to basically carry them through the process.

Yeah, so dig into that a little bit more. In the pre-training world, there are basically no examples where the provider that's selling the cluster actually really understands pre-training. It's not like you contract a cluster with CoreWeave and they're giving you tips on data mixes and architecture, like, “Hey, I heard this stuff about sparsity.”

They're going to work on the infrastructure layer, which means performance and reliability. So what's the analogy in RL? Not only is the infrastructure harder, you also need to have forward-deployed engineers, understand this company's business, and guarantee that the model you're training is going to work.

That was never a commitment that CoreWeave or Nebius had to make to one of their customers—that the training job was going to work well.

Yeah. I mean, it's an interesting thing. People assume that hosted training is going to be very easy. Honestly, I assumed that too. I thought, “You just train the model, the generator generates the trajectories, and you just train the trainer on these trajectories. Voilà, you're done,” right?

But it turns out the really valuable task in day-to-day work, which we do, is way more complicated than just generating trajectories or building an environment. Building an environment is very hard, and that's why Anthropic bought the most environments last year. That's why their coding model was so damn good: it's a hard thing, and they paid a lot of money for that.

A lot of hosted training providers basically thought it was like managed endpoints or managed inference—that you just provide an endpoint, which still, of course, is not an amazing experience. They basically thought that you just provide the infrastructure and the client enjoys it. But there's a lot of education to be done.

Of course, Rome wasn't built in a day. We still have more time. We're just at the beginning of this revolution, so we'll wait and watch what happens.

Last question about post-training. In the inference world, if you're making an analogy to that, there were some open-source frameworks that just dominated. vLLM and SGLang kind of simplified things for everybody. We got some really high-quality experiences really quickly, and then people built a whole bunch of more complicated stuff over time with prefill-decode disaggregation, training speculators, quantization, cache-aware routing—all these extra bells and whistles beyond the basic recipe and the kernels to get great performance out of the open-source models.

Let's compare that to frameworks in RL: Slime, MILES, veRL, PRIME-RL. What are you seeing people use? There's a lot of them out there right now, and some of the hosted training providers kind of abstract that away too, where you don't even know what framework you're using.

You're just bringing your dataset, or you don't even know about that, because the forward-deployed researcher—FDR, not FTE.

FDR, FDR.

We can't be telling these RL guys they're engineers, FTEs, right? They are FDRs. So, yeah. Tell me about framework wars and what you think is actually going to show up there.

Yeah, framework wars are very interesting because it's a very funny thing that all these inference providers started by providing amazing frameworks—SGLang, vLLM, et cetera. But the next market they thought to expand into was hosted training. That's why you can see great orgs capitalizing on MILES, which is absolutely amazing, by the way. They have made it very agent-friendly, so you can just set it up in one shot. It's pretty decent.

Of course, you have to set the hyperparameters, so it's not that easy a task to just train a model, but they have done a very good job. PRIME-RL is another very good one. When I was GPU-poor, before joining SemiAnalysis—shout-out to SemiAnalysis—I ran a lot of runs on PRIME-RL, and that was also a pretty pleasant experience.

Over time, my experience with these frameworks has gotten better. Of course, these teams have put in a lot of investment, and building these training frameworks is way harder than people think. I suggest the audience read the Human Sand[?] blog, where they basically show how native NVFP4 training is done at such a large scale. Trainer–inference mismatch compounds over steps, and as your trajectories get longer, you do want the optimizer to learn more technically, right? So you have to take a few more steps.

If your trainer–inference mismatch compounds over 100 steps, you'll realize that your model is diverging a lot, and that collapses training. Shout-out, shout-out to all these framework providers putting in work day and night, sending in PRs, even though they might be just pure slop. Shout-out to them.

Slop shout-out. Yeah. Yeah. Yeah. We got a term—we got a term: slop PRs, like SLPRs or something, you know, like NeoClouds, you know. So—

Yeah. I was at MLSys earlier this year, and one of the talks—Roger from Infact[?], shout-out, Roger—was basically like, “If you want us to merge your PR, you have to message us in Slack, because we're not reading every single one of your slop PRs anymore.” I was like, “All right.”

Good to know. Let's move to the wrap here, guys. Sam, what's on your mind as we finish up talking about NeoClouds? Where do we go from here, buddy? What's next?

Sam, you want to talk about Azure?

No, run it back.

Go company by company and criticize their internal organization, beginning with Azure and AWS.

I want fewer obscure bugs. I want nothing to write about in ClusterMAX 4.0. The better the cluster, the less there is to say. I want to log on to Slinky and just have it all work—have topology awareness, not blow my foot off every single time I try to see whether the health checks work.

I hope that all my performance engineer friends get good clusters that they never have to think about. No more providers randomly wiping your nodes, randomly rebooting them, randomly deciding that kubectl is going to go down for 3 days. We've heard some horror stories from guys, right?

Yeah, protect yourself with a good SLA, but better than that, just go with a good provider. You'd rather not be fighting to get back your whatever 5% because it was down for 3 weeks out of the year.

Yeah, trust people. Let's give everybody a way in which you can trust everybody else. Join hands and sing “Kumbaya.” We'll see a ClusterMAX 4.0 after we get some Vera Rubins in these data centers. Looking forward to it, guys.

What about TPUs? Are we ever doing it?

TPUs on ClusterMAX.

TPUs on cluster.

They're giving away too many secrets. Too many secrets. TPUs. Trainium. Some other novel accelerators that are being deployed in data centers across the country right now and around the world.