Shawn Wang
We've actually changed our SDK team to think about agent experience instead of developer experience. We think that the same benefits that apply for DX also apply for AX. Why would you have an agent read through hundreds of Kubernetes files and write YAML that's not even typed when it can basically make a couple of changes in a decorator and get this sort of self-provisioning runtime, being able to see its changes live in action?
We're here with Akshat of Modal, CTO of Modal, together with Vibu. Congratulations on your Series C.
Akshat Bubna
Thank you.
Shawn Wang
Your party yesterday was amazing.
Akshat Bubna
Yeah.
Shawn Wang
All the photos and all the swag.
Akshat Bubna
We had a bunch of art installations, which was kind of fun—seeing our products on pedestals next to Rodin.
Shawn Wang
When you started, it was not the GPU inference company. I mean, maybe it was in your mind. Take us back to the origin story.
Akshat Bubna
I actually first met Erik, who's the CEO, through an investor. Back then, Erik was already thinking about building a new kind of runtime, and he got there thinking through why workflow orchestration products are so hard to use. It's because you have to run them on Kubernetes. Kubernetes is hard to manage, it's not built for burstiness or custom images, and it has a terrible developer experience.
Shawn Wang
I'll inject for listeners who are new: we interviewed Erik 2 years ago, and there's a bit more of the story there, from Spotify and all those things. I actually came across Erik through Data Council because he did that talk on the serverless container stack that you guys built. That was my first, “Okay, I need to take models very seriously,” moment, but it was still very unclear: do I actually need all this for just my data pipelines?
Akshat Bubna
Initially, what we were thinking about was, if we build a better runtime, it's a very useful primitive in itself. There are a lot of things that get solved by serverless functions: you can do ETL stuff, you can do job queues, you can do all this bursty processing, which it turns out every company had needs for.
But then we were also thinking about this as a primitive that we could build a whole collection of products on, which are very versatile. Perhaps data engineering would have been the first one, but we were thinking about inference. Back then, it was more classical inference—computer vision stuff and running XGBoost and whatnot—but we added GPUs to the product a year before ChatGPT came out. We just didn't think it would be that big of a deal.
Shawn Wang
Yeah, just add an A100. Was there any early key problem that really sparked why you built it?
Akshat Bubna
Primarily, none of the tooling that was out there was built for a really great developer experience. There was also a general trend: a lot of the workloads we were seeing were—this is a word I wish there were a better word for—compute-heavy. They need a lot more resources, so you need to burst up and down a lot, versus Kubernetes, which is designed for slow scaling and more for web server use cases.
There's also a lot more specialization in the kinds of environments these workloads run in. Sometimes they need accelerators; sometimes they need different kinds of images. This is just a consistent thing that we saw across a lot of companies.
Shawn Wang
That would be the next step. Yeah, it'd be nice. I don't know how much this factored into the early story, but I wrote a post when I was at Temporal about software-defined infrastructure or something like that.
Akshat Bubna
The self-provisioning stuff.
Shawn Wang
Yeah, I couldn't even remember my own post, and then you put me on the landing page.
Akshat Bubna
Yeah, we really liked the term, and so we stole it.
Shawn Wang
Because you had the insight that everything can just be in decorators, next to—collocated with—the code, right? Was that a big part of the original story, or was it just a DX layer?
Akshat Bubna
That was really important because we didn't want people to spend so much time writing YAML. It seemed like you could really condense the surface area of what you're doing, put it in code so you can actually operate on it just like you can operate on other code, and build something that's more expressive and dynamic. So, yeah, that was always a very important part.
Shawn Wang
The pushback is: this is a DSL, it's closed source, and I'm locked in to Modal.
Akshat Bubna
Yeah, we never really got pushback for that because the nice thing about Modal is you can bring whatever code you have. Sure, the DSL is the configuration layer for what hardware you're using and how you're scaling things up, but you still own the code. That's been an important part of our story, even as we do inference now.
Shawn Wang
How much of it do you think still stays the same today? If you were to build something today, developer experience is obviously very important, but I feel like a lot of this has kind of been changed with just hooking it up to an agent, having Claude Code or Codex implement a tool. There are very agent-native primitives that are kind of different than if I'm doing this myself, right?
Akshat Bubna
We've actually changed our SDK team to think about agent experience instead of developer experience. We think that the same benefits that apply for DX also apply for AX. Why would you have an agent read through hundreds of Kubernetes files and write YAML that's not even typed when it can basically make a couple of changes in a decorator and get this sort of self-provisioning runtime, being able to see its changes live in action?
From the customers we talk to, they actually find Modal is way faster to use for agents versus operating on a different substrate.
Shawn Wang
Because you collocate the infrastructure requirements with the code that runs it.
The negative thesis now is that nobody's looking at their code anymore, so there's no point.
Akshat Bubna
People aren't looking at code. One thing we still see as really important is observability. How good is your dashboard? We push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to interpret what's going on, make judgment calls, and so on. I feel like that's maybe more important now than looking at the code itself.
Shawn Wang
Because you can try to treat the code as a black box, see the observable action that comes out of it, and then just prompt a change.
I actually think it takes a bit of restraint not to specialize—to say, “I want to ship a new primitive,” and then just be general-purpose. People ask you, “What are you for?” and you're like, “I don't know. We can do this, we can do that.”
I'd be curious to ask you: if we were to ask you what Modal is for, even at a high level—there's a lot you guys do, sandboxes, GPUs, everything—how do you answer?
Akshat Bubna
Modal is a cloud platform where we've built the primitives from scratch for AI applications. Right now, it basically covers inference, training, batch processing, and sandbox workloads, but we're building a lot more.
Shawn Wang
I noticed you didn't say web server, so there is still a role for always-on, large-scale Kubernetes-type things.
Akshat Bubna
Yeah, absolutely. We're not trying to compete with the Renders of the world because we think the differentiator for us is the other workloads that need specialized compute and need to scale up and down a lot. They're shaped differently.
Shawn Wang
I think you're building a lot of it alongside the startups, right? They're innovating quite a bit. Even in your latest blog post, in the Series C, the customers that you mentioned here—Cognition, Decagon, Ramp, and whatnot—they're innovating with you, right? That's not something AWS is doing directly.
Akshat Bubna
Yeah, absolutely. This is classic: we're a small team, so we can move really fast. Our engineers are working with their customers and figuring things out.
Shawn Wang
My first week at Cognition, I walked in and there was someone wearing a Modal shirt. I was like, “What are you doing here?” They were like, “Yeah, I am embedded inside of COG.”
Akshat Bubna
Yeah, I think that was Payton. We sent him over because the latency of communication was too high otherwise.
Shawn Wang
It's a distributed node. You have to place one in and collocate it.
So, actually, I had a direct personal experience. I worked on smol developer [?] 3 years ago.
Alessio Fanelli
It was inspired by Cloud Run. I think you onboarded me at some point just before that, and I was like, “I need some bursty compute.” I was just going to try using Modal, and it was a pretty pleasant experience. Apparently, I showed up in the board meeting because of the analytics.
Akshat Bubna
Yeah, you blew up on Hacker News, and we got a big traffic spike. I actually think the way you used smol developer was with Modal Functions for running stuff, which was a good use case.
Alessio Fanelli
Yeah, to me, that was proto-Cognition, right? If only I had stuck to it. If you just draw the tech tree out, it’s like, “Yeah, probably this will happen.”
Akshat Bubna
Yeah, he was so close. [laughter] But the funny story there is, at the same time, we were talking to a bunch of customers who needed something like sandboxing. This was in 2023.
Alessio Fanelli
So we needed a new API right after that.
Akshat Bubna
Yeah, we built sandboxes in May 2023, before anyone even knew this was going to be a thing. The first example we published was that we took smol developer and put it in a loop, so the agent could iterate on itself.
Alessio Fanelli
Loops are hard these days. [laughter] When was this—2023?
Akshat Bubna
Yeah.
Alessio Fanelli
smol developer.
Akshat Bubna
Yeah, it was around mid-2023.
Alessio Fanelli
For listeners, the problem was that the models weren’t built for any of this, right? They weren’t post-trained to understand looping and self-correction, and tool calling was there but also not that great. I don’t remember if you used tool calling in this one, but the models would just diverge after about 10 iterations and not produce anything meaningful.
Talking to myself 3 years ago, the answer would have been: collect all the failures, build a benchmark, collect all the examples, build the RL environment, sell it for something like $10 billion to Meta, and then also train a model and sell that for $60 billion to Elon. This is a money machine. It’s actually not that hard. [laughter]
Erik Bernhardsson
I mean, it’s hard to have that kind of inherent conviction that this stuff will get that much better.
Alessio Fanelli
In retrospect, it’s so [__] obvious. [laughter] Fair enough. What else were we doing back then? I don’t know.
Anyway, that was the start of your sandboxing journey, right? I feel like it didn’t really blow up until last year.
Erik Bernhardsson
Yeah.
Alessio Fanelli
So there were a couple of years of quietness.
Erik Bernhardsson
Exactly. Yeah.
Alessio Fanelli
Very underrated product value. My experience with Modal: Charles, before he had joined Modal, met this guy at a hackathon, and he really insisted that we wanted to run some small model—not hosted anywhere. He was like, “There’s this cool company, Modal. They’ll spin up a GPU sandbox so we can throw it on there. It’ll take a Hugging Face link.”
There’s so much value right there: instant hosting, spin it up, spin it down, let it stay cold, and when we run the demo a few days later, it’ll come back up. All this stuff, in retrospect, is still what we need today.
Erik Bernhardsson
Yeah, it’s still needed today. Obviously, workload shapes have changed a lot. We run stuff for people at really massive production scale, and there it’s not about scaling from 0 to 1. It’s about how we scale really elastically from 1,000 to 1,500 GPUs very quickly in a given region. It’s the same-shaped problem.
Alessio Fanelli
Okay, so look at, say, Cursor Composer. They had a setup where they would do RL on a model every couple of hours. You guys have a whole version of an RL inference gym and whatnot. When you look at workloads like that, you’re basically doing training runs where you need to scale up and scale down every hour—thousands of GPUs, right? That’s the example where we do need it.
Erik Bernhardsson
Yeah. Actually, I’ll take a step back and talk about how people use Modal today. Our biggest use case is actually elastic inference, and the thing we first found product-market fit with was inference for custom models. We stayed away from the LLM space initially. We were serving companies like Suno for audio, Runway for video, robotics companies, and computational biology companies that trained their own models elsewhere. Modal was the best black box for deployment and scaling to however many GPUs they needed as their traffic patterns changed.
We saw that all of them had a very unpredictable traffic pattern. Some days, the company would do a launch, and they’d need way more capacity. It’s not just 1 model that they deploy. All these companies deploy lots of different models in different regions, so the autoscaling problem becomes even harder. You have to scale within a certain region, and those cycles are offset, so at different times you need to scale up in different regions.
Alessio Fanelli
That in and of itself is a huge category. There are a bunch of inference providers—Fireworks does this as a service, Together does this as well, Baseten. That’s kind of carved into its own niche for language models, at least right now.
Erik Bernhardsson
Yeah, the thing we’ve specialized in is the autoscaling aspect. We found that it’s not universally true that everyone else can autoscale. We’ve gone deeper into it on the technology side, and we’ve incorporated GPU snapshotting into the product. You can actually take the GPU state—your Torch-compiled model—snapshot it, and then the next cold start is much faster.
Going back to your question, that’s why you need a lot of burstiness for inference. People also do a lot of on-demand training for RL stuff, and the rollouts are bursty, as you said. People also do a lot of batch jobs. We see a lot of companies that, before they have a training run, need thousands of GPUs to run encoding or something like that. Those things are much more bursty.
I agree that agents aren’t that bursty. Sandboxes are, except when you’re doing RL. RL is insanely bursty.
Alessio Fanelli
Yeah.
Erik Bernhardsson
When you’re doing rollouts, you sometimes need 100,000 sandboxes.
Alessio Fanelli
Yeah. I’m curious if you’ve seen early sparks of continual learning. Some people, like our friends at Engram, recently announced this. They’re trying to do training 24/7, per se. That also seems like a different workload, right? There’s a very weird dynamic in how you’re using GPUs between people and whatnot, but it seems like something that would work well for you guys.
Erik Bernhardsson
As you said, we’re fortunate to work with a number of customers at the frontier, and some of our customers are taking the primitives we have and trying to use them in very interesting ways, like continual learning. It’s possible that, as this stuff gets better, some of that will be part of our offering as well, if more people need it. We’re just waiting to see how all this shakes out.
Alessio Fanelli
Is there a primitive that you added after sandboxing that was the next step in the story?
Erik Bernhardsson
I guess we’ve been going much deeper into LLM inference because we realized that some of the advantages we have with autoscaling—especially in different regions and whatnot—aren’t present elsewhere. The place where we had a gap was that we weren’t working on the model layer itself. We were a black box, and we realized that we could actually get to frontier-level model performance by having great people who work on all of this.
We’ve been open-sourcing a lot of our work. Recently, we shared our work on dFlash, which is a block-based speculative decoder, and we open-sourced all of it. By using open-source dFlash, you can get the same performance as you would with 1 of the proprietary providers. The next thing we’re thinking about here—
Alessio Fanelli
I thought this was an interesting blog post as well. I think you make the point—or not a claim, exactly—about how effective speculative decoding can really be. Is there anything you want to point out from this about what people should know?
Erik Bernhardsson
Absolutely. Would it help to describe what speculative decoding is?
Alessio Fanelli
Yes. We’ve covered EAGLE and all this—Hydra and all those things—but that was about 2 years ago. I think it doesn’t hurt.
Erik Bernhardsson
Speculative decoding is where you have a smaller model, called a draft model, predict tokens ahead of the bigger model. Then you have the bigger model verify all the predicted tokens. The reason it’s faster is that, if you’re predicting 1 token at a time, you’re bound by memory bandwidth. If you can batch the verification of the draft model, then you’re much more efficient in your use of compute, and it’s faster.
As long as your draft model is producing a lot of tokens that can get accepted—which is called the acceptance length—you can get a speedup that’s multiple times the original model speed. That’s what we highlight here. People talk a lot about making kernels faster and whatnot, but improving a kernel only gives you a few percentage points of improvement. Increasing acceptance length literally gives you a multiplicative improvement—something like 2× to 4×—without much overhead.
Alessio Fanelli
Yeah.
Erik Bernhardsson
Yeah.
Alessio Fanelli
I think you are running a second model, right? So maybe it’s more expensive in terms of compute—
Erik Bernhardsson
I meant quality, but yeah. [laughter] There’s no drop in quality or performance because you’re never accepting a token that—
Shawn Wang
Exactly. Yeah.
Erik Bernhardsson
Right. Yeah. And so we've been working a bunch on dFlash, which is a block-based speculator. Instead of predicting 1 token at a time, it's predicting a block, and we've been open-sourcing our work with it. The next thing for us here is helping people train speculators and custom models.
Traditionally, it's very FTE-driven, support- and deployment-engineer-driven: you work with customers and help them do that. Our vision for this, and why we launched auto endpoints, is that we want to make frontier-level performance available to everyone. We teased this announcement; the next thing we're launching is basically that, as you run an auto endpoint, we shadow traffic and—
Shawn Wang
Do you want to explain what auto endpoints are at a high level?
Erik Bernhardsson
Yeah. So this is, I guess, going back to Modal: you touch the code, but sometimes people actually don't want to touch the code, and they want to get started with an endpoint that works and has all the great performance and scalability that Modal has. We've made that easier with a way to create an endpoint from our UI or from the CLI that has all of our optimizations already baked in, like the dFlash stuff, and there's full transparency.
We give you the code; you can go run it yourself, and if you want, you can eject out into the full Modal experience. As people get sophisticated, they do want to tweak the models and fine-tune stuff. You can still do all of that; it's not a black box.
The next thing, as we tease later in the post, is: how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves, again without having to talk to a person?
Shawn Wang
I guess, just to understand it directly: obviously, you have the GPUs, you have an endpoint that's compatible, and you serve an open model. If someone were to do this themselves, what's the delta that you guys provide? You do a lot of open-source, great work on effective inference. How does it compare to, say, I take the same model, GLM 5.2 2 FB8, take an off-the-shelf inference engine, vLLM, SGLang, get compute of similar capacity at a similar cost? What's the kind of delta that plugging into something like this offers outside of the benefit of scaling?
Erik Bernhardsson
It's interesting because we've taken the approach of open-sourcing our contributions and upstreaming them. We work closely with the SGLang team. We actually want the improvements that our team comes up with to be there and open-source for others to use, even outside of Modal.
Akshat Bubna
The benefit to us is that we have a team with significant expertise. If you do have something that's not there, our team can help you get that performance first. The other thing is that with these endpoints, we are way more elastic, as you said, than anyone else. You have true scaling to zero and true burstiness, and in practice, that matters a lot more to people than just finding a GPU and running model code on it.
Shawn Wang
Yeah, and I will say it's actually not that straightforward to just do what I said. It's easier said than done, right? I think, still, for the average person, it's hard to just gut-check using different combinations. There are quite a few combinations you can make there, and the trade-offs aren't really known at face value.
Akshat Bubna
Yeah, I mean, it's not just that. I think it's that running production-grade inference is a hard infrastructure problem, even if you subtract out the autoscaling.
Shawn Wang
Even if you subtract out the autoscaling, you still have to control things like tail latency and make sure every request is delivered at least once and whatnot. There's a lot of innovation that you can do here. I think it's very interesting that you're starting to encroach on—as you become a full cloud, you're starting to encroach on other people's turf. What will you not do?
Akshat Bubna
Well, we want to follow our users and make sure they get a platform that has everything that works well together. Right now, we're focused on the model lifecycle and the agent lifecycle: going from data prep to training to inference, and also, if I want to deploy a background agent—say, from a sandbox to persistent storage—and a whole bunch of other stuff.
Shawn Wang
We talked to Cole, who did Ramp Inspect.
Kurt Mackey
Yeah.
Shawn Wang
And obviously, Inspect is also on Modal.
Kurt Mackey
Yeah. So Ramp Inspect was a great example of a background agent that was really successful because they were able to use some of the primitives, like snapshotting and fast scaling, to make this something that feels really reactive and works well.
Shawn Wang
Yeah, that's the new CTO of Ramp right there. Yeah, Rahul.
Kurt Mackey
Yeah, Rahul. It was really, really fun.
Shawn Wang
Yeah, I mean, I think I'm very bullish. One of my reflections was that, obviously, when I met you guys, you weren't that much in the GPU game, and now you're all about inference. One of the points that I hinged on for Jensen's keynote at GTC this year was what we're calling the inference inflection: in AI workloads, or machine-learning workloads, it used to be, let's call it, an 8:1 GPU-to-CPU ratio, and now it's more like 1:1.
That's interesting because of how much agents are basically blocked or call out to CPU-heavy stuff. The actual limiting factor swings back and forth from GPU to CPU a lot more than it used to: all GPU, then occasional CPU-GPU-CPU, and now it's just constantly, and you have to colocate everything.
Kurt Mackey
Yeah. And that's one of the things that, again, we see as something appealing about Modal: we've built this capacity pool that spans 17 cloud providers. We're very good at running on various kinds of cloud capacity across the world.
Shawn Wang
You don't have your own data centers.
Kurt Mackey
We don't have our own data centers. We just run across a lot of neoclouds and providers.
Shawn Wang
You're running the math, and you're like, what's the cutover point where you're like—
Kurt Mackey
Yeah, it's a good question. Part of it is that we see our differentiator in the software layer, and being capital-light and focusing on the software helps us move really fast. So far, it's worked out well because there are so many other people building data centers that we're able to work effectively with them and, again, focus on what makes us special.
Shawn Wang
Yeah. Seventeen gets you into the local providers sometimes. Who's the most interesting one?
Akshat Bubna
There are actually a lot more neoclouds than you expect, and they all have various degrees, or various levels, of reliability. That's why something we've invested a lot of time in is building our own reliability layer on top. If the GPU falls off the bus or something happens, user workloads are not affected. That actually lets us use a lot more capacity than you, as a user, would be able to.
Shawn Wang
It's a useful thing to have because now everyone knows what layer you are, and you've sort of optimized for being the supercloud of all clouds.
Akshat Bubna
Yeah, that's the idea. And so I guess, when you mentioned colocation, that's another interesting thing. One thing we've seen is that people come to us when they want very specifically located CPUs or GPUs, like they want—
Shawn Wang
Oh, they pin it in, like, EU—
Akshat Bubna
Exactly, or EU-US—
Shawn Wang
Data locality or performance, or what?
Akshat Bubna
It's either data locality or latency.
Shawn Wang
Yeah, like, you want your—they're running sandboxes in Modal. They want them to be right next to—
That is important in all those things, and so you've kind of accidentally—I don't know if it's accidental, but you've built the perfect primitive for agents to express themselves. It's almost very funny how every extra development just involves more file system, just involves more CPU—just the things that you already have.
I don't know much about whether there are any networking usages that are interesting, but you've also done some good work on networking.
Akshat Bubna
Yeah, I mean, that's exactly right. We're sort of just taking compute, storage, and networking and building stuff on that layer for, again, the stuff people need. We see a few interesting network things coming up. One is that people actually want sandboxes, so we have a—
Shawn Wang
For, like, a Docker cluster-type thing. Sorry, Docker Swarm—what is it called?
Akshat Bubna
Compose. Compose-type thing.
Shawn Wang
Yeah.
Akshat Bubna
So, actually, if you want Docker Compose, our sandboxes now support this thing called Sidecar. A sandbox is actually a pod of containers, and you can run multiple containers in the sandbox. Also useful because, going back to networking, people want a lot of control over outbound networking from a sandbox.
You might want to run a man-in-the-middle proxy for maybe logging stuff for RL, controlling how egress can happen to a domain, or injecting credentials. We've had to build a lot of that stuff ourselves.
Shawn Wang
Yeah.
Akshat Bubna
But then also, sometimes people actually want sandboxes spanning multiple nodes to talk to each other, which is an emerging thing we're seeing. We have support for that for a different reason, and, yeah, we'll see if that becomes safe—
Shawn Wang
Like just an open socket? Is this directly, like, mTLS?
Akshat Bubna
We do support that. You can expose a tunnel inside a sandbox, and then you can either expose it to the public internet, or it can be private. You can add an HTTP auth layer above it, but we have this thing called i6pn, which we haven't talked about, which is an overlay network using IPv6 addresses.
So if Modal containers within the same workspace, when this is enabled, can actually address each other using this private IPv6 address, and no one else can. It's sort of like private networking for containers. We actually built it because we needed it as a primitive for our distributed training product.
We have this other feature where you can add a decorator to a function and get a cluster of GPUs. They have RDMA networking, so you can run a distributed-training job that's truly serverless. We need the overlay network for that, but then we've seen people using it for other reasons, and I'm intrigued to see what people would do with it.
Shawn Wang
Yeah, what would people do with it? Build primitives and let people figure it out, right?
Akshat Bubna
They read the docs: “Let me use that for something you never intended.” This is literally not even in our docs page. People somehow found it and they're using it.
Shawn Wang
I mean, the way you portrayed it, with RDMA versus TCP, was very well laid out. But the transfer-speed change at scale for RL—
Yeah, you have it built in. I'm sure someone found it to be a lot more efficient before you actually made a thing out of it, right?
Akshat Bubna
Yeah. And not to split hairs, I guess the overlay network actually is the TCP overlay network. The reason we have that is you need it to do the key exchange for RDMA before you set up the RDMA network on top of that. But then people found the TCP part.
Shawn Wang
Can I tell you, this is a big aha moment for me? I reviewed 2,200 submissions for the AI Engineer World's Fair. I got this from John Ousterhout. I don't know if you know John Ousterhout by name. He's a well-known professor who published a lot of interesting software-design books, and this is the talk he chose to submit. It's on TCP. It's on RDMA, and I'm like, you wouldn't think that this guy, who's kind of an operating-systems guy, would care about RDMA.
I mean, it makes sense to me because cloud, right?
Kurt Mackey
The way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL, there are a lot of degrees of freedom, and it's basically a systems problem of—
Shawn Wang
Moving memory around.
This shows you how primitive my understanding of networking stuff is. Is this the domain of WireGuard as well?
Kurt Mackey
Not quite—
Shawn Wang
Adjacent?
Kurt Mackey
So—
Shawn Wang
Explain everything.
Kurt Mackey
Sure. How do we move memory around GPUs?
Shawn Wang
Oh, sorry, yeah, that is memory. I was talking more about—maybe I was talking 5 minutes back—about the private IPv6 addressing that you've set up. Is it basically a VPN?
Kurt Mackey
Yeah, it's sort of like a VPN.
Shawn Wang
Yeah, WireGuard is—yeah, you're right. It is. It is—
Kurt Mackey
Yeah, you already moved on to topics.
Shawn Wang
A similar thing in the same space. WireGuard is encrypted, and this is—
Kurt Mackey
You don't need encryption.
Shawn Wang
Yes, it's not encrypted.
Kurt Mackey
That's the main difference. This is TCP, and we have eBPF programs that will reject or allow the TCP connection based on whether you're allowed to do it. It used to involve a full-size sidecar, but now you have eBPF in the Linux kernel.
swyx
Yeah. I don't know if this is a natural follow-on to the topic of my skepticism about distributed training, but people spend a lot of money on cables to hook up GPUs, and even that isn't fast enough—that's the bottleneck. Is your networking fast enough?
Erik Bernhardsson
So I guess you're talking about fully distributed training, like DiLoCo or something, which is cross—
swyx
That would be, yes, that's the extreme. You're kind of in the middle, and then other people would have the Mellanox cables up in their actual data center.
When you run multinode training on Modal, you use RDMA. I think Mellanox or InfiniBand is what you use RDMA with, but basically, it's a way to bypass the TCP networking stack and transfer stuff much faster between one node and the other. We have, I think, 3 terabits per second of internal networking, which is the standard that's needed.
swyx
Okay, so I misunderstood where TCP fits in the stack.
Okay.
swyx
Yeah, I mean, very impressive work. So effectively, you're extending the Modal philosophy to the training cluster—
Erik Bernhardsson
Yeah. And we're not going for, obviously, large-scale pretraining runs. The thing we've built multinode training for is—we see a lot of smaller-scale post-training. People are post-training medium-sized Qwen models so they can get higher quality on inference. This is a perfect fit for something like that.
swyx
Yeah, that is my impression of how a lot of these labs explore branches in post-training and then eventually merge whatever they find in—
Yeah, the other use case we've seen for multinode training is, even if you have a big cluster, your researchers are still doing small runs, and having elasticity there matters a lot more. This is actually the current limiting factor for autoresearch, which is—you basically need to give your model some GPUs. We have a blog post on autoresearch, and Modal turns out to be a pretty good substrate for that.
swyx
So my impression is autoresearch means many things. If anything, it's still science fair right now, right? Not actually—I don't know how many people are actually doing this.
I thought the same thing.
swyx
Yeah, you would know.
Both our internal training and inference teams actually use this general shape quite a bit. We have this one internal repo called AutoInference, which is essentially—we've automated our own FDE efforts using this harness. The agent will just spin up a sweep of different things. It'll even run an NVIDIA Nsight profiler, tweak configs, and arrive at the right thing. It'll change your GPUs from H200 to B200, and it actually works really well.
swyx
Nice. By the way, I enjoy that your FDE is so technical that you have to do these things. It's very different from FDEs at other companies.
Yeah. For our FDE team, they're essentially applied inference researchers or applied training researchers.
swyx
Someone told me they have to be able to build, but they also have to be able to sell. Do they have to sell, or are they like, “They're good”? Is this a post-sale type of thing?
It does. Being able to talk to a customer and engage effectively with them matters a lot. But it's not really a sort of sales thing. We pair them with solutions architects as well, who are more on the pre-sales side.
swyx
Okay, let's spend a bit more time on autoresearch. This is a big focus for me for this year. Where does this go? Have people explored enough? There are all these beautiful charts of improving, improving, then it sort of levels off a bit, and then you find the next thing.
Is this basically one abstraction up from normal training? Is that how we think about it, or do you think about it differently—model-level training versus basically AI-driven hyperparameter search?
Some people call it neural architecture search or whatever, right? The stuff I've seen people do with it is nowhere near the architecture level. It's pretty much tweaking parameters, but it's basically a hyperparameter sweep that's guided by some sort of model intuition. So it's much more efficient than whatever other sweeper you would have. It's just a question of where you want to spend your compute, right?
swyx
Because, yeah, you can just throw infinite amounts of money at this and somehow you'll bang on Shakespeare.
Infinite monkey.
swyx
Yeah, very good for Modal. I think it's also very important that agents can spin up other agents. They can spin up their own infrastructure—very good for you. How good are LLMs at generating Modal code? The benefit of existing pre-LLM is that you're in the data.
Yeah, they're actually surprisingly good. I think pre-Claude 4, they were not, and now they're able to one-shot stuff out of the box. We're playing around with releasing a Modal Bench for the harder things that the LLMs cannot do yet, and maybe—
swyx
What's an example of that?
I think the things that sometimes agents struggle with, without the right guidance and a skill, are how to use the rest of our observability: how to tell when something is failing, how to look at the logs, and then update the right thing. It's sort of reasoning about that.
swyx
But they're able to one-shot—
Yeah, you can't just add a skill to it. We have a Modal skill now, which is actually kind of why we built this Modal Bench. It's to find things like that so we can address them in our tuning skill.
swyx
No, no, I mean, it's good. Are you facing any shortages? We talk a lot about GPU shortages, but also CPU, also memory.
Yeah. We've had a lot of growth, which means we've had to be much better about—
swyx
Proactive capacity planning.
Yeah. So we have a—
swyx
Which, by the way, is like an MBA's dream job.
swyx
It's like just planning this stuff. I think last time you and I talked, it was maybe about this.
Yeah. I mean, we have a really competent team of people. The role's called compute strategy. So if anyone listening here [laughter] wants to work on compute strategy, that's the team.
swyx
Yeah. I mean, the normies call it FP&A or something.
Well, it's more than FP&A. There are a lot of interesting financial questions: What is the blend between 1-year and 3-year reservations? How do we forecast our own capacity? Especially since our capacity is very fungible across different GPU types and different regions, you basically have to model a lot of it.
You also have to have an opinion on how the supply chain is going to evolve, and then you have to take bets based on that. Tokenomics.
swyx
Yeah, this is probably not a real point, but I was trying to think about what other industries have had these kinds of problems. We can't be the first to have them. I was thinking about airlines with fuel, and how they have to hedge their fuel. For a long time, Southwest made a heroic fuel bet, so they were super low-cost compared to everyone else. [laughter]
Yeah, I had thought about that. We're having a fun time, too, you know. A lot of the compute business in general, for us, is also about being very good at capacity management. That is how you have great unit economics, but also, over time, how you can unlock more value for customers.
One of the things we're building now is a way for customers who don't care about latency to get much cheaper pricing. They'll get results back in the next 24 hours or something—a batch tier, essentially. Those are levers we have because we control the whole stack, scheduling, and whatnot, to give people a sufficient service.
swyx
Yeah. I feel like those aren't as popular as they should be. The frontier labs have all those APIs, but batch tiers aren't as popular as they should be.
The demand that we see for something like that is actually not for LLMs, although sometimes people want to run evals and do synthetic data preparation, and it makes sense there. But it's from a lot of non-LLM companies, like people doing computational biology. They haven't run really big batch jobs, and they don't care when they get it back.
swyx
Yeah. It's also like a cousin to the halting problem: Will this finish in time? [laughter]
Yeah. You can bound it. You can give people a bound on it.
swyx
Yeah. I think what's interesting is the next phase of Modal. What do people expect from you now that you're established and you're a well-known compute player among all these leading companies? You had an inference launch week, and we talked a little bit about the launches. What else should people know?
We are building primitives that make our users' lives much easier. With LLM inference, thousands more companies are going to post-train their own models and deploy open-source models for inference. So we're thinking a lot about what the best product shape for that is, and that involves everything from our training gym to endpoints that get frontier-level performance, again without having to talk to anyone.
It looks somewhat different in other verticals. We're also seeing a lot of real-time audio and video work, which is why we're working on things like regional routing with fallbacks. You can get GPUs as close to users as possible, so you get low latency for video streaming and whatnot.
On the agent side, we're still working very closely with our customers because things are changing so fast in terms of what they need. Beyond sandboxes and persistent filesystems, there are a lot of other things people need from this agent stack as they build production agents. We're thinking about those other things that fit in there.
swyx
I want to ask what the other things are.
Yeah, [laughter] probably can't share right now. I do think a lot about the principal components of cloud, and you do talk about compute, storage, and networking.
swyx
So far, for me, the first couple of generations of cloud are fine. What's qualitatively different about agents that means you need some new permission level?
A lot of people—obviously, okay, I'm just going to spew tokens at you until it hopefully sparks something—are saying the new level is whatever Claude Code does, which dangerously skips permissions, or allows an allowlist by command, or whatever, right? Sometimes they're like, “Okay, well, we have this adaptive thinking mode where—just trust me, bro—I will make the calls for you.” Is that it? Basically, is it LLM-mediated permission?
Now you're looping it with a goal and following through.
Shawn Wang
Yeah. I mean, I'm skeptical of LLM-mediated permission for stuff at the sandbox level because you do want hard boundaries. Obviously, someone can exfiltrate stuff.
But maybe that's old-school thinking. Maybe we're the dinosaurs. Maybe the AI OS, or the LLM OS, is really—the kernel is a goddamn LLM. [laughter] It makes you feel uncomfortable, but that's what trusting the LLM is. Imagine a spherical cow, a perfect LLM, right? [laughter] Maybe I want to test the boundaries. Obviously, I don't believe that, but I want to see where I'm wrong. That's the non-consensus.
Erik Bernhardsson
Yeah. I mean, I think you always need hard guardrails. You can pair those with softer guardrails, right, and ask an LLM to mediate it.
Shawn Wang
And to end with, a couple of your comments on the ecosystem outside of Modal-managed agents. Everyone has one: Gemini, OpenAI, Claude. They're very useful for you, but they're also a way for those companies to start edging into your space. What's going on?
Erik Bernhardsson
Yeah. We're very excited to partner with Anthropic and some of the other foundation labs. Building on them is a great place to start if you're starting out building an agent, but then when you get to building something more production-grade—say you're a company like Ramp that's building its own agent—Ramp also runs its accounting agent on us, one of its external-facing agents.
You need a lot more control over your compute primitive. How do you process different files that the agent has access to? How do you snapshot and restore? How do you control the networking? Maybe you want GPUs. When you get to that point, you want a specialized sandbox provider that gives you those things, and that's the role we're trying to play.
We don't really have an opinion on the harness—whether it runs in a cloud-managed agent and you hook it up to a Modal sandbox, or you run the harness in a Modal sandbox. We'll see where people converge with that.
Shawn Wang
Do you have any opinions on the meta-harnesses? Is it just another layer on top of these things?
Erik Bernhardsson
You mean, like OpenAI and—
Shawn Wang
OpenAI is one. I think Vercel had one, which I can't remember the name of right now. Fred Schott had one. Most recently, Databricks had Agent Bricks. All these are sort of meta-abstractions, kind of pseudo-agent-cloud-type things.
Erik Bernhardsson
I personally haven't played around with them or built anything on top of them.
Shawn Wang
I mean, everything's bullish for Modal as long as it consumes more infrastructure. That's why we're focusing on the infrastructure layer. It's somewhere where our relative competences are, and it's also a hard problem to solve.
Just generally reflecting on this as an infrastructure person—not as intense as you, but in that field—this has been the most exciting time in infrastructure. It was boring for a while, actually, and you couldn't really get people excited about data infrastructure. Erik would get on Data Council, everyone just watched the video, and it was like, “Look at how many sandboxes I spin up,” and no one gave a crap. [laughter]
Erik Bernhardsson
Yeah, that's true. It is a very exciting time, and I think a lot of that is driven just by the amount of scale all of this stuff needs.
Shawn Wang
I think a lot of your initiatives, a lot of your product directions, make sense in retrospect, which is the best kind, but I wouldn't necessarily have thought about them myself. We need the predictions, you know? I mean, I think there's a lot that you just don't even see, right? You have the batch, you have the voice, you have the multimodal. What else is coming up for us? Where do you see things going?
Erik Bernhardsson
In general, it's clear that there's obviously a huge shift happening. I think one thing that's not as obvious to people, because LLM inference gets talked about so much, is that we also work with a lot of companies doing things like drug discovery and computational biology. Big things are probably going to happen there—the Chai Discovery world.
We work with a lot of robotics companies that are actually putting robots in active deployments and getting good results out of them.
Shawn Wang
Is there an air-gapped Modal? Is there a version that's on-prem, air-gapped, whatever?
Erik Bernhardsson
No, we're cloud-only.
Shawn Wang
Yeah, okay. But, yeah, what you're saying is that because you're focused on primitives, and they're good primitives, you find use cases and all these kinds of things. That actually probably diversifies you a little bit away from LLMs all the time.
Erik Bernhardsson
Yeah, absolutely. Our goal isn't to only serve the LLM-first market. Just on the website, there's the audio, the bots, and the bio images. Yeah, I mean, there's a lot here. There's Qwen TTS, Customizing Whisper, and Chatterbox.
Shawn Wang
You know, there was Customizing Whisper.
Erik Bernhardsson
Yeah, yeah.
Shawn Wang
This screen reminds me of a fallen competitor: Replicate. What's your postmortem on what happened?
Erik Bernhardsson
This is one thing we've kind of stayed away from: providing an API for models, because I think some model APIs end up serving a really hobbyist market, which is much less sticky. We've always wanted to build for companies that are building products and need more flexibility that's not just an API.
Shawn Wang
You can build an API for a model, and this is clearly what it is, but you're saying you can wrap it into a more fully functioning backend that you run.
Erik Bernhardsson
Yeah. Actually, all of our examples—it's not just, “Spin up this model; here's an API token; use it.” They're actually all code.
Shawn Wang
Okay.
Erik Bernhardsson
The point is that this is example starter code.
Shawn Wang
Starter code.
Erik Bernhardsson
Yeah. But you can tweak it however you want. If you're a company building a product, like computational biology or whatever, you can go much further.
Shawn Wang
I guess I'm trying to tease out for listeners: when does it stop becoming, “Oh, you're just an API call and you're just a wrapper on an API,” and become what you call a product? Obviously, there are more lines of code, but beyond that, what is the substance that people add that qualifies it to be something more?
Erik Bernhardsson
I think there's a little bit of a selection effect. A lot of companies that do want to get deeper into that level are probably building something that's more differentiated. An example is that, with LLMs and friends, we originally worked with companies that were building their own post-training frameworks. Ramp, actually, early in the day, was training its own tokenizer and swapping out the tokenizer in LLaVA and whatnot. I'm not saying that was successful in that case.
A better example is, let's say Krea, because Krea does not use Modal for training.
Shawn Wang
Krea on Modal.
Erik Bernhardsson
Yeah. But they use Modal for all their inference, and that's because they have a completely custom model architecture. That means they have to be at the code level and tweak things that aren't an API.
Shawn Wang
It's interesting as well. We had Ethan, most recently on the xAI Grok team, make a prediction that the next tier in video models is a better model or an agent that orchestrates video models—a language model backbone that can use tools and write code. Yes, I can make my 6-second video or my 10-second video from Grok, but actually I want my 6-minute video, and I'm not going there through normal video generation.
Erik Bernhardsson
Yeah, that's interesting. We have GPU sandboxes, and recently we've seen a few companies doing agents that do video manipulation or—
Shawn Wang
Give it FFmpeg? That's not—you need to give it Adobe.
Erik Bernhardsson
Yeah. I hadn't put it together that it would actually be a video production thing. In my mind, these things were going more toward editing.
Shawn Wang
Yeah. I think about this a lot, obviously. [laughter] Sorry.
Erik Bernhardsson
Luma Agent is a version of this for video production, but it's a one-off.
Shawn Wang
I was going to get your quick takes on some other recent news and see if you have anything interesting. Gitpod is in a somewhat different market—they're in the CI/CD market—but they're technically very impressive. I don't know if you've taken a real look at them.
Erik Bernhardsson
Yeah. People on our team have talked to the Gitpod team, and they're technically very strong.
Shawn Wang
Yeah. We're very bullish at Modal on the CI market as well, because as there are more agents—more coding agents—they're going to run a lot more CI, and the primitives there can be much better.
Erik Bernhardsson
I think there's a lot of wasted CI.
Shawn Wang
Yeah. So, let's filter that. What is the highest-order bit in improving CI for agents?
Erik Bernhardsson
There's a lot of wasted time in CI on preparing your artifacts and basically preparing your dependencies and whatnot. Obviously, build systems help with that, but if you have primitives like memory snapshot and restore, can you just run CI more efficiently?
Shawn Wang
Oh, okay. Interesting. Yeah, I mean, another form of on-demand compute.
Erik Bernhardsson
Yeah, exactly.
Shawn Wang
Yeah. It needs the same agent platform. For those who don't know, Gitpod rebranded to Ona. There was this whole thing. I actually semi-sounded the alarm at Cognition. I was like, “You should take these guys seriously because their infra is very good.”
Then they joined OpenAI, and presumably we'll see Codex Cloud from the Ona team, which I think would be very strong. To me, teams like that, which can set up the networking and the secure boundaries for your agents to have their own cloud each, effectively, is what you're doing, kind of. I'm just trying to draw the analogy or the differences, if you've studied them. What is the philosophical difference?
Erik Bernhardsson
My sense is maybe they didn't go after the right market at the right time, because we, I guess, also got lucky with agentic use cases really taking off and with Modal landing more on a sandbox-shaped thing than a VM-hosting thing.
Shawn Wang
Sandboxes work.
Erik Bernhardsson
A sandbox is a sandbox. [laughter] It's just build-time sandboxes versus runtime sandboxes, and actually it turned out runtime was better.
Shawn Wang
Right. The difference there is that runtime sandboxes have a different configuration surface: how you configure images, how you attach storage, and so on. It's fascinating. Astral also joined OpenAI—Python tooling ecosystem people. Are you still bullish on building on top of Python?
Erik Bernhardsson
Yeah. We had Python as our first SDK language because that was the language that people did data and ML in. I actually now have Go and TypeScript SDKs as well, and our runtime is completely language-agnostic. It is in Rust, but it's not tied to Python by any means. We haven't seen much change there. With inference and training stuff, people are still very Python. The interesting thing with the agent stuff is that people use our TypeScript SDK a lot more because they're not actually doing anything with ML. I don't think we'll have to go beyond that super soon, because Python and TypeScript are still dominant.
Shawn Wang
The last 2 languages in the world.
Erik Bernhardsson
Yeah, that's it.
Shawn Wang
Well, English and prompting—
Erik Bernhardsson
English and prompting.
Shawn Wang
I occasionally talk to people who try to build new languages. Even Brett Taylor, who's chairman of OpenAI, says we need a new language for LLMs. No one has come across one, and I keep looking. Python and TypeScript have a lot of adoption, but they're also very imperfect as languages themselves.
My close is this: I think Modal used to be a big bet on developer experience, and you've pivoted the team to agent experience. Can entire companies and unicorns—multi-unicorns—be built on just having better agent experience? Do you need something else?
Erik Bernhardsson
It's a big part of our identity. It's not just the very tactical question of how an agent uses the CLI; it's also how easy it is to spin something up. What is your iteration time when you want to spin up a new service and get something going in production? In practice, that matters a lot to people, and I think it will continue to matter. People are building stuff even faster, and if you give them ways to do it quickly without overhead, I think that matters.
Shawn Wang
The debate for me has been: do you do anything fundamentally differently for developer experience versus agent experience? You seem to be on the side that they're the same.
Erik Bernhardsson
We have a blog post on that.
Shawn Wang
Cosine similarity of, like, 0.9 or whatever.
Erik Bernhardsson
Yeah, pretty much. The main shift for us has been, as I said, that we built this benchmark, ModalBench, to see where agents are lacking. We literally add surface area to a product. If they're reaching for something, maybe this should just be a CLI.
Shawn Wang
They hallucinate their own features.
Erik Bernhardsson
Yeah, and sometimes it makes sense. If they're reaching for this thing, it's product feedback—give it to them. We also used to only have logs and metrics in our UI, so we moved all those things into the CLI as well, so they're accessible in that form.
Shawn Wang
Simple as that. Cool. Thank you so much. This is a great update, and I can see why you guys have succeeded so much.
Alessio Fanelli
It is really focused, but also really good execution.
Erik Bernhardsson
Thanks. I mean, we have a long way to go.
Alessio Fanelli
All right. Thank you. Cool.