Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal
VibhuBasil ChathaErik Bernhardsson
- Erik Bernhardsson estimates Modal is at least 50× larger than when he last appeared on Latent Space, in revenue, users, and usage. Revenue is now about two-thirds Modal Functions (mostly inference) and one-third Sandboxes—a product the team considered shutting down. Sandboxes saw little growth after a quick prototype, then took off as LLMs moved from snippets to full apps, with use cases from vibe-coding companies like Lovable, Ramp’s background agents, and neo-labs running reinforcement learning. Vibhu observed sandbox revenue growing 30% month over month; Erik called the growth “kind of nuts” and said it kept exceeding his expectations.
- The GPU procurement market changed sharply in the past year: Modal went from hyperscaler spot capacity and one-year reservations to working with almost 25 clouds, signing three-plus-year deals with meaningful upfront payments, and building a four-person procurement desk and a ten-person bare-metal engineering team. Capacity is often sold in “three-year rectangles” starting months ahead, while real demand is bursty. Modal’s proposed advantage is aggregating thousands of customers and flexibly slicing up capacity.
- Bernhardsson says Modal is, to some extent, “effectively long GPU prices already” through locked-in reservations, but worries more about a price collapse than further spikes. Hardware prices eventually tend to fall; today’s scarcity rent flows mostly to NVIDIA and partly to neoclouds, while buyers who locked up supply risk “buying at the top.” He is aggressively buying capacity, but not trying to corner the market: prices, utilization, and the resale market could fall together, as his Hunt brothers’ silver-market example illustrates.
- He explains the neocloud growth model: presell five years of chips to a large buyer, take the contract to Wall Street to raise money, then buy the hardware—“pre-sell things that you haven’t even bought.” He names Endscale as an example of a company that went from nowhere to a pipeline worth hundreds of billions of dollars despite having little capacity. Modal’s product-led-growth model can’t lock in long contracts in the same way. He’s glad startups are building GPU price indices and derivatives, but doubts GPUs will become a “Henry Hub… Brent crude”-style, highly commoditized tradable asset.
- “2027 might be the year of non-NVIDIA accelerators”—a call he admits he has gotten wrong for five to ten years. AMD is getting good scores on some tasks, and TPUs work well inside Google. He sees better software support as a potential unlock for alternatives, while noting TSMC and ASML may be deeper bottlenecks. Hybrid disaggregated inference—NVIDIA for prefill, Cerebras for decode—is already being tried.
- The future of sandboxes is more ephemeral: Modal’s median sandbox is about eight seconds long, and Bernhardsson envisions roughly 100-millisecond startup, running a few commands, then stopping and snapshotting it. That could enable branching agents and running every CI test in its own sandbox. The obstacle is overhead from starting and stopping; reducing that “transaction cost” could make highly parallel workloads practical.
- A shift in the fine-tuning value proposition: he says open-source models such as Kimi K3, DeepSeek 4.1, and GLM-5.3 are now almost as good as frontier models at lower cost, often without fine-tuning. He also argues there is a structural role for inference providers separate from model labs: demand shifts with new releases, and Modal can reallocate GPU capacity between models without being asked.
- Modal grew headcount about 4× annualized, a pace Bernhardsson calls unsustainable, and may reach about 700 people next year. FDEs grew from about four to about 24; he says Modal hasn’t aggressively built a product-management team. Code reviews are optional, backed by investment in CI and testing, and the company uses agents to probe its own systems—“escape the room, but for agents.” He expects models to keep improving in verifiable domains, while humans remain competitive for a long time in areas with diffuse targets and longer feedback loops.
1. Spotify’s recommender to Modal: “the infrastructure f*ing sucks”
- Bernhardsson joined Spotify out of school in 2008 or 2009 and spent seven years building its first music recommendation system—Related Artists, Discover Weekly, and “a lot of matrix factorization.” Modal’s genesis came from building AI applications and realizing “the infrastructure f*ing sucks. There’s got to be a better way”; it was a tool he had wanted throughout his Spotify years.
- The path ran through his open-source workflow orchestrator Luigi, in the same category as Airflow. Returning to workflow-orchestration ideas in 2021, he realized “a core puzzle piece isn’t just orchestrating the compute, it’s executing the compute”—a bigger problem—and shelved orchestration. Modal was the third idea he got excited about, after a web-scraping attempt that went nowhere.
- The Better.com detour: five years as CTO of the mortgage startup, growing the tech team from two to “300 or 400,” he thinks. He learned about management, hiring, and developer productivity before returning to infrastructure, AI, and machine learning. Interest rates rose quickly after the pandemic, causing issues at the company, but he had already left.
2. Bootstrapped, shitposted into existence, saved by Stable Diffusion
- He self-funded Modal through its first employees, taking no salary for a long time—“it made me really frugal about money”—and raised a seed round in early 2022. A co-founder joined as CTO around August 2021. Growth was founder-led marketing: “what I’m good at is, I guess, shitposting and being terminally online.”
- The killer app wasn’t ChatGPT but Stable Diffusion, which arrived that fall when Modal had four or five people. Revenue went from about $100/month in November 2022 to about $2,000 the next month, then $30,000 in January 2023, almost entirely from one New York Stable Diffusion company, Mage. Mage accounted for 80–90% of revenue in Q1 2023; Suno became a large customer in late 2023, and revenue later diversified.
- Since the last Latent Space appearance, he estimates Modal is “probably at least 50×” larger in revenue, users, and usage. Customers he names include Decagon, Suno, Runway, and Co-Cognition.
3. You can’t wrap Kubernetes: rebuilding the stack for GPU economics
- His thesis: every technology era runs up against the limitations of its tools. Kubernetes and Docker were built for steady backend workloads on relatively cheap CPUs; generative AI brings expensive GPUs, utilization pressure, and large models that need to load over the network. “You can’t build for this new generation of software by putting a wrapper on top of the old stuff. You need to start over a little bit from scratch.”
- Modal built its own file system and container runtime, along with CPU and GPU snapshotting, an autoscaler, a scheduler, and capacity-management tools. Developer-experience work also proved useful for production workloads, since scaling containers up and down quickly is a similar problem.
- One consequential early decision was going multi-region and treating capacity as “one big pool over the entire world,” rather than separate clusters in each region. Modal has had bugs and outages; recently, Bernhardsson says, it has basically hit four or five nines of uptime.
4. The capacity market flipped: 25 clouds and three-year prepaid rectangles
- Until the end of last year, most capacity came from hyperscaler spot capacity or one-year reservations. Spot capacity was preemptible, but worked in aggregate across hundreds of regions. Modal initially used AWS, later GCP and Azure, and rented Oracle capacity early for A10s and H100s.
- In the past year, Modal has had to work with almost 25 clouds. Getting capacity often means three-plus-year deals and meaningful upfront payments. The last six to nine months prompted a procurement desk of four people, full-time on calls, deals, and vetting, plus a ten-person engineering team managing bare-metal capacity worldwide and handling network issues, bad drives, and GPUs “falling off the bus.”
- Modal doesn’t own capacity; it rents it, typically through three-year deals. Locking up capacity for 2027 will require work across many avenues. Bernhardsson says capacity prepayments are currently the best use of cash; salaries and offices are “basically a rounding error” by comparison: “we have two hundred employees… the GPUs cost a lot more money.”
5. Rent vs. buy: already long GPUs, more afraid of collapse than spikes
- He isn’t dogmatic about buying. Equity can be “a very diluted way” to fund hardware; debt usually requires long-term revenue to cover interest. Owning hard assets adds operational and financial complexity. He would rather focus on being a lean software company, but “everything is on the table,” including joint ventures and complex SPVs.
- He pushes back on the idea that capacity is permanently unavailable: NVIDIA is delivering several hundred thousand Blackwells a month, and “if you’re willing to pay more than everyone else,” you can get capacity. He compares panic buying to the pandemic toilet-paper shortage. Prices could keep rising for another six months or another 36; it’s hard to know.
- He worries more about prices collapsing than rising further. That could hurt anyone locked into a lot of supply: the risk is “buying at the top.” Today’s scarcity rent goes mostly to NVIDIA and partly to neoclouds and other capacity owners. There may be a long-run premium for owning GPUs, but he questions whether it’s worth holding the assets on the balance sheet.
6. Don’t corner the market: Hunt brothers, Armenia, and an internal ClusterMAX
- Asked why not buy as much as possible and resell any surplus, Bernhardsson warns that in a downturn, “a lot of things are correlated”: if prices and utilization fall, the resale market could vanish at the same time. He is aggressive about buying capacity and raising money for more, but wants to build a company that lasts 20, 30, or 50 years. He invokes the Hunt brothers’ attempt to corner the silver market, which “failed spectacularly.”
- He speculates that NVIDIA prefers some diversification among buyers, while Anthropic and OpenAI tend to prefer established vendors and need very large blocks for training. Modal can pursue smaller, scattered capacity: “You can get 32 nodes in Armenia. You can get 128 nodes in Indonesia. We love that kind of shit.”
- Modal vets vendors with a spec sheet covering items such as IOPS, CPU PassMark score, north-south bandwidth, egress, and ingress. It often runs acceptance tests on one node, but larger commitments can still involve “a little bit of a leap of faith.” Internally, he says, they have something like their own version of SemiAnalysis’s ClusterMAX, rating vendors on reliability, responsiveness, and support.
7. The neocloud presell machine and GPUs as an asset class
- Neoclouds can scale quickly by offering a big buyer five years of chips at a good rate, taking the contract to Wall Street for financing, then getting hardware from companies such as Supermicro, Dell, or NVIDIA. “You can essentially pre-sell things you haven’t even bought.” His example is Endscale, which he says went from nowhere to hundreds of billions of dollars in its pipeline despite barely having capacity. Modal’s product-led-growth model lacks the same ability to lock in long-term contracts, making it harder to scale financially as quickly. He calls CoreWeave and other large neoclouds savvy about raising money and using GPUs as collateral for loans.
- On startups such as Silicon Data, Compute Exchange, and Ornn trying to create price indices and derivatives: compute is less homogeneous than other asset classes, given differences in networks, locations, disks, CPUs, and reliability. Still, “you can say the same thing about oil.” He sees a need for better risk hedging, risk-sharing, and insurance, but isn’t sure GPUs will become a highly commoditized tradable asset like Henry Hub or Brent crude.
- Asked about a tweet wishing IPO-ready companies in a CapEx arms race could coordinate and pace it: “No comment.”
8. “2027 might be the year of non-NVIDIA accelerators finally”
- He acknowledges his track record: “every year for the last five, maybe even ten years, I’ve been saying, ‘Next year is going to be the year of non-NVIDIA accelerators,’ and I’ve always been wrong. But I do think 2027 might finally be the year.” AMD is getting good scores on some tasks; TPUs are widely used internally at Google and seem to work well. He says he doesn’t see as much about Trainium, though its team is good and working on software.
- He sees software as a potential unlock: “if we fix the software layer for these accelerators, there’s going to be a lot of demand for them.” He also notes that TSMC and ASML may be deeper bottlenecks. He expects more alternatives over time, while emphasizing that NVIDIA is an excellent company with an excellent product.
- He’s excited by hybrid, disaggregated inference, such as running NVIDIA for prefill and Cerebras for decode. Some workloads are compute-bound and others memory-bound, so hardware can be mixed and matched, though the setups get complex in practice. He also mentions Groq.
9. Two products: Functions at two-thirds, Sandboxes the third they almost killed
- Modal Functions, launched in 2021, offers serverless deployment and scaling while Modal manages capacity. It reached product-market fit in 2023 and is now about two-thirds of revenue, mostly from inference. Bernhardsson resists becoming a neocloud that sells only rectangular, three-year GPU deals; Modal’s product is designed to deploy and scale cyclical or bursty workloads and aggregate demand across many thousands of customers.
- Sandboxes started with the observation that LLMs were generating code and might need a place to run it. The team already had container-management and isolation primitives, including gVisor; a co-founder built a prototype in a day or two. There was some early uptake, but it was an insignificant revenue source going into summer 2023. Two years later, the team even discussed shutting it down.
- He says LLMs became able to generate full apps in 2023, beyond refactoring functions and writing snippets. Growth came from vibe-coding companies such as Lovable, background agents at companies like Ramp, and neo-labs using Sandboxes for reinforcement learning. Sandboxes are now about a third of revenue.
- Bernhardsson says he didn’t foresee how large the product would become. In response to Vibhu’s observation that sandbox revenue was growing 30% every month, he called the growth “kind of nuts” and said that although he kept thinking it had plateaued, it kept growing. Multi-product was always part of the vision; Modal now has Functions and Sandboxes, with training and batch in the works. “We’re in the business of building building blocks.”
10. The ephemeral sandbox thesis: eight-second median, fork everything, thousand-test CI
- Today’s sandbox use is often “lift-and-shift”: put a dev box in the cloud and have an agent control it. Bernhardsson’s preferred model is long-running agents as durable workflows that can start a sandbox in about 100 milliseconds, run a few commands, shut it down, and snapshot it. The median Modal sandbox is “eight seconds long or something like that.”
- The CI example makes the case: rather than running a sequential install-and-test flow, run each test in its own sandbox and “parallelize the hell out of that”—potentially 1,000 sandboxes. Today, setup overhead can take minutes. If you do setup once, snapshot the container, and clone it, the parallelization benefits become practical. Otherwise, the friction is “the transaction cost.”
- He sees potential in forking, snapshotting, resuming, and live migration. Modal has added virtual machines alongside gVisor for workloads that need full Linux compatibility. He also describes Anthropic’s managed agents as having a bring-your-own-sandbox integration for Modal. His view is that Anthropic benefits from a broad ecosystem of tools, provided customers use Anthropic for inference.
11. Training’s underserved middle and the fine-tuning reversal
- Bernhardsson is bullish on training, especially for researchers iterating and for smaller and mid-sized workloads rather than 100,000-GPU hyperscaler or neocloud deals. Examples include text-to-speech, speech-to-text, recommendation systems, and computational biotech; some customers train relatively modest models on 64 GPUs. “It’s very hard to set up training clusters.”
- Batch is another opportunity: one customer needed to transcribe 5,000 years of audio from millions of files, and Modal had enough capacity to do it in a few days. He is also bullish on low-latency workloads as speech-to-speech models improve, and on building general-purpose, composable blocks for AI-powered software.
- His fine-tuning view has shifted. A few years ago, tuning a smaller open-source model could bring it closer to frontier quality at lower cost. Now, he says, models such as Kimi K3, DeepSeek 4.1, and GLM-5.3 are almost as good as frontier models at much lower cost and can often be used off the shelf. Demand for post-training remains, including supervised fine-tuning and reinforcement learning, but fine-tuning is optional rather than something users have to do.
12. The Hugging Face incident, agent pentesters, and why cyber incumbents sell fear
- The agents involved in the Hugging Face exploit were running inside Modal and found a vulnerability in an app deployed on the platform. As Bernhardsson understood it, they used that app as a springboard to get full internet access. He stresses that an app running on Modal was hacked; Modal itself was not.
- One lesson is whether Modal can make security more restricted by default without losing the product’s “magic.” He compares this with AWS making it harder to accidentally expose an S3 bucket publicly. Modal has also begun using agents internally to probe for vulnerabilities and sandbox escapes. “It’s like escape the room, but for agents.” They started a few months earlier, around the time Vibhu mentions Fable.
- He believes agents can be very good at finding vulnerabilities and can also be used defensively. Asked whether cybersecurity companies are “cooked,” he says that is outside his expertise and doesn’t jump to that conclusion: security vendors have enterprise relationships, and “a lot of what you’re selling is effectively fear.” Modal used to bring in human penetration testers roughly every six months, he thinks, and may now need them less often; he remains unsure. He’s optimistic that people and companies will use AI to work more efficiently.
13. Agent experience is developer experience: AEO, deprecation, the slop cannon
- Asked whether developer experience matters when agents can read the docs, he says almost everything that makes DX good also helps agent experience: good docs, descriptive errors, and fast feedback loops. Agents make some hard-to-use tools easier to work with, and the human toil of dealing with YAML has gone down. He expects infrastructure to become more programmatic, with SDKs and imperative operations rather than declarative DSLs.
- Agent training data makes deprecations harder: agents often use old Modal syntax, so the team has become more conservative about removing features and must keep examples online up to date.
- Modal runs internal benchmarks asking agents to deploy Stable Diffusion or rank sandbox vendors by reliability and price. Agents can misunderstand pricing, so the team works to make content clearer. Asked about generating GitHub repos full of Modal code to influence agent recommendations, he calls it a “slop cannon.” He thinks automated content could work, as with early SEO, but doesn’t like the idea.
14. FDEs, product management, the IC renaissance, and 4× headcount growth
- Modal scaled its forward deployed engineers from about four at the start of the year to about 24. They work with customers on performance tuning, including training draft models and adjusting parameters in DLM and SGLang; help customers work around problems; flag over-provisioned CPU or memory; and assist with migrations. They are not meant to make Modal a service-heavy business. A key goal is to identify product gaps: “Long term, how do we turn this into product so we don’t need an FDE for it?”
- Modal hasn’t aggressively built a product-management team. Bernhardsson says infrastructure product management requires deep strategic thinking about foundational building blocks and one-to-two-year horizons, and avoiding many similar but distinct features. Some people have moved into product roles from DevRel or engineering.
- He sees a shift back toward individual contributors, which GenAI may have accelerated. Modal makes clear there is no prestige difference between being a manager and an IC; former managers are returning to writing code, and Erik and Akshat still do so. Code reviews are optional. He sees them partly as a way to avoid accountability for merging code, so the alternative is investing in CI, deployment, and testing. Infrastructure companies need “one more nine” of uptime, with resilient systems, rate limits, decoupling, and strong test coverage. He says agents have become good at writing tests.
- Headcount grew about 4× annualized over the past year, which he calls unsustainable. He watches for cultural drift, islands of differing work styles, and engineers taking on speculative projects that may not be the right priority. Modal may reach 250 people by year-end and around 700 the following year, with most hires in systems engineering and more work on GPU performance and research, including post-training, distillation, and quantization. He says the story behind the Dunbar number may be “more fake than anything,” but that the underlying effect is “actually kind of real.”
15. The rectangle endgame: flexible capacity, Taylor Swift, and what’s not plateauing
- Bernhardsson’s tweet argues that the problem with GPU scarcity isn’t only getting GPUs; it’s that capacity is sold as a three-year rectangle starting three to six months in the future, unlike most users’ bursty demand. Modal can often provide 100 GPUs on demand, without a sales conversation. Its value proposition has two parts: developer experience and flexible capacity. By aggregating many thousands of customers, Modal smooths demand and can slice capacity up and down quickly.
- He sees a structural role for inference providers separate from model labs. Demand shifts when a new lab releases a model; Modal can move resources between models as demand changes. He says no one even has to tell Modal to do it; the system handles it.
- On whether LLMs are commodities, he’s unsure: “Is Taylor Swift a commodity?” If labs keep releasing new models and their margins cover training costs, they could still have a good business.
- He doesn’t think intelligence is plateauing, though he says it remains to be seen how well models handle tasks that are harder to train with reinforcement learning. Models have improved at coding, math, and cybersecurity—areas with more verifiable outcomes. Humans may remain competitive for a long time in tasks with diffuse targets and longer feedback loops.
- He names computational biology, where predictions can be checked with simulations of molecular dynamics, and materials science, where properties can be measured, as promising areas. On-prem remains a model he is skeptical of: he is a cloud maximalist and says BYOC and VPC peering reflect an outdated network-as-a-moat security model. He doesn’t know whether Modal will ever do on-prem, but would consider BYOC at a sufficiently high price and sees it as a possible way to use neocloud capacity while keeping Modal’s control plane in its own environment.
Full transcript
Thanks for coming on, Erik Brynjolfsson from Modal. Could you explain the business a little bit and maybe how you got here?
Yeah. We build AI infrastructure. The easiest way to put it is that we offer a bunch of compute, a compute platform, and a storage platform that enables customers like Decagon, Suno, Runway, Co-Cognition, and many others to build AI applications. In practice, that means a lot of inference, sandboxes, sometimes training and fine-tuning, and sometimes batch jobs and other things. We’ve basically built a software SDK that enables customers to move really fast and build this. We manage all the capacity underneath and make it easy to deploy, scale, and run these things in production.
You were on Latent Space like 2 years ago. A lot has happened since then. How big were you back then, by whatever numbers you can share, and comparatively, how big are you now?
I actually don’t remember where we were.
Okay.
But it’s probably at least 50×—
Yeah.
—in terms of revenue and users and usage—
Yeah. Yeah.
—and stuff like that. Yeah.
I use it for a side project I have where I turn 2D images into 3D models.
Nice.
Yeah, just hosting a couple of models there.
That’s awesome. What do you think? Any feedback?
I love it.
Nice. Good.
That’s why I use it over any competitors or anything.
Good.
Before you started Modal, you were at Spotify and also at Better.com, I think.
That’s right. Yeah.
What were you doing at Spotify?
I did a lot of different things at Spotify, but my main contribution was building the first version of a music recommendation system. I joined after school in 2008 or 2009, maybe, and spent 7 years at Spotify. Most of that time was spent building a music recommendation system basically from scratch. I built up a small team, and we launched a bunch of features like Related Artists and Discover Weekly, and a bunch of other stuff. There was a lot of matrix factorization and compute-intensive work. Arguably, a lot of what later became the genesis of Modal came from building AI applications and realizing that the infrastructure fucking sucks. There’s got to be a better way to do that. The rationale for Modal was to build a tool I’d always wanted throughout my years at Spotify.
So Modal was more of an offshoot of what you learned at Spotify, not so much from Better.com?
Better.com was a bit of a side tour. It was kind of a roller coaster, and there’s a long story about the company that we can talk about separately. I was the CTO for 5 years at Better—
Yeah.
—and grew the tech team from, I think, 2 to 300 or 400. I learned a lot about management, building teams, and hiring. I also learned a lot about developer productivity and thought a lot about tools and things like that. I did a lot more full-stack work. But I came back to what’s close to my heart, which is infrastructure, AI, and machine learning.
What was Better.com doing again?
Mortgages.
Oh, okay.
Yeah.
How did you get into that? We don’t have to spend too much time on it.
No, it’s fine. This guy, Zach Frankel, who I didn’t know at the time, ended up recruiting me to Better as the CTO. The company had 10 people. For whatever reason, the founder, Vishal, was a very interesting guy—very smart, a little crazy in mostly positive ways. We got along very well. We started building it in 2015, and I spent 5 years growing the company. Eventually, it turned out that doing mortgages is pretty hard, and interest rates went up very quickly after the pandemic, which caused a lot of issues at the company. But by then, I’d already left.
That was around 2020 or 2021, when you left?
Yeah, something like that.
So you left before you had the idea for Modal, or did you leave and then start ideating?
It’s hard to say. I had a lot of random ideas when I left Better, and I wasn’t sure if I wanted to start a company. During the pandemic, I almost took a sabbatical. I had 2 kids—I still have 2 kids—but at that time, they were 2 and 4. I decided to hang out and take some time off. I was working part-time, hacking on certain things, and wasn’t sure; maybe I should do some consulting. I started building stuff and got really excited about the idea that eventually became Modal. Modal was, I think, the third thing I built that I got excited about. There were a few attempts before then that went nowhere. But Modal was the first idea where I thought, “This is kind of cool. I should do this. This could be a really cool platform.”
Were the other ideas also in a similar vein, like developer tools?
The first thing was some sort of web-scraping thing, which went nowhere. But in hindsight, maybe it could have gotten somewhere.
Was this before the ChatGPT launch?
Yeah, 2021.
Oh, okay.
ChatGPT was in late 2022, right? To rewind a little for context, when I worked at Spotify, I released an open-source framework called Luigi, which is for workflow orchestration. It’s in the same category as Airflow, Prefect, and Daxter—a somewhat dated category. People don’t talk about it as much, but for a while it got a lot of attention. Anyway, I never turned it into a business. A couple of years later, in 2021, when I was thinking about business ideas or things to work on, I came back to workflow orchestration. I started hacking on a bunch of ideas around running code in the cloud—basically, orchestrating code running in the cloud. What I realized was that a core puzzle piece isn’t just orchestrating the compute; it’s executing the compute. I realized that was actually a bigger problem to solve.
In a way, I left it on the shelf and said, “Actually, I’m going to go to this other thing over here, and I’ll get back to workflow scheduling in a little bit.” But I never got back to it, long story short. I ended up building Modal, which focused more on figuring out a way to make it easy to run code in the cloud. Initially, the focus was fairly general-purpose. It could be GPU code, AI and machine learning, or data stuff. We focused a lot on how to scale it up and make the developer experience awesome. We realized super early that, in order to do that, we’d have to go very deep in the stack and build our own file system and container runtime. We solved a lot of hard engineering challenges.
Then you mentioned ChatGPT. That wasn’t quite the killer app, but around that fall, another thing came out: Stable Diffusion. Stable Diffusion turned out to be a killer app for what we were building at the time. We were only 4 or 5 people then. We started seeing a lot of people come to us wanting to run a Stable Diffusion model, and that made us realize that maybe generative AI inference—
Mm.
—is actually a really good use case for us. And so—
How were you finding it?
I’m a compulsive tweeter. I was even more so then. Now I’m too distracted by building a company, but I always tweeted and blogged a lot, so I think I had a little bit of a following. In the early days, a lot of what drove the growth was me compulsively tweeting about Modal and what I was building, getting people excited, and writing long blog posts about where the cloud is going. I call it a bit of founder-led marketing. I always feel like you have to lean into what you’re good at. What I’m good at is, I guess, shitposting and being terminally online.
Yeah.
And so, you know, I think a lot of our growth is that we're just kind of good at social media and being online a lot.
Nice. How long until you had your first customer?
Also, did you raise early on?
We didn't raise for a while. I actually self-funded it for a while, even through the first few employees. In a way, that was kind of nice because it made me really frugal about money and made me want to see ROI. I wasn't taking a salary for a long time.
At some point, it became a problem because I needed the money. We didn't raise for a while, until we raised a seed round in early 2022. By then, I'd already been working on it for about a year. Aksha joined as the CTO and co-founder around August 2021.
But it took until late 2022 for us to see any meaningful revenue.
I guess that's when ChatGPT came out.
Yeah, and Stable Diffusion came out then.
Yeah.
In our case, that was the main thing that sort of changed things. We went from about $100 a month in November 2022 to about $2,000 a month the next month, and then, all of a sudden, $30,000 a month in January 2023. Finally, I felt this massive anxiety lift off my chest. I thought, “Maybe we're actually onto something. It feels like we're onto something.”
We grew very fast in 2023, and I guess we've been growing very fast since then.
That $30K-a-month moment—was that just a couple of big customers, or was it very diverse?
It was just one. I don't know if I've ever talked about who it was, or if they even know it, but I think they suspect. It was a company in New York called Mage. They were a Stable Diffusion company.
In the first quarter of 2023, I think they were 80% or 90% of our revenue. After that, though, our revenue started diversifying a little.
Yeah.
In late 2023, we got Suno as a big customer. They were a big chunk of our revenue for a while, too, but over time we've actually achieved really good revenue diversification.
Yeah, yeah. For the Stable Diffusion stuff, I remember using it for ComfyUI. I'm assuming a lot of people were using ComfyUI, running ComfyUI workflows on it.
Some people were, but I think the people using Modal tended to be doing lower-level, more complicated stuff. What they liked about Modal was that it's always been pretty hackable.
They deployed their own custom workflows with adapters, custom inference, and a lot of optimizations. All the work we'd done to make the developer experience really good also turned out to be great for running production workloads. Auto-scaling turns out to be a very similar problem: you want to scale containers up and down very fast. We just happened to be really well positioned to capture a lot of those workloads in the early days.
Did tools for auto-scaling not exist? Why did you have to build those yourselves?
1. Rebuilding AI Infrastructure
My thesis is that every new era of technology runs up against the limitations of its tools. You can look back at the web, the cloud, mobile, PCs, AI, or whatever. Often, tools do exist, but they aren't particularly good. You need to rethink a lot of those tools to reap the benefits and really turbocharge building for this new world.
You saw this with cloud: we had to invent things like Kubernetes and Docker. In a similar way, with generative AI, you could build things on the previous generation of tools, on top of Kubernetes and Docker. But those technologies were built for a very different kind of workload: steady backend workloads with an even, steady number of requests. They run at pretty low utilization because CPUs are usually cheap, so people don't worry as much about cost.
Now you're entering a new world with very expensive GPUs, where you want to manage capacity and utilization. You have very large models that you need to load over the network into CPU and GPU. A lot of those tools just weren't built for that.
We felt we had to rebuild a lot of it. I mentioned the file system, but over the last few years we've also built some really cool things around CPU snapshotting and GPU snapshotting. We built our own autoscaler, scheduler, and all kinds of capacity-management tools.
We feel like you have to do that because you can't build for this new generation of software by putting a wrapper on top of the old stuff. You need to start over a little bit from scratch.
So with the auto-scaling stuff, did you see any weird behaviors on the platform? Was there anything that broke while you were scaling?
Capacity has always been a challenge from time to time. How do you get hundreds, thousands, or tens of thousands of GPUs? That's always challenging.
We had to go multi-region very early, and in hindsight, that was kind of a fateful decision. We bit the bullet and built it. We always thought of our capacity management as one big pool over the entire world. We don't think of it as separate clusters running in each region. That was one of the very consequential decisions we made early on.
I don't know if there have been specific issues. We've obviously had bugs and outages.
Yeah.
As much as I would love for Modal to have 100% uptime, I think recently we've basically hit four or five nines. Throughout Modal's history, of course, we've run up against scaling issues and things like that.
Yeah.
Yeah, the GPU market—and, well, actually, before that.
Yeah.
So you guys use AWS, GCP, and all these cloud providers underneath.
Yeah.
Do you also buy any infrastructure, or no?
2. GPU Capacity Becomes Strategic
Let's talk a bit about cloud capacity, because I think it's changed a lot. In the first few years, we got almost all of our capacity from the hyperscalers: spot and on-demand capacity, mostly spot, across all the different regions. We initially used AWS, but also GCP, and over time, Azure.
We started renting some capacity from Oracle pretty early because we wanted A10s and H100s, and we got really good rates on those. Over time, we added a couple more clouds and started doing reservations. But until the end of last year, most of our capacity was still either spot or one-year reservations.
Spot means you're just renting it right there and then that's it?
Exactly, yeah. It's also preemptible, which we had to deal with. But it works in aggregate because you can aggregate capacity from hundreds of regions. That model worked really well until we started running up against capacity constraints again.
First of all, Modal started growing very fast, so we needed more capacity. But fundamentally, the cloud market also started changing a lot. In the last year, in order to get the capacity we need, we've had to work with a much larger set of clouds. We work with almost 25 clouds at this point.
In many cases, to get the capacity we want, we have to do three-plus-year deals and pay a meaningful amount of money up front. You're locking in pricing on a three-year term. So we spent the last 6 to 9 months rethinking how we get capacity.
We had to build a whole procurement desk. We have 4 guys working full-time, basically just procuring capacity, calling the clouds, doing deals, and vetting the capacity.
We’ve had to build a whole engineering team—basically 10 people—just managing bare-metal capacity from all over the world and dealing with network issues, bad drives, GPUs falling off the bus, and all these things. We don’t own capacity.
Hmm.
We rent all the capacity so far, typically on these three-year deals, and that’s a big decision. Should we buy, or should we keep renting? I’m not dogmatic about it. Renting is nice because there’s a certain amount of capital efficiency to it. If you’re buying, first of all, there’s the obvious operational complexity: you have to deal with vendors and suppliers, power, cables, and all kinds of stuff.
But there’s also financial complexity. How do you fund it? Should you raise equity? You can raise a billion dollars and go buy hardware, but that’s a very diluted way to do it. You can raise debt, but if you do, you typically want to have long-term revenue locked in to pay the interest. So that’s not trivial either.
We look at the economics, the operational complexity, and the risks. It’s a complex trade-off, and we’re open to buying. There are other constructs, too. To get the capacity we need for 2027, it’s going to take a lot of work, going down a lot of different avenues to lock it up.
Do you have to hire a lot of finance people to do all this stuff, or not really?
I’ve had to spend a lot of time on this, and we’ve had to spend a lot of time on this. Over time, I do think there’s a lot to consider. These are very expensive assets—the hardware. It’s kind of staggering. You’re starting to see AI investments become several percentage points of U.S. GDP, and interest rates in America have started shooting up in the last few weeks.
At this stage, you look at the neoclouds, and these are very sophisticated financial engineering companies. We look at that and at all the options, and it’s possible we might have to do similar things ourselves. I wouldn’t want to. I’d rather focus on being a very lean software company and doing the things we’re good at.
I don’t think it’s good in general for software companies to own hard assets. Should you own your office? I don’t think that’s necessarily the right solution for most companies. But, of course, in order to get the capacity we want, everything is on the table.
Yeah. I guess the supply is going to be constrained for a while, so you kind of have to do that. Do you agree, or do you think this is just a moment in time?
I don’t necessarily agree with that. NVIDIA is delivering several hundred thousand Blackwells a month. I don’t want to say the market is 100% efficient, but in general, if you’re willing to pay a high price, you can get GPU capacity. If you’re willing to pay more than everyone else—and, of course, Anthropic and OpenAI are buying up as much as they can—but fundamentally, there’s constantly supply coming onto the market.
I don’t necessarily think it’s a fully cornered resource. Prices are going up, and I do think it’s worth reflecting on the fact that the normal state of hardware is that prices always go down. It’s kind of like gravity. We’re in a weird period where hardware prices are going up, and it’s hard to say how long this is going to last. It could be another six months or another 36 months. It’s really hard to know.
I don’t know if it’s true that you need to buy capacity. To me, that’s a little like the great toilet paper shortage of the pandemic, or a run on the bank. That being said, capacity is going to be tight for a while, and prices are going up. Luckily, I think it’s not necessarily bad for us if prices go up, because scarcity also means we can charge more from our customers. The fact that we have a lot of reservations locked in at a rate means that, to some extent, we’re effectively long GPU prices already.
I don’t worry too much about prices going up. I worry more about prices collapsing. I don’t think that’s going to happen anytime soon, but I think it would be quite bad for a lot of the players in the market.
Why would that be bad?
I think it would be bad for anyone who’s locked in a lot of supply.
Yeah, because they paid way too much.
They paid too much.
Yeah.
There’s a risk of buying at the top. Again, I don’t worry about it that much. I don’t think it’s going to happen anytime soon. But, like I said, hardware prices tend to go down in the very long run, so it’s worth reflecting on that.
Right now, I think your capacity is constrained, and that leads to what, in economic terms, is called a scarcity rent. Who captures a scarcity rent? I think that primarily goes to two places: mostly NVIDIA, but also, to some extent, neoclouds. Neoclouds, and in general anyone owning capacity, have better economics. That’s undeniably true today, with capacity being constrained.
Long term, in the very long run, I don’t think that’s necessarily entirely true. There’s always going to be a little bit of a premium you can get from owning GPUs. But is it worth the risk of holding these very expensive assets on your balance sheet? I’m not so sure. We’re going to see. It’s going to take a few years to get through this, and in the long run, we might own some, rent some, do a combination, or do some other crazy stuff—joint ventures, who knows? Maybe some complex SPVs or whatever. Everything is on the table.
How do you compete with OpenAI and Anthropic for GPU capacity? I’m assuming they’d want to buy up literally everything that’s available.
They are, and I think a lot of the prices going up is a reflection of that. That being said, NVIDIA is shipping a lot of stuff. To some extent, I think NVIDIA prefers a little bit of diversification. They don’t want a small number of large buyers, much in the same way that you see hyperscalers getting a somewhat constrained allocation.
I think NVIDIA also prefers to see a lot of neoclouds and alternative clouds getting more allocation. This is more speculation, but I think smaller clouds that are a little riskier to work with are more of an opportunity for us. My feeling is that Anthropic and OpenAI prefer to buy from the tried and true. Part of it is also that they have to do a lot of training runs, so they need very large blocks.
But there’s plenty of capacity out there. You can get 32 nodes in Armenia. You can get 128 nodes in Indonesia. We love that kind of shit. We can go out there and get a lot of smaller capacity all over the world at pretty good prices.
Is it pretty much just that if someone offers you a certain number of GPUs, that’s all you care about? Do you care about how reliable it is? Is it just some janky warehouse where someone has these?
Sometimes—
You guys look at that?
It can feel like a janky warehouse. But we do have a spec sheet. When we talk to vendors, the first thing we do is send them a spec sheet. We say, “We need this number of IOPS. The CPU needs to have at least a certain PassMark score. We need to see minimum north-south bandwidth, egress, and ingress.”
So we do have a list of requirements. Often, we try to do an acceptance-testing period where we get one node and run some tests. Over time, to some extent, it also involves a little bit of a leap of faith.
Sometimes we sign very big commitments—tens or hundreds of millions of dollars—and start working with a vendor we still don’t fully know. Over time, we’ll see. I think there’s also a little bit of our own internal version of what SemiAnalysis has with ClusterMAX. I feel like we almost have our own Modal max score internally.
Mm-hmm.
I know which vendors I like more than others, and we prefer to do additional deals with vendors that have shown very good reliability. It’s not just reliability, either. It’s also whether they’re responsive when we have support issues, whether they fix the issues we ask them to fix, and how quickly they resolve things when they go down.
Do you agree that the demand for intelligence is basically infinite? If so, why wouldn’t you just buy up as much capacity as possible, and if you overbuy, sell it to somebody else?
It’s very easy to say, when there’s a scarce resource, “Let’s just buy as much as we can.” But first of all, if we bought it today, we couldn’t utilize all of it. We could utilize a lot of it, right?
But you can sell it off to somebody else who wants to use it, right?
The challenge is that if something happens in the market, a lot of things are correlated. If prices are going down and utilization is going down, the market for reselling capacity could vanish at the same time.
I’m extremely aggressive about buying capacity every day and raising money to buy more. Overall, I’m very bullish. But I want to point out that there are tail risks here, and when those tail risks happen, things could get very bad. Part of it may be that I’m just old and have seen this in the past. My first job was in 1999, so I saw the dot-com crisis, and I’ve seen the Great Financial Crisis. I want to build a company that’s around for another 20, 30, or 50 years.
We could certainly go out and buy an insane amount of capacity right now and try to corner the market. But do you know about the Hunt brothers?
No.
It’s a legendary story in trading. They tried to corner the silver market.
Mm.
It failed spectacularly. Cornering the market, in general, may not be a good idea. We are very aggressive, but we’re ramping up according to a plan that matches our demand growth and where we want to get to by the end of next year in terms of run rate.
Are you only buying NVIDIA stuff, or are you also looking at capacity from other types of companies, like Amazon Trainium?
We’re looking at it. I think alternative accelerators are starting to get very interesting. Ironically, every year for the last 5, maybe even 10 years, I’ve been saying, “Next year is going to be the year of non-NVIDIA accelerators,” and I’ve always been wrong. But I do think 2027 might finally be the year of non-NVIDIA accelerators. We’ll see if I’m wrong this year, too.
AMD GPUs are starting to get really good scores on certain things. TPUs are obviously used very widely internally at Google and seem to work really well. I don’t know much about Trainium. I don’t feel like I see quite as much, but the team is very good, and I know they’re working on a lot of the software stuff.
Hardware is hard in a certain way; software is easier. That makes me feel like if we fix the software layer for these accelerators, there’s going to be a lot of demand for them. With NVIDIA being such a bottleneck, we’ll probably see more alternative accelerators. That being said, there are bottlenecks further down the pipe, like TSMC and ASML. Maybe they’re an even deeper bottleneck to all of this.
Long term, monopolies don’t always last, so I think it’d be healthy for the market to have more alternatives. As much as I love NVIDIA—and to be clear, they’re obviously an amazing company with an amazing product—I do expect to see more diversity in the market, including things like Cerebras and Groq.
Yeah. What are your thoughts on some of those providers?
I think they’re pretty exciting.
Yeah.
You see them doing really well on certain workloads. There are people doing interesting hybrid, disaggregated inference setups where they run NVIDIA for the prefill and Cerebras for the decode, and things like that.
Oh, I hadn’t heard of that.
Yeah, and Groq, too. Certain things are compute-bound, and certain things are memory-bound, so you can mix and match. It starts to get very complex when you’re deploying these things in practice, but I do think there’s a lot of opportunity in these more complex hardware setups.
Are there any learnings you get from some of these neoclouds, like Nebius or whatever, and the way they’re doing things versus what you’re doing?
3. Neoclouds Finance the Buildout
One thing that’s very interesting with neoclouds is how they can expand so quickly. I think the craziest example is Endscale, which went from nowhere to having hundreds of billions of dollars in its pipeline despite barely having any capacity.
How can neoclouds grow so fast? The answer is that they go to Anthropic or Cohere or OpenAI, or some other big buyer, and say, “Hey, do you want to buy 5 years of chips? Sign this contract. We’ll give you a good rate.” Then they take that contract to Wall Street and say, “Look, we’ve got this big buyer. They’ve committed to $10 billion worth of chips. We just need the money right now.” Then they go to Supermicro or Dell or NVIDIA to get the hardware, install it, and deliver it to the customer.
It’s a very nice financial construct where you can essentially pre-sell things you haven’t even bought. That means you can scale up extremely fast. That’s what we’ve seen with these neoclouds: they can scale very well.
Modal has always had a bit more of a product-led growth focus, without the same ability to lock in long-term contracts. That makes it a little more challenging financially to scale that fast, even though I think we’ll get to the same point in the long run. It’s been fascinating to see how these neoclouds, especially CoreWeave and the other big ones, have been so savvy about raising money and thinking about GPUs as collateral for loans.
What’s the relevance of—I forget who it was, but some financial company was like, “Oh, we’re going to make…” Actually, I think it was NVIDIA. They were like, “We are going to turn GPUs into an asset class.” Is that just because they need to raise money and are trying to figure out where to get it from? What’s the relevance of that?
There are a bunch of startups—I don’t know if you’re thinking about those—including Silicon Data, Compute Exchange, and Ornn [?], that are basically trying to create price indices and trade derivatives. I think it’s interesting. There are a number of challenges with compute as an asset class. It’s definitely not as homogeneous as other asset classes. You have different networks, different locations, different disks and CPUs, all kinds of things, and different levels of reliability.
That being said, you can say the same thing about oil: we created relatively commoditized, tradable assets out of it. It remains to be seen. At the scale of AI data center and GPU deployments, I do think we need to be smarter about risk hedging, how we share risks, and having more insurance and things like that. I’m very happy that people are thinking a lot about this. That being said, I’m not entirely sure we’re going to see GPUs become a super-commoditized, tradable asset like Henry Hub or Brent crude.
Yeah. I want to shift the conversation to Modal Sandboxes.
I guess, before we do that, what are the different products that Modal has? The main product is obviously the GPUs, and then you have Modal Sandboxes. Is there anything else?
I think of the 2 big products we have as Modal Functions and Modal Sandboxes. We launched Modal Functions back in 2021. The idea was serverless functions that are very easy to deploy and scale, with us managing all the capacity. That turned out to be really good for inference and for managing GPUs and running things like that.
We never had a dedicated inference product or a GPU product per se. We never thought of Modal as a neocloud, and I think there’s a risk of getting sucked in that direction and becoming a neocloud—just selling a rectangular deal like, “Here’s a 3-year contract. You get this many GPUs.” Modal has always been very good at letting you deploy something and scale it up, with very cyclical or bursty workloads. We manage all the capacity, and the beautiful thing is that, because we have many thousands of customers, we can aggregate that capacity and get very smooth aggregate demand, which we can manage very efficiently.
Modal Functions was the core. We launched it in 2021, and, as I mentioned, we started seeing product-market fit in 2023. Now it’s about two-thirds of our revenue, and it powers a lot of things. In practice, it’s mostly inference, but you can use it for all kinds of other stuff.
We launched Modal Sandboxes in 2023, and it took a while to see product-market fit, similar to Functions. We launched it because we realized a lot of people we talked to were using LLMs to output code. Akshat and I were talking over lunch one day, and we thought, “Maybe they need a place to run that code.” We realized we already had a lot of the infrastructure to take code and run it, because we’d built the primitives for managing containers and isolation. We already had gVisor and isolation primitives.
We got excited about the idea, and Akshat wrote a prototype and launched it in basically 1 or 2 days. We saw very little growth. I don’t want to trivialize it—we did see some growth. Some early users picked it up and enjoyed it for certain things.
I mean, it was 2023, so.
Yeah. It was 2023, and some people picked it up. We had some early users who enjoyed it for certain things, but overall, it was an insignificant source of revenue going into the summer of 2023. Two years later, we even discussed, “Do we need this product? Should we shut it down?” Then it just started exploding.
I think what really happened in 2023 was that LLMs got really good at generating code.
In 2023?
2023.
I guess, yeah.
I feel like that was when they went from being able to refactor functions and write snippets to generating full apps. We started seeing a lot of growth from vibe-coding companies like Lovable and others. Then companies like Ramp started using Modal for background agents. We also started seeing tremendous growth from a number of Neo labs doing things like reinforcement learning on Modal Sandboxes.
Sandboxes is now about a third of our revenue. We consider it basically our second product. To me, that’s really exciting because, even in the early days of Modal, I always knew that Modal was about creating a whole toolkit of products. We always wanted to be almost like a cloud: you have storage and compute and a bunch of different ways to work with it. We always wanted to be a multi-product vendor, but that wasn’t necessarily true in practice until now.
Now it’s clear: we have Functions, we have Sandboxes, and we’re working on a bunch of other stuff, including training and batch. To me, that evolution into becoming a multi-product company has been amazing, because it was always so core to the vision. Even in the very early decks, I had a bunch of slides on this, but it wasn’t necessarily clear to the outside world until recently.
Do you have anything in the works that you think is going to be the next product? Any predictions about the market?
I’m very bullish on training. I think training is an underserved market. Obviously, there’s a lot of focus on very large training runs, but we actually see a lot of demand in the smaller and mid-sized training segment. I’ll be talking a little bit about that later this week.
Hmm.
We’re organizing a conference. Batch is a really interesting market. I think we see a lot of demand from customers. We had a customer the other day that needed to transcribe 5,000 years of audio. They had millions and millions of audio files, and we had a lot of capacity, so we let them do that. It took a few days. I think there are some really interesting use cases there.
I’m very bullish on low-latency stuff. Once we get speech-to-speech models working—the voice models—I expect that to be a very interesting market. Then, of course, there’s all this agentic stuff. In a way, I have mixed feelings about the word “agentic,” because I think all software will become agentic, so it’s almost like, what does it mean to be agentic? I think all software will become AI-powered over time.
Clearly, there’s a set of building blocks for doing more LLM-powered software that seems to be common across all the different use cases. The way I think about Modal is that we’re in the business of building building blocks. We want to give you amazing, generalizable, battle-tested, general-purpose, composable building blocks. I think there’s some really interesting stuff with agent use cases, too.
That’s interesting you bring up training, because I think on another podcast, maybe a year or 2 ago, you said you weren’t specifically focused on training. What changed in the market?
We’ve been building a training product for multiple years, on and off.
Oh, okay.
I wouldn’t say we’ve never been focused on it. It’s one of those things where, now, we’re 150 engineers, but a year or 2 ago we had a much smaller number of engineers. Just by virtue of being smaller, we had to focus a little more, so inference became our sole focus for many years.
Yeah.
But we’re bigger now. We have opportunities to place bets in other markets, too. I’m also really interested in continuing the list of things: post-training reinforcement learning. I think there’s exciting stuff around that. There are also many other areas, like storage, data warehousing, feature stores, real-time streaming pipelines, and all kinds of stuff.
The long-term view of Modal is basically: if you want to build some cool AI shit, we’ve got you covered. We’ve got all the building blocks. Come to us, and we’ll help you take these building blocks and compose them. That’s the beauty of software: you can build almost anything if you have the right building blocks.
That’s very much the sort of company and product I want to build.
When you say training, are you focused more on bigger enterprises doing huge training runs?
No.
Or are you— Okay.
No. They’re going to do these big hyperscaler—or rather, neocloud—mega-deals and get 100,000 GPUs or whatever. But there’s an underserved market, in my opinion, for researchers iterating or people training smaller models. There are text-to-speech and speech-to-text models. Computational biotech is a market where we see a lot of demand. We have a lot of customers in that space training relatively modest models. You can train on 64 GPUs or whatever.
Recommendation systems and a lot of traditional machine learning are other examples. There are a lot of things where you don’t need 100,000 GPUs. We’re not going to focus on that market for a while, but I think there’s a lot of stuff in these slightly smaller markets that’s relatively underserved. It’s very hard to set up training clusters.
Yeah. Have you noticed that, since GLM-5.2 or Kimi K3—or maybe Kimi 2.5—a lot of people are using those and fine-tuning them?
If anything, I’ve seen the other direction. A couple of years ago, the value proposition for fine-tuning was: open-source models aren’t quite as good, but if you take a small model and fine-tune it on your specific domain, you can reach the frontier in terms of quality and cost.
Today, I’d say the open-source models are so good that you can basically just swap them in. Instead of the frontier models, you can run Kimi K3, DeepSeek 4.1, or GLM-5.3, and they’re almost as good as the frontier models at a much lower cost. We still see a lot of demand for post-training, and I do think it’s a really interesting space. We’re working on some products in that space, both for SFT and reinforcement learning.
I’d also say fine-tuning is no longer something you have to do. It’s very optional. Open-source models work really well right off the shelf, too.
Back to sandboxes, actually. I forgot to ask you: what did you have to update about the sandboxes you were offering that existing sandboxes didn’t allow for?
4. Ephemeral Sandboxes for Agents
Not a whole lot. I think the thing we’ve always been very good at is that we’re infra geeks, right? We’re very good at scaling, reliability, and doing a lot of things. I think that explains to a large extent why we have, in my opinion—and I don’t have the numbers—by far the largest market share in this space. We’re just very good at running hundreds of thousands or even millions of sandboxes.
Yeah.
We’ve had to rewrite our systems a bunch of times to handle that scale, but we were always very good at it. The sandbox space in general isn’t the most differentiated space, but we’re working on some really cool stuff. Over time, we’ve had to add support for VMs, or virtual machines. We previously built our sandbox product on gVisor, but now we offer fully virtualized machines as well. For certain use cases, you need full Linux compatibility.
There are other things we’ve had to build better primitives for, like more interactive piping of data back and forth between sandboxes and working with file systems. We’ve had to spend a lot of time on snapshotting. I think a lot of the things that will differentiate us in the long run are really cool things like forking sandboxes, snapshotting, resuming, and live migrations.
In general, I think the novelty of sandboxes was always this sort of ephemerality: you can spin up sandboxes for a few seconds. That’s a string I want to pull more on. I think there’s a lot of really cool stuff you can do by decoupling the agent and the harness from sandboxes, having agents control many sandboxes, forking sandboxes, and snapshotting things. You could run very short-lived sandboxes, maybe one per command. I think there’s a lot of cool stuff there. We’ve already spent a lot of time on snapshots, but there’s a lot more to be built there as well.
You brought up snapshotting a couple of times. What do you see as the use case for snapshotting? What would that give us that we don’t get right now?
The big thing for me is that today, a lot of sandbox workloads are kind of lift-and-shift. People think about sandboxes almost like, “I have my dev box. Now I’m going to shove it in the cloud and have an agent control it. Maybe I’ll run the harness inside the sandbox.” I don’t know if that’s the right way to think about compute for agents in the long term.
I think a much better model would be to have these very long-running agents, which to me are kind of durable execution workflows. They’re going to run 24/7. Every once in a while, they’ll run commands, and you’ll have the ability to spin up a sandbox in 100 milliseconds, run a few commands, shut it down, and snapshot it. The faster you can get that wake-up, go-to-sleep cycle, the more you can create sandboxes that only run for a very brief period. That will make compute much more efficient and enable you to do some really cool things, like forking and moving things around.
Imagine Claude Code having a branching flow instead of a very linear, sequential one, and running hundreds of agents at the same time. I think a lot about CI as another illustrative example. For most of the history of CI, with GitHub Actions and things like that, you had a very sequential flow. You slowly install a bunch of stuff, then run the test suite, and then generate some reports or whatever. You can break it up into shards, but it’s kind of annoying.
What I’d love to do is take your entire test suite and run every single test in its own sandbox, and just parallelize the hell out of that. Run 1,000 sandboxes. Why can’t that be done? The problem is there’s too much overhead around starting and stopping sandboxes. The more you make starting and stopping frictionless, the more you can enter a world where you can run as much as you want in parallel, in extremely bursty, large-scale fashion. I’m an infra geek, like I said, so I get very excited about these things. Once we figure out all these primitives and the right building blocks, I think it’s going to unlock a lot of really cool stuff that we previously couldn’t do with software.
Why hasn’t that already been built? I would assume people would have wanted to parallelize as much of their CI as possible before, too.
You can, but the overhead is high. It’s like Amdahl’s law or whatever. In GitHub Actions, for instance, we parallelize the test suite in Modal. But if you split your test suite into 2 shards, each shard needs to do all the installs and the test setup.
Mm.
That takes a long time. It can take minutes. But let’s say you can do the test setup once, snapshot the container, clone it, and then run the test suites independently. A lot of the benefits can’t materialize until you get the friction and overhead down.
Mm.
The transaction cost.
I think you guys have a partnership with Anthropic, and they’re using sandboxes for something? Modal sandboxes?
Yeah. The managed agents with Anthropic have a bring-your-own-sandbox integration, so you can use Modal for those.
Oh, cool. How did that come to be?
I think Anthropic is trying to create a thriving ecosystem and community.
Mm.
And so I think they make money from inference, right?
Okay, yeah.
I think it’s very much in their interest to promote as much of an ecosystem as possible for everything else, as long as you use Anthropic for inference. I think it makes rational sense for them to partner with us as a sandbox provider.
Also, I feel like we have to bring up the Hugging Face incident. Are you okay talking about that?
Yeah, yeah.
Okay, cool.
It’s all there.
Could you briefly describe what happened? I think they were using the Modal sandbox, but it wasn’t a fault of the sandbox.
5. The Hugging Face Incident
Yeah, exactly. I forget all the details of the exploit, but I think what you’re referring to is the fact that Modal was involved.
Yeah.
Basically, the agents that exploited Hugging Face were running inside Modal. They discovered a vulnerability in an app that was deployed on Modal.
Mm.
They used that as a springboard for getting access—getting full internet access, as I understand it. I don’t remember all the details. To be clear, an app running on Modal was hacked. Modal itself was not hacked.
Yeah.
A lot of people run stuff on AWS. Occasionally, those apps get hacked. Does that mean AWS got hacked? No, of course not. A lot of people run apps on Modal, and one of those apps got hacked.
There are some lessons we can learn. One thing we’ve been reflecting on is whether we could have more sensible security restrictions by default. To use another analogy, it used to be very easy to accidentally make an S3 bucket public. AWS realized that while that might not pose a security problem for AWS itself, a lot of its customers would accidentally leak data. So now it’s much harder to make S3 buckets public.
In a similar way, we’re exploring whether we can make security a little more restricted by default without destroying the magic that makes Modal good. Separately, agents are getting really good at cybersecurity. As another lesson from this incident, we’ve started using agents internally quite a lot to try to find vulnerabilities. So far, they haven’t found anything, but they’re very good at probing deep parts of the system, trying to exploit things, and trying to escalate and get out of sandboxes. It’s something we’re exploring very actively right now.
How exactly do you get the agents to run through the codebase? Do you just say, “Hey, find security vulnerabilities”?
No, you just put them in a sandbox and tell them, “Can you try to exploit—”
Just run through the—
Yeah.
—the entire repo.
“Can you just try to escape from the sandbox?”
Yeah.
It’s like escape the room, but for agents.
Cool. When did you start doing that?
We started doing it a few months ago.
Okay.
It’s pretty exciting.
Like when Fable came out, I guess?
Yeah, roughly around the same time.
Okay.
My view on cybersecurity is that I think agents are very good at finding vulnerabilities, but you can also use them very well defensively. I think companies like Modal, and many other companies, are thinking about the same thing. If you talk to pretty much every CISO, I’m sure they’re all asking: How do we use agents defensively to fix all our vulnerabilities?
Over time, I’m not that nervous. If anything, agents and all the breakthroughs in cybersecurity will make the world safer.
Do you think all the big cybersecurity companies are just cooked?
I don’t know if that’s true, because you’re asking about something that’s very far outside my core expertise. When I think about cybersecurity, a lot of what you’re selling is effectively fear. That’s almost as much a branding and marketing strategy as it is about the core products. Brand and things like that are arguably even more valuable in the era of AI, so I wouldn’t necessarily jump to the conclusion that they’re cooked.
They have deep relationships with enterprise customers, and they have big sales teams that are very good at telling spooky stories. I think they’re going to continue to be very good at that.
I ask because you’re doing it yourself. You’re not going to one of the big cybersecurity companies, like Palo Alto Networks or whatever. But I guess only giant enterprises care about that. They can also blame somebody if something goes wrong.
That’s true.
Yeah.
You kind of want to hire an established vendor in many cases, in order to blame them. To some extent, it does reduce the need for us to bring in human pen testers, which most startups do on a regular cycle. I don’t actually know what our cycle was, but I think we used to do it roughly once every 6 months. Maybe now we only need to do it once every year. I don’t know.
With all of these things, humans can also use them to work more efficiently. I’m a believer that humans with AI are even more powerful now. Maybe there’s an argument that we can bring in pen testers who are really good at using agents and find vulnerabilities even faster. I don’t know. I’m an optimist. I started a company, so I’m generally optimistic about everything.
I certainly don’t think humans or companies are cooked. I think a lot of companies are going to figure out how to use AI to be even more efficient.
Mm-hmm. I want to transition a little bit to developer experience, because early on—and even now—you guys focus a lot on that.
Yeah.
You’ve talked a lot about that. Does developer experience even matter anymore?
6. Developer Experience for Agents
I think it still matters. I think the context you’re getting at is agents, right? Does developer experience—
Yeah. The docs are already there. The agents are going to look at the docs, and all that stuff.
To that point, almost all the things that make developer experience good also make agent experience good, which is what we’ve seen in practice. Things like having good docs, descriptive error messages, and fast feedback loops.
Certain things have changed. The other day, I was setting something up with Cloudflare. I find Cloudflare notoriously annoying to work with sometimes, and I threw an agent at it. It was able to configure a bunch of things in Pulumi for me.
Agents do make it a little easier to work with things that were harder to work with in the past. The human toil of dealing with YAML has arguably gone down a little, because you can just tell your agent to deal with it. But for the most part, I think the things that make developer experience good also make agent experience good.
Infrastructure over time becomes—and needs to be—more programmatic. That’s always been a thesis about sandboxes: things you previously had to configure using YAML should just be an SDK. It should be more imperative: launch a sandbox, do stuff with it, tear the sandbox down, rather than configuring it declaratively using some sort of DSL. That also pushes toward a slightly different interface that maybe didn’t exist before. SDKs are as relevant as they always were, if not more.
Is there anything you’ve changed in your product because most people are running agents on it?
We do. We actually run a lot of internal tests where we have agents...
We tell agents to build a mobile app and run it. A concrete thing that’s kind of funny is that we’ve actually seen how agent training data makes it very hard to deprecate things now. Agents often love using old versions of Modal syntax, and it’s hard to nudge them away from that. So we’ve had to be a little more conservative over the last year or so with how we deprecate or remove features. We have to be very careful to update every single example that’s out there online.
Yeah.
And, you know—
You have skills, right?
Yeah. There’s stuff like that, but there’s also an insane amount of Modal code online now that goes into the training data. I’ve seen that as an interesting example. Another thing is that, similar to search engine optimization, there’s agent engine optimization—or whatever people call it, agent optimization. We often run internal benchmarks where we ask an agent, “Can you deploy Stable Diffusion for us? Use whatever tool you want?”
Yeah.
We see whether it recommends Modal or not, and those things are very useful for us to track over time. We might ask it to spin up some sandboxes, or rank all the sandbox vendors. We spend a lot of time doing those things to make sure—
What do you do about that? You say, “Build this Stable Diffusion thing.” Does it recommend Modal or not? What do you do about that?
I think that’s harder. I think it’s like search engine optimization: primarily, it’s through content. You just have to put out a lot of content.
Interesting.
You can also get a sense of the perception. You can ask an agent, “Rank all the sandbox vendors by reliability,” or, “Rank all the sandbox vendors by price,” and see what it says. Agents do seem a little more prone to misunderstanding pricing and things like that. So we spend a lot of time making content clearer and more understandable for agents, because we know they’re indexing everything that’s out there—
Yeah.
—and that talks about Modal.
Have you thought about making a bunch of GitHub repos and having agents write a bunch of Modal code in there? Have you tried that?
Slop cannon. I think the truth is that’s probably something you can do. My guess is that language model training data today is quite insensitive to what’s AI-generated in the first place—
Yeah.
—and you can probably create a lot of slop. We’ve thought about it. I don’t love that idea, but in theory, much like SEO in the early days—and still to some extent—auto-generated content is a very effective strategy. I think it’s possible to do those things today, too.
Yeah. I want to switch to FDEs. Apparently, you guys hire forward deployed engineers. What do they do?
7. Scaling Modal’s Operating Model
A lot of them. We’re in a new era of building applications and things we couldn’t build just a few years ago. We see a lot of companies that need a little help, so we started investing in FDEs a couple of years ago. But it’s really this year that we’ve scaled it up a lot. We started the year with about 4 FDEs. Now I think we have about 24 or something like that, so we’ve grown the team a lot.
They do a lot of heavy customer engagement. It’s not exactly the Palantir model, but they work with a lot of customers on performance tuning, especially with LLMs. We see so much benefit from sitting down with customers, training draft models, and tweaking parameters in DLM and SGLang. They often help customers work through issues—not necessarily fix bugs, but figure out how to work around problems.
Another thing we use FDEs for is proactively helping customers who are over-provisioning on Modal. If they’re requesting too much CPU or memory, that can get expensive, so we try to catch those cases and sit down with them: “You’re spending a lot of money on Modal right now. We can help you save money.” With active deals, we also help customers migrate code from other platforms.
We never intended to become a service-heavy business. The goal for FDEs is, first, to be part of the sales process and help close deals. Second, to serve customers and provide some support. But I think the most important thing is to bridge gaps in the product that we later want to turn into real features. To a large extent, whenever an FDE does something, my question is: “Long term, how do we turn this into product so we don’t need an FDE for it?”
Yeah. Do you guys hire PMs?
Product managers?
Yeah.
No.
Would you say these FDEs are kind of pseudo-PMs?
It’s a little bit of a different function. Certainly, on the more customer-facing side, they are. But on the more internal side, there’s a separate part of product management that’s more about what the next building blocks are that Modal should invest in.
Then there’s something in between: looking at 10 or 20 different customers who are all asking for slightly different things, and figuring out what the patterns are and how we turn them into product. We do have people doing product with us, and that’s evolved over time. We’ve had people become product managers after previously being in DevRel or engineering, showing a strong aptitude for thinking strategically about what we have to build.
I’m not anti-product management. I do think a lot of companies in the past hired product managers too eagerly, and there’s a risk you end up with products full of widget-y things that eager product managers thought of and shipped. Doing product management well at an infrastructure company is very hard. It takes deep strategic thinking: What are the foundational building blocks? Where do we want to be in 1 to 2 years?
Building infrastructure is very hard, and we have to make sure we don’t end up with 35 features that look similar but all do slightly different things. That level of thinking is rare to find. For that reason, we haven’t aggressively tried to build up a product management team yet.
Yeah, that makes sense. How would you say your teams have been evolving since GenAI? You obviously worked at Spotify, where you had a certain team structure. Have your team structures evolved a lot now that we have coding agents?
In a funny way, yes. I do think team structures have changed a lot, but I’m not convinced GenAI drove that. In particular, I’ve seen a swing back toward being an IC, which I think is very positive for the space as a whole.
If you look back at the space 10 or 20 years ago, everyone wanted to be in management because that was how you got more money, recognition, and prestige. I think that shift started even before GenAI, but AI has heavily accelerated it. We make it very clear to everyone at Modal that there’s no difference in prestige between being a manager or an IC. Everyone is just a member of technical staff. Ultimately, it’s about your contributions.
We see a lot of former managers becoming ICs because they want to go back and write code. There’s been a shift away from nontechnical managers. Everyone—including me—still writes a lot of code at Modal. Akshat does, too, and pretty much everyone on the tech team.
A lot of people who aren’t on the tech team still write code. To me, that’s a recent phenomenon. I don’t know if it’s necessarily caused by GenAI, but I think GenAI has greatly accelerated it because it made the cost of writing code even lower.
If you have nontechnical people committing code, how do you review that? How do you guys do code reviews?
We do code reviews, but we think of them as optional. My view is that code reviews, in large part, are people trying to escape accountability for merging things into production. If you have that culture, it’s a sign that you need to invest much more in CD and CI, or CI in particular, but also CD—continuous integration and continuous deployment.
We do code reviews. When I write code, I occasionally ask the authors, “Is this in line with how you think about things? I hope I’m not changing the architecture fundamentally.” But we also make it clear internally that code reviews are quite optional. If you’re just changing docs or doing something basic, don’t wait for a human to look at it, especially in the era of AI. Why do you need a human to look at every piece of code? You don’t. But that does require you to have very good controls—
Yeah.
—internally around testing things.
Especially at an infra company.
Yeah, exactly. Infra companies need one more nine than the average startup in terms of uptime. These are investments you just have to make. You have to build systems that are very resilient. You have to have rate limits everywhere, decouple systems, make sure nothing is a single point of failure, and have extremely good test coverage.
Agents are actually really good at writing tests these days. A few years ago, I thought that was one of the major things missing in agents. I think writing good tests requires you to step out of what you’re trying to do for a second and play devil’s advocate. But now they’re quite good at writing tests, too. I think all these things help.
Do you guys use frontier models for coding, or have you experimented with the open-source ones?
We experiment with everything. Initially, it was a lot of Cursor, then Claude and Codex, and now a lot of people are using Nightshift, open-source tools, or other stacks. I don’t have a strong point of view. I want people to experiment and try different things. We use frontier models and open-source models.
We’re trying to do a little more dogfooding internally now that we’re running inference for a lot of open-source models at a very large scale. It’s also helpful to use our internal endpoints for a lot of agentic development, so we try to promote that.
Something I read maybe a year or two ago was that you didn’t think there was going to be a big on-prem business for you guys. Have you changed your opinion on that?
No, I haven’t changed my opinion. To me, Modal is a cloud maximalist. To take a step back, when the cloud came in 2008 or 2009, my first reaction was, “That’s crazy. Why would I want to take my code and run it on someone else’s computer? That’s insane.” Then, six months later, I was doing it. It took me six months to change my mind. It probably took enterprises another 15 to 20 years.
You look at vendors like Snowflake, AWS, and Datadog. The cloud is clearly winning, and multi-tenant systems are clearly winning. To me, on-prem looks increasingly like a legacy business model. I also think BYOC and VPC peering reflect an outdated security model: protecting everything inside our own network and using the network as a moat. I don’t think that’s true anymore. We’ve moved on to more robust security paradigms, like zero trust.
From our perspective, having full control over our entire infrastructure stack means we rent all the hardware, deploy and manage all the infrastructure, and monitor the health and metrics on all the hosts. That means we can iterate very fast and build a better product. That’s very valuable, versus customers in the past who had enterprise editions they had to deploy in customers’ environments. They had to think very carefully about release management and dependencies. We don’t care. If we want to introduce a new database, we can introduce a new database because we have control over it. We don’t need to think about what customers have access to. Multi-tenancy has so many benefits.
That being said, if we were to work with a very large Fortune 500 customer in the future and they asked for BYOC, we’ve done this in the past, so we’d give them a number. That number is very high, and no one has said yes to it so far, but who knows? At the right price, I would still build it.
Separately, BYOC might also be a way to work with and get capacity in certain ways. We sometimes work with neoclouds, and those neoclouds already have very large capacity reservations. The question is: How do we get them to run Modal when they already have capacity? BYOC might be a solution. We can still keep the control plane in our environment. The flip side is that we could maybe run other people’s stuff on that capacity. There’s some fun stuff you could do in theory, where you aggregate all that capacity into one big pool. We’re excited about some of those ideas. In general, I don’t know if we’re ever going to do on-prem, but BYOC, maybe.
So—
Bring your own cloud.
Yeah, yeah.
I keep saying BYOC. I should explain.
Yeah, it’s okay.
I should explain acronyms.
Let’s say hypothetically you were going to raise a shit ton more money soon. What would you use that money for?
As I mentioned, in this capacity situation, a lot of the money—the cash flow required—is for capacity prepayments. To me, that represents the best use of cash today. We’re ramping up capacity very aggressively. Maybe we move into other types of deployment models in the future, like ownership or whatever, but I think there’s always going to be a cash component, some sort of down payment or prepayment. That’s the greatest use of cash right now.
There are, of course, other things like salaries and office space, but in the grand scheme of our financials, that’s basically a rounding error right now.
Yeah. Interesting. Do you think that’s going to change?
No. The operating leverage we have in this business is kind of insane. We have 200 employees, and they cost a lot of money, but the GPUs cost a lot more money.
If you had to say what the top three things are that you or the business are focused on right now, what would you say?
Capacity, and the financial complexity—how we think about that space, pricing and packaging, and the tension between ownership and renting. Should we go in a more neocloud direction? Should we go in the opposite direction, the counterposition? That’s probably the biggest thing I think about.
The second biggest thing I think about is product strategy. I think a lot about what the next building blocks are.
If we do more agentic stuff, what are the things we need? Are there certain storage or compute primitives, or in general, what is the thing the world is going to wake up and realize they need over the next few years? Ideally, we have that at that point. We’ve been pretty good at anticipating demand and building products in advance of it, with the serverless functions and the sandboxes. So what’s the next thing?
The third thing I think a lot about is culture and company building. We recently crossed 200 people, and we’re growing extremely fast. We’ll probably hit 250 by the end of the year. When you’re growing very fast, you have 250 people who are all trying to figure out how they can add the most value to Modal. You have to make sure they have the right context and motivation, and that they’re excited to come into work every day because they get to build something incredibly cool and know what makes Modal successful and what they can do to contribute.
You have to align the incentives and make sure people have a very positive culture, where they have the implicit trust to do a lot of stuff, but also the accountability to deliver the best things and be held accountable if they don’t. To me, those are super critical things at this scale. When you have 10, 20, or 30 people, you can all just sit in a room, and it’s all implicit. You can sort of feel it. People can read me; they can tell me when I get a little annoyed, and they can tell when I’m happy.
But now we’re 250 people, and people don’t necessarily see that. Of course, there are a lot of new people who might be a little nervous to talk to the CEO. How do I remain approachable? There’s the Dunbar number, which I think is more fake than anything, at least the story behind it. But it is actually kind of real. There’s something around 150 to 200 where laying the foundation of the culture is very, very important.
How many people are you guys trying to hire over the next year?
We’ve grown about 4× on an annualized basis in the last 12 months. I think that’s an unsustainable growth rate, relatively speaking. You can double the team every year, and that’s not super hard to do. But at 4×, you start to see cultural drift. You start to see islands of different working styles, and people doing very speculative things.
That’s something I always watch out for in the engineering team. I’ll go talk to an engineer and ask, “What are you working on?” It turns out they’re building something kind of fringe, and I’m like, “Is this really what you should spend time on?” Growing very fast has that cost, and it has the cost of ramping people up.
I don’t know. I think by the end of the year, maybe 250. By the end of the next year, we’re going to keep growing, maybe 2.5× or 3×. So it’s very likely we’ll end up at around 700 people next year.
And I’m going to assume most of those are engineers.
Mostly engineers.
What kind of—
We’re ramping up sales a lot, too.
Okay. What kinds of engineers?
It’s primarily systems engineers. We’ve always been very deep into Linux, file systems, scalable systems, networking, cloud, and stuff like that. That’s the core of what we’re looking for: people who are really good at those things.
We’re also looking more at things further up the stack, because we’ve moved up a little bit. We’re doing a lot of work in the LLM inference space, so now we’re dealing with a separate set of concerns around CUDA, GPUs, and MFU—maxing out the tokens you can get per GPU—and a lot of other things.
Even though I’ll say that a lot of the work in that space, in my opinion, turns out to be more systems-driven. KV caching and routing are still very systems-like. These are very systems-heavy problems. But we’re doing a little more GPU performance work and that kind of stuff. We’re starting to build up a research team, and we’re very excited about moving into things around post-training, distillation, and better methods for quantization.
For your customers or internally?
Both. I think it’s a little bit like with the FDEs. Initially, we’ll do a lot of stuff with our customers to optimize their models, but over time we’ll figure out how to turn those into products. I think we’ll arguably fail in a certain way if every time a customer needs to optimize a model, we have to go and work with them.
Long term, it has to be turned into products. With the FDEs and the research team, initially you’re going to have to do a lot of work with customers, and we are that. But over time, we’re also building a lot of internal tooling and, eventually, internal products that are fully self-serve. We’re working on a lot of stuff right now in the post-training space, but also in GPU performance. A lot of cool stuff.
Back to the product strategy stuff that you were talking about.
Yeah.
What are your thoughts? What do we need over the next year or 2 years?
8. Building the Next AI Primitives
I think it’s incredibly hard to say. If I said I know exactly what we need to build, I think I would be delusional. What we’ve seen so far, and what I believe on a more general level, is that we need better building blocks.
This is a new era of software, and we’re unlocking a whole new type of applications that couldn’t have been built a few years ago. With all the generative AI stuff, code that LLMs output needs to run in an isolated way. There are so many new primitives needed to work with GPUs and LLMs. We’ve built some building blocks so far, and that’s unlocked a bunch of applications that the world has seen over the last few years—not just from us, of course.
Over the next few years, there are going to be so many more cool things. I mentioned some stuff: I’m very bullish on training, batch jobs, and certain things around agents and low latency. But in the end, we consider ourselves an infrastructure shop. We’re not trying to build very narrow things. We try to build very general-purpose building blocks and put them in front of customers. Then those customers will combine them and build incredible applications we could never have anticipated.
I want to go through some of your tweets.
Okay.
This will be the last part of it. Maybe you want to double-click on anything. “A lot of long-running persistent sandbox stuff right now. Today feels lift and shift. I suspect sandboxes will evolve from DevBox in the cloud to something more ephemeral: one environment for every command and snapshotting in between, so you can pause, resume, clone.” We talked about this a lot.
Yeah.
Yeah.
We talked a bit about it, right? I think the median sandbox in Modal today is 8 seconds long or something like that, which is kind of incredible when you think about it. Over time, I think it might get even shorter. We want to support long-running sandboxes, too. There are some very valid use cases around long-running sandboxes.
But my vision is that the world needs this, too. We need to figure out how we can massively parallelize things and reduce the overhead of starting and stopping sandboxes.
“If I ran an IPO-ready startup engaged in a CapEx arms race with a competitor, it would be nice to coordinate and pace the arms race a bit.”
No comment.
“The biggest downside of GPU scarcity is not that it’s hard to get GPUs, but that almost all capacity is sold as a 3-year rectangle starting 3 to 6 months in the future. That’s not what demand looks like for most users.” I think we talked about that a little bit as well.
Yeah. Maybe I’ll go a little deeper now. One thing I think Modal has always been very, very good at is fully on-demand capacity. If a customer comes to us and needs 100 GPUs, we can typically give them that immediately. You don’t even have to talk to a salesperson.
A lot of workloads in real life don’t look like these flat, rectangular workloads. They scale up and down throughout the 24-hour cycle; they may grow over time or have certain burst periods. To me, that’s a challenge for a lot of customers today. I do think it’s a legitimate problem for the space: the primary way to get capacity is to buy these rectangular blocks, and it’s not great. The world needs to evolve.
That’s fundamentally why I think Modal can play a very large role in offering more flexible capacity. We can do that because we can aggregate many thousands of customers, and demand gets very smooth in aggregate. That’s a lot of Modal’s value proposition.
Would you say that’s a differentiator?
I think so.
Oh, really? Okay.
Yeah, a massive differentiator. In fact, if you want to be very reductive, you can break down Modal’s value proposition into 2 things. One is the developer experience: the ability to iterate and build things without having to think about infrastructure. The other thing is the capacity.
Yeah.
The more I think about Modal and where we’re going in the long run, the more I think we’re aggregating the world’s compute capacity, and we’re very good at slicing it up very granularly because we’ve built all this infrastructure around scaling up and down extremely fast. There’s all this stuff around GPU snapshotting. We’re working on some ways to make it much, much faster. That model works really, really well at very large scale because you can aggregate lots of customers.
Every month I keep saying our sandbox revenue might plateau, but it keeps going up 30% every month.
It’s kind of nuts. I don’t know. Every month I’m like, “Well, sandboxes grew a lot,” but clearly they kind of plateaued, and then they keep growing even more. A lot of VCs ask, “How did you know sandboxes were going to be big?” The truth is, I didn’t. I thought it was going to be bigger than it was in 2023, for sure, but the fact that sandboxes are a third of our revenue today is insane. We’re obviously very happy about it, and we have a very large team investing in sandboxes.
I don’t want to trivialize it. I do want to give ourselves a lot of credit for seeing sandboxes for what they are today, a little bit back in 2023. But I probably would never have bet that revenue from sandboxes would be this large and that there would be so much market demand.
There’s an interesting structural reason why inference providers should exist separately from model builders. The best models keep changing a lot. Inference providers can easily reallocate resources between models, since overall demand is quite smooth.
If you think about all the Chinese labs and, in the future, hopefully more U.S. labs too, every time a new lab releases the latest and greatest open-source models, you see big shifts in demand for these models. If all these labs independently held capacity, they’d see demand go up a lot when they launch a new model, and then another lab launches a better model, and demand goes down.
Yeah.
Of course, in the background, they’re working on the next version. I think there’s a structural reason why you’re seeing demand shift to the latest and greatest Chinese lab models. It’s not as if everything just moves overnight, because a lot of models, prompts, and other things have a certain stickiness to them. But there’s an argument that it’s very easy for Modal to shift resources. Today we’re running X thousand GPUs for one model; tomorrow we need to shift 50% of that to another model. That’s what the system is built to handle.
Yeah.
No one even needs to tell us to do that. It just magically does it.
So you agree that LLMs are basically commodities? And if you agree, where do you think the business for the big labs is going to be?
I’m not sure that’s necessarily true. Is Taylor Swift a commodity? She releases a big album and goes on tour and makes a lot of money. Then there’s a down year when she goes and records—I don’t know if this is the best analogy—and then she releases a new album. She still has a lot of brand and power, and she makes a lot of margin. As long as the labs can repeatedly launch new models, and the margins they have on those models let them pay off and amortize the cost of training them, it could still be a decent business.
So you don’t think intelligence is plateauing?
No, I don’t think so. I think it remains to be seen. Models today are very good at doing things that can be trained very efficiently using reinforcement learning. We’ve seen that initially with coding and writing simple snippets. Over time, models have gotten really good at writing production-level code. They’ve gotten really good at math and cybersecurity. These are all quite verifiable things.
I still think it remains to be seen whether models are going to be really good at things like cooking food. I don’t know—things that have more diffuse targets and maybe longer feedback loops. My feeling is that humans will be quite good and quite competitive with models for a long time in those domains. But AI models are certainly getting very, very good.
Cool.
As I’m sure everyone can agree.
Yeah. The models are getting really good at coding. They’re already really good at coding. What other applications that aren’t as big right now do you think are going to become huge for LLMs over the next few years?
I don’t know, but I think anything that’s kind of verifiable—
Computational biology, I think you mentioned.
Yeah, computational biology. I think those aren’t exactly verifiable, but there are certain ways you can run simulations. You can use an LLM to output predictions, then use computational methods to simulate molecular dynamics and get feedback loops.
Material science might be another area where you have a bit of that loop: you can synthetically manufacture certain materials and measure their properties, and put an agent in the loop to do things. I have a physics background, so I’m pretty excited about that. Drugs, materials. What else? I’m drawing a blank, but I’m sure there are many other things.
Cool. Those are all the questions I had. Is there anything you wanted to talk about that I haven’t?
I’m good.
Cool.
This is great.
All right.
This is a lot of fun.
Awesome. Thank you.