Erik Bernhardsson on Creating Tools That Make AI Feel Effortless

Today I’m chatting with Erik Bernhardsson, founder and CEO of Modal. Modal developed a serverless cloud platform tailored for AI, machine learning, and data applications. Before that, Erik worked at Better.com and Spotify, where he led Spotify’s machine learning efforts and built the recommender system.

Well, Erik, thanks so much for joining me today on No Priors.

Yeah, thanks. It’s great to be here.

If I remember correctly, you worked at Spotify and helped build out their machine learning team and recommender system, and then were also at Better.com. What inspired you to start Modal, and what problem were you hoping to solve?

I started at Spotify a long time ago, in 2008, and spent 7 years there. I built a music recommendation system, and back then there was essentially nothing in terms of data infrastructure. Hadoop was the most modern thing, so I spent a lot of time building infrastructure.

In particular, I built a workflow scheduler called Luigi that basically no one uses today. I built a vector database called Annoy that, for a brief period, people used, but no one really uses today. I spent a lot of time building that kind of infrastructure.

Later, at Better, I was the CTO and thought a lot about developer productivity. During the pandemic, I took some time off and started hacking on things, and I realized I had always wanted to build better infrastructure for these types of applications—data, AI, and machine learning.

I pretty quickly realized this was what I wanted to do, and that was sort of the genesis of Modal.

That’s cool. How did that approach evolve, and what are the main areas that the company focuses on today?

I started looking into the challenges with data, AI, and machine learning infrastructure and thinking, from a developer productivity point of view, about what tool I would want to have. I realized a big challenge was that working with the cloud is arguably kind of annoying.

As much as I love the cloud for the power it gives me—and I’ve used the cloud since way back, around 2009—it’s actually pretty frustrating to work with. I had this idea in my head: What if you could make cloud development feel almost as good as local development? It would have these fast feedback loops.

I started thinking about how to build that and realized pretty quickly that we couldn’t really use Docker and Kubernetes. We were going to have to throw that out and probably build our own file system, which we did pretty early, as well as our own scheduler and container runtime.

The first 2 years of Modal were basically spent laying all that foundational infrastructure in place.

In terms of what you offer customers today, what are the main services or products?

We’re infrastructure as a service. On one side, we run a very large compute pool, with thousands of GPUs and CPUs, and make it very easy to get access to them. If you need 100 GPUs, we can typically get you that within seconds.

It’s one big, multitenant pool, which means capacity planning is something we solve for customers. They don’t really need to think about reservations. We always provide a lot of on-demand GPUs.

On the other side, there’s a Python SDK that makes it very easy to build applications. The idea is that you write code—basically, functions in Python—and then we take those functions and turn them into serverless functions in the cloud.

We handle all the containerization and infrastructure, so you don’t have to think about Kubernetes, Docker, and that kind of thing. The real killer app, as it turns out—we started this company before generative AI—was Stable Diffusion. A bunch of people came to us and said, “This looks kind of cool. You have GPU access, and it’s very easy. You don’t have to think about spinning up machines and provisioning them.”

Our first killer app was doing generative AI in a serverless way, with a focus on diffusion models. Now we have a lot more modalities. A lot of usage is still text-to-image, but we also see a lot of audio and music.

One example of a customer I think is super cool, building really amazing stuff, is Suno, which does AI-generated music. They run all their inference on Modal at very large scale. There are a lot of customers like that, building cool generative AI models, particularly in modalities such as audio, video, images, and music.

That’s cool. I think Suno is using a transformer backbone for a lot of its work now, versus a diffusion-model-based approach.

I think it’s a combination of both.

I think they talk about it publicly. That’s the only reason I mention it.

You wrote a post in October called “The Future of AI Needs More Flexible GPU Capacity.” In general, what I’ve heard in the industry is that a lot of the ways people use GPUs are reasonably wasteful. I’m curious about your view on flexibility around GPU use: how much is actually used versus wasted, and how much optimization is left, even with the existing types of GPUs that people are using today.

GPUs are expensive, right? I think that creates a paradox. A lot of cloud capacity can only be accessed by signing long-term commitments, which I don’t think is the right model for a lot of startups.

The amazing thing about the cloud, to me, was always that you have on-demand access to however many CPUs you need. But for GPUs, because of the scarcity over the last few years, the main way to get access has been to sign long-term contracts.

Fundamentally, I don’t think that’s how startups should do it. I understand it—it’s been a supply-and-demand issue—but looking at the CPU market, the fact that you have instant access to thousands of CPUs if you need them is how I’ve always thought GPUs should work as well.

That becomes especially important as we shift more toward inference. For training, it’s been less of an issue because you can generally make use of the training resources you need. But for inference, you don’t even know how much capacity you’ll need in advance. It’s very volatile.

A big challenge we solve for a lot of customers is that we’re fully usage-based. When you run things on Modal, we charge you only for the time the container is actually running. That’s a massive hassle for customers traditionally: doing the capacity planning and thinking about how many GPUs they need.

Then you either overprovision and pay for a lot of idle capacity, or you underprovision and, when you run into a capacity shortage, experience degradation in service. With Modal, we can handle these very bursty, unpredictable workloads well because we take all these user workloads and run them on a large pool of thousands of GPUs across many different customers.

One of the things that always struck me about training is that, to your point, you spin up a giant cluster and run a huge supercomputer for months in some cases, and then your output is a file.

Yeah.

That’s literally what you’ve generated. It’s kind of insane if you think about it. That file is, in some sense, a representation of the entire internet, or some corpus of human knowledge, or whatever.

Then, to your point, with inference, you need more flexibility in terms of spinning things up and down. Alternatively, if you’re doing shorter training runs or certain aspects of post-training, you may need more flexible capacity to deal with that.

Totally, and that’s something we’re really interested in right now. Traditionally, most of Modal has always been inference. That’s been our main use case. But we’re also really interested in training, particularly shorter, very bursty, experimental training runs—not the very large training runs, because I think that’s a very different market.

That’s a very interesting area we’re pursuing.

How do you think about meeting people’s end-to-end needs? I know there are a lot of other things that people do.

There's a lot of people using RAG to augment what they're doing. There's also a variety of different things that people are now doing at inference time in terms of using compute to take different approaches there. I'm curious how you think about the end-to-end stack of things that could be provided as infrastructure, and where Modal focuses or wants to focus.

Yeah, totally. Our goal has always been to build a platform and cover the end-to-end use case. It just turned out that inference was where we were well-positioned to focus as our first killer app. But my end goal has always been to make engineers more productive and focus on what I think is the high-code side of ML.

Our target audience tends to be more traditional ML engineers—people building their own models—but there are many different aspects of that. There's the data preprocessing, then there's the training, and then there's the inference. There's probably even more things, right? Having feedback loops where you gather data, online ranking models, and all these things.

My goal for Modal has always been to cover all of that stuff. It's interesting to see a lot of customers now—we don't have a training product, but a lot of customers use Modal for batch preprocessing. Maybe they’re training a video model. Maybe they have petabytes of video, so they use Modal, perhaps with GPUs, to do feature extraction. Then they train it elsewhere, and they come back to Modal for the inference.

For us to do training makes a lot of sense. In general, I think it makes a lot of sense to build a platform where you can handle the entire machine learning life cycle end to end, along with many other things related to that, such as data pipelines, nightly batch jobs, and all these things.

What you describe is a pretty broad, platform-based approach. I think there's a handful of companies that are in your general space or market. How do you feel that Modal differentiates from them?

I think, first of all, we're cloud-native. We're cloud maximalists. We went all in and said, "Basically, we're going to build a multitenant platform." The benefits of that are tremendous because we can do capacity management much better. That's one of the ways we can offer instantaneous access to hundreds of GPUs if you need to. You can do these very bursty things, and we just give you lots of GPUs.

I think the other benefit, or the other differentiation, is to be very general-purpose. We focus on what I think, as I mentioned, is the high-code side. We run custom code in our containers, in our infrastructure, which is a harder problem. Containerization and running user code in a safe way is a hard problem. Then there's dealing with container cold starts. As I mentioned, we have to build our own scheduler, our own container runtime, and our own file system to boot containers very quickly.

Unlike many other vendors, which are only focused on, say, inference or perhaps only LLMs, our approach has always been to build a very general-purpose platform. In the long run, I hope that manifestation will become more clear, because I think there are many other products we can build on top of this now that we have the compute layer becoming more and more mature.

When I talk to large enterprises about how they're thinking about adopting AI, many of them already have their data on Azure, GCP, or AWS. They're running their applications on those platforms. They've bought credits in the marketplace, and they want to spend their existing credits. They've already gone through security reviews. They've done a lot, and they worry about things like latency or pings out to other third-party services versus just running on their existing cloud provider, their hyperscaler, or their set of hyperscalers. Many of them actually work across multiple providers. How do you think about that in the context of Modal, in terms of your own compute versus hyperscalers versus the ability to run anywhere?

Yeah, totally. Of course, there's also the security and compliance aspect of this. It is a challenge. I look back at when the cloud came along, and I remember back in 2008 or 2009, when the cloud came along, my first reaction was, "How the hell? Why would anyone put their compute in someone else's computer and run that?" To me, that was insane. Why would anyone do that?

Over the next couple of years, I realized that it actually makes a lot of sense. I think now, even among enterprise companies, there's a recognition that our compute is probably safer in the big hyperscalers.

In a similar vein, I remember talking to Snowflake back in, say, 2012 or something like that. They had a similar approach where they basically said, "We're going to run databases in the cloud, and it's not going to be in your environment. We're in infrastructure as a service." I thought that was nuts. Obviously, Snowflake is now a very large, publicly traded company. I think they showed that software as a service makes a lot of sense.

I think there is a little bit of resistance to adopting this multitenant model, but when you look at security and adoption of the cloud, I think we have a lot of tailwinds blowing in our direction. I think security is moving away from a network layer into an application layer. Bandwidth costs are coming down, and there are a lot of tricks you can use to minimize bandwidth transfer costs.

You can store data in R2, for instance, which has zero egress fees. Realistically, we're going to have to push a lot to do that, but I think there are so many benefits of this multitenant model in terms of capacity management that, to me, it's very clearly a big part of the future of AI: running a big pool of compute and slicing it very dynamically.

You mentioned earlier that one of the things that really caused early adoption of Modal was Stable Diffusion and these open-source models around image generation. Are there any open-source projects or models that you're seeing become very popular in recent days or in the last couple of months that have really started taking off?

That's a good question. If anything, I think it's actually been a little bit of a shift toward more proprietary models—but proprietary open-source models, I guess. Most recently, Flux has been a model that's getting a lot of attention.

I'm personally very interested in audio. I think audio is very underexplored, and I think there's a lot of opportunity for open-source models in that space. But I don't think we've seen anything really cool yet.

What else do you think is missing in the world today in terms of AI infrastructure or infrastructure as a service?

I'm very biased, but I think Modal is missing a way for engineers to take code and run it. I'm very bullish on code, on people wanting to write code, and on people building things themselves.

Outside of the LLM space, which is a very different world, in my opinion, I think there are always going to be a lot of applications where people want to train their own models, run their own models, or at least run other models while having very custom workflows. I just don't think there's been a great way to do that. It's pretty painful. I think that's pretty exciting.

On the storage side, there are some other really exciting things. We haven't really touched storage at Modal; we focus very much on compute. I'm personally very interested in vector databases and how they're going to evolve. I don't think anyone really knows. I'm also interested in more efficient storage around training data.

I guess another thing I'm very fascinated by right now is training workloads. In order to train large models efficiently, you have to spend a lot of money and time setting up the networking. One of the things I'm really excited about is the possibility of making training less bandwidth-hungry, because I think that would actually change a lot of the infrastructure around training.

You could then tie together a lot of GPUs in different data centers and not have to have these very large data centers with InfiniBand and all that. That's another infrastructure development I'm looking forward to seeing more of.

There’s sometimes been a little bit of debate around vector databases, and you mentioned that you actually built one when you were at Spotify. I think Spotify today hit $100 billion in market cap. I think it’s one of the first European technology companies—

Yeah.

—to get there, which is pretty cool.

Yeah.

A lot of folks I know may use one of the existing vector databases or, in some cases, are just using Postgres with pgvector. How do you think about the need for vector databases as standalone pieces of infrastructure versus just adopting Postgres or doing something else?

Yeah. I feel like everyone’s debating that. I don’t necessarily know. I think there’s a case to be made that you can just stick everything into a relational database and you’re fine.

To me, the bigger question is, in the long run, what’s an AI-native data-storage solution? I don’t even know if it necessarily has the same form factor or the same interface as a database. That’s actually a bigger question that I’m more excited about. I think people look at vector databases and, whether they’re relational or not, sort of shoehorn them into this old-school model of: you put data in, you get data back. But I think there’s a lot of room to rethink that in the age of AI and have very different interaction models with that data. I know that sounds a little fluffy.

Mm-hmm. Yeah, it’s super interesting. Could you say more on that?

One thing I think a lot about is maybe the database itself could be the embedding engine, right? Instead of putting a vector in there and searching by that vector, I think the more native, AI-native storage solution would be that you put text in, you put video in, you put an image in, and then you can search by that. To me, that would be a more native, AI-native storage solution.

That’s one line of thought I’ve had: maybe we’re just so early to this that I think it’s going to take 5 or 10 years for it to really shake out.

Yeah, that’s really cool. I guess one other thing that you mentioned was that more people seem to be training their own models, at least in a lot of the areas that Modal works with. Do you think there’s any heuristic that people should follow in terms of when to train their own model versus use something off the shelf?

I think eventually, for any company where model quality really matters, unless you train your own model in the end, I feel like it’s going to be hard to defend that you have a better solution, because otherwise, what’s your moat? If you don’t have your own model, you need to find a moat somewhere else in the stack, and that might be possible to find. It might be somewhere else for a lot of companies.

But I think at least if you have your own model and that model clearly is better than anyone else’s, then that inherently is a moat in itself. I think it’s clearer outside of the LLM space, when people are building audio, video, and image models. If that is your core focus, it’s very clear to me that you kind of have to train your own models in that case.

Mm-hmm. Yeah. If I remember correctly, you were an IOI gold medalist.

Yeah, that’s right.

And obviously you think a lot about code and coding. How do you think that changes with AI over time? Or do you have any contrarian predictions about what happens there?

I don’t know if this is contrarian, but I actually think that this is just 1 out of many improvements in developer productivity. You look back at compilers, which were originally tools that made developers more productive, and then higher-level programming languages, databases, cloud, and all these things. I actually don’t know if AI is different from any of those changes in hindsight.

By the way, every time that’s happened, it turns out there’s so much latent demand for software that the number of software engineers goes up. I feel like if you look back at the last 40 years of software development, every decade engineers get 10 times more productive due to better frameworks, better tooling, or whatever. It turns out that just unlocks more latent demand for software engineers.

I’m very bullish on software engineers. I think it would take a lot to destroy that demand. People look at AI as a kind of fixed thing, but in my opinion, it’s like, no, it’s just going to unlock more latent demand for more things. I’m very bullish on software engineering.

Mm-hmm. And I guess the other field that you touched a long time ago was physics. I think you won a Swedish physics competition in high school.

Yeah.

I’m curious if you’ve followed any of the physics-based AI models or some of the simulation work. That’s an area that strikes me as very interesting, and—

The way you think about the models for it is different.

I did win the Swedish high school physics competition. I was a total mathlete nerd when I was in my teens.

Mm-hmm. Okay. Yeah, I think it’s a really fascinating area right now. It’s one of those areas where it seems like there’s some real reinvention needed and not as many people working on it. So it’s one of the areas I’m excited about, just in terms of there being lots and lots of different applications that you can start to come up with relative to it.

Yeah. Physics, in my opinion, looked very much like the golden era of physics—the ’20s, ’30s, and ’40s. I kind of feel like the field hasn’t really evolved much. So I don’t know. Maybe I would love for you to be right that there’s a resurgence of new physics-based models.

Yeah, I don’t know if it would necessarily help in the short run with basic research. I think it just helps with simulation. It kind of feels like physics as a field really doubled down on sort of the Ed, Edwin path of physics and maybe got a little bit lost there or something. I’m not sure. It’s kind of becoming—

Are you talking about more materials, like doing more compute-based methods for physics?

It’s like ANSYS or other companies where you simulate an airplane wing, or simulate load-bearing in a—

Oh, I see. So, like, HPC.

Correct. Yeah.

That’s always existed, right? Especially in oil and gas and stuff like that—

Yeah, exactly. Yeah.

But it’s a lot of small, bespoke, fine-tuned or hand-tuned models for specific things versus—

Yeah. I mean, meteorology is something I actually think deep learning should change, right?

Mm-hmm.

It sort of makes a lot of sense. Deep learning should be very good at predicting turbulence and things like that, because turbulence—

Yeah, totally.

—is actually very hard to solve in traditional physics models, right? So deep learning should, in theory, make a lot of sense.

Yeah, I think there have been a couple of papers on that out of NVIDIA, and then I think Google has a team that’s worked on it. So there are a couple of different weather-simulation teams that have started to publish some pretty interesting stuff—

Yeah.

It seems. Yeah.

Yeah. I would also point to an adjacent area: biotech. I think its computational methods have been enormously successful, right? If you look at protein folding in particular, but also other things like sequence alignment and things like that. That’s actually a field where we start to see a lot more usage of Modal as well. I feel like there’s a kind of resurgence of computational biology. It’s a really exciting field right now.

Oh, that’s really neat. Are there specific use cases that you see people engaging with most across your customer base relative to the sciences?

There’s a lot. I’m not a biology person, so this is kind of superficial, just looking at our customers. But one thing I’ve seen a lot is actually medical imaging, because my understanding is that with modern methods, you can do very automated experiments, get millions of them, and do automated electron-microscope imaging of that.

And so we've actually seen quite a lot of customers use Modal for processing those images and doing computer vision on them, which is kind of cool.

It's really cool. Is there any area that you're most excited about from a human impact perspective for some of these models?

With my background at Spotify, I think Suno is, to me, a very exciting thing. I think it's still very early in AI-generated music. You can still hear that it's not right. It's a sort of uncanny valley a little bit, but Suno is getting better and better with every generation of their model.

First of all, music in itself tends to be one of the first areas where you see real impact from new technologies, whether Spotify, iTunes, piracy, or gramophones going back, right? I always think music is an exciting area in that sense. It always shows the opportunity of new technologies.

I also think Suno is fundamentally something you couldn't have done before generative AI. That, to me, is really exciting. It's pushing the frontier and enabling a completely new product. There's no way Suno could have existed 5 years ago.

Mm-hmm. That's cool. Well, I think we covered a lot today. Thanks so much for joining me.

Yeah, thanks a lot. It was great.

Find us on Twitter at nopriorspod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.