[BidClub_]
Latent Space · · 69 min

The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin

swyxMatei ZahariaReynold Xin

YouTube ↗
TL;DR

  • Databricks’ open OmniGen agent layer is designed to make model and harness churn an interface problem, not a customer migration. Matei Zaharia traced it to engineers building workflows around Isaac, an internal wrapper for Claude Code and Codex, while Genie and other agents needed portable sessions, collaboration, history, search, security and spend controls. The common API accepts messages or files, streams text and tool calls, and can cancel turns; Matei’s open-source rationale is that an open layer with many integrations should outcompete a closed equivalent.

  • The core platform thesis is that useful agents need proprietary data in the right place, not merely another general-purpose model. Swyx articulated the view that traditional software may be rewritten by putting the data in place and adding capable agents; Matei agreed that “without the right data, you can’t really do that.” Databricks’ closing strategy is therefore to make data, governance and execution safely accessible to agents.

  • LTAP attacks the entrenched tax of moving transactional data into analytical systems through brittle CDC pipelines. Matei argued that Databricks can obtain roughly “99%” of HTAP’s benefit by unifying storage rather than query engines: idle storage-fleet CPUs transcode row-oriented Postgres pages into compressed columnar data that analytics can read immediately. The approach aims to avoid an intervening CDC pipeline while keeping investigations from overloading the production database.

  • Agent adoption creates a control-plane market because binary permissions cannot express enterprise risk. Zaharia’s contextual policies remember whether an agent installed a one-day-old NPM package or read 1,000 confidential documents, then allow, block or escalate later actions. The same session state can cap a task at $5 after a debugging job burned $500. Security is paramount, while cost exposure also compounds for consulting companies with 100,000 employees coding for customers.

  • Databricks cited operating scale both as infrastructure evidence and as a source of systems-design data. Matei said it launches 50–60 million virtual machines daily across three clouds and processes exabytes of data, while Neon launches roughly 13 million databases a day, with agents and branching contributing to that activity. Separately, Databricks described a new analytical engine whose sampled trace table contains about a quadrillion data points and whose rollout targets workloads finishing in tens of milliseconds.

  • Databricks’ competitive argument against Snowflake is path-dependent: “start open and start large.” Ion Stoica’s account emphasized Databricks’ origins in bulk ingest, machine learning and open formats, followed by faster serving, versus Snowflake’s narrower proprietary warehouse focus. For enterprises wary of decades of lock-in, an open foundation and a unified data-management plane can outweigh an individually optimized engine.

  • Mosaic’s role has shifted from general open models toward specialized models and compound agent systems. Ion Stoica said Databricks chose not to focus primarily on another frontier model. Zaharia’s document-understanding model returns structured JSON while being roughly 100× cheaper—and still better—than frontier-model parsing; Databricks also described open-model pipelines that generate training environments and beat Opus and GPT-5.5 on a task. The timing of mainstream self-customization remains uncertain, but stronger bases, better RL traces and synthetic data should keep lowering the barrier.

Digest · the substance, structured for research

1. OmniGen turns harness churn into infrastructure

  • Databricks’ Data + AI Summit had grown from a roughly 50-person Berkeley meetup where the founders taught Spark to about 100,000 people globally, including 30,000 in person. That scale contextualized the discussion of a new open agent layer, OmniGen.

  • Matei Zaharia traced OmniGen to two converging problems: Databricks engineers were building elaborate workflows and UIs around Isaac, an internal wrapper for Claude Code and Codex, while Genie, internal agents and customer agents needed to switch models and harnesses “every few months.”

  • His product requirement went beyond inference: “The agent is completely useless if you can’t share sessions with someone and have history and have search.” Coding agents and custom business agents looked different at the surface but needed the same delivery, collaboration, portability and control layer.

  • The common API treats an agent session as something that accepts messages or files, streams text and tool calls, and can cancel a turn. Databricks maps terminal-based Claude Code, Codex, Pi, the OpenAI SDK and other harnesses into that interface, absorbing upstream API changes while leaving users free to replace its UI.

  • The release was already attracting outside participation. The hosts cited roughly 400 merges shortly after launch; Matei guessed around half were from outside his team, including Kubernetes support, cloud-sandbox integrations and additional harnesses such as Cursor CLI and Antigravity.

  • Matei compared the layer to the modern data stack: ingest, transformation and visualization were useful abstractions, but customers eventually wanted fewer disconnected systems. He suggested a similar consolidation pressure may emerge around the many frameworks required to build a simple agent.

2. Persistent cloud sandboxes fix an awkward local workflow

  • Swyx’s origin story was visceral: while driving to a doctor’s appointment, he tethered his laptop to his phone and checked coding sessions at red lights so they would keep running. The productivity felt futuristic, but the operating model felt like “the dark ages of programming.”

  • His resulting wish list required more than remote chat: a persistent cloud sandbox, an interactive shell, file listing, log tailing and properly rendered Markdown. Coding environments may carry tens of gigabytes of source code, artifacts and installed libraries, so local persistence still matters even when storage and compute are otherwise separated.

  • Hosted sessions turn a personal prototype into team software. Engineers had built “five or six” similar internal agent frameworks, but without a server, authentication and security controls, they could not securely collaborate or connect the agents to protected company data.

3. Open protocols coordinate parties that cannot evolve together

  • Zaharia compared the agent interface to network protocols and data sharing. A retailer such as Walmart could expose a live inventory view to suppliers, which combine it with their own data and decide what to ship. When multiple parties move at different speeds, a stable interoperability layer is more durable than repeatedly building integrations on demand.

  • The same coordination problem appears among agents, users and tools. Vibe-coding can produce one-off software quickly, but it does not replace a protocol when independent systems need to work together over time.

  • Spark supplied the strategic precedent. Making machine learning, graph processing and connectors composable libraries let outsiders cover thousands of databases and formats that one team could never support, while every participant benefited from the shared substrate.

  • Databricks’ boundary is pragmatic: open-source layers where integrations and network effects compound, but sell operational services that cannot be delivered as code alone. Preventing a streaming job or the Lakebase database from losing data overnight requires an accountable operations team.

4. Stateful policy makes autonomy safer without constant approvals

  • Zaharia rejected simple allow-or-deny tool lists. An agent may legitimately read confidential documents, install NPM packages or publish to a company website, yet allowing all those actions in one compromised session creates an exfiltration path.

  • Contextual policies track the sequence and magnitude of actions. Installing a one-day-old package or reading 1,000 confidential documents can change whether later publication is allowed, blocked or escalated—making the system both more useful and more secure than static permissions.

  • Low-level APIs also need semantic translation. Databricks’ internal Google Drive MCP server exposes about 60 calls; reusable policy libraries can identify which ones share documents publicly, then let policies target those higher-level events. Zaharia connected this approach to Panther’s Python-based event processing.

  • Cost becomes another state variable. After an agent consumed $500 reading logs, Zaharia wanted to delegate the task with a $5 cap and an approval prompt for more. Databricks currently gives its engineers unlimited usage but analyzes traces for anomalies, efficiency and model strengths across languages such as Rust and TypeScript.

  • Asked what startups could build, Zaharia pointed to analytics for coding-agent usage, spend and quality: identifying weak tasks, missing skills and areas where teams should improve their agents. He described this as a likely management plane for agent development.

5. Agent workloads are changing the unit economics of compute

  • Matei said Databricks launches roughly 50–60 million virtual machines per day across three clouds and processes exabytes of data daily—often, he joked, before the listener has breakfast. He characterized Databricks as one of the largest CPU-compute orchestrators.

  • Neon now launches about 13 million databases daily, with agents and rapid branching driving much of that activity. Disposable databases and persistent coding sandboxes represent a different consumption pattern: autonomous software creates infrastructure repeatedly instead of treating each database as a carefully provisioned permanent asset.

6. LTAP removes the CDC bottleneck by unifying storage, not workloads

  • Matei divided databases into OLTP systems such as Postgres, MySQL and Oracle, optimized for row-level transactions, and analytical systems designed for aggregations, machine learning and scans. Running a sufficiently large analytical query on the production database can slow or crash the application it is meant to explain.

  • The standard answer is change data capture: read database deltas and reconstruct the state in an analytical platform. It is fundamental but brittle—schema changes break pipelines, engineers get paged at 3 a.m., and only “maybe two people” raised their hands when Matei asked a keynote audience who loved their CDC setup.

  • Earlier HTAP systems pursued one engine for both worlds, but Matei argued that they sacrificed the Postgres and Spark ecosystems while struggling to match either side’s performance. “It ends up actually sucking at both.”

  • LTAP is presented as “HTAP done right”: preserve separate transactional and analytical engines but give them one storage layer. Matei argued that this could deliver roughly 99% of the desired benefit—fresh data available for reasoning immediately—without forcing one query engine to serve every workload.

  • The discussion also emphasized that analytics can run on a separate machine fleet, so an investigation does not overload the live transactional database that is still serving users.

7. Idle storage CPUs made row-to-column conversion effectively free

  • The enabling step came from the Lakebase and Neon-style separation of storage and compute. Instead of persisting Postgres pages in the open data lake, the team explored writing column-oriented data that analytical engines could consume directly.

  • Meetings remained stuck on first-principles debate until an engineer built a prototype. The storage fleet had idle CPUs available for row-to-column transcoding; the resulting data compressed better, so services could write less data to S3 or another object store. Matei’s conclusion was that there was “no overhead” and “no compromise in performance.”

  • Matei initially did not believe the agent positioning his team had written. An Australian customer changed his mind: service telemetry could reveal SLA dips, but agents needed the underlying orders and users to explain them. Joining operational context could make those investigators “10 times more powerful”—“now I’m actually sold on our own message.”

8. Prototypes and named customers settle strategic debates

  • The LTAP breakthrough had no grand kickoff or formal design document: “Somebody actually went ahead and just tried to prototype it, and it worked.” The conversation drew the same lesson for OmniGen—working software and real users expose constraints that an abstract requirements document cannot.

  • Matei described a culture of incremental launches and tight customer contact. His first question to a new product team is: Who is the target customer, are you on a first-name basis, and are you texting them? Many products are built in weeks and revised through that feedback loop.

  • Matei recalled an early Delta Lake customer whose event-rate and freshness requirements were far beyond the company’s existing workload. The discussion placed this roughly four years into the company, around 2017–2018. Once engineer Michael Armbrust made it work for that customer, it worked for others.

  • Ali Ghodsi said Clean Rooms was initially built for only two customers. His argument was that the downside of overfitting to one or two customers is smaller than the risk of “boiling the ocean” and ending with none; he also cautioned that some customers should eventually be fired.

9. Enterprise adoption requires a different product instinct

  • Ali Ghodsi emphasized that older enterprises bring regulated data, privacy obligations, legacy systems and procurement processes involving many more stakeholders. A developer tool optimized solely for pristine technology companies may struggle to scale into the larger market.

  • Technology companies often answer a pitch with “I’ll just build that myself.” Retailers and other enterprises may explicitly refuse to make streaming reliability their business; they do not want production down because “some weird nerd” could not keep pipelines working.

  • Those customers still possess deep knowledge—aircraft engineering, medicine or clinical data—but want a bridge to it rather than a database education. Their desired interface is essentially: “I have this giant matrix with my clinical data; how do I cluster it?” They “never want to know” the infrastructure beneath the answer.

10. The new analytical engine is a factory trained on real workloads

  • Ali Ghodsi argued that nearly every analytical engine with meaningful traction is about a decade old. Each began with a narrow use case, accumulated expedient extensions and became “a gigantic pile” of compromises; rewriting it invites the classic second-system syndrome and a conventional five-year maturation cycle.

  • Instead of selecting algorithms mainly from papers, Databricks built “a factory for building databases.” A machine-learning model—not an LLM—learns from a sampled trace table containing about a quadrillion data points and predicts how candidate implementations will perform across real query shapes.

  • Relevant features include scale, throughput, latency, data distribution, sparsity, reuse and distinct-value counts. Even ASCII versus Unicode matters: if a string dimension has only 256 dense possibilities, an array lookup can replace a hash table.

  • The model informs both engineering priorities and runtime dispatch, because the algorithm optimized for tens-of-milliseconds latency may be wrong for scanning petabytes. The rollout begins with a new endpoint and incremental capabilities, avoiding a five-year hard cut while ultimately aiming to cover everything the prior engine did.

11. Open storage matters more than one universal query engine

  • Ion Stoica explicitly rejected collapsing every query workload into a single HTAP engine. The LTAP thesis collapses storage while retaining specialized execution layers; agents can already write both PostgreSQL and Spark SQL, so the human desire for one language is no longer a blocking architectural requirement.

  • Asked why Databricks outpaced Snowflake despite both embracing cloud architecture and separated storage and compute, Stoica named two differences: openness and AI. Databricks began with Parquet and evolved through open formats, while its platform was pitched as machine learning plus data well before ChatGPT.

  • Stoica described Databricks as starting with bulk ingestion, JSON logs, large-scale batch processing and low-cost open storage, while Snowflake focused on a smaller, valuable subset of business data for fast serving. The companies once partnered until customers asked Databricks, “Why do I need this other thing? Why can’t I just query your tables?”

  • His summary was “start open and start large.” Enterprises that have survived 30 years of vendor lock-in want an open foundation and ideally one way to manage corporate data. Stoica noted that this preference was controversial five or six years earlier, when a Snowflake co-founder argued for “Choosing Open Wisely.”

12. Specialized models convert proprietary context into advantage

  • Ion Stoica said Mosaic initially became known for early open LLMs, and Databricks later released DBRX above the Llama 3 scale. Databricks nevertheless chose not to compete primarily by pouring compute into another general frontier model; Genie instead applies external or customized models as a “virtual data scientist” that understands company data and relevant tools.

  • Specialized volume workloads still justify training. The discussion cited a document-vision model that converts PDF or Word pages into structured JSON, performs better than frontier-model parsing and is roughly 100× cheaper—avoiding the huge token bill and near-misses associated with sending page images to Claude Opus or another frontier model.

  • Databricks is also developing specialized sub-agents for coding workflows. Stoica said customization should become easier as stronger base models generate better RL traces and synthetic environments. Databricks has pipelines where an open-source model generates its own training environment, trains itself and beats Opus and GPT-5.5 on a task; the unresolved question is when that moves from expert researchers to ordinary users describing a job.

  • Databricks currently serves both customers who want AI Runtime—on-demand GPUs, libraries, data movement and monitoring—and those needing help building evaluations and synthetic data. The closing thesis ties everything together: models already reason well enough that much traditional software may become an agent over correctly placed data, but “without the right data, you can’t really do that.”

Matei Zaharia

One of the theses we have is that once you can get the data in the right place, the AI models are becoming pretty good. The generic agents are fairly good—I mean, Ali talked about AGI already here. They have pretty good reasoning capabilities. Actually, I think much of the traditional software will be rewritten with this new paradigm, which is just: get the data to be there, and then let's slap some AGI on top. Magic will come out.

swyx

Yeah.

Matei Zaharia

But without the right data, you can't really do that.

swyx

Matei Zaharia from Databricks, welcome to Latent Space.

Thanks for having us.

Alessio Fanelli

Yeah, thanks so much.

swyx

Thanks for taking time out. You have your Databricks Data + AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 people.

Matei Zaharia

Yeah, it was a little meetup at Berkeley, I think. We put together tutorials and taught people Spark.

Alessio Fanelli

Yeah. Obviously, now it's—I think the headline number is 100,000 people around the world, with 30,000 in person. It's a crazy community. I just saw the keynote. [laughter]

swyx

Ali is just—did you know that it was obvious back then that Ali would be such a great CEO? He's a great presenter. What do you think?

Matei Zaharia

I mean, I think among our group of founders, it was clear that he'd be the best at this, and it turned out great. He's ramped up on so many topics while running a company. He would just go in and study a topic and become as knowledgeable as all the experts. Even if you can't hire the person, you learn enough about finance and sales and whatever it was, and go from there.

Alessio Fanelli

I mean, he's obviously very high IQ and has very high EQ, but Ali today is quite different from Ali 10 years ago. [laughter] I think there's a lot of work that he put in to get to this point.

Matei Zaharia

Yeah. To me, the most appealing thing about him is that he's funny.

swyx

It's true. It's hard to make jokes about data, serious topics, security, and what have you.

Alessio Fanelli

Oh yeah, that's for sure.

swyx

You guys launched a whole bunch of things. I'll just name-check the stuff, because we're not going to cover everything: OmniGen, your baby; LTAP, your baby; your Dream Engine. We're also going to cover Genie and Customer 360. You acquired Panther, there's Delta Sharing, and there's Unity AI Gateway. A lot of these are things that you would expect Databricks to do. It's part of the roadmap, and everyone in your category has similar things. But I think the 2 of you are leading the 2 most unique and differentiated initiatives in the landscape. Maybe we'll start with OmniGen, and then we'll go into it. I do think that a lot of people are exploring this sort of meta-harness concept. What led you to it?

Matei Zaharia

There were actually a couple of converging lines, which I think is a good sign that you need something new. On the one hand, there's all the coding agents. Internally, we have a really great developer infrastructure team. They built something called Isaac that's basically a wrapper on Claude Code and Codex, and lets you use them either on the web in sandboxes, or just on your dev machine, laptop, or whatever.

They were adding all kinds of stuff there, and we saw that the more advanced engineers were building their own workflows with tons of agents. They were building their own UIs and stuff on top of that. The other line was us building agents. We shipped this data science agent called Genie on the research team, which I co-lead. We also build a lot of internal ones for various things, and then we have all the customer ones.

All of them were running into this issue of, “I need to switch models and harnesses every few months.” Plus, the agent is completely useless if you can't share sessions with someone and have history and search—all this layer on top of it for collaboration. I thought about it from both contexts, and at first people thought it was weird: Why are you doing coding agents and custom agents in the same thing? But I said it's basically the same problem. You just want to build the stuff that lets you deliver the agent, maybe control it if you care about security, and make it portable across things.

We prototyped some things as experiments. We said, “Yeah, actually, we can make it work,” and then we built it for real.

Alessio Fanelli

I'm wondering if this kind of architecture maps to anything in your careers in the past. I always think about how a lot of things tie back to operating systems.

swyx

A lot of operating systems tie back to databases, or the other way around. [laughter]

Matei Zaharia

The thing I do think it ties a lot to is network protocols, like the Internet Protocol. We also did stuff with data sharing, which most viewers probably won't know unless they—

swyx

Yeah, Delta Sharing is an open protocol for sharing.

Delta Sharing. Yeah, so you have a company maintaining some kind of table. Let's say Walmart or something: They have the inventory and what's been sold in each store. They also have suppliers, and the suppliers would love to produce more things and ship them exactly when you need them.

Why can't you share a view of that table in real time with them? Then they query it, join it with their data, and decide what to send. It's one of these things where you might ask: Today, since we can vibe-code anything so fast, why do we even need to design protocols or APIs or software? Why can't you just vibe-code things on demand?

Actually, for this type of interoperability, where multiple parties are moving at different speeds and building stuff, and you still want some layer on top to coordinate, you do want to design it and build it. It reminds me of agents talking to each other, and users talking to agents and tools.

Alessio Fanelli

Do we know of any other comments or alternative viewpoints?

swyx

I think, by the way, we had a debate on exactly this. We said the benefits matter a lot, and around the time we decided to do this thing, I was telling Matei, “It just happened to be a particular week when I was coding nonstop from the moment I woke up until the moment I went to bed. I was looking at my Claude sessions and my Codex sessions, and one of the things that was particularly annoying was having to keep my laptop open.”

I was actually driving to a doctor's appointment, and I remember wanting to make sure the whole thing continued working. It's so comforting to hear you say that, because I'm like, “I don't know if I'm a clown and I'm doing this.” Honestly, I was driving and tethering my laptop to my phone, keeping it on the side, and whenever I hit a red light, I started looking at what was going on on my laptop. I just felt that was ridiculous.

Yeah.

swyx

It felt like we went back to the dark ages of programming.

Matei Zaharia

I mean, the productivity you gain from all these coding agents is amazing, but have you heard of the cloud? [laughter]

swyx

It was crazy to me.

Alessio Fanelli

Was it the thing we were working on—the sandboxes—or was this before that?

swyx

It was a sandbox.

Okay, so you were—

swyx

I was approaching it from a very different angle. I wanted to say, “Hey, we're going to have cloud sandboxes that don't shut down. You can get one very quickly, but not just for running agentic sessions. It's also for running development.”

I was personally building that that week, and through building it, I ran into all these issues. Then I wrote a document for my case: “Here's my wish list of what the actual environment should do.” I think he ended up implementing almost every single one of them.

Matei Zaharia

Yeah, I remember Reynold saying—because my first prototype of this had just chats with your agent—“I have to be able to open a shell, like my own shell, and list files and tail them.” So I was—

Alessio Fanelli

Is this an SSH into a mainframe?

Matei Zaharia

Yeah, actually, it has that. [laughter]

swyx

Tailing my logs.

Yeah, yeah.

swyx

Another thing I think I asked for was—I still use Cursor for the sole purpose of rendering Markdown files.

swyx

Uh-huh. Yes.

Matei Zaharia

“Give me a way to see my Markdown files and render them properly. I don't need a separate tool anymore.” Yeah, I think you also built that in.

We did that. We had a lot of engineers building their own vibe-coding setups. But the other thing they all said was, “Hey, I built something that's amazing for me, but no one else on the team can use that because I don't have a server to collaborate.”

This is why we tried to set up OmniGen, so you can have a server and have the security set up in there. You can log in with Google or whatever and actually securely share stuff. That's why we've seen a lot of other agents hit things like this: people think they prototyped an awesome agent, but it's not allowed to connect to some really important data or whatever because of the security team.

swyx

At this point, for those watching along on YouTube, we're going to bring up an image of the structure here, and we can talk through a little bit of the architecture. I just want people to understand because, when we're talking about software, it can be very abstract. Here's actually what we're talking about.

You've worked out this entire platform in open source, basically. There's a runner component and a server component with a sort of uniform API that you've figured out, along with all sorts of other elements. Obviously, you can plug in all these persistence layers and compute layers. This is a whole cloud. This is Agent Cloud.

Yeah, it's got these components to work with it. A lot of the action happens on the machine where you deploy your agent to, so whatever you've got on there, you can run. But yeah, it's sort of the minimal thing you want to have hosted collaborative agents and to have that server.

One of the reasons we open-sourced it is that, for anyone building agents, this gives them an app they can start with and customize. We were seeing in Databricks, too, that someone would make a nice agent app, and then other teams would ask, “Can I just use yours for my agent?”

I think we had 5 or 6 different agentic frameworks built by every different team. They all do more or less the same thing.

Shawn Wang

Yeah, people want to take something that works and fork it, and you might as well have something open source. That was another question, which is interesting for Databricks: What do you choose to open source, and what do you choose to make proprietary? This goes back to Spark, right?

Matei Zaharia

Yeah. [laughter] One of the reasons to open source something is if you think it's a layer that will actually have some network effect. It'll benefit from many people collaborating on it.

For example, with Spark, I don't know if you know that, when Spark came out, we also focused a lot on letting you have libraries on top. They used to be different distributed computing engines for machine learning and graph computation. We said they should all be libraries that you can compose, and we made it super easy to add connectors to data sources, too.

We benefit because we don't have the time to write connectors to 1,000 different databases and file formats. We can just use the ones people make, and of course they benefit from joining this thing.

Another way to think about it is, I can imagine that if our thing wasn't open, we'd have some kind of agent-hosting thing, but it wouldn't be open, and then there's an open one. Which one's going to win in the long run? Here, because there is this benefit from people writing integrations, it'll be that.

Then there are other things that you just can't even deliver as open source that are things the company does. For example, how do you make sure your streaming jobs or your Lakebase database doesn't lose all your data at night? That requires an operational team that's going to sit there. There's no way around it; it has to be a service.

We want to make sure that, as a company, we're really good at those infrastructure services, and then we're as open as we can be in terms of what you build on top.

Shawn Wang

I mean, speaking from a benefits perspective, I think we're already seeing pull requests and ecosystem integrations, even though it was only released on Saturday.

Matei Zaharia

Yeah, Saturday. Someone—

Shawn Wang

Let's see what's going on. Yeah.

Matei Zaharia

You can look at the merge lines. I actually asked some legend this morning about the—

Shawn Wang

400 merges already?

Matei Zaharia

I would guess around half are not from my team. For example, someone added support for running it on Kubernetes. People added many cloud sandboxes. This can launch a cloud sandbox and run your agent in there, which is great for sharing, too, because it's not on your laptop with someone running sketchy code on it.

Many startups have put those in, and we expect to see more of them. We also have more agent harnesses already: Cursor, CLI, and Antigravity.

Shawn Wang

That's all beautiful. I feel like the last time this happened, there was the rise of the modern data stack. I don't know if it was that useful. I'm actually kind of curious about your postmortem. I think most people will agree that it is finally dead, but maybe this gives rise to a new modern AI stack that does the same thing. [laughter]

Matei Zaharia

I mean, I think the modern data stack was a pretty useful thing, probably even up until this day. For the audience who don't actually understand the history, I think the modern data stack effectively decomposed into this: You need a layer to ingest the data. You need a layer to transform your data. All of this is run, and then you need a layer to maybe visualize your data. All of this runs on some sort of data warehouse or, later on, as we're doing, a lakehouse.

I think those concepts are all very powerful and very useful. They enabled a lot of workloads. What people eventually run into is kind of a question of unification and consolidation. It's, “Hey, do you really need to chop all of this into different pieces and work with so many different vendors and platforms in order to get a very simple visualization done?”

Over time, everybody started realizing that customers are pushing us. We started to realize that, so we started building more and more capabilities and trying to consolidate. At the end of the day, customers don't have to worry about hooking up 5 different systems in order to produce a chart.

But I think, honestly, something like this is probably happening with how many different frameworks you want to hook up together in order to do a very simple agent.

Shawn Wang

Just to be clear, I would say the core of this is this common API on top of all the harnesses. The API is basically like: you've got an agent session, and you can send in a message or a file. Basically, that's what you can send in, and then you get out these streams as it's streaming text or as it's doing tool calls.

The other thing you can send in is a command to cancel a turn. So that's the API. What we did is we could get you that on top of Claude Code running in a terminal, Codex, Pi, the OpenAI SDK, all that stuff. We map them all to that same interface.

That's something that you'd have to maintain yourself if you built your own agent orchestrator. Whenever Claude changes its API, you have to tweak your thing, or it's going to lose some messages. So that's the thing that's valuable to maintain.

Then on top of that, we built a few apps. I think we built a pretty cool UI and stuff, and we built the security and control piece, which I'm excited about. But it's that common interface. It doesn't try to be a stack.

In fact, you could plug in your own UI on top of this server. That's one of the use cases we care a lot about because we want to use this in our own products.

Matei Zaharia

Yeah, it should be everywhere.

Shawn Wang

Yeah. I think one of the things that's really interesting to me is—first of all, I'll endeavor to do everything I can not to call it the modern AI stack because I think we have a name. But one of the first people who told me about compute sandboxing was Nikita from Neon.

A lot of people think of Neon as serverless Postgres with the separation of compute and storage, instant branching, and all those things. But actually, every database company is also a compute company.

Matei Zaharia

Yeah.

Shawn Wang

He was actually showing me his whole sandboxing solution. I don't think he ever launched it.

Matei Zaharia

So, our sandbox solution—the reason we could have built it so quickly was because we realized that if you just take the actual lakehouse architecture and remove the database from it—by the way, it was coming from you—there are some differences.

For example, in the ones that support this particular workload, it's important to have local persistence because you want your state to persist. Your libraries—you don't have to install your library every time, right? Whereas the Neon architecture, because of the separation of storage from compute, you don't need persistent local disk.

There are some differences, but at the end of the day, yeah, it's—

Shawn Wang

Yeah, so this is when you run a coding sandbox.

Matei Zaharia

If I use the dev infrastructure we had internally at Databricks, there are tens of gigabytes of data just for all the source code, artifacts, and stuff that I built, and I want that to come back next time. But yeah.

Shawn Wang

Before the show, we were talking about some adoption statistics that might be surprising. They could be internal or external, whatever comes to mind—just to impress people with the scale at which this is happening.

Matei Zaharia

On the analytics side, I think we launch maybe 50 or 60 million virtual machines a day across our 3 clouds. So we're one of the biggest compute orchestrators out there, for sure, for CPU compute. All of those, I think, process exabytes of data. I joked that, depending on which time zone you're in, typically before you have breakfast, Databricks will have already processed exabytes of data that day.

On Neon, it's actually pretty interesting too: it's launching, I think, 13 million databases a day now.

Shawn Wang

What do you mean?

Matei Zaharia

A lot of those were thanks to agents and branching experimentation. Because we made it so easy and so quick to launch databases—and thanks a lot to Nikita's team for that—it's changing the way people use databases.

Shawn Wang

Okay, we're going to go into more database talk in a bit, but I want to make sure we close up anything on Omnigen. You mentioned you're excited about the security and control side. A lot of companies are figuring that out right now, as well as the spend side. What have you found there?

Matei Zaharia

I spent quite a bit of time talking to internal users—developers, security teams, managers—and also lots of customers. There are a few things. First of all, one thing that immediately became obvious, as far as security, is that there's this tension between usability and security.

A lot of coding agents today have very basic controls: you can tell me which tool patterns are allowed or disallowed, or whatever. It's yes or no. That puts you in a very tough spot. As an example, should my agent be able to read some confidential documents? Or should it be able to install new packages from NPM, which maybe is compromised? It's a yes-or-no question, but maybe I want to allow it.

Should my agent be able to publish stuff to the company website? If I'm using the agent to code on the website, yes. But should it be able to do both? It could grab a confidential document, get prompt-injected, and leak it. Probably not.

The thing we decided we need is stateful, or what we call contextual, policies, where you keep track of the state of that session. It's not, "Is it allowed to push to the marketing site or not?" It's, "Hey, if it did a risky thing, like it installed a 1-day-old package from NPM, or it read 1,000 confidential documents, then no. Don't do it. Otherwise, maybe it's okay."

That's one example of moving that trade-off. It's both more secure and more useful by having a more powerful engine, essentially. This requires tracking sessions.

The other piece that was interesting there is that there are these very low-level events it's doing, and you want some libraries on top that parse them. For example, we have an MCP server on Google Drive internally. It's got 60 API calls. How do I know which of those will share a document with stuff on the internet and which ones won't? It's annoying.

So we designed the policy layer in OmniGen around functions, and you can have libraries. Someone can make something that maps low-level events to high-level ones, and then you write a policy about the high-level things that came out.

Shawn Wang

And was that related to Panther?

Matei Zaharia

Yeah, Panther will help with that. Panther is kind of a similar idea on the event-processing side, and it's Python-based versus a weird custom language. This is sort of more in real time.

Shawn Wang

Those things are happening.

Matei Zaharia

Yeah. These are the cool things. I think the contextual or stateful part, and then the way it can have libraries, was another reason to make it open source, because others will write libraries and we and our customers can use them.

The final thing is that, because it's stateful, one of the states we track is how much you spent in that session. I asked an agent to debug something, and it spent $500 because it decided to read a lot of log files and burn a lot of tokens. But I can literally say, "Okay, launch a sub-agent to do this and cap it at spending $5. Ask me for permission if it needs more."

Because we're counting that within the session, it'll pop up and tell me, "Okay, you spent $5. Do you want to go on?"

Shawn Wang

For more context here, Matei spent the last 5 years architecting Unity Catalog at Databricks, which is the governance layer for data.

Matei Zaharia

That's right. Yeah.

Shawn Wang

And it's sort of combining expertise at that layer together with all the AI governance here.

Matei Zaharia

Yes. But I also spent a lot of time being annoyed by coding agents and getting burned. As the CTO, I don't want to end up on the front page because I installed some weird npm package and leaked all the code.

So I'm especially paranoid, but I also have very little time. I don't want to sit there approving, "Do you want to run a 20-line bash script? Yes or no?" That's why I spend a lot of time figuring out how I can make it as safe as possible and not annoying.

Shawn Wang

Is safety—and let's call it security—a bigger concern than token maxing or token budgets?

Matei Zaharia

Yeah, they're both there. I guess it depends on the type of company you are. Some companies have limited budgets, and they really care—

Shawn Wang

About that. Or, I mean, you can be Uber and still be concerned, you know.

Matei Zaharia

Yeah, totally. For us, security is absolutely critical as a cloud provider. It's the most important thing.

We're not so worried about token maxing yet, but I've seen—for example, I talked to some consulting companies. They have 100,000 employees who are all coding for customers. If each of those spends an extra $1,000 a month, that's not fun. We have only a few thousand engineers.

Shawn Wang

What's the policy at Databricks? Is it just unlimited, or...?

Matei Zaharia

Unlimited, but we do use our own product to analyze the traces and stuff, and we have a team that's looking to optimize and see if anyone's doing something weird. We actually had some really cool insights just from analyzing current traces, like which models are better at, say, Rust versus TypeScript or whatever. At least in our codebase.

Shawn Wang

Yeah, amazing. Obviously, I have to ask the token-maxing question. I think it's a key thing, but yes, security and control are above that, and figuring out a sane layer where you can have some autonomy, but not too much.

Matei Zaharia

Yeah. We want to make it super easy. As an engineer, you should be able to set the thing. In OmniGen, you can ask your agent, "Set up a policy on yourself to do this."

Shawn Wang

If there's anything I should be showing, I don't see it on GitHub.

Matei Zaharia

It's in the docs there, so you can look at it later. Just look in the docs on contextual policies if you want to see them.

Shawn Wang

Yeah, if you want to follow up on this, this is exactly where to look.

Matei Zaharia

Yeah, and the story of these is that I wrote a doc with 10 ideas for things to work on. That was my wish list of things people asked for, and I told the team, "Hey, can you do at least 5 of these for the launch?" Then they got back to me with all of them.

Shawn Wang

Oh, wow.

Matei Zaharia

You can come up with more, but some of them are just meant to be examples. Really, you can intercept any event the agent is making, and then either block it, force it to ask the user, or allow it. You can update state to keep track of stuff.

Shawn Wang

Ultimately, I think of you as a systems designer: you let people plug in, right? That's the whole modus operandi of what you do.

Matei Zaharia

Yeah. We care a lot about composability too. Can someone else write a library that others use?

Shawn Wang

There's also a batteries-included philosophy here, probably very similar to how you did Spark: you could just start using it.

Matei Zaharia

That's right. It has to be good out of the box at certain things, and then you can build your own things on top. In Spark, if you just want to read a table or do an aggregation, it should be awesome out of the box.

Shawn Wang

People who want to catch up on OmniGen should watch your keynote and go through the GitHub and the docs. If they want to contribute or build on this ecosystem, where would you call out as the most high-leverage places to get involved?

Matei Zaharia

Get involved in the Discord and GitHub. Our team is there monitoring, and some of the things people ask for, we just build ourselves.

Some of them, we're collaborating with them to build that, and they also tell us how they would like to use it, because I think especially for developers, everyone wants it to work their own way. With a really good developer tool, you have to hear the feedback on all the ways and figure out the abstractions and how to let people customize. So we'd love to hear from you if you think, “Hey, I don't want it to work this way.” Tell us. We really just want to get that compatibility layer across agents and then let you do stuff on top.

Shawn Wang

Yeah. Is there anything, in terms of the startup side? I'm a founder, I see an opportunity, and I want to get in front of you. What's your request for a startup? What do you wish someone was working on?

Matei Zaharia

Oh, for a startup.

Shawn Wang

Yeah. You have your own startup and it's doing well, but if you weren't working on your own startup, what would be obvious to work on? [Laughter.] You advise many startups too, obviously.

Matei Zaharia

I do think that, as a company with a lot of engineers, anything that helps me make sense of how people are using coding agents and their spend, but also quality—like, “You should add this skill,” or, “You should write this thing,” or, “Your agents are really horrible at tasks involving this service,” or, “Go spend time.” That would be nice. Yeah.

Shawn Wang

Yeah, the closest I found is this team, Git AI.

Matei Zaharia

Oh, cool, yeah.

Shawn Wang

They started with, “We'll just do code and human attribution,” but they're basically building the analytics layer on top of that. I do think there are a bunch of them. Artificial Analysis is obviously doing super well with their stuff. There will be people—I think this is the domain of consultants first, but then people will actually build software that has the management plane for coding agents.

Matei Zaharia

Yeah, I think there'll be a lot of insights there. You have that in other areas.

Shawn Wang

Okay, well, the other big thing is your dream engine. [Laughter.] If you want to tell the story of OLTP, our background is that I'm going to make people listen to our Ankur Goyal episode, where we talked about SingleStore, HTAP, and all that history.

Matei Zaharia

Yeah, yeah. The OLTP idea is actually pretty simple. People have heard of Ankur's talk about HTAP; it's effectively the world of databases. Sorry, there's maybe a lot of context that needs to be injected here. The world of databases—

Shawn Wang

We're going to be the database podcast that's forcing people to learn their databases, guys. You cannot vibe-code with just Markdown files.

Matei Zaharia

It's one of the most important fundamentals of systems technologies out there. But the world of databases is effectively split into roughly 2 halves. There's what we call OLTP databases, which are transactional—think of your Postgres, your MySQL, and your Oracle databases. And the other side is what we call analytics, which we might sometimes refer to as OLAP.

The difference is that on OLTP, you typically run transactions on an event that looks up one specific row; we update that row, right? It's a very row-oriented data structure. In analytics, you're trying to reason about the data; you're trying to compute, “Hey, what's my revenue per store? How's my website doing every day?” And then you eventually want to run machine learning on it to predict, “Hey, how might my sales be going in the future?”

They are very different architectures, and everybody starts with OLTP databases because every app, when it becomes serious enough that it needs more than Markdown files, needs to have a database. You don't want to lose your data; you want to have some transactional consistency. But once you want to reason about the data, if you only have 100 rows, it's probably okay to run it on your Postgres or your MySQL database. But once you have more data and want to run more complicated analysis, the analysis might crash your Postgres database. So you start getting data out of the—

Shawn Wang

Replicate them into the analytic systems.

Matei Zaharia

Yeah. Some of them actually get into Elasticsearch for log analysis. A lot of our customers obviously get into Databricks to run more sophisticated things. And there's this term called CDC.

Shawn Wang

CDC

change data capture.

Matei Zaharia

And what it does, it reads the binlog of the database. If you don't understand what a binlog is, that's fine. But it's a little delta of the data, and then it reconstructs, based on the delta, the state of the database on the analytic side.

But CDC is a very painful thing. It's basically standard in the industry; everybody uses it. Many data engineers end up being woken up at 3:00 a.m. because of some pipeline issue.

Shawn Wang

My explanation is that everybody became a $5 billion company just doing CDC.

Matei Zaharia

Yeah, exactly. CDC is a very—it's one of the most boring but one of the most fundamental operations powering modern society. But it's so brittle that we joke it should be called continuous data corruption, because you might change your schema on your OLTP database and then the CDC pipeline fails to handle the schema change. And then everything goes out.

Shawn Wang

I mean, there are all sorts of tricks that you can do, like you add in some versioning or whatever, but—

Matei Zaharia

Yeah, but it's, in general, very complicated. At my keynote, I asked the audience to put up their hand if they love their CDC pipeline. Only maybe 2 people put it up. So, at SingleStore, about maybe a decade ago, I think the industry had this idea: “Hey, what if I built a single database that can handle both workloads?”

Shawn Wang

Which, by the way, every database person has always dreamed about.

Matei Zaharia

Yes, this is the holy grail of database engineering. Why not build a single system that can do both of these? But it ends up just being a lot of compromises. One, I think the first issue is that each— they say Postgres has a massive ecosystem, right? You want to be using the tools that are built for Postgres, and Spark, for example, has a massive ecosystem. There are a lot of libraries you want to use. If you were to create a new thing, you don't have an ecosystem; you tend to create a new, smaller proprietary API, and you're lacking both.

It's also very difficult to make it performant enough to be comparable on either side. So it ends up actually sucking at both. And our whole idea of LTAP—obviously a wordplay on the term HTAP—is that we think this is HTAP done right. HTAP wants to build a single engine for both. We think you can get 99% of what you need by unifying the storage and just having a single storage layer.

Once you have the single storage layer, if your Postgres databases are writing data in a column-oriented format, everything analytics can just go read that data directly without any delay, right? There's no pipeline in between, so all the data would immediately be available for reasoning and analytics.

I think I was telling some customers earlier that when we talked about this, it was going to be super useful for agents. At first, I didn't really believe in it myself, even though we wrote that positioning. But last night I was having dinner with an Australian customer, and they told me, “One of the big issues we have is that we have all these logs from our services, and we see SLA dips and want to investigate. But there's no way for those agents to even understand what's going on in the actual databases themselves.”

All we see is just product telemetry of the database and the services. You would actually make those agents 10 times more powerful if they understood, for example, who's actually placing those orders, what is happening, and what exactly they're doing. So now I'm actually sold on our own message.

Shawn Wang

Yeah.

Matei Zaharia

I think it really gets you almost all of the benefits of the HTAP holy grail, which is: make the data available immediately for reasoning and analytics.

Shawn Wang

Yeah, I think the way that humans are generally intelligent and want to have the ability and access to query anything, even while they do the work, they also need history and context. Where else do they get context? That's an analytical workload.

Matei Zaharia

Exactly. [Laughter.]

Shawn Wang

Yeah, I remember when we had incidents with our databases and the engineer said, “Well, I can't just run a giant query on it to see what's going on, because that's going to bring down the database and hurt it even more.” That's the kind of stuff that this gets rid of, because you spin up a whole separate fleet of machines that's doing the analytics. You're not overloading the main database that's still trying to serve stuff.

Matei Zaharia

Yeah.

Shawn Wang

So this has been a dream for a while. What had to get done in order to get to today? I feel like you've announced variants of this several times, but it wasn't as clear as LTAP. Yeah. I think LTAP is like, okay, we've got it, guys.

Matei Zaharia

I was talking to somebody at Meta, and he was asking me, “What's the catch? Why is it possible now?” I think the reality is we took a lot of time to actually work on the Lakebase architecture. Obviously, a lot of it came from the Neon team, which is separation of storage from compute. And it turned out it was just a tiny little step away, going from that to this LTAP idea. In the Neon architecture and in the Lakebase architecture, we're writing data in row-oriented format to the open data lake.

But in there, we're writing in Postgres pages. Actually, Ali and I were spending a lot of time debating: can we actually just change that to write in column-oriented format? We were just debating, and then one day, one of our engineers, who's actually super smart, came in and said, “Hey, I just prototyped it. It works.”

Shawn Wang

Wait, prototype what?

Matei Zaharia

Prototype instead of storing the data in the data lake in row-oriented format.

Shawn Wang

Like Postgres pages—write them in Parquet?

Matei Zaharia

Yeah. He just made the observation that our storage fleet has a lot of extra idle CPUs. We could use those CPUs to do the transcoding from row to column, where row is good for OLTP, but column is good for analytics.

So let's do the transcoding at that time. As a matter of fact, once you transcode the data, it compresses better. So, for those services writing to, for example, S3 or other data-lake-like object stores, you can actually write the data faster because it's smaller.

Shawn Wang

Yeah.

Matei Zaharia

So there's no overhead—there's no compromise in performance.

Shawn Wang

Overhead?

Matei Zaharia

Yeah, but we had extra CPUs anyway.

Shawn Wang

The fleet anyway, yeah.

Matei Zaharia

So the debate ended. It's one of the classics of the tech industry: a lot of debate, but then somebody actually went ahead and just tried to prototype it, and it worked.

Shawn Wang

But something this strategic and important to the company—I expect there to be a kickoff thing, like a design doc. Nothing like that?

Matei Zaharia

Nothing like that. We were debating in many, many meetings, debating whether it was possible or not from first principles, and then somebody just did it.

Shawn Wang

Yeah, if you set yourself up so people do that, that would be great. That happened a bit with OmniGen too. If I just had a doc and we could make this together, everyone would think, “What about this? What about this?” But if you try it out, it helps. If you have real users and they bash it, and it's still working—or, in this case, if you have the workload and you know what the workload looks like—you can just test the same pattern.

Tech aside, which is very cool, this is the most important thing: the culture of innovation. You don't have to ask my permission, and you don't have to go through a whole formal process. Just do it.

Matei Zaharia

Well, especially these days, I think with AI, it's actually easier to do that.

Shawn Wang

I think you're very right. I've met a lot of C-suite executives at large companies, and I think that at scale, things slow down. I'm sure you've felt it already, but somehow you have this core of people who are exempt.

How?

Matei Zaharia

I think we hire and work with really, really good people, and that's a very important part of it. Empowering them, but also spending a lot of time with us in the trenches, matters a lot too.

I think first, people could adapt to being in the larger company, so that helps. We want to make sure they know that they can try stuff, settle debates, and have a lot of examples of how it was done before, or launch a thing in beta, or whatever.

The other thing we encourage a lot is: instead of trying to boil the ocean for everything, let's figure out how to do it incrementally and very quickly. Many of our products are built in the span of weeks. Usually, my first question to whoever is building something is, “Who's the target customer? Who are you working with? Are you on a first-name basis with them? Are you texting with them?” I think having that very tight loop helps.

Shawn Wang

Can you bring up another launch that comes to mind within this kind of thing? I just want to give you some background on that.

Matei Zaharia

It was more of an internal thing, actually, because we would use that for our developers. Basically, the whole AI team got access to it and was using it, and we made sure it worked from the beginning with our internal code base, which is an enormous monorepo. We gave them some infrastructure and lots of token capacity. So it was all the developers.

Yeah, we had others. I don't—this is, I think, a public story, but—

Shawn Wang

I was going to ask about Marketplace and Delta Sharing. Did all of them have that? I just don't remember exactly which ones publicly referenced it.

Matei Zaharia

Yeah, they had others. Well, very early in the company, there was Delta Lake, which is the transactional storage layer we built. Our largest customer at the time said, “Okay, I need something in the cloud because, if the rest of our network is compromised, this thing needs to be separate to store and query the events.”

Then they talked to us and said, “Okay, this is the rate of events per second. This is the freshness I want. Can you do it?” That was way larger than any workload we had. We had our engineer, Michael Armbrust, working on that, and he worked just to make it work. Once it worked for them, it worked for everyone else. This was early in the company, probably 4 years in or so.

Shawn Wang

2018?

Ali Ghodsi

Yeah, 2017, 2018.

Clean Rooms, which is basically how you share data without sharing the underlying data, but still allow specific operations, was initially done just for 2 customers. I think the industry has a sense of, “Maybe if you overfit to 1 or 2 customers, it's going to be really bad for you,” but I think the downside of overfitting is much smaller than the upside itself. If you try to be too ambitious and boil the ocean, it's a much bigger problem.

Shawn Wang

Yeah, because you might end up actually having no customer.

Ali Ghodsi

Yeah, that's the more likely outcome. Then you can sort of pivot from there. I do think there is such a thing as a bad customer that sometimes you should fire.

One of the challenges we probably see—and maybe many AI and newer-generation companies are seeing—is that tech companies are very different from non-tech companies or traditional enterprises. If you optimize everything just for tech companies, you might have very big challenges scaling outside of tech companies.

Shawn Wang

Okay, what are the top 3 differences that you always think about?

Ali Ghodsi

Yeah, a big one is security, data privacy, governance, all that stuff. Usually, if you're building some kind of B2B or developer tool, your biggest market is going to be enterprises, but it's very different when you're dealing with a company that's existed for 30 years and has had some form of IT for that long. They have so many legacy systems, or they operate in a regulated space.

Whereas a startup, or even a more recent tech company, has everything new and sort of pristine. It's just different, and if you've never worked with enterprises or been in one, you just won't know about it.

Shawn Wang

And the procurement process is probably quite different. There are actually far more stakeholders.

Ali Ghodsi

That is one. Yeah. Another piece that's interesting is that, in some tech companies, people will say, “Oh, I can build that myself. I'll just build that myself.” So then you go—

Shawn Wang

I don't think people say that about Databricks.

Ali Ghodsi

They do. It depends on the teams and things. But, on the other hand, many of the enterprises say, “Actually, I don't—I never want to be in the business of building that. I don't want my—whatever, I'm a retailer or something—I never want to be down because some weird nerd couldn't get streaming pipelines working.”

Shawn Wang

Yeah, this makes them great customers, to be honest, right?

Ali Ghodsi

But you have to understand that it's hard without having worked there and stuff. You may not appreciate it.

Shawn Wang

Look, I think they're all great. Don't get me wrong. They have different challenges. But many of the tech companies, for sure, are far more DIY.

Ali Ghodsi

On the flip side, you have people who are very much experts in their domain. They're building airplanes, designing medicines, whatever, and they just want a bridge to the knowledge. They don't want to learn databases or whatever, as cool as we think it is, or as interesting as the average software engineer might think it is to read a little bit. They just never want to know.

They just say, “I have this giant matrix or whatever with my clinical data. How do I cluster it or whatever?”

Shawn Wang

Yeah, that's true. Okay, so then I wanted to actually build out the dream engine vision.

[Laughter]

Where does this all lead?

Ali Ghodsi

One of the things we realized maybe a couple of years back is that every single database engine out there, especially on the analytic side, is kind of a decade old. Pretty much everything that has reasonable traction is about a decade old.

They all started by targeting some very specific, narrow use cases. Over time, they became more and more successful, grew in their ambition, and tried to support more and more use cases. But the fastest way to support those use cases tends to be to hack around what was initially created. Those systems were not built for those use cases, but you can support them more or less okay. Before you know it, after 10 years of organic evolution that way, it becomes a gigantic pile of technical debt. That includes Databricks.

Very few companies or systems, I think, have the guts to say, “Let’s go start from scratch. Let’s go back to the drawing board and design, knowing everything we know today after a decade of workflows and probably billions in revenue. Let’s attempt to rewrite it from scratch and actually make sure it works and can support all these use cases.” So we started doing that.

But it’s a very ambitious project. By the way, you can search on Wikipedia for this thing called second-system effect.

Shawn Wang

Yeah, I know that. Every developer must know what second-system effect is.

Ali Ghodsi

It’s basically: You build your first thing, and it works out great, and the second one is bound to fail because it’s too ambitious.

Shawn Wang

You think you know everything, and then you’re like, “I’m going to design the perfect system this time.”

Ali Ghodsi

Yeah, and it turns out it’s not perfect, and then it starts failing. You’re too ambitious, you never launch, and you get killed.

The engineering team that actually started this was brilliant. I think we hired some of the best database engineers on the planet into Databricks, and they were brilliant. Thank God it’s not their second system. Many of them have built more than 2 in the past.

Shawn Wang

Nice.

Ali Ghodsi

But they were still worried about this. “Hey, building a database engine from scratch—I think the conventional wisdom is that it’s going to take, like, 5 years to mature. This will be a very long-term project. It could fail.”

One of the engineers was kind of joking and said, “Hey, maybe we just call it Project Stream Engine.” If we name it after a co-founder, maybe we don’t get canceled or killed.

But I think they built something pretty remarkable. They went back to the drawing board and changed the way database engines were built from a paradigm point of view.

Usually, when you build a database engine, you read a lot of academic papers, try to understand the latest algorithms and data structures, and put them together to see whether they work or not. There’s a high risk of failure there as well, because whatever looks really good on paper might work really well in 70% of the workloads, but then backfire on the other 30%.

They actually went and built more of a factory for building databases. They spent more time building this factory, and the factory takes the decade of traces we have. I think they count as a quadrillion data points in the trace table.

Shawn Wang

You don’t drop anything? Or do you sample?

Ali Ghodsi

We for sure sample, but there’s still a massive amount of data. They use that to build a model—a machine learning model, not an LLM. The machine learning model can very, very quickly tell us how any algorithm and any implementation will perform for any specific type of query with very, very high fidelity.

Based on that, they can pick the most likely algorithm and data structure that will actually help with the different kinds of workloads.

Shawn Wang

Mhm.

Ali Ghodsi

That happens both at runtime and at implementation time.

Shawn Wang

Mhm.

Shawn Wang

I mean, it sounds like you want to route to different data structures.

Ali Ghodsi

Yeah. If you think about it, a single database has many things implemented together, but you want to make sure they all work well with each other. For any given operation, there might be more than 1 implementation.

The reality is that an algorithm that works super well, for example, for very, very low latency might not work very well for, say, scanning through petabytes of data. Most often, there’s a trade-off between throughput and latency.

Shawn Wang

What are the key dimensions? Scale, throughput, latency—what else?

Ali Ghodsi

The distribution of data.

Shawn Wang

Yeah.

Ali Ghodsi

Right. How sparse the data is. That matters a lot. How frequently do you hit the same data?

Shawn Wang

Yeah. How many distinct values and stuff like that?

Ali Ghodsi

Those things matter a lot. The number of distinct values basically impacts the memory consumption of your aggregation hash table. At some point, there’s a hash table.

Shawn Wang

That’s something I’m going to try to list in my write-up. I really want a taxonomy. To me, taxonomies are so helpful because they cover everything you should think about.

Ali Ghodsi

I think if you actually try to list it out, there are probably a million different features.

Shawn Wang

[Laughter]

I always want, like, “Okay, give me 12.”

[Laughter]

You know, someone did an original paper, like, 40 years ago, about the 8 fallacies of distributed systems. That kind of thing is super useful.

Ali Ghodsi

Yeah, that’s it.

Shawn Wang

It’s like, “Okay, think through these 8.”

Ali Ghodsi

But let me give you a very weird example that actually has a profound implication for performance: Is your string ASCII, or does it have Unicode in it? How should I encode this?

Shawn Wang

I mean, strings are the most complex data types.

[Laughter]

Ali Ghodsi

For example, if strings are super dense, you could actually convert every string into a number. Imagine I have to do an aggregation. Instead of having a hash table, you could actually have an array, because if your string is dense enough—if you only have 256 options—you don’t need a hash table. You can just do an array lookup.

Shawn Wang

Like a country code or something.

Ali Ghodsi

Yeah. So there are probably millions of features in that model. Using that, they can, one, basically prioritize the different algorithms that might actually impact performance in practice.

Many of them are very counterintuitive. Things that you think might work super well actually don’t work that well in practice. More importantly, at runtime, you can dispatch the right algorithm and data structure.

Shawn Wang

I’m listening to the dream. I feel like Databricks is doing a really good job with the incremental evolution. Do you have to hard-cut to a new system at any point?

Ali Ghodsi

We designed it in a way that it can be incremental. First, we’re releasing a new endpoint.

But this goes to the broader notion of what we wanted to do. By design, this new engine should be able to do everything we’re able to do before, and better. The better part refers in particular to very low-latency workloads that can finish in tens of milliseconds.

But we want to roll it out incrementally, with incremental capabilities, so it doesn’t take 5 years to actually see the light at the end of the tunnel.

Shawn Wang

I think that’s a heroic task. I don’t know what other way to say it. I’m really interested in any sort of new workload and new databases. Obviously, I think I’ve maybe established that I’m a little bit of a database nerd.

The accounting databases, like TigerBeetle—I don’t know if you’ve seen those.

Ion Stoica

What do they do?

Shawn Wang

It’s a double-entry accounting database. It’s just meant to really model financial accounts and credit systems.

Ion Stoica

It’s like a very specific database.

Shawn Wang

Very high-throughput, yeah. Yeah. No, it’s exactly that. When you were talking about how everyone starts with a thing, then scales up, and then tacks on other things, it’s exactly that.

I recently interviewed Simon from Turbopuffer—same thing. Chroma as well. All the vector database companies of 2023 are suddenly now just generally general-purpose blob storage.

Ion Stoica

Especially, a vector database should never have been a separate category.

[Laughter]

Shawn Wang

I think that used to be a hot take. Now it’s the conventional wisdom nowadays. What should be a separate category? If everything becomes ELT, what’s—

Ion Stoica

I think the thesis of ELT is that we’re not collapsing the databases at the actual query layer. We’re just collapsing the storage layer. That’s a very important part.

We actually don’t think it makes sense to collapse the query layer into a single HTAP-style database. Part of it, by the way, is that a lot of people say, “Hey, it would be nice if there was only 1 query language I had to worry about instead of worrying about PostgreSQL and maybe Spark SQL. Why not just 1?”

But I don’t think that’s an issue for agents. Agents are very fluent in PostgreSQL or Spark SQL. They’re never going to get confused. As long as the data is there and accessible, agents will do fine.

That might have been—

Shawn Wang

Yeah, and the—

Ion Stoica

5 years ago might have been a problem for humans.

Shawn Wang

That could arise over time also, but it should. This leads to how to do things incrementally, right? We realize you don’t need it right now.

Ion Stoica

We don't need to solve that problem to have a lot of value from the current Delta app.

Shawn Wang

Yeah, okay. I'm going to end the pod with a bit of a spicier thing. Everyone has had the received wisdom around separation of storage and compute and tried to build the cloud. I had the same pitches from Snowflake. How have you succeeded where they failed?

Rough. Well, I respect that they're a competitor. Objectively, you have outpaced them.

What is the core insight, from your point of view, that you guys just took in a different direction?

Ion Stoica

Probably the biggest fundamental difference is that one is open. Both companies started around the same time. Both went to the cloud. Both focused on a separation-of-storage-and-compute architecture. But the biggest difference is that Databricks had never had a proprietary format, right? We started with the open ecosystem. We started with Parquet and then evolved into Delta and Iceberg and all that. It's like one big thing. I think that matters a lot. The other one is AI.

Before October 2022, when ChatGPT came out, we had always pitched Databricks as machine learning plus data. A lot of the platform was built with machine-learning use cases in mind. Obviously, AI is a little bit different, and Matei has spent far more time there than I do. But the whole platform was—we never felt, hey, we're just a data infrastructure platform.

Shawn Wang

Like Databricks only, yeah.

Ion Stoica

I think they started with—they thought, okay, we'll just manage the most valuable data and try to make it really fast. For that, we'll have our own storage, which is optimized with the engine. Then we'll just target the small amount of data that managers, finance people, and so on look at and make that super fast to serve.

It was a different space, whereas we started with: we'll do the bulk processing and ingest. You've got a bunch of JSON log files, you've got whatever—we do that very large-scale stuff because that's what Spark was for: large-scale batch-processing stuff. Then we'll keep the data in an open format. It might be slower, but it's already out there, and you can consume it downstream.

It turned out that it's easier to go from that batch thing that's really good at scale and ingesting and is super low-cost, and create versions of it that have the speed and features of the super-easy-to-use, smaller-data-for-business-users thing.

Shawn Wang

And then there's OPTIMIZE.

Ion Stoica

Yeah, start open and start large. In some sense, we started upstream of them. There was a time when we both listed each other as partners, because if you used both solutions together—Databricks for your ingest and compute, and then served the tables out of Snowflake—you got all the visualization and all the really fast stuff. That's great.

Then we both realized customers were telling us, "Why do I need this other thing? Why can't I just query your tables?" And we said, "No, we're horrible at that. Please use our partner for the SQL warehouse stuff." Then they realized, "Wait a minute, so much of the compute is moving upstream into this other thing."

Shawn Wang

You have to go into each other's territory, yeah.

Ion Stoica

But I think we did start with the bigger scope and with the open thing. That's important, actually. If your company's existed for 30 years, you've experienced being locked into Oracle and all kinds of crazy things. If you're the CTO there and you're setting up the architecture for the future of your company, you're going to want to pick a foundation that's open. You only want one way to manage data in your company, ideally. You don't want 7 different systems.

Shawn Wang

But I think the data formats have won. I think now every enterprise wants to put data in open data formats.

Ion Stoica

But it was actually very controversial back then. I think 5 or 6 years ago, one of the Snowflake co-founders actually wrote a blog called "Choosing Open Wisely," which basically argued against—

Shawn Wang

Yeah, yeah.

Ion Stoica

I think they might have taken it down. You have to find the archive now.

Shawn Wang

Oh, I mean, it's never going away now. No, no, it's still there. I love the perspective that only you guys will have because, obviously, you run the company. Thank you for indulging this. It's an incredible perspective.

Ion Stoica

Maybe one last one. As you were talking, I think I have to give Ali a lot of credit.

Shawn Wang

Mhm.

Ion Stoica

He's an incredible CEO. I think he's the perfect combination of IQ, EQ, technology obsession, execution, and business acumen.

Shawn Wang

Mhm.

Ion Stoica

He's also a founder, which makes it a lot easier for him—

Shawn Wang

Yeah.

Ion Stoica

—to mobilize and execute. I think that's—

Shawn Wang

Oh, that was it. So, did you have Ali? Okay.

Ion Stoica

Well, there's a whole lot of other things, but I think Ali played a pretty big role in the—

Shawn Wang

I thought there was going to be some technical choice that he contributed to.

Ion Stoica

He pushed for a lot of these. There were forks in the road where he pushed for one way, and then it became clear that that was the right way.

Shawn Wang

Second one, not a clear question again: Mosaic. A lot of people in our community are curious about the model story of Databricks. When you guys bought Mosaic, the thing was like, "Okay, well, we can do fine-tuning. We're going to do an in-house model," because they had the Mosaic models. It seems like you're not doing that, and it seems like you're going toward more of the AI apps and the harness stuff. What's the story there?

Ion Stoica

When Mosaic started, I think it was well known, or became most well known, for releasing open-source LLMs early on, and they were general models. Actually, before that, they were doing other things. They were focused on optimizing training systems, basically. They had the fastest image-model training stack in the world and stuff like that.

Then they decided to do LLMs, which was smart. They moved into it before ChatGPT, so they had some of the first open-source LLMs.

Shawn Wang

Yeah. We interviewed Jonathan Frankle and Abhi for MPT-7B.

Ion Stoica

Yeah, exactly. Oh, yeah, very cool. We decided that even though we did launch an open-source model, DBRX, and went up to above the Llama 3 scale, we decided that we really wanted to focus on the next step. There would be so many people releasing models, and instead of doing the general model, where a big part of the recipe is just throwing a lot of compute at scale, we wanted to focus on the next step: let's say you have the very smart model. How do you make it useful?

For us, it was a lot about automating how to make it very good at querying data. That's the first-party agent we have called Genie. It's like a virtual data scientist. Imagine there's someone who already knows all the stuff in your company inside out and knows all the machine-learning libraries, all the data libraries, all the stuff on the web, and you can ask them questions. That's what we wanted to do first.

That meant, let's not focus as much on just training some kind of frontier model, but let's build a system using either external models or fine-tuned, customized components. We're still doing quite a bit of model training, though. In fact, we're always procuring lots of GPUs and stuff to do it.

There are a few places where we're doing that. One is that there are many high-volume use cases where, if you have a specialized model, it's just so much better than any of the general models you get. A nice example of that is understanding documents—PDFs, Word documents, stuff like that—and parsing them.

If you've ever tried to do that, it's frustrating because you send it to Claude Opus or whatever, and it almost gets it, but it gets some things wrong. It's super expensive. You just burned a huge amount of tokens plopping an image into there.

Our team built this document-vision model that takes a page and gives you back nice JSON with all the components. It's very competitive. It's probably 100 times cheaper than those frontier models and still better. That's actually done by one of the researchers who came from DeepMind and was a co-founder of Adept, a very early LLM-scaling person, but focused on this.

Likewise, we're doing specialized sub-agents for part of what the coding agent does. If you've seen the stuff on advisor models from Harvey, also from—

swyx

And Anthropic's Claude Code as well, yeah.

—and UC Berkeley, actually, one of my grad students there wrote a paper called "Advisor Models." I think it came out before those did. I'm sure others had the idea at the same time, but that's something that helps a ton.

We actually showed some stuff just today at the keynote on—

swyx

Is it Parth? Oh, you know Parth?

Parth, yeah, yeah. Parth is—

swyx

—speaking at my thing.

swyx

He's speaking at Continual Learning Bench.

Yes, yes. I'm one of his advisors at Adept, yeah.

swyx

I interviewed his brother, Chai, because he's also at Adept.

Yeah. Yeah.

swyx

That family's very smart.

Ion Stoica

Yeah. They're awesome. [laughter]

swyx

We're doing some of that, and as we get experience with these in the first-party agents, we're also doing them with customers. My feeling is that customizing models is actually going to get way easier over time. That's what we're finding, because the base models are smarter, so they generate better traces in RL already, and RL is about learning from your own past traces.

Synthetic data generation is way better and way easier now. We have pipelines just using open-source models. The same model generates training environments and trains itself and beats Opus and GPT-5.5 and stuff at a task. So I do think it's going to pick up. The ease of training the algorithms is only going to go up over time.

There's a question of when it crosses into the mainstream. Instead of just the specialized document-parsing thing we did, where you need a hardcore LLM researcher, when does it get easy enough that anyone can plop in some stuff and describe a task?

Yeah. Well, you know what makes it easy? Interfaces and unified APIs. Obviously, if it's not interoperable, then you cannot switch.

swyx

That's what we're seeing with Omnigent and the composable agents. You can have sub-agents with specialized models, and then you can train the whole thing. I think that'll help a lot, too.

The last thing I was going to leave you with—I'm sequencing this, so I'm actually kind of proud of myself—is Satya. He's talking about this. I interviewed him at Microsoft Build a couple of weeks ago, and then he wrote this essay, which I'm sure you've seen, about the whole model-building frontier ecosystem. He sounded, when I was talking to him, more like a Databricks CEO. [laughter]

Uh-huh.

swyx

Is there—I mean, this thing presumably went viral in my circles. I don't know if it did in your circles. What's the theory of, I guess, tokens as IP, building up the context? He basically said everything, but data is the new oil, or context is the new oil—some version of that that you guys have heard before.

Ion Stoica

Yeah, I agree. I think the data you have, as you get better technology around that, you can just do more in your domain with it. It's not even just about AI. Even when people started collecting stuff in real time, I remember all the power companies put in smart meters and stuff, and all the car manufacturers started putting in sensors and cameras and stuff.

Any technology makes data more valuable and can give you some advantage—anything that helps you do something with it and make some decisions. AI is the same way. You had all this stuff that's just sitting there. Now you can have an agent automatically tell you. For example, I discovered a feature in my product was broken because a customer complained. The agent tells me, “I noticed no one is uploading files anymore because they got errors,” or whatever.

As you saw with Lakebase, as a database company, because we have all this history of all the queries and all the table layouts and how they work, we can build a new engine very quickly that's actually good, and we're confident that it's going to be good. So I think this is right. I think the question is exactly how it will land, but I do think custom model customization, which Sajid talked about, is going to get easier over time.

swyx

Yeah.

Which is why, by the way, I brought up the model thing, because they have their Mosaic AI things and you guys don't. That was the mental question.

Ion Stoica

Yeah, we do have—we're doing RL fine-tuning as a service with a bunch of customers. We have preview customers, and we have a general offering called AI Runtime. We get you GPU clusters on demand with a software stack in there that makes it easy to do training.

So we didn't—but that's existed for a while. We've had GPU compute for a while, and that's where a lot of the Mosaic stack went to help scale that.

swyx

Yeah.

Ion Stoica

But we found that the engagements—there are 2 types of customers. There are some who just want GPUs and libraries to get data in and out and monitors. That's what AI Runtime is. And then there are some that say, “Hey, can you actually work with me, build evals, build synthetic data—”

swyx

Yeah. The more forward-deployed solutions architects.

That's what we're doing. As more things transition from being custom to not, that's sort of how it is today.

swyx

Going back to the original question, I think one of the theses we have is that, once you can get the data in the right place, the AI models are becoming pretty good. The generic agents are fairly good. I mean, I think I heard you talking about AGI already being here. They have pretty good reasoning capabilities.

Actually, I think many of the traditional software systems will be rewritten with this new paradigm, which is just getting the data to be there. Then they slap some agent on top. Magic will come out.

Yeah. But without the right data, you can't really do that. And it's actually our approach to going into security and our approach to going into the customer data platform space.

We launched 2 products at Data + AI Summit, one targeting security teams and the other targeting marketing teams. Those all have a lot of existing technologies out there. I think our approach is just, “Hey, once you get the data in, everything is a lot easier with agents on top.”

swyx

Yeah. Yeah. Well, you guys have been fantastic guests. I just love this discussion. I love the ability to dive in on the tech side, but also culture and strategy. I hope this isn't the last time we chat. Congrats on all the success so far.

Thank you. Congrats on your success also. [clears throat]

swyx

Yeah.

Databricks is actually supporting my event. I run a conference. I've been an attendee of Data + AI Summit for a long time, and I noticed that it was kind of—this is back in 2022—90% data and then 10% AI. I thought, “Well, okay, we need a community thing that's just 90% AI.”

Ion Stoica

Yeah, yeah, that works.

swyx

So Databricks will be at the conference, and it's just amazing to see you guys build out the most interesting cloud that I have ever seen outside of the Big 3. It's amazing how far you've grown.

One of the most insightful things—I don't know, I'm not a VC, but I play one on TV—was when Ben Horowitz was talking to you guys, advising you on where this company was going. He was like, “Don't sell until $100 billion,” or some version of that story, right?

It was like, “The company should be worth a trillion dollars. You're underselling it for $10 billion.”

swyx

And he doesn't do that for everyone.

[laughter]

swyx

For some reason, I think he saw the vision, but also the infinite runway that you have.

We're lucky to have Ben. Yeah, he's a big supporter.

swyx

Yeah, amazing. Okay, well, thank you so much.

All right, thank you so much, swyx.

The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin | BidClub