Databricks：从数据到决策【Business Breakdowns 第238期】

All right, Alan, I am pumped to have you here to talk about Databricks. It’s rare that we go into the private sphere, but there are certain companies that are 100% worth analyzing in this space, Databricks being one of them. What I would say differentiates Databricks from a Stripe or a SpaceX or an OpenAI is that most people understand what those businesses do. I think Databricks is a little bit more of a mystery to most people who are not close to the business or who have not invested in it. So, if you could just start off with the simplest explanation you could give in terms of what Databricks actually does.

1. Databricks Processes Data At Scale

Totally. I think part of the challenge with Databricks is that they address so many different use cases. You may hear folks talk about, “Oh, we use Databricks for recommending movies,” or “pricing strategy,” or “fraud detection.” These are not necessarily super-related use cases, but it sounds like Databricks is very critical for all of them. Then you go a layer deeper and ask, “What exactly do they do?” You’ll hear folks say, “Well, they process the data.”

That leads me to the question: What does processing the data mean? For someone like myself, who does not have a technical background, even that can be hard to grasp. The example that resonates with me is that I think we’ve all had the experience of getting a data file—a spreadsheet in Excel—and being interested in running an analysis based on that data. It might be a very simple analysis, something as simple as wanting to understand the average price of a bunch of different items that were sold.

I think we’ve all had the experience of getting the data in the spreadsheet, but it’s not perfectly set up. Not every column is exactly where the price should be, and perhaps in certain cells the price is in a certain currency, while in other cells it might even be written out as text. You can’t just select all that data and say, “What’s the average?” You end up spending the majority of your time—sometimes 80% to 90% of your time—going through the process of unifying all the data into the same format so that you can run that very simple calculation of what the average is.

To me, that pain point of actually getting data into a format that allows you to ask even a simple question is this idea of data processing. In the case of Databricks, think of that at a completely different scale. You’re talking about tons of different types of data sources. There’s this concept of unstructured versus structured data. Anything that’s rows and columns and fits in a spreadsheet is more structured data, but the reality is that most of the data out there is unstructured.

It could be log files that are big streams of text. It could be images or video. For folks that are analyzing websites, it could be clickstream data. You take that problem of a lot of different data formats and ask how you get them into a state where you can actually run analysis against them. That’s how I would think about the core of what Databricks does.

They’ve since expanded, and they do all kinds of different things, but that data-processing concept is what underpins the primary pain point. You can tie that back to all these different use cases. If you think about an e-commerce company that is trying to determine how much inventory it should stock of a particular SKU or a T-shirt, whatever it might be, you could imagine that there are a lot of different inputs that might help you make that decision.

It could range from how digital advertising is performing for that particular T-shirt to how competing types of T-shirts have been selling. It could be credit card data. It could be all kinds of different data. If you could get access to that data, you could potentially put it into a process of creating a model to answer the question, “How many T-shirts should we keep in stock?”

That makes a lot of sense. I think painting the picture of the Excel model will resonate with pretty much anyone. You can talk to people about how simply taking data—whether it’s unstructured or just not in the proper format—ends up being 95% of the work. Often, I think that can create an environment where you have the idea of doing more analysis or running a more complicated model, but that workload up front stops you and ultimately limits what you’re doing.

I think that does bring it to life in a really thoughtful way. I want to go back to the beginning here, because a lot of what you mentioned ties back into some of the unique origins of Databricks. I wanted to start with the founding team and the academic background. There are all these clichés about academics and them not being commercial, but here you have this really impressive story of evolution.

Can you bring us back to the beginning stage of Databricks—who it was, what it looked like in those beginnings—and tell us a bit of the story, which I think is really interesting for this business?

2. The Cloud Data Open Source Bet

I think it’s a very unique part of the Databricks story that, to this day, you can trace a lot of the culture and DNA of the organization, as well as many of its decisions, back to this founding story. There were 7 founders who came out of Berkeley. They were all working in what’s called the AMPLab at Berkeley around 2009.

If you roll back the clock to that time period, it was actually the beginning stages of the cloud. What Ali, the CEO and one of the co-founders, would say is that the 7 of them were working in this building together, doing research. On the floor below was another team that was putting out some of the very early research around the data center being the next computer—the early concept of cloud computing.

You had, in one part of the building, a lot of innovation around cloud computing at the hardware layer. Then you had Ali and his colleagues thinking about what some of the software opportunities in the cloud might be. They were so close to the research around the early stages of the cloud. Unlike with AI today, when there has been a very clear recognition that AI is going to be a big deal, back then the idea of the cloud was still somewhat controversial.

Hard to imagine, but yes.

Totally. This group of people, who were rooted in research, gained conviction around the idea that the cloud was going to be important. Then they thought about what problems they should work on within the cloud. They actually considered a few different ideas, but where they ended up was this idea that data was going to be a really big problem—data at scale.

If you think about all the different use cases, going back to the beginning of the conversation, there are an infinite number of applications around data. The other interesting thing that Ali will say is that, during that time period, Twitter was becoming big, Airbnb was becoming big, and Facebook was still becoming big. It was a very positive time in technology, with a lot of optimism around entrepreneurship and starting startups, and that also fed into the energy of the group.

One of the co-founders was actually the creator of Apache Spark. They believed that data was going to be important, so they asked, “How do we create a business around that?” The other piece they thought about was using open source as another key bet, which was very aligned with the idea of coming from academia and research.

When you roll back the clock, there were 3 major bets they had a view on: The cloud was going to be big, data was going to be big, and open source was going to be a good way to build a business. In hindsight, all 3 of those bets turned out to be very good bets. The idea was less clear at the time, but because of that environment, where there was a lot of optimism around technology, all of it coalesced into the beginnings of Databricks.

On the point of connecting cloud to data, is it fair to assume that by transitioning to cloud, there would actually be more capacity for data to be stored or used? Is that connected in the sense that, prior to cloud, data capabilities might have been constrained?

Yeah. One of the paradigm shifts of cloud was generally this concept of scale-out architecture, which basically allowed the ability to use more commodity hardware to address larger and larger amounts of compute, storage, and data processing in this case. That was an important underlying trend that enabled this proliferation and this idea of just data explosion that I think you're touching on.

When you look at what Apache Spark was, it was leveraging this concept of distributed compute and applying that to data processing. That was very important.

The reason I asked is that it's always interesting to think about second-order impacts, or a certain market enabling another market on the back of it. Particularly with AI today, everyone's looking for second-order impacts.

Totally.

It's interesting to hear how they're connected. And to your point, those three ideas, those three bets they made, certainly compounded in many ways.

3. Open Source Becomes A Business

On the point of open source, there are all different ways to approach it. We have walled gardens versus open source, and there's the very famous Apple versus Windows example, or Apple versus Microsoft. Can you talk about the commercialization and how that played into things? I know that was a major stepping stone for the business in terms of evolving from a tool that gained usership.

Oftentimes, offering things for free is a great way to get people in. It has to be great, and it certainly was. But that feels like such a meaningful point in time for this business. I'm curious how you would describe what happened and how they really turned this into a great product and evolved it into a great business.

There are a lot of examples of successful open-source projects, and that's one of the magical things about open source: It does enable a level of adoption. In the consumer world, we talk about capturing magic in a bottle. In the enterprise world and with technology, open source is a really good way of getting massive amounts of awareness, mindshare, and adoption.

There are a lot of examples of successful open-source technologies, but there are actually not very many examples of successful businesses that have been built on top of open source. A lot has been made about Red Hat being one of the first companies that built a business on top of Linux, but the reality is that it's actually one of the hardest things to do.

You have to hit 2 home runs. This is actually one of the ways that Ali explains it: You need to hit the first home run, which is to develop an open-source technology that gets mainstream adoption. But what a lot of folks don't think about is the second home run, which is how you actually build a business on top of it.

Part of the problem is that the open-source technology ends up becoming one of the business's main competitors, because anyone can get the product for free. Anyone—including your customers, but also competitors—can leverage that technology. Those competitors that have more distribution and more customer relationships can actually do a better job of monetizing that open-source technology.

This is where, again, coming from academia really helped inform the strategy for Databricks. The benefit of not being immersed in the commercial market is that you don't have preexisting notions of what you should or should not do when it comes to building a business on top of open source.

Again, the biggest example before that was Red Hat, which basically provided services and support for Linux. That was the main thing. But because the Databricks team wasn't really aware of the precedents, they thought about things in a very first-principles way.

When you think about this challenge of how you monetize something where there's a free alternative, it's actually a simple answer: You need to create a better product that is worth paying for.

Simple answer, maybe not simple execution.

Totally. Part of the reason why it's not a simple execution is that if you've built your brand and gotten a lot of positive feedback for successfully creating the open-source technology, it can be weird to then say, "I'm actually going to create a competing product, and I'm not going to put all of the bells and whistles into that open-source technology."

For a lot of folks, that creates a lot of tension. You've got a lot of folks in the community who are like, "How could you do that?" There's almost a feeling of betrayal.

When you're successful with open source, it can almost be a curse because you become very popular. You're viewed by technologists as someone who's brought this great thing into the world, and then all of a sudden you need to be willing to be a villain.

I think a lot of people have trouble making that jump, but I think, again, in the case of Databricks, they realized that there's no way they can compete unless they have differentiation. Creating that differentiation was just a very important concept.

One of the things they did with Databricks, the company—and we can also talk about how they chose the name Databricks—was that they created a new implementation of Spark that was completely proprietary. It had a lot more performance and the things that enterprises want: reliability, scalability, and so on. They were just very unabashed about the fact that if you wanted to use this new implementation, you would have to pay for it.

Is that comparable to the free tier of an LLM versus the pro tier of an LLM today? You can feel the difference, and not everybody will have used the different tiers, but I'm trying to grasp the differentiation. To your point, it needs to be there.

It's very easy to describe it when you're signing up for a subscription to a website and it's like, "You get 3 free articles, but after that you have access to our proprietary database" for the subscription cost. What comparison would you make, if there is an analogy for the difference?

I actually think that analogy is pretty accurate, but there are nuances to it. As a consumer, we're used to this construct: There's a premium version, and then we pay for the premium version. I think that decision about what goes into the premium version is where the nuance is.

Again, going back to enterprise, the typical traditional wisdom is, "Oh, well, the average developer who's working out of their garage—we like that they can use this technology for free. We really want to monetize the enterprises. So why don't we monetize the enterprise features, things like single sign-on, governance, and security?" That actually makes sense.

But the reality is that while it is true that enterprises will pay for some of those additional features, how much will they pay for those additional features? At the end of the day, those additional features are not the core product.

Going back to your example of the LLM, there are different ways of deciding how to paywall a product. In certain premium products, you can use it 10 times, but then the 11th time you have to pay. In other situations, there are extra features that you have to pay for, and then you have to go to the premium tier.

The closest analogy in the case of Databricks would actually be the better model that is smarter and will give you better answers—you do have to pay for it. I think that comparison is not a bad one when you think about it through the lens of the core performance of the model, not just these ancillary things that you're paying for.

That differentiation—it's not added features; the core thing is actually higher quality—is a meaningful differentiator. This is something, to your point, that every business has to think through.

I talk to many people who want to roll out a premium tier of something, and the extra features are just not that valuable. Nobody's going to pay for it.

I did want to touch on Databricks—the meaning and what's the origin story to the name. To draw contrast, there are a lot of examples of companies that have been formed on the back of a successful open-source technology that have basically named their companies after the technology. Docker was a startup that was built to commercialize Docker, the technology. MongoDB is another example.

In the case of Databricks, the analogous thing to do would have been to name the company Spark. The reality is that there are a lot of benefits to that because Spark was a very well-appreciated name. The brand awareness that Databricks, the company, would have gotten from coming out with the name Spark would have been very beneficial.

But the reason why they went with Databricks is because, from day 1, they always felt like it was going to be more than just Spark. The way they thought about it was that there were going to be many, many bricks that could all be used around this broader problem of data.

It's a simple idea, but underneath the decision to name it Databricks as opposed to Spark is this reflection of long-termism—thinking about what is actually going to set them up to become much more over time.

I often think about that when companies evolve. There's an enterprise value of a brand that has multiple different products, and you could stick with that core product or you can evolve above it. It's a little artsy in terms of the way I describe it, but I do think there's something representative of that.

I think it alludes to one of the points that you made early on, which I think is a good thing to bring in now: They've evolved in terms of what they offer, and they have many different things under the hood now. I think this coincides with you and WCM's involvement in the business.

So can you talk a little bit about the evolution of what Databricks offers to customers and how it's evolved past that original state?

4. Databricks Builds The Lakehouse Platform

Databricks today has truly reached that point of being a platform. There's sometimes an easy framework of feature, product, and platform within enterprise software, and I think Databricks has made strides over the years along that journey.

Immediately following the success of commercializing Spark, what the Databricks team did a really good job of was recognizing, “Hey, who are we serving in the enterprise?” It's the data engineers and data scientists. These are folks who, after they process the data, are actually building machine learning models to run some of the predictions, forecasts, recommendation engines, and all those use cases we talked about.

There's actually a very complex tool chain to enable that process, which Databricks very naturally extended to. One of the products that they came out with, which ended up being open source again, was called MLflow. This was another product that extended the value proposition along the same use case for these data engineers and data scientists.

They then came out with another product called Delta Lake, which was a first step toward data warehouses. We'll get into the convergence between Databricks and Snowflake, but Delta Lake was another step in the direction of saying, “Okay, a lot of these use cases for machine learning are advanced scale-out use cases, but they don't necessarily need to have the same level of performance that allows for what's called transactional use cases.” That was actually an important next product as well.

The transactional use cases—does that have to do with speed?

Speed is part of it. There's a concept in the database world called ACID. It's an acronym: ACID. The actual letters stand for atomicity, consistency, isolation, and durability, which is a long way of saying that there are certain workloads that require a certain level of guarantees around the quality and integrity of the data.

For a lot of traditional analytical workloads—for example, taking it back to a data scientist who's running an analysis about how many shirts we should stock in inventory—it actually isn't that important that all the data underneath it is perfectly in sync, such that if one of the data sources is tweaked by a little bit, that totally throws off the analysis.

But there are other workloads for which you need to have that guarantee: that there are no inconsistencies in the data, and that one person isn't changing a piece of data here that has follow-on effects there. That's another segment of the market when you think about the types of workloads that databases can address, and it was an important extension and evolution of Databricks.

They then came out with a product called Delta Lake, which was a first step toward addressing these ACID requirements. One other really interesting thing about Databricks is that I think one of their core competencies is marketing. There's a funny story that, when they came out with Delta Lake, in order to explain to folks what it was, they gave out free T-shirts that said, “Delta Lake is Spark on ACID.”

That's amazing. So that was another extension of Databricks: introducing another product as a logical extension of their existing products, but also commercializing it well and understanding how to market the product to the broader community.

From there, to your question, in my view there was a very pivotal moment a couple of years ago, when we got involved at WCM in investing in Databricks. Following Delta Lake, Databricks started to show the ability to address traditional data warehouse workloads, which is the end part of the ACID journey that we discussed.

That was very important because, up until that point, you could largely say that the products Databricks had built were all addressing that core persona of the data engineer and the data scientist. But the types of folks who actually engage with the data warehouse are more traditional data analysts.

These are folks who typically use SQL to run queries against more structured data, versus data scientists running Python and building machine learning models against unstructured data. But because Databricks had built the foundation and logically laddered its way to the data warehouse, it was then able to come out with a SQL product that was more directly competitive with one of its peers, the public company Snowflake.

That was, to me, an incredible proof point. Databricks already deserved a lot of credit for expanding from a single product to multiple products, but then to expand further to multiple personas, for lack of a better term, was a tremendous TAM expansion in its own right. It was also a demonstration that there's so much that goes on underneath to enable the success of a product like that.

That was around a couple of years ago. They introduced their data warehouse product earlier this year, and they announced that it's on pace to be 1 billion dollars in revenue, which is an incredible amount of scale for a new product. That, to me, really started to demonstrate this idea that Databricks is successfully becoming a true platform.

Yeah, it's very interesting to hear. We had discussions before this conversation about your involvement in the business, and I'm very interested in late-stage investors in private businesses and what insights they glean. When you painted that picture to me of the evolution into a true platform, it checked out to me in terms of, okay, this was a unique moment.

They've evolved on that point about competition at the highest level. If I'm an organization using Databricks, am I potentially also using Snowflake? Am I using multiple vendors? Is it not necessarily a winner-takes-all market? How does that work, just in terms of how much dominance there is with customers when they choose one versus the other?

5. Databricks Expands Its Customer Footprint

The reality of the market is that there tend to be multiple vendors that enterprises will use. I think Databricks has contributed to a trend of enabling more types of tools for more types of workloads. Databricks will then be able to come out with more products to address those different workloads.

To answer your question, it is very much the case that you see customers using both Databricks and Snowflake, for example. If you roll back the clock to that core use case that Databricks addressed early on around data processing, it's a very classic situation where an enterprise might use Databricks first to process the data and then store that data in a Snowflake data warehouse.

Snowflake has started to try to move upstream to do more data processing, and Databricks has moved downstream to do more of the data warehouse. That can give you a sense of the way that these tools can live together within the same company.

I will acknowledge this is very easy for me to say from the cheap seats, but it would seem as though moving from processing the unstructured world into the warehousing, more structured world, might be a smoother evolution than vice versa.

Because in my mind, that unstructured world is a very complicated solution that was being offered—not that data warehousing is not—but do you think that's a fair representation? And definitely feel free to push back on that assumption.

Well, I would say that empirically, if you look at the numbers, that has played out to be true, with the data warehouse products scaling to 1 billion dollars. That has dwarfed the analogous revenues that Snowflake has had around moving to data engineering.

But what I would say is that, since we're not able to run an A/B test in different versions of the world, there was a lot of Databricks execution that led to it having more success moving toward structured data.

We touched on it briefly earlier, but I thought this was another example of the company having not just great technologists, but actually really good, savvy marketers and folks with a commercial gut instinct. When they recognized that they wanted to move into structured data, there was this concept that unstructured data—what people would call that world—would be data lakes, and then structured data would be data warehouses.

At the time, you could go back to some of the news coverage, and there was quite a bit of ridicule about this idea of a lakehouse. It was almost too clever: this idea that, “Oh, you're going to combine the 2, and you've come up with this name.”

You fast-forward to today, and the lakehouse is a very real, defined category that industry observers have all coalesced around. The credit Databricks deserves is not only for executing on the product and the technology to achieve what we've talked about with data warehouses, but also for doing all the work to educate the market around why the lakehouse architecture is the best of all worlds and why that is the future. That's an incredible piece of the story that I think Databricks probably doesn't get enough credit for.

Yeah, I think those things are very hard to measure, but you certainly can appreciate them more after the fact. I certainly give companies bonus points when they're having fun while doing this execution.

There's something that just seems to matter to me and shows a willingness to enjoy the aspects of business and competition and whatnot. There's a certain amount of fun, and I don't know if they would use these words, but I feel like irreverence. I think this ties back to the founding heritage and DNA, where it's, “Look, let's have an opinion about where the world is going.”

I think, as an investor, if you go back to the early bets, they would tell you, “These are the 3 bets that we're making. We're making a bet that cloud will be big, data will be big, and open source will be a good way to build a business, or at least build adoption.”

And here it was: We think the lakehouse will be big, and we think that this is where the world should go. We think that this will help customers, and we are going to bet behind that. It just makes it very clear that if you're betting on Databricks, you're betting on this future state of the world.

I find that sometimes with companies that try to have it all, they say, “Oh, we'll be good here. We'll be good there.” But the reality is that that detracts from your ability to execute in the way that Databricks did with something like the lakehouse. I think things like first principles can get thrown around a lot, but in preparation for this, watching and reading a lot of things that Ali has done in terms of interviews, it checks out in terms of the approach. There's a certain clarity to the academic world and being born out of that, in terms of understanding exactly what you're doing, having that focus, and having that clarity in terms of why you're going after things, which can sometimes get drowned out by some of the other baggage that comes with academia.

I would also say that a different way of describing this is that Databricks is helping to lead the industry to where they think the industry should go. They identify where the pain points are, and they come up with a solution that they think makes sense, as opposed to looking at an existing market and just saying, “Okay, well, we can do a me-too product just for the sake of expanding our TAM.” One of the underlying things that Databricks does a really good job of is recognizing true value creation, as opposed to just monetizing and driving revenue growth, if that makes sense.

Mm-hmm. It's based on customer challenges, but also having a predictive view in terms of what's going to happen in the future. There's a little Steve Jobs thing about designing for what the customer doesn't necessarily know they need yet in there—not to draw too hard on analogies.

On the point of market expansion, TAMs, and all of that, when it comes to both Databricks and Snowflake, both are relatively young businesses. Were they replacing industry incumbents? Was it all new-market creation? How would you describe the TAM that exists relative to what it was, whether it's prior to cloud or even within the cloud and using some of the incumbents?

Going back to the starting points of Databricks and Snowflake, Snowflake was really the next-generation cloud version of the data warehouse, which was a market that did exist. In the case of Databricks, it would have been that data lake market, but that had never been as well established because there was basically a lack of good enough technology.

There had been a lot of attempts at creating data lake companies. There was a technology that predated Spark called Hadoop, and there were companies that were built on top of Hadoop, like Cloudera, that went public at one point. But the problem there was simply that the technology wasn't good enough.

So, to your question of TAM, I think the market technically existed, but there was this period people forget in the early 2010s when big data was a very sexy topic. There was a period where companies had the recognition, or at least the inkling, that data was valuable. There was a whole period of a number of years where companies were storing a lot of data—volumes and volumes of data—with this underlying view that we should be doing that because big data—why not?

But the reality was that there was, in Gartner's term, a very hard trough of disillusionment, where it was, “Okay, now we've stored all this data. What are we going to do with it?” And it turns out it's very difficult to get anything out of it. So, again, going back to Databricks' core value proposition, it was really solving that problem, and that was massively TAM-expansionary.

Yeah, that makes a lot of sense. And data being the new oil, I think there's certainly some truth to it, but there's also this massive challenge of understanding that this probably has value, but how do we unlock that value and do something with it? Exactly. And that being a problem they solved is quite interesting.

I want to get a little granular, just in terms of a use case, so I truly understand what is going on, to the extent that you can answer this. One of the examples that I saw presented is: I make a credit card transaction. All of this data is flowing through the pipes of my credit card company, maybe my bank. Databricks is involved in that.

If they see I make a transaction with a nontraditional vendor, it's for a very large amount, and it's in a country that I've never made transactions with before, I can get a fraud alert. My understanding is that this flows through the Databricks pipes to some extent, just in terms of managing all the variables at play that would cause a fraud detection.

In that example, how does it work with Databricks actually having the ability to make the decision on behalf of the credit card company to send me that fraud alert, versus them presenting this alert back to the company and it coming back to me? It paints an interesting picture of how ingrained they are with their customers. I know it's going to differ by use case, but can you talk a little bit about that?

Sure. For any given credit card company, the implementation might be different from another. But to bring your use case to life, it is very accurate to think about that core value proposition that Databricks provides, which is that you could imagine the amount of data that goes into making a decision of whether or not a transaction is worthy of a fraud alert. There's a tremendous amount of input that can go into that, and there's probably never enough. You could always add more data to that analysis.

That hits on the core thing that Databricks provides, which is the pipelines to bring in all that data and process that data, because all the different types of data are going to be in different formats, and feed that into a machine learning model that gets fine-tuned by the data scientists and tweaked and will constantly be updated based off the facts on the ground and the empirical data of also evaluating these models: Are they actually accurate after the fact? And then tweaking those models again. All of that is core to Databricks' value proposition.

Now, once Databricks helps a company come to that decision, who is actually sending the fraud alert? Again, there could be different architectures here, but typically companies will build another application that actually takes the action, and the model output from Databricks will inform that action. That's maybe a classic way to think about the architecture.

That makes sense. The application that sits atop it is ultimately informed by the Databricks models that are analyzing all those things. And I can imagine it being rules-based: If a transaction meets these criteria and it gets sent up, then that triggers the fraud-alert warning.

Where I was going with that is I'm trying to grasp, again, you'll have companies that are using multiple different vendors. Obviously, there are so many different use cases. The one I just brought up is a cost savings; we've already talked about revenue growth and how this can be used.

But in terms of ways to measure stickiness and ramping with customers, what does that look like? It just seems the more you would use the model, the more ingrained it would be in your business, therefore less likely to churn.

Totally. They've disclosed that their net dollar expansion rates are greater than 140%. So there is embedded in that a high level of stickiness and also embedded growth. Quantitatively, those are the numbers that they've disclosed.

But I think qualitatively, the right way to think about it is that many of these use cases that we've discussed are very core to the fundamental product that businesses are selling. Sometimes in the world of data analytics, you can just envision a data scientist or a data analyst in the back office running an analysis for the strategy team. That may or may not feel sticky.

But when you think about these use cases where this is a content streamer suggesting the next movie you should watch after you finish a movie, that is core to the product, and that can often be revenue-generating and very mission-critical. So there is that level of stickiness in terms of once you get embedded into use cases in production.

Then there is also the added layer of stickiness around just the fact that there's a concept of data gravity. Once you put in the work to store and catalog data within a data platform, that becomes very sticky. So there are a lot of different dimensions with which Databricks becomes very embedded within a company.

I think the last thing I would mention is, if you've done the work to process data once, you could potentially use that for multiple use cases. You can then again imagine how that becomes very hard to unwind, even if a certain product gets sunset, but there's another product that's still leveraging the same data. That's very sticky as well.

You've made the allusion—and it's the elephant in every room now—of AI. I guess I'll just start with the very highest level: What are the impacts of AI on a business like Databricks, to the extent that they're beneficiaries, to the extent that there are risks associated with it? I'll let you wax poetic there.

6. AI Reshapes The Databricks Stack

Well, maybe just to start with some quantitative framing: Databricks—they're now over $4 billion in ARR. They have disclosed that about a quarter of it, $1 billion, is AI-related revenue. And so AI has already become a very large part of the business.

But I think beneath that, there are different ways to perhaps slice and dice the impact of AI on Databricks. And I think for me, one of the things that I've really liked about Databricks as an investment is that there are multiple ways to win. Starting with the core data-processing piece, there's almost a consensus understanding within enterprises now that you don't have an AI strategy without a data strategy.

Everyone recognizes, of course, that the model providers are doing what they're doing, and every generation of models is getting smarter and smarter. But at the end of the day, if you don't have good, clean, well-cataloged data, the models can only do so much.

One of the ways that AI has really benefited Databricks' business is that it's created a tremendous amount of prioritization and awareness around the importance of the core product that Databricks has always provided. In my mind, that is a durable tailwind for what companies will need to do and have always needed to do anyway. That is actually not dependent on whether we achieve AGI or not, or on what the next OpenAI model does. The reality is that as long as there is a general belief and understanding that AI is important, there will be a driver toward more data engineering and data processing.

That's a general tailwind for Databricks that, when you think about Databricks as a business and as an investor, actually paints a picture of a more durable growth trajectory that's perhaps not as spiky on the upside, but also not as volatile on the downside in the event that sentiment around AI changes.

That makes a lot of sense, just in terms of a heuristic at a high level for thinking about AI within the business.

Yeah, and there's probably a couple of other ways to think about AI's impact on Databricks. Another way is the fact that there is this huge cohort of AI-native companies, including the largest AI labs, that can and do use Databricks internally for themselves.

This is something that companies in the public market also talk about: How is AI actually impacting products and use cases? There's also the question of whether you're part of the stack that AI-native companies are utilizing, and Databricks very much is.

I think the final part of AI's implication for Databricks' business is actually the product. One of the big-picture bets that Databricks is making, going back to this idea that they have a DNA of having an opinion about where the world is going and where Databricks can add the most value, is really around the idea that, in the future, AI and LLMs have already proven—even if the models don't get any better than where they are today—the ability to automate more work.

When you think about how big of a TAM that is, it's probably just as infinite as the TAM that we talked about initially around data. What Databricks is doing is building products that, in the same way that they came out with MLflow, which helped with the whole process of a data scientist building a machine-learning model, create an entire stack. Products called Agent Bricks and Lakebase, all together, will help enterprises build their own agentic applications to automate specific use cases and actually automate labor and work, which is just a tremendous amount of ROI.

Yeah. I'll reframe it, maybe, and you can tell me how accurate versus inaccurate this is. Most of what we talk about with agentic is oftentimes just shrinking the context, giving it very deep context on specific tasks, and therefore the quality of the response is going to be much stronger. It's not going to be pulling from random places on the web. Databricks obviously has the richest data to use in terms of informing those models and therefore can partner with businesses that might want to develop those agents. Is that—

Exactly. Yeah. To your point, part of what the industry is realizing is that there are techniques that are important to leverage to build effective agentic applications. For example, RAG—retrieval-augmented generation. There's a whole set of processes around enabling that with vector databases, for example, and embeddings. Databricks has offerings there.

Another very important part of building agentic applications is model evaluation. Because of the very unpredictable nature of large language models, it's not quite as easy to always know exactly whether our application or agent is acting the way that we think it should. There's a whole set of technologies around model evaluation and being able to quantify how these models are behaving. Are they doing what we think they should be doing? Databricks, again, is building products around that.

It's parallel to going back to the machine-learning era, where you would go through a similar process around evaluating models. There, you can get a sense of the fact that while there's a lot of focus on the core large language model, if you actually want to build applications in production, there's so much around and beyond just the model. That's really where I think Databricks has a strong right to win.

Yes, the infamous question of where the value might occur within the layers of the AI ecosystem is quite interesting.

On the opposite side of the equation, there's been this interesting, solid, good relationship with the cloud infrastructure providers, AWS. Does that change at all with AI? Because, to your earlier point, the data cleanup and all of that becomes even more important. I've personally been able to clean up unstructured data into structured data much more cleanly with AI. It's a different story when it comes to me versus doing this at cloud scale.

What's your view on that relationship, given that the cloud providers haven't necessarily moved into this category? Does that change at all? Is that a risk? Is that something that you think about at all?

Well, it's funny you mentioned that cloud providers haven't really moved into the data market, because they actually do have offerings there. I think it's more a reflection of the fact that Databricks has done such a good job of both executing on the product and positioning itself in the market that you sort of have that premise.

I would say that, for as long as I've followed Databricks—and I first met Ali over 10 years ago—they had just signed their first strategic partnership with Microsoft. It was actually Databricks branded under Azure as Azure Databricks, and that was an incredibly important partnership to jump-start Databricks' modernization.

I bring that up because, from day 1, Ali, as a business leader, has always been extremely pragmatic and strategic about how they operate vis-à-vis the hyperscalers. What I would say is that there has always been co-opetition with the hyperscalers. The reality is that customers using Databricks will also be consuming infrastructure, compute, and storage from the hyperscalers.

There is a benefit for the hyperscaler clouds when customers are using Databricks on top of their infrastructure, and I totally expect that co-opetition dynamic to continue in the world of AI. I do think it's a very important question because this is a big enough market that the hyperscalers care about it. It's been another, I believe, underappreciated strength of Databricks: its ability to align itself with the hyperscalers.

There are a lot of examples of companies that came out with a great product, had a tremendous amount of momentum, and then Microsoft decided that this was too strategic for them to lose, so they were going to put all of their weight behind killing that product. I think we could all think of different examples here, and that has often been a real challenge for growth-stage software companies.

I think Databricks has done a very good job of never positioning itself in such a way that the hyperscalers are 100% incentivized to kill it. There's enough alignment, opportunity for partnership, and mutual growth that the relationship with the hyperscalers has generally been relatively synergistic, despite the fact that they do represent very real competition.

I bring it up all the time, but Amazon with something like FedEx and UPS, where they were relying on them and then FedEx and UPS couldn't deliver during the holidays, and it was a big problem. To the extent that Amazon built out a network and then eventually started to compete with it, there is something to the co-opetition being of high enough quality where it's not creating a problem. There are other dynamics that get involved, but that's very useful framing.

7. Databricks Funds Its Growth

I did want to get a little bit more into some of the financial dynamics. You mentioned the 4 million in ARR at this point. How does it work from a customer perspective? Is it a simple usage-based revenue model?

It is. Yeah. The way to think about it is that Databricks charges based on the actual compute being utilized for any of the workloads that are on Databricks. Going back to a tangible example, your credit-card fraud example: Every single time the customer wants to run an analysis, everything that happens underneath that in terms of the pipelines that pull in the data incurs compute cost, and that's how Databricks aligns itself from a modernization perspective.

I would say that we've talked about this open-source piece, but Databricks has been really smart beyond just the core usage-based pricing about recognizing when certain features or products are strategic, but when they actually have a right to charge for those products.

One of the examples that Ali has given is that when you think about your smartphone, one of its features is your address book. The reality is that the address book is an incredibly important feature, not just for phone calling, but a lot expands from having an address book in your smartphone. The reality is that no handset maker is going to be able to charge for the address book.

Databricks has a lot of different products that are analogous to the address book. In many cases, they're effectively using open source to give them away for free in order to get adoption, but they're still very strategic. One example of that is that one of their big value propositions is providing a governance layer on top of all the data, so that enterprises have a single pane of glass through which they can see all the metadata that they're processing.

And so, I guess all this is a long way to say that while Databricks does use usage-based pricing based on compute, the reality is that, at least from my perspective, they're actually monetizing more than just compute. It's a way of monetizing a lot more layers of value that Databricks is providing. There's an intertwined nature to all the different things interacting with one another, which represents value even if it's not directly correlated to the compute cost.

On your point of when they realize something has so much value that it should be charged for, does that come from it burning a hole in their pocket because of the compute cost—that's how much value it has—or is it more of a qualitative assessment?

I can't speak for all the conversations that are had internally, but I think my view is that there are a lot of different inputs into every decision around how to monetize and what to make open source. Sometimes it's more of a defensive stance, saying, “Look, we need to make sure that we get adoption of, for example, the governance product.” In other cases, it might be offensive, and they've also done a very good job historically of recognizing when they can be disruptive.

This might get a little bit into some of the technical details, but one of the big things that Databricks did very effectively to compete against Snowflake was embrace open formats. They purposefully decided not to charge for storage, which was something that Snowflake had historically charged for. That's another example of where, strategically, they're looking at a way to potentially be disruptive—not just from a pure lower-cost perspective, but actually architecturally, to enable customers to keep their storage and keep their data wherever they're storing it.

Don't force them to put it into Databricks. You can actually just run Databricks on top of where the data already sits, as opposed to how Snowflake historically worked: You actually had to move all that data into Snowflake. So that's another tangential point around this idea of making decisions about not only when to charge and when not to charge, but also having a strategic view around whether this is effective from a defense and offense perspective.

Yeah, competitive forces come into play. On that point, when it comes to general pricing trends, aside from what they charge for, I can understand that it all gets blended together. But does the pricing trend tend to correlate to the cost of compute, or would you say that they're able to raise pricing? Are there pricing wars? I'm just curious. There's the quality of the product, which is going to make customers make that decision. Some customers might depend on or lean in on the price. What drives pricing changes?

I would say that in this market, it's more about total cost of ownership relative to performance. What a lot of customers care about is whether they're able to run their workloads in a performant way. Again, because it's not apples-to-apples in a lot of situations, there's an infrastructure layer that the hyperscalers monetize.

If a vendor like Databricks can actually help customers run their infrastructure more effectively, that may not show up in Databricks's pricing, but from a customer perspective, that factors into total cost of ownership. A lot of that is a technology solution in terms of better understanding how certain workloads are behaving. How do you optimize the underlying infrastructure to serve those workloads?

In my experience, it's about getting workloads into production and really seeing what the total cost of ownership is to achieve the goals of any particular use case. The reality is that for a lot of these use cases, going back to the fact that Databricks is oftentimes embedded in the core products, or perhaps the nature of the decision is extremely strategic, if you can effectively provide the end value, the ROI is typically very clear for customers.

It's an interesting business in the sense that, with the use case of credit card fraud, you can actually draw a very clear ROI. Fraud is a major issue and a cost problem for credit card providers, and you're able to draw those connections. But there are other things that may be more difficult to draw direct ROI conclusions from, but are equally as valuable. It's very interesting to hear how different ecosystems and value chains work when it comes to this type of business.

On the cost side of the equation, cost of compute is theoretically passed through, and you have your overhead. Are there big buckets of cost that we didn't touch on that would be very important? Generally speaking, Databricks's model is fairly capital-light. They don't need to get very much into the GPU acquisition situation that we're all very aware of these days.

The reality is that a lot of core data-processing workloads are CPU-based. It is interesting that when you hear Jensen at NVIDIA talk about where he sees a lot of value for GPUs in the future, he does have a view that more of these workloads will transition to GPUs. But the reality today is that Databricks's products are not compute-intensive in the same way that you would think about a lot of AI-native companies.

Which is amazing to me. It just speaks to what training would actually require in terms of compute, but given how much they process, that's actually surprising to me. Interesting.

It could evolve. One of the things that Databricks is having success monetizing is called model serving, which is basically—when you go back to some of the examples we've talked about—this entire workflow around building the intelligence that underpins an application. You tie that into an LLM and create what's called an endpoint that then exposes that intelligence to an application.

More and more customers are asking Databricks to actually host that endpoint, basically like an API, on behalf of the customer. In those situations, Databricks will actually have GPU costs underneath that, but again, relative to some of the other examples out there, in terms of the scale of cost, it's a very different order of magnitude. Not all GPUs are created equal, and that's a whole other topic. But I interrupted you, I think, on the question of costs. Were there any other things that you would bucket in there that we didn't touch on?

So, Databricks has been free-cash-flow positive at the scale that they're at—$4 billion-plus in ARR. One could make the argument that free-cash-flow positive is a very low bar, and that perhaps they should be showing more profitability, but a big chunk of their cost is just around a traditional software business model. It's investing in people. It's investing in R&D.

Of the scaled software companies out there, I think it's one of the companies that, in my view, has as strong a track record as any in terms of demonstrating ROI against organic innovation. So, from a cost-structure perspective, there's nothing dramatic to call out other than the fact that they're still investing very aggressively behind R&D. That is a big reason why they've been able to maintain their pace of innovation even as they've expanded to so many new products and areas.

I do have to ask: It's capital-light, it's free-cash-flow positive, and they've done a lot of fundraising over the years. Where does that capital go?

This is actually a unique dynamic that is not necessarily specific to Databricks, but is more of a dynamic that I think more folks are aware of, which is that some of these tier-one, high-quality private assets are just staying private longer and longer. There's a dynamic where, once you reach a certain level of scale, there is a certain expectation for perhaps early investors, or oftentimes, more importantly, employees, to be able to get liquidity for their options or RSUs.

In the case of a lot of these companies, including Databricks, oftentimes the reason why they need to do these big fundraises is a tax consideration. Once you've provided employees enough opportunities to get liquidity, the IRS starts treating those RSUs and options as taxable. Historically within startups, one of the benefits of options was that they provided deferred tax compensation.

But this is something that I think the industry has started to learn: There's a real tax bill for basically compensating employees via equity. And so, to answer your question, the majority of the proceeds from the fundraises of Databricks have effectively been used to offset the employee stock compensation and the corresponding tax bill associated with it.

If I'm understanding correctly, it's not necessarily pure secondary in nature, where the employees are cashing out. Maybe that is true to some extent, but it's that when I get my equity grant, I ultimately only end up getting 66 of the 100 shares, because 34 are used to pay the taxes associated with that compensation.

Exactly. That makes sense. And on the point of more private companies staying private for longer, it's been a very interesting dynamic. As a former public-markets guy, I can actually understand some of those challenges. But what would you say, just from your seat? It's very interesting—you sit at the crux of all of this. Do you think it's a when, not if, about going public, or what would you expect that catalyst to be, whether it's specific to Databricks or you can talk about the industry as a whole?

It's literally a trillion-dollar question now with some of these assets and the scale that they've gotten to in terms of both the business and valuation. I think that, for that tier-one list of privates, there's increasingly a term that we all know of as the MAG 7 in the public markets, but there's effectively a MAG 7 in the private markets.

You've now seen that there is enough infrastructure in place from a fundraising perspective, both in terms of capital availability and the process of doing these very large-scale fundraises at late-stage growth, that it's basically made it pretty easy, if you're at a certain level of quality, to stay private.

Going public becomes less about the need to access public markets and more of a discretionary decision about the pros and cons, and for each business, there’s going to be a different decision around that. I think in the case of Databricks, it’s fair to say that because they were private in the 2022 cycle in growth tech, they were able to continue to play offense in a way that a lot of their public peers were unable to. It really helped to accelerate their business for a bunch of different reasons you could imagine, but generally speaking, the ability to continue to invest both behind sales and R&D was something that really benefited Databricks. I can’t speak for them, but I would imagine that was an informative experience, where the thinking would be that the reasons to go public would have to be sufficiently high to overcome the benefit that they’ve already experienced from staying private.

It’s a very interesting dynamic, and I think you put it incredibly well there with a trillion-dollar question. It’s going to be one that’s interesting to watch. There are a lot of headlines out there this week about what could happen next year, but it’s almost like a “believe it when you see it” type of environment now.

Totally.

I think we talked a lot about what has gone right and the opportunity set ahead. What stands out to you from a risk perspective? I think we’ve glossed over some of those, but when you think about risks for the business, is there anything that pops out the most to you?

8. What Could Break The Thesis

It is a dynamic enough market where the pace of innovation is still very important, so it can’t be taken for granted that continued R&D execution will be sustained. We’ve seen examples in this space of companies that have perhaps taken their eye off the ball, been slow to ship certain products, and how that can really show up in the numbers. So I think, first and foremost, it’s easy to say execution, but I do think in the case of Databricks, continued execution at scale is important because they are doing so many different things.

How they execute around the newer AI products is going to be important over time, maybe not necessarily over the next 2 to 3 years, because, again, of that dynamic that we talked about earlier, where the core data-processing tailwinds are so strong. But I do think for the ambitions that Databricks has, it will be important to execute on some of the AI products in the same way that they executed on the Lakehouse.

We are in a very similar dynamic right now where there’s a question of category creation. What exactly does an agentic application product portfolio look like? What do you even call that? And how does the industry coalesce around that set of tools and products? I think that’s not just a question of product execution, but also that marketing and commercialization DNA.

In the case of Databricks, time and time again, what I’ve seen is that the way they make decisions has been with a very long-term mentality in mind. Go back to the decision around how they named the company. There are always these trade-offs where, if you have a shorter time horizon in mind, you might make a certain decision. You might not open-source something. You might be tempted to monetize whatever feature you just came out with.

I think the ability for Databricks to continue to maintain that DNA is going to be really important because, especially as we enter AI, every single time you make more of a short-term-oriented decision, that inevitably opens you up to some sort of vulnerability down the road. I think it will be really important from a cultural perspective—which is something that we spend a lot of time focusing on—that they maintain that core culture of being long-term, but also that founding, academia-based DNA of thinking from first principles. It’s easier said than done to just say, “Oh, keep doing that.” For us, that’s an important thing to stay on top of.

Yeah, it might tie back to your original answer on staying private versus going public. Having that long-term mentality is a little bit easier to do in the private markets, I think, than in the public markets, which is notable and at such an interesting point in time with AI. It’s all very interesting.

We close these conversations out with lessons, and I think you might have tapped into some at the end of your answer there. What would you say are lessons that you can take away from Databricks? In the spirit of whether it’s pattern recognition or anything else, what lessons can you take away from Databricks and investing in that business?

Honestly, it would just be reiterating the last point around long-termism. I think that’s something that a lot of investors and founders talk about, but I think in the case of Databricks, it’s being able to actually point to so many specific examples where certain decisions are made and there is that clear trade-off.

There are certain times when people talk about being long-term, where it isn’t clear what the trade-off is. I find that the way the Databricks team is able to talk about the bets that they’re making, going all the way back to that original founding view of those 3 bets about where the world is going, and operate with that level of consistency, is something that really stands out about Databricks.

We talked about how there were ways that they could have monetized sooner. Even going back to the cloud example, where it wasn’t entirely clear that the industry was all in on cloud, they were so convicted that cloud was a real thing that they never came out with an on-premises version of their product. You can imagine, again, that there were a lot of examples where they could have, and that ended up leading to less monetization in the near term than they might otherwise have had. But they had this very clear view of where the world was going.

If you just go back through the number of examples we’ve touched on, you can point to a lot of examples of maintaining that long-termism and recognizing why certain decisions are made. I’ve found that, following Databricks, that’s been something that I’ve started looking for in other companies.

I think that’s a very interesting point on long-termism, but understanding what the actual trade-off is. It’s so easy to gloss over that second point. Very interesting. This has been a pleasure, Alan. Thank you for educating me on something that I only knew a very surface-level amount of material on. It’s been a pleasure.

No, this has been great. Thank you.