Arvind Jain on building Glean and the future of enterprise AI

You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. This is a conversation with Arvind Jain, whom I've known for a while. Arvind is the CEO of Glean, one of the most successful enterprise companies using LLMs—maybe right now the most successful enterprise company that foundationally uses LLMs.

And I really want you not to be fooled by Arvind's humble style. He's one of the most successful founders in Silicon Valley. Before Glean, he founded Rubrik, which IPO'd for, I believe, a valuation over twenty billion dollars. So he's been phenomenally successful twice. Very low-key, but clearly very driven and has some real insights. I hope you enjoy this conversation.

Arvind, thanks for taking the time to talk. It's been a while. I think you have an incredibly successful company, so I feel maybe embarrassed asking you this, but I think it's not a consumer brand, so people might not have heard of it. Can you explain what Glean does, please?

Yeah, absolutely. First, Lukas, thank you so much for inviting me to this. Glean is an enterprise company, an enterprise AI company. Think of us as ChatGPT, but inside your company.

Glean connects you with all of your internal systems, your data, and your knowledge inside your company, and then gives you a ChatGPT-like experience where people come and ask questions, and Glean will answer those questions for them. It's going to use all of the world's knowledge, but additionally, it's going to use all of your internal company data, knowledge, and context to help answer those questions for you. Simply put, that's what we do.

Of course, as AI continues to advance, the same platform that we use to build our Glean assistant can also be used as an agent-building platform to build all kinds of AI agents on our platform.

What's interesting about your company is that it's been one of the real successes, I think—or at least, from my perspective, one of the real successes—in terms of real LLM applications for the enterprise that work well. But if you look at the timing of when you started it, it was before it was obvious that LLMs were going to work so well, at least to most people.

Yeah.

How did your vision for the company start, and how did it evolve through this period of LLMs taking off?

1. Enterprise Search Before LLMs

Glean started in early 2019. In those days, nobody was talking about large language models, and the term generative AI did not exist, to my knowledge, at the time. We were not thinking about that either. We had a problem at hand that we were interested in solving, which was search.

This was my experience in every single job that I had before Glean: It was so hard to find things inside the company. When you do your work, you need some information; you're looking for a document. It's extremely hard to actually find information.

As enterprises, we have so much information and so much data, and it's spread across many different systems. In enterprises, we tend to have hundreds of thousands of different systems, and stuff is all over the place. When I'm looking for something—when I have a quick question I need an answer to, or I'm looking for a doc that I know exists but still cannot get to it—those things cause a lot of frustration.

In fact, there have been a lot of research studies. People would spend 1/3 of their working time just trying to find things. That was the first problem that we wanted to solve. We wanted to build a Google for people in their work lives: one place where you go to ask your questions, and we bring the right information back to you.

Sometimes, it may not even be the information. It may be that you're just looking to connect with the right people who can help you on a given topic. So we become that one place where you come and find data, documents, and people all in one place. You don't have to think anymore about where to go and look for things. That was the first product that we wanted to build.

The interesting thing is that in early 2019, transformers were already a thing. The rest of the world didn't really care as much, but inside search teams at Google, we were seeing a really good impact from transformer technology on some of the core components of a search engine.

We saw this real promise in those days, in early 2019, where we could conceptually understand information. We didn't have to actually go and build a keyword-based search system anymore, which is quite dumb. If you think about how we match information to people's questions, it's happening at a very surface level with keywords.

Google had actually put BERT-based models in open domain in late 2018, and we were able to use them and then actually go and pretrain what we now call small language models on your enterprise corpus. We could start to deeply and fundamentally understand what your business is about and what any given document is about, and then do this conceptual, semantic matching of information to people's questions.

Interestingly, transformers played a foundational role in Glean's evolution. We were probably the first company to use transformers in the enterprise in those days because it was really, really applicable to building a good search experience at the time. That's how we got started.

Over the last 6-plus years, as these models have advanced, what used to be a Google for you in your work life became more like ChatGPT for work life. In 2019, transformers had no capability to generate information. They had none of the reasoning capabilities that models have today. They had this embedding-based, conceptual sort of matching capability, which we were able to use.

As these new capabilities came, they allowed us to evolve our product. Now we don't have to just surface information to you; we can actually read that information using AI and give you precise answers to your questions. That's how the evolution has been over the last 6-plus years.

So you're politely telling me that you did see the power of transformers, at least in 2019, when you started the company, which is super cool.

Well, yes. We saw it, actually; it wasn't just a vision. We could take 2 documents—documents that had no real word overlap. If you look at the actual choice of words used in those 2 documents, they were very different, but still, a BERT-based model trained on your enterprise data could tell you, “Hey, these 2 documents are very close to each other semantically.”

That was it. Remember, in 2019, today we expect a lot from AI, but AI was not there in 2019. That itself was still a big step function in terms of our ability to really conceptually match people's questions with knowledge. We saw that power, and then we decided to bring it to enterprises.

You're in this world where these labs are investing massive resources and constantly coming out with new models. Do you still fine-tune a custom model for every customer, or is that important to you? How do you think about which parts to buy or use and which parts to build in-house?

2. Building Versus Buying Models

The thing I keep telling our technology team is, “Do not reinvent things that have been invented already.” That wouldn't do us or our customers a service. Our approach is always to maximally use innovation that's happening outside because it's available to us.

The good thing is that all of the innovation happening in the LLM space—all of these models from companies like Google, OpenAI, and Anthropic—they're all making those models available for us to use. We like to use those models, and we do; we use them quite a bit in our stack.

The way our technology works is that we actually pretrain and build some models for semantic matching. We have to do that ourselves because these models are all about your enterprise and your data, and sort of building an understanding of what your business is about.

And so there, we actually will go and connect with your enterprise data and knowledge. We'll take all the documents and actually build and train the model on them so that it can start to understand how your business talks and speaks, what code words you use, and what's your lingo. So for that, it's actually important to go and train a model on your enterprise corpus.

But these are very small models. We don't actually train large models. These models allow us to build custom embeddings for your enterprise content, so that when people come and ask for things, we can match their requests with documents in your corpus in a much better way using semantic matching. So those models we build ourselves.

But then when it comes time to actually do something with the information, imagine the workflow of when you come and ask a question in Glean. The first step is, well, we see your question and we're trying to understand what it's about. We're going to use our semantic embedding models—the ones that we train ourselves—to assemble the right pieces of information and get the best knowledge within your company that we think can answer those questions.

And now we're going to take this information and ask AI to synthesize an answer from it. For that, we don't actually use our own models. We use out-of-the-box models like GPT, Gemini, or Claude. Similarly, for reasoning, when you take a complex question and you're trying to break it down into a multistep agent workflow, we use the reasoning capabilities of the largest foundation models.

So that's the architecture. It's more, “Don't train, don't fine-tune when you don't need to.” But there are certain things in the search stack where it makes sense to do that. Maybe I'll give you 1 or 2 more examples.

Please, yeah.

Well, one thing—so in search, of course, part of it is that you get a question from a user and you're trying to semantically match it with the right information. That's what I described just now. But then there are smaller parts of the stack, for example, query expansion. In retrieval systems, you have to spell-check users' questions, find synonyms, and find acronyms.

For those models, what we've realized is that you can take these really small, open-domain language models and fine-tune them for those specific tasks. For example, for spell-checking, we have a fine-tuned version of a language model that will do it for us. For some pinpoint tasks—small, very specific things—these fine-tuned models actually help us do those things faster and cheaper.

They don't add more capabilities. You could take GPT, a large GPT model, to do a spell-check for you, and it would do a fantastic job. It's just overkill to use that model for that kind of task. So there's a little bit of that fine-tuning happening in the search stack for us. But otherwise, for reasoning and generation, we use out-of-the-box models.

3. Security For Enterprise AI

I think another one of your advantages, or parts of your technology that you talk about a lot, is the security model that you have. In enterprise search, there are such complicated rules around who can access what data. As these models get more complicated, if you're building an embedding model, you could imagine it somehow leaking information across embeddings. Can you talk a little bit about how that works and how you think about it?

Yeah. The way Glean works inside your enterprise is that, as we connect with all of these internal systems like Google Drive, Slack, or Salesforce, we're reading and understanding content that lives in those systems. But we're also understanding the governance of that data—who are the people who can access that information? For any given document, for example, we'll look at the list of people who are authorized to see it.

This is fundamental because if you're going to deliver AI in the enterprise in any way, you have to do it in a safe and secure way. You can't have AI-based systems start to leak information internally to your employees who didn't have the right to see that information. So the way our system works is that we connect with these individual applications, understand their permission model, and bake that into our core indexing technology.

The permissions are baked right into our indexing system. So now, when you come and ask questions in Glean, we'll retrieve information for you that can answer that question, but we'll only retrieve information that you have the right to use. We know who you are, you're signed in, and we'll use our enterprise identity to establish which documents you have access to and which are relevant to this topic.

Then we make the models work on it. That's the fundamental idea: in a RAG-style system, you can make AI safely work on data that you have permission to use. But you also mentioned this interesting question around embeddings. If you're building custom embeddings, what content do you actually train them on?

Because if you train them on all of your enterprise corpus, they will indeed start to leak information, and you won't realize in what form or shape they actually do it, because ultimately you can't understand, behind the scenes, how AI works. So when we're training custom models, we typically train them on content that we think is safe to train on. There are a lot of heuristics that go on behind the scenes to understand what information is actually safe to train on.

So if you use it only on a subset that is safe, then your embedding models are not leaking information.

I see. Speaking of security, I think a really interesting fact about you, at least from my perspective, is that you're also the founder of Rubrik, right?

Mm-hmm.

Which maybe not everyone has heard of, but it's an incredibly successful company that recently IPO'd. I feel like you're kind of a two-time unicorn, verging on a decacorn maybe, as a founder.

And it's so amazing. You're so humble and understated. I just wonder if you have any insights into what you're doing differently from other people. What do you attribute these 2 successes to?

I would think that a $10 billion-plus exit has got to be 1 in 1,000, 1 in 10,000, depending on where you start. So the fact that these 2 are correlated, I think, is a powerful indicator that there's more than luck involved, at least in your case.

Well, I think it is correlation, and no more than that. But what I would say from our experience is that when we started Rubrik, that was early 2014, and we picked a problem that we knew every business faced. It was really hard to protect your data, especially as the attacks were on the rise with ransomware and other things.

We felt that, in 2013, when we were contemplating starting Rubrik, all the energy and attention, and all the best tech people, were focused on consumer technology, and enterprise was sort of lagging at the time. We didn't have a lot of great companies. The SaaS revolution was still fairly early, and we felt businesses were being ignored.

Just helping them with this very fundamental problem—“Hey, we'll help you keep your data safe”—was something where there were no products being built that were less than 2 decades old. So we saw an important, large problem with a large market, and after that it was about just going and doing a good job building a product and doing a good job building a business.

I think we were fortunate in that we were able to go build a good team that then, of course, generated all this success. I would say that with Glean, similarly, we picked a problem that is not a niche problem. You can go and talk to every single person, anybody that you know, and ask them, “Hey, is it easy for you to find things inside your company?” And they'll all say no.

And so it was a pretty obvious problem. Again, nobody was working on it at the time, and we felt that we had to be the pioneers and solve this. I got a lot of advice, by the way, at the time, not to actually get into this business.

Well, yeah, because—sorry. I knew you were going to give me something really humble like this. But I think what these 2 have in common, in my mind, is that they're ideas that a lot of people have that don't work. I feel like I've heard those pitches for your 2 companies quite a lot in my life.

But then you were wildly successful doing both of these different things, where I think it must be really execution-oriented. It's not like you came in with some wildly new product that no one was expecting. It seems like you did a phenomenally good job in a market that didn't seem so great, or that seemed broadly crowded and specifically hard to execute in. I'm not sure, but I want to draw some insight out of you.

Yeah. I think I'll add 1 thing. Look, think about search. A lot of people actually told me not to work on search and enterprise search.

Mm-hmm.

Because there were only failures. In the 2 or 3 decades of enterprise search history, including products from Google, every time Google tried to build an enterprise search product, it always failed.

When you see that kind of history, people started to feel like this problem was maybe too hard to solve, or maybe it wasn't even worth solving. If nobody builds a big business, maybe there's no need to build a product like that. That was the mindset, and that's why there was no innovation in search inside the enterprise for a long time.

There were no startups. We started in 2019, and there was no search company that you would know of that got started in that 10-year period. This was a dead area, probably one of the worst areas where no investors wanted to invest.

But this is interesting: you see a problem that everyone has. Everybody faces this pain point, yet there are no good products. Then there are also technology trends. We saw some new things that happened at that time. One of them was SaaS, the whole SaaS transformation, which made the problem much worse, but also made the problem tractable in the sense that SaaS systems actually allow you to go and read data inside those systems. Before, we couldn't even do that. It was just so hard to get hold of data in the enterprise.

We also saw transformers as a really core new capability that would help you deeply understand content and actually build a better search experience. We observed those trends, which gave us the confidence to take that contrarian bet and say, “Look, we think we can now go and build a good product and build a huge business.”

I think part of it is that I’m an engineer, by the way, in my background and in who I am as a person, and doubts are part of my existence. That’s how I think most engineers are. I was surprised at myself when everybody else told me not to do this. I don't know what it was that actually gave me that conviction, so I was surprised at myself.

Maybe that's 1 thing to think about: if you're going to build large companies, of course you have to tackle problems that are very universal in nature—broad problems that have broad impact.

Do you think that you operate differently? Another thing that's interesting is that your product helps other companies operate better and more efficiently, and I assume you're dogfooding your own products.

Yeah.

If I were a new hire asking you, “Should I join?” would you tell me, “Get used to a special kind of culture that we have here at Glean”?

That's a good question. I don't know if we have a particularly unique culture. A lot of our learnings, and my learnings, have come from Google, where I spent a lot of my working life.

Google was a really special company. It was very different from the 3 companies I'd worked at before Google. It was so unique in the sense that it put technology and innovation in front of everything else.

As an engineer at Google, you were the king. Nobody got to tell you anything. You could go and build whatever technology you wanted to build. You got to decide whether it was important, whether you wanted to build that system, and you had all the resources to go and build those things.

That was the Google model: you hire really smart people, really good engineers who are very motivated to do something special, and then just let them be. They'll figure out the right things to figure out, they'll figure out how to organize with other team members, and they'll do a great job.

Frankly, for a manager, that's also a pretty good situation to be in. You don't have to do any work.

I love that model, and we've always followed it. I do it now at Glean, where we have a really amazing team, and my role is generally to stay out of the way. I can't help myself—I do get in the way sometimes—but often, it's mostly about building an innovation-first culture, and success will come.

4. The AI Powered Workplace

Do you think that having much better search—and now you're building these agentic systems, which sounds like they could be customized to what a company is trying to do—will cause companies to function dramatically differently in the future?

The way we work is fundamentally going to change, both at an individual level and at an overall business level. In fact, this is the vision for the future. I believe this is what our work lives are going to look like in the future.

For any person, regardless of how senior they are, you will have an amazing team of assistants, coworkers, and coaches that are going to help you not only do 90 percent of the work that you need to do, but also help you get better at the things that you do.

If you think about today, there are certain people in the company who have that luxury. If you're the CEO of a company, you have your coach, your assistants, a chief of staff, and an executive team. You have this really amazing set of people around you, which helps you become a true multiplier for the company.

AI is going to change that. Today, only the CEO or the senior executive can have that luxury and that help. AI democratizes that and brings that help to every single person.

Everybody's going to have this dream team around them, and you get to build that dream team with AI-powered agents. You can have 1 agent that works like your assistant, another that acts like your coach, and yet another that acts like your coworker and shares your workload with you.

That's the world that we're going to be in. You're no longer alone. You have this team continuously around you.

Similarly, for companies, most of the business processes that we have today could be envisioned as having parts that can easily be handled through some kind of AI agent. There's a big transformation ahead of us. It's going to reshape how individuals work, and it's going to reshape how a company—how an organization—looks.

People’s roles are gonna change. I think about software engineers. You hear that in the future we’re gonna become more code reviewers as opposed to people who write code. That kind of change in where we spend our time is gonna happen to all of us.

It’s funny. I think about what’s in our, at Weights & Biases, in our Notion and Google Drive and all these things. We have a lot of, for example, OKRs and our quarterly plans. I keep being tempted to paste them into Gemini. I know the context window is big enough—

Yeah.

And put in my current OKRs as I get through the quarter and just be like, “Which ones do you think we’re gonna miss?” Or, “What do you think about this strategy? Do you think it’s a good idea?” If you look back at what we’ve done, do you ever do stuff like that? Not tactical coding stuff, but sort of start to ask about broader strategic questions?

Yeah. Actually, today with Glean, these are the kind of things that it does a really good job at. Glean is connected to all those systems, so you don’t even have to—

Right.

Even have to cut and paste those OKRs from that—

Right.

Notion or Google Drive doc. It already is in that overall enterprise-wide search index that Glean has.

Some of my favorite activities with AI now are, number one, when I want to learn about technology, what a given team is working on inside our company, or how we designed a particular system. Instead of setting meetings and asking them to give me an update and walk me through some kind of presentation, I ask Glean to generate the latest, up-to-date tutorial on that topic for me. And that’s all I say.

I have this agent where we’ve told it that when I’m looking for a tutorial on a given topic, it should make sure to go and look at all the design docs that have been written. But design docs often become obsolete, so it should read the design docs, look at the updated results, and then start to look at more recent Jiras that got resolved and code that was committed into that component. Get the latest view of how things work. Don’t give me the stale doc; give me the latest on this system and how it works.

This is actually pretty cool, because now there’s this concept of evergreen documentation. I don’t have to deal with stale stuff anymore, because AI is doing that human-like thing where it’s reading stale information and new information, deciding what’s right and up to date, and giving it back to me.

Similarly, an example similar to what you mentioned: we’re a new business, and if you build a good product, you don’t have a churn problem. We actually didn’t have any dashboards around churn, and nobody had built any analysis on it because it wasn’t relevant.

I love how humble you are. That’s amazing. You’re old enough that you must have a very sticky product if, six years in or something, you’re not worried about churn.

Yeah. But guess what? We had some churn last quarter, and we certainly got worried: “Okay, what’s happening? What’s gonna happen to all the other customers? Where is the customer health?” I’m exaggerating a little bit. Some teams have it, but we didn’t have really good versions of it.

We wanted to do an analysis, so one of our team members in finance—she’s not an engineer and hasn’t built anything from a software-system perspective—went and asked this question in Glean: “Go and look at every single customer of ours.” You can ask Salesforce to get a list of all the customers. For each one, go and look at our shared Slack channels with them. Go look at what the conversations look like and what the sentiment is. Go look into our product dashboards and look at the usage.

Then take all of that data and give me a risk profile: whether it’s an account that is healthy—green—or red or yellow. Sure enough, AI is pretty good at this, and it generated a report that tells us all the top-risk accounts and the ones that are not.

Frankly, I would say it was better than any dashboard we would’ve built, because it was able to make use of subjective textual information, which dashboards obviously can’t. Dashboards can only look at usage trends and some structured information. These are certainly new capabilities, where you could tap into the 95% of your enterprise data that is unstructured. It’s full of insights, if only somebody had time to extract them. That’s where AI comes in super handy.

Totally. Wow, that’s such a compelling story. I want to do that as soon as we get Glean installed here. Wow. How do you—one thing I think about with you guys is your customers probably don’t let you pull metrics on their data.

Yeah.

Or maybe they do, but how do you know if you’re shipping a new model that’s better or if your embedding is better? How do you actually measure your progress?

Yeah. First, we have customer zero, which is Glean, our own instance. As we bring new models in, we have an evaluation framework. In any customer deployment, there’s plenty of information available, and you can have question-and-answer pairs: people have asked questions and somebody’s answered them.

You don’t even have to create these golden datasets. Sometimes you can obtain them from actual, real conversations that people have. For example, in Slack, somebody’s asking questions or people are responding with answers, and there’s a lot of reaction and thumbs-up to that information. That’s a really good example of a golden set of what a question is and how AI should be answering it.

We build these golden sets for each customer, and then we’ll bring a new model in. We would run all those questions through the new model. It doesn’t have to be a new model; it can be any change that we make in our own systems.

Our system has a lot of things happening in it, including changes in the search and retrieval system and the models that we use. So you measure how the answers are looking on the golden set with this new change that you’re rolling out. And then you also use AI to—because these are subjective answers, you also have to use LLMs as a judge to see how close you are to what the golden response is. That’s the fundamental way for us to measure how well our end-to-end system is working.

Hmm. Do you have issues with hallucinations? I feel like in almost every domain that kind of drives us—

Yeah.

Crazy. Do you take special measures about that? I would think of it from an enterprise perspective—

Yeah.

They might feel more sensitive to hallucinations than even a consumer.

Yeah, 100%. Hallucinations are there. We have some safeguards against them, but that’s not to say that Glean is always going to answer questions exactly the right way.

I think it’s not there because there are so many things that can go wrong when somebody asks a question. Sometimes the information isn’t even there to answer. Sometimes the information you’re using to answer people’s questions is stale or out of date. So the failure scenarios are numerous, and one of them is the hallucination of the model itself.

For the hallucination part of the model, remember that in our system, when you ask a question, we’re the ones who first assemble the raw materials—the knowledge that we’re going to use to answer that question. We give all of that knowledge to the model at prompt time, along with the question, and the model comes back with an answer. What we do after that is take the model’s response line by line and try to see if we can find that same piece of information in the input that we gave to the model.

We use that for our citation framework, and we cite, on a line-by-line basis, where the human-generated information was used to produce that particular line. When we don’t see it, we either suppress it or don’t show the citations. That way, you can tame the model a little bit. If you see information being produced that isn’t present in the input we gave it, then we can give up and tell the user, “Look, you asked about a relevant piece of information, but we weren’t able to figure out an answer.” That’s how we suppress hallucinations a little bit through citation and reference checking.

But the bigger problem in the enterprise is not that. The bigger problem, when people ask questions, is whether you can even get the right information from all the different systems you have, and whether you can get information that’s up to date, correct, and written by a subject-matter expert. Most of the losses actually happen on that side.

I see. Do you imagine a world where these agents start asking people if they can’t figure something out—routing to someone? I think I would kind of like that if I could deploy a bunch of agents to go collect information for my employees.

That’s an interesting use case. In Glean, one thing we do is that we don’t think of our product as, “Take all the information in the company, understand it, and then answer questions for people.” It’s very much a people-oriented product.

When we deploy Glean inside a company, we build a deep understanding of the enterprise itself: what the business is about, who the different people are, and what they work on. In fact, it’s a common scenario that when somebody asks a question and we can’t answer, sometimes our answer will be, “Look, we don’t have the information, but these are the people who are working on this topic, and you can go and connect with them.”

But with agents, you’re flipping the question. Instead of humans asking agents to do work, agents are actually pinging people to retrieve information from them. That also seems quite legitimate because that’s probably the future world, where there are deeply intelligent agents that work alongside humans. Humans can ask agents questions, and agents can also ask humans questions.

Do you have other thoughts on where organizations go as they interact with AI more deeply?

5. AI Augments Human Talent

I would just add one thing. There’s a lot of this ROI conversation that tends to happen with AI: “I’m going to have AI come and do a lot of the work that humans do today, and we’ll be able to replace them. Maybe I’ll have a small team. Maybe we’ll have a single-person company that does a billion dollars in business.”

There are all these viewpoints about how AI replaces humans, and I feel like that’s missing the point in a big way. Ultimately, a business and its value come from its people. That’s where the intellect is, and that’s where the capacity to do work is.

It’s much better to think of AI as an enabler for all of those people. As a business, I don’t want to shrink my team. I actually want to grow it, but I want everybody to be able to do 10 times more work so that we can succeed in a much bigger way.

So when you think about enterprises and how they change with AI, people are going to have one of two mindsets. Either they’re feeling the pressure and want to reduce their team size significantly, or they can keep investing in people and improve their top line significantly.

Do you think that the skills that make people successful will change in this world?

I think the fundamental skills remain the same. What makes one successful, in my opinion, is hard work, dedication, and the desire to succeed, and I don’t think that fundamentally changes.

Of course, all of us have to adapt. Think about software engineers. Maybe it’s no longer necessary to remember all the different APIs and frameworks that are available because AI can always bring that information back to you. You can spend more time thinking and designing, and less time writing code, perhaps.

These changes will happen incrementally, and we won’t realize that we’ve transformed ourselves by the time it happens. It won’t feel like a step-function change. Every day, there will be one new task that we start using AI for, and before you know it, you’ll be fundamentally different from what you were 2 years ago.

All right. That seems like a nice place to end. Thanks so much for your time, Arvind.

Thank you, Lukas. It was really fun to have this conversation.

I appreciate it.

Thanks. Yeah.

Take care.

Thanks.