The Rise and Fall of the Vector DB category: Jo Kristian Bergum (ex-Chief Scientist, Vespa)

Okay. Hi. So this is another lightning pod with Joe Christian Bergam. Is that—did I get it right? You're over in Norway.

I'm over in Norway—Trondheim, Norway, in the center of Norway.

Yes. What should people know about Trondheim?

It's a small city. It's easy to get around. There's a great technical university here. The climate sucks a little bit, but it's easy to get things done in the winter.

I've never been over. I've been to Oradev, I think, which is over near you guys.

What we're here to talk about is generally your hot takes on RAG, search, vector databases, and all that stuff. I think you've taken to publishing a lot more recently on X, and that's gone really well. I'll go into the main thing that everybody knows you for, which is your piece on “The Rise and Fall of Vector Databases.” Could you give us the background of why you felt compelled to write this?

Yeah, first of all, I think I have to go a little bit back. I have a long background in search and working on infrastructure for search. I've been in search, working on search systems for years at Yahoo and FAST Search & Transfer here in Trondheim, Norway, and also working on embeddings, neural search, and all of those things, leading up to ChatGPT—the ChatGPT moment in November 2022.

Then there was some kind of cookbook, I think, from OpenAI, where they said, “Okay, this is how you can connect ChatGPT with your data, and here are embeddings.” I think a lot of developers then got into this: this is how we can build search; this is how we can do RAG. There was this unnatural connection between retrieval in RAG and vector embeddings—that retrieval had to use vector embeddings.

By the way, I have a small role in that. I was actually the one who wrote the Chroma example in the OpenAI cookbook.

You did. Okay.

I was an angel investor in Chroma before they became a vector database, and I was just helping out. I'm actually a huge fan of Jeff and Anton from Chroma. I think Anton left, but I think they've done a great job promoting retrieval for AI and infrastructure, and they did a lot of great things. I really enjoy talking to them on X.

Anyway, then we had the whole vector database movement. I think Pinecone was one of the pioneers in framing it as a new infrastructure category: if you need to work on embeddings, you have to use a vector database. Naturally, if you want to do anything in AI, then you need to have a vector database.

That was my primary motivation for writing that piece and looking a little bit back at what happened, where we are now, and how I see it. That was the pure motivation.

The general thesis, I guess, if you want to recap it, is that it was a very fast rise and fall. Pinecone was a dominant player for a long, long time, and I don't know my exact sources because there's a lot of rumors going back and forth, but apparently they went up to around $100 million in ARR very quickly to raise a big round. Then suddenly a lot of people started leaving. It suddenly went from cool to uncool very quickly, and I don't understand why.

I don't understand that either. I think they also repositioned a little bit, going back to their core messaging. If you go to their website now, it looks more developer-focused. It's not “the memory for AI”; it's not enterprise-ish. It's more toward developers now.

I think they're trying to go back to their original roots, and I think that's good. But of course, there's been a lot of competition in the space and a lot of new companies. One of the upcoming stars is Turbopuffer, which has a kind of similar model, a little bit different pricing, and really talks to developers.

I'm not saying that the companies are dying. I'm just saying that the separate infrastructure category is dying, because you have vector search capabilities in almost any database technology nowadays. You also have them in more traditional search engines like Elasticsearch, Solr, and Vespa.

I think there's a convergence of features on both parts. Then you have things like pgvector in Postgres, and a lot of people get confused: “Okay, I already have a database that has vector search. Why do I need another database, like a vector database?”

So I think the database category is dying. There are lots of great technologies here, don't get me wrong, but I'm not saying that the companies are dying. I'm saying that the category is dying. There's a distinction, and I think a lot of people overlooked that and came at me because they had some kind of hate around some of these companies. They said, “Yeah, you know, go Pinecone,” or whatnot.

I actually want to call these new companies search engines. I want to go back to the natural abstraction, because I think that's a more natural abstraction for connecting AI with knowledge and for all the arguments around doing RAG. I think the natural concept there is search.

One of the insights I had came from using Windsurf a lot. I log into Windsurf and use its Cascade mode, and if you ask it what tools it has available, it lists 17 or 18 tools. There are tools to edit files, but there are also things like “search codebase,” “search the web,” and search tools in general.

I love the idea of connecting the reasoning model with these tools that are essentially search tools. That can help the agent or the LLM formulate the query: should I do a grep search, should I do more of a semantic search, should I do more of a keyword search, or should I just search the web? I think that's a more natural abstraction than jumping into vectors. How you represent that is more of a detail of how you implement search.

It's interesting that we fixated a lot on vectors—dense embeddings and all that—and I think now we're broadening out. I would also mention that Chroma, from the start, has always said they're going after information retrieval and not so much the narrow sense of RAG.

Broadly, I think this is the consensus: the category was never really going to last that long. There was just a brief period of time. One of my favorite early tweets in AI was from this post-ChatGPT phase, where I summed up all of the fundraising that happened in vector databases, and it was something like $230 million in total across all the vector databases.

That was more than the entire lifetime fundraising of MongoDB. Basically, they cannot all win because they've already taken more money than it supports for a single one of the de facto winner companies in NoSQL.

Interesting. I think also, with MongoDB, they brought a new category in NoSQL. Nowadays, all the other database players have also caught up, right? Now even MongoDB has relational SQL.

There's always this convergence, but MongoDB kind of sticks. I don't think Pinecone, which was originally leading that movement, will stick in the same way. It's too narrow.

It's too narrow.

But I would like to say one more thing about embeddings. People are like, “Okay, Jo, but embeddings are really important.” I also think that embeddings are really important, because you can represent more data than ever before, like multimodal data and whatnot.

You run it through your neural network, get an embedding representation, and then you can move this embedding representation around in vector space and adjust it to your domain or whatever you're doing. So it's really important.

But what happened was that it went mainstream. It went from these big tech companies like Google, Yahoo, and Facebook—all of them have been working on embeddings for a long time for a lot of different tasks—but post-ChatGPT, when we got the embedding APIs from OpenAI, it suddenly became mainstream. Every developer could start using embeddings and similarity search and so forth.

I'm not against embeddings. Embeddings are here to stay. It's just that it's not only about similarity searches in this kind of embedding space. More people realized that you actually need something more than just embeddings and cosine similarity to do search well, like freshness, authority, and all of the other signals that really play a role in web search.

I remember one of the OpenAI guys wrote, “You can embed the whole web, and then you can build the next generation of web search.” I thought, “Okay, just looking at semantic similarity isn't going to play out too well.” They're trying to sell you their model, so they're going to say those very hyped things.

The way I put it is, you're always going to want to do a hybrid query. You always want to add metadata and do all that stuff.

My question to you is maybe an age-old question: should they all be the same system? Your search system, like Elasticsearch, typically duplicates whatever your main system of record is. Then you have a search index that's almost a complete duplicate—you just copy over the documents. Do you believe in that? Do you think there's a convergence here?

This is a fantastic question. I think for a lot of use cases, if you're already using some database like Postgres, it has this great extension, pgvector.

And I know that I tweeted things about pgvector that were true at the start, around the limitations of pgvector. But there was a rally around pgvector, with new algorithms being added—actually, 2 algorithms: both IVFFlat and HNSW—along with half-precision and binary vectors. So, actually, what you can see is that pgvector is doing more in terms of vector-search capabilities than some of the real vector database players.

If you're only looking at vector-search capabilities, and you already have your data in Postgres and are operating at a reasonable scale, I think it's fair to use Postgres. If you're not operating at a really large scale, and you do some vector-search-related workloads while also using a database for other types of workloads, then it might make sense to just keep the data there. But if you're building something that really depends on search quality, and your business depends on it, then definitely consider using a real retrieval or search engine to represent the data there.

Yeah, yeah. How closely entwined are recsys and search in your mind?

With embeddings and embedding-based retrieval, there's a convergence because embedding-based retrieval has been used for a long time in recommender systems, like large-scale recommender systems such as TikTok or Yahoo News. Apparently, TikTok published their recsys recently, which is kind of interesting.

In a system that operates at a really large scale, there's always a cascade of different stages. You first have to retrieve from a candidate pool, typically using embedding-based retrieval, and then you have layers of reranking. Finally, you end up with 100 candidates or something like that that you actually present to the user.

So I think there's definitely convergence. Embedding-based retrieval is now more common for such systems, so there's convergence in how it is solved across the technology spectrum.

Yeah. Any other thoughts on the confusion for a lot of folks who are newer to this? They understand now that you cannot just have embeddings only and cosine similarity only. It's just the sequencing: What should I do first? What should I do second? What should I do third?

Everyone says reranking is super important, but it adds maybe 3% to 4% to your results, and maybe that's why it's the lowest-hanging fruit. I'm always trying to figure out what I should recommend to people: Should they start with Postgres or MongoDB as their transactional and vector store? Then they can split it out to maybe use Elasticsearch or Vespa. I don't know if that would be the recommendation there. Redis is also trying to push itself there very, very hard, and then you add the reranker. Is that a good sequence?

It's really hard to come up with general recommendations without knowing what you're doing. But if you're looking to build a RAG application, I think most people are interested in something related to RAG. When you have some data that you have to transform, I think the first thing is—does Hamel always talk about this?—look at your data.

First of all, know how to get your data in a cleaned-up way. If it's PDFs or whatnot, there are things to do there. I think a very strong baseline is the classical BM25 algorithm that's been around for 30 years. It's keyword matching, but it offers a very useful baseline for a lot of different search use cases.

Then you can start looking at using an off-the-shelf embedding model to also embed your data. Most of the engines have some kind of hybrid-search capability. Start to play with that, and then, if you can afford it both from a latency perspective and a cost perspective, you can look at adding a reranking layer on top of that.

How you stitch that together depends on your framework of choice, but you can stitch this together with multiple different APIs depending on your budget, I guess.

Yeah. I always tend to recommend that people do this offline as much as possible—benchmark offline and whatever. Most people don't need fully online systems.

Yeah, and that's a friction point because I've been used to working on constrained online systems at pretty significant scale. When everything always needs to be online and low-latency, I have problems adjusting to when you want to do things at a much lower scale.

I'll give you an example. Calling out to some kind of embedding API to get JSON floats is not something you want to do if you're running at thousands of QPS. You don't want to add that dependency. You want to have something local, something that's faster.

I've always been like, “Okay, I'm going to call out to this endpoint. It's going to take 300 milliseconds to get this large float.” It's something that I just shrug at. But now I'm shifting toward, “It's easy. It's an API-based service. You don't have to think about it; it's just there.” So it's much easier to build from, right? To have something that is API-based. I'm trying to embrace that mindset.

I see. I see. No, when I say offline, I mean more like not in the critical path—batch systems. Have you looked at PostgresML for running the models alongside the database? Are you bullish on that kind of stuff?

No, I'm not. I'm sorry, I'm not. I think we've also seen other players that try to move a lot of the logic into the database—agentic embedding inference and whatnot, LLMs. I think the right direction is to keep infrastructure a little bit separate from that because there are different scaling properties.

People can stitch those 2 things together instead of trying to do everything with one single platform. So no, I'm not bullish on that because I don't believe in the developer experience of writing these huge SQL statements for transforming data, embedding it, and then writing it back, and expressing this in the database. What does this do to my database? Is it calling out, or what's going on?

I tend to want to have more control over cost and performance and what's going on than just writing some really large SQL statement to execute it.

Yeah, it's interesting. I think there's this constant tension between what should live in the database versus what is an external system. I don't think it's clear-cut, like the classic cron service, which we have in Supabase.

Okay, cool. Any other hot takes? What are the biggest criticisms that you got after you published this? What do you agree with, and what do you disagree with?

Yeah. I think one of the things people pointed out is that, if something goes semi-viral, after a few days you discover that there's a lot of replies that you didn't see, and you're like, “Okay.” But one of the things that stood out was that people said that Jo is saying that RAG is dead because vector-database infrastructure is dead, right?

I think that was a misunderstanding as well, and I think that comes from people making the connection between RAG and vector databases. It's so strong that when I'm saying the vector-database infrastructure category is dead, people hear, “Okay, RAG is dead.”

I think RAG is definitely not dead. Augmenting AI with retrieval or search is still going to be relevant, and I think it's going to be relevant for a very long time. That was one of the things. And now we have 10-million-token models and longer context, and you have the same cycle repeat every time.

For me, I put out this cryptic tweet. I was like, “Llama 4 is going to reignite the long-context versus RAG debate, but it will actually resolve the debate—not in the way that you want.”

This is too cryptic to me.

No, it's just like there are 5 other guys saying, “Long context kills RAG,” or, “RIP RAG.” I'm just like, “Guys, you're idiots.” Or you're engagement farming, basically. Most likely, you believe they know what they're doing, and they're just saying nonsense to have fun, and people don't take them seriously.

Yeah, no, but there's a nuance to this, right? I've seen people do RAG when there's no need to do RAG. If you have 1 PDF with visual information and things, and you want to chat with that, that case is probably dead if you don't have high QPS and things like that.

I had a call with someone that had 300 articles, and I said, “This will just fit into the context window of one of these Gemini models. You don't have to have a vector database for this case.” They were so surprised when I said this: “Can you really do that?”

But look at it. We had a 4K context window, and now we have 10 million, and that's fast. People are still running their initial demos from early January 2023, where they were dealing with 4K or 8K. Some parts of that are not relevant now because we have longer context windows, but I think retrieval, of course, is going to be there for a long time.

One example I love to bring up is one of these small toy datasets from TREC-COVID: 170,000 documents, and it's already 36 million tokens. You're not going to load all of that for a single query.

Yeah. Awesome. Do you have a take on knowledge graphs and GraphRAG?

I think GraphRAG—well, I have a lot of takes around it. I think the one issue is that a graph database solves one particular problem, and it does it well: traversing the edges in the graph, random access, and jumping across. But the core issue is actually building the knowledge graph in the first place: the entities and the relationships.

If you say that graph databases or GraphRAG are going to kill Vector RAG and all that discussion, I think the first issue is to actually build the knowledge graph in the first place. If you use a search engine or a dedicated graph DB to speed up and accelerate the searches, okay, fine. But I think people are like, "Okay, if I'm going to do GraphRAG, then I need a graph database," and I hate that connection between doing something and connecting it to some specific technology.

I think a lot of people do that: you jump from some concept into some technology. You can also do graph exploration with a search engine, so you don't need a specific technology to do it. Can GraphRAG be better than Vector RAG? Yeah, for sure. In some cases, it might make sense, or hybrid, or whatnot. But I think people get caught up in some specific technology all the time.

Yeah, but I think that's okay. I'm still trying to validate the presence of knowledge graphs in LLM applications because, obviously, with LLMs, it is much easier—better—to create these entity triplets and all that. So theoretically, it should be better.

Yeah. I mean, in the past, knowledge graphs have been a dirty word, but now maybe it's not. Maybe, maybe, maybe. I think with LLMs, you can do a lot more things around data generation in general, so generating those triplets is a bottleneck. It's been a bottleneck, and now you have LLMs, so I agree. Now it could be easier to actually build what matters, which is those triplets.

Okay, awesome. Any other opportunities that you find? I know that you mentioned Jina AI. I think they're a prominent European startup in RAG. And I think over here, Voyage just got acquired by Nvidia. Do you know anything on the embedding side? Do we need a lot better embedding models? Are what we have in the big labs good enough?

I hope to see—I mean, Voyage was really leading the pack on doing domain-specific embedding models, like legal PDFs, and what I want to see is more embedding models in that direction, where you essentially represent this PDF as an embedding or multiple embeddings for the legal domain, finance, or health. I hope to see that grow so that you can have a better starting point than just those text models.

I've been a huge believer in using vision-language models as a backbone for embedding models, where you essentially take a screenshot of a page. You don't have to go through OCR, so you then get a much richer representation. You don't have to go through these complex processing pipelines. So I hope to see more innovation.

I'm not sure if it's going to happen because I think it's a difficult business model to be in: you have to have an API-based service, you have to do batching, and you have to make up for the compute. And then, are people willing to pay for it? I think maybe that's why Voyage got acquired. I think Jina AI is also doing a lot of great things in this space now, especially in European languages.

But I think every company is trying to move up in the value ladder, right? They want to move into enterprise search or move in a different direction. So, yeah, but I do hope that we will see more and better general embedding models.

Yeah. Yeah. I mean, I'm sure the Voyage guys are very happy because it seems like they got quite a lot. Okay, cool. Anything else before we wrap? Any calls to action? Any parting rants on the topics of the day?

No. If you want to connect with me, for the audience, you can find me on X. I'm under the handle Joe Bergen there. I love sharing on X. I hang out there quite often.

Yeah. I mean, it's where the AI community is, you know. Although I've always been trying to grow on LinkedIn or YouTube—I mean, there are a lot more people there. There's Twitter, sure, but it's like an echo chamber. Yeah, but it's not the same. I mean, we wouldn't have this meeting, me and you, without X there, right? So it's a great place for really high signal-to-noise, and I think the AI community there is really great.

Yeah. Awesome. Well, thank you so much for having me. This has been awesome.