[State of Evals] LMArena's $1.7B Vision — Anastasios Angelopoulos, LMArena

All right, we're here with Anastasios from Arena. I actually don't know your last name.

Angelopoulos.

Yeah, there you go. Congrats on all the success. You got the Arena handle.

Yeah, we did. We got the Arena handle. Thank you. Big branding moment.

I think X is becoming more commercial. Obviously, you bought it, but at least you have a place to go where you can be like, “Hey, we really like this.” But I do think dropping the LM has changed the feel of it.

I don't know. The reason we kept the LM at the beginning is because we started as LMSYS, right, out of the LMSYS organization at Berkeley. So we decided—

Language models.

Exactly. So we wanted to maybe broaden a little bit. We were the first Arena, so we felt like, “Let's kind of try to own that.”

Last time we had you guys on, you hadn't really spun out yet. I did a call with Alessio and I was like, “These guys are going to start a company.” [laughter]

I didn't know. I think you actually were already started at the time.

I don't remember. Maybe because I had chatted with Anjney, and he said he was your founding CEO.

He was indeed, which people don't know.

Anjney is a very interesting character. We have a podcast scheduled with him.

Yeah. Yeah.

He does a lot more than normal VCs.

He does. He's been incredible to us.

Do you want to shout out some of the stuff that he did?

Yeah, absolutely. The way the company started was as an incubation by Anjney.

Yeah. What he did was find us at Berkeley and pick us out of the basement. He was like, “Hey, these guys seem like they're onto something.”

He started working with us really early and gave us some grants. a16z was not the only one to do this; we also had a great grant from Sequoia. But Anjney was particularly supportive of us and gave us some resources to continue building out Arena before we were even committed to starting a business.

In that capacity, he formed an entity for us and said, “Hey, you guys can walk away at any time if you don't want to start a business.” It was really incredible—a very aggressive investment move by him. [laughter]

Right, because any money that he spent, at the end of the day, we could walk away and leave him with nothing.

But I think he wisely knew that the right thing for Arena was to start a company out of it. It was the only way that we could scale, and he knew that Wayne Yan and I would ultimately see that and be excited about doing it ourselves, which ended up being the case.

Was there a moment for you where you decided? I'm sure you were debating it yourself, and you had other opportunities. What was the deciding factor for you?

It became clear that the only way to scale what we were building was to build a company out of it. The world really needed something like Arena: a place to measure, understand, and advance frontier AI capabilities through real-world usage with real-world users, based on organic feedback.

In order to achieve the scale and distribution necessary—and, of course, the quality of the platform necessary—to do this effectively, we would need to start a company out of it. We considered other options: Were we going to keep doing this as an academic project? Were we going to do it as a nonprofit? But ultimately, under those constructs, we didn't feel like we'd have the resources necessary to accomplish our mission.

So you raised $100 million, or $80 million?

$100 million. That's a lot of resources. It's great.

What's it for?

Well, obviously—

On behalf of everyone—

Yeah, everybody—

Dude, it's an arena. How are you going to spend your money? [laughter]

First of all, we don't necessarily need to spend all that money, right? The purpose of money at a company is to give you cards to flip. It's to say, “Hey, you have enough resources so that if your first bet fails, you can make another bet and another bet.”

Of course, that's not to say that we're going to spend all of it. You want to spend things responsibly. Having said that, the platform is actually quite expensive to run. We fund all of the inference on the platform. The way it works is the platform—

You pay market rates. They don't give you discounts.

No, no, we get discounts, but they are standard enterprise discounts, the same that would be given to any other customer.

Have you disclosed any numbers? I see numbers of votes, but what's that in monthly tokens?

I don't know about tokens. I'd have to back that out. We have, let's say—again, this is off the cuff, but I can safely say—we have more than 5 million now, 5 or 6 million. We have probably 250 million conversations that happen on the platform. We're on the order of mid-tens of millions of conversations every month happening on the platform.

It's actually quite a large consumer platform for LLMs. Of course, nothing really compares to ChatGPT, but—

But no, still, I think the largest-scale ones—

The benefit of this is that it's actually quite a diverse population. For example, 25% of the people on our platform do software for a living.

Still, at this scale, how do you know?

We do all sorts of things. We either survey them or analyze the prompt distribution that's coming into the platform. I'm happy to share more. We've done something called Expert Arena, which is trying to understand the distribution of experts that are coming to the platform.

A lot of that can be unauthenticated, or whatever the usage is. But about half of our users now are logged in.

Yeah.

So we have some ability to understand them. We also have surveys that we run on the platform that tell us a bit more about who the actual users are. Of course, there's always response bias in surveys, so you have to take it with a grain of salt. Nonetheless—

If you don't know Anastasios's background, he's the guy to correct for response bias.

Okay. Yeah, there are a lot of guys like that—and girls.

Guys and girls. You're not the only player. There's Artificial Analysis. They started an arena. It's like some crypto people who started this. I don't know if you've had a conversation with them, like, “Hey, this is our thing,” or, “Let's work together on something.” I don't know if you've—

No, I've talked to Artificial Analysis. I've actually talked to both groups. Both seem—

Am I missing any major players? Is it just those two?

No, I think those are some of the larger ones. Depending on how you define the term, Artificial Analysis obviously has huge market mindshare around the analysis—

Of different AI systems.

Yeah, they told me they were going to be the Gartner of AI.

Yeah. That's kind of their goal, and I think they're going after that consulting market and so on. Artificial Analysis, from what I understand, is a team of consultants doing this.

Yeah.

They seem like really, really nice guys going after that particular market.

Their analysis is based on aggregating public benchmarks and turning those into analytics—

Independently rerunning.

And independently rerunning, which matters.

Yes. Using those to compile reports and so on that educate the field on the performance of all these different models.

But they also have arenas.

They have arenas, but the arenas are not based on organic usage. The thing that distinguishes our platform from theirs is that users are actually inputting their own use case. They're asking their own question. That gives a level of realism that their platform doesn't have.

Of course, they specialize in a slightly different thing, but I see those platforms diverging in that sense.

Yeah, and sometimes it's the only way to do this. For example, for Artificial Analysis, their video arena is pre-generated videos. You can't enter your own video.

That's correct. But we are doing it organically.

Yeah. Exactly.

As a voter, it does help in terms of not having to wait.

It does, but also, why would you go?

Do you actually care about other people's videos? Form your own intuition.

Maybe. Yeah. Maybe you're interested in comparing.

I'm a shitty prompter, right?

Are you? I don't believe that.

I'm a terrible prompter. I learn by example.

Don't denigrate yourself. Don't denigrate yourself.

There are many people whose prompts are much better than mine. Let's say that. That's a fact.

People have all sorts of cool ways of prompting. Oh, yeah. It's educational to see.

The only way to learn is by looking at other people’s prompts, seeing the results, and thinking, “Oh, I didn’t know you could do that.”

Totally. Yeah. Okay, so let’s go back to Arena. One thing I do want to say is that the number-one use of funds is getting off Gradio.

Oh, yeah. Yes, we—well, listen: Gradio is an incredible platform. Gradio scaled us to a million MAU.

Yeah, that’s incredible. And, of course, you tell the Hugging Face fans that—

Of course. Yeah, we’re really grateful to Gradio for taking us this far. Eventually, it became time for us to move off of it and go to React.

I’m sure Hugging Face would have loved you to stay on.

They would have. I’m sure.

Was there a technical reason? You just couldn’t get the performance?

Yeah, it just became hard to develop, and there were all these tools that we wanted in React. To do all the fancy things that you can do in React became kind of difficult. One example—I don’t know, what’s a feature that you really wanted?

Let’s say we wanted to create our own custom loading icons for video with notifications.

Okay.

How are we going to do that in React? It’s hard.

Yeah, I mean, you can make a custom component. I’m sure the Hugging Face guys are going to come in and say, “You can do that in Gradio,” which maybe you can. But fewer developers know how to do that. How are we going to hire for that? We’d have to reskill them, and people are less familiar with that stack.

So it’s all React, all that?

Yeah, all that.

Okay, cool. Other uses of funds that might be interesting?

Basically, that’s compute resources. It’s primarily inference; that funds the free usage of the platform, and then also hiring, of course—headcount. Yeah, we have an office. You know, it’s in San Francisco.

I’ll tackle one of the major things this year, which I’m sure you’re tired of thinking about, but for people who are not in the loop, this is going to be news to them: the Leaderboard Illusion, the whole thing with Cohere. Let’s summarize what they said and then your response.

Mhm. The Leaderboard Illusion is a paper that critiques Chatbot Arena, and the main—

Pretty brutally.

Well, I would say unscientifically. [Laughter] Let’s be clear: Cohere wasn’t doing that well on—

Cohere was around 74.

It’s all good. It’s actually a respectable place to have on the leaderboard. I don’t even think it was really Cohere—the model developers—doing this. It was more their research side. But in any case, what does the Leaderboard Illusion say? It says that Chatbot Arena was doing undisclosed, quote-unquote, private testing on our platform: model providers would send us prerelease models, and we would expose them, and so on and so forth. The claim is that this creates inequities in the leaderboard due to that prerelease testing. For example, they cited that Meta had, at some point, tested a number of models with us.

Of course, we can’t disclose all the details of how all that was done.

But that is the main claim of the paper. Our response to that paper is online; you can find it by searching for “Response to Leaderboard Illusion.” Our response is essentially pointing out a series of factual mistakes in the paper that question the validity of its claims. You can look at the first version of the paper on arXiv yourself and see the claims. I think most scientists would view that as—

Oh, they’ve corrected it.

They corrected it, of course, but they didn’t correct everything. They just corrected some aspects that were blatantly unscientific and false. For example, they said that we only sampled around 9% open-source models and around 60% closed-source models, and that this created a gap between open and closed source. In reality, we’re very supportive of open-source models, and it was more like 60/40. That was one example of an error in the claims.

Another example is that they claimed there was some sort of bias introduced by this prerelease testing and that it was undisclosed. In reality, as you probably know, we’ve been doing this prerelease testing for a long time. Our community loves it. They love getting the secret code names.

Yeah, the secret code names—Nano Banana and all that. So Nano Banana, by the way, started on you.

It started on us, right? And people loved it. It became a global sensation. A nonzero fraction of the global population was using it. About naming it Banana, was that their decision?

It was their decision, I believe. It was sort of randomly generated, and it just went—

No, no. So apparently, Nana, who’s a PM—

Yeah, yeah.

—is named after her because her nickname is Nana.

Oh, that’s sweet. I didn’t know that.

Nana put Banana on it.

Oh, that’s sweet. Yeah, I didn’t know that. I didn’t know the origin story.

To us, it just looked like a random thing.

But it was clearly head and shoulders above everything. Before that, there was Recraft Image, remember?

Yeah, I do. Of course. And also BFL and all those. All those models are great, and I think those teams are also improving quite quickly. But Nano Banana was a sensation. That moment alone changed Google’s market share.

Yeah, market share.

Yeah.

Seriously. Google’s stock—billions of dollars are moving because of Nano Banana.

And now there’s an OpenAI code red and everything.

I don’t know about that, but The Information reported this.

I would say image generation has been this weird part of AI overall because it’s not strictly AGI-critical. It’s not reasoning, and it’s not feeding more context into the model. It’s the model generating a visual representation. I always think, well, Gemini used to get a lot of complaints for generating racist images or whatever.

That was a hilarious moment, and ChatGPT also had it in the past. I’m like, can we just get rid of this? Do we have to do image generation? Let’s focus the positive reputation of AI in general on language models and coding and the other stuff.

Yeah.

But I’m wrong. [Laughter] I’m such a huge Nano Banana Pro shill.

Yeah, I totally agree. I was also kind of wrong about this. I didn’t see the positive benefits, but actually, I think these multimodal models are going to become some of the most economically valuable aspects of AI, both in the consumer space and in enterprise. One of the fastest-growing market segments in AI adoption is marketing and design.

Yeah, ads. I’m a content creator, right?

Yeah, of course. I’m sure you’re using it all the time.

Infinite supplies of diagrams and explainers and infographics.

Yeah. Soon, we’re not even going to be making them for our papers. Our paper figures are just going to be made by—

Yes. I do think they actually one-shot. DeepSeek came out with V3.2 recently. I took their explanations of the RL environment stuff and fed them into Nano Banana Pro, and it generated an image that helped me understand the paper better. The fact that I can casually generate a paper-quality diagram that would usually take a PhD student a month in Photoshop or something is incredible.

Yeah, it’s incredible. It is amazing.

I want to ask about your principles for running Arena. I think you manage a giant community—5 million MAU. What have you decided are the core principles, both before becoming a company and now that you’re a company? I don’t know if anything has changed for you.

I don’t think anything has really changed. We want to provide the north star of the industry and center the use cases of real users, foregrounding those so that people know what to target. The goal is to create a benchmark that is constantly fresh and does not suffer from overfitting because we constantly have new data points coming in. It tracks all the different new models and all the different new use cases of AI, and gives the whole world a sort of ground truth for how real users are using these models and how good they are on those use cases.

We continue to do quite a few open-source data releases. We’ve probably released more data than basically anybody on the real-world use cases of AI: millions and millions of conversations from real users that the community is using to study and improve on.

In terms of what you will build versus what you will not build, I’m not necessarily caught up on everything that you’ve launched. I know recently you’ve done the Dev or Code Arena.

Yeah, Code Arena.

Expert Arena.

Yeah, Expert Arena.

What’s in the critical path for you, let’s say, for next year? What have you decided you’ll never do?

Let me first talk about things that I’ll never do on the platform.

Integrity comes first to the platform. Basically, the public leaderboard that we show on LMArena, I think of as a charity. It’s a loss leader for us. We don’t really make money on the public leaderboard. You can’t pay to get on the public leaderboard.

It’s not like a Gartner in that sense. It’s not like any of these pay systems. It’s never going to be like that.

Models are going to be listed on the leaderboard—

Whether or not the providers pay and whether or not they’re getting a good score. They can’t pay to take it off either. And so, what that means is very important.

And so, what that means is that the leaderboard has a certain integrity that will never be compromised, of course.

Yeah. But not all preview models will make it onto the leaderboard.

No, but that’s okay. Those preview models have never been released.

Yeah, right. Who cares about putting unreleased models on a leaderboard? The point is that for every released model, the score that you see on the leaderboard is statistically sound. It reflects the real-world capabilities of the model.

Yeah. Why? Because millions of people from around the world have voted for it, and that’s where that number comes from. All we do to compute that number is let millions of people vote. We take those votes and turn them into a number that’s always going to remain a transparent and fair reflection of model performance.

Where are we going? Lots of different new categories. I don’t know if you recently saw that we exposed occupational and expert categories. Now, a single-digit percentage of our user base—we’re at millions to tens of millions of users, right? So a single-digit percentage means a lot.

A single-digit percentage of our user base is in medicine, legal, business, finance, accounting, creative, marketing, and things like this. We’re able to show the performance of these models in all these different verticals because we have all these users in our user base. We’re working more toward multimodal. Video is soon to launch on the site, at some point later this year or early next year. So, there are lots of things in the pipeline.

Amazing. Will you expose an API?

We’ve thought about it. Yeah, I think it’s a possibility.

Yeah. What are the counterarguments? Why not?

Well, there’s obviously a need for an API. The question is more one of focus for our company, just because we’re a startup, and so we really should be doing one thing well—

Arenas.

Yeah, arenas. So, I’m not sure how far we want to splay out, or on what timeline we’d want to do that.

Yeah. Any other community-management tips, more broadly? Every AI company really wants to grow its community. You’re obviously one of the strongest in the world. What’s really worked?

Well, first of all, I want to give a shout-out to our community manager, Greg, who is doing an awesome job managing our community, whether that’s on Discord or on LMArena. He’s really incredible. So, I would say, hire Greg—but don’t hire Greg.

Don’t hire Greg.

Don’t hire Greg. He’s ours. Find a Greg. I agree.

But in general, the question is: How do you get to so many users?

That is a tough question.

And keep and retain them.

That is a tough question because consumer is one of the hardest markets in the world. There are a lot of websites in the world that people can go to. Why should they go to yours? The reality is, if you want to create a really dominant product, you have to provide people value.

To be frank, I don’t think we’re all the way there yet. It’s not like I have the solution and answer for how to build a great consumer product. If I did, we wouldn’t be at tens of millions of users. We’d be at hundreds of millions, or we’d be at a billion users. We’d be like—

Is there a world where you’re bigger than ChatGPT?

I don’t know. I don’t know that we need to be.

Yeah.

And I don’t know that we ever will be, because that’s an extraordinary generational product that they built, right? It took a lot of time, and to some extent it also involved luck. There are a lot of lightning-in-a-bottle moments, like Nano Banana was for us, where our user base just goes up by a lot.

But when those users come, they can just as easily leave. The way I think about it is that every user is earned. You have to earn them every single day. They can leave at any moment. They’re fickle.

All the time, you have to be thinking about: How do I provide this person value? Learning how they’re using my website, what more could I give them, and how do I build in all the retention mechanisms so that they stay and then also bring their friends?

Is there one thing that’s working in terms of retention? You said a lot of people are signing in now.

Yeah, sign-in was a big driver of retention.

No, but what did you give them in order to encourage that?

Persistent history.

That’s it? That’s enough?

Yeah, that’s one thing that has had a big impact.

Okay. Yeah. What do you want from people? What are you looking for help on? Any call to action?

Yeah, we are always looking for people to come and join us. If you are one of the best people in the world in your area, whether that’s consumer product, machine learning, B2B go-to-market, marketing, or any of these things, we need you at Arena.

We’re building a high-performance team of real experts in everything that they do, and I’m always looking for excellent people to work with.

What about partnerships? Let’s say I’m at Cognition. I want to partner with LMArena, or just Arena. What works for you? What existing partnerships do you already have that are really fruitful?

Yeah. So, we of course partner with all the major model labs.

Yeah.

And that’s straightforward—

And that’s straightforward: “Hey, we have a new model here. Here you go.”

Exactly. So, I think the most straightforward thing for someone like Cognition would be: Let’s evaluate that—

Agent.

Yeah. But we should be continuing to shape our—well, Code Arena is an agent evaluation.

That’s true. That’s true.

And it’s more focused on— all these arenas tend to focus on the model rather than the harness. But maybe that should change.

Maybe we should be evolving in that direction.

And I think Code Arena is a good example of an arena that would support—

A full-featured harness like Devin.

Yeah. And so, in my view, if I’m talking to Cognition, I’m saying, “Hey, let’s get Devin on the arena and figure out how to loop the harness together so that we can—” I’m sure there’s something that could be really valuable there, especially given Devin.

Last week, people were talking about Devin being dead.

Did you see that?

Yeah. People were saying Devin’s gone. Devin’s not gone.

Devin’s everywhere.

Well, people were saying Devin was dead. So, can we highlight that for people and show them, “Hey, Devin is actually the best, or one of the best in the world, at doing what it does”?

LMArena can actually do that. Our place as a central evaluation platform allows that to happen.

Yeah, love it. All right. Thank you for owning the state of the art.

Thanks so much, and congrats on a wonderful year.

Appreciate it. Congrats to you, too. Congrats on all the growing momentum in your podcast and in your career. Thank you.

It’s really impressive to see.