Claude Plays Pokémon 是如何做出来的

Hey everyone, welcome back to another Latent Space Lightning Pod. This is Alessio, partner and CTO of Deel. Swyx isn't here today, but we have a special co-host, Vibu, who, if you're part of the Latent Space community on Discord, you've definitely seen. Welcome, Vibu, as a co-host for the first time.

What's up, guys?

We had David Hershey from Anthropic on today. He's the person behind Claude Plays Pokémon. It's funny: as we were first DMing about playing Magic: The Gathering together, people were like, “Of all the different nerd angles you can get me on,” and then people were like, “David is the person doing this.” I was like, “Okay, I'll DM him.” It was cool—we already had a touchpoint, so welcome to the show.

This is our second Anthropic guest; we had Eric Schon from the SWE-agent team before. Welcome.

Thank you. Glad to be here. Excited to talk Pokémon.

Let's give a little background on this. Claude 3.7 Sonnet came out a couple of weeks ago—I don't know, time goes by. Monday this week? I don't know, man; it feels like 2 weeks ago. Then you had this Claude Plays Pokémon thing that kind of went viral.

If people remember, there used to be this thing called Twitch Plays Pokémon, where people could go on Twitch and type in the chat, and then the system would figure out the next action that the emulator would take. What you've done instead is give it to Claude and basically have Claude figure out how to walk through it.

I'm looking at it right now. So far, it's been stuck in Mt. Moon for 52 hours. Poor guy has probably encountered 15,000 Zubats. Let's talk about what gave you the idea for it—the origin story—and then we can go through the implementation.

1. The Pokémon Origin Story

I started working on it in June of last year. I work with customers at Anthropic, and I really wanted to have some way for myself to experiment with agents in a real way—a framework, a harness where I could go to town, try different things, and see what actually worked to get Claude to do pretty long-running tasks in general.

I had that on the one hand, and then I asked, “What is the thing that will make me the most addicted to making this work? How will I grind the hardest actually trying this?” Pokémon was a pretty clear answer. Someone else at Anthropic had tried once to hook it up, so I had a little bit of the shell of what I needed to actually put it together and kick off what became a bit of an obsession for me in the coming months.

I played with it in June and was trying things out. Claude 3.5 Sonnet had just come out in June of last year, when I started kicking it around. It was very good, but you could see signs of life, though not much really happened.

Ever since then, as we've released new models, it's sort of been the way I get to know one of our new models a little bit. We released the new version of Claude 3.5 Sonnet in October, and I used this to really see what it was better at. It got better: you could see it start to get out of the house somewhat reliably, which was not always true, and it got a starter and even named it sometimes. It was doing stuff—not great, but it could move.

Along the way, we created a Claude Plays Pokémon Slack channel. As I posted GIFs and bot updates, it slowly grew in popularity into a cult following internally of people who were somewhat interested. A couple of weeks ago, I was testing an early version of Claude 3.7, and you could just tell it was a little different.

As you said at the top, it's in Mt. Moon for its 50-somethingth hour. This is a little bit worse than average from what I've seen so far, but it's more or less on brand. It doesn't really have a great sense of direction, and it's pretty bad at seeing things on the screen, but it plays the game. It gets Pokémon, it catches Pokémon, and it caught its first Pokémon. It got out of Viridian City the first time. A whole bunch of stuff happens for the first time, or you could squint and see a thing playing the game.

Internally, the updates were very fun. People were going wild at the fact that this was actually happening, and it was entertaining enough that I could kind of see it. The other side is that we finally got a sense that this was an actually useful way to measure what was going on with the model. There's one thing when it's fun to follow along, but internally, I think we got more of a sense that you could actually use this as a bit of a measuring stick for what's going on in the model.

I've spent—I know how many hours—staring at Claude Plays Pokémon. I've had to see and read millions of words that Claude has generated in the course of playing Pokémon over the last 8 months. You can get a feel for what's actually going better, what it's getting better at, and that kind of thing.

With this particular release, I think the fact that it got this much better at this kind of task reflects a lot of things that we wanted to be true about the model in the first place. Those things lined up, and we thought, “Okay, maybe this is an interesting way to actually tell people about what's going on here,” for a crowd that maybe doesn't know quite as much about software engineering and all the other ways we've told people about agents in the past.

Were there any other games that you considered? To me, it seems like Pokémon is good because it's, you know, isometric—it's kind of flat, so you can easily score it—and it doesn't have too many hidden facts about objects. Everything is described. Did you consider anything else, or was Pokémon by far and away the first choice?

I didn't, mainly because Pokémon was the first game I ever got as a kid. This is purely coming out of my own nostalgia. Twitch Plays Pokémon was also something that I cared a lot about a decade ago, or whatever that was.

Please tell me it's not a decade ago.

I think it's actually a decade ago. I'm sorry.

11 years ago. February 2014. Yeah, that is nuts. Pokémon Red is 20 years ago. Oh my God—25, at least.

For me, it was mostly just my love for Pokémon coming through here. Since then, there have been a lot of people saying, “Oh, we could do this. We could do this. We could do this.” Pokémon is actually really nice because if you don't do anything for 5 seconds, there's typically not a consequence. By the nature of doing inference on a model every snapshot of time, it's actually a pretty good game to be able to do this with.

You put together a very nice architecture diagram. Do you want to screen-share that so people on YouTube can follow along? We'll put it in the show notes if you're just listening.

Got it. I know Vibu had a bunch of questions on that, too.

Yeah, let's do it. Very straightforward questions, basically. Can we just double-click into all of it?

Yeah, yeah, yeah, it's easy. I found it off Twitch, and no one was talking about it, so I started sharing it around. I lost the original source, but basically everything in here is pure gold. The memory is a little interesting, but if you want to just go through it at a high level.

2. Inside the Agent Harness

Yeah, you got it. I want to preface this by saying that I do not claim this is the world's most incredible agent harness. In fact, I've explicitly tried not to hyper-engineer this to be the best chance that exists to beat Pokémon. I think it would be trivial to build a better computer program to beat Pokémon with Claude in the loop.

This is meant to be some combination of understanding what Claude is good at and benchmarking it—understanding Claude alongside a simple agent harness. What that boils down to is a pretty straightforward tool-using agent, from my perspective.

At the end of the day, the core loop is just having a conversation that rolls out. Essentially, you build the prompt including everything we've built up until now, call the model, receive some tool use, resolve those tools, and then talk about summarization. Basically, there are a few different mechanisms to maintain the information you need to do something long-running inside the context window.

When you think about what an actual prompt looks like as it rolls out, it's something like this: you've got tool definitions, which describe 3 tools that I'll get to in a second; a short system prompt, which is pretty boring; and instructions that basically tell the model how to use the tools. There are about 6 facts about Pokémon that I give it, along with a few corrective things that I've seen it do horribly wrong, like, “Hey, you might want to consider doing this a little bit better.” There's really not a lot of system prompting going on.

We have that knowledge base, which you referred to. I'll talk about that, but this is the main way it stores long-term concepts and memories as it's operating over time. The bulk of things is this conversation history, which is a chain of tool use. There are no user interjections at all, for the most part. It's just: go, the model uses the tool, it gets a result back, it uses another tool, and it gets a result back.

Cool.

3. The Vision Problem

Most of the meat of this is in the tools themselves. When you think about what's going on, it can really press buttons and mess with its knowledge base. I'll talk about Navigator separately because it's a patch for how it deals with some of its vision deficiencies.

Using the emulator, it can execute a sequence of button presses. It can say, “Press left,” “Press right,” or whatever, and it gets back a screenshot and a screenshot overlaid with coordinates of the game. These coordinates are used for the Navigator tool, which I'll dive into in a second. It's basically there to help Claude get a slightly better spatial sense of what's going on on a Game Boy screen.

Does that come with the emulator, or are you adding this in?

I add that in. I've somewhat extensively reverse-engineered Pokémon Red by this point to extract roughly every bit of possible information from it. I don't use most of it, but I have essentially everything you could know about the current state of the game exposed programmatically so I can tinker with it at this point.

I was just reading this diagram: “You get what spaces are walkable based on what's stored in RAM,” and I'm like, “Oh, you definitely reverse-engineered this.”

The good news is we also released Claude Code this week, if you saw that. This would all not have been possible without having Claude Code also go figure out how to do all this for me, because I could have done it, but there's a lot of tedious hardcoded addresses in a memory map that I had no interest in converting to a Python program. Thank goodness for Claude Code.

It gets these 2 screenshots. It gets a small blurb of state, which I read straight from the game. There's a lot of this here, actually. The thing that matters is location. Claude will pretty aggressively hallucinate that it succeeded in transitioning between zones if you don't tell it that it did not.

This just comes down to literal vision issues, and most of the patching of extra help I've given it has been attempts to make it so it could still play despite not being very good at seeing Game Boy screens in particular. Then it gets a handful of reminders. These reminders do a decent amount of work, but they're things like, “Remember to use the knowledge base occasionally,” and we tell it if it gets stuck.

If you detect that it hasn't moved in 30 steps or 30 time steps, you give it a reminder. I once saw it see a red box on the screen that was the door and think it was a text box, then spend 12 hours pressing A over and over to try to clear the text box. You see that happen once, and you add in some helpful reminders not to do that.

How much knowledge does the model have about the game itself? For example, types, right? It doesn't know about type weaknesses and things like that, or how much are you trying to put into it?

Yeah, if you go to Claude...

It will tell you about some things. I have not yet decided if the knowledge it has about Pokémon is helpful or harmful toward its playing the game. Half of the time, when it says, “Oh, I know this about Pokémon,” it uses that to hallucinate something.

For example, at the beginning of the run on Twitch, you saw it go out of the lab, see this NPC in the bottom of Pallet Town, and say, “It’s Professor Oak. I found him.” It’s very much not Professor Oak, but the fact that it has indexed on this concept is a little strange. It’s stuff like that where it’s unclear to me where it is, but it clearly has some information about it. There are a million game guides sitting on the internet, so it’s unsurprising that there’s a decent amount of information there.

I don’t really give it a lot of extra information. It picks things up. I watched on the stream the other day as it tried to use Thundershock on a Geodude, and it failed. It was like, “Hmm, I forgot about that. That does not work.” So clearly, it knows some things. It’s not perfect, but it picks some things up as it goes through the run. Ideally for me, I think it’s just interesting to see what it actually learns as it’s playing. The more it does that, the more I’m actually interested in it.

Yeah, one of our Discord members, Jung, had a good question about the sense of self. Sometimes it gets confused about who the actual playable character is in the scene. How do you steer that?

Yeah, I think sometimes it gets confused. That can be applied to many things in Claude playing Pokémon, particularly when it’s trying to look at the screen and understand what’s going on.

I’ve attempted to prompt it in all sorts of ways: “You are at this exact coordinate, you’re in the middle of the screen, and you’re wearing a red hat,” and things like that. That’s all neat, but Claude doesn’t particularly understand the middle of a Game Boy screen and a whole bunch of concepts like that. You can prompt all around it, but this kind of spatial awareness—where something is with respect to something else—is something that Claude is still just not great at in its current incarnation. Sometimes it loses track of who it is on the screen and thinks there’s something else there.

I’ll keep tracking through this. I hinted at this other tool that I give it called Navigator. This is the only other patch that I have for the vision issue. Navigator basically lets Claude say it wants to go to one of the coordinates that we provide in the screenshot, and then we automatically press the buttons to get there.

It has to be something on the screen. I’m not trying to let Claude navigate a whole map by asking it to play sight unseen. One thing you’ll notice if you run it without this tool is that if Claude wants to get from one side of a wall to the other side of the wall, it happily just tries to walk through the wall repeatedly because it doesn’t quite have the concept of what’s between it and the destination. I spent a lot of time prompting around this, and it just isn’t one of those things it’s very good at.

In order to make it somewhat fun to learn from Claude playing Pokémon at all, we use this Navigator tool, which helps it actually get around a little bit better.

We covered a bit about the different tools, the prompting, and the strategies. I’m curious how many tokens all this is using. There’s a part to the conversation history and truncating parts of the messages in state, but at a high level, how many tokens is this using? And can we go into where those are coming from and what’s being truncated?

4. The Context Window Tradeoff

You got it. When you think about the prompts here, essentially every step, something that looks like this gets sent. I’ll just go through what each of these looks like.

The system prompt is probably 1,000 tokens—pretty small, just a handful of paragraphs. I let the knowledge base get up to about 8,000 tokens. I put an arbitrary cap on it so Claude won’t write a whole bunch of BS in there if you just let it keep writing stuff. The cap helps constrain it to try to think about what’s actually important.

The conversation history is kind of finicky, but it basically rolls out 30 messages. That’s something you can tune. I’ve tuned it to 30 messages because that’s about the best performance I’ve gotten. What that means is that it basically uses a tool, gets a response back, uses a tool, gets a response back, and it’s allowed to do that 30 times. Then it triggers the summary, which takes that conversation history, summarizes it, makes it the first user message, and then we roll back out again.

The bulk of the tokens ends up being in the conversation history once it’s at its longest. In fact, most of that ends up being the screenshots, which are scaled up a decent amount to fit in. I do allow it to see a number of previous screenshots, but not all of them, because it ends up being a ton of context if you let it see even 30 turns’ worth of screenshots. I trim out a few. That’s where the bulk of the actual tokens are.

In practice, this rollout ends up at a maximum of around 100,000 tokens. That’s the longest message you ever send to the API on one of these turns. It will fluctuate with summarization, depending on the state of the knowledge base, probably between 5,000 and 100,000 tokens.

Is that per state of the game? Do you have a high-level ballpark estimate of how much this would cost to run? Let’s say people want to compete.

I think you’d really want to think about running this as a side project in terms of the impact on your personal wallet and how much you care about Pokémon. It’s not clear to me that, without the blessing of Anthropic, I would have decided to take on this project for my own wallet’s sake, especially if you want to experiment and try 10 different things.

It’s costly. I haven’t spent a lot of time on the exact number. It’s not that hard to estimate; I just told you a bunch of numbers, so you can kind of back it out. But to do a lot of experimentation, there are at least thousands of dollars of tokens being consumed. It is not a cheap rollout.

Yeah, but in the scheme of how some people use tokens, it’s not terrible. How many turns are you keeping in memory before you summarize?

Thirty right now. I’ve tried more and less. One thing you see a lot when you talk to people building agents is that there’s some effective context length that actually makes the model the smartest. That seems to vary slightly from model to model, but for this model, for whatever purpose, 30 messages worked better than 20 and better than 40. So it’s somewhere in between those where it worked pretty reasonably.

Does that change based on location? How many would you want to give it to get it out of Mt. Moon? We’ve got to bring Claude home; we can’t let him stay in there for another 57 hours.

I’m actually not sure it does. I’ve tried posting 20 or 30 screenshots at a time for it to be able to see, and it’s not obvious that this temporal concept is actually super relevant to it.

Again, trust me as someone who has spent a lot of hours obsessing over this: you can try to prompt Claude in a lot of different ways to understand how to navigate better, and anything short of telling it exactly what to do does not work. Actual navigation is just not a skill it’s great at. It’s good enough to random-walk its way through some complex mazes, and in good, easy areas it’s pretty good at popping around.

I could tell you if there were a way to prompt this slightly differently that would navigate better, and I would believe there is something, but it is not an easy lift.

I just asked Claude right now, “How do you get through Mt. Moon in Pokémon Red?” It does have a plan, but I don’t know if it’s the right plan. I’ve seen it come up with a lot of answers to that question, and most of them are right.

This is part of the pain when I talk about not being sure if its knowledge is better or worse. You see it say, “Oh, I know the exit is on the eastern wall,” and then it just spends 12 hours trying that. It’s unclear to me that we’re not just harming it by having it think it knows the answer.

Yeah, I think that’s the interesting part, right? You don’t want it to just know the answer. The model clearly knows a lot about the game. EV/IV-maxing Pokémon would be very extreme, but if that’s what you wanted, we could just hook it up to a knowledge base—hook it up to a guide for how to beat Pokémon Red.

The interesting piece here is actually: can it figure out what to do without just memorizing the route?

That’s exactly right. Part of what I’ve realized by putting it out in the world is that people will draw their line of where purity is anywhere on the spectrum. Is this cheating? Yeah, maybe. Who knows? Frankly, I don’t particularly care.

The main insight that I have is that when we put this out, you learn a lot about what the model is good and bad at by staring at it. That’s what I like about it. Evaluating the model is kind of separate from your emulator and how it can use an emulator, right? We can always improve those things.

I’m curious: as you switched from Claude 3.5 Sonnet to Claude 3.7 Sonnet, were there any regressions? Did it get worse at anything? And was the prompting somewhat consistent? A lot of what we’ve seen with different reasoning models is that you prompt them differently, right? You tell them what to do and let them figure it out. Any insights there?

5. Why Simpler Prompts Win

Yeah, that’s a good question. One thing that’s nice about Claude 3.7 Sonnet is that it’s this hybrid reasoning model, so it can kind of do the old thing and the new thing. It’s actually pretty good at just being an out-of-the-box model and having this thinking mode where it can spend time reasoning. I didn’t really run into any serious degradations.

The one thing I’ll say is that literally every model that has come out for Pokémon—the main change I’ve made to this agent is deleting prompt stuff. There’s a whole bunch of Band-Aid prompt stuff I’ve added in the past, trying to steer it away from doing many of the things that it got horribly stuck doing. As the models get better, I’ve found that making sure it’s as simple as possible and giving it as much free rein to try to solve a problem as possible is useful.

The way I think about this is that I’m less confident over time that I understand exactly how a model is intelligent. It’s capable of all these ridiculous things. It does PhD-level stuff in some ways and is unable to read a screen as well as a 4-year-old in other ways.

My confidence in exactly what I need to tell it to do to be smart at playing Pokémon is really small right now. If I tell it, “This is the way you need to solve this problem,” that might not actually be the best way for Claude 3.7 Sonnet to solve the problem. It’s just different from me in terms of how it thinks about these things.

I’ve found that pulling some of the unnecessary instructions—where I tried to use my intuitions about what would make the model better—out of the prompt over time is the thing that has consistently gotten more juice out of this as the models have gotten smarter.

I was watching the stream yesterday or the day before, and it was a very tense battle. I think they were down to 2 HP each, and the opposing Pokémon missed a Scratch or something, and it didn’t die. You could tell Claude was like, “Wow.” It was very dramatic.

I was thinking about the game: is there any thought being put into trying to have it be more rational? Do you prompt it to be more rational, to let it know that it’s not real life, that it’s a game? It feels like it gets very distressed when the Pokémon are actually going to die.

It’s funny. It knows it’s Pokémon. It’s like, “You’re playing Pokémon Red.” It does know that; it has a sense of that. But it clearly has some attachment.

I’ll tell you a fun story. We tell it to nickname its Pokémon. It will occasionally do that without being told, but it’s more fun if it nicknames its Pokémon, so that’s in the prompt: “It’s fun if you nickname Pokémon; you should consider it.”

One thing we found when we started doing that is that it got more protective of the Pokémon it nicknamed. It’s pretty obvious: when it catches a Pokémon and gives it a nickname, it will go heal it right away if it’s hurt. That never happened before, which is pretty interesting.

There are some cute little quirks about Claude, who really wants to protect its precious nicknamed Pokémon, which is great.

I will say it’s kind of normal. When I was 5 playing Pokémon Red and I had 2 HP in the midst of a Scratch, that meant everything. That was existential.

I agree completely.

6. Transferring Lessons Across Games

How about skill transfer? One question I had: you’re playing Pokémon Red, right? Say you want to play Pokémon Silver or Gold next. Have you thought about how models can learn from these games, store these learnings, and use them again in the future? I’m sure it’s not part of the project today, but I’m curious about your thoughts.

I’ve thought about it only a little bit. I think there’s some interest when you actually read one of the knowledge bases it has gained on some of the longer rollouts, when they’re good. There are actually some pretty decent tidbits about how it should act and try to do things, and some of the ways it’s succeeded.

One of the things that’s most unique about Claude 3.7 Sonnet that I’ve seen is that it will have meta-commentary on what it’s good at and bad at in its knowledge base: “I misperceived this thing, and so I need to be careful doing that again.” You occasionally see that show up there, which is pretty cool.

I could imagine there being some way to translate that knowledge base from one game to another. I think my knowledge base is frankly kind of clunky as an implementation right now. It’s more or less a Python dictionary that’s appended to the prompt, and I think you could find better ways, if your goal is to transfer across games, to manage a knowledge base that Claude can actually use better in different scenarios.

There are definitely pieces there that I think would get it off to a better start on the next Pokémon game. Or even if I restarted the stream, it would have some tidbits that would probably speed it up if it had access to things that it learned in the past.

That’s interesting. I always think of that in card games. You have the idea of tempo in a card game, and it’s the same in Magic as it is in Star Wars, Flesh and Blood, and all these different things. I feel like games are similar, where learnings you get from Pokémon can carry over to similar kinds of open-world games.

I think it’s also particularly interesting for some of the things around how Claude learns to play a game in general. Pressing too many buttons at once is a bad idea. “I lost—what’s going on?” That kind of thing.

There definitely is stuff it has learned that’s interesting in a meta way, and it’s hard to give it that sense of self necessarily in training. Sometimes it’s hard for it to know what it’s good and bad at in some scenarios, but it’s interesting to think about how it can learn across things.

Some of this is also due to the emulator, right? A lot of what it’s learning is, “How do I use an emulator? What am I good and bad at?” But the model internally should know quite a bit about Pokémon.

If you’ve played Pokémon, going from Pokémon Red to Emerald to Diamond, having played the first one doesn’t help you that much in the second. You get the general concept; you get what types are good against other types, and the model knows a good bit of this, right?

But it’s still interesting because it shows that knowledge bases help with understanding how to use the emulator. It struggled, and then it figured it out. So even though I know Pokémon, it’s this thing where it will learn how to use the emulator.

Yeah, which is pretty cool. That has been part of what’s been fun about seeing Claude make progress on this thing.

I had a bit of a follow-up question to the last one with Alessio. If people want to blow thousands of dollars and want to improve this a little bit, is there anything else that you’d want to see done, whether that’s improving the emulator or trying different things? Is there anything that anyone watching this could work on, or anything you’d hint them toward?

7. The Road To Victory

No doubt. If I had to guess, the biggest lift that exists around this is probably something around the memory, which I don’t think is hyper-optimized right now. The nice thing about the memory is that it’s always in the prompt; it doesn’t go away. Sometimes, if you leave it up to Claude to try to read, load, and save to knowledge bases, it will underutilize them, even for useful things. But I think there’s probably something there.

I will say that, in all of the many, many hours I’ve spent tweaking around the edges of this thing, nothing quite does it like a new model, though. Fundamentally, I think the limitations right now are mostly around some of the smarts. I’ve seen a lot of people on Twitch tell me about ways they could fix the navigation capabilities with a better prompt. People would be welcome to try, but I would guess that would be a somewhat fruitless avenue. I think it’s just not very good at understanding things the first time.

I’ll give you a very quick anecdote, which I think is my favorite example of why this is particularly hard. I have this clip of Claude leaving Professor Oak’s lab and saying, “Great, I left Oak’s lab. Now I need to go up to the north end to go to Route 1.” It just hits Up on the D-pad and goes straight back into the lab.

It’s like, “Shoot, I’m back in the lab. I need to leave,” and it hits Down. It’s like, “Great, I’m out of the lab. Now I can go up to Route 1.” It’s straight up. It just goes up and down 12 times, and you’re not fixing that with a prompt. It literally doesn’t get it; it doesn’t understand. So it’s pretty hard to make little around-the-edges changes that make a huge difference.

I’ve always been fascinated by the fact that Twitch Plays Pokémon actually beat the game. You just look at it and you’re like, “This cannot possibly work,” because you have people trying to sabotage it in the chat, too. Not everybody’s trying to solve it.

So I just looked it up: it took 16 days and 7 hours for Twitch Plays Pokémon to beat Pokémon Red. How close do you think we are to a model that can beat it in less than 16 days? And do you think it needs some core model, really big jumps, or do you think we’re close?

I think there is model work that needs to happen, at least from Claude. I’m confident there are model improvements that need to happen for it to be really capable.

I have 4 spots in the game stuck in my head where I think there’s literally no hope it’s going to get through them. I think there’s a gap that’s mostly around its ability to see, navigate, and remember visually what’s going on. I just don’t think we’ve figured that out yet. To me, that’s a pretty big gap.

I do expect it to keep getting better. I have no reason to believe that this isn’t just a fundamental ability to scale, learn, and understand problems—something that I think is getting better as we train models to be more capable at these long-horizon tasks. I actually do think this is a pretty reasonable proxy for that, and I think it will continue to get better for a little while.

I don’t know if there are affordances around images and videos and things like that that we need to figure out to make it work. It’s unclear to me if that’s true or not. But yeah, I think we have a little ways to go before we can beat the game in 16 days. I do not have a lot of faith that the current stream is going to be standing in Victory Road in 13 days.

What’s been your favorite moment from building this—from thinking of the idea to just seeing it play? Any major highlight?

I think the hypest I’ve been is when it beat Brock the first time. I was just like, you know, I’ve been doing this for 8 months, and then a few weeks ago I kicked off a run, woke up the next morning, and it was like, “Oh my God, oh my God.” The other good thing about it was that I woke up at 8 a.m.

And I have it send me updates to Slack. These are ridiculous things, but it was literally about to start the Brock battle. I opened up my phone, and it was like, “Oh, this is happening right now.” It was a pretty hype way to start a day.

I think that was my highlight. I have a lot of other cute things, like some of the cute nicknames it's come up with over time and things like that, which are endearing, but that was the peak hype for me: we beat a Gym Leader, we've got a badge, Claude's doing it.

A bit of a follow-up: I noticed that you mentioned it eventually started beating multiple Gym Leaders. Were these all the same run? Was it different runs?

Yeah, the run that you saw on the graph we put out alongside it in our research blog is a single run that I have watched get through at least Surge's Gym. Then it got a little past that, and the reason that's where we stopped reporting is because that's the physical amount of time that occurred between when I started it and when we launched the model. That was a very hyper-up-to-date graph on the best run we had.

Awesome. I know we're running out of time. My last question is: are we going to work on Claude Plays Magic next? Or maybe we can do the Magic Arena intro challenge?

Yeah, funny story. There was a project I did right before I joined Anthropic that was training an open-source model to be slightly better at picking cards in a draft. I was training it on the 17Lands data that exists to learn how to pick cards out of packs a little better.

I did talk about that in my interview to get hired at Anthropic, so if I've put time into this, I'm ready. I am ready for that project too. I have that code sitting around somewhere as well. I'm really getting into all my nerdy ML and gaming hobbies here.

Yeah, no, I'm ready. I don't know if you're planning on open-sourcing any of the Pokémon stuff, but if you want to work in open source on the Magic stuff, I'll be happy to collaborate.

Awesome. We've talked about it. I don't know yet what the plan is. There's a certain amount of “this is not my day job” that I have to figure out how I want to deal with. We'll see.

Awesome. David, any parting thoughts? Anything people have missed?

No. I think the one thing I do like to drive home when I've been talking about this is that I really do think this is just demonstrating a thing that is going to make agents better with this model. This is a very fun way to see it, but I think the thing is that it has some ability to course-correct, update, and figure things out a little bit better than models have in the past.

Even if there's stuff it's dumb at, it tends to have an ability to power through it in a new way. What excites me is that I think there will be some real-world stuff that comes out of this model once people play with it. I'm pretty excited to see how people take the skills we put on display a little bit here—or lack thereof in some cases—and figure out how to turn them into actual agents that do stuff.

I have a quick last question on that, actually. Is there any guidance, or any way that you quantitatively measure the evals of this system? A lot of it is vibes, a lot of it is how far it gets and where it gets stuck, but are there any lessons or specifics about how you measure how it actually does?

I've done a lot of small tests of putting it in a scenario and seeing what it does, but frankly, the best test I have is just running it 10 times on this configuration and seeing how quickly it progresses through milestones of the game.

I mean, it's the best thing about games, right? It's why games are such a useful thing: there are literal benchmarks of Gym badges that are moments of progress in a game, which are ways to evaluate what happens. I think how quickly it's able to make progress is actually a pretty reasonable eval, if a slightly expensive one to calculate. It's an integration test, not a unit test.

Awesome. David, thank you for joining. Thank you, Vibu, for filling in on the host side too.

Yeah, my pleasure. Thanks for having me, guys. I appreciate it.

Awesome. Good to see you.