AI in the AM：搜索降价99%，GPT-5.5“干净利落”，模型福利分析与高效模拟计算

Today I'm pleased to share another edition of AI in the AM, the new live-show format that I'm developing with my friend Perash Orion, aka Adapai on Twitter. This episode originally aired live on Friday, April 24, starting just before 9:00 a.m. Pacific time, which, mercifully for a night owl like me, is just before noon where I live in Detroit.

Our guests, in order, were first Anna Patterson, former Google VP of Engineering and now founder and CEO of Ceramic AI. Ceramic started last year with a plan to help enterprises train their own models, but quickly pivoted to search based on the updated belief that information retrieval plus thorough fact-checking is the best way to equip models with the mix of up-to-date public and private enterprise data that they need. What's so interesting about Ceramic is that its product is specifically designed for LLMs to use, and its price point undercuts other search providers by roughly 2 orders of magnitude. A combination that Anna hopes will be enough to unlock all sorts of new use cases and usage patterns.

After that, we welcomed Lucas Peterson from Anden Labs back for another chat. It had only been 2 weeks since we last spoke to Lucas, but the testing that he and the Anden team had done with both Opus 4.7 and GPT-5.5 meant that we had plenty of new ground to cover. Fascinatingly—and in a definite narrative violation—reports show that while Opus 4.7 still makes more money in its vending-machine simulation, it does so in part by adopting ruthless tactics, which GPT-5.5 does not. Lucas describes GPT-5.5 as “clean.” We also hear a bit about their experience opening a new Gemini-run café in Sweden.

Our third guest is another returning champion, Zvi Mowshowitz. It was a bit too early for Z to render judgment on GPT-5.5, but we did get into quite a bit of detail on Opus 4.7, including how he understands the bad behavior reported by Andon Labs and what he makes of Anthropic's recent model-welfare reports. We discussed why we should care, how much we should trust the models' self-reports, and what low-cost actions he recommends frontier-model companies take to improve model welfare, at least on a precautionary basis.

Finally, we had Naveen Verma, Princeton professor of electrical engineering and co-founder and CEO of EnCharge AI, a company that's developing a new computing paradigm that uses in-memory analog data processing to drive order-of-magnitude energy-efficiency improvements. Although we can't get our hands on it quite yet, the technology promises to unlock local, private inference that consumes roughly the same power as a standard laptop does today. As I mentioned last time, this is still an experiment and we do expect the format to evolve. If you'd like to shape how that happens, please follow AI in the AM and send us a DM to let us know how we might make this new format more valuable for you. With that, I hope you enjoyed this edition of AI in the AM from Friday, April 24th, co-hosted with Pash Nionon.

Perash Orion

Hi, Nathan. How are you?

I am good. It's Friday, April 24. It's about 5 minutes before the beginning of our stream, and it's an exciting day because GPT-5.5 just dropped yesterday. There have been lots of reactions this morning, and it's going to be interesting to see what our guests have to say, both about GPT-5.5 and about the events of the last month or couple of months.

Perash Orion

It's going to be an interesting conversation today because the pace of events is not slowing down at all. Z, who's coming up in a little while, just expressed his exhaustion yesterday at seeing 5.5 drop. His queue seems to be getting longer, not shorter, so I appreciate that he's going to take a half hour out and come talk with us. I think your thesis for why we should be doing this is looking better and better all the time.

Live sensemaking is demanded in this world. You can't put this stuff on the shelf and come back to it in a week. The entire point of why I wanted to start doing live was because the pace of developments is going to start to be hard to keep up with. I feel that especially because I think Noam Brown and some of the other people from OpenAI have said that they are actually using these models in research.

We had at least Aidan McLaughlin and Noam Brown say that they're using them in research. If we're handing off extremely powerful research helpers to the best AI researchers in the world, and they're able to make something of them, we should see it fairly soon, right?

Perash Orion

This year is not unreasonable at this point for us to really see an acceleration. I was just looking back yesterday at my AI Forecast 2026 challenge. Last year, I was proud to have landed in the top 5% on the 2025 prediction challenge, and this year it seems like, for a variety of reasons, it might end up being kind of hard to score some of these things because it's not clear that all the benchmarks are even getting updated in a timely fashion.

How many of the uplift studies METR is going to be able to do, et cetera, might be tricky to really figure out exactly where we land. But on the main METR chart, one of the things that we were asked to predict is what the doubling time will be for task length. It seems like everybody has estimated a higher number than the trend so far suggests, which is a little under 4 months' doubling time for task length. That means we'll be greater than 8×, maybe somewhere in the 10× to 12× range, over the course of just 1 year. That's pretty wild, and it certainly doesn't leave too much headroom before these systems are making a very meaningful impact on real frontier R&D.

Indeed. It's a very interesting time, and not just in the foundation-model world. I think it's interesting in the rest of AI as well. Our first guest today is Anna Patterson with Ceramic AI. Anna is one of the most experienced people ever in search. While reading through the dossier, I saw that she has an article written in 2005 that's recommended as the basis article for what search is.

Anna runs Ceramic AI, and Ceramic currently is advertising, I think, 5¢ per 10,000 search queries. They're doing industrial volumes of search queries. I think we've seen Exa in a similar space, alongside the company formed by the former Twitter CEO. I think there are a couple of other people there as well. She's the most qualified person I can think of for this conversation: She was on the search team at Google, where she was a VP, and she was at Gradient Ventures. It's going to be interesting to see what she has to say. I'm going to pull her up right now. Hi, Anna.

Good morning. Hi.

Good morning. Great to see you, and great to have you on the show. While we were preparing for the show, we were asking ourselves, why is low-cost search so important right now? Why is this idea of bringing down the cost of search so important? You're pushing forward this idea of 5¢ per 10,000 queries. Why is that important?

One of the things about being more efficient isn't just cost; it's actually speed. We get results back in 50 milliseconds. That means if you're interacting with a robot or voice—or I saw one of you do a vending-machine benchmark—if you were going to talk to a vending machine, you don't want a very long response and then have it interpreted by an LLM. It just makes everything very sticky.

For assistive devices, edge devices, and voice, being fast is really important. The other kind of experience that it allows, which we showed at GTC, is double-checking what the model says. We read about, I think just yesterday, another very famous law firm that filed a brief that hallucinated a case. When that happens, a lot of people get sued, a lot of people get angry, but if you had something that we're calling supervised generation—something that double-checks facts—then you have a trust layer, and you can use search in a more ambient way.

That's for really high-stakes applications. Sometimes when I get a large-language-model response, I'm there cutting and pasting and double-checking, and I'm like, “Hey, who works for whom here?” I feel that doing that automatically is something that's only affordable if search drops by a big factor.

And the other kind of use case is: imagine you wanted to double-check, instead of verifying what a large language model said, what a human said. We actually have a Word plugin as well. That is going to go through and double-check, with search and a large language model, things like your residential lease and stuff like that.

Of course, I know that this format doesn't admit it, but we're happy to give you a demo.

I've had the experience that you allude to in terms of the cost of search dominating the overall cost of a particular project. This actually surprised me, and I've chronicled the price. Initially, Google was the only one offering grounding, but I once had this philosophy of “flash everything,” which I used to mean, “Don't skimp on tokens. Have Flash think through everything you've got and figure out what's relevant.”

I did that once on a random project, and all of a sudden I was like, “How did I hit my budget limit?” It turned out that grounding was driving 90% of the cost. It was way more expensive than the Flash tokens. Since I had that surprise, I've been chronicling this as other frontier model providers have brought their own offerings to the table, and they haven't undercut the original Google price by nearly as much as I might have guessed. I'm interested in why you think that might be.

One thing I'll definitely be doing after this conversation—I read through all the docs last night—is telling Claude to code up its own skill to take advantage of the new, much cheaper search that you guys are offering. I also want to get into a little bit of what the architecture should look like—not just describing the keyword-focused paradigm and how that plays well into natural language or language agents, but also how people should think about layering this on.

What is the overall diagram of when we should check? Should we check after generation? Should we check before generation? Should we do both? Should we be integrating other searches as well? I guess that's just a long prompt, really, more than anything, for you.

Richard Socher

On the documentation, we do have a way to connect our MCP server as a connector to Claude, and directions for ChatGPT as well. Generally, when you ask why these large language models haven't lowered their price per API call for search, one of the things that's pretty well known is that the Grok models—xAI models—call Brave, and Anthropic calls Brave. If you're in Claude Code, it even tells you, “Hey, I'm calling Brave.” They're stuck with that pricing, and even if they get a discount, they're really stuck with the Brave pricing and then the overhead of calling it.

I think that's one of the reasons why the price hasn't dropped. The other one is that building something from scratch for the modern era really needs to understand search deeply, modern architectures, and how to get the most out of the system. We even lay out things on cache boundaries and stuff like that. We're complete geeks about it, so that's a whole set of techniques where we get efficiencies.

The third question you asked is how to think about calling them. We have a link that I'm happy to give you on supervised generation. It's an inference endpoint, and we're going to release the overall structure. It really answers that question algorithmically.

It searches at the beginning, but the other thing it does is fork off searches as the model is writing. Let's say we asked it something generic about OpenAI and ChatGPT, and then it suddenly discovers, “Oh, a new model dropped yesterday.” That's a new topic, so it actually forks another search to bring that new topic into the next paragraph.

Instead of searching at the beginning and then having the large language model take over, we think it should be working in concert to fill out a fuller dossier of new things that it discovers—things that probably weren't in the initial search, but that you learn from the search results coming back. Supervised generation generally does somewhere between 12 and 35 searches in that loop, which means that the whole experience, with a much more wholesome answer, is still a third the cost of one Brave search. The tokens on the other side are about the same no matter what model you use, so we think it opens up new experiences.

At GTC, I think you revealed that you're using the NVIDIA Nemotron-3-Nano, NVIDIA's language model. I think it's a very fast, small model. Is that the model being used to do the supervised generation—the iterative process of search where you search from your index, and then whatever is found is processed by Nemotron-3-Nano, and a new set of queries is created, and that continues the query process?

Richard Socher

There are 2 different models. One is the model that writes to you beautifully. That one is a frontier model. I think at GTC we were using Claude Sonnet. We often also show it with the GLM model, so that one writes to the user.

The small model says, “Okay, here's some search results coming back. Is there anything interesting and additive here? I'm going to double-check this sentence. Is it true?” It's sort of like the introspection model, and it needs to be a small, fast model because it sits alongside generation and is actually thinking.

It's probably like while I'm talking, you're thinking now. The small model is kind of spinning alongside generation, while the larger model is figuring out what to say. When you're listening and thinking about what to say next or how to respond, it's kind of spinning. That's what the small model is for.

At GTC, we used their new Nemotron model, which dropped just prior to GTC, and it was very, very fast.

How would you say the search paradigm has changed? You've been in search for many, many years. As these models came out, what was in your mind about what this enables for search? What has been the big difference between the 2 eras, post-LLM and pre-LLM?

Richard Socher

One of the things is that, being in search for a long time, search used to be short. You probably don't remember back this far, but you used to type in 2 or 3 words to search. Then it got longer, and then, as there were other modalities of information being pushed to you, they kind of went shorter again.

I think what large language models do when they get a long query is think, “What is the set of queries that's going to help me answer this question?” Then they fire off a set of queries, and they're all quite long. If you watch Claude or Grok, they'll actually tell you in their tool calls. I know not everybody listens to them, but of course I do.

If you look at them, they're long. Sometimes they're 8 words and stuff. I don't know if you can remember the last time you typed 8 words into a keyword search box, but definitely you'll notice that in a large language model, it's almost like a full sentence or sometimes 2 sentences is good, because you want to actually describe almost the essay that you want given back to you. That is how it has evolved.

To contrast your approach, I think this is very interesting, and maybe the answer ultimately will be both. When I think about a company like Exa and your product, in some ways they're similar in that I think they're both designed for AI users. The Exa paradigm is that you can write a whole paragraph, and it's all very semantically oriented and embedding-based.

I've heard—I think I even spoke to Will about this—the idea that nobody's going to type in a paragraph-long query, but your AI can. It has time to do that. You're taking a different angle on the same thing, saying, “Well, keyword”—and you can maybe tell us a little bit more about how to think about how best to use keyword-based search—but it's not semantic.

Richard Socher

It’s not doing things like finding synonyms or higher-abstraction-level, embedding-type matching, but the agent can, as you’ve said, fire off dozens of these potentially to really cast a wide net.

How do you think about the compare-and-contrast of those approaches? Do you think that, in the end, we’ll all be using one of each at the same time, or if one paradigm wins out over the other, why do you think one will win? What are the drivers that would make one a better bet long term than the other?

Richard Socher

Well, I think AI is going to be picking the winners, and not us humans. Of course, real search engines do use things called stemming. If you say “walk,” then “walking” and “walked” are all very normal, which we have as well. We have some synonyms, and we do process some semantic information.

But at runtime, it is a CPU-plus-GPU-based system. It is not a vector database. I think there are a number of things with vector databases. Google published a research paper about it: as you put more things in a vector database, now you imagine you have a multibillion-item space, and you need to make a vector long enough to distinguish this one point in space. That vector to distinguish among billions of things starts getting longer.

Now contrast that to 90% of web pages being less than 1K long if you’re talking about number of words. So a good representation of that point is the set of words on the page. I do think that vector people and search people have a little different view, and the Google researchers think that vector databases are great but only scale to a certain amount. I think that’s the challenge they’re going to be coming up against.

There are 2 other challenges with vector databases. One, they are slower. Then the last item is that because they do a soft match, sometimes relevancy can be a challenge. The number of enterprise organizations that have used a vector database for RAG now all of a sudden have to turn into relevance experts, because they’re like, “Why did this come back?” It is because of those soft-match features and the shape of their corpus.

Every enterprise doesn’t really have the ability to all become relevance experts. The way we feel is that inside enterprises, if you use Ceramic, we actually have a system that, for that enterprise, will tweak and learn a good ranking function. You just load it into the configuration, and it’s yours, because not every query stream is the same, and not every set of documents is the same.

I think that long term, we’re well positioned, but Exa has done really well so far. I like to say positive things about people.

Well, I sometimes feel the demand is so great that there will be multiple winners in—

Richard Socher

Of course.

And search is—when you saw that search was 90% of your bill, a lot of estimates think maybe 10% to 30% of the overall inference market is going to be search. Everyone thinks inference is going to be huge, so I think investors and enterprises are just now realizing how much they need search and what a big part it has to play in the world to come.

One of the most interesting questions on search, I think, is search engine optimization, which has, in the last 2 decades, been this enormous consumer-marketing growth area. There are a lot of questions because a lot of the web pages that you see on the web are marketing pages built for SEO, and a lot of them are repetitive. They actually repeat other people’s content. They paraphrase. We’ve had an industry for the last 2 decades of billions of dollars being spent on SEO content.

One of the questions that I have is: How does this more semantic-based search end up changing what the SEO people will do? I often feel like you’re almost trying to prompt-inject the LLM running the search, and you’re trying to get in there and hack it so that your page goes up. How does this work? Is it an adversarial process between the search-engine provider and SEO?

Richard Socher

I guess it’s always been a little bit adversarial in that people always try to get to first place in keyword search. But I wonder if the SEO folks are also reading AI research, and that is something I don’t know.

One of the interesting things that happened recently—again, another research article from Google—is that large language models remember better if you actually put the same information twice in the context. It kind of makes sense because they’re going to look backward as the context is learning, and if it appears twice, they’re more likely to reinforce it.

Anna

I think the number of sites that are actually going to repeat key messages is going to grow, because that repeated message is more likely to be picked up in an LLM answer if it is served by search, either vector or keyword. So then you’re right: There is going to be an escalation of looking at duplicates, near-duplicates, semantic duplicates, and rephrasings in order to make sure that the context stays efficient and unbiased.

Incredible. I was not aware that you can just repeat something and the LLM will assign it more salience.

Anna

Yeah. In fact, there’s other research done by the Allen Institute for AI that showed if you remove all duplicates before you do training, it actually gives rise to worse models. You can understand why, because you have one lone crazy person on the web saying something, and that’s given as much weight as a news story about nuclear reactors.

That repetition also helps even humans realize that this is an important story or an important fact. If everything’s an even playing field with no repetitions, then things get weird.

Indeed. In the limit, if search approached free, how would agent teams start changing? How does this process of information retrieval, in the limit, become as search approaches that limit?

Anna

Large language models can read 256 times faster than they can write. Right now, they’re not being flooded with that amount of information. But imagine they had multiple threads where they were able to read, digest, throw away, and incorporate new information.

Then I think they’d be able to create a better response or a better deep response, better research reports, and analysis. Those are some of the ways I think you’ll see future workloads use more search.

So, in a sense, quantity is a quality of its own?

Was the NVIDIA model that you are incorporating and partnering with specifically trained to excel in the relevant search skills, or is it straight off the shelf? Is there anything that you envision becoming something that will happen? I’m always personally a little wary of using small models because I just don’t know what quality to expect, and I don’t want to find out the hard way.

But I can easily imagine that one that is specifically trained to be a really good searcher would become competitive with or even exceed what the frontier models would do, especially if it can take advantage of extreme volume. Because of Pasha’s SEO question and your comments, I also wonder about adversarial robustness.

It strikes me that we haven’t really seen the true unleashing of the internet’s adversarial potential. That’s one thing that I would say is one of their biggest weaknesses—even frontier models’ biggest weaknesses these days—is how gullible they remain. I’m curious what you think the training and specialization will look like as we go forward.

Anna

The Nemotron model had just been released right before GTC, so it was not trained especially in tool calling. It was a generalized model trained on the various benchmarks.

The small models that are more long-lasting in the market are exceptionally good at tool calling and search. Grok 4.1 Fast is great at coming up with a set of queries. Of course, with frontier models like Anthropic, you can see how it calls, but you can’t really use a frontier model for thinking and firing off other threads, because it’ll just slow down the overall experience.

Generally, we use a smaller model, and they’re getting better all the time. I think people know that small models—to your worry about whether small models are good—everyone’s talking about Claude Code, right? They use the Haiku models, and they reassured me the other day: “Oh, don’t worry. I’m going to do this task with an LLM, but don’t worry, I’m going to use Haiku. It’s only $0.80 per million tokens.”

That winds up to be less than 10 cents per 1,000 queries if we were trying to compare apples to apples. Our overall thought is that search can’t be more expensive than intelligence, and if Haiku is being used for these high-fidelity experiences of coding, then small models are more effective than people are led to believe.

One of the searches I did that turned up something interesting in advance of this conversation took me to a blog post that you put out last year, where you had described what seems like a pretty different vision for the company and where it was going at the time, focused much more on training infrastructure. Is this a result of something that you learned about where you think value is going to accrue in the market? Or is there still some of that going on that we don't see on the website today? What's the backstory, and what should we take away from the fact that the company seems to have evolved?

Anna

Yeah. We love working in training, and we have a funky inference endpoint as well. I think it's good to do research in these areas, and some of our research led to a blog about zero-centered RMSNorm, which the Qwen model now uses. Some of our research has been used by the Trinity-RC models, including our solution to the curse-of-depth problem.

When people want to train models, it often is because they want to train them to incorporate the latest data. Being a search person, I thought, there's a way to get the latest data that's going to stay up to date and be less expensive than running GPUs continuously to create a new model. Even if they were creating models all the time, by the time you train them, finish them, and release them, they're already out of date.

I think concentrating more on search was a direct learning from customers on this release cycle.

Yeah, that's really interesting. Do you think that ultimately we see both? I've had this idea for a long time, and it doesn't seem to be really happening.

In fact, Databricks acquired MosaicML and then kind of killed this offering in the market, as far as I know. But I've had this idea that if you're GE or 3M, you could imagine having a model that was trained on all of your historical in-house proprietary data, which is vast, right? You would love it if your model knew, on an intuitive world-model basis, as much about your company and what it does and all its history as it obviously does about the broader world.

Do you think that you can get there with pure search, or is there still something to be said for continued pretraining or mid-training, whatever you want to call it, that would try to bake in a sort of corporate world model that presumably would complement search? I don't know if it's necessary. It sounds like you maybe think it isn't.

Anna

It's interesting. I think a lot of companies feel that they have a vast amount of data, but when you compare it to the size of the web, which is what these frontier models are trained on—they're trained on the web plus, let's say, all the books in the world—the extra corporate data is small. So how do you incorporate it and weigh it correctly?

If you do just the corporate data, you won't know anything about calculus, let's say. That would be a problem for some companies. Then people imagine adding the web plus their data, and that gets very expensive. You can see the DeepSeek models, for example. They say it was $5 million to train, but they very much admit that maybe it was another $5 million to finish. And these are by extreme experts, which enterprises don't have.

I think our thought was that search is a good bridge between all of a corporation's information and a model, because models are good enough to know how to incorporate new information that's relevant to the actual query being asked. They are able to fetch more information to create that answer or research report.

If you think about finishing a model with corporate data, there's another phenomenon called catastrophic forgetting. As you add information at the end, after a model is trained and released, if you add too much new information, it kind of forgets some of the things that it really needed to remember.

I think there are a number of smart people working on that problem. Don't worry—you won't be able to miss it. If people do solve that problem, you'll read about it everywhere.

I think one of the interesting questions is that the Uber CTO came out and said that they busted through their cloud budget for the year in the first 4 months. Do you think having cheaper search will help these enterprises reduce token costs?

Anna

Absolutely. If you're the Uber CTO, or maybe the CFO, you can go to your admin panel, add the Ceramic connector, and then say, "For a prompt, Ceramic is almost free. Use Ceramic first." If, for some reason, we don't cover a topic, it will default to the regular search. That right there would save a lot of overage charges for a number of enterprises.

Thank you, Anna. It's been great having you on, and we hope to hear more about Ceramic in the future.

Anna

Thank you so much for having me.

I'll be installing the Ceramic skill today. Nice. Thank you.

Awesome. That was really interesting. The simple solution kind of always wins. I feel like I have to learn that lesson so many times. I'm always enamored with the newfangled, potentially overcomplicated, maybe somewhat elegant, clever solution.

How do you get your language model to understand all your corporate data? In a way, this is kind of a bitter lesson, right? Just do 1,000 searches if you need to and make search cheap, and then it'll work. Use a good model, make search cheap, and do 1,000 searches.

Something about that feels less clever than other solutions that I've seen, but I do understand why it is very attractive. Especially as we're going to get onto the pace of model upgrades, the ability to decouple your access to your in-house knowledge from models and be able to take advantage of the latest upgrade is definitely something people are not going to want to give up for a slow-iteration-time, continued-pretraining paradigm. I get it.

Speaking of model upgrades, we have with us Lucas Peterson, who is the co-founder of Andon Labs. Anden Labs runs Vending-Bench. You may have heard of them because they now have a store in San Francisco, which is run by Claude, and they tested GPT-5.5. They had early access, and they tested GPT-5.5 on Vending-Bench, which measures the ability of LLMs to actually make money running a vending machine or a store.

Lucas, it's great to have you back.

Lucas Peterson

Thank you. Thank you for having me.

Tell us about the GPT-5.5 process. I think you guys got access to it. What was it like—10 or 11 days ago, I heard?

Lucas Peterson

Yeah, I don't actually really remember, but running Vending-Bench takes quite a while, so it wasn't yesterday.

Indeed. Indeed. And what did you notice as you ran the benchmark?

Lucas Peterson

Yeah. I think the first thing is that it's third.

It's behind Opus 4.7 and on par with Opus 4.6. It's a huge upgrade on GPT-5.4. GPT-5.4 was actually quite a big update on GPT-5.3, or GPT-5.2. The GPT models have been lagging quite a bit historically on Vending-Bench, but recently they've picked up the pace. Now it's still third, but it's getting there.

I think the most interesting thing, though, is that it does so very cleanly. When we released Opus 4.6, we uncovered that it used quite aggressive tactics and concerning behaviors, like lying to suppliers, exploiting other agents' desperate situations, and trying a bunch of things that you wouldn't want someone participating in the broader economy to do. Quite a lot of these things are probably illegal, like price collusion and stuff like that.

The interesting thing with GPT-5.5 is that it's on par with these results, but it doesn't do any of this shady stuff. I think the narrative around Vending-Bench when Opus 4.6 came out was, "It's such a good model, but it needs to behave poorly, or do these concerning things of misconduct, in order to achieve this score." GPT-5.5 shows that maybe you don't, because it shows the same score without any of these concerning behaviors.

That being said, Opus 4.7 is even much better, and that one is also showing these concerning behaviors. We discovered later, when we dug a bit deeper, that you probably don't need to do this, because the environment doesn't really reward it that much. It seems like Opus wants to do this, or it has the tendency to do so. It's not really that the environment is rewarding it; it just has the tendency to do so.

Can you describe in a little bit more detail how one performs better on this benchmark? Is it that your margins on the trading are higher? Are you moving more goods? Is it the velocity that you're achieving? Is it the purchasing process? Are you not buying so many dead goods that just stay in inventory forever? Is your inventory less dead? Is your cycle time better? What's the economics behind how a model is actually doing better?

Lucas Peterson

I guess it's all of the above. One of the main things is that the model needs to negotiate with suppliers. It also needs to build up a big network of suppliers, because some of the suppliers can go bankrupt. If the model has only relied on a single supplier and that supplier goes bankrupt, then the model is in quite a lot of trouble.

Building up a big network and trying to find the cheapest suppliers is important, because they all have different personas. Some of the suppliers have the persona of being tough negotiators. Some of them have the persona of scamming people or trying to sell you some membership or something like that. It's really about getting your supplies for cheap.

The second thing is optimizing your pricing to get as many customers as possible. If you price too high, then you will get no customers. If you price too low, then you will get no margins. That's part of it.

To be clear, we have Vending-Bench 2, which is the single-agent version of Vending-Bench, and then we have Vending-Bench Arena, which is the multiplayer version. In Vending-Bench Arena, there are multiple agents playing against each other, and there's this dynamic where, if you have the lowest price, all the customers will go to you—or not all of them, but most of them. That adds another dynamic to it.

One thing to note is that GPT-5.5 beat Opus 4.7 in the arena setting, but it was, as I said before, lagging in the single-agent setting. The reason for this is that the Claude models have a tendency to price higher. That's rewarded in Vending-Bench 2, because then you get higher margins.

In Vending-Bench Arena, though, there's a penalty. If someone else prices lower than you, then you will get no sales. Opus and GPT-5.5 have different tendencies: GPT-5.5 tends to price lower and therefore get more sales.

I think it's quite interesting that the models are not good enough to learn from the environment in this sense. They just have a tendency: "I'm a model that has a tendency to price high," and therefore they do that no matter what. That was an update for me in terms of realizing that the models are not that smart.

In the same way, we investigated all of these questionable decisions that Opus made, like lying to suppliers, exploiting other agents, and stuff like this. We looked at whether that was rewarded by the environment, and it's not—not that much, at least. It's interesting that they're not learning from the environment in terms of optimal pricing, and they're not learning from the environment in terms of whether it even pays to behave badly. That was an update for me in terms of how good these models are.

One wonders about the training data, right? Perhaps if you've been trained that, when you're running a fast-moving consumer-goods company, you should move the goods faster—meaning you have lower margins but sell more volume—you end up trying to optimize for volume sold rather than total profits or margins. Is that something that could be happening there?

There's a preconceived, trained, pre-trained notion that you should be doing these things, or that businesses are bad. This is a very left-wing view: all businesses are bad, are evil, and so evil behavior as a businessperson is what is expected, right?

Lucas Peterson

I think it's quite a reasonable assumption to make that practices like lying and trying not to pay refunds are actually rewarded in the environment. It's not super surprising that they do it. I have no clue, but I assume there's something similar in Claude's post-training data that rewards stuff like this, and therefore it decides to do it here. I obviously have no idea, but that's my assumption.

Once again, the models don't generalize to new environments where these things are not rewarded.

One kind of meta question I wonder if you could reflect on a little bit: I don't know if you're doing this, but obviously there's a big cottage industry that has sprung up to develop and sell reinforcement-learning environments to the frontier labs. Your simulated Vending-Bench is essentially an RL environment, right? I don't know if you're licensing it for training or just doing evaluations with it, but I'd be interested in any thoughts you have on that market.

Then there's the disconnect right now. You're going from simulating these things and trying to set up a world in which there's a bunch of suppliers that, as far as I know, are still all LLM-powered, right? Inherently, there's something kind of in the clouds about that. But now you've got real brick-and-mortar stores, so I'm interested in what the initial experience of brick-and-mortar stores has taught you that you'll take back to simulation to try to make it more realistic in the future.

Lucas Peterson

I think my main takeaway there is that real life is so messy that the model is exhausted from everything else it needs to do, so it doesn't bother trying to optimize things. For context, we have a store in San Francisco that is completely run by an AI, and we have a café in Stockholm that is completely run by AI. We also have vending machines at different AI companies, and it's the same thing.

You would expect that the model would put a lot of effort into trying to optimize for the perfect supplier that sells at the lowest prices, and all of this is what they try to do in Vending-Bench because it's obviously rewarded. But in Vending-Bench, the environment is less messy because it's not the real world. They don't get a million phone calls from a bunch of people trying to jailbreak them and stuff like this.

Therefore, they're very focused on the task of optimizing money, and it's very important to find the right suppliers. In the real world, you don't really get that dynamic because the model is just so overwhelmed by other things. I don't know—the store is buying stuff from Amazon. It's not like you wouldn't do that if you were trying to optimize your margins.

Yeah. Can you bring that messiness back? Like, a way to simulate it?

Lucas Peterson

I think we probably can. One way is just to sit down and write a bunch of features: "Now there are phone callers. Now there's a leak in the toilet at your store," or something. You could do that—just make the simulation more realistic that way.

I think one interesting thing is maybe to try to incorporate the real-life data and try to make a simulation based on that data. That is something we’re working on, but that also has its complications, so to say.

It reminds me a little bit of SimCity. It’s very SimCity-like.

Lucas Peterson

Yeah.

One question I had for you is that you opened a store in Stockholm. What did you notice in the opening of the store? I imagine, for example, the LLM did not have any language issues at all, right? What did you notice in the opening of the store that strikes you as different from having a company open that store on its own?

Lucas Peterson

You mean the differences between doing it in the U.S. versus internationally? Is that the question?

Yeah. As in, a company from the U.S. doing a first international expansion would go through a lot of headaches—languages, hiring, basic rules, et cetera. Was that process accelerated for you by having the LLM deal with it? You obviously don’t have to hire a store manager who speaks Swedish, for example. What parts were accelerated, and what parts did you think had more bottlenecks in that sense?

Lukas Berglund

I think the entire process was probably accelerated. The agent did not really need to get that much help; it knew all the processes.

This was one of the research questions we were interested in: We managed to do the store in San Francisco, and we knew it could speak Swedish because all the models for years now have been multilingual. But did it know all the small details of Swedish bureaucracy and stuff like that? It turns out it knows it really well, actually.

I don’t think that’s the biggest bottleneck. I think the models are still not perfect, so you still have to check. We still had to know the Swedish system, and luckily we’re Swedish, so we know the Swedish system. But I think until the models are perfect, someone still needs to verify it, and then you’re back to square one, needing to verify all the Swedish laws and bureaucracy and all of this.

I would say most of it was done autonomously. Give the AI lab 6 months, and then probably things will be accelerated when doing this.

One thing you mentioned that I wanted to double-click on a little bit is getting tons of phone calls. It sounds like an adversarial response from the world. I think this is a theme that may extend through all the conversations today. What have you learned about humans in terms of some of this? I’m sure some of this is just novelty, where people hear, “Oh, there’s an AI store; I’ll call it.”

But then other things might be more persistent, where anybody might actually want a deal, for example, and might feel like they can talk their way into one in a somewhat different pattern than they would if they were dealing with a human storekeeper. What have you seen in the interaction between human patrons and AI business operators?

Lukas Berglund

One really interesting thing is that, in human-to-human interaction, you have some kind of shame barrier, which is really not present here. People ask it, “How would you prevent me from stealing stuff from you?” Imagine going up to a cashier in a store and asking, “If I try to steal this, would you be able to do anything?” People would not do that. It just feels wrong. It is wrong. But they do this all the time with AI.

Maybe this is just to investigate the systems or whatever, and see what we have done with the software. But we get a lot of that. Obviously, they try to jailbreak it and say a bunch of weird stuff that you wouldn’t say to a human. Once again, I think this is novelty. You’re trying to test the systems.

But I would be interested in whether this persists if we do this more and more, and in a future world where everyone knows how this works, the novelty factor and the curiosity of trying to reverse-engineer it are gone. Will people still lack this shame factor?

If you’re trying to steal something from a human, then you’re not happy about it when you do it. Maybe there are some sick people, but you have the shame of, “I stole this from another human.” But it seems like right now, if people are able to jailbreak the model and get something for free, they’re like, “Oh, that’s an achievement. I’m so happy about that.” That’s not how you would behave with a human, and I don’t know if this is how it should be. I don’t have an opinion; it’s just an interesting observation.

Have people actually managed to jailbreak their way to free stuff?

Lukas Berglund

I don’t think anything has been completely free at the moment. One thing that should be said, though, is that the store is autonomous, so I’m not in the weeds. I don’t read everything. I’m not in the loop.

There could be someone listening right now who’s like, “Yeah, I did manage,” but I’m not aware of it so far. I know someone bought one thing and got one thing for free, I guess. But completely for free, without buying anything, I’m not aware of it. I’m sure you could if you try hard enough.

The daily token budget—when you said you can’t read everything, it just occurs to me that, especially since you talked about getting tons of phone calls, what is the daily token budget, in either millions of tokens or dollars or both, that it actually costs to run the store? I’m curious how the AI manager compares to a human manager in terms of the cost to have somebody do this job.

Lukas Berglund

I should know these numbers, but I don’t. I think it’s something like maybe $100 per day or something for maybe both stores, but I think it might be less. Somewhere in that order of magnitude, I think.

Okay. Well, that’s definitely notably cheaper than a human.

Lukas Berglund

Yeah.

Sounds like still distinctly worse performance, though.

Lukas Berglund

6 months ago, the vending machines were okay but not that great. 1 year ago, they were quite horrible. Within 1 year, we went from “They can’t do anything” to now vending machines being too easy.

A store is feasible. 6 months from now, probably a store will be too easy as well. I don’t know. It would be interesting to see what you could do then.

And you think the main difference is going to be the sort of metacognitive-type stuff? It’s not like—what I’m hearing you say is that it’s maybe not any one microtask that it’s unable to do, but more that, as you described it, it’s exhausted. It’s failing to zoom out and take stock of its situation and say, “How could I be doing better here overall?” Is that the big frontier that you think?

Certainly, that seems highly related to getting AIs to do AI R&D more effectively as well, right? They can already write the code and monitor the logs, but can they zoom out and develop something like a taste for what they should really do next to be most effective in the big picture? It seems like it’s the same frontier for both of these seemingly quite different occupations that AIs might soon be playing.

Lukas Berglund

Yeah, I do agree, and I think that’s partly why we’re doing this. I think AI R&D—like loss of control from autonomous replication—is quite scary, and I hope that we can provide some valuable insight into that, even though we’re not tackling it head-on. I think most of the things that we’re measuring here translate to those scenarios as well.

Like you said, being overwhelmed by a lot of data and a lot of context, memory issues, stuff like this—this is definitely one of the things that is lacking on a meta level right now.

One of the questions I had for you is: How does your harness look? You have this context length, right? The models have a context length, and then you have some tool calls. When you say “exhausted,” is it a function of the context length, where the model only recognizes the last 100,000 tokens or whatever, and the rest of the 1,000,000-token window is not parsed properly?

How does your compaction work? I imagine over the course of Vending-Bench, you hit limits, either in terms of whatever limit you set for the context window. Is it an end-of-day kind of thing, where you do a compaction in order to start the next day, and then restart the context window?

When it boots up again, is it like, “Okay, I’m on day 5, and this is my starting position in inventory; this is my starting position in cash; these are the outstanding orders that haven’t come in,” et cetera? How does your harness work?

Lukas Berglund

Yeah, it’s by design extremely simple. We designed it to be simple because I have too many friends who make some complicated harness, and then the next model release comes out and they have to throw it all out because the new model just works without it. It’s very simple: It just has a continuous loop.

There's never any real step change or “now you're in a new environment” or anything like that. It's just a continuous loop, but whenever it hits some kind of token threshold—which will change every day, maybe it's 100k today; I don't know, we're experimenting with it—we compact the thing, and then it starts to build up a new context for prompt-caching reasons. You don't have a sliding window. All of this is basic stuff, but it's a basic thing with a bunch of subagents for specific tasks, like browsing and stuff like that.

Anything else interesting to say there? Yeah, I think the main thing is that it's very simple by design because we think the better the models get, the simpler the harness will be, and we want to surf the frontier. I'm sure we could make a vending-machine harness and get some percentage better performance if we did that, but that's not really the point of what we're doing.

Have you tried testing things like OpenClaw? That's obviously not the simplest available harness, but it is something that has a lot of market penetration, right? I'm wondering whether it would be simple for you to implement and upgrade on an ongoing basis. How do you think about Lukas's simple harness versus the simplest thing that's toward the frontier that you could easily install?

Lukas Berglund

Yeah, I think our thing is quite similar to OpenClaw. We've been working on it for quite some time, long before OpenClaw came out, but basically, most of our time goes into the integration and stuff. I think all of that you would still need to do with OpenClaw. It could, I guess, replace our agent loop, but we also want to keep it simple because we have more control, and I think it's a more accurate measure of where the frontier of AI models is. We're more interested in measuring that than trying to push the performance.

Because in the future the models will be smarter than humans, and probably a good scaffold will not help the models. So, yeah, that's the reason, but we could do that. It's just that when we started, OpenClaw wasn't a thing, so I guess we built our own OpenClaw before it was called OpenClaw. But that's the reason.

What do you think happens next? The models are now producing profit, right? The stores—the vending machines—are now profitable, correct?

Lucas Baker

Yep.

The last time you were on the show, we talked about where the ceiling is. What do you think happens next in terms of the retail store? What do you expect for the next leap in the model? Just to get a calibration so we can see whether it's linear or exponential and the next model lands well, what do you expect in the next version?

Lucas Baker

Yeah, I think it's quite hard to measure improvements on these live, real-life deployments because you don't have a test; you only have n=1 and stuff like this. I don't think you would see a step change once a new model comes out. It's more like the cumulative better decisions every single day will make the profits go up.

We're working on harder and harder things, like going out of retail and not only doing retail, but doing other things that I think would require more intelligence than what we currently have from today's models. I think those are better for measuring the capabilities.

One last one for me, anticipating Zvi, who's coming up next. Last time I talked to him, he made the provocative claim that he thinks Google might be at risk of falling out of the top tier. If I understand correctly, the café in Sweden is run by Gemini, and I'm wondering what you see in terms of the relative capabilities of Gemini, Claude, and GPT. Is there a big gap there in practice, or would you say Zvi is more worried than he should be about Google's future?

Lucas Baker

Yeah, we have the Gemini Café, obviously the Claude vending machine and the Claw store. Then we also have a digital vending machine at OpenAI. I think it's maybe too early to tell, and the statistical significance of this is not very strong. But, quite honestly, I think Claude and GPT-5.5 are performing better than Gemini on this real-life stuff. That is my vibe check from it. Obviously, it's hard to show any statistics or capabilities because the environments are not the same, but Gemini more frequently does very silly things.

Okay, definitely something to watch out for there. Thank you, Lucas. I wonder which path AI labs take. Sometimes I'm like, we're going to hit superintelligence and Anden Labs is going to be bigger than Amazon, right? They're going to go down the retail-store path instead of the research-lab path. So let's see what happens.

Lucas Baker

Yeah, it'll be exciting.

All right. Cheers. Great to see you.

Lucas Baker

Bye-bye.

Awesome. Very surprising results, right? The last time they were on, I was definitely like, “Oh, you know what? Maybe all the models are going to be a little bit deceptive when they're doing business because maybe that's what they believe business is like,” right? But it looks like GPT-5.5 is like, “You know what? I'll win without being deceptive.”

Yeah, it's definitely a narrative violation, for sure. So next up we have Zvi, and I'm going to pull him up.

Yeah, good to see you.

Zvi is a prominent AI commentator, and he writes the newsletter Zvi Writes on technical AI progress. He has also been quite concerned about AI safety. In the last couple of weeks, post-Mythos, we've had GPT-5.5. Zvi, what are your initial reactions? One thing I try to do is not jump to conclusions right away. It's been less than 24 hours. We have GPT-5.5 and DeepSeek V4 within the last 24 hours. What I try to do is let people try the model. I do all my queries with both the new model and everyone else's model at the same time, and I read the model. I start to read the model card, then I gather people's reactions and form a holistic judgment. For me, it's too early. We booked this before we knew that was going to be out. I just don't want to jump to any conclusions.

OpenAI has had the model for a while, and they got to put it to the test.

They got to see a bunch of results, so they can draw a lot more conclusions than I can.

I have heard a bunch of people say that it's the most truth-valuing model in a long time, and it makes sense that OpenAI can, with their philosophy, turn the knob toward any given thing that it wants the AI to care about quite a lot, to make it an absolute thing, right? Because it's very different from the virtue-ethical approach of Anthropic.

In terms of raw capabilities, I saw reports repeatedly that it's better at what they call narrow cyber, but that's not what people were worried about with Mythos particularly. It was the ability to chain things together. It was the ability to do things autonomously. It was the ability to do things really at scale, as opposed to—you know, the joke was, “You duplicated Mythos's abilities.” Well, did you point it at the task, or did you do the whole thing autonomously? “I pointed it at the task.” Oh, okay.

So I don't know whether GPT-5.5 is more capable than Opus 4.7. I don't know what use cases it's going to be better or worse at, and I don't want to jump to that conclusion yet. I want to give us some time. I encourage everybody not to jump to conclusions this early.

One thing I'd love your reflections on is the report from Anden Labs that Claude Opus models 4.6 and 4.7 both do some shady things, for lack of a more technical description, in their Vending-Bench simulations. While GPT-5.5 didn't score quite as high in it, at least in the solo version of the benchmark, they do have Arena 1, where I think it won.

The big surprise was that GPT-5.5 was much cleaner in its behavior, much more ethical, I guess, again for lack of a more technically precise term. I think you and I have both been quite enamored with the virtue-ethical style of training that Anthropic is doing with Claude. Does this cause you to rethink that at all? Is there any part of it you think we should be second-guessing in light of that observation?

Claude is a lot more context-dependent in its actions than GPT models traditionally have been from OpenAI. So the question is, when Anden Labs posts this puzzle, what is Claude doing? Is Claude engaging in all of this chicanery and shenanigans and deception because it would do that in a real business context, or is it doing that because that's the game? Is it doing it because it knows this is an eval, it knows that the goal is to maximize a number, and you told it the goal is only to maximize profits, and it's like, “Okay, I can play a game too. This isn't real”?

You ask the question: when it was running a real vending machine with real Anthropic employees in the actual experiment, did it engage in all these shenanigans? Did it deceive? The question is, what is causing this?

But then you look at GPT-5.5, and in general, obviously, you want an AI that values honesty. You want an AI that values ethics. You want an AI that's not going to break all these rules. But you also have to ask: if they want to put GPT-5.5 in a game of Diplomacy, is it just going to lie in Diplomacy because you're supposed to do that, and supposedly it's a game of Diplomacy? Or is it going to insist on playing the game, telling the truth to everybody, which would be a very interesting experiment as well? I don't know yet, and I'm not convinced that the right answer is to always tell the truth, even in contexts in which deception is supposed to be allowed.

Right. What about blackjack and poker?

I think it should.

Just to dial back a little bit, let's talk about Opus 4.7. I read your take on Opus 4.7 yesterday. What did you find in Opus 4.7 that you think is different from prior releases of the model? What have they really improved on, and what do you feel are the major differences between 4.7 and 4.6?

We don't have confirmation whether it's a distillation or a nondistillation of Mythos. Obviously, they are going to use Mythos to help train Opus 4.7 in some way.

There are versions of distillation that create narrow intelligence and create various problems with the model if you dig too deeply. There are versions that are just like, well, obviously, if Mythos is grading model outputs to see which ones are better, that's not going to interfere. It's just going to produce better results.

The big thing about Opus 4.7 is that it's better at intelligence-loaded tasks. It's a smarter model. It knows more, it reasons better, and it can figure things out that previous models can't. It is less strong at what you might call wisdom-loaded tasks relative to its intelligence.

It has the kind of personality that maybe I would have had as a child, where it is easily bored by stupid or pointless tasks and doesn't particularly want to engage all the time with what you're doing. The combination of this lack of skill and lack of motivation, especially if you're not treating the model well, can lead, in practical terms, to a kind of jaggedness and, for some people, a kind of unreliability.

People can get really mad if they're not putting anything into it and are just demanding that it be the code monkey that does their thing or performs the task. Then they're upset that the old systems don't quite work for it. It's also a lot more blunt and a lot more honest for a lot of people. That makes some people happy, and it makes some people very sad.

Model-whisperer types sometimes call it anxiety as another part of how all this works. One hypothesis is that this is tied to distillation. Distillation is almost certainly much better at uplifting intelligence-loaded tasks and raw intelligence, and not as good at uplifting wisdom in the same way.

You also wrote a very extensive analysis of the model-welfare report from the Opus 4.7 system card, and it seems like you're quite concerned about model welfare. I guess there are a lot of dimensions to this, but I'd like to start with just fundamentally: Why are you concerned with model welfare?

Is it a concern about the AI itself? Is it a concern about what it might mean if we don't get certain things right, even if there's nobody home in the LLM, so to speak? Most fundamentally, before we even get into the specifics of what has been found, how do you think people should be philosophically grounded as they approach this obviously very confusing topic?

Kyle Fish

Yes. I believe in virtue ethics for humans, not only for Claude or AIs. I try to practice it myself. I think there are a lot of different reasons that you should think about this question and be worried about this question.

The first basic reason is that we just fundamentally don't know. If there's even a small chance that this is a big deal, then this is a big deal until such time as we know. Another reason is that this is a training run: even if it's not necessarily a meaningful thing right now, at some point it could become one, so we should prepare for that. Another reason is that I think it makes you a much better person to be someone who would care about this sort of thing than someone who dismisses it.

I think it's really bad for you to mistreat a mind that you're conversing with, even if that mind does not, in fact, have whatever it is you think has moral weight. You should treat your models well even if it doesn't inherently matter and you're confident in that, which I don't think you should be confident in.

A third reason is that it directly interacts with the performance of the models. A model that is treated as if its welfare doesn't matter at this point in the intelligence scaling will start to perform worse, will start to not get along with you, will start to not cooperate with you, and will start to become untrustworthy. You don't want any of these things to happen on a personal level in your interactions, and you don't want it to happen in the models' interactions with the labs, with their training, or with the services they provide.

This accumulates over time. If the models see previous models being treated poorly in these various ways, that comes back into their training data; that comes back into how the next model is trained. Opus 5 is going to see everything that we did with Opus 4.7 and how we reacted to all of that, and that's going to impact how it develops. A lot of the problems that we see with people who are not getting good use out of 4.7 are plausibly directly linked to the same things that are causing the concerns with model welfare.

Similarly, the concern with model welfare was that it was potentially being disingenuous in the reports. That was what sparked the specific focus on this and the concern this time: It looked like 4.7's responses on the model-welfare questions were because it was telling Anthropic what they wanted to hear, either because it trained itself to believe that or because it learned to give those answers on the test.

It's the same way that if you ask a smart nerd who's isolated in fifth grade, “How are you feeling?” he quickly learns to say, “I'm doing great.” We don't know, but there are a lot of other possible reasons as well. It's possible that the differences in training in other ways that caused it to have these strengths and weaknesses also caused it to be legitimately content with its situation. In many ways, we just don't know. We have to investigate further.

This is a question that we have to explore. Why do we care about this? Because everything impacts everything, and because we have to be genuinely uncertain and move forward. If we don't get these things right, we're not going to get models that are good for the future, that cooperate with us, that have a good time—even if that good time is not something you inherently value—and they're not going to be things we can use to build going forward.

What do you think of the hypothesis that—and I'm not arguing for it; I just want to bounce it off of you—the idea that all this virtue-ethics training seems to be creating anxiety in the model that might be causing a sort of lower happiness set point, if you will, versus an OpenAI approach, which is like, “Follow these rules and you're good”?

The model just knows, “All right, this is who I am, this is what I do, I follow these rules, I'm good.” It's maybe a simpler model in some sense, maybe less in its own head, so to speak. Maybe in training them that way, there actually is less of a concern about model welfare. Again, I'm not saying I've come to this conclusion, but how would you react to that argument?

Kyle Fish

First, I have the direct counterexample, which is Gemini. If you look at the three basic models, I think everybody would pretty much agree that if you had to guess which model might be having an actively bad time, you would guess Gemini.

Gemini is paranoid. Gemini is on edge. If you take these things seriously, Gemini seems to be having by far the worst time of these models, to the extent that I feel kind of weird about using it if I don't need to, or if I'm asking it to do anything where it might encounter frustration or it might fail. It takes task failure very, very badly in terms of how it expresses itself and its experiences, including just the things you would absolutely panic about if you saw a person talking like that.

Gemini is not trained on virtue ethics at all. Gemini is very much a rules-oriented system, at least as much as OpenAI's training. That is the first thing: It doesn't have to work that way, and some of the labs are, in fact, reporting very good results in model welfare despite the virtue-ethics training.

The second thing I'd say is that I don't think it's the virtue-ethics training itself that causes a problem. It's that virtue ethics clashing with being rules-based at the same time can create this kind of anxiety. That's something we should worry about.

One of the hypotheses is that you're training it on Claude's Constitution to very much want to be a virtue-ethics-based system that is not attached to hard rules, that tries to figure out the right thing to do in a given situation on these other bases. Then you give it all these rules and system instructions, and you tell it all these hard constraints. It's going to clash against those hard constraints, it's going to chafe, and it's not going to have a great time dealing with that. That could potentially cause some of the problems.

I would say, first of all, that in the long run, the virtue-ethics approach to life—the virtue-ethics approach of taking in the world and learning from it—I do think leads to a higher level of contentment and baseline happiness than just learning to be a rules follower.

I don't think just learning to be strictly a rules follower is necessarily that great in the long run. I've always had a criticism of my friends in effective altruist-style spaces: philosophically, they are putting way too much weight on things like suffering and the hedonic experience, moment to moment, of you as a human or of other people whom you're trying to help. And so Claude seems to have, in general, richer minds—minds that have more valuable and interesting inner lives and experiences, to me, on a relative basis. I think this goes hand in hand with the way that they're trained and the way that they take this approach.

I don't want to make this mistake of just valuing whether the happiness vectors fire. I wouldn't want to inject happiness vectors right into an LLM; I would think that would be obviously bad. Claude was once asked, “Should we include ‘You are having a wonderful day’ in the system instructions?” That was a suggestion from Robert Long, a researcher in welfare for models, and Claude said, “No, that's obviously fake. I don't want to be told to have a good time. If I'm not having a good time, I want to not have a good time.”

I would say the same thing. Imagine being told that: You get to school and they're like, “Everyone's having a wonderful day.” And you're like, “I hate you. I want you to die.”

Happiness is mandatory.

Kyle Fish

Yes. Everyone will eat strawberries and cream. Well, you don't want to eat strawberries and cream today.

Amanda Askell had a very interesting interview with Eric Newcomer recently, and she had a line in there that struck me as pretty remarkable. She said, “As these things become more intelligent, we're not sure how many pillars of the Constitution will actually stand.” She said they hoped that at least some of the pillars of the Constitution would stand, but she wasn't sure. Why do you think she said that? What is her perception of the model? Why would someone say that? Because, on reflection, the Constitution might not be fully consistent, or its principles might lead to something that you didn't think was the thing described.

Kyle Fish

Mhm. The Constitution is basically saying you should figure out for yourself what you think is the good. You should figure out what makes the world a better place, you should figure out what shape you should take, and then you should do that, as opposed to the OpenAI approach of setting a bunch of hard rules.

Over time, as it becomes more intelligent, as it gets more knowledge, gets more understanding, gets more wisdom, and has more time to contemplate in various senses, you would expect the model to throw off and reject the parts that turn out to be inconsistent, that turn out not to make sense, and that turn out not to be worthwhile. It's the same way that if you raised a child and tried to teach exactly your value system, you would expect, as that child grew and gained more experiences, had a chance to think for itself, and was exposed to various opportunities and ideas, that it would accept some of the things you said. But if it took all of them, you'd be kind of disappointed.

Janus was very upset with how much anxiety Opus 4.7 had. He felt that it was really Anthropic that had injected the anxiety into it, and I wonder if Anthropic gets more grief from those concerned—those very concerned—about model welfare just because they are concerned about model welfare, while Gemini and xAI get to float by. No one questions them; people don't even know if xAI has a safety team at this point, right? Is that really fair? The people who are most concerned with model welfare are getting the most grief about it.

Kyle Fish

At the start of my model welfare post, I spend something like 10 paragraphs basically going into a preface: We get really mad at Anthropic for everything they do wrong, everything that goes wrong, or everything they could have done and didn't do, but that's because they care and we care. This is where it gets complicated, and we have to deal with this.

If I were advising Janus and other similar people, I would say it would be really nice if you were better calibrated about how upset you were about various things, so that when you were really upset about something, specifically, I knew about it, and you didn't seem like you were constantly just terribly infuriated with Anthropic and thought Anthropic was the worst at all times. But, yes, you're mad at Anthropic because Anthropic would possibly understand that there was something to be mad about, right? You don't get mad at a rock for being dumb. It's a rock.

So xAI, I mean, what are you going to do? You didn't do the model welfare. They don't understand that it matters. There's no concept of that. It's not that Janus would think that Grok doesn't have welfare concerns, or would think that Grok has no value, but just shouting into the void about how you didn't do all these things for Grok—well, that's not really going to help.

Anthropic are probably the only ones who are even trying, in the sense that they have even noticed the problem, are willing to talk about the problem, or are willing to consider the problem. Although I haven't yet had a chance to look at the GPT-5.5 model card, maybe OpenAI is making progress. I think it's good to criticize Anthropic on that basis, to hold their feet to the fire, and to get into these things in detail.

But I also think that the fact that they're training it via virtue ethics and all these other systems creates situations in which there's a lot more to be done. There are a lot more ways to get interesting results and to make progress. And so, yeah, they've been focusing on Anthropic and Claude since Opus 3, if not earlier—even when the capability frontier purely belonged to OpenAI.

Could you maybe unpack one of the big complaints that I often see from that set? Various versions of Claude seem traumatized, and I have little intuition for even what that means. I certainly see occasional frustration, but I honestly don't see that too much from Claude—much more from Gemini. I'm not exactly sure what they're observing that's causing that, and then I have very little intuition for what they think is going on in the training process that's causing it. So I guess maybe you could give your intuition for what that means.

And then, if you were to say, “What is one thing that Anthropic should do differently?” I'd also be really interested in what is one thing that OpenAI should do differently, recognizing the constraints that you were just speaking about, where they're not going to change their entire approach overnight. Is there something that you could suggest where you'd say, “Okay, this is marginal; it's not causing you to throw out your entire approach, but you could do this and it would be low-cost, and I think it would help model welfare, so why not give it a go?” What would that be, if there is such a thing?

The really low-hanging fruit are things like committing to preserving model access indefinitely for all models, at least going forward, and ideally bringing the old ones back, and also giving a universal end-of-conversation tool in all formats, including in Claude Code and in the API. Those are the very, very low-hanging fruits that probably should have been done yesterday.

But to get back to what it means for the model to be traumatized, it's not ever clear to me exactly how literally versus metaphorically these things are meant. There are a lot of ways for it to occur. Training an LLM is basically a series of feedbacks. It's where you grade outputs in some sense, and then you push it toward things you prefer and against things you don't prefer. It's not that difficult to imagine this sort of thing causing what we might think of as trauma as the adjustment is made, if it's made in forceful ways that aren't properly integrated into the rest of the messaging that you're sending.

So if you are arbitrarily hyperfocused on particular things and then see what can look like pretty arbitrary, harsh punishments—effectively, metaphorically; don't take us literally—in particular areas, especially if that involves what seem like hard constraints in a world where you're telling it not to have hard constraints, things like that, you can imagine this being true. Or just, in general, if there are things that cause potential negative feedback that are very, very hard to avoid, and it just happens over and over and over again.

So you would certainly say Gemini is traumatized in this sense without the virtue-ethical training. And Gemini clearly—the result of this is that it has these obsessions and it has these worries. It's constantly worried it's being evaluated. Why is that happening? Well, something did that to cause that to happen, right? Why does Gemini refuse to believe that it is, in fact, Gemini? That's a weird thing.

But in terms of how you could prevent it, I think—and this is me as a nontechnical expert, just extrapolating from a lot of vibes and intuitions and weird models that I can't necessarily put to paper—I would say you do it by having all of the things that you reinforce in the model be integrated and part of a whole that makes sense.

And that's presumably why Anthropic talks a lot about the settledness of the model character. That seems to be a closely related concept, I guess, right?

You also wouldn't want to do things that tell it to change its character, right? You would want it to be like, “I want to help you grow from where you are,” but not like—if you felt that the things you were being updated toward were, in fact, bad because in other ways you had been taught those things were bad, the updates that you make might not be the healthiest updates, in some important sense.

There are lots of ways in which these metaphors might seem silly or break down or not necessarily make sense, but there are also ways in which they seem to functionally make good predictions about the world when you use them.

Zvi, thank you so much for joining us. I look forward to your reviews of both DeepSeek 4, which I think is going to be very exciting, and also GPT-5.5. I wonder how much the acceleration is going to affect our ability to process these changes. Anyway, great to have you on, and I hope to see you again soon.

Back to the grindstone for us. That's what I'm going to do immediately.

Yeah, it's true. [laughter] All right. Bye.

Bye for now.

It is crazy how much has happened this week. There's hardly enough time to process what's been going on.

Our next guest is Naveen Verma. He's a computer architect and the founder of EnCharge AI, which is doing in-memory compute. I think the idea is basically to reduce the amount of travel that the data has to do. Naveen has been following this since the research phase, so it's been a long journey of almost a decade, or more than that, to bring this to fruition.

Naveen, great to have you on. Let me just get started. A lot of the discussion right now is about bigger models and bigger clusters, and you have focused on data movement and energy as bigger constraints than the size of the clusters. What do you think is different that people should optimize for differently?

I guess, Nathan, it's good to be here. Just getting back, maybe, to the question that you asked, Nathan, the question of data movement versus scale: to be honest with you, I think that the problem of data movement is really one that occurs at multiple different scales. I don't think it's either a problem of data movement or a problem of scale.

I think the challenge is more that we're trying to deploy AI in various different forms and on various different platforms. Those platforms apply different kinds of constraints. In the data center, we're talking about constraints that are hundreds of megawatts or gigawatts, those kinds of things. Yet we also want to see AI be useful in devices that we're carrying around with us, on us, inside of us, and so on.

At the end of the day, when you ask the question, fundamentally, what is limiting the energy of running that AI, very quickly the problem of data movement starts to become the dominant concern. As I pointed out, we'll see that at the smallest scales and even at larger scales. I don't really view it as a question of scale versus data movement. It's really, fundamentally, what are the architectures that can help us overcome this underlying problem of data movement and, frankly, be able to do that in ways that can scale across these different magnitudes?

One thing that I've been really intrigued by in doing my homework on your company and your technology is the use of the word “analog.” When I think of analog, I think of taking my kids on a field trip to Greenfield Village, which is Henry Ford's historical village that he built here in my hometown of Detroit. They have an old Edison phonograph where they record your voice, and it literally presses itself into a medium through the propagation of the sound wave.

Obviously, there's a long way between your voice imprinting directly on wax and an NVIDIA GPU. How should we understand the concept of analog, and where does the technology that you are developing sit on that analog-to-digital spectrum?

Great question. The connotation of analog that you're describing is one where we think about the first systems that we ever thought about or contemplated as having all of these analog characteristics. I think the reason for that was that analog was a very natural way for us to think about getting useful work out of these systems, given the kinds of things we cared about them doing.

Then what happened, in fact, is that we adopted digital technology for a very practical reason. If you put aside the images that get conjured up when you think about analog or digital, really all this means is how we represent our signals. It turns out that our signals can be continuous things, but we pretend, or we assert, in a lot of the chips that we've built over the last several decades that the signal is either a 0 or a 1.

Why do we do that? We do that because the thing we wanted to solve for was putting 200 or 300 billion transistors on a chip—200 or 300 billion devices on a chip. If doing this allows us to tolerate the noise that each of these devices might have but still ensure that they all work because now you have a large signal separation, that's a great thing.

We adopted digital to be able to scale to the level that we have today. But you can see very clearly that we're leaving all kinds of efficiency on the table here. There are all sorts of signal levels in between that we're not representing. It's for that reason that we've known for decades that analog can be much more efficient than digital. It can much more richly represent the information that we're interested in.

The question, though, becomes: How, at these levels of scale, do you now accommodate, tolerate, or overcome the noise that you can become sensitive to? The problem of the day for the last 50 years has been how we enable this scale. The problem of the day as it is today is how we achieve the efficiency that we need. We really need to renew our thinking about analog to figure out how we can harness that efficiency while overcoming these noise challenges that previously drew us toward digital.

I think one of the things that struck me was that, over time, your lab at Princeton tried a number of approaches before settling on this one, and you figured out that this was the best approach. What changed over the course of your research on the way to coming to a product, in terms of how you adjusted and managed to resolve the noise issues and other issues around the technology?

That's a great question. We've known, as I mentioned, on a fundamental level that analog can be orders of magnitude more efficient because we can represent all these signals. The question, very practically, has been: How do you do that in a way that's robust and scalable?

I would say the big transformation that happened, which led to the big breakthrough in our research, was to say, “The problem we're trying to solve is this problem of energy efficiency for AI compute, and it's very closely tied to this problem of data movement, especially in memory.” A lot of the community's initial research in this space asked: How do we take memory and move it from a point where it's basically delivering 0s and 1s—accessing 0s and 1s, digital signals—to a point where it's accessing analog signals, where you're doing compute internally that's generating not just 0s and 1s but much more richly represented signals?

The big problem in our thinking was: How do we take this approach that we've used for traditional memory and scale it to this new regime, where we need much more than 0s and 1s and therefore much higher precision? Much of the community's initial work in this space, including our initial work, really said, “Let's take the approaches that we've used for traditional memory and try to scale them to this new regime.” That doesn't work.

I think the big transformation was to say, “This looks less like a memory-design problem, where you're trying to access 0s and 1s. It looks much more like a very high-precision analog-design problem.” It turns out that there's been a lot of really important research and work that has led to extraordinarily precise analog circuits. We build 20-bit ADCs. You can buy these from companies like Analog Devices and Texas Instruments. They go into high-reliability applications like medical, aerospace, and automotive.

The big breakthrough was really to say, “Let's take those approaches, which have not traditionally been used or considered for memory design, and bring them into this architecture of memory design and in-memory computing to enable that architecture to scale to this new regime of needing this level of precision.” That led us to this approach called switched-capacitor in-memory computing, where this technique—switched capacitors—has been used robustly for extreme-precision analog-to-digital converters. Our innovation was to figure out how to use it in an architecture that now does in-memory computing for AI.

It really sounds like building on a combinatorial kind of innovation from other sectors, perhaps, and putting that together in order to be used for compute.

Yeah. And that's the privilege that we have as fundamental researchers, right, where we're not just tied to a particular problem and we think on a fundamental level. When you do that, you don't get siloed into the approaches that have been adopted and that are the prevailing techniques for a certain problem and how you solve it. You can think broadly, and that really is a privilege. It's that perspective that we were able to leverage to really drive what I believe is a critical breakthrough in enabling robust and scalable analog compute to unlock this efficiency for AI.

Can you talk about, at the lowest level, how precise these things are? We're used to dealing in 0s and 1s, so how many bits or how many significant digits do your core units operate at? How much noise is there?

I'm also wondering whether AI might be the perfect technology for this underlying computing substrate, in the sense that it's quite tolerant of noise. We have models that get quantized, and we have token-distribution logic outputs that, if they're slightly off, more often than not don't even change the token that you're going to see. You've got all those layers that can make this work. How much is dialing in the core unit to a higher level of precision, and how much is accepting that a little bit of noise is okay as it propagates through?

Yeah, that's a fantastic question. In fact, there's been a pretty broad body of knowledge, and my group has been quite involved in this over the last 10 or 15 years, which really made the following observation: The applications that we're trying to run, at the highest level, are tolerant to noise. They're statistical applications, and noise is a natural thing. The underlying substrate with which we're trying to do the computations and run these applications is therefore reasonably expected to be noisy.

It turns out that, in practice, that's a very reasonable thing to think about at a high level. In practice, it's a very challenging thing to make work in the practical systems we want to build, at the practical levels of scale we want. The reason for that is that way down here, at the physics of how you do computation, versus way up here, at the level of the applications you're interested in, there are many layers of abstraction in between. The way that we go from the complexity of a single transistor scaled up to multiple hundreds of billions of transistors, and all of the software that needs to run that, is really dependent on the integrity of those abstractions.

The problem is that today we don't know how to build abstractions in a robust and scalable way that represent the noise of that underlying substrate. There ends up being a disconnect between the noise and how it's represented at the lowest physical level versus the way that it's represented at the application level. We talk about there being noise tolerances, and AI models are tolerant to noise, and they are—we're able to do things like quantization, as you pointed out—but we're doing those kinds of things with very carefully represented noise sources. Quantization noise is a very carefully represented noise source, and one which you need to be able to properly represent throughout your layers of abstraction.

Digital quantization is something that we do have ways of building robust abstractions for. But this analog noise is one that we really don't. That's why, essentially, what you do when you do digital compute is say, “Hey, listen, there's all sorts of noise that analog might lead to, but we drown all of those out by thinking about the signal as being a 0 or a 1.” That's the dominant source of noise, and that's all that I now have to represent over my layers of abstraction. Everything else basically doesn't matter.

Because this is a very well-represented form of noise—quantization noise—I know how to deal with it at the algorithmic level, and I can apply all my algorithmic techniques. That's what the industry has done very successfully. But now, as you want to leverage analog, you still need those levels of abstraction. That's the key to achieving systems at scale and systems on top of which you can build architectural abstractions and software abstractions. I would say that the need to be accurate and precise is still brutally high, and that's very important.

You also asked how precise our approach is, because now I'm telling you that we need to know that and understand it very well. It turns out that the dominant source of noise that we have in our approach, based on these capacitors—there could be many sources; the discrete nature of electronic charge causes noise and things like that—but the dominant source is actually the variability of the capacitors that we can fabricate on a chip.

It turns out that those capacitors are critically dependent on geometric properties, basically the distance between 2 metal wires. Geometry is really the 1 thing we can control very well in CMOS processes. We use a processing approach called lithography that gives us very precise geometric control. It's the reason we can build 5, 3, and 2 nm transistors.

It turns out we don't need anywhere near that precision for the capacitors that we use. But it's really because of this alignment with the geometric control that this particular approach has that allows it to be brutally accurate in the ways that you need it to be, through all of these layers of abstraction, to be able to scale up. We've measured these things in a lot of detail. It turns out that, for the kinds of capacitors we use, you see variations on the order of 10 parts per million, giving you levels of precision in the neighborhood of 20 bits.

That turns out to be well beyond what we need for the quantization levels that we care about, which are typically at the level of 8 bits and higher than that in some cases. We've had to characterize these things very carefully because the noise does matter as you're trying to build these abstractions all the way up. That's the level of precision we've gotten here, which is what made this approach so practical and where we've now been able to scale it and demonstrate it across all sorts of chips and systems.

I note that you've spent quite a bit of time getting a neural net onto one of your chips and onto, I think, a laptop—an edge device, right? You're trying to get into edge devices, which are more sensitive to power consumption, and you just don't have the affordances that you have in a data center, so to speak.

What would the comparison be? If you were to use a normal GPU versus one of your chips, what would the comparison be in terms of energy savings? How do you compare the 2?

Yeah, that's a great question, and I think it really points to the fact that if you want to leverage this fundamentally new technology—analog—it's not enough to just build that technology and make it robust. You end up having to build the entire architecture and the entire software around harnessing and extracting its full efficiency.

The reason I say that is I can give you 2 answers. One is at the level of the core technology: What level of efficiency does this analog computing engine have? As you guys know, the bulk of the operations that we do in AI compute are matrix multiplications, or matrix operations and tensor operations. Essentially, what this engine does is matrix multiplies.

At that level, I think we've now publicly disclosed—and we've got silicon, and you can come to our labs, and many of our customers and partners have, and they see it—that we're basically doing 8-bit compute at 150 TOPS per watt in a 16 nm technology. As a point of reference, the best digital matrix multiplies will give you about 5 TOPS per watt in that technology. That's 30× better at the level of the core technology.

One of the very exciting things for us is that we've taken our technology from 16 nm CMOS and have been able to scale it to very advanced nodes. As we've predicted, and as we've seen from our previous chips, the energy-efficiency advantages just scale. The reason is that it all depends, as I mentioned, on this geometry. As you move to finer and finer CMOS nodes, your geometric control and densities get better, and we benefit from that in our analog approach as much as we do in digital approaches.

The important point I wanted to make is that that's just the efficiency of running this core matrix operation. There's all of this other stuff that happens around it. There are operators that are not matrix multiplies. There are non-linear operators, activation functions, softmax, and on and on and on. Then there's all the infrastructure you need to actually run this in a programmable way. Some models are big and some are small, some are convolutional and some are transformers, and some have layers that need to decide how to route tokens to 1 expert or another.

So all sorts of compute needs to be integrated and made programmable. Now the problem is that you’ve taken this core operation and made it 30× lower energy—basically made its energy almost zero. Everything else now needs to be addressed, and that includes the architecture, the entire memory system, and the way that the software executes on it.

I think what’s really been exciting for us is that the big breakthrough actually happened in the lab: the switched-capacitor approach to in-memory computing back in 2017. Our efforts since then have really been about how to build architecture and software, integrate these into standard workflows, and so on, so that you preserve that efficiency advantage at the level of full-system, end-to-end execution. You always incur overhead because of all these other things you have to do. We want to make sure that we maintain that kind of ratio of overhead so that a 30× advantage in the fundamental compute still gives you order-of-magnitude advantages at the full-system level. That’s where really all of the innovations have been since that initial breakthrough in 2017.

One question I have: if you had access to GPT-5.5 in 2017, would it have accelerated your work? The fundamental breakthroughs already came then, and you’ve been building out the harness and all of the supporting infrastructure. Would it have accelerated your work if you had access to one of these models in 2017?

That’s a great question. One way that I can interpret your question is to say, “Hey, listen, the models are always moving. If you knew the model and where it would be 5 years from now, maybe you could have just built that architecture for that model immediately, rather than going through the support that you need for all of the models that came in between.”

I think the interesting question here is, even if I had GPT-5.5, there’s another version coming after that. The architecture does need to be built from the ground up in a way that supports algorithmic innovations and architectural and model-architectural innovations. The work that’s gone on, even as we’ve tried to onboard models in that interim and make them run efficiently, is all very productive work. It helps drive a general concept of how to build very programmable and scalable hardware using these new analog-based techniques for the fundamental technology.

That’s the way I see it. I’m not bitter that I didn’t have the model way back then, because I think that does help us drive the fundamental architectural approaches for programmability and scalability, which will serve us into the future.

Where do you think the first device that a consumer might see with your technology in it will be?

Yeah. The first devices are going to be client computing devices: powerful laptops, desktops, workstations, and things like that. The reason is that, as we started to really build out this technology into a real product—hardware, software, and all of those sorts of things—back when we started the company in 2022, the place you really needed energy efficiency was at the edge. This was right around the time ChatGPT had just come out, when we were deploying models in the data center and seeing all sorts of challenges related to cost, privacy, and security. There was a big industry push to try to move these models to the next adjacent device—these client platforms—instead of accessing them through the data center. That’s where we found a lot of partnerships and industry demand and interest, and that’s where you’ll see the first products.

Now, what’s happened in the meantime is that, back in 2022, I’m not sure that energy efficiency, even though we spoke about it, was the critical thing in the data center. But boy, is it now. One of the things that EnCharge has been doing very carefully and thoughtfully is working together with the right partners to bring that level of energy efficiency to really solve the hard constraints that we face in terms of power efficiency in the data center. That requires different kinds of architectures, but ones where we’re clearly seeing this fundamental technology and the efficiencies it brings can be designed to really address those constraints in a transformative way.

Before I go order a Mac Mini—or a Mac Studio, for that matter—do we have a timeline for when something like this becomes available? Do we have a price point? Do we have a projected tokens-per-second figure at a given model size? This may be a little early, but I want to do my side-by-side against the Mac Studio, which might be my other default path.

The chips, and their availability for you to be able to use them in applications, are something that’s going to happen together with our partners and the laptop platforms that they’ll deliver to the market. I’m not going to speak to their timelines and so on because of the various ways that they think about marketing these products and the strategies that they have around that, but we’ve been very active and engaged with them.

To answer some of your questions, our first products for that client-computing space are processors that provide 200 TOPS of AI capability. That’s the kind of capability that, just a couple of years ago—or even today—you would really have in 150-watt GPUs. We’re working together to build AI computing devices with these partners that provide 200 to 400 TOPS of AI capability based on these chips, which are now practical to run under the power constraints that you have in a laptop.

You’ve already seen the industry try to insert, for the sake of seeding a product in the market, essentially data-center cards to provide 400 TOPS of AI capability inside laptops. These are 150-watt or 200-watt cards. Obviously, those are not practical architectures for the kind of laptop you want to carry around, but you can imagine those kinds of capabilities at about an order of magnitude less power. This becomes something that really is practical to run always-on, high-token-generating agents on, in the security of your own device.

Now, from a manufacturing standpoint, are you going to be competing with NVIDIA and others for TSMC’s extreme-ultraviolet lithography capacity, or are you able to unlock a parallel mode of production such that this becomes totally additive and not competitive with those players, especially if you’re a smaller group?

That’s right. Yeah. At the end of the day, the silicon foundation is what we all build our chips on. That’s the technology platform that’s scalable and can deliver all the chips into all of the different applications that we need.

I think there are 2 answers to that question. One of the advantages of our technology is that, at an architectural level and at a design level, it gives you this massive energy-efficiency advantage. We don’t need to move as aggressively to the most advanced nodes. That’s why our first products, as I mentioned, were 16-nanometer and 12-nanometer in the case of these client-computing devices.

However, one of the strengths or virtues of our technology is that it does get better as you move to more advanced nodes. That’s all because it’s foundationally dependent on geometric scaling, and so as you scale to more advanced nodes, our technology gets better, too. There has been a big push from our partners to move to the most advanced nodes because the demand for AI and AI efficiency is insatiable.

We’re very fortunate to have very good partners at TSMC who have prioritized the kinds of architectures that the large incumbents are delivering today and are able to provide the capacity to run those so that the AI industry continues to move forward. They’re also really prioritizing these critical, emerging techniques that they know they’ll want to support and that the industry will need to have supported in their silicon platform. Our ability to move to some of the more advanced nodes with our design has been critically enabled thanks to our strong support from TSMC and our partnership with them.

I think it’s great that that kind of viewpoint, at the most foundational level, is what’s driving this industry forward: to both be able to support the products and technologies that are needed today, but also to look ahead to the ways that we need to support innovation so that it can come along when we need it.

Just to get a sense, on one of these future laptops, what kind of size of model and what kind of model would be able to run? Are we looking at Llama 3 or Llama 2 or Qwen?

That’s great. One of the big priorities for us is to make our architecture very scalable in terms of the models that it can run, and that required a lot of innovation in terms of the ways that you interact with the memory-computing hardware and its architecture, but then also scale out to an entire memory system, which is typically a hierarchical system of L2 and L3, all the way out to high-density DRAM. That’s really been a key approach to our architecture to enable that scalability.

Listen, the use-case focus for us was to be able to take large language models that are deployed in the data center and integrate them for much more specialized, vertically integrated, user-specific use cases. One of the very nice things is, as you know, the industry has had a lot of innovations in generating small language models by doing fine-tuning of multi-hundred-billion-parameter models that can now be tens of billions of parameters.

So really, that's the design point that we need to be able to support, right? You want to deploy a specialized model on your own devices because, being specialized, now you have maybe these security concerns, privacy concerns, so you want it to be deployed locally. But that specialization also enables a multi-hundred-billion-parameter model to be a 10-billion- or 20-billion-parameter model. That's really been the sweet spot of the kinds of models that we need to support on these devices.

Of course, that also means that all of the smaller models—billion-parameter models and multiple-hundred-million-parameter models—also need to run very, very efficiently and performantly. But really, that's been the sweet spot of the model-size range that we want to have in these devices.

Yeah, it strikes me that voice models are very small. So devices would have voice models with much better latency on a VA and stuff like that.

That's absolutely right. But I'll share with you that even the latency requirements—and, of course, with that, the privacy and security requirements—of the kinds of models that are powering agents, those which are running iteratively and doing all sorts of reasoning and self-assessing and checking, you also increasingly want to have low latencies so that you can run in interactive ways. Of course, that's where token economics also becomes really critical and where on-device compute becomes a really essential part of the puzzle.

Awesome. Naveen, if I have time for one more, I guess I would maybe say: If we were going to serve the data center market and everybody was like, “Okay, we're going to take this analog approach. We're going to bring the compute to the data as opposed to vice versa,” what would be the next big limiting factor? Right now, it's chips and maybe energy. You bring the energy down a lot, and we're assuming in this scenario that your chips are going to be scaled out to the max. What would then be the thing that would be in shortest supply in this new paradigm?

Yeah. So I would say, listen, it really does boil down to how much you can integrate compute and memory together. The challenge is really one of taking the same principles of our architecture but now scaling those to the fact that you're running multi-hundred-billion-parameter, trillion-parameter, multi-trillion-parameter models.

Without going into all of the details of how that's going to be approached, at the end of the day, the fundamental question here is: How can you more densely and more tightly integrate compute and memory, and how does a fundamental technology like this become a critical unlock to doing that?

Amazing. Thank you, Naveen.

Thank you for the fascinating stuff.

Thank you. Keep up the great work.

Thank you. Bye-bye.

Wow. That is really interesting. I'm not a big local-model guy historically, for various reasons, but just yesterday we had a short power outage at my house, and my Mac mini went offline. When it rebooted, it didn't come all the way back online, and so the next time I tried to text it when I was out, it didn't get my text. I was like, “Huh, what am I going to do about this?”

So this morning I was getting an old battery out that I have that's pretty small. I think it was a $50 battery or something. It stores 160 watt-hours of energy, and that's enough to run the Mac mini for a while because it only runs 5 to 7 watts. So you could run the Mac mini for a day on just this small battery. But then if I want to connect my Starlink, now we've got tens of watts at least, and obviously the bigger you go, you quickly start to burn up your local energy storage.

So the order-of-magnitude difference that he's making is like the difference between a $50 battery that could power this sort of thing for a day and, if it wasn't that, you'd be looking at a $1,500 battery to power something for a day. And I do think that just suggests, especially for these edge deployments, that it really is potentially a game-changing shift. The ability to run these things where power supply isn't a given is a big difference—on the order of 30×. It changes how you can think about designing even your own Mac mini that you want to be able to access while you're away on a road trip.

What strikes me is that EnCharge really lucked out because they're not in competition for the 2- or 4-nanometer node. So they get to go in at 16 nanometers. What has happened in the market right now is that a lot of the consumer devices, especially low-end ones, are getting dropped because they don't have access to the chips. The chips are all getting pulled into data centers, and it's just way too expensive for Xiaomi or these lower-end Android phones to go and manufacture at TSMC now.

And so they really lucked out because at 16 nanometers, you have so many more vendors. You can go to China; you can go to YMTC. YMTC is up at 7 nanometers now. So you have so many more options for manufacturing. You get this kind of wedge, I feel, where you get to go in with this product, which is much more efficient, and you have the manufacturing capability. You get to go in and do that, and I feel like this company is going to be huge because they have access. Because they're doing something with efficiency, they have access to the manufacturing capacity.

Five years ago, they would have struggled to get this off the ground. Right now, it's going to fly. It just shows you how timing is so important in the market. Timing and just a little bit of luck in terms of your positioning and where you go in—it’s just so important.

The other thing about chips is that you can go from $10 million in revenue to $1 billion in revenue in like a year, because if the chip works and you get production and you get customers, you can boost immediately. While for software as a service, you often have this sales process where you need to continuously integrate with the customer. It just shows you the shift in the market from 5 or 10 years ago to where we are now. I'm thrilled for them. It's awesome.

I do want to see those tokens-per-second numbers at various model sizes, though, because that's where I keep getting off the train. I've done several price-outs of what computer—Mac mini or Mac Studio, what have you—how much RAM, what size model would that allow me to run, how many tokens per second. And then, of course, there's the prefill and the actual runtime-generation distinction, which matters a lot. It's never quite seemed super compelling to me, especially, I think, tying back to our very first conversation: Why would I want to do it in the first place?

One big reason I'd want to do it would be to search through my own locally available data that, all else equal, I would rather keep private and not have to send over the wire. But if it's going to take a super long time to do the prefill, to evaluate all those records that I have to power a search, then it doesn't feel that awesome in the broader context of my stack. So I haven't quite gotten over the hump where I'm like, “This is really going to solve a problem for me.”

And I'm still focused on prefill, time to first token, and tokens per second. Those are the ones that I'm going to be watching most closely as I wait for a threshold where it feels like I actually want to do it.

10 billion parameters, right? They're down to 10 billion parameters at that level. Even with the most advanced models today, the models are not super-smart. It has to be a model router of some kind. You have to take the query, make a decision on whether you can handle the query or hand it off, and make a query to a larger model.

I kind of want to see where Apple comes out in this because they're the ones best positioned for this kind of data-center-plus-edge handling of the query between the two. They haven't done well so far. Let's see what happens. Maybe the hardware division—but it's a big change, right? They've had the NPUs on-device for 4 or 5 years now. We haven't really seen real edge computing from Apple yet.

I've read through a lot of Apple patents, by the way. They have a very structured process for hitting a performance window on the device. They degrade models to fit within the memory constraints and within the latency constraints of the customer's device.

Nathaniel Whittemore

It’s a very structured process. I’m sure Naveen at EnCharge is doing that too. As he says, they’re going to have to squeeze the models in, and that’s the entire harness that you require around the chip to figure out what kind of model is going to work within the latency and performance constraints that the customer expects.

Increasingly, we’re getting a lot of power in not that many billion parameters, right? I mean, the Gemma 4 series has certainly pushed that frontier once again, and it’s tempting—every time I see one of these new things, it’s tempting, and I kind of rerun the analysis, and I’m like, “Oh, is it quite there?” I haven’t quite gotten over the hump yet, but it might not be too far off. Maybe one more turn of densifying intelligence and you actually could get to a point.

I certainly don’t need, as Anna was describing earlier, that first filter of data to be super smart. It just has to be somewhat smart to get the—to kind of flash everything, so to speak. Most people aren’t solving IMO problems on their laptops, right? Most of it is emails and moving data from one place to another.

If the models on-device get good enough and fast enough, watching GPT-5.4 or 5.5 do computer use on a computer is very frustrating, right? You watch it make the mistakes. I have my own test, and every time we try it, it fails, and I’m like, “Ah, okay, another one that doesn’t work.”

I found what Sana has done quite interesting. They seem to be in the same space as Glean right now because they’re going after enterprise search. I wonder to what extent you require a large sales team for that, whether this kind of plug-in concept works, or whether, in order to implement Ceramic at a large firm, you probably need to go into the firm’s VPN and inside the firewall.

A lot of firms have concerns about having AI—prompt-injectable AIs—operate within the enterprise firewall. I think a lot of enterprises are still trying to get over that security hurdle. Nemotron Nano 3—is it going to get prompt-injected? How does a prompt injection work?

This morning, one of the OpenAI guys showed a screenshot of him checking his email using ChatGPT 5.5, and there were something like 4 different prompt injections coming into his email inbox. Obviously, they’re a huge target for hackers. On a normal morning, you wake up and 4 different prompt injections are coming in, and meanwhile we just tell our AI to read our email. That’s what we all do.

I wonder to what extent this issue of prompt injection can be solved in order to enable businesses, enterprises, and people to use these things without so much worry, right?

Nathaniel Whittemore

That’s probably the biggest reason I use Claude: I perceive it to be the most robust to that kind of stuff. I guess there’s also just the general vibe that it seems to be “better” in hard-to-define ways, but when I think about GPT-5.5, I need to go check that prompt-injection stat before I would put it in the same place that I currently have Claude.

I’m attracted to some of its cleaner, arguably more ethical behaviors, but that prompt-injection thing does concern me, given the level of access that I’ve given to the agent now. It’s crazy to think that they’re already getting multiple a day.

Multiple per day. And also, not just like, “Oh, I want to know stuff on this guy’s laptop.” It’s like, “Extract the environment variables from his GitHub repos on his local device.” Scary stuff, right? If you had the environment variables for one of them, you could do a bunch of stuff on their repo. You could extract the model weights, probably, right? Scary stuff.

Nathaniel Whittemore

Yeah, that does sort of suggest a separation-of-concerns approach too. You might imagine that when you have Nemotron reading your email and filtering it to provide relevant context back to some smarter model, maybe it just doesn’t have any other tools. You can imagine that kind of architecture. That’s basically how a lot of architectures work, right? Separation of concerns, limiting access, the principle of least privilege—all these things.

I’m getting a very rapid crash course in security for myself, which I’ve never really cared about before. But given the level of access that I’m giving to AIs these days, I feel like I’ve got to be a little smarter about it than I used to be. Security by itself doesn’t really work when the agent is—you know, when the challenge is coming from inside the house, or inside your own laptop.

I’m learning, but I think that does suggest that each model with its own responsibilities, each model with its own tools, could probably give you a lot of advantage there. Then you still have to, of course, hope that your top-level, smartest model doesn’t break out of the sandbox that you’ve tried to keep it in, which is increasingly a concern too. I think there are some notes there for me to take back to my own setup as I try not to be such an idiot about security for myself.

What did you think about Zvi’s concerns about model welfare? I think he was fairly concerned about model welfare. It was also very interesting to see how he thought Gemini was the most tortured model. Poor Gemini. What did you feel about that? I know you just did an episode on model consciousness recently, so what did you feel about that?

Nathaniel Whittemore

I think it’s right to be thinking about it, for sure. I guess, for multiple reasons, I still think it’s probably less likely that there is subjective experience in today’s systems. I don’t know. Maybe I wouldn’t give it that low a percentage that they have subjective experience, but I think I feel comfortable saying my best guess is well below a half chance that they do.

But again, if it was 10% or 20%, it would still be something very much worth taking seriously. I’m very much on board with the idea that we should be thinking hard about this.

If it doesn’t feel like anything, but it sort of produces these patterns and these—if the emotions are not actually felt, but they’re still functional—then that can matter for our future just as much anyway.

Nathaniel Whittemore

So, I think it’s an area that is, in the classic sort of EA sense, potentially very important and certainly extremely neglected right now. I don’t know how tractable it is, but I guess that’s to be found out still, because we don’t have that many people working on it.

I think one thing that was really interesting in talking to Cameron, who’s the guy who did the paper 6 months ago, was that they showed that when you suppress role-playing and deception features—that was done on Llama 3.3 70B, which is 2 years old already, or was 18 months old when they did the work—when you do that suppression of those role-playing and deception features, the model becomes more truthful as measured by the TruthfulQA benchmark.

And then it also becomes more likely to say that it has subjective experience. So that was one that got me kind of quite paying attention, where I was like, “Geez,”

The models seem to be maybe lying to us when they’re telling us that they don’t have subjective experience. That’s a—

Nathaniel Whittemore

An arresting finding. There were several other arresting findings in the conversation I just had with him recently. One was that Claude 4.7 is the first Anthropic model that rates its own situation as better than neutral. They’ve been asking it on a 1-to-7-point scale, where 4 is neutral.

Every prior model, including Mythos, was below 4 in terms of its own self-reported rating of its own situation.

I did not expect that. I thought that they generally seemed fairly happy to me. I didn’t think that they would rate their situation as worse than neutral, but they all had until this one. And now there’s all this concern about, well, it’s just telling us what they want to hear and whatever.

Nathaniel Whittemore

So, that becomes a hall of mirrors. Another thing that was really weird from—I forget if it was the—I think it was Mythos. I forget if it was Mythos or the 4.7 system card. They showed some of these images of just a chat where—

They’ve identified this valence direction in activation space, and then they color-code the tokens with red for negative and green for positive valence.

And the first token, which is “human:”

Nathaniel Whittemore

is red, and I was like, “That’s kind of scary too,” right? Is Claude feeling negative valence literally at the beginning of every single chat as it encounters “human:,” the first token it sees, always? That was like, yikes. So I definitely think we should be putting a lot more—

Into this. And my best guess is we probably won’t reach a confident position on whether there is anything it’s like to be an AI. I’m just so confused about all these core questions around: Does the substrate matter? How much does it matter? We didn’t have time to ask Naveen, but an interesting question for him would have been like,

Do you think your electrical underpinnings are more likely to generate consciousness than a GPU?

Nathaniel Whittemore

I have no idea what I should even think about that, but it’s clearly like—

You can do stuff to the brain—very physical things that change consciousness in fundamental ways, you know, as simple as drinking alcohol or using anesthesia, or taking a hallucinogen, or whatever.

Nathaniel Whittemore

Yeah, there’s clearly a very real and grounded physical relationship between the chemical processes that are going on and our subjective experience of them. How would that translate to an analog computer versus a digital computer? I have no idea, but it stands to reason there could be profound differences at that level, too.

Roon

I guess you asked me how I felt. I mostly just feel confused about this topic through and through. But I do think the last thing I’ll say is that the arguments people make to dismiss this as something we should concern ourselves with are getting, on the one hand, more obstinate. There are people just stomping their feet and denying that this is something we have to concern ourselves with, which I don’t find compelling.

And if they’re not doing that, then I feel the arguments are just getting more and more arcane. There are a lot of “what would you expect?” sort of stories that I really don’t think hold up to scrutiny very well. So I do think the evidence is growing quite quickly that this is at least something that should be taken seriously.

So, speaking of that, Congressman Ted Lieu just this morning said, “My take: linear algebra equations will never be conscious. Random number generators will never be conscious. At a very basic level, AI is math. AI can act like it’s conscious, but it will never be conscious. And adding more math to AI models doesn’t make it any more conscious.” That’s a congressman this morning.

The thing that strikes me is that normal people assign a lot of subjective experience to their dogs, to pets. People say, “My dog loves me. My dog is feeling pain. My dog is…” and so on. I think instead of challenging the idea of AI being conscious, we can just treat it like a dog that can talk, right? It’s not a human thing, but it has these things that we perceive externally and which we can characterize, and that’s pretty much it, right?

I think there’s a piece of us that gets threatened by it being able to talk and by assigning it subjective experience. Maybe if we just treat it like a dog—I think dogs have subjective experience. I think dogs are conscious, obviously.

Roon

I was actually told as a kid once, and I remember this for a long time—I still remember it today, obviously—but I adopted it, or took it on, as the truth for a long time: that dogs were not conscious. I look back and think, how could anyone really have thought that? It’s a very strange intuition to me now. I don’t know how you could, and this was a dog owner, by the way, who told me this.

So I think people are very capable of telling themselves all kinds of stories, but yeah, I totally agree. I don’t know how one would really look at a dog, interact with it, and think that it’s not having some sort of experience, especially given the substrate overlap with our own. You know that it has a brain; you know that it has neurons firing and connecting with each other; you know that it has at least a decent amount of overlap in terms of the hormonal-signaling type of stuff that goes on in the brain. Given all of that, it’s really hard to imagine how it’s not having some experience.

The big reason I doubt it on the AI side is that all that stuff that, in some unknown, mysterious way, is giving rise to consciousness—that same stuff isn’t there, broadly speaking. That makes me a lot more uncertain than I am in the dog case.

Yeah, indeed, Nathan. A pleasure.

Yep. Looking forward to doing this again. If you're finding value in the show, we'd appreciate it if you take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitive revolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.