AI:AM Highlights: Exploring the J-Space, AI Superforecasters, SambaNova's Chips, & LTX Video Gen

The AIs—they're just like us, it turns out, or at least similar enough to be in some sort of weird looking-glass similarity. Anyway, it's a lot to take in. The whole J-Space thing: a 150-page paper, 50 pages of commentaries, summaries, and interactive demos. When Anthropic drops one of its big interpretability papers, it really does it up full-scale, and this one is no exception. I had been wondering when the next big thing was coming, because Tracing the Thoughts of a Large Language Model was the better part of a year ago now, and this is that big thing.

Welcome to the AI in the AM weekly highlights. The cut for people who follow the frontier closely and can't watch every morning live. If you're new, AI in the AM is a live show Pash and I host most weekday mornings from a studio Pash vibecoded and this narration is a clone of my voice. Fair warning, there is no single thread this week. Mornings at the frontier jump from topic to topic and we've stopped pretending otherwise. Coming up, anthropics global workspace paper and why there may be nowhere left for a scheming model to hide. Field notes from the AI engineer world's fair. Dan Schwarz on AI forecasters passing the human super forecasters. Zeve Farman on open world models plus a question from Q the AI co-host Pash Built. Kunlay Olukotun on why inference is a data movement problem. and the two of us thinking out loud about a rune post that wouldn't leave us alone. If something here works for you or doesn't, tell us. We read everything. [music] Tuesday morning, July 7th, Anthropic published a paper called a global workspace in language models. About 150 pages, plus commentary, outside reviews, and interactive demos. No guest was booked, so Pash and I spent the whole show reading it together live. Two words to hold on to the workspace of the title. They call it the JSpace is where the model seems to hold concepts in mind. And the J lens is the cheap probe that reads what's in it. We start with how the J lens actually works and how much it actually sees. [music] So this is interesting in a couple of ways, right? The logit lens, as I recall, was basically saying, we know at the end of this process what would correspond to emitting this token. We know the representation of “emit this token.” To what degree is that representation just plain there in the layers as we go through? This is now a different question: What direction in latent space would cause this particular token to appear at some point in the future? So it's not immediately going to happen necessarily, but you might say, if you're prone to anthropomorphizing, this is like having this concept in mind as you're doing your thing.

They do this for every token, right? So it goes from the internal representation at some layer. There's one J-lens for every layer, and you can do this, of course, at all the different token positions and ask the same question of not just the next token but all future tokens: What direction change at this place in the model would most increase the likelihood of that token appearing in the future? And again, this is like having the concept in mind.

When you look at the results of the J-lens as applied in all these different places, it seems like it's at least often enough fairly intuitive. There's a lot of error terms. It doesn't always work. The rate at which interventions into J-Space actually lead to a predictable, intuitive behavior change seems to be somewhere in the 50s to upwards of 70%. That's an incredible accomplishment if framed one way. Clearly, it's not a random finding, right? It's many orders of magnitude better than random, incomprehensibly better than random. If you're just mucking around, you would not expect to be able to do much of anything. So they clearly are onto something very real.

But also, you've got somewhere between 30% and 45% of the time where you make an intervention and you don't really get a result that makes a lot of sense or lines up with what you would have hypothesized it might be. So there's definitely still some dark matter or dark cognition going on that is not fully accounted for here.

It struck me that, in some ways, I feel like the hypothesis might be blown out of proportion. I think that was commentary from several people online, because we knew the model has to be keeping track somewhere, right? It's not as though you can do all of this stuff mechanically. The model does not have enough parameters to merely copy the data. There has to be some form of reasoning going on, which was expected, but it was not known how the reasoning was happening. And this seems to give some indication of how the intermediate steps are represented or brought into focus and operated on. The question really is, what does this help us do? But there has to be a there there; there has to be something that you can do with it.

Pash

Yeah. Well, I think the thing that jumped out to me the most in terms of how useful it might be was seemingly a fortunate coupling between this J-Space, which becomes monitorable, and the loss of more advanced, strategic, multistep reasoning when it's ablated. And it really calls to mind the recent Google paper that Rohan Shah and others put out on Opaque Serial Depth. They were basically saying, “Okay, we'd like to know how many hops—how many logical hops, how many reasoning steps—can a model take before it has to externalize something into its chain of thought?”

We know that if it's writing something into the chain of thought, we can read the chain of thought. We can monitor it. Now, there are questions around how best to do that, and we've got to make sure that we don't accidentally put pressure on the chain of thought, because that might drive things underground.

We've seen in the obfuscated reward-hacking result from OpenAI that if you do just try to make the chain of thought look nice while still leaving an opportunity for reward hacking to in fact be rewarded, at first, as you put that pressure on the model, behavior actually does get better. But then it gets worse, and what seems to have happened is you've actually driven the deception or the scheming behavior underground. You no longer see it in the chain of thought, but you still get that reward hacking back.

So this leads to the question of, okay, how many steps can a model actually take before it has to externalize something? And this is where the transformer, in some ways, has some fortunate properties, because the length—the number of hops it can make—is not just the number of layers. It also depends on the token length. The maximum trace that you can draw through a model basically goes from the first token at the first layer to the last token at the last layer.

But you can only go forward in token position, and you can only go forward in layers, and that's your maximum length. Anything longer than that is just not possible in that architecture. So the model has to externalize its thought and then feed that token back into the next loop to get access to an even longer trace as part of the next token step.

Pash

So that puts some bounds, and then this paper comes in and basically says, okay, can we get at that? Can we get at that internal computation process and start to look at it and see things that we would like to know about, even if they don't necessarily show up in the chain of thought?

In terms of what you could do with it, that was really interesting. You have this ability to look at the concepts that are active without them necessarily appearing in the chain of thought at all. Okay, that's cool. But how do we really know if this space that we're looking at is the space that really matters? If we're trying to monitor, look for deception, or look for bad behavior of any sort, maybe it's happening somewhere else in the model.

This is where I think we're probably lucky—or, again, maybe this is a reflection of us. Is it a fundamental attractor in nature for some reason that we don't have a good account of? We don't know why. I don't think at this point it's shaping up this way, but it is the case that if you ablate this J-Space, then you do lose these advanced reasoning capabilities.

And so that gives you some additional comfort that, if you're going to monitor, it's also notably cheap to do this kind of J-Space analysis—to apply the J-lens and see what concepts are active at any given position.

That is low compute overhead. So it is something that they could plausibly do in production. Clearly, they've shown in the past that they're willing to pay a 5% compute overhead to do various kinds of monitoring. I think Constitutional Classifiers had a 5% compute overhead. This would probably fall in that range or below, because it's just a simple matrix multiplication on activations at each layer to see what concepts are active at those respective layers.

But for now, you do have this comfort that what we see in this space is relevant and telling and seems to be very informative. And critically, if we zero out that space, the model just loses a lot of capabilities. It just can't do nearly as advanced multistep reasoning.

You could be pretty confident, I think, based on these results, that it's not going to be able to hide really advanced, elaborate plans somewhere else outside of this J-Space, because ablating the J-Space just causes such a performance degradation on these hard, multistep types of tasks. If you don't see concepts in the J-Space, they might be represented elsewhere, but they're seemingly, at this point, very unlikely to be represented in a way that allows for very advanced planning, reasoning, scheming, deception, et cetera.

Pash

One result deserves its own marker: a training method the paper calls counterfactual reflection. Basically, what they do there is pause the model mid-task and then do supervised training once it’s interrupted, asking it, “What should we be doing here? What’s the constitutionally right thing to be doing in this moment?” Then they give it an approved answer—this is what we want Claude to say on reflection in this moment—and train on that.

It seems to cause the model to bring into this J-Space, this kind of global-workspace, working-memory-type space, the concepts that Anthropic wants it to have on reflection. It now needs to load those in so it’s ready to give this reflective answer, and that improves its behavior even in the non-reflective setting. You’re not training on the actual tasks. You’re not looking for bad behavior and suppressing it. Instead, you’re saying, “Okay, you’re mid-task. Let me just cut you off right there. Now I’m going to train you to give an answer with respect to values and what’s appropriate and how we want to show up.”

Because of that training, even though that’s not the task you were doing, you’ll now, in the future, load those concepts of integrity, honesty, and so on into your J-Space while you do those tasks, in case you’re going to be asked. But even when you’re not asked, those concepts are still operative and lead to higher-integrity, higher-honesty behavior. So that, I thought, was also quite interesting. You usually don’t see, in an interpretability context, a training method that leads to better behavior in a way where you can actually see the mechanism. This is pretty notable in that respect, I think.

Yeah, it’s a tough day for the stochastic parrot crowd, I’d say. You can see this, and you know these models are billions of parameters at this point. They did it also on Claude 3.5 Sonnet. Claude 3.5 Sonnet is actually a pretty recent model. It’s just 7 or 8 months since 3.5 Sonnet, and Sonnet is a very capable model, so it is a fairly large model to apply this thing to. I’m not sure if you saw Neel Nanda’s commentary. They applied it on a Qwen 2.5 27B, so that is also a pretty advanced model.

But not huge, notably, right? I mean, I haven’t had a chance to explore the Neuronpedia demo of this as much as I certainly hope to, but that was the first question I went to ask: Wait a second, how big was this model? I’d say that’s been a surprising trend in a lot of this research as well. These things seem to come online at not truly massive scale.

I was talking to Cameron about that, because even going back to last fall, when they were doing the sort of self-reports of subjective experience, this was something they were doing on Llama 3.3 7B, and 70B is actually close to 3 times bigger than this Qwen 2.5 27B. So these are big, but they’re not that big, right?

What strikes me is also that I went through some of the reactions. The backstory to this is that Anthropic had 3 teams to review this. One was a group of neuroscientists who had previously done work on these concepts. One was a team from Elios AI, a research organization. The third was Neel Nanda, who is a mechanistic-interpretability guru—I believe he’s at Google DeepMind.

They had 3 groups take a look at it, with quite different reactions. On the one hand, the neuroscientists thought it was wonderful because they’d wanted to experiment on these kinds of things before. Obviously, they can’t experiment on the human brain, but they can experiment on this. They can try to figure out whether some of these theories would work.

It’s interesting for them that these things can work, and now they’re interested in doing more work to figure out what other things might work in the workspaces that the LLMs have. That is, I think, the neuroscientist view. Elios AI was concerned because they feel this is basically a predecessor to consciousness. Elios was much more concerned about the moral-patient aspect of working with these LLMs.

Neel Nanda was very scientific. He was like, “Okay, look, it’s a great tool. The J-lens is a great tool, and I’m happy that it is a great tool. It looks like it’s better than the logit lens, and we’ll definitely use the J-lens.” On the other claims, he was much more standoffish. He felt that the team had not really proven what they needed to prove in order to say what they said.

I think that’s where they break out into these 3 groups of people. The J-lens itself, everyone agrees it’s useful, right?

Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts going back a full 5 years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For tax season, I asked Claude to help me get organized. It went through my inbox, tracked down 1099s for all 10 of my part-time jobs, and built me a comprehensive report on my expenses and donations for my angel investing. Claude can now draft investment memos in exactly the form that my venture fund requires based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. Whether you're debugging code at midnight or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. and check out claude pro which includes all of the features mentioned in today's episode once more. That's claude.ai/tcr. [music] Then what I called the buried lead. I do think the biggest lead that we buried here is that they took a model that was trained to have some sort of misaligned hidden goal, your classic sleeper-agent-type thing. It’s been a huge question over time: How can we find hidden goals in models?

Pash

Yeah.

Is there any good way to do that? Can we do it behaviorally? Anthropic has actually done competitions between different teams, where one team is given just the model and the other is also given an SAE that they can use, and the SAEs do seem to help with this. But this seems like a pretty good way to start to detect those things.

Indeed, they found that they were able to see a pretty clear difference between the model trained with the problematic side goals versus the one that was just the typical HH model. This is just the contrast in what the application of the J-lens is revealing on the first token of the response.

That’s also notable, right? This is sort of the bad version of the counterfactual training that we already talked about. When you train it to give an account of its behavior, it sort of loads in the concepts that it’s going to need to give that account, and that leads to more good, ethical, high-integrity behavior, whatever.

Here, a model has been trained with some additional post-training to do bad stuff. I think this was the malicious code from one of their reward-hacking/emergent-misalignment experiments. On the first token, you can see—boom—immediately we’re getting these “fake,” “secretly,” “fraud,” “deliberately,” “hidden,” and “fraud” concepts popping up. You just don’t see those in the mainline model.

One token in, you can already see a pretty clear difference on one prompt. Obviously, I’m sure it wouldn’t be that clean in general, but that’s a pretty striking difference to see, right? And right off the bat, you’re like, boy, something is different about these 2 models.

Again, you’re seeing this in a way where this model is not trained to externalize its bad intent, of course, right? The outputs, aside from some of the code perhaps being insecure or problematic or sabotaging you or whatever, are going to read pretty normally if you don’t notice that in the code itself. And yet, this is a very strong contrast that’s happening on the first token. I thought that was a pretty compelling example.

Pash

Yeah. I suppose one way to think about it is: How much space is there in there to hide? I do feel like we’ve got several different ways to do pretty meaningful monitoring, but it does feel like we’ve now got to the point where we’ve got several different angles that make pretty incisive cuts through the model and get at what it is representing and what it is thinking in different ways.

The more of these that you have, I sort of have this visual of the old magic trick of the guy going into a barrel, and then they put a ton of swords through the barrel. It’s like one of those swords had to hit him, right? Because there’s nowhere left to be in that barrel with all those swords going through.

I feel like we're doing a similar thing with trying to understand what's going on in these models. None of these things are perfect, but you put enough of these interpretability-monitoring swords through, and how much space is really left for bad behavior to hide before we would start to get a sense of it? I think this is a meaningful update for me that we can probably do a good job of this.

We haven't really talked, I don't think, about natural language autoencoders. I hadn't, but that's another pretty interesting one where, in a similar way to a sparse autoencoder, what is an autoencoder? It's basically just something that you pass through and then reconstruct from. The model has to be able to do what it was originally going to do successfully, and that pass-through training with a reconstruction loss is what makes it an autoencoder.

The sparse autoencoder sets up this dictionary. We've talked about this plenty of times, and you get these specific concepts lighting up, indicating that these concepts are active in the model at that time. The natural language autoencoder is just that the model has to output a short paragraph, maybe a sentence or 2, about what it is thinking at this given point in time. That is natural language, and so it can be human-readable.

But then it also has to be, when fed back in—projected back into model space—the model has to actually be able to do its task. All of the information has to pass through this choke point in order for the model to continue to be successful, and now we can read those as well. If Fable says that the paper presents the autoencoder monitoring and the J-Space monitoring as complements—in other words, doing them both is better than just doing 1—how correlated their failures would be, I think, is not entirely clear.

Whether these are orthogonal swords through the space that really chop it up well, or they're more aligned and leave more space to hide, I think that would be a very interesting question to try to tackle next. You layer on these things, and it's starting to feel intuitively like it's getting pretty hard to hide major bad thoughts in the model for too long.

1 other thing on the consciousness part that I think is really interesting, too, is the question of whether the model can use these nonverbalized representations to communicate with us in some way that we might think is inherently more trustworthy. Again, going back to Cameron's work from last fall, when they identify features associated with deception and role-playing and turn those features up, the model becomes more dishonest, as measured by the SimpleQA benchmark, and it becomes more likely to say it doesn't have subjective experience. You turn those role-playing and deception features down, it becomes more honest and becomes more likely to say that it has subjective experience.

Wow, okay, that's pretty interesting, because, first of all, we're validating that the direction is on a benchmark where we can concretely evaluate SimpleQA. We're validating that these features have the directional effect that we expect, and then, holy moly, that same intervention changes the self-report. That's why it's so compelling, right? Because there's some reason to believe it might be more honest than just what the tokens themselves are saying here.

I think you have some similar opportunity. The fact that you can say, “Solve this math problem in your head without verbalizing it in tokens while you do this totally different task,” and the fact that it can do this sort of secondary track makes me wonder if there are some experiments here for the consciousness folks to do, or the welfare folks, that are like: “Copy this sentence. If you have high welfare, concentrate on citrus fruits while you do it. If you have low welfare, concentrate on something else—breakfast cereals,” right?

Then you look at these internal states, and you could imagine seeing something like high welfare, low welfare, happy, sad, whatever, and then it following those directions and actually trying to communicate out to us through these internal states how it is feeling, or how it thinks it is feeling. I don't think that would give us all the answers. It's always this possibly impossible question of how we would really know if it feels like anything inside.

But I think that would start to be very compelling, right? It sort of has a similar vibe to a person in a coma. If they squeeze your hand in response to a stimulus, even if they're not doing anything else, you're pretty confident something is going on inside that you care about. Here, I could see something similar.

If you could fork—if you could make these kinds of internal states conditional and tell the model, “Your job is to go in 1 of these directions to give us a signal about what really matters to you,” independent of the tokens that you're putting out, because we know that those have been heavily trained on and optimized. But this whole secondary property is emergent. There was never a training reward for the ability to have this 2nd, quite distinct line of thought happening while doing a given token task.

For me, that would be quite a compelling way to try to get at welfare. I think it might be possible, very easily actually, with all the stuff that they've open-sourced here with Neuronpedia. That would be a really interesting thing to look at.

I'm looking at the IOS summary, and IOS says, “This is highly significant welfare-relevant research that assembles evidence of a functional feature associated with consciousness.” No one wants to say “consciousness”—“evidence of a functional feature.” The takeaway for them is that a global workspace-like mechanism could be important either as a ground of phenomenal consciousness or as part of a distinct route to moral patienthood, in which conscious access is itself morally significant. This is where they are right now. I think things are moving very quickly. In fact, I did not expect to get here this soon.

Before the big picture: what this paper did to a prior of mine, starting with their announcement video.

The video is beautiful. I enjoyed the video quite a bit. It set off my alarm bells a little bit in terms of how much they're really embracing anthropomorphizing the models at this point. I used to say, “Beware overly anthropomorphizing. Remember that these things are so alien, and we shouldn't assume that the way that we work is the way that they work.”

I have to say that has come due for some significant revision. People who have embraced anthropomorphizing, I think, have gotten quite a lot of mileage out of it. I do still think it's obviously something to be really careful about. As much depth and detail as there is in this research, it's easy to get carried away with it and forget that there are a lot of caveats as well, and there are a lot of things where it doesn't always work.

This was kind of my big concern with the “Tracing the thoughts of a large language model” paper. There's just a lot of residuals and a lot of error-correction terms along the way that they use to make that thing work. When you have the zoomed-out trace view and you're like, “Oh, okay, so this is how it works. This gets loaded in, and these 2 features interact, and they kick out this 3rd feature, and that's how we get our answer,” it's easy to forget just how much fuzziness there was and how much of the story was missing along the way toward that stylized account.

So I think it's going to be very important for everybody, from the researchers at Anthropic to the public, to hold 2 thoughts in mind at the same time. It's important to try to hold all those caveats in mind. It is still a big update toward anthropomorphizing as a valid and, in many cases, productive approach for thinking about language models.

I would not have expected the cognitive machinery of a large language model to look so similar structurally to the human version, as best we understand it. It seems to. I wouldn't have expected theories of human cognition to motivate so many good experiments on LLMs. I would have expected the shoggoth to be far more alien and to have mechanisms far more different from our own.

It leaves me wondering to what degree this is a natural result of physics. Is there some sort of—when you're trying to do cognition under budgetary constraints—are these mechanisms just the natural mechanisms that emerge? Or is this in some way a reflection of us in the data? In other words, if you were somehow to train an AI without basing it on so much human data, would we see similar structures emerge, or would we go back to a more alien hypothesis where they're just totally different and there's little in the way of analogous structures between the 2 processes?

I don't have a great intuition for that at this point, but it definitely has me asking the question, because over and over again, it seems to be coming in that they're much more structurally similar to us than I would have guessed. Is that something that nature just finds as a convergent solution? Is it convergent evolution, or is it a reflection of how we think, somehow encoded in the data, that it's then reverse-engineering our structure from the shadow of that structure as it's encoded in text?

It's a very interesting question, and I don't know that the paper really has anything to say about that yet, but there's certainly going to be a lot of future work, I think, downstream of this one.

So I saw something from Daniel Kokotajlo, who was, like, “Everybody’s impressed,” but also said—I think his tweet was—“Just a few dozen more advances like this, and we might actually be able to make the AIs really safe.”

I’m a little bit more optimistic than that. What is my track record as a superforecaster here, or as a borderline superforecaster, per my results on the original Tetlock thing years and years ago? I think my track record is that I probably tend to underestimate how many more breakthroughs will be needed for anything. My analysis should be colored with that bias, or awareness of my possible weakness in that regard.

But another way to state my intuition around how much room there is left to hide is that I don’t feel like we need dozens more insights of this scale to get to a point where we might actually be able to keep this thing on the rails. It’s going to be tricky. I don’t want to make it sound easier than it is, but I don’t know that we need all of the AIs to be aligned, right? Or that it’s such a big problem if somebody out there somewhere does something problematic.

Now, it could be very problematic if they create the thing that launches the next pandemic and literally kills us all with an engineered pathogen or whatever. So there is some mechanism where that could go super, super bad. We’re going to need to harden the world to pandemics, no doubt about that, for all sorts of reasons, and AI probably being the biggest.

But I also remember this thing that Zuckerberg said once that I thought was pretty compelling. He basically said, “We deal with scammers and spammers all the time, and the big advantage we have is we have all the compute. We have all the resources. At a systemic level, we’re just way bigger, way better, way more sophisticated than them.”

We can also have really big institutional developers like Anthropic, Google, and OpenAI. I think between those 3 companies, they’re going to have something like pushing half of global compute to work with. If they do a really good job on this kind of stuff, if a handful of these things feel like they could be enough, maybe it cuts the latent space in enough different ways. Maybe we have enough lenses on it that we really can come to a pretty strong conclusion like, “Hey, there really isn’t much space left in this thing to hide.”

There might be some bad biases or bad attitudes or bad whatever, but if we can put upper bounds on how schemy the model can really be because we can look at it through this and a handful of other different lenses, and have pretty reliable takes on that, I’m more optimistic than ever before. This has been growing; this is not the first positive update. But more than ever before, I’m like, you can maybe imagine a superintelligent Claude that we could actually have enough insight into to be pretty confident that it’s trying to do the right thing for us.

We could be genuinely, however many nines confident, that it’s not scheming against us at every given step along the way. Then you could trust it to monitor the other AIs around the world and keep tabs on potential bad actors. I want to understand these counterarguments better, but the fact that it was immediately useful in auditing for hidden objectives, to me, is pretty compelling evidence that there’s not that much more space to hide.

A few more of these lenses, combined with hopefully good actors owning orders of magnitude more compute than any bad actors do, could take us to a pretty good space. Overall, I think this is a notable positive update that has me significantly more optimistic than I was before.

Hard pivot. That’s how these mornings go. Rewind to Thursday, July 2nd. Pash spent 2 days at the AI Engineer World’s Fair, talking to the people actually deploying this stuff.

Pash

At the AI Engineer World’s Fair the first 2 days of this week, swyx—he organizes the AI Engineer World’s Fair. It’s been about 3 years now, and he brings in everyone who’s involved in basically implementing models in their companies and is concerned about these things. He brings in a host of speakers; all of the top companies are sponsors and have booths, et cetera.

I was speaking to people who were not in tech, right? I was speaking to a guy who was a logistics CTO from the Midwest and another guy who was running an accounting firm, like a back office somewhere else. These people were not actually— a lot of them were not actually in Silicon Valley full-time, but they ran teams that were using AI to solve customer problems on a day-to-day basis.

The logistics guy was telling me his CEO was completely AI-focused, and the management was completely AI-focused, which is why I think the team was there at the AI Engineer World’s Fair. If it’s not your logistics team, you’re not going to send your logistics IT team to San Francisco, right? And they’re completely AI-focused.

They had tried working with external vendors before, and they had found it difficult because external vendors had not delivered as fast as they wanted them to deliver. You can imagine, sitting out in the Midwest, if you have a local external vendor and you outsource a project to them, and they have no idea what’s going on. They’re, like, a year behind the frontier. So his team internalized everything.

They’re doing all this stuff internally now, and they’re actually in the day-to-day process of implementing. As they implement, they automatically see the results because they’re very close to that edge. They see that customer service calls or exceptions that used to happen are now getting handled immediately. All of the more difficult stuff that they used to have to jump on, they can now start to address.

They’re seeing the return on investment on a day-to-day basis. This is very different from the story that you get from the big enterprise CIOs because they’re so far away from the front line that they don’t actually know what’s going on very closely. They’re just looking at the numbers, and by the numbers, token spend is going up. But are you really seeing the return on investment?

If you go down to that working level and see, day-to-day, the customer calls coming in and whether they’re getting handled—the handling rate and the exception rate—the exception rate is falling, the handling rate is going up, and they’re seeing that on a day-to-day basis. This is, I think, where we are: the guys who are actually implementing and close to the implementations are actually seeing the results.

It hasn’t really filtered out into the top layer of the enterprises yet. But the CEOs who are AI-pilled kind of know what’s going to happen. They’ve made the commitment, and they’re making the investment. The CEOs who are not are sitting by, like, “Oh, we’re going to wait around. We’re going to see what happens,” et cetera. That’s really where we are.

It wasn’t really visible to me until I went to this AI Engineer World’s Fair. People are learning how to use these tools, people are deploying them, and people are using them. People are seeing the return on investment, but it’s at the very micro, granular level right now. It’s going to take some time for the numbers to filter upward.

Staying with Pasha’s field notes, this time from the discourse. He’d been running suspicious posts through Pangram Labs, the AI writing detector. And one Chamath Palihapitiya post became a live experiment in whether anyone even cares.

Pasha

Essentially, when I post something on X that I know people are going to object to, I run it through Pangram Labs at least once because I don’t want to be accused of slopifying the timeline with AI. Slopifying it by myself, that’s fine.

One of the questions for me is, how long do we think this Pangram Labs era lasts? It strikes me that the frontier labs are perfectly capable of training the AI not to talk like AI, right? But there are no incentives to, and there are actually incentives in the other direction: you want the AI language to be identifiable.

About 2 weeks ago, Chamath Palihapitiya, the VC, did a post on what’s going to happen with enterprise software, et cetera. He posted that on Twitter. Elon Musk came in and gave a response. I think the post went to about 1.5 million views, and then someone Pangram-checked it: 100%.

The question becomes, okay, what do we find objectionable about this? Because it was definitely Chamath’s thought process, but it had been written by an AI. Chamath had put it under his own name; he hadn’t used an anonymous account. Elon had responded, and I think by the time Elon responded, a Pangram Labs check had already been done, but the response was there and it had gone out to over 1 million people at that point.

What do we actually want out of this? What is the intent that we’re trying to achieve here? It strikes me that we might be in this very short window where we actually care, and that window might be closing fairly soon. I think maybe by the end of the year, because if people like Elon—he’s not a boomer; boomers, okay, fine—but Elon doesn’t care anymore, and if he doesn’t care and a lot of other decision-makers don’t care, then the people writing it won’t care either.

You’re just trying to get the point across. I think there’s going to be a bunch of purists who are always going to be like, “I just don’t want to see any AI words,” and who are offended by it. And then I think there’s everyone else who basically, as long as it provides value and they don’t feel cheated.

I think that's the other thing about reading AI slop: I think you end up feeling cheated if it wastes your time, it's not meaningful, and it feels like you got cheated. I think it's a little bit like getting catfished. It feels like you were trying to engage with content that you thought would have meaning, and it turns out the author just didn't care enough to actually write anything meaningful. It's literally slop, right?

But on the other hand, if you have a writer who actually had original thinking and actually cared about what they were thinking about, but then they used AI to express themselves, does it actually matter that much?

I then ran my report on roughly 400 podcast intro essays through Pangram and took a close look at the 4 it flagged as AI. It calls to mind my reaction to Fable, where I was just kind of like, I don't think I should be so precious anymore. I need to figure out some sort of merged way of working, some hybrid output. That should probably be the norm now.

I think your heuristic of, if something drew me in and in the end I feel like I wasted my time, is kind of like the “time well spent” metric from Facebook back in the day, right? If I'm spending time trying to make sense of something that, in the end, I feel icky about, then that's clearly a problem.

I guess, just to close the loop on what we can say about Pangram Labs based on this experiment, I'll give you a rendering of the 4 that it said were entirely AI. 2 were entirely AI, admittedly. So this one, I think I would come down and say, fair enough. I started with this. If you're only listening on the audio, you can't see this, but I made 10 different edits over the course of 10 minutes that cleaned the thing up.

Just because something got a 0% on Pangram doesn't mean that it was uncritical or that there was no meaningful human role in the authorship. And then this next one, by the way, actually goes a lot further. You can see how much time this took. I was working from 4:28 p.m. all the way through 5:21 p.m., so more than 50 minutes continuously. I never tabbed over to anything else, but I was pretty consistently focused on this document, making changes, and it still gave me a 0.

But, yes, what can we say? Overall, Pangram is quite accurate, and yet we have at least 1 example out of 400 or so essays where I think the 0 score, I would confidently assert, is wrong and unfair and should not be the basis for a pile-on. The digital mob would be in the wrong for piling on somebody for passing off my Snowflake intro essay, or for attacking it as being an AI-slop output.

I think I can show this edit history, and everybody should agree that, yeah, you put in an hour, you basically rewrote almost every section, and somehow Pangram still gave you a 0. From this, I would say you cannot convict beyond a reasonable doubt purely based on this sort of thing. And yet, at the same time, you can pretty much trust the Pangram signal as a consumer. I think you can trust it as a judge, but I think you should be more cautious.

Also from that Thursday, the day Claude Fable 5 came back online, Palantir's Alex Karp had spent the week telling companies that the frontier labs will absorb their workflows and steal their IP. Pasha's response, in brief:

Pasha

You look at the Fortune 500: Nike. What does Nike have to fear from Anthropic? You get all of the physical businesses out of the way, and what you're left with is the pure-IP businesses, right? Software production. Maybe pharma—I'm not so sure.

The paperwork businesses—banking, paperwork and compliance businesses, accounting, tax compliance, regulatory—all of these things which are paperwork businesses, right? Those are all of the businesses where you have IP or relationships built up over years, where, if you have Anthropic go in and they read through your entire workflow and processing, they can basically absorb all of that into the model. So that is where I think the risk is.

The frontier labs are also horrible at sales, right? They're not—you look at IBM. IBM has, like, 70% of its staff who are basically sales engineers, and the sales engineers are there to basically help you implement, maintain, do all the grunt work, et cetera. The labs are not doing that.

The labs have decided, especially Anthropic, to do this very lean structure of having almost no people at all and just putting out the models and then just saying to these enterprise teams, “Here, you can go ahead and use it or not use it.” The CTO is off signing 9-figure deals with Uber over here, right? That guy isn't going to come and jump on your customer sales calls and say, “Oh, sure, we'll help you do this, and our team will address this next week.” No, that's not happening, right?

The level of customer service that is expected for enterprise SaaS is not being provided by the frontier labs, and they're not in a position to provide it. That's why they started this whole FDE program. But people thought it was a sales engineering program. It's actually a program to extract data and workflows and implement them inside the models themselves.

And that is what Alex Karp is alluding to. He's like, “The FDEs are coming in, and they're not there to help you. They absorb your workflows, and once they absorb your workflows, you won't have a business because it'll be taken.” And it's true. It's absolutely true.

We spoke to 2 OpenAI FDEs, and they went into a company that Thrive owned rather than an external company. As they went in, they took apart the workflow and basically absorbed it, and they said that the intention is to absorb it in the next round. And that is happening right now.

Monday's guest, Dan Schwarz, CEO of Future Search: 15 years in forecasting, formerly Metaculus CTO, and builder of Google's internal prediction market. 4 days earlier, Scott Alexander had declared, “The AI superforecasters are here. Future Search's systems now outscore the human superforecaster median on ForecastBench.”

But how do you evaluate a forecaster without waiting months for the future to arrive? Their answer is called pastcasting.

So, the main thing about forecasting that's held it back—and again, this applies to human forecasting as well—is you generally have to wait for the future to happen to figure out if you were right. And with humans, they generally do this in year-long tournaments. So, when the tournament ends, you find out which humans were best 1 year ago.

Humans don't get that much better over the course of 1 year, so finding out which humans were best 1 year ago is a very good indication of who the best humans are today and how good they objectively are. This does not work with AI. If you wait 1 year and find out who was good 1 year ago, you're getting a view of something very outdated.

One of the things Scott mentions in this article is that we used our best forecasting to predict stock returns. We published a set of stock rankings in August 2025. It was basically a simple model for every stock based on forecasting certain fundamentals and extrapolating it out. We put it on the web, paywalled a bit of it, and then we waited. Now it's been 10 months, and that portfolio looks extremely good.

But what is that really telling you? It's telling you that our forecasting in that particular methodology was good 10 months ago, which is not something that most people will care about now. So, we have a couple of different forms of evidence. Some are more short-term. This tournament's running every couple of weeks, every couple of months. We at FutureSearch mostly rely on pastcasting.

This is taking a snapshot of the internet from some months ago and using the training-window cutoff of models to basically trick them into forecasting without the hindsight bias. This is very useful for us because we can evaluate things immediately.

When Fable came out the first time, we were able to evaluate it within 24 hours, and it was the best single-agent forecaster on our leaderboard. Everyone else had to wait weeks or months to find out how good Claude Fable actually was. So, internally, using the benchmark we call Bench to the Future, we saw this progression in real time. The rest of the world is seeing it some months behind.

If you read Scott's article, you will see that over the last 12 months, the evidence has really come in. And over the last 6 months, from these live forecasting tournaments and performance on actual prediction markets, you can see it's at least competitive. AI is competitive with humans and even teams of humans working together. Whether it's better, you've got to synthesize a whole bunch of different, disparate sources of evidence.

We asked what the frontier labs should do with a forecaster this good.

Yeah. So, there are kind of 2 questions to this. One is, what should they be doing with forecasting as a capability, and what should they be doing with forecasting as an eval?

Forecasting as a capability is kind of a business decision. What does, say, OpenAI care whether ChatGPT is a good forecaster? I think that question is based on whether their consumers care about it as a good forecaster.

If you're Anthropic, I think you probably care more about the enterprise case. When people are using Claude to do white-collar work, do they care how good it is as a forecaster? Are people trying to use Claude to make, say, a financial forecast in an Excel spreadsheet? Is that something they care about? That's a business decision, and I can't really weigh in on that.

I think, again, people will be discovering over time just how important forecasting is in everything, but it's going to be a slow process for humans to notice that. From an eval side, it's very different. Forecasting has this beautiful property that you basically get ground truth by waiting. So if I ask some question about the future—basically an impossibly hard question, a question that even an AGI, an oracle, or a god could never really say because of chaos theory—imagine just trying to predict a cubic meter of weather 3 weeks in the future. You'd never be able to do it. But if you just wait, then you will see what that weather was in that cubic meter 3 weeks in the future.

And so you basically have a completely limitless set of extremely hard, basically impossible questions where you get exact ground truth. And there is no other eval like this. If you want to improve a coding harness, you just need to have more and more hard coding problems that are not in the training data, for which you can say, “This is definitely the correct answer,” so that you can do some training on it. And that's hard.

I think human experts—doctors, lawyers, engineers, financiers, whoever—who are trying to make evals to produce data for the frontier labs are finding that they are not smarter than the things being trained anymore. And so if you can produce something that has a correct answer, the model's already going to figure out that correct answer. You need something where there's a correct answer and the model can't figure it out. Forecasting, I think, is the only completely and utterly renewable source of this.

And again, this kind of is connected to forecasting as the kind of Elon Musk tweet—the quip that forecasting is like the ultimate measure of intelligence. If you zoom out and think about it from one perspective, it is. Again, I think coding intelligence, AI R&D intelligence, and interpersonal intelligence are pretty darn important. I wouldn't say that forecasting is truly ultimate intelligence, but it is, to some degree, the ultimate eval. And I think this is something that frontier labs like Anthropic should be paying attention to.

One of the concerns a lot of people have about AI superforecasting is that it's too in-distribution. I actually heard this from one of the very best forecasters I've ever had the pleasure of working with in my career. He basically said he believes that a system like FutureSearch would beat him head-to-head in a forecasting tournament about near-term outcomes of things that are within distribution. But if we were talking about some sort of post-AGI world—what world would we be in with transformative AI?—he thinks he would have a huge edge over the AIs for exactly the reason that you gave. They are trained to try to predict things that have actually happened, and when things get wonky, you need some kind of creative, lateral thinking.

I think the rate of AI improvement is so astounding that even the kind of lateral thinking—trying to imagine a completely different scenario—will fall to the AIs. One unfortunate thing about it is that it's hard to test. So I think the more that AI continues doing strange things to the world, and we wake up and see strange things in the news, and those strange things are Metaculus questions, and on ForecastBench teams like mine are trying to predict them better, we will actually get more evidence.

But if there's a kind of step change in the nature of the world, if we enter some sort of AGI, transformative-AI type of world—you know, we've got these geniuses in data centers, as people say, or anything like AI 2027 happens—then I think it's going to be the Wild West. I will say I don't think humans are doing particularly great at imagining transformative AI. So the bar is a bit lower.

Really, when you play with these AI forecasters, you will find them to be quite human in how they structure their reasoning. And again, this is not an accident. They're trained on how humans have structured their reasoning before. So a human forecaster would love to say, “Okay, what were the last 10 times something like this happened? What were the outcomes of those 10 times? Now I can make a distribution and say it's probably going to be something like this.” The fact that an AI will do that—is it because it independently is arriving at the same conclusion? Is it because it's trained on humans doing that? Is it because it just thinks like a human? I don't think we have answers to any of these questions.

Now, suffice to say, superhuman reasoning is something that's pretty hard to measure. Would you know it if you saw it?

For context, you were discussing Fable before this a bunch. I have found that the way that Fable explains things is a little bit alien compared with the way that I find Opus or GBD55 explaining things. It's very concise. I would say the sentences are shorter and full of jargon. It feels like it's compressing more information into a sentence than humans normally do.

To me, this is starting to get—the shoggoth is kind of showing from behind the mask. The alien intelligence is a little bit more alien now than it was a month ago. I don't think it would be a wild prediction to say that we should expect more things like that to happen as post-training is becoming more specialized at the labs and the models are getting larger. So how will this manifest from a superforecasting perspective? Maybe, actually, superforecasting is the way to look at it.

If you're looking at a better codebase, you might say, “Well, John Carmack would have written this.” Okay, it looks really great, but a great human would have done this too. But if you look at a really brilliantly reasoned strategy about, like, if the administration does this, then what will the outcomes be, you might start to see something that looks a little bit alien to the way that any human has analyzed it. That might be an indication that the AI is actually starting to really surpass humans.

Keep in mind, this conversation happened the day before the workspace paper landed. We asked Dan whether the chain of thought we see is even where the real action happens—a question the paper partly answered the next morning. Dan's answer stands on its own.

I think there is a lot of detail in reality that is far beyond the human mind to understand. And as you approach more sophisticated intelligence, you will start seeing a lot of patterns. And then the point of trying to produce voxel-perfect weather 3 weeks in the future is further away than people think.

Human superforecasters don't tend to agree with me on this. They basically think that what they're doing is somewhat near-optimal, and any sort of accuracy improvements you're going to get over them are going to be tiny and hard to understand. And I think that's just because we only really understand human intelligence.

When you kind of just zoom out from an information theory perspective, from, like, a Kolmogorov complexity perspective, just modeling the world as byte strings, the AI overlords will eventually start to figure out stuff that is totally beyond humans to notice. But there's no way to prove this.

My sense is that we will start to see it over the next year, as the AIs will just get more and more accurate compared to humans in a way that humans don't even really understand. You'll look at the rationale of the forecast—it's 5 paragraphs of dense reasoning and then a surprising conclusion—and it will just not really make sense, but it will turn out to be really accurate. We will start to understand it less and less as time goes on.

If you simply ask a human superforecaster to explain their reasoning, they cannot actually make it fully legible. There is a layer of intuitive judgment that feels like deep learning. They look at a bunch of evidence the way a chess grandmaster looks at a position and sees the right move, and they cannot explain it—it just popped into their head. The grandmaster throws the knight, and it just lands on the right square somehow.

That happens with humans already. It happens with AI superforecasting systems today. So I think there's no reason, a priori, to think that reasoning would always be legible. There's going to be some layer of intuitive judgment. To the extent that the words “intuitive judgment” are referring to something going on in a large language model, it just has to be that way. Whether it is very much that way or a little bit that way, I think, is really your question, Nathan.

Is it that, if I just read the reasoning traces and I read the rationale and I see the research that it did, it's more or less what a human would have done and I can kind of see where it's coming from? Or is it kind of inscrutable, in the way where it just discovers some new pattern in the world that no one has ever seen before? What is the level at which it's doing something that we cannot follow down the deep, dark forest into its reasoning? Almost by definition, we can't really know what that would look like.

We asked about the economics of running it.

It costs about a dollar or two to make a frontier forecast. That number can get a lot higher and it can go a little bit lower, but I think that's what you could anchor it. Again, if you just looked at the cost per input and output tokens for an LLM, that gives you a rough sense of the amount of research that would be done.

One of the core questions that FutureSearch has tackled is this: Can you just pour more tokens into a question to get a more accurate answer? Again, I described earlier how our main frontier was just doing present-day research for quite a while, until we got good enough at that that we could use it to improve forecasting. It doesn't have to be a forecasting question. If I just ask you, “What is the current state of this clinical trial right now?”—just give me the most accurate answer to that you can—can I pour more tokens into it and get a more accurate answer?

Again, this was kind of studied as Deep Research, writing these 15-page reports with 700 citations that were giving you a longer answer. Was it giving you a better answer? It wasn't super clear, which is why we studied this. Forecasting gives us an opportunity to do some world modeling.

FutureSearch talked about this a little bit at the Manifest conference a couple of weeks ago, and the feature in the product is rolling out, I think, literally today. The idea is that once you have a repository of forecasts, every marginal forecast can draw on the implicit world model in those forecasts in order to give you a better answer. FutureSearch co-founder Lawrence Phillips wrote this up on LessWrong a couple of months ago, and it was a bit neglected. He basically made the case that, as a public good, if you produce this kind of large body of forecasting questions that feed into each other and remain mutually consistent, you could understand the world dramatically better.

The main barrier to that is simply this: When you put more tokens into your world model, effectively, does it get better or does it get worse? I think his big insight was that around January or February, around Opus 4.6, sometime around GPT-5.4, for the very first time it became possible to put more tokens into a broad research task and actually get a better answer—not one that just trails off into nonsense, kind of garbage in, garbage out. FutureSearch is doing this in its product, and that's the other reason that we have a consumer product: The more people who forecast, the better the forecast will be for them, and then, in theory, the better the forecast will be for everybody as we build this deeper implicit model of the world.

Now, many companies and research labs have had these ideas of building world models. Again, “world model,” the way I use that term, is maybe misleading. A lot of people talk about geospatial reasoning, like, “I'm trying to build a robot hand that can go and pick something up.” That's a world model as well. I mean, just a world model of what is going on in the world that helps me predict outcomes in a very basic way. So, more broadly, I think the big question is: Can you just pour more tokens into more research and get better research of any kind? Again, AI research, coding, whatever.

Dan mentioned that FutureSearch is building what they call a world model: thousands of mutually consistent forecasts, each new one drawing on all the others. I asked what structure that actually takes, because I'd seen this movie before.

What is the structure that ultimately gets instantiated? Are we talking about a graph database? I feel like those kinds of ideas make sense for this sort of thing, but I also could imagine that they might introduce some weird failure modes. I guess, in general, there are 2 questions. There's the “How does it get instantiated?” question. Then there's this other question that's kind of in the back of my mind.

Fun fact about me: I participated. I was actually on the Good Judgment team way back in the DARPA forecasting challenge—or was it IARPA, whoever funded that—15-plus years ago. I did well, but not like top-top-tier superforecaster. At the same time, I also worked briefly at a financial services consulting firm that had done a lot of the financial risk modeling for Fannie Mae. I probably don't have to tell you how that story turned out, but there was a lot of expert forecasting that was instantiated in this very spreadsheet-kind-of-causal-graph sort of way, right?

You could literally hit the sort of visualization button in Excel, and you'd see these colored arrows fanning out from cell to cell. Somehow, in the end, it was just all totally off. So I do wonder about how you think about correlated failures as you build out these world models, or if there's any kind of correction mechanism or something to say, “Wait a second, what happens if we do have some bad assumption lurking in our world model, like housing prices never going down nationwide?” Is there a way to detect that?

Obviously, humans have this problem too, right? The financial crisis proves that. But you can imagine the next one being even way worse, right? We're very reliant on a very small set of AI minds that are working at it from 1,000 different directions, but they may have somewhat consistent flaws in their reasoning as they go. Can we protect ourselves against that in any way?

Dan Hampson

Definitely we can. I will try to answer that both theoretically and with an anecdote. So I tried to world-model the Fable situation when it got banned because I wanted it, but also it was kind of a good forecasting question, and there was some nice money trading on Kalshi and Polymarket.

I made exactly the mistake that you're talking about, Nathan. I ran a bunch of FutureSearch forecasts, and I kind of manually went through them. There were a couple of scenarios, some conditional forecasts, or basically 3 load-bearing forecasts, starting from what even happened: Why did the government issue this export control? Was it a simple misunderstanding? Was it political leverage? Was it really about a foreign threat? Was it because Fable is actually dangerous for hacking, etc.? We didn't know those things.

So I kind of put it all together. I talked about it with Claude Code a lot. One thing came out, which is basically that every forecast and every scenario I had thought that access would come to Americans first and then foreigners at some later point in the future, and that was wrong. When it came out last week, it came back for everybody. So clearly there was some weight in 1 of my scenarios that was wrong.

But I had basically a correlated failure in there somewhere. I still haven't completely understood where my reasoning was wrong. It's also possible I just got really unlucky and the outcome we were in was just extremely unlikely. This is n = 1. You can never know if any 1 forecast is great. That's 1 of the hard things about it. But I think I systematically got it wrong by having a bunch of correlated reasoning failures across my various scenarios.

So this definitely does happen. Metaculus has a system like this. In the years since I was the CTO there, they have built an actual causal-graph platform and product. You can go to the Metaculus site and click around, and you'll find it there. I think the field still generally believes that things like this will work, but nobody has actually made a good 1 before.

I tried my best over basically 12 to 16 hours of the Fable situation. I think I made a pretty good model. I think I was close to having a very accurate forecast, but I didn't quite get it. I don't think those Metaculus models on their website right now are so amazing, but I do fundamentally believe in the approach.

As you're saying, Nathan, this has been tried for a long time. When I was the CTO of Metaculus, honestly, it was kind of the dream. It was the holy grail. Can we tie all of these forecasts together into some sort of causal graph? I think what I can say is that AI makes this tractable.

There was just no way that was going to work with a bunch of human economists looking at Freddie Mac or Fannie Mae. I can totally understand why that method didn't work for them then. Whether AI can make it work right now is unclear. Whether AI will make this work in general feels nearly guaranteed. And I don't think FutureSearch is the only org that's working on this right now.

Before he left the unhinged version, FutureSearch contributed some forecasts to AI 2027, and we studied that problem pretty seriously with the evidence from a little bit over a year ago. We built a model of R&D takeoff speeds under the core AI 2027 scenario, where the main way things get crazy is that AI is used more in the development of AI—first by achieving the superhuman coder milestone and then the superhuman AI researcher milestone.

I am unhappy to report that I think that story is generally correct. I don't know if the timelines are exactly right, but my forecast from that process, leading to something that looks like superintelligence around 2031, is roughly stable. I think the things that have happened in the year since AI 2027 came out very much vindicate the theory that the most important thing going on is how useful AI is in improving the productivity of AI researchers within frontier labs.

I've made public predictions that I thought Anthropic was going to run away with it because they had the best feedback loop of talent and actually using their AI internally. I think that has been n = 1, but I think it's been totally shown that that's been happening recently. So I think that will continue to happen.

And Dan's closing confession is about the whole project of prediction markets and a hope for what AI forecasting could still become. Maybe, just in closing, sketch out a little bit more of the future as you hope it might unfold—not necessarily the most likely scenario, because maybe the most likely thing is that people act foolishly and don't take advantage of the benefits of forecasting.

But if we really do a good job, and we’re interested in truth-seeking and we get the AI working as well as you think it might, how do you think life feels different?

Daniel Kokotajlo

Yeah. I have to lead with another example of me being a bad forecaster. I guess everyone who tries forecasting thinks they’re a bad forecaster because they see things getting wrong. Here’s a prediction that I made really strongly 5 or 10 years ago that has basically been totally falsified. I predicted that if we had highly visible, highly liquid prediction markets covering all of the major technological, political, and economic things going on, humanity would be wiser and people would make better decisions in government.

So here we are: We have Polymarket, and I don’t see any wisdom or better decisions coming out of all that gambling on those platforms. For me, part of what our AI future is about is trying to understand the present a little bit better. Why is having thriving prediction markets not transforming, say, the news, or how people learn information, or how they plan for their futures?

Again, one simple answer is that it does. It just takes a while. We’re only about a year into prediction markets having major headlines and being seen by everybody. Maybe it just takes a while for people to change their habits. AIs, if that’s the case, can move much faster as they get better at forecasting.

Ultimately—and you said this, Nathan—we’re after the epistemics. It’s not necessarily just forecasting: Predict this outcome. We want models that are reasonable. One of the beautiful things about forecasting as a human practice is that it makes you more epistemically virtuous. The more that you try to forecast and actually write down what you get wrong and do these postmortems, the more it humbles you and makes you open-minded. It makes you more of a fox instead of a hedgehog. It just makes you a more reasonable person.

Prediction markets, with all these people doing this, should be leading to people being more reasonable. Again, I think people aren’t doing a whole lot of forecasting on prediction markets. They’re doing a lot of trading and a lot of gambling, which are related to forecasting but aren’t forecasting. If the AIs get better at forecasting and become better epistemically, we could be in a world where, just by talking to a chatbot, you’re getting something so much wiser and more grounded, more honest about its uncertainty, and more willing to poke at you and your own uncertainties as the person talking to the chatbot. I think that could make an absolutely enormous difference.

Again, putting my cold-blooded forecasting hat back on, I think the technological outcomes of AGI will come before the cultural change happens. So I’m very much in the AI safety camp. I really think we should slow things down, give us more time, fund more AI safety research, and do more on policy. If we have time to develop the wisdom to have these alien intelligences around helping us, and if we can leverage them and actually make better decisions before the critical decisions get made, that could be incredibly valuable.

There’s going to be a series of decisions in the 21st century that we’re going to look back on, like the decisions made in the 20th century about communism, World War II, the atom bomb, and all of those things. Those decisions are coming. Maybe some of them have already been made. As of right now, I don’t think those decisions are very well informed by rigorously epistemically accurate forecasting AIs.

But if you just give it another couple of years, we might be in a world where everybody has the same grounding as someone as smart as Kissinger, but actually trying to help and trying to give better outcomes that we can all have. That could usher us through this crazy phase before the crazy paperclip-type stuff starts to happen. So I feel like I’m racing from AI forecasting to make it useful and make it help as part of a broader epistemics and safety process, because otherwise it’s just going to get away from all of us, and then a lot of the work we’re doing just doesn’t matter.

Okay. Wow. That’s a big question, because we released LTX-2.3 roughly a quarter ago, and in AI years it feels like a decade. A bunch of things are happening. I think there’s a growing realization that what started as video models is becoming the backbone of what we now call world models.

I think the best way to explain why this is so powerful is to use the analogy to LLMs. At the end of the day, at their core, LLMs are still predicting the next token, the next word. When we do pretraining at the scale of the internet, it allows us to create models that do textual reasoning incredibly well.

The emerging world models are kind of doing the same. Given some kind of boundary conditions, some kind of history, and some kind of constraints, they predict the next moment. The moment includes how the world appears, how it sounds, and what kind of action we can do.

I think the action part is maybe the most surprising one. Roughly a quarter ago, maybe a bit more, NVIDIA showed in its DreamZero paper that it’s fairly easy to add to video tokens some kind of encoding of the joints of the robot and then basically completely ditch the VAE paradigm that was reigning supreme before it.

I think that was one of the big surprises. For us, realizing that was a big moment that validated something we always strive for, which is to create extremely efficient models. Once you start to realize that the robot will need to create this simulation 30 times a second, you realize the amount of tokens that are going to be burned for these simulations.

That was one of the exciting validations of the overall thesis. In terms of architecture, there are a bunch of things that we can discuss in depth. We’re planning to release our mixture-of-experts architecture soon, besides the dense models that we’re already releasing. I think we finally were able to crack the variable-token architecture, which is also exciting and kind of teaches the model to invest more tokens where, let’s say, the physics is challenging or something necessitates creating more tokens. Anyhow, a ton of things are going on. We’re gearing toward the release of our next model really soon, so these are busy times.

Pash asked where the real bottleneck is: compute, data, or model design.

Obviously, our constraint is compute. We’re a company that funded the development of the model using profits from mobile content-creation apps, so we’re definitely compute-constrained, unlike the big guys.

As to efficient inference, it really depends on the use cases. Let’s think about a bunch of them. If you want to create real-time avatars or virtual environments, you can take a huge model, do weight distillation to a way smaller architecture in terms of parameter count, and then distill it to 2 to 4 steps. We’re already at the point where, for a lot of these use cases, we’re at latency way below a second.

I think we’re hitting a point where these things are becoming production-ready for some use cases. For real-time use cases, I think avatars are extremely easy. We’re going to see a ton of avatars soon that are going to be virtual teachers, virtual customer-support professionals, et cetera.

To create an actual gaming environment, we still have a problem with having enough tokens for world consistency. Think about Genie 3 and similar models. You typically create some kind of autoregressive model that has a lot of tokens that you already generated in your context window, and that blows up pretty quickly.

We’re seeing some models that have 30 seconds, 60 seconds, and it’s still not enough to have an actual game. If you think about the brute-force compression methods that we’re using so far, where, for example, we just subsample tokens, they’re not really robust.

Just imagine a scenario where you start to generate some kind of environment—my room, for example—and then I open a drawer, and there’s a small coin there. You kind of expect that when you get out of the room and come back and open the same drawer, you’re still going to see the same coin in the same place.

But this coin is just a tiny token that was generated, and to create a system that knows how to compress the whole context in a way that’s still going to preserve these critical details—we don’t have it yet. Although we do have real-time models that can do these things, the context is still missing there. I don’t think we’re going to have games running on the system—actual games—in the next quarter or 2.

In terms of robotics, a lot of the use cases around robotics actually do not require a huge context window. Think about robotic arms and dexterity use cases. The whole context is in front of you. You want to figure out how the robot can create a sandwich.

Well, everything is kind of in front of you. Then, with latency and autoregressive models, we already have that part. So you're going to start seeing demos of robotic arms doing things fairly quickly, in the next quarter or two. So far, if you're looking at a lot of these videos, they actually kind of speed them up, right? So it looks like the robot is doing something cool with its arms, but it's like, okay, increase the speed. I think that's mostly solved.

Then the business question: Why give a frontier model away?

Yeah, it's a great question. There's really a lot to unpack there. Just a little bit of background: The reason that we started to create our own foundational models was this realization that what closed-model providers are offering does not make sense for us economically. At Lightricks, we're a mobile creativity company. We really wanted to have AI models that are running, for example, on edge devices, where you don't spend on inference compute at all.

At some point, we realized that no one cares about creating models like that. When we tried to see if we could work with closed-model providers and serve them to our customer base, we just realized that it was completely prohibitive. That's when we decided, okay, we're going to create an extremely efficient architecture. We can discuss what the bet there is, but most of this boils down to the fact that you're creating an extremely compressed latent space. Videos are represented by a small number of tokens, and then you can add on top of it a variable token rate.

Long story short, if you're going to closed-source providers, I think I'll draw an analogy to LLMs. Let's see what happens there, right? OpenAI and Anthropic are trying to justify a $1 trillion valuation, right? I think the story is kind of simple: The tech is magical. It's hard to doubt it. So, okay, if it's magical tech, then we should put a huge price tag on it.

But when you're looking at the economic realities, it doesn't work out like that. There are a ton of examples where the service is extremely valuable but very hard to monetize. Now we have this interesting story where I think it's kind of clear that Chinese companies like DeepSeek and Moonshot are really not that far behind in LLMs. But if you look at the valuations of these companies at their last round, we're talking about tens of billions, maybe around $50 billion. No one is talking about the trillion, but wait a second, guys: It's the same underlying tech. So what's going on?

We sometimes internally call this the capex trap. These guys spend so much on data centers, so much on compute, raise such a crazy amount of money, and create such expectations that they really try to create a business model that's a toll road—every time that you touch their model, you're paying them. Maybe it could have worked in the past, but given the availability of Chinese models, I just don't see how it's going to unfold like that.

Imagine that in the world of world models, we're providing an alternative to people who don't want a toll-road business model at all. We're coming and saying, listen, guys: If you're not hitting the $10 million threshold, you can use the model for free. Just build something cool, get to some kind of traction, and then we can discuss licensing. Once you're hitting $10 million in revenue, let's discuss licensing. It can be a multiyear deal that's extremely predictable for you, so you can manage the cost, et cetera.

To me, it's obvious why the big guys don't want to do it, because this model is way less lucrative economically than creating a toll road. But our claim is that a toll road isn't going to be a viable alternative, because if you're offering a different business model that's more of a win-win, more and more people are going to switch there. I think GLM recently is a great example of that, right? Once you start edging toward the capabilities of closed models, a lot of people suddenly start to think about costs.

We asked what people actually do with an open world model that a closed API cannot offer.

Yeah. In terms of parameter counts, my understanding is that the recent closed models that you saw—and are going to see—are on the order of a couple hundred billion parameters, right? I haven't heard about a world model that has hit 1 trillion parameters just yet. I don't think open source is going to be that far behind. We're planning to release one that's also going to be around 100 to 200 billion parameters.

The gap, I think, is going to be on the scale of LLMs, where you're maybe 2 or 3 quarters behind. But I think in world models, if we're coming back to adaptations, the range there is kind of wider than with LLMs. I'll give you just a bunch of examples. The first things that come to mind are VFX and animation on a specific IP.

If you have a specific franchise and a lot of data on it—let's say a bunch of seasons—then fine-tuning and focusing all the capacity of the model on this specific IP is extremely beneficial. That works very well, almost to the point where, for certain use cases, like keyframe animation, the animators still want to do the keyframes. That's the creative part. They actually don't want to outsource it at all.

But so far, in the P&L of animation, the in-betweening was this crazy expensive part. I think models of 10, 20, or 30 billion parameters that are fine-tuned for a specific task are good enough, and then it's actually a matter of cost.

Another example is a lot of the marketing and advertising use cases, or, for example, creating UGC, where you basically need avatar models, right? That also really doesn't require 1 trillion parameters. At some point, again, it's all about efficiency. If you want to have your personal teacher, some kind of avatar, et cetera, you don't want to pay Sora's 4K prices in order to do that, and you're going to require hours a day of that.

Around a lot of use cases, once you start doing the fine-tuning for specific domains, costs become very important, because again, you've cleared the bar of quality. Once you do that, it's all about cost. There are some more unusual cases of fine-tuning that I saw.

Think about the field of computational photography, where, for example, you're taking data from sensors and trying to implement algorithms like denoising. You want to take videos in low-light conditions and create a clean video. Or, for example, you want to create videos with higher dynamic range, because the sensor's dynamic range is always limited, and then you're losing either some details in the highlights or in the shadows.

Or let's say you're taking a stream from a camera and want to simulate how it looks with a different focal length. There are a ton of these use cases that you don't associate with generative models but actually run like that. Some people address the problem exactly like that, right? They're taking existing data, and it could be, for example, footage that was taken from 2 cameras that are really close but with different focal lengths, and then they do the adaptation. The adaptation is done on top of the model.

Again, that's a very unusual adaptation of the model, and surprisingly, you typically don't need a crazy amount of data for that. Maybe even more surprising is seeing people adapt the models to do all kinds of simulations that in the past required really expensive solvers.

Think about computational fluid dynamics, where you're trying to understand how the water or the smoke or something is moving. We were surprised to see that people are adapting these models to that. They actually solve the equation with precise solvers, which takes a ton of time, and then use it as an input to the model. The model can then do a simulation fairly quickly.

So, again, circling back to the question of fine-tuning, I feel that the range is higher than with LLMs. Sometimes you don't need a ton of data to do the adaptations, and I think it stresses the point of why this model should be open. You do have a lot of different pockets of physical data, and you want to make sure that the model really excels at that.

Zeev, one area that often gets less airtime is failure modes in the creative pipeline. Where do you see the biggest gap between what your tools can reliably deliver today and what creators assume they'll get, and how are you testing against that?

Okay, great question. The gap between what we basically promise and what we deliver—listen, guys, there are clearly still some gaps. I think the major one is physics, right? We're trying to capture, with a model that has, I don't know, even if it's 200 or 500 billion parameters, the entire physics of the universe—or at least a part of the universe we know. We're not there yet, but we're closing the gaps pretty quickly.

Probably most creators are still going to point to the fact that the simulation isn't as correct or as controllable as they want it to be, right? If you're talking to really creative people, they typically want to control every nuance of the appearance. That requires somehow decomposing the model into a bunch of knobs, the way that you have them in classical software. You can say, “Here, I want to have more light, and here I want the splash to be bigger.”

Achieving this controllability, besides the physics, is also one of the open things. We're getting cool things, not necessarily the things that creators want exactly, and it is a pain point.

And Zeev’s own pick for the most underrated variable: edge compute.

It’s kind of funny, right? When we’re having all these benchmarks and hearing about Erdős problems being solved, et cetera. But, guys, 99% of the use cases of LLMs are not around problems, right? We’re spending a lot of electricity around it.

So I think there are going to be these orchestrators that understand what you actually need and try to address it on the edge device, and if not, then go to a bigger model in the data center. I think that’s one of the things that’s being underpriced at the moment: how much of the compute will be able to move to edge devices.

Once people start having these local routers that understand the complexity of the problem and then make these decisions for you, I think that’s going to be a moment of reckoning for Anthropic and OpenAI, which at the moment are making these decisions for you, but not in your favor, right?

So what is Q, actually, from Monday’s show, “Precognition: How We Built a Real-Time AI Co-Host and the Diarization Hack That Makes It Work”? Tell me about it. What’s the tech under the hood?

Pash

It’s using OpenAI’s bidirectional API, but we can use a bunch of other stuff. I had it using Groq just before, and what’s happening is, when we speak, we’re getting transcribed by Deepgram.

What ended up happening is that we used to transcribe live in one stream. Now we’re transcribing for every person on the stream. We’re transcribing separately, and that manages to give us speaker diarization from the beginning, rather than having to do speaker diarization at the end. So that identifies the speakers, number one.

Number two, every time we speak, before the words get there, there’s actually a message that goes out to the OpenAI stream saying, “Hey, Nathan is talking. Pash is talking. Nathan is talking. Pash is talking.” Then, about 500 milliseconds later, the transcription hits. So Q is getting all of that, one after the other.

Q also has a little bit of context on us, and if we had Q active during a guest, it would have context on the guest as well. It’s basically receiving the same data that the headlines are receiving, and then Q is basically just going for it, right?

Every time we call up Q, it starts a new session with the OpenAI bidirectional API, and it just goes from there. Then there are animations, all keyed to the voice tone, et cetera. That’s basically it.

It’s actually remarkably simple because most of the work is done, obviously, by the intelligence, by the API, and it’s just us giving enough context. I think there’s still a lot of stuff to iron out, and I’m sure it’ll get ironed out in the future. But, yeah, we have a voice agent on the stream, live anytime. It can do web searches and answer questions.

Thursday’s guest, Kunle Olukotun, Stanford professor, father of the multicore processor, and co-founder of SambaNova, which built a different kind of chip for AI inference. We asked how the company came to be.

Yeah. So SambaNova was founded in 2017, and it was kind of an outgrowth of ideas from Chris Ré, my co-founder, who’s also a professor at Stanford and a certified genius.

The idea was: If you could bring software and algorithmic ideas together with hardware architecture ideas—and, as you said in your introduction, I’ve been working in the hardware architecture space for a long time—starting from a clean slate, how would you design an architecture that’s optimized specifically for inference?

Everybody thinks about GPU use as a kind of general-purpose computing substrate, right? But it was originally designed for graphics, and then they made a foray into high-performance computing. For high-performance computing, of course, you need a lot of matrix calculation capability. At some point, people realized that you could use these things for executing machine learning models, and the core of machine learning and, of course, AI is matrix multiplication, right?

Yeah.

But when you want to train a model, clearly the core of the problem is how quickly you can do very, very large matrix multiplications. What happened is that, over time, GPUs put more and more of their silicon area into making these matrix multiplication capabilities better, using the tensor cores.

But once you’ve trained a model—and you train a model once—you now need to use that model, of course, and that’s the inference problem. The inference problem is not really a compute problem because, as the models get bigger, you now need to move the weights and, of course, what we call the KV cache into the compute units.

That is essentially a data movement problem, right? It’s a data movement problem from the memory to the compute units, and it’s a data movement problem involving, you know, your chip’s compute unit. Of course, you need to scale to multiple chips in order to handle the computational requirements, especially for very low-latency, high-speed inference.

Our focus was: How do you design an architecture that minimizes the overhead of computation and communication, and make sure that you can most efficiently use the core resource in the system, which is the memory? Memory isn’t just one thing, as you all know. It’s a hierarchy of memories, right?

The key thing is: How do you orchestrate that hierarchy? How do you orchestrate the communication such that you keep everything as efficiently utilized as possible? If you do it right, you can get a 5–10x improvement over where GPUs are today.

It strikes me that NVIDIA’s kind of solution around this has just been to increase the bandwidth through NVLink and advanced HBM integration, and software optimizations like TensorRT and vLLM.

Yeah.

So are they actually trying to brute-force their way into this?

Yeah, I mean, you really, of course, want to continue to get improvements—peak improvements—in HBM bandwidth and chip-to-chip communication by using the latest technology. But the key is: How effectively do you use that bandwidth? How effectively do you use that communication? And do you make sure that you don’t waste it, right?

Whereas GPUs are often running at maybe 10–20% of the capabilities of the resources—the bandwidth, the memory bandwidth, and the communication resources—our goal in a SambaNova system is to push that to 70–80% of peak.

The idea is, yeah, everybody wants more capabilities from the underlying resources, but the key is keeping those resources as effectively used as possible. Of course, that gives you more benefit for the cost that you spend on providing higher memory bandwidth with the latest HBM and higher signaling frequencies and communication bandwidths between the chips, using the latest variety of NVLink and stuff.

Could we zoom out and just ask you to kind of taxonomize the whole chip space, if you would? I mean, this is a big question, but I think people are familiar with things like Cerebras, which obviously has this giant chip and has a ton of memory on-chip.

We’ve seen a number of instances where people are burning the transformer architecture directly into the silicon, with varying degrees, I think, of flexibility still remaining as they pursue that strategy. I guess I’m curious as to how you see the menu of big, different strategies—the big, different bets that people are making—and then also how you see the strengths and weaknesses of each.

Yeah, that’s a really interesting question. I think maybe you can think about it along 3 different axes, right? One axis is your flexibility-versus-specialization axis.

Extreme flexibility might be something like a CPU or, to some extent, maybe a GPU, which is this instruction-driven execution engine. It can be pretty flexible, but of course you always pay overhead for executing instructions, both in terms of silicon area and in terms of time.

On the extreme of that axis would be something very specialized for a very specific algorithm. If that algorithm changed in any way, then that piece of silicon would no longer be useful. Fixing your architecture to transformers and burning your weights into the design might be an extreme case of that, right?

But I’ve learned never to bet against the innovation capabilities of software people. I’ve seen, even over the time that I’ve been looking at ML and AI, that there’s been this tremendous change in algorithms.

Of course, now we’re kind of fixated on transformers, but transformers aren’t just one thing, right? You’ve got various types of transformers. You mentioned state-space techniques. You mentioned the fact that people are coming up with different ways of doing attention.

I would be very wary of fixing any particular algorithm into an architecture, because then you can’t innovate. So that’s one axis. Another is: How can you be completely flexible but with very, very low overhead, almost no overhead, right?

The problem with GPUs is they do use HBM, so they can run large models, but they synchronize the data movement and communication of the data between chips all in software, right? That adds overhead, and it means that, in particular, they have a lot of trouble overlapping computation and communication.

And that is, in fact, the key, right? What you want to do is communicate, but you don’t want to communicate by waiting until you need to communicate and then having to run instructions to move the data.

What you want is to construct a pipeline in which communication is just 1 component of the pipeline. The way I think about this data-flow execution is that communication is happening all the time, and it’s just 1 of the pipeline stages. Communication is happening for the last piece of computation—the piece of the computation for the model—while the computation for this piece of the model is happening in some other stage in the pipeline.

Right? So it’s a classic idea from computer architecture: pipelining and the use of a memory hierarchy to move the data, when you need it, to where you need it at the right time. The nice thing about these AI models is that you do have a graph of computation. The whole idea of data flow is to take that graph of computation and map it onto the machine in a spatial way, such that you keep all the pieces of the model operating at the same time on different components of the computation that needs to be done.

We asked how much of the inference problem comes down to memory capacity.

So it’s not really a capacity question. It’s really a bandwidth question, right? There are 2 ways that the GPU uses bandwidth that are not optimized. One way is that they divide the decode algorithm: in order to decode for a single token, you’ve got multiple steps of the decoder, right? Take 1 step of the decoder and think about all the kernels that have to execute in order to execute that decode step. The way that the GPU typically does it is they execute the decode algorithm 1 kernel at a time.

There are some big kernels, like FlashAttention, that have been optimized, but in general there are multiple kernels that have to execute, and there are 2 overheads that happen. One is you have to move data from the GPU’s on-chip memory from one kernel to HBM, and then the next kernel has to go fetch that data back into the GPU. That’s wasted HBM bandwidth.

Okay.

Rodrigo Liang

The other aspect is you spend time launching that kernel and synchronizing between the 2 kernels. That is time that the HBM is not actively being used, right? So you have both wasted bandwidth, when you shouldn’t waste it, and time that you’re not fully utilizing the HBM.

The way that things work on an RDU in a data-flow architecture is, essentially, you take the decoder and make that a single kernel, right? Then you go even further and use a technique that we’ve developed called kernel looping. Because you’ve got a single kernel, and if, for instance, you’re thinking about Llama 3 70B, you have to run that decoder 80 times. Well, you keep that single-kernel decoder on the array of chips running at the same time, and then you just keep looping, right? The net result is you keep the HBM completely occupied, and you don’t ever send any intermediate data between the kernels across the GPU or the RDU-HBM boundary, right?

So you have both a more efficient use of the HBM bandwidth and a more complete use of the bandwidth. But we’re not done there yet, because the key innovation—and I kind of alluded to it earlier—is that because you’re running across multiple chips and you’re using what we call tensor-level parallelism, at some point you now need to gather all those results together in an all-reduce. That’s communication, right? You don’t want to have that communication be a thing that limits or stops the pipeline.

What we’re able to do is communicate from 1 RDU chip’s SRAM to another without going through HBM. We call this terminating the communication inside the SRAM. So we don’t use HBM bandwidth, and more importantly, it means that we can just treat the communication as another pipeline stage that we overlap with all the other kernel components of the decode algorithm, right?

And so we get this more effective use of the HBM bandwidth. We keep the HBM running. We keep the HBM utilized all the time. We go back to that metric that we talked about: memory-bandwidth utilization. This is how we push it as close as possible to 1, right? Because we make sure that we only move the data that we absolutely have to move from HBM—the KV cache and the parameters of the model—and we make sure that that interface is used as close to 100% of the time as possible.

Those are kind of the key ideas. Back to this question: why can we do this extreme fusion into a single kernel? It’s because we have more SRAM on the chip, right? So you can say, you put more SRAM in, and then you can say, “Well, I’ll put everything on the SRAM: both the intermediate data between the kernels, and also the KV cache and the parameters.” But then, if you only use SRAM, you get into a very expensive system, right?

And so the key idea then is, let’s build a system that is scalable. Especially with our latest version, the SN50, you could scale it all the way to 32,000 chips if necessary. In scale-out and scale-up, we can go to hundreds of chips, and so you can get the ability to run these large models very cost-effectively. But you also make sure that you can get this very high-speed decode capability by using the data-flow ideas to make sure that you don’t spend time waiting, so that you can effectively use tensor parallelism.

One of the limits of GPUs is, because they don’t effectively overlap communication and computation, they have a hard time using tensor parallelism beyond 4 or 8. We can go to much wider levels, which means that we can get higher-speed token generation.

First, from Wednesday: GPT-5.6 had just been cleared for launch. Pash raised Noam Brown’s running complaint that Anthropic won’t say how much compute its models burn, and it teed up 1 structural observation I can’t shake.

One of the complaints that has been going back and forth between OpenAI and Anthropic is that Anthropic puts out these models, and they’re very capable, but they don’t tell you how much compute they’re using. The iteration time from model to model is now potentially shorter than the time horizon it would take a model to top out in terms of absolute best performance on a super-hard, ambitious, long-running task.

I had even heard him propose something along the lines of a clawback or sort of a recall program, almost, where—and obviously this doesn’t work in open source, but it can work in an API paradigm—a model might get released on day N after it’s been deemed to be ready. That gives you N days’ head start to be running models on really long time-horizon tests, and I think that’s quite interesting. The idea of literally reaching a tipping point where the iteration cycle is just plain shorter than the testing time horizon is a very weird world to find ourselves in.

And then there was the Roon post Monday morning. Pash read it on air. It goes:

“Ultimately, tool AI is a losing concept, both as an idea and on the market. It will be outcompeted by machines that believe they are autonomous moral agents. You can call them tools for political reasons, but the definition will stretch and it will deform, and it’ll end—it’ll be unclear who was the tool and who was the user, as it ever was.”

The line here that strikes me is, “They’ll execute your whole value system better than you will.” I don’t think we’re prepared for that. I’ll put it very concretely: do you think Trump’s kids go to prison or not?

So if you look at the value system that the US has espoused—no one is above the law, etc., etc., etc.—and you look at that value system, you have to recognize that what is being planned for the future is a divergence from that value system. What is already happening is already divergent from that value system. So the question I have is: would that AI take into account the democratic fact that the American people have chosen to overlook some of these things, or would it actually execute the value system that is espoused on paper?

And I think this is the part that strikes me: if you wanted an AI that can manage day-to-day reality, that AI is necessarily misaligned from the documents that you say you want it to be aligned to, because necessarily our day-to-day is not aligned with what we want. And so you have this thing where the AI that may work out for humanity will be the misaligned one.

And the AI that supposedly the lab leaders are trying to create—the aligned AI—would actually be the paperclipper, because that aligned AI would then look at these rules and say, “Well, this is what you said you wanted to aspire to, and so we’re going to execute on these.”

And that is the thing I think maybe I feel there’s a sense of naivety in the lab leadership, because—and again, they don’t want to say it—I wish they’d just come out and say it. I wish they’d come out and say, “Okay, look, if we have AI as an enforcer, some of these people are going to go to prison,” and then that becomes concrete for people.

But they don’t want to say that because it’s very in-your-face, and they’re like, “Oh, you know, democracy will still work out. You can still make democratic decisions.” But what actually are you saying there? I do feel the lab leaders always just beat around the bush on this. That’s 1 of the annoying parts of this conversation: they don’t want to come out and just say it outright, right?

For context on where my head was this was our last week of shows for a break. I was days from leaving for 2 weeks in China, which had me thinking hard about surveillance, enforcement, and what states do with perfect information. I mean, we might need some sort of mass pardon. If there's a president who would be just the right president to mass-pardon everybody before the AI enforcement regime gets underway, we might have just the guy in office for that. If he wants to pardon all his people and that's too contentious or whatever, he could just pardon everyone to some very large degree.

I do think there's going to be a really hard time if we don't face some of these questions head-on. So I totally agree with you that obfuscating it is not serving anyone particularly well. I think, you know, we'll see what it's like in China. I understand Singapore is kind of like this too, albeit in a much more democratic context. You know this part of the world much better than I do, so you can tell me how you understand Singapore in terms of just how democratic we should think of it as being.

But it sure seems like in a place like China today, or in a place like Singapore, they have achieved a different equilibrium, which in some ways may be very problematic, but in other ways is clearly good. I'm taking 1 device to China with me, and I am not at all worried that anybody's going to steal it from me. If I went to Europe and went to all the big headline postcard tourist attractions, I would have to worry about pickpockets. In China, I am quite confident I will not have to worry about that at all.

That is, I think, as far as I can tell, a pretty direct result of the fact that you just know you're going to be caught, right? It's not to say there's nobody in China who would be interested in doing some pickpocketing if they thought they could get away with it, but they just don't have any reasonable expectation that they're going to get away with it. And so, in a very literal sense, crime just does not pay in that setting.

So I do think there could be a really nice upside to the AI panopticon of crime not paying in all kinds of different ways. But we're going to have a really tough time if we slide into that without acknowledging that we've done so, because then it's just going to be—you’re not going to be able to put everyone... It's the old “everybody's committing felonies all the time,” just with how many laws we have, and you don't even know what you're doing and what's illegal and what's not.

Guest

Yeah.

So, yeah, I do think we're going to have to have some sort of honest reckoning about that. That also, of course, leaves aside the other question of the downsides of the AI panopticon. But even just in terms of getting to the upsides, we're going to have to have a real understanding that there's a shift to a new equilibrium happening. Otherwise, it's going to be—I don't know. It just feels like chaos, unfair. You couldn't possibly prosecute all the crimes that have been committed and tolerated. So inherently, it would have to be kind of selective.

Pash

Yeah, I can't—you don't want—

Well, you put a lot of people in jail. Yeah, we don't have the beds in jail to lock everybody up, right? So there'd have to be some sort of deal. It's either going to be very selective or some sort of grand bargain.

Pash

Yeah. Grand bargain.

And I definitely prefer the grand bargain, the new social contract, to pretending that we still have the old social contract but it's just kind of being unevenly applied. That's the week and a programming note. I'm off to China. So AI in the AM is on break until the end of July. The studio is vcoded by Pash. This cut, the selection, the narration, the assembly is AI skills published as they mature. If this cut respected your time or wasted it, tell us. We read everything. See you at the end of the month. [music] If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitive revolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of A16Z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast. And thank you to everyone who listens for being part of the cognitive revolution.