Balaji Srinivasan：AI 将如何改变政治、战争与金钱

So polytheistic AGI, I think, is one very useful macro frame. It means every culture has its own AGI, and eventually every culture has its own social network, cryptocurrency, and AI. The AI is sort of like their oracle at the center of society, and they’ve got deterministic law with cryptocurrency, probabilistic guidance with AI, and the social network that binds the whole thing together. Those 3 technologies are like social technologies that are almost like the reactor core of the network state of a modern internet-first society.

Martin and I were talking offline about how amazing your thread was on AI. Normally, or often, you’re a crypto guy and a network-state guy, but you’re a technologist. You’ve been thinking about AI for quite some time, so why don’t you trace us through your evolution a bit?

Totally. Martin and I are roughly contemporaries at Stanford. I got my PhD in 2005 or 2006.

I got mine in 2007.

Yeah. I’m a little gray down over here, and Martin’s a little gray up over here. We’re complementary gray.

We both have that ambiguous kind of Middle Eastern look.

Exactly. Middle Eastern.

That’s correct.

That’s exactly right. So we’re both men of a certain age, I think is fair, as the phrase goes. The funny thing is, I actually taught machine learning and computational statistics in the context of genomics at Stanford for the mid-2000s, and founded a DNA-sequencing company. My original career expertise for 10 years was, in a sense, machine learning full-time.

Then I got into crypto in the early-to-mid-2010s, just as the deep-learning revolution was getting underway with ImageNet and that series of papers in the early and mid-2010s. I have a foundation in probability and statistics, multivariable calculus, and so on, so I’m conversant with the space.

The thing I will say—and I’m sure Martin has thoughts on this, and we’ll get to the specific tweet—is that in the 2010s, we were all tracking diffusion models and language models and so forth. They were improving, but style transfer, for example, was working by the mid-2010s, right? GPT-2 and so on were interesting. It could kind of blurt out a sentence, but I admit that I never really thought it was going to get past Markov-chain-like stuff.

I was surprised at how much better GPT-3 and DALL-E were—a hint in early 2022—but I was surprised at how coherent ChatGPT was. I think everybody was. It was a huge jump up from what had come before in terms of being Markov-chain-like.

I’ve been observing over the last 2 years, being originally very deep in machine learning and then very deep in crypto. You can’t be deep in everything. You can’t be deep in everything—maybe Elon can. Aside from Elon, it’s pretty hard to be at the cutting edge of so many different fields at the same time because they’re deep fields with a lot going on.

With respect to AI, I think there are several realizations I’ve had over the last few years in terms of the unarticulated limitations of the space. Some of these I think I came to relatively early in the history of modern AI, and others I’ve come to more recently, but let me enumerate them in no particular order. I’ll make Martin jump in anytime.

The first is that something that motivated Eliezer Yudkowsky, Sam Altman, and a bunch of other folks who built OpenAI was almost the same kind of sentiment that motivated people to build the Sistine Chapel. It was sort of an implicit Abrahamic monotheism, which was like summoning God—communing with God, but in the Abrahamic God sense. That also meant the vengeful God who would turn you into paper clips, or turn you into pillars of salt. That was sort of an implicit thing behind it.

When they talk about AGI, they talk about AGI as implicitly a unitary thing. We will get to AGI, and then it will go to infinity, and we’ll be raptured into a singularity kind of thing, even though that’s implicit.

Because I’ve been thinking so much about crypto and other kinds of things, I was like, “Well, there’s a different sort of implicit school of thought, which is polytheistic AGI.” Rather than the vengeful God, do we have a war of the gods? Do we have many superhuman intelligences, all from different cultural backgrounds, that have the mores and values imprinted on them?

Very early on, I had a tweet that said, “At a minimum, there’s going to be American AI and Chinese AI.” If we’re lucky, there’ll be decentralized, open-source, crypto-style AI. At the time, American AI was highly woke. This was 2022. I knew China was going to stop at nothing to copy it, so I knew we were going to get at least 2. I thought we might get N if we were lucky enough to get decentralized.

That wasn’t obvious at the time because the cost of training models was so high, and OpenAI was so far ahead. It took a while before other people caught up. But now it’s very clear that we’re going to have lots and lots of high-quality, open-source, decentralized models. A new one comes out almost every week, and China is going to work hard on this. That’s a big thing, with all the DeepSeek models and so on.

So polytheistic AGI, I think, is one very useful macro frame because it takes away some of the AI-apocalypse tones. I don’t think image generators or text chatbots are going to cause the destruction that people thought they were going to cause—that they would bust out and do systems programming. Martin can speak about that. Martin and I were talking about how there are certain limits, and it’s now more clear that they’re worse at systems programming than they are at visuals. We’ll come back to that.

What does polytheistic AGI mean? It means every culture has its own AGI, and eventually every culture has its own social network, cryptocurrency, and AI. The AI is sort of like their oracle at the center of society, and they’ve got deterministic law with cryptocurrency, probabilistic guidance with AI, and the social network that binds the whole thing together.

Those 3 technologies are like social technologies that are almost like the reactor core of the network state of a modern internet-first society. They’ll be customized for each different kind of group, and certain things will be disallowed or allowed. Image generation might not be allowed in some subcultures, or NSFW content, or whatever. All that kind of stuff would be tweaked.

That’s 1 concept. The second concept is about Eliezer Yudkowsky, who, even if I disagree with a lot of his ideas, did a lot to promote AI and get people into the field. Even if I disagree with bombing the data centers and various other things, I give him significant partial credit for getting people motivated to look into the space. Directionally, there was something there, so let’s try to see the right side of things.

The second big concept that I really disagree with, and that I think is being borne out, is the idea that AI could just cogitate for millions of years and figure things out, and could outmaneuver you all the time. We know that’s not true because turbulence, chaos, and cryptographic equations don’t work that way. You can come up with turbulent or chaotic systems where you simply cannot forecast indefinitely with finite-precision arithmetic.

In fact, you get fracturing and breaking. Cryptographic hashes are set up in such a way as well to be hypersensitive to initial conditions, where a small change of 1 character can get a totally different MD5 sum or something like that.

You could come up with a thought experiment where you inserted turbulence or chaos into your decision process—sort of like shaking a turbulent clock before throwing a pitch—and the AI wouldn’t be able to predict your actions. That’s a simple experiment in real life. The flow of a fluid is turbulent, so that actually puts bounds on what AI can predict—quantitative, physical, and mathematical bounds on what an AI can predict.

Can I just add a little bit of color here? I think this is great. We need to call out why the way that you describe AI as gods, and monotheistic and polytheistic, is great at describing how human beings view AI, right?

Gods, and monotheistic and polytheistic.

It’s great at describing how human beings view AI, right? But in reality, we’re talking about software running on computers that are bound by those limitations. I don’t view them as gods personally. I got my PhD in systems, so I view them as system software.

I actually think the original sin in all of this AI—the anthropomorphic fallacy—started with Bostrom, right? It was 1 of these thought experiments. When Nick Bostrom wrote Superintelligence, he was talking about this Platonic ideal of AI, and this Platonic ideal of AI just happened to be able to recursively self-improve and just happened to have these superphysical capabilities that no AI today has.

But it happened that the conversation around AI started then, and then, coincidentally, LLMs showed up 4 years later. Somehow, our thought process on these 2 things dovetailed. People would take all of these mental ruminations and thought experiments and apply them to actual systems.

The problem with taking some Platonic ideal—whether it's Bostrom's, the Abrahamic view of God, or any kind of religious view—is that it is very blinkered to limitations, and you've pointed out a great limitation. We've been doing computer simulation forever. We totally know the limits of simulating physical phenomena, particularly chaotic systems. We have very hard bounds on these. It's not just the limits on the size of a computer word or an integer; there are actually very strong limits on time and the amount of compute necessary.

You can talk about these in 2 ways. For this conversation, we should talk about what this Platonic ideal is and how we should have a mental model for it for the non-computer specialist. What I would love to do as we go through this conversation is talk about how these are still bound by computer systems. We know the limitations of computer systems, so let's see how they're bounded. You beautifully did both of those things in the same one. I just want to make sure that we tease apart both of those things as we have this conversation.

Totally. I feel I can speak both languages here. I understand where those guys are coming from because I'm also a tech radical, but I'm also a tech pragmatist, so I think I straddle that boundary. The danger is that if we don't say this is a Platonic ideal, people will map it to existing systems incorrectly.

That's exactly what happened in 2020 and 2021. They took this thought experiment that started with Bostrom—and if you go back to the original book, you're like, listen, this has nothing to do with real systems—and applied it to a real system. I think that was kind of the original sin.

Totally. Totally.

I know, I know. Well, let me poke at that a little bit, and then both poke at it and defend it. The Turing test was a thought experiment that was a Platonic ideal, with no reference to neural networks and no reference to implementation details, and yet it served as something that went from a thought experiment to an applied thing with, you know, the Lovelace test. The CAPTCHA was like V0 of that. You can argue that it became commercially important, and now obviously AI has blown past the Turing test. It can be people in this—

There’s also the Chinese room, John Searle's thing, about machine translation—another thing that was a Platonic ideal. There are other things like that, such as six degrees of separation, which social networks actually made real. So I'm not necessarily against Platonic ideals.

Thought experiments are very important. I just think we need to be very clear, when we're having a conversation, not to conflate them with the actual system.

But that's the thing we've kind of fallen into in this conversation—not you and me, just the broader discourse.

The broader conversation. That's right. So now your point, which is a very good one, is that these are real systems and they actually have real limitations. I think one of the more interesting things for me over the last few months and years has been defining exactly where those limitations are, because I think where they landed was kind of counterintuitive.

One of them is that you have this decentralized AI rather than AGI. That alone, I think, kind of nukes a bunch of the concepts of “we'll get to AGI and just win,” because it's clear that there's a rapid onrush of new models, and it's more of a continuous kind of thing. The fast-takeoff scenario didn't happen.

On the other hand, can an AI write a sonnet? It can. It can do it better than most humans. Can it write a screenplay? It can do that, again, better than most humans. There are a lot of things that we thought might be harder. We thought locomotion might be easier to solve than what we think of as higher cognitive functions, but it's actually locomotion that's still harder in some ways.

Why do you say that?

Yeah. So, for example, when you're competing with a human brain on 3D navigation in space, you're competing with a 4-million-year-old mammalian brain in a body that's been running away from predators and picking berries for 4 million years. It's incredibly highly evolved.

When you're competing with a prefrontal cortex—which handles language learning and creativity—how old is it? 250,000 years. If you take this question from an economic-equilibrium standpoint, you're saying, well, do you want to compete with the most evolved system that's solving the much more difficult problem, which has much higher dimensionality and has to deal with chaotic, nonlinear systems, as you said? Or do you want to deal with the very new evolution that deals with a much denser space and actually does a pretty good job with linear interpolation?

The problems that work very well with linear interpolation are the ones we're solving. I think you're totally right. It wasn't obvious. AI has been solving what we thought was the easier problem, but our fallacy is that it's easier for humans because we're really good at it, because we've been doing it for a really long time. It turns out the harder problems are just harder for us because we've only been doing them more recently.

This is a great case where our intuition about the problems to solve was wrong because of our own anthropomorphic fallacy—our own notions of what problems are easy and hard.

Yeah. I mean, one of the surprises I had with GPT-3 and ChatGPT was how far you could get with language.

Yeah.

Before the ChatGPT moment, it wasn't obvious to me that language was sophisticated enough to encode almost any concept about the world. Or rather—

If I put you in a very dark room—and that's an arbitrary construction—and I describe how you navigate and pick something up, I think you'd have a very tough time doing it.

No, no. I know what I'm saying. Basically, if you generalize language to be streams of symbols—

Yeah.

Right. You could send telemetry to somebody. Basically, I think I was surprised by how many concepts were encoded in language that could be learned, even to the point of rough world models—like the map of the Earth or the proximity of things. You can back out those kinds of things just from this.

The distinction is that it could have been the human being who looked at the world, did the reasoning, and created the world model, and that is cached in language. Then language is—

Yeah, the same thing, but from the world to the world model was the human, and then everything else, and then the language—

I agree. Yeah.

That's right. But it was a little surprising to me that you could get that far with language models, as opposed to spatial reasoning—that predict-next-token would get as far as it did. That was surprising to me. The reason is that you and I both did so much stuff on Markov chains and conditional random fields. Of course, the transformer is a different architecture, but I just wouldn't have believed that method, taken to this scale, would get as far as it did.

Another thing I think was very counterintuitive for me in the late 2010s was double descent. As a classical machine-learning guy, that's just very counterintuitive: that you could go past overtraining and back into a good regime.

But I want to talk about one thing you did say, which is self-replication. I don't think of that as a forever constraint on AI. I think of that as a today constraint.

The reason I think of that as a today constraint is that they're not embodied, so they're not in robots. Because they're not in robots, they can't build data centers and mines and replicate themselves, and so on and so forth.

The whole concept of consciousness initially was that consciousness evolved so that, if you're running away from a boar or something like that and there's a branch ahead, you have a model of yourself and know whether you'll fit under that branch or not. If you just had a generic model of the self, the more self-conscious you are, the more you can simulate your run under that branch—whether you're going to die or survive.

That's one theory for why consciousness arose: to help with survival. Reflection helps you stand outside yourself, to be able to see the—

That's right.

That's right. So, right now, AI does not really have goal setting. It doesn't have reproduction or embodiment, and it can't act independently of humans. This is one of the big things I think people were really scared about in late 2022, and they've calmed down on this.

The thing that you and I both poked on, Martin, I think, was: Is this thing going to jump out of the box and code itself? We laugh at that now, but the reason I think that hasn't happened is that AI can't prompt itself yet. And prompting, I argue, is actually a much harder thing than people realize.

Did you see—did we talk about my analogy of the spaceship?

We were talking about this point.

Okay, so let's say you have a really fast spaceship, close to the speed of light or something. You still have to point at the phi and psi coordinates on the surface of a sphere, in coordinate space, to determine where you're going to point that ship.

If you're going to take it on a journey, then you have waypoints: Here's this heading, and here's that heading, and so on and so forth, like a series of phi-psi pairs on the surface of the sphere. That's only 2 floating-point variables, right? By contrast, how high-dimensional is the vector that you're giving as input when you talk about a prompt?

If you just take UTF-8 code points, or even ASCII, and you have a few words, that's much higher-dimensional than a vector of 2 floating-point variables, right? So, you can get to a very high level of dimensionality in terms of the direction vector you're pointing this AI spaceship in.

A prompt is a very high-dimensional direction vector, even if you account for the fact that many potential prompts consisting of strings of random characters wouldn't be interesting. It's still a very, very high-dimensional vector. So, it's like you've got a fast spaceship, but you still have to point it in a direction to go somewhere. I think that's a good analogy.

Well, I think there's one more level of complexity you have to add to your analogy, which talks about how difficult it is to make these things—let's call it—autonomous, which is closing the control loop.

That's the verifying part. Go ahead. Yes, go ahead.

Well, it turns out that the directions that you point in have to be understood, right? They have to be in-distribution. You can't point it in a direction it doesn't understand, because it does the worst thing if you point it in a direction it doesn't understand.

It just crashes right into that wall, or it tries a random direction. It's the worst thing ever, right? And so the problem is—

If it's producing a direction that has to go into itself, it doesn't know what it knows, and it doesn't know what it doesn't know. Does that make sense? So, it could produce—

In fact, it's optimized to fake it.

Yes. So, if you could tell it—if you could say, “Hey, listen: Produce a bunch of directions by feeding the last direction in”—you have no idea whether a direction it spits out is going to be in-distribution when it comes back in. And that's what closing the control loop means. It's a very tough problem.

I think this is such an important point for us to go into, because in theory you can close a control loop on these things. But as scientists, we want bounds on what that means, right? For example, clearly you want to gather new information to update your model. Then we want bounds on how much information you need to gather.

It turns out that the model was trained on everything humans have ever gathered, so is the incremental experiment going to update that? Maybe information theory says probably not, but we don't even have bounds on these things.

Now, you said previously that, when it comes to a chaotic system, we know that computers take a long time to compute a nonlinear system.

Actually, I just want to pause you there. You just gave me an idea for a great prompt: What areas do you feel your knowledge is thinnest on?

This is the key. That's a great prompt.

Yes. Does a model know to what extent it is in-distribution or out-of-distribution?

Yeah, I'm going to try that one.

But this is the key. Self-reflection is the key, because if it produces an output that's out of distribution, then of course you have error in that, and you're not there to nudge it back.

Yeah. So, it's like real-time events, obscure or niche academic fields, specialized subfields behind paywalls, local and regional information, human emotion, intent, or experience, and private or proprietary systems. Actually, that's a pretty interesting, quick off-the-cuff response, right?

You should ask if it can always produce a response to which it has a lot of data. If you're closing the control loop, it's going to spit something out that you're going to feed back in, right?

That's the whole point. The question is, will it always spit stuff out that, when you feed it back in, will give you nonsense?

Oh, I see. Yeah. I mean, people have actually tried that experiment: Take the image and exactly replay the previous image, and then it morphs into something totally different. Another angle I have on AI is that prompts are tiny programs. That idea is more common today, but I think I articulated it and it went viral a while ago.

They're programs in a hidden API, because normally you have an API that is fully documented but very error-intolerant, right? Prompting is the opposite: It's completely undocumented but highly error-tolerant. It will usually do what you mean.

But the better your vocabulary, the better you can prompt it. So now art history is an applied subject, right? Knowing a vocabulary like Cézanne versus Picasso means you can actually pull up the style that you want on demand. So, the broader your vocabulary, the broader your subject knowledge, the more you can get out of it.

We're in the age of the phrase, which is the prompt, the 140-character tweet, and the 12 words for your crypto password. These phrases of power in AI, in social media, and in crypto just unlock everything. So, the better your vocabulary, the more you can do, right?

I think of prompts as tiny programs, and one of the things I've gotten in the habit of doing is writing them. It's the total opposite of search. With search, you learn to type things in keyword-ese, and you figure out the word that has the most specificity—TF-IDF, you know, on the page or whatever.

I will sometimes write these long memos to an AI. Continuing the polytheistic analogy, I'll give them to Brahma, Vishnu, and Shiva. I'll give them to ChatGPT and Claude, and now Grok and whatever else, right? I'll consult all the gods, and then I'll make my decision on that basis. Sometimes I'll have them argue with each other.

Why do I say “gods,” kind of half-jokingly? The Hindu frame on that is not the “fearing God” thing. It's not the same kind of thing, but in a sense it is a superhuman intelligence that knows everything about your culture, and if you ask it the right question, it can tell you something that you didn't know.

In the Hindu tradition, they're not infallible, right? That is to say, it's not the same as the all-knowing, all-seeing. It's more like superhumans, almost more like superheroes. People will argue with me about that, but I think that's more true.

Yeah, the Norse tradition is similar to the Hindu tradition in some ways, right? The gods were not infallible, but they were superhuman.

I think this is a great and useful framing. Just remember that, when it comes to computer systems, we can put formal bounds on them. We can do this information-theoretically; we can do this computationally. That's going to come, and once that happens, we will understand these systems fully. It'll be very hard to think of them as gods at that point.

Well, that's right. The interesting thing is that the interpretability work that Anthropic and others have done, and the work on grokking or what have you, is actually really good stuff. You can pick apart neurons, and you can find the Golden Gate neuron if you saw that kind of thing. You can dial that up and dial that down. You can start actually taking apart these AI brains in a way that hasn't happened before.

A few other kinds of things: That thread you guys made actually summarized several of my recent ideas. I should probably put this into a post so it's there for the record. I'm just going to do a bunch of these and maybe get your thoughts, right?

Okay, so first concept, in no particular order: AI doesn't do it end to end. It does it middle to middle. Basically, you still have to prompt it, and then you have to verify it. People talk about prompting, but they talk less about verifying. Karpathy and I had this good conversation a few weeks ago where, basically, AI is going to create massive numbers of jobs in proctoring and verification because it's so good at faking things.

So one of my other concepts is: AI makes everything fake, and crypto makes it real again, because AI is a probabilistic technology and crypto is a deterministic technology. Crypto is, in some sense, what AI can’t fake. It’s the hard cryptographic equations. It can’t fake a Bitcoin private key. It can’t fake even an on-chain NFT. That’s what AI cannot fake. Those are the hard barriers, right?

I generally agree. I don’t think crypto solves the grounding problem, right? I mean, it’s a mechanism you could use, but—

It’s a mechanism. The grounding problem is—

So, when you say “ground,” the grounding problem is—

Grounding in reality.

Yeah, the actual physical grounding problem. Yeah.

All right, I disagree with you on that, and here’s why. Or let me give you a counterargument, at least. Let’s say you ask Perplexity to summarize the FTX hack in 2022. Among the citations it would give you would be links to a block explorer, right? That would actually have on-chain data that you can cryptographically verify, showing that this transfer of these funds happened at this time. If you want to go even further, you can pull out the digital signatures, hashes, and timestamps from the block explorer.

Now, here’s my argument. That works for financial data. But what’s happening now with Farcaster and other kinds of things is that, with the increase in block space, you could put more and more kinds of data on-chain. We’re going to have to, because you’re going to need crypto instruments, cryptographically hashed posts, and crypto IDs to know that it was posted by a human or to know that the data wasn’t tampered with.

More and more kinds of data are going to go on-chain, and then that will eventually mean that an AI’s citations are to on-chain data, which is both financial data and social data. At least then it will map back, in terms of grounding, to an on-chain, cryptographically provable assertion of some kind.

You might say, “Well, at least that’ll be an assertion at the metadata level.” We can prove that this digital signature made this assertion at this timestamp, with this probability. I’m talking about real-world grounding. I say something; I am a human being. You have no idea whether what I said is true or not true. There’s a geographic place where there’s a picture of the geographic place taken from a 1970s photo. Was that doctored or not? The actual physical-world grounding problem remains, simply because you can’t yet encode the physical world as digital data.

No, and overall, it’s a great mechanism to do that once we can solve the ingest problem. But—

Let me talk about something that’s happening now that I’ve been funding on the side. It’s not a full solution, but I think it’s a partial solution: crypto instruments. The idea would be that, when you capture a frame of data—for example, with DNA-sequencing machines—the data coming off the machine is TIFF files, which are actually image data that gets processed to A’s, C’s, G’s, and T’s. Many other kinds of instruments have a stream of data coming off the machine as you’re capturing it. Cameras are like that, right?

You could—and there are things that do this already—take a hash of that and post it on-chain at that time. What that would at least say is that that frame of data existed at that time. If you had something like a scientific experiment, such as a preregistered, double-blind trial, you could have not just a crypto instrument, but also other people with proof of humanity there who have a sort of attestation ceremony. Now you have a number of different kinds of on-chain data that start to get harder to fake in a coordinated way. Not impossible, but harder.

Totally. As soon as you get it into the system, crypto is a great mechanism for ensuring end-to-end guarantees. The data-ingest problem is a long-standing problem in computer science, and over time everything you’re saying is going to become more and more true, because we’re going to be increasingly incentivized to make sure the stuff going into the system is true.

I totally agree. I’m an old-school networking guy. For us, there’s the internet, and there’s the stuff that goes into the internet, and then you use different mechanisms for both. It’s worth calling out.

That’s totally right. Okay, great. So, the next concept to discuss—and I think this is a useful division, a relatively recent point that I made to myself and thought was useful—is that AI is good for the visual and less good for the verbal.

What do I mean by that? When it’s generating images, video, or user interfaces, like Vercel’s v0 or Replit’s user interfaces, the great thing is that you can instantly see them. With the GPUs we have in hardware, you can cheaply verify whether they’re good enough, because you can instantly get the gestalt of it, right?

Whereas when it’s backend code, legalese, or mathematical equations, you have to slow down and use System 2 thinking, not System 1. It’s not just your gestalt impression. You actually have to go line by line and check whether it’s right. That is the expensive step: verifying.

That’s a nonobvious thing. The more front-end, video, and visual the task is, the easier it is to verify.

For me, I spend most of my time in software and engineering, and the big distinction is stateless versus stateful. If you’re generating code that’s going to have semantics that evolve while you’re running it, it’s impossible to spot-check. Some things are computationally irreducible; you actually have to run the computation to get the answer.

The image is the perfect example. It’s visual, and it’s basically stateless. All of the state is there. There are no runtime semantics.

Totally agree.

That’s right. Whereas even a relatively small snippet of backend code could have a fairly complex finite-state machine underlying it, or even an infinite-state machine. Simulating the time dynamics of that requires something different—maybe formal verification if it’s algebra.

Or you actually have to run it if it’s computationally irreducible. There’s no way to do it statically. It reduces to the computation-verification problem, which has been a longstanding problem in computer science.

Formal verification, at least for a subset of programs, has become commercially viable for smart contracts because they’re so high-value and so small that it’s worth doing. It’s not going to work for the general case, but you can do a constrained case.

That was one major division: visual versus verbal. Another, when you get to stateful systems and so on, is that I think the limits of AI are the things I’m interested in—the fine distinctions around what it can do. What I think AI is particularly bad at, and what people are trying to use it for and are going to fail at, is markets or politics.

Let me explain why. For systems that are time-invariant, like mapping an image to the label “cat,” or the rules of a game like chess, checkers, or even Go, or something with a static rule set or static mapping, you can use the train-test paradigm and train a model.

However, when you have something that is time-varying, especially rule-varying and adversarial, as markets and politics are, the same trade will quickly start resulting in a loss. The other participants are also using AI against you. It’s decentralized AI again.

That argues that the CEO, influencer, or creator who is constantly sensing the market or sensing the political winds and has a thesis based on human nature or other things is the sensor that prompts the AI. That’s a job that is hard for AI to do at a really deep level, because the system is time-varying, rule-varying, and adversarial.

It goes back to what you said in the very beginning. If you look at these types of equilibria, they’re complex differential equations that are nonlinear. In order to predict what’s happening, we’d have to do this nonlinear extrapolation, which we know these things aren’t very good at.

In fact, I wasn’t even thinking of the stock market as complex differential equations, but you’re right.

Mandelbrot wrote this great book about the fact that these things are chaotic. They’re super chaotic.

Yeah, you’re absolutely right. It would be useful to show, with a toy example, a chaotic system that is time-varying, or another one that—

The thing about that, though, is, to argue against my point, they’ve gotten AIs that are actually pretty good at StarCraft, which starts to stretch the boundaries of what I was saying because it’s definitely adversarial. It’s more time-varying than chess. You could argue it’s not rule-varying, but it’s time-varying.

Let me go to another point here. The commercial implication of that point on prompting and verifying is that business spend moves toward prompting, proctoring, and verifying—basically checking all the stuff that AI can generate. That’s going to be a huge, huge, huge thing.

That maps to KYC. In a bad way, it maps to the glass cases in Walmart. In a sense, a low-trust society is spending more and more on verification and proctoring, and so on.

Next. AI means amplified intelligence, not agentic intelligence, because the smarter you are, the smarter the AI is. Better writers are better prompters. What are your thoughts on that?

Yeah, I think it’s interesting in the coding space that we actually start to have numbers on this now.

Oh, interesting.

Yeah. So if you actually look at relative productivity gains, it just turns out that if you’re a more senior developer, you will have better productivity gains.

Oh, I hadn’t seen that graph. So actually, it is something that makes the smart smarter, basically.

But also on a relative basis, which is really surprising, right? And then, if you think about it, it’s actually not surprising. You know what the fundamental trade-offs are, you know what to ask, you know how to interpret the results, and you know how to throw away bad stuff when it’s bad. So clearly, if you kind of know what you’re doing, you can both verify, to your point, the output, but you can also be more specific in your asks.

I think it’s really important for all of us to realize that formal languages came out of natural languages, not the other way, right? If you could explain all of this stuff in English to each other, we would, but it’s just really inefficient. So we came up with more efficient ways to explain trade-offs.

Constrained.

More constrained.

Yeah. Basically, constrained languages that reduce ambiguity.

This is strictly an efficiency thing, right? Someone who knows how to speak these formal languages to the models is going to articulate what they want better and is going to be able to interpret the results better if the response is formal. So it is kind of a nice codification of exactly what you’re saying.

Yeah. The thing about it is AI means everyone’s a CEO, because you speak to the AI like you do to a great employee, where you give clear written instructions—

And then you can verify the output.

And then you know, it actually kind of turns management into a skill. It hyper-deflates the cost of trying one’s hand as a CEO or as a manager, because you have to give those instructions. So the better you are at communicating what it should do, often the more people you can manage, and so on and so forth.

By the way, this gets to the next point, which is AI doesn’t really take your job. It takes the job of the previous AI. What I mean by that is you now have a slot on your roster at every company for an AI image editor, an AI text or chatbot tool, an AI coding or IDE tool, and so on and so forth. Each new release of Grok or Claude or whatever competes against ChatGPT, Grok, and Claude, right? So the AI takes the job of the previous AI. Because they’re complementing you, you kind of have a whole raft of AI augmenters that are augmenting your humans. But those AIs are competing in AI space, to a large extent, with the previous AI, because once you’ve onboarded an image generator into your flow—

Then it just keeps improving, and you start using it in more places, but it’s an AI taking the job of the previous AI. Let me know your thoughts. Can I actually—this is adjacent to what you’re just saying—but can I push on something you said previously? I actually agree with your polytheistic view of the world. I totally agree, but let me provide the counterargument for us to noodle on here: Have you seen this kind of thing, where all the AIs you ask to produce a random number produce the same number? Have you seen this?

Like 4?

Yes. It’s like 7 or something like that, right? One thing that was non-intuitive but remarkable about these models is how easy they are to distill. As soon as someone creates a leader, everybody uses that leader and kind of sucks the life out of it, and then all the models kind of converge on it very quickly—

Which you could argue is a counter—

To the polytheistic—

Yeah, kernel intelligence, basically. There’s like a core—

Maybe there are 10, maybe there are 100 AIs, but it just turns out they all have the same capabilities. Is it just a technicality that they’re actually different and they’ve all learned from each other?

Well, it’s an interesting question, and my view is—and I’ve not called this a strong view yet—but my view is that’s almost like the human body plan and spinal column, and then you differentiate on top of that core spinal column, maybe. It’s like you’d have some sort of—

It’s like every human, to first order, can see and speak and hear, and so on and so forth, but some people have much better vision, or they have much better speech, or something like that, right? So there may be some distilled kind of thing.

By the way, another interesting part of what you’re saying in general is that text on the internet is not emitted equally by every group. For example, liberals tend to write more text on the internet, and conservatives tend to be more visual. So you’ll actually have an ideological skew that’s hard to unsee, because the people who are training AI disproportionately are writers.

I totally agree with what you’re saying. I’m just saying the counterargument. By the way, I agree with what you’re saying, but the counterargument would be: You just ask the AI model, “Conservatives don’t write so much, so give me the answer using the mediums that they use,” or whatever I mean—which is, all of that information would be in the model.

Here’s how I would distill my view on this, which is very much in line with what you’re saying: I think the universe is very complex, and I don’t think it gives up its secrets easily at all. I think the universe is full of fundamental trade-offs. You can’t have both; you have to choose A or B.

And so these models will align with those fundamental trade-offs, right? Maybe it’s performance, maybe it’s correctness, maybe it’s whatever it ends up being. As soon as you want a specific solution for a given problem where it hits one of those trade-offs, you’re just going to need a different model, because otherwise you just can’t end up having both. Again, because I know the coding space the best, we see this a lot, right? A model that’s very, very good for certain parts of code is just not going to be generally good at other things, because those are the trade-offs made when training it. I think this is kind of the future plurality of models.

Well, is that true? I thought somebody said something—I may be wrong about this—but I saw some counterintuitive result that said making AI specialized in one area makes it worse in other areas. Did you see something like that?

Well, yeah. This is a very big debate, but the debate goes as follows: The first wave of AI was pre-training, where everything you threw into it just got smarter. That’s kind of a 10-for-10 technical win, right? It’ll be as good at writing code as Sonnet is. But as soon as you’re doing RL, where you’re training it in a specific domain with a specific verifier, you’re likely losing other areas. So if you make it really, really good at playing chess, it’s going to be less good at something else, like writing piano scores.

I think the current debate, and the current data, seems to suggest that RL doesn’t generalize in the same way. So now we are in this case where you would have a plurality of models, because you’re always kind of robbing Peter to pay Paul when you make it good at a certain domain. That’s right.

Yeah. I think also, somewhat related to that, in terms of what domains it’s good at and so on and so forth, at least right now, I think—I’m not sure if you agree with this—AI doesn’t really take your job; it allows you to do any job, because you can get to an okay level as a user interface designer, sound-effects designer, or something like that. But you need a specialist for polish. Though, I wonder, maybe with enough RLHF from specialists, maybe that won’t be as necessary.

So here’s my current mental model. There are 2 personas. Persona number 1 is the expert in the space, and persona number 2 is the non-expert in the space. If the non-expert in the space is using AI, it’s taking the place of the expert, right? Maybe you’ll ask it and it’ll give you something. Let’s say I want a 3D asset for a video game and I’m a programmer. Then I’m going to ask it for a nice 3D asset, and then it’ll give me 1, right? So I’m the non-expert.

The expert user, to our previous point, will actually know how to ask it better, and it’ll likely get better results because they’re actually a domain expert and that expert is using it. I think we see both of these. If you look in the market, you see both of these uses, and I think both of them will persist. If I’m a programmer, I—

It’s like a doctor talking to their AI, and they can instantly go to specialist language. So you’re right: Even if there’s specialist RLHF, you may not be able to—

Yeah, that’s right. Why would I want to learn the entire domain and make all of the trade-offs of 3D design when somebody else could have done all of that work for me, and they could talk to the model in the specialist way, when I can just talk to my model using code? In order to very efficiently use a specialist model, I would have to become a specialist, would be the argument to our previous point.

So I think, listen, for casual use, I can use these models for whatever I want, but to really use them very well, again, to our previous conversation, I’d have to become a specialist, and somebody else may have already invested all of that time.

By the way, if you actually look at products, these products have both of these distinctions. Some products are very clearly for the casual user trying to replace—think about Cursor versus Lovable, right? Lovable is, “I’m a casual user. I want to create a website. I don’t even have to know about code.”

And that’s great. You can create amazing things.

Cursor is, “I am a professional software developer. I have an IDE, and I know an IDE.” Over time, maybe these things converge. That could be the case. But thus far, these are very different user bases, right? There’s professional coding versus basically casual coding.

If you think about what a computer can do, it can do the job of an accountant. It can do the job of a physicist. It can do the job of all kinds of professionals, but then you clad it in something, and it’s adding up numbers in Excel. You clad it in something else, and it’s doing simulations for MATLAB, right?

We already know that when it came to logical System 2 thinking, computers were actually really good at that—much better, superhuman at it. Now you have something similar where these models are very versatile, but you clad them in the power-user interface, and you clad them in the casual interface. There’s implicit contextual prompting, probably as well as a system prompt, that makes them do those things.

The thing that’s interesting to me about something like chain-of-thought—and I think this is where people were freaking out in late 2022, and maybe they’ll still be right to freak out—is that computers had historically always been good at the logical style, much better and superhuman at that. Now they’re also superhuman, in a sense, at the probabilistic style, at least in text generation and so on.

It’s not inconceivable that someone could figure out a way to merge those two—a quantum-gravity kind of theory of things, where you take the probabilistic and the deterministic and pull them together, right?

Yeah. This very old-school-systems part of me thinks that there’s a fundamental trade-off here, which is that you can trade off—you can build a system for determinism, and you could build a system that basically cuts a bunch of corners, but you can’t build a system that does both, right?

What’s that?

Well, I feel like you can build a system for determinism, and you can build a system that basically cuts a bunch of corners, but you can’t build a system that does both, right?

What about the tool-use stuff? AI has gotten pretty good at figuring out when it wants to generate an image, when it’s supposed to search, and when it’s supposed to read a PDF, right?

So now you’re acknowledging exactly what I’m saying, which is that some things you want the fuzzy thing, and some things you want a traditional system, like the tool use, basically. Yeah, for sure. But then maybe it just becomes a consumption layer, and all that hard work is still being done by traditional systems, right?

Well, I’m just saying the full argument is: if you have a model that does everything, you wouldn’t need tools, right? Tools are literally the API to the traditional system, and so that’s almost like a capitulation that some things you want traditional software to do.

No, I know. But what I’m saying is, if you’re just a pragmatist and you don’t care, could a hybrid system actually get there? I can’t say it couldn’t, right? Maybe it’s something where what we actually need is something like Elon’s billion miles of Tesla driving, if we have enough context—not from LLMs, but from pointer movements and mouse clicks on iOS or other macOS.

Yeah. Yeah. An AI OS could.

Yeah. So your question is: can I have one trained neural net, one trained LLM, that can do both the fuzzy stuff and the hyperprecision? Is that possible? My gut—again, this is total intuition—is that the universe is way too heavy-tailed. It’s way too nonlinear, and so the state space is too high for that to actually encode all of that.

But humans can do it.

No, we don’t. We use calculators, and we use software. The whole reason we built software is because humans can’t do it.

I know, but software is necessary because humans can’t do it.

Well, maybe we’re just talking about two different things. Clearly, AI with traditional software can do great stuff. To me, the question is: can you have one AI that does all of those things without traditional software? Can you build one LLM? I think the answer is obviously no, but I’ve heard arguments that it can.

That’s why I’d say I’m on the borderline of the tech pragmatist and the tech radical. I always want to identify the limitations of the systems today and then see how you could push beyond them. I consider calculators, which were made by humans, to be ultimately humans doing it. It’s a tool that we came up with.

To give you an example, earlier I was saying that AI is good at visual but not verbal. But you can turn audio into spectrograms, which you can then look at visually. At least you can see radical deviations and so on. Maybe not the entire sound, but you can see.

Can we transform things in other outputs of AI so that we could quickly inspect them visually? Is there some grid or visualization where we can turn something into a visual problem?

Yeah, I love this. A thought experiment I have is: can I literally create a bunch of audio outputs so that I can listen to my AI like I listen to a car? When your car is rumbling, you can say, “I don’t know what’s wrong with my car, but I know it’s not normal.”

Yeah. Yeah. You can tell if it’s rumbling. We’re very good at atmospheric inputs, and I think this is a great idea. Can we start exposing the internals so that we understand when it’s working well and when it’s not working well?

Yeah, like colored text, for example, in terms of its level of confidence. Yellow, red, green, right? There’s a lot of AI UX that one can do.

Let me make a few other points that I think are interesting. Killer AI is already here, and it’s called drones, and every country is pursuing it. We don’t really have to care about the image generators and chatbots. All the worry about superpersuaders or whatever is pretty stupid.

Strong agree.

Strong agree. And the thing is, when I push people on this, what’s interesting is that some of the people who were saying, “Oh my God, we need to regulate everything,” are now actually on the side of, “We need to build it before China.”

In both cases, first it was safety, then it was security, but it all comes down to control. You might argue that the security argument is a better argument. I think it’s a more realistic argument in some ways. But the concept that killer AI is already here is interesting because they put so much stock in, “Oh, it’s going to persuade everybody to do things. It’s a superpersuader.” Persuading is statistical, and drones are deterministic, or at least the guns on a drone are deterministic.

Yeah. I think the interesting question around AI, attacks, and defenses is: does it change the equilibrium? Did the internet actually introduce the notion of asymmetry, where the more that you rely on it, the more vulnerable you are? The United States is more vulnerable than some random third-world country.

It’s not clear to me that you get the same thing with AI. It could just enable everybody to have bigger weapons, but the equilibrium is the same.

Well, I think that it actually has really huge impacts for borders. Unfortunately, I think China is well positioned here for a very specific reason: their justification for the Great Firewall is that they’ve justified it as digital borders. They say, “We can intercept physical packets. Why can’t we intercept digital packets?”

And now, with the whole Ukraine-controlling-drones-in-your-territory thing, that becomes more than simply a metaphor. It’s a real thing. It’s like controlling cloud space, right?

If you can allow somebody to script drones or script humanoids in your jurisdiction, then they can blow things up, right? That’s no longer theoretical. The counterargument is, well, maybe you just have them preprogrammed and autonomous, so they don’t even need an internet connection and can just do it with cameras or whatever.

That’s true. In Ukraine, there are these drones on cables—the crazy thing they’re doing there.

You know those big unwinding cable things that you sometimes see on ships, right? They have these cables for one-way drones. They’re ridiculously long, multikilometer-long Ethernet cables, like Cat 1 cables, on a drone, so it can be offline. It goes past signal jammers or something like that, then goes and blows up on its target. It goes forward, and the cable gets tangled in the trees.

They don't care. It just goes because it's not going to go on a reverse trip where it has to yank the cable, so when it's a one-way path, it doesn't matter. It's a one-way drone, which is kind of crazy. So that's an argument that, at least at short distance near the border, drones wouldn't be able to get in, right? But that concept of digital borders becoming hard borders, I think, is going to become more of a thing.

Basically, your immigration policy becomes your firewall because, with telepresence, you can move robots around, and that's starting to become real. So that is something where I think that has real implications for the geography of a country, because the alternative to having so-called defensible borders is basically an encrypted state where you don't even know where it is on the face of the Earth.

What I mean by that is: can you make a map of Bitcoin? Not really, right? It's so dispersed, and you don't know every holder. They're moving around the world, and there's no single map of every miner. Even if you got a map, you wouldn't know if it was complete or erroneous, or outdated, or something like that.

You actually have, in a sense, security through obscurity. So you couldn't just go and blow all those things up, versus something that's outlined on the map, which is sort of seizable and vulnerable in a certain way. That's something I think about a lot in terms of what future borders look like.

You might have hard digital borders, and China might preserve its territory, but those that can't enforce hard digital borders, for whatever reason, can't stop these kinds of drone incursions. Let me know your thoughts.

This assumes drones are not autonomous.

Well, even an autonomous drone would still need to be given a control signal to do something.

No, on the other hand, fully autonomous. You'd just be like, “Go blow up this building. Here's a picture.” A flight on top of a zone would not need any packets.

That's true. And then also, if you think about it, are you going to block every single telephone call in every—it's really difficult.

That needle in a haystack.

We have some super-spooky stories of being in China. I remember this—this is kind of a non sequitur, but I have to tell it because it was so spooky. I was in China probably 10 years ago. I used to work for the government. I used to work for the intelligence community when I was a kid—really, I was a kid. It was in 2001, my first job out of college.

Anyway, 10 years later, I was on a business trip in China, and I called a friend of mine. I was telling him about my day, and the phone dropped. So I called my friend again, and I was telling my friend about my day, and the phone dropped again. I'm like, “What's going on here?”

What I was recounting was where I had been. So I called my friend one more time, and I said, “1, 2, 3, Tiananmen Square,” and the phone dropped. I think there was a bug in whatever software they had, and somehow I had been picked up. But I don't think it's too crazy to assume every conversation on every phone call can be monitored and has been for a while.

Well, now they have something—this is another way AI changes the balance of power—in the following way. In China, they had this saying, which is, “The mountains are high and the emperor is far away.”

Sure. On the one hand—and this is a broad generalization over thousands of years of history—but very broadly, because the state had all the weapons and the army and so forth, that chair could morph into an agent of the emperor, or a CCP guy today, if the government so desired, because there are no true limits. It can just do whatever it wants.

On the other hand, whatever the law is written down, the people just do what they want, okay? They pragmatically say the limit is really the limit of what people can enforce. If the state has lots of power, then any written limit doesn't really matter. But if the state can't find you because you're on the other side of the world, then the written law doesn't matter either.

That's a different conception from the progressive-versus-libertarian debate within the West, where they'll always quote law against each other back and forth: what is written is what is permitted, or whatever.

But AI does change that balance, because now the mountains are never high and the emperor is never—

The long arm is incredibly—

The long arm is infinite.

Yeah. They can synthesize. There was something—maybe you know this thing, Martin—I think it was called TIA, Total Information Awareness, in Iraq at a certain point.

The idea was they had satellites covering Iraq, and every time some guy was putting down an IED or something like that, they would rewind the satellite to find who the guy was who did that and where he came from, and then put a bomb through his window or whatever.

In a sense, it was like tracking someone for their whole life, because you were sewing together their trace through all of these cameras. That's totally possible for China to do now. The difference is that AI makes it possible to—for a long time, that data was ingested, but it couldn't really be parsed or queried because it was too difficult to look through 5,000 hours of video on one person or whatever.

That's increasingly becoming queryable, ingestable, and summarizable in a way that it never was. So I think the real check on something like that is going to have to be cryptography, exit, and so on and so forth. Ultimately, it's getting out of the jurisdiction—having property that they cannot actually seize. You go back again to the limits of power and what have you.

Anyway, let me pause there. Those are some thoughts on the balance of power, since you talked about that.

Okay, last one.

I think there's going to be—and there already is—an anti-AI backlash that's like the anti-crypto backlash, and it will be part of the anti-tech backlash, because a lot of people are not using AI for what we're using it for. They're using it for therapy, or as a companion or something like that.

And that's the top of the pyramid of needs. It's kind of funny: if you actually look, it's self-actualization, spirituality, and therapy. Finally, computers are addressing the further reaches of the pyramid.

Exactly.

The top of that.

That's right. And there's another aspect to it that I think hasn't gotten as much press, but it's interesting to understand. Just like the tariffs are meant to ward off Chinese competition—I'm not sure if they'll work; in fact, I'm skeptical—there's a similar, much less publicized thing happening at many media corporations, where they're unionizing to try to ward off AI competition.

They have union contracts that say editors, owners cannot use AI. So they're making their organizations very brittle. They think they own the market, but they're not allowing themselves to use AI. Eventually, they're going to be beaten by AI-enabled competitors that pull all of their followers and views away from them because they're just more efficient.

So I think that's going to result in an anti-AI backlash. I think it's already kind of here, where, on some artist forums, they'll say, “Are you an AI supporter?” Have you heard that?

You know, AI artists spend as much time building things as traditional artists. It's just a different tool set.

But yes, yes, yes, that's true. Basically, they feel that it's similar to the reaction by master craftsmen—

Yeah—

in the 1800s, right, when mass production started taking over what they were doing in the physical world. This is now happening in the digital world, right?

And the other aspect of this is that I don't think people have thought about the international aspect. If you've got, let's say, a lawyer who's making $200,000 a year in the U.S., or a doctor in the West, and then you've got somebody from the Philippines or India or anywhere in the world who's currently making $2,000 a year, maybe the converged wage with AI plus their IQ—or whatever you want to call it, AI-plus-human convergence—is $20,000 a year.

Which is a 10x increase for the person abroad, but one-tenth the wage for the person in the West, and it radically increases consumer surplus and so forth.

But I do think that that's going to be a big deal in the years to come, and we'll have to figure out how to mitigate that.

To your previous point, I just think this is so important. I agree there's going to be a huge backlash, and I think some of it is going to be rooted in the experience of individual people. My job is shifting, and I am very sympathetic to that. I think we should address it, but I think there's something more pernicious going on, which is—

And listen, maybe this is my cynicism, but more and more I view politics as: you've got pretty sophisticated people, and they have clientele classes, and then what they—

Patron—patron—

Yeah, what they say is basically what will move the clientele class the most. That's what they do: they just look for sound bites that'll move the clientele class.

And actually, the patrons are sophisticated people who can hold nuance in their heads. They know their complex topics, right? But they dumb it down on purpose, right? And what better talking point than AI? This goes back to the Promethean legend.

I mean, we're terrified of technology. You can anthropomorphize it. You can talk about it as gods. I mean, it is the perfect tool to mobilize, and we're seeing this on both the right and the left, right? So this is not in any way beholden to one party. So I think this is the ultimate political tool for any purpose, and we're seeing it used for that. And I think, for that, even more than crypto, by the way, I think AI strikes at the heart of people's insecurities more than crypto ever could. And so I think that this is the big battle, and I think it's going to be bigger.

Well, it's interesting. It's all of the above, right? Because AI is disrupting media, crypto is taking power over money, robots are taking power over manufacturing, and drones are taking power over the military. And, by the way, there's a crypto angle to at least 3 of them: there's a crypto angle to money, there's a crypto angle to AI in terms of constraints, and there's a crypto angle to the drones because you're going to want the control plane for the drones to be on-chain, since that's the part that can't get hacked, whereas the Pentagon could hack it. So I do think, yeah, there's—

This is something going after quite a few—

3,000 years ago, if you said, “Listen, we're going to do this new crypto thing,” people would be like, “But—” But if you said, “Listen, we're going to create AI—these artificial intelligences that have unbounded power”—I think you're really getting at a core human insecurity that we've seen in myth and legend for 3,000 years and probably longer.

That's true. That's true. I think we'll see what happens with currencies, but I think you're right: it's both.

It's both, for sure.

Perfect. Love you, Martin. Thank you so much.