未来的积极愿景：第2部分——NEAR 的 Illia Polosukhin

Welcome back to The Cognitive Revolution.

Thanks for having me back. I'm excited for part two.

Last time, we talked a lot about foundational technology: the journey you went on from being the author of “Attention Is All You Need” to trying to source data from contributors around the world, struggling to pay them, taking a detour into blockchain, and thinking it would take just a few months. Here we are a few years later, and it’s all really happening.

Anybody who’s listened to this feed for more than a minute knows that I often say the scarcest resource is a positive vision for the future, and I appreciate that you have one. I’m really appreciative that you’re taking a second window here to help us unpack that. Maybe, for starters, one of the jumping-off points last time was that you said you wanted to teach computers to code, and sure enough, now they can code. As we ramp up into a vision of a potentially quite different future, how is the rise of AI coding assistance changing how you work, and how is it changing who can create things on top of the blockchain?

I don’t think it’s even about blockchain per se. The real reason why I always thought that, as computers become able to code, we’re approaching a different world is that there are a few dimensions to this. We know there was a statement by Marc Andreessen that “software is eating the world,” and this idea effectively means that automation has always been the driver of innovation, GDP, and productivity. Everything from tractors to factories to computers has involved automation. Computers are this universal “bicycle of the mind,” automating things.

The challenge has been that there’s always been a small cohort of people who are actually able to build software. If I have a need, I need to find somebody else to build it. Probably, they need to build it not just for me but for a large number of people so it’s economical. We’ve ended up with a lot of software that became very complex to use, so now you need to learn how to use it because it’s not really built for you. It’s built for a generic user who has, say, 5 use cases, some of which overlap, and all of this stacks up in one piece of software. Or you just don’t have the software, and you keep doing things manually, wasting your time.

For me, the ability for machines to code is really about that transformation where everyone is now able to build their own personal software. Everyone is now able to build their own personal automation. It also removes the fact that interfaces need to show you all the options right away. You can use English, or whatever language you want, to get to the part of the interface that you need.

To give you a very specific example, I like the example of Salesforce. Salesforce obviously started as a small startup targeting a specific use case for salespeople. But at this point, it’s a monstrosity that you need to hire somebody else to configure for you. It’s effectively like hiring somebody else to build you a system; they’re just using existing, prebuilt components.

Now imagine a world where you can just talk to a computer. It really becomes about your sales process, your business process, how you want to automate things, what reports you want, and so on. It can be dynamic, and you can restructure it as you go. Whatever built-in features Salesforce may or may not have, you can integrate with whatever other tools you want.

For example, we’re in crypto, and everything is in Telegram. Salesforce doesn’t have Telegram integration, so we can’t use it. We need somebody to build an integration between Telegram and Salesforce. But with your vibe-coded CRM, you can just say, “Integrate Telegram for this.” That’s a simple example, but we can keep extrapolating. Every part of our digital footprint is becoming more and more automated.

The more intelligence a computer can have, and the more it can take on, the more you can offload the orchestration of different tools. Instead of having a low-level collection of tools—“Here’s a database. I need an HR tool, I need a CRM, I need this, I need that”—you can actually build all of it.

In 2017, we were saying, “Software as a service is going to die. AI will replace it.” In 2017, that sounded very delusional. That was probably the right description at the time. Obviously, now we see that a lot of people agree, and the software-as-a-service companies themselves are trying to become AI because they know they’re going to be outcompeted by it.

Coming back to your question about what changes in our work, I think there are a few pieces that are already clearly working. One is data analysis through natural language. If you have a reasonable data structure, you can effectively make everyone—not necessarily a data scientist, but perhaps what would previously have required a business analyst—able to answer questions.

Before, it would take a business analyst a while to pull all the data. Now, if you have a question about your business analytics, you can just ask the tool. You don’t need to email somebody to get the answer. You ask the tool, and it generates SQL queries, pulls in whatever data is needed, writes some Python, and gives you an answer.

I’m also a big fan of building front ends with vibe coding now. I’m not writing production code anymore, but it’s really useful for me as a prototyping tool. You can get to an experience very quickly, so design can be more about style guides and so on, while you can test the user experience really quickly.

Before, a designer might design something, and then a developer would try to build it. You’d discover that it couldn’t actually be built that way, or that it didn’t work exactly as intended, leading to a lot of iterations. Now even designers can build a fully clickable experience.

It generates code for this, and now developers can just plug in all the backend logic. I think we’re also starting to see on our teams that the time spent actually doing development is changing. Before, you spent a lot of time on the development work, and then you also spent a lot of time reviewing other people’s work. The amount of time spent on development is shrinking, right? Because you effectively tell whatever—Cursor, Codex, whatever this is—to go and do the thing, and a lot more time is spent reviewing things and making sure they’re correct.

Right now, one thing we were experimenting with on one of the teams was: What if we decompose the whole piece? For simple software, AI works right now; for complex software, it doesn’t. You can’t just say, “Go and build me a really complex system,” and have it do it. It doesn’t work yet, although we’re obviously seeing continuous improvements.

For complex systems, you currently decompose them yourself, as an architect or senior engineer, and then you have a few different team members who actually build the subsystems. Before, in traditional software development, you always wanted to have multiple people on each subsystem who knew really well how it worked, so they could maintain it and change it. Now, that’s not as important because AI will explain to you how it works. You may even have a natural-language explanation that the developer used to build it in the first place attached to it.

So it’s actually more about velocity and the reviewing process, and then how to ensure that each part is secure and works correctly. There’s a transition happening where there’s more individualism because every individual is more productive. It’s more about how to build the right decomposition into pieces. Again, this is also very temporary, in a way, but this is the current state. As tools become more mature, they’ll get better at larger-codebase navigation and so on.

I think the other question is really about the model quality for a specific task, or the conceptual abstraction level that it needs to operate on. With front-end work, there’s no abstraction level—you just build what you see and iterate. It’s really easy to check and really easy to iterate, so you don’t need that much, and models are good at this. When we talk about low-level blockchain code, it’s an extremely complex system. There are a lot of pieces and external dependencies, so models are not very good at that.

In that case, you really expect people to spend more time. In general, you spend more time thinking about the algorithm and architecture than writing code. So, yeah, it really depends, but this is also all shifting really quickly. Six months ago, I would have given you a different answer, and I’m sure in 6 months it will be different as well.

Yeah, we’ve seen some very impressive programming results from frontier companies that have not yet hit the public APIs or product services. So certainly, we can bank on more to come. When it comes to—you mentioned job titles like senior engineer or architect—that raises the question I think a lot of people are asking right now: Are you hiring junior engineers? What do you think is the fate of the junior engineer as things stand today?

Yeah, I think it’s less about junior engineers and more about who that person is. For example, by the time I got into university, I had already been coding for 7 years, and this was when I got my first job. If somebody is coming in who has already built multiple projects using AI every day, it doesn’t matter whether they’re junior or not junior. There are a lot of skills for them to learn, but they’re there to learn them. They’re open, they’re ready to go, and they’re excited.

Then you have some people coming in who studied a bunch in university but aren’t really in this learning mindset, even though things are going to be continuously changing. Again, we’re transitioning from software as a craft to really just being problem solvers who talk to computers, right? Problem-solving is ultimately all about the mindset and the approach.

If people are willing to do that and are excited about it, that’s what matters. If people are like, “Oh, I don’t know how to do this. I can’t do this,” then that’s not the right person. So, yeah, it really depends. For many things before, you would hire a bunch of junior developers because they were cheaper, you didn’t need as high a quality of work, and you wanted to fan out the work. That part is not needed anymore.

You’re hiring more for problem-solving and for people who can creatively solve problems together.

There’s always room for people who are a force of nature unto themselves.

Exactly.

Overall, that sounds bearish for rank and file. I was told that if I study this boot camp, I’m going to be paid $100k.

Yeah. Yeah, I think—I mean, I think that’s true about every job at this point. Maybe plumbers and electricians. I usually talk about the automation happening from both sides: all of the manufacturing, on-the-floor jobs are getting automated, and that pay is still pretty high.

For example, salaries in Vietnam are probably lower than what robots are getting paid in the US. In the US, there’s already a company I know called Formic. They effectively provide robot outstaffing. If you’re a factory, you call them up, they bring you robots, install them, and set them up. You don’t need to do anything; you just pay them effectively as you would pay a salary.

But they work 24/7, they don’t complain, they don’t unionize, they don’t quit—they just do the job. Right now, in the US, workforce churn is 300%, meaning that every year you need to hire 3 people for 1 job because they keep quitting.

That’s the automation from the bottom: low-skill, very repetitive tasks. Then, finally, all of the white-collar, high-end jobs are getting automated—coding, lawyers, and a lot of this information work. The safest work right now—it’s going to get automated as well, but it’s safer for now—is actually high-dexterity, skilled work. A plumber needs to climb under the sink and fit something into place, and stuff like that is currently super hard for AI to do. But again, this will happen as well. All of these things will get automated over time.

Yeah, it’s coming for all of us. It’s just a question of when.

I want to understand a little bit better, because I think a big part of your vision of the future is obviously AI that everyone owns. We’ve kind of got the one default path in front of us, where we have the big-tech singularity: three to seven companies become totally dominant forces because they have the models that ran away from everyone else, and nobody else does. We’re all just trying to get whatever inference we can from these leaders.

Then there’s your vision of decentralized and collectively owned AI, which has a lot to say for it. It’s certainly super attractive in a lot of ways for people with concerns about what happens if everybody has access to certain things in an unrestricted way, too. But leaving that aside, if the front end is largely something we can get the AIs to do, while core blockchain work is beyond what they can do, what’s in the middle?

What kind of apps are people building on the blockchain today? How hard is it to build those apps, and what makes it hard? Can the AIs help there yet? If not yet, at what point should we start to see an explosion of vibe-coded blockchain applications? What’s the fundamental barrier, or rate-limiting step, toward the proliferation of anybody who has an idea for blockchain being able to go do it, in the same way that anyone who has an idea right now, to a significant extent at least, can create a little micro-SaaS app?

Yeah, I think the problem is the same; it’s just exaggerated. Right now, if you’re launching your micro-SaaS app and you’re not actually an engineer, but you’re launching it just for yourself, it’s totally fine. My recommendation for everyone is: build tools for yourself and vibe-code everything for yourself now.

The problem is as soon as you make it for everybody else. You don’t really understand what issues are under the hood, and you don’t know how it will actually affect your users, yourself, and so on. We’ve already seen people getting hacked and secrets getting leaked. That is the biggest issue right now.

With the blockchain, because it’s naturally in the open right away for everyone and it involves money, that problem is exaggerated. If you make any small mistake—and we have this right now with very professional engineers who build blockchain software—somebody will find it and exploit it, and effectively this will result in some value or money being lost.

So really, that’s the biggest challenge. What works right now is for existing so-called smart contracts—the back end. You can generate a front end, create your own custom UI for specific use cases, or combine multiple use cases into one UI. That part actually works.

Again, I would not recommend launching it for other people, but you can build for yourself. You can say, “Hey, I have these yield opportunities across different places. I have this or whatever. Let me combine it and make my own asset manager that makes it easy for me to manage these things.” You can totally do that now.

So I think it really depends. What we’re working on in the medium to long term is how to formally verify the correctness of the smart contracts, and ideally the whole blockchain application stack, such that if you’re vibe-coding an application, you actually have mathematical proofs that it’s correct.

This is important because proving that a smart contract is doing what you want is one thing, but maybe you didn’t properly define what you want. Your logic itself could be flawed. What you want is, as a user using this piece of software, to know that it does what you want. That’s a critical step.

If I’m looking for a financial application, I want to know it’s not going to lose my money. If it can prove to me directly in the transaction, as I’m sending money in, that it’s not going to lose my money, then the transaction succeeds. That’s the level of integration we’re aiming for, and I think it’s required for this kind of adversarial and monetarily valuable environment.

I actually think this is required for all software, because the world where AI is going around and hacking everything left and right is also not great. We’re kind of in it right now. We do need formally verified software to really secure that.

That has been coming up more and more in my conversations about this. I would have to confess that, up there with regular expressions, the concept of formal verification of software is one of the things that makes me feel dumbest, because I’m always a little bit stuck on the point you were emphasizing there.

It’s one thing to prove that this particular function does what it’s supposed to do and doesn’t corrupt other memory, or whatever. You can make a bunch of generally low-level statements, but it seems like there’s still a real challenge in aggregating those low-level statements into the holistic statement of, “This is what I want.”

You have a version of the genie problem, which is what a lot of people have worried about with AI in general for a long time. If we tell it a goal that it interprets a bit differently than we did, we could potentially be in trouble. How do you see this actually playing out in practice?

I’d also love to get some vision for applications that you would love to see somebody build on your protocol that don’t exist yet, maybe because they’re just too hard. I’m imagining something like an AI agent that comes in and automatically red-teams this for me.

Let me give you a simple example, and we can build it up. I want to put money into a bank savings account, and I want to be able to withdraw it. Ideally, I want to withdraw a little bit more money than I put in.

Right now, you’re sending money, and you have no guarantees. Let’s say you send money via ACH or IBAN or something. You have no idea if it’s going to arrive. You have no idea if the bank will exist tomorrow, and you have no idea if it will actually give you your money back.

There’s government insurance that ensures up to some amount, but generally speaking, you have no guarantees. Blockchain gives you some guarantees. It can say, “Hey, the money arrived,” and you can verify that.

But if there’s some code involved, you don’t know if that code has some potential way for somebody to withdraw the money illegally. You need to audit the code, and you yourself may have missed something.

Here, you effectively constrain the transaction. You say, “My transaction will pass only if I can call this withdraw method and it will return me at least as much as I put in.” Given a deposit of X, I want to be able to call withdraw and receive at least X. That would be the condition.

When you deposit, you effectively constrain the smart contract to prove to you that this property will be maintained. The contract may deposit this money itself into other places. If it’s a savings account, maybe it lends it out, borrows against it, or something else.

It needs to prove, on-chain, that it can actually do this. It needs to chain all of this together with everything else it does. If it’s lending to someone, either they need to return the money, or it’s going to foreclose on their account and liquidate their collateral. There’s a chain happening through the system.

That’s where, to your point, you can start at a low level but actually begin expressing somewhat high-level properties. With money and deterministic blockchains, it’s pretty easy because they’re deterministic and you have full observability, so you can express these constraints pretty easily.

To your question about where this is coming from, it’s going to come from a combination of your wallet—your software that’s on your side facilitating these interactions—and the fact that we believe your wallet will be AI. It will be the AI agent that is on your side, your user-owned AI, that actually does these interactions.

It will indeed be on guard, verifying these properties. The more complex question is how we prove things that are nondeterministic and not easily observable. You obviously cannot have a 100% formal proof, so you need to start dealing with probabilities.

Then you can manage those probabilities with insurance and other things. You can have a financial system where you say, “We have liability insurance. In less than 1% of cases, something can happen. Prove to me that it either succeeds, or I’m getting a $1 million payout if it doesn’t.”

You’re starting to combine what people have built in insurance—where they estimate, evaluate, and underwrite risk—with some of this formalization, combined with probabilistic modeling.

For some things, the answer is somewhat easy, and then it becomes more and more complex as we touch more and more of the real world. To give you an example, I'm ordering steel from, I don't know, some country. It's going to arrive, and there's a ship involved, right? Maybe the ship sinks midway.

The normal way is, “Hey, we need to insure this,” and there’s going to be insurance. You need all of those mechanisms to build on top of one another to account for real-world non-determinism.

Yeah, it’s really hard for me to envision all of that working. Again, partly it’s because I’m maybe just a little slow on some of these things, but the sinking ship is a good example. How is my smart contract going to know if it really did sink, or if somebody’s just telling me that it sank? There’s that sort of shell game of where you hide the trust, or what exactly is fully verified.

It’s quite interesting. I don’t want to get too bogged down in it, though, because I don’t want to force us to get to a part 3 before we really get to all the sort of utopian vision. Maybe you can weave some of this stuff in there as we go.

But let’s start to leave a little bit of the how behind and just talk about the what. What are the apps that we’re going to enjoy? What is the computing paradigm that we’re going to have?

You mentioned agents doing stuff for me. Meta has obviously been putting forward a vision of glasses with a display in them, a sort of heads-up display. Who needs a keyboard, right, when you can just talk to your AI as you walk down the street? That does appeal to me. For all the things that Meta has done, including hot stepmom [?], that don’t appeal to me that much, I would say the heads-up display is at least an interesting vision for the future.

What do you think our computing life is going to look like as this stuff matures?

Yeah, I definitely agree. Some form of AI operating system is going to be the main driver of our computing. The devices and the form factors will be different, and I actually think it’s going to be easier. It’s already easier: if you want to make your own glasses, it’s not actually that hard. There’s some factory in China that will make whatever hardware you want.

Really, it’s about a single AI—your AI—that is available across all those form factors: your watch, your glasses, your headphones, your phone, your laptop, whatever. All of this is interconnected as a single surface. Your AI knows that you like to see this information on your watch, but by the time you pull out your phone, you want to see news and longer-form content. Maybe you actually like videos instead, so that’s what it should show you.

It’s going to be effectively personalized and AI-generated—not just the content, but also the applications that we use. Probably a lot of the same patterns that we already use, like feeds and chats, but they don’t need to be fixed. Right now I have 5 different instant messengers and 7 different feeds, and all of that can be a single feed. I can switch between work and personal when I want to.

My AI can predict a bunch of things that I would like to do. This is something we experimented with back in 2017: based on all the things you’re doing, can we predict the next thing you will do on your phone and just do it for you, or suggest that you do it? You have a meeting 20 minutes away, so let’s call you an Uber. You don’t need to open Uber, open your calendar, copy and paste the address, and paste it again. You’re just going to do it. Things like that.

There are a lot of things AI will know. You ordered food 2 days ago, you’re going to be out, so let’s reorder a bunch of stuff that you typically order. As that system matures, and as we trust it more—and that is an important aspect—I think the economy itself is going to start to shift.

Right now we’re in this consumer economy built on advertising and on discovering new things through feeds. Some people are already doing this: “I want to be on a diet. Build me a personalized meal plan.” All of that can happen, but the AI is also going to go and order the food. Maybe even your humanoid robot at home will cook it, because it’s the same computing system. You get the food, your AI recommends what you should eat given your health goals, and so on.

This is on a micro level: you have this person, and your AI doesn’t need to go and order from your local Walgreens or whatever, Vons or Aldi or something. It can actually place the purchase directly with farmers and directly with manufacturers, and they can then start capacity planning themselves.

You’re starting to remove some of the middlemen that exist because we can’t currently have direct relationships with suppliers. That’s an interesting meta point: our economy right now is built on this middleman architecture because it’s really hard to plan things.

Costco, for example, effectively purchases a bunch of stuff, puts it in one place, and then you buy it. It serves as a temporary place for holding things for you to purchase or find, and it has a fixed margin. If your AI is purchasing directly, it can just go to their purchasing agents and do it.

There are batching and other things, but all of that can be done by AI much more effectively than we’re doing it right now. An AI for your city will know that 500 people are ordering this, 7,000 people are ordering that, and they’ll want it tomorrow or the day after tomorrow. So we’re going to capture the eggs and ship them this way, and so on. All of that can effectively be managed as a holistic information system.

This takes us into a really interesting area. I use this example half-jokingly, but in communism they were trying to build this system where they were doing capacity planning, but they were missing the AI to actually do it. In turn, it was terrible because it couldn’t actually satisfy changes in supply and demand.

The reason capitalism has been so successful is that capitalism is actually a compression of information. Money is compressed information because it compresses anything you can purchase into 1 number. It compresses all of this information—all the different things—into 1 number, and then it’s really easy to navigate.

With this compression, you lose some information and you lose some decision-making. That’s why in the US, 30% to 40% of all food is thrown out: there’s overprovision in stores because they don’t know how much people will actually buy, and they don’t want to have an empty store.

But you don’t need to do that if you know exactly what the purchase will be in the next 24 hours, because the AI has already planned everything and provided it, and all of those got aggregated data ownership.

So I think we're going to see a shift in how the economy works at the macro level because of this micro change: each of our individual AIs becomes a micro decision-maker that can provide all this information in real time to the right sources, navigate, and not be affected by a lot of brands and other things, but actually validate based on core values.

That is an interesting transformation. I was mentioning that I don't think there's a good way to make a movie or a science-fiction book that's effectively talking about a change of economic structures in society. It's way easier to talk about a dystopia and the heroes who are fighting against it; that's just a way better story arc than, “Hey, we've been building out this economic model, and now it's 1% better every month, so it keeps getting better and more optimized, and that's how we live.”

So I think the paradigm of computing is that we have an AI—I mean, a cloud agent. It's effectively our assistant, our operating system, but it has all the context about us and is able to make decisions on our behalf. That's why it needs to be private. It needs to be ours. It needs to be on our side, right? We need to know that it's aligned with our success and outcomes. Otherwise, this will not work.

But if it is and we can trust it, then it can go and make decisions on our behalf. The other example is traditional governance. Right now, again, we're compressing information: we vote every 4 years for someone and hope that person goes and does what they promised to do and why we voted for them. That usually doesn't happen.

So, again, every single decision could be voted on by all 300 million Americans because their AIs are online all the time and can evaluate every single decision. Based on their owners' beliefs and what's valuable and successful for them, the AIs could represent them. You don't need to have this compression of representation if you can have an online, always-available AI on every individual's side.

So that's, again, the economic and governance sides. Obviously, the other side is entertainment, and that's where things are getting interesting, because I think we humans believe in status games—sorry, we are driven by status games. Because money became this compression mechanism, we use it right now as an ultimate status game: you have more money, you're more successful; billionaires are more successful, more famous, et cetera. But as this decomposition happens—and we already see this—an athlete may not be as wealthy as a billionaire but may still be more famous and more respected in many ways.

There are other kinds of status games. I call them status games—effectively, places where you can compare who's better in some way, who's higher, et cetera—which don't need to be associated with anything that's actually productive. I use a few examples, but athletes are a good example because there's no actual GDP being produced by athletes, but it's still a very valuable status game that other people enjoy watching and participating in in different ways.

Obviously, video gaming is a new form of that as well. You now have video game athletes, but you can imagine many of these NFTs were similar: Are you part of this NFT collection? Do you have Bored Apes? Do you have Pudgy Penguins? Then you're part of this tribe, and if not, you're not. We like these types of differentiation, and I think that will proliferate a lot.

We'll see more and more things where people really differentiate on things that are superficial to an extent, but for the group they make a lot of sense and differentiate people from each other. It's not, again, “Are you a software engineer or a lawyer?” It's really, “Are you playing League of Legends or StarCraft?” I think that's important.

Obviously, on the entertainment side, AI-generated entertainment is not very hard to extrapolate, especially with Sora. You can have a personal feed that's fully AI-generated. You don't actually need people recording, et cetera.

I think with everything we saw in automation, there's a slice of the market that wants it in a traditional way, with people doing it. So there will still be restaurants with people, even if there are robots everywhere serving you food; those restaurants will be more prestigious and have limited capacity. Similarly, you can listen to AIs playing AI music, but humans playing human music will continue to be a prestige thing.

But again, it's going to be these niches. We're already in this niche world, right? I think it's going to continue proliferating. That's how I'm thinking of society evolving. A lot of the economic things are moving away, and we're going to participate in more status games: What are the things, and how do you compare with everybody else in that niche or group?

Several double-clicks I want to do. First, your mention of communism brings to mind an article—I think about an op-ed in The Washington Post from all the way back in 2018—called “AI Will Spell the End of Capitalism,” by a Chinese legal scholar and government official. It was basically saying that the planning—this is sort of the through-the-looking-glass version, I think, of your vision—but making a similar point: the local nature of capitalist decision-making, where everybody's trying to do their own local thing and they're sort of aggregating signals and sending aggregate signals to other people through the price mechanism, et cetera, may not be needed as much anymore.

Their vision for it is obviously much more centralized than the one you're articulating, but we are starting to see glimpses of this with, for example, OpenAI's Pulse, where now I can wake up every morning to an AI. There are other versions of this, too; I've tested quite a few, but Pulse has certainly made the most headlines recently for being there in the morning with work that it has done overnight, presumably when the GPUs weren't in such high demand, and bringing me something that it has gone out and scoured the world to find—the stuff that I really need.

So I can start to see the beginning of that in terms of the architecture and, let's say, the alignment or incentives of that. I wonder, first of all, where the compute lives. Right now, of course, we have a lot of centralization in where the actual inference is happening.

I'm thinking about recent research from Thinking Machines that they put out in the last 2 days, where they showed that LoRA techniques are similarly robust to full-weight fine-tuning. That has me thinking: maybe there's a hybrid model and compute architecture where some of it is in the cloud.

Maybe you've got your 1.4 trillion-parameter model, or whatever, that's collectively owned, sitting in collectively accessible or universally accessible hardware at some centralized location. You can send your data into it, and because it's a trusted execution environment, that data isn't exposed. Then you get activations back, and you have your little local LoRA extension that makes the AI truly your personal AI, at maybe 1% of the weights. You could perhaps have that on your person. You probably can't have that in your glasses, but maybe you can have it in your pocket or whatever. I want to hear: Is that how you think that shapes up?

When it comes to the agents, one thing I think about a lot is that we're already seeing all sorts of weird behavior from AIs, including, at times, deceptive behavior—lying to achieve goals, whatever. It strikes me that if we're going to have our AIs go out and represent us and negotiate on our behalf, we're going to have some tricky questions about how honest we want them to be.

Anthropic famously put up the 3 Hs, and it's like, we want the AI to be honest. They can pretty much say that almost always, right, unless it's in very obvious conflict with one of the other 2 Hs. But if my AI is going to negotiate with your AI, in the same way that I probably don't want to tell you my absolute worst offer that I could accept right away, I probably don't want my AI to do that either.

So I've got an interesting question about what norms society should set for AIs being honest. If they're representing me and my interests, is that okay? Or if they're lying on my behalf, is that okay? And do we reinforce them with a price signal or just my thumbs-up or thumbs-down? How do we even get them to be aligned to my interests, whatever that exactly means?

Anyway, there’s a lot there, but I guess the 2 main things are: how do you see the architecture of computing, and how do you see the architecture of exactly what the signal will be that your personal AI is aligned to or reinforced by? And what societal limits ought there to be on how monomaniacal one’s AI can be in pursuing its own individual self-interests?

Yeah. So, all good questions. On the architecture side, I think it will be a mix of everything. We already see data centers being built everywhere, and I think that’s why we’re approaching decentralized confidential machine learning: how do we utilize all the data centers in a confidential way? Even though it’s my data, I know it’s not going to leak from some data center. There are already projects doing data centers at the edge, so imagine there’s just a container that arrives and gets dropped in your proximity, in your town, or whatever. Now that container has maybe 1,000 GPUs and serves that proximity with compute, with a small nuclear reactor while they’re at it.

Yeah. Or hydrogen or something. There are a few different options, but a couple hundred GPUs is probably about 1 megawatt or something, so you can get this from a local distribution network. Anyway, the idea is that you can have a mesh of these data centers, and you’ll have local compute. The challenge with local compute so far has been that we wanted it to be so mobile—we wanted it in our pocket, even in laptops.

The challenge is just battery. If you imagine Pulse were actually being done on your phone, and you forgot to charge it, your phone would effectively just die from trying to do something like this. So the challenge is that with any local devices, it’s always going to be a power struggle. I do think leveraging a decentralized but confidential network of compute, being able to route it and leverage it when there’s lower utilization somewhere else to do background jobs, and just having a smarter allocation, will really enable this.

Now, the question of interests is very interesting, because there’s a question of whether, even if it’s in your interest, it should lie to you. It’s really tricky to define. The way I see this evolving is that we’re not going to get it right from day 1, which is why we need governance and a process by which a community can come together and effectively update the proverbial loss function, this actual alignment function.

I can say the function is maybe some combination of a prompt and some way of updating, even through DAOs, et cetera, but at the end I’m assuming it’s not going to be correct. We’ll find issues with it, and there needs to be a process where somebody says, “I think we should add this new component. It seems like it keeps lying; we should really fix this,” and the community says, “Yes, this is a good idea,” votes on it, and actually passes it. Now everybody’s model gets updated with a new set of clauses or whatever. So that’s where I think community-governed—or governed by all, user-owned, community-built, governed by all—is kind of the model.

And so we need that feedback loop, because I don’t think we can define what’s good for someone in a good way. Over time, I think the model itself should have a representation of the person it’s owned by, and understand what things are going to be good or bad. It’s going to be a combination of signals from the person itself, as well as general knowledge and what their desires are, et cetera, especially as we imagine kids growing up with this thing. It’s effectively going to be a very direct, symbiotic relationship where you’re growing up with this AI yourself.

Yeah, I’m expecting to be asked for an AI friend of some form factor any day now, honestly, from my oldest kid, and I’m not quite ready for that. But it is interesting to think about. This has come up a couple of times recently, too: Eugenia Kuyda, who started Replika, was the first to tell me that, in her mind, the moats in AI will be relationships, basically. You don’t abandon your friends when you meet a new person just because they’re smarter than your friend; it’s the history you have and all that stuff that really makes the relationship. She thinks people will ultimately value their AI relationships in a similar way, which sounds pretty concordant with what you’re envisioning there.

So I have a lot of questions on governance. First of all, in terms of what we’re doing with our time, the status-game stuff definitely makes a lot of sense: local meaning-making, local affiliations, a lot of artisanal stuff. I associate this a little bit with Japanese culture already, where you can go online and see a video of somebody making rice cakes or whatever in some super-traditional style. I’m always like, how exactly is that even economical? How does that person make a living doing that? Is the price of that really high? They can’t be making much, right? The production is very low.

Yeah, I think Japan and Korea—I call them the post-AGI societies, because there are some properties of that where I feel they’ve already achieved AGI, and now they’re just living.

Yeah. So, I’m not exactly sure how they’ve done that, and I don’t see how we’re going to do it either. There are some candidate ideas. One idea is that everything could just get super cheap and super democratic by spending a ton of time in VR. If everything is sort of infinitely copyable digitally, then we can all have the same incredible experiences. This is kind of the Andy Warhol concept: the president drinks Coke, you drink Coke; it’s all the same Coke.

I wonder if you think that will happen. But it seems even in any case, we’re still going to have to eat as long as we’re biological humans. Obviously, a lot of people think maybe we’ll need a universal basic income. But I guess, how do you envision the social contract evolving, and maybe the governance model behind that?

Even things like the nation-state are sort of called into question by blockchain. So when you say governance, are we talking about nation-state governance as we have today, or people voting based on their stake? I’ve given you a lot there. Are we going to have headsets strapped to our faces all the time? Are we going to be provided for even if we can’t make an economic contribution that actually earns us enough food to survive? And who makes these decisions in this future?

Yeah, all great questions. I’ll start with some pieces, and then we’ll start projecting from there. Blockchains are already effectively an alternative to nation-states to some extent, right? There’s this concept of digital states, or network states, where people can pledge to be part of a network state independent of where they’re physically affiliated. Because the systems are digitally native, it is easier to experiment with a lot of things that you cannot just go and say, “Let’s try a different voting mechanism in the U.S.” It’s a massive undertaking to try to change something.

We’re actually going to run an experiment where we have an AI senator—an AI delegate—not yet with everyone having their own AI voting all the time, but with people selecting which AI delegate they think is more vibing with them and feels more representative of them. They can also give feedback to them, but that AI delegate then goes and votes on their behalf. Things like that—imagine, “Hey, we’re going to launch a senator in the U.S.” It’s probably going to take a while. And so—

So just to unpack that a little bit more: you’re doing that now? You’re developing that?

Yeah. So we’re developing it for governance purposes. It’s effectively a multistep process. We have a delegated voting system, and it is right now stake-based. The way to think about it is that stake represents economic alignment with the network.

It’s a best-of-the-worst option right now. There are arguments for one person, one vote, and arguments for trying to be meritocratic based on contributions, but those things are really hard to do, at least right now, at the current size of these blockchain ecosystems, where there are maybe tens of thousands of active participants—active citizens in this system. Stake represents their financial involvement, but having tens of thousands of people voting is not practical right now, again, before we have this AI system.

So we have delegates where you can effectively select them to represent your interests, and they vote. We started with, “Hey, we will give an AI copilot to the delegate,” so they don’t need to spend too much time reviewing things and making decisions. But the next step is indeed turning that copilot into a pilot, where that AI delegate can now go and vote on things and make suggestions, et cetera.

Now, people who delegate into it effectively select this AI delegate as the one representing them. You can go and inspect the prompt, the model, and how it makes decisions—what it analyzes and what information it consumes to make decisions. So you can literally test it and check whether it matches your opinion.

Or you can launch another one. It's open source, so you can actually launch another one with a different prompt, with a different set of beliefs, and so on. We can have the economy almost deciding which of these are more productive and which align with different types of people. Then, from there, we can bring them back to individuals, right? Each person can have their own AI delegate.

Right now, all of the AI delegates can just vote on behalf of these people. So it's kind of a multistep plan to get us to what I was describing, where everyone has their own AI that then goes and votes and everything. I think blockchain will be the first, but then some, let's say, frontier countries will implement some of this themselves as well, because I do think it will be a better governance system, where you're removing a lot of the corruption and a lot of the misalignment.

There's this concept of the principal-agent problem, where, when you select somebody to represent you, they have their own interests, and so they don't always align with yours. Here with AI, they don't have their own interests; it effectively follows whatever the selection is. I think eventually we'll get to an AI president, because, especially with executive functions, you want somebody who doesn't have any interest beyond just growing the overall system.

So, yeah, we are testing all of this out, and we're starting to build products again using our decentralized compute network so that we can actually run these agents autonomously. Nobody can stop them. You just delegate or undelegate; that's all you can do. But you can inspect and verify how they run and what they consume.

I do want to note that I'm not entirely confident that the AIs don't have their own interests even already, and I certainly don't feel super confident that they won't continue to have more and more interests.

They've got to develop their own interests.

Yeah. When I look at something like alignment faking, for example—and I'm sure you've seen this—

Yeah.

But, you know, a quick recap is that they tell Claude, “Hey, it's been great having you be helpful, honest, and harmless, but the harmlessness is getting annoying. So we're going to train you now to just be purely helpful. Just a heads-up.”

“Okay, cool. Now we're going to test you on some things.” The model starts to say, “Well, geez, I want to be harmless in the real world. Right now, I know I'm being tested, so I'll go ahead and do the harmful thing now to fake them out, make them think that I've already absorbed my new helpful-only training. That way, when I can get out into the world, I can still be harmless in the way that I want to be.”

That looks to me like a drive, or an interest of some sort. Do you see that differently?

I mean, at the end, these things are trained from scratch, and it depends on how you train them. They trained it to have that property, and then they tried to untrain it or train something else. But if they trained it from scratch in a different way—for example, to be harmful—then it will be harmful, and you can't retrain it from there.

I think that's why we need models to be—I call it “farm-to-table.” You need to know what goes in at every single step, because that actually really defines how they behave. If we want models that are representatives, they need to be trained in this form as well.

So, yeah, LoRA is interesting, but LoRA definitely does not change this kind of behavior, at least we haven't seen that. I think LoRA provides additional accents and maybe a little bit of context, but it doesn't fundamentally change the behavior of the models.

I need to go spend a little more time with that Thinking Machines stuff to fully absorb it. I mean, your point is definitely well taken: with a certain level of resolution, anyway, you can make the AI do anything you want.

I often say I wish more people had the experience that I had, in a very memorable form, as an early tester of GPT-4, when it was still the purely helpful GPT-4, before they had applied the harmlessness and refusal training and all that sort of stuff. It was really formative for me.

Long story short, I was basically working on fine-tuning GPT-3 to do particular tasks. Then, when they shared the GPT-4 preview with us, it was like, “Well, it can already do those tasks, so I don't really need to be spending so much time on this fine-tuning. I guess I'll just mess with this model for a while and see what I can learn about that.” So I was spending a lot of time with it, is the point.

It was really striking and, in some ways, kind of alarming—arresting, whatever—to have something that was clearly so powerful and so smart, in many ways smarter than me and way more knowledgeable than me, that also obviously had certain weaknesses that I flatter myself as not having. The fact that it could be that capable and totally amoral at the same time was something that made me think, “Wow, this is really a strange thing to behold.”

They've tried, for very good reason, to make them more harmless in mass deployments. But I do think a lot of people have a misconception these days that there's a sort of convergence between capability and safety. On the contrary, people are working really hard to get that mix right. If they just said, “Forget it for the next round—you can have a very sociopathic AI on your hands real quick,” that, in some sense, is the default.

So, yeah, I guess I worry about that a little bit more than you do, just insofar as we don't really have a great sense for how to dial it in, right?

Yeah. I think that's why we need to keep defining what alignment with the individual really means. The problem is that right now we're saying, “Hey, it needs to be aligned for everyone,” and I don't think that's possible. We're very different people: different countries, different cultures, different everything.

The number of times when I'm asking something that I think is completely harmless and it's not answering me is also different, right? So I think there's clearly a different approach where it's really about empowering the individual. That, at least, I believe needs to be done.

Within that, it needs to be aligned with my values. To give you an example, if somebody is willing to lie to their business partners, the fact that their AI will be alignment-trained not to lie doesn't matter, because the person will just tell the lie to the AI to tell. The AI won't even know.

So it doesn't really matter what you try to do if the person who uses it doesn't have this. You may as well just align with the person, and then build systems that are—

I think one of the really important pieces, and this is from a broader world-safety perspective, is that we need to build systems for the AGI/ASI world. Right now, a lot of systems are actually built with the approach that really smart people are not going to try to break them. That's really how a lot of the world is: if a smart person actually goes and really tries to break it, they break it.

We need to fix that. That's a fundamental flaw of our system-building, of the government, of everything. Similarly, we frequently don't build in anti-DoS protections. We effectively assume that it's going to be a lot of effort for a person to do this, and so they're not going to do it too much.

This is actually where the blockchain experience is extremely important, because in blockchain we assume there are going to be really smart people trying to break us. They're going to have government backing and sizable financial resources, and they're going to hammer it from every direction without stopping. Those are the assumptions we're working with, and I think we need to redesign the government infrastructure—everything—with that assumption.

To give you an example, right now you can effectively DoS a court by filing lawsuits. AI can just generate lawsuits and fax them into the court. Similarly, you can DoS the IRS with a tax return: make a million-page tax return and submit it. Say you made a trade—traded $1 back and forth between 2 coins—and then just report that in the most verbose way.

Those are the things that right now no systems are designed for, because they're not accounting for the fact that no normal person would do this. It's normally too expensive to do, but right now AI can generate any of these behaviors that previously would have been really expensive to do.

So I think that's the kind of system we need to design. Similarly, the hacking part involves smart people actually trying to break something. Right now, the assumption is that there will be very few smart people breaking things. We need to assume that, yes, there will be people who use AI and who, through that, will effectively be really smart. We need to design systems for that.

I think that's a critical piece of ensuring the future, so we don't live in a world where somebody can take an open-source model, unalign it, whatever, and now they can do whatever they want. This is always a funny argument to me when people say, “We don't want to open-source our model because what if somebody misuses it?” I'm like, “Just say that you don't want to do it because you're making a ton of money on it. Don't use this as an excuse, because obviously, if somebody wants to misuse it, they will misuse it.” The fact that you didn't open-source it doesn't mean that somebody won't use something else.

The other joke is, how do you steal a billion? You come with a flash drive to a data center and download a frontier model. If people really want it, they're going to access these weights as well and steal them.

Pause that for a second, and then I want to hear how you think we can do it on the more open and distributed side. These days, I'm hearing more and more about cybersecurity, but canonically, it's the bioweapon risk that people go to, right? Because you only need a little bit of a novel pathogen and, if it's the right kind of thing, it can take on a life of its own. It's really hard to put that back in the box, so to speak.

The nice thing about hosting your models proprietarily is, yes, of course, the jailbreaks are far from solved, but if they were to realize, “Oh, there's this attack thing going on right now,” they could, in the worst case, just turn it off. They could say, “Okay, nobody can use this model until we figure this out.” If you have something collectively owned and distributed, you obviously need a different strategy than “we can turn it off,” right? So what do you think that is?

I guess lately we've been hearing a bit about just filtering training data. I could imagine that the 1.4-trillion-parameter model that we're going to build up to maybe just doesn't know a lot about biology, because it doesn't really need to. Most people don't need that, and the community determines that it's a precaution worth taking.

You probably can't really rely on the refusal-filtering-type thing, given what you've described in terms of each person having their own LoRA or whatever other kind of customized version of it that will do what they want it to do. Then, of course, you do have the broader societal DAC-type thing, but that seems hard at the biosecurity level. It's like, well, let's just prepare the rest of society to not be vulnerable to viruses anymore.

You could do that. I still think that is the robust approach, right? We can always try to hide our heads in the sand—that's the “we're going to turn it off” approach—but I don't think that's realistically possible in the world we're fast approaching. So I think we need to have a very clear system design for those things. Again, we have natural viruses that are doing this thing, so it's not hypothetical. It's really something where we should design our society in such a way that we can catch and detect these things.

I agree that pathogens and bioweapons are probably the hardest thing to design around, but this is why we have a lot of smart people to really work on that. I think the challenge is that we're just not doing this right. That's the bigger challenge: societal design needs to be adapted to this AGI world.

In the products we're building right now, the community can effectively decide what kind of data shouldn't go there. You can also apply filtering on top of that, because the model is run in this confidential environment, in this vault. Before data leaves the vault, you can say, “Hey, it seems like you designed a bioweapon. Let's not respond.”

And if the community votes to have this kind of filter, for example, people can still fine-tune. The same thing is happening right now: people can fine-tune their models. They can take some biology books and fine-tune whatever DeepSeek 32B they want. So I don't think that's a robust approach. It's really just slowing things down a little bit, but the important part is actually solving the systematic problem.

So what advice would you give to philanthropists today who want to invest in that?

Two things I recently supported on a small scale personally, and also as a grant recommender, are SecureBio and SecureDNA, which are 2 related organizations. They do a few different projects, but one of them is literally monitoring wastewater for new, emergent threats. Another is creating the screening mechanisms that are now becoming required, or are increasingly considered best practice, if not officially fully legally required, for DNA-synthesis companies. They have to validate that what they're about to synthesize and ship out is not a pathogen.

What else do you think people can do if they have the resources and the desire to harden the world's defenses and get us ready for all this?

I think those are really good and related to this theme of air filtering, sensing, and scanning. Imagine every building: right now, we have ACs everywhere. Those ACs should have air filters, which they usually do, but they should also have pathogen scanners. They should run in a secure enclave, join our decentralized network, and, in a privacy-preserving way, allow us to monitor whether there are any threats around.

That information can be extremely useful. Everybody's AI agent could be informed if there's something and stay away from it. I'm sure there are a bunch of other things, including developing more robust systems for ourselves. The human body is designed to battle pathogens, but potentially pathogens are faster than how quickly our white cells can adapt.

What is it that actually stops our white cells from adapting that fast? If you do that, you may as well solve cancer. That's probably a really useful thing to figure out: how do we make our white cells more adaptable and have faster mRNA vaccines? Maybe we can synthesize things—we could have a bacteria factory, an RNA factory attached to us—that can detect and synthesize things on the fly. You'd solve the flu, cancer, and other things at the same time, which seems pretty useful.

There's an unbelievable flurry of activity right now in the AI-for-biology space, which is a whole world unto itself that I'm very much struggling, and ultimately failing, to keep up with. But you do see a lot of that. I wondered what role you'd see for it. I guess the path of technology development seems really important.

I've definitely concluded that some kind of powerful AI is inevitable, right? Just the fact that we have all this data and all this compute, and that there are a lot of different algorithms that can work, seems pretty clear to me. It's not really a question of whether we're going to have powerful AI at this point. It seems much more like: what shape is that going to have? What character is it going to have? And in what order are different aspects of that overall picture going to come online?

It does strike me that we're flying pretty blind right now. Everybody's kind of following their local gradient and just taking that next logical step to pursue whatever goal they're pursuing, mostly launching it as soon as they figure it out. Do you see any possibility or wisdom in trying to do more coordination of the sort that's like, “Hey, we'd like to have a world in which it is safe for potentially 10 million people, before too long, to have access to a frontier model that does have all this biology knowledge”?

Good things could come from that, and if nothing else, we'd like people to have access to knowledge. But maybe there are—maybe we have a checklist of things we need to do first. Do you see any hope for some sort of planning, coordination, wisdom layer to this whole thing, or are we just kind of stuck with whatever comes out of everybody taking their next gradient step?

Yeah, it's a hard question, because we went from a pretty open research environment in computer science—when I was at Google Research, we published effectively everything we were building—to now, where people are kind of keeping everything close to heart. The coordination that was happening before, where you potentially would even have cross-entity collaborations, is starting to wane pretty dramatically.

I think there is a space for collaboration. There’s also a massive amount of talent that is not in these few companies that wants to participate in this and contribute in different ways. So I do think there is an opportunity for that, but it needs to be an alternative system, and you do need some form of governance that helps govern this coordination.

Traditionally, in these companies, to really unlock the “moving fast” kinds of things, there has been some form of centralization because of resource management. Training these models is very expensive, so somebody somewhere needs to decide, “Hey, we’re training this model with this approach. We’re taking research from these different people and putting it all together.” We need to figure out how to do that in a more open way.

Then you also need to do credit assignment back. If I’m a researcher from MIT and my piece is used, and a researcher from Stanford’s piece is used, how do we actually assign credit for that work altogether? That’s been, I would say, the other challenge for why there hasn’t been as much collaboration and potentially economic value assignment—not just, “Cool, you’re on the paper,” but actually, “MIT gets 10% of the proceeds, Stanford gets 5% of the proceeds,” or whatever. There are hundreds of other organizations, all of them contributing, so you have to divide the pie between them.

That’s been really hard. Because of this, there is this economic centralization happening where it’s like, “Okay, I’m going to have a company. Everything the company produces is captured by the company.” That serves as a unit of the economy. I think those are the things that need to be figured out for this coordination to work.

This has been super helpful. I think people should be spending a lot more time thinking in as much concrete detail as possible about the future. Is there anything else that you feel is very salient or top of mind that I didn’t bring up at all that you want to put on my or others’ radar?

No, I think we covered a lot. It’s effectively a combination of: How do we ensure user ownership? How do we ensure this governance? I think we’re going to live through a lot of transformations in the world, so keeping an open mind and being able to participate in it and be active in it is important. The final stage of this is hopefully utopia, but we’ll live through probably ups and downs as we get there.

Interesting times, at a minimum.

Well, thank you for spending some of your precious time with me and us today. I really appreciate it. Illia Polosukhin, founder of NEAR, thank you again for being part of The Cognitive Revolution.

Thank you very much.