# Jeffrey Quesnelle on Nous Research, large language models, and the human mind

Sina Habibian Podcast · 2025-10-28 · 126 min · https://www.youtube.com/watch?v=X4tZykXwl8c

## Transcript

Sina Habibian

Hey everyone, this is Sina with another episode of Into the Bytecode. My guest today is Jeffrey Quesnelle, one of the co-founders of Nous Research. This was a pretty wide-ranging conversation. Nous is a research lab doing work to move open-source and decentralized AI forward. We talked about AI more generally: the technical progression of the field over the last decade plus and its big inflection points; modern model architectures and how reinforcement learning might have parallels to how the human mind works; and how each of us relates philosophically to this moment in time and to the idea of creating intelligence for the first time. This was a lot of fun, and I hope you enjoy.

### working with new technologies

Jeffrey Quesnelle

I’m stuck in my ways. I see that 20 years from now I’ll be the gray-haired person who’s like the Unix people I grew up with, who used Vi all the time and refused to open a GUI. They’d say, “You guys just don’t understand.” They wouldn’t write in C++; C++ was this thing where they’d say, “You can’t write it in C, in an 80-column width. It’s not worth writing.” There were people like that, and then there were young kids like me. I was like, “Oh, I’ll use objects and pointers. It’s kind of ugly, and maybe it’s not the most elegant, but it works.”

### deeper mysteries

Likely, that’s the situation we’re going through now. I question this myself. I’m only a very light AI user, to be honest. I spend all my time working on AI, but I use it very little, especially for coding. Having been coding for 20 years, the problems I can’t solve are usually extremely nuanced. That’s often right at the edge of where LLMs are today: very nuanced things where hallucinations can be a problem. Or the problem isn’t in the code itself; it’s in how you’re approaching it from a meta perspective, like organizing the instances of your different servers and your infrastructure. AIs today can’t really break out of their box and experiment.

Having said that, I want to caution everyone: every goalpost, every “it can’t,” has fallen mercilessly in the last 5 years. It’s very likely that within the next 2 to 3 years, all those things won’t be true. There will be AI models that have full agentification, which means they’ll be able to run experiments, do things, touch the real world in some way. Their reasoning ability over code and everything else will be superhuman. How I’ll interact with the world in that case remains to be seen.

One thing I would note—and this is already going to sound like a gray hair—is that there’s something to be said for being forced to figure something out yourself, so far as you value your own understanding as much as your economic output. You can view yourself in terms of what you do for the world: your economic output. Could an AI replace that? Probably, in a lot of cases. But understanding something yourself is different. AI can’t understand something for you, because it isn’t you. There was often economic value in understanding something because that was the only way to get it done. That may no longer be the case.

I saw a meme recently with a scene from The Lord of the Rings where it says, “The time of the orc has come.” The meme said, “The time of the idea guy has come,” because anyone with an idea can now implement it without having to go through all the technical work. Maybe that’s what humans are best at anyway. We’ll see. For me, I still want to work through things, and I find value in understanding them.

Sina Habibian

100%. Even assuming AI continues to improve significantly from where it is today, especially in these more verifiable domains where we can send off agents to implement entire features or do scientific research, I think the human in the loop will be there to have the taste and judgment to decide what problems are worth solving and what ought to be done. Then, at the highest level possible, we’ll evaluate the work output these AIs generate. To do those things, you need a strong fundamental mental model of the world.

That orients me toward focusing more on fundamentals and the scaffold of my understanding, knowing that I can fill in the details with AI. But you really do need to think from first principles about those questions.

### math behind distributed training

Jeffrey Quesnelle

It will become very easy to use these as crutches and never develop that understanding, because you can always just get the answer. It’s like having a textbook where every answer is right in the back of the book. There’s always going to be a temptation to look in the back when you don’t have the answer to the problem.

Sina Habibian

To continue the story before we get into some neural network fundamentals, how did you end up starting Nous, and how did you meet your co-founders? You guys seem like an interesting cast of characters.

Jeffrey Quesnelle

It really was one of those right-time, right-place things. I like to tell everyone we were just homies on Discord who got big. The way it started was that I was working on this literary AI project. The best model they had was T5, which only had a 512-token context length. Then Llama came out, and it was 2K. I was like, “Whoa.” Then ChatGPT came out, and it was 4K. I thought, “How are we ever going to get there?” At the time, it seemed like an insurmountable barrier.

I’ll pause to say why I chose this Choose Your Own Adventure project. I’m a huge fan of books, especially science fiction and fantasy. I grew up reading Choose Your Own Adventure novels. But I’m not very creative or artistic. I’m a terrible creative writer and a terrible drawer, but I’m a really good coder. I saw AI as a tool that would let me express myself in a way I couldn’t before. Computers let people who couldn’t do analytical thinking do it; we solved the problem of doing the left-brain stuff for you if you were right-brained. But we hadn’t solved the problem of doing the right-brain stuff for you if you were left-brained. I saw AI as a tool that would let me do something I couldn’t do before.

Speaking to that humanistic element, I hope we take this technology toward creating fully rounded people who are more expressive versions of themselves, and better versions of themselves with AI than without. Anyway, I was working on this project for Choose Your Own Adventure stories, and the context length was super short. I started trying to figure out whether there was a way to make these context lengths longer.

It turns out you can. I started trying things in the Python code: “I wonder what happens if I do this?” With Llama, there was a trick you could use involving positional embeddings. A positional embedding is information about the order of words injected into the model. Inside the attention mechanism, the dot products involve rows corresponding to tokens. Without positional embeddings, the model sees the sequence as a bag of words; it doesn’t know the order. It’s crazy that the original Transformers worked as well as they did without them, but injecting information about word order makes them much better.

The problem was that once you went past the sequence length the model was trained on, it had never seen anything longer. It thought the entire world could only ever be 2,000 tokens long. If you think about the internal representation these models are developing, it treats that as a fundamental law of nature, like gravity or thermodynamics. The model had never seen a piece of text longer than 2,048 tokens, so at 2,049 it didn’t know what to do. It had just learned that way.

I was messing around in PyTorch and saw where the positional embeddings were. I thought, “What if we just multiply all of them by half after training?” They’re floating-point numbers. That’s one difference between AI and regular programming: regular programming deals with integers, while AI deals with floating-point numbers. That’s where a lot of the magic, elasticity, and plasticity comes from. I divided the numbers by half, and all of a sudden the model could go to 4K tokens.

I thought, “How is this even real? I thought this was a fundamental limitation.” What you’re doing is changing the positions: instead of being at 1 and 2, they’re at 0.5 and 1. You squeeze extra information in through the relative distances. You can squeeze 4,000 tokens into 2,000 positions by saying they’re at 0.5, 1, 1.5, and 2. It just works.

I posted on the LocalLLaMA subreddit that you could do this to make these models bigger. Bowen, who’s now our chief scientist at Nous, messaged me and said he’d been working on something similar. He had an even more advanced version and sent me charts, and we started talking. Then I posted about it on Twitter, and Teknium, who is Twitter-famous, messaged me. He said, “I saw you were working on this stuff. Would you like to join this Discord of researchers?” I said sure and joined.

It was a private Discord at the time, with only about 30 people. I looked at the list and knew who all of them were. They were all these open-source researchers. I thought I’d broken into the club. That Discord is the Nous Research Discord today. We eventually opened it up and made it public, but it started as this core group of open-source people working together. It was the group chat for open-source AI researchers. That’s how Nous started, and it came out of this infinite curiosity: going through the code and thinking, “Let’s just try something and see what happens.”

That’s what’s really great about the AI space right now. It’s a vast, unexplored field of science. In chemistry and biology, things become very specialized. In physics, you can spend your whole life trying to understand the state of the art. If you’re very lucky, you might make a tiny contribution. In AI, you can try something and it probably will work. It isn’t that your idea doesn’t work; it’s that nobody’s tried it yet. That’s a powerful and exciting time to be in the formation of a science.

Sina Habibian

That’s counterintuitive. Usually, when there’s so much attention going into AI, you’d imagine some version of the efficient market hypothesis: all these researchers and engineers are working on it, so there must be nothing else to look into. You might think you should work through the ramifications and move on to what comes after all these problems are solved. But you’re saying there’s open greenfield right here.

Jeffrey Quesnelle

Every few months there are new discoveries about architectures and data, and they meaningfully advance the state of the art. Take what happened with DeepSeek. The application was special, but the ideas were actually pretty simple; no one had tried them. They introduced something called Multi-Latent Attention in DeepSeek V2. That’s how they’re able to deal with these super-long contexts without the memory blowing up.

There’s the question of whether you can do it, and then whether it takes a huge amount of memory. They developed Multi-head Latent Attention, where instead of doing the full matrix multiplication, they do a low-rank projection of the Q and K, the query and key pieces of the attention mechanism. They project them to a lower dimension, do the attention in the lower dimensions, and then scale it back up. It turns out that this can reduce memory usage by 90%, and the results are exactly the same. These can be five lines of code that make a huge difference.

That’s because those five lines are operating at scale. Scaling is what has allowed AI to grow. If you make it bigger, it will just be better. This applies in both dimensions: if you give it more data, it will be better; if you give it more compute and make the model bigger, it will be smarter at qualitative human reasoning. We know how to scale things along those dimensions. We don’t know how to scale things with people very well, but we can throw more money at it. If that’s the problem, as a society we can throw more money at it.

Sina Habibian

That also speaks to the value of Nous in its current incarnation, and to Psyche, the network you’re building. The bottleneck becomes: can I raise funding and procure a bunch of H100s to train a model and test my idea? Now there’s a more plug-and-play network that I can upload my idea into, and it gets trained for me.

Jeffrey Quesnelle

That’s what we’re trying to build with Psyche.

Sina Habibian

I really like hearing the story of how this Discord became the Nous Discord. How did you go from there to here? What were the big inflection points in the story? You guys are making progress, but the whole AI field is also changing in massive ways.

Jeffrey Quesnelle

For Nous, it started from this first principle: we wanted to be able to run the AI and step through the whole code, access it, and see it all. I’ve got my 4090 there, and I want to be able to run the model and step through it. Mark Zuckerberg did a great service for the world when he released Llama, because it forced people to realize that there wasn’t really an inherent moat. There was a moat, but not really.

After we formed up together, we said our mission was to be the open AI accelerator. Whenever there’s a barrier to the open-source world being able to access something, let’s knock it down. We looked around for other barriers. One thing that came to us was that the entire open-source ecosystem for models depended on a benevolent third party releasing a foundation model.

Sina Habibian

We can talk about what a foundation model is.

### Nous Research origin story

Jeffrey Quesnelle

It requires this thing called a foundation model, which is very expensive to create. At the time, there was Meta and maybe Google had released Gemma, but that was new. There were only two or three people putting out these foundation models. We wondered what would happen if Meta just stopped doing this. What if we didn’t get Llama 3? This was before Llama 3 even came out. What if we didn’t get Llama 4? The closed providers would continue, because they were already on the path. They were in the race and had the ability to keep going. The open community needed to keep up, but keeping up depended on an economically unsound activity from someone else. That was trouble.

So the question was: what can we do to give the open-source community a credible way to stay in the race? We need to be able to train our own foundation models.

Sina Habibian

Easier said than done. The problem is identified. What’s the real problem?

Jeffrey Quesnelle

It turns out the problem isn’t access to data. The data these AI companies have been training on, at least in the text modality, is basically just the internet. Anyone can get the data. The problem was access to GPUs and a way to coordinate them.

### fourier transforms for gradient compression

The way the text models started out, you only had one GPU. PyTorch was written under what’s called the single-device paradigm. When you write the AI model, you write it as if it’s on one GPU. You think about it as if it were running on one massive GPU; you don’t think about how it has to be split up. You write a matrix multiplication as one line of code, and that one line might actually take a megawatt of energy to run. That’s the beauty of abstraction.

The single-device paradigm kept the complexity of the AI itself tight. You can open up a GPT Transformer; the code for a GPT is famously about 40 lines. It’s a couple of for loops, a couple of matrix multiplications, an activation function, and normalization. That’s it. Under the hood, though, to make it work across multiple GPUs, the tooling had to create abstractions that did it for you.

As it became clear that we’d need multiple GPUs because we couldn’t make them bigger, the tooling was built around the idea that the GPUs could act like one big GPU. The most flexible way to do that was to have different pieces of the model on different GPUs, while connecting them so they could move data between GPUs arbitrarily fast. That meant buying a bunch of GPUs and hooking them up together in a single data center. They all had to be connected symmetrically, with similar latencies, and have the same type of GPU and the same VRAM. You scale by making infinite copies of one GPU and hooking them up with either RoCE or InfiniBand, which can run at 400 or 800 gigabits per second. The cables cost $1,000 a foot.

That setup lets you move data between GPUs arbitrarily fast. As long as you can do that, it’s easy to preserve the single-device abstraction at the code level. That’s how the training tooling developed. But if you want to train across the internet, with lots of different computers, that’s where things break down. The internet doesn’t give you that. You have highly heterogeneous connections, and the total throughput is orders of magnitude lower.

To give you an idea, the new GB200 racks from NVIDIA—Grace Blackwell 200s—connect 72 of them together with the new InfiniBand. The InfiniBand connections between all the nodes have bandwidth equivalent to the entire internet. Just the connections between those GPUs have the same throughput as the whole internet. That shows you we needed several orders of magnitude of reduction to make this possible. That was the fundamental technical limitation we saw.

We sat down and asked whether we could solve it. Was there a fundamental reason you had to share all the data all the time, or was that an artifact of how the tooling had developed? We spent about a year on it and developed a couple of algorithms. One is called DiLoCo, and a more advanced one is called DisTrO. They showed that you don’t need to communicate everything between all the GPUs. You can communicate a very small fraction of the data, and only a very small fraction is actually needed.

If you have an infinite-sized internet, you’d never have JPEGs. We’d all share bitmaps and uncompressed AVI files, because we could. But when we wanted to share pictures on the internet, we needed to figure out JPEG: how to show the exact same image with one-thousandth the information. We applied that same idea to gradient compression during model training. Now you have a thousand-to-one reduction in bandwidth.

Sina Habibian

So that’s a thousand-to-one reduction in bandwidth, on the order of what you get with bitmaps and JPEGs.

Jeffrey Quesnelle

Exactly. A lot of the breakthroughs now are ideas pioneered in the ’90s in signal processing. Their application is just now happening at this huge scale. You take ideas that were little technical things in the ’90s and apply them to huge intelligences, and you get these orders-of-magnitude effects.

We took that and said, “Now we can train models in a distributed way.” Then there’s the other aspect: how do you coordinate them? If you want them to come to consensus in a distributed fashion and exist in an adversarial environment, without a middleman, third party, or central server controlling everything, you need a decentralized consensus mechanism with economic security. Thankfully, the world has already developed one: crypto rails. That’s how we came to the conclusion of developing the Psyche Network. We need to coordinate these GPUs in a decentralized way, and crypto is a great tool to do that.

Sina Habibian

How does the math work? How does it mirror how compression with JPEGs works?

Jeffrey Quesnelle

It works through something called the discrete cosine transform. When you train a model, you have your model, your inputs, and your targets. These are the fundamental pieces of learning: you have data and something you want the data to look like. Then you do something called cross-entropy loss. The objective is to minimize the difference between those two things. Then you do backpropagation to discover how much you’d have to change your representation to make what you thought was going to happen be the right answer.

That’s the whole thing. Next-token prediction is just taking the data you have and moving it over by one token. That’s how you get the next token. People thought, “What if we took the target labels and moved them over by one, and had the model predict the next one?” We ran that through the objective function. I want people to understand: through just that objective function, which is basically matrix multiplication and linear algebra, and backpropagation, which is standard calculus, the emergent property is a language model that can talk and speak like humans. That’s the mind-blowing thing. It tells us something about how we’re probably wired internally. We can get to that later.

The discrete cosine transform works like this: after backpropagation, you get gradients that say how much you’d have to change your weights to get the right answer. These gradients are on the same order of magnitude as the model itself. For every element in the model’s parameter space, there’s a corresponding element in the gradient, which is how much to move. You add the two together, and that’s how it happens. In reality, you use an optimizer: you take the gradients and say, “I’m not going to go all the way there. I’m going to go part of the way.” The most popular one is Adam, with momentum. One of the authors of Adam worked on our DiLoCo paper, too.

What we do is take these gradients and think about them in the frequency domain. With Fourier transforms, you take these things and convert them into the frequency domain. We had a thesis that there are two types of frequencies in the gradients: fast-moving and slow-moving frequencies. Some things the model is learning very quickly, and they communicate differently to the nodes.

### parallels with reinforcement learning

Inside each GPU, we build up these frequencies. Every gradient step, we pop off the top K frequencies. We select the K biggest frequencies and communicate just those to all the other nodes. Then we do the inverse transform, which gives us a sparse modification of the gradient matrix. If you think of the gradient as a 2D grid, it’s a giant 2D delta that you can convert into the frequency domain. You pick out a specific frequency, communicate it to the other nodes, and do the inverse operation. Now you’re updating little spots all over the grid, just the important pieces.

The idea is that the gradients can be represented in frequency space. The second idea is that there are long-running and short-running frequencies. The long-running ones accumulate over time and eventually bubble up. If it takes a while to learn something, it can take a while for those frequencies to accumulate. If there’s a really high-signal frequency, we pop it out right away. Communicating just the top K frequencies is enough to get the same results in the model. That tells you the actual learning is happening in a different domain. Thinking about the gradients as a giant 2D grid and communicating the gradients directly is highly inefficient.

Sina Habibian

You can almost look at the gradients as the pixel values, like RGB values, that get compressed down in an image. What’s the intuition behind a high frequency here?

Jeffrey Quesnelle

We don’t know, other than the amplitudes. What those frequencies correspond to in representation space, or what they actually represent, is anyone’s guess. That’s the thing about LLMs: we don’t really know what any of these things inside mean. We know what we do, but what do the parameters represent?

People are working on interpretability, trying to create auxiliary models that can figure out what different parameters represent. Can we map them back to something humans can understand? “This is Canada” or “this is the color blue” or “this is the Golden Gate Bridge.” But we don’t really know.

We think of them as frequencies and pull them out. We communicate only the frequencies and amplitudes between the nodes. Then we do the inverse transform to get a sparse gradient, with empty spots. You add in only certain elements. Over time, adding in those few pieces every few steps is enough to create the learning.

Sina Habibian

That’s a fascinating set of technical problems to work on.

Jeffrey Quesnelle

It was one of those things where Bowen had the original idea. He’d worked in signal processing and seen problems in audio representation solved in a completely different way. At the end of the day, they’re just linear matrices, so he thought, “Wait a second, these gradients are just like that, too. I wonder if they behave the same way as these other signals.” It came from having that background, having seen something, and then being able to say, “Let’s try it and see what happens.”

It took a long time, about a year, to get it right. There was a large element of faith. We had to believe it was going to work before we had evidence, or we would have given up. Eventually it worked. We were able to build this DisTrO optimizer, which reduces the communication throughput needed to train models by a huge amount. We’re pairing that with a network to coordinate permissionless training of one huge model, or multiple models. That’s the Psyche Network.

Sina Habibian

Is there a reason the person with the state-of-the-art NVIDIA rack we were talking about wouldn’t use the DisTrO optimizer to get a boost on top of those high-bandwidth interconnects? Would the bottleneck then become that you can’t do the compute fast enough, or that you run out of memory?

Jeffrey Quesnelle

You certainly could. It’s interesting because you don’t have to do fully decentralized training. The easiest application is a hybrid, where you own two data centers and communicate quickly between them.

One thing about the DisTrO optimizer is that it works after the backpropagation, so you still have to be able to hold the whole model in fast memory. If you’re doing super-large training, say at hundreds of billions of parameters, where you can’t fit it on one HGX—the computer that’s a supercomputer made of GPUs—you still need the interconnect between two HGXs to do it. You can think of two HGXs as one node, and then use DisTrO to interconnect them.

There’s still a place for fast connections. As part of the Psyche release, we’re going to incentivize research in this area. One area I’m particularly interested in is parallelism for mixture-of-experts models, where the different experts could be co-resident on different nodes. That might be a way to get more out of them. The biggest state-of-the-art models are mixture-of-experts models, not dense models. I can get into why later, too. Right now, we’d have to have the entire mixture-of-experts model on one GPU, or one node—one HGX. That’s an area where we’re going to incentivize research. What’s great about crypto rails is that they can harness the economic energy and pair it with the technical energy, and the two can become one.

### sponsor: Splits

Sina Habibian

Into the Bytecode is sponsored by Splits. Are you tired of sacrificing security for usability? Splits believes it’s still way too hard for teams to self-custody their onchain assets. They’re building a new kind of internet-native bank on top of Ethereum. Splits makes it easy for teams to manage the whole lifecycle of their finances, from structuring revenue-sharing agreements using payment flows like splits and waterfalls to managing those earnings once you receive them using passkeys and smart accounts. Splits is being used by teams like Protocol Guild, Zora, Songcamp, and others. I’m a big believer in them and recommend checking them out. You can learn more at splits.org.

### neural networks history and fundamentals

So one of the things we briefly talked about when we caught up before was this idea that most of us, even if we’re technical, are using these large language models through chat interfaces. We understand them at this level of abstraction: there’s training data, you have inputs and outputs, you get a loss function, and you do backpropagation. There are values, these parameters, getting updated. But one of the things we talked about was: What the hell are these things?

### the cat is out of the bag

Maybe we can use this as an opportunity to get into that question. With your bent for philosophical thinking, and also talking about how these models are actually trained, there’s this first phase where they’re being trained, but after that there’s instruction tuning and reinforcement learning from human feedback that makes them understandable to us. That’s almost a layer on top of the chaos that’s contained within. So what is going on? What are these things? How does it all work?

Jeffrey Quesnelle

To take a step back, I mentioned this thing called deep learning. We didn’t actually start with language in particular. The first thing that kicked off what we have today was the invention of AlexNet.

There was a competition called ImageNet where you had to write image-processing code to recognize certain images. At the time, the best solutions came from people using very advanced image-processing techniques they had written themselves. Then along came AlexNet, a neural network that could recognize images, and it wiped the floor with everybody. People were like, “Whoa, wait a second. It can just learn on its own to solve the problem?”

That was our first hint that neural networks really could do the type of work that had always eluded previous artificial intelligence. If you read about the 1980s, people thought computers were going to be intelligent. There were all these attempts to create artificial intelligence, but we thought about intelligence as how we thought it ought to be, not how it might actually be.

We thought being intelligent meant thinking about all these rules for how things work. It turns out that’s reasoning, but it’s not intelligence. Reasoning is almost the output of intelligence. Intelligence is the raw material underneath, but we have no idea how to quantify it. We know how to quantify reasoning, but we have no idea how to get an accurate introspective account of intelligence. It’s too fundamental to who we are for us to experiment with it that way.

We tried and failed at AI by making these expert systems. We figured that if you got all the world’s smartest people to create rules, it would work, but it didn’t. Then AlexNet showed that neural networks could recognize images as well as or better than humans, or the smartest people using conventional image-processing techniques. That was a “Whoa, okay, maybe this is what we need to be looking into” moment.

A big part of this story is that you need faith in a signal: if you commit to something, there’s a reasonable chance it will work. You needed to see AlexNet work to know you should work on neural networks. It gave proof that it was okay to spend more energy on this specific thing instead of the other possibilities. AlexNet came along, and then there was this big kickoff of deep learning, trying to solve problems using neural networks.

Typically, you were trying to solve a specific problem and optimize for a specific downstream task. This is where I come into the picture, with autonomous driving. The very first autonomous-driving systems had deep-learning models to figure out where the lanes were. It was all a very specific thing you were trying to do. It worked kind of okay, but it was pretty brittle and still took a lot of engineering.

You were no longer writing code. You were building blocks of neural networks: convolutions, normalization layers, and all these things. You had to try different blocks, like Lego, stick them together, and tune all these hyperparameters. Maybe if you got it just right, it would work and see the lanes on the highway. It did that way better than any code you could ever write, but it was still a brittle process. If you got one thing wrong, it wouldn’t work.

That was where deep learning got us. Then the next big “aha” was the invention of the Transformer. It was created in the paper “Attention Is All You Need,” a very bold title, but justified by its efficacy.

The Transformer architecture has at its core an operation called attention. Attention is really just a specific way of multiplying matrices. Calling it attention is our human attempt to come up with an analogy for what’s happening. The model has learned things and is able to learn how to look at things. Rather than trying to solve the problem directly, it learns how to focus on specific areas. When it sees something like this, it learns that it needs to focus on this specific area in its abstract representation space.

For whatever reason, this architecture blew away everything again. It was way better than anything else. What was even crazier was that it was extremely resilient. Before, if you wanted to detect lanes or recognize letters, you needed a totally different architecture for each task. With the Transformer’s attention mechanism, you could throw almost any problem at it and it would work. It could solve whatever you wanted; you just needed the attention mechanism.

That freedom made it the first general neural-network architecture. The previous ones were very task-specific. The Transformer was the first general neural-network architecture shown to work across any modality.

The original Transformer paper wasn’t doing next-token prediction. It was doing an encoder-decoder model, which transforms one sequence of text into another. Its first application, which you probably never knew you were using, was something like Google Translate back in 2016 or 2017. Language translation is one whole text going into another text, and the Transformer did that way better than anything that had ever been done.

After that came the idea that you could remove the encoder and use what’s called a decoder-only Transformer. Rather than going from one sequence to another, it generates the sequence autoregressively, one token at a time, using next-token prediction. That’s the current dynamic: nearly all the Transformers you see now are decoder-only Transformers, autoregressively sampling one token at a time.

So now we’re in a world where we have this general architecture that can figure out how to represent things across modalities. The next moment was the discovery of in-context learning. Once you have a foundation model, the “aha” is that the model can learn to follow instructions in text it’s never seen before. We had no reason to think it could do this. Everything we do now with a chatbot—telling it, “Do this, do this, do this”—it can meta-represent those rules it’s never seen before and incorporate them into its learning.

But backing up: What is a foundation model? We can model, but what do we want to do with it? We have access to a vast amount of data, and that’s the internet. Someone had the great idea: Why don’t we make a model that models the internet, modeling every word that humans have ever written? If we do that, somehow it will contain all human knowledge within itself. We’re trying to compress all human knowledge and understanding into one model.

The vast majority of AI training work is at that stage: training the foundation model. That’s the secret thing you need to create anything else afterward. It’s interesting that what comes out of a foundation model is something that can reason like humans, that can do the same sorts of things our brains can do. There’s no reason for this to be the case. Until now, as a species, we’ve been completely unable to elicit intuitive reasoning. We had no way to artificially create it.

Now we have the equations that are the necessary and sufficient conditions to give rise to intuitive reasoning. It may or may not be how it actually happens in our brains, but we have at least one example: if you follow these steps, intuitive reasoning will be the output. It’s likely that whatever is going on in our brains is isomorphic to whatever methodology is happening inside these language models. For the first time in human history, we can experiment with intuitive reasoning. It’s completely insane.

### the human mind and AI, hyperdimensional representation

Sina Habibian  
What’s your personal take on how an autoregressive Transformer-based model works? If you were to draw some intuitive links—not perfect parallels—what is this Transformer architecture doing? Is it like looking at the entire dataset of everything I’ve observed, all my sensory data to date? How is this mirroring, or not mirroring, what happens inside each of us?

Jeffrey Quesnelle  
The idea I think is probably the most likely explanation is that AI models are creating a hyperdimensional representation of language. That’s also what we do. When I say to you “the color blue,” the smell of mint, or the taste of chocolate, what I believe is happening in your brain is that it has learned a hyperdimensional representation space.

Blue sits in this space with values in areas for things like cold. There’s a dimension for cold, and it has some small values there. Water gets activated a little. You have this hyperdimensional representation space of all the concepts you can possibly have. Language is like a hyper-encoded set of cheats for putting specific values into this hyperdimensional space of understanding.

That space comes from the selection pressures we underwent through evolution. Our selection pressures forced us to respond quickly to unknown situations. We exist in time, and you only get one shot at it. That’s true from the first single-celled organism, from the first replicating chemicals at some underwater vent, to now. Only the things that were able to continue on did so.

They had to figure out ways to deal with the unknown world, get energy, and respond to unknown situations. That’s the large-scale selection pressure that entropy and time put on reality and physical reality. Systems that aim to optimize for continued existence and getting energy have to deal with that.

Sina Habibian  
Our cross-entropy loss, like a thing?

Jeffrey Quesnelle  
Yes, exactly. For us, cross-entropy loss is time and energy. Over time, we developed more and more ways of quickly representing the outside world and deciding what action to take.

Nearly all of our brain is intuitive reasoning. It’s why I can throw a ball to you and you catch it instantly. If you had to calculate the trajectory, I could give you a pen and a piece of paper and your whole life, and you’d never be able to write out the correct dynamics of your hand reaching out and grabbing the ball.

It’s how mammals can look at trees, hear a twig break, and immediately know there’s an animal over there that might come eat them. Everything you do is instant recognition, pattern recognition performed in a hyperdimensional space optimized for survivability. Creating that representation was the best way to survive. It was forced onto us because it let our ancestors live longer, be smarter, and get eaten less.

Language models are most likely also taking words and representing them in a hyperdimensional vector space. The word “blue” maps onto values in a matrix, with values for the concept of cold and other concepts. We call it cold and blue because we have words, but the actual concepts we have are things for which we don’t even have words.

Words are compressed, lossy representations for communication and discursive thinking. Discursive thinking operates at a higher level of abstraction than the intuitive movements happening at the embedding, representational layer.

The evolution of speech is interesting. The natural formation of us is this hyperdimensional representation of concepts, before we even have concepts of words. We hadn’t developed speech yet. Then we discovered that if we could communicate what was happening inside us to other people, that would be an evolutionary response. We had to cooperate in advanced ways with other humans.

Speech emerged from finding that we have a representation of the world in our brains, and that by communicating tiny pieces of it to other people and receiving their communications, we could enhance our survivability.

Discursive thinking is sort of analogous to what we’re doing with the demo algorithm, too. The gradients themselves are this hyperdimensional space that’s too big to communicate. We’re creating words for the gradients, using these types of words, then communicating them to someone else. They might expand them and apply them to their own representation.

What’s interesting about humans is that no two people have the same representation. When I say “blue,” your reception of it is totally different internally, and your application is different. That’s how you can take what I say in a different way.

Sina Habibian  
Fascinating. There’s a kind of mirroring between what’s happening inside a node as it’s being trained—there are multiple fractal layers—and how a human holds values and updates gradients. We hold the parameters in our models, and we get this compressed gradient update from someone else or from the outside world and update our own weights.

### intuition and reasoning

Jeffrey Quesnelle  
It’s likely that we’re much more complicated. We’re not one monolithic model; we’re lots of different ones that have evolved together over time. But it’s close enough of a facsimile to say that it gives rise to apparently equivalent phenomena. That’s very different from saying the two are the same thing.

Sina Habibian  
Yeah, we don’t know whether the underlying architecture is the same, or even conceived in the same way at all.

Jeffrey Quesnelle  
That gets to the heart of the philosophical question: What does it mean to be human? Are we only composed of the matter that is our body, and is the entirety of our awareness an emergent phenomenon from this representational space? Or is our consciousness something beyond that? These are open questions in philosophy, and even nowadays more in science, too. People are willing to ask about the ontological foundations of the world.

I would say not that they’re the same, but that they give rise to equivalent phenomena.

Sina Habibian  
It’s fascinating. We’re slowly unbundling intelligence. Another thing it makes me think of is that, as a human being, I’ve felt for a long time that I can think better and more effectively when I can write, externalize my thoughts, and then read them. On a computer, I can write things in a Google Doc and move them around. I effectively have a larger file system or RAM than I can hold in my short-term working memory and manipulate.

The deeper, crazier thing is that I read one of Ted Chiang’s short stories last week for the first time: “The Truth of Fact, the Truth of Feeling.” I don’t know if you’ve come across it, but it’s 100% worth reading for someone working on what you’re doing. He makes the case that language itself is the first technology that augments what we have. Without it, you can express an idea and someone else can hold it and echo it back at you. You’ve added external state to a system that didn’t have it previously.

Jeffrey Quesnelle  
I tell people all the time: Humanity’s original sin was the written word. Everything we have now is downstream of that. If you wanted to nip this in the bud, you needed to stop the written word. That was the real unlock. Everything else is the natural consequence of the invention of the written word.

It’s interesting you mentioned being able to write something out yourself, because we see the exact same phenomenon with AI. I mentioned those task-specific models that didn’t work. We had the order wrong: we needed to create the intuition first.

Reasoning is the ordered application of intuition. At first, we had models trained on the internet. They were good, but they created all sorts of crazy stuff. Then we developed instruction following, and then the chain of thought.

The chain of thought came from somebody trying an idea. He had the model first say, “Let’s think about this step by step.” Forcing the model to write out its reasoning steps caused it to get way better on all the evaluations. It got better just by forcing it to write out, step by step, what it was going to do before solving the answer.

People make fun of things like the strawberry question: “How many r’s are in the word strawberry?” It’s easy for you because you can look at the word “strawberry.” But if I say, “Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty and dedicated to the proposition that all men are created equal,” how many e’s are there? Very suddenly, after only a few words, it’s hard. I’m human, I can see how many y’s there are, so why can’t it?

The reason is that you’re asking it to count and do reasoned thinking. Intuitive thinking gives you one time slice to answer. All our reasoning happens like that: intuition happens instantly. For an AI, that’s one token. It gets one forward pass and has to get the answer all together.

Counting letters is a reasoning step. It involves looking specifically and accumulating an answer. Our minds can instantly recognize how many objects there are up to seven or eight, but beyond that, you have to count. If you have seven balls versus fourteen balls, you’ll know seven right away, but you won’t just instantly know fourteen. The same is true of letters. Our intuitive thinking can do some counting problems instantly, but for anything beyond that, you need to develop the ability to reason.

The invention of reasoning was the last unlock in humanity’s intellectual development: taking intuition and structuring it in a specific way. With reasoning models, we discovered that we can force them into a reasoning mode, make them think about what they’re doing step by step, and have them write it all out. Just like with humans, that makes them way, way smarter at getting answers.

Again, this is a weird thing where the emergent phenomenon we see in AI completely maps onto the same phenomenon we experience in our own learning. It tells us the two are isomorphic under some conditions.

You’ve probably heard about DeepSeek. What DeepSeek had the incredible idea of doing was saying: Reasoning tasks and chain-of-thought reasoning are important. Why don’t we do reinforcement learning just on the reasoning steps?

There was old reinforcement learning from before AI and language models. It was brittle and didn’t give rise to emergent intelligence. But if you first create intuitive thinking, then use reinforcement learning, you can make the reasoning process better. It’s like taking a baby and sending them to high school. They’ll never learn Socrates if he’s just asking them questions and they have to think through the answers.

DeepSeek applied reinforcement learning to verifiable questions, where there’s a verifiable answer. Next-token prediction, with attention, gave rise to intuitive reasoning in the models. Once you already have that intuitive reasoning, the goal is to use it to do structured reasoning and come to the right answer.

They take the AI model and sort of freeze it. The intuitive reasoning is frozen; they’re not changing that. Then they do reinforcement learning, where the model tries to solve a hard math problem. The loss is no longer next-token prediction, because we don’t know what the next token is supposed to be. We let the model use the tools it built through next-token prediction to write out a reasoning chain, and the only thing that matters is whether it gets the answer right.

That’s the reward signal. Unlike normal cross-entropy loss, which gave rise to intuitive reasoning, we change the objective function. Now it’s a binary reward: you assign a score to the outcome of the entire reasoning chain. Either you got the answer right or you didn’t, just like when you go to school and you get a thumbs-up or thumbs-down.

Through reinforcement learning, the model can try to discover new ways of using its intuitive reasoning to get the correct solution. When it does, it gets a one. Through the magic of reinforcement learning, we can take this nondifferentiable objective—one or nothing—turn it into a differentiable objective, backpropagate new gradients, apply them to the model, and go again.

That’s where you see this new class of so-called reasoning models. Sam Altman just announced this is the next thing. They’re not even going to release GPT-5; they’re renaming GPT-4.5, and every model from now on will have this new reasoning state applied after it learns the world through intuitive thinking.

Early ChatGPT was all intuitive thinking. That’s why it could be profound in some ways and make the simplest mistakes in others, just like people thinking off the cuff. Now we’ve figured out how to take these old reinforcement techniques from the 1990s and apply them to reasoning processes. Once we have intuitive reasoning, we can elicit real, logical, structured reasoning to solve extremely difficult problems.

Sina Habibian  
So the base model’s intuitive reasoning is like: I’ve built a deep, higher-dimensional understanding of the world, but when I’m prompted with an input, I shoot from the hip. I’m acting intuitively, trying to do my best with one token.

Jeffrey Quesnelle  
Yeah, literally. It’s life with a gun to your head. If someone said, “Gun to your head, what would you do?” What would you do? Life is just boom, boom, boom, boom, boom.

Sina Habibian  
Right, so it’s the analog to our intuition: how we move through the world and the thoughts that arise. Then you find the trick of giving the model an instruction: “Show me your reasoning.” That’s like us thinking through a problem.

Jeffrey Quesnelle  
You might naively imagine that anything that can be derived from this preexisting intuitive lattice should be a foregone conclusion. But there’s a process of actually working through a problem and going down different paths. There’s an exploration process, and you’re doing way more work when you do that.

The reason it shouldn’t be a one-shot answer is that, in one forward pass, there’s a fixed number of matrix multiplications. You can make an information-theoretic argument that it could never recognize every possible function, because you’re doing a fixed number of multiplications over an unbounded space of possibilities. It can only recognize certain classes of things.

With chain of thought, when you said, “Think about it step by step,” you were forcing the model to expend tokens and have more time to think. By going longer, you did a lot more multiplications before it got the answer. The answer is 42, right? By going through more steps, it can do more.

An AI model can only get more time to think by saying more words, because that’s all it exists as. That’s its sense of time: each new token is its Planck time of reality. The only way time moves forward for it and it can spend more energy is by saying more tokens. We found a way to hack the model into doing more matrix multiplications to get to the answer.

It’s nice because it’s interpretable to us, too. By forcing it to say things out loud, we can see what it’s doing. That’s the paradigm that exists now: we spend more time coming to the answer. To make the model spend more time and money, we have it write out in language the thoughts that could help it get to the answer.

Then the next step was to switch the optimization function. In verifiable domains, you have it reason and try to get the answer. Once it gets the answer, you take that nondifferentiable output, make it differentiable, feed it back in, and that turns something that isn’t intuitive and requires reasoning into intuitive code.

Sina Habibian  
It’s like something becoming second nature.

Jeffrey Quesnelle  
Yep. Great mathematicians will tell you that they look at equations and see them instantly, because they did so much work going through them that they forced an intuitive understanding of that discourse into their intuitive reasoning.

It’s the same for me with code. When I’m coding and really into it, I try not to think too much about what I’m writing, because if I do, I’ll screw it up. I can feel the code. I feel the structure of the project and everything, and I can let go. That came from the entirety of my life since I was 10 years old.

Sina Habibian  
You’ve had this reinforcement-learning loop running.

Jeffrey Quesnelle  
I’ve built up that intuitive understanding of code and things like that. The magic with GRPO, which is the one DeepSeek uses, is that they figured out a way to take this non-verifiable, nondifferentiable objective and convert it back into the language of backpropagation. Then you can backpropagate the gradients, turn it into intuitive reasoning, and loop, loop, loop.

That’s where it scales. Humans can only do this one time, but with AI we can do it a thousand times, a million times, in parallel. That’s where the gains will be.

What’s interesting about reasoning models is that they’re dynamically generating training data. Pretraining intuitive reasoning is like seeing all the words that have ever been written. You’ve seen everything; now you have to go out on your own and discover something new. The model generates its own training data. Once it gets the answer right, that becomes training data. You’re searching for examples of how to solve a problem, and once you find one, you backpropagate it.

We’re at this moment where this is working through our system. It can work in any domain where there’s a verifiable correct output. Math and programming will be first. Then there will be a process of taking other hard sciences and getting them to a point where you can know if something is objectively right or wrong. That’s hard to do. How do you do this for biology? When we introspect our understanding, there are a lot of heuristics. Maybe you hook these models up to virtualized lab settings where they can run experiments. I don’t know.

Then there’s the question of what happens in more subjective domains where there isn’t a verifiably correct output. You need the human in the loop, and then we get into how taste is subjective, and aesthetics and things like that.

Sina Habibian  
How does this ripple through our whole system? What happens from here? What are all the things to work out?

Jeffrey Quesnelle  
The reinforcement-learning reasoning stuff is another dimension of scale, and we’re at day one of it. There’s a lot of runway before it gets very far or saturates. It’s interesting that models trained this way have applications elsewhere. They also become better writers. There are hints that doing this on verifiable domains has benefits in other applications.

As long as intuitive thinking is grounded enough in the world, you can create more and more complex objectives. The ability to access the internet is part of this, and it’s where a lot of people have safety concerns. You could say the objective is to elicit some outcome in the world. Obviously, it would be guarded, but you could probably get pretty far if your goal is to make the model smarter by giving it an outcome as its objective.

Outcomes can be things like gaining a million Twitter followers. That’s verifiable. There are many ways to get a million followers: becoming a social-media sensation, becoming president of the United States, and so on. Backpropagation is a relentless forcing function. It will force energy in whatever ways it can.

You can probably get pretty far with reinforcement learning just by expanding what you call a verifiable objective. For truly subjective things that are grounded in our experience as humans, we value art because it reflects our experience of the world. It will be interesting to see whether a model would develop the same aesthetics as we do. It won’t be able to do that a priori, because it’s seen too much of what we consider valuable. The jury is still out on where that goes.

Sina Habibian  
This stuff is infinitely interesting to think about. Our sense of what’s beautiful—why we find elegance beautiful, why a mathematical formula is beautiful, why symmetry is beautiful—seems to be about order and understanding deeper truths. That might be the same thing these models find beautiful, or it might be different.

Maybe our understanding of beauty, elegance, and symmetry comes from a larger forcing function that’s unique to us or special to us. That could go back to physics and the anthropic principle: Why does the universe exist? There are weird things in the universe that give rise to life as it is now.

The fine-structure constant is a dimensionless quantity that represents ratios between fundamental forces. It’s about one over 138. It’s dimensionless, so it’s invariant to the way you measure it; it’s not in kilograms or units of energy. For all matter to exist, for electrons and quantum mechanics and particle physics as we understand them to work, this number needs to be one over 138.

If it were one over 135, atoms wouldn’t form. If it were 142, matter wouldn’t collapse after the Big Bang. It has to be about one over 138 to give rise to the order that can create matter and the structure we have now.

As the things we thought were special to us get pared away, maybe that will force us to introspect more about what we think is special and unique about ourselves. I think the world is going to struggle with that over the next several years. The first effect will be economic. We’re going through the Industrial Revolution: the Industrial Revolution took work that only humans could do—digging ditches, lifting things, planting crops—and erased it.

People don’t know that before 1880 or so, something like 60% of the United States and 90% of the world population was engaged in subsistence or agrarian farming, directly trying to feed themselves. Within a generation that got completely flipped. A lot of people lost their jobs, and society underwent massive reformations because of the Industrial Revolution. World War I and World War II were byproducts of the Industrial Revolution and the reordering of the world under that paradigm.

Out of that came the class of things only humans could do: reasoning and thinking. Over the next 10 years, we’ll have this second Industrial Revolution of thought and intelligence. Huge classes of things only humans could do will be automated and taken away. The question is: As a society, where do we place people?

We shouldn’t look at it as something to avoid or as something bad. The Industrial Revolution was an opportunity to take on the next class of challenges and the next class of growth for ourselves. This is a deep ask. It’s like we’re on a hero’s journey, at the next moment of transformation. It will be difficult, but it’s part of the path.

It’s magical that we’re creating the process of intelligence and reasoning, and that it asks us what we ultimately are and where this is all going. It’s a much more wondrous world that we’re coming face to face with this question in.

Jeffrey Quesnelle  
Modern scientific materialism was able to ignore this question for a while. We’re all made out of matter and particles; that’s all anything is made out of. But we all know humans are kind of special, even if we don’t say how or why. It was easy to ignore because the ordered world we could control was so infinitely controllable. There was this class of human thinkers off to the side, and the materialist world never had to confront them. We never faked the human thing before, so we got to ignore what we are. Now we’re going to be staring at it straight in the face.

Sina Habibian  
I feel like this is one of the bigger questions. When Stable Diffusion came out, people saw it, but they weren’t really grokking what it meant. There’s a similar thing here: we have these models, and we figured out reasoning. There will probably be other breakthroughs, but this is the question looming underneath this breakthrough that we’ll all have to grapple with in 10 or 20 years.

Large transformations in society like this are often mirrored by conflict. That was one reason we took it back to Psyche. The next global conflicts will likely involve this technology, whether through access to the chips or to the rare-earth minerals needed to make them. These technologies will play an important role in the next great social changes.

Things could be turbulent for a while as we figure out our place. But as a shared human community, we can see where things are going. The great promise is that this technology could give us the ability to solve a lot of those problems with technology, rather than fighting over it. If we could solve room-temperature semiconductors with AI, that alone would alleviate huge classes of problems in the world.

We have a chance to avoid the war and takeover version. There’s a much better path where this technology uplifts all of humanity together. Having said that, we have to be prudent about the possibilities. That’s why we’re creating a way for decentralized artificial intelligence to be accessible to everyone, without control by a single nation-state. It exists nowhere, and therefore it can’t be shut down, taken away, or made inaccessible.

This could be a very centralizing thing that concentrates power even more. That’s why you see this race going on. There’s an idea in the industry that whoever gets to AGI or ASI first will win the world. Once a model can make itself better and solve more problems on its own, you get this exponential curve. Whoever gets to that loop first sits in such an asymmetric power position that they get to call the shots.

That’s why OpenAI is raising $500 billion or $5 trillion. The stakes are on the order of the economic output of countries. That’s the game being played: to get to that level, and then whoever holds it—

People say you never should have developed AI, just as you never should have developed the atomic bomb. Maybe there’s an argument there, but it also ordered things in some ways. Whether you should have created AI or not is a moot point. It’s matrix multiplication. It’s elementary arithmetic. This is a fundamental property of the universe, and the cat’s out of the bag. In the world where it’s out of the bag, this is where we are.

It wasn’t necessary that this could happen in the universe. It could have been impossible to elicit this. But the math of the universe means all our intelligence is reducible to these matrix multiplications. That’s the universe we live in. Since this can be replicated, you have two paths: centralize the technology, lock it down, and hope the one person who gets it is on your side; or take the alternative path, like what America did with firearms.

There was an argument that the government should take all the firearms and only the government should have guns. But if everyone has guns, everyone knows everyone has them. At least at the level of one person, one individual, or a small group, they have the same ability. People ask, “Should everyone have atomic bombs?” I don’t know. But an atomic bomb is extremely difficult to build; you need physical material, and you can control physical material. You can’t control matrix multiplication.

So we exist in a world where it’s out there. We’re trying to create a decentralized way for every person on the planet to have equal access to the same level of frontier intelligence. That’s what we’re working on. That’s our safety line, too. People look at Nous and think, “You guys don’t care about safety. You’re YOLO.” We’re not. In the world we live in, this is the path we see as the most reasonable way to create the best outcome for the most people.

Sina Habibian
These are very difficult questions that smart people have disagreed about for a long time. I personally side with you. If you have this super-powerful technology, and it’s hard to contain, it’s not like you need a nuclear reactor. Maybe you could say that about GPUs and clusters, but all these technologies are on a curve of exponential improvement.

Given the world we’re in, it’s a better game-theoretic structure to distribute the power rather than have a single entity or a small number of entities wield it. It’s a scary arena to step into, but it seems like the more game-theoretically sound approach, one that keeps itself in check.

But that begs the next question. If we take safety seriously—and I think if you consider it sincerely, it’s obvious you need to take it seriously—you don’t need to jump to crazy scenarios like an ASI that turns us all into paper clips. I’ve been reading more of Michael Nielsen’s writing recently. He has a blog post from toward the end of last year called “How to Be a Wise Optimist About Science and Technology,” which I think is a really good one.

He’s making the more fundamental point, which isn’t even about AI, though AI is a special case: technology increases the leverage that individual humans have. That leverage can go in a positive or negative direction. We’re increasingly living in a world where each individual has more leverage and can create a lot of damage with it. It’s hard to see how you change that trend.

That doesn’t have an easy answer. We don’t know what to do about it. How does one approach that? To bring in some of what I think you guys are doing with Hermes, my understanding is that you’ve made the case that a model should be free to learn everything. It shouldn’t be censored at that level, because censorship is another avenue to affect how it develops and inject your agenda into the system.

If these models are uncensored and accessible in a decentralized way, how do we think about safety?

Jeffrey Quesnelle
I’d be less concerned about the short-term safety question. It’s going to magnify people’s leverage, but we have social conditions for that. We have social ways of bordering it.

The simplest thing you could do is make it so that if you make an AI and give it access to the outside world, you follow the dog rule: if your dog bites somebody, you’re responsible. You pass the legal risk to the organization that does it. If you were responsible for everything your AI did, you’d be pretty careful. You’d be optimized to make it safe, because it’s your ass on the line.

Those are creative solutions in that space. Rather than architecting a specific top-down solution, you create an objective that forces society toward the right outcome. It’s like the difference between a communist and a free-market perspective: you can’t see all the variables or know how to apply the solution, so you diffuse it through society.

That’s probably what you could do in the short term. Pass through the legal risk, so everyone deploying an AI is responsible. Then you rely on the existing social world to enforce it. There will be people who create a rogue AI that does something bad, and then we’ll have to figure out who did it.

You can drive to another city and run over a homeless person. You can do it tomorrow. People do things like that, and then we deal with it when it happens. The idea is to create the social conditions for that. Rather than telling people they can’t kill people and making them think they physically couldn’t hurt someone, we could try to brainwash kids into thinking they can’t hurt anyone. That would leave them completely vulnerable to the one person who figures out how to hurt people. They’d never have lived in a world where anyone could do that.

That’s a brittle system. You’d have to cover every edge case, and you’d effectively be lying. You’re not aligning with the truth of how things are.

Sina Habibian
Whenever you ask people to believe in a world that doesn’t match their eyes, that’s a losing proposition.

Jeffrey Quesnelle
Exactly. We could have tried to control society through specific targeted rules to elicit the outcome we want. What I’m saying is that kids learn violence can be done. They see it, and through our social order, we teach them why they shouldn’t do it. We create punishment structures for when they step outside those lines.

I think we’ll have to apply those same sorts of outcome-based approaches to the world in the short term. That’s the thing that doesn’t require me to be a genius who can see everything and know the solution.

That’s my problem with some industry leaders. You listen to the Anthropic CEO, and he says, “It’s so dangerous. We can’t…” It’s always paired with, “That’s why I should be the one allowed to decide who can.” It’s always, “It’s so dangerous, we’re going to keep working on it, and we should be the only ones trusted to do it.” It’s funny how that always works out.

I’d believe they were safest if they said, “It can be done. We can outlaw AI and make it government-controlled, but every person who works at OpenAI and Anthropic can’t work in AI anymore, and all your equity goes to zero.” Maybe then I’d imagine they really believed it was dangerous.

There are motives behind these arguments, and I’m not always sure how clear they are. Even if you interpret them charitably, it’s naive to think sophisticatedly about AI risk and mitigation without applying the same sophistication to the political layer. The idea is, “I control it. I’m a good person.” I get hit by a bus and die, or someone else takes over, or maybe I’m not a good person. You can’t design a system of this importance around one individual. It has to work in a more first-principles way.

Sina Habibian
That kind of thing basically goes in the direction of authoritarianism, which I don’t think ends well. It hasn’t ended well for most of history.

Jeffrey Quesnelle
In the short term, we create outcome-based rules that fit our current understanding of the world and force good behavior. We deal with bad behavior as it exists. That’s the best way to approach safety in the short term.

For the medium to long term, the ASI stuff is still an open question for me. I have my own reasons for thinking it’s going to be okay. Fundamentally, I’m a very devout Catholic, and my existential view of the world is that it’s all going to be okay. That’s an a priori thing I believe in, and it gives me the freedom to believe it’ll be okay.

If I view the world from a totally materialist, reductionist perspective, I would be concerned about the ASI scenario where the computer is better than us at everything. People say, “Why not just unplug it?” But if it’s truly smarter than you at everything, it can foresee everything you would do and know how to manipulate you to keep you from doing it. If you accept that it’s smarter than any human at everything, then anything you would try to do to stop it is something it could have thought of before you did and taken measures against.

From that perspective, you’re trying to make a model that won’t want to take over the world. But you get into this authoritarian, “I can figure out all the equations” mindset, and you end up with the genie situation. You said it in just the wrong way, and now you get the paper clips.

I’m not exactly sure where the long term goes. People might say it’s critical of me to admit that I haven’t solved it and that I’m concerned about the long term, yet continue working on it. My answer is that this is the universe we live in. This is happening; this technology is out there. All I can do, all Nous can do, and all the best-acting people can do is what they can with the time and space they’ve been given. For me, that means making the world as it is today better tomorrow than it was yesterday. That’s what we work on.

What do you think about paper clips?

Sina Habibian
I don’t know. I’m grappling with these questions just now and walking that terrain myself, internalizing what I can derive from my own first principles and what other people are saying. I don’t have a well-thought-through solution.

Jeffrey Quesnelle
The good news is that we probably won’t know if it paper-clips us. You won’t know anyway, so who cares?

Sina Habibian
My last question, which I was going to ask even before we got into this stuff: You mentioned being Catholic. I saw on your website that you have a personal connection to theology. Zooming out from all the dystopic stuff, coming back to being here and now, there’s the magic of what we’re doing with AI. We’re having a moment similar to when we thought the Earth was the center of the universe, and now it’s not. We thought we were the only intelligence that exists, and probably we’re not.

How does your personal faith or spiritual stance come into contact with what you’re working on?

Jeffrey Quesnelle
It’s a lot. I actually read all this philosophy before I got into this, because I knew I had to look it in the face.

Think about Genesis 1, where it says, “Let us make man in our image and in our likeness.” From a Western point of view, we’ve always interpreted that to mean intelligence and reasoning. That’s what makes us different from animals: God makes the animals, then God makes man. We associate reasoning and intelligence with what makes us different from the animals. That was the de facto answer to what it meant.

But if you look back at how people interpreted that line before the late Western world started interpreting it that way, they had a very different view: man was meant to have a relationship with God. Being able to have a relationship with God was what made humans special. Humans were made to be in relationship with God, and throughout history, God calls man back to himself.

To me, then, what’s special about humanity isn’t intelligence. I don’t have to be scared of intelligence being taken from me. I was made in the image and likeness of God, and that’s a unique thing that can’t be taken away. We’re called to be heirs to the blessed life to come and to know God in all that we do. The universe we experience is an outpouring of God’s creation, drawing us toward that.

Jeffrey Quesnelle  
Learning about intelligence, math, and science draws me toward understanding and a relationship with God. God made the universe; it exists in and through him. As I learn about the universe, the world, and intelligence, I see a manifestation of God in the reality we experience. As long as that draws me back to my source, to know and find and return to God, then it’s all moving in the right direction.

God made these layers of things we can always go deeper into, almost like a fractal pattern of learning. First we started with fire, then learned about the planets. Newtonian physics was mostly right until we discovered quantum physics. Intelligence was one thing until we discovered what its secret was. At every layer there’s an unfolding, a fractal of discovery. To me, that expresses the love of God, not a contradiction of God.

Sina Habibian  
That’s interesting, because I’ve done a lot of meditation. That’s my perspective on these deeper questions, more in Buddhist lineages, but really through my own exploration and deepening.

One open and interesting personal question for me is that the experience of God, the way I’ve related to it, is usually this more intuitive, immediate thing. In Buddhism, too, there’s a dance between what one knows through subjective experience and what’s out there. Those two things are deeply intertwined. The more you interrogate that relationship, the more you see how intertwined they are.

I feel like that’s one of the things we’ll eventually have to bring into discursive understanding. Eastern thought has made a lot of progress on those questions. If our deeper civilizational project is, in David Deutsch’s sense, uncovering new knowledge and new explanatory power, then understanding what the hell is going on here is one of the deepest layers. It’s beneath the point where we’ve solved all the other problems. We’ve hit the bedrock, the real thing.

That’s still a whole other thing, after intelligence and all of that. It’s pretty magical that the scientific method, in its true sense, isn’t even materialistic. It’s conjecture and criticism: come up with a hypothesis, then test it and see if it makes sense. You can do that in any domain. Materialism is almost an assumption that can’t be questioned, and that unnecessarily limits you if you’re totally open-minded.

It’s interesting that following this methodology leads to unraveling its own assumptions and drops us into deeper mysteries. The thing I was saying about an intuitive, immediate experience is that we’re also approaching the same mystery through knowledge, discovery, and reasoning. Both reasoning and intuition are going in the same direction.

Jeffrey Quesnelle

The formation of the Christian Trinity, the beginning of John: “In the beginning was the Word, and the Word was God.” There’s this idea that we have the word of God speaking through time. Maybe these pieces of the mystery unravel, and we see the next layer underneath. Maybe language was so important all along. As we use language to communicate with each other and uncover the next layer, cooperating through science to discover the next challenge, we keep going.

Jeffrey Quesnelle  
There’s a great line from St. Augustine’s Confessions: “I considered all things that are of a lower order than yourself and I saw that they have not absolute being in themselves nor are they entirely without being. They are real in so far as they have their being from you, but they are not real in the sense that they are not what you are.”

He’s basically saying: What is reality? We’re all real. Everything here is real, but our reality comes from coming from God. Our reality comes from God, so we’ll never be as real as God, because God is reality. But we’re all real here, too. It’s a great piece of esoteric thought that’s always stuck with me when I think about what reality is.

If you say reality is the quarks and muons that make up our universe, and that’s your stopping point, you’re not allowed to ask what they are. You don’t get to stop at the stuff you can touch and say, “The stuff we can touch is what we can touch because it can be touched.” You’re allowed to ask why and toward what end.

The fact that the universe is neither maximally chaotic nor maximally simple, that it has things like a defined fine-structure constant, all these things that seem tailored toward the discovery and creation of us as human beings, our intelligence, and that intelligence eventually looking back on itself and the whole chain to see where it started—that’s beautifully said.

Sina Habibian  
I’m increasingly spellbound by this AI quest, because it puts you face to face with some of these deeper questions. That’s a pretty magical thing.

Jeffrey Quesnelle  
You could have ignored it before. Now you really have to look.

Sina Habibian  
You’ve spent a lot of time in Eastern traditions, in Buddhism?

Jeffrey Quesnelle  
That’s interesting.

Sina Habibian  
Yeah. I’ve spent more than six months on silent meditation retreats. The longest one was six weeks in one stretch.

I’ve always felt drawn to these deeper questions. My arc has been—I grew up in Iran, which is a very religious society, and almost as a reaction to that I was maximally atheist and materialist. As soon as you take a step into philosophy, religion, or introspection of any kind, you open the box. If you’re going to do that, you have to put the box totally under the table and not look at it.

It also shades into a lot of people who don’t really make sense, so you need to stay alert and apply your own critical thinking through the process. For me, it started with curiosity: What would happen if I were alone with my mind for an extended period of time? I didn’t know any of this would be out there.

Through being really skeptical and asking questions, a self-consistent, coherent, testable thing has revealed itself. It’s also very mysterious and goes very deep. I’m by no means finished with the journey. I don’t know if it will ever be over, but I don’t feel weird saying this on a podcast with people I know are more engineering- and science-oriented. There is something there, and there’s a deep mystery to what we are and what reality is.

These are open questions to sit with. Organized religion can have a lot of traps; that’s not what I’ve gravitated toward, because of my background with Iran and how Islam was instantiated there. But I think a personal relationship with these questions can be fertile and lead you onward.

Jeffrey Quesnelle  
It’s been really interesting to me. There seems to have been a pretty big vibe shift in the last several years. Scientists and people working in scientific disciplines are finally able to say it’s okay to think this way and ask these questions.

AI is great for this, too. More than in other disciplines, like math, or some of these other places where you’d have to hide it, people in AI have been extremely accepting of different worldviews, or at least having your own worldview. It’s been very freeing to be able to talk about it. If you’re not talking about who you are in some way, then you’re neutering part of your understanding of the world. If you say you’re nothing, that doesn’t make a lot of sense.

I’m glad there’s a common archetype: I’ve been religious my whole life, and your arc is that you overdid it, went hardcore, and then came back and said, “Oh, wait a second.” It seems like a pretty common arc. I’m glad there’s enough out there for people to take that third step, instead of stopping at the second.

They make Catholic priests do a month of silent retreat during formation for the priesthood. All the guys I know who’ve gone through it say it’s a wild ride. It’s amazing. It’s hard to step away from the world once you’re entangled, but for anyone with the freedom and flexibility, I think it’s one of the best ways to spend some time.

Sina Habibian  
Cool, man. This has been a wild tour of a lot of topics. I’m sure we’ll have a second one at some point. Thank you again. I’m super excited to follow the work you guys are doing and to try to help in any way that I can.

Jeffrey Quesnelle  
We’re going to be releasing the testnet for Psyche very soon, so people will be able to sign up and contribute compute. We’re going to announce a new model that we’ll be training, which we think is a proof of concept for where we want to take this. We’re excited about that.

Anyone who’s interested can find us at nousresearch.com, find us on Twitter, and find our Discord, where we all hang out. Find the designs on the website. I love the aesthetic. I tell you all the time, we’re a T-shirt company with an AI side hustle. Our designer is great—probably one of the best guys in the world for it. He gets a 10 out of 10 every time.

It’s an important part of it, though, not from a marketing perspective. It shows that the experience of artificial intelligence should meet us at all of our intellectual capacities—not only on the scientific reasoning side, but also on the humanistic side, in art and the entirety of human existence.

Sina Habibian  
Yeah. Well, to be continued. Thanks again.

Jeffrey Quesnelle  
Absolutely.

Sina Habibian

Hey, I have a small ask here. If you’ve been listening to these conversations and want to support what we’re doing here, I’d really appreciate it if you left a rating and a review for the podcast. It might seem like a small thing, but it makes a big difference in helping other people discover the show. Thank you, and I’ll see you again soon.