[BidClub_]
Machine Learning Street Talk · · 73 min

He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow]

Llion JonesLuke Darlow

YouTube ↗
TL;DR

  • Llion Jones believes Transformers have become an “oversaturated” local minimum and has drastically reduced his own research on them. His warning comes from the RNN era: years of publishable gains from 1.26 to 1.24 bits per character were abruptly eclipsed when a deep decoder-only Transformer reached roughly 1.1. “There’s some breakthrough that will occur at some point,” he argues, potentially rendering today’s endless architectural tweaks similarly redundant.

  • A successor architecture must be “obviously crushingly better,” because Transformer infrastructure now creates enormous switching costs. Researchers already understand how to train, fine-tune, inspect, and serve Transformers, while scaling an incumbent model 10× can overwhelm a modest architectural improvement. For investors, the implication is that benchmark edge alone is not disruption; a challenger must offset an entire installed base of software, skills, and compute practices.

  • Sakana AI’s Continuous Thought Machine combines sequential internal computation, neuron-level models, and synchronization-based representations. Developed over roughly eight months and awarded a NeurIPS 2025 spotlight, the CTM treats a thought as something unfolding through time rather than a single activation state. Hewitt’s maze example makes the distinction concrete: predicting a 100- or 200-step path in one shot failed, while learning to trace it sequentially became tractable.

  • Adaptive compute emerges naturally in the CTM instead of being bolted on with a hand-balanced computation penalty. In a 50-step ImageNet setup, training uses both the point of lowest loss and the point of greatest certainty; easy images resolve in one or two steps, while harder ones consume more time. The model also emerged “nearly perfectly calibrated,” which Hewitt calls a “smoking gun” rather than definitive proof that the architecture is better.

  • The most compelling CTM evidence is behavioral: constraints cause it to discover different algorithms. During maze training it sometimes follows a path, recognizes an error, backtracks, and tries another; under a tight thinking budget, it jumps ahead, traces sections backward, and leapfrogs toward a solution. The research case is not yet that CTMs replace Transformers, but that internal sequential search produces richer behavior than a fixed-depth, one-shot computation.

  • Jones sees research freedom as both Sakana’s operating model and a scarce strategic asset. Product demands, publishing pressure, and investor expectations progressively narrow autonomy, encouraging safe positional-embedding papers over ideas that “might be too weird.” His counter-bet is open-ended exploration—including evolutionary searches whose largest runs may only be in the tens of thousands despite hundreds of millions spent elsewhere—while acknowledging that commercial success could eventually create the same “technology capture.”

  • SudokuBench suggests headline progress in reasoning still breaks on genuinely novel, compositional tasks. The best models score around 15% on handcrafted variant Sudokus, and GPT-5 improves but remains unable to solve puzzles humans can handle; models revert to “I’ll try five, I’ll try six, I’ll try seven” instead of finding each puzzle’s unique break-in. Thousands of hours of Cracking the Cryptic thought traces offer training data, yet Jones says present RL cannot reliably sample the rare reasoning trajectory required.

Digest · the substance, structured for research

1. Transformer success has captured the research agenda

  • Jones’s personal call is unusually direct for a co-inventor: earlier in the year he decided to “drastically reduce” his Transformer research. The space is oversaturated—not exhausted—and he wants to use his unusual freedom to turn up exploration of architectures that are not already consuming most of the field’s attention.

  • The host frames Sakana through Kenneth Stanley’s Why Greatness Cannot Be Planned: discovery comes from letting researchers follow their “gradient of interest,” while committees and fixed objectives converge toward “gray goo.” Jones says the philosophy is discussed internally, and one of his central jobs as co-founder is protecting that autonomy as the company grows.

  • Jones’s organizational diagnosis is causal: more talent and capital intensify competition; investors eventually request returns; companies need products; pressure to publish or commercialize rises; autonomy falls. He still tells new hires, “I want you to work on what you think is interesting and important,” and says he means it.

  • His neglected-scale example is evolutionary search. The field has spent “hundreds of millions of dollars,” while the largest evolution-based searches are probably only in the tens of thousands; he expects something interesting when somebody truly scales them. Pitching that direction inside an environment going all-in on one technology produced “zero interest,” helping motivate his own company.

2. The RNN era shows how quickly optimization work can become obsolete

  • Before Transformers, recurrent networks produced the same feeling of architectural finality. Character-level language-model papers rearranged LSTM or GRU gates, changed initialization, or added hierarchical computation; improvements from 1.26 to 1.25 to 1.24 bits per character were small but legitimately publishable and exciting.

  • Then Jones’s team applied very deep, decoder-only Transformers to language modeling and immediately reached roughly 1.1 bits per character. Colleagues came to their desks to suggest a calculation error—perhaps the result was in nats—but “it really is the correct number.” Excellent lines of RNN research suddenly looked redundant.

  • The parallel he draws is deliberately uncomfortable: today’s normalization placements, training recipes, and other Transformer permutations may be another elaborate optimization of an architecture awaiting discontinuous replacement. “I personally don’t think we’re done,” he says; continued scaling is not, in his view, proof that this is the final architecture.

  • The host worries that foundation models are causing architectural skills to die out as data scientists and ML engineers become prompt-focused AI engineers. Jones pushes back: “We have plenty of very talented, very creative researchers out there, but they’re not using their talents.” Incentives make a safe positional-embedding paper more rational than a strange idea that may fail or prove difficult to publish.

3. Incremental superiority cannot overcome the Transformer installed base

  • Jones says research already contains architectures that outperform Transformers—but not by enough. The incumbent comes with accumulated knowledge, fine-tuning methods, inference systems, knowledge of how its internals work, and mature training software; therefore “being better is not good enough. It has to be obviously crushingly better.”

  • Transformers cleared that threshold against RNNs through much faster training and sharply higher accuracy. Deep learning similarly overcame symbolic-AI skepticism by becoming impossible to ignore. A new design now faces a gravitational pull: a clever accuracy gain can disappear when an established lab simply makes its Transformer “10 times bigger.”

  • The symptom Jones finds hardest to dismiss is “jagged intelligence”: a model solves a PhD-level problem and then says something obviously wrong in the next sentence. Current networks may be universal approximators that can be forced to do almost anything with sufficient data, compute, and patience, “but I don’t necessarily think that they want to.”

  • His poster-child analogy is a matrix-exponentiation paper’s spiral classifier. ReLU and tanh networks technically separate every point using fragmented boundaries; the alternative represents the spiral as a spiral and extrapolates its continuation. Scarfe extends the point to video-generation models: fixing hands from the wrong finger count to five may reflect more brute force, not a representation that understands a hand.

4. CTMs represent thought through time rather than a static hidden state

  • Hewitt says the Continuous Thought Machine took roughly eight months—long by current AI-paper standards—and rests on three ideas: an internal sequential thought dimension, neuron-level models, and synchronization as the operative representation. He stresses that it is not wildly outside the local minimum; it is a simple, nature-inspired departure developed carefully enough to earn a NeurIPS 2025 spotlight.

  • Mazes became the “hello world” because conventional deep learning can output an entire solution mask in parallel, whereas a human-like formulation requires a sequence: go up, right, up, left. Once the model must emit that route from start to finish, an apparently trivial vision task becomes substantially harder.

  • Each CTM neuron is itself a small model. Rather than mapping one pre-activation directly to one output, it consumes a finite history of that neuron’s activations and generates the next post-activation, letting dynamics accumulate at the level of individual units.

  • The representation then measures how pairs of neurons synchronize across their activation histories, using a dot product rather than reading only the recurrent state at one instant. Each pair can use an exponential decay at a different timescale, from immediate co-firing to long-run coordination, enriching the order-(d^2) relational space. Hewitt’s premise is that “the concept of a thought is something that exists over time.”

5. Adaptive computation falls out of the training objective

  • In the maze setup, the CTM continually sees the full image and retrieves information through attention while receiving, illustratively, about 100 internal thinking steps. Because neither it nor other tested models could predict 100 or 200 route steps in one shot, the team built an autocurriculum that trains just beyond current competence—one step conceptually, five in practice.

  • For ImageNet, the model runs for 50 steps and training identifies two indices from 0 through 49: where loss is lowest and where confidence is highest. Cross-entropy is averaged at those points. The induced behavior is clean: easy examples finish in one or two steps, while difficult ones naturally use more available thinking time.

  • Hewitt contrasts that result with Alex Graves’s Adaptive Computation Time work, which required a massive hyperparameter sweep and a loss penalty on computation. Tommy Leung explains why: neural networks are “greedy,” so using all available compute generally lowers loss. In the CTM, no explicit objective demanded adaptive time; “adaptive computation times seem to just fall out naturally.”

  • Synchronization also appears to help gradients propagate through long recurrent runs, touching neurons across many time steps; Hewitt says the model “just kind of worked” across broad hyperparameter settings. After training, it was also nearly perfectly calibrated—its confidence tracked empirical correctness—though he presents this as suggestive, not conclusive.

6. Internal sequential search opens a different scaling dimension

  • Hewitt treats chain-of-thought reasoning as, among other things, a new dimension for adding compute. CTMs try to internalize that sequential process rather than express every intermediate step as language, using synchronization across multiple temporal hierarchies to search a high-dimensional latent space.

  • His proposed advantage for ARC-like tasks is the ability to project one step, 100 steps, or 200 steps ahead while breaking a sparse problem into a process. He does not claim the architecture has solved ARC; the narrower call is that this representation may offer flexibility missing from fixed-depth Transformers and deserves direct experimentation.

  • Asked about neural Turing machines, Hewitt explicitly declines the strongest claim: “I wouldn’t go so far as to say that the Continuous Thought Machine is definitively” Turing-complete. The affinity is latent reasoning unfolding over time, without requiring the same difficult discrete read/write operations associated with external neural memory.

  • The architecture also changes how difficulty is represented. A ViT or CNN must nest the reasoning for an obvious cat and an obscure, underrepresented class into the same fixed stack; a CTM can stop at different times, naturally segmenting examples from easy to hard. That architectural curriculum resembles how humans learn without requiring a manually partitioned dataset.

7. Maze behavior shows compute constraints can invent new algorithms

  • During intermediate training, the team watched the CTM begin down one maze path, appear to recognize “oh no, damn, I’m wrong,” backtrack, and choose another. Later it became more efficient, using multiple attention heads to refine solutions in a more distributed way. The camera-ready supplement records roughly 14 such unexpected observations.

  • Tightening the thought budget produced the strangest specimen. Instead of tracing a long maze continuously, the model jumped approximately forward, filled a segment backward, leapfrogged ahead again, and repeated. “Turns out that there’s a faster algorithm to do it,” Hewitt says, though he remains unsure what the behavior means beyond demonstrating that constraints alter the learned algorithm.

  • Language may be another ambiguous maze: many trajectories can reach viable outputs, and a system must recognize, revise, or backtrack from a bad route. Hewitt is actively exploring language modeling, but keeps the claim conditional; adding ambiguous maze solutions is itself still an experiment he had not yet run.

8. Memory and AI scientists could turn solitary search into collective search

  • Hewitt describes a possible memory experiment that would restrict an agent to perhaps a 5×5 local view of a maze, forcing it to save and retrieve enough history to revisit a location and remember, “I did the wrong thing last time.” Multiple agents could share that store as an almost “cultural memory,” collectively solving a global task from limited individual views.

  • Jones expects models to become increasingly powerful research partners. Sakana’s AI Scientist demonstrated an end-to-end pipeline—from a seeded idea through code, experiments, results, and paper—and produced a “100% AI-generated paper” accepted to a workshop. He presents that as proof of possibility, not his preferred working arrangement.

  • His near-term model remains interactive: seed an idea, debate alternatives, inspect generated code, and discuss results as they arrive. The analogy is an intern: no sufficiently rich research vision can be fully transmitted once and then left unattended for four months; steering communicates tacit intent and path-dependent understanding.

  • Jones concedes that this collaboration may eventually become harmful. Human-plus-engine once surpassed a chess engine alone, but “adding a human into the mix actually makes the bots worse” now. If AI scientists reach the point where his input is detrimental, whether humans should step aside becomes a broader question rather than a technical workflow choice.

9. SudokuBench exposes the gap between fluent reasoning and novel breakthroughs

  • SudokuBench is built from handcrafted variant Sudokus: standard row, column, and box constraints plus unique natural-language rules. One puzzle says a number in its rule description is wrong; another overlays a maze in which a rat’s route to cheese must obey numerical constraints. Solving therefore requires interpreting and sometimes meta-reasoning about the specification.

  • Best current models score around 15%, mostly on the smallest and simplest puzzles. Jones says GPT-5 delivers a jump but still cannot solve puzzles that humans can. Every puzzle contains a distinctive “break-in,” while models typically retreat to dull enumeration: “I’ll try five, I’ll try six, I’ll try seven.”

  • The training asset comes from Cracking the Cryptic, whose professional solvers verbalize their reasoning through videos sometimes lasting four hours. With permission, Sakana extracted thousands of hours of detailed thought traces and made them available for imitation learning—an approximation of the hidden reasoning Andrej Karpathy wished could accompany humanity’s finished text.

  • Expert humans possess many “reasoning Lego blocks,” recognize which motif a rule suggests, test it, discover insufficient disambiguation, and backtrack. Present models do not show comparable meta-reasoning on the benchmark, and current RL fails because the necessary breakthrough is too rare to sample reliably. Jones argues that meaningful progress on SudokuBench would represent meaningful progress in AI reasoning generally.

Llion Jones

Despite the fact that I was involved in inventing the Transformer, luckily no one has been working on them as long as I have, right, with maybe the exception of the other 7 authors. So I actually made the decision earlier this year that I'm going to drastically reduce the amount of research that I'm doing specifically on the Transformer because I have the feeling that it's an oversaturated space. It's not that there's no more interesting things to be done with them. I'm going to make use of the opportunity to do something different, right? To actually turn up the amount of exploration that I'm doing in my research.

We just released the Continuous Thought Machines. It's a spotlight at NeurIPS 2025 this year. You should care about it because it has native adaptive compute. It's a new way of building a recurrent model that uses higher-level concepts for neurons and synchronization as a representation that lets us solve problems in ways that seem more human by being biologically and nature-inspired.

The atmosphere in AI research was actually quite different back during the Transformer years because it doesn't feel like something similar could actually happen right now, because of the reduced amount of freedom that we have, right? The Transformer was very bottom-up, right? It's not that somebody had this grand plan that came down from on high that this is what we should be working on. It was a bunch of people talking over lunch, thinking about what the current problems are and how to solve them, and having the freedom to have literally months to dedicate to just trying this idea and having this new architecture fall out.

We've spent hundreds of millions of dollars. The biggest sort of evolution-based search is probably in the tens of thousands. We have all this compute. What happens if you scale up these search algorithms? I'm sure you'll find something interesting when someone eventually does bite that bullet and really scale up these evolutionary sort of artificial-life experiments, because I pitched it in an environment where people were just going all in on this one technology. I got zero interest. So now I have my own company and I can pursue those directions.

This podcast is supported by Cyber Fund.

Omar

Hey folks, I'm Omar, product and design lead at Google DeepMind. We just launched a revamped vibe coding experience in AI Studio that lets you mix and match AI capabilities to turn your ideas into reality faster than ever. Just describe your app and Gemini will automatically wire up the right models and APIs for you. And if you need a spark, hit I'm feeling lucky and we'll help you get started. Head to a.studio/build studio/build to create your first app.

Two for AI Labs is a research lab based in Zurich. They've got a team of amazing ML engineers and research scientists. They're doing some really cool stuff. If you look at their website, for example, you can see what their approach was for winning the ARC AGI 3 public competition, which closed out a few months ago, and they are hiring amazing ML engineers and research scientists. They also care deeply about AI safety. So if any of that is a fit for you, please go to Two for AI Labs and give it a go.

Tim Scarfe

The audience will know I'm a huge fan of Kenneth Stanley's ideas. His book, Why Greatness Cannot Be Planned, changed my life. It was absolutely insane. What he was speaking to is that we need to allow people to follow their own gradient of interest, unfettered by objectives and committees and so on.

That's how we do epistemic foraging: when you have too many agendas involved in the mix, you kind of end up with gray goo and you don't discover interesting novelty and diversity. I suppose that's basically the thesis of your company, Sakana, to lean into those ideas.

Llion Jones

Yes, exactly. At the company, we're massive fans of that book. We're hoping to have him come and talk at our company next week, actually. It's a philosophy that we do talk about internally, right? We have copies of the book, including the recent Japanese translation.

As you know, as one of the co-founders, one of my main jobs—one of the main things that I have to keep doing for this company—is making sure that we protect the freedom that the researchers currently have, right? It's a privilege, really, that we have the resources to be able to do that. Inevitably, as I've seen happen, as the company grows, more and more pressure comes in and it narrows the freedom. But because we believe in this philosophy so strongly, I'm hoping that we can give people all the research freedom that we do now for as long as possible.

Tim Scarfe

What are those processes that curtail freedom as a company matures? I mean, how would you describe that?

Llion Jones

It's great that there's never been so much interest, people, talent, resources, and money in the industry, but unfortunately that just increases the amount of pressure people have to compete with all the other people working on it and trying to get the value out of this technology and make money. I think that's what just happens, right? As a startup, you have a feeling of excitement and trying something new. Right at the beginning, you have a bit of a runway, so you have the freedom to try different things.

But inevitably, people start to ask for returns on their investments or they're expecting you to churn out some product. This just unfortunately reduces the creativity that researchers have because the pressure to publish, or the pressure to create technology that's actually useful for the products that we have, goes up. So the feeling of autonomy, I think, starts to go down.

But I literally tell people when they start working for the company, “I want you to work on what you think is interesting and important,” and I mean it. On YouTube, there's a phenomenon called audience capture.

Tim Scarfe

Right. I think there might be a phenomenon called technology capture, which is that in the early days of Google, it was quite open-ended. I mean, the Transformer is now the ubiquitous backbone of all AI technology, and it's a huge achievement that you're involved in.

But there's a similar story with OpenAI. They're now starting to see all of these commercialization opportunities. They can—I mean, they're going to become LinkedIn. They're going to become an application platform. They're going to become a search platform. They're going to become a social network.

I guess this could happen to you guys. There's a very strong chance, especially with your new paper that we're going to talk about today, these Continuous Thought Machines. It could be a revolutionary technology, but then it will become obvious how it could be commercialized. That's how those pressures come in.

Llion Jones

I like the audience-capture analogy. I think there's definitely been some kind of capture by large language models, right? They worked so well that everyone wanted to work on them. I'm really worried that we're kind of stuck in this local minimum now, right? We need to try to escape it.

So we spoke about the Transformers, but there's a time just before the Transformers that I'd like to talk about because I think it's quite illustrative. Of course, the main technology before Transformers was recurrent neural networks, right? There was a similar feeling, right? When recurrent neural networks came in and we discovered this new sort of sequence-to-sequence learning, that was also a massive breakthrough, right? The translation quality went up massively, right? Voice-recognition quality went up massively.

There was a similar sort of feeling then, like, okay, yes, we've found the technology and we just need to perfect this technology. Back then, even my favorite task was character-level language modeling. Every time a new RNN-based character-level language-modeling paper came out, I got quite excited. I would want to quickly read the paper: How did they get the improvements? But the papers were always just these slight modifications on the same architecture, right?

It was LSTMs and GRUs, and maybe initializing it with the identity matrix so that you could use the ReLU function, or maybe if you put the gate in a different place, or if you layered them in a slightly different way, or if you had gating going upwards as well as sideways. One of my favorites was this hierarchical LSTM, where it would actually decide to compute or not compute the different layers.

If you trained on Wikipedia and looked at the structure of when it was decided to compute or not compute, it kind of looked like the structure of the sentences was actually being picked up by the model. I used to love that sort of stuff, right? But the improvements were always like 1.26 bits per character, 1.25 bits per character, 1.24. That was a result that was publishable, right? That was exciting.

But then after the Transformer, the team I joined afterward applied, for the first time, very deep Transformer models—decoder-only Transformer models—to language modeling, and we immediately got something like 1.1, right? Something that was so good that people actually came to our desk and politely told us, “I think you made a calculation error. Do you think it's nats, not bits per character?”

And we're like, “No, no, no, it really is the correct number.”

Tim Scarfe

What struck me later is that, all of a sudden, all of that research—and, to be clear, very good research—was made completely redundant.

Llion Jones

Yes.

Tim Scarfe

Right. All of those endless permutations of RNNs were suddenly seemingly a waste of time. We're kind of in the situation right now where a lot of the papers are just taking the same architecture and making endless amounts of different tweaks, like where to put the normalization layer and slightly different ways of training them. We might be wasting our time in exactly the same way, right? I personally don't think we're done. I don't think this is the final architecture and that we just need to keep scaling up. There's some breakthrough that will occur at some point, and then it will once again become obvious that we're wasting a lot of time right now.

Llion Jones

Yeah. So, we are a victim of our own success and this basin of attraction. There are so many basins of attraction. Sarah Hooker spoke about the hardware lottery, and this is a kind of architecture lottery.

It actually made me think of the agricultural revolution, which is that this kind of phase change happened, and all of the folks who had these skills that were so necessary—these diverse skills for living and surviving—they died out. That's actually quite paradoxical, because we need those skills to take the next step.

Tim Scarfe

And so, we're now in this regime where we've got the term “foundation model,” and the implication is that you can do anything with a foundation model. In the corporate world, we used to have data scientists—you know, ML engineers doing these architectural tweaks, even in midsize enterprises—and now we just have AI engineers who are doing prompt engineering and so on.

So, you're saying that the fundamental skills we need in order to be diverse, think of new solutions, and develop new architectures are dying out. I think I'm going to disagree with that. I think we have plenty of very talented, very creative researchers out there, but they're not using their talents.

For example, if you're in academia, there's pressure to publish, right? If there's pressure to publish, you think to yourself, “Okay, well, I have this really cool idea, but it might not work. It might be too weird. It might be difficult to get it accepted because I have to sell the idea more. Or I can just try this new positional embedding.”

The problem is that the current environment, both in academia and in companies, is not actually giving people the freedom they need to do the research they probably want to do. There's also this interesting thing that, even in spite of great new research—I was speaking to Sebastian Hoyer, and he's got all of these new architectural ideas—OpenAI aren't implementing them. Google are doing this diffusion language model, which is quite cool, and I'd like to know your opinion on why that is.

There are a few philosophies floating around, like this concept of a universal representation: that there are universal patterns, and the transformer representations resemble those in the brain. That has led to the idea that we don't need to use different architectures, because if we just have more scale and more compute, all roads lead to Rome. Why would we bother doing it any differently?

Llion Jones

There are actually better architectures, right? There are architectures that have already been shown in research to work better than transformers. But they're not better enough to move the entire industry away from such an established architecture, where you're familiar with it, you know how to train it, you know how it works, and you know how the internals work. You know how to fine-tune them. All the software is already set up for training transformers, fine-tuning transformers, and inference.

If you want to move the industry away from that, being better is not good enough. It has to be obviously, crushingly better. Transformers were that much better than RNNs. When you applied them to a new problem, they were just so much faster to train, and you got such higher accuracy that you had to move.

I think the deep learning revolution was also another example of that, right? You had plenty of skeptics, and people were pushing neural networks even back then. People were saying, “No, we think symbolic stuff will work better.” But then they demonstrated that neural networks were so much better that you couldn't ignore it.

This fact makes finding the next thing even harder, right? That's the gravitational pull, always pulling you back to, “Oh, okay, but a transformer is good enough.” You made a cool little architecture over here that looks like it's got better accuracy, but OpenAI over here just made it 10 times bigger and it beats that. So, let's just keep going.

Tim Scarfe

May I also submit that there could be an additional reason? I love that “Fractured Entangled Representations” paper. There's this shortcut-learning problem, and I think there's a bit of a mirage going on here. There might be problems with these language models that we're not fully aware of.

There's also this thing where we're starting to bastardize the architecture. We know we need to have adaptive computation for reasoning. We know we want things like uncertainty quantification. What we're doing is bolting these things on top, rather than having an architecture that intrinsically does all of these things that we know we need.

Llion Jones

Yeah. I think our Continuous Thought Machine is an attempt at addressing those more directly, which Luke will be able to tell you more about later.

There's something still not quite right with the current technology, right? I think the phrase that's becoming popular is “jagged intelligence”: the fact that you can ask an LLM something and it can solve literally a PhD-level problem, and then, in the next sentence, it can say something so clearly and obviously wrong that it's jarring. I think this is actually a reflection of something probably quite fundamentally wrong with the current architecture.

As amazing as they are, the current technology is actually too good. That's another reason why it's difficult to move away from them, right? They're too good in the following sense. You spoke about the fact that we have these foundation models, so we have the foundation that we can do anything with them.

Yes, I think current neural networks are so powerful that, if you have enough patience, enough compute, and enough data, you can make them do anything. But I don't necessarily think that they want to. We're sort of forcing them. They're universal function approximators, but I think there are probably spaces of function approximators that will want to represent things more in the way that a human represents them.

There's actually quite an obscure paper that is my poster child for this. It's called “Intelligence via Matrix Exponentiation.”

Tim Scarfe

And I think it was actually rejected. You can probably project the image of Figure 1, but there's an image of it solving the classical spiral dataset, where you need to separate the 2 classes in the spiral.

Llion Jones

Yes. It has the decision boundary for both a classic ReLU multilayer perceptron and a tanh multilayer perceptron. You can see that they both solve it, right? Technically, they both solve the problem, because they classify all the points correctly and get a very good test score on this very simple dataset.

Then they show you the decision boundary for the MEX layer that they built in this paper, and it's a spiral. The layer represented the spiral as a spiral. Shouldn't we, if the data is a spiral, represent it as a spiral?

If you look back at the decision boundaries for the spiral and the classic ReLU multilayer perceptron, it's clear that you just have these tiny little piecewise-linear separations. That's what I mean. If you train these things enough and push these little piecewise-linear boundaries around enough, they can fit the spiral and get high accuracy. But when I look at that image, I don't get the feeling that the ReLU version actually understands that it is a spiral.

When you represent it as a spiral, it actually extrapolates correctly, because the spiral just keeps going out.

Tim Scarfe

You're touching on something fascinating there because we were talking about the need for adaptivity and adaptive computation. I'm really inspired by Randall Balestriero's spline theory of neural networks. We've had him on many times, and you can look at the TensorFlow Playground and see what happens when you have a ReLU network on this spiral manifold.

You'd be forgiven for thinking that these things are basically a locality-sensitive hashing table, right? They partition the space, and they can predict the spiral manifold. But we want to do something a little bit different than that.

It also comes into this impostor thing, because just tracing the spiral manifold but not continuing the pattern—there's a big difference between that. From an impostor perspective, just tracing the pattern is not learning it abstractly or constructively.

If we learned it constructively, as you speak about in your paper—with this complexification, these abstract building blocks—we could do adaptive computation. You understand the spiral. That means that, with adaptive computation, you can continue the spiral and then update the model's weights, so it has adaptivity, because that's so important for intelligence.

So we know that we need models that can do these things. But for some reason, they’re so sycophantic, they’re almost better than an adaptive intelligent system because they tell us exactly what we want to hear. They seem so intelligent, but we know that they’re missing these fundamental properties.

I’m still fairly skeptical when I see video-generation models. We went through a phase where you could detect them because of the number of fingers on somebody’s hand, right? With more data, more compute, and better training tricks, they improve, and now they usually do have 5 fingers. But did we fix the problem, or did we just use more brute force to force the neural network to know it’s 5 fingers, rather than having something that actually had a much better representation space?

It’s almost mad that it’s controversial to say that we should represent a spiral like a spiral. Something that could do that generally—if it represented a human hand the way that maybe I represent a human hand—might make it much easier to count how many fingers are on a hand. It’s unfortunate that they work so well. It’s unfortunate that scaling works so well, because it’s too easy for people to just sweep these problems under the carpet.

You guys have possibly created what I think might be the best paper of the year. This could actually be the innovation that takes us to the next step. Did you get the spotlight in Europe as well?

Luke Hewitt

Yeah.

Tim Scarfe

This year, and congratulations on that. I think that’s testament to how amazing this paper is.

Luke Hewitt

The CTM, the Continuous Thought Machine, is actually not that far outside of the local minimum that we’re stuck in. It’s not as if we went and found this completely new technology. We took quite a simple, biologically inspired idea: the fact that neurons synchronize—and not even necessarily in a biologically plausible way. Brains don’t literally have all their neurons wired together in a way that they work out their synchronization.

But it’s the sort of research that I want to encourage people to do. The way to sell it is quite easy. I think at no point did we have to worry about being scooped, right? That stress was taken away from us completely. There was no pressure to rush out with this idea, because we thought, “Well, there’s probably somebody else working on exactly this.”

I think the reason that we were able to get a spotlight is because we were able to create such a polished paper. We took the time to do the science properly, get the baselines that we wanted, and do all the tasks that we wanted to try.

Encouraging researchers to take a little bit more of a risk—to try these slightly more speculative, long-term ideas—I think the sad thing is, I don’t think it’s necessarily a very difficult thing to sell. I want to have the CTM as a poster child of “it works,” right? It was a bit of a risk. We didn’t know if we were going to find something interesting, but it was our first shot, and we did find something interesting, and it became a successful paper.

Tim Scarfe

If we do find a system that can acquire knowledge, design new architectures, and do the open-ended type of science that you’re speaking to, can you see a future where, at some point, the locus of progress will be mostly driven by the models themselves?

Llion Jones

I think so. Whether or not that’s going to replace us completely, I go back and forth on. Powerful algorithms are helping us do research, right? I think it might just end up being a more powerful version of that.

The AI Scientist that we released showed that you could actually go end to end: go from seeding the system with an idea for a research paper, take your hands off, and just let it go. Think about the idea, write the code, run the code, collect the results, and write the paper—to the point that we were actually able to get a 100% AI-generated paper accepted to a workshop recently, right?

But I think we did that to show that you could do it, as a sort of demonstration in a real system. I think I would want it to be much more interactive. I would want to seed it with an idea and then have it come back with more ideas, have a discussion with me, and then go away to write the code.

I want to look at the code and check it, and then discuss the results as they’re coming out. So that’s the sort of near-term future that I would envision, or how I would like to do research with an AI.

Tim Scarfe

And could you introspect on that? Is it because you feel we need supervision because the models don’t yet understand? There’s this path-dependence idea, so we need to do supervision because we have the path dependence, so we can guide the generation of the language models. Maybe in the future, language models will just understand better themselves.

But there’s also the output dimension, which is that we want to produce artifacts that extend the phylogeny of human interest. We want it to be human-relevant.

Llion Jones

Yeah, I think it’s more that, in that initial seed idea, it’s probably impossible to actually describe exactly what you want. It’s exactly the same with when I have an intern. I can’t just have an intern come into the company and say, “I have this mad idea,” explain it to them, and then leave them alone for 4 months.

There’s a back-and-forth because I have a particular idea that I want to explore, and I need to keep steering them in the direction that I had in my mind originally. So I think it’s more like that, basically.

You have such a deep understanding. You have this rich provenance and history and path dependence, and that means you can take creative, intuitive steps that respect the phylogeny. They respect all of this deep, abstract understanding that you have, and interns don’t yet have that.

Tim Scarfe

But maybe AI models in the future will have that.

Llion Jones

Yeah, sure. If they get to the point where my input becomes detrimental, then, yeah, that’ll be a thing. It’s kind of like chess, right? There was a point at which chess-engine-and-human fusion actually beat chess engines. That’s not true anymore, right? Adding a human into the mix actually makes the bots worse.

Tim Scarfe

Oh, interesting. I wasn’t aware of that.

Llion Jones

Yeah. So what to do when that day comes for AI scientists is a broader discussion, I think.

Tim Scarfe

I think now is a good segue to talk about this paper in a little bit more detail. This Continuous Thought Machine—you were just pointing to it before. Luke, first of all, mate, introduce yourself and set this thing up for us.

Luke Hewitt

My name is Luke. I am a research scientist at Sakana AI, and my primary area of research is the Continuous Thought Machine. It took us somewhere in the region of about 8 months to work on this project with the whole team. I did a lot of the work, but we also had a lot of people in different areas, doing different parts of it. I think an 8-month life cycle for a paper seems a bit long for AI research at the moment.

But yes, to the actual technical points of the paper. We call it Continuous Thought Machines. It originally had a different name. We called it Asynchronous Thought Machines before, but every single time people asked us, “What’s the asynchronous part?” it became a bit confusing.

Continuous Thought Machines basically depend on 3 novelties. The first one is having what we call an internal thought dimension, and this is not necessarily something new. It’s related conceptually to the ideas of latent reasoning, and it’s essentially applying compute in a sequential dimension.

When you start thinking about ideas and problems in this domain and in this framework, you start understanding that many solutions to problems that look intelligent are often solutions that have a sequential nature. For instance, 1 of the primary tasks that we tested in the Continuous Thought Machines was this maze-solving task.

Solving mazes for deep learning is quite trivial. It’s really easy to do if you make the task easy for machines. One of the ways to do this is to give an image of a maze to a neural network, like a convolutional neural network, and have it output an image the same size as the maze. It’s 0s where there isn’t a path and 1s where there is a path.

There’s some really brilliant work showing how you can train these in a careful way and scale them up essentially indefinitely. This is a fascinating and really interesting idea of how to solve this. However, when you take that approach out of the picture and ask what is a more human way to solve this problem, it becomes a sequential problem.

You have to say, “Go up, go right, go up, go left,” whatever the case may be, to trace a route from start to finish. When you constrain that simple problem space and ask a machine-learning system to solve it like that, it turns out to actually get much more challenging.

So this became our hello-world problem for the CTM, and applying an internal, sequential thought dimension to this is how we went about solving it. There are 2 other novelties that we can touch on and talk about.

We sort of rethought the idea of what neurons should be. There is a lot of excellent research in this world, particularly in cognitive neuroscience, exploring how neurons work in biological systems. Then, on the other side of the scale, we have how deep-learning neurons work. The quintessential example is a ReLU: it’s off or on, in a sense.

This very, very high-level abstraction of neurons in the brain feels a little bit myopic.

So we approached this problem and said, well, let’s, on a neuron-by-neuron basis, let this neuron be a little model itself. This ended up doing a lot of interesting work on how to build dynamics in the system.

The third novelty here is, as I said before, we have this internal dimension over which thinking happens. We ask the question: What is the representation? What is the representation for a biological system when it’s thinking? Is it just the state of the neurons at any given time? Does that capture a thought, if you wish? If I can be controversial and use the terms thinking and thought, my philosophy with this is no, it doesn’t. The concept of a thought is something that exists over time.

So how do we capture that in engineering-speak? Instead of measuring the states of the model that is recurrent, we measure how it synchronizes—how neurons synchronize in pairs along with other neurons. This opens up the door to a huge array of things that we can do with this type of representation.

Tim Scarfe

You were talking about the sequential nature of reasoning. Devil’s advocate: there was that Anthropic biology paper, and they were talking about planning and thinking. They were saying that this thing is planning ahead, and I think your system actually does planning. It’s computationally different. Can you explain that?

Llion Jones

Yes. I think the boundary in terms of computation, from a Turing machine perspective, if you wish, is really interesting. The notion of being able to write your tape, read from a tape, and then write again—to be in a Turing-complete system—is obviously an incredible idea that has completely changed the world.

I think the primary difference—let’s talk about transformers versus what we’re trying to do with the CTM—is that, in the process through which the CTM thinks, we can apply that internal process to breaking down a problem. The problem itself can have a single solution, and you could do that in one shot. As I explained with the maze, you could just process that in one shot, but there are certain phrasings of problems—real problems—where doing so becomes exponentially more challenging.

In the maze task, a really good example is that if you try to predict 100 or 200 steps down the path in one shot, no models that we could train—not even our model—could do that. We needed to build an autocurriculum system where the model first predicted the first step, and then, when it could predict the first step, we started training it on the second, third, and fourth steps.

The resultant behavior of this is where it gets interesting. One of the ways that I like to do research, and that I encourage people who work with me to do research, is to understand, if you wish, the behavior of a model. We’re getting to a point now where the models that we build are demonstrably intelligent in ways that keep surprising us, and breaking that down into a single set of metrics, or even a finite single metric about performance, doesn’t seem to me to be the right way to do it. Understanding the behavior and the actions that those models take when you put them in a system and train them in a certain way seems to reveal more about what’s actually going on under the hood.

Tim Scarfe

Very cool. I think I didn’t pick up on this. So you’re doing a fixed number of steps, so you have a context window. Did you say that you’ve set that around 100 steps?

Luke Hewitt

For the maze task, the model always observes the full image at every step. The CTM will observe the full image; for argument’s sake, those inputs could be tokens from a language model or the output of a language model. Those inputs could be numbers that the model has to sort. Whatever the case may be, it should be agnostic to data. That’s how we’ve tried to build it.

But in the maze task, the model can continuously observe the data. No matter where it is, it can look at the whole image simultaneously, but it uses attention to retrieve information from the data. It has, let’s call it, 100 steps that it can think through.

What we do is pick a point where the model solves 3 steps through the maze. It says, “I’m going to go up, up, and right,” and then it’s correct. But then it makes the wrong turn. At that point, we stop supervision. We only train it to solve the fourth step—1 more than what it could. In practice, we do it for 5 steps, but the principle holds.

When you do that, it’s a self-bootstrapping mechanism. I think the intuitive listener will understand how that extends to other domains, other sequential domains—for instance, language prediction many tokens ahead, that sort of thing.

Tim Scarfe

I’m really interested in this idea of adaptive computation. I guess the first question is: How sensitive was the performance to the number of steps? The next question would be: Could you have an arbitrary number of steps? Perhaps, based on uncertainty or some kind of criterion, you could do fewer steps. And the final question is: Could you potentially have an arbitrary or unbounded number of steps?

Luke Hewitt

That’s a really good question. I’ll answer the uncertainty question first, about the sensitivity to steps. A very good example of this is that we just trained the model on ImageNet classification, and our loss function is quite simple.

We run it for, for example, 50 steps, and we pick out 2 distinct points. The first one is where it’s performing the best—that is, where the loss is the lowest—and the second one is where it’s most sure, or most certain. Those give us 2 indices between 0 and 49, inclusive, and we apply cross-entropy at both of those points. We make the loss the average of the cross-entropy at those points.

What this does is induce a behavior where easy examples are solved almost immediately, in 1 or 2 steps, whereas more challenging examples naturally take more thinking. It enables the model to use the full breadth of time that it has available to it in a natural fashion, without having to force it to happen.

Tim Scarfe

You’ve decided to model every neuron as an MLP, which is really fascinating. Talk about that, but also, there’s this notion of synchronization, and I think you use the inner product to determine the extent to which the parameters are synchronized. This kind of unfolds over time as the driving force. Can you explain that in a bit more detail?

Luke Hewitt

Absolutely. I think it’s a good point to explain the neuron-level models, as we call them in the paper, or NLMs, first, because it ties into this. You can imagine a recurrent system as a state vector—a state vector that’s being updated from step to step. We track that state vector, and that state vector unfolds. For each individual neuron, each neuron in the system, we have an unfolding time series. It’s a continuous time series. Well, it’s discrete, but it’s a continuous value. Those time series define what we call the activations over time.

Synchronization is quite simply measuring the dot product between 2 of these time series. So you have a system of d neurons, and essentially you have d² different synchronization pairs. Neuron 1 can be related to neuron 2 by how they synchronize, and neuron 1 can also be related to neuron 3, et cetera.

The neuron-level models function by taking in a finite history, like a FIFO of neuron activations coming in. Instead of being just a ReLU activation, they use that history as information to process a single activation out. That is what moves from what we call pre-activations to post-activations.

The principle here is that this might seem rather arbitrary. Does it help with performance? It turns out that it does, but that’s not really the catch-all solution here. That’s not what we’re after. What we’re after is trying to do something biologically plausible: find the line somewhere between biology, which is how the brain implements things in the biological substrate that we have, and deep learning, which is highly parallelizable, super-fast to learn, backpropagatable, and has all of the nice properties that have got us this far.

We want to find a line somewhere where we can take some sprinkling of biological inspiration but still train it with deep learning. It turns out that neuron-level models are a nice interim that we can do this with. The concept of synchronization is applied on top of the outputs of those neuron-level models.

Tim Scarfe

On the scaling, I think the time complexity is quadratic with respect to the dimension of the synchronization matrix, right? In your paper, you were talking about subsampling to improve performance. How did that affect the stability? Did doing that cost you anything?

Luke Hewitt

Yes, it’s a neat question. In terms of stability, what we found was kind of fun, and this was a sentiment that we had throughout the experiments that we ran with this paper: It tended to work, no matter what we tried it on, across a wide range of hyperparameters.

The problems that you have with backpropagation through time, typically with recurrent models like RNNs and LSTMs, are a challenge. You run for many internal ticks with the RNNs or the LSTMs, and the learning seems to break down. But the fact that we use synchronization, in some sense, touches all of the neurons across all of the time steps, so it really helps with gradient propagation.

A nice, interesting point that’s maybe a bit oblique to what you asked about synchronization is that we have a system of d neurons and, like I said earlier, there are d² possible combinations.

This essentially means that our underlying state, or underlying representation of the system, is quite a lot larger than what you would get by just taking those d neurons. As to what that means in terms of downstream computation and performance, and the things that we can do with this, is what we’re actively exploring right now.

Tim Scarfe

You guys used an exponential decay rate?

Luke Hewitt

You have a system that unfolds over time. It would be maybe a little bit too constrained if the synchronization between any two neurons depended on the same time scale. For instance, there are neurons in your brain that are firing over very long time scales and very short time scales. The way that they fire together impacts other neurons and causes those neurons to fire.

Everything in biological brains happens at diverse time scales. It’s why we have different brain waves for different thinking states, for instance. Besides that point, what we do with the exponential decay in the Continuous Thought Machines is that it allows us to have a very sharp decay, to say that for these two neurons that are pairing together, what really matters is how they fire together right now.

But if we had a very long and slow decay, essentially that’s capturing a global sense of how those neurons are firing over an extremely long period of time. This was essentially a way of capturing the idea of how different neurons could maybe fire together very quickly, while other neurons can fire together very slowly or not at all.

This lets that representation space that I spoke about—that d² representation space—become richer again. We can enrich that space with more subtle tweaks to how we compute those representations.

Tim Scarfe

We were speaking about this yesterday, Luke. When folks apply transformers to things like the ARC challenge, or to things that need reasoning, we need to do lots of domain-specific hacks. The ARChitects, who were the winners of last year’s challenge, did depth-first search sampling, and some folks have been experimenting with using language representations or DSLs.

Some of this is to do with the reachability of language, right? Language is quite dense, which means you can kind of monotonically increase. But if I understand correctly, your system might have some interesting properties for reasoning and for discrete and sparse domains, and also for sample efficiency, because we want to build a system that can actually do well on things like the ARC challenge.

Can you explain in simple terms why you think this architecture could be significantly better than transformers for doing those things?

Luke Hewitt

I think a lot of the really fascinating work in the last few years that I found fascinating in the literature of language models has been related to what one can actually call a new scaling dimension. I see continuous chain-of-thought reasoning as a way of adding more compute to a system. That’s obviously just one small part of what that really is and what that really means, but I think it’s quite a profound breakthrough in some sense.

What we’re trying to do is have that reasoning component be entirely internal, yet still running in some sort of sequential manner. I think that’s rather important. You spoke earlier about Gemini’s diffusion language modeling, and I think there are a lot of different directions exploring this right now.

I do think that the Continuous Thought Machine, with the ideas of synchronization and multihierarchical temporal representations, gives a certain flexibility in that space that other people are not yet exploring. The richness of that space—the ability to project the next step to solve the ARC challenge, and the next 100 or the next 200 steps—to break that down into a process that a model can then very quickly search in its high-dimensional latent space becomes something that feels like a good approach to take.

Tim Scarfe

Do you see any relationship between this architecture and Alex Graves’s Neural Turing Machine?

Luke Hewitt

Yes, that’s really interesting. I do. I think one of the most challenging parts about working with a Neural Turing Machine is the concept of writing to memory and reading from memory, because it’s a discrete action. That has its own associated challenges.

I wouldn’t go so far as to say that the Continuous Thought Machine is definitively neural Turing complete, but the notion of doing reasoning in a latent space, and letting that space unfold in a way that is rich toward a different set of tasks, is interesting.

This actually brings me to a point that I find quite interesting and that I’d like to share with you. Consider again the ImageNet task, or any sort of classification task. It’s a nice test bed. There are many images that are really easy, and there are many images that are really difficult.

When we train, for instance, a ViT or a CNN to do this task, it has to nest all of that reasoning in the same space. It has to put all of its decision-making process for a very simple, obvious cat versus some complex, weird, underrepresented class in that system and in that data set. It has to nest it all in parallel, in a way that gets us to the last layer, and then we classify.

I think breaking that down, where you have different points in time where you can say, “Now I’m done. I can stop,” lets you take a data set or a task and actually naturally segment it into its easy-to-difficult components. I think we know that curriculum learning and learning in this continuous sense again seems to be a good idea. It’s how humans learn.

If we can get at that architecturally and just have that fall out in a model, again, this seems like something worth exploring. I’m not sure if you know much about model calibration and how neural networks tend to be poorly calibrated.

Oh, go for it, Tommy. It’s a bit of an old finding, but if you train a neural network for long enough and it fits really, really well, and you’ve regularized it really, really well, you’ll find that the model is uncalibrated. This essentially means that it is very certain about some classes where it’s wrong and uncertain about some classes where it’s correct.

What you want for a perfectly calibrated model is that if it predicts a probability that this is in the correct class of 50%, 50% of the time you want it to be correct about that class, and so on and so forth. So if a well-calibrated model is predicting a probability of 0.9 that it is a cat, then 90% of the time it should be correct.

It actually turns out that most models that you train for long enough get poorly calibrated. There are loads of post hoc tricks for fixing this. We measured the calibration of the CTM after training, and it was nearly perfectly calibrated, which is again a little bit of a smoking gun that this actually seems to be probably a better way to do things.

The flavor of this kind of research is such that we didn’t actually go out and try to create a very well-calibrated model. We didn’t even try to create a model that was necessarily going to be able to do some kind of adaptive computation time. I was a very big fan of the paper Adaptive Computation Time by Alex Graves, but that paper had a massive amount of hyperparameter sweeps because it needed to have a loss on the amount of computation that was being done.

Tommy Leung

Anytime you try to do some sort of adaptive computation time research, what you’re fighting is the fact that neural networks are greedy, right? Obviously, the way to get the lowest loss is to use all the computation that you have access to.

So unless you had an extra loss with a penalty that said, “Actually, you’re not allowed to use all the computation,” and very carefully balanced that loss, you didn’t get the interesting dynamic computation-time behavior falling out of the model in that paper.

What was really gratifying to see with the Continuous Thought Machine was that, because of the way we set up the loss that Luke described earlier, adaptive computation times seemed to just fall out naturally. That’s more the way that I think research should go.

Llion Jones

Okay, because we don’t actually have a specific goal or a specific problem we’re trying to fix, or something we’re trying to invent. It’s more that we have this interesting architecture and that we’re just following the gradients of interestingness.

Tim Scarfe

Yes. On that point, I think maybe the most exciting thing about your paper is that we were talking about path dependence and having this understanding that is built step by step, this process of complexification. Maybe this is apropos of the theme of world models in general, and also active inference.

I say “active inference” in big quotes because it’s not Karl Friston’s active inference—maybe adaptive inference or something like that—but we want to build agents that can continue to learn, that can update their parameters, and most importantly can construct path-dependent understanding.

That’s completely different from just understanding what the thing is. How you got there is very important, and this architecture potentially allows these agents, using this algorithm, to explore trajectories in spaces, find the best trajectories, and actually construct an understanding that carves the world up at the joints.

Luke Hewitt

Yeah, that’s a really neat perspective. I haven’t actually thought about it like that, but yes, I think that particular stance becomes really interesting when you think about ambiguous problems, because carving the world up in one way is as performant as carving it up in another way.

Yeah. Perhaps the hallucination in language models is carving the world up in some fine way, but it's just not performant by our measure—this is hallucination, and actually that's not true. But in some other trace down the path of wanting to carve the world up through an autoregressive generation of tokens, you end up in a different carve-up of that world. Being able to train a model that can be implicitly aware of the fact that it is actually carving up the world in a different way and explore those manners, those descents down the carve-up, is something that we're after.

I think it's quite an exciting approach: taking a stance of, let's break up this problem into small, solvable parts and learn to do it like that. How can we do this in a natural way without too many hacks?

Tim Scarfe

Yeah, it's something I've been thinking about because François Chollet—as much as I love his measure of intelligence—the idea for him is that adapting to novelty is getting the right answer, and the reason why you gave that answer is very, very important. In machine learning, we have this problem that we come up with this kind of cost function that rather leads to this shortcut problem.

But we could just build a symbolic system; we could be AGI, and we could say, okay, we need to do this principled kind of construction of knowledge, maintaining semantics. Well, we're not doing that. We're doing a hybrid system. But there must be some natural way of doing reasoning where, in spite of the end objective being this cost function, because of the way that we traverse these open-ended spaces, we can actually have more confidence mechanistically that we're doing reasoning which is aligned to the world.

Llion Jones

I think that's a great way of seeing this particular avenue of research, and obviously we're not the only people thinking like this and we're not the only ones trying to do this. What we have is an architecture that's amenable to it, and surprisingly so. Again, it wasn't the goal. It's not the goal to do this type of research. It's not the goal to be able to break the world down into these small chunks that we can actually reason over in a way that seems natural.

Instead, what we did was pay respect to the brain, pay respect to nature, and say, well, if we build these inspired things, what actually happens? What different ways of approaching a problem emerge? And then when those different ways of approaching a problem emerge, what big philosophical and intelligence-based questions can we then start to ask? And that's where we're at right now.

So it might feel at times, especially for me, too many questions and too few hands to answer those questions. But I think the fun and exciting thing, and the encouraging thing I can say to other younger researchers out there is: do what you're passionate about and figure out how to build the things that you care about, and then see what that does. See what doors that opens up and see how to explore deeper into those domains.

Tim Scarfe

We were talking about this yesterday, weren't we? That you can think of language as being a kind of maze.

Luke Hewitt

Yes. What is to stop us from taking this architecture and building the next-generation language model with it? I mean, that's honestly, as you know, something that I am actively trying to explore right now. I think the maze task gets really interesting when you add ambiguity to it, when there are many ways to solve the maze.

Honestly, this isn't something I've tried yet, and maybe it's something I should try next week. But essentially, you can imagine an agent—or the CTM in this case—observing the maze and taking a trajectory. Surprisingly, we saw this: we have a section in our recently updated paper on arXiv—the final camera-ready version of this paper—where we added an extra supplementary section that is not in the main technical report.

That supplementary section is basically: Hey, we saw this cool stuff happen, and we list, I think, 14 different interesting things that happened while we were doing the research that obviously didn't make it into the paper, but we wanted people to know about these strange things that happened. And this is one of the strange things: we watched during training what was happening.

At some time during training, maybe halfway through the training run, we could see what the model would do. It would start going one path in the maze, and then suddenly it would realize, “Oh no, damn, I'm wrong,” and backtrack and then take another path. But eventually it gets really good, and it does some sort of distributed learning in this because it's got an attention mechanism with multiple heads. So it can actually just figure out how to do this pretty well and refine its solution.

But sometime early on in the learning, it descends multiple paths and comes back and backtracks. We have a really fascinating set of experiments that also showed—and we actually have some supplementary material online showing this—and I don't really know what this says. It's kind of a deep philosophical thing, but if you're trying to solve a maze and you don't have enough time, it turns out that there's a faster algorithm to do it.

This blew my mind when I saw it. So if we constrain the amount of thinking time that the model has but still get it to try to solve a long maze, instead of tracing out that maze, what it does is it quickly jumps ahead to approximately where it needs to be, and it traces backwards and fills in that path backwards. Then it jumps forward again, leapfrogs over the top, and traces that section backwards, and then leapfrogs, and it does this fascinating leapfrogging behavior that is based on the constraint of the system.

And again, this is just an observation we made, and what that means in a deep sense, and how it's related to giving a model time to think versus not—and is it enough time to think? What happens? What different algorithms does the model learn when you constrain it in this way? I find that quite fascinating and an interesting thing to explore.

Does it tell us something about how humans think? Does it tell us something about how we think under constrained settings versus open-ended settings? There are a number of cool questions you can ask on this front.

Tim Scarfe

You guys are both huge fans of population methods and collective intelligence, and because we can scale this thing up and we can scale it out, what would it mean to scale this thing out not only just in a kind of trivial parallelization, but in terms of having some kind of weight sharing between parallel models and so on? What would that give you potentially?

Luke Hewitt

This is a fun area of research. So one of the active things that we're trying to explore in our team is concepts of memory, long-term memory, and what does this mean for a system like this? An experiment that one can construct, for instance, is to put some agents in a maze and let them try to solve this maze—not how we did it in the paper, but in a very constrained setting where an agent can only see maybe a 5×5 region around it—and we give that agent some mechanism for saving and retrieving memories.

The task, if you wish, is to solve that maze, find your way to the end, and the model needs to learn how to construct memory such that it can get back to a point where it's seen before and know, I did the wrong thing last time, and go a different route. You can then see this with parallel agents in the same maze, with a shared memory structure, and see what actually happens when you can all access that memory structure and have a shared global, almost like a cultural memory that we can access and solve this global task by having many agents trying to use this memory system.

I do think that memory is going to be a very key element to what we need to do in the future for AI in general.

Tim Scarfe

So the subject of reasoning came up just a second ago, and I think there's a perception that recently we made a lot of progress in reasoning, right, because it's actually one of the main things that I think people are working on. We released a dataset recently called Sudoku-Bench, and I was actually quite happy to see it come up organically on your podcast a few weeks ago.

Llion Jones

Chris Moore,

Tim Scarfe

Right?

Llion Jones

Yes. So, I wanted to tell you a little bit about this benchmark because I think I've been having a little bit of an issue promoting it because it doesn't, on the surface, sound particularly interesting. Sudoku has a sort of a feeling that it's already been solved, right? So how interesting can a collection of Sudokus be for reasoning?

Exactly. We're not talking about normal Sudokus. We're talking about variant Sudokus. And variant Sudokus are usually normal Sudokus, right? So put the numbers 1 to 9 in the row, the column, and the box, but then literally any additional rules on top of that. And they're all handcrafted. They all have extremely different constraints.

Constraints that actually require very strong natural language understanding. For example, there's one puzzle in the dataset where it tells you the constraints of the puzzle in natural language and then says, “Oh, by the way, one of the numbers in that description is wrong.” Right? So you have to be able to meta-reason about the rules themselves even before you start solving the puzzle.

There are other puzzles where you have a maze overlaid on the Sudoku and the rat has to work out a way through the maze by following a path to the cheese. But there are constraints on the path that it takes, like what numbers and what they can add up to. It's difficult to really describe how varied these variant Sudokus are, and I think they're so varied that if anyone were actually able to beat our benchmark, they would necessarily have created an extremely powerful reasoning system.

Right now, the best models get around 15%, but they're only solving the very simplest and smallest Sudoku puzzles in the set. We're going to be putting out a blog post about GPT-5's performance, and it is a jump, but it's still completely unable to solve puzzles that humans can solve.

What I really like about this data set—and what was actually the catalyst for me creating it in the first place—was that there was a quote by Andrej Karpathy saying, “Okay, so we have all this data; it's from the internet. But what you really want, if you wanted AGI, you wouldn't want all of the text that humans have ever created. You would actually want the thought traces in their head as they were creating the text,” right? “If you could actually learn from that, then you would get something really powerful.”

And I thought to myself, well, that data must exist somewhere. My first thought was maybe philosophy. There's a type of philosophy where you just write down your thoughts without thinking, like stream of consciousness. I thought maybe that could work.

But then, when I wasn't thinking about it and I was in my leisure time, I was watching a YouTube channel called Cracking the Cryptic.

Tim Scarfe

Yes.

Llion Jones

These two British gentlemen solve these extremely difficult Sudoku puzzles for you. Sometimes their videos are 4 hours long, and they're professionals; this is their job. What was perfect, I realized, is that they tell you in agonizing detail exactly what reasoning they used to solve those particular puzzles.

So, with their permission, we took all of their videos—which represents thousands of hours of very high-quality human reasoning, like thought traces—and scraped them and made that available for imitation learning. We did try to do this internally. It turns out that I did a little bit too much of a good job of really creating a very difficult benchmark. So we're still trying to get that stuff working, and we'll publish it if we have some success.

I want to really sell the fact that this reasoning benchmark is different. Not only do you get something that's super grounded, where you know exactly if it's right or wrong, so you can do RL to your heart's content, but you can't generalize very easily. Each puzzle is deliberately designed by hand to have a new and unique twist on the rules, called a break-in, that you have to understand.

Right now, despite all the progress we've made, current AI models can't take that leap. They can't find these break-ins. They'll fall back to, “Okay, I'll try 4. I'll try 5. I'll try 6. I'll try 7,” right? The reasoning becomes really boring and nothing like what you see in the transcripts that we've open-sourced from this YouTube channel.

So I just want to put the challenge out there: this is a really difficult benchmark, and I think progress on this benchmark will really mean progress in AI generally.

Tim Scarfe

Could you reflect a bit? After watching this Cracking the Cryptic YouTube channel, how diverse were the patterns? Chris was saying to me, “Oh, you know, these guys, they go on Discord servers, they get these creative, crazy ideas,” and I'm obsessed.

Maybe I'm just being idealistic, but I love this idea of there being a deductive closure of knowledge, right? There's this big tree of reasoning, and we're all in possession of different parts of the tree to different depths. So the smarter and more knowledgeable you are, the deeper down the tree you go.

In this idealized form, there is one tree, and all knowledge originates or emanates from these abstract principles. We could, in principle, build reasoning engines that could just reason from first principles, and it might be computationally irreducible. So you have to perform all of the steps.

It feels like, because we're not in possession of the full tree, what we need to do is fish around. We fish around to find Lego blocks: “Oh, that's a good Lego block. I can apply that to this problem.” Maybe that's just what we need to do in AI for the time being: we need to acquire as much of the tree as possible. But could we just do it all the way down?

Llion Jones

Yeah, fascinating question. That tree is probably massive, right? And as a human is solving these puzzles, they're definitely learning in real time and discovering new parts of this tree. It's sort of a meta-task, right? Because it's not just reasoning; you're reasoning about the reasoning.

I don't think we have that in AI right now. If you watch the videos, they'll say something like, “Okay, this looks like a paraskew, or this is a set-theoretic problem, or maybe I should get my path tool out and trace this around.” Of course, the professionals do have this already massive collection of reasoning Lego blocks, as you say, in their heads. So they'll recognize, “Okay, that type of rule usually needs this kind of Lego block.”

It's fascinating to watch how good they are at just intuitively knowing where to look. Someone like me, who hasn't solved as many, needs to spend a lot of time looking around: “Okay, maybe I should try this, or maybe I should try this one.” But even they're not perfect.

You can watch them take a certain kind of reasoning and start building up: “Okay, maybe we should solve it like this,” and then go, “No, that doesn't disambiguate it enough,” and then backtrack and go down another path. Again, that's something that we do not see current AI doing when they're trying to solve this benchmark.

The tree is very big, and I guess the phylogenetic distance between many of these motifs in the tree is just so large. It's so difficult to jump between, and I think that's why, as a collective intelligence, we work so well together: we actually find ways to jump to different parts of the tree.

Tim Scarfe

Right?

Llion Jones

I think that's probably why the current state of the RL algorithms that we're trying to apply to this just isn't working. In order to learn how to get these breakthroughs—to understand what the nuanced reasoning is to solve these puzzles—you have to sample them. That's such a rare space; it's such a specific kind of reasoning that's required to get to the specific breakthrough that this kind of technique doesn't work, right?

There's definitely a feeling in the community like, “Okay, this is how you just solve things now. We have RL; yes, we can get these language models to do what we want.” It doesn't work for this data set.

Tim Scarfe

Guys, it's been an absolute honor having you on the show. Just before we go, are you hiring? We've got a great audience of ML engineers and scientists, and I think working for Sakana would be the dream job.

David Ha

That's very kind of you. Yes, we are definitely hiring, and as I said earlier in this interview, I honestly want to give people as much research freedom as possible. I'm willing to make that bet, right? I think things that are very interesting will come out of this. I think we've already seen plenty of interesting things coming out of this. So if you want to work on what you think is interesting and important, come to Japan.

Tim Scarfe

And Japan just happens to be the most civilized culture in the world. [laughter]

All right. It might be the opportunity of a lifetime, folks. So, yeah, get in touch, guys. Seriously, thank you so much. It's been an honor having you both on the show.

Llion Jones

Thank you very much.

Llion Jones

Thank you so much. It's been great.

He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow] | BidClub