边缘智能：Liquid AI 的 Ramin Hasani 探索设备原生基础模型

Happy 4th of July to everyone in the United States. Today my guest is Ramin Hasani, CEO of Liquid AI, a company founded by MIT researchers that's developing device-native foundation models.

I'll say up front that, just before recording, I encouraged Ramin to go deep into the weeds on the technical details of Liquid AI's work. And, as you'll hear, he did a truly excellent job demonstrating a mix of technical sophistication, differentiated vision, and a contagious passion that, in my humble opinion, makes this episode an instant classic.

We start with an overview of the team's research into tiny, biologically inspired, differential-equation-based neural networks that Ramin and his team developed at MIT and which inspired them to start the company. Some of the capabilities they demonstrated, such as parking a car with a control module that consisted of just 12 liquid neurons, still sound a bit like science fiction today. And while those systems haven't scaled up to today's capability frontier, the company has maintained the liquid philosophy, which today means taking a neutral, empirical approach to designing and optimizing neural networks to perform under all sorts of exotic constraints, including, most commonly, the need to run on edge devices with limited memory and processing power.

Considering the fact that the global smartphone and laptop market is roughly $800 billion per year—a number that the global AI data-center buildout is only now surpassing—and that so many enterprises and individuals value privacy and the ability to control their own information, the demand for inference threatens to price much of the world out of the frontier-model market. This is an absolutely massive market opportunity unto itself.

Liquid has serious proof points, including holding the number 5 spot on the Hugging Face U.S. Downloads leaderboard, plus notable partnerships with companies such as Shopify and Mercedes-Benz. Anyone who doubts this can do a quick download and demo of Liquid's Apollo app, which shows, in my experience, that even a 1-billion-parameter model—which combines a small number of attention layers with a very simple gated learned convolution, while admittedly far from the frontier—can run fast enough on an iPhone to be a real option for basic use cases such as privately searching through and classifying one's own local documents.

Perhaps most interesting is the network architecture search process that Liquid uses to develop networks for particular use cases and runtime environments. Having found that proxy metrics too often lead the process astray, they now evaluate models on real downstream tasks on the actual target hardware that their customers intend to use. Ramin shares a lot more detail on their findings, but, in short, while attention-based architectures continue to generalize better than any known alternative and therefore continue to dominate the frontier, the more specific your use case and the more limited the compute resources you have available, the more likely their search process is to land on an exotic architecture. For now, this is where architectures like Mamba and other subquadratic innovations really shine.

Toward the end, Ramin teases a platform that Liquid will soon be introducing to allow customers to fine-tune small models for their own use cases on a self-serve basis. For multiple reasons, including the potential to ease demand for frontier models and improve access to AI globally, I, for one, will be very excited to see that come online. And so, without further ado, I hope you enjoy this high-energy look at how Liquid AI is squeezing as much intelligence as possible out of any given computational resource with co-founder and CEO Ramin Hasani.

Ramin Hasani, co-founder and CEO of Liquid AI, welcome to The Cognitive Revolution.

I've been following Liquid AI from afar for a number of years, and I'm fascinated by some of the architectures that you guys have developed going back to your time at MIT together. I'm also fascinated by the trajectory that I perceive the company to have taken over the last few years as it's become a more customer-focused, commercial entity. So maybe, for starters, how would you tell the broad story of Liquid AI leading up to what you're doing today, and what the company's mission is today?

Yeah, absolutely. So, 3.5 years ago, we spun out of MIT CSAIL, building on a technology that we had actually been working on for 10 years before—basically, a decade before we started the whole company.

Our objective function at MIT has always been maximizing the amount of intelligence that we can pack into the smallest format of algorithms. Efficiency has been the cornerstone of our research. We have been working specifically on robotics and systems that were coming into the real world. The idea of liquid neural networks was discovered as part of my PhD thesis, and together with my co-founder Matthias Lechner and my other co-founders—the 4 co-founders that I have—we've been researching ideas about how we can build machine-learning solutions that can go on robots and don't have millions or billions of parameters.

That way, we can actually host them directly on, let's say, CPUs, NPUs, or smaller GPUs that are mounted on top of physical systems, while delivering the reliability of much larger instances of artificial-intelligence systems. Essentially, what we try to do is build alternative algorithms in order to get creative in the algorithmic space, to see how we can build machine-learning systems that can generalize beyond the data that they have seen.

When you go into the real world, distribution shifts become a real thing. Imagine deploying a robot in an open-world environment: an autonomous car, a flying drone, or a fixed-wing vehicle. All of these systems are going to go out, and very, very rapidly, you're going to get out of distribution. So you have to actually build systems that are really comfortable with being able to get out of distribution.

Humans are extremely good at it. Humans do that at an extreme; natural learning systems—animals—also do it the same way. They actually follow a really nice trajectory out of distribution as well. That means the learned concepts can generalize to data that you have not seen before.

This has not just been our goal. I think the entire field of artificial intelligence has been working toward this, especially in robotics and closed-loop environments, where you have an agent acting in an environment. You want to have those kinds of properties.

Naturally, the place that we started looking into was brains. We started looking into animal brains, and one thing that I really liked was looking at them from a first-principles kind of approach: How do neurons exchange information with each other?

We started looking into the brains of worms and small animals. Why worms? This specific worm, C. elegans. When I started this type of research in 2015 as part of my PhD with my co-founder Matthias Lechner, this was the only animal whose entire nervous system we knew as a whole.

This animal exhibits a massive amount of sensory-reactive behavior—amazing levels of control with 300 cells in its nervous system. That was fascinating for us because this is much smaller than any neural network that performed control at that time, let's say, on autonomous systems. Yet this worm can perform more dexterous movements than the best robotic systems that we had in the world.

So we thought, okay, let's start understanding how neurons exchange information in the brain of the worm. From there, let's start building nervous systems with more complex neural circuits, so that we can get to the stage where we can, let's say, build the next animal—basically, follow the path of evolution of nervous systems and see how we can rise and evolve as part of this thing.

There are equations that describe the neuronal dynamics of, let's say, 2 neurons in the brain of C. elegans. Because the worm is very small, the neurons do not spike. They are graded neurons; they behave in an electrotonic way.

They're very similar to artificial neural networks that we have because they're also very differentiable. You don't have spikes in the activity of the neurons, so they're very differentiable. That's why it was even nicer and more attractive for us, because we could apply learning theory to these types of neural networks.

Once constructed—let’s say two neurons, four neurons, eight neurons, or 100 neurons next to each other—you can start training them. You can apply backpropagation as a form of differential programming on top of a system built from these inspirations we got from nature. The type of differential equations involved were also very well behaved. You can make them as complicated as you want, and when they become more complicated, they mimic biology much more closely.

Of course, from a computational-footprint perspective, they become more complex to scale. But you can use artificial neural networks to abstract away the complex differential equations that describe the neural dynamics between two cells. We represent them with sigmoidal functions, gated sigmoidal functions, and now matrix multiplications that capture the impact of the inputs coming into a system with a transformer-based architecture, an attention mechanism, and all of those things. These are all simplified computations that allow us to scale machine-learning solutions.

From a neuroscience perspective, it was very interesting for us to explore whether bringing back those differential-equation-based computations in a more elaborate form—putting more computation into the behavior of every single neuron and mimicking how two neurons exchange information with each other—might unlock something greater than what we had seen from an artificial-neural-network perspective. Early on, the results were fascinating. We saw that with 12 neurons, you could autonomously parallel-park a small car; with 19 neurons, you could drive a car; and with 30 neurons, you could autonomously fly a drone, navigating it and receiving and processing sensory information.

We called these more complex and elaborate versions of neural dynamics liquid neural networks, with “liquid” referring to their adaptability. We called them liquid time-constant neural networks—that was the LTC paper that we published—and we coined the name “liquid” because the dynamics of these systems remain flexible even after training. The models could react to new types of inputs they received and learn during backpropagation how to adapt to those inputs. They encoded more degrees of freedom in the flexibility of a neural network’s learning dynamics compared with artificial neural networks and other systems whose dynamics we had seen.

That form of dynamics allowed us to scale this technology beyond robotics. We applied it to predictive AI in financial services, the medical domain, and many other areas, including audio modeling and multimodal video understanding. We applied this technology over and over, and we saw that it was very promising. Every neuron is much more complex than a neuron in a normal neural network, but at the same time, you don’t need that many computational operations to get the results you want.

We also saw that the out-of-distribution performance of these models was extremely good, so they were well suited for robotics applications. We had interactions with the United States Air Force. At that time, Boeing was hosting my postdoc at MIT, and there we showed that you could even fly jets with these types of neural networks, using a handful of neurons. It was interesting to see how far you could push out-of-distribution generalization with a very small system powered by these liquid neural networks, as opposed to artificial neural networks.

As I told you, each node in a liquid neural network is very complex. It’s a differential equation that you have to solve, and the more neurons you have, the more complex the forward pass and backward pass through the network become. Therefore, we needed to think about efficiency. These systems are highly nonlinear, and we tried not to sacrifice nonlinearity because adding nonlinearity to learning systems allows you to build more expressive systems.

This is something I proved in my PhD thesis. I showed that nonlinearity directly contributes to performance, especially in smaller models. The idea for us was: How can we scale these systems? You have nonlinear systems, but it is extremely difficult to scale nonlinear systems. The reason behind that difficulty is that the computational complexity of the models can reach cubic complexity. It’s not even quadratic; you’re talking about cubic complexity, which is too much.

Usually, you have a set of differential equations that you can roll out with a numerical solver. You run the numerical solver and step by step compute the desired outputs. The more steps you use, the more accurate the results become. But the problem is scalability again: you cannot really get to infinite precision with these numerical solvers.

One idea that came to us was: What if we solve the whole system in closed form? You have a differential-equation system with these liquid neural networks, each representing the neural dynamics of two cells exchanging information with each other at a certain level of abstraction. We thought, “Let’s take this system and try to solve it in closed form.”

It turns out that a closed-form solution for this type of equation had not existed since 1907. In 1907, a scientist called Louis Lapicque modeled the membrane potential—how to mathematically model the membrane potential in cells—and that form of the equation became fundamental to channel modeling: how information propagates through ion channels inside a cell, and how neurotransmitters are propagated to another cell.

In 1907, this was Louis Lapicque’s membrane-potential equation, an ordinary differential equation. Later, scientists called Alan Hodgkin and Andrew Huxley started working on the biological grounding of this type of differential equation and adding more complexity to the model. They developed a neuroscience model of a neuron: how a neuron reacts to a membrane potential. They started in 1953, and in 1963 they won a Nobel Prize for developing a better and more accurate representation of neuronal dynamics.

From there on, in every textbook you read, these types of equations do not have a known closed-form solution. Liquid neural networks were also part of that type of equation. For the first time, in 2022, we solved the interactions of liquid neural network neurons with each other in closed form. This became a Nature Machine Intelligence paper published in November 2022, called “Closed-form Continuous-time Neural Networks.”

The closed-form solution has massive implications. Why? Because now I don’t need to use numerical solvers to run a liquid neural network. I can have not only hundreds of neurons but billions of neurons next to each other, and I can scale these computations while still keeping the nonlinearity as part of the system.

At the beginning of February 2023, an article came out in Quanta Magazine profiling me and my co-founder, Mathias Lechner. It was about the implications of finally having a closed-form solution for neuron dynamics and how important this could be for both machine learning and brain science as a whole.

My inbox was full of VCs. Silicon Valley started talking; everybody wanted to put a term sheet in front of us to get started and scale this technology because it was fundamentally different from a transformer-based or attention-based architecture. It was grounded in biology, in the type of mathematics that we had developed.

Around the same time, alternative models were coming out, such as state-space models. You saw faster iterations of state-space models and convolutional neural networks that were designed to be scalable, all of them in linear form. To scale alternative models, you have to linearize them. We had another category that added operators we learned from biology and physics as we grew the research.

We got to the point where we could become very competitive in building models for solving increasingly general-purpose tasks. When I say “more general-purpose tasks,” I’m talking about modeling signals beyond predictive performance: modeling signals for language, audio, and vision in a way that humans understand. That’s the complex format that humans understand.

The mission of Liquid AI became building efficient, general-purpose AI systems at every scale. We coined the word “efficient” in our mission, and we care about it so much because, from a foundation-model company’s perspective, I think we are probably the most efficient foundation-model company on the planet.

I’ll tell you why in a minute. The reason why we set the mission as efficient machine learning is that we saw how many considerations we had to make in order to really build a closed-form solution, and then take that closed-form solution and scale it to, let’s say, a system that can model language, a system that can model audio, and a system that can model video and vision in general.

We had to consider so many things so that we could, in a computationally tractable way, scale our neural architectures to the point where we are today. The base of our technology—I’ll talk about the core technology of the company later on—but today the technology has become liquid foundation models, or LFMs. Liquid foundation models are pretty popular right now. We are ranked number 5 in the United States in terms of the number of downloads, from a popularity perspective. I believe we have over 1 million downloads per week on Hugging Face.

That’s a lot of downloads of these small models that we’re building. They’re very popular. The top ones from the United States are Google, Meta, Microsoft, and NVIDIA, and the 5th one is Liquid AI. We got ourselves there by using about 1,000 GPUs and having 1,000 GPUs in-house.

From a foundation-model perspective, getting to that level of popularity and releasing more than 50 model instantiations that people are using in enterprises—that’s the place where we brought Liquid, building on top of the inspirations that we got from nature and the path that I was portraying for you.

Brilliant. There are so many different directions I want to go in as a follow-up there. For starters, one thing that jumps out at me on a very fundamental level is that when you describe liquid neural networks, you describe them in terms of the number of neurons, whereas we’re used to hearing about the number of parameters. Something tells me there’s a paradigmatic difference underlying that difference in description.

Help me develop my intuition a little bit more. I guess it probably connects to nonlinearity, and this is so fundamental, right? There’s been incredible progress, but on things like really robust out-of-domain generalization, we’ve still got a lot of work to do. On things like adversarial robustness, we still have a lot of work to do in the mainline paradigm. So unpack this a little bit for me. Why are we counting neurons versus parameters, and what does that tell us?

That’s a great question. Early on, we actually started talking about the number of neurons as the unit of computation in our research because that was very interesting. You can associate every neuron with a process that can be modeled by a mathematical equation. Now you can allocate gates, basically, in a sigmoidal kind of form. The more functions you have, the more parameters get added to that single cell.

In a liquid neural network, in terms of parameters, you can say, “Multiply by 7,” and that would be the number of parameters of a neuron. In that differential equation, that cell itself, because of the equations that it has, computes more than just a forward-pass computation. It has internal feedbacks; it has three-degree feedback mechanisms inside. All those parameters are actually one-to-one analogous to any artificial neural network system that you would build in terms of parameters.

A rule of thumb is that when I say “number of neurons,” you multiply by 7 and you would get the number of parameters of the system. But this is for the early versions of liquid neural networks. As we started standardizing parallelization schemes for the types of mathematical functions that we wanted to explore, and for the inspirations that we were getting, we converged on the number of parameters.

Today, when we’re talking about a liquid foundation model that has, let’s say, 1 billion parameters, we literally mean 1 billion parameters, just as with any other GPT that is being built out there. Does that make sense?

Yes. But let’s zoom in a little bit more on that neuron. I’m wondering, too, what the fundamental limit is, or what the first bottlenecks are, when you try to scale the original liquid approach with the sigmoid-type functions that we have today.

Those are carefully selected to be easily run on GPUs. I’m guessing that these multiple degrees of freedom inside a single neuron may present challenges in terms of executing that process on available hardware—how we can get these amazing results from tens of neurons.

The obvious bitter-pilled question would be: What if we take that exact paradigm and go to millions or billions? You must be hitting some bottlenecks along the way. What are they?

The main challenge is turning sequential computation into parallel computation. When you start going from a single-neuron dynamic to multiple-neuron dynamics, the weight parameters of your system, instead of being scalars or vectors, become matrices and tensors. Now you’re talking about matrix multiplication. You want to turn scalar computations into tensor computations, right? This is how you parallelize sequential computations.

The problem is that, if you have nonlinear relationships in an equation, you cannot trivially create a one-to-one map between a vectorized computation and a tensorized computation. Those nonlinear attributes of liquid neural networks—and any other nonlinear recurrent neural network—add degrees of complexity to the math of the whole equation.

Let alone if the recurrence has a nonlinearity on it. It becomes a lot more complex to disentangle vectors, or write them using typical linear-algebra methods, into tensors. You have to be able to turn these computations into tensors to compute them in parallel. You compute them once at a time, you see? That’s the mathematical challenge.

It’s not related only to liquid neural networks. Any mathematical operation that you want to tensorize, if it’s nonlinear, is going to have the troubles that we’re talking about. That’s why the state-space models that came out are all around linear dynamics. We’re talking about linear dynamical systems, right?

The reason those linear dynamical systems are important is the simple fact that we do not have proper ways to scale nonlinear systems. You can apply nonlinearity to a linear dynamical system. For example, in a gated system, you can add a sigmoidal function after you perform the dynamical system in a linear way. You compute all the matrices and everything in parallel, and then you can do a pointwise application of a nonlinear operation on top of the entire tensor. That’s what you can do.

But what if the relationship between the parameters themselves is governed by some nonlinearity? You cannot really use typical linear algebra. You have to always approximate that nonlinear system into a linear system, and then you would be able to parallelize the systems. Does that make sense? That’s the fundamental bottleneck.

How much does the closed-form solution address that? With the available computing resources, how far has the original liquid-network paradigm been able to scale so far, to the present?

Fantastic question.

So, one of the properties of liquid neural networks was the fact that they have multiple feedback mechanisms. They're not just one form of feedback; they have three layers of feedback between two cells, within the synaptic dynamics itself, because we wanted to mimic how it is done in the brain. So, they have multiple degrees of feedback and nested nonlinearities on top of each other. That nested nonlinearity itself would also add a degree of complexity.

Even the closed-form solution version of a liquid neural network, from a dynamics point of view, means that instead of 100 neurons, you can now have 100,000 neurons, maybe 1 million to 10 million neurons. But you still have to compute the models sequentially because of the nested nonlinear relationships that actually happen in the circuit. So even the closed-form solution would be limited in scalability, in the sense that you cannot parallelize it very effectively.

But there are also people working on these topics a lot—not just parallelizing, but speeding up sequential computations. We're talking about sequential scans. A scan is an operator that you can run on top of, let's say, matrix multiplication. Instead of doing it in cubic or quadratic time, you can do it in subquadratic time, so you can reduce the complexity of computations even on nonlinear operators without parallelizing them, without linearizing them.

There is ongoing research in this area. We're doing some of that research ourselves to find ways to run those nonlinear systems sequentially, because clearly there is an advantage to having those forms of nonlinearities when you do function approximation. Scaling laws also become very important here, I believe. Scaling laws define architecture.

When we're talking about transformers being this revolutionary thing, we're talking about massive scale. The reason why the transformer architecture and attention mechanism are such brilliant architectures is the fact that they're unstructured. There's no structure. When I'm talking about nested nonlinearities and all this complexity that we have in liquid neural networks, you don't have that in transformers, right?

You have basically just matrix multiplication as the core functional operation, and it is unstructured. You can literally multiply any matrix into any matrix of any size. So, the whole idea here is this: the larger the neural networks you make, the more you want them to become less and less structured. We've seen the success of transformers at trillions of parameters. Now we're talking about tens of trillions of parameters. The next generation of models that are going to come—we're talking about trillions of parameters—and you can do that with a transformer architecture.

As soon as you start adding a little bit of bias to that architecture at scale, things become completely messed up. So, we're talking about liquid neural networks and these alternative architectures that we're discussing: there is a scale to them. There is a regime of parameters where you can just do better—let's say, up to 100 billion parameters, up to 1 trillion parameters. That's kind of the range we're operating in right now.

The smaller the model architecture is, the more you want to specialize it for a certain application. They are mathematically biased to solve a certain type of task better than other types of architectures. I would say that biases in algorithms—the more biases you put on, and by biases, what I mean is adding a lot more nonlinearity, adding multiple gating levels on top of a neural network, adding recurrence, adding multiple different types of recurrence, or adding convolutions—become a function of the scale of the neural networks as well as the use cases you want to solve.

Convolutional neural networks are also pretty unstructured if you just keep the convolution, because you can apply convolutions to neural networks of any size. So, the question becomes: how much bias, and what kind of problems do you want to solve? The bias in neural architectures becomes a function of the scale of the neural networks as well as the use cases you want to solve.

The spectrum would be that the more you go toward larger sizes, the more you use unstructured mathematical operators: pure matrix multiplication, pure convolutions, pure, let's say, sequential scans—pure operators. You get much more of that. Forget about gates; forget about adding fancy control, like forget gates and all those things. At scale, this is something that we've seen, because we've scaled neural networks and we have a really good understanding of that architectural variation at different scales.

Just for calibration, when you talk about the liquid neural networks that have gone up to hundreds of thousands or are pushing a million neurons, what kind of hardware does that run on? Is that a CPU supercomputer?

You can do that on a CPU, even simple CPUs or simple GPUs, because they don't have that much. Probably one of these liquid neural networks would fit in 1 to 25 megabytes of file size, basically. You can literally put them there, and a CPU would be enough. A Raspberry Pi would be enough to perform the computations.

We have shown that in a lot of predictive, specialized applications of AI, these systems can be very powerful. These closed-form variants could be as powerful as the ordinary differential equation version of the neural networks in speech synthesis. That's one of the applications. In predictive sequences, imagine you have complex forms of sequences coming in—multivariate sequences—and you want to perform some sort of prediction on top of them. These neural networks are actually pretty good.

If you're talking about more out-of-distribution generalization, to the extent that you're within a bounded period, you cannot have open-ended continual learning. Liquid nodes are not continual-learning systems. They are more adaptive forms of computation because of the many gates, many feedback mechanisms, and many input-dependent parameters that they have in their mathematical equation.

What is the definition of continual learning that you're using there, which they don't satisfy?

Continual learning would be a system that is continuously receiving new data and retuning itself. It also changes the parameters of the system. Liquid neural networks have dynamics that are input-dependent. That means when new data comes in, the dynamics of the system behave according to those inputs, but the parameters of the system, like any other neural network, are fixed.

It's just because the format of every neuron, or every node, in a liquid neural network is like a differential equation. You would have different dynamics when you're operating. Dynamics are another dimension that you add to the number of parameters. It allows you to compress more information, to compress more knowledge. That's why, with smaller instances of the models, there's no free lunch, or there is no magic here.

With liquid neural networks, the axis of dynamics is something that we added to neural networks. Why? For being more adaptable. I'll give you a very tangible example here. Imagine you're driving and, all of a sudden, it starts raining. Depending on the form of the rain, your autonomous-driving system, which is taking control of the car, could react to those driving scenarios.

If you have a traditional neural network that hasn't seen that kind of environment, it might get biased because the rain hitting the camera is noise—a certain type of noise, or a certain type of adaptation of the input that is coming in. But the reality hasn't changed. The confounding variables of the whole environment haven't changed, right?

That's why liquid neural networks react to the input completely differently. They absorb the input and apply low-pass filtering on top of the input, so they are much more adaptable to those kinds of scenarios. That doesn't mean that they change the parameters of the system to be more adaptable, because there are 2 axes here: one is changing the parameters of the system, and the other is changing the dynamics of the system.

I would consider continual learning to be where you continuously change the parameters of the system as well. In liquid networks, we don't do that.

Gotcha. Okay, helpful. So, let's fast-forward to the present. You guys are now in the market, working with customers. One notable conversation I heard with one of your customers was with the CTO of Shopify on the Latent Space podcast, who had some very good things to say about you and your technology. I was struck by the fact that you've gone neutral in a sense.

What sort of jumped out at me in terms of your approach is that you've developed an architecture-search process where the promise to customers is not, “Hey, we developed this one paradigm.” An old calculus teacher used to say, “When all you have is a hammer, everything looks like a nail.” You're explicitly promising people that even though you came from the lineage of this particular kind of network, you're not just going to blindly apply it to their problem. Instead, you've created this higher abstraction, or more meta-level process, for searching through architecture space to find the thing that's going to work best for them.

The 2 notable details are, first, that the proxy metrics you found don't work that well. Just measuring perplexity or whatever isn't enough; you found you needed to go farther and test models on the actual downstream tasks that they're going to be asked to perform. Second, you have hardware in the loop—actual target hardware in the loop—to test the architecture subject to the very real, physical constraints of the robot, the sensor, the phone, or whatever it's going to be running on. Maybe I understand that the LFM model came out of that, but before we even get to the LFM, could you sketch out the range of different types of problems that we're putting into this architecture search? And then maybe also—since that implies, of course—what kind of hardware are we targeting, and what kinds of different architectures are winning for different kinds of problems under different kinds of constraints?

Absolutely. There's a system that we developed in-house. We call it Automated Foundation Model Design, or AFMD. That's a meta-learning system that puts hardware in the loop and then tries out many different operators with an evolutionary strategy.

The evolutionary strategy optimizes for a couple of things: memory consumption on that device, latency, and speed, with no sacrifice in quality. When we talk about quality, perplexity is not the measure. It's actually the downstream applications that we care about. It's also not just public benchmarks; we're talking about 100 different benchmarks. So, from a meta-learning perspective, the problem space becomes a very, very complex kind of problem.

Now, I'll tell you why we took this approach to designing an architecture. We wanted to remove all the human biases early on as we were building architectures. One of the things that we realized culturally at companies like this—and what I can tell you is that even at the largest foundation-model labs in the U.S. right now, Anthropic and OpenAI, there are a bunch of people coming from the science side—is that there are what I call the Avengers of the architectures, or the Avengers of post-training or, let's say, pre-training.

These groups of people usually consist of a very small set of people who are calling the shots: “Oh, you know what? You're going to tweak this portion of this architecture so that it performs better.” Why? Because in their personal experience, it has started working better. If you really think about it—and this is something that is broken in all the foundation-model labs—you cannot say that somebody has a fix for this.

But now the recursive self-improvement kind of process is actually fixing that, because people are finally realizing that you have to give it to the algorithms. You have to have Bitter Lesson people. You have to give it to a systematic process to actually find out what the true architecture is for the problems that you want to solve.

You can build a general-purpose computer. The insights that I shared with you in the format of the scaling laws of neural networks are coming out of our massive exploration of the space of architectures. The fact that, in the smaller category of models, some biases in the architecture help, while in the larger instances of the models you don't need to bias the systems—you can actually go with pure convolutions, pure transformers, or pure recurrences that are very simplified—is a result of that exploration.

You don't need to add any specialized treatment, like gating, there. You have gated delta networks, Mambas, Jambas, and many different variations of these architectures that are coming out. What you want to do, though, is to be completely unbiased.

Even we ourselves, on day 1, when we started Liquid, had one of the inventors of SSMs as our founding scientist, Jimmy Smith. He invented S5; he basically brought the parallel-scan idea to the SSM world. Then we had Stefano Massaroli and Michael Poli. They were designing the Hyena Hierarchy. There was a convolutional path going on. All of these are linear systems within the realm of linear systems.

Then we came out with these nonlinear, hyper-control-theory-enabled foundation models, which are liquid neural networks. There were a lot of big heads in the room, and you have to call the shots. Again, it's the same problem in all the labs: you have to figure this out.

We thought, “Okay, let's fundamentally solve this problem. Let's systematically, from first principles, put all the operators of interest—whatever we think could be an operator that can give rise to a general-purpose computer—into the system.” A liquid neural network on its own is a general-purpose computer. You can scale it; theoretically, you can get there.

So, you put all those equations and all those things into a unified theory. All the space of operators can be encapsulated inside these linear input-varying operators, or, in general, liquid input-varying operators. Input dependence is something that we've been talking about for the last 10 years. That's extremely important, and we see right now that input dependence is also extremely important in transformers. It comes naturally with the attention architecture as well, but in a very, very unstructured way. It's not as structured as what is done in RNNs, liquid neural networks, or SSMs.

Now we have various variants of convolutional operators and various variants of recurrent operators. Then there are the attention mechanisms themselves. There's grouped-query attention, the original transformer, and many other variations. Along with some of these variations of dynamical systems and various types of convolution, we also brought in the liquid computational blocks.

For example, we added gated, double-gated convolution, or a certain type of bias, onto different dynamical systems or different operators. We added those, and the space becomes something like 50 to 100 different operators. Then you want to build hybrid models so that you can reduce—basically, what's the goal here? The goal is to maximize computational efficiency without a loss of accuracy. That's the objective function of the search space.

We said, “Let's run all of our compute at this problem and try to see how the system is going to design this.” We started doing scaling laws on this. We went as small as a neural network with 10 million parameters and ran the scaling laws up to 72-billion-parameter models in these hybrid structures.

In the early days of Liquid, 2023 and 2024 were always about proving out, on a certain type of processor, what the most efficient type of architecture could be. It turned out that, when we put all of our biases aside—all the gating mechanisms that we were putting around operators—them being simplified was the key.

In Mamba-style architectures, you have a bunch of gating mechanisms. In gated delta networks, you have a bunch of operators. In linear attention, there are gated variants. Everybody is tweaking a certain parameter in the network by hand because, in their own experiments, they observe something.

It turns out that all of this has to go away if you want to get to the most efficient form of architecture. It became the double-gated convolution that came out of this massive search space—AFMD, the original system that we designed. This was one of the candidate architectures that came out, and it turned out to be very good on general-purpose computers, which are called CPUs.

On CPUs, because they have a special kind of structure, we tested different operators to find the generic structure and computational graph that would yield the most simplified architecture. There are no added, hand-tuned features anywhere; it's literally coming out of the systematic tests that we've done. This became the de facto architecture of the LFM2 structure that we announced.

While exploring these things, we observed many different candidates popping up. We tried to figure out, for example, for an NPU—a neural processing unit inside an AI PC powered by AMD, Qualcomm, or Intel—which of these variants would be a better neural architecture. That could give those hardware providers and silicon platforms a boost in the efficiency they unlock, the speed of computation and latency they get, and the memory footprint they get, while having a computational graph that doesn't sacrifice any form of quality.

So this was the whole thing: removing all the human bias with a systematic approach on 2 architectures, including our own biases. The only thing that actually remained in LFM2 on CPU computation is this double gating. As I told you, we have nested computation in liquid neural networks. Originally, this nested format of computation was something that became very interesting to include in these very simplified kinds of neural networks that we built, along with unstructured 1D convolutions as the layers of choice.

70% to 80% of our networks are structured by these gated convolutions that we have—double-gated convolutions. They're extremely simplified, and they replace attention. They reduce the computational complexity by a lot, reduce the memory footprint by a lot, and maximize the speed of computation by a lot, at scale as well. At the same time, in terms of quality, you've seen that some of these models are extremely competitive with the transformer-based alternatives. I think you asked a bunch of other follow-up questions as well, but I would pause here for any follow-ups.

Yeah, let's get into use cases in a minute. I'm definitely interested in that, and I also want to talk about the future of hardware and what your work implies for the future of hardware. But I think it would probably be helpful for a lot of people, including myself. Although this is something I've studied in some depth, I still would like to grok it better than I do.

I especially noticed this with Mamba when I went down that rabbit hole and became very excited about it. It seems that the trick that's played over and over again with these gating mechanisms is that we want the transformation that's done on the data to be input-dependent. So it's not enough to learn a transformation. We want to learn a transformation, but then also have something relatively simple—typically, the gate is relatively low-dimensional.

Sometimes it could just be a scalar that's applied to that transformation. It could obviously be more complicated than that, but it's some relatively simple mechanism that says, “For this learned transformation, here's how we're going to modify it given the input currently under consideration.” That seems to be great. It seems to unlock a tremendous amount. Help me understand more: is there anything you think I'm missing there, or can you help me deepen my intuition for why that is such a powerful and recurring theme in all these different architectures?

That's a liquid structure. When I say this input-dependent kind of thing, that's what's coming into Mamba. A year and a half before Mamba came out, we released a paper called Liquid-S4. If you just read the abstract of that paper, we were actually introducing, for the first time, this idea of input-dependent SSMs.

Basically, the idea there was: let's bring the fundamental building block that we found has a lot to do with representation learning—the capacity to learn this input-dependent element—and really bring that format of gating to neural architectures, to SSMs. Then we brought that to convolutions as well, with the other kinds of systems that we're designing today, including Liquid Foundation Models.

That kind of format of gating actually adds a lot. You should just think about it naturally; it makes a lot of sense. For a neural network, it's not about the forward pass of the neural network being adaptive. Once you're input-dependent, you have input dependence in the learning. From a learning theory perspective, all the magic happens in the backward pass. When you're computing the gradients backward, that input-dependent operator itself is going to represent itself.

So when you're learning from the data that you're seeing, you're also learning some sort of dynamics—input-dependent dynamics. That second axis that I was talking about is not just the number of parameters of the neural network; it's also the dynamics that the neural network actually learns. You have some representation of that dynamics being learned in the backward pass.

The complexity of that gate is going to make a huge change. It also turns out that the gate is extremely important in language modeling, especially when you have sequence models like RNNs, classical control systems, continuous-time RNNs, discretized RNNs like LSTMs, liquid neural networks, and linear versions of recurrent networks such as SSMs.

When you add this type of gating, you can improve the language capabilities as well. It seems that for discrete sequences, this gating also helps a lot. It's not just for continuous-time sequences, but also for discrete sequences, because I would consider language to be a discrete sequence of information. So you have that sequential dynamics being added by this gating.

It is a very right frame of mind to think about that kind of feature. For us, we think that this input dependence is one of the fundamental discoveries that we've made, and that we also learned from biology. It happens; it comes out of physics. If you just put the math together, you're going to realize, “Oh, what is special about this dynamical system is the fact that it has this nonlinear input dependence.” That's a kind of structure.

By biases, I mean that the complexity of the gating—even the existence of that gating—is a bias that you're adding to your system. Now, is it needed or not at scale? That's a big question mark. We're going to see, when neural networks have 100 trillion parameters, whether matrix multiplication is enough to really perform like a general-purpose computer and get to AGI, or not. That's a big question mark.

But we also need to put this discussion about architecture, and the obsession with architecture, in perspective. We have to consider the learning theories themselves. Right now, we're talking about different learning schemes that are emerging. For example, you can train a model with next-token prediction. You can train it in a world-modeling context. You can train it sequentially over a long-term horizon and with a long-term history.

There are so many ways you can construct the objective function of a learning algorithm itself. That would also contribute a lot to the learning process. In terms of architecture, I'll tell you that we found the purpose of architecture—and the most important application of architectural research—has been efficiency. It's about getting to efficient formats of computation without a loss of quality.

That's an extremely important thing, because you're talking about the resource-allocation problem right now. As the models are becoming bigger and the demand for AI is growing exponentially, you want to have more efficient versions of these systems to actually run them at scale. Otherwise, how can we provide access to AI to everyone? We're seeing instances of this already: the larger labs are wiping out the compute on the planet. Why? Because they have to train and then host these models for all of us.

So efficiency becomes a fundamental property of the model architectures that we're researching. If you want to get into architectures that enable the next generation of neural networks—sorry, the next generation of intelligence systems—in a way that allows them to become like the human brain, which performs computation with 20 watts of power and is already an AGI that does that with 20 watts, that's a massive exploration that goes beyond architecture.

It goes into memory research. It goes into learning algorithms. It goes into data. It goes into prior research. Looking at it from a statistical-learning perspective becomes important, and then learning theory itself—the limitations of learning theory—also gets imposed on today's learning systems. The definition of learning theory itself is actually broken at scale.

Right now, everything is i.i.d. We're building averaging machines. You've seen the quality of writing and the sequencing that comes out of AI systems because of the fact that we haven't gotten there yet. Maybe, to some extent, multi-agent systems have been becoming the solution, and test-time scaling became a solution for the caveats we're seeing from systems that are just learning with autoregressive modeling as a pretraining method.

But I think innovations will be needed not just in architecture, but in the whole thing as a whole. You have data, algorithms, models, and learning algorithms all together. They can design the future—the ultimate Holy Grail in this space, basically.

Yeah, that's very interesting. So basically, your perspective is that as we get into the recursive self-improvement era, the real advances will come more from new learning paradigms and new objectives, and then those advances will be made efficient through architectural optimization. But the architecture comes after the paradigm question: what exactly are we learning, and how are we giving a signal?

It is one component of it. Looking at it from a data problem, data representation is one axis, architecture is another axis, and the other axis would be the learning algorithms themselves.

Then we get into that recursive self-improvement, which is kind of the continual-learning characteristic of what the new system is—the fancy name for the continual-learning kind of research that has been happening for probably more than 4 decades now. That's how I would characterize the whole space. You cannot just look at architecture in isolation as the fundamental thing that actually changes everything.

Yeah, you remind me a little bit of Ali Behrouz's “Nested Learning: The Illusion of Deep Learning Architectures.” If we have time, maybe we can touch on nested learning. But let's stay focused on your work for the moment, at least.

So, in LFM-2, this is the result of this architecture search, and the surprisingly simple thing that comes back is some reduced but still critical number of attention layers. The other layers are—and we've seen this kind of thing with SSM-attention hybrids as well—but I think the surprising revelation from the result of this search process is that the non-attention layers can actually be extremely simple, as long as they have some gating. So you've got the gate, and then you've got just a really simple convolution that, I think, considers only a very short span of tokens, right?

And so, if we were going to update the headline “Attention Is All You Need” from however many years ago now, we would maybe say attention is something that you really do still need, at least at certain scale. But you also need gating on your other layers, and you don't actually need anything super crazy, fancy, or sophisticated—the state-space model and all that kind of stuff. It turns out that keeping the gate is actually the part that really drives the most value, and then you can have a really simple mechanism behind the gate. You can have 70% of that and 30% attention, and, subject to resource constraints, that ended up being the winning formula. Am I getting anything wrong there?

No, you're touching on the right things. Basically, it's the regime you're operating in and the goal of your system. Are you trying to build superintelligence? Are you trying to build the most powerful version of the AI system? You need the most unbiased version of an algorithm.

Attention is an extremely rich, unbiased format of algorithms. Even if the computational complexity of attention is n to the power 2, maybe we really need n to the power 2 to get to that kind of level. Maybe we even need more complex architectures. We've always tried to reduce the complexity of architectures for the sheer purpose of the fact that we are resource-constrained as humanity as a whole. We are resource-constrained right now.

I would say the discoveries that we have right now show that there is a gradient on architecture that you can follow as you scale models. The gradient that you're following is the fact that, for smaller and specialized models, you can put in as many biases—like these gating mechanisms that you're bringing in—and you can play around with as many operators of interest in your computational graph. It is going to work and give you some sort of a boost.

If you're really maximizing for linear time complexity—you want to implement linear attention systems, just the fastest kind, and speed is so important that you're willing to sacrifice a little bit of quality—you can bring in linearity, and the whole system could be linear. You don't even need some of those hybrids. It just boosts that accuracy because, as we see, the O(n²), basically the computational complexity, at a certain level and at a certain scale is needed for us to really get to the performance that you want.

The larger the network becomes, the more unstructured you can make it. That's kind of the learning from that whole algorithmic approach that we started designing neural architectures.

Yeah, very interesting. Okay, let's look at the other end of the spectrum, then. As you work with customers, what are some interesting examples of when, given resource constraints and the narrowness of the domain of interest, other kinds of biases are actually winning in the architecture search process?

Great question. For example, if you go to biology and you want to model sequential data, we're talking about DNA data. From a vocabulary perspective, DNA data is very limited, right? It's not like language, which has a maximum amount of vocabulary and so on. In DNA language, the vocabulary is very simplified, but the lengths of the sequences you have to process are, let's say, for a human being or maybe for a bacterium, somewhere between 1 and 100 billion sequence elements.

For that long-context kind of thing that you want to perform, when you do not have that large of a vocabulary, you don't need attention. You can actually run on these kinds of data with pure convolutions, pure SSMs, pure liquid neural networks—systems like recurrent networks and parallelized versions of these recurrences—and linear attention for extremely long contexts. You cannot do it any other way. The reason is that the context has become so large that the quadratic cost of attention just kicks in.

On biological data, you would want to have some sort of structure there. There are places like video modeling and so on where you might want to have different architectures and even different learning algorithms. You've seen the success of diffusion there, right? Diffusion is also, I would say, a format of a prior that you put on a certain architecture. Still, there's a debate between diffusion and learning algorithms: is diffusion actually part of the architecture? You can make that connection, and you can make that distinction as well.

On video and scene understanding, probably elements of diffusion would be needed. Some people still believe that with autoregressive modeling, you could actually get there anyway, but we'll see if that stays true.

When we're talking about an audio signal as well—if you're talking about audio alone, and language is not part of the whole thing, just pure voices, voice-to-voice, let's say noise-to-signal kinds of tasks, signal-to-signal—these are places where recurrent neural networks are still very, very powerful. They're extremely powerful, especially in the low-data regime.

Models that have a lot more biases in them, in this smaller regime where you do not have that much data, are where you can bring a lot of value. Recurrent neural networks and biases in architecture can help you in the low-data regime to fill that gap with the feedback mechanisms that they have.

The more complex the architectures, the more you would be able to handle. The closer the architecture is to the dynamics of the data set you're trying to solve, the better of a learning system you're building. In physics modeling, when you're talking about a class of models like physics-informed neural networks, these are, again, another architecture we're talking about here. They would be really good for physical simulations.

This is the other side of the spectrum that I'm talking about: the properties of the data, the context lengths, and all the other considerations that I mentioned would change the architecture by a lot. Then, again, you want to scale this to the largest regime and make it unbiased. Again, a transformer would be able at scale to beat this, but at smaller scale, transformers would not be able to beat any of the other formats of dynamical systems that we describe.

And how about the differential-equation-inspired original liquid neural networks? It's clear that the worm is running on a very small amount of watts. In practice, do you encounter things that are so resource-constrained that you have to go to these extremely specific or extremely biased architectures to actually deploy today?

Yeah, 100%. Think about places where latency matters—you know, computation has to happen in microseconds. You cannot really afford to have larger computational complexity. You want to have the simplest type of system that handles that, and there you can have adaptive systems. You can have many different formats of systems.

The other thing is simulation as a whole. As I mentioned, in physics you have physical data coming in, and you want to build a digital twin of, let's say, a physical or chemical reaction that happens at a factory. In those kinds of places, you would go toward an extremely biased, and maybe even just differential-equation-based, model. Liquid neural networks today are getting applied to many, many different applications.

I see that because the original repository from years ago is still open source, people are still building predictive machine-learning models on sequential data, physical data, or sensor data. That makes sense because these systems operate in continuous time, and it makes sense to have a continuous-time dynamical system to apply to modeling that kind of behavior. Today, with larger instances like the Claudes of the world, you could also direct your agents to try out a bunch of things in an auto-research format. You could direct them: “Go pick the neural networks that would be the best fit for these types of datasets, and here is the space of possibilities you want to explore.” Based on the biases you have, machine-learning people are now mostly orchestrating these automated agent pipelines as we are building them.

That’s what’s happening at Liquid AI itself. We are trying to contribute to open source, and we are open-sourcing some of the instances that are coming out of our search spaces, like LFM-2. LFM-3, again, is going to have the objective that each next generation should always beat the previous generation on the criteria that we care about, without sacrificing quality. That’s extremely important for us. Then, for example, we decide how large of a neural network we want to make based on the neural networks that we have, and this becomes the open-source version of our models.

Then we work very closely with silicon companies. We work with AMD and Qualcomm, and what we do with them is try to understand the silicon roadmap—the hardware roadmap—that they have. Based on that roadmap, we try to inform what they have to do even in the next generation of their ASICs. When we have this understanding of the algorithmic aspects of intelligence and the variety of things that they could support, we can dramatically reduce the cost or satisfy the constraints of the use cases they want to enable.

These are the considerations that you have to make. We can even build specific foundation-model graphs for them. Those are the kinds of projects that we do with semiconductor companies.

In terms of the commercial side of things, we also take the models and apply them in many different places. I call this a category in which Liquid AI is very enabled: device foundation models. Anything—all the processors that are outside of data centers—we try to apply our technology to those kinds of places.

Inside data centers, we apply it to constrained use cases: low-latency and ultra-low-latency applications of AI, extremely long sequences, and situations where you want a very small memory footprint for your AI system. You want a cost-efficient implementation of the foundation models that you have. We contribute to memory, speed, and latency without sacrificing quality or cost. These are the elements that define use cases where Liquid can come in, in terms of access to our technology.

With any new generation of our architecture that comes out of our architecture searches, or with new modalities that we explore, we check a range of applications. At Shopify, for example, we are exploring recommendations, search, product-catalog understanding, and multimodal systems very, very well. Our models are in production at Shopify right now, and they are improving the quality of the click-through rates and the internal criteria that Shopify cares about.

Our models also go inside cars. One of the use cases we enable is in-car intelligence. Recently, we signed a contract—actually, a historical one—with Mercedes-Benz, where our models are going to power the audio and visual elements inside the car. Whenever you want to talk to your car, basically, the new voice would come out of a Liquid foundation model. We control that with a model that gives you the quality of the best models you’ve seen so far, such as the audio models you’ve seen all around, but at the same time it is 600 megabytes. It can actually fit inside the smallest processor in that car.

That changes the game because you’re enabling local AI at scale, in places that matter—in cars, for example, and on mobile phones. There are a lot of devices where this can be extremely useful. You’re talking about billions of devices in the world, and mobile devices can be extremely useful. That market itself, by the way—the mobile business alone—is a $500 billion market. It is absolutely insane, and it is as big as the data-center market, so you can imagine there’s a parallel here to be made. The efficient market and the constrained-intelligence market are something that Liquid AI is going after.

Then there’s the laptop market, wearables, robotics, manufacturing, IoT systems—anywhere that we have a processor inside a system, Liquid can bring intelligence on top of that. The aspirational goal that I have is that we really build an intelligence layer on top of the diverse formats of hardware and processors that are available in the world. That would be something that I would want to have.

I’m glad you said that statistic. I had that noted to make a point of at some point, because it’s worth repeating. The global annual smartphone market is about $500 billion. For all of the hundreds of billions going into the data-center buildout, it will get bigger than the smartphone market, but it’s only now getting to the scale of—and getting bigger than—the annual smartphone market.

That means there’s a lot of dark compute out there from an intelligence perspective that is nowhere close to being used at its maximum. That does not include the laptop market. So there’s $1 trillion, generally speaking, worth of compute going out into the world on an annual basis, sitting on people’s desks and in their pockets. There’s a massive substrate there to—

Take advantage of it.

And we need to do it, because we don’t have enough energy to host it. I feel like we’re realizing right now how difficult it is to really power this format of intelligence at scale. We really have to work smart. We cannot just use the simplest format for every use case.

For more constrained use cases—let’s say you have a one-shot predictive task or a data-extraction task—you don’t want to call the fanciest type of intelligence, like a frontier-level model, to perform data extraction for you. You can have a variety of different intelligence systems doing many different things. Then, for the most sophisticated problems in the world, you can go to the most sophisticated AI systems that exist in the cloud.

It’s inevitable that the processor world outside of data centers should get enabled as we speak. I think there are actually some talks coming out, and there’s a lag among Silicon Valley investors who are just saying, “Oh, okay, now it makes sense.” That gap is getting answered right now, and I think we’re in exciting times for efficient AI in general.

Okay, so one challenge with this other $1 trillion worth of compute that’s going out into the world is that it’s super heterogeneous, right? There are many, many different devices and different chips. How much of the work that you’re doing is about trying to get closer to optimal use of that compute?

Obviously, we talked about scans earlier, and the transformers in data centers—that’s been, not that I’m sure there are no more optimizations to be made, but a ton has gone into making sure that you can get as close to maximum throughput as possible. When you do an architecture search targeting some random cellphone or sensor in a factory somewhere, how much do you have to work on kernels and scans as a foundation even to be able to do that search?

Yeah. Well, that’s one of the different layers of abstraction. We’re testing out AI to see how good it can get on the kernel-design side of things. For the things that it has seen in the open source—the kernels that are available in the open source and the GPU structures that it has seen out there—we can do a lot. But not the NPUs: the most hidden types of NPUs, whose IP is not disclosed. Those require another architecture search because we don’t have that knowledge.

I would say kernel design is at a level where we can actually start automating it. You can definitely have loops where you can juice out the best post hoc optimization. Some of the things that we’ve worked on and talked about so far have been optimizations that we do before the design of a foundation model—all sorts of considerations before we start pre-training a foundation model.

The approach that Liquid takes before getting into pre-training the model is that you run this massive architecture search before coming up with the hardware in the loop.

But then right after you design the architecture, there are all these post hoc things, like quantization-aware training, changing the bits of the system, going into the kernel level, and trying to juice even more out of the system. All those post hoc things are also something that you can automate, automatically orchestrating them with a kernel engineer who does and performs those optimizations. We do those post hoc—we do that as a post hoc optimization step.

That being said, we have the capability in-house to define architecture and define operators at the kernel level as well. For example, you could say matrix multiplication has probably 100 ways that you can actually structure it in terms of kernels: with caching, without caching, how you distribute the workload between the CPU and GPU, and how you perform the entire forward pass. As you perform all of these optimizations, they could also, in principle, become part of that massive search space.

These are the kinds of questions that some of the silicon partners are asking us: How can we, even before designing a foundation model, figure out the computation graph first, but without having the guarantees that, when the training is finished—which is a costly process, since multiple millions of dollars have to go into the full training of a neural network—even if the network is small, you want to have all sorts of inference optimizations before and post hoc, like quantization-aware optimizations? In principle, we can definitely launch something like that as part of our search.

So what do you think hardware makers should be doing differently? What are you telling them to prioritize to help us realize this more efficient, distributed AI future?

I feel like with every iteration of a software technology, the level of abstraction that the hardware providers should be building for it is coming up. For example, we've been talking about the CUDA moat for a long time now. There is no CUDA moat anymore. Why? Because if you look at AMD, there are agents at the kernel level now that can automate that stack.

The stack that hardware companies should start working on is the intelligence layer, like what NVIDIA is doing with its Nemotron project. If you see how this project is actually quite successful, a couple of billion dollars probably went into the design of these Nemotron projects. If you really look at what it does for NVIDIA, it's basically building an intelligence layer on top of NVIDIA's compute so that you have an easier barrier to entry as an enterprise to buy NVIDIA solutions.

The sales of the solution are always around those things. I'm just talking about the enterprise business of these hardware providers. If they want to sell more hardware, they're going to have to move up their stack from kernel-level optimizations. Those are kind of post hoc things, and they have to get into the intelligence layer. Does that mean they have to become a foundation-model company to some extent? Yes. They have to be able to train that intelligence layer themselves, and NVIDIA is a successful example of how this is actually paying dividends for them by building the Nemotron project.

If you look at the other hardware companies, they haven't done this yet. They have started strictly optimizing their models for the open-source architectures that are out there. That's also extremely valuable—you have to do that. That's a given. But again, if you want to create differentiated value and you want to be successful, I would say you've got to be able to bring the stack to the intelligence layer. That intelligence layer should naturally fit on top of your hardware.

If your hardware has limitations, let's say, compared with your competition, software can always help to give you that juice. For example, if you're talking about maximum token speed on an AI PC between an Intel computer, an AMD computer, and a Qualcomm computer, they might have their own trade-offs. The company that's going to have an edge is the company that actually owns that efficient intelligence layer on top of its hardware, so that it can say, at the end of the day, you're going to run tokens on top of this system.

It doesn't matter whether the bandwidth of my CPU is a little bit lower or higher than yours, or whether the amount of memory that I'm actually having in there is lower or higher. With the intelligence layer and software-optimization techniques, I can actually get myself to the place where I'm the winner here. All of my laptops or all of my PC solutions that I'm building as a hardware provider are naturally coming with an intelligence layer on top of them.

NVIDIA is entering this game heavily as a competitor in the CPU space now. You're seeing devices like Google actually enter with Android, with the Android ecosystem that they have. They're replacing Chromebooks with Android laptops. This is what they're putting out there—I think it's called Aluminium OS or something like that. I don't actually know exactly what it is. Meta is going to come with a sort of devices themselves as well.

All the processor builders need to get a little bit closer to that intelligence layer and try to leverage software and efficiency considerations when they're planning ahead—where things are going and why that intelligence layer is important. Right now, AI is something that everybody wants. All those applications you want to run on silicon, you want to run them on a base of intelligence that's already provided to you.

You want to build harnesses, and all sorts of harnesses are basically applications that you can build for solving a certain problem in the traditional sense of an application. That market can get enabled on top of chips if you have a very nice stack that enables intelligence on top of what you have.

The other thing that I would say is that silicon is always—if you're going outside of the data center, I mean—there's a massive diversity of silicon, but the resources are constrained no matter what. You don't have that much power, so you've got to be very careful about what sorts of applications are the most important uses of your hardware, medium, and platform.

If you already have an intelligence layer, you could identify when it's time to sunset, basically, a set of devices. For example, I strongly believe—I think glasses could be a form factor of computing later on, like devices. I think glasses would be a very interesting kind of medium, and I think somebody's eventually going to get it right. It might actually be a replacement for laptops, but is it? We need to really figure out what it is.

If you really want to define a market and define where the field is going from a hardware perspective, I would say you've got to up your software game by a lot and get more hands-on, investing a little bit more energy and capital into the R&D side of things on the foundation-model side. That's the new base of software, inevitably.

So does that imply a future where we have just a lot of vertical integration and a lot of coupling? The model will come with the hardware that I buy, and it may not be so swappable in the future because the model is heavily optimized for the hardware and vice versa, such that these things are not so modular in the future as they are today?

Or why do you want to change the model? It brings you to that choice kind of question. You're given basically a default. Now, you want to switch this thing, by all means, but why do you want to switch it?

If the model—the intelligence layer that's in there—is not fixed, it's an adaptable system. It is, let's say, a self-improving system, with a call to your data sets and stuff. You can actually have a platform that does, let's say, full fine-tuning of that system.

It is enabled, and there is not just one model that you can load into the system. There's not just one cloud model or one on-device model that you're going to use. You've got to be able to orchestrate between many different instantiations of this model to be able to build applications. We're talking about a model class that would actually sit on top of, let's say, hardware on a laptop, for example.

I would say you need to have that. If the default is just giving you that efficiency and it's ready to go, that's the choice. That's something that I think NVIDIA is trying to propose in enterprises when they go and sell the Nemotron project.

It makes a lot of sense because you already have a product. You already have a multimodal model loaded on top of, let's say, the PC that you bought. Why should I switch? They have already done all sorts of optimizations for me, and it's running extremely fast. Why do I need to change it? The model itself is tunable; I can use their Megatron framework to actually tune the model.

If you don't want to do that and you want to choose another model to host, as I said, this has to be a given. Your hardware should already be optimizing for the entire open-source ecosystem and the models that are available. At the same time, it would give you an advantage, to yourself and to your customers.

Okay, maybe in the last few minutes, how about a practical application of this? You have a blog post on “Local Co-Work: No Cloud, No Waiting—Tool-Calling Agents on Consumer Hardware with LFM2-24B-A2B.” I think people know that’s a mixture of experts: 2 billion active out of 24 billion total parameters. Let’s say I want to make that a part of my life. I’m interested in how you would coach me on setting this up today.

For reference, I sometimes try to make the transcript of the podcast something that I can feed to my agent. You can think of this as partly coaching me and partly coaching my agent. I’ve got this deep-context database that is the last 5 years of my digital output. This podcast will be recorded and transcribed, so everything you said and everything I said will be in there and searchable. It’s got all my email and Slack messages and everything.

Okay, cool. Now I’ve got Claude on my desktop, which can call tools locally to get data back, but then it sends all the results to the cloud to decide which of those results are actually the right ones to be looking at. So far, I’ve been okay with that. The benefits are certainly worth whatever risk I’m taking, I feel, but I would maybe love to run that data through a local model first so that I don’t have to send all my data to the cloud every time.

Notably, what gets filtered through, I’m probably still going to end up running through a foundation model, so it’s not going to entirely skip the cloud. But I’d also like to save some tokens, because I’m going to want to use Fable whenever I get it back for whatever it’s most appropriately used for. So how do I get really good performance on my local computer with this model? Do I need to be doing fine-tuning? Do I need a distillation strategy? How do I actually take the base model that you have and recover as much of, let’s say, Opus or even Fable performance in terms of searching through and understanding my data as I possibly can?

How far will I be able to push that process? How close to parity with frontier models can I get? Tell me everything I need to know, and I’ll go have my agent do it.

Yeah. Well, that’s a great question. Obviously, the local co-work, as it stands today, is just to open minds: this type of application can enable a whole class of local agents. Basically, you have a local computer with an orchestrator. If something is so complicated, it should be able to send it to the cloud and fetch the answers for you. If it’s not sensitive, you have, let’s say, smaller models that are PII models. It should be able to use those models to filter out all the personally identifiable information and send it to the cloud for you.

You don’t even need to see these models. These models should be running in the background. The model that you have to tune is that orchestrator, which should be able to route between many different services, or even smaller specialized models that are doing things, and some of the cloud models that are out there. That router is the computer. That’s the local computer. When you open your laptop, it should just be that, and then you start working with all the services you want to have.

It’s the same way that you communicate with your assistant. It should be in the format of an assistant that does all sorts of those jobs. It’s especially weird to have the user interface just be that and not see all the file formats and everything you have to do, but you’ve got to get used to it.

Now, Claude Code, even as far as an IDE, is at a quality where it does all of that stuff for you. Is the 24B off-the-shelf model going to be there in terms of quality? No, it’s not. None of the local models today are there. With the local models today, you’ve got to fine-tune them. You’ve got to get them specialized for the things that you want to do, with proper explanations for your cloud agent.

Is Claude able to go out there and actually build this model for you—do the fine-tuning and get it to that place? No, because today, even Fable-level models would not be able to do that. First of all, you wouldn’t have access to that, because Anthropic actually has access to the auto-tune and automated model-generating-models kind of platform.

There are companies, and also ourselves, that are building platforms that enable you to do fine-tuning of this whole thing. That would cost you between tens of dollars and, let’s say, low thousands of dollars to actually get you to cloud quality, with all the checks for a production-quality kind of model. It’s not going to cost you tens of thousands of dollars. It’s going to be between tens of dollars and low thousands of dollars. That’s the scale that we’re thinking about.

Again, efficiency here matters a lot. We want this fine-tuning to happen on the compute that you already have. If you don’t have it, you would actually put it in a secure and proper kind of data center and host it with the providers, basically. You would be able to get to that quality of the models.

Hopefully, as we go forward in the next few months, we’re going to announce some platforms that you can hook directly into your terminal. You don’t do anything. You just say, “Hey, go call this platform for fine-tuning this thing.” This would give you a production-grade foundation model that you can deploy for yourself, either fine-tuned on that data or depending on the use case and what you want to do.

It doesn’t even have to see that data. It can synthetically generate data on its own and actually train the model to be the perfect, reliable tool caller. It understands what its shortcomings are, and it can actually go away and deliver it to some other places.

Yeah, I would say we’re extremely close to getting to that.

Wait for your platform to be ready. I don’t have to DIY it—that’s the message.

This has been brilliant. Maybe one last question, and then I’ll just give you the floor to close however you’d like. How far do you think this goes in terms of the miniaturization of intelligence, if you will? One intuition would be that the biological world we see is maybe on some sort of Pareto frontier already, and so the watts that go into our brain are maybe getting us close to the maximum that we could get for that amount of power. But maybe you have a different intuition.

What would you expect in terms of the upper limits of intelligence that I could have on my phone or on my laptop as we really get to the physical limits of the technology?

When you think about intelligence, the way I look at it right now is that transformer-based networks and also our type of, let’s say, current architectures—the landscape of architectures that are available—gave us, with scale, in-context learning capability. The thing that actually emerged from next-token prediction—you see, the word “emerged” is important—is intelligence. Intelligence, for me, is an emergent property.

If you want to miniaturize intelligence and bring it to the physical world, I don’t believe that, with the current set of algorithms, we would be able to get close to the intelligence per watt that the human brain is providing. You’re not going to get there. Why? Because I believe that, with the human brain, we also have to consider the amount of energy that went into the design of humans as a whole. Biological evolution is a very, very long process.

A lot of the pretraining process is what people talk about: “Oh, you’re going to see the entire world.” Humans do not need to see the entire internet to be able to reason about something. Humans have gone through years of evolution, so I would attribute a lot of it to the evolutionary aspect of things.

But again, what I would say is that human intelligence came with multiple mechanisms of in-context learning. When our current AI systems do in-context learning, they learn a vague representation of one algorithm, which is least squares. It’s basically least squares. What they figured out is basically gradient descent in a mushy way. For some use cases, with some examples, you would be able to make the system understand and give you the next example.

These are kind of beautiful properties. This is intelligence. This is what I would call an emergent property of these systems. You set the algorithm to be next-token prediction, and you get a vague version of gradient descent and in-context learning capabilities. For humans, you don’t have only emergent gradient descent. You can learn by examples, but you can also do reinforcement learning. You can run simulations in your head. You can do all sorts of algorithms. You can do Bayesian statistics in your brain.

You see what I’m saying? So you have a diverse set of algorithms emerging from the way that humans actually got designed and got intelligence. For me, I believe if you start optimizing for those—if you try to force your way into the system to become a reinforcement-learning learner, learning from trajectories—that’s not the right way to actually get to an emergent property of intelligence.

I believe the form of intelligence that is going to be miniaturized—being able to get the maximum amount of intelligence in the smallest amount of time, like the human brain—we need to come up with these inductive biases, these foundational kinds of algorithms, of which next-token prediction is one. What else do we have to do at the beginning of the design of these systems so that reinforcement learning and curiosity-driven intelligence emerge with a limited amount of energy from the final system that actually comes out of it?

I would say that emergent property is something that we really need to research a lot more in that direction, along with multiple variations of ways of learning that would enable the next generation of artificial intelligence.

I think that could be a great note to end on, but let me just give you one more opportunity. Anything else that I didn’t touch on, or anything else you would want to leave people with before we break?

No, I think we covered a good lot of information. I think we went into some things that we don’t usually talk about now. In my current role as the CEO of the company, I get to talk a lot more about the business opportunities and the market aspects of intelligence and stuff, but at the core, we are scientists, and we are really pushing the boundaries. We are actively thinking about it day to day.

You can imagine that even right now, while we’re talking, I’m already in the back end. I’m actually training some stuff, and I really don’t want to get away from that mentality, because I think everybody can actually now build. I think the times couldn’t be better. The agents are pretty amazing at giving you this opportunity to really do frontier research. If you’re curious about something, it’s just that we need to find people who are a lot more curious and improve curiosity in people as well, to really get to curiosity instead of just fear of different things.

I’m a techno-optimist. You’ve probably observed that. I usually talk about technology in a very positive light, and one thing that I always say is that we need to get to the level where people are as curious as scientists were from day one. What gets us into science? The purpose of science has always been satisfying our own curiosity and understanding the world around us. That’s the purpose of science.

With the current superpowers that are given to us by these AI systems, I think everybody could actually start contributing to understanding the world around us, generating more value, and satisfying our own curiosity. But it also requires a certain degree of rebasing our biases around what is work, what is not, what automation is, and what an army of agents working for me is. What does that definition even look like?

It requires a change in culture and a change in the way of thinking about this—not just for individuals, but also for enterprises, where we are dealing with this as well at Liquid AI. I think that change is something that I’m looking forward to, and I think this is something that’s going to happen. The pace of it is extremely fast, but I think we have to do it one way or another.

I love that. I consider myself extremely fortunate to live, to a really remarkable degree, a curiosity-driven life these days, and that’s one candidate for a positive vision for the future that I think should inspire a lot of people—especially because, as you note, there’s an opportunity to move into that phase already with the AI systems that we have.

This has been excellent. I really enjoyed it. Thank you so much for the time. Ramin Hasani, co-founder and CEO of Liquid AI, thank you for being part of The Cognitive Revolution.

Thank you.