Tensor Logic“统一”AI范式 [Pedro Domingos]

TensorLogic unifies not just symbolic AI and deep learning; it also unifies things like kernel machines and graphical models. I'm Pedro Domingos. I'm a professor of computer science at the University of Washington in Seattle and a longtime machine learning researcher. My dream from my PhD onward has always been to unify all the different paradigms of AI into a single one.

My PhD unified 2 of them. My best-known research unifies a couple of others. I wrote this book that turned into a big bestseller, surprisingly, called The Master Algorithm, which is precisely about this goal and where we are toward that goal. My latest work, which this podcast will talk about, is a new language called TensorLogic that, I would say, for the first time brings this dream of a unified representation, a unified solution to AI within reach.

Tim Scarfe

I can set the temperature of GPT to 0, and it still hallucinates. And I can have a poor deductive system that hallucinates all kinds of things. So, to me, those are separate problems.

No, very good. Precisely, the problem, or one of the problems, with GPT is that it hallucinates even when you set the temperature to 0. What the hell, right? I want to have a mode—

Tim Scarfe

Yeah, yeah, yeah.

Right? Not I, but every Fortune 500 company, if it's going to use AI, needs to have a mode where the logic of the business is simply obeyed, the security isn't violated, the customer doesn't get lied to, et cetera. We've got to have that, or they will not take off, right? Transformers can do that. TensorLogic can do that precisely because, in the reasoning mode that I just described, if you set the temperature to 0, it does purely deductive reasoning. By the way, the temperature can be different for each rule.

There is this thing called predicate invention, which is discovering new predicates, discovering new relations that are not in the data but explain it better. I would say that, in some sense, discovering representations like that is the key problem in AI; it is the holy grail.

What was Turing's achievement that we now take for granted? Turing's achievement, for which he is deservedly famous, was to postulate this notion of a universal machine. The amazing thing about computers is that they're universal machines, which in his time was a completely counterintuitive notion. What do you mean, a machine that can do everything?

The typewriter can type; the sewing machine can sew. You're telling me there's a machine that can type with one hand and sew with the other? What are you talking about? That's the genius, right? So, the first step is to have this property of having a machine that can do anything. What we're missing to be able to do what the universe and evolution do is universal induction. What is the Turing machine equivalent for induction, for learning? That's what I'm after.

Tim Scarfe

Professor Pedro Domingos, it's amazing to have you back on MLST. I've lost count of how many times we've had you on the show now, so it's amazing to have you back. The main reason that we've invited you today is that you've just released a brand-new paper called TensorLogic: The Language of AI. You said fields take off when they find their language. You gave the example of calculus in physics and Boolean logic when designing circuits. What's the idea behind this paper?

TensorLogic, in many ways, is the goal that I've been working toward my entire professional life because I really do strongly believe that a field cannot take off until it has really found its language. TensorLogic, I believe, is the first language that really has all the key properties that you need in the language of AI.

For example, it has automated reasoning right out of the box, like Prolog has. The classical languages had a number of things that we just took for granted. Transparent and reliable reasoning—you didn't even have to worry about it; it was just already available. At the same time, you don't have that in PyTorch at all. You have all these hacks to try to do reasoning on top of it.

At the same time, Lisps and Prologs never had automatic differentiation—the ability to learn. One of the beauties of the current moment, in many ways, is that you barely have to look at most papers; people barely talk about the learning because it's already implemented under the hood. So, you want that as well, right?

You want scalability on GPUs; that's the other thing that things like PyTorch, TensorFlow, and whatnot give you. There was no language before that had all of these, and there are a number of others, but these may be some of the key ones. TensorLogic is basically a language which, as the name implies, is a marriage—a very deep unification—not just some superficial combination of the tensor algebra that deep networks are all built out of and the logic programming that symbolic AI is built out of.

There's only 1 construct in TensorLogic, and it's the tensor equation. You can do everything with tensor equations.

Tim Scarfe

Are you saying that there's only 1 language of AI? Certainly, in some fields, like physics—you gave the example of calculus—almost all of physics involves quite a bit of calculus. There are other fields where, actually, there are multiple languages that play almost equal roles. So, I'm wondering: do you think TensorLogic is going to be 85 or 90-plus percent of the way that we should be talking about and thinking about AI, or will it be a mixture of different languages?

That is a very good question. In fact, we know very well in computer science that there's no one programming language that's better for everything. There's just people who think it is. Everybody has their favorite language that they believe is the universal solvent, but it never really is. We know also, for fundamental reasons—going back to Shannon and whatnot—that there is no language that is the most pithy for anything you might want to say.

Having said that, physics is a good example because calculus is so fundamental. Feynman famously said that he thought in calculus. The thing that I found with TensorLogic is that I don't know how much of AI it's going to be or how much it should be, but what I have found, in many ways to my surprise, is that TensorLogic is more than just a programming language. It really, I think, captures the fundamentals of what you need in AI in a way that, going in, I didn't even think was possible.

All of tensor algebra can be reduced to this operation which, going back to physics, is called the Einstein summation. Einstein summation was something that was introduced by Einstein when he was working on relativity and got tired of writing summation signs. It was all about tensors. General relativity is all about tensors. He jokingly called it his great contribution to mathematics.

But the bottom line—and there's this great paper, or blog post, by Tim Rocktäschel saying, “Einstein Is All You Need”—is that you can do all of deep learning with just einsum. All of the matrix multiplications and tensor products, all of that, are instances of einsum.

On the one hand, in symbolic AI, it's all about rules. TensorLogic is based on this, to me, gobsmacking observation that an einsum and a rule in logic programming are the same thing. They are actually the same thing. The only difference is that one is operating on real numbers and the other one is operating on Booleans, but that's just a different atomic data type.

To summarize, at this point I think that it would be crazy not to do these things with TensorLogic.

There may be other, better things coming later, but at this point I would say TensorLogic is probably better for what people are doing across the board. But hey, that’s me. I may be a little biased.

Tim Scarfe

First of all, shout-out to Tim Rocktäschel. I read that blog post from 2018 earlier, the one you were referring to. But I suppose the thought occurs that, if it is mostly about einsum, you might make the argument: Why do we need an abstraction when we already have a great abstraction in einsum? Folks can now use PyTorch and JAX. What exactly does your abstraction allow them to do that they can’t do with PyTorch?

No, very good. It does several things. First of all—and this is going to be in increasing order of importance—the syntax of einsum in these languages—there’s also this package called einops—is incredibly clunky. So, at a very basic level, TensorLogic is just a much pithier, more compact, easier-to-write-and-understand way to write einsum expressions.

Physicists and mathematicians are famous for liking to say that a good notation is half the battle. This might not seem like a big deal, but my experience is that you can just think better and faster once you have this notation rather than this funky procedure call with these indices, these arrows, and these arguments. It’s a nightmare. The syntax of TensorLogic is especially natural: You write an einsum the way you would write a rule. There’s a tensor equation with a tensor on the left-hand side and a join of tensors on the right-hand side.

Another very important aspect—and one that I think could prove decisive—is that people don’t use einsum much because it’s not very efficient. Under the hood, it’s not as efficient as it could be. Sometimes I think, “This could be done so much better,” right? But I’ve done some programming of this, and I wound up not using einsum because it’s so slow and clunky. All of that can be fixed once you have this one abstraction of the tensor equation and you implement it on CUDA, for example. You can optimize the heck out of it, and einsum will finally be able to reach its potential, right?

Actually, none of these things should be the most important part. The most important part is that einsum, as we know it, is only good for tensor algebra. TensorLogic is a language where the same construct does all the symbolic and all the numeric parts, and any mix and variation between them, including learning the symbolic part and whatnot. These are all things that in the einsum world just didn’t exist, right?

You talk about the people who knew einsum, whether in AI or mathematics or physics, and they just had no idea that any of this had anything to do with reasoning. You look at all the ways that people are trying to do reasoning today and just want to pull out your hair.

Tim Scarfe

Let me ask a very concrete question. In some sense, I’m a simple man. I need a very concrete example, because I completely agree with you: The symbols we use, the language we use—simplicity is so fundamental to our ability to reason at higher and higher levels.

So let’s take one example from your paper, which is that a logical OR of a bunch of values is equivalent to an einsum with a Heaviside function applied to it. You give this example, right?

To be precise, what I did in the paper—maybe this is an important piece of context—if you look at the simplest form of logic programming, it’s Datalog, right? That’s the foundation of databases. Most SQL queries are variations on Datalog rules, and Datalog rules are composed of 2 things: joins and projections. This is databases 101.

What I have done is generalized join and projection to numeric values. There’s this thing that I define called the tensor join and the tensor projection, which, when the tensors are Boolean, becomes the regular symbolic database version, but now the numeric version has all these things as special cases. And, by the way, it’s also more general than einsum, right? So another benefit of this is that it actually goes beyond einsum.

Now, an OR—the way you get an OR is just as in Prolog or Datalog. How do you get an OR in Prolog or Datalog? By having multiple rules with the same head. Those rules are then implicitly disjoined. If I have “A if B and C” and “A if D and E,” then that means “A if B and C or D and E.” The same thing happens here.

You could also, of course, just put them all in the same equation, because it might be more convenient just to say, “AB + CD,” right? So doing an OR is a completely straightforward thing, but it’s really not where the main action is. It’s in the tensor joins and tensor projections.

Tim Scarfe

Thank you for laying all that out. Completely agreed; it makes sense. I just wanted to clarify the very example I was giving: An einsum over a particular index of Boolean values, followed by a Heaviside function applied to that element—which is just 0 if it’s less than 0 or 1 if it’s greater than 0—is equivalent to a logical OR over all the same Boolean values of that index.

Yeah, sorry. I see—I understand your question. So, again, it’s more than that. It’s a DNF, right?

Tim Scarfe

A DNF is a disjunction of conjunctions.

Yeah, exactly. And so what happens is that the einsum is—in numeric land, think of a dot product. The dot product is just the sum of products, which in Boolean land will be a disjunction of conjunctions. If more than 1 is true, you get a number that’s greater than 1, which you need to pass through a step function to reduce all the values greater than 1 back to 1.

Tim Scarfe

Yes, but my question was more like, okay, I have these 2 different representations of the same operation, at least at the element level. An einsum over an index followed by a Heaviside on that element is equivalent to an OR over all the same Boolean values of that index.

My question is per element. So my question to you is: I give up 1 thing, which is that instead of having a single symbol that’s kind of like an OR, I’ve now got 2 operations, einsum and Heaviside. And there are many examples of that, right? I can build every single circuit out of NAND gates. I think we discussed this once, actually. Or I can have other kinds of gates, and it’s useful to have other kinds of gates.

So, in your language, do you foresee people not having syntactic sugar like an OR operator, which under the hood is einsum plus Heaviside, or would they still retain those? It’s just that the fundamental, most basic constructs of the language are Tensor Logic.

We can do everything with NANDs, so why do we need high-level programming languages at all, right? The point is that there are 2 things that you want the language to be. First of all, you want it to be universal, so you can—For some things you don’t, but in general, for AI, surely you want a universal language. You want something Turing-complete and Turing-large, and Tensor Logic is that.

But this is actually the most important and most difficult part: You want something that’s at the right level of abstraction for the things that you want to do. And NAND definitely is not. I can show with a lot of examples that I have in the paper, for example, that you can code a transformer in a dozen tensor equations as opposed to a vast mass of code, right?

And then what happens when people have a language that suits their needs is that they just get used to it. They often wind up using it even for things that it wasn’t the perfect thing for, but at that point it’s what they’re comfortable with. So my guess is that at the end of the day people are just going to do everything in Tensor Logic, and they know, in the back of their heads, that, yes, there are ORs going on here, and you can think of them as ORs, but they just think of them as joins and projections and tensor equations.

Tim Scarfe

Very good. And by the way, you can implement transformers and anything else in Tensor Logic. It’s so easy, in fact, that I fed your paper into Claude Code, and I got it to implement the whole lot this afternoon. Maybe I’ll publish that on GitHub if folks want to have a look, but it’s quite straightforward.

Just to get the trajectory a little bit here, Pedro, you’re famous for writing The Master Algorithm. In that book, you spoke about all of these different tribes of machine learning, like Bayesian folks, logic folks, kernel methods, neural networks, and all of this. Do you see this as a step toward unifying these things together? Because now in Tensor Logic, you can actually create a composition of different modalities of AI, and it just works, but this might seem a bit weird to people. Can you explain what that might actually look like?

Absolutely. So, in a way, The Master Algorithm was laying out my agenda, right? It was asking the question, “What is the master algorithm?” I did say at the outset, “I’m not going to give you the master algorithm in this book. I’m just going to tell you where we are and why I think this is the central goal of AI.”

I would say that Tensor Logic is that answer. We haven’t talked about that yet, but Tensor Logic unifies not just symbolic AI and deep learning; it also unifies things like kernel machines and graphical models. The things that graphical models, for example, are built out of, and that you can compute probabilities with, are a direct— I didn’t do this on purpose, but it just fell out.

The fact that the factors that graphical models are made of are just tensors. The marginalization and pointwise products that probabilistic inference is made of are just tensor joins and projections on those tensors that represent potentials and, in the case of Bayesian networks, conditional distributions.

At this point, we do have this very simple language where you can do the entire gamut of AI, which, honestly, I didn't think was going to be possible going in. I thought the answer would be much more complicated. Now, is this the master algorithm? Tensor Logic per se is not the master algorithm because it's just a language. I would say that it's the scaffolding on top of which you can build a master algorithm.

Tensor Logic is not just a language; it's also the learning and reasoning facilities under the hood. For example, one of the best things about Tensor Logic is that the autograd graph is incredibly simple, because there's just one construct: the tensor equation. The gradient of a Tensor Logic program is just another Tensor Logic program. So this is all there. The learning and the reasoning are all there.

However, what I would say is that this is not the master algorithm per se, but it's what we need to produce one, and I intend to produce it on short order.

Tim Scarfe

You've described a language, and certainly, if components of that language are Turing-complete, that's a big, vexed issue. We'll come back to that a little bit later. But because of computational equivalence, from an expressibility point of view, we can describe anything in the universe.

So we've got this framework, but to me, the challenge in AI is structure learning. As well as being able to express stuff, it's being able to adapt to novelty and create, perhaps from building blocks that we already have, a new structure to allow us to do something useful in that domain. I can't quite make that leap with your technology yet. So how do we do the meta thing where we actually build the Tensor Logic constructions to represent the kind of world that we're seeing?

Oh, very good. I actually go into that in the paper, but briefly. The paper is just an informal introduction to these ideas. Inductive Logic Programming is the field that deals with discovering rules from data. But it does this by things like greedy search or beam search, and it's a very large search space and extremely inefficient, right? Which is actually one of the things that killed it, even though it could do all these things that people in deep learning are just painfully rediscovering.

In Tensor Logic, this is one of the best parts of it: structure learning falls out of gradient descent. Gradient descent actually does structure learning. And on top of that, this is actually the best part as far as the learning is concerned: there is this thing called predicate invention, which is discovering new predicates, discovering new relations that are not in the data but that explain it better.

I would say that, in some sense, discovering representations like that is the key problem in AI; it's the Holy Grail. Everything that we know—when you look at the world, you don't see pixels. You don't see photons hitting your retina; you see objects. The objects are invented predicates. All the way up to science, Newton's genius was to introduce a new quantity, which is force, and energy and entropy, and all these, et cetera, right?

In Tensor Logic, that also just happens by gradient descent. It's hard to believe, but let me just give you a hint as to why this is the case. There's this other thing that's folded into Tensor Logic, which is tensor decompositions. Tensor decompositions are a generalization of matrix decompositions.

If you think about matrix decompositions, to take that simple case, what a matrix decomposition does is take a matrix and decompose it into 2 new matrices that together are more compact but essentially reproduce the same data. There's a generalization of that to tensors called the Tucker decomposition. There are others, but the Tucker one is the most relevant one here.

So, to answer your questions very directly, if you write in Tensor Logic a rule schema, including a data tensor on the left-hand side—and by the way, your entire data can just be reduced to a single tensor, represented as a single tensor; we can touch on that later—you write a rule expressing that as a function of a few other tensors, and gradient descent, just as in matrix factorization, will discover the best values for those.

Then, if you want to, for example, discretize it, say, “I'm going to threshold this and make it Boolean again,” you will see what concept it learned, or you can leave it in numeric form. So the learning is actually extraordinarily powerful.

I've always thought—and I think a lot of people in deep learning really believe this—that gradient descent can do amazing things, provided you give it the right architecture to operate on. In a way, what all these million papers are about is finding the right architecture for gradient descent to operate on. And of course, transformers are a great leap forward, but I think Tensor Logic is an even greater leap forward.

Tim Scarfe

How so? Because, for example, I can picture—suppose we want to get rid of Python. So I'm over here in PyTorch, and I've described all my layers in the clunky syntax and said, “No, now I have the Tensor Logic programming language on GitHub. Let me go do it there.” I'm still going to construct my layers, right?

Because, for example, you do, of course, allow for the nonlinearities, right? So after every einsum, I can apply whatever kind of nonlinear function I want—a ReLU or sigmoid or whatever else, right? That's still going to be described in my program. I'm still going to have to do that kind of structuring of the network, if you will, except now in Tensor Logic.

In my opinion, that's one of the biggest limitations right now. These are all just divine-incantation structures that people have come up with: let's put in a dropout layer here and this kind of layer there. We don't actually allow the machines to learn the overall topological structure. We only allow them to find weights within that structure.

No, but I understand your question. Tensor Logic does allow that. Step 1: you can encode a multilayer perceptron—the entire multilayer perceptron—and I do that in the paper with a single tensor equation. All the layers, provided that they all use the same nonlinearity, can be encoded in 1 equation.

You can also have different equations for different layers, or typically sets of layers, however you please. But from the point of view of structure discovery, the thing to realize is that if you set up one of these very general equations that you can create in Tensor Logic, which, in some sense, can represent a very broad class of architectures, then what the learning does is discover the architecture within that space.

If you think about it, at some level, when you compare an ordinary multilayer perceptron with a set of rules, you can see the connection. In fact, there was a system called KBANN in the early days that did this very cleverly. It initialized a multilayer perceptron with a set of rules because each neuron is a rule, right? But it's also more flexible because now you can have weights, right? A single neuron can represent a conjunction, and therefore a layer can represent a disjunction, and so forth.

When you're learning weights in an ordinary neural network, you can actually see it as learning the structure of a set of rules. What Tensor Logic is doing is this at a more powerful level. That was just propositional, and now this is at the full level of generality of first-order logic.

But you can learn the structure, and then, of course, there's more than 1 way to do that. You can also decide how black-and-white you want the structure to be, what you want to leave as weights, and what you want to discretize. But the structure itself can be learned by taking a tensor equation. A tensor equation is a very general thing, right? When you learn the weights of those tensors, that materializes into a specific network structure.

Tim Scarfe

Yeah, so I understand that. Let me bring this back to the folks who are familiar with PyTorch or traditional techniques. What you described is, yeah, I can just create a fully connected network with however many layers I want and let SGD find all the weights. That doesn't work. It doesn't work in practice, and it's not going to work with Tensor Logic. It's just a different representation of the same fundamental problem, which is that there are too many degrees of freedom. It's not going to learn anything useful.

This is why so much alchemy goes into structuring constrained networks to have certain built-in inductive biases, right?

No, absolutely. To take another example, you can also do an entire ConvNet in just 1 tensor equation.

The quintessential example of fully connected layers not working is a multilayer perceptron for vision, which you replace with a ConvNet that actually has the local structure. That is also a Tensor Logic equation. Now, you're saying, “Well, how do you choose between the ConvNet and an MLP?” Very good question.

These days, you can start out with a very general structure because GPUs and large server farms give you an amazing amount of power for something like this. You can almost, I would say, brute-force that search, provided you have the data. I'm not actually recommending that you do it.

You can also, however, and more interestingly, write down what you believe are properties of the structure. When you program a network in Python, you have to commit. You say, “Here's the structure,” and the only thing that happens is the learning of the weights.

In Tensor Logic, you don't have to do that. You can set up one of these very general structures and then say, “Let me give you a bunch of equations that are things I believe to be true about the structure but do not completely determine it.” Those work like priors and, indeed, like soft priors. You can turn the temperature up or down and say, “You've got to obey this equation and that one.” Sure, you can override them.

In my experience, this is actually what is important: that gradient descent, instead of starting from a tabula rasa, has this kind of soft knowledge. Most importantly, you—the developer, the AI researcher—get to do this. This is really the essence: every deep-learning researcher or data scientist knows that you don't set everything up a priori and then push the button and hope for the best. There's an iterative loop where you set up the structure, learn, get the results, and then refine the structure.

What this does is make that process much more efficient because, in your interpreter, you just have to write one more equation or modify an existing equation. Also, the entire stack of what you learn is much more interpretable than it was before. That's actually, in some ways, one of the most important properties of Tensor Logic: you can understand what's going on much better than you could before.

There are 2 ways. One is that the code is much more transparent than the whole pile of things sitting under a bunch of PyTorch procedure calls. The other is that the result of learning—at least if you do it in certain ways that I discuss in the paper—is transparent in a way that a transformer simply can't hope to be.

Tim Scarfe

We've covered some interesting topics on MLST before. Of course, there's geometric deep learning, which is this idea that symmetries are fundamental. We've spoken with Andrew Wilson from NYU recently about soft inductive priors, and I've just spoken with Yi Ma about his CRATE series of architectures.

I guess the prevalent idea here is almost Platonistic: there are real, natural patterns, and if we bias the model, as you're alluding to, it will converge on really good representations that describe reality. The alternative view is that reality is constructive and gnarly, and that won't work.

You were talking about your Tucker decomposition earlier. That's the idea that we might have a large, sparse matrix, we might want to densify it, and we might want to factorize it. The factorization will pull out some of these natural orderings of the universe, perhaps. I was thinking, isn't it a bit like a gzip algorithm? What if these factorizations are just semantically meaningless? How do you know that you've got a good one?

Great question, and you've touched on several things there. Let me start with geometric deep learning. I'm a big fan of this. In fact, I gave a keynote at the 2nd ICLR on something that I called symmetry-based learning, which is in some ways an ancestor of geometric deep learning.

I really do think that the universe possesses these fundamental symmetries. Actually, I don't think that—that is known, right? In physics, the Standard Model is basically a bunch of symmetries. It's extraordinarily powerful that such simple things could be such universal regularities, so that you can basically build everything else out of them.

If you think about it in machine learning, the problem is: what is the learning bias that you should start from? Should you pull in a lot of knowledge? Should you have a very vague architecture? There's the no-free-lunch theorem, which says that if you don't assume anything, you can't ever learn anything.

What's amazing about machine learning is that, with very weak biases, you can get very far. I would submit that those weak biases—fundamentally, at the end of the day, the most important ones—are these symmetries. Tensor Logic is precisely the perfect language for expressing those symmetries, as physicists will tell you. It's what they use—not the logical version, but the numeric version.

I think we can discover those regularities. I have some suspicions as to what they might be, but we're not quite there yet. Once we have those regularities, in some sense, they will play in AI the role that the Standard Model plays in physics.

Now, of course, there are people who say, “Oh, forget that. Going back to Marvin Minsky, there is no small set of AI laws or anything. It's just one damn thing after another. You're dreaming.” I respect that point of view, and we will find out empirically. But if I had to guess how this is going to play out, at the end of the day, it's going to be like this: this stuff gives you the 80/20. It gets you 80% of the way, and then for the other 20%, you have to do a lot of these things and a lot of hacks.

Since Tensor Logic still makes it much easier and faster to do those hacks than if you didn't have it, it actually gives you a benefit both in the 80% part and in the 20% part.

Tim Scarfe

There are folks in complexity science. There's this guy called David Krakauer, and in his book, on the first page—actually, in the very first sentence—he discusses the scientific and social implications of differences between:

A, closed, reversible, symmetry-dominated, and predictable classical domains; and B, open, self-organizing, dissipative, uncertain, and adaptive domains.

I think that's what you're talking about—the kind of Roger Penrose-type world. Now, I think the latter is where all the interesting stuff in the universe is. It's where life and intelligence and all the other things we want to model are. Could it be the case that those things are not reducible in the way that you're arguing they are?

I'm glad you asked that question, because this really is the crux of the matter. Also, you're probably familiar—I know you're familiar because we've talked about it before—with Steve Wolfram's notion of computational irreducibility, right?

Tim Scarfe

Yes.

And, of course, we now understand very well that many systems are chaotic and therefore inherently unpredictable. There's complex systems and all of that.

Tim Scarfe

Exactly. The whole notion that “more is different” is a very famous notion in condensed-matter physics, and I'm a very strong believer in it. So doesn't that contradict what I just said?

Actually, no. I would say the following: from physics all the way to AI, with biology in the middle, the universe is basically composed of 2 things: symmetries and spontaneous symmetry breakings.

God made the symmetries. The symmetries are the laws. As far as we can tell, none of these systems at any level violate the laws. Those symmetries are there. There's a lot to be said there, but essentially, the great majority of people—maybe with some exceptions—believe that the laws of physics apply to everything. My brain obeys the laws of physics. Society obeys the laws of physics.

The problem is that the laws of physics are useless at some point in understanding even biology, let alone psychology, sociology, or AI. Why are they useless? Because we have inherited, from the beginning of the universe, a series of spontaneous symmetry breakings. My brain is doing spontaneous symmetry breakings one after another, continuously.

Some of those die out, become irrelevant, or stay the same, but others balloon into very big things. That's actually what evolution is: one of these things after another. Once you have that, the computational irreducibility problem is that, at some level, although in principle this is all predictable and reducible, in practice it isn't.

But here's the point: how do we handle that? Our brains know how to handle it in a way that AI doesn't.

The way they handle it is this: You predict—you computationally reduce everything you can to begin with. I've actually talked with Steve at some length about this, and I'm much more optimistic about how much is reducible than he is. The thing is that your overall universe is not reducible, but it's full of these reducible pieces. In a way, evolution is a cumulative process, and our brain is an accumulation of these reducible pieces.

You want machine learning to discover them, and you want inference to exploit them. But after that, you have no choice but to keep gathering data and using it to inform your predictions, right? In a way, the physics goal of “I give you the initial conditions, and then I just predict”—the Laplace's-demon dream—is a dream. But I think the problem that some of the complex-systems people have not realized is that we don't have to do that.

Ask any engineer, any aerospace engineer using a Kalman filter. What you do is predict just what you need to, as in reinforcement learning, right? You want to have a sense of where you're going, but at every time step, you recalibrate your predictions with the new data that comes in. You only need to predict things well enough to control them, to make them predictable, right?

We humans are always controlling the world to make it more predictable, and this is what robots need to do as well. This is what I'm trying to support with a language like tensor logic. I'm increasingly more of a believer in Hofstadter's concepts: that there are multiple levels of description, and even within a level of description, there may be multiple languages to describe things at that level.

I think part of the lesson is not only that we observe a particular level, and that we try to reduce things and come up with theories at finer-grained levels—higher-resolution theories, or whatever—but that we also observe a certain layer and are able, by whatever sort of miraculous mechanism, to almost pull a theory out of thin air, or abduct one. Here's thermodynamics. Somehow we came up with that, right?

Even if we learn theories at lower levels, or higher-resolution theories, most of the time we don't replace the older ones. Within their domain of operation, Newtonian mechanics is still extremely useful for lots of things that have to do with our scale, our scale of activity. General relativity is useful at a different scale, and quantum mechanics at a different scale.

Tim Scarfe

I'm hearing that tensor logic is a great language for a certain layer of description and for the activities of AI, but you're not arguing that it's the language to replace all other layers, right? You still buy into the idea that there are other languages at different levels?

I'm glad you asked that question. I am absolutely arguing that tensor logic is the language to use at all these layers, and let me give you some evidence for that. Express general relativity in tensor logic. It's tensors, differentials of tensors, and whatnot. Tensor logic does that out of the box.

Do the same thing with quantum mechanics. Do the same thing with all these others, with all the different pieces of AI that I know. Why is that possible, and why does tensor logic do that? Again, I think this gets at a very deep fact about the universe, which complex-systems people and physicists have suspected as well: The universe has this amazing property, without which it would not be comprehensible, that you can have a lot of complexity at one level that then organizes itself into a new level at which a different set of laws applies, right?

In a way, what we do with computers is do that by design, right? But here's the key: What you want is a language in which to express this process—the whole process by which multiple levels get created, by which multiple representations get created, including different representations at the same level.

For example, going back to Herb Simon, many people, at least in AI, have believed that the essence of human intelligence is your ability to switch between representations as the problem dictates. As long as you pick one representation, you've stuck yourself in a box. But at that level, tensor logic is a meta-representation. It's the way to construct representations.

A large language model, to take a very salient example—what has that transformer learned when it looks at all that text? Precisely, I would say, a lot of its power comes from the fact that it has looked at all these different pieces. It's like, as Bill Blake says, it has learned these super-algorithms. There are all these different pieces and different ways of doing things that it has gathered from different places, and it doesn't choose between them.

It's the prompting and the fine-tuning and all that that then pull out the parts that are better for one thing or another. So we absolutely have to do this in AI. I think it also reflects a deeper truth about the universe. I think there are going to be laws of this.

We're not then describing laws of the universe, and I think tensor logic, at least, is my best attempt at having a language in order to do both this type of AI and this type of scientific discovery. I also believe, and I discussed this briefly in the paper, that tensor logic is not going to be just a good language for AI. It's going to be a good language for science in general, for several reasons.

One of them is this, but the other one is that if you look at the difference between the equations on the page and the resulting program from implementing them, often there's a lot of complication. In tensor logic, the tensor equation is almost a symbol-for-symbol translation of the equation on the page. So now you can just do science on a different level.

Also, if you look at scientific computing, it's usually tensor operations with some logic wrapped around them. Tensor logic does the tensor operations and the logic in one language, but more importantly, the logic now becomes learnable. You can learn the logic as well.

Tim Scarfe

Let me challenge you on this. For example, in your paper, when you got to the RNN section, tensor logic can represent RNNs, but then you hacked in star t. You were like, “Oh, I need this little star t here.” What's star t? Well, star t is a virtual index that doesn't create new memory. That's not tensor logic. You hacked in star t because you needed it in order to express RNNs, right?

No—great question. There are 2 very important things to distinguish here. The first is what star t is not: The RNNs also illustrate that it is syntactic sugar, right? You want to express X of t plus 1, right? Tensor logic is Turing-complete, but it doesn't have t plus 1. Adding this is a very simple piece of syntactic sugar. Why wouldn't I do that, right?

Again, there's an 80/20 rule about which of these constructs you want to have. But star t is actually a completely different thing. Star t is there for computational-efficiency purposes. Star t is a hint about how to implement that tensor that saves a ton of memory.

You know this notion of a leaky abstraction: All abstractions are leaky, famously, in computer science. Tensor logic is no exception. For the most part, when you write tensor logic, you don't have to worry about what goes on under the hood, but sometimes you want to. This is precisely one of those things.

The idea of star t is that—we don't have for loops anymore, right? Which is great. Forget all of that. But sometimes I don't want to be computing a new tensor, or even just a new vector, for every new thing that I do, because that would be a waste of memory.

Star t is just saying, “You have 1 vector, and you reuse it at every iteration.” So you have the initial X zero, and then X one overwrites it, right? This is a piece of the language. You can do everything without it, but it would be silly not to use it.

Tim Scarfe

Let me push back on something because you mentioned it twice now, which is the term incompleteness. So your paper relies on Siegelmann's 1995 paper. She herself now, decades later, has admitted that that thing is a total toy that has no practical relevance whatsoever, okay? Because it requires infinite-precision rational registers that encode in a fractal way, et cetera. And by the way, in her paper, all she demonstrated was that under these infinite assumptions she could build a particular RNN that was a universal Turing machine. The problem with you using that for your tensor logic is two things. One, that restricts the field over which you can have your tensors. It must be one of these fields that has infinite precision—so infinite-precision rationals or whatever. I can't use any other fields, like modular arithmetic, which is actually what runs on GPUs, for example. And secondly, it would restrict the actual structure of the weights to her universal Turing machine. Therefore, it wouldn't be a general-purpose tensor logic. Do you realize this problem?

No—no, no. So, actually, there is no problem there. Let me tell you exactly why. Let’s do this in 3 steps.

First of all, Turing completeness doesn’t matter at all, whatsoever, because the only difference between a Turing machine and a finite-state machine is the infinite tape. In the real world, there is no infinite tape.

Tim Scarfe

Why do you keep mentioning it?

That’s part 2. This is actually a very interesting set of questions. Let’s set it aside for a moment and get to that part.

Turing completeness doesn’t matter. What matters is that you want to be able to express any computation that you might want. That’s what matters, right? You might choose a specific language for specific purposes. For something like tensor logic, you want that generality. You have that generality irrespective of Turing completeness. So, this is part 1. We can debate it, but let’s set that aside for just a second.

I don’t get to change the way computer science is, and Turing completeness is a shorthand for universality. I just want to show people that tensor logic is universal. Now, I have a proof that tensor logic is computationally universal that does not rely on the Siegelmann construct.

I chose not to publish it in this paper because it would take too long. The beauty of that is that, in 1 paragraph, I can just say, “Look, the equation in the Siegelmann paper—you can implement it here, and we’re done.” There are so many ways to prove that things are Turing-complete.

I completely agree with you that that construct is ridiculous. It’s silly. It has no practical significance, but the reason I used it is just my way of telling people in 1 sentence that tensor logic is Turing-complete, and why. But the real action is—I’d love for you to share it. I’d love to see the other proof.

Tim Scarfe

Oh, I can. I mean, the other—actually, there’s more than 1 other type of proof that’s possible. Let me tell you what that one is. So, there are 3 ways. There’s the Siegelmann way, right? Another one is that you have a finite control with access to an infinite external tape. That’s a much more reasonable thing, in my view. You have a memory—

Yeah. The memory is infinite, but all that you have to do in tensor logic is know how to access that memory. It gets back to—remember, a Turing machine is a finite control and an infinite tape, right? So, if tensor logic can realize the finite control, which obviously it can, and you give it an infinite tape, then we’re done.

On that note, you can even do it the following way. For example, Dell Sherman has a great paper about this. People have come up with various very simple ways to set up a Turing-universal computer. One of them is a set of rules that sets up that machine. Without going into details, you can just write that set of rules in tensor logic without even having to wake up from your sleep. So, there you go. I totally agree with you.

I often say to people, “A Turing machine is just”—and I really hate to use the word “just” because it doesn’t do justice to Turing, to Alan Turing, and the genius of his creation, the theory of computation—but it’s a finite control with unbounded read/write external memory. I’m totally on board with that.

Absolutely, tensor logic is a finite control, but then you need to add to it these operations to manipulate external memory. So, it’s tensor logic plus some operations to deal with external read/write memory, no? I mean, those operations are just read, write, move left, and move right. That’s all there is.

Tim Scarfe

I know, but that’s an extension of—at least in my view, I don’t know if before you there was such a thing as tensor logic. I’m not sure. I know that a lot of people have talked about tensors for a decade or more, but it seems like some kind of extension to the way tensors are typically used. It’s certainly an extension to the way tensors are used in general relativity. There’s no read/write to external memory in that.

Of course, but that is why tensor logic is more than tensors in mathematics. The tensors that people use in mathematics just don’t do this, but tensor logic does because of the logic-programming side. If tensor logic can do logic programming, then it can do everything that a computer can.

Tim Scarfe

Have you specified fully all the operators in tensor logic somewhere, on a website or something?

There are only 3: tensor projection, tensor join, and univariate nonlinearities. The nonlinearities are crucial. Tensor algebra is multilinear. Algebra is linear; tensor is multilinear, right?

Tim Scarfe

I totally agree. Where do the memory operations fit in there? Are they projections? Are they joins? Are they—

Oh, no, they’re not even projections or joins. Think of a trivial projection where you’re not summing things; you only have 1. That’s what a write is.

Actually, let’s not even worry about tensor joins and projections. Let’s just think about propositional rules. If you want to implement propositional rules in tensor logic, all that you need is tensors with no indices, with 0 indices. So, we’re only dealing with scalars. A write is just a rule that says the target of the writing is on the left-hand side, and what you want to write is on the right-hand side.

Now, to get very quickly to the issue of an infinite memory: What is an infinite memory? An infinite memory is just an infinite vector indexed by the memory address. That’s all it is. So, how do you write this infinite memory in tensor logic? You just have the memory as your tensor on the left-hand side. It’s so simple it almost seems like there’s nothing to think about.

Tim Scarfe

I’ll have to work through some examples. So, just to finish that thought, how do you advance the tape?

Well, you just increment the index. And how do you move it left? You decrement the index. It’s done.

Tim Scarfe

Could we come up with a solid example? I don’t think we sufficiently described the star notation. Roughly, as I understand it, rather than becoming a dimension, it becomes a transition function. We don’t need to model the full trajectory.

Just to give an example, if I wanted to compute, say, the nth digit of π, or approximate it, would I not need to fix the size of the tensors beforehand? Did you see what I mean? The way I understand it, these things have a fixed size. How could it possibly solve unbounded problems?

No, very good. To clarify, the star notation is not a function. The star notation is about an index. For example, if I have a vector, Xᵢ—or a better example, a matrix Mᵢⱼ—this occupies, if i and j are each 100, 10,000 positions in memory.

But if what I do is Mᵢⱼ* on the left-hand side of my tensor equation, then instead of being 100 by 100, it’s just 100. What this is saying is, “Run through i, and for every j, overwrite the result.” You can do this in either dimension, but pick one or the other. It just says, “Keep overwriting the results.” So, you lose your old one.

Let me put it this way: Mᵢⱼ* is actually a vector where the only dimension is i. j is actually just an iterator for a for loop. You see what I’m saying? Concretely, for example, in an RNN, this is what you want because Xᵢ is your vector and the j—let’s call it t—is Xᵢₜ.

At every new step in time, when the state evolves, you don’t want to—I mean, you could, but in general, you just want to overwrite the old state with the new one, as in any state-transition system. Does this make sense?

Tim Scarfe

It does, but you’re describing an accumulator. Do you lose something by losing the history? If you think about it, you’re overwriting what went before with new information, and you’re just unrolling in time. Do you lose anything doing that?

Of course you lose. If you don’t want to overwrite it, then don’t put the star in.

But now, to answer your question about π: How would I compute all the digits of π, in infinite Turing-machine land? I have a vector of the digits of π that has a start but not an end. What the computation in tensor logic does is compute each successive digit.

We didn’t talk about this, but how is inference done in tensor logic? Forward chaining or backward chaining. They are both generalizations of the corresponding operations in symbolic AI.

If you applied forward chaining to a set of rules that computes the digits of π—actually, just 1 rule, because it’s very simple—what it will do is, in each iteration, fill in the next digit of π. If your vector is infinite, this will go on forever, as it should.

If your vector is finite, at some point you run out of memory and you're satisfied with the number of digits, which is what we do with any real computer in the real world.

Tim Scarfe

I don't want to always get us bogged down in Turing issues. I think we should move on, but I think it'd be fun to talk about it more at another time or just to work through some examples. I think I'll probably work through some examples, but I think this was an interesting one. There's a strange attractor with Turing conversations, and normally it goes in the Schmidhuber direction where the universe is finite, there's no difference between an FSA and a Turing machine, and I felt that we actually had some information gain in this conversation.

Well, actually, on that point—and this is a bit of an aside; it doesn't actually have anything to do with tensor logic, so I hope you don't mind me asking—but since we have a computer science professor, I want to run something by you. I always get this kind of pushback from people. I'll say, for example, that autoregressive transformers—and I mean classic autoregression, not extended autoregression, not generalized autoregression, just autoregressive transformers—are not Turing-complete. DeepMind admits this, and they write a paper showing how you can extend them to become Turing-complete.

I'll say something like that, and somebody will be like, “Oh, yeah, but if I can't do 100-digit multiplication with this context size, all I have to do is just have more context, and then I'll be able to do it.” I keep making this point. Here's the crucial difference, right? You brought this up beautifully when you said, “Look, a Turing machine is a finite control with an unbounded read-write memory.” And here's the really cool thing about those Turing machines: they can run in a way where they're churning, churning, churning, churning, and then they say, “Out of memory.” All you have to do is just give them more memory and hit Continue. You don't have to reprogram them. You don't have to retrain them when you've increased their context size, right? That's the whole difference: with a neural network, a traditional transformer, if you increase its context size, you have to go back to the drawing board—you've got to retrain it, right? Because you've run out of memory. Is that a fair point that I'm making?

This is actually extraordinarily simple, and it's incredibly frustrating to me that there's so much confusion about it, starting with computer science and theoretical computer science and now playing out in AI and transformer land. It just boils down to this, right? You said earlier—and I violently agree, so correct me if I misinterpreted it—but you said, “Turing completeness is not important,” but that shouldn't cause us to underrate Turing's achievement.

Absolutely. What was Turing's achievement that we now take for granted? Turing's achievement, for which he is deservedly famous, was to postulate this notion of a universal machine. The amazing thing about computers is that they're universal machines, which in his time was a completely counterintuitive notion. What do you mean, a machine that can do everything? The typewriter can type; the sewing machine can sew. You're telling me there's a machine that can type with one hand and sew with the other? What are you talking about?

This is the genius, right? The first step is that you want to have this property of having a machine that can do anything. This is the foundation of computer science, of computers as a revolutionary technology, right? So, point 1. But point 2—I'm getting to the transformer part, right? Unfortunately, these confusions build on each other and never get resolved. It's one of those symmetry breakings, right? We went down this road of defining things a certain way and worrying about infinity, and now we're stuck there, right? NP-completeness is another example, but ignoring that.

The problem with transformers—the real problem is the following. The thing, for example, that inductive logic programming has and that we want is that you can learn things from very small examples, like children do in elementary school. You learn to do addition on tiny examples, but then, if needed, you can do addition on numbers of any length. Of course, your life is finite. You will never add infinite numbers, but that's not the point. Infinity is just a shorthand for something that's so large it doesn't matter how large it is.

What I want in machine learning is to be able to learn to handle problems, graph structures, knowledge bases, inference problems, whatever, of any size from very small ones. That's the limitation that a lot of these transformers have, and that's the one that you want to fix and can fix, and tensor logic helps you do that.

Just to cap off the discussion about Alan Turing, because I think he deserves a mention: you mentioned that this was the real achievement, this universality. It wasn't just a machine to do typing and sewing; it was even within computation, right? In his time, people didn't know this. They're like, “Well, what if I have a machine that just has a separate read tape and a separate write tape? I don't know. Well, how about if we add 2 write tapes? Does that make it more powerful? What if it's read-write? What if it's just a stack? What if it's lambda calculus? What if it's...”

There were so many myriad L-systems and all these different computational models, right? Nobody knew that they were all equivalent. That was the really remarkable thing. To be fair, Turing wasn't the only one doing things like this, and precisely now we know that there are all these things that are equivalent, and extensions of that power.

Here's actually a really important point, right? The question that has been on my mind for decades is this: a Turing machine is a model of deduction. It's universal deduction. What we're missing to be able to do what the universe does and evolution does is universal induction. What is the Turing machine equivalent for induction, for learning? That's what I'm after. That's what the Master Algorithm is.

I know it exists, and again, just as you can have 1,000,000 different versions of Turing machines that are all equivalent, you can have 1,000,000 different versions of the Master Algorithm that are all equivalent, and that's okay. The point is that first we have to realize that there is one. We have to prove what it does, and then we can refine it with the syntactic sugars and whatnot, and that's all good. But the main point is having gotten the universal induction machine, which I think we are pretty close to.

Tim Scarfe

But Pedro, I know the answer. It's Bayesian tensor logic. No, I'm just kidding.

No, if you're Bayesian, it is Bayesian tensor logic.

Tim Scarfe

This is a good segue because we are talking about reasoning and deduction. Transformers don't really reason, right? I think of them as a kind of collection of fractured bits of knowledge, maybe with a little bit of understanding 2 levels down, but we understand many levels down. And when we do reasoning, what we're doing is respecting all of the constraints of this epistemic understanding phylogeny thing that we have, and that allows us to build new knowledge, right? Because you can build new knowledge, you can create new things when you respect all of the understanding that you already have.

Transformers don't do that, but let's talk about how this works in tensor logic. You have this temperature parameter. For example, you could do something akin to deduction even in an embedding space, right? And certainly with an MLP. This is where I was a bit confused, because I can appreciate that if we have a logical model which is in the domain of certainty, we can do deduction, right?

And then if we have something like an MLP and we learn the weights and we turn this temperature parameter up, it's actually introducing some degree of randomness. Why would that be anything like the kind of logical deductive reasoning we do? Would that not just do what neural networks do now, which is look for similarity in some embedding space, and the type of reasoning it's doing isn't actually semantically meaningful at all?

I actually say that, of all the things in that paper, this is the most exciting and important one: you can do sound and transparent reasoning in embedding space with tensor logic. How come, right? Why is that possible? To give the gist of it, here's the key.

Think of kernel machines for just a second, and the Gram matrix, right? The similarity matrix—what is it, right? You're in feature space, and for every pair of objects i, j, it's the dot product of their feature representations, right? Now, if you embed all your objects, we already know who they are. There's a matrix with the embedding vector for every object, whether it's a word or a token or anything else, right?

Now I can do the dot product of the embeddings of 2 objects, right? Let's suppose they're all unit vectors to keep things simple, right? And now what happens is—and let's, for the moment, say you're not learning the embeddings yet, right? Let's say you just have random vectors, right? Your embeddings are random, right? That's actually already useful for a lot of things, but of course it's not where the action is, right?

And now there's the following very interesting property: the dot product of a vector with itself is 1.

The dot product of 2 random vectors in a high-dimensional space is approximately 0. So, your Gram matrix, your similarity matrix, will be approximately the identity matrix. And now, what happens if I have a tensor logical operation that operates in this way and then has something like a sigmoid nonlinearity? Then what's going to happen is that it's going to clean out that noise, and it turns into the identity matrix.

Now I have all these rules that are just operating in a purely logical mode. They're Boolean tensors going in, meaning relations, and Boolean tensors going out. That way, you can do pure deduction in embedding space with these random embedding vectors. That's already something interesting, but now let's say you learn the embeddings, which of course is the whole point.

When you learn the embeddings, what's going to happen by trying to minimize the loss function is that the embedding vectors of objects about which you tend to make the same inferences will get closer. If I'm saying something about one object and another one is similar to it, gradient descent, to minimize the loss, is going to increase their dot product. So, you're going to wind up with a similarity matrix that has high values for objects that are quite similar—in the limit, 1 on the diagonal—and has low values for objects that are quite dissimilar.

Now, if you turn the temperature parameter, meaning the stiffness of the sigmoid, at one extreme, at 0 temperature, you have a step function. The similarity matrix is discretized back to 0 and 1. So, at the 0-temperature extreme, you have pure deduction. You see where I'm going with this?

Tim Scarfe

I do, but could I challenge that a tiny bit? When we train neural networks, we think reasoning is good when we're building—let's use the LEGO analogy—these blocks. The neural network that we've created is a good one if it represents the world in an abstract, causal way.

I can see how you've framed this as deduction, in the sense that you've got this Boolean operation and you can build from it. But what if you're building on a sandcastle? What if the component—let's say it's an MLP component—doesn't represent the way the world works?

No, that's a very good question. Again, there's more than one thing you can do with tensor logic. One of them is that you can just reimplement existing things like MLPs and transformers. If all you did was reimplement them, they would have all their pros and cons. It's the same thing, just implemented much more elegantly.

What I'm talking about here, and what I talk about in that section of the paper, is doing something different. It's not an MLP, and it's not a transformer. It's actually doing these things where you embed objects, you embed relations in a certain way that follows from the objects, and you embed the rules and the reasoning. This is a different process.

What this different process allows you to do is that, when you raise the temperature, you get to do analogical reasoning. Douglas Hofstadter came up before. I think Douglas Hofstadter would like this because it's analogical. He has this whole 500-page book arguing that all of cognition is just analogy.

Again, this is one of the schools of thought, one of the tribes in The Master Algorithm: reasoning by analogy. You do reasoning by analogy because what happens is that you generalize from one object to an object that has a high dot product with it. Now I get to borrow inferences from similar objects. The higher the temperature, the looser the inferences, and the more analogical the inferences can be.

For example—and again, Douglas goes into this in some of his books—and any mathematician, like Turnstall, I just heard him say this the other day: “Mathematicians reason by analogy. They notice similarities between things. But at the end of the day, you need to have a proof.”

In tensor logic, in this particular scheme of embedding and reasoning in embedding space, this is just simulated annealing. You start out with a high temperature, being very analogical, and then you lower it. At the end of the day, you have a proof. It's a deductive proof that is guaranteed to be correct, but you couldn't have gotten to it because the search space is so large without the analogical part.

Tim Scarfe

Okay, I understand what you're saying. You can generalize reasoning outside the domain of certainty. But the reason I'm asking is that the reason we have metaphor and analogy is because there's this incredible process of evolution and intelligence. It's led to the coarse-graining of all these concepts that we use in our language, and there's this rich, beautiful phylogeny that represents the causal reality of what's happened.

Why is statistical similarity the same thing as analogy?

It's not. Again, I skipped over some steps here. It isn't. The most powerful type of analogy—kernel machines, in some sense, are the least powerful type of analogy—is just, “Here's a similarity,” or nearest neighbor. I have a distance function. That's not really where the action is.

The action is in what's called structure mapping. Structure mapping was proposed by Dedre Gentner, where you solve a problem by mapping its structure to the structure of problems that you know. The canonical example is Niels Bohr's model of the atom, which he came up with by analogy between an atom and the solar system. The nucleus is the sun, and the planets are the electrons. It turns out to be a bad analogy, but it was crucial in the development of physics.

There's also this whole subfield of AI called case-based reasoning, where I might be a help desk. You come up with a problem, and I don't try to solve it from scratch because I don't need to. That would be a waste. I go to my database of similar cases, find one, and then tweak it.

Structure mapping is an extraordinarily powerful thing, but it's the combination of similarity and compositionality, which kernel machines per se don't have, but tensor logic does. The point in tensor logic is that you have all the power of kernel machines, but all the compositionality of symbolic AI.

The structure mapping just comes out of the box. You don't need to do anything more to have structure mapping and all the power of analogical reasoning that comes with it.

Tim Scarfe

Can I suggest that a good analogy is to ad-lib? Do you think that's fair? You've got the general structure there, and you can plug parts into the blank spaces and get a solution, right?

That's one mode in which things can function. The whole process of structure mapping, or of case-based reasoning, can actually be very rich. I can combine, for example, 2 big pieces, but that's one example.

Tim Scarfe

Yeah, that's fair. It has this nice nested-structure property. Since we're on this topic, let me ask you about something that I was confused about in the paper. I don't understand your connection between hallucination and deduction, or determinism, because in my mind, I can set the temperature of GPT to 0 and it still hallucinates. I can have a poor deductive system that hallucinates all kinds of things.

So, to me, those are separate problems. Where was I misunderstanding?

Very good. Precisely, one of the problems with GPT is that it hallucinates even when you set the temperature to 0. What the hell? I want to have a mode—right? Not me, but every Fortune 500 company, if it's going to use AI, needs to have a mode where the logic of the business is just to behave. The security isn't violated, the customer doesn't get lied to, and so on. We've got to have that, or AI will not take off.

Transformers can't do that. Tensor logic can do that precisely because, in this reasoning-in-embedding-space mode that I just described, if you set the temperature to 0, it does purely deductive reasoning. By the way, the temperature can be different for each rule.

I think this is what almost all applications are going to have: there are some rules that are either mathematical truths or logic that you must guarantee will not be violated. They are the laws, and those have infinite temperature. Then there are all these other rules that involve more qualitative reasoning, more accumulating evidence, and perhaps things that you mine from the web. Those will have higher temperature.

That temperature parameter can be learned for some rules and fixed for others. Now you have this whole spectrum between the deductive and the more qualitative—or even fantasizing, truly hallucinating—at the far end of the high-temperature range.

Precisely the point that I'm making in the paper is that, with LLMs, the best thing you can get at 0 temperature is still a lot of hallucinations. Then there are things like RAG, but all they do is retrieve, and even then you still hallucinate. Compare tensor logic in this mode with RAG. It doesn't just retrieve things. It computes the deductive closure of your knowledge, which is an exponentially more powerful thing to have. And with zero hallucinations.

Tim Scarfe

Well, it is if the model represents the world. What does hallucination mean? Or, actually, what does slop mean? My definition of slop is when a creative artifact is produced by something that doesn't understand. If I understand the domain deeply, that artifact looks incoherent to me because it was generated by a process that doesn't understand the world.

Isn't it even the same with Tensor Logic? Deduction is great, but if the model isn't a good one, wouldn't that just be a hallucination as well?

Oh, absolutely, but let's make some distinctions here. The only claim I'm making here, because it's the only one you can make, is that Tensor Logic at zero temperature in this mode will give you the soundness properties that logic has. Soundness in the technical sense of soundness.

All that means is that you only reach conclusions that truly logically follow from the premises. You don't say anything about whether the premises are valid or not. If the premises were hallucinated, so will the conclusions be. There's no magic there.

That is a very important property to have. Again, if I give a transformer a bunch of true facts, it still hallucinates. That's what I can guarantee will not happen in Tensor Logic. Coming up with the true facts is a different part of the game. You can write them down, you can learn them, and you can refine them. You never know for sure if you have the perfect model. Of course, that's more the machine learning and knowledge acquisition part.

I do think I have a very important guarantee here of non-hallucination, but it's not a guarantee that the model you're working with came from the real world. That's a whole other neck of the woods.

Tim Scarfe

Who's going to adopt this first? How are we going to bootstrap this as a community? How do you see this progressing?

Good. The last section in the paper discusses adoption, what needs to happen, and things like that. Let's suppose that everybody agrees Tensor Logic is a beautiful, perfect language and what we need for AI. Just for that reason, it would not be enough to make it take off, sadly. People are still using COBOL these days. I rest my case.

Legacy is an irony in computer science, or in the information technology industry. It moves faster than anything else, but at the same time, things never die. You can't kill them. You can't kill COBOL.

I like Python. I program in Python. It's very nice in many ways, better than Fortran for some things, and so on. Even if it was never—or NumPy, if you will—you get the point. For AI, it's just a terrible thing, but I'm a Python programmer. I'm not going to rewrite all my code. Forget that.

So what is going to make it happen? We can look at what has made this happen in the past. There are several things. For example, look at how Java took off. Java took off at the time of the internet because it was the language of networking, allegedly. You could debate that, but people wanted to do things that were very hard to do with things like C and C++. And so Java took off.

Tim Scarfe

It was the language of embedded programs and web browsers. That was the only option we had, right?

Exactly. There are big arguments about this, but they aren't relevant to us here. The point I'm trying to make is that we're in a different ballgame now.

Why did languages like Lisp and Prolog fall out? They were better for AI than Fortran, C, or Java. They were niche languages. The network effects of the more widely used languages and all their surrounding aspects completely overrode that. We understand that very well now; people didn't in the '80s.

Now, the big technology, the center of everything, is AI. If you have a better language for AI, that is the one that is going to have the biggest users. Moreover, if you have a language that solves the big pains, that's what makes people adopt a new language—or a new anything, like a new app. It needs to solve some big pain.

Is there a big pain that Tensor Logic solves? Hell yeah. It potentially solves hallucination. It solves opacity. We're in a world right now where there are multibillion-dollar corporations and systems driven by this black box. I've talked with CEOs of big tech companies who say, “I can't sleep at night because I don't know what this thing is going to do. The people who trained it have left the company, and who knows what it will do?”

If we can make a dent in that, people will convert to it very quickly. I also think that when people experience how easy Tensor Logic is to use compared to the big pile of stuff that lies under PyTorch and whatnot, they will be very motivated to migrate quickly.

There are several other things, such as developing the open-source community and vendor competition. But one important thing is that Tensor Logic is ideally suited for AI education. It's one language with very little extraneous stuff, and you can teach the entire gamut of AI very well and do the exercises. It will be a language that the professors, the TAs, and the students will like.

History shows, going back to things like Unix, that if you have something that takes off in computer science education, people go into industry and say, “I want to use this because it's what I like.” A generation later, it's what everybody is using.

One more thing is that the transition to Tensor Logic from Python doesn't have to happen all at once. You can have, for example—and I already have one, actually, as do others—a preprocessor in Python. Again, because it's very easy to do, you read the paper and you can write it in the next 30 minutes. You can write a preprocessor in Python. All it does is make a one-to-one mapping between the syntax of Tensor Logic and einsum.

Making things efficient, as we discussed, is another matter. But from the point of view of developer uptake, there's a long history of people doing this with different languages. You have a preprocessor that lets you write tensor equations, and then it converts those equations into PyTorch or just NumPy, in Python. You do everything else in Python that you did before. You don't lose anything; you don't lose any existing code. It's just that a set of things, and in particular reasoning, have now become much easier than they were before.

Then, once you have this little lollipop, you think, “I can do this, and let me have that piece of syntactic sugar.” Before you know it, people are saying, “I don't need all that Python stuff anymore. I'd rather live in the Tensor Logic world.”

Tim Scarfe

You said that Tensor Logic is good for AI education, and Tensor Logic is a declarative language, which means it's the “what,” not the “how.” It's this incredible coarse-graining that screens off a lot of unnecessary detail. But is it unnecessary? I guess that's the question. Do you think people learning about AI should know how the underlying things work? Certainly, folks working at Google might need to do some domain-specific optimizations for certain components of the machine behind the scenes. Do you think we can screen off all that detail?

Great question. Actually, let me start by correcting something. Tensor Logic, like Prolog and Datalog, has both declarative and procedural semantics. You can look at a tensor equation. That's actually the whole beauty of logic programming, in some sense: a Tensor Logic equation is like an equation, like Einstein's equation. It's a statement about the world.

But you can also look at it and treat it as a function call. The left-hand side is the call, and the right-hand side is the body, which is a bunch of other calls and the way to combine them. In fact, in my experience using Tensor Logic so far, I tend to use it in procedural mode. It's a set of equations, a bunch of statements, like you would have in any imperative language. It's very important to bear that in mind.

To the heart of your question, which I think is very important: when you're teaching people something, I would actually say this is the tragedy of computer science education. From high school to introductory courses to the most advanced subjects, you want to teach people the beauty of what they can do and the essence of the algorithms and so on. But then—and particularly the students—they spend all their time bogged down in all this crap.

All these details where you get the semicolon wrong and the program doesn't work anymore, and they hate it. They decide that computer science is not for them, or at best, they waste 10 times more time than they should, right? So, precisely, the whole point of having the right abstraction is to avoid that. I would say this is one of the best features of Tensor Logic: to do that for AI.

But you also say, correctly, that a lot of the time you need to go beyond that level of abstraction, for example, from the point of view of efficiency and a lot of other things. But I would say—and again, we won't know until Tensor Logic is used widely and we see what happens—that Tensor Logic is a language that, at some level, is like C, right? It's very low-level. The beauty in my mind—again, this gets back to the multiple levels of abstraction—is that you can use it to say very high-level things.

You can also use it to express the lowest-level possible computations, right? A tensor equation is something that you can map onto a GPU with almost no change, and then optimize the heck out of it. In fact, I've joked with folks at NVIDIA that CUDA is a nice moat, but Tensor Logic could be the end of that moat.

Tim Scarfe

I sometimes feel like—and I'm not sure exactly how much money was spent on bigger and bigger transformers, deeper and deeper, wider and more data, and whatever, more-parameter transformers—but it's got to be a lot, like a trillion dollars or something like that. I feel like sometimes we spent a trillion dollars to learn, yet again, lessons that people could have learned if they'd taken certain basic courses in computer science. I'm wondering if you sometimes feel like that, and what lessons, if any, you think people should have known before spending a trillion dollars.

I violently agree with that. In fact, the paradox of the current moment in AI is that, on the one hand, this is super exciting, right? This is what we've worked all our lives toward. It's like the dream is happening. I used to tell people, when I went into grad school, that one day machine learning was going to take over the world, and they'd be like, "What?" And I'm like, "See, it is taking over the world. There, take that."

On a more serious note, transformers are a great leap forward, and anybody who's used a chatbot is like, "Wow, look at the things this can do. This is great." But at the same time, the sheer amount of wastefulness, stupidity, and ignorance going on is just unbelievable. It's like, "Why are you reinventing things?" For example, I've talked with people at OpenAI who do the reasoning. Many of them are very good people, so I'm not trying to pick on anybody, but it's like, "Oh, what is reasoning? We need to figure that out." And then they say a bunch of stuff that is completely wrong.

I'm thinking to myself, "Why don't you spend an afternoon reading a couple of chapters of Russell and Norvig and save $100 billion in wasted compute? Please, just do that," right? Part of what I'm trying to do with Tensor Logic is make things go in that direction, because the current direction is just too damn painful. And it's not just that it's painful. This is going to end badly, right?

Tim Scarfe

Oh, yeah.

In a way, spending all this money on data centers is not wasted, because it's not like the fiber that went dark, right? We in AI have an appetite for unlimited compute. But they're spending all this money prematurely on stuff that isn't ready for that yet, right? The demand is probably not going to be there, and we're going to look back and go, "Wow, 99.9% of that compute was completely wasted."

That's because of a lot of the reasons that we've been talking about, including that you didn't know how to do reasoning, so you brute-forced it, et cetera. We have to change the direction of this ship. It's like that well-known quote from Matt Damon in Good Will Hunting, to paraphrase it: "You've wasted a trillion dollars on an education you could have got for a buck fifty in late fees at the library."

Tim Scarfe

Exactly. Exactly. Well, Professor Pedro Domingos, it's an absolute honor to have you on the show. Thank you so much for joining us.

Thanks for having me. Thank you. Always a pleasure.