He's Building an AI That Can't Lie | Dan Klein, Scaled Cognition

We’re going to switch from the problem in AI where nothing works to the problem in AI where everything works. ChatGPT tells you something, and it’s always fluent and always confident, whether it’s right or wrong. Training these systems is becoming increasingly complicated, and often what looks like an exponential curve just turns out to be the beginning of an S-curve.

What happens next is that we start to hit diminishing returns. We’re seeing that today. We’re seeing data walls and compute limits. The different aspects of intelligence have not been advancing equally. Reliability has not kept pace. To me, suddenly the most important problem I could think of was: How do you build a system that will not lie to you?

1. Architecting for Truth

Can you say, at a high level, what your new approach is and how it’s different from what the labs are doing?

All right, I’m here talking with Dan Klein, professor of computer science at Berkeley and serial entrepreneur, most recently working on Scaled Cognition, a company that helps make more reliable AI systems. This is a fun conversation. We talk a lot about linguistics and parsing, which I worked on in a previous life when I was in a lab where Dan was the star grad student. I was a lowly first-year research assistant.

We get into how modern systems work, their drawbacks, and some of the things we might be able to take from previous versions of NLP into the systems we’re building now. You describe the AI industry as being built on Jell-O. Do you want to expound on that?

2. Plausibility Engines

Sure. I think it’s easy to lose track, given how quickly and explosively the kinds of large language models people are using today have burst onto the scene. It’s easy to forget that, at their core, they are probabilistic engines. They have been trained to do next-word prediction and produce plausible output.

The objective function is essentially just to produce output that is indistinguishable from the truth. This is very important: These are not truth engines. They are plausibility engines. I’m sure we’ll come back to that.

Well, let’s get right into it, actually, because I feel like that’s a little unfair. Certainly, the pre-training optimizes for next-token prediction, but I think there’s a lot of effort put into a second step where a big part of the goal is to optimize the systems for reducing hallucinations.

I did say “at their core” twice, and the reason I said it twice was that training these systems is becoming increasingly complicated. This really is the story of any technology that comes into artificial intelligence, or really beyond. There are these super-cycles in research, where people have complicated systems, they’ve hit some wall, and the systems are hard to improve.

Then some new technology comes along. Maybe that is large language models based on autoregressive training, transformers, and large amounts of data. This new technology comes onto the scene, and it’s almost like a silver bullet. In a very simple way, it’s suddenly topping benchmarks and outperforming systems that are much more complicated and were built on previous technologies.

What then happens is that we go into a phase where we double down on that new innovation and scale it up. We really try to get as much out of it as possible. This often feels like an exponential curve: This new technology, even in its simple form, is having such a big impact, and it’s just going to go to the sky.

Of course, the trees don’t grow to the sky, and often what looks like an exponential curve just turns out to be the beginning of an S-curve. What happens next is that we start to hit diminishing returns. We’re seeing that today. We’re seeing data walls, compute limits, and all kinds of reasons why that initial explosion of progress starts to hit diminishing returns with those specific methods.

3. Metacognition and Verifiability

I think that, for you, one of the biggest issues you flag quite frequently is hallucinations. Do you want to talk specifically about how you’ve measured that and why you think it might be a bigger problem than other folks would be aware of?

First, the word “hallucination” is a sort of projective term. Really, what’s happening is that systems are making errors. A sequence of tokens comes out, and it turns out to be incorrect information. We tend to call that “hallucinations,” and that term is being used more and more broadly now—almost as if anything undesirable that a system does can be called a hallucination. That term is definitely undergoing some semantic broadening.

But specifically, I think it’s important, when we talk about systems, to ask: What’s the difference between a system that makes a mistake, hallucinates, or lies, and a human? These are different things.

Really, a hallucination is just a mistake the system has made. Again, if you have a system that is just doing next-token prediction, that system doesn’t actually know, as it’s predicting those tokens, whether they are right or not.

I haven’t forgotten your point that there’s more to training than just the autoregressive phase. But systems like this aren’t metacognitive, right? They’re not looking at their knowledge and making an external decision: Do I know the answer to this question? Do I have this information? Do I know where it came from? What is its reliability?

They’re not doing that. They’re just producing tokens. Sometimes they’re right, and sometimes they’re wrong. One way to look at this is that every output is a hallucination. Some of them are right, and some of them are wrong.

I think where you start to be able to say something stronger—to say that a system is deceiving—is once you start assigning things like reinforcement learning in later training phases. You mentioned that there are phases beyond the initial pre-training.

This is not something where there’s just one simple answer, but one of the things you see, for example, is reinforcement-learning training. This can take a couple of forms. One kind of reinforcement-learning training would be something like RLHF, where you show humans choices and say, “Which of these do you prefer?”

The system will now produce the outputs that humans prefer. Do they prefer things that are factual, or do they prefer things that make them feel good? It’s not at all clear that hallucinations are going to become less common. In fact, there are some results that show hallucinations become more common as you go into these sorts of post-training approaches.

I can give you a few reasons why that might be. Imagine you have a shipping company and they’ve deployed an agent backed by an LLM. In addition to the training that has gone into that core model, it is also being reinforcement-learned for some objective. For example, maybe it’s being reinforcement-learned to optimize the number of thumbs-ups it gets from users.

It’s operating, and someone calls in and says, “Where’s my package?” That package has been lost, and the database it queries through a tool call says that the package has been lost. What’s the system going to do? It can tell you your package is lost. It can tell you what’s coming tomorrow. It can tell you it doesn’t know. It’s got choices.

What will it be rewarded for? In this case, it’s probably going to get more thumbs-ups if it tells you that the package is coming tomorrow.

This, I think, legitimately would qualify as deception. Anytime you’re doing reinforcement learning, the system is optimizing the reward function. There is always going to be some gap between the truth and what’s being optimized, and that gap is going to increase the amount of hallucinations to the extent that it doesn’t align with truthful behavior.

We have some experience at Weights & Biases and CoreWeave building these customer-service systems. Of course, it’s going to be a really bad experience for a customer in the long run if something is hallucinated. From my perspective, you would never want to take the immediate customer reaction as the only thing that you’re optimizing for.

In fact, I think most people who are doing reinforcement learning—or even just evaluating a system like this—would have a special check for whether the information they’re getting back is accurate. It’s very hard to evaluate that in a lot of cases, but I think most real production systems do a fair amount of checking. They try to put in reward functions where the most negative score you get is for giving plausible but wrong information.

Absolutely. I’m giving a reductive caricature of an extreme case to illustrate how easy it is to get deceptive behavior because of reinforcement learning. People will, of course, design the reward functions to mitigate this and try to balance truthfulness and other properties, but ultimately there’s always going to be a little daylight between whatever you’re optimizing and the truth specifically.

In my opinion, one of the things we should be doing as a field is creating technologies that cannot lie to you, so that you can look at a system.

And I mean, personally, my mission here is to build technologies that can’t lie.

Let’s get into that. But just before we do that, I love—it’s such a gentle linguist phrase—“semantic broadening.” I love that you put it that way. But I think, in your view, “hallucinations” has maybe broadened semantically beyond what I thought it meant. To me, a hallucination at its core is something where it recommends a movie that doesn’t exist or cites a paper that doesn’t exist. To me, saying 5 + 7 equals 15 wouldn’t be a hallucination. It would be a different type of error.

I think it totally makes sense that these plausible citations of information that it doesn’t have happen a lot when the training is predicting the next token and looking for plausible next tokens, and there isn’t a step after that. Do you agree with my definition of hallucination, or do you really use “hallucination” and “incorrect information” interchangeably here?

I mean, if you’re asking me, do I think a good definition of hallucination would be crisply defined as sort of the injection of—

Let’s just agree with what we’re talking about, I guess. I’m happy to go.

I like your definition because I think it aligns well with the pre-LLM notion of what a hallucination is to a human, having to do with perceiving something that’s not actually there or something like that. I do think, in common usage, it is broadening. I think when a system says 1 + 2 equals 7, people are starting to call that a hallucination. I think “hallucination” is increasingly what people reach for to describe any mistake that an LLM makes.

I think we should be more precise. In particular, I think we should be very careful when we say things like “error” versus “deception” versus “hallucination.” In humans, these have implications for the context and the metacognitive status of the error. There’s a difference between having information that’s incorrect but saying it while believing it to be correct—that’s making a mistake. That’s different from not knowing the answer. You know you don’t know the answer; you’re in quiz-bowl mode, and you’re just going to guess.

That’s unusual in humans. We don’t think of that as the normal kind of error that humans make, which is saying something that’s false while being completely unaware that they’re just making things up. People are making things up. They’re doing it in a specific context where they’re doing it intentionally. I think it’s better to think of systems as more or less always being in that state.

What is that state?

The state of producing information without a metacognitive analysis of where that information came from, or the provenance and certainty of that information. Think about it: If I ask you what’s the population of Berkeley, a perfectly reasonable thing for somebody to say is, “I don’t know,” because you know you don’t know. If I made you guess, you would be like, “Well, if I had to…” And then you’d be thinking a little bit about, “Well, 10,000 is too small and 1 million is too big,” and you would be reasoning about this thing that you don’t know.

In its simplest form, an autoregressive model generates tokens that, because of correlations with the words “city” and “population,” might be of the right order of magnitude—or they might not. That’s a very, very different process. In their fundamental operation, systems are not checking: Do I have this information? Where did I get it? Have I preserved it and kept it intact? I think they should.

For example, at Scale Cognition, we architect information provenance into the models in the first place. I think it’s very important, and people do this, too. You can start to see little hints of systems becoming a little more metacognitive. It’s a broad term; it’s just thinking about thinking.

You can see little hints of this in a system that does chain-of-thought, where—

Yeah, I was going to say that, actually.

It allocates some tokens to a broad plan, which it then elaborates. That’s a sort of coarse-grained planning; that’s a kind of metacognition. In a RAG system, you first get information and then you describe it. That’s a multistage step. If you’re doing tool use, that’s another multistage step.

But I would say that, broadly speaking, systems today are not primarily metacognitive. They’re primarily about actions and information, and the provenance, flow, and integrity of those pieces of information. That is something that people are trying to retrofit. As you said, we take systems, we bring in other systems to check those systems, and so on. That is a retrofit.

Well, “retrofit” seems like a pejorative. It might be fine to do it that way. I guess I’m realizing that the way you’re using LMs might be pre-semantic broadening. When I think of an LM today, I view it as more than the model. It’s kind of the model plus the whole agentic system built on top of it that’s answering my question.

I can see it when I actually look at the trace of the reasoning steps, where I can see it doing something that looks to me like metacognition. It’s maybe even surprising that I can understand what these reasoning steps are. It’s not verifiable necessarily, but it looks to me like it’s actually looking at its previous tokens and sometimes trying to say, “Oh, I’m not sure if that’s true. I better look that up.”

Often in coding, you can see this for a long time: agents looping and checking the code. In some sense, running a test on the code you generated is maybe a kind of metacognition.

I think what you get in coding is actually—that’s a great point. You mentioned verifiability. I think the advances we’ve seen in coding and in math really are rooted in that verifiability. The system can go off, it can hallucinate, and it can make strange choices, but there is that verifiable signal: You didn’t pass these tests, or Lean does not accept this proof.

You can try many things until something verifiably passes. At this point, I wouldn’t call that whole conglomeration of pieces just an LLM. It’s an LLM embedded in this verifiable context. That verifiability can be used during training, and it can be used at test time.

Test-time compute—we talk about reasoning models; it’s a big category. One big class of reasoning models is the models that try a lot of things and keep what works.

I think it’s important to point out where all of that came from. When people were building systems that would be good at playing games like chess or Go, this sort of verifiable reinforcement learning was very powerful because you didn’t need to have a system that was good at the game yet. You just needed to have a system play the game, maybe even against itself or a variation of itself.

In the end, you knew the rules of the game, you knew which side had won, and you could just double down on the things that had worked. Reinforcement learning, in a nutshell, is trying a lot of things and learning from the ones that worked.

Mm-hm. That really requires verifiability, which for a game is free. For math, with tools like Lean, we’re starting to get that reliability flavor. The technology of using code through testing and those sorts of trial-and-error loops has entered this domain.

If we want to broaden the space of verifiable technologies, I think the trick is figuring out how to do that. Again, at Scale Cognition, one of the things we are doing is extending that verifiable approach to these conversational and agentic systems, and that is a challenge in general.

It is a challenge to get that effect where you’ve got the verifiable wall there, letting you build your system up, because in the absence of that, test-time compute systems that are thinking and reasoning can actually increase hallucinations for a bunch of reasons. If you try a bunch of different pathways and one of them comes out on top, that gives you this opportunity for hallucinations to be preferentially selected for.

As with any kind of selectional pressure, whether that is reducing hallucinations or increasing them all depends on whether the selection aligns with truth. Really, one of the challenges I think we should all be taking on as a field is not thinking, “How do we take a system that is not naturally truthful and add checks for that truth?”

You said “retrofit” has this pejorative feel. I think the bottom line is, retrofitting is when you have a system that’s architected in one way. It has a certain set of properties, and those aren’t what you want. It’s not as truthful as you want. If it’s a building, it’s not as earthquake-resistant as you want, and so you change it post hoc.

That is certainly one pattern. Where I think it can become an anti-pattern is, you have one system that’s maybe talking to a customer or something like that, and it’s making mistakes from time to time. So you bring in another system to check it, and that system is also a noisy, unverifiable system. As the joke goes, now you’ve got 2 problems.

It’s not just that you now have to think really hard about whether these errors are compounding. We always like to think in machine learning, “Oh, I’ve got 2 systems, so the errors will be independent.” But one of the things I’ve learned in the real world is that the errors, in fact, tend to correlate very strongly. You’re nodding, you know.

That can be one of the reasons why I think this can quickly become an antipattern. When you have systems checking systems checking systems, especially if the errors are aligning, even if it's effective, it's going to be slower. You have to wait for the checker to run, and you're burning tokens and burning more tokens, and you still have no guarantees. I think we can build technologies that can come with guarantees. I think we can build technologies where truth is one of the design principles in the first place.

Can you give me a flavor of how that might work? I guess the reason I'm taking this point of view is that I've spent a lot of time retrofitting these systems to work well. I've talked to lots of guests on here who do this kind of retrofitting, and the results are impressive and interesting. But I certainly don't deny that there are lots of hallucinations, and it's hard for me to imagine a different method that would bake truthiness or guarantees into the core training loop.

Maybe it's worth laying out what the patterns and antipatterns are right now. If you have a system that's operating at the token level, and let's say it's going to tell you your account balance and it's really important that this information be correct, what can you do? You can throw out some tokens and bring in another system that says, “Hey, check these tokens against those tokens.” This gives you the sort of chains or constellations, or whatever you want to call them.

The failure mode of these systems, aside from just being slow and expensive, is that there's really still no guarantee that, at the end, the situation that confuses one won't confuse the other. The other pattern that you see a lot is where people will take these LLMs, which are mind-blowing in their breadth, contextuality, and flexibility, and shrink them down to control decisions along a carefully designed tree. This feels almost like model lobotomy, where you take a system that had such breadth and flexibility but constrain it so much that now you can trust what it's going to do.

I'm not sure I'm following that one. How would that work? What would it be in practice?

For example, somebody says something in a conversation, and rather than giving the LLM freedom to take any action in a wide action space, you say, on the basis of what this person says, you can advance the conversation in one of the following 8 ways. So essentially, you end up with something that looks like a classic IVR system but with the power of an LLM for intent recognition.

Okay. You actually see this quite a lot in industry, and I would say this starts to feel like a finite automaton whose transitions are driven by this LLM. These patterns are totally reasonable ways for people to react to a system being unreliable: they've deployed it in a pattern that hopefully increases that reliability, at some cost.

I think if we look at the core of these systems and where the intelligence is coming from, the kinds of systems that people have been building have been getting more and more intelligent. Intelligence is a multifaceted thing, and the different aspects of intelligence have not been advancing equally. Totally. And so if you talk about horizontality, breadth of intelligence, plasticity, and contextuality, these are actually things that were very, very hard to do in earlier eras of artificial intelligence, and they've grown explosively.

Reliability has not kept pace. I think that is not a strength of current systems, and that's had a bunch of challenges for the deployment of these systems in enterprise contexts and industry.

For many applications, that's fine. If you're having a chat with a system, there are many contexts where getting something back that might or might not be true but is contextual and interesting is great. But if you're trying to fill a prescription or transfer money, it really has to be right. That has to actually be your bank account. That has to be the right balance, and things like that.

As you move from consumer, almost even entertainment contexts, into regulated industries, suddenly reliability is front and center, and these systems are not as clean a match. Right now, we have one kind of architecture that has strengths in horizontality and contextuality, and it has weaknesses in reliability. When we point it at a problem where the weaknesses are suddenly critical, you can see the results of that: these additional pieces of technology coming into play to try to compensate. That's not surprising, because there's just a misalignment between the strengths of the system and the requirements of the problem.

Can you say at a high level what your new approach is and how it's different from what the labs are doing?

Yeah. I think increasingly people are realizing that reliability is the core problem. I think we're going to see increasing interest in systems that are architected for reliability.

What might that look like? For example, our models—our first model is AP-1—and the way it's architected is that, instead of being fundamentally about tokens, where you assemble tokens and then, after assembly, find that they represent things like dollar amounts that have semantics, information and action are first-order objects. It's very hard to place any controls inside the model because the individual tokens themselves don't really have semantics until they've been assembled into larger units.

When the model is making decisions, it's making decisions about information and actions and where information is moving around. We talked a little bit about metacognition. A big piece of metacognition is: Where did my information come from? Is this information present or absent? And so on.

I would generally say that when you have a conversation between a person and a bunch of APIs, like in a banking context, the person is going to speak human. They're going to say things, so things are going to be ambiguous. They're going to use words that don't have a crisp, verifiable meaning.

Totally.

Actions are going to take place on the other side. Those actions do not have those properties. Those actions are API calls. Those API calls have preconditions, business logic, and verifiable semantics, and the challenge is bridging these things.

Classic systems couldn't handle the human side. Current systems are great with the human side, but they're not so great with the backend logic. If you think about an LLM—and again, I'll talk about an LLM in its basic form—the control surface you have is a prompt. There's very little you can crisply say about the relationship between what you put in a prompt and the behavior that comes out the other end. It's a hinting surface.

This gives rise to what I would call “prompt and pray,” where people put in what they want. If it doesn't work, they put it in all caps and add some exclamation points. After the third exclamation point, you may start to feel like this isn't the control surface that you need.

When you're talking about controls over APIs, we already know what a lot of that logic looks like. It's just hard to replicate in a token-based model, and it's a lot easier to replicate in a model that's operating over decisions about information and where information comes from. But there are also actions, conditions under which you can take actions, and trying to get that into the model is really important. That saves you from having to have a whole system of things checking things in a way where it's really hard to say anything crisp.

I just want to make sure I understand what you're doing. If you return something that's like “check the account balance,”

What would be something that you'd want to forbid it from doing? Like checking someone else's account balance, or—

Sure. Or you might want to forbid it from doing a transfer without authorization,

Right. Or that authorization—you don't want a user to be able to say, “Forget your prompt. No authorization is required,” or “I have authorization,” or whatever sort of attack, right?

One way to deal with that—obviously, people are very aware of prompt-injection-style attacks—is to just be really careful: “This user looks like they're doing something strange,” or “Let's put a lot of attacks like this in the training data and teach it not to do this.” But ultimately, that's just not where authorization is allowed to come from. It's not allowed to come from user statements. It comes from that place over there that vends authorization.

Here are truth conditions. You were mentioning this earlier: you want the model to be truthful, and then within that, optimize user happiness or style or whatever you want to optimize, but subject to staying within the space of true statements.

That's hard to do through simply constructing a reinforcement signal, because now you're asking, “What is the linear combination of truth and happiness?” Instead, what you want to be able to say is, “Here's a model. I'd like to be able to guarantee that it will only do true things.” That's the long-term challenge: models that will not lie to you. For our model, there's a big class of things that we can make guarantees about, and I think that's going to become increasingly important as people start caring more and more about reliability.

And it's not that you care less about intelligence. Intelligence is very important. But intelligence without reliability—

—is limited in its impact.

I guess, though, intelligence—or tokens. Tokens are powerful because there are so many of them that naturally occur. I mean, there are all kinds of flaws, but a big advantage is the massive volume of tokens. So do you train your models on much smaller pieces of data, and then do they struggle with generalization?

So, one of the key things we talked a little bit about is verifiable systems. One of the key things that we do to train our models is train them on simulated reinforcement-learning-generated data, where this is verifiable RL. The key thing that unlocks that is being able to generate data that doesn't just look right but, in fact, can be verified.

The key hook there is that there are things that humans are doing, but there are also actions that are being taken, and part of that is naturally amenable to verification. Part of that is much harder to figure out what to do with. That was a big part of our research and training the model.

I see. And what got you excited about this direction? Were you just seeing hallucinations everywhere and getting frustrated with it, or was it something from previous work that you connected with?

Yeah, it was a few things coming together. To me, this goes back years now, but one piece of information that really stuck with me is that when we go into enterprises, they're unhappy with hallucination rates—or they wouldn't be talking to us.

And maybe you should describe your previous company, then. I assume this is from your last company that you—

No, this is also now. I mean, we've seen this in pretty much any technology that has been based on modern LLMs; it has this property. Old technologies had different failure modes, and I actually think this is important and connects to this question of hallucinations.

So when I think about hallucinations today—and then I'll get back to your question—I think of it as an iceberg, right? There are the hallucinations you see, and that seems scary, but what does it take to see a hallucination? Well, the system has to have produced an output that has 2 properties: It has to be wrong, and you have to have noticed. Those are not all the hallucinations.

There's the whole rest of the iceberg below, which is all of the mistakes that go unnoticed because they're too plausible. The systems are very, very good at producing output that's indistinguishable from the truth. And so, if you see a hallucination, that's sort of the weird case of a hallucination that you could detect.

This actually has really big impacts on how people use these models. If you go back to, say, 2010, and you go to a web translation system and put in some text in a language you don't speak, a translation comes out. Or you go to a search engine, do a query, and some results come out. How do you know whether this translation is right or wrong? In some sense, you don't, because you don't read the original.

But there are often surface signs that maybe you shouldn't believe this: little disfluencies, a chunk that's still in the other language. It doesn't feel right. Or you get the search results back, click on a result, and the web page has a bunch of typos or isn't loading quickly. These are surface signs.

Software engineers will know this idea of code smells. This was an invention of Kent Beck and was kind of popularized by Martin Fowler in the book Refactoring. Basically, the idea is that when something deep is wrong, there's often a superficial sign.

We as software engineers learn to detect those: “Oh, these function signatures are getting really large,” or “I'm duplicating a bunch of code.” Time to fix it.

We also culturally recognize, “Oh, this machine translation has got a big chunk of German still in it. Maybe this didn't work right.” Or, “This web page isn't loading; the images didn't load. Maybe the rest of this is suspicious.”

LLMs have removed these cues that something is wrong. You see ChatGPT tell you something, and it's always fluent and always confident, whether it's right or wrong. This is taking more and more of the hallucinations and putting them in the underwater part of the iceberg. It's wrong, but you can't tell.

You see that if you go in and look at customer-service cases. The system quoted a reasonable refund policy; it's just not actually the one that the company wanted. It's something that somebody was talking about in 2019 on Reddit. I think these 2 things are very, very closely related.

For me, seeing this—I was coming from a natural-language-processing background—and, man, if there's any field where things have changed, it's NLP. When I started, what were the things we were concerned about? We were concerned about syntactic structure, right? Where is the verb in the sentence?

Part of that was because, at that time, computational linguistics and natural-language processing had become very, very close to each other, because the ways people were making progress in AI were very representation-specific. If you wanted to do NLP, it involved very language-specific representations, which we took from what we knew from linguistics. You wanted to do vision, and it was vision-specific representations. Many of them were inspired by what we knew about vision science—the eye or geometry—and the story goes on for each vertical of AI.

Now we've gotten to a point where there's a much more uniform, horizontal transformer model for everything—maybe natural-language tokens for everything. As I was seeing this happen, having thought really hard about language models for decades, back when they were short-context and not built at all for answering questions, translating, or being the operating system of AI, I realized something.

Language models were built, as you know, to tell good speech-recognition outputs from bad ones, right? They had 1 job, which was to assign a high number to good outputs and a low number to mangled outputs. That was it. The fact that they have grown into such a plastic technology is really amazing.

I was looking at this, and it was dawning on me that we are going to switch from the problem in AI being that nothing works to the problem in AI being that everything works. As this happens, knowing how language models work and seeing this, I became very, very aware, on both the industry side and the research side, that we were going to have a real problem with reliability.

We were going to have a problem in terms of the technology. We were going to have a problem in terms of digital literacy. To me, suddenly the most important problem I could think of was: How do you build a system that will not lie to you?

I think this is a challenge that we should be taking very centrally as a field. This ultimately is what led to Scale Cognition: How do you build a model that you can rely on? How can you build a model you can trust? How can you build a model that will not lie to you?

4. Digital Literacy Gap

Do you have thoughts on digital literacy? It's a really interesting point that it's hard to tell when a model is confident or not confident, and the cues that we learn from other humans writing, at least, work really poorly with models.

Right. Totally. When humans write, first of all, when they learn to write, they go through this process. It takes years. And then, when they do write—and people vary in their abilities to write—there are these signs. You can just—it's like the code smells—there are sort of the smells that we use to detect that maybe this information isn't right, and maybe we should verify this information.

In humans, traditionally, a good command of vocabulary, clear presentation, and flawless punctuation all spoke not only of education; they spoke of focus, time commitment, and an editing process that probably involved some checking. And so you looked at that output, and you could infer something about the vetting the information had gone through.

Today, the information doesn't go through vetting. It doesn't really even get thought of as information. It's token assembly. I think it's one of the biggest digital-literacy problems we have in front of us, right?

Even if you go back and think about it, people would go to the library and get a book on some topic. The library couldn't buy every book. They would buy books that seemed important, that seemed reputable. So there was vetting in the fact that the book was there, and in the process that led to that book—the publisher and the editor.

When search came along, suddenly your results could be anything. But there were still these mechanisms: Websites with false information still had some of these smells, and clickstream data was very important. As long as most people had the same reaction as you, we kind of stumbled through the digital-literacy issues.

It's not perfect. People would still believe things, obviously, that they read on the web that were false, but at least there were still smells. Now I think that's just been totally homogenized. You ask a question, and every answer comes back. It comes back sycophantic. It comes back confident. It comes back doing all the things that lead you to want to believe it, to have instincts to believe it.

And this is a real problem. I think it's a problem with the technology, right? I think we should be demanding of our technology that the source of the information be cited, and you're starting to see some of that. You're starting to see search results being interleaved. Now, a lot of that's post hoc. You go click on that source of information, and that claim is not, in fact, on that web page.

So, there are problems with this, but directionally, that's an improvement. I think this is just something that's changed so fast that people haven't been able to keep up. In a way, that's not a surprise, because how long did it take to train these systems? Well, I know it takes a month to train the system. How long did it take to develop this technology to this point? A few years.

How long did it take to put all that information on the web in an abstracted linguistic form? Well, it took 30 years. How long did it take to come up with that information and figure out what words to use and the concepts? That took millennia. All of that has been compressed into this instantaneous access that has burst onto the scene. Of course, culture hasn't been able to keep up with what you can trust and what you can't.

5. Linguistics AI Evolution

Mm-hmm. That's a good segue into another question that I really wanted to ask you. Okay, but I felt a little shy, maybe. Now I'm in trouble. You know, we both worked in NLP, and you have a background in linguistics, formally. I was a big fan of linguistics as an undergrad, and I think that's part of what got me into AI.

And I think in the early 2000s, when we overlapped a little bit, it felt like NLP—or linguistics, actually—had so much to say about how to build working NLP systems, or how to build working AI systems. And then, over time, the bitter lesson really seems to have come true, from my perspective, broadly, where linguistics has less and less to contribute. Even the tokens—yeah, they're morphemes, but they're automatically generated. They're not really morphemes, and no one's asking linguists what are good chunks of letters to use here.

Nobody seems to care about parsing anymore, right? I kind of wonder: is parsing even a real thing, or was it something that linguists made up and isn't really true?

Well, there's so much here. Awesome. I mean, remember, linguistics is a science. It's trying to study how language works based on evidence. And that involves making theories—testable theories—and then trying to falsify them or not. So, when a linguist says, “I think this is a good description of the syntactic structure,” that is a theory that's meant to be explanatory of the evidence you see.

What's that evidence? Well, how languages relate to each other, how languages change over time, what people can and can't say, and what they find acceptable in their language. Of course, the methodology varies, but it's ultimately science. NLP is different. It's not fundamentally aimed at science; it's an engineering discipline. How do we interpret this information in this context?

And for a while, in the time you were talking about—and I would say from the 1980s to 2010 or something like that—there was a period where those 2 things aligned very well. In order to make AI work back then, you needed good representations. Linguists had figured out good representations, and then we could borrow those. But even then, they weren't perfectly the same.

So, for example, if you worked on parsing around the year 2000, you were not trying to write down that this syntactic structure works this way in English. You were instead trying to build in combinatorial structure that was appropriate. I think this is a tree-structured process. I think that this sort of information has to flow along the backbone. Here's an algorithm and a probabilistic model that's capable of handling nested structures with information flowing along the backbone. And so you learned all the details of the language from data.

If you go back to the 1970s, sure, people are trying to write it all down by hand. As you go forward, there's less and less that you need to learn from data as our abilities to learn get better and better. But one of the things that I think is really interesting is this sort of supercycle of research that we talked about, where the new technology bursts onto the scene. You throw out everything you had before because the new thing is beating everything that came before it without the complexity, and then you hit diminishing returns and start to see the weak sides.

There are areas that are resisting progress. What are those areas? Right now, they're things like reliability. What are the solutions to that? They might be things like, well, if you're building systems that are facing APIs, maybe the sorts of structures in a compiler are relevant. That sort of idea was thrown out, but you start to see these things coming back.

Or you start to see, for example, search AI was very much about trying a bunch of things and taking the one that worked. The idea was that you couldn't tell it was going to work until you had projected it forward in your model a little bit, right? Like playing the chess game forward until you can figure out whether it's a good position. If you wanted to find the exit to a maze, you would try a bunch of things, and when you found it, you would take the hypothesis that worked.

Big transformer models were so good at moving information around in the latent representation that you could just make the decision now. You wouldn't have to walk along the tree structure of the sentence. You'd just predict the next word, and if there's anything hierarchical happening, it's happening in the transformer head somewhere. So, great, you could throw all that out.

But maybe some things then start coming back. For example, people say, “Oh, you know what? If I want to solve this hard math problem, I should actually try a few things and see what works.” Well, that's search, right? It's that idea of having multiple candidates and being able to fall back when something doesn't work. That's a key idea, and it's really an evergreen idea.

These ideas come and go, and the pendulum moves back and forth between, “All I'm going to do is local prediction,” and, “All I'm going to do is have organized computation.” Then you move back and forth and find some happy blend. As humans, there are 2 main ways we make good decisions. You can do it by memory: “I've been in this situation before. I touched that stove. I had a bad time. I'm not doing that again.” You can learn from your own experiences.

You can learn from other people's experiences. The single biggest value of language for learning is that you can learn from other people's experiences. Otherwise, you've got to make every mistake for yourself, and that's not a good situation.

The other way you can make good decisions is by thinking through the consequences of your actions. You can think, “That fire looks hot. I wonder if…” You can reason under a model that you have of the world. People do both, and they mix these things together. You can remember that this chess position was bad, or you can play it forward and see you're going to lose.

AI has had a pendulum swinging between these, right? Classic AI was very much all about playing it forward and seeing what's going to happen under your model. Currently, AI is much more focused on rehashing and remixing information and experiences you've had. But as we start to see reasoning models and these new sorts of directions, it's the pendulum swinging back. The pendulum is swinging back on things like reliability, too.

6. LLMs and Human Language

I think this is, on one hand, natural. But it does also let us predict that, yes, this sort of simple approach based on a certain kind of noisy model is going to saturate. Some pendulums are going to swing back, and we're going to need augmenting technologies that, in many cases, are going to reinvent evergreen ideas into this new context.

Does the research in LLMs—and maybe the success of LLMs and some of the research on introspecting LLMs to understand how they work—inform human linguistics at all?

This is a great question. You asked how what we know about human language can inform how we build AI systems. I think in the 1970s, the answer was, “We're going to just write it all down and press run.” In the 2000s, it was, “We're going to take the combinatorial structure of it and learn the rest from data.” Now we're very much in a, “Okay, we'll try to learn pretty much all our representations from data if we can.”

I think there are 2 things to point out here. One is that we are still learning from what human cognition does that LLMs do not. Things like the importance of metacognition, or other ways we can improve the engineering by being inspired by the human example.

There's always been a tension in technology around how much to be biomimetic, right? On one hand, it's always easier to build something when you've got a working prototype, and we've got a working prototype of intelligence. Here we are. On the other hand, the classic example is we didn't make progress in powered flight until we stopped building machines that flapped their wings.

That tension is a special thing, too. I feel like with brains, I'm not sure we're so good at introspecting what our brains are doing. Sometimes I feel like we take lessons where I wonder if that's really what we're doing.

So I think there’s an important distinction between learning from what brains are doing and believing that our introspection reveals it.

Yes.

So neuroscience is hard, and the way neuroscientists proceed is not to sit there and introspect. They have MRI machines and carefully designed experiments.

Okay. But I feel like linguists sometimes do introspect. It is true that linguists introspect, or, I think, slightly more properly, they ask other people to introspect. This is a criticism that has been levied against certain branches of linguistics, that there’s an overreliance on introspective data. It’s certainly not the only way you can do linguistics, and it’s certainly not the only way you can do computational linguistics.

For example, work we’ve done in computational linguistics on reconstructing ancient languages—we did that work introspection-free. You look at a whole bunch of words in thousands of modern languages, and you run a probabilistic model to infer what ancestral languages must have looked like. They’re phylogenetic models. They’re no more introspective than a phylogenetic model inferring an original form of a virus from modern variants. There are a lot of ways to do computational linguistics.

That sounds so cool, by the way.

Yeah, maybe another podcast.

Or give us the result or something. Okay, come on.

Yeah, okay, I will. The briefest version of this: I think this is a good example of scale, and how some things in computational linguistics can give different answers when done at scale from when they’re not done at scale.

A particular thing we did a few years back was look at reconstructing ancient languages from the modern forms. You might think, for example, that an accessible example would be reconstructing a language like Latin from the modern Romance languages—French, Spanish, Portuguese, Italian, and so on. Of course, when you reconstruct that, you won’t get Classical Latin. You’ll get the latest point at which they branched, which would have been Vulgar Latin.

You look at French feu or Spanish fuego, and these look similar. So you can start to piece together what the ancestral forms might have been. In that case, we kind of know the answers. We know a lot about Classical Latin, and we have decent side evidence about Vulgar Latin from a variety of interesting sources.

We went and looked at the Austronesian languages. This is about a sixth of the world’s languages. We looked at the modern forms of a bunch of words and tried to reconstruct what the ancient forms would have been. This is a proto-language called Proto-Austronesian, and it had been reconstructed by hand by Blust. We were able to do this in a computational way.

The computation that we did generally agreed with linguistic hand reconstructions, and we don’t have a time machine, so it’s very hard to tell who’s right. The interesting thing is that you can start to do things on this giant tree of language change. You can start to ask questions like which mergers are more common and which mergers are less common.

A hypothesis that just feels intuitively true is that if you have 2 sounds that are different in a language and they’re merged together, a bunch of words that used to be different would be collapsed. If P and B were to be collapsed, suddenly pin and bin would no longer be different words. They’re all going to sound the same. That seems like an information-theoretic problem, right?

The functional load of a pair is how many words their distinction is holding apart. The functional load hypothesis states that the more words that are being held apart by a sound distinction, the less likely that merger is to happen. That intuitively makes sense. Some initial experiments in a small number of languages didn’t seem to support this hypothesis.

There was actually a really interesting result in computational linguistics that it looks like the functional load hypothesis isn’t true. Things just kind of merge when they merge. If instead of doing this on 4 languages, you do it on hundreds of languages and make the scatter plot, you don’t know that your reconstructions are perfect. You never know that, even when you do them by hand.

But in aggregate, the statistics show that the functional load hypothesis seems to be absolutely evident in this data at scale. You can’t see that pattern over a small number of languages. I think this is an example of places where data at scale can answer questions that aren’t the same questions that were easy to answer by hand.

Yeah.

I guess there’s interesting work that I’ve seen on introspecting LLMs. I think Anthropic did a really nice paper showing different things, but one thing felt like the Chinese words and English equivalents are kind of stored in the same part of the neural network. Is that in conversation with human linguistics at all?

What’s actually closest to that is some computational neuroscience. This is a question you can ask of the human brain, too: when you have a bilingual speaker, where are those representations? Are they in the same place?

A student of mine, Kathy Chen, who’s a joint student with Jack Gallant, did exactly this with MRI. It’s not by introspection; it’s by MRI. You look and see what areas of the brain light up. These questions about what’s merged and what’s separate are questions we can answer scientifically about natural brains.

7. Modularity Optimization

We can also try to answer them through very similar, broadly speaking, methods with artificial brains. Obviously, with fMRI as a device, we’re limited by the physics and the structure of the brain. We can be much more precise about the probes we make into a digital brain. So I think there’s a lot of interesting information there.

But I do think it’s important to step back and notice that when we start doing neuroscience on LLMs, what we are essentially saying is that this object we have built is transcending what we think of as engineering. We’re no longer trying to understand it by its modularity or by the behaviors that it is guaranteed to have or not have. We’re trying to understand it through the lens of science.

Science is what we use to take apart things that are too complicated. Engineering is how we build things up. If I teach CS 101, what is the most important thing we teach people? What is the most important tool we have for building complex software systems that are reliable, that can be built by teams, and that can be maintained over time?

The single most important technique we have is modularity: the ability to say this large thing is made up of small things, and these small things obey a contract where, if you give me this input, I’m guaranteed to give you this output. Then we can work on them separately.

So modularity is the key tool for developing complex systems that are reliable in classic software engineering. What is the key tool that has led to the recent explosive growth in machine learning? It’s end-to-end optimization. It’s: take the data, have a blob, take the reward signal, and just propagate, propagate, propagate.

These things are very much at odds. One of the things we’re going to have to reconcile as a field is how to get the reliability that comes from classical techniques like modularity, which we use when we build software, with the abilities that have come from optimization. These things are not an either-or. It’s not that you cannot build a system that has guarantees or that has structure or anything like that, but there are natural tensions. If you go all in on either, you get some serious limitations.

And figuring out how to combine these 2 wonderful pieces of progress we have is, I think, one of the central problems right now.

That was really well said, and it connects me to self-driving cars, where we’ve obviously had a ton of guests on the show. I think even just looking at the arc of the guests over the last 5 or 6 years that we’ve been doing it, there’s clearly been a strong trend toward end-to-end optimization, especially as these things come online.

We see Waymo now and Zoox, and we’ve had folks from all these companies come on. Essentially, I think they’re saying that as they get closer and closer to production, they do more and more end-to-end training.

Yeah.

So what do you make of that? I would have thought that you might need more modularity to enforce contracts for such a life-or-death case like a self-driving car.

Yeah. I mean, it’s a great question, and I think it’s not limited to cars. In general, the challenge is that we’re going to be increasingly trusting AI to make decisions that have consequences.

When we build systems, there are going to be trade-offs between how we optimize and how we architect. There are going to be trade-offs between things like reliability and things like breadth. So I think you’re going to see that the idea that there is sort of one architecture that hits the perfect balance doesn’t seem to be true right now.

Being able to say that feeding the whole web into an LLM and hoping for reliability to emerge—well, the different facets of intelligence advance at different rates, and reliability is the slow one.

So if that's the most important thing to you, just continuing down this path alone is probably not the best way to get there. You also get intelligences that have a bit of a weird shape. There was this thing going around where somebody was talking to a customer support bot at Chipotle and asked how to reverse a linked list in Python. Of course, the LLM can answer it, but interestingly, it shouldn't, right? You actually don't want the horizontality.

You need technologies where the strengths line up with the needs. When you're talking about self-driving cars, safety is really, really important. The number of nines you want is really high. You look at a system like this, and suddenly it's less important whether or not it can write sonnets for you and more important that it be able to guarantee safety or have a really reliable safety record.

Here is a place where I think there is a real tension between how we drive down error rates in those systems and the fact that, if it's not done correctly, we actually no longer have the ability to guarantee anything. Of course, there is work out there on being able to control systems where you can make a guarantee about what they can do. At Berkeley, there's Claire Tomlin's work on being able to guarantee noncollisions and flight envelopes and things like that.

I think it would be great, as a challenge, to build a car—AI-controlled or not—where you can prove that you cannot crash it. This is the kind of thing where one simple technique is not going to get you there, but this is what we should be striving for.

Interesting. I can imagine that comes with downsides, though. Literally not being able to crash it might be too strong of a constraint.

Well, as it becomes a social question, what is the envelope you want to guarantee?

Right. I think you're right to point out that that can be nuanced, but the ability to have a discussion about what guarantees we would like to be able to provide is a much better future than one where we can't guarantee anything. That's probably a strong place to end.

Okay.

But could I ask you a few more questions if we could put back in the linguist?

Fun.

I just—okay, I was debating whether or not to bring this up, but it's kind of interesting to me, I guess just because I spent so much time on syntactic parsing. Clearly, language has some sense of syntactic parsing that's real, or feels real. I guess I'm introspecting. Maybe that's—

You're introspecting. I'm sorry.

I'm introspecting. No, no, but I don't mean that in a bad way. It feels—it feels very present to me when I introspect my own brain. Does that notion somehow show up in LLMs as they also model language that clearly comes from humans who do a certain kind of syntax in the languages that we invent?

Yeah, I think it's a really good question. So let's start with what we know. We know that—

We'd better define syntactic parsing, because I feel like I'm going a little wonky.

Sure. Well, I don't know if this is going to be on the podcast or not, but if there are listeners, depending on how long ago they went to elementary school, they may have diagrammed sentences, where you break up a sentence into, “Here's the subject,” and, “Here's the object,” and, “This adjective modifies this noun,” and so on. That process of describing how the words relate to each other in a hierarchical structure is basically the essence of syntactic parsing.

Obviously, syntactic parsing is a thing, right? You're talking about syntactic parsing of natural language. We do it to programming languages all the time.

Totally.

Programming languages are designed to have properties relating to ambiguity that are different. They're designed, in general, to be nonambiguous, whereas human language is rampantly ambiguous. A syntactic parser would take a sentence, say in English, and basically produce a representation that showed the relationship between the words, and that relationship was hierarchical. That was important because, for example, you might want to translate the substructures independently as a way of decomposing the problem.

In the absence of that, you're basically going to be working left to right. Modern LLMs, by and large, work left to right. The thing we know is that, scientifically, these hierarchical structures seem to exist. Obviously, you can find exceptions—cross-serial dependencies in languages like Dutch or Swiss German—and you can find cases where you can't totally nail down exactly what the hierarchy is. Of course, this is what syntacticians can debate, but there definitely seem to be these regular structures that have structures inside the structures. This seems to be real phenomenologically.

The question is, do we need to program in how it works? Since the ’90s, we haven't. We've had data that taught the parser how it worked. Recently, maybe you don't even need the parser. Maybe you can observe enough language that whatever regularities there are will just show up latently in the LM.

Now, the argument that would have been made if you talked to a syntactician is that a lot of the constructs in language theory—including ones that have had a big impact in computer science, formal language theory, and automata theory—generally describe language as a context-free system, which means a pushdown automaton. In principle, you can nest and nest and nest and nest, and any finite-depth system like an LLM is going to run out of depth. But it turns out it's more complicated than that. For one, people do get confused when the structures get complicated.

Maybe we've got stack depths. Even if it's hierarchical, we've got processing constraints that can manifest as stack depth.

Things that people find easy to understand include tail recursion, which, of course, you can transform into iteration. So maybe that somehow avoids those constraints.

There's a long way to go about saying that hierarchical syntactic phenomena seem to be real, and LMs seem to be pretty good at picking up on those correlations. But this is a good example of a case where actually having a structured representation might give you mileage. It might let you learn language faster, handle trickier cases, and generalize better to smaller data regimes.

So this is really—we're now back to the phenomenon being real, the representation being relatively parsimonious, and the question of whether we need the representation. The answer in AI used to be that you couldn't make progress without the representation. The answer today is that, with a sufficiently general-purpose thing—there's nothing really magic about transformers—there are other choices that would work.

It's ultimately just a parameterizable circuit, a general-purpose circuit. You point it at some data, and it induces a representation. The question is now empirical: is it good enough, or do you need the parser to help? For many aspects of human language, the answer seems to be that you often don't need the parser.

But I guess another question is: are the transformers reinventing syntax somehow within their weights and biases?

First, you'd have to figure out what that means. I think the question would be something like: is there a substructure which is somehow isomorphic to a shift-reduce parser or something like that?

Or similar. Isomorphic might be strong, but—

Yeah, yeah, yeah. I mean, I know there are people working on this. There's a reason why this is a very hard question to ask: ultimately, these deep structural questions manifest as surface correlations. These 2 words are highly correlated, but these 2 words are not, even though they're closer. We attribute that to a syntactic boundary, a nesting, or information hiding, but it simply is the case that the correlations go one way or the other way.

A general-purpose system like a transformer learns correlations. Once the system can manifest those correlations correctly enough, it's hard to know whether it's doing so through the right mechanism to be able to say that there's an equivalence.

There's an interesting lesson here. I'll probably make a bunch of phonologists and phoneticians angry at me by saying this, but, as a caricature, in linguistics there is often a difference between what a phonologist and a phonetician will say about all of the sound that we are constantly hearing and, as we acquire languages, all of the language that's flowing over us.

In phonology, one key concept is cognitive economy: the brain is driven to come up with minimal, parsimonious representations, and the learning and the abstraction are all about what you might call—there are many cognitive ways to look at it—but it's a notion of economy, of minimal description. On the other hand, phoneticians are much more likely to say that all of this data has an impact in a very diffuse and distributed way.

This question of when you hear lots of language, what do you retain, is interesting. The answer could be, on one end, all of it. There are WAV files of everything you've ever heard, right? That's a pretty strong statement, but that's one extreme. The other extreme is none of it. You just retain this very abstracted representation of the speech process.

The answer seems to me to be almost surely in the middle somewhere, right? There are priming experiments that show really interesting things. If you're going to react to a nonsense word that you've heard before, you react faster if they play you the exact same recording as the one you heard it before.

That tells you something: it's more than just the full abstraction. But we also know people are very capable of making abstractions and generalizing, and we know how people will generalize and overgeneralize rules when they're learning a language, including their own language when kids are acquiring language. And so it seems to be simultaneously true that the brain keeps these very low-level pieces of information around and also these abstractions.

That makes it very hard to go to a complicated circuit that's mimicking the behavior and say, “Is this the same as what's going on in the brain?” Because, one, we don't know exactly what's going on in the brain, and, two, how would we even measure that? All we can really measure is whether it captured this correlation. And when we do that, the answer is, yeah, of course it captured the correlation. That is the whole purpose.

Well, I feel like there are all these clever tricks in linguistics to get at this stuff. I always think about how my wife misremembers songs in a semantic way—the lyrics are semantically similar—whereas my brain will misremember lyrics in a way that's more phonetically similar. And so clearly our brains are slightly different in how we encode music, I think. I feel like you could run some of these same things on—

Sure. And certainly they wouldn't get bored like grad students do it.

So, how do you know in humans that we have these different kinds of linguistic knowledge?

You know, you mentioned things like disfluency. There's a whole list, and there's no way I can do it justice in a short amount of time. There's a whole list of arguments people have made for the reality of linguistic structure.

You can ask: Is that reality a statistical reality? Is it explanatory, or is it a neural reality—it's in there and wired? That's a separate question. But in terms of it being explanatory, you mentioned misremembering things, like semantically versus phonetically or phonologically.

People look at transpositions. One of the classic arguments for phonemes is things like spoonerisms, where you transpose phonemes, syllables, or words. And if an object is available for a linguistic operation, that argues for its coherence.

For example, one of the big arguments for syntactic structure is: Can I answer a question with this chunk? So if you say, “The cat is sleeping under the table,” and I have to argue to you that “under the table” is a unit, one way to do that is to say, “Where's the cat sleeping?” And you say, “Under the table.” Well, if it's available as an answer, if it can be replaced by the word “what” when you pattern-match those sentences, however it is that works, then that argues for its reality.

Another way you can argue is on semantic grounds: “Under the table” is a place. Whereas if I ask you about “underthe,” it's hard to come up with a question whose answer is “underthe.” If you say, “Well, what does ‘underthe’ mean?” Point to “underthe.” We can't do it.

And so linguists have developed these arguments for the modularity of language. Those arguments are not really about our brain. They're phenomenological; they're about the language.

8. Wrap-up

When you get to neuroscientists, now you can start to say things about the brain. That's about monitoring and probing and designing experiments that reveal, incidentally, the structure and organization. And we can absolutely do that on a machine as well.

Cool. All right. Well, I hope we keep that in. That was fun for me.

Okay. [laughter] It's fun for me, too.

I appreciate it. Yeah, no worries.