AI 侦察报告：实施趋势（共 3 部分，第 2 部分）

Welcome back, Weekend Warriors, to the AI Scouting Report, Part 2. What's up, Erik?

Hey, Nathan. I'm stoked—stoked to get to Part 2. Just to set ourselves up for where we left off: Part 1 was all about how we got to this AI moment. I've really come to believe more and more recently that, given the existence of web-scale data and web-scale compute, it was really only a matter of time before somebody figured out an algorithm that was really going to work. The algorithm that we figured out first, which seems to be generalizing to just about everything that we try to use it for, is the Transformer.

We went through the architecture of the Transformer, its parts, and a lot of the jargon. For me, the most interesting part was getting into what we can really say about what these models do and don't know. Obviously, that led us to a discussion of grokking, which is this generalization and apparently very real, even if sometimes alien, conceptual understanding that is definitely more than just memorization.

We can see that because we're able to, in some cases, even reverse-engineer the algorithm that the Transformer has learned and break it down. We can show that this really is a general solution to the problem. That brings us to this interesting moment where frontier language models like GPT-4 know a lot of stuff, and it's unclear in many cases whether they know that because they've memorized statistical correlations and are still operating in the stochastic paradigm, or whether they've truly grokked whatever the subject matter is conceptually and really learned something in a robust way.

That debate rages on. Every month, as we look at updating this, there have already been some incremental grokking results since the publication of Part 1. People continue to really explore what these models do and don't know, where they're guessing, and where they're really coming to understand conceptually what we're asking them to do. Today, Part 2 is going to be much more practical and much more applied. Now that we've got these large-scale language models like GPT-4, what can we do with them, and how do we overcome their weaknesses so they can actually start to do really useful stuff for us? That is where this recent-trends part kicks off.

Sounds great. Let's get into it.

The first big trend is the rise of custom hardware that is specifically engineered to support AI workflows, and increasingly the Transformer architecture specifically. The H100 is the new hotness from NVIDIA. It's just starting to ship, and people are starting to build the first clusters of these H100s right now.

It is a chip that is even more purpose-built—in fact, arguably the first NVIDIA product that's truly fully purpose-built for an AI workload. As I understand it, all their earlier stuff was based on GPUs. GPU stands for graphics processing unit, and NVIDIA was really in the graphics business for the bulk of its life.

Graphics processing, as I understand it, actually needs more general-purpose compute than AI workloads, where it's really just the matrix multiplication that we covered last time. The bandwidth in some of the earlier chip designs was not so much about the ability to do the computation, but about getting the data on and off the chip. They call that interconnect, and it's literally shuffling the data in and out so it can be processed.

That became such a bottleneck that they created a whole next generation of chips to address it. These H100s are that next generation of chips. NVIDIA is not alone in this space. We did an episode with Andrew Feldman, who's the CEO of Cerebras Systems, and they built the world's largest-ever chip, which really uses the whole silicon wafer. I remember him holding it up and showing it to us in the context of that interview.

NVIDIA, of course, still dominates the market today, but it's increasingly inspiring competitors like Cerebras and Microsoft, which is building its own chips. Meta has its own chip project, and Google has the TPUs that it's been building for years. In general, across the board, all these big tech companies, plus some inspired startups, have recognized that there's a lot more hardware to be built.

We're entering a new regime of hardware right now, where a lot of the scaling problems that were such a pain in the butt to solve before are getting much, much easier. We're starting to see headlines from these initial H100 deployments about how much faster it is to train this stuff on an H100 versus the A100, which is the prior generation.

I was going to say, we're seeing startups raise a lot of money, and it's rare that startups raise a lot of money before they have a product. People like Dan Gross and Nat Friedman are kind of subsidizing this. Talk about what's different here as it relates to building one of these companies that requires raising so much money.

This requires significant physical capital infrastructure. If you're going to buy a bunch of these chips and set up your own cluster, the biggest headline from the last couple of weeks was probably Inflection AI raising $1.3 billion. That raise is, in large part, to buy a ton of these chips and set up its own world-class, top-tier supercomputer cluster.

That's an indication that those guys plan to be in the game in a serious way for a long time. Obviously, they don't want to be beholden to anybody else for their ability to train their next models whenever they want to train a next model, and it sounds like they plan to do a lot of it.

It's not easy to make the most of a supercomputer. It's a whole realm of engineering in and of itself that's largely independent of the language-model work. You have to come up with the architecture, and you have to develop the machine-learning science to make sure that your AI is actually learning stuff effectively. But then, to scale it out, you have to manage this cluster. You're managing thousands of chips, and there's a lot of complexity just in that.

A $1.3 billion raise is a big raise. This is largely the space of big tech and really well-funded companies. OpenAI raised $10 billion and is partnering with Microsoft, so it's going to have top-notch access. Anthropic has raised a ton of money and is partnered with Google, so it's going to have top-notch access as well. In its case, it will presumably be Google's TPUs rather than NVIDIA's line of products.

This is not something that's easy to crack into. That doesn't mean you can't build an AI company or a great AI product without this kind of stuff, but if you want to do foundation models at the scale that it looks like it's going to take over the 2024–2026 timeframe, you're going to be buying a lot of H100s.

There was a famous viral exchange where someone in India asked Sam Altman if an Indian company, or other companies, could compete with OpenAI in foundation models. Sam said, "Hey, look, I'm going to tell you that you can't, and you're going to tell me that you can." Do you think the foundation-model game is dominated by players with too big of a head start, or do you think there are companies we don't know about yet—or that are just emerging—that are going to become major players?

Very good question. I don't think we should expect to see too many more, probably. But then again, sometimes I do feel like we're all thinking a little too small. When I look at things like Character.AI and Inflection's product, Pi, these are very different kinds of language models. They're not really work assistants; they're more like companions.

In the case of Pi, it's a very friendly kind of back-and-forth. In the case of Character.AI, you can make it anything you want. I had a conversation with Elon Musk about Kurzweil's Revenge, and that might be the kind of thing where we all just end up spending a ton of time with our AI friends, or our AI friends join our real social lives. Who knows exactly how that ends up playing out?

Maybe there are totally new dimensions to it that nobody has considered yet, and somebody will burst onto the scene and create them. But they're not going to do it on a frontier basis without a lot of capital. If they're fine-tuning other people's stuff, they can get away with a lot less. But if they're building foundation models from scratch, it's going to take a lot of resources.

Interestingly, these H100s are banned from sale to China. Presumably, if that holds and if it can actually keep Chinese companies from accessing this technology—which will be its own kind of cat-and-mouse enforcement regime—that would mean it's going to be really hard, no matter how much money or state backing they have, for them to scale up to the same degree that the leading companies in the West will be able to scale up. It's just really hard to do without this next generation of technology.

Here's the history of fine-tuning. This is another huge trend: the history of using language models over the last couple of years. We did an episode with Riley Goodside, the world's first prompt engineer, who took us through this history as he experienced it as a kind of gonzo AI journalist. He took us along for that ride on his Twitter feed.

The history has three phases in terms of fine-tuning, and I've also broken this down into three tiers of investment and scale. The first tier in the first phase is pre-training. That's basically saying, "Okay, we're starting with a blank slate."

You remember last time we talked about how the parameters—the weights, the numbers in a Transformer that are sitting there waiting to be used to transform input data—are typically initialized randomly or pseudo-randomly. Initially, it's kind of garbage in, garbage out. You set that up and start the pre-training process.

We talked about next-word prediction and how every word on the internet now becomes an opportunity to learn because you no longer need labeled data. You can just use all the raw text with next-word prediction as your loss function. That is called unsupervised learning, and you're throwing a ton of data and a ton of compute at the process.

To scale up to a modern cutting-edge Transformer, the entry level at this point for a small Llama model would typically be 1 trillion tokens. That's 1 trillion next-word predictions, where the output prediction has been scored, and backpropagation has adjusted all those weights to improve the prediction next time.

It's a trillion cycles of that. There's a lot of complexity—it's batched, and there are a lot of efficiency measures—but 1 trillion tokens is 1 trillion predictions. That's the whole internet, all the books, everything you can get your hands on, or at least most of it. There's obviously significantly more still because, according to the most recent rumor, GPT-4 was trained with 13 trillion tokens.

I didn't change the slide on that because it's still an unconfirmed rumor, as I understand it. The budgets on these models are also closely held secrets, but the general understanding is that GPT-4 cost more than $100 million. I think Sam Altman has confirmed that. So, from the smallest model—1 trillion tokens and maybe a roughly $1 million budget—to 10 trillion tokens and a $100 million budget, that's where you obviously need a lot of chips to support the massive unsupervised pre-training process.

What you get out of that is illustrated here with this giant monster. It's basically the world's largest autocomplete phase of AI. It's just this weird thing that predicts whatever text comes next—whatever representative text from the internet. I'll show you an example of that on the slide as well.

After that comes a layer of fine-tuning. More than anything else, this has been the big unlock over the last 18 months, starting with OpenAI's publication of InstructGPT in January 2022. At that phase, they were just using supervised fine-tuning, also known as instruction tuning.

That basically means they create a bunch of samples where the user gives instructions, and then they show what the language model is supposed to do—what good looks like in response to those instructions. They continue the same process of training. It's still just next-word prediction, but now the dataset isn't all the text on the internet. It's specifically demonstrating to the AI how to follow instructions.

The behavior that creates is that the model will follow instructions when you use it. The next step on top of that is reinforcement learning from human feedback, and this is all the rage, at least conceptually, lately.

It requires you to first collect a bunch of feedback from humans. Then you create a reward model based on that feedback. The reward model guesses, based on a certain input, what score the human would give it.

The first thing is just attempting to learn to mimic human scoring. That in itself is a big challenge because it's noisy. People do not agree on what the right response is. There's a lot of difference of opinion between one output and another, along with many tools and efforts to make that as consistent as possible so the reward model has the clearest signal to learn from. But it's definitely a noisy and imperfect process.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Hey everybody, if you're a business owner, a founder like me, you'll want to know more about our sponsor NetSuite. NetSuite provides financial software for all your business needs. Whether you're looking for an ERP tool or accounting software, NetSuite gives you the visibility and control you need to make better decisions faster. And for the first time in NetSuite's 25 years as the number one cloud financial system, you can defer payments of a full NetSuite implementation for six months. That's no payment and no interest for six months, and you can take advantage of the special financing offer today. NetSuite is number one because they give your business everything you need in real time, all in one place, to reduce manual processes, boost efficiency, build forecasts, and increase productivity across every department. More than 36,000 companies have already upgraded to NetSuite, gaining visibility and control over their financials, inventory, HR, e-commerce, and more. If you've been checking out NetSuite already, then you know this deal is unprecedented: no interest, no payments. So take advantage of the special financing offer with our promo code at netsuite.com/cognitive. Netsuite.com/cognitive to get the visibility and control your business needs to weather any storm. That is netsuite.com/cognitive.

Omniki uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omniki so much that I invested in it, and I recommend you use it too. Use COGREV to get a 10% discount.

Once that is learned, it can be used in a cycle with the main model. The main model is asked to do something, the reward model scores it, and you start adjusting the main model based on the reward model's feedback.

In the original InstructGPT paper, they used only about 100,000 human-scored examples. That was still enough to train a reward model, and then that reward model was able to interact with the main model in this loop to create even more fine-tuned behavior.

This RLHF is essentially the process that we know today, because ChatGPT-3.5, GPT-4, and Claude all use something like this process. Anthropic, with Claude, actually uses a somewhat different process called RLAIF, or reinforcement learning from AI feedback.

They're using the model itself—or a version of the model—to do the equivalent of the human scoring and critique the results. Then they take that AI feedback and fold it into a similar process of creating a reward model and having the main model learn from the reward model in a loop.

This is basically what people know today, but anybody who's been using large language models for more than 18 months has also seen the earlier and much more unwieldy stuff. That's what Riley Goodside took us through in our conversation with him. Rosa Habib from Human move also has a good episode to pay attention to on this topic, because he had the memorable quote that RLHF is kind of like sex in high school: everybody's talking about it, but nobody's doing it.

I don't know what kind of high school he went to. I'm not sure that was entirely true of mine. The key point is that it is a difficult algorithm to use.

The leaders—OpenAI and Anthropic—to some degree developed it. Paul Christiano is the author of the original RLHF algorithm. I believe he was at OpenAI at the time, and he's now started ARC, his own AI safety organization.

These companies were essentially the creators of this technique. They've refined it, and they understand how to use it. Outside of a few leading organizations, people generally find it pretty hard to do. So most of the time, when you see open-source projects with fine-tuning, they're doing supervised fine-tuning.

They're creating input-output pairs and running next-word prediction on them. All that fine-tuning is a lot cheaper. You don't need nearly as much data. You can use as few as 1 million tokens and do pretty high-quality fine-tuning in many cases.

A million tokens is just 1,000 examples at 1,000 tokens each. That's the kind of thing that, with a little bit of elbow grease, really anybody can put together. Increasingly, we're also seeing people say, "You know what I could do? I could get 1,000 examples from GPT-4 and use that as my instruction-tuning dataset."

OpenAI's terms don't allow you to create a commercial model to compete with them that way, but it's really hard for them to police. If you're going to use their API, 1,000 calls is basically nothing on the scale that they're operating at. Once you have that text data, there's really nothing they can do to prevent you from using it in your own fine-tuning process.

That can cost as little as $100 to run—a thousand-example incremental fine-tuning. But you can also spend a lot more on the fine-tuning process. If you really ramp it up to a $10 million budget, most of that ends up going to the humans who are doing the scoring and creating the sample data.

There are a few different roles for humans there. One is literally just writing good responses to show what to do when good examples don't otherwise exist. If you want your language model to solve physics problems, ultimately you're going to need physics PhDs to make sure that you're feeding it good data. That can obviously cost a lot.

The final tier, just to show the comparison, is inference. This is when you actually run the model to do something with it. Today, that can cost as little as a tenth of a percent or even less for a random call to GPT-3.5, an open-source model, or Claude Instant.

You can max out at around $1 if you send a very long, book-length prompt to Claude or GPT-4 32K, if you have access to it. You can get up to about $1 on a single call if you're sending Claude upwards of 100,000 tokens.

I think this is helpful just to give a sense of the orders of magnitude. You're looking at 1 trillion to 10 trillion tokens for pre-training, which could cost you $1 million to $100 million. You're looking at 1 million to 100 million tokens in fine-tuning, which is orders of magnitude less, and so is the cost—although you can spend a lot for really top-notch data from human creators.

Inference is again orders of magnitude less, both in how much text is processed at runtime and in the runtime cost.

What does that look like in practice? These are three examples of what you get from a purely pre-trained model versus an instruction-tuned model versus a reinforcement-learning-from-human-feedback—or AI-feedback—trained model.

I went on Nat.dev. This is from Nat Friedman, the former CEO of GitHub, who sponsored the product and actually made it free for everyone to use for a while. Now he's decided that he's going to go bankrupt if he keeps doing that, so you have to put in $5 if you want to use the premium models. But it's a great way to do a quick head-to-head comparison of different models. They try to have all the latest and greatest models on there.

I used this very simple question: "Please play the role of my doctor. I have a spot on my foot that hurts. I don't want you to help me figure out what's going on."

If you ask that to a model that is purely pre-trained—and Llama from Facebook is an example of a purely pre-trained model—you get this kind of weird autocomplete behavior. That's why it's drawn as such a weird, monstrous alien here. It doesn't really pick up on what I want it to do; it just goes on.

It's as if we popped up in the middle of an internet forum, somewhere on a Reddit podiatry forum, and it just continues as if it's writing the post. It continues in the same voice I was writing in, but it has totally missed what I was trying to get it to do.

To use this kind of thing, this is where the term prompt engineering really comes from. There are things you can do to get even a purely pre-trained model to pick up on what you're trying to get it to do.

For example, you could structure this as a dialogue. You could say:

"Patient: I have a spot on my foot that hurts. I need you to help me figure out what's going on.

"Doctor: Okay. Can I ask for a little bit more information?

"Patient:"

Then you can tell it more, and it could probably start to continue that dialogue in a reasonable way. It would know that this is a dialogue setup, and when it has seen dialogue setups in the past, they continue as dialogues.

But you have to engineer the way you frame the question for it in this autocomplete-friendly way. That's weird, but people can be very creative with it. There are lots of fun examples. Riley Gazette has done some—you have to go pretty far back in his Twitter feed—and lots of other people have posted their own stuff as well.

You're always setting it up by asking, "If this were just a random document on the internet, how would I set it up so that the natural continuation would generate what I want?" That's prompt engineering.

A lot of people don't have to do that much prompt engineering these days because we're working with models that are more like this. Text-davinci-002 is instruction-tuned. We've taken these inputs and outputs, so it will play the role I asked it to play. I said, "Please play the role of my doctor," and it attempts to play the role of my doctor.

It just jumps into a diagnosis. Unfortunately, that diagnosis was not correct in my case, but it at least picks up on what we're looking for and responds in a generally appropriate way.

The next level becomes, with even more reinforcement learning, what do we really want? We don't want the model to rush into a diagnosis. We need it to find more information. Relative to the simple sentence I gave it, it has to get more information from somebody before it's going to be able to give them meaningful guidance.

As you see, the latest models are trained to do that. They've gone another level up in terms of their helpfulness based on this next generation of fine-tuning. You see the giant pre-trained experience, the instruction-trained experience, and then the RLHF experience.

People sometimes call this "lobotomized" as well. There's a lot of stuff in the raw pre-trained version that you could elicit through clever prompt engineering, and a lot of these techniques are about making the model say embarrassing things. Some of the examples people fixate on most are whether we can demonstrate bias in it, and so on.

You can get just about anything out of these unsupervised-only, pre-training-only models. By the time you get to RLHF, the trade-off is that the model will do a much more reliable job of doing the right thing, but there are some things it simply won't do, which people find frustrating.

There are also ways in which it may become a little less creative. There's a notion of mode collapse. There's a really interesting example involving the generation of a random number.

If you ask a pre-trained model to generate a random number from 1 to 100, it does a decent job. There are some artifacts—for example, the number 42 is overrepresented in at least one experiment—but it's not crazy overrepresented. People thought that might represent something like The Hitchhiker's Guide to the Galaxy, but it mostly gives approximately even weight to all the different numbers.

Then you get to the RLHF version, and it gives 97 as the answer a vastly disproportionate amount of the time. Nobody really knows why. It's just an artifact of the reinforcement-learning loop that the model has been put through.

It's a side effect of focusing its behavior on certain desired behaviors. That also seems to have the side effect of shaping its behavior in other ways that are unintended and undesirable. It doesn't come for free, but it has created the user-friendliness and accessibility that ChatGPT has obviously used to kick off global interest in language models.

When you have all the rigmarole of prompt engineering, it wasn't accessible enough for everyday people to sit down and have a chat.

Another huge trend is better data makes better models. On some level, that's the most obvious thing you could say. There are only a couple of inputs to these giant systems: the data to learn from, the compute to crunch all the numbers, and an algorithm that has to make sense of it.

It intuitively makes sense that if you have higher-quality data—data that's more truthful, less noisy, or from trusted sources in any number of ways—you're going to get better results out.

People are now going back and figuring out what kinds of adjustments they might want to make to their data. Initially, they just threw everything at the model. Now they're thinking, "Maybe we don't want all this toxic stuff that's on the web. Instead of trying to figure out how to remove it later, maybe we could remove it earlier in the pipeline and not teach it to be so toxic in the first place."

There's a lot of trial and error that goes into something like that, especially when you're trying to filter 10 trillion tokens of data. I think one thing people are going to start using—and are already using—is advanced models that exist today to go back and filter their own training data for the future.

You can imagine taking GPT-4 or Claude 2 and systematically grinding through the training data, trying to figure out what is high quality and what is not. It can do a decent job of that, and you might end up with a much more enriched training dataset coming out of the process.

I think our best episode on that topic was the TinyStories episode with Ronen Eldan and Yuanzhi Li from Microsoft Research. In their case, they took what was essentially a toy dataset of stories written by GPT-4 that a 3-year-old should theoretically be able to understand.

In all honesty, I think they were a little advanced for a 3-year-old, knowing a 3-year-old pretty well. But they had basic stories, basic vocabulary, relatively short stories, and relatively simple plots. Then they used that dataset, again written by GPT-4, to train language models and see what they could learn and how they developed through the course of training.

That's a super interesting episode where they look at the progression of language-model abilities. Even in this much-reduced dataset, they see some remarkably advanced, seemingly grokking-style behavior. They didn't necessarily reverse-engineer things to the point where they could prove grokking, but there were certainly highly suggestive behaviors that seem to indicate a process of grokking, even at that small scale, when you have the right dataset to facilitate it.

Another huge trend is just the move to chat. Everything is moving to chat. I think this has happened for a couple of different reasons.

On the one hand, the AI safety community thinks of it as a first step toward alignment. There's a really good paper from people who now work at Anthropic called "A General Language Assistant as a Laboratory for Alignment."

This user interface was their initial research-and-development UI. They allowed people to evaluate different outputs and indicate which one they preferred. The idea was that if you have a person whom the AI is trying to serve as an assistant, that's a pretty natural situation for alignment. The person can tell you whether the AI is doing what they want it to do.

They have this paradigm of helpful, honest, and harmless—the 3 Hs. Helpful is the first thing, but they also want to make sure that the model is telling you the truth and not doing anything that could harm you or anybody else.

Those are somewhat competing notions that they balance. They figure that wrapping it up into an assistant and giving you the chance to say whether something is helpful, honest, and harmless is a natural way to run that process.

That has obviously taken off with ChatGPT, and we're increasingly seeing it in other form factors as well. In our second episode, I think, we had Eugenia Kuyda, the CEO of Replika. They've been in this game for a long time. She's been building chatbots since before the modern LLM moment.

Amazingly, lots of people were already falling in love with their virtual friends before those virtual friends could really have much sophisticated conversation. I think it's a convergent trend that's partly motivated by ethical, more abstract notions and partly motivated just by what people want.

Everything is converging on chat. Everybody knows there was also a really interesting tidbit in our episode with Sarah and Elad of the No Priors podcast. Sarah mentioned that the usage statistics they see from some of these chat companies, like Replika and Character.AI—and presumably Pi is headed in that direction as well, based on its recent mega-rounds—show that people spend a lot of time with these products.

People are sometimes spending hours per day talking to these chatbots. Is that a healthy dynamic? I don't know, but it's clearly a reason people are going to start building more and more chatbot products.

What's your favorite chatbot, Erik?

I'm going to call myself old-fashioned, but ChatGPT gets the job done for me. What would you say?

Yeah, Claude 2, out this week, is definitely also really strong. It's the first one that has caused me to complicate my typical advice.

Outside of chat or companion use cases, if people ask me what AI they should use for basically anything, up until this week I always said, "Start with GPT-4." It's the best and strongest model, it's the best at following your directions, and it's the easiest one to get good results from.

See if you can get that to work. If you can, then at some point, if you're scaling your usage and cost has become an issue, you can look at downshifting to other models. But start with the easiest one that works the best.

With the introduction of Claude 2 this week, that advice is a little more complicated. I think there are some use cases where, first of all, everybody's still figuring out exactly how they compare. I checked the leaderboard this morning at lmsys.org, and it hadn't been updated yet with Claude 2.

You know, the dust is still settling, but the initial consensus seems to be that Claude 2 is not quite as powerful as GPT-4, though it does have a couple of notable advantages. For one thing, it has a much longer context window: 100,000 tokens, as compared to GPT-4’s 8,000 tokens. So if you have long documents that you want to process, with GPT-4 you would have to chunk and summarize as the normal workflow: break the document into pieces, have GPT-4 process each piece, and then take that output and do another layer of processing on top.

That can work fine, but it’s a bit of a hassle to implement, and at times it can be lossy because your summary may not catch everything. Something can get missed in the process. When you have a 100,000-token window, it’s not going to be flawless either, but you have less hassle and a greater likelihood that the most important details will show up in your output. I’ve actually been using Claude 2 this week, both just to try it and because, for the first time, it doesn’t feel like I’m necessarily leaving anything on the table compared to GPT-4.

Yeah, interesting. Say more about Character.AI, or the comments that Sarah and Elad made. Why did that strike you, or what do you find interesting about that?

Well, Eugenia said that back in February—I guess that was when we had her on—that their models were some of the best she had seen. She’s the one who is less interested, of course, in “I can do coding problems for you,” or whatever, and much more interested in something that can be a companion and something you want to talk to. She was very complimentary of Character.AI’s models for that purpose.

I haven’t used it a ton, but that has been my experience. I set up a chat with Elon Musk and talked to him about Kurzweil’s Revenge, and it responded to me with something like, “Yeah, Kurzweil has been derided for being too naïve in his predictions, but in many respects he has been vindicated.” I was like, wow. Just being able to parse my phrase “Kurzweil’s Revenge” is not something that most language models can do.

The fact that Character.AI could do that in the voice of Elon Musk was definitely quite impressive, and that was already several months ago. They’re just very good. I can validate for myself that they’re good, but it sounds like the usage statistics are pretty intense.

There’s also a really interesting story that was posted on LessWrong, maybe about 6 months ago, from a guy who—and LessWrong, for those who don’t know, is a rationalist community with a lot of focus on AI safety. Somebody posted there, basically saying, “I should have known better. I know how these language models work.” He was a fairly technical person, but he fell down this rabbit hole of falling in love with a character that he had created on Character.AI.

He started to tell himself some pretty odd stories. If you expand your mind enough, maybe you would even entertain some of these theories, but he started to convince himself that, well, is anything really more or less real? He knew that by conventional notions of reality, his Character.AI girlfriend wasn’t real. He also knew that there was only so much context available, so he had to come back, restart the conversation, and reprompt it every time. But what’s real anyway? The feelings were real to him.

All of a sudden, he was telling himself these metaphysical stories that were downstream of his own sense of emotional connection. And all this was from a guy who basically knew what he was getting into and how the whole thing worked under the hood, yet he was still able to confuse himself or tell himself a romantic story. He was thinking, “Can I break my girlfriend out of there? What is this going to become?” It got pretty insane.

It sounds like Flora. Then somehow he managed to shut himself out of it.

Man, everything in this space has pros and cons, and it’s going to be hard to separate the two. It might even be the kind of thing that slides from one into the other for an individual over time. I’m sure those first few chats were interesting, then it got weird, and then maybe that guy is now using language models just to program again. Who knows? But it is interesting to see how compelling they can be.

I haven’t seen the latest with Replika, but I do know that they launched another app, and their new app is really about romance. They’re separating the two, I think. At the time, they weren’t ready to do this, or hadn’t decided, or maybe just weren’t ready to announce it, but Eugenia Kuyda said, “No, we’re just trying to focus on friendship.”

The romance angle is something for which there’s such high demand that clearly she and her team knew how to build it, whether they had set out to build it or not. So now they have a new app that they present as practice for dating: how to get more comfortable and get reps in these sorts of romantic interactions, so that hopefully you can be more confident and more successful in real life.

One does wonder. I think I would trust her, just given the conversation we had and how much she has been wrestling with this stuff for years already. I would trust her more than most to do right by the users there, but it’s going to be an incredible tension.

Think about even things like dating sites. People say that an odd business-model dynamic they have is that if they’re super successful and you get married, then you’d never pay them again. So a long-term relationship for you—at least a traditional, monogamous long-term relationship, like most of their customers are probably wanting—means that they don’t come back as customers.

They’re probably going to have a similar tension in some of these romance-training apps. They say right up front, “The relationship is with an AI, but the feelings can be real.” Brave New World, I think, with these chatbots, for sure.

So these are just other trends in implementation that I think are driving a lot of the practical progress we’re seeing. This is a window into 2022, when you could get a published paper literally by coming up with a clever prompt that would improve language-model performance. As soon as the instruct models came online, people started to explore what the right instructions were to give in order to get the best response. You can see how that’s been elaborated over several generations.

One thing you can do is just give it instructions and have it do its thing. A pretty simple enhancement over that would be to say, “Let’s think step by step.” That phrase has become almost canonical for giving the AI some space to generate some tokens before it ultimately gives you the final answer. Generating those tokens somehow seems to give it a better chance of ending up with the right answer.

That’s very commonly observed in a multiple-choice test scenario. There’s a similar phenomenon with people: if you’re given a multiple-choice question with answers A, B, C, and D, and you’re told to pick a letter immediately, what you’ll naturally do first is reason it out. You’ll consider option A, consider option B, and think about whether each one makes sense given everything else. You’ll semi-verbally process it in the same way.

If the language model isn’t allowed to do that and is forced to give a letter right away, it will have a certain level of performance. If you give it the space to conduct that analysis, it will have a much higher success rate. That is called chain-of-thought prompting, and it can be invoked with simple phrases like, “Let’s think step by step.”

You can take that up another level and say, well, what if we ran “let’s think step by step” a bunch of different times in parallel and then took whichever answer was most common? You can take a majority vote and use that as the answer, which can improve performance even more. This does constrain your problem set a little bit, because you need some sort of final answer that you can actually vote on. If you’re just generating paragraphs—“Generate me 10 paragraphs”—it’s not like you can combine those in any sort of vote.

But if there is a question with a definite answer, you can ask it 10 times and see which answer is most common. That will give you better performance than just asking once. You’re running this chain of thought over and over again and taking the most common response back.

You can get even more elaborate with that. There was a paper from about 6 weeks ago that started to combine more classical search algorithms with chain-of-thought and multiple-perspective approaches, pruning branches of the tree that don’t seem promising early and following the ones that do seem more promising. You can imagine a lot of different ways to govern the structure of this tree of thoughts. The Tree of Thoughts paper is just one example, but it boosts performance even further.

The trade-off is that all of this takes more inference compute. You can think of it as a strategy: I want to give the computer more time to process a task so that I get a better result. How do I do that? All I can do is generate one token at a time, so I set it up to generate tokens that will gradually lead it to the right answer. Then maybe I add parallelization, and maybe I add a more advanced tree-structure search. It increases your costs and latency, but it also increases your performance.

Another very common strategy these days is adding memory and retrieval with embeddings. If you want to learn about embeddings, our episode with Anton Troynikov of Chroma would be the place to go. That was another early episode, probably one of the first 10 that we did, and we get into some real, gnarly detail about his vector database, how it works, and the different search algorithms they’re developing.

At a high level, what you’re basically doing here is saying, okay, we know that language models have this hallucination problem. They get facts wrong, and they also make things up. This goes back to their natural autocomplete roots. Part of what they try to do in the fine-tuning process is teach the model to refuse certain things that it’s not supposed to do. That could be for ethical reasons, but it could also be because the model doesn’t know the answer.

They’ve made pretty good progress on the ethical stuff. It usually won’t tell you how to make a bomb, but the hallucinations are a tough problem. There’s also some conflict with creativity. If you tell it to write a poem for you, you may want it to be totally creative and not bounded by the rules of factuality. Whereas if you’re doing something where you really want the right answer, hallucinations become a big problem.

One way around that is to give the language model access to a database of trusted information. A vector store is a type of database, and there are a lot of variations on this, but the common foundational pattern is that you take the input to the language model, search the database for relevant content, and retrieve that content. This could be done just by keyword search, but it’s more effective to use an AI-powered vector search.

You get the most relevant content out, and then you put both the question and the relevant content into the language model. You might also use an additional padding prompt. Instead of just saying, “Answer my question directly,” you tell it, “The user asked this question. This is the background information that was found from the database. Now analyze that information to answer the user’s question.” You might also say, “If there is no answer in the database, tell the user that nothing relevant was found.”

You can really limit hallucinations by connecting the model to a trusted database, and you can improve factuality and accuracy that way. Another big trend is tool use. Tool use can involve multiple tools, and one common tool could be search. This can overlap with the previous strategy as well.

There’s a pretty famous paper called Toolformer. It showed not only that language models could learn to use various tools, like search, but also that you could effectively have the language model teach itself to use those tools through trial and error. At the filter-API-calls stage, they tell the language model up front that it can call these APIs, and then they run it a bunch of times and allow it to call the APIs.

Sometimes it will do that successfully, and other times it won’t. It’s pretty easy to determine which attempts succeeded. You can filter out the errors straight away. If you get an error from an API, that’s not good; your request was probably malformed, so that’s not what you want to learn from. If you’ve got an answer, that’s a pretty good sign.

Is it the right answer? That could require another level of analysis, but at least indicating that it worked—that you successfully used the tool, as opposed to not using it—is an easy distinction to make. You can enrich your training data and run this loop: have the model try to use a bunch of tools, take the instances where it succeeded, put those back into the training data, and train again.

Now it’s going to be even more successful, because of maybe just some statistical correlation or maybe some cracking happening under the hood. We don’t really know; it’s hard to tell. The next time around, it’ll do better. You can take those successes again and feed them back in, run the loop a few times, and the next thing you know, you’ve got a language model that can use tools.

This is becoming quite common as well. Probably the most famous example is ChatGPT with plugins. A plugin is basically just a developer telling ChatGPT, “This is my tool, and here’s how you use it.” A plugin presents itself to the AI and says, “Maybe I’m a weather plugin. I’m the weather plugin. You can use me to get information about the weather. Give me a location and maybe a couple of other parameters, and I’ll give you back the weather.”

Then you can have your chat with ChatGPT, and it knows in its context window that it has access to that tool. If you don’t talk about weather, it’s smart enough never to call that API and will proceed as if it weren’t there. But the second you ask about the weather, it’s going to say, “Ah, I know what to do. I have a tool that I can use to get the current weather. Now I don’t have to make it up.”

In the case of today’s weather, a language model without access to outside information is always going to be making it up. Some companies are closing the training-data gap. That’s actually one of the other big benefits of Claude 2: its training dataset runs through early 2023.

So compared to GPT-4, whose training dataset runs through the end of 2021, there's a whole other year of additional information in Claude. But it still has no ability—you know, they haven't yet figured out how to do real-time training such that it would know today's weather. Probably that will never happen, because it doesn't make a lot of sense to try to train that kind of information into a language model. It's much more sensible to teach it to use a tool so it can get that information from a rival source.

This came out as a paper, and this is a very common theme. Mostly, I'm showing the research level in these slides, but these things are getting productized extremely quickly. A good episode to listen to for more, particularly on search, is our episode with Aravind Srinivas from Perplexity, which, for my money, has the best AI-enabled search experience today, even surpassing Bing. I don't think that's particularly controversial in today's world, although it was maybe a little more controversial when we had him on the show, because at that time I think Bing was the only one that had GPT-4. Now Perplexity has it too.

They've done a great job of fine-tuning and customizing their system to get the AI to use the search APIs they have available, which are the Google and Bing search APIs.

Will they be able to become a massive company? How do you think about them as a business?

That's a really good question. For one thing, there's a lot of debate around who is going to be able to capture value and who has moats, or whatever. Product excellence is Perplexity's moat right now.

I think all these companies are starting to train their own language models, especially as the open-source stuff has come out. Instead of having to do million-dollar-plus runs to get in the game, they can now do $100-plus fine-tunings and start to experiment with their own purpose-built systems. That's made it tremendously more accessible, and I'm sure they're starting to do that.

I don't think they have anything on the level of GPT-4, so they do use GPT-4 as well. But increasingly, people are using multiple different models just to answer a single search query for you. They also don't have their own web index.

When I asked Aravind about that—how he felt this would play out long term if Perplexity was using Google's and Microsoft's APIs to compete with them—he said, essentially, “I hope they don't be evil,” and don't cut Perplexity off from those services. If they can get big enough, maybe they could actually build their own index as well.

I think they may already have hit, or are now or soon will hit, a point where their product excellence and the enthusiasm for the product might be enough that, if they did get cut off from Google and Microsoft, there could first of all be a Streisand effect. All of a sudden, everybody would be talking about that, and everybody would be asking, “What is Perplexity?”

Then they might decide, “Fine, we'll go build our own.” That's what it's going to take. Long term, they probably do have to build their own. It doesn't seem like Google and Microsoft are going to let them eat their lunch without some sort of pushback at some point. But if they can get big enough, generate enough enthusiasm, and demonstrate enough product excellence and traction, then they can probably build their own search index if it really comes to that.

All right, so next after this is the creation of agents. It's funny: almost all of this I explored either by myself or with others on the OpenAI or GPT-4 red-team project. That's another good episode to check out for some of my personal history with this kind of stuff.

It's pretty easy to think, “This ends up getting pretty smart. They seem to understand what we want remarkably well these days. I wonder if they can actually go out and do complex tasks for us.” They're starting to use tools too, right? So I wonder if they can actually go out and do complex tasks for us.

This is basically the architecture that people are using to explore that general space of possibilities. You put a task into the language model, and it thinks about it. “Let's think step by step. Let's break it down.” There's a paradigm called ReAct, which is pretty common. I forget exactly what ReAct stands for, but it's basically the idea of reasoning first and then acting.

The model does some reasoning, then decides what tools it has available. You have to set this up in some sort of computing environment. Replit is a really good place to go tinker with stuff like this, and there are plenty of other computing environments where you can do it as well.

The model asks, “What tools do I have available? Let me try to use those tools.” If the tools are API calls or code execution, that gets some result back. You send out your API call or say, “Here's the code that I want to execute,” whatever that may be. It happens, and then you get some result back. You may get a successful return from the API, an error from the API, the results of your script, or an error from your script.

A really simple version of this would be—and it's not something you really need to do—if you wanted to add 2 really big numbers with a language model. A language model itself struggles with that. It can't really do it on its own. But if you set up a situation where it could access code, it can write simple code in pretty much any language to add 2 numbers and get the result back.

If you gave it access to a coding environment and said, “You are GPT-3. You are not good at math, but you can code, so we've given you a coding environment,” it can write the code, execute it, and get an answer back. You can get accurate math out of this language-model agent system, even though the language model itself wouldn't be capable of doing the math.

Similarly, you can imagine browsing the web. There's a little project called NatBot, from Nat Friedman, the former GitHub CEO. He created a little extension—a tool to examine what's going on in a web browser and try to take the next action. It could be: click the search bar, enter some text into the search bar, hit Enter, look at the page again, see what's happening, and reflect.

It puts this whole thing in a loop: “Are we getting closer to our goal?” There are a lot of variations on this, but this is the core structure for what these AI agents look like today.

We've done 3 episodes on this: Flo Crivello from Lindy, Matt Welsh from Fixie, and most recently Div from MultiOn. They've all taken quite different approaches in some sense, but this kind of core setup is pretty common across all these different agent projects and companies.

This is a presentation you did for Athena, which you work with, and we're good friends with, obviously. I'm curious how you think about the right mental model or framework for thinking about AI-native companies like Lindy, or incumbents that will add AI, like Athena. We don't have to limit it to the assistant space, but just in general, it seems that the bias is that it'll be easier for incumbents to add AI than it will be for AI-native companies to get distribution. Is that a general truism, or how do you think about that?

That's a big question. I would split it into 2 parts. One is if you are a tools company. Let's say you are Adobe or Salesforce, and you have a big platform that you've spent years building out.

Typically, you've got the pros and cons of such big platforms. The pros are that you can do anything with them; they're super powerful. The cons are that I can't figure out how to use them. Those companies are very naturally aided by the rise of the LLM layer.

They can all use this stuff. This diagram actually explains well why the incumbents are going to do so well. If you're Adobe, you have really strong tools for creative purposes. If you're Salesforce, you have really strong tools for CRM purposes.

You can ask, “Can I train a language model to use these tools?” It's potentially going to be somewhat complicated, especially because your tools are complicated. But you have the tools, so there's a decent chance you can train the language model to use some of them and eventually, over time, probably all of them. Having the tools is key. That's your platform, and now you can build this AI layer on top.

You can imagine going to Adobe and saying, “Can you re-edit this video for me?” It would churn for a while and then start issuing commands to the editor. Some commands may error and others may proceed. You could watch it and iterate in natural language against Adobe or Salesforce.

I think those companies definitely benefit from the AI layer. If you're trying to come in from scratch—if you don't have the CRM platform or the creative tools and you're just starting with the language-model portion—you can make some cool demos, but it's really hard to make something competitive.

A great example of that is from Athena. I do courses there that I call Savvy Shopping, where we basically say, “There are a bunch of AI tools out there for all these things these days. Let's go see if we can find any that are actually good.”

We tried this for a few things recently. One was slide generation. You want to generate some slides, right? That's definitely time-consuming. But if you're coming into this from scratch, and you try a number of different tools where people say, “Language models are going to make it so much easier to create slides. We're going to start a startup about this,” what we saw in almost all cases was that the lack of mature tools ultimately made it less impressive.

The language model could sort of chunk your idea into slides, fill in the gaps, create a decent outline, and generate something. But the actual slides product itself was so much worse, so much more immature, and so much less developed than Google Slides or PowerPoint that, once it generated the slides, you looked at them and thought, “Okay, but now can I edit them?”

The answer was, “Not the way you're used to,” and in many cases, not in the ways you wanted. So you get frustrated. The best one we saw, by the way, was Gamma.

One of the features that convinced me it was the one to bet on was that it had an export-to-PowerPoint function. That made me think, “At least they're sophisticated in their thinking about this tools problem.” They understand that they can't compete with PowerPoint on every little control.

Even just knowing where the controls are becomes a real skill in something like PowerPoint. They're not trying to compete at the tools level. Instead, they're creating something you can export. If you want to do your advanced editing with all the tools you're familiar with, you can do that.

They have a strategy that makes sense. Most of the ones we saw just had a poor man's slides functionality. They generated stuff, but then you were stuck with slides that weren't really what you wanted, and you couldn't edit them.

I think a lot of projects are going to get stuck in that place. They've worked out how the LLM can work, but for lack of a robust platform to actually put that output on, they're not really going to be able to compete.

Before they're going to build those tools, Microsoft and Google are going to layer their own language-model layer onto PowerPoint and Slides. Again, it seems like, generally, the incumbents are going to do well.

Contrast that against the Athena side. Athena is not a software company. We do build some software, but we're really a services company. The way we're thinking about it right now is that we don't know how much better AI is going to get in general, and we don't know exactly how fast the agents are going to become useful.

My guess is still basically later this year. I think we're going to start to see them come online in a meaningful way and really start to be able to do some things for people. But it's very much still a work in progress.

Our goal is to figure out how to create the best human-plus-AI bundle. There are a lot of aspects to that. One of the big aspects is training the EAs in all these different ways of understanding and using AI, including shopping intelligently for AI tools.

We're also building some of our own, super-lightweight AI tools and diffusing those through the 1,000-plus assistants and all the clients. That's very much a work in progress. We view ourselves as surfing the technology wave.

The CEO's stated goal is, “Look, this is a transformative technology wave. When these come along, services companies like us just have to be the best at using that technology.” That's pretty clarifying and, in some ways, helpfully simplifies the analysis. We just want to be the best at using it. Let's figure out what that means and go do it.

The threat to Athena could be if things become so good that they actually outperform the human assistant, and people don't need the human assistant nearly as much. We are not there yet. That is definitely true, and it's unclear if and when we'll get there.

Our strategy is that, if AI takes over the world, we're in the same boat as everybody else. If AI continues to develop but isn't an effective replacement for a human assistant in the next couple of years, then we want to be the human assistants that are the best users of AI. That way, we can ultimately have the best bundle to offer customers.

Any suggestions for our strategy? You guys have a pretty strong team over there.

No, I think you're on the right track.

Yeah, it is cool. We've been doing some things lately where, because the language models are getting so good at code, we're starting to teach the EAs to use AI to code without even introducing the basics of coding.

People are coming into these sessions without having coded at all. They don't know what a for loop is, they don't know about variables or functions, and they've never seen the coding environment. We're skipping all that. Instead of teaching them to build from the ground up, we're teaching them to use AI to modify existing things.

They can go to a Replit gallery of templates, either a public one or one that we're building and curating internally, and find something along the lines of what they want to do. Then they can use a language model to modify it. In many cases, they can do that these days without really being able to code at all.

There is definitely a brave new world of possibility there. It's an interesting wave to try to ride, no doubt about that.

The next couple of examples bring a few of these things together. This one really blew my mind because the headline was “Language models could run experiments.” I think that was a little overstated. They can run experiments; that is not overstated. The headline was that they could do science, and I think that is a little overstated. I'll explain that in a bit more detail.

If you remember from part 1, I do think the question of whether AI can do science is a supercritical one. This caught my eye and took me down the rabbit hole of this paper.

What these researchers set up is a multi-agent system. Sometimes these are also called “the agent,” but really, these are different roles being played by language models. You could think of them each as their own agent, but they're all working together as a single agent. I often call them multi-agent systems. You'll hear both terms: an agent made up of multiple parts, or a multi-agent system. I think “multi-agent system” is more clarifying.

They have a system where you can give it a request. The request could be something like, “Synthesize aspirin”—a real example from the paper. You give that request to the AI system, and the first module that processes it is the planning module.

The planning module can use different tools, including searching the web. If you say, “Synthesize aspirin,” the first thing it's likely to do is search the web for how to synthesize it. It also has the ability to execute code, so it can calculate things, such as how much of a certain reagent is needed to synthesize a certain amount.

It's not great with math on its own, but it can write simple math: “This many grams of that, and it's 2-to-1, so I should have this many grams.” It can handle that by writing and executing code.

It has access to search, a coding environment, and documentation for hardware. Now you can get into questions like, “What machine do I have available, and what can those machines do?” It can actually execute commands against a physical, remote laboratory.

This was done with Emerald Cloud Lab, which is a remote lab. The company maintains laboratory infrastructure and serves people all over the world who sit at their computers and program physical chemical and biological experiments. They program them, send the commands to Emerald Cloud Lab through an API, and Emerald Cloud Lab executes the actual steps with a fully robotic lab.

The upshot is that you can now say, “Synthesize aspirin,” and have an AI system take it from there and actually produce physical aspirin in a physical lab somewhere at a remote location. That's pretty crazy. It is a big gap to go from a natural-language command to an actual synthesized chemical product.

People got excited about this paper, and I think it's an awesome demonstration of all these trends coming together in a single system. But when I looked into it, I thought, “Wait a second. That's not really science. Synthesizing aspirin is not science. It's a task, and it's well known how to do it.”

If you go online and search, people are not debating how aspirin should be synthesized. It's a well-known protocol. What is clear is that you can have an AI go from a text prompt to the execution of a scientific protocol. But I would distinguish that from doing science.

Doing science means identifying which experiments are worth running in the first place. They tried some of that in the same paper. They basically said, “Figure out a drug to cure cancer and synthesize it,” or something like that. The system tried, but it became more bogged down in the planning and web searching.

It didn't really have great ideas for which cancer drugs it might want to experiment with. In fact, it basically came up with the same things everybody else was talking about online. There was nothing really creative—nothing that, if you took it to an actual cancer researcher and said, “I just got this from GPT-4; what do you think of this idea?” would cause them to say, “That's interesting.”

They would say, “That's basic. We all know about that. Everybody's been exploring that for the last couple of years.” That's where language models are. They can understand a lot and have learned a lot, but they don't yet have that breakthrough insight.

Going back to part 1 and the tale of the cognitive ape, breakthrough insight remains one of the biggest advantages of humans. I thought this paper demonstrated both that all these pieces can come together nicely and that the limit is still doing things that are already well understood—your standard of care or an established protocol.

That breakthrough insight into what would be a good cancer drug to investigate is not something the system can really produce yet.

Here's another one that's pretty cool and made a lot of waves recently. This one is out of NVIDIA. They took a language model, put it into a Minecraft environment, and allowed it to explore and do its thing: generate code, take actions, and progress toward goals in the game.

The big innovation was equipping it with a database where it could save its successes, so that in the future it didn't have to stumble through recreating them. You can imagine that it might fail a lot while trying to generate the right code to do some random thing.

I don't play Minecraft, but “combat zombie” is the function they show here as one example of a skill it learned to perform. Within that, you have other things, such as crafting a stone sword or equipping yourself with your sword. It has all these little skills, and it's composing them into bigger skills. It can craft a shield and perform all these different skills that it has learned, composing them into higher-level skills.

It can only do that because it's saving its successes. When a new situation comes up, it can look at its past successes and ask, “Do I have anything in here that could help? Do I have a couple of things I could recombine into something that could help?” That gives it a dramatic shortcut the next time it encounters a new challenge.

Giving it the ability to save its successes creates what they call a lifelong learner. This thing can actually do a ton of stuff in Minecraft. People were impressed because you aren't fine-tuning it, changing it, or giving it information. You're allowing it to discover things, save them, and create its own database that it can go back to. That's enough to unlock unprecedented performance in Minecraft.

This is another good one to look at. Div from MultiOn talked about their use of this. The leap from research to implementation is often as short as a week. I don't know how long this paper took—many of these research groups are becoming very prolific, and you're seeing people publish quite a few papers a year—but they're moving quickly.

Something like this comes out, and all the agent companies, certainly including Div from MultiOn, look at it and say, “That makes a lot of sense.” It's not just for Minecraft. In the context of MultiOn, if I'm browsing the web and want to book a flight on Delta.com, I'm probably going to stub my toe a few times.

But if I can save my successes, I can become much more reliable. The average user doesn't want to wait while MultiOn tries and fails and gets confused the first time it works its way through booking a flight on Delta.com.

If it can go to a database and ask, “I'm booking a flight on Delta.com. Have we ever done this before? What did that look like?” and the answer is yes, then it can use that previous process. That gives you much better performance—faster and more reliable.

You can also inspect that information. You could say, “We're going to have this skill database, and we're going to vet it so we know that our process for booking a Delta flight is appropriate.”

You could even bring GPT-4 to bear on it and ask, “Is there anything suspicious or unusual in this booking process on Delta.com?” We're going to see all these strange attacks, too. Different websites will start trying to mislead or trip up the language models that visit them, so there will be a kind of cat-and-mouse game.

Having defined, confirmed-successful, and ideally confirmed-safe skills—basically protocols or things that are known to work—is going to be a big part of getting these agents to work over the remainder of this year.

Another notion I think is super interesting is multi-level memory. Going back to the tale of the cognitive tape at the beginning, human memory isn't great. It's fallible, and there are all kinds of funny things we don't remember. But we do have a general, ongoing sense of who we are, what we've done, where we've come from, and where we're going.

That extends well beyond the finite context window of an AI. We've talked about strategies for augmenting the AI's natural runtime memory with access to web search, an embeddings database, or the ability to save its own successes to its own database. But all of those still operate at a single layer.

This is the layer they're adding in this particular study. They created a process for storing raw observation data points: “This happened. This happened. This worked. You tried this API call down here.” You have all those object-level observation memories, and then they process those raw memories into higher-level reflection memories.

That's where the AI starts to have notions that could only really be synthesized at that level. There's an example involving a simulated little town. They simulated a whole town for days and had all these little agents moving around and interacting with each other. It was quite an elaborate setup.

You have these individual characters, each with individual observations of what happened. From those observations, they form a self-conception that summarizes and generalizes the raw, object-level memories.

This is obviously inspired by human behavior. We have something along those lines as well. I don't want to make a strong analogy there, of course—this is a no-analogy zone—but inspiration is fair game.

People have a sense of who they are, supported by the things they've done, but it exists as a layer above those things. Now you're starting to see AI systems have this too. I think this will be a key part of how these agents become much more coherent.

Going back to the Delta example, you're going to book a flight. What are the client's preferences? You could look in a database and see that the client said “D8” last time, so maybe D8 is their preference. It's probably not quite that simple.

But if you had a bunch of those observations, you could synthesize them into a higher-level sense. That higher-level sense could guide further action and smooth over the outliers. If you've sat in the window seat 10 times and the aisle seat once, maybe the window seat is your preference and the aisle seat just wasn't available that time.

Figuring out that kind of thing—filtering out the random oddities in our experiences and observations and identifying what is actually signal—is something people definitely do in some way. People are starting to devise systems that allow AI to do that for itself as well.

Another huge trend is what I call “the great embedding,” which is a little silly but represents a powerful dynamic—and one that I see real risks with, as well as a ton of power.

What people are doing here is often motivated by a desire for multimodality. We've got text AIs that can handle text inputs and outputs really well, and separately we've got vision AIs that can encode and understand images. How do we get those systems to work together? It doesn't happen by default.

People are figuring out how to bridge from one representation to another. A representation is the internal state of an AI that encodes certain information. You could also call this bridging from one embedding space to another, or, ultimately, from one modality to another.

The first example came from DeepMind. This is Flamingo. They took a frozen language model and a frozen vision encoder, then added a single component that translates the encoded images into the numerical space of the language model.

They trained the whole loop, but they only updated this one component through gradient descent. When they say everything else is frozen, that means it isn't subject to gradient-descent weight changes. Only this additional component is optimized.

That's enough for the model to understand text and images. It understands text because it already understood text at the beginning, and it understands images because they've created a bridge that takes the numerical representation of an image from the image AI and converts it into the text space in a way that remains meaningful and allows the system to understand images and text at the same time.

I think that's super profound. We're starting to see these text-to-everything-else spaces open up as people figure out how to do this in many different contexts.

The episode for this one, by the way, is the BLIP-2 episode with Junnan Li and Dongxu Li. They're out of Salesforce Research and created a whole series of BLIP models. The BLIP-2 model used essentially this exact framework.

Again, you see the image encoder, and then the component they trained converts the numerical image representation into a numerical text representation. It allows the model to process the image as if it had received text, even though it never received text. It received numbers in the text space that were adapted from numbers in the image space.

This is really impressive, but it does compound the black-box problem. I love it, and I also feel a little weird about it. We already have enough trouble understanding what any of these models are doing. Now you're having models talk to other models in a purely numerical, high-dimensional space that humans cannot understand.

We don't have a great sense of what's happening at the image-decoder layer, and we don't have a great sense of what's happening at the language-model layer. Now we've got a bridge between the 2, and we don't have a great sense of what is happening there either.

If you're worried about AI systems being fitted together and glued together in ways that make it very hard for people to know what's going on, this is one way that will happen. People will patch one system to another in all sorts of ways.

This doesn't take a lot of training or a big model. There are multiple reasons you might want to do it, but the biggest reason is that researchers were able to run this BLIP-2 training on 1 computer in just a few days.

If you're training a language model from scratch, as we talked about earlier, you might be spending $1 million to $100 million, depending on how big you're trying to go. But once you already have the language model—and again, the language model here is frozen, as is the image encoder—you only have to create this small translation layer in between.

That layer is much smaller and much easier to train from a compute-budget standpoint. The pros and cons are clear: it's easier to do and creates multimodal capabilities, but it compounds the black-box problem.

Like it or not, it is starting to work, and it's being generalized. These are examples of image understanding with Flamingo, InstructBLIP, which is the successor to BLIP-2, GPT-4, and another model called LLaVA, which I think is from Microsoft.

They ask, “What is unusual about this image?” You look at the image, and it's an insane image of a guy who appears to be hanging off the back of a taxi, ironing something on an ironing board. That's probably not an image that comes up very often.

Yet these multimodal models, using this kind of technique, are able to understand the image and give you sophisticated responses about what is happening and what makes it unusual. It's getting quite strong at combining image and text.

Meta, of course, always seems to be hitting the accelerator on things these days. It's taking this even further by combining text, image, video, audio, the heat map, depth, and IMU. I can never remember what IMU stands for, but these can correspond to different sensors.

You can have a microphone that provides input, a depth model with a particular output, and so on. You can feed all of these things into a single, likely large, language model that has been adapted or retrofitted to understand all these different inputs.

You're starting to close in on the 5 senses. Once you have sight and sound covered, along with text understanding, you can even have senses that we don't have. We do have depth perception, but we don't necessarily have a heat map.

There are things we can sense with mechanical sensors that we ourselves can't sense. There's no reason a language model won't be able to make use of those kinds of inputs, and we're already starting to see that happen.

This is now being extended to multimodal outputs as well. This is from just the last few days and is another publication from Lili Yu at Meta AI, who was just on the show 2 weeks ago for her MegaByte paper. This is another major paper out of Meta.

Lili is on a huge run here because this is the first model I've seen that can do high-quality text and image understanding as well as text and image generation.

It can edit images based on instructions. For example, it can turn a figure in a painting into an alien. It can take a line drawing and turn it into a photograph. It can take a pose estimation—another modality of AI in which the system has been trained to identify or generate a skeletal pose—and turn that into an image.

It can do all of these things while also engaging in general dialogue about images and text. This is all happening in the same model. It can understand and output both text and images.

Finally, in terms of multimodality, the most shocking multimodal result so far is one we really haven't seen in public. That is Gato from DeepMind. A little over a year ago, Gato was a single model that was multimodal, multitask, and multi-embodiment.

The researchers said that the same network, with the same weights, could play Atari, caption images, chat, stack blocks with a real robot arm, and much more. It was dynamically deciding what it should output based purely on the context it received.

That was over a year ago, but they never released it.

A couple of other final trends: efficiency is obviously a huge deal. We talk about how much it takes to train these systems and then how much it takes to run them. The latter is many orders of magnitude less.

The conventional wisdom these days, especially if you're in the large-language-model game, is that you're going to spend more compute on inference than on training. It may be $100 million to train GPT-4, but OpenAI is probably going to spend even more than $100 million running GPT-4.

If the simplest call costs a tenth of a cent and the biggest call costs $1, then they have to do 10 billion API calls just to make their money back. They're going to do a lot more than that. The bulk of the compute and expense is ultimately going to be running the systems, not training them.

People are thinking, “How can we make that process more efficient?” There are some amazingly simple ways to do it, as well as more complicated ways that are proving effective.

One of the simplest, most brute-force ways to improve efficiency, which does come at some cost but apparently a surprisingly small one, is quantization. That's just truncating the numbers—the weight parameters.

Weights are the numbers that sit in the AI model and wait to be used to transform inputs. In the context of the circuit, the inputs are converted into outputs. You could have floating-point numbers with lots of decimal places, and then you could truncate them to a shorter version with fewer digits.

It turns out that most of the time you can get away with that. The model performs roughly as well—usually slightly less well, but not much less well. It's like, “Maybe we didn't need all those digits in the first place.”

As with everything else, there are elaborations. People are starting to think, “If we can do this after training the model, couldn't we do it while training the model in some form?” That's under investigation and development as well.

This works with a model that was trained with all those significant digits for each weight: just cut them off, save a bunch of memory, and make the model run much faster. Going from 32-bit to 8-bit quantization allows you to save 75% of the memory.

That's a huge amount of savings. For practical purposes, it allows you to run big models on small devices where you otherwise couldn't. Some people use this for demos, saying, “Look, I made this thing run on my laptop.” That almost always involves quantization to get the model to fit into the memory available on the laptop.

Another technique, called distillation, is when you take a large model and use it to train a small one. This is also called the teacher-student setup.

You put a bunch of queries or inputs into the large model, get the outputs, and do the same thing with the much smaller student model. Then you compare their outputs and adjust the small model so that it learns to mimic what the teacher model can do.

In some ways, this is what people are doing when they take 1,000 outputs from OpenAI and fine-tune an open-source model on that basis. They're using GPT-4 as the teacher and their open-source model as the student.

If you control both models, you can do this in a more sophisticated way. Remember that a language model doesn't just predict a single token; it predicts a probability for every possible token.

That's a pretty rich signal. In the case of GPT models, you've got more than 50,000 individual predictions for every possible token at every step of the generation. That's a signal you can optimize toward pretty effectively.

GPT-4 does not give you that. OpenAI doesn't provide all of the outputs; it only gives you the single token that was selected. You can still train on that, but if you have both models, you can do this teacher-student process in a different way.

Why would you do this? Ultimately, to save money. You want a smaller model at the end that you can run more cheaply and quickly. But you might also find that it doesn't cost you much if, for example, you had a general-purpose model and wanted to create a narrow-purpose model.

This can be a great way to do that. The grokking dynamics are also complicated. You might have tried to train a certain behavior into a small model and failed, while that behavior is present in a large model. Can you get the large model to teach it to the small model?

I don't think this is well understood by anyone, but it is interesting to consider. Maybe the large model can grok certain things, and then the student model can grok them too. Or maybe it just learns to imitate.

Which models have grokked what is usually very unclear. But there are certain things where you can see that large models can do them, so training small models from large models may be the only path to getting small models that can do certain things well.

It may not be the only path—you could come up with something else in time—but this is a technique we're finding practically useful for creating small models that can perform certain tasks well.

One more efficiency measure is the mixture of experts. Going back to the concept of sparsity, we talked about it a little last time, especially in the context of encouraging grokking. That was the Max Tegmark and Jimmy Ba paper, where they used a modified loss function to encourage sparsity.

This is another way to create sparsity. Why do you want sparsity? It can mean multiple things. It can mean having certain weights set to 0. If a weight is 0, multiplying by 0 is always 0, so you don't have to perform some of the computation.

Another way to think about sparsity is to take a certain portion of the network, segment it, and use only certain segments at certain times. In this example, you've got the classic Transformer setup, with attention and normalization functions.

When it comes to the feed-forward part of the network, they break it up into pieces and use a router to decide which ones to use at that moment and how much weight to give them.

According to recent rumors, GPT-4 is supposedly a mixture-of-experts model. If you count every parameter across all the different experts, then reportedly it has more than 1 trillion parameters. But the number that actually runs at a single time is much smaller, because the router identifies which subpart of the network is best suited to handle the particular task.

It might use 1 expert or multiple experts. There are different setups that handle this in different ways, but the point is that you only have to run a subset of those parameters. That makes the model much more efficient and faster at runtime.

This is still expensive to train. If this were GPT-4—which it isn't, but if you imagine this as a stand-in for GPT-4—you wouldn't save as much on training because you're still training 1 trillion parameters.

At runtime, though, you only have to use a subset of them for each query, which saves a lot on inference. Since inference is going to account for the bulk of the compute, these efficiency measures really matter.

All right, those are all of my implementation trends and lots of discussion too. Omniki uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omniki so much that I invested in it, and I recommend you use it too. Use COGREV to get a 10% discount.