语言模型的信息论：Jack Morris

Today, we're here with our special guest, Jack Morris. I guess you're from Cornell. That's your affiliation right now?

Cornell. It's actually confusing because I'm at the New York City outpost of Cornell. So you have the city, right? But it's Cornell Tech, which is like a small Cornell campus in New York.

You're a student of Professor Rush, who teaches at Cornell, so I should have made that connection. Okay. I'm sorry. That's a horrible mistake to make right off the bat, but there aren't that many PhD students who make an impact with their research. The last time something like this happened was Shinju from Princeton, and he joined the OpenAI Operator team quite shortly after he graduated.

So you're one of those high-profile PhD students coming out of the program, and I figured it was a good time to talk about your work and also the fact that you're looking for which lab you're going to join. That's a whole interesting meta-discussion, especially with the insane market for AI talent these days. What's it like to be an AI grad student these days?

Thanks for having me. Maybe we can go back to when things first started, or put yourself in my shoes. In 2017–2018, I really learned a lot about machine learning. I went to a state university. It's a good school, but they didn't have a deep-learning research department or anything. They had people doing it, but it just wasn't as big at that time.

I was getting really interested in those topics, especially as they applied to language. Then, in 2019, I was starting to do research and thinking about my career. At that point, I was 20 or 21, and I was thinking about where I wanted to be career-wise and who was doing the coolest stuff right then, looking at what kind of work was coming out at the time.

I thought AlphaGo was really good. At that time, I was playing a lot with BERT and BERT-based models. Google and DeepMind were doing great work. GPT-2 and GPT-1 from OpenAI were interesting, but I think most people were into BERT at that time. I still have a soft spot for that 100-million- to 1-billion-parameter scale of models.

This is all to say that, at that time, I felt like the people doing a lot of the most impactful work were professors and PhD students. There were a ton of interesting ideas being explored and cool opportunities in academia, so I ended up applying to grad school.

First, I did the Google AI Residency Program, mostly during the pandemic, in 2020 and then 2021. I was also applying to grad school, and I started grad school in 2021. That was still what was going on at that time—around when GPT-3, with 175 billion parameters, had been released, but not InstructGPT. We had pretraining, and the science of pretraining was emerging, but that's where the models were.

I'm glad that I went to grad school and had a great experience, but the last 5 years have changed a lot. The whole meta has shifted, and the power dynamics are completely different. The ideas are coming from different places. Most things were open; now, most things are not open. The types of questions people are asking are different.

For better or for worse, I did go through the full grad-school experience, and here I am. It's been a really interesting perspective watching the science emerge alongside the products. The biggest thing that happened by far was ChatGPT coming out, right in the middle—before Christmas in 2022, in November.

I remember that year, my grandma was asking me about it. That's when it hit me: This is actually becoming a real area that people will know about and understand. I was trying to explain it to my parents, and that's when I think things really started to change in terms of the types of questions you wanted to ask that couldn't always be answered with academic resources.

A lot of the fundamental boundary-pushing in AI science moved into companies. That was the year when, just around NeurIPS as well, everyone in NLP and deep learning was very confused. Some people were kind of expecting this already, in the sense that they were obviously more clued in to large language models. But I think the sheer amount of consumer-level interest in 2022 completely changed the world. Now we're just in a different sphere.

Did you have to pivot your research, or were you already just going from BERT to other stuff? You've done a lot of embeddings work. You're always heads down, working on a problem, so I don't think most people in academia are the type to say, "Oh, look at this new product that came out. I'm going to abandon everything I'm doing." That can be the right move, you know?

It definitely can. Honestly, if I were to give advice to a younger grad student, I think the way to do it would be to sit and wait until the next paradigm shift, then immediately start working as fast as you can to reimplement it. I don't think that's necessarily the best way to do science, but it's probably the best way to play the academic game in the days of AI.

You've seen that so many times, most recently with the reasoning models. o1 came out of OpenAI in September 2024, and then there's been this explosion of abstraction layers on top of that. First it was reimplementation: How do we even do this? Now it's a lot about the data. What's the right data? What are the right evaluations? What are the right training schemes? There are so many different axes you can test and publish research in.

I think the easiest way to do that is probably just to work in a field that's existed for less than 1 year, so no one has any big advantage.

That is mostly correct. I think anyone who jumped on reasoning and reinforcement learning for large language models is doing super well. I just saw this morning that one of the recent Stanford grad students who worked on reinforcement learning started a company, and they're worth $500 million.

It's absolutely bonkers right now. There's no product—just 3 dudes sitting in some basement somewhere. I mean, undoubtedly cracked, but also not worth $500 million.

Yeah, but maybe you're not paying for the product, right? It's the ideas behind it, or the—

Yeah, yeah.

There was this big shift in scale from working with 100-million-parameter models. Really, what happened is that I think the companies invested a ton more in training and infrastructure, and we all had to catch up.

I go to Cornell and work with the professor there. He has to buy GPUs. Should he buy last year's GPUs or this year's GPUs? How many should he get? We were trying to figure that out, and there was a big lag where, basically, at the 7- and 8-billion-parameter scale, there's a huge difference between the BERT-size models, which are 125 million to 200 million parameters, and the 8-billion-parameter models.

Obviously, it's 2 orders of magnitude, but there's also this idea of emergence. If you're talking to a model that's 100 million parameters, no matter how well it's trained, it knows nothing. If you ask it, "What's the capital of a state?" or "Who was president of the United States in 1990 or whatever?" it will always say George Washington, because it just associates the words "president" and "United States" with George Washington.

When you get to the 8-billion-parameter scale, suddenly it knows every single president. It knows every single capital of every single country. I really do think that changes the type of research you can do.

It took us a while in academia to catch up—getting good 7-billion-parameter models, then running them and getting GPUs to run them. Now I think things have stabilized a lot. We have access to compute, and we can fine-tune and run inference on models at that scale, and that's fine. But there were 2 years when everyone in academia was working on smaller models, and none of it really mattered.

I can branch that discussion in 2 ways. We should get to your research at some point, but I'm enjoying this because I don't think we get to talk about this on the podcast too often.

One is that there's often a bit of advice from people in industry to grad students: Give up. Don't work on models. Just do benchmarks, right? A really good benchmark will get our attention, and then we'll hire you. You can switch to models later.

You have, for better or worse, avoided that, which is cool, and we can talk about that as well. But the other thing, I think, is that around 7B or 8B—maybe 4B—is when you start switching from a single-GPU setup to a distributed setup. Do grad students get high-performance computing training? How much do they teach you about working with large clusters of stuff?

To be clear, they don't teach you anything. If you see a paper coming out of even Stanford, probably the best school in AI if you had to choose, it's not like they're learning how to do multi-node, distributed FSDP training with whatever DeepSpeed. You have to learn that from the internet and from other people. There are no classes that really do that. I mean, it's hard to facilitate that as one person.

I would say most grad students are doing stuff on a single GPU. Some people are doing multi-GPU training. There are probably basically no grad students doing multi-node training. There are probably a few, especially if they have company affiliations, but that's really unusual, I think.

For grad students who are looking to get up to speed on that, I would recommend the GPU MODE Discord, where basically the PyTorch team is hanging out, just waiting to help you. The other one would be the fast.ai team. If you have some kind of thing, Jeremy Howard would basically help you out, and they have some distributed training. Honestly, try to reach out to the DeepSpeed team at Microsoft. They're actually reasonably accessible. Nobody talks to them. It's so funny. I met them at NeurIPS, and they had nobody at their presentation of DeepSpeed 3. I was the only one asking questions.

Yeah, that's good advice. Listen to this guy.

People are there if you want to ask. This is very valuable experience. Once you're a GPU god, you're basically in a different tier as a researcher because you don't rely on someone else helping you out. You can just be your own research engineer.

Yeah, I'll comment on that quickly, because if someone has been listening to this and also following me online for a while, I think I've made a couple of comments saying something like, “You shouldn't learn about CUDA,” or things to that nature. I'll give some more color to that.

So it's definitely a great idea to learn CUDA if you can. I think my point was that if you're trying to enter this space, learn about the models, learn about how they're trained, what the data looks like, and what the compute looks like. One axis of that is how to do more efficient training and inference. One part of doing more efficient training and inference is studying the hardware, which is GPUs. I think that's a very small subset of all possible knowledge that you could acquire, and it's probably not the best place for a lot of people to start.

That said, if you do it, you've got to be one of the most hireable people in the world. If you really deeply understand the architecture of the new GPUs coming out and how to control it, you're in a very small handful of people, and everyone will want to hire you.

Actually, the sweet spot isn't even CUDA right now. I would say it is Mojo. I don't know if you've been paying attention to Mojo from Modular.

Oh, I listened to your podcast, man. You had that guy on the other day. The whole story is Chris Lattner—an industry legend: LLVM, Swift, all these things. Now he's turned his attention to the Python-CUDA relationship, right? He wants to basically create a viable CUDA replacement. It's basically Python married with Rust.

For the last 2.5 years, it was basically kind of stealth, not ready for production. When he came on our podcast, he was basically announcing to the world, “We're open for business. You can use us now for most models, and we're actually faster than the native implementation, sometimes the PTX implementation.” I don't know how that works precisely, but he's a compiler-languages god.

I think there's one of those windows now. Like you said, bet early on something that's a shift. It's one of those windows where you try to implement things. The Mojo team is 100 people. If you run into issues, you'll get Chris's personal help on things. I'm not promising it, but probably, because he wants to work on improving the toolkit.

It's not really about becoming a CUDA god, because once you wrap your head around the general concepts and principles, you can probably translate between ecosystems pretty effectively. A lot of people switch from JAX to CUDA. The thing is being able to experiment very quickly on a limited budget. Efficiency isn't just about trying to be an efficiency guru and making that your career, which is kind of boring. It's also about being able to experiment very quickly and find these ideas.

I also think vLLM and SGLang seem really good and important, and here to stay. They'll probably just get larger and more complex to accommodate future systems, but if I were a grad student starting out and working in that area, I'd probably want to learn more about how they work.

Awesome. Let's go to your research. I like to mention that I first came across you because of CDE, the Contextual Document Embeddings paper. You can tell me the story about that, but I just want to show you proof that I get 1 slot per day to highlight the number-one AI story: you were the slot of the day for October 5.

No way.

I mean, obviously you were producing work before that, but I thought CDE was a really cool exploration of how embedding models are kind of stuck in a rut—here's actually how to make them very efficient by just doing it in 2 stages. That seems like a relatively simple insight that was done very well. But you have a general information-theory thing that we should maybe start with, and then we can create it in a similar way.

Yeah, sure. That sounds good. We can circle back on that.

That's really cool that you wrote about it. What was that, almost coming up on 2 years ago?

Yeah, this is the post I wrote. I called it “A New Type of Information Theory.”

We don't need to go into the details of the paper, but there's a concept called V-information. Maybe I'll give the simplest explanation. If you say you have 2 text files, 1 text file contains a paragraph of information about New York City, and the other text file contains the same text but encrypted with SHA or whatever encryption algorithm, so it looks like random letters, if you decrypt it, it has the same text as the first text file.

From the perspective of Shannon's information theory, these 2 files contain the same information content. Relative to everything, they have the same number of bits, but it's very clear to the observer that the first text file, which is plain English text, is much easier to read and easier to process, even though they have the same information.

There's this theoretical framework proposed in a 2020 paper, “A Theory of Usable Information under Computational Constraints.” It really doesn't have that much press. There aren't as many citations as you would think, but I think it's a really neat idea. We should measure information with computational power as a constraint. They have this idea called V-information, which is how much information is extractable from a given file or code. In that case, we could say the first text file actually has more extractable information than the second text file.

I think that's really good. That captures a lot of our ideas of how these deep learning systems work. Why does pre-training work? If you have 2 sets of weights and you want to train on some downstream data set, 1 set of weights is pre-trained and 1 set of weights is randomly initialized. Why is the pre-trained model better at all, even though it's never seen your data? Maybe one way of looking at that is that it makes the information more extractable somehow. There's this concept of computational processing that you can almost store up.

I like this as a lens to view problems with: how much information is stored where. If you get a set of model weights or an activation vector and you print some tensor or NumPy array, it looks like random numbers, right? There's nothing human-intelligible about that, but really, it's this complex combination of the training data and the training algorithm that gets compressed into model weights. Then the actual computation that the model is doing involves manipulating these numbers in ways that we don't understand. It's this highly compressed, nonlinear combination of all these information sources, mixed with computation. I just think we don't have the right words for discussing this.

I like the information-theory analogy because, back in the day, we had phones and telegraphs, and people were building the phone system with these crazy heuristics to send information across the country or send telegraphs across the Atlantic. People were just trying stuff. Then we found stuff that worked and ran with it, but that wasn't really optimal.

It wasn't until someone came along and proposed this concept of a bit—a 1 or 0 that tells you something. Once we have a bit, we can do all these things: count the amount of information in a signal, do really good error correction, measure properties of distributions of things, and build a really good system for phones, which eventually led to computers.

I'm bringing this all up because I don't think we have—I don't think we know what a bit is yet in terms of deep learning models. I'm going to graduate with my PhD this year, but I didn't figure it out. So if you're listening to this, maybe you can spend more time on it, or you're smarter than me, or you have a group of collaborators who can all get together and figure out what the right lens to look at this stuff is.

But even by just asking these questions, I think I was able to conduct this research agenda that I'm still working on, actually.

Yeah. What do you call this field?

I don't know.

I don't know. I called the post “A New Type of Information Theory.” I don't think it exists yet, I guess. Maybe it'll get a name once someone comes up with the right set of definitions.

I think information theory is a really good start. There are a couple of related threads. First of all, you don't know this, but I've actually been trying to accumulate data about Shannon—sort of a Shannon information theory view of language models. I have a lot of notes. This is actually on my GitHub for people who are watching along.

At the limit, if a language model has 175 billion parameters using 16-bit precision, it would take up 350 GB. You can compare that to Wikipedia. Wikipedia is about 150 GB. Let's say GPT-3 can store 2 Wikipedias. But is that a relevant measure of information storage?

It is not, because you can compress Wikipedia a lot. There are a lot of repeated patterns. Tokenization is the first form of compression, but I think there's a related talk from Ilya Sutskever about how deep learning is, in a sense, compression. You have a data set, and you compress it into a model that is smaller than the data but generalizes and has some amount of acceptable loss.

I think one of your commenters on the post made this direct comparison with Kolmogorov complexity, which is how Ilya sees it. I think people have this information theory idea or approach to language models; it's just not precise because, exactly as you say, we don't know what a bit really means.

We don't know what the most legible representation is. Legibility is a word that comes to mind in terms of how much it matters to us that something is human-readable. Even if it's SHA-1 or SHA-256, I don't care, but that is less readable and therefore less useful. Entropy isn't the word, because it's directly convertible. It's just less useful.

Yeah, yeah, yeah. Useful is a good word.

I think maybe useful information or usable information is the right lens. Kolmogorov complexity is a really interesting connection. I think that's a really good concept for computer scientists.

I'm not sure exactly about this specific talk or what he was trying to say, but I think that we have a very good understanding of language model pre-training, and there's a deep connection between language models and compression. Actually, maybe let's start with the embeddings. We can come back to that.

Okay. So, are we good to the first paper? Actually, let's go to your Wikipedia numbers if you still have access to that. This 50 GB for the text of Wikipedia sounds pretty high to me. Is that uncompressed, like text files or something?

I got it from Andrew Ng, so I don't know.

Okay. No, I'm probably off. I just have the sense that when you store text, it's generally very, very small, especially when you zip it.

Maybe he's including all the languages and all the edits.

Yeah, that could make sense. Because I guess, if you want to do apples-to-apples comparisons, you say GPT-3 can store 2 Wikipedias. Is that right?

2.3 Wikipedias.

Yeah. I thought it would be a lot more.

And this is actually an experiment that you could do. You could just train a model on Wikipedia and keep training it until you can perfectly extract all of Wikipedia. That would be a good way of knowing how many Wikipedias GPT-3 can store.

I like that idea.

But I think this type of back-of-the-envelope math is really useful for thinking about problems and grounding yourself in the real world, even if you can never quite answer the questions you want to answer, at least in 4 years.

If we think about embeddings—vectors that people use for search—we can do the exact same kind of math. If you use the OpenAI embeddings, which, last time I checked, I think have 1,536 dimensions, and you say there's 16 bits per dimension with half-precision floating point, it's something like 20 KB of information in a vector.

If you want to store 20 KB of text, that's a lot of text—many, many paragraphs that you can perfectly compress into 3 KB. I think this is kind of the idea we had.

I'll give you the practical explanation. First of all, I'm a second-year grad student. I'm going to these conferences, seeing all these other things people are working on, and thinking, “What the heck? How am I going to have my own little area to do work in that no one else is working in already?”

I spent a lot of time coming up with bad ideas, and my advisor would say, “No, that's not a good idea to work on.” Many times this happened. Even my first year and a half of grad school was a lot of exploration and coming up with bad ideas.

Honestly, I'd be interested to see how he remembers it, but I think I wrote a sequence of proposals about different projects. Then I came up with this idea. I was like, “Oh, we should just try to do as well as we can at reverse-engineering the text that's in embeddings.”

We were talking about it, and he was like, “Oh, yeah, you should just do that.” Then that was the end of the proposals. After that, I just worked on that problem for a long time.

At the time, I was really motivated by that because I thought, “Cool. As a grad student, this is my first sort of official sign-off on coming up with a good research idea.” At the same time, there was this big rise of a startup business model called a vector database. There were all these companies popping up, raising money, getting crazy funding, and then actual applications being built where, instead of exchanging customer data, they exchanged vectors.

We had this very grounded question: What data are they actually sending when they send the vectors? First of all, you have this information-theoretic argument that when you send 1 vector, there should be a lot of text recoverable, just in terms of a lot of these things representing very short documents, but actually having many, many bits. The problem seems tractable.

Second, we had this justification based on how the product is actually being used. If someone hacks into a vector database, what do they actually find? If that makes sense.

I think I have the talk that you did that Sasha highlighted. Is this the one?

Oh, yeah. Maybe that has the graphic.

Go 1 before, I think.

1 before? This one's good. I like having visual aids. I like giving people breadcrumbs to follow up on if they're interested in digging more.

I remember this was a pretty hot area of research at the time, and there have been some really interesting follow-ups. We ended up building a system that can do this quite well. Our highlight number, at a certain length—a long sentence length—is that we can get 90% of the text back exactly.

A lot of people were able to do stuff with that. For example, I know these people who work on the problem of debiasing embeddings. In 1 data set, they have a procedure for removing all latent features that correlate with gender, so they can produce useful embeddings that, from some perspective, have no information—or usable information—about gender.

They'd been doing that for a while, and then they actually used our tool. They would put in a sentence like, “This woman is a doctor at Wild Cornell Hospital in New York,” or, “This woman is a doctor. She works at Weill Cornell.” Then they would run their procedure, and afterward they would run our embedding-to-text model.

Now it would say, “This person is a doctor. They work at Wild Cornell,” which is pretty cool. They had text-based evidence that their method was actually removing gender features.

But let me talk for a second about the research phase here, because I thought it would be—I mean, if you've ever heard me talk about this, I probably told you about it, but just for a wider audience, I like thinking back on this. In some sense, it was probably my greatest victory of grad school: working on this embedding inversion problem for quite a while, proposing a lot of approaches, and testing stuff.

Sometimes you do stuff and it's clear it was a bad idea. Sometimes you think you should have figured it out earlier. Then sometimes you do stuff and realize it's really complicated and probably not worth it.

I was testing different decoding algorithms for embeddings, or text that was closer to the text in embeddings. I was testing these inference-time adaptation models for samplers. I think we tried a lot of architectures and training tweaks. We should have tried RL. I think that would work.

But we finally found something that ended up working. I guess I'm just saying all this because I thought it was so rewarding. We were just banging our heads against the wall. I'd have biweekly meetings with my advisor, and he'd suggest things.

Sometimes we'd agree we were mutually stuck. Sometimes I would get feedback one way or another and try something new or try a couple of things. We had this idea that it was possible from the information theory arguments and this other thing where we would take our best guess at what the text was, re-embed it, and see that it was kind of far from the true embedding.

So, we had this proof that a better method could leverage this kind of information. When we finally solved it, it was awesome. We had this number that was around 30 for months. I think at one point I got it to 35, and I thought, “Oh, I’m done. I got it to 35.”

My advisor told me, “Oh, no. You can’t really just propose a new problem and show that you pushed a metric from 30 to 35. That’s confusing and probably not that meaningful to people.” I think that was kind of a local minimum for me where I was bummed. But then we ended up getting the number to around 97 or something, which neither of us knew was possible. We were just staring at this graph, thinking, “Oh my God, who knew you could get this much information from an embedding?”

That was so rewarding. It was invigorating, honestly—the research process of picking a good problem and then spending so long trying things that didn’t work, which I’m probably forgetting how frustrating that was. I’m sure it was terrible. But actually solving, or at least coming up with a much better way of solving, the problem was great. I don’t know if I’d say we solved it, but we definitely learned a lot from where we started. It completely solidified for me the fact that I should have gone to grad school to have this life experience, and it makes me want to do research forever.

You’re clearly in love with the journey, which I think is important because it keeps you going through the tough parts. Is this a good time to talk about the universal geometry side?

Yeah, let’s do that next. I think that’s a good idea.

We have this more recent follow-up. The first part I was talking about ended up in this paper called “Text Embeddings Reveal (Almost) As Much As Text,” which was published in 2023. We recently had a paper come out on arXiv, which will hopefully be published at some point, called “Harnessing the Universal Geometry of Embeddings.” That was probably the second of 3 times where I felt like we had made a real discovery about the unknown. It was very rewarding for its intrinsic elusiveness.

I’ll start by explaining it in terms of the prior paper. We built a system that can do embeddings to text, and it works very well. We were all very pleased with ourselves. Then we went to a conference and talked to people about it. We talked to the vector databases, and I think some of them changed their privacy policies, which was somewhat gratifying.

We kept getting this perpetual question: “Well, you’re just assuming we use the OpenAI model. Or you’re just assuming we use the most popular text-embedding model. If they fine-tune their own model, or if they use a model that you’re not training an adversary for, then you can’t solve the problem.” That’s true. None of the vector-to-text stuff works unless you have the assumption that you know the encoder and can make a lot of queries to it.

But we had this underlying theory that all of the models learn very similar things. We have some preliminary evidence for that. Certain models that are fine-tuned from the same base can have their representations swapped without doing much. Or, if you look at the nearest neighbors, a lot of the models will give you the exact same nearest neighbors, even though they have completely different training bases.

Then there’s this paper that came out last year called “The Platonic Representation Hypothesis” from some folks at MIT, which is really compelling. I think it’s a great intersection of philosophy, representation learning, and deep learning research. I love this paper. It’s such a beautiful idea: all models are trained on data from the world, and there’s only one world. As the models get better by scaling data and model sizes, they’re converging toward learning the exact same thing.

In this paper, they have evidence based on correlations supporting this idea with vision and language models. It’s very neat. Basically, think about it: you’re us, and you see this Platonic Representation Hypothesis paper. A lot of people have this shared idea that Claude and GPT-4 probably do very similar internal computations because both of them are trained on trillions of tokens of human-written text. Even if they have different architectures, the actual basis, or the numbers if you look at them, may look different. But in some way, they’re computing the same thing.

I think it’s even more true with these embedding models, which really only have one objective that works. They’re probably all trained on MS MARCO, which is a really popular dataset, and perhaps pretrained on Wikipedia. We wanted to combine this Platonic Representation Hypothesis idea with the vector-to-text thing and produce a system that can align models so that we can do embedding inversion. It’s valuable for more than just embedding inversion: you can use this to glue together models. That’s what actually got me super excited.

By the way, I think there are a few related threads. I think we did an episode with Nicholas Carlini where he had an extraction attack on one of the GPT models, and they got it fixed. The other thing I was just realizing, for all of you, just in case you’re not thinking it through, is that being able to invert embeddings also means that you can back out secret prompts or context that might leak customer information. That’s potentially harmful and obviously an attack-vector issue.

One of the things I had a question about was whether or not positional embeddings affect it, and whether extending the positional embeddings affects it, because contexts are going to get longer and longer. Your ability to invert will obviously decrease with longer context.

What now? Maybe not that important. No, no, no, you’re totally right. We’re operating in this space in our work where the sequences are relatively short and the embeddings are relatively large. I think we’re at a great advantage from that perspective.

You’re definitely right: if you embed an entire book into a 500-dimensional vector, there’s just no way you could get the entire book back. There must be these kinds of collisions. In information theory, if you have lossy compression, 2 different inputs map to the same code, which means that you can never determine which input formed the code.

I think that’s probably what will start to happen. If you have 2 books and swap just 1 word and embed them, you’ll probably get a perfect collision. Someone can try this. In that case, inversion is impossible. Even when you don’t take it to the limit, it probably just gets very, very hard. Things get super-compressed.

I don’t know how well this work scales. It’s a great question: exactly how much information can you cram into 1 of these vectors? I don’t have a sense of where the boundary is. It would be interesting to talk to someone from the math department, maybe someone who works in linear algebra, about how literally we can take inversion. What measures of a matrix do they have that we can run and use to try to get some meaningful information out of that?

This is where information theory starts to collide with linear algebra and all the other stuff. Totally.

Yeah, there’s always this detail where we’re running these on computers. We don’t actually have real decimal numbers or real numbers; we have floating-point representations of numbers, which kind of throws a wrench into the mix.

Do you have any consideration of superposition? I mean, nonlinearity? You could stuff information in the lower bits, but I don’t know if that matters.

It’s a great question, and I get the sense that a lot of the less important bits are more useful for computation. Maybe the higher-order bits are more important for storing data or something like that, but I’m not sure.

These are the kinds of questions I’m hoping to explore over the next few years. I’ll skip ahead for a second. We have this result that’s maybe the third discovery I was alluding to: a way to measure the exact capacity of a language model.

If you train a language model on a ton of random data and measure its rate of memorization, you hit this perfect-ish plateau in total memorization, no matter how you scale the training size. We call this the model capacity. The question I’ve been stuck on in the back of my mind for a while is how that’s actually implemented.

This is a transformer that’s trained for many, many data points and many, many training steps. It’s almost like, if you have the 10⁶ point on the x-axis, the capacity—we don’t have to actually say the numbers—is basically perfectly dividing its computation between all of the data points.

Every one of the 10^6 data points gets a tiny sliver of the model parameters because they're completely independent random strings. So I don't really know if superposition is occurring here. It seems possible to me that the model would learn completely independent columns of computation, 1 per data point. But it's also possible it's learning some kind of combined thing where maybe it learns a load and a store, and it's sort of loading and storing bits using these generic operations. Then, in the end, it reconstructs the random string.

Even though the data is completely independent, the computation is very similar in terms of predicting random strings. But I guess this is all to say that, about superposition and everything, I have no idea how the mechanisms are actually implemented inside the models. That's 1 thing I'm hoping to learn about in the next couple of years. It's a reasonable question whether it's meaningful to learn. I think there are a lot of things that are nice to know but maybe not that useful. Latent space alignment is very, very useful. Dataset efficiency is theoretically cool, but practically, people are just going to go for the biggest dataset they can. The scaling laws are kind of worked out, insofar as the relationship between compute and data. A lot of memorization, I don't know.

I think maybe this is a good point to bring in the idea that Andre has been pushing for the last year and a bit: the cognitive core. What is the dumbest possible model that knows nothing but is smart enough for tool use to do everything else? You can run it on-device, get fast inference, make it open source, whatever. Gemma 3n is a really good candidate right now because it's a 4B model that's claimed to be better than Llama 4 and GPT-4.1, according to certain arenas that shall not be named.

This is where things get complicated. It feels like language models kind of implement things and know things almost in the same way, and it's really difficult to disentangle whether they're memorizing facts or learning useful ways to generalize about new stuff. But I agree this would be really nice. I don't think we have a lot of evidence that we can build a system like this that's really, really good at reasoning but really dumb about the world. I don't know if we have the tools.

Yeah, maybe, maybe not. I think the existence proof is humans, right? People always lean on humans as the existence proof. It's not a great existence proof, because if you talk to people about the number of neurons that we have and make a neuron roughly equivalent to a parameter, we have something like 100 trillion in our brains. We consume something like 20 watts of energy. It's nothing. We're so much better than language models, it's not even funny.

The last feature of us is that we're self-pruning, which is not something that language models do as well.

Oh, like, we will forget stuff.

No, we're not deeply, densely connected. Connections will drop, therefore we're more efficient, you know? So, unlike a language model where everything is always connected all the time—

Yeah, or you preset the skip layers or whatever, and that's it. It's not really anything we've evolved with learning. It's just something you do based on ablations and guesstimates. Even if we did want that, I'm not sure we have the right frameworks or methods for actually building what you're talking about yet.

I think the world is much closer to where you're at than where Andrej is at. Our conversation with Noam Brown was like, yeah, reasoning is emergent. If you gave the o1 harness on top of GPT-2, you would get nothing because GPT-2 didn't know enough. You need GPT-3 and GPT-4 in order to then get o1, with GPT-4 as the base model.

Which is, yeah, I mean, that's reasonable. The way I put it is, in order to use tools—in order to search Google—you need to know at least some search terms in order to search Google and then learn what you need. If you don't know what to search, then you might just be too dumb.

I like the ethos. Maybe you could do some kind of pretraining where, whenever the model doesn't know something, it can just Google for it. That way, you try to encourage it to learn words without—or to guess words correctly without—actually storing the information in its weights.

Yeah, it seems like a nice goal, at least.

Yeah, you need some kind of online learning, probably, or memory, and some combination of that. It's exciting. If that's the direction where this all ends up, that's great. But people aren't doing that. Instead, we're building 500-billion-dollar data centers in the middle of Texas. All hail the God cluster that eventually wraps around the Sun and consumes solar energy, because that's all we need.

Do we finish out the universal geometry thing?

Let me finish the methodological description. So, back to the embedding universality: we started with going from embeddings to text. We know about this Platonic Representation Hypothesis. Maybe I'll skip over the details, but basically, we took total inspiration from computer vision and this model from 2017 called CycleGAN, which is, among other things, a way to map between 2 different distributions without any underlying notion of which thing should be mapped where. It's just based on some kind of idea of closeness.

The cool thing about this—if you look at the top left, I guess the top left is Monet, so impressionist paintings, and the picture on the right is a photograph—is that it's learning this kind of semantic notion of what content goes where just by mapping a distribution of Monet pictures to a distribution of photographs, without actually telling it which Monet picture should map to which photograph.

Maybe go to the middle 1, if you don't mind: the zebras and the horses. It's clearly learning what an animal is, what legs are, and more abstract stuff, like what the camera position should be, what grass is, and so on. It's learning what a horse that looks like a zebra is, which is actually a complicated semantic concept. We don't have a dataset that has a horse and then that horse as a zebra. We just have separate horses and separate zebras. But somehow, this GAN system is able to elicit this sort of mapping property.

It's kind of a magical connection that it learns, and I'm still in awe that it's possible at all. But we more or less repurposed this system. We built our own, but we took this idea and applied it to model embeddings. Instead of zebras and horses, we have BERT embeddings and GPT embeddings—or 2 completely different models with different architectures.

I think these are GTR, which is a T5-based retrieval model, and GTE, which is based on BERT. They have different training data, different architectures, different downstream objectives, and different embeddings. Yet when we do this CycleGAN in the embedding space, they just perfectly snap to the same place, which is amazing and has some pretty deep implications for the Platonic stuff. Maybe the models are actually learning a lot of the same functions, or something, and in some semantic way they're very close.

And, yeah, this is a diagram of how our system looks.

It's weird to me how profound it seems. You seem deeply impressed by it. And then the other thing is, when we talked to Emmanuel from Anthropic, who did the circuit tracing and mechanistic interpretability work, they were excited that the same thing in different languages maps to the same circuits. I'm like, that's what you would expect.

Yeah, yeah. I don't know why. I think this feels more profound to you than it does to me. I'm like, yeah, obviously.

No, that's so fair. Maybe it's just self-congratulatory, and we're happy that we're the people who got it to work.

Yeah, exactly. It does seem obvious in retrospect. I think that's constant feedback I've gotten from research: people will tell you that this seems obvious to them. But you have to realize that you came from a perspective where no one had ever done this before, and they're coming from a perspective of you telling them it's true. If someone had told you that this was true, it would maybe be obvious to you too, if that makes sense.

The way I put it is that we have the intuition, but not the proof. You did the work and have at least some evidence that it's true, whereas we just have intuitions, right? So part of research is just confirming intuitions. The applied part comes from, okay, now that you know this for a fact, what do you do with it?

Yeah, right. I think the details can be really interesting: the details of the proof, which models are most similar to one another, to what degree you can get them to align, and on which distributions this property actually emerges. That's why reading papers can be fun sometimes: they answer all those little questions.

Yeah, I would say—okay, I'll pull out something very current, which is Gemma 3n, which launched and became generally available yesterday.

I would say that for me—and you can correct me if I’m wrong—the most immediate implication is mapping adapters to language models. The dream is that you have a language-model backbone—let’s say this one is a 2B language-model backbone—and then you offload your vision, so you only load in the vision adapter parameters when you need vision. You only load in audio, speech, and text-to-speech, whenever you need it, because these are all separately trained. You’re just aligning latent spaces, and you can train them separately.

I think this helps make us more confident in 2 things. One is that it’s more efficient. That’s a given. 2, it makes us confident that we can add capabilities without taking away or catastrophically forgetting others. They’re just stacking more parameters. Just stackable.

Yeah, swappable, stackable. It’s like a fatter version of LoRAs that’s not really model-specific.

I would say Apple and Google are pursuing this with their own on-device stuff, is my sense.

Is this open source?

Gemma, yeah. For a given definition of open source, which is, “We release the weights on Hugging Face. Here you go.”

Oh, that sounds like open source to me.

I guess it’s open weights, but not the data.

Not the data, not the code.

Not the code, yeah.

Right. Yeah, I would say that this is quite SOTA in terms of efficient models. Maybe a small LM also from Hugging Face would also be in that category. There aren’t that many people working on very good, very efficient models.

Yeah, this is a very deeply related question and something that really interests me: what is the limit of a 100 million-parameter model? If you imagine 100 years from now, when maybe our computers are gelatinous blobs and we all communicate through telepathy, will we have 100 million-parameter models at the level of today’s o3 Pro or whatever? If so, how would that even be the case? Based on scaling laws, do we have special data? Do we come up with a brilliant new training scheme or some type of magical architecture? I really don’t know. Or maybe we’re really on the plateau already.

It seems like when we’re doing things like calling a 27B model small, that’s what Mistral is doing. We’ve plateaued a little bit in terms of what we can do to compress things.

I have a fun theory that this is where we mix quantum computing with models. We have to change what a parameter means. We have to search through very high-dimensional space and resolve it much quicker than we can with conventional compute. That would be my pie-in-the-sky thing.

I said 100 years. That’s very reasonable to me. Throw quantum at it.

Yeah, I’ll probably have to get a second PhD to know what’s going on there. I think we should establish the definition of a small model as being a model that a grad student can run inference on in reasonable time on a single GPU. Which is probably 7B. Maybe. I don’t think 27B is small under any reasonable definition. Is it MoE? What is it, Mistral?

I don’t think so. I think their stuff is default dense. Don’t quote me on that. This is coming off of just a lot of pre-trained data that has potentially collided.

Okay, there were 2 more papers that we wanted to cover, and then we can wrap it. You had “Approximating Language Model Training Data from Weights.” I think this is a little bit newer. How does this rank in terms of your overall work?

Yeah, let’s return to the kind of information theory question. Maybe we’ll skip over the contextual embeddings in the interest of time, but we’ll group those papers.

Great paper. Hopefully people start training with that technique. It’s kind of a free lunch.

Those questions are all about information and model activations: how much can we recover from a given vector? What data does this vector represent? What computation does this vector represent? There are really 2 types of—if you want to taxonomize them—dense information-storage mechanisms. One of them is activations or embeddings, which we were discussing already. The other is weights, which are the things that are used to perform the computation, but not the computation itself.

We now have 2 papers in this direction of what is stored in the weights. The first one is about language-model capacity, which is called “How Much Can Language Models Memorize?” Or “How Much Do Language Models Memorize?” I never remember which one we settled on. The other one is called “Approximating Language Model Training Data from Weights.” The first one, I think, has a lot of deep messages about how language models store information and how they work in general. The second thing is a proof of concept of maybe a longer-term research project.

Let’s start with the capacity stuff, if that’s good with you. Do I have the paper for that? I don’t know.

You know, we can return to the question you asked me, which is something like, “Why do we care?” Or, “What is this useful for?” I don’t know if I have a good answer for this. I think this is somewhat profound. It’s kind of like in physics, when they try to measure these constants, like gravity. People tried to measure the rate of acceleration due to gravity for a long time. Or those Greek guys back in the BC era, when they were trying to approximate the radius of the Earth based on shadows. We’re trying to take the GPT architecture—the main one—and just measure how much information it can store. We did this through the lens of memorization, which I think we can skip over for the podcast. We’ll just talk about information storage in weights.

These curves, to me, are pretty crazy. Again, maybe it’s the sort of discoverer’s folly, where I’m like, “Oh, this didn’t exist before, so it seems so cool.”

But then you’re saying it seems somewhat obvious.

No, don’t let me take that away from you.

Again, I independently was asking how come there aren’t enough people exploring LLMs from information theory. Then you come along and your embeddings work has become an information-theory exploration, and I’m suddenly very aligned with exploring this, promoting this, and encouraging more people to figure it out. That’s ultimately how we figure out this whole compression issue and what Andrej wants, which is the cognitive core, right? The most efficient model for the most capability. That is an information-theory question.

Totally agree with that. We could start here. Transformers that are trained in 32-bit precision, we estimate, can store about 3.6 to maybe 3.9 bits of information somewhere in there per parameter. For some perspective, this is quite bad. If you have 32 bits available and you can only use 3 to 4 of them, you’re—

Just store 32, bro.

Yeah, then you could build your own AI lab if you can make these models that much more efficient. I don’t know how they’re implementing this mechanism or where the bottlenecks come from. Even now that we know this, I don’t know what it’s necessarily useful for.

I guess the tools that would be interesting to me are knowing, given a dataset, if you could predetermine the exact model size and maybe the architectural properties required to get a certain level of performance. That would be really neat, and we don’t even know how to do that. We don’t even know what the difference is between doing LoRA training, which trains less than 1% of the parameters, and full fine-tuning, which trains all the parameters. We don’t really understand the difference there. I think this is maybe a baby step in that direction, but there’s a lot of unknown ahead of us.

Do you think this is a hard limit? Do you think someone can come up with a better algorithm or architecture and then just change the slope?

There are 2 axes here. One is the ability of the model to store data, and I think we can definitely improve that. Maybe even if we tested this with the Llama architecture, there's sort of like a GPT++ architecture, I would guess that it could store data better, just because the numerical flow is a little bit better. The nonlinearities are maybe a little bit more suitable to training. That will probably raise the bound a little bit.

The second axis is that our measurement tools are just not that good. I’m a grad student, running all these hyperparameter sweeps and drawing conclusions from them. Even so, there are probably ways to measure this better, but all that would do is push the number up. It’s possible there’s a way to store 5 bits per parameter if you have a better optimization technique, or if you were a super-genius and could perfectly set the weights to store the data. Maybe you can do better. I would be happy if someone came along with a much better measurement tool.

And this is just the first measurement. I would guess that in the future, people will look back and say this was somewhat off in one direction or another for whatever reason. That’s just how science goes, and I have no problem with it.

What we do is we call this the Morris constant, 3.6, right?

I would never.

And then we set a challenge—a leaderboard: beat this. And let people go.

Yeah, that assumes we know the true constant ahead of time and can measure the error rate.

It's doable. You laid it out here. Yeah, that makes sense. One minor doubt I have is that the goal actually isn't memorization; it's generalization, right?

Mhm.

The best memorizer model may not be the best generalizer model. Incentivizing people to max this number might actually be fruitless in terms of actual intelligence. You just get the best actual compressor; you're going to get gzip.

That's totally true. And there's this pattern in research, time after time. Someone poses a question, and then people answer it over and over and over again. But it's often much more fruitful to ask a new question. Maybe it just doesn't matter how much GPT models can store, and you should work on something else. We'll figure that out.

Did you want to dwell on this side at all?

Yeah, let's just talk about it real quick. Definitely not the algorithm itself.

By the way, what are your tools for doing these kinds of charts and diagrams? I'm kind of curious about the behind-the-scenes tools.

I think visualization has definitely been a fun hobby of mine during grad school. This one actually Oscar, my co-author, made. Maybe I gave some prompting, but he made it. I think most of the last few papers have all used Google Drawings. I was using Figma for a while, and Illustrator. I think Illustrator is actually the best tool.

Oh, did you know the Transformer diagram was made in Adobe Illustrator?

Oh, yeah, I did know that, actually.

Yeah, because that's the only way you can get arrows that curve like that. And they've got good shadows.

Google Drawings is the least robust, but it's the most accessible. Honestly, if you're good, you can make pretty good stuff. Excalidraw is nice, too, if it's not going in a paper.

Yeah, it's just too rough for a paper. But you need something professional-looking. If you're going to publish your work, you need to make it look nice, professional, and official, right?

I think there's something worthwhile about saying, "Okay, if I'm going to put my name behind this, I want to spend time making all the references perfect, all the diagrams professional, and all the captions correct." I think it's important to put that level of detail into your work.

That's a little chuggy, but let's finish this off. So, okay, we're talking about bits, information theory, what information is stored in embeddings, and language model capacity. I think a much more practical question is—maybe this is more analogous to the vector database hacking and embedding threat model we discussed—if you have access to a set of model weights, what can you learn about the data?

You were just mentioning Gemma 3B, which came out yesterday. You can download it, and it takes up a certain amount of space on disk. It was trained on some data, but we have no insight into what that data was. It's probably English. It's probably some distribution of web text. I guess there's a lot of code.

And we seem to have a lot of information about the model, right? You have this file with many 1s and 0s, which means something, but it's kind of like a highly compressed version of the training data. I would be extremely surprised if they did any type of private training. There are mechanisms for doing differentially private language-model training, or even just anonymization in the pretraining pipeline. I bet they don't do any of that. They just train on the data, and they know that we don't have the right tools to decrypt the model weights.

My dream is that we can come up with some way of translating model weights back into text datasets. In the most recent paper drop, there's a paper called “Approximating Language Model Training Data from Weights.” It turns out to be a really hard problem. Trying to go from model weights to text is really hard, and we do something a lot simpler.

There are 2 ways we make it simpler. The first is that we assume access to 2 checkpoints, which is probably not the case with Gemma. But in the case of DeepSeek, if you download the 400-billion-parameter model weights, it's this giant file, and you can actually get 2 of them: the base-model weights and the fine-tuned-model weights.

The way we put it is that you have this difference in parameter space telling you what DeepSeek fine-tuned on. It's very controversial. There are geopolitical implications; definitely at the corporate level, they're really interested in what DeepSeek trained on. They've released this treasure trove of information about what they trained on—the actual model weights—but we have no tool for interpreting or decrypting this weight difference.

We started with something really simple. Instead of trying to regenerate the training data, we take a web corpus and select the training data that looks like the true training data and gives us performance as close as possible to the true training data. There's a complicated method, but it's something like looking at the data-point gradient, seeing if it points in the direction of the fine-tuning in weight space, and then taking the top dataset. There are some tricks to it, but it's basically gradient-based selection based on this weight difference.

It seems to be okay. It can get us pretty good training data. If you wanted to use this, it would be if your competitor released a base model and a fine-tune and you were trying to recreate their dataset. You could take that weight difference and a giant web dataset. If I were doing this at a company, I'd probably try to scale it up to trillions of tokens and select the exact data points that try to produce the model. It turns out you can train a pretty good model with that. We don't get quite to the performance of the original model, but it does seem to be trending in that direction.

This is very creative. I don't know what the use of it is.

Yeah, when would you be in this exact situation?

Decently often for the open-model labs. DeepSeek, for example, has released an update. Mistral does it pretty frequently. Llama does it frequently. It's not impossible, but I really like the creativity in using “synthetic checkpoints” to do this, which I don't think I've heard from any other place. I don't know if you came up with the idea. It's like linear interpolation in weight space.

Okay.

That's a bunch of the recent work. I wanted to cap things off with the datasets question. If that's a good—

You can ask me whatever you want.

Well, it's not a good—it's not an ask. It's just that I think this is a very good thesis. I think it's a hot take. I almost invited you to speak based on this alone, but it was a little bit late.

Oh, for the conference?

Yes. When I look for conference keynotes, I look for something that has a broad overview—something that can put the last few years in perspective—or an insight that you can reasonably rely on to last for a while, so you can get some mileage out of it. A lot of ideas in AI come and go, but things that are trends—scaling laws, trend lines, things like that—are the kinds of things I pay attention to. So maybe you want to recap the backstory, if there was one.

Yeah, sure. The meta backstory is that I've started writing on Substack, and this is a post I wrote a few months ago.

“The Highest Art Form of Humanity.”

Yeah. Publishing papers wasn't doing it for me anymore, and I moved to Substack. That's the name of the post: “There Are No New Ideas in AI, Only New Datasets.” One guy pledged to me, but then I found out he was my former student from a class I was teaching, so I don't think it really counts.

It counts. He's a friend. He's your first supporter.

A pledge is a pledge, man. I'll take whatever I can get.

The underlying thesis is that whenever—maybe I'll lay out this framework first—there's this book called The Structure of Scientific Revolutions by Thomas Kuhn that I read near the beginning of my PhD. It suggests that science moves in cycles. Not very often, there's something he calls a paradigm shift, which is like a 0-to-1 innovation where everything changes.

Then it's followed by a rapid period of small innovations and a lot of reapplication of previous techniques—pre-paradigm-shift techniques—to the new era. Then things slow down as we wait for a new paradigm shift. I was asking myself what was unique to the paradigm shifts that we've seen in AI.

And, by the way, to me, AI and language models are somewhat synonymous at this point, at least for the foreseeable future. I'm certain that will change, but basically everything that's pushed the boundary to what we have now that resembles intelligence has come from language models.

Those breakthroughs came in a few steps. I think the idea is also a meta-commentary on the research community, because what everyone wants as a researcher is some kind of cute new method that no one has thought of before that just works on the existing data better than the previous methods.

That’s, for whatever reason, the kind of most glamorous thing people think you can do as a researcher. Mamba—it’s like a transformer, but it’s more efficient and works better. So that’s what a good idea looks like, and I think everyone wants to find something like that. But if you look at what’s actually borne out in practice, it’s never been like that. I think all of the things that I would consider paradigm shifts in the Kuhnian sense came from a new technique trained on new data, and I think the new data is super, super important.

So I wrote it as a series of 4 paradigm shifts. The first is the emergence of deep neural networks with AlexNet, which I think was the 2010 to 2012 era, where we just started training on ImageNet, a scale no one had ever seen before: millions of images. Then the second thing was transformers, BERT, the paper “Attention Is All You Need” in 2017, and the first GPT in 2018, which was web-scale pretraining.

No one had ever done that before. No one had ever tried to scrape all the text off the internet, tokenize it, and feed it into models. It’s a crazy idea, and I think we should be honest: transformers are incredible, and their staying power is never going to cease to amaze me. They’re much more optimal than I think anyone ever knew, and I don’t know if we’ll ever beat them. But the real innovation is web-scale pretraining, and I think we honestly probably could have gotten this with RNNs.

I know the Scaling Laws for Neural Language Models paper shows that RNNs have worse scaling curves, but probably people would have said, “I bet you could have built ChatGPT with a very sophisticated RNN.” You didn’t even need transformers. What you need is web-scale pretraining.

The third innovation is instruction tuning, which we thought came with reinforcement learning. But I think the big innovation of instruction tuning is actually the human preference data: gathering positive and negative pairs of what looks good in terms of a chatbot interface. It turns out you can do supervised learning on that, too. You can do DPO, which is a form of supervised learning. You don’t even need the InstructGPT techniques; you just need the data.

So I’m playing devil’s advocate here, but I actually think this is true: if we had the right datasets, we almost could have scaled 2015-era techniques and gotten something that looks like at least InstructGPT. Reasoning models are a little different. I’m not sure if we could have done that with RNNs or not. I don’t know if I’m in a position to comment on that with certainty, but they do fall into this framework: they really did emerge from a new data source.

In this case, it’s something a little different. It’s verification with symbolic systems—math calculators, coding environments, unit tests—things where we can provide numerical feedback to language model outputs. But we built a way to learn that and leverage it to get more intelligent systems.

Whatever the 5th thing is, whether it’s video, embodied AI, or some kind of crazy innovation on reasoning models, whatever comes next will probably be some type of new data source that we’re not using yet.

That’s a really good thesis. I would say that the researchers I talked to would somewhat disagree. Obviously, this is a hot-take type of thing, and you already acknowledged that RNNs don’t scale to the same extent. They operate on the slope of the curve, whereas the amount of data, the type of data, or the core insight just changes the order of magnitude of the x-axis that we’re mostly working on, right? But both are important.

The way that I think someone put it to me was that an improvement in compute or data efficiency is the equivalent of having a whole bunch more data that otherwise would be a lot more expensive to collect. It’s likely that the frontier models right now are just a collection of hundreds of these small experiments that stack up. You mentioned Muon in your post, which seems to be the atom killer. Curiously enough, still none of the big models use Muon, but the vibes are good.

Yeah, and the value of building better optimizers is really incredible. It’s just a free lunch. You can plug in a slightly better training mechanism, and then you save a ton of compute and a ton of training time. That’s hugely valuable.

I think this is cool because it puts us in a mode of asking: if you were ever to ask what comes after reasoning, it has to be something on the order of this. And most ideas are not. Most ideas are not. This is cool in the sense that it just jolts you out of incremental thinking into asking what really is missing for the next paradigm. I don’t have an answer. Do you have one? Do you have candidates?

Oh, I really haven’t even considered that too much. I guess scaling reasoning to do the autocomplete for step 5. I mean, you got us all the way there, and you were like, “You’ve got to show us the way now.”

We can say it’s an exercise left to the reader, but the reality is that predicting the future is too damn hard. Maybe it’ll be obvious to me in hindsight in 5 years, but sitting here today, I really can’t derive from first principles what the next wave of innovation will come from.

Yep. I think we have a few years left. Each of these phases lasted for a few years. Reasoning just started last year. We’ve got some juice on this one.

Cool. I think that’s a broad overview. We went way over time, but I really enjoyed this. I guess my parting question for you is kind of a meta one.

I’m not an academic. I’m self-taught. I just read a bunch of papers and talk to people all day as part of the podcast. How do I rate in terms of my questions? Could I pass as a grad student, or what’s my distribution like? Maybe I’m more industry-oriented than academic.

I think you have to realize that the only person who’s an expert in your area as a grad student is you. Even eventually, your advisor defers to you for a small set of questions that fall within your very niche expertise.

So I think you’re clearly a very good generalist and have a huge amount of background on these topics, to the point where I would say you’re passing the grad student Turing test. I think if you went to a talk, people would just assume you have some weird research area of your own that they don’t understand.

My research area is AI engineering. I’m trying to make it up as I go. But no, this is super helpful.

Okay, well, that’s about all we prepared. All the best in your search. All the best in your PhD. Apparently, the current PhD meta is that you do a bunch of small papers, staple them together, and find an overall theme. You do a defense, and that’s it. That’s the journey, which is kind of cool. I would love to do that. I’m too old to do it, but it’s cool.

Yeah, it’s a great thing to do at any age.

Well, it’s better to do a Substack, right? Then you have people subscribing and pledging along the way, getting validation. That’s better than a PhD. “Substack better than PhD”—that’s the title of the episode.

But no, thanks for your time.

Yeah, thanks for your time. This is really great.

Where can people find you? What are you looking for, really?

I’m online. You can follow my Substack and Twitter. I tweet pretty consistently.

And you’re putting papers out.

I guess the most meaningful thing, to be honest, is to engage with the research and send me an email if you really care.

That’s amazing, and I love having those kinds of discussions.

And you mean what I’m looking for in a job or out of life?

Your research direction. What interests you above anything else? If there’s someone out there looking who has a problem and is looking for someone to help them with it, you’re the guy for this.

Oh, yeah. Hopefully, if you’ve listened this long, I think my research is much more well-connected than some people’s PhD research in that it all falls into a very small manifold of all possible problems.

So if you want to work on anything within that space, or that’s adjacent to the problems we discussed in terms of language models—and maybe not even language models, but model weights and activation information—I think anything that can be described as that is very interesting to me, and I would love to talk.

Awesome. We’ll put your contact information in the show notes. Thanks for your time.

Thank you.