《更好的数据就是你所需要的一切》—Ari Morcos，Datology

So, you first came across my radar. I mean, I guess Datology is a relatively exciting or well-hyped startup, at least with the fundraising and the high profile of the people that you hire. I reached out to book this interview after you worked on the RC. I don't even know how to pronounce it.

RC.

Yeah, the RC foundation models. You guys have been doing a lot of data work. How would you describe Datology today?

Our mission at Datology is to take everything around the data side of machine learning. You have a bunch of data sitting in storage, and you're going to feed it into a model via a data loader. There are a ton of choices you would make in that process, ranging from how you're going to filter the data, how you're going to sequence the data, what synthetic data you're going to generate, if any, and how you're going to batch the data.

All of those things will have a tremendous impact on the performance of the model that you train on the data. One of my favorite catchphrases is, “Models are what they eat.” If you show them great data, they're going to be really high quality. If you show them low-quality data, they're going to be low quality.

But this is a frontier research problem. How do you actually do this effectively? How do you do this automatically at scale? It has to be automatic to be able to process trillions of tokens, billions of images, and things like that. Our mission at Datology is to take that whole process and make it really easy, so that anybody can get access to state-of-the-art data curation without needing to be an expert themselves.

In doing so, we help the folks we work with train models much faster and to much better performance. We also help them train much smaller models to the same or better performance, which I actually think is some of the most exciting stuff going forward. Fundamentally, what we do at Datology is help people curate their data so they can train models faster, better, and smaller.

So, the key words for that are data curation as a service, data efficiency, and all those terms. In the pre-chat, before we started recording, you mentioned that there's a cool story around how you got into data in the first place, right? You were at Google DeepMind; you were at Meta as a research scientist. Describe how that became an interest.

My PhD is actually in neuroscience. I come much more from an empirical science background. I actually spent time trying to teach mice how to count and then analyzing the activity of thousands of neurons in the brain while the mice counted, trying to understand how that actually happened—what the neural dynamics were that enabled that.

That's actually initially how I got into machine learning: as a means to analyze my neural datasets. I also started my PhD in 2011, so AlexNet came right after that, Char DQN right after that, and there was lots of evidence that AI was going to be very exciting, which led to me transitioning.

As a result, because I had this somewhat different background of being trained as an empirical scientist rather than as a computer scientist, my real first mission when I joined AI was to try to build more of a science of deep learning. Something that I think is still true today in many cases is that deep learning is an empirical science.

Most people who have computer science backgrounds were trained more in the context of a branch of theory. Everything was very provable. The initial pushback to deep learning was actually that you couldn't prove anything in it. But deep learning is, at its core, an empirical science. We have to run large experiments. We understand the rules for how we design these systems, but the properties that come out of them when we actually train them on a ton of data are emergent and unexpected.

I always really wanted to write papers where they had 2 halves. The first half of the paper was trying to understand why a representation was desirable or undesirable, or why a model was good or bad. Then we would use that understanding to improve the model. That was always my goal. That was kind of the perfect paper.

Rather than just throwing spaghetti against the wall and seeing what stuck, we were able to really understand why something didn't work and then use that understanding to improve it. Unfortunately, it turns out that it's not so difficult to do the first half—to try to understand the system—but it's really, really difficult to actually use that understanding to improve the system.

A lot of times, what would happen is that you optimize for a variable and find, “Hey, here's this property of representations that makes models good.” You go and optimize for that, and then it turns out that wasn't a causal variable; that was a correlate. It doesn't actually work. I maybe wrote 30 papers where we did that first half and maybe only 3 or 4 where we did that second half. That was always kind of frustrating and dissatisfying to me.

Then, around 2020, I had several papers that all slapped me in the face at the same time with the same insight: all that really matters is the data. I had come into all 3 of these papers very much focused on inductive biases. How do we put better inductive biases into models, either through changing the objective or through changing the architecture? That's where most of the field was, and it's still where you see a lot of the papers at the big conferences: architectures and various tweaks to architectures.

But I had these multiple papers, all of which made this clear takeaway that the data is the only thing that mattered. I'll give you 1 example. There's a paper we had called ConViT, where the idea was to take a Vision Transformer and initialize it as if it were a convolutional neural network. That way, you could start with this inductive bias of convolution, but the model could choose to unlearn it if it wanted to.

The idea was that it was a soft inductive bias, not a hard inductive bias. ConvNets have a hard inductive bias. You can't not be convolutional in a ConvNet. But in this case, you initialize the Transformer that way, and if it wants, the model could learn not to be that.

Just to follow up: there's a one-to-one mapping of a ConvNet to a Transformer, and you can map it directly onto the weights?

Exactly. You can map it exactly. If you have a 3×3 kernel, you can have 9 heads; each head corresponds to a different part of that kernel, and then you can initialize it.

So, it's like a very coarse thing that can then be refined with training.

Exactly. Then it can choose to change its weights so that it can undo the weight tying that you impose on it this way. We actually had a follow-up paper that showed you could take a trained network and instantiate a trained CNN as a ViT as well. So there's a way to do this.

It turns out that in the small-data regime—and when I say “small data” here, I mean less than 500,000 data points—and in the context of image self-supervised learning, this is super helpful. This paper has actually been cited in a whole bunch of niche scientific problems where there's very little data, such as volcano prediction, where you have 1,500 data points.

But the advantage of using this soft inductive bias decays as the data size increases and eventually becomes harmful. If you see enough data, the threshold at which this changes is around 1 million data points. So it's not massive by any stretch by our current model standards.

Basically, once you get past 1 million data points, that soft inductive bias no longer helps you, and it actually becomes mildly harmful. I had this paper and a couple of other papers that all made the same point: when you get to enough scale, inductive biases matter not at all. All that really matters is the learned posterior from the data distribution, and that's really what defines everything.

Then, of course, the rise of the Transformer really showed that starting with models that have fewer inductive biases built into their architecture is the right thing. We had this combination of factors, which ultimately was very confronting for me because I had spent the last 6 years of my career working on inductive biases. Now I'm faced with several different papers, all of which show me that what you've been working on isn't actually that important.

Bitter lesson.

The bitter lesson was indeed very bitter for me. That was really my inculcation in it, I suppose. At the end, I thought to myself, “Clearly, the bitter lesson is true here. What should I do in this new world?”

It became clear to me that there were really 2 options that made a ton of sense: either go work on making GPUs go brrr—and I'm not a hardware engineer; I don't know how to make GPUs go faster—or work on data. For a whole bunch of reasons, data has been dramatically underinvested in relative to its impact.

Something I've said before, and I'll say again, is that data is the most underinvested-in area of research relative to its impact. I don't think it's even close. There are a whole bunch of reasons for this, some of which have to do with the culture of machine learning and some of which have to do with the incentives that have been set up.

But data has generally not been considered. Even if you look at the scaling laws work from Kaplan and Chinchilla and all these other things, they all assume IID data, which is insane. We know that all data are not created equal. “Garbage in, garbage out” is the oldest adage in computer science, and yet all these scaling laws assume that all data is created equal. That makes no sense whatsoever.

That's what led me to start working on this problem. It turns out that there's a really cool thing about data research. In addition to it being impactful relative to the investment, which makes it a great research area and makes it an even better company, there's a connection between the scientifically interesting questions and the practically relevant ones.

What I had said previously was that with representations, you have this disconnect where there are questions that are scientifically interesting about understanding why a representation is good, and then there are the questions that are practically relevant: How do I use this to improve it? What was so frustrating to me early in my career was that those were different questions a lot of the time.

The questions that I wanted to ask, which were curiosity-driven and really interesting to me as a scientist, often ended up not being the questions that were practically relevant downstream. But it turns out that with data, this is no longer true. With data, if you can understand what makes a given data point useful or what makes a given data point uninformative, you can almost always use that insight to make a data set better and therefore make a model better.

What this means is that the set of questions that are scientifically interesting and the set of questions that are practically relevant in data research are largely the same questions. That's really rare to find in research, period. What this means is that we can ask the questions which, as scientists, are extremely motivating to us, but then have very high confidence that the answers to those questions are going to help us build models that train much faster, train to much better performance, and can train with far fewer parameters.

So that's a little bit of a high-level overview of how I got into the data problem and the pain that I had to go through to get there in the first place.

You mentioned something about the incentives in data not being aligned. Can you unpack that? From the outside, you have companies like Scale AI that have obviously become super successful, so people are investing a good amount of money. But what you're basically saying is, NVIDIA is $4 trillion and Scale AI is not $4 trillion. Why do you think there's that inefficiency?

First off, we have to divide the research community from the industrial community, because I think they're very different. In general, data work has been far more valued in industry than it has been in the research community.

First and foremost, part of this is that data work has often just been considered second-class work. It's the grunt work. It's the plumbing. It's the stuff that you don't want to work with as a super hoity-toity scientist. There have even been tweets recently going around saying that data cleaning is boring and low-value work.

Whereas I think what you'd find is that if you talk to the most talented AI researchers and ask them what's the secret to their success, they'll largely tell you that they look at the data. Ultimately, these models are a reflection of the data that you show them. It can be tedious and challenging, but it is so critical to get this right.

I think, first off, there's this general perception that this is lower-quality work—or not lower-quality, but lower-prestige work—and that's been there for a long time. I think part of this had to do with the way that research incentives were set up. The data set was viewed as a given.

If you think about research circa 2018, given ImageNet, you maximized performance on the validation or the test set, right? But the data set—ImageNet—was given as something you don't change. Even Kaggle had this framework: Given the data set, go and make this better. People might try things like bootstrapping or stuff like that, but generally the assumption was that you were going to improve the model through better modeling, not through improving the data set.

Part of this also was that, in the supervised learning era, this made sense. We generally weren't compute-limited; we were generally very data-limited. Data was very scarce. If you wanted to assemble ImageNet, you had to go to MTurk and get a whole bunch of people to label the data set.

There was generally some quality floor, because a human had looked at every data point in the data set. Even if there were still a lot of errors, at least it wasn't going to be as bad as just scraping the internet. But then, in 2019, the field underwent this pretty massive change. We figured out how to train without labels.

One of my more controversial viewpoints is that I think the Transformer is a great advance, to be sure, but I think it's one of a very large set of equivalently good architectures that we could have found. There are many ways we could get to the same performance without the Transformer. But I do not think there's any way we could get to where we are today without self-supervised learning and the ability to train on unlabeled data.

That was the real advance, to my mind, that enabled us to get these incredible increases in capabilities.

Which is the masked objective?

It's not just the masking objective. I think the masked language modeling objective is one, but even next-token prediction. Generally, it's this notion that instead of having to get an external label from a human, we can ask the model to predict one aspect of a data point from other parts of that data.

That is really powerful, because think about it: That meant that we went from ImageNet's 1 million data points to literally trillions of tokens—a million-fold increase in data quantity in a matter of several years. That's completely unheard of.

That also changed everything, because now we went from data being scarce and having a high-quality floor to, all of a sudden, data being absolutely massive. All of our models are basically always underfitting the data, whereas previously we would do 160 epochs on an image data set, and they would generally all be overfitting the data.

So now we moved to this underfitting-the-data regime. There's no more quality floor, and now we have all of these problems with redundancy, low quality, low information gain, and all these various things that come with these massive unlabeled data sets.

I think the problem also changed pretty dramatically from the 2010s to the 2020s. I think that's what makes it so exciting as a scientific question: This didn't really make sense to study prior to 2020, but now it makes tremendous sense and is absolutely critical for us to solve in order to enable these models to continue to improve, and also to enable the cost-effectiveness of these models so that they don't just stay as something that's only possible to achieve if you have hundreds and hundreds of millions of dollars.

Making the data better can be a massive compute multiplier. It can change the performance per dollar by orders of magnitude. In many ways, that's our whole goal: How do we make that easy and effective for everyone?

Totally. You were at Meta from 2018 to September 2023, which was during both Llama 1 and Llama 2. At what point inside Meta did some of these learnings become apparent—when you thought, “Okay, we should start to spend resources working on this”? You mentioned 2020, so I'm wondering if that was—

I think Llama 1 was already a big breakthrough. Llama 1 definitely put more effort into data filtering, I think, than many others, and definitely started to change us. But even then, I would say that when I left Meta, the idea of actually curating the data to figure out what the high-quality, high-value data was still remained fairly underappreciated.

If you talk to a lot of the folks on the data teams within the big frontier labs, what you'll find is that they've invested really heavily in crawling. Oftentimes, they've worked on getting better crawlers and trying to clean up the source of the data that's coming in, which makes sense.

But ultimately, I think what you really need to do is take this perspective: Given everything that the model has seen so far, and given a potential candidate set of data, what data point is going to teach the model the most the next time it sees a data point? That's a pretty different framing for how to think about this problem.

I think there's certainly been some great work done, although it's all secretive within the bigger labs. But that's a really hard problem. That's a frontier research problem, and I don't think we still know how to solve it.

I think data creation is also a hard problem to solve, quote unquote, because it's not one where there's a single silver bullet. There's not just one trick you can use and all of a sudden things work. Rather, here are 50 different things that you can do, each of which provides a pretty modest gain on its own.

But then, if you can figure out how to make them combine, you then get a really big gain. But you have to figure out, first off, what are all these different things you want to do? And then, 2, how do you make them play nicely with each other? Because by default, they don't play nicely with each other.

Yeah, I'll make a quick observation. You mentioned self-supervised learning. I definitely agree that just getting rid of labels altogether is great, or forming your own labels, right? And I have a general observation that I think extends to things that are not just learning: self-supervised optimization, self-supervised neural architecture search, self-supervised curation. If you can just automate everything, I think that's the lesson—just get the machines to do it, because we are the rate limiters if we must label everything.

Yeah, I think this is very true. It's actually something I think about a lot: are we actually falling prey to the bitter lesson again here by trying to have human-guided methods of data curation? Probably the best open effort on data curation is DCLM, DataComp-LM. It was led by Ludwig Schmidt, a professor at Stanford, and about 30 students across many different institutions. A really wonderful effort to curate Common Crawl-style datasets.

Yeah, we've actually covered DataComp and DCLM on the podcast.

Awesome. Great. But DCLM had a really cool study at the end of the paper that I don't think gets nearly as much attention as it should.

They had these 30 grad students spend 2 years basically trying to design the optimal filtering criteria for these models. And they built a system that's pretty good at this. Then they asked all those students to predict what that system was going to do. So, given a data point, is the system going to say, “Keep the data point,” or is it going to say, “Reject the data point?”

These are nominally the best experts you could ever hire to do this. These are students who have just spent all of their time looking at NLP data for 2 years. They could not predict what the DCLM classifiers would say above chance.

This comes up a lot of times where people often ask me, “How can you possibly do this without a human in the loop?” It just seems impossible. You need to have a human to actually rate these data. But obviously we have to be automated because humans just can't scale to billions of data points, trillions of tokens. It's just not possible. But even if we could, we actually wouldn't want that: humans are not good at this task.

To give an intuition as to why humans aren't good at this task, the easiest way to think about this is that the value of a data point is not just a function of that data point itself. It's rather a function of how that data point relates to every other data point in the training set. If I have 10,000 copies of slightly variable summaries of Hamlet, I don't need all of those. But if I were to look at any one of those individual summaries, I might say, “Hey, this is really high quality. This is really accurate. It tracks all the characters. It's well written. It's clear. But I don't need 10,000 of those.” That's just a task that a human would never be able to do because a human can't keep the whole data set in their head, obviously. So even if you could have this scale with humans, you wouldn't want to.

But what's the right number between 1 and 10,000?

The unsatisfying answer is: it depends, but it's also the right answer. It depends on how complex the concept is. Redundancy is really useful, right? Removing all redundancy is a bad thing. If I remove all redundancy, then I'd only be able to understand, say, a golden retriever in the one situation that I've ever seen it in before. I wouldn't be able to generalize, and that would be bad, right?

So some redundancy is good, but I think we all have the intuitive understanding that infinite redundancy is not good. It's bad. Where is this line for different concepts? One example I like to give for this is elephants versus dogs.

Elephants are pretty stereotyped. There are 2 kinds of elephants in the world: Asian elephants and African elephants. They're all gray. They all have floppy ears. They all have a trunk and some tusks. They all have wrinkly skin. African elephants are bigger than Asian elephants, but largely they're all pretty similar. There's not too much variability.

So I don't need that much data or that much redundancy to understand the concept of elephants fully and completely. But dogs, on the other hand, are totally different, right? Dogs are super variable. There are hundreds of breeds, not to mention all the mixes of different dog breeds. They have different shapes, sizes, textures, colors—all of these different things. The amount of data that I need in order to properly understand dogs is going to be a lot higher than the amount of data I need to understand elephants.

So this gets to some of the challenges when you're actually trying to do this sort of curation, at least on the filtering side. You don't get a dataset where you're given, “Hey, these are a bunch of dogs; these are a bunch of elephants.” Instead, you just get, “Here's a bunch of data,” right?

First off, you have to, in an unsupervised way, discover what these concepts are; use something about that concept in order to make some inference about how complicated or complex it is, and therefore how much data you need to understand it. Figure out, “Okay, this is a really complicated concept. I probably should keep a lot of redundancy. This is a really simple concept. I don't need that much redundancy,” and then make that appropriate choice of what you want to remove.

So this is, I think, where a lot of the challenge comes from, but these are the sorts of factors that you have to keep in mind when you're trying to design these systems.

How do you draw the line of a concept, though? Because then it's like, well, the elephant and the dog, but what about mammals? And then what about—you know what I mean? How should people think about it? Maybe that's why you need an ontology, because it's hard to talk about.

Yeah. No, I think that's right to some extent. Look, it's an empirical question, like all things are, right? With every dataset, you can choose a different level of fine-grainedness. Ultimately, it's a hyperparameter, a knob that you can tune for how aggressive you're going to be with respect to creating new concepts versus keeping concepts together.

It's one of these things where, to your point, it's why we've run hundreds and hundreds of thousands of experiments to try to figure this out. I think this is something where it requires just a lot of experimentation to understand how to do this.

And I think one of the challenges we have is not only do we have to make this work on one dataset, but we also have to build a system that can automatically adapt to any arbitrary data distribution and be able to make the appropriate inferences zero-shot on a new data distribution.

So we kind of have these 2 sets of questions. First off, how do we push the frontier of data curation forward? And then, second of all, how do we do out-of-distribution generalization, where we say, “Hey, we have this great data curation approach. How do we make sure that this generalizes to a novel data distribution?”

I don't know if this is a good time, but I was going to ask for a brief history of datasets. It might be too much. I don't know. I'll just list off, because we've done a Dataset 101 episode. It might be one of our earliest episodes by far, because we want people to know the datasets, and I think everyone starts at Common Crawl.

I think every lab has their own web scrape. Would you say that's true, or do they start from Common Crawl—

At this point? Yeah, like I said, this is where most of the labs have actually invested most of their time and effort: building better versions of Common Crawl for themselves.

Yeah. I'll just name-check some of these if you have commentary, just chime in. GitHub, the source of code; maybe Stack Overflow, even though that's cut off these days. I don't know—do people get code from anywhere else?

I think there are obviously places where you buy code data, but for public code, I think those are the most common. I think some interesting things about those that I just personally find surprising: stars are not a good predictor of whether data is useful for models or not.

Not surprising.

I think the most popular repositories are not necessarily higher quality, at least with respect to whether they improve a model's coding capabilities. I haven't done it, but the StarCoder paper has, and there have been a couple of other papers that have all shown that. That's something that I just consistently found to be a little bit surprising. There's a lot of things that are kind of counterintuitive about data curation.

I haven't read the paper, but did they find anything good that was a sign of a good codebase?

There wasn't anything that was super predictive.

Oh, man.

Honestly, in some ways, some of them were based on length, and some of these simple heuristics actually ended up being better, but nothing was super-discriminative there, which is kind of interesting.

Okay, cool. I'm going to keep going. arXiv, which is, you know, GitHub for papers. Books1, Books2, and obviously Books3. Controversial—I think Anthropic is getting sued over Books3. Yeah, I think a bunch of people are getting sued. Meta has also been sued over Books3. In some sense, can we just look past it? I don't know. It's like Books3 is transformative use—I don't know if you have a view on this.

Well, I think the recent ruling was interesting, although it was an appellate court ruling, so presumably it's going to go to a higher court afterward. But what they ruled was that it's fair use so long as you purchase the book. You can't download Books3 and use it, because that's piracy—you've stolen the books in the first place. But if you bought a copy of all of those books, then you can train on it, and it just counts as fair use, which I think is an interesting—and to me, pretty reasonable—line there.

One fun thing about Books3 is that it also has a lot of not-safe-for-work stuff in it, which is kind of interesting if you actually go and look through it. There should be a Stripe one-click checkout with Books3: just buy Books3, get a warehouse, and get them all shipped there.

I wonder what the cost would be. I'm sure somebody has run the numbers. I'll look it up. I don't know if you can comment on this at all, but in the Meta lawsuit, I remember there was an email thread with some of the research scientists inside Meta talking about Books3, and Zuck was like, “Just do it. This is public, right?”

Yeah, that was public and part of the lawsuits. Any reflections or comments? All I can say is that when I was at Meta, legal stuff around datasets was very challenging and becoming increasingly challenging. There were a number of situations where the only person who could approve things was Zuck because of the scale of the risk, I think, and it definitely made publishing at Meta near the end more challenging around just what we could do with any dataset. Realistically, companies like Meta and Anthropic are big targets for these lawsuits.

Yeah. So my conspiracy theory for what happened to Llama 4 is that the lawyers got to it. The lawyers got to the datasets.

And they had to change what they used.

They couldn't. Yeah, they were just like, hands tied behind their backs when other labs were not, because Meta had an active lawsuit.

I think that's possible. I think probably more of it just has to do with the challenges of continuing to scale and having that be the goal. This is actually a lot of the reason why I got into data and started DatologyAI: the scaling laws always were terrible.

What the scaling laws paper showed was that there was a predictable relationship between performance and compute and data. The Kaplan one, that there's a predictable relationship between performance and compute and data. That's really useful, but it was a bad predictable relationship. Power-law scaling is terrible. It means that every time you 10× your data, you get a diminishing marginal return on performance. You know, this is why you had these prognostications: “Oh, GPT-N is going to cost a trillion dollars to train.” It's because you take that scaling curve and just naively extrapolate it.

I think that's what we've seen to some extent with the failure of the mega-models, right, with GPT-4.5 and Llama 4 and others. I think there's a challenge of just continuing to do that naively, and you have to figure out how to break it. I think there are a number of theories of ways to break it, and I don't think they're mutually exclusive. My bet is that data quality is a massive way to do this.

In many ways, the paper that was foundational for DatologyAI is called Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning, and we were fortunate to get a best paper at NeurIPS. What that paper showed was that if you use your data correctly, you can actually bend the scaling laws themselves. An interesting technical part of this is that what we really care about is how much new information you learn from the next data point. Technically, that's the marginal information gain per data point. Perplexity is another variant of it. There's a duality between them.

It turns out that we were able to prove it for perceptrons, at least, because that's generally what you prove at small scale. This work was led by Ben Sorscher, who was a really fantastic grad student I worked with on this paper. What he showed was that there's a direct duality between power-law scaling and the fact that the marginal information gain per data point also decays as a power law. That's why you get power-law scaling: every successive data point is teaching you less and less and less, and it follows a power law. So you get performance decaying as a power law as well.

If instead you can keep that flat, then you bend the scaling law, and all of a sudden you learn dramatically faster because the amount of information you're learning is not decaying with data set size. Now, that was all in theory—what you could accomplish—and we proposed a couple of metrics that got us one step there. But in many ways, I would actually say that the whole point of DatologyAI is: How do we realize the potential that was shown in that paper? How do we actually make that a reality? Fundamentally, if we want to get scaling to work well, we need to do a better job here.

Are you measuring the quality of these open data sets over time? Are the most recent open data sets better than the older ones at a good rate, or just marginally?

They do get better, but I don't think they're improving relative to the headroom and potential, I would say. Nemotron is actually pretty similar in quality to DCLM. It came out about 6 months later. It has more unique tokens. They made a really big deal about it having more unique tokens, but on average, the quality is pretty straightforward.

When we think about what we're able to accomplish at DatologyAI, we usually think about these three axes I mentioned: train faster, train better, and train smaller. Typically, the first question is: Given a certain baseline data set, how much faster can we achieve the same performance? In how many fewer tokens?

We're now able to get to the same performance as DCLM about 12× faster. In fewer than 10% of the tokens, we can match what you get from training to convergence.

When you say performance, do you mean GPQA, or do you mean loss?

We typically take the accuracy across 15 standard benchmark tasks that are relevant for a given model size: your MMLUs, your ARCs, your RACE, et cetera.

The problem with those is, are you training to the test? I'm sure you know this.

That's something we're super careful about, because it's really easy to overfit to these benchmarks and then end up with models that are really brittle. I think this is something we've seen especially with synthetic data, and synthetic data is a big part of what we do at DatologyAI. We found that it can drive pretty dramatic gains if you do it correctly. There are lots of ways to do synthetic data incorrectly. We've seen a number of models that are trained on a lot of synthetic data and end up doing really well on benchmarks but then don't pass vibe checks, and people don't really use them.

We do a lot to try to prevent this. First and foremost, we keep a held-out set of test sets that we only look at very occasionally. We also evaluate on a whole bunch of other evals, which we then have models evaluated on, to really try to ensure this. But yeah, this is fundamentally how we measure. We look at an average of benchmarks, just trying to think about what's fair and reasonable with respect to what we can do.

That's the first thing we typically look at. Then we look at train better. Of course, under the same compute budget, how much better can you do with a given data set? We're able to beat the best open data sets by anywhere from 4 to 5 points, depending on the specific data set and eval. Four to 5 points on average, and those are absolute points.

We generally find that in order to get that same performance from training longer on the baseline data sets, you'd have to train on those baseline data sets at least 5 to 10 times longer to try to match that performance, because every successive point of accuracy gets harder and harder to achieve.

Finally, train smaller basically says: Given performance held constant, what's the smallest parameter-count model that we can get to outperform it? We can already get models that have fewer than half the parameters, also train faster, and outperform the larger models trained on the uncurated or alternatively curated data sets by a large margin.

So, this is a big roundabout way of getting to the answer of whether the open data sets have kept up with this improvement. With a fairly small team—we're now a team of about 30—most of the results that I've discussed were achieved with a team of under 20, because we've grown quite a bit in the last couple of months, and with not that much compute by common standards.

You know, more than academics, but certainly nowhere close to the frontier labs, we've been able to achieve, I think, pretty dramatic results. I think the reason for this is because there's so much headroom here. We've already been able to get 10x gains. I think there's at least another 100x behind this that still remains to be done.

There's so much stuff that we're not even doing right now that I know makes sense to do, let alone all the things that we are doing that I know we can be doing better, where we're still very suboptimal with respect to how we're doing this. I know that the way we do our synthetic data right now could be much better; the way we do our filtering could be much better; the way we do our model-based filtering, our embedding-based filtering—all these different aspects could be much stronger. So I think there's just so much headroom here.

I think the challenge is that there's not a huge incentive to do this in the open data set community. The labs, which have the biggest incentives, obviously have strong incentives not to share anything with respect to that. So you're left with the Allen Institute, things like Dolma, Hugging Face, et cetera, to make progress there. But I do think that this is such a hard problem that it really demands a whole company that is focused on this.

I think what you see in all the frontier labs is that they have data teams. If you talk to the folks who work on those data teams, what you'll systematically hear is that typically they're under-resourced relative to the gains that they're delivering, and that they're always having to fight for attention. This is just a fundamental thing that I saw at Meta, I saw at DeepMind, and I've heard at all these other places.

It was a big part of why I decided to start DatologyAI instead of doing this within Meta. I had the opportunity to start a data team there that was trying to centralize this. But fundamentally, I think that this is such an important problem that it needs to be the end itself, not just the means to an end, which I think is what you see in many of these big groups.

You need to have a large team of really talented people who are really passionate about looking at the data. There aren't that many people who are that passionate about just focusing on how we build the best possible data sets for model training. I think it's hard to do this as a data team. I think there's a real benefit to being a data company, and that's a lot of why I started DatologyAI.

How do you think the economics—or the open-source data set world—will evolve? You basically have these open-source data sets that are good, but maybe they're not quite as good for making production data systems, and then you have companies like yourselves that are sitting on top of them. Do you think at some point there's going to be some sort of rupture between, “Hey, why are you just taking my open-source data set and making it better in private for people without contributing back?” Do you guys have plans to then open-source other sets? I think there's kind of this open question of whether these things are actually useful in the open or whether you should just do it in private.

Yeah, it's a great question, and one that we've thought a lot about. First off, one thing to note is that while we do work with folks who are just training on open models in general, we really built our product and designed it to be able to work with companies that are training on a combination of open-source and proprietary data.

That proprietary data could just be data they've been collecting as a matter of business for the last decade. Or it could be data that they've sourced from a data annotator or another data provider. Some folks we work with have all 3, right? They're going to use open data, they're going to use data that they've acquired, and then they're going to use data that's been part of their business to begin with. I think a lot of our focus goes there, although of course we are excited about working with lots of folks who are training on more open data sets.

I published for a decade, more than that even. This was very near and dear to my heart, and it's something that we thought a lot about at DatologyAI. I think one of the challenges of building a startup today, especially a startup for which science is a critical component—which, as I mentioned, is one of the things that really attracted me to starting DatologyAI—is this tension. Fundamentally, we have to build a business, and in order to do that, we have to have a moat.

You can think about 3 places where our moat could come from. One is scientific know-how. One is engineering infrastructure and the challenge of just implementing this yourself. Finally, there's a brand moat that you can eventually reach. We're very far from a brand moat at this point in our journey. Eventually, I would love to have a brand moat where, whenever anyone thinks about data and AI, they think of DatologyAI: “That's where I should go first.” I hope that we get to that point, but in the meantime, we have to rely on the other 2 moats—the scientific know-how and the engineering infrastructure.

I think on the open-data side, what we've seen is that the engineering infrastructure definitely can be a moat. Unfortunately, I think that the scientific know-how moat is actually pretty important, and a lot of the evidence that we've seen so far has suggested that it's meaningful.

As an example, many of the customers we talk to, one of the first things they'll ask is, “Hey, how do you compare to the best open-source data set?” So, if we were giving away everything needed to build that best open-source data set, some folks would just go there. I think that's been where our challenge has been.

What we've tried to do, and I think we've done a good job of, and I'm generally happy with the balance we've struck, is to try to give a lot of intuition in the blog posts that we put out as to what we're doing and how it works without necessarily getting to the point of reproducibility. That's much more open than you see most of the big labs be.

If you look at the data section of the Gemini technical report, it basically says, “Data quality was the single most important thing for making a great model.”

One paragraph: “We used algorithms and heuristics.” Great, you know? I think some people were even pointing out recently that there's been a lot more attention on rephrasing as a method for using synthetic data.

Was it the Apple paper?

The Apple paper, the Kimi paper, and a bunch of others have mentioned this. Some folks recently pointed out that, hey, in our blog post from November, we were talking a lot about that. That's something that we do a lot—producing rephrased data. The guy who first came up with rephrasing was one of our first employees.

So, we've improved on that pretty dramatically and taken it to new places, but that's something where I think there would have been an incentive to just not even talk about it at all.

Sorry, just on that, do you feel like this is a great example of what you were talking about? You were talking about it in the data post, and then the Kimi paper comes out with a model, and people are like, “Oh, rephrasing is important.” But you're like, “Hey, I was telling you that before, but I just didn't have a model to show you that it was important.”

Do you think that's still, even in open science, a limiter for people—that if you don't have a model, people don't care? Same with DeepSeek. A lot of the things in the paper were kind of known, but then once you have them applied, people care.

I think that's certainly something that happens, and I think it speaks to the same sort of cultural incentives that we talked about earlier, where people tend to think about this very much as ultimately being a means to an end.

I understand why that is, of course. Ultimately, when we sell better data, we're selling a better model at the end of it—a more cost-effective model. But I think the fact that people don't care about it as much unless they're really smacked in the face with it is both a tragedy and an opportunity.

I would love it if it weren't the case, but given that it is, that's the opportunity we see at DatologyAI to really make an impact here.

This might be a little bit of a tangent, but you mentioned synthetic data and rephrasing, so I figured now is a good time to go into it. I figured that most of the work at DatologyAI is filtering, but I see synthetic data as something slightly different. It's in the general domain of improved data quality, but it's different from filtering.

Yeah.

Am I right to understand synthetic data as rephrasing, or are there other parts to synthetic data in your mind?

Yes, I think there are different parts of synthetic data. There are 2 parts, but let me first actually just comment on the filtering-versus-synthetic-data distinction.

I used to actually use the words “data filtering” or “data pruning.” That paper I mentioned that was at NeurIPS actually has “data pruning” in the title, and that's how you beat scaling laws through data pruning. When I started DatologyAI, I really changed the language to “data curation” over “data pruning” or “data filtering,” and that's because curation is a lot more than just filtering.

Filtering and saying, “Hey, this is a bad data point. We want to get rid of it,” is absolutely an important part of what we do. But it’s also about rebalancing data sets: upsampling certain data distributions and downsampling others. That might not mean filtering; it might just be changing the weight with which you take it.

The order in which you present data can be really impactful. Curricula—and we now have seen this with discrete curricula for multiphase training and things like that—that’s not filtering. The way you batch the data can be an important factor. Synthetic data can be an important factor. The way you mix sources—all of these sorts of things go beyond just filtering.

Filtering is a very important part of what we do, and it will always be something that we care a lot about. But it’s much more than that. Okay, so now to the question about synthetic data. I think, at a high level, there are 2 approaches to synthetic data, and we have focused more on one of them—the rephrasing one—although I think there’s opportunity in the other one.

The first approach is to create new data where the knowledge in that data is largely coming from the model that’s generating the synthetic data.

Oh, that’s distillation.

It’s a version of distillation. I think this version of synthetic data could be construed as distillation in disguise. I think it’s a very clear version of this. When you think about the criticisms around model collapse and stuff like that, I think they largely apply to this version, where you have net-new data creation coming out of these models.

So that’s one—I’ll slip one in there. There’s also model steganography, where you can sort of hide preferences in a model and distill it down.

Absolutely. Now we’ve seen the recent Anthropic Othello stuff around that.

If people search for “Anthropic Othello,” you’ll see it.

Yeah, exactly.

The other way is this rephrasing or rewriting approach. The information in the data is actually coming from the data that you’re conditioning the rephrasing on in the first place. All the model is doing is reformatting the data or presenting it in a new way that may be easier for a model to learn.

Yeah, cleaning, right?

It’s cleaning it in some way. It could be cleaning it. It could be making the information more accessible. It could be putting that information in a format that’s more representative of what the model is going to be faced with downstream.

One of the things that definitely happens with synthetic data is that we’re bringing more post-training-like data into pretraining. In general, one of my beliefs is that most of what we do in post-training is better done in pretraining and mid-training, and earlier on in training in general.

It’s just the scale. You don’t have that scale until now.

It’s just that—yeah, exactly. I think if you assume this paradigm where pretraining is incredibly expensive and something that you can only do very, very rarely, and then post-training is cheap, it makes sense. But as soon as you break that assumption—and I think DeepSeek showed that already—you can get a frontier model for a marginal cost of a couple million dollars.

That’s gone down since then because we’ve gotten better at it, and compute has come down in price since then. I believe that getting to a frontier model should cost $1 million or less for most organizations, at least in a specialized domain. When you think about what enterprises need, that’s generally what they need. They don’t need a model that can do everything. They need a model that can do a constrained set of tasks to very high accuracy, for as low an inference cost as possible.

I think that will be under $1 million very, very soon, and that changes a lot of these dynamics. But going back to the synthetic data question and these 2 different types, I think one is toward this net-new creation. That’s where you have a lot of risk. That’s where you get the model collapse concerns.

I train a model—a generative model—on a given data distribution. It overfits the modes and underfits the tails. So then, if I have it generate a bunch of data, it’s going to be more mode and less tail. Then I do that a bunch of times, and eventually I get a spike. I get a delta function.

Only mode.

Only mode, exactly. That makes sense. I understand why that happens. I will note that if you filter the data after each point, that’s now information injection, and that can break all of this. I think it can prevent model collapse.

Which a little bit is what RL is.

Which is a little bit what RL is. I think you can absolutely view it that way. A lot of the work that has suggested that RL is really just eliciting the capabilities of pretrained models—random rewards or a single example, and then it’s just changing the distribution; it’s like aligning to the distribution the model has in the first place—is very much in line with that way of thinking about it.

You’re distilling from a perfect model, which is the environment, the verifier, or whatever, and then you’re distilling that into the model. So, yeah, it’s amazing. It’s beautiful.

The cool thing about rewriting is that the model that’s doing the rephrasing just needs to know how to rephrase. It doesn’t need to know anything about the content itself. It doesn’t need to understand it. That means you can use a pretty weak model to do the rephrasing and have it generalize and generate data that can teach a model that’s much better than the model doing the rephrasing.

With this distillation in disguise, I’m generally quite skeptical that you can get a model that will be better than the teacher generating the synthetic data when you do this sort of net-new data creation. It’s possible you could through some sort of heavy rejection sampling on the big model, because you’re effectively inserting new information when you say which of the synthetic outputs is good or bad, right? There’s some new supervision coming in there.

We’ll actually have a blog post coming out in the next week or 2 about our synthetic data generation, which we call Beyond Web. We’ll have some cool scientific experiments in there, too, as part of trying to figure out this balance where we can share some of the science but also do so in a way that’s sustainable for our business.

One of the things we show there is that by doing this, you can get a model to do much, much better than if you had trained on all the raw tokens in the first place. By doing this rephrasing, you can effectively break this data wall and get models that are better than either of the models that generated the data.

With rephrasing, I think this is super possible because most of the information is coming from the data. It’s not coming from the model itself.

A couple of follow-ups on that—just things I’ve always wondered. Are textbooks all you need?

No, they are not all you need. I think textbooks are great, and there’s a lot of really great content in high-quality data points like that. But obviously, textbooks are also a very narrow data distribution.

If there’s only one thing that you should take away from this entire interview about what is good for data quality, it’s diversity. In many ways, I used to do all this work on out-of-distribution generalization. We had all of these very careful studies where we would say, “Okay, let’s make this corner of the data distribution, then leave this held out, where it’s never seen this combination of things, and let’s see if it can generalize.”

Then LLMs and the modern way of training models came along and said, “Hey, what if nothing were out of distribution? What if we just made it so that we train on everything and everything’s now in distribution?”

And by the way, that is in line with AGI, right? So you might as well—

And that’s basically what we’ve done, and it’s worked. It’s worked shockingly well, way beyond what anyone—or most people, I think—would have expected. I certainly was shocked by it. I made a strong bet that there was no way you could get compositionality just from scaling.

Well, you can. Turns out.

It does work when you get big enough. What I was really referencing was the Microsoft Phi papers—Phi-1, Phi-2, Phi-3, and Phi-4. A lot of them do the rephrasing or rewriting in textbook format, and I feel like that’s a little bit of cargo culting: “Oh, just because you write like Wikipedia or write like textbooks, the models learn better.” That’s not automatically proven to be the case.

I think that’s also probably part of the reason why you see a big difference between the benchmark scores of those models and their real-world use.

They went to too narrow a distribution. I think this is the problem with synthetic data fundamentally: you’re always going to have some bias here. You can do a lot to make it more diverse, and we’ve put a lot of effort into finding ways to do that. For example, we rephrase into many, many different styles and formats. That’s really important to get stuff that’s good.

But I think this is the risk: you go on way too narrow a distribution, and models are always going to be fairly peaky with their output distribution. That actually results in reducing diversity.

That said, I will say that there is a takeaway from “Textbooks Are All You Need” that I think is correct: repeating higher-quality tokens is almost always better than seeing net-new, lower-quality tokens. Epoching over higher-quality data is almost always better than getting the same amount of new data of unknown or average quality. Average in this case means what you get from an internet dump or something like that, or even a reasonably filtered internet dump.

Yeah, it’s always better. The modification I made, or the study I would want to commission out of that, is: instead of having another epoch on high-quality data, if you found high-quality data, good—go and paraphrase it, and then train on that. Maybe that will get additional gains. I don’t think I’ve seen any papers that have been to that effect.

The Phi paper actually had an experiment to that effect, where they tried adding multiple epochs and looked at how many rephrasings they did of each of them. They had some results that were to that effect.

Amazing. The other question was more on curriculum. Curriculum learning had a bad reputation for a while. How come it’s back? What’s changed?

A bunch of things. This is really interesting because when I was initially deciding whether to start DatologyAI, and raising and talking to various initial recruits and so on—it was mid-2023. At the time, I was saying that curricula were going to be a really important aspect, and a lot of people were basically just like, “No, curricula don’t work. We tried this a bunch of times, and curricula don’t work.”

Curricula are one of these ideas that I think always had to work, in the sense that it just made too much sense. There are a number of these things where it’s like, it might be hard to figure out how to make them work well, but they always had to work. There’s actually a really cool paper from Stanford that had a nice way of conceptualizing this. Imagine a graph where each of the nodes is a different concept or idea that you want the model to understand, and the edges represent the dependencies between those concepts. If concept A helps you learn concept B, there would be an edge from concept A to concept B.

Now imagine this graph of all the concepts in the world and all the different edges between them—a huge graph. If that graph is empty, it would mean that nothing is helpful for learning anything else. In that case, curricula would not make any sense; you should just randomly order things. If that graph were complete, so that there were edges of equivalent weight between every pair of nodes, then similarly, it would mean that everything is equally useful for learning everything else. Curricula would not work, and you shouldn’t use them.

For any other graph besides those 2 graphs, curricula make sense. It’s pretty obvious that neither of those is a graph of the actual world we live in. Clearly, the world does have dependencies—some very obvious, like the fact that it would be hard for me to do division and multiplication if I didn’t understand addition and subtraction, and some much more vague. I have always believed that this has to work.

The challenge has largely been that if you’re fully saturating your data, there’s really no advantage to a curriculum unless you wouldn’t be able to learn it otherwise. Generally, I think the idea behind curricula is that they make you much more efficient. But in the supervised learning world, we were fully saturating these datasets. Maybe a curriculum would get you there faster, but that wasn’t the bottleneck or limiting factor.

There wasn’t a clear incentive to actually do these hard experiments to try to figure out how to make a good curriculum. Who cares if I can get you to ImageNet performance in 80 epochs instead of 160 epochs? That’s nice, but it’s not a big deal in the first place.

Now we’re in this totally different world where, all of a sudden, our models are underfitting the data. This is super important, and getting a curriculum right could literally make the difference between spending 10 times as much on model training—potentially hundreds of millions of dollars.

All of a sudden, curricula make a ton of sense. I think that’s why the problem didn’t really make sense to put a lot of effort into previously. Now we’ve seen pretty clearly with discrete curricula that this makes a big impact. Largely, what we talk about when we say “mid-training” is really just a later phase of your discrete curriculum, which is another way of thinking about it.

You could even think of post-training as part of a curriculum. One of the things that I’m really excited about is that we’ve mostly focused on pre-training and mid-training at DatologyAI so far, but one of the most consistent asks from every one of our customers has been, “Can you do more on post-training? Can you also help us curate the post-training data?”

So we’re starting to invest pretty heavily there. One of the things I’m really excited about is viewing this whole thing—from pre-training to mid-training to post-training—holistically, as a single process. Then we can ask questions like, “How do we optimize our pre-training data to make post-training more effective?”

These are really exciting questions, and you don’t see them happen even at the big labs because they have entirely separate teams. There’s a pre-training team, a mid-training team, and a post-training team. The mid-training team is a customer of the pre-training team, and the post-training team is a customer of the mid-training and pre-training teams. It’s quite hard to actually have signals propagate through all of these, so I think this is a really exciting area.

I’ll push you a bit on this. I think a popular view is that post-training is the elicitation of capabilities that you already train in pre-training. What dependencies can you have that feed back into pre-training?

I’m inclined to agree with that view, and I think that view would lead very strongly to the fact that you should be trying to optimize your pre-training data to make post-training processes more effective. You should try to figure out: How do I optimize my pre-training data so that the slope of the test-time compute curve, or the slope of the reinforcement learning curve, is as steep as possible?

Alternatively, how do I optimize my pre-training data so that the slope of the jailbreaking curve is as shallow as possible? Fundamentally, I don’t think alignment and post-training make sense as a long-term solution. If you can easily align a model through post-training, you can easily misalign a model through post-training. If it’s easy to put it in, it’s easy to take it out. If it’s really hard to put it in, it’s really hard to take it out. That’s just a truism of models.

If you do alignment during pre-training, you’ll end up with models that are, I think, largely impossible to misalign without putting a massive amount of data into them. I think there are a lot of benefits to that. I think we’ve also seen evidence for this, looking at the difference between Llama and Qwen with respect to their ability to be post-trained. It’s much easier to do reinforcement learning on Qwen than it is to do it on Llama. Likely, that has to do with the fact that Qwen put a lot of synthetic reasoning traces into its training data.

Even with wrong examples?

Yeah, but even with wrong examples, that’s where it’s still there, which is wild. I think that pretty clearly shows that it’s the base model that’s doing it; it’s not the rewards you’re giving. If you give random rewards and the model still learns, it’s probably not the reward signal that’s doing it.

That’s cool. I’m curious about customer usage. How many people are doing post-training? Obviously, nobody today because you don’t have it, but when people come to you, are they mostly looking to do post-training on open models, on OpenAI models, or what do they ask for?

We usually work with folks who are either training their own models from scratch or doing continued pre-training on an open model with a bunch of domain-specific data that’s unique to their use cases and their businesses. We typically focus on folks who are doing training with a significant cost. Typically, that means at least a couple of tens of billions of tokens, oftentimes more. The standard small-scale post-training fine-tuning case is not something we focus on as much.

That said, this has been a question that a lot of people have asked us consistently: “Who’s actually training their own models? Why don’t I just rely on the open models?” I think there are a number of reasons why we see people do this.

First off, sovereign AI has been a pretty big area where we’ve seen a lot of demand. Lots of countries want to have models that they own, which are unique to their language and culture. That requires them to have really good data curation in order to do this effectively.

Just to double-click, countries owning models isn’t actually a thing that I know about. I’m from Singapore. We have the SEA-LION model, but it’s not owned by a country, and I can’t name any other country that owns a model.

Yeah, I think that’s actually correct.

It’s largely what you see right now: public-private partnerships where governments are making pretty large grants.

TII in the UAE is the closest.

Yeah, I think you have those. I think you also have these places where the funding is the country, and it becomes a little unclear where it comes from. But, yeah, I think usually what you see is that countries are doing big grants to private companies or public-private partnerships to go and build that sort of thing.

That’s a big thing. I think we’ve seen a lot of larger enterprises that have a lot of their own data that want to do this. And when you think about this, ultimately what we see is that, across those 3 value props—train faster, train better, train smaller—which matters and when?

Train faster, in principle, is the easiest one to compute. You know, I say, “Okay, this model would have cost you $10 million to train. I get it to you for $1 million or for $800,000 or whatever, right? Great. I saved you a ton of money.” In practice, though, nobody wants to train a $10 million model for $1 million. They want you to already have the model.

They already have that. They want to train a $100 million model for $10 million.

They want to train better. So train faster usually doesn’t matter so much from the perspective of, “Hey, this model is now a lot cheaper.” It does matter a lot more from the perspective of being able to iterate much faster, right? Because when you think of the workflow of most ML engineers, you start training, and you sit on your hands until the training finishes. You find something else to do, but largely you’re waiting, and your iteration is bounded by how long that takes.

If you can take something from 10 days for a model to finish training to being overnight, now your existing team is way more productive and can do far more iterations and stuff like that. So that’s where we usually see that matter the most.

Most people care the most about train better, right? I can get a better model for the same compute, and we can absolutely deliver that through data. Data is effectively a compute multiplier, right? Because all models are underfitting their data sets. If you can make your model more data-efficient, you effectively make your compute more valuable.

If you think about compute as, “I inject a certain number of dollars and I get a certain performance back,” if I use better data, then I will get more performance back per dollar invested, and now my compute is more valuable. So that’s where train better, I think, tends to be the most meaningful thing.

Interestingly, for the companies that are most advanced on their AI transformation journey, train smaller is the one that I think actually means the most. Because when you think about the total cost of ownership of these models, it’s going to be very, very heavily weighted toward inference. It’s all inference.

You think about a company that’s spending, say, $50 million a year on inference, which in the scheme of things is not very much, right? If you deploy a model that’s twice as big as it needs to be, that’s going to cost you $25 million in year 1. The cost to train a model that has fewer than half the parameters but is just as good or even better at your particular use cases is, say, $2 million or $3 million. That’s a no-brainer if you can do it easily, right? If it’s really hard, then you’re never going to do that. But if you can do it easily and you can get it right on the first try, that’s a no-brainer.

And then $50 million a year is not going to be very much, right? We know that all of these products have a tiny, tiny fraction of what their eventual user bases will be, right? We’re still very much in the first inning here. Everyone that listens to this podcast is using AI nonstop, but the rest of the world is not yet. So the inference costs are going to skyrocket with these models.

If you use a general-purpose model and then constrain it to say, “Hey, this model knows about everything, but now only do this one thing,” that model is going to have a ton of parameters that do not need to be there, and those are going to massively increase the cost of serving that model.

So I think that when you think about the use case of an enterprise where they need a model that’s an inch wide and a mile deep—it can do a small handful of things, but it can do that really, really effectively to five nines of reliability, and it can do it for as low a cost as possible—the economics make it so that it really makes a lot of sense to do this yourself if you can do it easily.

The way we think about it is that there were 2 big barriers. First, you have to get training right, and then you’ve got to get data right. On the training side, I think 3 years ago this was super hard, right? But MosaicML was the first one to really recognize that there is a huge opportunity in making this easy.

Now this has largely been commoditized by things like SageMaker and Together AI and lots of different folks to help you on the training side. But on the data side, the barrier is just as high as ever. In many ways, that’s our mission at DatologyAI: How do we bring that barrier down so that anyone who wants to train a model can do so with the best-quality data on their first try?

They don’t have to go and spend 40 years in the desert. They don’t have to get it wrong 100 times first, which is what will happen if you don’t have this experience. Instead, on the first shot, they get a really great model.

Yeah, just a follow-up question on train smaller. I fully agree, and I think that this is something a lot of people are investing in. You are primarily doing work on the data side—data pruning, which maybe is a bad word now, data curation, whatever.

I think a lot of people—you know, Jonathan Frankle was on the podcast very early on—but a lot of people were betting on pruning the model itself. You have a working model at scale, and you just lop off anything below a certain epsilon. Is that confirmed to just be dead?

So it’s funny: Jonathan actually interned with me when I was at Meta, and we worked on this stuff together. He had the Lottery Ticket Hypothesis, which is a really beautiful paper.

Which he now completely disowns.

Which he largely disowns. I had this whole idea when Jonathan and I worked together that we wanted to create a lottery-ticket initialization. It would just be an initialization you’d sample from for initializing the weights that would then be one of these perfect winning-ticket initializations.

We actually found out that the problem was that the lottery ticket was actually data-dependent, and that was where the fundamental problem came: As soon as you change the data distribution a little bit, the winning tickets changed in a really big way.

I don’t think pruning is dead. Parameter pruning still absolutely has a place, but I think certainly we found it challenging to really realize the potential of it. I think one of the big tricks with parameter pruning, just to be clear, was that unstructured pruning—when you would prune weights randomly, so you view all the weights as a smorgasbord and just prune them randomly—worked really well, and you could remove massive quantities of the weights with unstructured pruning.

The problem is that unstructured pruning doesn’t really give you a clear compute advantage because you need to have a sparse matrix now to reflect this, and there’s a pretty huge overhead of sparse matrix multiplies. GPUs are not very good at sparse matrix multiplies. There’s some support for them now.

There are some hardware optimizations for that.

There’s some hardware, and people have talked about building ASICs that would be really good at unstructured pruning, but I don’t think I’ve seen one that works super well. I think if someone did make something that worked really well for models that were pruned in an unstructured way, that could be effective.

Structured pruning, in which you just remove a unit—you just remove a neuron—is really easy to make faster on a GPU, but that just doesn’t work nearly as well. So I think there’s still potential here. I don’t think it’s the panacea that I, and I think many others, had hoped.

That said, I think one thing that’s cool about using better data to train smaller models is that it’s complementary with any other approaches for optimizing inference. I think pruning and quantization obviously still have a role to play in helping inference go faster, and that would stack on top of anything that we’re doing, which I think is kind of cool.

Yeah. One grand challenge—a golden question—that would be very valuable for you, and just in general, is this idea of what is the smallest possible model for a given capability? Do you have any insights on that?

I did a podcast with Jack Morris, who’s out of Cornell, and I think there’s some information limit. I think he had some answer like 8 bits per parameter or something like that. I forget what the conclusion was.

Yeah, I’m not sure that I would put out a specific number, but I would definitely say far, far smaller than what our current models are trained to be, right? We are nowhere close to this.

I am generally of the belief that most of the models that the vast majority of people will be using in, say, 3 years will be single-digit B or smaller. I think we’ve seen this very clearly. You look at just the Llama series—if you want to exclude Llama 4, do so.

But Llama 1 through 3, you can see pretty clearly that the 7B variant from the N+1 generation is pretty close to the 70B variant from the prior generation. It may not be quite there, but there’s still a very clear trend here. We’re seeing this with the Qwen models, too. You look at some of these small Qwen models, and they’re incredibly performant relative to what state of the art was a year ago.

I think it’s pretty clear that these models are way too big. I personally would bet against the next frontier being trillion-parameter models. Rather, I think we’re going to really optimize the inference cost. I think test-time compute as a paradigm also pushes you toward smaller models, right? Because if your cost of solving a problem is the cost of inference times the number of thinking steps, and you have to do a lot of thinking steps, minimizing the cost of inference is really important. Anything we can do to make the inference model that’s doing the one step of thinking a lot faster enables test-time compute to be a lot more effective.

There’s another version of this, which is the Andrej Karpathy cognitive-core concept: a model that doesn’t know anything but can use tools a lot to figure things out. Another information-theoretic limit that would be very helpful to figure out is, what is the minimal viable model for that? Something like 0 on GPQA and 100 on BrowseComp. I really like that idea, and I think it’s very possible to do that, because knowledge storage takes a lot of capacity and a lot of parameters. You don’t need it.

One of the first papers I ever wrote was about showing that when you train models on randomized labels—because this was a common test to do—the one way you could prove that a model was memorizing was to randomize all the labels. There’s no actual true association, so it would have to memorize them. Models could do this really well. There was an ICLR Best Paper from 2017 that showed this, and people were really surprised that models could memorize all of ImageNet. At the time, that was crazy: “Wait, they could just memorize a million labels? That’s wild.” Of course, models can memorize the whole internet now.

What we found was that if you deleted units from a model that had memorized, it would be really damaging to that model. But for a model that had actually learned a generalizing solution, you could delete a lot of units and it would be pretty robust to that. It’s a very clear demonstration of exactly this concept: the more you memorize, the more capacity you’re using.

Dropout regularization.

There are a lot of dualities to dropout. I think there’s an argument to be made that dropout helps prevent memorization and helps learn more generalizable solutions, and that’s part of why it worked well. I think it’s very possible to do this, and I think we’re wasting a ton of capacity in these models on knowledge that is totally unnecessary for them to have.

Before we wrap, just because we started with the RC models and then never talked about them, I think the most interesting thing to me was that they started with 23 trillion tokens of data and then you helped them get down to 6.6 trillion. Any learnings from that? This is a 4.5B model, which is on par with Gemma 4B and a little worse than Qwen 3, but roughly the same. Any learnings there—experiences or things that other models should adopt?

For that one, we started with a combination of DCLM, Nemotron, and FineWeb. We basically just concatenated them all together. There were about 25 trillion tokens combined, and we produced 7 trillion out of that.

What was exciting to us was seeing the speed at which the model learned. It was beating Gemma pretty consistently before the 1 trillion mark, which was pretty cool to see. I think it really highlighted how higher-quality data can get you much better performance much more quickly.

As one of our first real customer engagements, it was exciting for us. RC is the first customer that we're talking about and being public about since starting the company. More generally, it’s a good showcase of the fact that combining all of these different techniques can give you a really big gain. That’s one of the things we’ve been saying, but it’s nice to have a real demonstration of it.

This wasn’t something where synthetic data took us there, or filtering took us there. It was really about thinking about how to combine all of these techniques. One of the things we’ve consistently found is that when you take these different techniques and try to make them work together, they don’t generally work together. You can make them work together, but it’s quite hard to do so. What was exciting for us was showing that it’s possible.

People tend to think that you can’t stack curation. The fact that we started with some of the best-curated open data sets and were able to make them dramatically better is a pretty good indication that there’s still a ton of headroom left here. We didn’t need to go to Common Crawl to get those tokens. We are, of course, doing work on that, and we think there’s a lot we can do to improve there. But just starting from that, we’re now making bigger data sets from that corpus. I think we can get up to 15 trillion while still having pretty identical quality, which is pretty neat.

The other thing is that it really stacks. If we apply our curation on top of, say, DCLM, and then apply it on top of FineWeb, the gap between FineWeb and DCLM is maintained in the gap between DatologyAI-curated DCLM and DatologyAI-curated FineWeb. They both get a lot better, but DatologyAI-curated DCLM is still better than DatologyAI-curated FineWeb.

There’s so much still left to do here. We’re just scratching the surface. We’re pretty excited about what these results showed. We already have better data sets than what RC trained on, because that model was largely trained in May, and we’re pretty excited about all the next trainings we’ll have that go even bigger.

I have a couple more lightning-round questions. Based on your customer conversations, what data does everyone want? What data does everyone want but is really hard to get?

I think expert data is the obvious thing.

Domain expertise.

That said, I would also note that most people don’t know what data they actually should be getting. They just show up with whatever they have. Something we’ve found shockingly frequently is that we talk to folks who have been planning for a really expensive training run—a training run costing millions and millions of dollars. They’ve been thinking about the architecture they’re going to use and all this other stuff, and then they reach out to us: “Hey, we realize we need a good data set, and we’re planning to kick off training in 2 weeks. Can you help us?”

A lot of it is, “You probably should be thinking about your data set before all the other things.” If anything, that’s actually the most important thing. The most surprising thing is maybe how often people don’t even have a conception of what good data is. Oftentimes, what people think is good data isn’t. That goes to the DCLM point that we mentioned in the past. It’s very counterintuitive and really hard for humans to identify what’s high-quality and what’s low-quality.

This is a little bit of a recruiting question. If somebody had an answer, they should join DatologyAI immediately.

The first thing I would say is that if you’re one of these people who keeps finding yourself staring at the data, going into the data set—if you can tell me what your favorite and least favorite C4 example is—you belong here. You should come join us and a bunch of other nerds who love doing that exact same thing.

In many ways, that’s the single biggest predictor of whether someone is going to be really happy at the company: how much you look at the data in your own work. You’d be surprised by how many really talented researchers don’t do it very often. They just view it as a given. I think that’s been pretty surprising across the board.

That said, there are so many questions that I’m super excited about on the science side. I mentioned the interactions between pre-training and post-training. That’s definitely one that we’re really excited about. One of the things we care a lot about is making it so that our product and curation automatically adapt to novel data distributions.

Right? If you have this, it has to be fully automated. We didn't talk about this too much, but one of our challenges often is that if we're working with an enterprise that has a lot of proprietary data, they obviously don't want to give that to us. So we bring our curation to their data, but this means that it has to adapt automatically. We have pretty limited access to go and look at that data.

So that's actually a really hairy and interesting out-of-distribution generalization problem. But it's also really important because there's no golden curation. Curation is only optimal with respect to a given set of downstream use cases or tasks, right? We need to be able to define, based on the model needing to be able to do XYZ, how we should use that information to adjust the curation that we do, to make sure that we're giving the data that's most relevant for solving task XYZ. That needs to happen automatically.

We have a number of ways that we can do that across a number of our techniques, but that's a very broad and general question that we want to apply to every part of our pipeline. The way we do synthetic data should differ based on the downstream use cases. The way we're doing every different part—filtering, and so on—is going to change based on that. So that's another question that we're really excited about.

Fundamentally, we're trying to answer this question of how you value data with respect to a target. When I think of DatologyAI and our core competency, I think every company needs to have an unfair advantage, or some core competency that they do better than anyone else. For us at DatologyAI, I want us to be—and I think we already are—the best in the world at valuing data with respect to a downstream use case.

In many ways, I think that's kind of the NP-complete problem of AI. If you can do that, you can kind of do anything. That's the thing that we're really focused on. Of course, curation is the very obvious, direct application of that core competency, but when we think about the vision for the company in the long term, it's about asking what all the other ways are that we can operationalize that same core skill set. I think there are tons of really interesting things you can do there. That's the fundamental question that we really want to answer, and there are tons of different entry points to that question.

Awesome. And just to wrap on some gossip, let's talk about Meta and superintelligence. In the notes, when you talk about science moat and whatnot, you raised a lot of money from very prominent people. You have Yann LeCun as one of your investors, along with Geoffrey Hinton and Jeff Dean. So when Ari says that they have a science moat, believe him.

Maybe since you have Yann as an investor, this is more of a touchy question, but what do you make of the whole Meta superintelligence team? Yann was also on LinkedIn saying, “I'm actually working at FAIR. We're focused on the next generation of AI, not on this current generation, so my role is the same.” But maybe people might say, “Then why didn't you do the current generation 10 years ago?”

What do you make of the whole change, and whether or not you think this is an interesting direction for Meta, especially given the large platform and user base that they have?

Well, first, with respect to Yann specifically, Yann's an incredibly talented scientist, of course, but I think his preference has always been to do science rather than to run an organization. I think he ran FAIR organizationally for a year or 2 right at the very beginning, but pretty quickly he handed that off to other people. When I was there, it was Joelle Pineau and Antoine Bordes, and then Joelle for most of it, who really were running FAIR. She was an incredible leader. I really respect her deeply, and I couldn't have asked for a better kind of advocate for science within FAIR.

When she left, people were saying, “This is the end of FAIR.”

I hope that's not true, but I also had that concern. I think Yann always really wanted to do the science himself. For much of the time I was at FAIR, he generally operated with his own group of a couple of postdocs and visiting scientists, and then he'd have a couple of students through NYU. He would do his own research there.

So I don't think he was ever—or at least not since the beginning—in a role where he was defining AI strategy for Meta. I don't think that's the role he wanted at any point. I think he really wanted to be doing that research, so I don't think his role is probably changing very significantly, in the sense that he wasn't doing that previously, and I don't think it was what he wanted to do.

I think one thing that's pretty cool about it, obviously, is that it showcases the importance of data: Meta is willing to spend quite this much on Scale—not an acquisition, but the kind of deal we're seeing today.

Alex Wang is not going to underrate data, let's put it that way.

Yes, he's certainly not going to underrate the importance of data. I do think that this is an area where the stuff we've done is quite different from what we've seen from the data annotators, which have been more focused on collecting the data versus actually optimizing and curating it. I think there's quite a bit you can do on top of those things. So I think it definitely draws some attention to that.

I will also just say generally that when Zuck makes a very big bet, it's not proven wise to bet against him. Historically, that's been the case, and most of the big bets have panned out. I think the one that's still really up in the air is the metaverse, but I would actually argue that I think that's going to end up paying off in the long run.

I think the Ray-Ban glasses are pretty darn cool, and a lot of the foundations of what was in Reality Labs will go into those. Also, FAIR was part of Reality Labs for about a year and a half after one reorganization. Initially, FAIR wasn't, and then it got reorganized into Reality Labs.

So I think when I left, FAIR was officially part of Reality Labs, if I recall correctly, and there was at least a 1.5- to 2-year period where that was the case. Some of the AI investment that laid the foundations came out of that metaverse investment in the first place.

That said, I think we talk about data as being a compute multiplier all the time. Talent, I think, obviously, is a compute multiplier, and given the amounts that they're spending on compute, I think you can make a good argument as to why spending a crazy amount on talent is also worth it. So I'm excited to see what they do. I hope that they put a lot of focus on data—

—and become customers.

Yes.

Awesome. Well, thank you so much for chatting and coming by, and insisting on doing it in person because you're actually very charismatic in person. I'm glad you did this.

Oh, thank you very much. Thanks for having me. It was a joy to chat in real life.

Awesome. Cool.